Dependency Hell: The Linux Experience

This week was one of the most frustrating that I have had since I first learned to program a computer at the age of 17.

The reason was that this week I discovered the true nature of the Linux Experience: Dependency Hell.

I had heard of this before but did not truly appreciate the nature of this beast.

In short, unlike Windows, which uses compiled executable programs, Linux largely expects users to download source code and compie their own binaries. This is likely due to the sheer number of Linux distros out there, releasing a binary for all of them would be impractical. The problem with this is Linux make extensive use of common libraries, or dependencies, that are shared between various programs, and are, unfortunately, often incompatible.

So, you could have one program which requires at least version 2.0 of a library, and another program that will not run on anything later than 1.0. If you want both programs to run, you are in “Dependency hell”.

Unfortunately, in the AI world, there are a LOT of common libraries (CUDA, PyTorch, Torchvision/audio, xFormers, etc), leading to the issues that I had this week.

It should be noted before I continue that when I built my Linux-Powered AI rig I went into it with significant optimism, I loved the idea of Linux. I can’t stand the bloatware on windows, the tracking, the constant forced updates and restarts, the ads, and the lack of control over the operating system. When I bought my current windows 11 laptop (They didn’t have any with windows 10) I clicked on the start button and saw a new article about Prince Charles’ Coronation. It’s bad enough that I am bombarded with ads on the internet, I do NOT want to see ads when I click on the start menu of my own computer!

The rule is EITHER paid OR ad supported, not both! If your OS costs money, you can’t have ads too! Microsoft haven’t learned this.

I even installed Linux on my current laptop briefly, but the drivers for the trackpad didn’t work, so I reluctantly installed Win11 again.

However, the difficulty that I have had this week just getting programs to run has seriously soured me on the Linux experience.

The issues all started when I tried to install xTTS (A Text to speech extension) for SillyTavern.

I did a Pip Install, I had to update some packages, but everything seemed ok.

Then I realised that when I updated the dependences to allow xTTS to work, I had broken several of my other programs, including comfyUI!

Reinstalling those programs, of course, broke xTTS, because the versions of the software used by each were incompatible.

I eventually learned about Conda and Virtual environments, which solved some of my issues, but not all (Never in my time using windows have I had to install a virtual environment just to get a program to run!).

However, even with Conda/Venv, I was STILL having serious problems just fixing my existing programs, let alone installing new ones. At one point, at least 3-4 of my main AI programs were in a broken state due to dependencies, and I had completely lost faith in the system. It got to the point where I was genuinely ecstatic just to start up a program and have it actually work!

What I learned eventually is that when you run pip install inside a conda environment it installs the packages to the location where pip was installed, which is your systems pip directory. IE, you are installing the packages globally, potentially overwriting other packages.

Using conda install fixes this problem, but most of the packages I was using didn’t have a conda version, so I was forced to spend many hours pip installing dozens of packages to attempt to find a solution.

At one point, I restarted my system, and found that I couldn’t log into the gui! Apparently my hard drive was full, which, for some reason, prevented login?

It turns out that I had used up 75 GIGABYTES of storage page just downloading difference packages! CUDA, for example, is about 2 GB’s. Torch vision is about 1 GB, so every time I do a pip install that’s another 2 GB’s of data.

I purged the pip cache and moved some data from the main drive to the backup drive (Luckily remote desktop still worked, even though the local login didn’t?) and I was able to log in again, but I still had the dependency issues to deal with.

Eventually, after several days of misery, I discovered that you can install pip locally to a conda environment, so that if you then run pip install it will install the packages to the local version of pip, not the system version, allowing you to use pip install without overwriting your system packages.

So far, this seems to work.

Without bragging, I have nearly 25 years experience working with computers. I am new to Linux, yes, but I have extensive experience in programming, the command line, libraries, and the complexities of operating systems. If it takes me three days just to get a program running without breaking another program, then there is a problem with the operating system. It simply should not be that difficult.

Just to clarify, all I am trying to do here is install a program, that’s all. I’m not trying to write code, just run it!

ALL of the above could have been accomplished on windows by just double-clicking on an exe!

There was one program (A web based gui for training voices for Realtime Voice Conversion, or RVC) that I spend several hours trying to resolve the dependencies for. Everything I pip installed something, it broke something else. I had to reinstall comfyui at least seven or either times!

Eventually, I gave up and downloaded a fork of the program that worked on windows. I ran into ONE error (My firewall was blocking the clone of the git repo. I simply disabled it temporarily until the download was completed) and the program was up and running in less than three minutes.

This has consistently been my experience: Fighting with Linux to resolve dependencies that would require just a few clicks on windows.

I know that I have extensive experience on Windows which I lack in Linux, but we are just talking about basic functionality here, you shouldn’t need to be a Linux guru just to get a program to run!

The other discovery that I made is that the AI world doesn’t appear to be quite as Linux-centric as I initially thought. Before I built my AI rig I read that the vast majority of AI-related programs and libraries are designed for Linux, and I would have difficulty running them on windows.

In actual fact, in my experience, the only thing that I have found that was easier on Linux than windows was installing Sage-Attention 2. Sage attention 2 can be installed with just a few lines of code on Linux, it is, apparently, a lot harder on Windows.

Apparently Deepspeed is not properly supported on Windows either.

Everything else either had a windows version, or there was a similar program that worked on windows. In some cases in fact I found more easily accessible information for windows than Linux (Such as Lora training for Hunyuan Video).

It seems to me that a lot of the people singing the praises of Linux are people who are card-carrying members of the Linux club, and either fail to see it’s substantial shortcomings, or obstinately refuse to acknowledge them.

Windows isn’t perfect, I mentioned the issues with bloatware, privacy, etc, but honestly, after dealing with Linux “dependency hell” for the last 3-4 days, I will happily allow Bill Gates to come to my house personally and watch me take a shower.

Running a debloat script on windows 11 is far, far, easier than dealing with Linux’s issues.

Linux is for people who want to work on Linux, not for people who want to work on anything else.

Windows, despite its issues, just works, and when you’re a professional who values their time, you want something that works.

I cannot understand how anyone gets any work done on the OS when you are losing so much time resolving dependencies.

There are some things that are better on Linux, yes, and it has extensive support within the server space and academic/scientific computer space, but you have to balance that against the sheer man-hours wasted doing things that can be done with two clicks on windows.

I was hoping that Linux would eventually become a competitor to Windows and MacOS. That it would be the OS that you could recommend to your mom or pop, an OS free of cost, free of bloat, free of ads and spyware, with all the features that one would need from a modern operating system.

I believed that user-friendly distros like popOS or its parent Ubuntu, or Windows-Like Distros like Mint, would eventually evolve to the point that they can enter the consumer space.

However, having gained some insight into the Linux experience, and the way that Linux functions “under the hood”, I now believe that it is destined to always remain what it is: A niche OS used only by technically minded people who use it either because they have to, or because they have the time or inclination to spend on it.