Dependency Hell: The Linux Experience
This week was one of the most frustrating that I have had since I first learned to program a computer at the age of 17.
The reason was that this week I discovered the true nature of the Linux Experience: Dependency Hell.
I had heard of this before but did not truly appreciate the nature of this beast.
In short, unlike Windows, which uses compiled executable programs, Linux largely expects users to download source code and compile their own binaries. This is likely due to the sheer number of Linux distros out there, releasing a binary for all of them would be impractical. The problem with this is Linux make extensive use of common libraries, or dependencies, that are shared between various programs, and are, unfortunately, often incompatible.
So, you could have one program which requires at least version 2.0 of a library, and another program that will not run on anything later than 1.0. If you want both programs to run, you are in “Dependency hell”.
Unfortunately, in the AI world, there are a LOT of common libraries (CUDA, PyTorch, Torchvision/audio, xFormers, etc), leading to the issues that I had this week.
It should be noted before I continue that when I built my Linux-Powered AI rig I went into it with significant optimism, I loved the idea of Linux. I can’t stand the bloatware on windows, the tracking, the constant forced updates and restarts, the ads, and the lack of control over the operating system. When I bought my current windows 11 laptop (They didn’t have any with windows 10) I clicked on the start button and saw a new article about Prince Charles’ Coronation. It’s bad enough that I am bombarded with ads on the internet, I do NOT want to see ads when I click on the start menu of my own computer!
The rule is EITHER paid OR ad supported, not both! If your OS costs money, you can’t have ads too! Microsoft haven’t learned this.
I even installed Linux on my current laptop briefly, but the drivers for the trackpad didn’t work, so I reluctantly installed Win11 again.
However, the difficulty that I have had this week just getting programs to run has seriously soured me on the Linux experience.
The issues all started when I tried to install xTTS (A Text to speech extension) for SillyTavern.
I did a Pip Install, I had to update some packages, but everything seemed ok.
Then I realised that when I updated the dependences to allow xTTS to work, I had broken several of my other programs, including comfyUI!
Reinstalling those programs, of course, broke xTTS, because the versions of the software used by each were incompatible.
I eventually learned about Conda and Virtual environments, which solved some of my issues, but not all (Never in my time using windows have I had to install a virtual environment just to get a program to run!).
However, even with Conda/Venv, I was STILL having serious problems just fixing my existing programs, let alone installing new ones. At one point, at least 3-4 of my main AI programs were in a broken state due to dependencies, and I had completely lost faith in the system. It got to the point where I was genuinely ecstatic just to start up a program and have it actually work!
What I learned eventually is that when you run pip install inside a conda environment it installs the packages to the location where pip was installed, which is your systems pip directory. IE, you are installing the packages globally, potentially overwriting other packages.
Using conda install fixes this problem, but most of the packages I was using didn’t have a conda version, so I was forced to spend many hours pip installing dozens of packages to attempt to find a solution.
At one point, I restarted my system, and found that I couldn’t log into the gui! Apparently my hard drive was full, which, for some reason, prevented login?
It turns out that I had used up 75 GIGABYTES of storage page just downloading difference packages! CUDA, for example, is about 2 GB’s. Torch vision is about 1 GB, so every time I do a pip install that’s another 2 GB’s of data.
I purged the pip cache and moved some data from the main drive to the backup drive (Luckily remote desktop still worked, even though the local login didn’t?) and I was able to log in again, but I still had the dependency issues to deal with.
Eventually, after several days of misery, I discovered that you can install pip locally to a conda environment, so that if you then run pip install it will install the packages to the local version of pip, not the system version, allowing you to use pip install without overwriting your system packages.
So far, this seems to work.
Without bragging, I have nearly 25 years experience working with computers. I am new to Linux, yes, but I have extensive experience in programming, the command line, libraries, and the complexities of operating systems. If it takes me three days just to get a program running without breaking another program, then there is a problem with the operating system. It simply should not be that difficult.
Just to clarify, all I am trying to do here is install a program, that’s all. I’m not trying to write code, just run it!
ALL of the above could have been accomplished on windows by just double-clicking on an exe!
There was one program (A web based gui for training voices for Realtime Voice Conversion, or RVC) that I spend several hours trying to resolve the dependencies for. Everything I pip installed something, it broke something else. I had to reinstall comfyui at least seven or either times!
Eventually, I gave up and downloaded a fork of the program that worked on windows. I ran into ONE error (My firewall was blocking the clone of the git repo. I simply disabled it temporarily until the download was completed) and the program was up and running in less than three minutes.
This has consistently been my experience: Fighting with Linux to resolve dependencies that would require just a few clicks on windows.
I know that I have extensive experience on Windows which I lack in Linux, but we are just talking about basic functionality here, you shouldn’t need to be a Linux guru just to get a program to run!
The other discovery that I made is that the AI world doesn’t appear to be quite as Linux-centric as I initially thought. Before I built my AI rig I read that the vast majority of AI-related programs and libraries are designed for Linux, and I would have difficulty running them on windows.
In actual fact, in my experience, the only thing that I have found that was easier on Linux than windows was installing Sage-Attention 2. Sage attention 2 can be installed with just a few lines of code on Linux, it is, apparently, a lot harder on Windows.
Apparently Deepspeed is not properly supported on Windows either.
Everything else either had a windows version, or there was a similar program that worked on windows. In some cases in fact I found more easily accessible information for windows than Linux (Such as Lora training for Hunyuan Video).
It seems to me that a lot of the people singing the praises of Linux are people who are card-carrying members of the Linux club, and either fail to see it’s substantial shortcomings, or obstinately refuse to acknowledge them.
Windows isn’t perfect, I mentioned the issues with bloatware, privacy, etc, but honestly, after dealing with Linux “dependency hell” for the last 3-4 days, I will happily allow Bill Gates to come to my house personally and watch me take a shower.
Running a debloat script on windows 11 is far, far, easier than dealing with Linux’s issues.
Linux is for people who want to work on Linux, not for people who want to work on anything else.
Windows, despite its issues, just works, and when you’re a professional who values their time, you want something that works.
I cannot understand how anyone gets any work done on the OS when you are losing so much time resolving dependencies.
There are some things that are better on Linux, yes, and it has extensive support within the server space and academic/scientific computer space, but you have to balance that against the sheer man-hours wasted doing things that can be done with two clicks on windows.
I was hoping that Linux would eventually become a competitor to Windows and MacOS. That it would be the OS that you could recommend to your mom or pop, an OS free of cost, free of bloat, free of ads and spyware, with all the features that one would need from a modern operating system.
I believed that user-friendly distros like popOS or its parent Ubuntu, or Windows-Like Distros like Mint, would eventually evolve to the point that they can enter the consumer space.
However, having gained some insight into the Linux experience, and the way that Linux functions “under the hood”, I now believe that it is destined to always remain what it is: A niche OS used only by technically minded people who use it either because they have to, or because they have the time or inclination to spend on it.
July 29, 2025 @ 5:24 am
Wow. Tell me you have no idea how to use Linux without telling me. No. Linux does not “Linux largely expects users to download source code and compie their own binaries.”. And learn how to use a package manager properly. “So, you could have one program which requires at least version 2.0 of a library, and another program that will not run on anything later than 1.0. If you want both programs to run, you are in “Dependency hell”. Yes. Windows has this as well. Was common with programs that needed specific versions of DirectX and Visual basic libraries. Ever wonder why later versions of Windows had the “compatibility tool” that would emulate and fake earlier versions of Windows to certain programs? Or why you upgrade to a newer version of Windows and suddenly some of your hardware stops working because the manufacturer didn’t bother making updated drivers (Rare in Linux as it supports old hardware for AGES before it gets kicked out of the kernel.).
“Without bragging, I have nearly 25 years experience working with computers. I am new to Linux, yes, but I have extensive experience in programming, the command line, libraries, and the complexities of operating systems. If it takes me three days just to get a program running without breaking another program, then there is a problem with the operating system. It simply should not be that difficult.”
Trust me you aren’t bragging. You admit you are new and know virtually nothing about Linux then go on to bash what you don’t know anything about. If it takes you 3 days to get a program running without breaking another in Linux then either you are trying to setup something very obscure or you just aren’t a competent user. The fact you “have 25 years experience working with computers” and can’t figure simple stuff out like how to use a package manager which basically does everything for you… Yea. It’s for sure an 1D10T error.
July 29, 2025 @ 12:34 pm
I’ve decided to approve this in the interests of Freedom of Speech, and because you do make some good points, but your tone is a little more hostile than I’d like.
Many of the programs that I was using were not available with the package manager, or the issues still occurred regardless. I spent, as I said, considerable time reserching the issues that I was having, and discovered many other people encountering the same problems, with the same programs (Specificially pytorch, torchvision, and CUDA, etc), did they not use the package manager properly as well? These are known issues with Linux. In addition, windows doesn’t need a package manager, because it doesn’t have the same issues.
You are correct, I used to use a games engine called the “Torque Games Engine” from Garagegames, and we did have issues with DirectX compatibility. However that was 20 years ago, and it was for a games engine that would have been relatively obscure at the time. I haven’t run into it on windows since. I’m not saying windows is perfect, but issues like this are vastly more common on Linux. The term “dependency hell” exists for a reason.
With respect, you’re comparing different versions of an OS! There’s a huge difference between a program, or some hardware, breaking when you upgrade from Windows 7 to Windows 10 and having multiple programs randomly stop working because you downloaded and installed another program onto the same system that happens to use a common library somewhere.
There’s no need to be as hostile as this, but I will address your points anyway. I said I was new to Linux, but that I have extensive computer experience. I was making it clear that I am not a novice user, and if someone with that much experience in computers is having difficulties, it’s likely that the fault is not entirely my own.
The programs that I was using were described in the post. Whether you consider them “obscure” or not is a matter of some debate. I doubt the average linux user is using Cuda, Pytorch, or Torchvision, or running packages from Github, etc, so maybe you’re right, however it doesn’t change the fact that I could, (and in some cases, did) run the same or similar packages in windows with no issues. Like I said, Sage Attention is supposedly the only thing that I was using that is easier to run on linux.
Like I said, I was using a package manager where it was available. I was following tutorials to the letter while working on setting up this software, and I researched them well before installing, and still I had issues. I also, as I said, ran into several other users encountering the same issues (Which was fortunate, because I could use their experience to help solve my problems). I am a competent user, and if all I had to do was “use a package manager that does everything for me” then I wouldn’t have needed to make this post.
October 2, 2025 @ 9:27 am
Supposedly, developers are not supposed to break stuff in newer versions of libraries.
If I define ‘I want torchvision >2.0.0’ in my config, then torchvision 4.0.0 should still work with my program. As long as I use the ABI and don’t depend on internals, things should keep working.
That is the philosophy of the kernel and a few important libraries, and it works wonders… unfortunately many developers don’t get the memo and break stuff left and right, sometimes for questionable reasons. And you’re right. That’s a problem. It’s a cultural issue.
If you do want to break what you want to do with your library… provide compatibility. Assume the worst case, that a closed-source program built by a now-defunct company that obfuscates its internals with nasty DRM code a user depends on for interfacing with their 40-year lifespan $4 million dollar MRI machine uses your library somehow and code accordingly.
It’s really not that hard. In the win32 API, they used all sorts of outdated legacy stuff such as UCS-2 encoding instead of UTF-16 or the more modern UTF-8, because windows 95 predates modern unicode. But instead of changing the behaviour of loadLibrary, they wrote loadLibraryA, and loadLibraryEx and loadLibraryExA, then updated their documentation to tell users ‘here is the newest version you should use’, while keeping the old functions with the same signatures and (mostly) behaviour in the kernel. Only security issues are allowed to break things.
I’m not the only one complaining about this. Linus Torvalds (who maintains the linux kernel) has the same things to say. If other developers depend on your code, don’t change what it does from under them without a very good reason!
As a result, even the most insane DRM programs (hello, secuROM) kept running under windows for two decades.
And like you said Microsoft goes even further by adding compatibility hacks. (https://devblogs.microsoft.com/oldnewthing/ is a great resource) When a program depends on an internal quirk of the memory manager, then windows will restore that quirk in newer versions. While I don’t think Linux developers should go quite that far for everything (not really feasible with the lack of manpower) some core libraries should entertain the thought of doing more (looking at you, libc, for breaking my linux games every two years).
And for the AI library scenario, it’s quite the same thing. While mostly open source, projects get abandoned left and right, or rely on certain versions of complex tools and don’t have the resources to keep chasing breaking bugs when those well-funded tools play fast and loose with compatibility to keep up with the rapid development in the space. The users are left with broken systems, or with having to do virtualization to keep things compatible (or running docker containers), and using massive amounts of hard drive space.
I’ve had the same problem, filling up a half-full 512GB hard drive in a doomed attempt to compile ROCm for both diffusion and text generation by hand. I called it quits after the script reached 50 pages worth of code (and I suspect so did my distro maintainer) and tried the manufacturer’s ubuntu based version on my arch distro (which required some major dependency juggling). It shouldn’t be this hard to multiply some matrices!
October 3, 2025 @ 5:11 am
Thanks for your comment!
Yeah, I get that Linux isn’t as bad now as it used to be (Or so I read, anyway), and I’m sure if you’re using Linux for regular computing tasks, it’s probably fine.
But if you’re on the cutting edge (Such as installing new repos from GitHub, or using rapidly changing AI toolkits, etc), you are going to break stuff a lot.
You made a great point about the win32 API, and I design my code like that too (At least, I try to!), so that new functionality doesn’t break old functionality.
Yes, it can be messy to have mylibraryA, mylibraryB, etc, but you know for a fact that you aren’t breaking anything, and creating hell for the dev’s relying on that functionality.
Linux doesn’t seem to have the same attitude.
I think that Linux assumes that the people using the OS are experienced devs with the knowledge and time needed to fix broken code and find workarounds. That’s what I mean in my post: Linux is really a dev’s os, which is great! But Windows is for when you want to get stuff done, and not waste time fixing things.
I’m not saying Linux can’t be used for professional work, of course it can, and it is, but in general, the attitude regarding reliability and functionality is different.
Exactly, users really shouldn’t have to rely on virtualisation to get code to work. I use virtualisation for 20-year old games, not software that’s just a few months old!
October 13, 2025 @ 2:50 am
As someone currently stuck in dependency hell (also related to ComfyUI), this was cathartic to read.
October 15, 2025 @ 2:26 pm
What areas are you stuck on? I might be able to offer some advice?