Thank you for your reply.

The numbers and algorithms posted above were always rough rules of thumb, and having tested them in real world conditions, they are certainly not entirely accurate, however, they are not entirely inaccurate either.

I was running 120b Q8 models with two 3090s (The rest of the model being on the CPU) and I was getting anything from 1-2 tk/s depending on the context, etc.

The graph above shows 3.43  tk/s for 140 gb/s RAM bandwidth, which is noticeable higher than I was getting in reality, but again, it’s a rough estimate.

The formula I used (memorybandwidth / model size) was based on internet searches, reddit, etc. I admit, not the most reliable sources, but I couldn’t find anything better at the time. Can you tell me where you got yours?

Another problem that I have noticed is that you are dividing by the GPU bandwidth, so, as the GPU bandwidth increases, the token generation rate decreases, which, surely, should be the opposite?

For example, you said:

(2 * 120 * 1000) / (936) = 256.410

So, for a GPU with a bandwidth of 936 gb/s, we would get 256 tk/s.

Ok, lets now assume that we double the bandwidth, to 1800 gb/s. We would expect to get twice the tokens, right? But with your algorithm:

(2*120*1000)/(1800), we get: 133.33

So instead of doubling, the tk/s is halving? Which can’t be right surely, unless I’m missing something?

(2 * n * 1000) / (b_w)

where

n – number of billion parameters
b_w – GPU bandwidth in GB/s

Taking your example:

> So, if the model is running entirely on the GPU (With a bandwidth of 936 GB/s) and the model size is 120GB, then the token generation speed would be:

we would have

(2 * 120 * 1000) / (936) = 256.410

which again, would be close if the model wouldn’t need to be split across several GPUs and the card was 100% memory bound – this depends on the model as much as on the card

What areas are you stuck on? I might be able to offer some advice?

You are very welcome, thank you for your comment!

Thanks for your comment!

Yeah, I get that Linux isn’t as bad now as it used to be (Or so I read, anyway), and I’m sure if you’re using Linux for regular computing tasks, it’s probably fine.

But if you’re on the cutting edge (Such as installing new repos from GitHub, or using rapidly changing AI toolkits, etc), you are going to break stuff a lot.

You made a great point about the win32 API, and I design my code like that too (At least, I try to!), so that new functionality doesn’t break old functionality.

Yes, it can be messy to have mylibraryA, mylibraryB, etc, but you know for a fact that you aren’t breaking anything, and creating hell for the dev’s relying on that functionality.

Linux doesn’t seem to have the same attitude.

I think that Linux assumes that the people using the OS are experienced devs with the knowledge and time needed to fix broken code and find workarounds. That’s what I mean in my post: Linux is really a dev’s os, which is great! But Windows is for when you want to get stuff done, and not waste time fixing things.

I’m not saying Linux can’t be used for professional work, of course it can, and it is, but in general, the attitude regarding reliability and functionality is different.

Exactly, users really shouldn’t have to rely on virtualisation to get code to work. I use virtualisation for 20-year old games, not software that’s just a few months old!