AI

Following on from my previous post on AI, I have come up with a very rudimentary algorithm to try to estimate the token generation rates that my AI rig will be able to produce when it is built.

Algorithm

The algorithm that I am using here is simple.

The token generation rate is, very roughly, the memory bandwidth divided by the model size in gigabytes.

So, if the model is running entirely on the GPU (With a bandwidth of 936 GB/s) and the model size is 120GB, then the token generation speed would be:

936/120= 7.8 tk/s

When running on the CPU, assuming a RAM bandwidth of 140 GB/s (See the graph in my previous post HERE).

140/120= 1.16 tk/s

These numbers do NOT take into account the context size of the model (Which can be a lot, for higher context models) or the time spent on prompt processing (Which can also be significant). In addition, it doesn’t take into account the overhead involved in, for example, combining multiple GPU’s together (IE, 48 GB’s VRAM Split over two 3090s is not going to perform as well as a single 48 GB GPU).

However, it seems that, as a general rule of thumb, these numbers work.

But what about hybrid systems? Where the processing is occurring on both the GPU and CPU?

This is where I had to made some assumptions, and I don’t yet know how well these assumptions hold up in the real world, if at all.

To generate the table below, I simply combined the GPU bandwidth and CPU bandwidth speeds together into an average bandwidth speed based on the percentage of the model that is on the GPU vs the CPU.

For example, assuming a 127.9 GB model, and 48 GBs of VRAM:

37.5% of the model is on the GPU, 62.4% of the model is on the CPU.

So, the average bandwidth is:

vram bandwith * (percentage of model on gpu /100) + system ram bandwidth * (percentage of model on cpu /100)

Or:

936*(37.5/100) + 140*(62.4/140) = 438.73 GB/s

Now, to get the tk/s, simply divide this by the model size:

438.73/127.9=3.43 tk/s.

Again, this is very approximate, and I won’t know how close it actually is to being accurate until the system is actually built, but I felt that this algorithm, and the data below, could be useful for some people.

Table

For the Table below, I have tested four conditions:

CPU and GPU, and CPU only (No GPU) for both a RAM bandwidth of 140 GB/s and a RAM bandwidth of 100 GB/s.

The reason why I have chosen to test two assumed RAM bandwidth speeds is because, first of all, RAM bandwidth is the most important factor is determining the performance of an AI system, and second of all, there can be a huge variation in the actual, real world, RAM bandwidth of a system (See my previous post).

NB: The following graph assumes that the system has access to 48 GB’s of VRAM. The numbers will of course be different if more or less processing is being done on the GPU (Although the numbers for CPU only will not change).

Conclusion

The numbers above look very good. With a 120b Q8 model and 140 Gb’s RAM bandwidth, the estimate is 3.43 tk/s! This, as I said, doesn’t take into account prompt processing time, context, and other real world considerations, but even as a rule of thumb, this is encouraging.

It will be interesting to see how well these numbers match up (If at all!) once I get the actual system built.

Introduction

As I mentioned in my previous post (HERE), I have purchased a Threadripper PRO 3955WX CPU for the purposes of building an LLM inference machine.

However, I have since discovered that there is a serious issue with using some Threadripper and Epyc CPU’s (Including the 3955wx) for this purpose.

The issue is that these CPU’s use 2 CCD’s (Core Chiplet Dies), which substantially reduces the memory bandwidth.

The issue is discussed here:

AMD Epyc 7002 Rome CPU’s with Half Memory Bandwidth. (Serve the Home)

And here:

Comparing Threadripper 7000 memory bandwidth for all models. (Reddit, r/Threadripper)

There are also many other sources confirming the results for various generations and models of chips.

The fundamental issue is that in order to reach the maximum bandwidth of 8-Channel RAM (About 200 GB/s) it is necessary to have not just 8 Channels supported, but 8 CCD’s as well.

Only extremely expensive CPU’s have 8 CCD’s.

The cheaper CPU’s have only 2, which effectively limits their ram bandwidth to quad channel speeds (Around 80-100 GB/s) or slightly more.

Ordinarily, this would not be a problem, since very, very, few use cases are going to come close to saturating the memory bandwidth like that.

The issue is that Large Language Model inference is one of those use cases (Computational Fluid Dynamics is another one). Not only that, but for LLM’s VRAM/RAM bandwidth is the single most important factor determining the performance of the system.

This means that the 3955wx which I bought is going to be extremely sub-optimal for this purposes, and needs to be replaced.

The problem is that it is extremely difficult to get actual, real world, values for RAM bandwidth online.

Most sources only quote maximum bandwidth values, which is 200 GB/s in 8 Channel mode. They make no reference to the CCD bandwidth issue, which can cause users to make bad purchasing decisions.

After spending some time researching this issue, I have decided to prepare a small graph showing my results, with sources, so that hopefully other users will be more informed than I was.

Data

I am focusing entirely on DDR4 CPU’s here, since DDR5 is out of my budget at this point.

I am open to expanding this graph if I can come across more data in the future.

The HTML Graph is quite squashed, so I have uploaded an image instead (Click for full image):

HTML Graph:

Analysis

At my price point, it seems that 4 CCDs with 8 Channels is basically the best that can be achieved. This means that the actual read/write speeds in the real world are in the region of 150 GB/s read 100 Gb/s Write.

It seems, based on my research, that there is no inherent difference between Threadripper/Pro and Epyc CPUs that have the same number of CCDs.

IE, A 4 CCD Threadripper Pro will have similar read/write speeds to a 4 CCD Epyc.

This conflicts with some users that indicate that the Epyc series have superior RAM bandwidth even with the same number of CCDs. I have come across multiples sources, for example, that indicate tha the 7302p, with 4 CCD’s, can reach 200 GB/s RAM bandwidth. I believe that this is incorrect, based on my research.

For my budget, a CPU like the Epyc 7302p probably would have been a better choice.

It is (substantially) less expensive than the threadripper, but has similar RAM bandwidth, due to it’s 4 CCD’s.

However, I eventually chose to go with a Threadripper PRO 3975WX.

There were several reasons for this.

Firstly, the Threadripper PRO is a substantially more powerful CPU.

It has a higher clock speed, higher boost clock, and twice as many cores (32 vs 16), and comfortable outperforms the Epyc in both single and multi core applications:

Comparision between Threadripper PRO 3975WX and Epyc 7302.

CPU speed and Core count are not priorities for LLM inference, however they do make a difference, and more/faster cores are still preferable.

The motheboards for the Threadripper PRO are also better suited to my use case. They have more PCI-E x16 slots, and seem to have better support for peripherals, including graphics cards, etc. They also seem to be cheaper, at least in my location.

Secondly, I did not want to have to return my Motherboard in addition to my CPU, since unlike the CPU, this is a bulky item.

The Threadripper PRO 3975 is quite a bit more expensive than the 3955 and (especially) the 7302, but it was just about within budget.

Conclusion

It is vitally important that users buying hardware for LLM’s and other memory bandwidth intensive applications be aware of the bandwidth limits caused by CCD’s, otherwise the performance of CPU-based inference can be substantially lower than expected.

In my case, for example, the 3955wx would have had a read speed of 82 GB/s, when compared to the 137 GB/s of the 3975!

Of course, if the inference is mainly happening on the GPU, the CPU is far less important, but I intend to run large models, (70b, 120b, etc), which means that these memory bandwidth limitations will have a substantial effect on the performance of the system for its intended use case.