{"id":10391,"date":"2024-10-02T21:43:00","date_gmt":"2024-10-02T20:43:00","guid":{"rendered":"https:\/\/phoenixgamedevelopment.com\/blog\/?p=10391"},"modified":"2024-10-07T01:16:53","modified_gmt":"2024-10-07T00:16:53","slug":"ai-memory-bandwidth-comparision-for-selected-ddr4-cpus","status":"publish","type":"post","link":"https:\/\/phoenixgamedevelopment.com\/blog\/ai-memory-bandwidth-comparision-for-selected-ddr4-cpus\/","title":{"rendered":"AI: Memory Bandwidth comparison for selected DDR4 CPU’s"},"content":{"rendered":"\n

Introduction<\/strong><\/p>\n\n\n\n

As I mentioned in my previous post (HERE<\/a>), I have purchased a Threadripper PRO 3955WX CPU for the purposes of building an LLM inference machine.<\/p>\n\n\n\n

However, I have since discovered that there is a serious issue with using some Threadripper and Epyc CPU’s (Including the 3955wx) for this purpose.<\/p>\n\n\n\n

The issue is that these CPU’s use 2 CCD’s (Core Chiplet Dies), which substantially reduces the memory bandwidth.<\/p>\n\n\n\n

The issue is discussed here: <\/p>\n\n\n\n

AMD Epyc 7002 Rome CPU’s with Half Memory Bandwidth.<\/a> (Serve the Home)<\/p>\n\n\n\n

And here:<\/p>\n\n\n\n

Comparing Threadripper 7000 memory bandwidth for all models.<\/a> (Reddit, r\/Threadripper)<\/p>\n\n\n\n

There are also many other sources confirming the results for various generations and models of chips.<\/p>\n\n\n\n

The fundamental issue is that in order to reach the maximum bandwidth of 8-Channel RAM (About 200 GB\/s) it is necessary to have not just 8 Channels supported, but 8 CCD’s as well.<\/p>\n\n\n\n

Only extremely expensive CPU’s have 8 CCD’s.<\/p>\n\n\n\n

The cheaper CPU’s have only 2, which effectively limits their ram bandwidth to quad channel speeds (Around 80-100 GB\/s) or slightly more.<\/p>\n\n\n\n

Ordinarily, this would not be a problem, since very, very, few use cases are going to come close to saturating the memory bandwidth like that.<\/p>\n\n\n\n

The issue is that Large Language Model inference is one of those use cases (Computational Fluid Dynamics is another one). Not only that, but for LLM’s VRAM\/RAM bandwidth is the single most important factor determining the performance of the system.<\/p>\n\n\n\n

This means that the 3955wx which I bought is going to be extremely sub-optimal for this purposes, and needs to be replaced.<\/p>\n\n\n\n

The problem is that it is extremely difficult to get actual, real world, values for RAM bandwidth online. <\/p>\n\n\n\n

Most sources only quote maximum bandwidth values, which is 200 GB\/s in 8 Channel mode. They make no reference to the CCD bandwidth issue, which can cause users to make bad purchasing decisions.<\/p>\n\n\n\n

After spending some time researching this issue, I have decided to prepare a small graph showing my results, with sources, so that hopefully other users will be more informed than I was.<\/p>\n\n\n\n

Data<\/strong><\/p>\n\n\n\n

I am focusing entirely on DDR4 CPU’s here, since DDR5 is out of my budget at this point.<\/p>\n\n\n\n

I am open to expanding this graph if I can come across more data in the future.<\/p>\n\n\n\n

The HTML Graph is quite squashed, so I have uploaded an image instead (Click for full image):<\/p>\n\n\n\n

\"\"<\/a><\/figure>\n\n\n\n

HTML Graph:<\/p>\n\n\n\n

CPU<\/strong><\/td>Read Speed<\/strong><\/td>Write Speed<\/strong><\/td>Copy Speed<\/strong><\/td>CCD’s<\/strong><\/td>Ram Type<\/strong><\/td>Ram Channels<\/strong><\/td>Ram Speed<\/strong><\/td>Sources<\/strong><\/td><\/tr>
Ryzen 9 5950x<\/td>54 GB\/s<\/td>54 GB\/s<\/td>–<\/td>2<\/td>DDR4<\/td>2<\/td>3600<\/td>Link<\/a><\/td><\/tr>
Threadripper 3960x<\/td>95 GB\/s<\/td>93 GB\/s<\/td>101 GB\/s<\/td>4<\/td>DDR4<\/td>4<\/td>3200<\/td>Link<\/a><\/td><\/tr>
Threadripper 3970x<\/td>96 GB\/s<\/td>98 GB\/s<\/td>102 GB\/s<\/td>4<\/td>DDR4<\/td>4<\/td>3200<\/td>Link<\/a> <\/td><\/tr>
Threadripper Pro 3955wx<\/td>82GB\/s<\/td>51 GB\/s<\/td>94 GB\/s<\/td>2<\/td>DDR4<\/td>8<\/td>3200<\/td>Link<\/a><\/td><\/tr>
Threadripper Pro 3975wx<\/td>137-139 GB\/s<\/td>102 GB\/s<\/td>137 GB\/s<\/td>4<\/td>DDR4<\/td>8<\/td>3200<\/td>Link<\/a>
Link<\/a>
Link<\/a>
<\/td><\/tr>
Threadripper Pro 3995wx (64C\/128t)<\/td>–<\/td>149 GB\/s<\/td>–<\/td>8<\/td>DDR4<\/td>8<\/td>3200<\/td>Link<\/a><\/td><\/tr>
Epyc 7302p<\/td>115  GB\/s<\/td>85 GB\/s<\/td>128 GB\/s<\/td>4<\/td>DDR4<\/td>8<\/td>2933<\/td>Link<\/a><\/td><\/tr>
Epyc 7443<\/td>136 GB\/s<\/td>–<\/td>–<\/td>4<\/td>DDR4<\/td>8<\/td>3200<\/td>Link<\/a><\/td><\/tr>
Epyc 7551p<\/td>154 GB\/S<\/td>156 GB\/S<\/td>145 GB\/S<\/td>–<\/td>DDR4<\/td>8<\/td>2600<\/td>Link<\/a><\/td><\/tr>
2xEpyc 7302p (Dual CPU)<\/td>219 GB\/s<\/td> <\/td>–<\/td>4*2<\/td>DDR4<\/td>8×2<\/td>2400<\/td>Link<\/a><\/td><\/tr>
Threadripper Pro  5995wx<\/td>160 GB\/s<\/td>171 GB\/s<\/td>–<\/td>8<\/td>DDR4<\/td>8<\/td>3200<\/td>Link <\/a>
Link<\/a><\/td><\/tr>
Threadripper Pro  5975wx<\/td>147 GB\/s<\/td>171 GB\/s<\/td>–<\/td>4<\/td>DDR4<\/td>8<\/td>3200<\/td> Link <\/a><\/td><\/tr>
Threadripper Pro  5965wx<\/td>147GB\/s<\/td>172 GB\/s<\/td>–<\/td>4<\/td>DDR4<\/td>8<\/td>3200<\/td> Link <\/a><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Analysis<\/strong><\/p>\n\n\n\n

At my price point, it seems that 4 CCDs with 8 Channels is basically the best that can be achieved. This means that the actual read\/write speeds in the real world are in the region of 150 GB\/s read 100 Gb\/s Write.<\/p>\n\n\n\n

It seems, based on my research, that there is no inherent difference between Threadripper\/Pro and Epyc CPUs that have the same number of CCDs.<\/p>\n\n\n\n

IE, A 4 CCD Threadripper Pro will have similar read\/write speeds to a 4 CCD Epyc.<\/p>\n\n\n\n

This conflicts with some users that indicate that the Epyc series have superior RAM bandwidth even with the same number of CCDs. I have come across multiples sources, for example, that indicate tha the 7302p, with 4 CCD’s, can reach 200 GB\/s RAM bandwidth. I believe that this is incorrect, based on my research.<\/p>\n\n\n\n

For my budget, a CPU like the Epyc 7302p probably would have been a better choice. <\/p>\n\n\n\n

It is (substantially) less expensive than the threadripper, but has similar RAM bandwidth, due to it’s 4 CCD’s.<\/p>\n\n\n\n

However, I eventually chose to go with a Threadripper PRO 3975WX.<\/p>\n\n\n\n

There were several reasons for this.<\/p>\n\n\n\n

Firstly, the Threadripper PRO is a substantially more powerful CPU.<\/p>\n\n\n\n

It has a higher clock speed, higher boost clock, and twice as many cores (32 vs 16), and comfortable outperforms the Epyc in both single and multi core applications: <\/p>\n\n\n\n


Comparision between Threadripper PRO 3975WX and Epyc 7302.<\/a><\/p>\n\n\n\n

CPU speed and Core count are not priorities for LLM inference, however they do make a difference, and more\/faster cores are still preferable.<\/p>\n\n\n\n

The motheboards for the Threadripper PRO are also better suited to my use case. They have more PCI-E x16 slots, and seem to have better support for peripherals, including graphics cards, etc. They also seem to be cheaper, at least in my location.<\/p>\n\n\n\n

Secondly, I did not want to have to return my Motherboard in addition to my CPU, since unlike the CPU, this is a bulky item.<\/p>\n\n\n\n

The Threadripper PRO 3975 is quite a bit more expensive than the 3955 and (especially) the 7302, but it was just about within budget.<\/p>\n\n\n\n

Conclusion<\/strong><\/p>\n\n\n\n

It is vitally important that users buying hardware for LLM’s and other memory bandwidth intensive applications be aware of the bandwidth limits caused by CCD’s, otherwise the performance of CPU-based inference can be substantially lower than expected.<\/p>\n\n\n\n

In my case, for example, the 3955wx would have had a read speed of 82 GB\/s, when compared to the 137 GB\/s of the 3975!<\/p>\n\n\n\n

Of course, if the inference is mainly happening on the GPU, the CPU is far less important, but I intend to run large models, (70b, 120b, etc), which means that these memory bandwidth limitations will have a substantial effect on the performance of the system for its intended use case.<\/p>\n\n\n\n

<\/p>\n\n\n","protected":false},"excerpt":{"rendered":"

Introduction As I mentioned in my previous post (HERE), I have purchased a Threadripper PRO 3955WX CPU for the purposes of building an LLM inference machine. However, I have since discovered that there is a serious issue with using some Threadripper and Epyc CPU’s (Including the 3955wx) for this purpose. The issue is that these […]<\/p>\n","protected":false},"author":1,"featured_media":10551,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[37,12],"tags":[],"class_list":["post-10391","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai","category-tutorials","has-post-title","has-post-date","has-post-category","has-post-tag","has-post-comment","has-post-author",""],"aioseo_notices":[],"builder_content":"","_links":{"self":[{"href":"https:\/\/phoenixgamedevelopment.com\/blog\/wp-json\/wp\/v2\/posts\/10391"}],"collection":[{"href":"https:\/\/phoenixgamedevelopment.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/phoenixgamedevelopment.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/phoenixgamedevelopment.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/phoenixgamedevelopment.com\/blog\/wp-json\/wp\/v2\/comments?post=10391"}],"version-history":[{"count":34,"href":"https:\/\/phoenixgamedevelopment.com\/blog\/wp-json\/wp\/v2\/posts\/10391\/revisions"}],"predecessor-version":[{"id":10567,"href":"https:\/\/phoenixgamedevelopment.com\/blog\/wp-json\/wp\/v2\/posts\/10391\/revisions\/10567"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/phoenixgamedevelopment.com\/blog\/wp-json\/wp\/v2\/media\/10551"}],"wp:attachment":[{"href":"https:\/\/phoenixgamedevelopment.com\/blog\/wp-json\/wp\/v2\/media?parent=10391"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/phoenixgamedevelopment.com\/blog\/wp-json\/wp\/v2\/categories?post=10391"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/phoenixgamedevelopment.com\/blog\/wp-json\/wp\/v2\/tags?post=10391"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}