Running Large Language Models: System Specs

by PhoenixGames on Sep.15, 2024, under AI, Tutorials

I have recently become very intersted in running AI Large language Models (LLM’s).

With a view to furthering my research in this area, I have been planning a build for a machine dedicated to AI inference.

My goal is to be able to run 70b models at Q6 or even Q8 quants with a tk/s of 3-6 tk/s at least, and, hopefully, a 120b model with at least a Q5 quant at at least 1 tk/s.

The spec that I arrived at is:

These have 24 gbs VRAM each, for a total of 48 GBs, and they have just two PCIE power connectors, not three, making it easier to power them.

2×3090 Asus TUF Gaming GPUs	These have 24 gbs VRAM each, for a total of 48 GBs, and they have just two PCIE power connectors, not three, making it easier to power them.
Threadripper PRO 3955WX	I went with the PRO threadripper because of it’s support for more than 256 GBS RAM, and it’s 128 PCE lanes. I could have probably went with the 3945 model, since the clock speeds are similar, and the extra 4 cores (16 vs 12) of the 3955 probably won’t make that much difference for inference.
256 GB 3200 MHZ DDR4 RAM	3200 Mhz DDR4 is not the fastest, but it’s the fastest speed that the 3955wx supports, and I don’t think that over clocking 8x32GB sticks is going to work. I need 8 sticks because I want to use 8 channel memory. Memory bandwidth is very important for LLM’s, and 8 channel memory has about a 200 GB/s bandwidth, vs 100 GB/s for quad channel.
WRX80-E SAGE Motherboard	This actually cost more than the CPU, but it has 7 PCI-x16 ports, which I will need in the future if I intend to add more GPU’s, and because it has 8 channel memory support.
Corsair HX1500	A 1500 watt PSU should be ok for two 3090’s, maybe even 3 if I underclock the card. If I get any more in the future I will have to get another PSU and connect them together.
2 TB M.2 SSD
Noctua Cooler
Mining Case	I went with an open Air mining rig because it is the only setup that would allow me to add more than 2 GPUS.