r/LocalLLM • u/Ill-Werewolf9775 • 1d ago
Question V100 or not?
I have a 4060 8GB on my main pc, and I started playing with some small models, but the resources are too limited.
I am now looking to buy a couple of V100 with the external board (it is connected on PCIe) that can use them with NVlink. Due to the price I can afford the 16 gb version. I plan to use them on one of the nodes of my proxmox cluster, with a dedicated psu. Do you think it is too late for buying this kind of setup?
1
2
u/FullstackSensei 1d ago
IMO, get the native PCIe cards. By the time you account for the adapters, heats inks and cables, they cost the same but are quite bigger and unwieldy to deal with.
A pair of PCIe cards, meanwhile, can fit neatly in your ATX motherboard
1
u/Ill-Werewolf9775 1d ago
I am thinking to the sxm2 version, so I can keep them outside the case... This stuff and a couple of those big (passive?) Heatsink. Is it also possible to mix a 32 and a 16 GB GPUs?
2
u/fastandlight 1d ago
Just don't. Better uses for your money now. I'd recommend a new Intel GPU card over V100s. No native FP8 or 4 bit hardware either. Yeah, you can do it in software, but even then you only have 16 or 32 gb of memory. And everything is dropping support for cards that old. I'd vote for saving longer for newer hardware.
1
3
u/Trademarkd 1d ago
I started a sub r/v100
It's still a good card with a lot of flexibility. The primary issue you'll run into is lack of vram on either model, leading to distributing over multiple, and then you learn how important pico and nano seconds of latency are in autoregressive. You also only get FP16 cores so you run into issues with BF16 values and you can never process faster than FP16 (even though you can store the weights as lower quants in HBM)
SXM2 or PCIE?
the sxm2's are cheaper, have more tdp and have nvlink capability, but you'll need an adapter board, heat sink, fans, etc.
For reference, I now run a cluster of 8.