r/LocalLLaMA • u/iwinux • 19h ago
Discussion A very confusing report from Puget Systems
https://www.pugetsystems.com/labs/articles/amd-radeon-ai-pro-r9700-dual-gpu-ai-inference-performance/Just to name a few:
- running Qwen3 8B on a 32GB GPU
- running Qwen3.6-27B Q4_K_M on 2 x R9700
- quote: "each prompt was sized at 500 input and 500 output tokens"
- for a full system that costs $18,775??
I don't understand what they are doing. Am I reading something wrong?
117
u/o0genesis0o 19h ago
What BS is this? They claim that 27B does not fit on 1 R9700 (at Fp16). You need 2xR9700. Okay, fair enough.
However, for the 2xR9700, they tested with Q4 Llamacpp. Dafuq? Why did they yap yap about FP16 for 1 card, but then run Q4 for two card? At Q4, 1 card can run both 27B and 35B well enough already.
And they tested ancient "deep seek R1" distill 8B? And what kind of abysmal number they got out of two kickass R9700?
Seriously. I feel like an absolute impostor fraud when I try to suggest my friends what hardware to build and how to setup their inference backend. But now I read whatever these clowns write, I suddenly don't feel like an impostor anymore.
All of us crazy people in this sub should open company to build AI server for people. We would kick those clowns ass.
63
u/RedditNamesAreShort 18h ago
It gets even better:
For context, the largest open-weight models (DeepSeek V4 Flash at 568 GB) remain firmly out of reach for any workstation-class hardware — those are datacenter territory.
Yup they just multiplied param count by 2 completely ignoring that its not an fp16 model to begin with.
6
u/o0genesis0o 17h ago
4 DGX Spark should be able to run that nodiff, right? It's too fancy for my 16GB VRAM blood, but I vaguely remember that model is fp4 by default. So 256GB weight, plus 256GB KV. Could likely serve a whole small company with that.
6
10
u/michaelsoft__binbows 17h ago
Never underestimate the power of getting famous on knowing what the fuck you're talking about...
6
u/Nefilim314 10h ago
I feel you about the imposter syndrome bit.
I feel like I sound crazy when I say I am doing meaningful work using only my laptop 5070 Ti with 12GB of memory that I have exhaustively configured to run the best possible speed/resolution for my workflow using the most minimal configuration of pi that I could muster.
I talked to one guy who confidently sang about his complex agent workflows and was practically humble- bragging about his insane token usage that the company paid for. He had agents that managed agents that managed other agents to run these elaborate jobs where he would write some vague text that one would convert into an implementation plan and another model would vet that plan and have a back and forth, then cut the plan on and delegate it to other agents to build which would delegate to other agents to write tests for. Then an agent would read everything write technical documentation for all of it which seemed entirely pointless since no one actually wrote any code in the code base. Then another agent would run the deployment.
He had managed to run up something to the tune of $12k in spend doing this. I saw the breakdown of usage and it was entirely set to Fable and Opus. Rarely did he ever use Sonnet, and never once used Haiku to do any of this… let alone even considering just using an alternative provider.
He spends more time trying to diagnose why his agents aren’t picking up tasks correctly rather than fix the actual bugs because he truly doesn’t understand the code that was built.
But damn, he sounds so confident about his AI chops.
2
u/o0genesis0o 10h ago
Maybe with one or more generation of model, we would be able to hammer good common sense into models, but right now, something is still not quite right with them, even if they are insanely knowledgeable in coding and can code tirelessly, and they can indeed design elegant architecture. The problem is which part to choose. I spent the entire day today pulling my hair out with Minimax M3 sometimes. It offers excellent paths to overcome architectural problems and technical problems and most of the time it is right. But sometimes, usually at the crucial decision, it makes very questionable design choice. Letting these agents running fully autonomous sounds like a heck of a waste of money. For now. I have no doubt we would be able to RL them to be even better software architect very soon.
I worked with real developers, who coded by hand like crap. And then they had cursor, and then Claude Opus. And they produced crap at a faster pace. And they don't know what their code means or how they work. They have been reduced mostly to reacting to pressure from boss and praying to Opus to fix the problem, even do things like adding K8s without knowing much, if at all, about K8s. Now that I work with agents only, I try very very hard to never lose control of the codebase like that.
What can you run on 12GB for real coding? I have 16GB and I'm already too squeezed. Can run Qwen 27B at IQ3XXS with 100k context, and I can push features with it, but it's not fast. But to be fair, Minimax at rush hour is just as slow, so.
1
u/Nefilim314 7h ago
My 12GB workflow is using Tiel Coder 35B Q4 but I had also used Qwen 3.6 35B Q5.
Basically I make the architecture decisions myself and my prompts are things along the lines of "make an api endpoint at /some/path in the foo module. The payload should be shaped something like { this } and the response should just be a 204 acknowledgement or a 400 error. The required fields are X, Y, Z. Use zod to validate the payload and generate swagger docs."
Stuff like this is seems to nail pretty consistently, but occasionally it'll do some silly stuff and I can ask probing questions about decisions it made and it'll rectify it.
It's a much slower process than just vibing out a whole feature, but the benefit is that I know exactly what the code is doing and why so the review process isn't just skimming through 35 pages of over-designed slop.
2
u/baron_von_noseboop 7h ago
Share your 12gb sauce? I'm on 4070 getting 20 t/s and 300 prefill with 60k context from qwen3. 6 35b a3b mtp q4 with q8_0. Speed is tolerable, but barely. Output quality has underwhelmed me... it's pretty poor compared to hosted models that cost << $1/M out tok, even with me feeding it small, well-specified tasks.
2
u/Nefilim314 7h ago
I've noticed Q4 3.6 keeps making some really silly mistakes. I know people really vouch for it 'being extremely close' but it seems to always have a little something thrown in that I don't trust it for any long agentic sessions.
Key settings for me:
- MTP on, max set to 3
- 128k context
- q8 kv
- MOE cpu offload set to 31 but occasionally can drop it down to 30
Settings for Tiel are very similar but cpu offload may be adjusted to fit mmproj if I actually need it. Q4 seems to do a good enough job unlike the standard 3.6 Q4. It still makes silly mistakes every now and then but the speed is good enough that it can fix it quickly.
Just using qwen3.6 Q5 just now and seeing 580 t/s prompt eval time and token generation at 48 t/s, but that number drops to about 40 during long sessions.
Side note: I'm using CachyOS and have configured practically everything to use the integrated intel GPU so that I can use the full VRAM for the model. I have to be more selective when I am doing work with Godot or Blender running.
Using podman and pointing at the unsloth Q5 MTP, I used this command:
podman run -d --name qwen36 \ --device nvidia.com/gpu=all \ --ulimit memlock=-1 \ -v "$HOME/Desktop/models:/models:ro" \ -p 127.0.0.1:8080:8080 \ ghcr.io/ggml-org/llama.cpp:server-cuda \ -m /models/mtp/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \ -a qwen3.6 \ --host 0.0.0.0 --port 8080 \ -ngl 99 \ --spec-type draft-mtp --spec-draft-n-max 3 \ --n-cpu-moe 31 \ --load-mode none \ -c 131072 \ -fa on \ -ctk q8_0 -ctv q8_0 \ -np 1 \ -t 8 \ --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 \ --reasoning-preserve \ --jinja3
u/senseven 11h ago
We are setting up systems to offload security irrelevant jobs to providers that rent gpu hardware. Sometimes we get a call by a random high level admin if we are willing to give them a 15 minutes quick shot to explain to them what we are doing because they don't understand our multi agentic workflows. That is the proper response to not knowing, not asking gpt three questions and getting hallucinated answers.
2
-7
u/timmeh1705 15h ago
Nah it's because we're all nerds trying to squeeze the life out of hardware.
The people who wrote this post probably have friends and fun outside activities to do
12
u/o0genesis0o 14h ago
IMHO, it's not excuse.
I check the link again, the found that the author dude was all snazzy on his profile with "provides clear guidance for the most complex AI use cases, ensuring clients can build and scale their development journeys with confidence." and then do something absolutely stupid like running full f16 unquantize, with vLLM, on one card. But when it comes to running two cards where vLLM is good, the dude use llamacpp.
If one poses as "expert" and take people money, one should do try to do things well. This article is nothing but stupidity. I mean for god's sake the guy got a R9700 and manage to serve an 8B model at 29tk/s decode, and claim "usable for interactive"? It's the card that can run 3.8 27B at good quant, full context, and get real work done, and this is what the "expert" got?
Leaving a pair of R9700 in the hands of this clown is a crime.
1
u/my_name_isnt_clever 7h ago
Then they should go outside with their friends and let someone who knows what they're talking about get paid to write an official article.
24
u/Muhlwa_Sholanke 18h ago
$18,775 and the headline test is an 8B model on a 32GB card. For that price you'd want at least one number that actually needs the hardware, not a 500/500 token prompt a single card could run in its sleep.
3
u/gh0stwriter1234 18h ago
If they were serious they would have tested Qwen 3.8 Flash Next... It runs passably even on my dual MI50 system, way slower than 27B though like 1/5th to 1/10th the speed.
2
u/stankmut 16h ago
This article was written before Qwen 3.8 flash next was released.
1
u/gh0stwriter1234 7h ago
Any company worth thier salt would have updated the article.. or released a new one... they are honestly kinda meh.
1
u/Savantskie1 11h ago
How are you running it on two MI50'S? I've tried and it's performance was abysmal.
1
u/gh0stwriter1234 8h ago edited 8h ago
It starts out at about 25t/s and degrades to about 4.5t/s at full context but its gradual
also I think -sm tensor does not work correctly for this model yet, so have been using -sm layer, also q8_0 on the kv cache I also have lazy loading enabled but not sure if I have it working correctly, in any case I do have enough ram it wouldn't matter anyway. 64GB on the video cards plus 128GB ram.
Accutally with my current settings it only degraded to 10t/s at 218k so far.
Also I am running The unsloth Q4 XS
12
18
u/Sexecute 17h ago
This is what happens when an LLM with a knowledge cutoff writes your research plan.
5
9
u/CatalyticDragon 19h ago
"Real AI workloads", you sure there, bud? Because I have 2x9700s and I've never run those workloads.
40
u/TableSurface 19h ago
They're in the business of selling hardware and support, not AI expertise
16
u/AprilWatermelon 18h ago
Their GPU encode/decode/scaling articles are very solid. I guess they are just not familiar with their new customers.
2
u/SkyFeistyLlama8 12h ago
Puget have been around for a long time and their graphics workstations for photo and video editing are solid. It's sad to see them failing so hard on LLM inference when the hardware is essentially the same.
13
u/CATLLM 19h ago
Poor excuse and low effort on their part. If they want to sell systems then do a better job of selling it maybe?
21
u/LetsGoBrandon4256 transformers 19h ago
A single R9700 comfortably runs everything up to 8B parameters. The 27B tier, which is currently the sweet spot for capable local inference, requires both cards working together.
Won't be surprised if there are buyers out there taking this shit at face value.
1
17
u/Serprotease 17h ago
It’s an AI generated report.
Not sure exactly their internal process but this convoluted way to say little is typical of sonnet/glm5.x . At the very list, the text of the report was AI guided/generated.
Even past that, testing fp/bf16 inference is just weird. No one does that.
Inference is almost always fp8 with downgrade to fp4 during peak times.
So… jumping bf16 with a note that the 27b model does fit all the way down to Q4 that’s… a choice. Especially when 9700 are designed for fp8 workloads.
Same with models and benchmarks. I can understand llama3 8b because it’s used everywhere, even currently in litterature/papers but they picked a lot of old models. Like, a lot.
Which goes to their issues with vllm. Vllm works with 2x 9700. Maybe not the nightly/latest one but it does work for the Qwen3.x series and older models. So picking old models and pointing that new vllm releases are not working out of the box is… correct but arguably not genuine as vllm does work with older stuff.
Also… no llama bench or equivalent? Why not using the standard to measure performance?
8
8
u/More-Catch-1331 10h ago
You're not reading it wrong. This is written by heavy Claude and OpenAI users. It's extremely stupid because at 64GB you would put a Q8 weight of Qwen 3.8 27b on one card and have oodles left over for cache, context, mtp, mmproj, everything. These people simply do not know what to test. Honestly it reads like someone put in a search query for "most used models" in Google and got back a result from a model with a cutoff at mid 2025 at best
6
u/Gargle-Loaf-Spunk 18h ago
that is embarrassingly bad. what a random mix of configurations and terrible comparisons.
4
4
u/FullOf_Bad_Ideas 9h ago
I think it's written by a business development manager with guidance from LLMs, they probably don't know any better.
0
-12
19h ago
[deleted]
10
u/LetsGoBrandon4256 transformers 19h ago
Ask your clanker for a better prompt. This shit is not fooling anyone.
155
u/Igot1forya 19h ago
I see stuff like this with techtubers too. "We built a 4-node super-cluster, but the scaling of our 9B parameter model didn't scale as well as we expected" and then proceed to test other tiny models that should only live on a single GPU/node as well as other idiotic configurations that never once push the hardware properly.