r/LocalLLaMA • u/Ne00n • 1h ago
Discussion Slow interference is great
No seriously, I kinda like it.
You have something to solve, you put it.
You know its gonna take like 20 mins to cook.
Every search adds another 30 minutes.
Yes I could boot up my debian on my gaming rig, run the same model at 10t/s + but why?
I rather let the poor server without GPU burn and run the same model at 2t/s and chill.
Its great, I love it.
28
u/Bulky-Priority6824 1h ago
20 mins to cook a task @ 2 tg/s ?
Bruh at 77 tg/s I'm hitting 23mins on average task.
What are you cooking?
5
2
1
14
4
u/ohrelia 1h ago
There's some truth to this. I used to have the GPT x5 plan but I would be constantly using it way too much. My life has improved again since going local. Now I just run Deepseek 0731 at 12-20tks, Qwen 3.8 flash at 12-25tks and GLM 5.3 flash at around 6-10tks. I prefer it immensely. Now I only have the cheap plan for 5.6 Sol and I use it whenever my local stuff is struggling.
Addiction is real!
3
u/bigattichouse 1h ago
Same.. bigger/slower model, and let it cook overnight on a task, get something interesting in the morning.
3
u/ProdoRock 1h ago
It gives you that punchcard feeling of 50s/60s computing. But now it’s early local ai on consumer hardware!
There is also this notion of optimizing. What else can I squeeze through this 2020 based M1?
2
u/SeriousPanic34 1h ago
I like the relatively slow 27 t/s of my qwen3.8 27b because i can read the thinking it is doing - sometimes it's enough to steer me in the right direction or make me realize something that i can interrupt the llm and add immediately. With cloud models they think so fast that you only really get to read the final outputs, which may not contain the interesting bits the llm found along the way
2
u/BorpMyGurd 1h ago
Power efficiency is a pig. Electricity is too expensive for that shit. It's hard enough to justify the cost of local inference as it is without throwing power away.
1
u/ryfromoz 31m ago
Looks at solar array with battery. Power is nothing to me but id still not match OP enthusiasm for that kind speed.
2
u/GregAbeI 1h ago
Big models better, always.
Things that need human checks should not move faster than humans can check.
1
u/TripleSecretSquirrel 1h ago
Increasingly, I agree with you!
Most of us here (myself included a lot of times), seem to gravitate toward trying to replicate the Claude/ChatGPT experience locally. Which is to say we’re looking for the smartest model and tinkering to get it to run as fast as we possibly can on enthusiast or consumer grade hardware. Optimization makes sense of course, but the reality is that unless I’m dropping $100k+ on hardware, I just cannot run a frontier model at cloud speeds. I could run it very slowly or I could run a smaller model much more quickly. Sometimes the right choice is the big slow model, sometimes it’s the small fast model.
I don’t think the goal needs to be to replicate the cloud API experience. I think the goal of squeezing more performance and intelligence out of our relatively limited hardware and models is really fucking cool, fun, and useful in and of itself. With the right harness, scoping, task definitions, plans, management, and loops, you can get incredible performance out of local models. I’m learning way more about how to effectively deploy models and agents this way! Plus it’s the best way to learn what models are capable of what tasks and when to invoke a larger slower model or (god forgive me) a cloud model.
1
1
u/SomewhereAtWork 1h ago
Yes I could boot up my debian on my gaming rig, run the same model at 10t/s + but why?
Turn it into a Debian gaming rig. Then you can do both.
Gaming on Linux actually doesn't suck anymore.
1
1
u/Mingay_cat 52m ago
I love my 0.7 tok/s qwen 27b and also vibe coding the shit out of colibri to just run models that barely fit on my disk at ridiculously low speeds to say I ran it locally.
1
u/AnnoyedAvocado21 45m ago
I have the ASUS DGX Spark clone so I have 128gb unified memory. Not shabby but not a speed demon that runs the biggest models either. I'm testing qwen38-27b-nvfp4 in the Deepseek harness and the Hermes harness and using ChatGPT to design the tests. I've explained to ChatGPT that I don't care about tokens, I don't care about token speed - I only care about the quality of the result. I'm giving it some coding tasks while I work on other projects. We're developing a personality profile and a communications profile. Qwen38 is very good at long tasks and tool use and uses vision as well while in the Deepseek harness. What I'm learning is there is a good way to prompt the model and a bad way and I'm generating a document I can use in the harness to ensure all request are formatted to get the best out of the model-harness pair.
Some talking head made the comment that the chat interface taught us wrong - it got us used to near immediate responses. Using agents is something where you prep, launch, and come back hours later - like baking a cake or cooking a Thanksgiving turkey.
Trust me - if I come into great wealth I'll upgrade my rig but until then I'll work to squeeze the best quality out of the rig I have.
1
u/No_Lingonberry1201 39m ago
Know what you mean, choom. A small feature takes 40 minutes, I might as well go and play some Factorio.
1
u/hobopwnzor 20m ago
Friction often makes systems better. People want instant results but they will often get better results from slower work, because it makes them think. If there's a job that will take 30 minutes they'll think before they submit. If it takes 5s but costs the same they won't.
0
0
u/Constant-Simple-1234 1h ago
I got addicted to speed. Even at 130 I am merely satisfied. Can get a better quality at 60, but it makes me impatient.
42
u/Max-_-Power 1h ago
Whatever floats your boat, mate