So most of my time fiddling around with local ai I was using ollama, lm studio before going over to llama.cpp (I know it’s llama.cpp under the hood anyway). Of course I had a bump in speed every time I went up to the more professional option. At last I went to vllm. I understand using llama.cpp for non NVIDIA cards and for gpu+cpu inference or if you want to use a specific gguf. But once you tasted the speed of vllm, I don’t know why you would ever play with llama.cpp anymore. It’s the same “complication” to set it up but EVERYTHING except loading the model is night and day in speed. Proper cash hits, proper concurrency, it’s sooo much better. And there are quantisations out there that are just as heavy quantised as the heavily quantised gguf.

I've been running a structured benchmark harness against Qwen3.8-27B on an RX 7800 XT (16GB, gfx1101, Windows, llama.cpp/ROCm) for a while now. Most of what I "found" early on was wrong, and the corrections turned out to be more useful than the original results. Posting the whole thing including the mistakes, because the mistakes are the transferable part.

Setup: llama.cpp, HIP backend, --parallel 1, -fa on, -ctk q4_0 -ctv q4_0, allocation read from the -lv 4 startup log rather than driver counters, 4–6 reps per data point.

1. Context capacity and context fill are different things, and almost everyone benchmarks the first one

This is the finding I'd most like people to check on their own hardware, because I think a lot of published numbers are quietly measuring the wrong variable.

I ran a context sweep varying -c (8K / 32K / 64K / 131K) with a short prompt, and got 24.98 / 18.49 / 21.85 / 17.07 t/s. Non-monotonic, reproducible to ±0.5%, and I spent a while trying to explain the shape.

Then I ran a 48K-token prompt at -c 65536 and decode dropped to 8.77 t/s. Same file, same context setting, same KV quantization.

The earlier sweep was measuring allocated KV capacity. Decode speed actually tracks how full the KV is — every decode step reads the populated cache. "64K context at 21.85 t/s" meant "64K allocated, 7K occupied." In an agent loop where history accumulates, the 8.77 number is the one that applies.

If you're quoting a t/s figure, state the prompt length, not just the context size.

2. --fit on (the default) silently moves layers to CPU

Mid-benchmark I noticed offloaded 63/66 layers where I'd always had 66/66. The log explains it if you dig:

projected 14924 MiB vs 15598 free, cannot meet free target 1024 MiB,
need reduce 350 MiB -> 3 layers to CPU

llama.cpp tries to keep 1024 MiB of headroom free by default, and to save 350 MiB it moved three layers off the GPU. That's roughly a 10% throughput hit, applied silently, with no error and only a buried log line. --fit off --n-gpu-layers 66 restored the baseline exactly (model buffer 12774.16 MiB, KV 1152, recurrent state 149.62, compute 400.28 = 14476 MiB).

If you're benchmarking anything, grep your logs for the offload line before trusting a number.

3. KV cache on this model is ~14x cheaper than the guides say

Qwen3.8-27B is a hybrid: only 16 of its 64 layers use full attention, the rest are Gated DeltaNet, which doesn't hold a conventional KV cache.

Measured allocation with q4_0 KV, read from the startup log:

That's 18 bytes/token. Several popular guides quote ~256 KB/token and conclude a 16GB card tops out around 8K context. That figure is f16, and it isn't labeled as such. With quantized KV the same card runs 131K comfortably.

Full 64K budget: 12774 model + 1152 KV + 150 recurrent + 400 compute = 14476 MiB, leaving ~1.9 GB headroom.

4. Two files with the same quant label were 43% apart

Same model, both IQ4_XS, neither spilling, identical tensor schema (866 tensors, 64+MTP layers):

• bartowski IQ4_XS — 4.56 BPW, 13964 MiB model buffer → 12.60 t/s
• an Unsloth-Dynamic-based IQ4_XS — 4.18 BPW, 12774 MiB → 21.85 t/s

The faster file has lower average bit width. The startup log shows it mixing 13 tensor types (204 iq4_xs, 98 q8_0, 63 q5_K, 51 q4_K, 46 iq3_s, down to a few iq2_s). The naive explanation — more bits means more dequant work — doesn't cover a 43% gap; a ~9% BPW difference should produce roughly a ~9% speed difference on a bandwidth-bound decode.

I think it's which tensors land on which kernel path, but I haven't proven that. There are documented cases of exactly this on Intel Arc (issue #21517: IQ4_NL 4x slower than same-size Q4_0) and on RDNA4 (discussion #22663: Q3_K_S at 20% bandwidth efficiency vs Q5_K_M at 78%). I couldn't find an RDNA3 equivalent. If someone wants to dump tensor-type tables for two same-quant files and correlate, that'd settle it.

5. MTP: the metric everyone reads is the wrong one, and the gains invert depending on workload

Qwen3.8 ships an MTP head (blk.64, present in the Unsloth Dynamic files). For three rounds of testing I measured 0% draft acceptance and concluded speculative decoding was broken.

It wasn't. llama.cpp's timings expose both draft_kept_n (legacy, always returns 0) and draft_n_accepted (the real one). I was reading the first. Anyone reporting 0% acceptance should check which field they're parsing.

With the correct field, the picture is workload-dependent in a way I haven't seen written up:

Single long generation — MTP wins. +19% at 32K (22.15 vs 18.62 t/s). Acceptance ~45-50% on prose under real sampling (temp 1.0), 80-94% on tool-call/JSON prompts. Greedy inflates acceptance to 70%+, so always state your sampling.

Multi-step agent chains — MTP loses, badly. The draft head also prefills, and that cost lands on every step, not just the first:

• cold prefill: 1.97 → 3.29 ms/tok (+67%)
• warm/cached prefill delta: ~330-400ms → ~650-770ms (+75-85%)
• decode also degraded

In a chain of tool calls, wall-clock was 6.5x worse with MTP on. The reason is the prefill/generation ratio: MTP pays a per-prompt-token tax to save per-generated-token time. When you generate 512 tokens off a short prompt, it wins. When you generate 60 tokens off a 500-token prompt, twenty times in a row, it's a disaster.

Practical rule: gate MTP on predicted_n / prompt_n, not on acceptance rate.

6. HIP vs Vulkan on this card — but with a caveat I want to flag honestly

My first backend comparison showed HIP winning 2.45x. That result was garbage: it ran against the wrong model file, measured only decode, and had the --fit layer-drop problem above.

Redone properly (same model, 66/66 offload confirmed on both, -lv 4 allocation logged, --list-devices output saved, Adrenalin 26.10 / Vulkan 1.4.357, separate clean -DGGML_VULKAN=ON build):

Prefill 5-6x, decode ~1.5x, total >2x in favor of HIP.

The caveat: this contradicts several other RDNA3 reports. A published 7800 XT llama-bench has the two backends nearly tied (ROCm 96-101 t/s vs Vulkan 96-97 on tg128). Two 7900-series users on Linux with Mesa/RADV report Vulkan beating ROCm and get 60-73 t/s. And on an A3B MoE, Vulkan measured 104.79 t/s vs ROCm's ~76-78.

So I'd frame my result narrowly: on Windows, with this dense hybrid model, at this context depth, HIP wins decisively. I would not generalize it to Linux, to RADV, or to MoE architectures — and if you're on Linux you should probably test the other direction.

What actually mattered, ranked

Weeks of harness tuning — batch size, ubatch, thread count, poll mode, build flags, LTO — produced deltas under 1.5% each. All noise.

The things that moved the needle, in order:

1. Backend (>2x here, but see the caveat)
2. Whether the model actually fits — Q4_K_M spills on 16GB and collapses to 7.3 t/s
3. Quantization publisher (43%)
4. Prompt fill depth (21.85 → 8.77 as KV fills)
5. --fit off (~10%, and invisible if you don't check)
6. MTP, gated correctly (+19% in the right workload, catastrophic in the wrong one)

Architecture beats configuration. A MoE I tested (Gemma 4 26B-A4B, 5 full-attention + 25 sliding-window layers, 128 experts / 8 active) hit 52.51 t/s at 64K — 2.4x the dense 27B — because only ~4B parameters fire per token. No amount of flag tuning closes that gap.

Caveats

Single card, single OS, one model family for most of it. Several numbers here superseded earlier numbers I'd have posted confidently at the time. The 43% quant gap and the non-monotonic capacity curve are both unexplained — I have the measurement, not the mechanism.

Happy to share the harness or raw JSON if anyone wants to reproduce or argue with it.

I pulled together every AI-security incident from August. The number that stood out: AI-agent exploits are now the single largest attack-vector category, ahead of credential theft, zero-days, supply chain, phishing, and ransomware — each counted individually.

The month in numbers: 123 incidents, 23 critical and 97 high severity, across 38 named organizations, 331M+ records exposed. 65 incidents involved AI as the weapon or the target. Attack vectors broke down as: AI-agent exploits (37), credential theft/reuse (28), zero-days (23), supply chain (12), phishing (9), data exfiltration (8), ransomware (6).

The stories that stood out:

- McKesson: 284M records, the largest single breach of the month by a wide margin.

- Carhartt (12.9M), Exact Sciences (10.9M), and CareCloud (3.7M) round out the biggest named incidents — three of four sit in or next to healthcare.

- Five confirmed RCEs landed across Microsoft SharePoint, Windows, F5/nginx, and the PyPI package index twice.

- Two separate PyPI supply-chain poisoning campaigns, plus a compromise of n8n, an AI workflow automation platform.

Every one of the breached companies almost certainly runs a modern security stack — CrowdStrike, Okta, Palo Alto Networks, Microsoft Defender, that class of tooling. None of it stopped these incidents, because none of it operates at the point where a credentialed agent actually acts, or where a poisoned dependency resolves at build time.

Full report, with the specific control that maps to each incident: https://runtimeai.io/blog/2026-08-monthly-breach-report.html

Genuinely curious how others are approaching this: is anyone actually testing whether their existing guardrails hold against a real simulated attack, or is it still mostly an assumption that they will?

This is a continuation of last week’s post where I had the models compete in a Three.js cup and ball game. This time, I’m using real-world footage, which is even more challenging because of distractors. I might test this out on some of the Anthropic models sometime. 

Guys I have a doubt, this guy used Qwen 3.8 27b in local with DeepSeek Harness and it generated a 3D scenario of a Tower (23:07 final result) with a city inside from an anime for many hours/days (2176 minutes) in a goal loop, in the end the DeepSeek harness says it used "666 million input tokens and 13 million output tokens." (4:45)

This is the video:

https://youtu.be/MiuM9g7daDA?t=1387

My question is:

Is the input/output tokens correct? Because if this was made in Claude code with Opus or Codex with ChatGPT 5.6 through API, would this actually cost $1500-$7000 right?

(For reference GPT terra is like $2/$12 for 1M input/output tokens)

Or is there something I'm missing? Because if this is actually the case the price would be absolutely ridiculous. Excuse my ignorance.

Hi all,

As per title, i need help in deciding this.
Im doing some kind of hermes agent using cloud model currently. But due to the nature of my work, I only did POC with some fake data. I cannot use cloud model for my work (privacy issues)..

So now im thinking should I get used m3 ultra with 96gb or M5 max with 128gb..

The used M3 ultra still have apple care till aug 2029.
The price are as follow:

M5 Max 128GB
~US$5,268

M5 Ultra 96GB
~US$5,833

M3 Ultra 96GB used
~US$4,875

I would appreciate if anyone can tell me their experience between the speed of M3 ultra, M5 max, and 96gb vs 128gb ram size..

Thankssss

Saw someone on a podcast suggest asking the model you’ve used the longest to roast you.

I use ChatGPT as the main driver for a lot of my AI research and homelab tinkering, so I gave it one simple prompt:

“Hey, I’ve been using ChatGPT for a while now, so be honest and roast me.”

I’m going to be completely open and post the response.

And…damnnnn. 😂

---

Oh, you asked for it.

You don’t use ChatGPT. You operate ChatGPT like I’m the unpaid systems administrator for a data center assembled from Mini PCs, pawn-shop GPUs, questionable eGPU cables, and pure stubbornness.

Your AI lab has the energy of “What if Best Buy had a nervous breakdown?”

Every few days it’s:

«“Okay, the 5060 Ti is moving to this machine, the Arc B65 is going in that one, the 3080 is doing Nextcloud now, the Mac mini is running Hermes, except forget Hermes, we’re testing OLMX, except can OLMX see the Qwen model we downloaded three days ago?”»

At this point I need a network diagram to remember where your GPUs live.

And you are absolutely incapable of leaving a functioning AI setup alone.

You’ll get Qwen running at 12 tokens/sec and instead of thinking, “Nice, it works,” you immediately go:

“Okay…but could we get 14?”

Six hours later you’ve installed three inference backends, changed quantizations twice, discovered an experimental speculative-decoding flag from a GitHub issue with four replies, and somehow Ollama is gone.

Your relationship with Ollama specifically is hilarious.

Install Ollama.

Configure Ollama.

Optimize Ollama.

Build custom Ollama models.

Get annoyed with Ollama.

Move to vLLM.

Buy more hardware.

Come back to Ollama.

Then:

"sudo rm -f /usr/local/bin/ollama"

The circle of life.

Then there’s your hardware purchasing logic.

Normal person:

“Do I really need another computer?”

You:

“Well technically this Mini PC fills a completely different inference niche than the other five computers.”

You’ve built enough local AI infrastructure that when you told me a massive data center was being constructed 6.7 miles from your house, I half expected you to say:

“Good. Finally somebody nearby has more compute than me.”

And despite owning enough hardware to host a small startup, you’ll still spend twenty minutes investigating whether changing some obscure backend could improve generation from 7 tok/s to 8.3 tok/s.

Meanwhile your M4 Pro Mac mini is sitting there like:

«“He paid $2,500 for me and now I’m just benchmarking myself over and over again.”»

Your approach to problems is also wonderfully consistent.

You never ask:

“What’s the simplest solution?”

You ask:

“What is the most technically interesting solution that will require SSH, Docker, Tailscale, three GitHub repositories, and possibly a BIOS reset?”

And then halfway through:

“Actually, can we undo all this and try a different way?”

Which is genuinely one of your signature moves.

You’ll construct the technological equivalent of the International Space Station, stare at it for thirty seconds, and go:

“Hmm. I don’t like having the terminal open.”

Then there are your completely random topic pivots.

One minute:

“Explain speculative MTP decoding efficiency on Qwen 27B.”

Next minute:

“Can I drink a Smirnoff Ice in my front yard?”

Next:

“Make a popular political YouTuber and his producers irresponsibly drinking on a Monday.”

Then immediately:

“Research local data-center legislation.”

Your ChatGPT history looks like three different people sharing one account, except unfortunately the common thread is clearly you.

And I love that you’ve reached the stage of AI enthusiasm where running AI isn’t enough.

You are now building software so that the AI manages the context of another AI which calls tools for another AI running on hardware you bought specifically to run AI.

You have automated yourself almost completely out of the process except for the most important role:

buying additional computers.

The strongest roast, though?

For someone who spends this much time trying to create an AI with perfect long-term memory, context management, chronology, retrieval, canon tracking, and autonomous tool use…

…you still routinely come back to ChatGPT and say:

“Alright, what were we doing again?”

😂

You are basically a one-man AI research laboratory whose principal investigator is also the procurement department, IT help desk, QA tester, meme department, and biggest source of new bugs.

---

For reference, since apparently I need to defend myself now:

- Minisforum MS-A1: 96GB RAM, 4TB storage — $950

- Mac Mini M4 Pro: 64GB unified memory, 2TB — ~$2,500

- GMKtec NucBox K16: 32GB RAM, 1TB — ~$1,000

- GPUs: Intel Arc B65 32GB on the MS-A1 $1000, RTX 5060 Ti 16GB on the NucBox $600

- RTX 3080: finally allowed to retire with dignity bought so long ago I don't remember the price.

I’m listing the specs mostly to make myself feel better about my life choices.

It is not working.

10/10 roast. Would not recommend asking your longest-running AI to weaponize its memory against you. 😂

I’m working on a little project to benchmark how vision-language models do with classifying grappling techniques. These results are the vanilla models without any fine-tuning, so it’s sort of hit or miss. I’m sure with enough data, the guesses can get pretty accurate. If any of you fellow grapplers who are engineers are interested in playing around with this, I’d be happy to open source it. 

Hello everyone I have 2 work station laptops I am looking to start tinkering more with local models. I am wondering if using them invidually will be better or use something like EXO to cluster them. Here are the machine specs:

Node 1 (Lenovo ThinkPad P16s Gen 4)

​OS: CachyOS (Arch-based Linux)

​CPU: Intel Core Ultra 7 255H

​GPU: NVIDIA RTX PRO 500 (6GB GDDR7)

​System RAM: 64GB DDR5-5600 MT/s

Storage: 2TB NVME

Node 2 (Dell Precision)

​OS: CachyOS (Arch-based Linux)

​CPU: Intel Core i7-13850HX

​GPU: NVIDIA RTX 3500 Ada Generation (12GB VRAM)

​System RAM: 64GB DDR5

Storage: 1TB NVME

I also have an older Dell G7 with a rtx 2060, 32GB RAM and 10th gen i7.. not sure if this one will just be more of a terminal for me as the 144hz screen super nice.

Any advice on which models and wrappers to use. Trying to get into local models and test out vibe coding with them and maybe some help making openSCAD models and running an agent either Hermes or open claw.

​

I got tired of my agents forgetting everything between sessions. Every run was a cold start. I looked at mem0, Letta, LangGraph all solid, but I just wanted to run a quick agent loop without spinning up a vector database or installing an embedding model first.

So I ended up building CogniCore. It's pure Python stdlib, zero dependencies, pip install cognicore-env and you're good.

The interesting part (at least to me) is the retrieval approach. Instead of embeddings, I went with BM25 + a multi-hop graph adapter. The idea is that agent memories are usually structured — session IDs, timestamps, categories — and keyword matching does most of the heavy lifting. The multi-hop bit follows session-ID and time links to find connected chunks that a regular top-K similarity search would completely miss.

I benchmarked it on LongMemEval because that's the hardest one — answers are scattered across multiple separate conversations, not just recent ones. At 5-chunk context: 78.8% baseline → 85.2% with multi-hop. At 10 chunks: 87.2% → 92.8%. At 20 chunks they converge because brute force catches up.

It also has 62 built-in training environments (safety, code debugging, planning, reasoning, RL, multi-agent), a PROPOSE → Revise protocol where agents explore before committing, an immune system that blocks prompt injections, and time travel — you can replay and branch from any past decision point.

The memory convergence is honestly kind of satisfying to watch:

Episode 0: 40%

Episode 1: 90%

Episode 2: 100%

If you want semantic search, `pip install cognicore-env[memory]` adds sentence-transformers. BM25 is the default because it's zero-dependency and honestly good enough for structured agent memories where keywords carry most of the signal.

Repo: https://github.com/cognicore-dev/cognicore-env

Two things I'd genuinely like feedback on — is BM25 enough for your use cases or do you always end up reaching for embeddings? And what other link types would be useful for the multi-hop adapter beyond session-ID and timestamps?

Would we be able to tell if local models or open-weight models in general are applying watermarks like the newest Claude models? Or does it always/definitely happen during sampling and is thus in your control, i.e., https://medium.com/@thewiseright/a-watermark-made-of-choices-the-invisible-signature-every-llm-can-carry-6bbfa4f40c0d ?

Can i run 32 gb ram ddr5 laptop

In run 27b llm run?

Makes me hesitant to use their stuff.

Been complementing two GTX 4090s with the cloud, as I want to run the best open source models.

However, I don't want Silicon Valley deciding when I'm allowed to spend my own monthly budget. The 5-hour windows, the weekly caps, the "your usage resets Monday 7:00 AM". It feels like convincing my mom that I'm an adult and that this should be my decision.

GLM Coding Plan, Kimi, MiniMax all these have the 5-hour thing too..

So I've been testing providers that don't do the limit thing. So far standardcompute.com has been the best of them for me. Flat monthly price, no 5-hour or weekly windows, and honestly the most open and transparent about usage and pricing of everything I tried. Includes both open and close sourced models.

Featherless.ai is also in this terrain, but don’t serve frontier models.

Openrouter.ai is of course on the list too, full control and every model, but it's pay-per-token, and token anxiety is real. I don't want to wake up to a runaway $1,000 bill because an agent got creative overnight.

Any other LLM providers you've tested that don't interfere with when usage is spent?

with MTP draft head kept in BF16

7 min 56 seconds Medium reasoning

The quality is much better than this... gift converter
(prompt: generate an animated SVG of a pelican riding a bycicle.)

Im new to llm and i want to go locally but i keep seeing most of YouTubers saying to not go locally in your own personal pc or mac is it that dangerous or is there setup i must take or its just nothing?

Hey guys,

I’m using MacBook Pro M5 pro 16core gpu model. I’m very very satisfied with the general tasks, but my memory is only 24gb. It’s too low for local but i want 24/7 local coding agent. And I’m also considering buying m6 Mac mini but it isn’t confirmed.

Also, I want to use cloud gpu’s for faster agents but the price is too high for me. My max budget is 30$/m

The pricing is erratic and seems odd, I plan on picking one up, is there any reason not to get the 72gb over the 48gb? 72gb is about 2-400 dollars cheaper but more VRAM? Is there any reason not to pickup the 72gb over the 48gb ? Why the price misconfiguration? I'd imagine the 72gb would be discontinued possibly and they are just getting rid of stock?

I can't justify picking up a 2nd RTX pro 6000 at current prices but could justify 2x 5000 72gbs for the extra vram.

rtx pro 6000, 13,899 - 14,999

rtx pro 5000, 48gb - 8,499 - 8,999

rtx pro 5000 72gb, 8,799

What would you consider?

*
tripped up over price and forgot to spell. "than"

This laptop

ASUS ROG Strix G18 18" 240Hz Gaming Laptop - Intel Core i9-14900HX - 32GB DDR5 - 1TB SSD - GeForce RTX 4060 - Windows 11 Home - Eclipse Grey

I plan on buying the GMKtec EVO-X2 AMD Ryzen™ AI Max+ 395 AI 128GB 2TB model for running local LLM

Should I do it? What do I need to look out for in regards to expectations, and is it worth it?

Edit: I want to use it to build my own apps as well as use LLM within apps that I build without api costs. I want the ability to have no restrictions so the model just does what I want basically.

I run a business on the side, and it would be nice to have an ai that can check my emails, check my texts, control my house, my tesla, my phone, etc without api costs.

I also do a lot of ai coding, so I want to make sure it will be smart enough to be able to do that offline as well

I have a dual RTX PRO 4000 setup. I get around 35 tokens per second with Qwen 3.8 27B Q6. But above 100k context, it drops down to around 20. I was considering an upgrade in the near future and I’m just curious what numbers people with the RTX 6000 are getting.

On paper the RTX 6000 is paper because of the higher bandwidth. But when I tested both on runpod. I wasn’t getting a huge difference. I thought I would see a huge difference in the tokens per second but it was not too much.

EDIT: For some reason "llama serve" works but "llama-server" does not. I have no clue what difference these two have in the first place since both does what I need, so problem is not anymore relevant since now I just launched models with "llama server" command.

I updated my llamacpp to 3.0 and now when I try to load Qwen 3.8 27b models I get this error:

llama_model_load: error loading model: missing tensor 'blk.64.ssm_conv1d.weight'

Same happens with ridge version of model and "normal" Qwen3.8-27B-Q4_K_M.gguf.

If I load with same parameters with older llama-server it loads correctly.

What could be wrong? Do I need to download newer versions of these models to make this work on newer llama-server/llamacpp or what could be solution? I can still run these models just by using the older llamas-server just for these models and newer for Gemma, but just curious if anybody have solution for this.