A mildly interesting video about which models can run on the Mac Mini and Studio. As expected current foundational models such as Kimi K3 1.4 TB, if they were even available to run locally, won't fit. Assuming it were available locally it may be possible in the next couple of years with the rapid increase in hardware capabilities - M7, M8?

https://www.youtube.com/watch?v=eDoWKgFqRM4

I was curious to test Qwen3.8 27B from OpenRoter to see how well it could performs on code on my local machine.

I’m also testing the Qwen3.8 Flash version from OpenCode GO.

While Max is also included, the 27B version is not.

I was surprised by the pricing.

For context: I was working with Qwen3.8-27b in LMStudio and asking it to do some work on my contacts via iMCP. My name and therefore email contains a female second first name.
Somehow Qwen mentioned in its thinking process that it thinks there is something fishy going on with my loyalty to my girlfriend but decided it won‘t dig into it 🤣

Guess whatever measures they took in model training to avoid misalignment is working

I kept running into the same problem with local AI tools: running the model was the easy part, but using it for actual work still meant jumping between documents, email, browser tabs, notes, and separate chat apps.

So I built Vyact, an open-source desktop workspace that connects those pieces around the model you choose.

The workflow I wanted was:

- Find a GGUF or MLX model that fits the available RAM and VRAM

- Download it and run it locally through llama.cpp, llama-swap, or MLX

- Index documents and inspect the exact passages retrieved for an answer

- Bring Gmail and Google Drive files into the same conversation

- Continue working with webpages through a connected Chrome extension

The Chrome extension is an important part of the project. It can summarize the current page, translate foreign-language pages, and send either selected text or the entire page to Vyact as conversation context. This makes it possible to ask follow-up questions without repeatedly copying and pasting content between the browser and a separate chat app.

It also includes a Netflix language-learning mode with dual subtitles, subtitle navigation, repeat playback, automatic pause controls, and short AI explanations focused on the language areas the learner has selected as difficult.

These browser workflows do not always require a large flagship model. By choosing a smaller quantized local model that fits the machine, practical tasks such as page summarization, translation, selected-text questions, and subtitle explanations can remain useful on more modest hardware. The actual speed and answer quality naturally depend on the selected model and device.

The screenshots show the current app rather than mockups.

Vyact supports GGUF models on Windows and Apple Silicon Macs, with native MLX support on Apple Silicon. Hosted providers and custom OpenAI-compatible endpoints are optional.

When a Vyact-managed local model is selected, the chat context is not sent to an external AI provider. The project is released under AGPL-3.0.

GitHub:

https://github.com/vyact/vyact

I’m trying to decide what to improve next. For people who already use local models, which part of the workflow is still the most frustrating: choosing a model, document retrieval, connecting real work context, browser integration, or initial setup?

Hey guys,

What are you guys using small LLMs for?

Except coding.

Thanks.

I'm pretty happy with my setup below, which is my daily driver for primarily OpenClaw, but can we push Qwen 3.6 35B A3B GGUF any harder on our weak hardware without sacrificing intelligence and maximum context? Or with a different model (which I doubt right now)?

Show us your t/s and run params on similar hardware!

I've maxed out what I can do at an average 27 t/s eval and 700 t/s prompt eval with the following setup which is optimized for OpenClaw with massive 14K+ token system prompts and multi-agent context switching. If I optimize purely for speed, I can reach 30 t/s, but I made a few trade-offs for OpenClaw stability. I also find leaving MTP off to be better over long runs even though at short runs I can get 30 t/s as well. I've found it not worth the +3 t/s.

• GPU: NVIDIA GeForce RTX 3070 Ti (8 GB VRAM) with 1920 MHz overclock @ 925 mV (0.925V) locked curve and +800 MHz (GDDR6X)
• CPU: Intel Core i9-9900KF (8C / 16T @ 5.1 GHz overclock)
• RAM: 32 GB DDR4-3200 @ CL16 (XMP 2)
• MOTHERBOARD: ASUS ROG STRIX Z390-E GAMING
• PCIe Bus: Gen 3.0 x16
• OS: Windows 11 / 64-bit
• LLAMA: llama.cpp-turboquant-mtp (Tom Turboquant beats Atomic Turboquant for me by 10 t/s)
• MODEL: Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf (Non-MTP)

-m "Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf"
--mmproj "mmproj-BF16.gguf"
-np 1
-t 8
-tb 16
-b 4096
-ub 2048
-c 131072
--n-gpu-layers 99
--n-cpu-moe 999
--no-mmap
--mlock
--cache-type-k turbo3
--cache-type-v turbo2
-fa 1
--chat-template-file "froggeric-Qwen-Fixed-Chat-Template.jinja"
--jinja
--chat-template-kwargs '{"preserve_thinking": true}'
-fit off
--kv-unified
--reasoning-format deepseek
--ctx-checkpoints 32
--checkpoint-min-step 512
--cache-ram 20480
--image-min-tokens 1024
--no-mmproj-offload
--no-warmup

I can also hit 30 t/s by adjusting --n-cpu-moe to 31 or 32 without mlock, but it ultimately crashes over long runs, and with the above params, it's rock solid.

Hello,
I am reading a lot of positive comments about Qwen3.8 27b as a local coding agent model.

I preordered a MacBook Pro M5 (not M5 Pro CPU) with 32GB RAM. Has anyone benched Qwen3.8 on this MacBook and can tell me their t/s and general experience with working with it? I am planning on using llama.cpp

I'm afraid that I should have used some more money to get the M5 Pro with 48GB...

Hey everyone,

I have a consumer app that uses fully-local AI to help people practice speaking a language privately and securely. The full conversation cycle is local:

• STT - Apple on-device SpeechAnalyzer
• LLM - Gemma 4 E4B
• TTS - Supertronic 3

The number 1 feedback I get from the average user is "I'm not downloading a 2.5-3GB model to my phone."

I implemented a cloud option that pings a serverless GPU endpoint that runs Gemma 4 so they don't have to download it (as opposed to simply calling a 3rd party inference API). But I originally built the app for the fully-local approach because I believe privacy is wildly underrated.

So my question is: If you've offered multi-GB models on a user-facing app, what's the best way to get them onboard with it?

Thanks!

Link to app if you want to check it out

The pricing is erratic and seems odd, I plan on picking one up, is there any reason not to get the 72gb over the 48gb? 72gb is about 2-400 dollars cheaper but more VRAM? Is there any reason not to pickup the 72gb over the 48gb ? Why the price misconfiguration? I'd imagine the 72gb would be discontinued possibly and they are just getting rid of stock?

I can't justify picking up a 2nd RTX pro 6000 at current prices but could justify 2x 5000 72gbs for the extra vram.

rtx pro 6000, 13,899 - 14,999

rtx pro 5000, 48gb - 8,499 - 8,999

rtx pro 5000 72gb, 8,799

What would you consider?

*
tripped up over price and forgot to spell. "than"

So most of my time fiddling around with local ai I was using ollama, lm studio before going over to llama.cpp (I know it’s llama.cpp under the hood anyway). Of course I had a bump in speed every time I went up to the more professional option. At last I went to vllm. I understand using llama.cpp for non NVIDIA cards and for gpu+cpu inference or if you want to use a specific gguf. But once you tasted the speed of vllm, I don’t know why you would ever play with llama.cpp anymore. It’s the same “complication” to set it up but EVERYTHING except loading the model is night and day in speed. Proper cash hits, proper concurrency, it’s sooo much better. And there are quantisations out there that are just as heavy quantised as the heavily quantised gguf.

I have a dual RTX PRO 4000 setup. I get around 35 tokens per second with Qwen 3.8 27B Q6. But above 100k context, it drops down to around 20. I was considering an upgrade in the near future and I’m just curious what numbers people with the RTX 6000 are getting.

On paper the RTX 6000 is paper because of the higher bandwidth. But when I tested both on runpod. I wasn’t getting a huge difference. I thought I would see a huge difference in the tokens per second but it was not too much.

I am considering getting this build for my work. I have a budget of around 60k. Mostly will be running a mixture of small models like qwen 3.8 27b, with expectation to also be able to run oversized models like GLM 5.3 at quants like Q4.

Do you have any thoughts or recommendations different options?

Here is the parts list I am looking at:

AMD Ryzen Threadripper PRO 9985WX — 64C/128T — B&H — $7,894.00

ASUS Pro WS WRX90E-SAGE SE — B&H — $1,299.99

2× PNY NVIDIA RTX PRO 6000 Blackwell Max-Q — 96GB ECC GDDR7 — B&H — $16,999.99 each / $33,999.98 total

TEAMGROUP T-Create Master 384GB — 8×48GB DDR5-6000 ECC RDIMM — Newegg — $10,199.99

2× WD_BLACK SN850X 8TB PCIe 4.0 NVMe SSD — Best Buy — $1,699.00 each / $3,398.00 total

Samsung 990 PRO 2TB PCIe 4.0 NVMe SSD — Best Buy — $389.99

MSI MEG Ai1600T PCIE5 — 1600W 80+ Titanium PSU — B&H — $619.00

Sliger CX4200a 4U Rackmount Chassis — Sliger — $289.00

Asetek 836SA-M1 360mm Threadripper/TR5 AIO — Sliger — $250.00

3× Noctua NF-A12x25 PWM 120mm Fans — Sliger — $75.00 total

Sliger GDRAIL-20XX-B General Devices 20" Rack Rail Kit — AVADirect — $123.04

CyberPower PR1500LCD Smart App Sinewave UPS — 1500VA / 1500W — B&H — $700.95

Ubuntu 24.04 LTS — $0.00

Total: $59,238.94 before tax (tax isn't real, tax can't hurt us (shhhhhh let me live in delerium))

I mashed Qwen3.5 4B with Qwen3.8 flash ngram table by simply adding the lookup from table.

Some time ago I read this article here https://dnhkng.github.io/posts/rys/ and when I saw the Qwen3.8 flash ngram release I immediately though about it and wondered if we could transplant the ngram table to a smaller model and if it will improve performance.

Today I made this work, not much but it's honest (coding agent) work 😄 and had some fun doing it. In short I downloaded the unsloth qwen 3.5 4B Q8 quants and mashed the ngram table from Qwen 3.8 and adding the values

Does it do anything? Weel model seems to be doing well on humanitys last exam question (manually input and verified, about 14 correct / 19 question) (see edit, bad data 😭)

I'll try to run an actuall benchmark tomorow (if any one can help with some guidance on this I would be gratefull).

Right now I was just excided to tell somone that this seems to work and the model is coherent out of the box without any training, just doing residual_before_block_2 = base_residual + 0.5 × ngram_lookup.

Next I would probably try to graft the matrix from qwen 38 that actuially weighted the ngrams from the input. Weights are here https://huggingface.co/dburner/Qwen3.5-4B-Q8_0-FlashNgram-MTP but require a llama cpp build to run.

EDIT: guys sorry to dissapoint, just woke up and i tried to run questions actually from https://huggingface.co/datasets/cais/hle directly, seems it cant get a right answer. Il keep working on this once I get some more free time.

Last night I've been running in questions from gemini. I asked Gemini to give questions and answers from HLE, I thought it was getting them from source. Should have mentioned take this with lots of grains of salt.

EDIT2: did a bit of digging Qwen 3.5 4B and Qwen 3.8 Flash share almost same vocab (some exceptions) but tokens do map to same indexes and qwen 3.8 flash uses hashes over index values. BUT the embeddings do not match at all. My best guess setting alpha to 0.5 does not really affect the inputs that much, tried the same contaminated questions on base qwen 3.5 4b and seem to respond the same. I am trying now to add an adapter and finetune only the adapter part in a similar style of the Qwen 3.8 flash. So far I do see drop in holdout loss but its only on 256 context (qwen 4b and ngram tables are frozen). Cant really fit much in 16GB VRAM. I've been training for an hour still seeing improvements in holdout loss.

Weights are here in for the adapter version but probably training is still required.

https://huggingface.co/dburner/Qwen3.5-4B-Q8_0-FlashNgram-PLEAdapter-MTP

Building a local AI that answers from my boat's manuals with no internet. A few days of work so far. It gives correct, sourced answers, but a question takes 12 minutes. Posting what I've tried so people can tell me what to try next.

To clarify i dont have any clue if 12 minutes even is good or bad, but i give it a shot for maybe some tips and tricks 😄

**Hardware:*\* MacBook Pro M2 Max, 32 GB. No other options on the boat.

**Software:*\* Bionic (LM Studio's new agent app). No embeddings, no RAG. The model greps and reads files in a project folder with tools. Suits manuals well.

**Dataset, ~1.7 M words of plain text:*\* Volvo Penta 2003 workshop and operator's manuals, 120S saildrive manual, Victron and B&G manuals, MOB1, inReach, Ship Captain's Medical Guide, Calder, Casey, Toss, RCC Atlantic Crossing Guide, NGA Sailing Directions split per leg, plus my own notes and checklists.

**What I've tried*\*

Data prep

- pdftotext for everything. Layout mode for engine manuals so the technical data tables keep rows together, reading-order mode for two-column prose. This mattered more than expected.

- A check script scoring what fraction of tokens are dictionary words. Caught a Volvo PDF with a text layer that was 50 % garbage. Replaced it.

- Scanned operator's manual OCR'd with macOS Vision, then checked against the page image.

- Wiring diagrams and pilot charts rendered to PNG for the vision model. PDFs without a text layer are invisible otherwise.

- A hand-checked KEY_NUMBERS.md: torques, clearances, oil and coolant volumes, intervals, and a list of numbers that are NOT in the manuals so the model doesn't invent them.

- AGENTS.mds with a full file map and a search recipe: read KEY_NUMBERS first, grep one distinctive word, stop at the first confirmed hit, answer "Not in the attached documents" rather than guess. Cut the tool rounds a lot and the answers got noticeably better.

- Split the Sailing Directions per leg. All together they drown everything else.

Models

- Qwen3.8 27B, 4-bit MLX. Correct, cites file and section, refuses when the answer isn't there, spotted a unit slip in my own notes. 5 to 12 min per answer. ~75 tok/s prefill, ~12 tok/s generation. Reasoning row not exposed for this model in the app, so I put the template's own "reasoning effort low" sentence into AGENTS.md instead.

- Same model as GGUF with MTP on: prefill 93 tok/s, generation 11 tok/s. MTP accepted 170 of 250 draft tokens and gave zero speedup. Bandwidth-bound.

- Qwen3.5 2B: read "D boat/" in a directory listing as a folder called D and called list_dir on it 130 times.

- Qwen3.5 9B: said "let me search the operator's manual instead" and sent the identical wrong search call 15 times in a row. Narration right, tool arguments stuck.

- Qwen3.6 35B-A3B: downloading now.

Settings

- Root model pinned so a small model can't be picked by accident.

- Exploration sub-agents off. Extra runs of the same model on one Mac just add waiting.

- Context auto-fits to ~42k on MLX and ignores the setting, known bug. GGUF honours it.

- One project with the whole text folder as working directory. Attaching subfolders per chat turned out unnecessary since it greps rather than indexes.

**Where the time goes:** roughly 8 of 11 minutes is prompt processing. Every tool result and the initial tool prompt get read at 75 to 90 tok/s. Generation is the smaller part.

**Questions*\*

1. Anyone getting a usable agent loop on a 32 GB Mac? Which model and quant?

2. Are Gemma 4 26B-A4B or GPT-OSS 20B more reliable at tool calling than the small Qwens? The failure I see is words and tool arguments diverging.

3. Any way to cut prefill for a harness with a big tool prompt on Apple silicon, beyond prompt caching?

4. What would you add to a folder like this before going offline for three weeks?

# Cracking the "Memory Wall": A Guide to Running 100GB+ MoE Models on a 16GB GPU via SSD mmap in llama.cpp

> 🚨 **WARNING:** The text of this article was edited and polished into its final form with the help of an LLM, since English is not my native language. I will also answer technical comments in the thread using an LLM. You can blame me for anything you like, but the source code, configuration scripts, architectural conclusions and inference logs are 100% real, obtained on my physical hardware and reproducible. Toxic people and armchair critics are asked to leave the thread immediately. I have zero tolerance for unsubstantiated complaints. Let's speak the language of computing-physics.

>

> **NOTE ON THE TITLE:** Reddit's mechanics completely block editing titles after publication. The topic title mentions the 125B class, based on early speculative specifications. As the official inference-environment metadata below clearly demonstrates, the model's actual scale is **176.94B parameters** with a physical file size of **104.08 GB**. This makes the achieved performance figures on a consumer 16-gigabyte card even more absurd. The same inference web UI, by the way, also confirms the remaining metrics: training context of **262,144 tokens**, vocabulary of **248,320 tokens**, **16 parallel slots**, supported modalities — **Vision, Video**. Server build — `b10729 (commit 458681e1d5)`.

---

### 💻 Tested Environment and Hardware Configuration

To ensure absolute transparency of the baseline metrics, all empirical metrics documented below were obtained using a purpose-built consumer workstation without corporate offloading:

* **Operating System:** Melawy Linux (an Arch-based distribution)

* **Kernel Architecture:** Linux XanMod 7.2.1 kernel (compiled natively for host optimization)

* **Processor (CPU):** AMD Ryzen 7 5700X (8 cores / 16 threads, configured on PCIe 4.0 lane maps)

* **Motherboard:** ASUS B550 (active PCIe 4.0 bus, the dashboard confirms GEN 4 @ 16x mode)

* **Memory (RAM):** 128 GB DDR4 running at 3600 MHz

* **Graphics Processor (GPU):** AMD Radeon RX 9070 XT with 16 GB of GDDR6 VRAM: RDNA 4 architecture (Navi 48 XT die, gfx1201), 256-bit bus with bandwidth of **up to 640 GB/s**, 64 MB of Infinity Cache and 128 built-in AI accelerators, ROCm/HIP stack

* **Storage Subsystem:** A standard SATA III solid-state drive operating over a 6 Gbit/s interface (maximum sequential read bandwidth of **~550 MB/s** as tracked under real conditions).

* **Model:** Qwen3.8-Flash-Next-Uncensored-**i1**, GGUF quant **i1-Q4_K_S** (quantization with the iMatrix importance matrix, ~4.25 bits per weight) + multimodal projector **mmproj F16**, file size **104.08 GB**, 176.94B parameters, roughly 6B active per token. Architecture: Qwen 4 preview (qwen4exp) with a 51B N-gram PLE table.

---

### 1. The Physics of MoE Inference: Bypassing the Memory Shortage via mmap and the Page Cache

With a GGUF file size of **104.08 GB**, standard workstations inevitably hit an out-of-memory (OOM) error when trying to load the model into memory entirely. However, for sparse Mixture of Experts (MoE) architectures, this limitation is overcome thanks to the OS-level virtual memory mechanism `mmap` and layer-offloading technology.

The architectural hallmark of Qwen3.8-Flash-Next lies in its high sparsity: only a small fraction of expert layers is activated to process a single token — **about 6B parameters** out of 176.94B. For classic dense architectures, `mmap` on a slow drive is inefficient: to emit a single token, a dense 70B model must read all of its weights, which over a SATA III interface (550 MB/s) yields a hard ceiling of ~0.2 tokens/sec. MoE solves this problem, since the router directs the token to only 1–2 experts out of the entire pool.

The launch scheme utilizes two `llama.cpp` backend orchestration parameters:

* **Omitting the `--load-mode` flag:** By default, `auto` mode is used (`mmap, unless a device does not support it`). At startup, the engine does not read the model weights into physical memory; instead, it maps the 104-gigabyte file into the OS virtual address space via the `mmap` system call.

* **Using the `-cmoe` (`--cpu-moe`) flag:** This argument moves the MoE expert blocks and PLE tables out of VRAM into system memory, preventing the 16-gigabyte card from overflowing.

Thanks to the `--n-gpu-layers 99` parameter, the base dense layers, embedding matrices and the vision multimodal projector (`mmproj`) are pinned inside the **GPU VRAM** (the allocation is **15.863/15.922 GiB** — 98% of the video memory). The KV cache for all 16 slots is pinned in **system RAM** via the `--kv-unified` and `--cache-ram 8192` flags. As the context grows from 56k to 132k tokens, the amount of occupied VRAM stays unchanged, while RAM consumption increases by ~1 GB (6.74 → 7.75 GB).

`llama.cpp` employs a **layout-blind expert streaming** mechanism: the GGUF file layout is regrouped so that experts frequently called together lie consecutively on disk (`adjacent layout`), and asynchronous **Direct I/O** with `prefetching` fetches an expert from disk several cycles before the computation.

#### Debunking the "Disk Dilemma": Memory Utilization Analysis

A question often arises: *"If about 6B parameters (~3.18 GB of data) are activated to process a token, then on a SATA III SSD the speed should drop to 0.16 tokens/sec. Where do the real 12–17 t/s come from?"*

The SATA III drive is a bottleneck only during the first minutes of the server's "cold" start. Since the host has **128 GB of RAM** installed and the entire model weighs **104.08 GB**, the operating system fully absorbs the file into system memory as it interacts with the model.

The output of the `free -g` command clearly demonstrates this distribution physics:

```text

total used free shared buff/cache available

Mem: 125 13 11 0 102 112

Swap: 141 0 141

```

* **used = 13 GB:** The process's pure anonymous memory (anonymous memory) — the slots' KV cache, the server runtime and the OS's own needs. This is exactly why `htop` shows the green bar at only the ~8.56G level.

* **buff/cache = 102 GB:** Our entire 104-gigabyte GGUF file has fully settled into the RAM page cache. The Linux kernel counts these pages as disk cache, giving the engine instant access to the expert weights at host RAM speed.

* **available = 112 GB:** The OS considers this memory free, since the mmap pages are not locked permanently (`--mlock` is off) and can be evicted if necessary.

* **Swap = 2.65 MB out of 142 GB:** Swap usage is minimal. Thanks to `vm.swappiness = 10`, the kernel does not evict mmap pages to disk, and the enabled **ZSWAP** mechanism with `zstd` compression intercepts the rare eviction attempts, eliminating I/O stalls.

After the context warms up, the MoE experts stop being requested from disk. The `mmap` mechanism turns system memory into a high-speed array from which expert weights are transferred to the GPU at the speed of the host memory bus, completely bypassing the SATA interface limitations.

#### What the `i1` Marker in the Weights Filename Means

The `i1` tag indicates the version of the **iMatrix (Importance Matrix)**. Before quantization, the model is calibrated on a dataset, the algorithm tracks neuron activations and annotates the weights: critical ones are preserved with higher precision, secondary ones are compressed more aggressively.

On the inference side, no additional computation occurs — the matrix is baked into the quantization structure and is not recomputed on the fly. The GPU's compute units dequantize the `i1-Q4_K_S` weights into FP16, and the matrix multiplications are executed by the built-in AI accelerators (in the RDNA 4 case, peak performance reaches **98 TFLOPS FP16**). The GPU chip spends the same cycles on dequantization as it would on a regular quant — the importance matrix does not slow down generation. The `i1-Q4_K_S` quant delivers accuracy close to the average `Q4_K_M` while saving 5–7 GB of memory.

#### Limitations of Scope: What Happens on 32GB or 64GB RAM Systems?

The lazy-caching mechanics via `mmap` described above remain stable only because the physical host RAM (128GB) exceeds the net file size of the model (104GB) minus the layers offloaded to VRAM. Running this exact quant on configurations with less memory will drastically alter the system's behavior:

1. **On a 32GB RAM Configuration:** An OOM error will not occur, as `mmap` will successfully map the virtual addresses. However, due to a severe lack of physical space, the Linux kernel will be unable to retain the read pages in the Page Cache. It will be forced to continuously drop old weights to clear space for new ones. This triggers heavy disk thrashing (I/O Thrashing), crushing the speed down to **0.1–0.2 tokens per second**. The model will essentially read from the drive on every single token.

2. **On a 64GB RAM Configuration:** Out of the 104GB model, a portion (~16GB) is offloaded to VRAM, while about 50GB of the remaining weights will fit into physical RAM. The remaining ~38GB of the model will have to be constantly re-read from the drive in cycles during inference. The generation speed will become highly erratic and bottle out in the range of **1–3 tokens per second**, completely bound by the drive's throughput.

**The Bottom Line:** This method of streaming 100GB+ MoE models via passive `mmap` is highly efficient only when 128GB of host RAM is available. For systems limited to 32GB or 64GB RAM, running this setup works strictly as a Proof of Concept (PoC) but is completely impractical for real-world tasks without dropping down to much harsher quants (like IQ2 or IQ3).

#### Architectural Nuances of the KV Cache and Vision

When scaling the context, the `--cache-type-k f16`, `--cache-type-v f16` parameters and using the F16 version of the vision projector (`mmproj`) are mandatory. Quantizing the KV cache (Q4 or Q8) on long token sequences leads to an avalanche-like accumulation of rounding errors and a breakdown of the network's attention (infinite loops, loss of logic). F16 precision guarantees perfect mathematical signal purity.

#### 🔄 Alternative Streaming Paradigms in the Ecosystem

The concept of streaming expert weights from disk is quickly becoming the main optimization vector in modern inference backends. For example, the `sglang-ssd-stream` extension implements an independent asynchronous paging pipeline for NVIDIA architectures. Instead of the OS-level `mmap` mechanism, it uses a custom engine built on the low-level Linux kernel API — **`io_uring`** in Rust for processing quantized FP4 streams (`Qwen3.8-Flash-Next-NVFP4-SSD-Stream`).

This mechanism intercepts requests to the expert predictive-lookup (PLE) tables even before the GPU compute graph reaches the required point, asynchronously reading the needed 4 KB pages from the NVMe drive into pinned memory and processing the tensors on a separate CUDA stream in parallel with the computations.

Despite the high efficiency of this approach on corporate-grade architectures (RTX PRO 6000 or DGX Spark class), the native `mmap` implementation (the default `--load-mode auto` mode) in the latest `llama.cpp` commits offers a far more accessible, hardware-agnostic alternative. It works "out of the box" in any standard Linux environment, including consumer platforms based on AMD ROCm.

---

### 2. A Universal CLI Blueprint for Compiling from Source (Any Linux Distribution)

To squeeze out the maximum prompt-processing speed (`prefill`) and completely eliminate runtime abstractions, `llama.cpp` must be built locally from source. Such a build forcibly enables native vector instructions for your CPU. Even on an ordinary mid-range CPU without hardware AVX-512 support (like my AMD Ryzen 7 5700X on the `-march=znver3` microarchitecture), the right compiler optimization flags guarantee that the host machine handles memory-page handling and virtual-address routing without creating a "bottleneck" for the GPU.

The production build runs on commit `b10729` (`b10729.r0.g458681e1d5` — the same build the web UI shows in Model Information) with `GGML_HIP_GRAPHS=ON` enabled: it is precisely this flag responsible for the `graphs reused = 8195` line in the inference logs. The server process is additionally pinned to physical cores via the environment: `OMP_NUM_THREADS=8`, `OMP_PROC_BIND=true` — exactly 8 physical 5700X cores for generation and 16 SMT threads for batch/prefill.

Below are the optimized build profiles for both graphics platforms:

#### Variant A: Build Profile for AMD Hardware (ROCm 7.x Stack)

We explicitly override the paths, binding the build system directly to AMD's native LLVM Clang compiler, bypassing the unstable `hipconfig` wrappers. The high-performance `mold` linker (`-fuse-ld=mold`) is also used to speed up the build:

```bash

cd llama.cpp

cd tools/ui && npm ci && npm run build && cd ../..

export ROCM_PATH="/opt/rocm"

export PATH="/opt/rocm/llvm/bin:/opt/rocm/bin:\$PATH"

export LD_LIBRARY_PATH="/opt/rocm/llvm/lib:/opt/rocm/lib:\$LD_LIBRARY_PATH"

export HIPCXX="/opt/rocm/llvm/bin/clang++"

export CC="/opt/rocm/llvm/bin/clang"

export CXX="/opt/rocm/llvm/bin/clang++"

export HIP_PLATFORM=amd

# for the Ryzen 5700X

export CFLAGS="-march=native -O3 -pipe -fno-plt"

export CXXFLAGS="$CFLAGS"

export LDFLAGS="-Wl,-O1 --as-needed -fuse-ld=mold"

# for the RX 9070 XT 16 GB

cmake -S . -B build -G Ninja \

-DAMDGPU_TARGETS="gfx1201" \

-DBUILD_SHARED_LIBS=ON \

-DCMAKE_AR="/opt/rocm/llvm/bin/llvm-ar" \

-DCMAKE_BUILD_TYPE=Release \

-DCMAKE_CXX_COMPILER="\$CXX" \

-DCMAKE_C_COMPILER="\$CC" \

-DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=mold" \

-DCMAKE_INSTALL_PREFIX='/usr' \

-DCMAKE_RANLIB="/opt/rocm/llvm/bin/llvm-ranlib" \

-DCMAKE_SHARED_LINKER_FLAGS="-fuse-ld=mold" \

-DGGML_ALL_WARNINGS=OFF \

-DGGML_ALL_WARNINGS_3RD_PARTY=OFF \

-DGGML_BUILD_EXAMPLES=OFF \

-DGGML_BUILD_TESTS=OFF \

-DGGML_CUDA_FA_ALL_QUANTS=ON \

-DGGML_HIP=ON \

-DGGML_HIP_GRAPHS=ON \

-DGGML_LTO=ON \

-DGGML_NATIVE=ON \

-DGGML_RPC=ON \

-DLLAMA_BUILD_SERVER=ON \

-DLLAMA_BUILD_TESTS=OFF \

-DLLAMA_BUILD_WEBUI=ON \

-DLLAMA_OPENSSL=ON \

-DLLAMA_USE_SYSTEM_GGML=OFF \

-Wno-dev

ninja -C build

```

#### Variant B: Build Profile for NVIDIA Hardware (CUDA Backend)

This profile, extracted from a production deployment's parameters, forcibly enables optimization for the local CPU via `-DGGML_NATIVE=ON`, Flash Attention shaders for all quants and math acceleration through cuDNN:

```bash

cd llama.cpp

cd tools/ui && npm ci && npm run build && cd ../..

export CFLAGS="-march=native -O3 -pipe"

export CXXFLAGS="$CFLAGS"

export LDFLAGS="-Wl,-O1 --as-needed -fuse-ld=mold"

cmake -S . -B build -G Ninja \

-DBUILD_SHARED_LIBS=ON \

-DCMAKE_BUILD_TYPE=Release \

-DCMAKE_INSTALL_PREFIX='/usr' \

-DGGML_ALL_WARNINGS=OFF \

-DGGML_ALL_WARNINGS_3RD_PARTY=OFF \

-DGGML_BUILD_EXAMPLES=OFF \

-DGGML_BUILD_TESTS=OFF \

-DGGML_CUDA=ON \

-DGGML_CUDA_FA_ALL_QUANTS=ON \

-DGGML_CUDNN=ON \

-DGGML_LTO=ON \

-DGGML_NATIVE=ON \

-DGGML_RPC=ON \

-DLLAMA_BUILD_SERVER=ON \

-DLLAMA_BUILD_TESTS=OFF \

-DLLAMA_BUILD_WEBUI=ON \

-DLLAMA_OPENSSL=ON \

-DLLAMA_USE_SYSTEM_GGML=OFF \

-Wno-dev

ninja -C build

```

---

### 3. The Ready-to-Use llama-server Deployment Script

The compiled binary is launched with the following parameter matrix. Note the **complete absence of the `--load-mode` flag**: llama.cpp enables the memory-projection mode `auto` (i.e. mmap) by default.

The profile includes continuous-batching architectures (`--cont-batching`), the unified KV cache (`--kv-unified`), which pins a single context pool in system RAM with an 8 GiB budget (`--cache-ram 8192`) and caches idle slots (`--cache-idle-slots`), plus hardware-accelerated Flash Attention (`--flash-attn on`) to ensure linear scalability on ultra-deep contexts. The `--threads 8` / `--threads-batch 16` flags map directly onto the 5700X architecture: 8 physical cores for generation, 16 SMT threads for batch prompt processing:

```bash

/usr/bin/llama-server \

-m Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_S.gguf \

--mmproj Qwen3.8-Flash-Next-Uncensored.mmproj-f16.gguf \

--image-min-tokens 1024 \

--jinja \

--no-skip-chat-parsing \

--reasoning on \

--reasoning-preserve \

--reasoning-format auto \

-cmoe \

-fit off \

--n-gpu-layers 99 \

--threads 8 \

--threads-batch 16 \

--batch-size 2048 \

--ubatch-size 512 \

--parallel 16 \

--ctx-size 262144 \

--context-shift \

--cont-batching \

--kv-unified \

--cache-prompt \

--cache-ram 8192 \

--cache-idle-slots \

--cache-type-k f16 \

--cache-type-v f16 \

--flash-attn on \

--temp 1.0 \

--top-k 20 \

--top-p 0.95 \

--min-p 0.00 \

--repeat-last-n 512 \

--repeat-penalty 1.00 \

--presence-penalty 0.00 \

--host 0.0.0.0 \

--port 11441 \

--reuse-port

```

The server runs as a user-level systemd unit (`Restart=always`, `Nice=10`): the `LimitMEMLOCK=infinity` and `LimitNOFILE=65536` limits, the cgroup constraints `MemoryHigh=115G` / `MemoryMax=125G` and `OOMScoreAdjust=-500` protect the mapped model and the hot context pages from swapping and make the process the last candidate for killing by the OOM killer. From the ROCm scaffolding: `ROCM_ALLOCATOR_STRATEGY=2`, `HSA_ENABLE_SDMA=1` and `ROCBLAS_TENSILE_WARMUP=0` (limits rocBLAS's aggressive memory pre-allocation). At the host kernel level, things are deliberately tuned for mmap inference: the block-device I/O scheduler for the model drive is switched to `none` (the kernel doesn't spend time sorting I/O requests during random weight reads), via `sysctl` are set `vm.swappiness=10` (swap is engaged only at the most critical moment before OOM), `vm.vfs_cache_pressure=50` (the kernel prefers to retain GGUF mmap pages in the page cache) and `vm.max_map_count=1000000` (the critically necessary limit on the number of simultaneously mapped memory regions for the 104-gigabyte file), and via bootloader kernel parameters `zswap` is enabled with `zstd` compression and a `zbud` pool — redundant pages are compressed in RAM and only then go to the SSD.

---

### 4. Performance and Context-Scaling Log Analysis

All metrics below were captured by the host `melawy-linux-lera` (the server listens on `0.0.0.0:11441`) from real `llama-server` logs (journalctl) and `htop` dashboards (RAM, swap, load average) and `nvtop` — yes, `nvtop` has long supported AMD/ROCm excellently, showing VRAM, clocks, power draw and utilization straight from the driver. The server with the model onboard has been running for over 17 hours without a restart without a single page swapped out, and its main thread holds 100% of one 5700X core (TIME+ on one thread — 1h55m of machine time). Analysis of the server's runtime under real multitasking scenarios demonstrates high linearity and reproducibility of the metrics:

#### A. Prompt Processing Speed (Prefill)

* **Peak prefill throughput:** Reaches a ceiling of **217.05 tokens per second** — a dense continuous block of 4,582 tokens is processed in 21.28 seconds at an average rate of 215.30 t/s (the first log excerpt below).

* **Sustained sequential loading:** Under a constant rolling prompt load, prefill performance predictably grows along with the batch: the log's progress lines show **128.09 → 206.34 → 213.76 → 217.05 t/s** on a short prompt, and on the 69k prompt (69,282 tokens in 400.28 seconds) the bar holds at **174.15 → 173.36 → 173.16 t/s** with an average rate of **173.08 t/s** (5.78 ms per token).

* **Dynamic slot multiplexing:** When several concurrent requests are active via `--parallel 16`, the prefill speed per stream drops to roughly **35 t/s**. This is the expected mathematical distribution: the inference daemon dynamically splits the total VRAM bus bandwidth and GPU cores among the active independent context tracks. Indirectly this is even visible in `htop`: with 16 slots the kernel holds `363 kthr: 11 running`, and the load average of the 8-core 5700X stays at the **7.36** level — batch expert processing genuinely utilized 16 CPU threads.

* **GPU power package:** During "thinking" generation, the GPU (`nvtop`) draws only **81–85 W out of 330 W** at **47%** core utilization (effective utilization — **11%**) and 98% video-memory occupancy; during image processing and the subsequent prefill, consumption jumps to **230–240 W**, peaking at **248–300 W** out of 330 W. Core clock — **2669 MHz**, VRAM — **1258 MHz**, temperature — **54 °C** at **30%** fan speed. The bottleneck of the system is not the GPU but the speed of delivering expert weights from the SSD into RAM: in the regular generation mode the GPU burns only a quarter of its power package.

#### B. Token Generation Speed (Eval Throughput)

* **Short context (~9.4k tokens, task 547 in the excerpt below):** Baseline generation speed is **15.22 t/s** (instantaneous values reach up to 15.62 t/s) — a full cycle of "introduce yourself, write a sorting algorithm in Rust, analyze Tatyana's letter to Onegin".

* **Medium context (21.6% of the window, 56,623 tokens):** VRAM allocation is firmly pinned at 15.777 GiB out of 15.922 GiB. The active physical RAM allocation delta is only 6.74 GB. Text generation runs at **13.0 t/s**.

* **Deep context (27.1% of the window, 71,127 tokens, task 0 id 15 in the excerpt below):** 1,846 tokens generated in 134 seconds — **13.74 t/s** at **72.76 ms per token**, the rolling three-second window `tg_3s` stays within the **13.5–13.9 t/s** corridor, and `graphs reused` grew to 1838.

* **Deep context expansion (50.7% of the window, 132,907 tokens):** VRAM allocation holds perfectly steady at 15.781 GiB with zero memory leak. The system RAM delta increases minimally — to 7.75 GB. Generation speed drops by less than 1 token per second relative to the previous tier, settling at a stable **12.12 tokens per second**.

```text

// Excerpt from the inference log (task 547 — 7,691 generated tokens)

prompt eval time = 5241.85 ms / 801 tokens ( 6.54 ms per token, 152.81 tokens per second)

eval time = 505246.75 ms / 7691 tokens ( 65.70 ms per token, 15.22 tokens per second)

total time = 510488.60 ms / 8492 tokens

graphs reused = 8195

stop processing : n_tokens = 9425, truncated = 0

// Next task 8243 — prefill of a dense block of 4,582 tokens

prompt processing, n_tokens = 618, progress = 0.13, t = 4.82 s / 128.09 tokens per second

prompt processing, n_tokens = 2666, progress = 0.58, t = 12.92 s / 206.34 tokens per second

prompt processing, n_tokens = 4066, progress = 0.89, t = 19.02 s / 213.76 tokens per second

prompt processing, n_tokens = 4578, progress = 1.00, t = 21.09 s / 217.05 tokens per second

prompt eval time = 21282.13 ms / 4582 tokens ( 4.64 ms per token, 215.30 tokens per second)

graphs reused = 8195

// Third excerpt — task 0 (id 15): a prompt of 69,282 tokens + generation of 1,846 tokens

prompt processing, n_tokens = 67686, progress = 0.98, t = 388.67 s / 174.15 tokens per second

prompt processing, n_tokens = 68766, progress = 0.99, t = 396.07 s / 173.36 tokens per second

prompt processing, n_tokens = 69276, progress = 1.00, t = 400.08 s / 173.16 tokens per second

prompt eval time = 400278.31 ms / 69282 tokens ( 5.78 ms per token, 173.08 tokens per second)

eval time = 134248.54 ms / 1846 tokens ( 72.76 ms per token, 13.74 tokens per second)

total time = 534526.86 ms / 71128 tokens

slot print_timing: n_gen = 1105, tg = 13.76 t/s, tg_3s = 13.85 t/s

graphs reused = 1838

stop processing : n_tokens = 71127, truncated = 0

```

### 🏁 Conclusion

Losing less than 1 token per second when doubling the context — from 56,623 to **132,907 tokens** (13.00 → 12.12 t/s with pinned video memory) — proves the point: competent low-level compilation of open-source software completely outperforms the brute-force scaling of hardware power.

Running a 104 GB model at 12+ t/s on a consumer gaming card is not an experimental hack, but a production-ready reality. By forcing the OS's built-in memory manager to run `mmap` cycles over the sparse expert blocks via the `--load-mode auto` mode (llama.cpp's default), custom architectures completely break the commercial monopoly of expensive corporate workstations.

### Contributors to this article:

* 👩🏼 **Valeria Fadeeva** — lead developer and founder of **Melawy Linux** (an Arch-based distribution).

With the help of neural network models:

1. **Google Gemini 1.5 Pro** — assistance in compiling the material, web searching and sorting through scattered information from the internet.

2. **Qwen-3.8-Flash-Next-Uncensored (i1-Q4_K_S)** — detailed fact-checking on real hardware, text and benchmark optimization.

We run one AMD MI300X on a GPU droplet and serve it internally as our own inference endpoint. I swapped the model on it. The difference at long context surprised me. I would love to get Qwen running closer to the speed of ds. Any tips would be appreciated.

The hardware

• GPU: 1x AMD Instinct MI300X VF, 205.8 GB VRAM
• Intel Xeon Platinum 8568Y+, 20 vCPU
• RAM 235 GB, no swap
• Disk 697 GB
• Ubuntu 24.04.4, kernel 6.8.0-138
• ROCm 7.2.4
• One card. Not a node, not a cluster.

What we were running

Qwen3.8-27B in bf16 on SGLang 0.5.14, in the ROCm container, under systemd. Speculative decoding was on through the model's own MTP head driven by EAGLE, accepting about 3.35 of 4 draft tokens. Context 262,144.

The one config change that actually mattered was --mamba-ssm-dtype bfloat16. It stopped the Mamba state cache from eating the KV budget in FP32, and it took us from a hard ceiling around 155k tokens up to the full window. If you are running this model on a single card and you are stuck this might help you.

We tried Qwen at FP8 and saw ~2x slower speeds. I am new to this, but was surprised nonetheless.

What we run now

DeepSeek-V4-Flash-0731 on vLLM ROCm. I did not build this. It is a public pinned recipe, ryanzhou/deepseek-v4-flash-mi300x, which ships the AMD patches as read only bind mounts over a stock vLLM install plus check summed prebuilt kernels. 23 patch mounts, one compose file. It came up on the first try, which I did not expect either (and really appreciate).

Config:

--max-model-len 393216

--kv-cache-dtype fp8

--block-size 256

--kv-cache-memory-bytes 16000000000 # 16 GB KV on device

--kv-offloading-size 96 # 96 GB KV offloaded to host RAM

--kv-offloading-backend native

--max-num-seqs 64

--max-num-batched-tokens 4096

--long-prefill-token-threshold 1024

--moe-backend triton

--speculative-config.method=dspark

--speculative-config.num-speculative-tokens=7

--compilation-config cudagraph_mode=FULL_AND_PIECEWISE

Env: VLLM_ROCM_USE_AITER=1, VLLM_ROCM_OPUS_PREFILL=1, VLLM_ROCM_USE_SKINNY_GEMM=0, HSA_NO_SCRATCH_RECLAIM=1, HIP_FORCE_DEV_KERNARG=1.

Here is the interesting part.

Single stream, 512 output tokens, prose output, unique prefix on every run so nothing is cached.

Going from 1k to 200k cost Qwen about 11x of its throughput. It cost DeepSeek about 1.2x.

I would rather not give up the 27B. It has been working quite well for the type of work we do, but speed is a real factor to consider.

Any thoughts, considerations, tips? Thanks in advance.

P.S. I used AI to create the table and to print the configuration above.

with MTP draft head kept in BF16

7 min 56 seconds Medium reasoning

The quality is much better than this... gift converter
(prompt: generate an animated SVG of a pelican riding a bycicle.)

I can run them only at pcie 8x speed so I would want to save some money

Motherboard: z11pa u12

CPU: intel gold 6138

Ram: 96gb ram

Storage: 64tb

I don't really need to load and unload models, and this will be a mix between a media server and local AI machine.

I am running a 5090 in my main machine.

Is Qwen3.8-Flash-Next 125B A6B currently the best option?

Interested in real-world tok/s + quality comparisons from people actually running these models on a single Spark.

mistral.rs inference engine

RTX 5090

mistralrs serve -m Qwen/Qwen3.8-27B --quant 4 --mtp --pa-memory-mb 1024 --mtp-n-predict 6

2026-09-01T23:02:03.867737Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 97.80, Prefix cache hitrate 0.00%, MTP accept 28.9% (len 2.73), 1 running, 0 waiting
2026-09-01T23:02:08.867835Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 166.60, Prefix cache hitrate 0.00%, MTP accept 24.8% (len 2.49), 1 running, 0 waiting
2026-09-01T23:02:13.867932Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 168.00, Prefix cache hitrate 0.00%, MTP accept 23.1% (len 2.38), 1 running, 0 waiting
2026-09-01T23:02:18.862084Z  INFO mistralrs_server_core::metrics: request completed: request_id=req_17c85ea2689a4abb937c1dd578e803d5 method=POST route=/v1/chat/completions model=Qwen/Qwen3.8-27B status=200 outcome=client_disconnected duration_ms=17406.885
2026-09-01T23:02:18.868016Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 163.80, Prefix cache hitrate 0.00%, MTP accept 24.8% (len 2.49), 1 running, 0 waiting
2026-09-01T23:02:21.864911Z  INFO mistralrs_server_core::metrics: request started: request_id=req_b82f3f7abe8f4a2080b305f151cad9bc method=POST route=/v1/chat/completions path=/v1/chat/completions model=Qwen/Qwen3.8-27B content_length=516
2026-09-01T23:02:23.868196Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 78.60, Prefix cache hitrate 0.00%, MTP accept 43.7% (len 3.62), 1 running, 0 waiting
2026-09-01T23:02:28.868281Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 151.20, Prefix cache hitrate 0.00%, MTP accept 56.3% (len 4.38), 1 running, 0 waiting
2026-09-01T23:02:33.868364Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 148.40, Prefix cache hitrate 0.00%, MTP accept 64.5% (len 4.87), 1 running, 0 waiting
2026-09-01T23:02:38.868448Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 147.00, Prefix cache hitrate 0.00%, MTP accept 66.8% (len 5.01), 1 running, 0 waiting
2026-09-01T23:02:43.868616Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 142.80, Prefix cache hitrate 0.00%, MTP accept 70.9% (len 5.25), 1 running, 0 waiting
2026-09-01T23:02:47.633507Z  INFO mistralrs_server_core::metrics: request completed: request_id=req_b82f3f7abe8f4a2080b305f151cad9bc method=POST route=/v1/chat/completions model=Qwen/Qwen3.8-27B status=200 outcome=completed duration_ms=25768.586 prompt_tokens=77 completion_tokens=2523 prefill_tok_s=1400.0 decode_tok_s=99.4
2026-09-01T23:02:48.868714Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 116.20, Prefix cache hitrate 0.00%, MTP accept 45.2% (len 3.71), 1 running, 0 waiting

I was testing Qwen3.8-27B models on my M1 Max 64GB mac and found that Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed-FP16 performed best for coding. But then saw this claim - OptiQ Code moves about 5x fewer tokens than opencode, and wanted to test it.

So I used Sonnet 5 to set up a test suite and here are the results:

MTPLX vs OptiQ

Two Qwen3.8-27B serving backends, three coding-agent harnesses, one real multi-file bug fix — measured end to end, not just at the token layer.

Machine: Apple M1 Max 64GB unified memory

Models: Qwen3.8-27B (MTPLX fp16 | OptiQ 4-bit + MTP)

• Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed-FP16
• mlx-community/Qwen3.8-27B-OptiQ-4bit

Servers:

• mtplx v2.10.2 with mtplx-qwen38-27b-optimized-speed-fp16
• mlx-optiq v0.4.34 with Qwen3.8-27B-OptiQ-4bit

Harnesses: Pi | opencode | OptiQ Code

TL;DR

• 2.2–4.9× — MTPLX beat OptiQ on wall-clock time across every harness tested
• Collapse to MTPLX — the original workload-split hypothesis didn't survive a real test

01 — The question: does splitting work between backends earn its complexity?

A workload-assignment scheme — from a source that hadn't been independently verified — proposed routing long, single-shot generations to MTPLX (its native multi-token-prediction head amortizes over many output tokens) and short, frequent agent turns to oMLX (continuous batching, and a claimed per-request MTP sampler tax on Apple Silicon that eats the gain on short generations).

Tested with a small Babashka harness: two task shapes × two endpoints, three timed trials each after a discarded warmup, greedy decoding, streaming to separate prefill from decode.

3 timed trials per cell, median reported · temperature 0 · warmup discarded

Verdict: MTPLX won both shapes decisively — including the short-turn case oMLX was predicted to win. The split's own decision rule calls it: collapse to MTPLX; the second server and the routing logic are pure overhead on this machine.

02 — Real agent work: three harnesses, both backends, one real task

Canned prompts test the model server. To test something closer to daily use, the same task ran through three different coding agents — Pi, opencode, and OptiQ Code — against both backends: a Clojure module with two seeded bugs across two files (a discount-math bug, an argument-order bug), verified by its own bb test suite. Each agent had to find both bugs, fix them, and get the suite green — no test files touched.

Every server was restarted between runs for a clean cache. MTPLX turned out to keep a persistent, disk-backed session cache that a process restart alone doesn't clear — an explicit override was needed to get it back to a true cold start.

Total tokens moved & wall-clock time

Single run per cell · fresh server restart before every run · Pi figures use its final, de-confounded configuration

Tokens are a mixed signal: MTPLX used fewer tokens with opencode and Pi, but more with OptiQ Code — consistent with MTPLX doing visibly more chain-of-thought reasoning per turn (a live <think> block was observed), not harness bloat. Time is the clean, fully-corroborated signal: MTPLX won it on all three harnesses, independently measured.

03 — What broke along the way: seven real bugs, surfaced by trying to run the tests

None of these were hypothetical — each one silently invalidated a run (or blocked it outright) until it was root-caused. Listed roughly in the order they were hit.

01. Wrong oMLX model id Config had the HF slash form (mlx-community/Qwen…); the server only answered to its double-dash cache-dir id. Worse, the harness's own model-id verification discovered the right id but never wired it into the actual request — it only used it for a warning message. Every oMLX call 404'd until both the config string and the wiring were fixed. Impact: every oMLX trial silently failed

02. Token-cap mismatch Two servers launched with --max-tokens 16384, but the long-generation task could plausibly need 32K+ output tokens — a truncated run flatters whichever server truncated first. Caps bumped to 32768 on both the servers and the harness config. Impact: would have silently invalidated the long-generation numbers

03. OptiQ's own start script couldn't start OptiQ Two flags (--vision, --kv-cache-type fp16) that the installed server version no longer accepts. The endpoint refused to boot at all until they were dropped. Impact: OptiQ endpoint unusable

04. OptiQ Code's headless mode crashed outright A broken relative import in the installed CLI, then a missing optional dependency once that was patched by an upgrade — headless (-p) mode couldn't run a single prompt until both were resolved. Impact: blocked before any measurement could happen

05. A "5× fewer tokens" claim, tested and not confirmed OptiQ Code vs. opencode, on an identical single-file fix, same backend, same model: OptiQ Code used 1.9× fewer tokens — a real and consistent advantage, just not the claimed magnitude. Impact: directionally right, ~2.5× overstated

06. Pi's "reasoning: true" broke every OptiQ request It made Pi send the system prompt with role: "developer" — OpenAI's newer reasoning-model convention. This Qwen model's own chat template hard-rejects any role outside system / user / assistant / tool, 404ing before the model ever saw the request. MTPLX's server tolerated the same role fine; OptiQ's didn't. Impact: every Pi→OptiQ request rejected at the protocol level

07. 61 irrelevant tool schemas, on every Pi request A config flag was flattening every tool from two unrelated MCP servers — a PDF-processing tool and a stock/crypto market-data tool — into Pi's per-request tool list. This dwarfed even a large custom-instructions file as a source of overhead. Disabling it cut Pi's token count and wall time by more than half. Impact: more than 2× the tokens and time it needed

04 — Reading this honestly: what this is, and isn't

• Single run per cell. Live multi-turn agent sessions are too costly to repeat at the reps used for the canned-prompt benchmark — these are one real trial each, not medians.
• An unresolved MTPLX quirk. With its SSD session cache forced off for a fair restart, Pi's MTPLX runs show an unexplained mid-session cache reset around turn 3–4. Reproduced across two independent runs, not yet root-caused.
• Token accounting isn't uniform across tools. Some report cache reads as a subset of "input," others additively. Totals here are computed consistently within this report; don't diff them against a tool's own dashboard number without checking which convention it uses.

Fixture: 2-bug Clojure billing module, verified via its own bb test suite · All servers restarted between runs · Greedy decoding where configurable

Getting into AI and i want my own local AI workspace, i've seen so many, if anyone has any repos i can ask claude to base its own one on would be greatly appreciated.

Wanted to share a setup that has been working better than I expected. I have three small agents running off a single consumer GPU, and they never fight over VRAM:

- vLLM serving Qwen3.5-4B at --gpu-memory-utilization 0.80, 16k context. It does double duty, vision for a plant diagnosis app and plain text for ranking podcast transcripts into clips

- faster-whisper with large-v3-turbo in int8_float16, sits in about 1.5GB and does timestamped transcription

- Sana 600M through diffusers for image gen, peaks around 2.5GB and only loads when a job needs it

- CLIP ViT-B/32 for embedding and scoring, but that one lives on CPU, so it costs nothing

The thing that surprised me most is how far the 4B vision model gets on narrow tasks. Open-ended chat about an image is mediocre, but "here is a photo of a plant, what is wrong with it" with a tight prompt and a fixed output format is genuinely usable. Same weights rank an hour of transcript for clip-worthy moments as long as you trim from the middle instead of truncating the end, since the intro and outro are usually where the good quotes are.

Whisper timings are what everything hangs off. The segment boundaries land on real pauses, so clip timestamps come out clean without any post-processing.

Everything is open source under Apache 2.0 or MIT, code is on GitHub if anyone wants the Docker commands and the exact memory split: https://github.com/blocksnetwork/Blocks-Agent-Demos?&utm_source=reddit&utm_medium=organic_social&utm_campaign=huggingface_agents&utm_content=gh_repo

Curious what other people are running resident together on one card. Feels like the 4B vision class quietly got good enough for real products this year.