MTPLX vs OptiQ

[u/Trampled-dad]

I was testing Qwen3.8-27B models on my M1 Max 64GB mac and found that Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed-FP16 performed best for coding. But then saw this claim - OptiQ Code moves about 5x fewer tokens than opencode, and wanted to test it.

So I used Sonnet 5 to set up a test suite and here are the results:

MTPLX vs OptiQ

Two Qwen3.8-27B serving backends, three coding-agent harnesses, one real multi-file bug fix — measured end to end, not just at the token layer.

Machine: Apple M1 Max 64GB unified memory

Models: Qwen3.8-27B (MTPLX fp16 | OptiQ 4-bit + MTP)

• Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed-FP16
• mlx-community/Qwen3.8-27B-OptiQ-4bit

Servers:

• mtplx v2.10.2 with mtplx-qwen38-27b-optimized-speed-fp16
• mlx-optiq v0.4.34 with Qwen3.8-27B-OptiQ-4bit

Harnesses: Pi | opencode | OptiQ Code

TL;DR

• 2.2–4.9× — MTPLX beat OptiQ on wall-clock time across every harness tested
• Collapse to MTPLX — the original workload-split hypothesis didn't survive a real test

01 — The question: does splitting work between backends earn its complexity?

A workload-assignment scheme — from a source that hadn't been independently verified — proposed routing long, single-shot generations to MTPLX (its native multi-token-prediction head amortizes over many output tokens) and short, frequent agent turns to oMLX (continuous batching, and a claimed per-request MTP sampler tax on Apple Silicon that eats the gain on short generations).

Tested with a small Babashka harness: two task shapes × two endpoints, three timed trials each after a discarded warmup, greedy decoding, streaming to separate prefill from decode.

Task shape	MTPLX decode	oMLX decode	Winner	Margin
Surgical AST edit (short turn — oMLX predicted)	53.5 tok/s	13.0 tok/s	MTPLX	+39.8%
Full-namespace scaffold (long gen — MTPLX predicted)	27.2 tok/s	7.4 tok/s	MTPLX	+56.3%

3 timed trials per cell, median reported · temperature 0 · warmup discarded

Verdict: MTPLX won both shapes decisively — including the short-turn case oMLX was predicted to win. The split's own decision rule calls it: collapse to MTPLX; the second server and the routing logic are pure overhead on this machine.

02 — Real agent work: three harnesses, both backends, one real task

Canned prompts test the model server. To test something closer to daily use, the same task ran through three different coding agents — Pi, opencode, and OptiQ Code — against both backends: a Clojure module with two seeded bugs across two files (a discount-math bug, an argument-order bug), verified by its own bb test suite. Each agent had to find both bugs, fix them, and get the suite green — no test files touched.

Every server was restarted between runs for a clean cache. MTPLX turned out to keep a persistent, disk-backed session cache that a process restart alone doesn't clear — an explicit override was needed to get it back to a true cold start.

Total tokens moved & wall-clock time

Harness	OptiQ tokens	MTPLX tokens	OptiQ time	MTPLX time	Speedup
OptiQ Code	21,285	30,247	271.5s	121.4s	2.2×
opencode	105,238	55,612	739.4s	150.1s	4.9×
Pi (latest run)	52,009	42,529	422.7s	163.9s	2.6×

Single run per cell · fresh server restart before every run · Pi figures use its final, de-confounded configuration

Wall-clock time by harness

Tokens are a mixed signal: MTPLX used fewer tokens with opencode and Pi, but more with OptiQ Code — consistent with MTPLX doing visibly more chain-of-thought reasoning per turn (a live <think> block was observed), not harness bloat. Time is the clean, fully-corroborated signal: MTPLX won it on all three harnesses, independently measured.

03 — What broke along the way: seven real bugs, surfaced by trying to run the tests

None of these were hypothetical — each one silently invalidated a run (or blocked it outright) until it was root-caused. Listed roughly in the order they were hit.

01. Wrong oMLX model id Config had the HF slash form (mlx-community/Qwen…); the server only answered to its double-dash cache-dir id. Worse, the harness's own model-id verification discovered the right id but never wired it into the actual request — it only used it for a warning message. Every oMLX call 404'd until both the config string and the wiring were fixed. Impact: every oMLX trial silently failed

02. Token-cap mismatch Two servers launched with --max-tokens 16384, but the long-generation task could plausibly need 32K+ output tokens — a truncated run flatters whichever server truncated first. Caps bumped to 32768 on both the servers and the harness config. Impact: would have silently invalidated the long-generation numbers

03. OptiQ's own start script couldn't start OptiQ Two flags (--vision, --kv-cache-type fp16) that the installed server version no longer accepts. The endpoint refused to boot at all until they were dropped. Impact: OptiQ endpoint unusable

04. OptiQ Code's headless mode crashed outright A broken relative import in the installed CLI, then a missing optional dependency once that was patched by an upgrade — headless (-p) mode couldn't run a single prompt until both were resolved. Impact: blocked before any measurement could happen

05. A "5× fewer tokens" claim, tested and not confirmed OptiQ Code vs. opencode, on an identical single-file fix, same backend, same model: OptiQ Code used 1.9× fewer tokens — a real and consistent advantage, just not the claimed magnitude. Impact: directionally right, ~2.5× overstated

06. Pi's "reasoning: true" broke every OptiQ request It made Pi send the system prompt with role: "developer" — OpenAI's newer reasoning-model convention. This Qwen model's own chat template hard-rejects any role outside system / user / assistant / tool, 404ing before the model ever saw the request. MTPLX's server tolerated the same role fine; OptiQ's didn't. Impact: every Pi→OptiQ request rejected at the protocol level

07. 61 irrelevant tool schemas, on every Pi request A config flag was flattening every tool from two unrelated MCP servers — a PDF-processing tool and a stock/crypto market-data tool — into Pi's per-request tool list. This dwarfed even a large custom-instructions file as a source of overhead. Disabling it cut Pi's token count and wall time by more than half. Impact: more than 2× the tokens and time it needed

04 — Reading this honestly: what this is, and isn't

• Single run per cell. Live multi-turn agent sessions are too costly to repeat at the reps used for the canned-prompt benchmark — these are one real trial each, not medians.
• An unresolved MTPLX quirk. With its SSD session cache forced off for a fair restart, Pi's MTPLX runs show an unexplained mid-session cache reset around turn 3–4. Reproduced across two independent runs, not yet root-caused.
• Token accounting isn't uniform across tools. Some report cache reads as a subset of "input," others additively. Totals here are computed consistently within this report; don't diff them against a tool's own dashboard number without checking which convention it uses.

Fixture: 2-bug Clojure billing module, verified via its own bb test suite · All servers restarted between runs · Greedy decoding where configurable

[u/tommythorn]

I seem to be missing an element. You describe the model and the harness (like pi which connect to a process serving the model over a socket), but not what's serving the model, llama.cpp, vllm, ... ?

[u/Trampled-dad → u/tommythorn]

Thanks, I should have mentioned it .. here are the servers I used:

• mtplx v2.10.2 with mtplx-qwen38-27b-optimized-speed-fp16
  
• mlx-optiq v0.4.34 with mlx-community/Qwen3.8-27B-OptiQ-4bit

Updated the post too.

[u/tommythorn → u/Trampled-dad]

Thanks, I'm trying to follow your steps right now. Maybe I'm missing it, but did you find a solution for pi's `role: "developer"` issue?

"Token accounting isn't uniform across tools" is indeed very annoying.

UPDATE: MTPLX is super slick! It calibrated on my 48 GiB MacMini M4 Pro and reported "40.something tps", but when I ran the "Make an HTML flappy bird." example I got 33.7 tps gen, 33.5 tps end to end (44.8 tps prefill), which is still a very good result.

[u/Trampled-dad → u/tommythorn]

Awesome! Incase it helps, here is my MTPLX server startup config -

mtplx serve \
  --model "Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed-FP16" \
  --host 127.0.0.1 \
  --port $PORT \
  --profile turbo \
  --mtp \
  --batching-preset agent \
  --context-window 131072 \
  --max-tokens 32768 \
  --ssd-session-cache on \
  --no-stats-footer \
  --fan-mode smart \
  --no-auth

For the `role: "developer"` issue with Optiq, I changed the Pi models config to:

"reasoning": false,

[u/whatsupnorton]

Correct me if I’m wrong, but I’m pretty sure that the mlx-community/qwen3.8-27B-OptiQ-4bit model is BF16 and won’t run as well on the M1 Max (I have a 32GB M1 Max and have noticed a significant difference between running FP16 and BF16)

[u/Trampled-dad → u/whatsupnorton]

You are right - HF says "The vision tower is kept at bf16 in a sidecar, so this repo loads text-only under stock mlx-lm and image+text under OptiQ."

I initially tested for tok/s comparing MTPLX and oMLX servers. And MTPLX turned out to be much better:

Task shape	MTPLX decode	oMLX decode	Winner	Margin
Surgical AST edit (short turn — oMLX predicted)	53.5 tok/s	13.0 tok/s	MTPLX	+39.8%
Full-namespace scaffold (long gen — MTPLX predicted)	27.2 tok/s	7.4 tok/s	MTPLX	+56.3%