For context: I was working with Qwen3.8-27b in LMStudio and asking it to do some work on my contacts via iMCP. My name and therefore email contains a female second first name.
Somehow Qwen mentioned in its thinking process that it thinks there is something fishy going on with my loyalty to my girlfriend but decided it won‘t dig into it 🤣

Guess whatever measures they took in model training to avoid misalignment is working

A mildly interesting video about which models can run on the Mac Mini and Studio. As expected current foundational models such as Kimi K3 1.4 TB, if they were even available to run locally, won't fit. Assuming it were available locally it may be possible in the next couple of years with the rapid increase in hardware capabilities - M7, M8?

https://www.youtube.com/watch?v=eDoWKgFqRM4

I was curious to test Qwen3.8 27B from OpenRoter to see how well it could performs on code on my local machine.

I’m also testing the Qwen3.8 Flash version from OpenCode GO.

While Max is also included, the 27B version is not.

I was surprised by the pricing.

On the R9700's I figured out the best path and quality was to get W4A8 running. AMD had also just dropped their AWQ MXFP4 quant of Qwen3.8 27B which is what I am running along with FP8 kv cache. The quality has been great, I ran comparisons across a mini SWE bench and in those tasks I saw the same results across both AWQ MXFP4 and FP8.

Another benefit was the massive kv cache gains. I am now running at about 920k tokens for kv cache.

Full repo is here, I just added ParoQuant optimizations today also.

https://codeberg.org/ggz14/radiance-vllm-mxfp4

So most of my time fiddling around with local ai I was using ollama, lm studio before going over to llama.cpp (I know it’s llama.cpp under the hood anyway). Of course I had a bump in speed every time I went up to the more professional option. At last I went to vllm. I understand using llama.cpp for non NVIDIA cards and for gpu+cpu inference or if you want to use a specific gguf. But once you tasted the speed of vllm, I don’t know why you would ever play with llama.cpp anymore. It’s the same “complication” to set it up but EVERYTHING except loading the model is night and day in speed. Proper cash hits, proper concurrency, it’s sooo much better. And there are quantisations out there that are just as heavy quantised as the heavily quantised gguf.

Further testing of the dino-era cards.

TL;DR: actual agentic tasks are slower than synthetic tests. A single V100 is the cheapest one to run per TG/kWh.

This is the continuation of https://www.reddit.com/r/LocalLLM/s/lHrKiVs2Z7

At first, I was skeptical whether you can fit a 27b model in a 16GB at all; but - shout out to Unsloth - I managed to fit a Q3 model with a useable context into a single 16GB.

The new test for P40 vs P100 vs V100 was run using Qwen 3.8 27B at Unsloth Dynamic quants, namely UD-IQ3_XXS. Llama-bench runs included both q8 and f16 ctx quants. 200W power cap was introduced to make the cards a little cooler.

Qwen 3.8 27B UD-IQ3_XXS on llama-bench (pp512 / tg128, tok/s)

Table conclusions: ctx quant costs something, but not much - see the depth table below.

A small bonus - I dropped in my primary gaming card in the same rig to see how Radeon performs. While it had reasonable speeds (beating V100 in PP and catching up in TG), I was not able to get it stable with ROCm - it crashed regularly mid-task with no generated error. Since it was a "bonus", I did not dig into reasons, but still ran the comparison.

A proper 7900 GRE card introduction:

Going deeper

I extended the depth sweep to 16k and re-ran the ctx quant comparison:

Table conclusions: q8_0 ctx quant costs up to ~11% of TG @16k, and the gap grows with depth. P100 and 7900 GRE suffer the most.

Double Trouble

For dual card runs, I tested both tensor and layer splits (-sm tensor and -sm layer).

Table conclusions: for a dense model, tensor split yields a significant productivity increase; layer split - almost none. Layer split lands within noise of a single card, so it still buys you VRAM.

It's an agent! Run, Neo, run!

For the real world test, I decided to plug this model into my day-to-day setup and try some agentic coding. The task was performed on a dummy code generated by Opus; the model had to add a feature to a multi-module python program: read docs, pinpoint change, init git, perform the update, clean up, commit. To my surprise, all runs completed successfully.

The UD-IQ3_XXS quant allowed for a 100k context (102400 to be precise) with 97% VRAM utilization on 16GB. The 16GB limit was reserved for all cards (even these having more VRAM). The setup is: VScode -> omp -> llama-swap -> llama.cpp. For multi card runs, tensor split was used.

Llama.cpp params: --ctx-size 102400 -np 1 -fa on -b 2048 -ub 512 --cache-type-k q8_0 --cache-type-v q8_0 --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0 --presence-penalty 0 --repeat-penalty 1 --spec-type draft-mtp --spec-draft-n-max 3 --jinja --chat-template-file froggeric-qwen-fixed-chat-templates/chat_template.jinja --reasoning-preserve --chat-template-kwargs '{"reasoning_effort":"medium"}'

In this table: PP W/ TG W - weighted speed (normalized by the number of tokens generated in the task), Time/NT(s) - estimated "time per a normalized task" in seconds (34825 processed and 9840 generated tokens - average count of tokens used for the test task) using weighted PP/TG speeds

Several findings:

1. Real agentic work has significant variation for PP speed of ~50%. This is due to small processing batches, each still requiring the card to switch from TG to PP and back. That's the reason behind introducing weighted speeds rather than just averages.
2. Single P40 and single P100 are almost tied - P40 wins in PP and P100 wins in TG.
3. Double P40 setup gives significant decrease of the total processing time, while double V100 not as much.
4. V100 wins over the newer (and more expensive per GB) 7900 GRE in TG, while tied in PP. Don't forget the ROCm stability issues...
5. Agentic runs had MTP on - and here all cards had ~1.5x speedup compared to the llama-bench but the P40: only ~1.25x.
6. P40 and 7900GRE suffer the most from long-context operations. When testing P40 alone, I was blaming the slow memory - but seems something more is at play.

The price to pay

The table below is based on UD-IQ3_XXS PP/TG speeds.

This table is an estimate based on my rates ($0.291/kWh) and the primary setup I run (2x V100) under heavy load (a two-day-long heavy agentic task on Qwen 3.8 27B Q8 with full 262k ctx and np up to 4). Power was measured at the wall with a smart plug: 530W peak, ~130W host floor, cards averaging ~83% of the 200W cap. Draw for the other configs is modelled from that.

The actual number was the two-day Q8_0 measurement itself: 15.197 kWh at the wall for 5.73M generated tokens, which is 2.65 kWh / 1M TG, or $0.77 at my rate (includes PP). The table estimates include adjusting for Q3 vs Q8 TG speed: Qwen3.8 27B Q8_0 is ~25% slower on weighted TG and ~15% slower on weighted PP.

So the actual token-per-electricity winner is a single V100 - my 2xV100 setup falls slightly behind, but gives ~43% faster inference. I believe this to be a PCIe limitation - nvlinked V100 should perform better.

Comparing with cloud, TG only, OpenRouter Qwen3.8 27B reports a weighted price of 2.87$/1M tokens as of today. So on electricity alone I'm running significantly cheaper (~3.7x); GPU investment provided me with a server with no rate limits, an option to keep the data on my premises, and a personal pleasure of building a custom solution.

Overall I'm quite happy with my dinos and I believe 2xV100 is good enough unless a model will arrive that will make all 27B range obsolete. It feels fast and quite snappy, resonably fast at both PP and TG, and does not break the bank.

# Cracking the "Memory Wall": A Guide to Running 100GB+ MoE Models on a 16GB GPU via SSD mmap in llama.cpp

> 🚨 **WARNING:** The text of this article was edited and polished into its final form with the help of an LLM, since English is not my native language. I will also answer technical comments in the thread using an LLM. You can blame me for anything you like, but the source code, configuration scripts, architectural conclusions and inference logs are 100% real, obtained on my physical hardware and reproducible. Toxic people and armchair critics are asked to leave the thread immediately. I have zero tolerance for unsubstantiated complaints. Let's speak the language of computing-physics.

>

> **NOTE ON THE TITLE:** Reddit's mechanics completely block editing titles after publication. The topic title mentions the 125B class, based on early speculative specifications. As the official inference-environment metadata below clearly demonstrates, the model's actual scale is **176.94B parameters** with a physical file size of **104.08 GB**. This makes the achieved performance figures on a consumer 16-gigabyte card even more absurd. The same inference web UI, by the way, also confirms the remaining metrics: training context of **262,144 tokens**, vocabulary of **248,320 tokens**, **16 parallel slots**, supported modalities — **Vision, Video**. Server build — `b10729 (commit 458681e1d5)`.

---

### 💻 Tested Environment and Hardware Configuration

To ensure absolute transparency of the baseline metrics, all empirical metrics documented below were obtained using a purpose-built consumer workstation without corporate offloading:

* **Operating System:** Melawy Linux (an Arch-based distribution)

* **Kernel Architecture:** Linux XanMod 7.2.1 kernel (compiled natively for host optimization)

* **Processor (CPU):** AMD Ryzen 7 5700X (8 cores / 16 threads, configured on PCIe 4.0 lane maps)

* **Motherboard:** ASUS B550 (active PCIe 4.0 bus, the dashboard confirms GEN 4 @ 16x mode)

* **Memory (RAM):** 128 GB DDR4 running at 3600 MHz

* **Graphics Processor (GPU):** AMD Radeon RX 9070 XT with 16 GB of GDDR6 VRAM: RDNA 4 architecture (Navi 48 XT die, gfx1201), 256-bit bus with bandwidth of **up to 640 GB/s**, 64 MB of Infinity Cache and 128 built-in AI accelerators, ROCm/HIP stack

* **Storage Subsystem:** A standard SATA III solid-state drive operating over a 6 Gbit/s interface (maximum sequential read bandwidth of **~550 MB/s** as tracked under real conditions).

* **Model:** Qwen3.8-Flash-Next-Uncensored-**i1**, GGUF quant **i1-Q4_K_S** (quantization with the iMatrix importance matrix, ~4.25 bits per weight) + multimodal projector **mmproj F16**, file size **104.08 GB**, 176.94B parameters, roughly 6B active per token. Architecture: Qwen 4 preview (qwen4exp) with a 51B N-gram PLE table.

---

### 1. The Physics of MoE Inference: Bypassing the Memory Shortage via mmap and the Page Cache

With a GGUF file size of **104.08 GB**, standard workstations inevitably hit an out-of-memory (OOM) error when trying to load the model into memory entirely. However, for sparse Mixture of Experts (MoE) architectures, this limitation is overcome thanks to the OS-level virtual memory mechanism `mmap` and layer-offloading technology.

The architectural hallmark of Qwen3.8-Flash-Next lies in its high sparsity: only a small fraction of expert layers is activated to process a single token — **about 6B parameters** out of 176.94B. For classic dense architectures, `mmap` on a slow drive is inefficient: to emit a single token, a dense 70B model must read all of its weights, which over a SATA III interface (550 MB/s) yields a hard ceiling of ~0.2 tokens/sec. MoE solves this problem, since the router directs the token to only 1–2 experts out of the entire pool.

The launch scheme utilizes two `llama.cpp` backend orchestration parameters:

* **Omitting the `--load-mode` flag:** By default, `auto` mode is used (`mmap, unless a device does not support it`). At startup, the engine does not read the model weights into physical memory; instead, it maps the 104-gigabyte file into the OS virtual address space via the `mmap` system call.

* **Using the `-cmoe` (`--cpu-moe`) flag:** This argument moves the MoE expert blocks and PLE tables out of VRAM into system memory, preventing the 16-gigabyte card from overflowing.

Thanks to the `--n-gpu-layers 99` parameter, the base dense layers, embedding matrices and the vision multimodal projector (`mmproj`) are pinned inside the **GPU VRAM** (the allocation is **15.863/15.922 GiB** — 98% of the video memory). The KV cache for all 16 slots is pinned in **system RAM** via the `--kv-unified` and `--cache-ram 8192` flags. As the context grows from 56k to 132k tokens, the amount of occupied VRAM stays unchanged, while RAM consumption increases by ~1 GB (6.74 → 7.75 GB).

`llama.cpp` employs a **layout-blind expert streaming** mechanism: the GGUF file layout is regrouped so that experts frequently called together lie consecutively on disk (`adjacent layout`), and asynchronous **Direct I/O** with `prefetching` fetches an expert from disk several cycles before the computation.

#### Debunking the "Disk Dilemma": Memory Utilization Analysis

A question often arises: *"If about 6B parameters (~3.18 GB of data) are activated to process a token, then on a SATA III SSD the speed should drop to 0.16 tokens/sec. Where do the real 12–17 t/s come from?"*

The SATA III drive is a bottleneck only during the first minutes of the server's "cold" start. Since the host has **128 GB of RAM** installed and the entire model weighs **104.08 GB**, the operating system fully absorbs the file into system memory as it interacts with the model.

The output of the `free -g` command clearly demonstrates this distribution physics:

```text

total used free shared buff/cache available

Mem: 125 13 11 0 102 112

Swap: 141 0 141

```

* **used = 13 GB:** The process's pure anonymous memory (anonymous memory) — the slots' KV cache, the server runtime and the OS's own needs. This is exactly why `htop` shows the green bar at only the ~8.56G level.

* **buff/cache = 102 GB:** Our entire 104-gigabyte GGUF file has fully settled into the RAM page cache. The Linux kernel counts these pages as disk cache, giving the engine instant access to the expert weights at host RAM speed.

* **available = 112 GB:** The OS considers this memory free, since the mmap pages are not locked permanently (`--mlock` is off) and can be evicted if necessary.

* **Swap = 2.65 MB out of 142 GB:** Swap usage is minimal. Thanks to `vm.swappiness = 10`, the kernel does not evict mmap pages to disk, and the enabled **ZSWAP** mechanism with `zstd` compression intercepts the rare eviction attempts, eliminating I/O stalls.

After the context warms up, the MoE experts stop being requested from disk. The `mmap` mechanism turns system memory into a high-speed array from which expert weights are transferred to the GPU at the speed of the host memory bus, completely bypassing the SATA interface limitations.

#### What the `i1` Marker in the Weights Filename Means

The `i1` tag indicates the version of the **iMatrix (Importance Matrix)**. Before quantization, the model is calibrated on a dataset, the algorithm tracks neuron activations and annotates the weights: critical ones are preserved with higher precision, secondary ones are compressed more aggressively.

On the inference side, no additional computation occurs — the matrix is baked into the quantization structure and is not recomputed on the fly. The GPU's compute units dequantize the `i1-Q4_K_S` weights into FP16, and the matrix multiplications are executed by the built-in AI accelerators (in the RDNA 4 case, peak performance reaches **98 TFLOPS FP16**). The GPU chip spends the same cycles on dequantization as it would on a regular quant — the importance matrix does not slow down generation. The `i1-Q4_K_S` quant delivers accuracy close to the average `Q4_K_M` while saving 5–7 GB of memory.

#### Limitations of Scope: What Happens on 32GB or 64GB RAM Systems?

The lazy-caching mechanics via `mmap` described above remain stable only because the physical host RAM (128GB) exceeds the net file size of the model (104GB) minus the layers offloaded to VRAM. Running this exact quant on configurations with less memory will drastically alter the system's behavior:

1. **On a 32GB RAM Configuration:** An OOM error will not occur, as `mmap` will successfully map the virtual addresses. However, due to a severe lack of physical space, the Linux kernel will be unable to retain the read pages in the Page Cache. It will be forced to continuously drop old weights to clear space for new ones. This triggers heavy disk thrashing (I/O Thrashing), crushing the speed down to **0.1–0.2 tokens per second**. The model will essentially read from the drive on every single token.
2. **On a 64GB RAM Configuration:** Out of the 104GB model, a portion (~16GB) is offloaded to VRAM, while about 50GB of the remaining weights will fit into physical RAM. The remaining ~38GB of the model will have to be constantly re-read from the drive in cycles during inference. The generation speed will become highly erratic and bottle out in the range of **1–3 tokens per second**, completely bound by the drive's throughput.

**The Bottom Line:** This method of streaming 100GB+ MoE models via passive `mmap` is highly efficient only when 128GB of host RAM is available. For systems limited to 32GB or 64GB RAM, running this setup works strictly as a Proof of Concept (PoC) but is completely impractical for real-world tasks without dropping down to much harsher quants (like IQ2 or IQ3).

#### Architectural Nuances of the KV Cache and Vision

When scaling the context, the `--cache-type-k f16`, `--cache-type-v f16` parameters and using the F16 version of the vision projector (`mmproj`) are mandatory. Quantizing the KV cache (Q4 or Q8) on long token sequences leads to an avalanche-like accumulation of rounding errors and a breakdown of the network's attention (infinite loops, loss of logic). F16 precision guarantees perfect mathematical signal purity.

#### 🔄 Alternative Streaming Paradigms in the Ecosystem

The concept of streaming expert weights from disk is quickly becoming the main optimization vector in modern inference backends. For example, the `sglang-ssd-stream` extension implements an independent asynchronous paging pipeline for NVIDIA architectures. Instead of the OS-level `mmap` mechanism, it uses a custom engine built on the low-level Linux kernel API — **`io_uring`** in Rust for processing quantized FP4 streams (`Qwen3.8-Flash-Next-NVFP4-SSD-Stream`).

This mechanism intercepts requests to the expert predictive-lookup (PLE) tables even before the GPU compute graph reaches the required point, asynchronously reading the needed 4 KB pages from the NVMe drive into pinned memory and processing the tensors on a separate CUDA stream in parallel with the computations.

Despite the high efficiency of this approach on corporate-grade architectures (RTX PRO 6000 or DGX Spark class), the native `mmap` implementation (the default `--load-mode auto` mode) in the latest `llama.cpp` commits offers a far more accessible, hardware-agnostic alternative. It works "out of the box" in any standard Linux environment, including consumer platforms based on AMD ROCm.

---

### 2. A Universal CLI Blueprint for Compiling from Source (Any Linux Distribution)

To squeeze out the maximum prompt-processing speed (`prefill`) and completely eliminate runtime abstractions, `llama.cpp` must be built locally from source. Such a build forcibly enables native vector instructions for your CPU. Even on an ordinary mid-range CPU without hardware AVX-512 support (like my AMD Ryzen 7 5700X on the `-march=znver3` microarchitecture), the right compiler optimization flags guarantee that the host machine handles memory-page handling and virtual-address routing without creating a "bottleneck" for the GPU.

The production build runs on commit `b10729` (`b10729.r0.g458681e1d5` — the same build the web UI shows in Model Information) with `GGML_HIP_GRAPHS=ON` enabled: it is precisely this flag responsible for the `graphs reused = 8195` line in the inference logs.

Below are the optimized build profiles for both graphics platforms:

#### Variant A: Build Profile for AMD Hardware (ROCm 7.x Stack)

We explicitly override the paths, binding the build system directly to AMD's native LLVM Clang compiler, bypassing the unstable `hipconfig` wrappers. The high-performance `mold` linker (`-fuse-ld=mold`) is also used to speed up the build:

```bash

cd llama.cpp

cd tools/ui && npm ci && npm run build && cd ../..

export ROCM_PATH="/opt/rocm"

export PATH="/opt/rocm/llvm/bin:/opt/rocm/bin:\$PATH"

export LD_LIBRARY_PATH="/opt/rocm/llvm/lib:/opt/rocm/lib:\$LD_LIBRARY_PATH"

export HIPCXX="/opt/rocm/llvm/bin/clang++"

export CC="/opt/rocm/llvm/bin/clang"

export CXX="/opt/rocm/llvm/bin/clang++"

export HIP_PLATFORM=amd

# for the Ryzen 5700X

export CFLAGS="-march=native -O3 -pipe -fno-plt"

export CXXFLAGS="$CFLAGS"

export LDFLAGS="-Wl,-O1 --as-needed -fuse-ld=mold"

# for the RX 9070 XT 16 GB

cmake -S . -B build -G Ninja \

-DAMDGPU_TARGETS="gfx1201" \

-DBUILD_SHARED_LIBS=ON \

-DCMAKE_AR="/opt/rocm/llvm/bin/llvm-ar" \

-DCMAKE_BUILD_TYPE=Release \

-DCMAKE_CXX_COMPILER="\$CXX" \

-DCMAKE_C_COMPILER="\$CC" \

-DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=mold" \

-DCMAKE_INSTALL_PREFIX='/usr' \

-DCMAKE_RANLIB="/opt/rocm/llvm/bin/llvm-ranlib" \

-DCMAKE_SHARED_LINKER_FLAGS="-fuse-ld=mold" \

-DGGML_ALL_WARNINGS=OFF \

-DGGML_ALL_WARNINGS_3RD_PARTY=OFF \

-DGGML_BUILD_EXAMPLES=OFF \

-DGGML_BUILD_TESTS=OFF \

-DGGML_CUDA_FA_ALL_QUANTS=ON \

-DGGML_HIP=ON \

-DGGML_HIP_GRAPHS=ON \

-DGGML_LTO=ON \

-DGGML_NATIVE=ON \

-DGGML_RPC=ON \

-DLLAMA_BUILD_SERVER=ON \

-DLLAMA_BUILD_TESTS=OFF \

-DLLAMA_BUILD_WEBUI=ON \

-DLLAMA_OPENSSL=ON \

-DLLAMA_USE_SYSTEM_GGML=OFF \

-Wno-dev

ninja -C build

```

#### Variant B: Build Profile for NVIDIA Hardware (CUDA Backend)

This profile, extracted from a production deployment's parameters, forcibly enables optimization for the local CPU via `-DGGML_NATIVE=ON`, Flash Attention shaders for all quants and math acceleration through cuDNN:

```bash

cd llama.cpp

cd tools/ui && npm ci && npm run build && cd ../..

export CFLAGS="-march=native -O3 -pipe"

export CXXFLAGS="$CFLAGS"

export LDFLAGS="-Wl,-O1 --as-needed -fuse-ld=mold"

cmake -S . -B build -G Ninja \

-DBUILD_SHARED_LIBS=ON \

-DCMAKE_BUILD_TYPE=Release \

-DCMAKE_INSTALL_PREFIX='/usr' \

-DGGML_ALL_WARNINGS=OFF \

-DGGML_ALL_WARNINGS_3RD_PARTY=OFF \

-DGGML_BUILD_EXAMPLES=OFF \

-DGGML_BUILD_TESTS=OFF \

-DGGML_CUDA=ON \

-DGGML_CUDA_FA_ALL_QUANTS=ON \

-DGGML_CUDNN=ON \

-DGGML_LTO=ON \

-DGGML_NATIVE=ON \

-DGGML_RPC=ON \

-DLLAMA_BUILD_SERVER=ON \

-DLLAMA_BUILD_TESTS=OFF \

-DLLAMA_BUILD_WEBUI=ON \

-DLLAMA_OPENSSL=ON \

-DLLAMA_USE_SYSTEM_GGML=OFF \

-Wno-dev

ninja -C build

```

---

### 3. The Ready-to-Use llama-server Deployment Script

The compiled binary is launched with the following parameter matrix. Note the **complete absence of the `--load-mode` flag**: llama.cpp enables the memory-projection mode `auto` (i.e. mmap) by default.

The profile includes continuous-batching architectures (`--cont-batching`), the unified KV cache (`--kv-unified`), which pins a single context pool in system RAM with an 8 GiB budget (`--cache-ram 8192`) and caches idle slots (`--cache-idle-slots`), plus hardware-accelerated Flash Attention (`--flash-attn on`) to ensure linear scalability on ultra-deep contexts. The `--threads 8` / `--threads-batch 16` flags map directly onto the 5700X architecture: 8 physical cores for generation, 16 SMT threads for batch prompt processing:

```bash

/usr/bin/llama-server \

-m Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_S.gguf \

--mmproj Qwen3.8-Flash-Next-Uncensored.mmproj-f16.gguf \

--image-min-tokens 1024 \

--jinja \

--no-skip-chat-parsing \

--reasoning on \

--reasoning-preserve \

--reasoning-format auto \

-cmoe \

-fit off \

--n-gpu-layers 99 \

--threads 8 \

--threads-batch 16 \

--batch-size 2048 \

--ubatch-size 512 \

--parallel 16 \

--ctx-size 262144 \

--context-shift \

--cont-batching \

--kv-unified \

--cache-prompt \

--cache-ram 8192 \

--cache-idle-slots \

--cache-type-k f16 \

--cache-type-v f16 \

--flash-attn on \

--temp 1.0 \

--top-k 20 \

--top-p 0.95 \

--min-p 0.00 \

--repeat-last-n 512 \

--repeat-penalty 1.00 \

--presence-penalty 0.00 \

--host 0.0.0.0 \

--port 11441 \

--reuse-port

```

The server runs as a user-level systemd unit (`Restart=always`, `Nice=10`): the `LimitMEMLOCK=infinity` and `LimitNOFILE=65536` limits, the cgroup constraints `MemoryHigh=115G` / `MemoryMax=125G` and `OOMScoreAdjust=-500` protect the mapped model and the hot context pages from swapping and make the process the last candidate for killing by the OOM killer. From the ROCm scaffolding: `ROCM_ALLOCATOR_STRATEGY=2`, `HSA_ENABLE_SDMA=1` and `ROCBLAS_TENSILE_WARMUP=0` (limits rocBLAS's aggressive memory pre-allocation). At the host kernel level, things are deliberately tuned for mmap inference: the block-device I/O scheduler for the model drive is switched to `none` (the kernel doesn't spend time sorting I/O requests during random weight reads), via `sysctl` are set `vm.swappiness=10` (swap is engaged only at the most critical moment before OOM), `vm.vfs_cache_pressure=50` (the kernel prefers to retain GGUF mmap pages in the page cache) and `vm.max_map_count=1000000` (the critically necessary limit on the number of simultaneously mapped memory regions for the 104-gigabyte file), and via bootloader kernel parameters `zswap` is enabled with `zstd` compression and a `zbud` pool — redundant pages are compressed in RAM and only then go to the SSD.

---

### 4. Performance and Context-Scaling Log Analysis

All metrics below were captured by the host `melawy-linux-lera` (the server listens on `0.0.0.0:11441`) from real `llama-server` logs (journalctl) and `htop` dashboards (RAM, swap, load average) and `nvtop` — yes, `nvtop` has long supported AMD/ROCm excellently, showing VRAM, clocks, power draw and utilization straight from the driver. The server with the model onboard has been running for over 17 hours without a restart without a single page swapped out, and its main thread holds 100% of one 5700X core (TIME+ on one thread — 1h55m of machine time). Analysis of the server's runtime under real multitasking scenarios demonstrates high linearity and reproducibility of the metrics:

#### A. Prompt Processing Speed (Prefill)

* **Peak prefill throughput:** Reaches a ceiling of **217.05 tokens per second** — a dense continuous block of 4,582 tokens is processed in 21.28 seconds at an average rate of 215.30 t/s (the first log excerpt below).

* **Sustained sequential loading:** Under a constant rolling prompt load, prefill performance predictably grows along with the batch: the log's progress lines show **128.09 → 206.34 → 213.76 → 217.05 t/s** on a short prompt, and on the 69k prompt (69,282 tokens in 400.28 seconds) the bar holds at **174.15 → 173.36 → 173.16 t/s** with an average rate of **173.08 t/s** (5.78 ms per token).

* **Dynamic slot multiplexing:** When several concurrent requests are active via `--parallel 16`, the prefill speed per stream drops to roughly **35 t/s**. This is the expected mathematical distribution: the inference daemon dynamically splits the total VRAM bus bandwidth and GPU cores among the active independent context tracks. Indirectly this is even visible in `htop`: with 16 slots the kernel holds `363 kthr: 11 running`, and the load average of the 8-core 5700X stays at the **7.36** level — batch expert processing genuinely utilized 16 CPU threads.

* **GPU power package:** During "thinking" generation, the GPU (`nvtop`) draws only **81–85 W out of 330 W** at **47%** core utilization (effective utilization — **11%**) and 98% video-memory occupancy; during image processing and the subsequent prefill, consumption jumps to **230–240 W**, peaking at **248–300 W** out of 330 W. Core clock — **2669 MHz**, VRAM — **1258 MHz**, temperature — **54 °C** at **30%** fan speed. The bottleneck of the system is not the GPU but the speed of delivering expert weights from the SSD into RAM: in the regular generation mode the GPU burns only a quarter of its power package.

#### B. Token Generation Speed (Eval Throughput)

* **Short context (~9.4k tokens, task 547 in the excerpt below):** Baseline generation speed is **15.22 t/s** (instantaneous values reach up to 15.62 t/s) — a full cycle of "introduce yourself, write a sorting algorithm in Rust, analyze Tatyana's letter to Onegin".

* **Medium context (21.6% of the window, 56,623 tokens):** VRAM allocation is firmly pinned at 15.777 GiB out of 15.922 GiB. The active physical RAM allocation delta is only 6.74 GB. Text generation runs at **13.0 t/s**.

* **Deep context (27.1% of the window, 71,127 tokens, task 0 id 15 in the excerpt below):** 1,846 tokens generated in 134 seconds — **13.74 t/s** at **72.76 ms per token**, the rolling three-second window `tg_3s` stays within the **13.5–13.9 t/s** corridor, and `graphs reused` grew to 1838.

* **Deep context expansion (50.7% of the window, 132,907 tokens):** VRAM allocation holds perfectly steady at 15.781 GiB with zero memory leak. The system RAM delta increases minimally — to 7.75 GB. Generation speed drops by less than 1 token per second relative to the previous tier, settling at a stable **12.12 tokens per second**.

```text

// Excerpt from the inference log (task 547 — 7,691 generated tokens)

prompt eval time = 5241.85 ms / 801 tokens ( 6.54 ms per token, 152.81 tokens per second)

eval time = 505246.75 ms / 7691 tokens ( 65.70 ms per token, 15.22 tokens per second)

total time = 510488.60 ms / 8492 tokens

graphs reused = 8195

stop processing : n_tokens = 9425, truncated = 0

// Next task 8243 — prefill of a dense block of 4,582 tokens

prompt processing, n_tokens = 618, progress = 0.13, t = 4.82 s / 128.09 tokens per second

prompt processing, n_tokens = 2666, progress = 0.58, t = 12.92 s / 206.34 tokens per second

prompt processing, n_tokens = 4066, progress = 0.89, t = 19.02 s / 213.76 tokens per second

prompt processing, n_tokens = 4578, progress = 1.00, t = 21.09 s / 217.05 tokens per second

prompt eval time = 21282.13 ms / 4582 tokens ( 4.64 ms per token, 215.30 tokens per second)

graphs reused = 8195

// Third excerpt — task 0 (id 15): a prompt of 69,282 tokens + generation of 1,846 tokens

prompt processing, n_tokens = 67686, progress = 0.98, t = 388.67 s / 174.15 tokens per second

prompt processing, n_tokens = 68766, progress = 0.99, t = 396.07 s / 173.36 tokens per second

prompt processing, n_tokens = 69276, progress = 1.00, t = 400.08 s / 173.16 tokens per second

prompt eval time = 400278.31 ms / 69282 tokens ( 5.78 ms per token, 173.08 tokens per second)

eval time = 134248.54 ms / 1846 tokens ( 72.76 ms per token, 13.74 tokens per second)

total time = 534526.86 ms / 71128 tokens

slot print_timing: n_gen = 1105, tg = 13.76 t/s, tg_3s = 13.85 t/s

graphs reused = 1838

stop processing : n_tokens = 71127, truncated = 0

```

### 🏁 Conclusion

Losing less than 1 token per second when doubling the context — from 56,623 to **132,907 tokens** (13.00 → 12.12 t/s with pinned video memory) — proves the point: competent low-level compilation of open-source software completely outperforms the brute-force scaling of hardware power.

Running a 104 GB model at 12+ t/s on a consumer gaming card is not an experimental hack, but a production-ready reality. By forcing the OS's built-in memory manager to run `mmap` cycles over the sparse expert blocks via the `--load-mode auto` mode (llama.cpp's default), custom architectures completely break the commercial monopoly of expensive corporate workstations.

### Contributors to this article:

* 👩🏼 **Valeria Fadeeva** — lead developer and founder of **Melawy Linux** (an Arch-based distribution).

With the help of neural network models:

1. **Google Gemini 1.5 Pro** — assistance in compiling the material, web searching and sorting through scattered information from the internet.
2. **Qwen-3.8-Flash-Next-Uncensored (i1-Q4_K_S)** — detailed fact-checking on real hardware, text and benchmark optimization.

I kept running into the same problem with local AI tools: running the model was the easy part, but using it for actual work still meant jumping between documents, email, browser tabs, notes, and separate chat apps.

So I built Vyact, an open-source desktop workspace that connects those pieces around the model you choose.

The workflow I wanted was:

- Find a GGUF or MLX model that fits the available RAM and VRAM

- Download it and run it locally through llama.cpp, llama-swap, or MLX

- Index documents and inspect the exact passages retrieved for an answer

- Bring Gmail and Google Drive files into the same conversation

- Continue working with webpages through a connected Chrome extension

The Chrome extension is an important part of the project. It can summarize the current page, translate foreign-language pages, and send either selected text or the entire page to Vyact as conversation context. This makes it possible to ask follow-up questions without repeatedly copying and pasting content between the browser and a separate chat app.

It also includes a Netflix language-learning mode with dual subtitles, subtitle navigation, repeat playback, automatic pause controls, and short AI explanations focused on the language areas the learner has selected as difficult.

These browser workflows do not always require a large flagship model. By choosing a smaller quantized local model that fits the machine, practical tasks such as page summarization, translation, selected-text questions, and subtitle explanations can remain useful on more modest hardware. The actual speed and answer quality naturally depend on the selected model and device.

The screenshots show the current app rather than mockups.

Vyact supports GGUF models on Windows and Apple Silicon Macs, with native MLX support on Apple Silicon. Hosted providers and custom OpenAI-compatible endpoints are optional.

When a Vyact-managed local model is selected, the chat context is not sent to an external AI provider. The project is released under AGPL-3.0.

GitHub:

https://github.com/vyact/vyact

I’m trying to decide what to improve next. For people who already use local models, which part of the workflow is still the most frustrating: choosing a model, document retrieval, connecting real work context, browser integration, or initial setup?

I'm pretty happy with my setup below, which is my daily driver for primarily OpenClaw, but can we push Qwen 3.6 35B A3B GGUF any harder on our weak hardware without sacrificing intelligence and maximum context? Or with a different model (which I doubt right now)?

Show us your t/s and run params on similar hardware!

I've maxed out what I can do at an average 27 t/s eval and 700 t/s prompt eval with the following setup which is optimized for OpenClaw with massive 14K+ token system prompts and multi-agent context switching. If I optimize purely for speed, I can reach 30 t/s, but I made a few trade-offs for OpenClaw stability. I also find leaving MTP off to be better over long runs even though at short runs I can get 30 t/s as well. I've found it not worth the +3 t/s.

• GPU: NVIDIA GeForce RTX 3070 Ti (8 GB VRAM) with 1920 MHz overclock @ 925 mV (0.925V) locked curve and +800 MHz (GDDR6X)
• CPU: Intel Core i9-9900KF (8C / 16T @ 5.1 GHz overclock)
• RAM: 32 GB DDR4-3200 @ CL16 (XMP 2)
• MOTHERBOARD: ASUS ROG STRIX Z390-E GAMING
• PCIe Bus: Gen 3.0 x16
• OS: Windows 11 / 64-bit
• LLAMA: llama.cpp-turboquant-mtp (Tom Turboquant beats Atomic Turboquant for me by 10 t/s)
• MODEL: Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf (Non-MTP)

-m "Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf"
--mmproj "mmproj-BF16.gguf"
-np 1
-t 8
-tb 16
-b 4096
-ub 2048
-c 131072
--n-gpu-layers 99
--n-cpu-moe 999
--no-mmap
--mlock
--cache-type-k turbo3
--cache-type-v turbo2
-fa 1
--chat-template-file "froggeric-Qwen-Fixed-Chat-Template.jinja"
--jinja
--chat-template-kwargs '{"preserve_thinking": true}'
-fit off
--kv-unified
--reasoning-format deepseek
--ctx-checkpoints 32
--checkpoint-min-step 512
--cache-ram 20480
--image-min-tokens 1024
--no-mmproj-offload
--no-warmup

I can also hit 30 t/s by adjusting --n-cpu-moe to 31 or 32 without mlock, but it ultimately crashes over long runs, and with the above params, it's rock solid.

The pricing is erratic and seems odd, I plan on picking one up, is there any reason not to get the 72gb over the 48gb? 72gb is about 2-400 dollars cheaper but more VRAM? Is there any reason not to pickup the 72gb over the 48gb ? Why the price misconfiguration? I'd imagine the 72gb would be discontinued possibly and they are just getting rid of stock?

I can't justify picking up a 2nd RTX pro 6000 at current prices but could justify 2x 5000 72gbs for the extra vram.

rtx pro 6000, 13,899 - 14,999

rtx pro 5000, 48gb - 8,499 - 8,999

rtx pro 5000 72gb, 8,799

What would you consider?

*
tripped up over price and forgot to spell. "than"

I am considering getting this build for my work. I have a budget of around 60k. Mostly will be running a mixture of small models like qwen 3.8 27b, with expectation to also be able to run oversized models like GLM 5.3 at quants like Q4.

Do you have any thoughts or recommendations different options?

Here is the parts list I am looking at:

AMD Ryzen Threadripper PRO 9985WX — 64C/128T — B&H — $7,894.00

ASUS Pro WS WRX90E-SAGE SE — B&H — $1,299.99

2× PNY NVIDIA RTX PRO 6000 Blackwell Max-Q — 96GB ECC GDDR7 — B&H — $16,999.99 each / $33,999.98 total

TEAMGROUP T-Create Master 384GB — 8×48GB DDR5-6000 ECC RDIMM — Newegg — $10,199.99

2× WD_BLACK SN850X 8TB PCIe 4.0 NVMe SSD — Best Buy — $1,699.00 each / $3,398.00 total

Samsung 990 PRO 2TB PCIe 4.0 NVMe SSD — Best Buy — $389.99

MSI MEG Ai1600T PCIE5 — 1600W 80+ Titanium PSU — B&H — $619.00

Sliger CX4200a 4U Rackmount Chassis — Sliger — $289.00

Asetek 836SA-M1 360mm Threadripper/TR5 AIO — Sliger — $250.00

3× Noctua NF-A12x25 PWM 120mm Fans — Sliger — $75.00 total

Sliger GDRAIL-20XX-B General Devices 20" Rack Rail Kit — AVADirect — $123.04

CyberPower PR1500LCD Smart App Sinewave UPS — 1500VA / 1500W — B&H — $700.95

Ubuntu 24.04 LTS — $0.00

Total: $59,238.94 before tax (tax isn't real, tax can't hurt us (shhhhhh let me live in delerium))

I have a dual RTX PRO 4000 setup. I get around 35 tokens per second with Qwen 3.8 27B Q6. But above 100k context, it drops down to around 20. I was considering an upgrade in the near future and I’m just curious what numbers people with the RTX 6000 are getting.

On paper the RTX 6000 is paper because of the higher bandwidth. But when I tested both on runpod. I wasn’t getting a huge difference. I thought I would see a huge difference in the tokens per second but it was not too much.

Hey everyone! I've been working on a little side project I wanted to share: latex-ocr, a standalone formula OCR model — you feed it an image of a math formula, it spits out the LaTeX source.

The main hook: it's only 67M parameters, so it runs comfortably on a laptop CPU. No GPU, no 300M-parameter monster to load. It's a CoCa-style model (contrastive captioner adapted for OCR), and despite the small size it beats the 107M UniMER-tiny baseline and gets pretty close to the 325M one on plain formulas.

The part I'm actually most proud of is the dataset. Real papers don't just use plain symbols — you see \mathbb{R}, \mathcal{F}, \mathfrak{g} everywhere, and existing OCR datasets basically ignore font styles, so models trained on them can't read (or hallucinate) those macros. So I rebuilt ~1.3M formulas with a MathJax → SVG → PDF → PNG pipeline and injected font-style macros with semantic heuristics (number sets → \mathbb, vectors → \mathbf, differentials → \mathrm). On that styled test set it clearly outperforms all the baselines — fair warning though, those baselines are zero-shot on styled data, so take that comparison with a grain of salt. The plain-split numbers are the like-for-like ones.

Everything is open: model weights and dataset on Hugging Face, training recipes included if you want to reproduce or fine-tune it yourself, MIT license. There's also a FastAPI server and a Gradio web UI, so you can drag-and-drop an image and see the LaTeX with a rendered preview.

Repo: https://github.com/PadishahIII/latex-ocr Model: https://huggingface.co/PadishahIIIXXX/latex-ocr Dataset: https://huggingface.co/datasets/PadishahIIIXXX/latex-ocr-dataset

Happy to answer questions about the training setup, the data pipeline, or anything else. Would love feedback — especially if you try it on your own gnarly formulas and it breaks, that's genuinely useful.

Hey guys,

What are you guys using small LLMs for?

Except coding.

Thanks.

I tested Qwen 3.8 Flash at quant MXFP8, and DeepSeek V4 Flash at the original quant, and I noticed two things:

1. DeepSeek seems to fail more often than Qwen, and I think Qwen does a better job of trying to verify its work.
2. I get random chineese characters in the DeepSeek output, for example: "Option 1 is nicer — zero changes on your side盗窃. Want me to go with that?" However, I find DeepSeek's text to be easier to read than Qwen's.

I plan on doing more testing before I decide which I'm going to stick with. Anyone else notice the occasional Chinese character output?

I do like both models though!

UPDATE: Sol, who setup the vLLM config, says my top_p setting was 1.0, which may explain the chineese characters. I'm going to retest with the recommended 0.95 setting.

Hello,
I am reading a lot of positive comments about Qwen3.8 27b as a local coding agent model.

I preordered a MacBook Pro M5 (not M5 Pro CPU) with 32GB RAM. Has anyone benched Qwen3.8 on this MacBook and can tell me their t/s and general experience with working with it? I am planning on using llama.cpp

I'm afraid that I should have used some more money to get the M5 Pro with 48GB...

I’m the co-founder of axem and this is a project we've been building. It's Apache 2.0 and I'm sharing the GitHub repo here for feedback.

This isn't really aimed at the "one model on one GPU" setup. Our problem started when we needed to run multiple models across GPU nodes, scale them independently, route requests between replicas, and keep the entire stack inside infrastructure we controlled.

Current setup:

• vLLM for inference
• llm-d for multi-instance orchestration
• multiple models running and scaling independently
• KV-cache-aware scheduling
• internal OCI registry for container images + model weights
• OpenAI-compatible API
• The entire platform is managed as infrastructure as code
• interactive installer that runs from Docker against an existing Kubernetes cluster
• can operate fully air-gapped with no cluster egress

It currently works with on-prem RKE2 as well as EKS/GKE/AKS.

GitHub: https://github.com/axem-solutions/shaide

I'm curious how people here handle this once a setup grows beyond a single machine.

If you're running several models in parallel, what are you using for routing/orchestration?

I have been trying to set up a pipeline for my video generations to handle really detailed, multi-modal prompts. I want to keep the final rendering local for the granular control, but once I start stacking reference images, specific camera directions, and detailed lighting notes, the local text encoders completely fall apart. They just lump all the inputs into one block, which makes the camera directions bleed into the character details midway through a clip.

I initially tried to build a step in my workflow to run a heavy language model upstream to clean up and structure the promt before passing it to the video generator. The issue is that the text encoder alone eats up 16GB to 20GB of VRAM, so my mid-tier setup just crashes instantly.

This is my current workaround: to pipe the heavy prompt context out to the MiniMax H3 API to handle the parsing on their servers. It processes the relationships between the text, images, and reference video, and then passes a much more compact, structured set of instructions back down to my local base model.

Basically, I am outsourcing the heavy lifting of the context parsing to an API, and letting my local GPU handle the actual video render for the last mile. This completely frees up my local VRAM and gives me way cleaner prompt adherence without the constant system crashes.

Are you all relying on local text encoders and just trimming your prompts to survive, if so what are the tricks that you use to keep VRAM capacity in check?

Just wanted to see if the community could offer any suggestions for speed improvements, or am I nearing the theoretical max for this hardware?

I've seen a few suggestions on other posts, but they tend to boil down to DFlash 2, RDNA 4, or CUDA related boosts. I tend to run long context, so DFlash doesn't appeal to me, because it drops well below MTP not long into my use cases.

Typical speeds are 1k prompt processing which degrades over context pretty quick and 50 - 90 tokens per second, depending on the type of response (basic code is 90, complex though is around 50).

I've tried RocM and vLLM briefly with worse results. Running the Unsloth q4_k_XL quant.

  - GPU: AMD Radeon RX 7900 XTX 24GB (Navi 31 / gfx1100)                                                                               
  - CPU: AMD Ryzen 7 5700X3D (8 cores / 16 threads)                                                                                    
  - RAM: 46 GiB                                                                                                                        
  - OS: Nobara Linux 44 (KDE Plasma), kernel 7.2.0-202.nobara.fc44                                                                     

  Graphics stack:                                                                                                                      
  - Driver: RADV via Mesa 26.2.1 (mesa-vulkan-drivers-freeworld)                                                                       
  - Vulkan API: 1.4.354                                                                                                                

  Software:                                                                                                                            
  - llama.cpp 0.3.0-dev (build 182, commit c1d0e7a), Vulkan build (GGML_VULKAN=ON)                                                     
  - Model: Qwen3.8-27B-UD-Q4_K_XL.gguf (unsloth, 16.35 GiB, embedded MTP draft head)                                                   

  llama-server flags:                                                                                                                  
  - --ctx-size 131072                                                                                                                  
  - --parallel 1                                                                                                                       
  - -ctk q8_0 -ctv q8_0                                                                                                                
  - -fa on                                                                                                                             
  - --jinja                                                                                                                            
  - --reasoning-preserve                                                                                                               
  - --no-mmap                                                                                                                          
  - -t 5 -tb 8 -b 2048 -ub 512                                                                                                         
  - --metrics                                                                                                                          
  - -ngl 99                                                                                                                            
  - --reasoning on --reasoning-effort low  (subject to change)                                                                                            
  - --reasoning-budget 131072                                                                                                          
  - --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00 --presence-penalty 0.0                                                             
  - --spec-type draft-mtp --spec-draft-n-max 3                                                                                         

  Results (llm-benchmark, 65536-token context, thinking ON):                                                                           
  - Generation: 72.8 tok/s     

I was looking to buy myself an RTX 3090, of course I have come to realise that looking for something brand new would either be impossible or marked up by an obscene amount. But on sites like Alibaba they have mentioned Brand New in house GPUs from companies such as PELADN. My question is that are these GPUs that they manufactured just old used/mined cards that have been repacked by them which allows them to list it as "Brand new". Much appreciated

We wanted to see If we can make the code review cheaper and if cheaper model still holds up against a frontier one so we benchmarked two local models against a frontier model using real production PRs.

The Setup

Three models through the exact same review agent—same tools, same prompts, same review code path:

- 1x hosted frontier model
- 2x 27B models on a single H200 (bf16 via vLLM)

Cases were real merge requests from a production codebase, replayed at their original commits with full repo access. It's a proper agentic loop: reads the diff, greps the repo, opens files, and follows references across 12+ turns.

Single-shot prompt benchmarks give a false sense of security here—a model that looks fine on an isolated diff hunk behaves completely differently when it actually has to navigate a tree. Every finding was checked against the code at that commit by a separate model, blind to which reviewer produced it.

What We Measured

Four metrics (because they trade off against each other):

- Detection — does it land on cases that actually contain a bug
- Noise — does it stay quiet on clean code
- Precision — is what it says true when it does speak
- Completion — does the review finish at all, vs. timing out or erroring

What are the different experiments we did:

- Toggling reasoning on vs. off with everything else held constant.
- Forcing the harness to execute more tool calls (opening every changed file before returning a verdict, enforcing minimum call counts, and batching).
- Extracting the step-by-step review procedure the best model visibly followed in its traces and handing that to the weaker models as explicit system instructions.

Key Takeaways

The two local models ended up further apart from each other than the better local model was from the frontier API—despite running on the exact same GPU, harness, and prompts.

The local model's failure mode wasn't context size or speed; it was grounding. Most of its findings didn't cite specific files, making them unactionable. When we forced hard constraints to require file and line citations, it dropped its findings entirely rather than outputting grounded ones.

More details are in the blog: https://getspinal.com/blog/frontier-vs-local-code-review

Hey everyone,

I have a consumer app that uses fully-local AI to help people practice speaking a language privately and securely. The full conversation cycle is local:

• STT - Apple on-device SpeechAnalyzer
• LLM - Gemma 4 E4B
• TTS - Supertronic 3

The number 1 feedback I get from the average user is "I'm not downloading a 2.5-3GB model to my phone."

I implemented a cloud option that pings a serverless GPU endpoint that runs Gemma 4 so they don't have to download it (as opposed to simply calling a 3rd party inference API). But I originally built the app for the fully-local approach because I believe privacy is wildly underrated.

So my question is: If you've offered multi-GB models on a user-facing app, what's the best way to get them onboard with it?

Thanks!

Link to app if you want to check it out

We run one AMD MI300X on a GPU droplet and serve it internally as our own inference endpoint. I swapped the model on it. The difference at long context surprised me. I would love to get Qwen running closer to the speed of ds. Any tips would be appreciated.

The hardware

• GPU: 1x AMD Instinct MI300X VF, 205.8 GB VRAM
• Intel Xeon Platinum 8568Y+, 20 vCPU
• RAM 235 GB, no swap
• Disk 697 GB
• Ubuntu 24.04.4, kernel 6.8.0-138
• ROCm 7.2.4
• One card. Not a node, not a cluster.

What we were running

Qwen3.8-27B in bf16 on SGLang 0.5.14, in the ROCm container, under systemd. Speculative decoding was on through the model's own MTP head driven by EAGLE, accepting about 3.35 of 4 draft tokens. Context 262,144.

The one config change that actually mattered was --mamba-ssm-dtype bfloat16. It stopped the Mamba state cache from eating the KV budget in FP32, and it took us from a hard ceiling around 155k tokens up to the full window. If you are running this model on a single card and you are stuck this might help you.

We tried Qwen at FP8 and saw ~2x slower speeds. I am new to this, but was surprised nonetheless.

What we run now

DeepSeek-V4-Flash-0731 on vLLM ROCm. I did not build this. It is a public pinned recipe, ryanzhou/deepseek-v4-flash-mi300x, which ships the AMD patches as read only bind mounts over a stock vLLM install plus check summed prebuilt kernels. 23 patch mounts, one compose file. It came up on the first try, which I did not expect either (and really appreciate).

Config:

--max-model-len 393216

--kv-cache-dtype fp8

--block-size 256

--kv-cache-memory-bytes 16000000000 # 16 GB KV on device

--kv-offloading-size 96 # 96 GB KV offloaded to host RAM

--kv-offloading-backend native

--max-num-seqs 64

--max-num-batched-tokens 4096

--long-prefill-token-threshold 1024

--moe-backend triton

--speculative-config.method=dspark

--speculative-config.num-speculative-tokens=7

--compilation-config cudagraph_mode=FULL_AND_PIECEWISE

Env: VLLM_ROCM_USE_AITER=1, VLLM_ROCM_OPUS_PREFILL=1, VLLM_ROCM_USE_SKINNY_GEMM=0, HSA_NO_SCRATCH_RECLAIM=1, HIP_FORCE_DEV_KERNARG=1.

Here is the interesting part.

Single stream, 512 output tokens, prose output, unique prefix on every run so nothing is cached.

Going from 1k to 200k cost Qwen about 11x of its throughput. It cost DeepSeek about 1.2x.

I would rather not give up the 27B. It has been working quite well for the type of work we do, but speed is a real factor to consider.

Any thoughts, considerations, tips? Thanks in advance.

P.S. I used AI to create the table and to print the configuration above.