AMA We’re the Team Behind Apodex 1.1 — Ask Us Anything!

53 Upvotes

We’re Apodex, the team behind Apodex 1.1, our new model family built to scale agentic intelligence for complex work. We’re excited to be here and answer your questions directly.

Apodex 1.1 is designed around sustained, verifiable progress toward real-world objectives—from reasoning and search to working with files, executing code, recovering from failures, and coordinating multiple agents.

Open models

Apodex 1.1

Apodex 1.0

Alongside Apodex 1.1, we released our open-source agent harness and two papers:

Participants

u/TechnologyCertain757 — Chris
u/Eric-LRL — Ruilin Li
u/shawnlinn — Shawn Lin
u/wowfingerlicker — Rock, STEM
u/RepulsiveDish6416 — Simon, agents and post-training
u/Ok_Student7211 — Shaoliang Nie, model behavior
u/Ok-Space3044 — Xinqi Wang, coding post-training

The AMA will run from 8–11 AM PT today, and we’ll continue monitoring and answering questions over the next 48 hours.

Ask us anything!

68 comments

r/LocalLLaMA • u/No_Afternoon_4260 • 6d ago

Megathread [Megathread] GLM-5.3-Flash - former ox-alpha

gallery

297 Upvotes

Megathread for discussing the release of GLM-5.3-Flash.

Quants
Fine-Tunes & Abliterations
Chat Templates
Inference Server Support & Configuration
Experiences, Benchmarks & Model Comparisons

We'll try to clean up future duplicates around the release and point them here.

Highlights

GLM-5.3-Flash is the first natively multimodal model in the GLM-5 series, and the first open-weight release of the glm5_next architecture. Z.ai's pitch: outperforms GLM-5.2 at one-tenth the price while approaching Claude Opus 4.8 on coding and agentic benchmarks. It introduces:

Hybrid Sparse + Linear Attention: 45 layers laid out as repeating blocks of 3x KDA linear attention followed by 1x DeepSeek-style sparse attention (34 linear / 11 sparse layers). The sparse layers use a lightning indexer (32 heads, dim 128) with a top-k budget of 2048 tokens, sharply reducing long-context serving cost.
Manifold-Constrained Hyper-Connections (mHC): widened residual streams with manifold-constrained mixing between layers, adopted to further improve scaling efficiency.
Natively Multimodal: a 24-layer ViT (448px, patch 14, 2x2 spatial merge) with temporal patching, so image and video tokens are in the vocabulary. Trained on a 30T-token multimodal corpus.
MTP head shipped in the weights: 1 next-N prediction layer; the official vLLM recipe uses it with 5 speculative tokens.
FP8 first: the main repo is FP8 (e4m3, dynamic activation scaling). A separate official BF16 repo exists.

Model Overview

Type: Causal Language Model with Vision Encoder (Glm5NextForConditionalGeneration)
Training Stage: Pre-training (30T multimodal tokens) & Post-training
License: MIT

Language Model

Number of Parameters: 320B with 18B activated
Hidden Dimension: 4096
Vocabulary: 154,880
Number of Layers: 45 (first 3 dense MLP, remaining 42 MoE)
Hidden Layout: 11 x (3 x (KDA Linear Attention -> MoE) -> 1 x (Sparse Attention -> MoE)), plus 1 trailing linear layer
KDA Linear Attention (34 layers):
- Number of Heads: 64
- Head Dimension: 128
Sparse Attention (11 layers, DeepSeek-style):
- Number of Attention Heads: 64
- QK / V Head Dimension: 256 / 256
- Indexer: 32 heads, head dim 128
- Budget: top-2048 tokens
Mixture of Experts:
- Number of Experts: 288 routed + 1 shared
- Number of Activated Experts: 8 routed + 1 shared
- Expert Intermediate Dimension: 2048
- Dense Intermediate Dimension (layers 0-2): 12288
mHC: enabled
MTP: 1 layer
Context Length: 1,048,576 tokens (max_position_embeddings); evaluated at 300K text / 164K vision

Vision Encoder

Depth: 24 layers, hidden 1024, 16 heads
Image Size: 448 x 448, Patch Size: 14
Spatial Merge: 2 x 2, Temporal Patch: 2 (video)
Output Projection: 4096 (LM hidden)

Weights

zai-org/GLM-5.3-Flash - FP8 (e4m3, dynamic), 62 shards, ~331 GB
zai-org/GLM-5.3-Flash-BF16 - BF16, 120 shards, ~640 GB

Architecture diagram

Benchmarks

Benchmark chart

Recommended sampling parameters

From generation_config.json and the eval setup on the model card:

Default: temperature=1.0, top_p=0.95 (max generation length 163,840 for evals)
Agentic coding (NL2Repo): temperature=1.0, top_p=1.0
DeepSWE: temperature=0.95, top_p=1.0
Vision (BabyVision): temperature=1.0, top_p=0.95

Inference

vLLM (official recipe, vLLM 0.27.0+, FlashInfer 0.6.17+ for NoPE sparse MLA, Hopper and newer):

vllm serve zai-org/GLM-5.3-Flash \
  --tensor-parallel-size 4 \
  --kv-cache-dtype fp8 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5}' \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice \
  --served-model-name zai-org/GLM-5.3-Flash

SGLang: official cookbook has verified configs for H100/H200/B200/B300/GB200/GB300 (TP4/EP4), with adaptive MTP for low-latency and --mm-feature-transport cpu to offload vision features.

Also supported at launch: TokenSpeed, KTransformers (CPU/GPU hybrid tutorial linked below).

Official Links

HF: https://huggingface.co/zai-org/GLM-5.3-Flash
HF (BF16): https://huggingface.co/zai-org/GLM-5.3-Flash-BF16
MS: https://modelscope.cn/models/ZhipuAI/GLM-5.3-Flash
Repo: https://github.com/zai-org/GLM-5
Blog: https://z.ai/blog/glm-5.3-flash
Docs: https://docs.z.ai/guides/llm/glm-5.3-flash
Paper: https://arxiv.org/abs/2602.15763
vLLM: https://recipes.vllm.ai/zai-org/GLM-5.3-Flash
SGLang: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-5.3-Flash
TokenSpeed: https://lightseek.org/tokenspeed/recipes/models#glm-5-3-flash
KTransformers: https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/kt-kernel/GLM-5.3-Flash-Tutorial.md
Discord: https://discord.gg/QR7SARHRxK

My aunt is 85 and lives with me now. She has written over 150 stories in her lifetime. They are mostly detective fiction and old west outlaw fiction. She also has macular degeneration that has taken most of her eyesight. She has given up on everything else she used to do, but she still writes and edits her own stories. Lately she has talked about quitting this too because it's just too hard to keep writing even with a giant screen and high-contrast tools.

After some discussion she agreed to try an interactive AI tool of some kind. I picked up a new desktop with an RTX 5080 (16gbVRAM) and 32 GB RAM.

I got Unsloth desktop installed this weekend and I grabbed Gemma4 as my first model.

But now I think I may be woefully out of my depth.

I've mostly only written prompts for existing online models before. I've never actually started from scratch like this and I'm not sure how much prep I need to do before I start interacting with the model. There are a ton of videos and articles about running AI locally, but it's not easy for me to figure out which ones I can trust or which steps apply to me. I would really appreciate a link to a guide for total newbs like me.

The use cases seem pretty simple to me:

Interact with my aunt solely through voice and audio.
Always be listening and available to her.
Read her own stories to her in a natural voice that she gets to choose.
Answer questions about existing stories to help her maintain continuity and bring old characters back from time to time. (She does this with handwritten notes right now and she's really struggling.)
When prompted, suggest technical edits (spelling, grammar, etc) and help her stay consistent with those edits across all her stories.
When prompted, suggest stylistic edits (clarity, pacing, etc) and help her stay consistent with those edits across all her stories.
Prepare her stories for publication in KDP format (this one is mostly to help me do this for her).

Here is the approach I was thinking I would follow, but after looking through all the Unsloth features I'm not sure how many steps I'm missing.

Write instructions that I want the model to always follow.
Place the instructions into the Unsloth System Prompt under Run Settings.
Have my aunt converse with model via microphone.

My instructions cover a LOT.

- Brief description of my aunt and her writing goals and style.

- Outline of her work (the types of stories and any connections).

- Location of her existing stories.

- Description of the AI's role and its primary tasks including definitions of terms.

- A set of detailed rules to be followed when helping her edit.

But I still have so many questions:

- Do I need to create an unsloth project for this?

- What's the best way to have the model listen to voice inputs from my aunt?

- What's the best way to set up the model so it always listens for her input (kind of like an Alexa)?

- How can I have it keep a log of all its work and make backups of files before it makes changes? (similar to how Google Docs keeps a revision history and allows you to go back and grab an older version of a file)

Any insights folks want to share or resources you want to point me to would be most welcome. Thanks!

36 comments

r/LocalLLaMA • u/po_stulate • 4h ago

Discussion Deceptive model quantization from AtomicChat?

60 Upvotes

I kept seeing guys in this sub saying how AtomicChat's Qwen3.8-Flash-Next quant is so good, fits in their machine when unsloth's can't, runs faster than other quants etc, so I went check out what's happening there.

First thing I noticed was that AtomicChat's Q4_K_M quant is suspiciously small when the ngram table is removed (only ~56GB), it seems like most of the tensors in this quant are IQ2_S instead of the usual Q4_K, Q5_K and Q6_K that you usually find in Q4_K_M quants, the GGUF filetype metadata also says IQ2_S instead of Q4_K_M. In their model card, their Q4_K_M also has suspiciously high KLD (0.084).

It seems pretty obvious to me that they're pretending a IQ2_S quant as a Q4_K_M, but at the same time I'm genuinely not sure because it can't be only me who found this right? How can nobody be pointing this out? Am I missing something or what may they be doing?

Their HF repo ID: AtomicChat/Qwen3.8-Flash-Next-GGUF

31 comments

r/LocalLLaMA • u/iamMess • 8h ago

Discussion I pushed Qwen3.8-27B to 2.000 prefill per second and 132 decode per second on A RTX 3090.

83 Upvotes

Yoyo

I'm back with updates to the fastest inference engine with minimal quality loss for Qwen3.8-27B.

The last few weeks I've been optimizing decode speed and I don't think it can be pushed further, until a newer/better drafter is invented.

So I focused on prefill, which I this morning was around 1.300 per second at 4k and now is just below 2.000.

The main improvement came from a custom kernel, which matches the quality of fp32 with 0.99997 similarity at int8.

Try all of the improvements here:
https://github.com/syv-ai/qwen38-27b-rtx3090

66 comments

r/LocalLLaMA • u/zannix • 9h ago

Discussion Qwen 3.8 27b (Q4KM) oneshot a Super Mario clone

96 Upvotes

I am absolutely blown away. Yes my setup is crap but the fact that it managed to do this in a single take is unbelievable (and I'm a developer).

Hardware used:
- Windows PC with 4070ti (12GB VRAM, 32GB RAM)
- Macbook M5 Air (LLAMA.cpp RPC connection to Windows PC)

Software used:
- LLAMA.cpp (Q4KM, xhigh, 8bit KV, MTP=1)
- Lmstudio Qwen 3.8 27b (Q4KM) GGUF
- Deepseek harness (mode: minimal)

Prompt: "please create a fully self-contained super mario game with only one short level, put everything inside mario.html inside the current directory"

context: 64k
thinking: xhigh
time took: 117 minutes
avg tps: 7.6

resut: https://pastebin.com/qyBu64sP

https://reddit.com/link/1w4821c/video/qpukeg1y4wmh1/player

73 comments

r/LocalLLaMA • u/Ne00n • 28m ago

Discussion Slow interference is great

• Upvotes

No seriously, I kinda like it.
You have something to solve, you put it.

You know its gonna take like 20 mins to cook.
Every search adds another 30 minutes.

Yes I could boot up my debian on my gaming rig, run the same model at 10t/s + but why?
I rather let the poor server without GPU burn and run the same model at 2t/s and chill.

Its great, I love it.

22 comments

r/LocalLLaMA • u/Unstable_Llama • 13h ago

News ExLlamav3 Recent Updates : CPU offload, GLM-5.3-FLASH, Qwen3.8-Flash, SC Quants ++

gallery

158 Upvotes

More new massive updates from turboderp:

- CPU offload of MoE experts
- Qwen-3.8-Flash-Next ngram disk offload
- GLM-5.3-Flash
- New self-calibrated optimization technique
- Countless other optimizations and improvements

If you have an NVIDIA card and haven't tried it lately, you might be missing out.

The attached cat image was made with Qwen-3.8-Flash-Next-3.05bpw-exl3 and this prompt:
Create a detailed SVG image of a cute kitten riding a magic turtle into space.

Come join the crew at the exllama discord
More frequent news on the exllama sub

105 comments

r/LocalLLaMA • u/Mysterious_Hearing14 • 4h ago

Resources Keenable SELECT: an agent that searches the web in SQL

keenableai.github.io

14 Upvotes

thats how deepresearch should look like nowdays

5 comments

r/LocalLLaMA • u/dangerous_inference • 3h ago

Discussion Question: Why is prefill unbelievably faster in vLLM than other inference engines?

12 Upvotes

I only started using some vLLM forks recently in a 4 x 48GB 4090 system.

DS4F - ~5000pp/180tg (DSpark)
Qwen3.8 Flash next - ~7500pp/135tg (MTP)

This is amazing, like having the API in my house. But it's also really hard to go back.

It's weird that we never come close to prefill numbers like this in llama.cpp or ik_llama. The narrative is that vLLM is around the same speed for single requests, but that is clearly not true.

There must some HUGE difference that constitutes an insurmountable obstacle to achieving such speeds in llama.cpp and many other inference engines. Does anyone know exactly what it is?

edit: These results are from my benchmark script that actually times the response, not the vLLM log. And they are not cache hits. My benchmark script deliberately busts cache. Actual cache hits, which I also measure, are like 20k-100k+.

43 comments

r/LocalLLaMA • u/perelmanych • 8h ago

Resources All currently popular local models in one table + Opus 4.8 results

31 Upvotes

If you are thinking what model will fit best your HW specs and tasks you are doing here is one table with all currently popular models that still can be considered as local.

LLM Test Scores

Feature	DeepSeek-V4-Flash-Vision-Exp	DeepSeek-V4-Flash-0731	Qwen3.8-Flash-Next	GLM-5.3-Flash	Qwen3.8-27B	Opus-4.8
Total parameters	≈285B	284B	125B	320B	27B	not published
Active parameters	13B	13B	6B	18B	27B	not published

Agentic benchmarks

Benchmark	DeepSeek-V4-Flash-Vision-Exp	DeepSeek-V4-Flash-0731	Qwen3.8-Flash-Next	GLM-5.3-Flash	Qwen3.8-27B	Opus-4.8
Terminal Bench 2.1	83.9	82.7	–	82.6	73.0	85.0
NL2Repo	57.7	54.2	48.1	52.1	42.3	69.7
DeepSWE	59.3	54.4	58.7	61.1	42.2	58.0
Toolathlon-Verified	75.9	70.3	73.5	72.1	–	76.2
Agents' Last Exam	27.3	25.2⁷	24.3	28.1	20.4	25.7
AutomationBench (Public)	25.7	25.1	–	25.3	–	27.2
GDPval-AA v2	–	68.1	–	72.3	–	75.1
Cybergym	75.3	76.7	–	–	–	78.3
DSBench-Hard	63.6	59.6	–	–	–	71.7
DSBench-FullStack	–	68.7	–	–	–	71.6
ApexBench (Pass@1)	36.5	26.2⁷	–	–	–	39.4
HLE with tools (full set)	–	16.8	–	22.9	–	25.4

Coding benchmarks

Benchmark	DeepSeek-V4-Flash-Vision-Exp	DeepSeek-V4-Flash-0731	Qwen3.8-Flash-Next	GLM-5.3-Flash	Qwen3.8-27B	Opus-4.8
SWE-bench Pro	–	56.0	62.5	–	61.7	69.2
SWE-bench Multilingual	–	–	81.0	–	73.8	84.4
CoWorkBench	–	45.1	73.9	–	70.7	–
JobBench	–	41.3	55.7	–	33.4	–

General benchmarks

Benchmark	DeepSeek-V4-Flash-Vision-Exp	DeepSeek-V4-Flash-0731	Qwen3.8-Flash-Next	GLM-5.3-Flash	Qwen3.8-27B	Opus-4.8
GPQA Diamond	–	90.8	91.7	–	89.2	93.6
HLE (without tools)	–	33.8	35.9	–	30.8	49.8
LiveCodeBench v6	–	90.6	91.9	–	90.3	–
IFBench	–	79.2	81.3	–	79.5	–

Multimodal benchmarks

Benchmark	DeepSeek-V4-Flash-Vision-Exp	DeepSeek-V4-Flash-0731	Qwen3.8-Flash-Next	GLM-5.3-Flash	Qwen3.8-27B	Opus-4.8
Chartography	64.3	–	–	–	–	65.0
ZeroBench (Pass@5)	35.0	–	–	–	–	34.0
BabyVision	–	–	–	73.0	65.7 / 85.6	34.1
MathVision	–	–	90.6 / 95.7	–	90.0 / 94.6	–
RealWorldQA	–	–	88.5	–	85.9	–
AndroidWorld	–	–	84.5	–	81.9	–
OSWorld 2.0 (partial credit)	–	–	52.3	–	48.0	–
Vision2Web	–	–	64.0	–	62.9	–
ClawEval-MM (Pass@3)	–	–	64.4	–	57.4	–
RecreationBench	–	–	49.9	–	47.1	–
ERQA	–	–	72.3	–	65.5	–

Note: I used GLM-5.3 to compose the table from official HF pages of the models.

Note2: Opus-4.8 results are presented only for illustration and are omitted from selecting the best model in a row.

Upd: Added SWE-bench Pro, SWE-bench Multilingual, GPQA Diamond and HLE (without tools) scores for Opus 4.8 from its System Card.

17 comments

r/LocalLLaMA • u/Significant_Focus134 • 5h ago

New Model Multilingual Tiny (3.7B) Reasoning MoE pretrained from scratch on a consumer-grade GPU

15 Upvotes

Hello!

I've just uploaded a recent checkpoint of my model trained from scratch:

https://huggingface.co/piotr-ai/polanka_3.7b_exp_wip_260901

It was pre-trained, mid-trained, and fine-tuned on a single 4090 over many months. How many tokens? I lost count.

Feel free to use it as a research artefact.

13 languages: PL, EN, ZH, CS, SK, UK, RU, IT, ES, FR, DE, PT, LT — with extra upscaled data for PL/EN/ZH.

9 comments

r/LocalLLaMA • u/milpster • 6h ago

Resources Update: llama.cpp for Radeon VII / MI50 / MI60 — +14% PP, +9% long-context fill vs upstream + adaptive Flash Attention

19 Upvotes

I posted a new gfx906 based llama.cpp fork a few days ago. One of the main points of critique was that i did not provide sufficient numbers for the gains to be achieved.

--

TL;DR: After switching our Qwen 3.8 27B production setup to DFlash2, several of the old gfx906 optimizations turned out to be neutral or outright regressions. We went back through the existing gfx906 work, isolated the problem areas, reworked the small-Q Flash Attention path and added adaptive native/convert selection.

Against current llama.cpp mainline, the resulting fork is now +14.1% in first-batch PP (379.2 vs 332.3 t/s) and +9.3% in 120k-context fill (252.6 vs 231.1 t/s), while deep-context TG is effectively tied at 13.6 vs 13.5 t/s. DFlash acceptance is identical at 0.691, and deterministic output matches byte-for-byte.

---

Our thread is here:

https://forum.level1techs.com/t/glm-and-i-created-a-llama-cpp-fork-optimized-for-amd-gfx906-mi50-mi60-radeon-vii-gcn-hip/254257/3

This is the github for it:

https://github.com/milpster/gfx906-llama-cpp

3 comments

r/LocalLLaMA • u/iwinux • 17h ago

Discussion A very confusing report from Puget Systems

pugetsystems.com

135 Upvotes

Just to name a few:

running Qwen3 8B on a 32GB GPU
running Qwen3.6-27B Q4_K_M on 2 x R9700
quote: "each prompt was sized at 500 input and 500 output tokens"
for a full system that costs $18,775??

I don't understand what they are doing. Am I reading something wrong?

50 comments

r/LocalLLaMA • u/No-Name-Person111 • 17h ago

News Mac ← USB-C cable → Linux box is becoming a thing.

github.com

130 Upvotes

32 comments

r/LocalLLaMA • u/Miserable-Dare5090 • 42m ago

Discussion CMP170Hx “Spark” Machine

gallery

• Upvotes

I got the CMP170 cards and unlocked them. I wanted to share my set up for CUDA since maybe it would be useful to others.

First off, I hate e-waste and we are in a special time for RAM. I wanted to have a DIY CUDA box, and I had started by adding additional cards to an old asus predator prebuilt I had around, which also had 64gb DDR5. To add the CMPs I needed more CPU lanes and newegg had some really good deals on CPU/MB/etc combos. Didn’t need a combo with RAM, otherwise I would have gotten it in newegg.

Anyway, I got a cheap case, some noctua fans for the cards, and transferred the memory/ssds. Placed previously owned cards on oculink slots, and used the main x16 for the GPU switch that houses the two CMP170s, so their effective speed is 2x16 across and with the other cards (which are 4x4, and therefore same speed).

Qwen Flash Next, turns out, fits very nicely in these cards. There is also a repository for deepseek, but you’d need at least 3 64GB cards to run it, and with prices rising, it will be hard to justify the gamble of buying ex mining cards for LLMs.

However…so far, these cards are great. Concurrency is good, prompt processing averages 4000 tps on Flash Next, decode is 80+ on a single stream. No MTP added. Third picture shows the 3 models I am now running in this CUDA box (flash next, qwen 27b, gemma 26b).

Anyone else trying out Flash Next on these cards?

12 comments

r/LocalLLaMA • u/jacek2023 • 11h ago

News qwen4exp fixes in llama.cpp

47 Upvotes

if you are on Qwen Flash Next make sure to update your build often

merged already (by ServeurpersoCom)

https://github.com/ggml-org/llama.cpp/pull/27978

https://github.com/ggml-org/llama.cpp/pull/28011

https://github.com/ggml-org/llama.cpp/pull/28023

https://github.com/ggml-org/llama.cpp/pull/28123

merged (by 0cc4m)

https://github.com/ggml-org/llama.cpp/pull/28032

in progress (by danielhanchen) MERGED NOW

https://github.com/ggml-org/llama.cpp/pull/27941

MTP in progress

https://github.com/ggml-org/llama.cpp/pull/27836

Discussion Don't sleep on Vision support for coding!

214 Upvotes

Normally, whenever a new model dropped, I always chose the non-vision version just to save VRAM; I though that only use case was when you were the one sending the picture.

However, with the release of QWEN 3.8 27B I decided to give it a shot, and it has been one of the best decisions I have made, as this makes the model way more capable for autonomous coding.

With no vision, the model will try to complete the task and get back to you once it thinks that it is done with no problem. But there are a lot of silent errors that do not get reflected via the code or the tests performed, so you could go back to an error screen or a broken page after getting a confirmation of your request being implemented correctly.

On the other side, when I ask something to QWEN with vision support, it will work on it, and then proactively take a screenshot to confirm if everything is right. This has helped numerous times with spotting errors that were missed. The model will continue to reiterate and take screenshots until it gets a visual confirmation of the issue being fixed. Just magnificent.

Btw, I currently run my local set up via Hermes with QWEN 3.8 27B (Qwen3.8-27B-UD-Q5_K_XL.) powered by a 5090.

93 comments

AMA We’re the Team Behind Apodex 1.1 — Ask Us Anything!

Megathread [Megathread] GLM-5.3-Flash - former ox-alpha

Highlights

Model Overview

Benchmarks

Recommended sampling parameters

Inference

Official Links

Popular

Discussion Fingers crossed for a 122b or really anything above 31b.🤞

Discussion Intel hints it may get back into memory business

New Model New Gemma models on arena ai

New Model New Model: Spark-X2.5-4B, Spark-X2.5-1.7B

Resources Keeping up with model launches

Discussion Kaitchup posted Qwen3.8 27B Benchmarks for quants from Q4 to Q1

Resources MTP released for Qwen3.8-Flash-Next-GGUF

Question | Help Help me set up local AI for my 85 year old aunt who is blind.

Discussion Deceptive model quantization from AtomicChat?

Discussion I pushed Qwen3.8-27B to 2.000 prefill per second and 132 decode per second on A RTX 3090.

Discussion Qwen 3.8 27b (Q4KM) oneshot a Super Mario clone

Discussion Slow interference is great

News ExLlamav3 Recent Updates : CPU offload, GLM-5.3-FLASH, Qwen3.8-Flash, SC Quants ++

Resources Keenable SELECT: an agent that searches the web in SQL

Discussion Question: Why is prefill unbelievably faster in vLLM than other inference engines?

Resources All currently popular local models in one table + Opus 4.8 results

LLM Test Scores

Agentic benchmarks

Coding benchmarks

General benchmarks

Multimodal benchmarks

New Model Multilingual Tiny (3.7B) Reasoning MoE pretrained from scratch on a consumer-grade GPU

Resources Update: llama.cpp for Radeon VII / MI50 / MI60 — +14% PP, +9% long-context fill vs upstream + adaptive Flash Attention

Discussion A very confusing report from Puget Systems

News Mac ← USB-C cable → Linux box is becoming a thing.

Discussion CMP170Hx “Spark” Machine

News qwen4exp fixes in llama.cpp

Discussion Don't sleep on Vision support for coding!