Gemma 5 or something else?

Can't wait to test! This should significantly boost TPS!

Now we just need more llama cpp optimizations to be merged in!

Edit:

For anyone who wants to test this: https://github.com/unslothai/llama.cpp/pull/144/changes

More info: https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/blob/main/MTP/README.md

What’s y’all’s best guess on parameter size based on these weird-ass names?

Looking at ... some of the new memory architecture. ... I hired my good friend, Seok-Hee Lee, who used to run SK Hynix. ... We are not ready to unfold it.

Normally, whenever a new model dropped, I always chose the non-vision version just to save VRAM; I though that only use case was when you were the one sending the picture.

However, with the release of QWEN 3.8 27B I decided to give it a shot, and it has been one of the best decisions I have made, as this makes the model way more capable for autonomous coding.

With no vision, the model will try to complete the task and get back to you once it thinks that it is done with no problem. But there are a lot of silent errors that do not get reflected via the code or the tests performed, so you could go back to an error screen or a broken page after getting a confirmation of your request being implemented correctly.

On the other side, when I ask something to QWEN with vision support, it will work on it, and then proactively take a screenshot to confirm if everything is right. This has helped numerous times with spotting errors that were missed. The model will continue to reiterate and take screenshots until it gets a visual confirmation of the issue being fixed. Just magnificent.

Btw, I currently run my local set up via Hermes with QWEN 3.8 27B (Qwen3.8-27B-UD-Q5_K_XL.) powered by a 5090.

These are screenshots from the r/Singularity comment section. I'm speechless. This doesn't even have downvotes. How can someone cheer for a monopoly run by a few elites?

More new massive updates from turboderp:

- CPU offload of MoE experts
- Qwen-3.8-Flash-Next ngram disk offload
- GLM-5.3-Flash
- New self-calibrated optimization technique
- Countless other optimizations and improvements

If you have an NVIDIA card and haven't tried it lately, you might be missing out.

The attached cat image was made with Qwen-3.8-Flash-Next-3.05bpw-exl3 and this prompt:
Create a detailed SVG image of a cute kitten riding a magic turtle into space.

Come join the crew at the exllama discord
More frequent news on the exllama sub

I was browsing HF for small LLMs and run into this model. It does not seem to be a fine tune - the model has its own architecture.

https://huggingface.co/XHToken/Spark-X2.5-1.7B
https://huggingface.co/XHToken/Spark-X2.5-4B

There are 4B/1.7B versions - the benchmark is quite interesting (4B is neck and neck with Qwen 3.5 9B). The HF page claims both models support native 1M context size.

Currently does not run out of the box on llama.cpp - pending this PR: https://github.com/ggml-org/llama.cpp/pull/27868

They have a custom fork of llama.cpp that works. Anyone has tried this?

Update:
GGUFs (require custom fork for now):
https://huggingface.co/XHToken/Spark-X2.5-1.7B-GGUF
https://huggingface.co/XHToken/Spark-X2.5-4B-GGUF

Just to name a few:

• running Qwen3 8B on a 32GB GPU
• running Qwen3.6-27B Q4_K_M on 2 x R9700
• quote: "each prompt was sized at 500 input and 500 output tokens"
• for a full system that costs $18,775??

I don't understand what they are doing. Am I reading something wrong?

Just saw this. https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp

I am absolutely blown away. Yes my setup is crap but the fact that it managed to do this in a single take is unbelievable (and I'm a developer).

Hardware used:
- Windows PC with 4070ti (12GB VRAM, 32GB RAM)
- Macbook M5 Air (LLAMA.cpp RPC connection to Windows PC)

Software used:
- LLAMA.cpp (Q4KM, xhigh, 8bit KV, MTP=1)
- Lmstudio Qwen 3.8 27b (Q4KM) GGUF
- Deepseek harness (mode: minimal)

Prompt: "please create a fully self-contained super mario game with only one short level, put everything inside mario.html inside the current directory"

context: 64k
thinking: xhigh
time took: 117 minutes
avg tps: 7.6

resut: https://pastebin.com/qyBu64sP

https://reddit.com/link/1w4821c/video/qpukeg1y4wmh1/player

Yoyo

I'm back with updates to the fastest inference engine with minimal quality loss for Qwen3.8-27B.

The last few weeks I've been optimizing decode speed and I don't think it can be pushed further, until a newer/better drafter is invented.

So I focused on prefill, which I this morning was around 1.300 per second at 4k and now is just below 2.000.

The main improvement came from a custom kernel, which matches the quality of fp32 with 0.99997 similarity at int8.

Try all of the improvements here:
https://github.com/syv-ai/qwen38-27b-rtx3090

Feels like maybe we have one more present left, for Christmas.

Kaitchup just posted results of his benchmarks for Qwen3.8 27B for quants from different labs, Q4 to Q1, .

All the details are hidden behind the paywall, but high level result is visible and looks like for people with 16GB cards UD Q3_K_XL is a winner - it has accuracy of 100% and size is only 12.8GB.

I kept seeing guys in this sub saying how AtomicChat's Qwen3.8-Flash-Next quant is so good, fits in their machine when unsloth's can't, runs faster than other quants etc, so I went check out what's happening there.

First thing I noticed was that AtomicChat's Q4_K_M quant is suspiciously small when the ngram table is removed (only ~56GB), it seems like most of the tensors in this quant are IQ2_S instead of the usual Q4_K, Q5_K and Q6_K that you usually find in Q4_K_M quants, the GGUF filetype metadata also says IQ2_S instead of Q4_K_M. In their model card, their Q4_K_M also has suspiciously high KLD (0.084).

It seems pretty obvious to me that they're pretending a IQ2_S quant as a Q4_K_M, but at the same time I'm genuinely not sure because it can't be only me who found this right? How can nobody be pointing this out? Am I missing something or what may they be doing?

Their HF repo ID: AtomicChat/Qwen3.8-Flash-Next-GGUF

Hello all you smarter people. I recently retired and have taken on a task that is going to stretch me a bit.

TL;DR My aging aunt is going blind and wants to keep writing stories that she's been writing for over 70 years. I think local AI has the ability to make this possible but I'm looking for a little guidance on the steps and the order.

FULL VERSION

My aunt is 85 and lives with me now. She has written over 150 stories in her lifetime. They are mostly detective fiction and old west outlaw fiction. She also has macular degeneration that has taken most of her eyesight. She has given up on everything else she used to do, but she still writes and edits her own stories. Lately she has talked about quitting this too because it's just too hard to keep writing even with a giant screen and high-contrast tools.

After some discussion she agreed to try an interactive AI tool of some kind. I picked up a new desktop with an RTX 5080 (16gbVRAM) and 32 GB RAM.

I got Unsloth desktop installed this weekend and I grabbed Gemma4 as my first model.

But now I think I may be woefully out of my depth.

I've mostly only written prompts for existing online models before. I've never actually started from scratch like this and I'm not sure how much prep I need to do before I start interacting with the model. There are a ton of videos and articles about running AI locally, but it's not easy for me to figure out which ones I can trust or which steps apply to me. I would really appreciate a link to a guide for total newbs like me.

The use cases seem pretty simple to me:

• Interact with my aunt solely through voice and audio.
• Always be listening and available to her.
• Read her own stories to her in a natural voice that she gets to choose.
• Answer questions about existing stories to help her maintain continuity and bring old characters back from time to time. (She does this with handwritten notes right now and she's really struggling.)
• When prompted, suggest technical edits (spelling, grammar, etc) and help her stay consistent with those edits across all her stories.
• When prompted, suggest stylistic edits (clarity, pacing, etc) and help her stay consistent with those edits across all her stories.
• Prepare her stories for publication in KDP format (this one is mostly to help me do this for her).

Here is the approach I was thinking I would follow, but after looking through all the Unsloth features I'm not sure how many steps I'm missing.

1. Write instructions that I want the model to always follow.
2. Place the instructions into the Unsloth System Prompt under Run Settings.
3. Have my aunt converse with model via microphone.

My instructions cover a LOT.

- Brief description of my aunt and her writing goals and style.

- Outline of her work (the types of stories and any connections).

- Location of her existing stories.

- Description of the AI's role and its primary tasks including definitions of terms.

- A set of detailed rules to be followed when helping her edit.

But I still have so many questions:

- Do I need to create an unsloth project for this?

- What's the best way to have the model listen to voice inputs from my aunt?

- What's the best way to set up the model so it always listens for her input (kind of like an Alexa)?

- How can I have it keep a log of all its work and make backups of files before it makes changes? (similar to how Google Docs keeps a revision history and allows you to go back and grab an older version of a file)

Any insights folks want to share or resources you want to point me to would be most welcome. Thanks!

if you are on Qwen Flash Next make sure to update your build often

merged already (by ServeurpersoCom)

https://github.com/ggml-org/llama.cpp/pull/27978

https://github.com/ggml-org/llama.cpp/pull/28011

https://github.com/ggml-org/llama.cpp/pull/28023

https://github.com/ggml-org/llama.cpp/pull/28123

merged (by 0cc4m)

https://github.com/ggml-org/llama.cpp/pull/28032

in progress (by danielhanchen) MERGED NOW

https://github.com/ggml-org/llama.cpp/pull/27941

MTP in progress

https://github.com/ggml-org/llama.cpp/pull/27836

more in progress for example

https://github.com/ggml-org/llama.cpp/pull/28136

Add 4 experts into Dense model and confirmed recovering model's ability up to "general level".

Hey, google. Please release official 124B MoE model!!!!!!!

If you are thinking what model will fit best your HW specs and tasks you are doing here is one table with all currently popular models that still can be considered as local.

LLM Test Scores

Agentic benchmarks

Coding benchmarks

General benchmarks

Multimodal benchmarks

Note: I used GLM-5.3 to compose the table from official HF pages of the models.

Note2: Opus-4.8 results are presented only for illustration and are omitted from selecting the best model in a row.

Upd: Added SWE-bench Pro, SWE-bench Multilingual, GPQA Diamond and HLE (without tools) scores for Opus 4.8 from its System Card.

I used a famous Simon Willison's pelican riding a bicycle prompt on the biggest local LLMs that can run on 128GB Apple Silicon. U used quantizations by Unsloth.

Qwen3.8 Flash-Next gives a lot of details. DeepSeek V4 Flash is strangely underwhelming. Qwen3.8 27B still rocks, and I like its consistent minimalism.

Is Qwen3.8 27B still large at 31GB? It is! But for this tasks 2-bit quantizations (at around 12GB) will give the same results. For more complicated coding, 4-bit are more than enough. RTX cards are well enough!

See:

• Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses - Terminal-Bench 2.1, GPQA Diamond and IFBench
• Do Qwen3.6 27B quantizations break the pelican?

Vellium is an open-source, local-first desktop app for AI chat, character roleplay and long-form writing.

Recent updates have focused mostly on making local voice and model setups easier to use.

Live mode now supports microphone input, local or Whisper-compatible speech recognition, streaming TTS, attachments, screen context and the usual chat tools—all inside the same voice interface.

Local speech can be installed and configured directly in the app. Whisper Large v3 Turbo Q5_0 is available for recognition, while TeraTTSv2 provides English and Russian voices with realtime playback. The TTS process stays active between responses, avoiding a full model reload for every reply.

The llama.cpp setup has also been simplified. Vellium can detect existing llama-server installations, GGUF models and running local endpoints, then configure them as a managed backend.

There have been plenty of smaller fixes as well: more reliable TTS streaming, safer runtime archive extraction, better endpoint discovery, improved timeout handling, system certificate support and easier settings navigation.

Chats, characters, LoreBooks, writing projects and knowledge collections are stored locally in SQLite. Vellium runs on macOS, Windows and Linux and supports OpenAI-compatible APIs, OpenRouter, LM Studio, Ollama and KoboldCpp.

GitHub: https://github.com/tg-prplx/vellium

Feedback from people using local voice or roleplay setups would be especially useful-particularly about anything that still feels awkward or unnecessarily complicated.

Hello!

I've just uploaded a recent checkpoint of my model trained from scratch:

https://huggingface.co/piotr-ai/polanka_3.7b_exp_wip_260901

It was pre-trained, mid-trained, and fine-tuned on a single 4090 over many months. How many tokens? I lost count.

Feel free to use it as a research artefact.

13 languages: PL, EN, ZH, CS, SK, UK, RU, IT, ES, FR, DE, PT, LT — with extra upscaled data for PL/EN/ZH.

I posted a new gfx906 based llama.cpp fork a few days ago. One of the main points of critique was that i did not provide sufficient numbers for the gains to be achieved.

--

TL;DR: After switching our Qwen 3.8 27B production setup to DFlash2, several of the old gfx906 optimizations turned out to be neutral or outright regressions. We went back through the existing gfx906 work, isolated the problem areas, reworked the small-Q Flash Attention path and added adaptive native/convert selection.

Against current llama.cpp mainline, the resulting fork is now +14.1% in first-batch PP (379.2 vs 332.3 t/s) and +9.3% in 120k-context fill (252.6 vs 231.1 t/s), while deep-context TG is effectively tied at 13.6 vs 13.5 t/s. DFlash acceptance is identical at 0.691, and deterministic output matches byte-for-byte.

---

Our thread is here:

https://forum.level1techs.com/t/glm-and-i-created-a-llama-cpp-fork-optimized-for-amd-gfx906-mi50-mi60-radeon-vii-gcn-hip/254257/3

This is the github for it:

https://github.com/milpster/gfx906-llama-cpp

No seriously, I kinda like it.
You have something to solve, you put it.

You know its gonna take like 20 mins to cook.
Every search adds another 30 minutes.

Yes I could boot up my debian on my gaming rig, run the same model at 10t/s + but why?
I rather let the poor server without GPU burn and run the same model at 2t/s and chill.

Its great, I love it.