What’s y’all’s best guess on parameter size based on these weird-ass names?

Looking at ... some of the new memory architecture. ... I hired my good friend, Seok-Hee Lee, who used to run SK Hynix. ... We are not ready to unfold it.

These are screenshots from the r/Singularity comment section. I'm speechless. This doesn't even have downvotes. How can someone cheer for a monopoly run by a few elites?

Gemma 5 or something else?

I was browsing HF for small LLMs and run into this model. It does not seem to be a fine tune - the model has its own architecture.

https://huggingface.co/XHToken/Spark-X2.5-1.7B
https://huggingface.co/XHToken/Spark-X2.5-4B

There are 4B/1.7B versions - the benchmark is quite interesting (4B is neck and neck with Qwen 3.5 9B). The HF page claims both models support native 1M context size.

Currently does not run out of the box on llama.cpp - pending this PR: https://github.com/ggml-org/llama.cpp/pull/27868

They have a custom fork of llama.cpp that works. Anyone has tried this?

Update:
GGUFs (require custom fork for now):
https://huggingface.co/XHToken/Spark-X2.5-1.7B-GGUF
https://huggingface.co/XHToken/Spark-X2.5-4B-GGUF

Kaitchup just posted results of his benchmarks for Qwen3.8 27B for quants from different labs, Q4 to Q1, .

All the details are hidden behind the paywall, but high level result is visible and looks like for people with 16GB cards UD Q3_K_XL is a winner - it has accuracy of 100% and size is only 12.8GB.

Feels like maybe we have one more present left, for Christmas.

Can't wait to test! This should significantly boost TPS!

Now we just need more llama cpp optimizations to be merged in!

Edit:

For anyone who wants to test this: https://github.com/unslothai/llama.cpp/pull/144/changes

More info: https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/blob/main/MTP/README.md

Hello all you smarter people. I recently retired and have taken on a task that is going to stretch me a bit.

TL;DR My aging aunt is going blind and wants to keep writing stories that she's been writing for over 70 years. I think local AI has the ability to make this possible but I'm looking for a little guidance on the steps and the order.

FULL VERSION

My aunt is 85 and lives with me now. She has written over 150 stories in her lifetime. They are mostly detective fiction and old west outlaw fiction. She also has macular degeneration that has taken most of her eyesight. She has given up on everything else she used to do, but she still writes and edits her own stories. Lately she has talked about quitting this too because it's just too hard to keep writing even with a giant screen and high-contrast tools.

After some discussion she agreed to try an interactive AI tool of some kind. I picked up a new desktop with an RTX 5080 (16gbVRAM) and 32 GB RAM.

I got Unsloth desktop installed this weekend and I grabbed Gemma4 as my first model.

But now I think I may be woefully out of my depth.

I've mostly only written prompts for existing online models before. I've never actually started from scratch like this and I'm not sure how much prep I need to do before I start interacting with the model. There are a ton of videos and articles about running AI locally, but it's not easy for me to figure out which ones I can trust or which steps apply to me. I would really appreciate a link to a guide for total newbs like me.

The use cases seem pretty simple to me:

• Interact with my aunt solely through voice and audio.
• Always be listening and available to her.
• Read her own stories to her in a natural voice that she gets to choose.
• Answer questions about existing stories to help her maintain continuity and bring old characters back from time to time. (She does this with handwritten notes right now and she's really struggling.)
• When prompted, suggest technical edits (spelling, grammar, etc) and help her stay consistent with those edits across all her stories.
• When prompted, suggest stylistic edits (clarity, pacing, etc) and help her stay consistent with those edits across all her stories.
• Prepare her stories for publication in KDP format (this one is mostly to help me do this for her).

Here is the approach I was thinking I would follow, but after looking through all the Unsloth features I'm not sure how many steps I'm missing.

1. Write instructions that I want the model to always follow.
2. Place the instructions into the Unsloth System Prompt under Run Settings.
3. Have my aunt converse with model via microphone.

My instructions cover a LOT.

- Brief description of my aunt and her writing goals and style.

- Outline of her work (the types of stories and any connections).

- Location of her existing stories.

- Description of the AI's role and its primary tasks including definitions of terms.

- A set of detailed rules to be followed when helping her edit.

But I still have so many questions:

- Do I need to create an unsloth project for this?

- What's the best way to have the model listen to voice inputs from my aunt?

- What's the best way to set up the model so it always listens for her input (kind of like an Alexa)?

- How can I have it keep a log of all its work and make backups of files before it makes changes? (similar to how Google Docs keeps a revision history and allows you to go back and grab an older version of a file)

Any insights folks want to share or resources you want to point me to would be most welcome. Thanks!

I kept seeing guys in this sub saying how AtomicChat's Qwen3.8-Flash-Next quant is so good, fits in their machine when unsloth's can't, runs faster than other quants etc, so I went check out what's happening there.

First thing I noticed was that AtomicChat's Q4_K_M quant is suspiciously small when the ngram table is removed (only ~56GB), it seems like most of the tensors in this quant are IQ2_S instead of the usual Q4_K, Q5_K and Q6_K that you usually find in Q4_K_M quants, the GGUF filetype metadata also says IQ2_S instead of Q4_K_M. In their model card, their Q4_K_M also has suspiciously high KLD (0.084).

It seems pretty obvious to me that they're pretending a IQ2_S quant as a Q4_K_M, but at the same time I'm genuinely not sure because it can't be only me who found this right? How can nobody be pointing this out? Am I missing something or what may they be doing?

Their HF repo ID: AtomicChat/Qwen3.8-Flash-Next-GGUF

Yoyo

I'm back with updates to the fastest inference engine with minimal quality loss for Qwen3.8-27B.

The last few weeks I've been optimizing decode speed and I don't think it can be pushed further, until a newer/better drafter is invented.

So I focused on prefill, which I this morning was around 1.300 per second at 4k and now is just below 2.000.

The main improvement came from a custom kernel, which matches the quality of fp32 with 0.99997 similarity at int8.

Try all of the improvements here:
https://github.com/syv-ai/qwen38-27b-rtx3090

I am absolutely blown away. Yes my setup is crap but the fact that it managed to do this in a single take is unbelievable (and I'm a developer).

Hardware used:
- Windows PC with 4070ti (12GB VRAM, 32GB RAM)
- Macbook M5 Air (LLAMA.cpp RPC connection to Windows PC)

Software used:
- LLAMA.cpp (Q4KM, xhigh, 8bit KV, MTP=1)
- Lmstudio Qwen 3.8 27b (Q4KM) GGUF
- Deepseek harness (mode: minimal)

Prompt: "please create a fully self-contained super mario game with only one short level, put everything inside mario.html inside the current directory"

context: 64k
thinking: xhigh
time took: 117 minutes
avg tps: 7.6

resut: https://pastebin.com/qyBu64sP

https://reddit.com/link/1w4821c/video/qpukeg1y4wmh1/player

No seriously, I kinda like it.
You have something to solve, you put it.

You know its gonna take like 20 mins to cook.
Every search adds another 30 minutes.

Yes I could boot up my debian on my gaming rig, run the same model at 10t/s + but why?
I rather let the poor server without GPU burn and run the same model at 2t/s and chill.

Its great, I love it.

More new massive updates from turboderp:

- CPU offload of MoE experts
- Qwen-3.8-Flash-Next ngram disk offload
- GLM-5.3-Flash
- New self-calibrated optimization technique
- Countless other optimizations and improvements

If you have an NVIDIA card and haven't tried it lately, you might be missing out.

The attached cat image was made with Qwen-3.8-Flash-Next-3.05bpw-exl3 and this prompt:
Create a detailed SVG image of a cute kitten riding a magic turtle into space.

Come join the crew at the exllama discord
More frequent news on the exllama sub

I got the CMP170 cards and unlocked them. I wanted to share my set up for CUDA since maybe it would be useful to others.

First off, I hate e-waste and we are in a special time for RAM. I wanted to have a DIY CUDA box, and I had started by adding additional cards to an old asus predator prebuilt I had around, which also had 64gb DDR5. To add the CMPs I needed more CPU lanes and newegg had some really good deals on CPU/MB/etc combos. Didn’t need a combo with RAM, otherwise I would have gotten it in newegg microcenter.

Anyway, I got a cheap case, some noctua fans for the cards, and transferred the memory/ssds. Placed previously owned cards on oculink slots, and used the main x16 for the GPU switch that houses the two CMP170s, so their effective speed is 2x16 across and with the other cards (which are 4x4, and therefore same speed).

Qwen Flash Next, turns out, fits very nicely in these cards. There is also a repository for deepseek, but you’d need at least 3 64GB cards to run it, and with prices rising, it will be hard to justify the gamble of buying ex mining cards for LLMs.

However…so far, these cards are great. Concurrency is good, prompt processing averages 4000 tps on Flash Next, decode is 80+ on a single stream. No MTP added. Third picture shows the 3 models I am now running in this CUDA box (flash next, qwen 27b, gemma 26b).

Anyone else trying out Flash Next on these cards?

I only started using some vLLM forks recently in a 4 x 48GB 4090 system.

DS4F - ~5000pp/180tg (DSpark)
Qwen3.8 Flash next - ~7500pp/135tg (MTP)

This is amazing, like having the API in my house. But it's also really hard to go back.

It's weird that we never come close to prefill numbers like this in llama.cpp or ik_llama. The narrative is that vLLM is around the same speed for single requests, but that is clearly not true.

There must some HUGE difference that constitutes an insurmountable obstacle to achieving such speeds in llama.cpp and many other inference engines. Does anyone know exactly what it is?

edit: These results are from my benchmark script that actually times the response, not the vLLM log. And they are not cache hits. My benchmark script deliberately busts cache. Actual cache hits, which I also measure, are like 20k-100k+.

Hello!

I've just uploaded a recent checkpoint of my model trained from scratch:

https://huggingface.co/piotr-ai/polanka_3.7b_exp_wip_260901

It was pre-trained, mid-trained, and fine-tuned on a single 4090 over many months. How many tokens? I lost count.

Feel free to use it as a research artefact.

13 languages: PL, EN, ZH, CS, SK, UK, RU, IT, ES, FR, DE, PT, LT — with extra upscaled data for PL/EN/ZH.

If you are thinking what model will fit best your HW specs and tasks you are doing here is one table with all currently popular models that still can be considered as local.

LLM Test Scores

Agentic benchmarks

Coding benchmarks

General benchmarks

Multimodal benchmarks

Note: I used GLM-5.3 to compose the table from official HF pages of the models.

Note2: Opus-4.8 results are presented only for illustration and are omitted from selecting the best model in a row.

Upd: Added SWE-bench Pro, SWE-bench Multilingual, GPQA Diamond and HLE (without tools) scores for Opus 4.8 from its System Card.

I posted a new gfx906 based llama.cpp fork a few days ago. One of the main points of critique was that i did not provide sufficient numbers for the gains to be achieved.

--

TL;DR: After switching our Qwen 3.8 27B production setup to DFlash2, several of the old gfx906 optimizations turned out to be neutral or outright regressions. We went back through the existing gfx906 work, isolated the problem areas, reworked the small-Q Flash Attention path and added adaptive native/convert selection.

Against current llama.cpp mainline, the resulting fork is now +14.1% in first-batch PP (379.2 vs 332.3 t/s) and +9.3% in 120k-context fill (252.6 vs 231.1 t/s), while deep-context TG is effectively tied at 13.6 vs 13.5 t/s. DFlash acceptance is identical at 0.691, and deterministic output matches byte-for-byte.

---

Our thread is here:

https://forum.level1techs.com/t/glm-and-i-created-a-llama-cpp-fork-optimized-for-amd-gfx906-mi50-mi60-radeon-vii-gcn-hip/254257/3

This is the github for it:

https://github.com/milpster/gfx906-llama-cpp

thats how deepresearch should look like nowdays

Just to name a few:

• running Qwen3 8B on a 32GB GPU
• running Qwen3.6-27B Q4_K_M on 2 x R9700
• quote: "each prompt was sized at 500 input and 500 output tokens"
• for a full system that costs $18,775??

I don't understand what they are doing. Am I reading something wrong?

I need a good model that feels smart ish in this regard but also runs with all my other stuff (audio gen, video gen, etc) enabled.

if you are on Qwen Flash Next make sure to update your build often

merged already (by ServeurpersoCom)

https://github.com/ggml-org/llama.cpp/pull/27978

https://github.com/ggml-org/llama.cpp/pull/28011

https://github.com/ggml-org/llama.cpp/pull/28023

https://github.com/ggml-org/llama.cpp/pull/28123

merged (by 0cc4m)

https://github.com/ggml-org/llama.cpp/pull/28032

in progress (by danielhanchen) MERGED NOW

https://github.com/ggml-org/llama.cpp/pull/27941

MTP in progress

https://github.com/ggml-org/llama.cpp/pull/27836

more in progress for example

https://github.com/ggml-org/llama.cpp/pull/28136

I used a famous Simon Willison's pelican riding a bicycle prompt on the biggest local LLMs that can run on 128GB Apple Silicon. U used quantizations by Unsloth.

Qwen3.8 Flash-Next gives a lot of details. DeepSeek V4 Flash is strangely underwhelming. Qwen3.8 27B still rocks, and I like its consistent minimalism.

Is Qwen3.8 27B still large at 31GB? It is! But for this tasks 2-bit quantizations (at around 12GB) will give the same results. For more complicated coding, 4-bit are more than enough. RTX cards are well enough!

See:

• Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses - Terminal-Bench 2.1, GPQA Diamond and IFBench
• Do Qwen3.6 27B quantizations break the pelican?