For offensive cyber attacks. Have you ever wanted to take over Russia or something similar? I hear closed A.I. say it is pretty easy. :P

Let me start this with a caveat, DSV4Flash is usually my go to for "hard stuff" where I'm going to dump a ton of context and need real "deep" thought. I was battling with Hermes this evening trying to get dynamic model switching working with alias so I can do "/model local" or "/model cloud" in the middle of session to escalate manually if I need more power/speed and want to go to DS.

There were a bunch of little "gotchas" setting this up and DS kept having trouble with it and falling into "you need to change this" bugs. No bro, YOU change it. Finally I got tired of it and switch back to 27B, gave it the problem and walked away (yeah, not fast). Came back 10 minutes later to "found the problem and fixed it" and I was like "yeah, we'll see".

Well, color me shocked, it sure did find and fix the problem! I could not believe it, I must have gone 30 turns with DS (now, to be fair, I did give Qwen all the context from the DS conversation, so it knew where to look/what we were trying to do), but.. Bob's your uncle, freaking 27B just "one shot" the fix that DS was all over itself trying to fix.

Also, for anyone not aware, you can alias models in Hermes so you can switch mid conversation with the slash commands above. That's really useful for me, so I really wanted to get it working and, well, here we are. Well done!

Here's what it looks like for anyone interested:

[8/31/2026 10:34 PM] Trinity in reply to Mike:
> ‎⁨/model cloud⁩
Model switched to openrouter/deepseek/deepseek-v4-flash
Provider: LiteLLM-Hermes-Auto-Router
Context: 1,000,000 tokens
_(session only — add --global to persist)_
[8/31/2026 10:34 PM] Trinity in reply to Mike:
> ‎⁨/model local⁩
Model switched to Hermes-Auto-Router
Provider: LiteLLM-Hermes-Auto-Router
Context: 262,144 tokens
_(session only — add --global to persist)_

Run WebGPU compute kernels drawn from real AI workloads directly on your hardware and earn a personalized card built for your device.

On top of that, we're open-sourcing hundreds of these WebGPU kernels, our first step toward making browser inference as fast as physically possible. Contributed results show how they perform across real hardware and help make them faster.

Benchmark your GPU: https://webgpu-kernels-fleet.hf.space
Blog post: https://huggingface.co/blog/webgpu-kernels
Kernels: https://huggingface.co/webgpu-kernels/kernels

You may have heard of Kiro, it's owned by Amazon, has existed since July 2025 and is the replacement for the older Amazon Q Developer. For various reasons it's not very popular. Maybe because they insist on the free tier only having old models like Sonnet 4.5 and DeepSeek 3.2.

Up until recently, it was composed of Kiro IDE (VSCode fork), Kiro CLI (command line client, proprietary), both quite unremarkable. But last month they launched a third product, Kiro Crew. It's a "chat-first" client in the style of Codex or the "Agents" window in VS Code, but most importantly, like the title says, it's open source (Apache 2.0).

• Blog post: Introducing Kiro Crew
• GitHub page

It surprised me because it looks pretty well polished, yet I had never even heard of it. According to the blog post, it has "been adopted internally by over 39,000 Amazon builders".

It's made in react+python for the backend, and can be used locally or remotely from any browser. It internally connects to Kiro CLI through ACP and relies on it for the models. It also has features like

• Scheduled tasks, subagents
• Importing data from other clients
• Browser/Computer use,
  
• Skill management
• Integration with Slack/Discord/Teams chats

Now here's the problem: it does not natively support any providers outside the Kiro subscription.

I don't know if Amazon will want to add this at some point or not (they might just not want to), but being that this is open source, it should be possible to add it anyway, and given the upside, it would certainly be worth it.

If anyone is interested in contributing, there is already a third party patch (not mine). Right now it has a few problems (from my testing: it's not seamless to install, it doesn't have a model switcher, and some models can't do tool calls), but with a bit more work all those could be solved.

No seriously, I kinda like it.
You have something to solve, you put it.

You know its gonna take like 20 mins to cook.
Every search adds another 30 minutes.

Yes I could boot up my debian on my gaming rig, run the same model at 10t/s + but why?
I rather let the poor server without GPU burn and run the same model at 2t/s and chill.

Its great, I love it.

I used a famous Simon Willison's pelican riding a bicycle prompt on the biggest local LLMs that can run on 128GB Apple Silicon. U used quantizations by Unsloth.

Qwen3.8 Flash-Next gives a lot of details. DeepSeek V4 Flash is strangely underwhelming. Qwen3.8 27B still rocks, and I like its consistent minimalism.

Is Qwen3.8 27B still large at 31GB? It is! But for this tasks 2-bit quantizations (at around 12GB) will give the same results. For more complicated coding, 4-bit are more than enough. RTX cards are well enough!

See:

• Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses - Terminal-Bench 2.1, GPQA Diamond and IFBench
• Do Qwen3.6 27B quantizations break the pelican?

I used llama-perplexity to check Qwen3.8 27B NVFP4 GGUF models vs Unsloth as a baseline. Perplexity (lower better) is a measure of quality and is only comparable for the same model family.

As you can see for the Unsloth quants, perplexity increases with smaller file sizes. What is disappointing is all the tested NVFP4 GGUF models have worse perplexity for their file size.

In fact all are worse than Q4_K_XL, which is my daily driver.

Tested models

• https://huggingface.co/felippeburk/Qwen3.8-27B-NVFP4-MTP-GGUF
• https://huggingface.co/esatapedico/Qwen3.8-27B-NVFP4-MTP-GGUF
• https://huggingface.co/utautako/Qwen3.8-27B-NVFP4-MTP-Q8attn-GGUF

Baseline https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

This probably explains why Unsloth doesn't publish NVFP4 GGUF.

I used AI to generate the image, but otherwise I did all the work.

Hi! I've been considering getting a local AI station for video generation and light coding (I have coding AI subscription from work for heavyweight). My intended models are probably Minimax-H3 and Qwen 3.8 27b.

I see many people recommending as much RAM as possible when you buy, but I feel like 64GB of unified memory fits my needs well - runs H3 and Qwen 3.8 27B with a lot of headroom for context. Is there a reason I should spend $1500 more for 128GB? Do you foresee video/small coding models getting inflated in size in the future? Also open to good alternatives to the Strix/Framework Desktop. Thanks a lot!!

Good morning!

I was doing some research yesterday and I came across this Gemma 4 120b a12b coder model:

https://huggingface.co/LLMWildling/gemma-4-120b-a12b-coder

And I was wondering if anyone had seen it or had played with it and what their thoughts were (and maybe be able to talk Bartowski or Mradermacher or Unsloth into turning it into a GGUF for me to test with)

Thanks!

-TheSilentHobo

This app is for *writing* finetune examples, not training the model

You can use this app for free just go to the repo

FineTune Studio: https://github.com/StarpowerTechnology/Fine-Tuning-Studio

If you have a low amount of examples it can enhance your training by adding more datasets:

https://huggingface.co/datasets

i made examples explaining how to get a specific shot for film-making to show you how your skills can be used to develop a language model in a useful way ..

I used to think that finetuning was some type of process that only revolved around code and math (it is partially), but in reality its mostly the stage of sitting down and writing examples examples for the model to say in a given situation

10-50 examples can give you a good start but you have a chance of overfitting

50-200 can make a consistent response pattern

200-1000 can help you achieve specialization in narrow tasks

1,000-100k consistent patterns can be extremely effective in task adaptation & generalization

If all of your examples are consistent and use the same patterns across diffferent domains, then the model will adapt easily. Make a language model from scratch or finetune an existing pre-trained model to save time .. it doesnt take long to do this every model i made took less than 24 hours to make & small dataset can be trained within minutes

if you are new or you just dont want to write the examples yourself, you can go to huggingface to find all type of datasets for this .. go local & build your own experiments .. its getting easier and easier to achieve high level capabilities with the available distillations from frontier models .. theres many ways to do it but this app is meant for people who want to create datasets from scratch & shape the behavior intentionally. If you have any question or if you want to add your own pointers leave a comment

I currently have the following rig cobbled together:

MSI mpg z890 carbon
Intel Ultra 7
64Gb DDR5 6000
RTX 5070 Ti (16Gb)
2 x RTX 5060 Ti (16Gb each)

The 5070 and one of the 5060s are in the main CPU connected PCI slots (running at x8).

The second 5060 is on a CPU connected M.2 slot via an M.2 to PCIe 4.0 x4 riser.

I have one remaining CPU connected M.2 slot that I could use for a fourth GPU (also at PCIe 4.0 x4).

My max budget is around $3,500 (£2,500 as I'm based in the UK).

So, the options I have worked out so far:

1. Buy another Nvidia card (perhaps another 5070 Ti or I could possibly stretch to a 5090 but that feels like too much money / Gb for me) and push the PCIe mounted 5060 onto the spare M.2 slot.

This would get me to 64 or 80Gb VRAM and all Nvidia (better software support, more quants to choose from). It would also give me two fast cards for 27B which should get me a nice speed up. However, I'd still have very limited RAM for offloading larger MoE models, and although it works on paper I am slightly worried about it running two GPUs over those M.2 slots smoothly.

1. Sell some of the Nvidia cards and put a pair of R9700 32Gb in my two PCIe slots for a total of 64Gb.

I like this because I could host something like Qwen3.8-27B at Q4 with 256K context on each card, or across both with a higher quant and more context, but I'm concerned about software and quant support, although it sounds like those are improving. Plus still only 64Gb of RAM for offloading.

1. Sell my motherboard, CPU and RAM then buy a retired Epyc server with 256Gb of DDR4.

I like this because it unlocks larger MoE models. Realistically I'd probably end up with 2666 or even 2400 speed memory at current prices, but that's still 150-180ish Gb/s bandwidth which is much better than consumer single channel DDR5. This would also get all 3 GPUs on proper PCIe lanes with room for another 2 further down the line.

---

Options I'm not keen on:

1. Buying more DDR5 for my current rig. Prices are insane, and speeds are poor. Even if I could max it out at 256Gb I'm not convinced large MoE models would be usable with that memory bandwidth.

2. Buying a separate Strix Halo / DGX Spark / Mac type unified memory box. These would be slower than what I already have for small dense models, and in order to unlock meaningfully bigger MoE models they would need to be at least 512Gb really (because the Epyc option already gets me close to 300 with a chunk of that being fast VRAM). 512Gb unified memory systems are hard to find and prohibitively expensive.

---

I would really appreciate advice from people who have experience with similar systems, especially the DDR4 server route and the AMD cards.

I don't want to blow a load of cash only to find it just lets me run what I can already run at a slightly better quant. If that's the case, I'd just go for the cheapest option (another 5070 Ti).

Thanks for your help!

I ran OpenAI Privacy Filter and Layrin on all 200 RedactionBench documents: 11 categories and 8,273 mandatory entities. I got a result I wasn’t expecting.

OpenAI Privacy Filter had a much better overall R-Score, but Layrin missed far fewer entities that RedactionBench says should always be protected. At first I thought my scorer was wrong.

Full disclosure: I built Layrin,a local privacy layer for protecting sensitive text before AI use. English isn’t my first language, so I used AI to help clean up some of the wording, but I ran the experiment and checked the underlying results myself.

The extra metric here, Mandatory Entity Coverage, is not another official RedactionBench score. I added it to answer a narrower question: when RedactionBench says an entity must always be protected, how much of it was actually protected?

A completely missed entity gets zero coverage. Micro coverage pools coverage across all 8,273 mandatory entities, while exact recall only counts an entity when the whole span was covered.

Why did the result flip?

R-Score does not only measure leakage. It also penalizes unnecessary redaction, which makes sense. A system that hides half the document may be safe, but the result might not be very useful.

The problem is that these are different failure modes. Over-redaction hurts utility, while a miss can expose confidential information. Putting both into one score is useful for ranking systems, but it can hide what caused the result.

Before reading too much into this, I checked the scorer. My paper-faithful implementation passed 29/29 conformance tests covering grouping, partial coverage, contextual selection and benign-gap penalties.

On the frozen OpenAI Privacy Filter predictions, it produced:

• Mean R-Score: 0.6003 vs ~0.58 published
• P20: 0.335 vs ~0.31
• P50: 0.615 vs ~0.59

The category pattern was also close. I then ran the exact same scorer unchanged on Layrin and got 0.3705.

So the result seems real: OpenAI Privacy Filter clearly wins the combined R-Score, but Layrin protects much more of the information RedactionBench labels mandatory.

Context is where it gets messy

RedactionBench separates information into mandatory, contextual and unannotated gaps. Its human study included 85 participants, with agreement around:

• 89.4% for mandatory information
• 47.7% for contextual information
• 94.1% for preserving gaps

That 47.7% stood out to me. Once the answer depends on context, people disagree a lot.

Take a date like September 18, 2026. It could be harmless, or it could be a termination date, treatment date, confidential acquisition date or the timestamp of an internal security incident.

RedactionBench also evaluates documents without the full user request, conversation history or system prompt. In a real AI workflow, those can change what someone is comfortable sending.

Layrin also uses reversible typed tokens instead of simply deleting values.

Sarah Chen signed the agreement with Northbridge Capital on September 18, 2026 for $4.2 million.

becomes:

[PERSON_1] signed the agreement with [COMPANY_1] on [DATE_1] for [AMOUNT_1].

The model does not see the real values, but it still understands the structure. That makes me wonder how much utility is really lost when the exact value is not needed for the task.

What was being penalized?

Across the benchmark, 23,476 Layrin-protected spans landed entirely inside RedactionBench-defined gaps, with no overlap with mandatory or contextual annotations.

Some are clearly over-redaction. I’m not claiming otherwise.

But manual checks also found things like production AWS Secrets Manager ARNs, RDS hostnames, internal package-registry URLs, S3 paths to production user exports, private IPs and application .env paths.

Logs alone contained 11,986 of the 23,476 gap protections, or 51.06% of the total.

It was also the category with the largest mandatory-coverage difference:

• Layrin: 92.04%
• OpenAI Privacy Filter: 45.78%

So the category where Layrin received the biggest over-redaction penalty was also the one where it protected much more mandatory information.

That does not mean every extra protection was necessary. It wasn’t. But the trade-off is pretty visible.

The gap protections were not only infrastructure values. They also included:

• 2,698 date/time spans
• 2,490 organization/company spans

RedactionBench can reasonably classify these as values that should remain visible under its policy. A company can also reasonably decide that an exact company name, date or internal resource is not needed by an external model.

That is why I’m hesitant to treat every benchmark false positive as information that was pointless to protect.

It wasn’t only Logs

Layrin had higher micro mandatory coverage in all 11 categories.

Even Files, the only category where OpenAI had slightly higher mean document coverage, looked different when mandatory entities were pooled:

• Micro mandatory coverage: 70.68% Layrin vs 64.44% OpenAI
• Completely missed mandatory entities: 778 vs 978

So one unusual category was not carrying the whole result.

Where I ended up

I don’t think R-Score is bad. It measures selectivity, which my mandatory-only metric intentionally ignores.

What I’m less sure about is treating the benchmark’s protection boundary as a universal privacy boundary. A hostname, date, company name or internal resource can be considered unnecessary redaction by the benchmark while still being something a real user does not want to send outside their environment.

For me, one combined number is not enough here. I would want to see at least two things separately:

1. How much mandatory information escaped?
2. How much additional information was protected outside the benchmark boundary?

In this experiment, those two dimensions separated a lot. OpenAI Privacy Filter had the much better combined R-Score. Layrin protected much more mandatory information, but also protected much more outside RedactionBench’s selected boundary.

Should privacy benchmarks report protection failures and over-redaction separately, instead of letting one offset the other in a single score?

Methodology

Both systems received the same 200 RedactionBench documents, with ground-truth annotations unavailable during inference.

Layrin Desktop 0.1.4.0 used its frozen production local-protection and tokenization pipeline, with the production configuration unchanged during the evaluation.

For some structured inputs, I used deterministic inference segmentation. This only changed the inference boundaries. The source text was unchanged, predictions were mapped back to the original offsets, and every source file still counted as one benchmark document.

OpenAI Privacy Filter was run locally using its public implementation.

Links

Full study, category tables, methodology and reproducibility details:
https://layrin.com/research/openai-privacy-filter-vs-layrin-redactionbench

OpenAI Privacy Filter:
https://github.com/openai/privacy-filter

RedactionBench paper:
https://arxiv.org/abs/2606.18782

I'm thinking of grabbing some of these, but people are saying they have very slow bandwidth due to being mining cards and this causes problems.

I don't fully understand what problems are that are caused from this, but my biggest question is: if I am able to load the entire model in VRAM with no offloading, wouldn't that solve the issue?

And, if there are multiple GPUs that load a bigger model by splitting the layers, wouldn't the bandwidth issue not be too much of a problem as it would only need to move the already computed forward passes to the next layer loaded in GPU?

Which wouldn't be as big as moving the entire model weights so I should still be able to load something like deepseek flash, provided I have enough cards for the VRAM needed, or is this wrong?

If someone could please explain this to me before I drop a grand buying a bunch of these cards, I would greatly appreciate it!

Hi people of LocalLLaMa,

I have been wondering for quite some time now - and this all started after I read some comments complaining about the pricing on Qwen 3.8 27B as opposed to DSV4 Flash that it mainly was driven by how massive its KV cache overhead was. And while I did agree with that, what I did wonder later on was why could we not finetune that away.

Apparently, I am not the only who has thought of that - Arcee, an open source friendly company that does a lot of neat work and gave us AFM had a similar idea. They took their model, and also opensourced their 'DistilKit'.

Among the notable work on that article, what stood out the most was the fact that it was feasible. However, they did face some challenges, finetuning this newer layer to learn from the teacher (in this case imagine the GQA layers from the teacher has the goal to teach the newly placed KDA layer in the student to mimic its representations/embeddings similarly (can be measured by cosine and other things to see how well that is going) - what they found was that while it could fairly close on a pretty small finetuning task (I believe they did on a 1B tokens only), they noted the performance nose dived for GSM8K while for some other datasets they measured remain almost equivalent even though that finetune was much smaller than the original training.

I then decided I would do this for this model, and designed a strategy on how layers would be loaded, their representations would be cached, how the student layer would then be loaded and so on. In the initial runs each update on a DCLM (initial run was on smaller sequences sized 512, then I did a 2048, and then a 4096) but it was roughly about 262K tokens in unique total. And the performance was not surprisingly poor, yes it was not as good as a straight launch, and definetly fell apart just as Acree said especially on areas the new layers werent familiar with and hadnt seen the teachers behaviour.

So, why the long post which is just text?

Well, I was wondering, is there a way we could as a community pool our resources (I don't actually know how we would do this) and do this finetune together? Because I have tried, and alone it might not be feasible - I have already spent over 100 dollars this month on various experiments and using vast ai for the most part. This might just be our own community win, and all of us would put our names on the HuggingFace and come as collaborators, and might even point out issues and fix them as we go along. Most of the design stage and what parameters and datasets to use and how to use them and what to look for and where to look for is done by me before the LLMs take over the agentic role of ensuring the run runs, the code works, the eval comes out and what it looks like and we could work together to find holes in that and see well we missed x that is why the behaviour y is observed etc.

I don't know though. This is just me thinking out loud with the community. Y'all tell me what ideas you have on how we could do this resource-sharing so that we could do this finetune at scale rather than me doing it at say just 1B tokens and then it being good enough for most benchmarks but not really so at others.

Interestings reads on this: https://www.arcee.ai/blog/distilling-kimi-delta-attention-into-afm-4-5b-and-the-tool-we-used-to-do-it

You can look me up here: https://huggingface.co/amkkk or https://darthamk97.github.io/ (I don't really keep this as up to date as I wished)

i just don’t see the use case for them. other than web crawling ?

With same prompt (4081 tokens) and same params except for the context size change:

the context 131072 have pp 778.91 t/s and tg 40.68 t/s

the context 65536 have pp 137.87 t/s and tg 13.36 t/s

Using RX 9070 XT, all gpu offloaded

Based on this GPU specs, it seems the correct behavior is the pp 778.91 t/s and tg 40.68 t/s from context 131072. But I interested to know why 65536 degrades to much with this params. There is any rule that I broken with 65536 context? The trained context of 262144 is divisible for 65536, so I out of ideias.

With --ctx-size 65536

llama-server --host 0.0.0.0 --port 8078 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 --kv-offload --threads 14 --device Vulkan0 --no-mmap --log-verbosity 4 --ctx-checkpoints 10 --slot-prompt-similarity 0.4 --cpu-range 0-13 --cpu-strict 1 --cache-ram 10240 --model Qwen3.8-27B-UD-IQ3_XXS.gguf --parallel 1 --ctx-size 65536 --temp 1.0 --top-k 20 --top-p 0.95 --min-p 0 --jinja --spec-type draft-mtp,ngram-mod --spec-draft-n-max 2 --spec-draft-type-k q8_0 --spec-draft-type-v q8_0 --spec-draft-ngl all --spec-draft-device Vulkan0 -ngl 99 --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64 -ub 512 -b 1024

0.40.844.179 I spec        begin: ngram_mod occupancy = 3974/4194304 (0.00)
0.43.380.083 I cmn  common_reaso: deactivated (natural end)
0.45.110.170 I slot print_timing: id  0 | task 0 | prompt eval time =   29600.10 ms /  4081 tokens (    7.25 ms per token,   137.87 tokens per second)
0.45.110.173 I slot print_timing: id  0 | task 0 |        eval time =    4265.11 ms /    58 tokens (   74.83 ms per token,    13.36 tokens per second)
0.45.110.174 I slot print_timing: id  0 | task 0 |       total time =   33865.21 ms /  4139 tokens
0.45.110.177 I slot print_timing: id  0 | task 0 |    graphs reused =         21
0.45.110.188 I slot print_timing: id  0 | task 0 | draft acceptance = 0.29091 (   32 accepted /   110 generated), mean len =  2.33
0.45.110.189 I slot print_timing: id  0 | task 0 |      acc per pos = (0.833, 0.500, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000)
0.45.110.212 I spec common_specu: statistics        ngram-mod: #calls(b,g,a) =    1     24      1, #gen drafts =      1, #acc drafts =     1, #gen tokens =     64, #acc tokens =     1, #mean acc len = 2.00, #acc rate/pos = (1.000), dur(b,g,a) = 0.355, 0.044, 0.001 ms
0.45.110.215 I spec common_specu: statistics        draft-mtp: #calls(b,g,a) =    1     23     23, #gen drafts =     23, #acc drafts =    20, #gen tokens =     46, #acc tokens =    32, #mean acc len = 2.39, #acc rate/pos = (0.870, 0.522), dur(b,g,a) = 0.002, 161.875, 0.016 ms
0.45.110.419 I slot      release: id  0 | task 0 | stop processing: n_tokens = 4138, truncated = 0
0.45.110.425 I srv  update_slots: all slots are idle
^C0.52.485.471 I srv    operator(): operator(): cleaning up before exit...
0.52.486.333 I common_memory_breakdown_print: | memory breakdown [MiB]                  | total   free     self   model   context   compute    unaccounted |
0.52.486.335 I common_memory_breakdown_print: |   - Vulkan0 (RX 9070 XT (RADV GFX1201)) | 16384 = 2091 + (13774 = 10953 +    2624 +     196) +         517 |
0.52.486.335 I common_memory_breakdown_print: |   - Host

with --ctx-size 131072

llama-server --host 0.0.0.0 --port 8078 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 --kv-offload --threads 14 --device Vulkan0 --no-mmap --log-verbosity 4 --ctx-checkpoints 10 --slot-prompt-similarity 0.4 --cpu-range 0-13 --cpu-strict 1 --cache-ram 10240 --model Qwen3.8-27B-UD-IQ3_XXS.gguf --parallel 1 --ctx-size 131072 --temp 1.0 --top-k 20 --top-p 0.95 --min-p 0 --jinja --spec-type draft-mtp,ngram-mod --spec-draft-n-max 2 --spec-draft-type-k q8_0 --spec-draft-type-v q8_0 --spec-draft-ngl all --spec-draft-device Vulkan0 -ngl 99 --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64 -ub 512 -b 1024

0.33.776.812 I spec        begin: ngram_mod occupancy = 3974/4194304 (0.00)
0.34.529.238 I cmn  common_reaso: deactivated (natural end)
0.34.859.457 I slot print_timing: id  0 | task 0 | prompt eval time =    5239.38 ms /  4081 tokens (    1.28 ms per token,   778.91 tokens per second)
0.34.859.459 I slot print_timing: id  0 | task 0 |        eval time =    1081.71 ms /    45 tokens (   24.58 ms per token,    40.68 tokens per second)
0.34.859.460 I slot print_timing: id  0 | task 0 |       total time =    6321.08 ms /  4126 tokens
0.34.859.463 I slot print_timing: id  0 | task 0 |    graphs reused =         15
0.34.859.475 I slot print_timing: id  0 | task 0 | draft acceptance = 0.25510 (   25 accepted /    98 generated), mean len =  2.39
0.34.859.475 I slot print_timing: id  0 | task 0 |      acc per pos = (0.778, 0.611, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000)
0.34.859.496 I spec common_specu: statistics        ngram-mod: #calls(b,g,a) =    1     18      1, #gen drafts =      1, #acc drafts =     1, #gen tokens =     64, #acc tokens =     1, #mean acc len = 2.00, #acc rate/pos = (1.000), dur(b,g,a) = 0.348, 0.032, 0.001 ms
0.34.859.500 I spec common_specu: statistics        draft-mtp: #calls(b,g,a) =    1     17     17, #gen drafts =     17, #acc drafts =    14, #gen tokens =     34, #acc tokens =    25, #mean acc len = 2.47, #acc rate/pos = (0.824, 0.647), dur(b,g,a) = 0.002, 118.512, 0.018 ms
0.34.859.738 I slot      release: id  0 | task 0 | stop processing: n_tokens = 4125, truncated = 0
0.34.859.743 I srv  update_slots: all slots are idle
^C0.41.190.565 I srv    operator(): operator(): cleaning up before exit...
0.41.191.274 I common_memory_breakdown_print: | memory breakdown [MiB]                  | total             free     self   model   context   compute    unaccounted |
0.41.191.276 I common_memory_breakdown_print: |   - Vulkan0 (RX 9070 XT (RADV GFX1201)) | 16384 = 17592186044285 + (16014 = 10953 +    4800 +     260) +         500 |
0.41.191.276 I common_memory_breakdown_print: |   - Host

I need a good model that feels smart ish in this regard but also runs with all my other stuff (audio gen, video gen, etc) enabled.

I guess when k3.1 comes out, it will be fable 5 lev, so maybe this month followed by minimax m3 pro and glm 5.5 . I guess open mods will reach Fable 5.1 level by December 2026 to January 2027 . Deepseek seems to be behind other labs on performance and benchmarks

Found this a few months back for more compute-efficient (relative to memory size) MoE models. Nowadays Qwen3.6-35B-A3B exists along with Qwen3.8-27B but MoE SLMs might not get another around. Maybe Marco can be recycled somehow along with the other two Qwen models to make things extra fast? https://www.reddit.com/r/LocalLLaMA/comments/1sgzt0p/marcomini_173b_086b_active_and_marconano_8b_06b/

And this is not just finetuning and knowledge distillation, there has to be some way of recycling individual layers or even do per-layer distillations. nVidia might have tried something like this but not that many people approached this from a "faster training than token-level" perspective.

I used Claude Code to help write this pipeline - Gemma4 to turn an idea into a short story, then IndexTTS 2.5 running on an NVIDIA 3080 to use voices from LibreVox and VTCK voices to pin as character voices. It uses Whisper to do Quality Control on the output clips and assigns 0-1 score if the Whisper transcribed output matches. It also has some tools it wrote for tone analysis but they're not great.

You can:

1) Set speed and emotion for the sentence

2) Insert pauses (ffmpeg) using the notation (200ms) between words

2) a) adjust the pause location +/- 1 ms because a lot of the pauses would somehow end up inside a word instead of between them

3) Insert Pauses between sentences

4) Do fine tuning with per-word/per-phrase emotion and speed

5) Chapter wide speed

6) Take a particularly good take and apply that as a reference

7) Re-cast characters chapter wide and book wide

8) Stitch all the sentences together and play the whole chapter

9) Download the completed chapter

10) Add filters to a sentence (reverb, muffle, telephone)

11) Bake filters into a character and make it part of the character.

I started by using kokoro but I wanted something near human. I first had Claude make a LaTeX inspired tool for me to change the Index TTS settings INSIDE the book/Script but that became too unweildy, so I had it make a web UI. I kept adding features out of frustration.

I have a separate agent analyzing the audio of various librevox books that are also in project gutenberg to learn the standards for tone analysis. The goal is to make a one-shot near human (or at least pleasant to listen to, and expressive) idea -> audio book by having an agent keep re-rolling takes on IndexTTS, running QC for pauses, pronunciation, tone, and sending me the final audio book when done.

Apart from Claude helping write the code, the actual models are self hosted.

Here is a close up of the voice editing:

I was shocked that even with ultra elevenreader, YOU CAN'T DO MULTIPLE VOICES??

My current workstation has 32gb ram and I have three 16gb cards of all different brands laying around. With a riser and some Lego I managed to wedge the AMD xt 7900 and Nvidia Quadro 5000 card into my case (of the top of my head)

With llama.cpp I get about 20t/s on Qwen 3.8 27b q4_k_m with 128k context fully GPU resident. It's totally viable as a coding model, just too slow...

Of course you can spend 10k on a machine which is not happening. I'm wondering if there is like a 1k upgrade that unlocks interesting capabilities? System ram for moe models or a less mismatched GPU setup...

I am absolutely blown away. Yes my setup is crap but the fact that it managed to do this in a single take is unbelievable (and I'm a developer).

Hardware used:
- Windows PC with 4070ti (12GB VRAM, 32GB RAM)
- Macbook M5 Air (LLAMA.cpp RPC connection to Windows PC)

Software used:
- LLAMA.cpp (Q4KM, xhigh, 8bit KV, MTP=1)
- Lmstudio Qwen 3.8 27b (Q4KM) GGUF
- Deepseek harness (mode: minimal)

Prompt: "please create a fully self-contained super mario game with only one short level, put everything inside mario.html inside the current directory"

context: 64k
thinking: xhigh
time took: 117 minutes
avg tps: 7.6

resut: https://pastebin.com/qyBu64sP

https://reddit.com/link/1w4821c/video/qpukeg1y4wmh1/player

Add 4 experts into Dense model and confirmed recovering model's ability up to "general level".

Hey, google. Please release official 124B MoE model!!!!!!!

hello,

I'm using Deepseek flash regularly and from time to time i see it deviating and start doomlooping or generating gibberish. It's somethign i already saw in heavily quantized model buthere i used official deepseek release https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731 .
I would be curious to know if anyone encountered such thing and how they solved it .

Here is my config :

vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \
--trust-remote-code \
--safetensors-load-strategy prefetch \
--dtype bfloat16 \
--tokenizer-mode deepseek_v4 \
--reasoning-parser deepseek_v4 \
--tool-call-parser deepseek_v4 \
--enable-auto-tool-choice \
--attention_config.use_fp4_indexer_cache True \
--block-size 256 \
--kv-cache-dtype fp8 \
--enable-prefix-caching \
--max-num-seqs 32 \
--max-num-batched-tokens 16384 \
--max-model-len 131072 \
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
--max-cudagraph-capture-size 256 \
--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}' \
--moe-backend deep_gemm_mega_moe \
--enable-expert-parallel \
--gpu-memory-utilization 0.93 \
--no-enable-flashinfer-autotune \
--host 0.0.0.0 --port 8000

Thanks guys!

I kept seeing guys in this sub saying how AtomicChat's Qwen3.8-Flash-Next quant is so good, fits in their machine when unsloth's can't, runs faster than other quants etc, so I went check out what's happening there.

First thing I noticed was that AtomicChat's Q4_K_M quant is suspiciously small when the ngram table is removed (only ~56GB), it seems like most of the tensors in this quant are IQ2_S instead of the usual Q4_K, Q5_K and Q6_K that you usually find in Q4_K_M quants, the GGUF filetype metadata also says IQ2_S instead of Q4_K_M. In their model card, their Q4_K_M also has suspiciously high KLD (0.084).

It seems pretty obvious to me that they're pretending a IQ2_S quant as a Q4_K_M, but at the same time I'm genuinely not sure because it can't be only me who found this right? How can nobody be pointing this out? Am I missing something or what may they be doing?

Their HF repo ID: AtomicChat/Qwen3.8-Flash-Next-GGUF