Deepseek flash 0731 doomlooping

[u/Best_Sail5]

hello,

I'm using Deepseek flash regularly and from time to time i see it deviating and start doomlooping or generating gibberish. It's somethign i already saw in heavily quantized model buthere i used official deepseek release https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731 .
I would be curious to know if anyone encountered such thing and how they solved it .

Here is my config :

vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \
--trust-remote-code \
--safetensors-load-strategy prefetch \
--dtype bfloat16 \
--tokenizer-mode deepseek_v4 \
--reasoning-parser deepseek_v4 \
--tool-call-parser deepseek_v4 \
--enable-auto-tool-choice \
--attention_config.use_fp4_indexer_cache True \
--block-size 256 \
--kv-cache-dtype fp8 \
--enable-prefix-caching \
--max-num-seqs 32 \
--max-num-batched-tokens 16384 \
--max-model-len 131072 \
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
--max-cudagraph-capture-size 256 \
--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}' \
--moe-backend deep_gemm_mega_moe \
--enable-expert-parallel \
--gpu-memory-utilization 0.93 \
--no-enable-flashinfer-autotune \
--host 0.0.0.0 --port 8000

Thanks guys!

[u/locbuilds]

if it only started once dspark was in the serve command that's where i'd start, speculative decoding plus draft_sample_method probabilistic is a pretty common way to get the main model stuck repeating after a bad draft gets accepted. first ablation: drop the whole --speculative-config and keep fp8 kv, prefix cache, 131k, expert parallel, all of it identical. if the loop dies it's the drafter not the official weights. next try greedy on the draft instead of probabilistic, then num_speculative_tokens 1-3 instead of 7. if killing spec doesn't help, turn off --attention_config.use_fp4_indexer_cache and try kv-cache-dtype auto, those two plus FULL_AND_PIECEWISE graphs on a new 0731 kernel path can garble long context. i wouldn't touch block-size 256 until those are ruled out.

[u/Best_Sail5 → u/locbuilds]

thanks mate il will try

[u/Easy_Werewolf7903]

Try to use the exl3 quants, they are smaller and more accurate compare to GGUF quants. https://huggingface.co/turboderp/DeepSeek-V4-Flash-0731-exl3

[u/Best_Sail5 → u/Easy_Werewolf7903]

i dont use gguf but the native fp4 deepseek arch

[u/Square_Light1441]

lower the temp to like 0.7, recovers a lot of quality

[u/EmilPi]

Seeing same with GGUF. Some queries send it into loop.
Maybe this is same case as with Qwen3.5/6: common wisdom is that lack of / short system prompt increases likelihood of loops, larger system prompt decreases. But this needs testing for DeepSeekV4F.

[u/EmilPi → u/EmilPi]

Also, I see you use fp8 cache - this likely increases loop likelihood too.

[u/Nepherpitu → u/EmilPi]

Deepseek is fp8 kv native, it doesn't support fp16.

[u/EmilPi → u/Nepherpitu]

Ok... I checked https://vllm.ai/blog/2026-04-24-deepseek-v4 and don't see mentioning this.
Its weights are fp4/fp8 native, depending on layer, yes. But KV cache?..

[u/Nepherpitu → u/EmilPi]

Yep, kv cache. Check recipes on vllm and sglang

[u/EmilPi → u/Nepherpitu]

Link please?