Resources
Which current local models that can run within 128GB generate the best SVG pelicans?
I used a famous Simon Willison's pelican riding a bicycle prompt on the biggest local LLMs that can run on 128GB Apple Silicon. U used quantizations by Unsloth.
Qwen3.8 Flash-Next gives a lot of details.
DeepSeek V4 Flash is strangely underwhelming.
Qwen3.8 27B still rocks, and I like its consistent minimalism.
Is Qwen3.8 27B still large at 31GB?
It is! But for this tasks 2-bit quantizations (at around 12GB) will give the same results. For more complicated coding, 4-bit are more than enough. RTX cards are well enough!
In my opinion, that test doesn't make sense because the models were trained on that specific task. You should be more creative and try something different to avoid benchmaxxing
Left to right preference is common among people using the Latin alphabet as their primary alphabet. I guess it would also be common without benchmaxxing a pelican
tbh the direction thing is probably more likely because bicycle photos are typically taken from that side (generally referred as 'drive side') to display the parts that are on it.
Thanks for sharing. I guess Deepseek V4 Flash under-performing is reasonable as it's run on IQ3, quite natural performance drop as trade off to fit in 128 GB unified ram.
I guess Deepseek V4 Flash under-performing is reasonable as it's run on IQ3
Usually, a bigger model at a smaller quant should work better than a smaller model at a bigger quant, since the sizes here are comparable, and DeepSeek is even bigger, so the comparison is fair.
Still, the quants could be of different quality, and the Deepseek V4 could simply have less training data like that and more data for something else.
This is speculative.
At some point the quant will be degraded so much it will fail. Equally some models may be so compressed that quantizing will have exponential loss in contrast to smaller models.
Tldr there isn't 1 rule, and it's why the pelican riding a bike is important.
This is speculative. At some point the quant will be degraded so much it will fail.
Sure, but usually that will happen after the size of the bigger quantized model becomes smaller than the size of the smaller, less-quantized model (and often even then bigger model stays better). Here, the total size of DeepSeek Flash is 104 GB, while Qwen Flash is 94 GB.
Also, bigger models are much more resilient to quantization. They may lose some precision, but their reasoning tends to suffer less.
I had Qwen 3.8 Flash Next vibe up a little reiterative SVG making script where it generates a SVG then self-corrects any minor flaws before the final output, I've been pretty impressed with what it can do with novel prompts.
People are quick to jump to the conclusion of the training data being contaminated by this "test". First, I don't imagine SVG creating ability is remotely a focus of the training, especially not specifically the pelican
Secondly, that doesn't account for the ability to structure SVG for things they will never have been asked to do before:
I know this is a little disjointed compared to the almost pixel perfect SVG they can create when it's an ambiguous prompt they are free to make assumptions on. But being able to oneshot from a photo and largely maintaining the positioning and vibe is still insane
I honestly believe their SVG creation capability is fundamentally just a byproduct of their raw coding ability. Being able to create styled UI components in general demands skill with positioning and composition. So I don't think they are benchmaxed on SVG, they are just very capable in the domain that lends well to making SVG
Nice comparison. I would be curious to see SVG validity, render success, token count, latency, and editability measured alongside visual quality. For local use, the best model may not be the one with the prettiest one-shot pelican, but the one that reliably emits valid, compact SVG that survives small prompt edits and quantization.
86
u/jacek2023 llama.cpp 18h ago
In my opinion, that test doesn't make sense because the models were trained on that specific task. You should be more creative and try something different to avoid benchmaxxing