r/LocalLLaMA • u/ChemistNo8486 • 21h ago
Discussion Don't sleep on Vision support for coding!
Normally, whenever a new model dropped, I always chose the non-vision version just to save VRAM; I though that only use case was when you were the one sending the picture.
However, with the release of QWEN 3.8 27B I decided to give it a shot, and it has been one of the best decisions I have made, as this makes the model way more capable for autonomous coding.
With no vision, the model will try to complete the task and get back to you once it thinks that it is done with no problem. But there are a lot of silent errors that do not get reflected via the code or the tests performed, so you could go back to an error screen or a broken page after getting a confirmation of your request being implemented correctly.
On the other side, when I ask something to QWEN with vision support, it will work on it, and then proactively take a screenshot to confirm if everything is right. This has helped numerous times with spotting errors that were missed. The model will continue to reiterate and take screenshots until it gets a visual confirmation of the issue being fixed. Just magnificent.
Btw, I currently run my local set up via Hermes with QWEN 3.8 27B (Qwen3.8-27B-UD-Q5_K_XL.) powered by a 5090.
79
u/BawbbySmith 20h ago
Sounds like it’s specific to ui/web dev. Working on backend systems it hasn’t really played a factor
39
u/EstarriolOfTheEast 19h ago
Not just webdev, all UI work. Also game dev (screenshots, level design), visual fx (shaders, procedural materials, particle effects), 3d design (3d modeling and level or interior design), various visualization types.
2
u/Themash360 13h ago
Yeah for sure in these cases it’s an extremely important part of the feedback loop
1
u/simplyeniga 3h ago
Fair but sometimes you want a UI test for your backend APIs to see the experience from someone consuming your service
15
u/cogitech2 21h ago
I've just discovered this benefit recently, too. I was watching a long autonomous session of Qwen3.8-27B building a game and I kept seeing it think "Oh, this model doesn't support vision, so I can't see the game screenshots..." etc.
So I grabbed the mmproj and got it up and running and continued the session with "You can see now".
Boom. It continued the session and used vision to inspect and correct its work.
60
u/XiRw 21h ago
Never saw anyone who did. It can and always has been helpful. If you want to save vram just offload it to the cpu. It’s not as slow as people think
6
u/KingCpzombie 19h ago
Wait, you can offload just vision to the CPU? How do you do that? Sounds like a good way to squeeze some extra context in!
14
u/XiRw 18h ago
If using llamacpp it’s —no-mmproj-offload
4
u/slavik-dev 16h ago
Yep. Just know that it's 5-10 times slower whenever it works with images. Textual t/s not affected.
2
u/CapsAdmin 14h ago
When I tried it, it was just too slow. I tried giving llamacpp all my 32 threads (16 cores) and it felt like it took 1 minute for one image. But GPU it felt instant. Maybe the images were too big?
So I just have a profile with and without vision that I switch between depending on the task.
2
18
u/Randommaggy 19h ago
I've offloaded vision to another machine through a skill firing requests at Gemma 4 12B QAT and for a lot of tasks Gemma's vision works better than Qwen's and it's faster.
5
1
u/ancapsaicin 12h ago
12B is an odd choice for vision.
Even with resolution maxed out, I find that E4B —or 26A4B if your machine affords— can see more details in the pictures. E4B may not recognize a Vermeer painting and attribute it to Lorenzo Linguini the famous Renaissance painter from Pizza but it will see the girl.
8
u/FullOf_Bad_Ideas 19h ago
I'm always ignoring vision for coding. The things I work on are usually not visual tho, just backends.
6
-1
u/klymaxx45 21h ago
Just thinking in my head, why would you not want vision?
19
u/howardhus 21h ago
to save vram. not everyone has vram to spare like you..
i mean.. why would *anyone* not just go on vacation with a private jet??
12
-10
u/klymaxx45 21h ago
I still don’t see the point. You’re trying to stuff a model into max hardware capacity, but while doing that, you’re capping the model’s capability and quality.
7
u/Bulky-Priority6824 21h ago
It took its own screenshot and "looked" at it https://imgur.com/a/WPeeDYM
They'll use this to snipe us in 2035
7
u/kiwibonga 20h ago
For me the thing that made even more of a difference was the ability to inject code into the running application -- some solutions already exist for game engines but you can usually write your own for your specific application; a HTTP server inside your app that can compile and run code it receives from the LLM on the fly. Your agent can just use CURL to send code to the running app, and receive a response in json. It can workshop ideas and verify APIs immediately in the right context (whatever menu is open, whatever application state you're in), and it can poke and prod the running app as it wishes, reading memory, performing spatial queries, etc.
7
u/Chinalke 20h ago
How did you get it to automatically take a screenshot?
6
u/KURD_1_STAN 19h ago
Don't like most harnesses have tools for that? I dont run local models so cant give u any answer also nobody can answer u without mentioning ur harness. Either it is built-in and alread does that if needed, need an extra install/plugin or doesnt have it. A sinple google search will answer u based on the harness u are using
2
u/cantgetthistowork 10h ago
What if the harness is not running on where the frontend will be deployed?
2
u/cinnapear 18h ago
Qwen3.8 did it on its own. Started the game in a window, took a screenshot, and looked at it.
1
u/terablast 9h ago
Most of the tools that give the capability to control apps (Playwright CLI/MCP, Chrome DevTools MCP, FlaUI MCP, etc.) will also come with something to allow taking screenshots.
-2
u/Constandinoskalifo 20h ago
!RemindMe 1 day
0
u/RemindMeBot 20h ago edited 19h ago
I will be messaging you in 1 day on 2026-09-02 01:08:36 UTC to remind you of this link
1 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.
Parent commenter can delete this message to hide from others.
RemindMeBot is switching to username summons. Instead of
!RemindMe 1 day, useu/RemindMeBot 1 day. More info.
Info Custom Your Reminders Feedback
5
u/gpt872323 20h ago
For me, vision support is critical; it’s what makes the model genuinely useful, especially in today’s model lineup. Maybe usage of other cloud models has set the baseline expectations. If you’re coding, you often need to share a screenshot with tools like Chrome DevTools MCP, so this is very much a use-case decision, not a blanket rule. For backend-only work, your point makes sense. But if you’re juggling multiple projects, constantly switching models is a hassle unless your GPU can keep both in memory.
3
2
u/Lumpy-Comedian-1027 15h ago
No webdev without playwright tests and let it take screenshots - or videos
2
u/createthiscom 7h ago
I was thinking about this the other day. How many of you are shadow AI’ing via qwen3.8-27b’s vision via a software kvm paired with playwright or something?
4
u/how-can-i-dig-deeper 21h ago
how much vram does it add for you?
17
u/cogitech2 21h ago
Zero if you use no-mmproj-offload = true
-2
u/ancapsaicin 12h ago
that sounds like the exactly wrong flag to use
3
u/cogitech2 11h ago
I didn't create the flag, I am just reporting that it is the flag to use to ensure the mmproj is handled by the CPU, therefore it won't use any extra VRAM.
Look it up if you don't believe me. It's what I use and it works.
-3
u/ancapsaicin 11h ago
lol
Getting code review flashbacks from back in 'Nam?
I just commented on the naming of the flag which you claim to never have created.
4
2
u/LukeSchlather 17h ago
I've honestly just been using Claude, but I would highly recommend looking at OmniParser as well as straight OCR (tesseract/easyocr.) Even with Fable I've found LLMs tend to hallucinate stuff within an image. Like I was trying to use it to debug some VM driver stuff and it straight-up read a blank image and claimed it was such and such. Running OCR falsified a lot of its claims. Omniparser is a pretty small and it can actually tell the LLM which UI controls are active if you're tabbing through some dialogs. I actually settled on a pattern where I have a thing monitoring the screen as things progress and detects state changes, and it also compares OCR and Omniparser which often give subtly different results. Another thing that can help is when the LLM wants to click on a button, run omniparser and make sure the button is actually there. Cropping just to the button can be a time-saver here since omniparser can be a little more expensive at the page-level.
1
1
u/ieatdownvotes4food 20h ago
agreed, no way vision goes off. and it will use it on its own without asking for it
1
u/awitod 20h ago
If you are using any harness all you need to do is make sure the environment is setup with the Open AI SDK in your environment and your model runner’s Open AI completions compatible endpoint.
I just finished a job where I am making a slide. It searched the web for images and then used the SDK to examine the images to find ones that matched what it was looking for.
The slide is html, so it then used playwright to take a screenshot and used it again to QA the rendered html visually.
1
u/HenkPoley 18h ago
Qwen3.8-27B is at the top of the open weights image-to-website leaderboard at arena.ai
https://arena.ai/leaderboard/code/image-to-webdev?license=open-source
1
1
u/Extreme-Pass-4488 17h ago
also assemble a sepparate agent that takes care of the images so they do not trash your context .
1
u/No-Refrigerator-1672 15h ago
Cool, but how? Every single OSS coding harness I know doesn't work with vision out of the box. How are you getting it to work autonomously? Is it a playwright mcp? Is it a plug-in for opencode? Or have you just rerouted Claude Code to your endpoint?
1
u/sumane12 15h ago
Deepseek harness and openclaw work perfectly with vision.
0
u/No-Refrigerator-1672 15h ago
They only let the you paste screenshots, or load images from disk. They won't let your model to use vision automonously. Openclaw has playwright, but it'll only let it take pictures of web pages; no autonomous coding for native applications.
1
u/virtualicex 13h ago
under PI I asked to create an extension to take a screenshot and produce the description of the image accepting a prompt: so the llm at startup reads about the extension and how to use it and every time it needs makes use of it
1
u/lemondrops9 12h ago
Ive just told Opencode my vision model can be accessed from this address and away it went.
1
u/returnity 6h ago
Pi just accesses Chrome Devtools and opens a headless browser, no setup needed on my system
1
1
u/South_Hat6094 9h ago
vision support matters most when the model can verify its own UI work. The silent-failure angle is the part people miss; screenshots catch the weird, half-broken states.
1
1
u/deadsoulinside 8h ago
I need to really use Qwen for coding, but I could not agree more with this statement. I am using claude for coding tasks, but the preview/screenshots it takes helps a ton, since it can see when things are not displaying right and immediately go and fix things.
Way better than having to explain to a model what this ui element is not doing correctly and a possible suggested fix.
1
u/zipeldiablo 21h ago
You can do snapshot testing even without vision support you know.
Vision helps on the implementation part and targeted debug i would say
2
u/ChemistNo8486 21h ago
Oh, yeah, but they are different tools. Both super useful, but the snapshot testing will not detect visual bugs that the DOM snapshot can't see.
For example, I am working on a fix for a music interface I am creating, as Navidrome did not have all I wanted.
The issue is was that my playlists were not showing, so Hermes ran a DOM and found no errors. However, he did a double-check with a screenshot and that is where he spotted that there was a visual issue with the render. Without the vision he couldn't have done it.
1
u/zipeldiablo 21h ago
For navidrome just use it as the library backend and there are plethora of frontend app using the subsonic api.
Not sure you need to make your own, what feature do you want that you couldn’t find?
Hum that’s weird 🤔 but arent there mcp tools that still can test the rendered ui directly not the dom? I think chrome devtools do that no?
And yeah i would’ve thought it was a db issue 😂
1
u/ChemistNo8486 21h ago
My music is not from official sources so a lot of it is "contaminated" (AKA songs with the same names, covers, instrumentals, etc) and I have +3000 songs.
I need a button that can "quarantine" that song by removing it from the playlist and then looking for different sources with a pipeline so I can then check it later.
I am not trying to reinvent the wheel, just that this issue is just super specific and none of the deterministic methods like acoustic analysis works, as some are just covers and the audio is basically the same. And I could't find any repo either. Any other methods would be appreciate it lol
1
u/zipeldiablo 21h ago
Oh. My music library got destroyed but tagger tools for the same reason :(
You can use soulsync for that
1
u/ChemistNo8486 20h ago
My problem is that the tags are all correct, it is just a content issue, as the songs are just wrong despite having the right tags. I basically need to remove the wrong song completely and try to look for other download source.
I am not paying and not using any official method, so it explains why some data is corrupted. Just the price to pay for data ownership I guess.
0
u/Fair-Perspective7352 15h ago
That failure mode is real - tests only check what you thought to assert, and rendering/CSS bugs almost never show up in test output. A screenshot pass catches a whole class of silent breakage.
Tradeoff for me is the extra tokens and latency per loop, so I only enable vision when the task actually has a UI to look at. For pure backend work there is nothing to screenshot anyway.
Are you having it screenshot on every iteration, or only when it thinks the task is done?
-4
u/acadia11x 21h ago edited 21h ago
I see the last line, you are running with Hermes. So it’s doing the screen cap? The model itself doesn’t do that … that’s the disconnect. Hermes is doing that is what you are saying? And since you are using a vision capable model hermes can feed the model with screen cap it took. You leave alot for someone to assume. Are you a developer? Bad design document
5
u/ChemistNo8486 20h ago
No offense but it is super clear, it just looks like you are confused on what is Hermes and what it does. Hermes is an agent, not a model. It provides the harness and all context needed to execute coding tasks, so it is just the architecture.
In this case, the LLM is always QWEN 3.8 27B, so there is no disconnect, QWEN is indeed executing the screen caps with the agent's architecture. The same is possible with Claude Code or any other agent.
-2
u/acadia11x 19h ago edited 19h ago
Dude I know what Hermes is as an agent framework and Hermes is also model by the way but that’s beside the point, I have open claw with Hermes framework just playing with where open claw is the landlord and Hermes agent framework handles more complex tasks … but that aside I’m asking because I don’t understand how qwen the model is doing screen cap, it’s a model, not a harness. I’m running qwen visual in vLLM which by the way is huge for 5090 which I also have, and tested it with open web on the visualization. The part I was unfamiliar is the auto screen capture … the point of asking is to understand, if it was clear to me I wouldn’t ask? Claude code is also an agent not model but uses Claude model… qwen is a model, where is it an agent to do this screen cap as a loop? Maybe something I don’t know about qwen that’s why I’m asking? You can choose to share knowledge or not
3
u/cogitech2 20h ago
You are simply wrong about the way this works. The mmproj adds vision ability to the MODEL. So if the model decides to take screenshots of its work, it can literally "look" at the screenshots, identify issues, and then fix them - with no operator input.
This works regardless of what harness is used (Hermes, Pi, etc. etc.)
1
u/acadia11x 19h ago edited 19h ago
Im not trying to be right I’m trying to understand
A model that supports vision works with its mmproj but my question was different , the agent self correction loop is part of Hermes agents?
1
u/returnity 5h ago
Yes the loop orchestration where the model executes a headless browser, takes a screenshot, the loads the file for the mmproj to read is a property of the harness. Hermes, opencode, pi, etc are all capable of this. The model just brings its visual understanding capability through the mmproj to the table, the harness provides the capability being discussed.
1
-5
u/acadia11x 21h ago
I have and never noticed this behavior … what are integrating it with? I don’t understand your post?
3
u/cogitech2 21h ago
Re-read the original post. It is extremely clear what the benefit is and how it works.
-3
u/TA-420-engineering 19h ago
Ninfer with nvfp4. Thank me later.
2
u/Iory1998 19h ago
What is Ninfer? I've been seeing this a lot. nvfp4 is lower quality than Q8_K_XL right?
1
u/Tormeister 2h ago
A custom engine for Qwen3.6 & 3.8 on the RTX5090.
Yes, nvfp4 is incomparably worse than Q8 ggufs.
-5
u/Southern_Sun_2106 21h ago
Can we please stop with the "Don't sleep on X" please? This expression has been beaten to death and is annoying af at this point tbh.
Yes, good point about Qwen. I think most people have noticed that by now with how popular the model is.
55
u/wayward_buffalo 18h ago
Pro-tip if you're tight on VRAM
I know I am, need every bit for context with how long the thinking traces are with Qwen3.8 27B)
--no-mmproj-offloadto keep your vision layers on CPU/RAM.Yes, your image processing is slow, but I don't mind waiting a little longer for the occasional screenshot verification if it gets me a bunch more ctx or fits a larger quant in VRAM.