Hey everyone,
I wanted to share a strange crossover project I’ve been working on. I've been messing around with music production and modular synths for over 25 years. Recently, I've also been doing professional AI data-curation work and computer science coursework.
A couple nights ago, I completed and launched my very first public GitHub repository: EvidenceBench.
The Project:
I wanted to see if frontier multimodal AI models understand complex spatial routing with regard to modular patches and signal types, or if they just confidently guess what's on screen using training data and best guesses based on potential signal type where a connection might be headed.
To test this, I built a benchmark using a fairly dense modular synthesis patch in Cardinal (VCV Rack).
Why modular synth patches?
Because they are the ultimate visual logic test. A single cable crossing, an unusual routing path, or a color-coded schema forces the model to actually trace the cables. It can't just cheat by "reading" a text layer or by having specific modules included in training data.
How the test works:
I put the models through a three-turn gauntlet:
- Turn 1: They get a set of screenshots and have to document every visually identifiable connection.
- Turn 2: I hand them a potentially incorrect JSON file built by my “new engineer" containing every connection and match it with newly labeled screenshots. The models are then asked to audit it against the image to catch any incorrect or missing cables.
- Turn 3: I hand them a massive 4x resolution version image of the patch and instruct them to self-correct their previous answers based on the clearer evidence.
Early Results:
I benchmarked Qwen 3.8 Max and Gemini 3.1 Pro. Both of them struggled hard with listing actual connections in turn 1, and overcorrected in turn 2. Turn 3 was straight up baffling, but it’s pretty clear that both of them hit their absolute limits trying to trace the paths.
Why I'm sharing it here:
The entire framework is portable built using open-source, written using native Python packages, and includes the .VCV patch file so anyone can load it up in Cardinal.
I’m a bit nervous because it's my first ever public repo, but I wanted to keep it completely transparent so I’ve included my early failed attempt and prompt mistakes in the repository to show how the methodology evolved.
If you want to check out the patch, the python validation scripts, or the full model evaluations, you can find it here:
https://github.com/apparent-symmetry/EvidenceBench
I'd love to know what you think—especially if you have ideas for absolute nightmare patch configurations that would completely break an AI's brain!