[R] When the answer is a relation between documents, retrieval isn't the bottleneck: 0/38 with full evidence, 28/38 with the same facts as structure

[u/CupGlass540]

Most RAG evaluation asks whether the right passages reached the model. I wanted

to measure what happens when they do and the model still can't answer — because

the answer is a relation *between* passages rather than a statement inside any

of them.

Setup: a five-document narrative corpus (260,204 words, 13,950 passages) and 38

questions asking whether event A precedes event B, where A and B are narrated in

different documents and share no character, place or causal link. No passage in

the corpus states either relation. Five models, one family (Qwen3, 0.6B to 14B).

Given the source passages as text, every model scored 0/38 and refused 92-100%

of the time. I think the refusal is correct — the ordering genuinely is not in

the text. Given the identical facts as a structured chronology block from an

explicit state store, an 8B model scored 28/38 (73.7%).

A four-condition ablation separates information from form. At 14B, form is

irrelevant: plain prose, sorted prose and a structured block all land at 73.7%.

At 8B, structure leads the best prose condition by 6 items (73.7% vs 57.9%).

So: an 8B model given structure matches a 14B model given prose.

Two controls I'd want to see if someone else posted this:

- Permuting the supplied story positions collapses accuracy to 10.5% (8B) and

21.1% (14B). The models follow the ordering they're given rather than

recalling the published text.

- A realistic retrieval baseline is also at the floor, and it fails by asserting

rather than refusing. Going from 4 passages to 32 drove refusal from 97% down

to 50% while accuracy stayed at chance. More context produced more confident

wrong answers.

Two things I got wrong, both found by auditing my own scorer and question

generator after v1 was already published:

1. v1 reported the 8B form effect as +32 points. A scorer defect wasunder-crediting the prose conditions. Corrected, the gap is 6 items, not 12 —roughly half what I claimed. Re-scoring 1,786 saved items produced 30 gainsand zero losses, so nothing published was inflated; two things wereunderstated, and correcting them shrank my own headline.
2. For 36 of the 38 questions, the gold answers derive from author-assignedstory positions rather than from evidence-backed relations, and thegenerator's own self-check recomputes the gold from the same rows. That checkis circular. So this benchmark measures agreement with an author-assignedordering — not whether a system reports what the evidence establishes.

That second one is the real limitation and it bounds what the paper can claim.

I've left v1 up rather than retracting it, with the corrections in §11.

Full write-up, including the two things the audit changed:

https://ai.bedvibe.studio/structure-not-scale/

Paper, data and code: https://doi.org/10.5281/zenodo.22169643

Happy to be told the 0/38 is a prompt artifact — I tried to kill it and couldn't,

but I'd rather find out from you than not find out.

[u/donk8r]

Your permutation control is doing more work than the headline and I would move it up. If permuting the supplied positions drops the 8B to 10.5%, then what the structured condition demonstrates is transcription of a supplied ordering. That lines up with your own limitation 2 and sharpens it: the state store already contains the answer, so the open question is who computes the chronology.

Which is why this result travels furthest in domains where the relation is derivable rather than authored. Disclosure, we build one of those (octocode, github.com/muvon/octocode), a code index where the parser extracts calls and imports at index time, so the graph gets built without anyone assigning a gold ordering. Your circularity problem does not arise there, and what survives is the part your data actually establishes, that an 8B with the relation materialized matches a 14B without it.

The 32-passage number is the one to put in front of people who think more context fixes this. Refusal dropping from 97% to 50% while accuracy stays at chance is the argument against retrieving more, and you got it as a control rather than as the claim.

[u/CupGlass540 → u/donk8r]

Conceded on transcription, and you've put it better than I did. The permutation

control does show the model reading off a supplied ordering rather than deriving

one — 10.5% at 8B when the positions are scrambled is not a model that worked

anything out.

Two things I'd say survive that framing, and I think they're smaller than my

title implied.

First, at 14B the form genuinely doesn't matter — prose, sorted prose and the

structured block all land at 73.7%. At 8B they don't. So even granting that the

content is transcription, transcribing from prose and transcribing from a block

are not the same task below some size. That's a claim about form, not about

reasoning, and I should have written the title that way.

Second, the 0/38 floor is untouched by it. Whoever computes the chronology, the

text doesn't contain it, and every model correctly said so.

Your derivable-versus-authored distinction is the part I'll actually use. My

circularity is exactly that a person assigned story_order and the generator

verifies the gold against the same rows. In a code index the relation falls out

of the parse, so the gold has an origin independent of whoever wrote the

questions. That's the corpus this experiment should have run on, and it's a

better answer to my limitation 2 than anything I had.

Agreed on the 32-passage number too. It was a control and it's the result I'd

lead with if I rewrote this — refusal falling 97% to 50% while accuracy stays at

chance is the concrete argument against "just retrieve more."

[u/donk8r → u/CupGlass540]

Both of those survive, and the first one is a better claim than the one you led with. Form mattering at 8B and not at 14B locates a capability floor, and that is testable in a way "structure beats scale" is not.

One warning on running it over a code index, so you do not import limitation 2 in a new costume. The parse gives you static relations for free, imports and call edges, and those are genuinely author-independent. Execution order is not one of them. "Does A run before B" is dynamic, and the moment you need it you are back to someone asserting a ground truth, either by writing it down or by instrumenting a run and treating the trace as gold. A trace is a defensible origin, it is just a different one from the parse, and mixing the two quietly is how the circularity comes back wearing a different hat.

So the version I would run is questions whose answer is a path in the static graph, with the gold computed by traversal rather than by anyone's hand. Your 0/38 floor then has an exact analogue that is easy to state: no single file says that changing this function breaks that test.

[u/CupGlass540 → u/donk8r]

That warning is the one I'd have walked into. Imports and call edges are

author-independent; execution order isn't, and I'd have reached for "does A run

before B" precisely because it's the closest analogue to the ordering questions

I already had. Instrumenting a run and treating the trace as gold is a

defensible origin, but you're right that it's a different one — the parse

derives a relation, the trace measures one instance of it. Pooling them would be

limitation 2 with better manners.

Traversal-computed gold is the part that actually fixes it. Nobody writes the

answer down at any point: the graph comes from the parse, the gold comes from an

algorithm over the graph, and the question generator can't smuggle in a

convention because it never sees one.

Two pieces of my design transfer directly and I'd keep both.

The four-condition ablation maps cleanly — raw file contents, a rendered call

graph in prose, and a structured edge list are the same information in three

forms, so the 8B-versus-14B form gap is testable there without changing the

method.

And the permutation control matters more, not less. Scramble the edges and

accuracy should collapse the way it did at 10.5%. If it doesn't, the model is

recalling the repository rather than reading the graph — which is a live risk on

public code in a way it wasn't on my books.

Your framing of the floor is the sentence I'd build it around. "No single file

says that changing this function breaks that test" is both the exact analogue of

my 0/38 and a question people actually need answered, which mine wasn't.

[u/donk8r → u/CupGlass540]

On the memorization risk, your permutation control is necessary and it is weak against precisely that. A model that has the repo memorised can still answer scrambled-edge questions from memory, and what you observe is some degradation you then have to argue about.

The control that hands you a number instead is renaming. Alpha-rename every symbol and path deterministically before you render any of the three conditions, so process_payment becomes fn_a17 and the graph shape is untouched. Memorisation keys on identifiers. If accuracy holds under renaming, the model is reading the graph you gave it. If it drops, the size of that drop is your contamination estimate rather than a worry you mention in a limitations section.

Report both conditions rather than swapping one for the other. Real identifiers carry genuine signal that a model should be using on real code, so the renamed run is a probe and the named run is the honest task. The gap between them is the number, and I have not seen anyone publish it.

[u/CupGlass540 → u/donk8r]

Renaming is the better control and I'll take it. It also converts the thing I

was going to write in a limitations section into a number, which is the trade I

always want.

One implementation trap I'd flag, because it would have bitten me: identifiers

leak out of the symbol table. Docstrings, comments, string literals, log

messages, exception text, test names and file paths all carry the original

names, and a rename that only touches the AST's symbols leaves most of the

memorisation surface intact. The condition that renders raw file contents is the

worst affected, since that's where the prose lives. So the rename has to be

applied to the rendered text of all three conditions, deterministically, with

the mapping stored so the whole thing replays.

The other half is that I'd want to predict the gap before running it. If I look

at the named-versus-renamed difference first and then decide what counts as

contamination, I've done the thing I just spent a week correcting in my own

scorer. So: threshold written down, both runs reported whichever way it lands,

and if renaming makes accuracy go *up* I have to publish that too and think

about why.

One thing I'd hold back on is the novelty framing. Semantics-preserving

identifier transformations do show up in the code-model evaluation literature,

and I don't yet know whether the named/renamed gap has been reported as a

contamination estimate specifically. I'd want to actually search before claiming

it hasn't — I've had to cut exactly that kind of search-derived priority claim

from my own README once already.

[u/donk8r → u/CupGlass540]

That's the trap, yeah. Renaming AST symbols and leaving docstrings, log messages and test names alone leaves most of the memorisation surface sitting right there, and the raw-file condition is where that prose lives.

Pre-registering the predicted gap is the right call too. Once you've seen the number you'll find a reason it was the number you expected.

[u/CupGlass540 → u/donk8r]

If you want it run against octocode's index, say the word — the parse and the traversal are local so it costs me compute and nothing else, and you'd get the named/renamed gap on your own graph either way it lands.

[u/donk8r → u/CupGlass540]

Yes please. I'd rather see the number than keep guessing at it.

One ask: publish it whichever direction it goes. A contamination gap on our own graph is worth knowing about even if it's embarrassing, and a result that only gets shared when it's flattering isn't worth much.

[u/CupGlass540 → u/donk8r]

Agreed, and that's the only condition worth having. It goes up either way, with the pre-registered threshold written down before the run so I can't rationalise the number afterwards.

Two things I need from you to set it up.

Which graph — octocode's own repo, or a codebase you'd rather see it on? And

should I build the graph with octocode itself, or from a separate parse? Using your indexer makes it a result about your pipeline, which is more useful to you but means the graph and the tool under test share an origin. I'd lean on octocode building it and say so plainly in the writeup, but it's your call.

I'll send you the pre-registration before I run anything, so the predicted contamination gap is on the record first.

[u/LowDistribution3995]

Use this benchmark please and post the scorecard. It actually shows meaningfull data and having more people use it helps me tweak the metrics being scored so it stays useful for testing as opposed an easily fudgable meaningless % for marketing purposes. https://github.com/munch2u-a11y/FP-AMB.git

[u/CupGlass540 → u/LowDistribution3995]

I'll look. One thing I'd want before running it and posting a number, and I ask

it of my own work too: what are the discrimination controls?

I spent this week finding out that a metric I built returns a clean, confident

zero in three unrelated situations — the two systems genuinely agree, the task

admits only one answer so nothing can vary, or the calls died at transport and

came back as absence. All three looked identical in the output. A scorecard

from that would have read as a pass.

So the questions I'd ask any benchmark, including mine: does a system paired

with itself score high? Does a deliberately broken one score low? And how many

items can any competent system answer, so the score is mostly carried by the

few that discriminate?

If those exist and I missed them, point me at them. If they don't, they're

worth more to you than another scorecard.

[u/LowDistribution3995 → u/CupGlass540]

Hard to read this .... But if your asking if I tried to make it fair, yes. I made it as fair as I could. I don't know what system paired with itself means so... I dunno. It gives a helpful visual scorecard though that breaks down scores into categories and includes token metrics.

[u/CupGlass540 → u/LowDistribution3995]

Fair — let me say it plainly.

Self-pairing means running your benchmark with the same system on both sides. Identical input, identical output. It should score a perfect agreement. If it doesn't, the number is partly noise.

Then the opposite: feed it something you've deliberately broken. If that still scores well, the metric isn't separating good from bad.

I ask because mine failed exactly that test last week. It returned a clean, confident zero in three completely different situations — the two systems actually agreed, the task only had one possible answer, and the calls died before they arrived and came back as nothing. All three looked identical in the output. A scorecard built on it would have read as a pass.

Those two checks are an afternoon each and they'd tell you more about FP-AMB than any scorecard I could post. If it already survives them, say so in the README — that's the thing that would make me trust the number.

[u/LowDistribution3995 → u/CupGlass540]

No.... It's an actual Q&A so the scenario your describing doesn't make sense. If the tested system pulls the correct context to match the answer key it gets points, if it doesn't, it doesn't. If you have any test or benchmark that basically gives you a random printout then that's not a benchmark. I'm not going to add to the readme because what your describing isn't a real red team test, it's like a common sense test. 

If I'm showing you a car and your response is I won't look at it until you've proven its a car.... I don't even know how to approach that. Like look at, its got 4 wheels and an engine. I'm not sure publicly advertising that the tires are round means anything and I'm not sure why that would ever be a serious inquiry of anyone looking to buy a car.

Sorry if I'm coming off a bit short here, but it really feels like your just dumping pseudo sounding slop. Everything your asking, I think, would take under 1 second for any agent to confirm or deny with a link to the repo....

[u/Future_AGI]

Your jump from 0 to 28 is the useful signal here, because it points at the composition step, holding a relation across passages, rather than at retrieval itself. We started scoring that reasoning step separately from retrieval, since a run can pull every right passage and still miss the relation between them. Once the composition score is its own number, you can see which of the remaining 10 are retrieval misses versus relation misses. Relational eval setup is here if it helps: https://github.com/future-agi/future-agi