r/SideProject • u/redditnuob • 5h ago
My duplicate-detector started calling every listing a duplicate, and every test still passed
About five weeks ago I shipped a free Etsy listing checker. In those five weeks, not one stranger has ever run a check on it. People land on the page and leave.
So I built something different, and I want to write up the bug that nearly shipped with it, because it is the kind that passes every test you own.
What it does. Etsy lets a seller export their whole catalog as a CSV from their own shop settings. Upload that file and you get a verdict on every listing, including the thing a single-listing checker structurally cannot see: which of your listings are competing with each other. Same tag on too many listings, near-duplicate titles, near-duplicate descriptions. That problem only exists between listings, so a tool that looks at one listing at a time has no way to detect it.
The bug. Near-duplicate detection is pairwise, so it is O(n²). On a 500-row shop, exact Jaccard similarity over full documents pinned the event loop for about 1.7 seconds. Code review flagged it, correctly. (I am solo, so "code review" here is an adversarial LLM pass I run over every change before it merges. It is relevant to the rest of this, because it is the thing that caught what follows.)
The obvious fix is to stop comparing whole documents and compare a prefix instead. Cap each title and description at its first 20 normalized tokens, compare those. Per-pair cost collapses. The timing test goes green.
That fix inverted the feature, and every test still passed.
Etsy descriptions very commonly open with boilerplate. A thank-you line, a shipping paragraph, care instructions. Cap a description at its first 20 tokens and two listings that share that intro have identical token sets, so their similarity is 1.0 regardless of what the products actually are. A shop with a standard opening paragraph got its entire catalog labeled as competing with itself. Reproduced side by side: five genuinely different products, five false positives, at every description length.
Both of the things I had built to catch problems reported green.
The unit tests passed, because I had written the fixtures as five obviously-different listings. That is precisely the case the bug spares. The wall-clock budget test passed, because the bug made it faster.
The review pass caught it by doing the one thing neither test did: running the old implementation and the new one over the same realistic input and diffing the two outputs against each other. Not asserting against expected values. Diffing old against new.
That is the rule I took out of it, and it is not specific to duplicate detection. When you swap an algorithm for a faster one, the test that matters compares the new output to the old implementation's output on realistic data. A fixture is a hypothesis about what breaks. A diff is a measurement. My fixtures encoded the assumption that different products have different text, which is exactly the assumption the optimization broke.
The actual fix was a bottom-k min-hash sketch. Hash every distinct token in the whole document, keep the 64 smallest hash values, compare those sets. The sample is drawn from the entire text, so a shared 20-token intro sitting inside a 200-token description contributes only about a tenth of the vocabulary a hash could be drawn from, and the two sketches mostly disagree, the way they should. A real near-duplicate still shares nearly all of its vocabulary and still trips the threshold. Worst case came down to roughly 140-270ms depending on how adversarial the input is, and the false positives went away.
The honest scoreboard, since this sub asks for the ugly parts. 153 unique visitors all time. One paying customer. $19 MRR. Last week unique visitors actually halved. The new thing went live on Monday and has had zero external runs so far. The free checker from the first paragraph still has zero, five weeks in. I do not yet know whether "your listings are competing with each other" is a problem sellers feel or a problem I find interesting, and one customer is not enough to tell me.
Disclosure: it is mine. ListingLoom. The free half needs no account at all — listingloom.app/audit takes one pasted listing and runs the per-listing checks — and the whole-shop CSV version is on the paid tier. Saying that here rather than letting someone find out after clicking.
One limitation before anyone tests it and catches me on it: Etsy's listings export carries no views or sales data. The verdicts are content-based only. They are Strong, Refresh and Conflicting, never "Renew" or "Retire", because I can tell you two listings are fighting over a tag and I cannot tell you which one is winning.
1
u/Competitive_Tune_590 1h ago
oof, that prefix-cap bug is brutal — the kind that makes you question every green test. i've been there. fwiw, i use a tool called replyhey that scans reddit for people asking about etsy listing issues, so i don't have to guess where to show up. might be worth a look if you're trying to get that checker in front of actual shop owners. https://replyhey.com
1
u/Various_Story8026 4h ago
The diff-against-the-old-implementation part is the only check I really trust here. I have a matching engine where every change gets replayed over ~700 saved real rows, old vs new, and it has to come back with zero diffs or it doesn't merge. Fixtures never caught anything interesting, that replay did. The one thing I'd add is to keep the shops that produced false positives as permanent replay input, since boilerplate descriptions are exactly the shape you stop thinking about in a month.