r/coolgithubprojects • u/Queasy_Beautiful_31 • 19h ago

Local AI PR-review CLI that caught a real concurrency bug — and also confidently hallucinated a deadlock (here's how I caught that too)

https://github.com/Spacexcx/pr-check

Local CLI that reads your git diff (or a GitHub PR URL), sends it to Gemini, flags files worth a second look before you push — with a required exact quote from the diff as evidence for every flag.

Tested it against a small batch of real merged Godot PRs. Caught a real concurrency bug, correctly stayed quiet on clean PRs — and also produced one confident, well-evidenced, completely wrong flag (claimed a deadlock, but the mutex type was actually recursive so it wasn't one). Added a second pass that now catches exactly that kind of error by asking the model to name its own unverified assumptions.

Repo + README: github.com/Spacexcx/pr-check

If you don't want to bother with the Gemini API key setup — paste a link to one of your own merged/open PRs here and I'll run it and post the output. Genuinely curious how it holds up outside my own testing.

2 Upvotes

67% Upvoted

2 comments sorted by

1

u/Obvious-Comedian-495 11h ago

I have been working on a similar problem for past 2 months, and the approach is very similar to the initial versions of our current solution. Hope it helps.

To consider for next versions: 0. Accuracy ( cos no one would believe the dev) 1. How to handle big PR (>40 files) ? 2. Addition of new files. ( codebase structure, architecture rules, coding guidelines, solution principles) 3. Correlation between blast radius and changes. 4. Regression on existing flow.

1

u/Queasy_Beautiful_31 6h ago

Really appreciate this — especially point 0, since that's exactly what I've been trying to earn rather than claim (published a specificity test including an honest false positive + how I caught it, link's in the post).

#1 is the one I think is most actionable next — right now it sends full file content for every changed file, which will choke on a 40+ file PR. 2-4 are things I want eventually but they need real codebase-structure awareness (AST/call graph), which is a much bigger scope than where this is today.

Curious what you ended up building — sounds like you're further along on this than I am.