GitHub published ReviewBench, an AI code review benchmark, on October 5, 2026. It draws from 219 real pull requests across 187 open source repositories. GitHub designed it to score how well an AI agent catches real bugs. It is not a benchmark for how often an agent leaves a comment. The benchmark matters because GitHub Copilot ranks first on it, while an independent benchmark from Martian ranks Copilot fourth. That gap is the real story for any team picking a review tool.
What Is ReviewBench, GitHub’s AI Code Review Benchmark
ReviewBench is an open, offline benchmark for AI code review agents. GitHub built it with Microsoft after analyzing 103.9 million pull requests to find realistic review patterns. The test set has 219 public pull requests in 19 languages. GitHub weighted it toward medium and large changes, not trivial one-line fixes.
Each pull request carries a set of golden findings. GitHub built that reference set from human reviewer comments, later author fixes, static analysis tools, and multiple frontier language models. Claude Sonnet 5 grades whether a candidate finding is true, relevant, and non-trivial. Separate engineers who had no role in building the dataset checked every finding by hand. Their judgments matched GitHub’s labels 96.6% of the time.
Why GitHub Built It Now
AI code review tools have multiplied, and teams now struggle to compare them. Vendors publish their own numbers, usually from their own test sets. That makes direct comparison unreliable. DevX has covered how generative AI is changing software development. Code review is one of the fastest-moving parts of that shift.
A shared, public benchmark lets any vendor submit results under the same rules. GitHub publishes the dataset, the grading rubric, and the matching logic so other teams can check the work. That transparency matters more than who ranks first.
How ReviewBench Scores a Pull Request
ReviewBench reports four headline metrics: grounded precision, grounded recall, augmented precision, and augmented recall. Grounded metrics only count matches against the known golden findings. Augmented metrics also credit real issues the golden set missed. The benchmark doesn’t penalize a tool for finding something new.
Grounded recall is the main metric for comparing different review agents. Augmented recall only works as a per-tool diagnostic. Each agent’s own findings change its denominator, so augmented scores do not compare cleanly across tools. GitHub’s ReviewBench methodology write-up explains the distinction. Users can also slice scores by severity and issue category, from security bugs to maintainability nits.
To submit a score, a vendor registers an agent on the review-bench.ai site. The agent needs a container image and its own model key. Vendors iterate against a 25-pull-request practice set. Then they run the full 219-pull-request set three times for a final score. A maintainer approves each new leaderboard entry.
Copilot’s Results and the Vendor-Run Controversy
GitHub Copilot’s code review tool, set to Balanced mode, tops the public ReviewBench leaderboard. Its grounded F1 score is 40.1%. GitHub says offline ReviewBench results have tracked the direction of its production A/B tests. An internal multi-model ensemble test showed an 8% gain in addressed rate and a 13.6% gain in recall.
The catch is that GitHub ran the initial comparison itself. Vendors did not run or verify their own entries. GitHub tested Copilot on October 1, months after testing rivals Cubic and Greptile in June. Products change within months, so the two scores may not reflect the same software version.
The Independent Benchmark Tells a Different Story
Martian is an AI research company with no coding product of its own. It launched a separate Code Review Bench in February 2026. As of October 6, 2026, Martian’s online leaderboard ranks Cubic first at 64.9% F1. Greptile and CodeRabbit follow, with Copilot fourth at 60.9%. Its offline leaderboard puts Qodo Deep and Cubic ahead of Copilot too.
Martian scores tools using its own dataset and method. Those numbers don’t line up one-to-one with ReviewBench’s. A vendor-run benchmark that favors its own product deserves scrutiny. An independent benchmark, run by a party with nothing to sell, carries more weight.
Choosing a Code Review Tool: What to Check First
Do not pick a code review tool from a single leaderboard. Check whether the benchmark publishes its full dataset and grading rubric, the way ReviewBench does. Check who ran the test and whether every vendor’s result came from the same test window.
Run a short pilot against your own repository before committing to any tool. ReviewBench’s documentation flags a real limit. A tool that scores well on 219 public repositories might behave differently on your private code. DevX has written about integrating AI tools into real development workflows. The same rule applies here: test on your own code, not a vendor’s demo.
Key Takeaways
- ReviewBench is GitHub’s open benchmark for AI code review agents, built from 219 real pull requests.
- GitHub Copilot ranks first on ReviewBench but fourth on Martian’s independent Code Review Bench.
- GitHub ran the initial ReviewBench comparisons itself, and vendor test dates didn’t align.
- Grounded recall is the primary metric for comparing tools head to head on ReviewBench.
- Pilot any code review tool on your own repository before trusting a public leaderboard.
Frequently Asked Questions About ReviewBench
What Is ReviewBench?
ReviewBench is an open benchmark GitHub published on October 5, 2026. It scores AI code review agents against 219 real pull requests from 187 open source repositories.
Is GitHub Copilot the Best AI Code Review Tool?
Copilot ranks first on ReviewBench, GitHub’s own benchmark. On Martian’s independent Code Review Bench, it ranks fourth online and fifth offline as of October 2026.
How Does ReviewBench Differ From Martian’s Code Review Bench?
ReviewBench uses GitHub’s own dataset and a grounded recall metric. Martian uses a different dataset and tracks both an offline score and an online developer response rate.
Can Other Vendors Submit Their Own Results to ReviewBench?
Yes. Vendors register an agent on the review-bench.ai site. They run it against the test set, and a maintainer approves the entry before it appears on the leaderboard.
What Is Grounded Recall in ReviewBench?
Grounded recall measures how many known golden findings an agent catches in a pull request. It only counts findings already in the benchmark’s golden set.
Does a High ReviewBench Score Guarantee Good Results on My Codebase?
No. ReviewBench’s documentation flags a real limit: scores on its public pull requests might not hold on your private code. A pilot test still matters before you adopt a tool.
Final Thoughts
ReviewBench is a real step toward comparing AI code review tools on shared, public ground. It is also a benchmark that its own creator’s product currently leads, tested on a timeline that favored that product. Read any single leaderboard as one data point, not a verdict. Before you adopt a tool, pilot it against your own pull requests. Check whether rivals got a fair, same-window test too.
Photo by Igor Omilaev: Unsplash
Johannah Lopez is a versatile professional who seamlessly navigates two worlds. By day, she excels as a SaaS freelance writer, crafting informative and persuasive content for tech companies. By night, she showcases her vibrant personality and customer service skills as a part-time bartender. Johannah's ability to blend her writing expertise with her social finesse makes her a well-rounded and engaging storyteller in any setting.
