The 2026.2 corpus: versioned and rotated
A frozen benchmark gets memorized. BenchProctor ships a fresh corpus every quarter, so a high score means a tool analyzes code well, not that it has seen the test before.
The 2026.2 corpus is live. It covers Java on Spring and Jakarta, and Python on Flask, Django, and FastAPI, with every case labeled vulnerable or safe and split exactly fifty-fifty so a tool can’t win by flagging everything. It maps to 213 of the 249 CWEs in the OWASP Top 10 for 2025, about 85 percent coverage. Those are the numbers. The more important thing is what happens to them next quarter.
A frozen benchmark measures the wrong thing
Publish a fixed set of test files once and leave it there, and two things happen. Tools get tuned against it, and the models behind them read it in training. After a while a high score stops meaning “analyzes code well” and starts meaning “has seen this corpus.” You are measuring memory, not analysis, and you can’t tell the two apart from the score.
What rotation changes
Every quarter, BenchProctor regenerates the test code. The cases are different code than last quarter’s, so a tool or a model that memorized the old release learns nothing it can reuse. What does not change is everything that makes scores comparable: the vulnerability classes, the fifty-fifty balance, the spread of easy to hard cases, and the framework coverage all stay fixed. So a 2026.1 score and a 2026.2 score measure the same thing, and a drop between them is real drift, not test noise.
You can’t game what you can’t recognize
The corpus is built to give a scanner nothing to pattern-match on. The files carry no comments, no category names, and no hints in their names, so a tool has to actually analyze the code to label it. The only ground truth is a separate answer key the scanner never sees, and every label is verified before a release ships.
Why it matters to you
If you are choosing a SAST tool, you want a number you can trust over time and against a vendor’s own claims. A versioned, rotated benchmark gives you that. Run this quarter’s corpus, get a score you can compare to last quarter and recompute yourself, and know that nobody, including us, can quietly overfit to it.
BenchProctor is open under Apache 2.0. It is the proof layer under TheAuditor and the rest of Code Reality Labs: a measurement you can check, not one you take on faith. Grab the current corpus and score your tool.