Built to Test What It Can Generate
The benchmark deliberately covers a wider language surface than any single scanner, so a tool is graded beyond its home turf. Because the graded artifacts carry no labels and the answer key is held separately, it still cannot be tuned to the test.
There is a quiet failure mode in tool-plus-benchmark pairs: the benchmark drifts to cover exactly what the tool already does well, or the tool gets tuned to the exact cases the benchmark ships. Either way the score climbs and means less every release. BenchProctor and TheAuditor share a founder, so that risk is real here, and worth addressing head on rather than hand-waving past.
The answer is in how the benchmark is built: broader than any one tool, and impossible to quietly tune toward.
A surface wider than any one tool
A benchmark that tested only what a given scanner already handles would be easy to look good on, and close to worthless. BenchProctor is built the other way. The corpus emits eleven code languages, C++ among them, and it grades every tool across that whole surface. Not every scanner analyzes all eleven, and the benchmark does not shrink to match any one of them. Covering more than any single tool is the design, so a score reflects real reach rather than a surface curated to flatter.
Why you still cannot tune to it
Covering the ground a scanner runs on would be dangerous if it let the scanner learn the answers. It cannot, and the reason is construction, not a promise:
- The graded artifacts carry no labels. No comments, no CWE tags, no names that reveal whether a value is clean or tainted. There is nothing on the page to train on.
- The answer key lives separately, as an open, machine-verifiable file the scanner never reads during a run.
- The corpus rotates per release. A tool fit to one release’s exact cases learns those cases, not the analysis, and the next release draws fresh ones behind the same published scoring contract.
Put together, a scanner cannot be quietly shaped to the test, even by the people who build both, because the thing you would shape it to is not in the artifacts it sees. The guarantee comes from how the corpus is constructed, not from one organization watching another.
A capability, not a rule
A benchmark proves a capability. It never defines the rule. BenchProctor’s job is to measure whether a scanner can follow taint across a real, mixed codebase, and to report the result honestly whether the answer flatters anyone or not. TheAuditor’s job is to do that analysis on the code you actually ship. Construction, not coincidence, is what keeps the first an honest measure of the second. More on the wider toolset at the Code Reality Labs blog.