An Honest Score With Public Inputs
BenchProctor provides a versioned corpus, public answer key, and scorer for interpreting compatible SARIF results.
Ask any SAST vendor how accurate their scanner is and you get a number. Ask where the number came from and the story falls apart. It was measured on a frozen public suite the tool has long since memorized, or on the vendor’s own private cases, graded by the vendor. Nobody shows you an honest score against a corpus the tool could not have trained on, because almost nobody has one. That is the quiet bottleneck under AI-at-scale: unprovable correctness. Claims are cheap. Proof is what is missing.
BenchProctor is a public benchmark corpus designed to reduce simple label leakage and make its scoring inputs inspectable.
Designed to reduce label leakage
The trick with any benchmark is that a tool can look good without being good. A frozen corpus rewards memory. A leaky one rewards reading the answer off the page. BenchProctor is designed to reduce both risks.
New versions can rotate the specific cases while preserving the published scoring contract. The published files carry no comments, no CWE numbers, and no category names, so there is nothing to pattern-match against except the code itself. And it seeds in decoys, code that looks alarming but is perfectly safe. A scanner that flags on appearance walks straight into them and loses points. You cannot ace this by recognizing shapes. You have to actually analyze.
Cross-language, so breadth is not optional
The cases span multiple languages and real frameworks, the kind of software your team actually ships. A tool that does well on one narrow slice and falls apart everywhere else cannot hide here. The score reflects whether the analyzer holds up across the surface area of real code, not whether it was tuned to a single familiar suite.
One honest metric
Point compatible SARIF output at the corpus, and the scorer reports its true-positive rate minus its false-positive rate. Catching real bugs pushes the number up. Crying wolf on safe code pulls it back down. The balance is what makes it honest. The laziest strategy in security tooling is to flag everything and call the noise thoroughness. On this corpus that strategy nets you roughly zero, because every true positive you catch by flagging everything comes bundled with a false positive that cancels it out.
That is the whole point. The number is not a leaderboard rank you take on faith. It is a measurement you can recompute yourself, against ground truth the tool never saw.
The public measurement layer
This is where BenchProctor stops being a standalone benchmark and becomes the receipt for an entire stack. Read how the pieces fit in the sovereign AI engineering stack. The stack ships its own SAST engine, TheAuditor, and a stack that makes strong claims about correctness needs something that can prove them without grading its own homework.
BenchProctor is that measurement surface. It can score TheAuditor or any other compatible tool against the published release. The resulting number is tied to a specific benchmark version and the provided SARIF output.
BenchProctor supplies the public corpus and scorer for that measurement. A buyer can inspect the release version, answer key, scorer, and their own SARIF output before interpreting a result.