The Benchmark Caught Its Own Mistake
A benchmark is only as good as its answer key, so BenchProctor audits its own key before publishing. In one pre-release audit it found a batch of artifacts that would have punished a correct tool, and fixed them before anyone was scored.
Every SAST benchmark is, underneath, an answer key: a pile of artifacts, each labeled vulnerable or safe, and a score that rewards a tool for agreeing. That means a wrong label does not just cost a data point. It inverts the result. A tool that correctly reports a live bug in a file the key calls safe is scored as if it raised a false alarm. The tool that got it right loses, and the weaker tool that missed the bug wins. A benchmark that punishes accuracy is worse than no benchmark, because it points a buyer at the wrong tool.
So before BenchProctor scores anyone, it checks its own key against what the code actually does. Here is a case where that check earned its keep.
What the audit found
In a pre-release audit of a set of compound artifacts, the applications where several weaknesses combine into an escalation, the numbers did not line up. Of 256 compound artifacts, 96 had shipped a working authorization control in the code while the answer-key row for that artifact asserted the control was missing.
| Compound artifacts audited | Rows that disagreed with the code |
|---|---|
| 256 | 96 |
Read what that would have done to a scorecard. A tool that correctly saw the authorization control was present and stayed quiet would have been marked wrong on 96 artifacts, because the key insisted the control was not there. The most accurate scanners would have taken the biggest penalty. Exactly backwards.
Why it was caught, not shipped
The fix is not clever. It is discipline: the key is not trusted because it was produced, it is checked against the real behavior of the code it describes before a release goes out. In this case the disagreement was found, the rows were corrected to match the working control, and the corrected artifacts went into the release. No tool was ever scored against the bad rows.
This is a self-audit, not an outside seal of approval. BenchProctor and the answer key share an origin, so the honesty here comes from the process being run and its results being stated plainly, not from someone else vouching for it. The public release still ships the answer key and the scorer, so you can recompute any score yourself and challenge a specific row if you think it is wrong.
The general rule
An answer key you trust blindly is a liability. An answer key you check against the code’s real behavior is an asset. The concrete story here is one instance of the general rule laid out in a wrong answer key punishes the tools that get it right. It is the same yardstick every scanner the stack relies on answers to, TheAuditor included.