Reading a SAST scorecard
A single SAST score can hide as much as it shows. Here is how to read recall, false positives, and the traps that make a good tool look bad.
A vendor tells you their SAST tool scores 92 percent. On what corpus, balanced how, against whose answer key, and 92 percent of what? A benchmark number is only useful if you know how to read it. Here is what BenchProctor reports and what each part actually tells you.
Two numbers, not one
Every tool gets scored on two axes. The true positive rate, or recall, is the share of real vulnerabilities it caught. The false positive rate is the share of safe code it wrongly flagged. You need both, because either one alone is easy to fake. A tool that flags every file has perfect recall and is useless. A tool that flags almost nothing has almost no false positives and misses the bugs.
BenchProctor reports both and combines them into a single Youden’s J score, which is just the true positive rate minus the false positive rate. A perfect tool scores 100. A tool that flags everything scores zero, because its recall and its false-positive rate are both total and the score is their difference. That holds at any split; the fifty-fifty balance is there so the false-positive rate is measured against a full half of known-safe code.
Averaged how
A score can be averaged two ways, and they answer different questions. Average across categories, weighting each vulnerability class equally, and you learn whether a tool is broadly competent. Aggregate every case into one pile and you learn how it does on the mix as it actually appears. BenchProctor gives you both, so a tool that is excellent on the two common classes and blind on the rest can’t hide behind one flattering average.
The trap: a wrong answer key
Here is the failure that quietly ruins benchmarks. If the ground truth is wrong, if a case is mislabeled or the vulnerable line is recorded in the wrong place, then a correct tool gets marked wrong and a broken one gets rewarded. The score inverts. BenchProctor keeps the answer key separate from the code the scanner sees and states that releases pass its label gates before publication. The public artifact includes the answer key, scorer, manifest, and checksums, while per-file proof metadata remains private.
What a high score does not promise
BenchProctor’s current corpus is single-file: each case stands on its own. That is the right place to start, and it already separates real analyzers from pattern matchers. But real vulnerabilities cross files, services, and languages, and those cases are on the roadmap. A great single-file score is necessary, not sufficient. Read it as “this tool can analyze code,” not “this tool will catch everything in your monorepo.”
Why it matters
The point of a scorecard is to let you compare tools honestly, including against their own marketing. Know what recall and false positives cost you, insist on a verified answer key, and treat one number with suspicion until you have seen the two behind it.
BenchProctor is open under Apache 2.0. Score compatible SARIF output and inspect the public scorer and answer key yourself.