Reading a SAST scorecard
A single SAST score can hide as much as it shows. Here is how to read recall, false positives, and the traps that make a good tool look bad.
A vendor tells you their SAST tool scores 92 percent. On what corpus, balanced how, against whose answer key, and 92 percent of what? A benchmark number is only useful if you know how to read it. Here is what BenchProctor reports and what each part actually tells you.
Two numbers, not one
Every tool gets scored on two axes. The true positive rate, or recall, is the share of real vulnerabilities it caught. The false positive rate is the share of safe code it wrongly flagged. You need both, because either one alone is easy to fake. A tool that flags every file has perfect recall and is useless. A tool that flags almost nothing has almost no false positives and misses the bugs.
BenchProctor reports both and combines them into a single Youden’s J score, which is just the true positive rate minus the false positive rate. A perfect tool scores 100. A tool that flags everything scores zero, because its recall and its false-positive rate are both total. On a corpus balanced exactly fifty-fifty between vulnerable and safe code, “flag everything” gets you nowhere, by design.
Averaged how
A score can be averaged two ways, and they answer different questions. Average across categories, weighting each vulnerability class equally, and you learn whether a tool is broadly competent. Aggregate every case into one pile and you learn how it does on the mix as it actually appears. BenchProctor gives you both, so a tool that is excellent on the two common classes and blind on the rest can’t hide behind one flattering average.
The trap: a wrong answer key
Here is the failure that quietly ruins benchmarks. If the ground truth is wrong, if a case is mislabeled or the vulnerable line is recorded in the wrong place, then a correct tool gets marked wrong and a broken one gets rewarded. The score inverts. That is why BenchProctor verifies every label before a release ships and keeps the answer key separate from the code the scanner sees. A benchmark you have to take on faith is not a benchmark.
What a high score does not promise
BenchProctor’s current corpus is single-file: each case stands on its own. That is the right place to start, and it already separates real analyzers from pattern matchers. But real vulnerabilities cross files, services, and languages, and those cases are on the roadmap. A great single-file score is necessary, not sufficient. Read it as “this tool can analyze code,” not “this tool will catch everything in your monorepo.”
Why it matters
The point of a scorecard is to let you compare tools honestly, including against their own marketing. Know what recall and false positives cost you, insist on a verified answer key, and treat one number with suspicion until you have seen the two behind it.
BenchProctor is open under Apache 2.0, the proof layer for Code Reality Labs. Score any tool that emits SARIF and check our numbers yourself.