How to Actually Measure a SAST Tool's False-Positive Rate

Ask a vendor for their SAST false-positive rate and you will get a number. Ask what it was measured against and the conversation usually ends. Most quoted SAST accuracy figures have no ground truth underneath them, which means they are unfalsifiable. There is no corpus with known labels, so there is no way to check whether “3 percent false positives” is true, generous, or invented. A number you cannot check is not a measurement. It is a claim.

Measuring a false-positive rate honestly is not hard, but it has requirements, and most published numbers skip all of them.

You need a balanced corpus with known labels

A false-positive rate is the share of safe code a tool wrongly flags. To compute that at all, you need code you already know is safe. That is the piece almost every informal estimate is missing. Running a scanner over your own repo and eyeballing the results does not work, because you do not actually know which findings are wrong. You need a corpus where every file carries a verified label, vulnerable or safe, decided in advance.

It also has to be balanced. A defensible corpus is split exactly fifty-fifty between vulnerable and safe cases, so the false-positive rate is measured against as much known-safe code as vulnerable. The metric that then defeats the oldest trick in the book comes a moment later.

You need a confusion matrix

Once you have labels, sort every result into four buckets:

                detected   ignored
 vulnerable        TP         FN
 safe              FP         TN

A false positive is the FP bucket: a safe file the tool flagged anyway. From the two safe-code buckets you get the rate directly:

FPR = FP / (FP + TN)     of the safe code, how much got flagged?

That is the whole definition. No labels, no denominator, no rate. This is exactly why an informal scan of your codebase cannot produce a real FPR: you have no verified TN pile to divide by.

You need Youden’s J so “flag everything” scores zero

A false-positive rate alone is still gameable in the other direction. A tool that flags almost nothing has a beautiful FPR and misses every bug. So pair it with the true positive rate and subtract:

TPR = TP / (TP + FN)     of the real bugs, how many caught?
J   = TPR - FPR          Youden's J

Here is where subtracting the two rates pays off. A tool that flags every file catches all the real bugs, so TPR is 1.0, but it also flags all the safe code, so FPR is 1.0. Its J is 1.0 - 1.0 = 0. That stays true at any ratio of vulnerable to safe, because J is a difference of rates. The flag-everything strategy scores a flat zero instead of a fake win. J rewards detection and penalizes noise in equal measure, which is precisely what an accuracy number should do. The full scoring model is laid out in how BenchProctor scores a SAST tool.

Where the ground truth comes from

All of this depends on having a corpus with trustworthy labels, and that is the gap BenchProctor fills. Release 2026.07.22 is a large, balanced body of test code with a separate answer-key CSV: 2,938,418 labeled cases across 11 languages, 21 framework targets, 242 emitted categories, and 231 distinct CWE IDs. BenchProctor states that the release passes its gate suite before publication.

Point compatible SARIF 2.1.0 output at the corpus, hand the results and the CSV to a zero-dependency scorer, and you get a TPR, an FPR, and a J you can recompute from the public inputs.

Why it matters

An unfalsifiable accuracy number protects the tool, not you. The moment there is a balanced corpus with verified labels behind the measurement, “trust us” turns into “check it.” That is the only version of a false-positive rate worth quoting.

BenchProctor is open under Apache 2.0. Score compatible SARIF output and inspect the public scorer, answer key, and checksums yourself.