> BenchProctor / blog
· scoring, methodology

How BenchProctor scores a SAST tool

The whole scoring model is a confusion matrix and one subtraction. Here's how true-positive and false-positive rates become a single number, why we average per category, and how the benchmark checks itself.

There is no machine learning in BenchProctor’s scoring, no learned judge, no opaque aggregate. It is a confusion matrix and one subtraction. That is deliberate: a benchmark you can’t audit is just another tool you have to trust.

Four buckets

Every test case has a known label, vulnerable or safe, in the CSV answer key. After your scanner runs, each case lands in one of four buckets:

                detected   ignored
 vulnerable        TP         FN
 safe              FP         TN
  • TP (true positive): vulnerable, and the tool flagged it. Good.
  • FN (false negative): vulnerable, and the tool missed it. A real bug shipped.
  • FP (false positive): safe, and the tool flagged it anyway. Noise that erodes trust.
  • TN (true negative): safe, and the tool stayed quiet. Good.

A case counts as detected when a SARIF finding points at the test file and, in the default mode, carries the expected CWE. The scorer accepts common CWE locations in SARIF rule IDs, properties, tags, and taxa. A filename-only mode exists for tools that emit no CWE, but it can reward over-flagging.

Two rates and one score

From the four buckets come two rates:

TPR = TP / (TP + FN)     detection rate    (of the real bugs, how many caught?)
FPR = FP / (FP + TN)     false-alarm rate  (of the safe code, how much flagged?)

The headline score is Youden’s J, the difference between them:

J = TPR - FPR

It runs from +1.0 to -1.0:

  • +1.0: every vulnerability caught, zero false alarms. Perfect.
  • 0.0: no better than a coin flip. A flag-everything tool lands here.
  • -1.0: inverted, so it flags the safe code and misses the real bugs.

A single number that rewards detection and penalizes noise in equal measure is exactly what you want from an accuracy metric. A tool that finds every bug but drowns you in false positives is not a good tool, and its J score says so.

Why flag-everything scores zero

Subtracting the two rates makes the laziest strategy collapse on its own. Suppose a tool reports every single file as vulnerable. It catches all the true positives (TPR = 1.0), and it flags all the safe files too (FPR = 1.0). Its score is 1.0 - 1.0 = 0.0. That holds at any ratio of vulnerable to safe, because J is a difference of rates measured within each class. The 50/50 balance earns its keep elsewhere: it gives the false-positive rate a large, equal pile of known-safe code to measure against.

Category-averaged is the headline

There are two honest ways to combine per-category results, and BenchProctor reports both:

  • Category-averaged (macro). Compute TPR and FPR for each category independently, then average across categories. Every vulnerability class counts equally, so a tool can’t earn a great score by nailing one enormous category and ignoring a dozen small ones. This is the number we lead with.
  • Flat aggregate. Pool every case together and compute one TPR and FPR. Useful for comparison, but it lets large categories dominate.

When the two diverge, that gap is itself a finding: it usually means a tool is strong on a few common classes and weak across the long tail.

$ python score_sarif.py results.sarif corpus/expectedresults-2026.07.22.csv

  category-averaged   TPR 0.962   FPR 0.044   J 0.918
  flat aggregate      TPR 0.961   FPR 0.046   J 0.915

  weakest categories
    xxe                J 0.71
    open_redirect      J 0.68
    ssti               J 0.64

Public checks and release assertions

A scorer is only as trustworthy as the answer key it reads. The public release ships the answer key, scorer, bundle manifest, and hashes so users can inspect the artifact and reproduce their own score. BenchProctor states that its pre-release gates also check label and scorer consistency, but the per-file proof metadata and oracle output are not public.

That’s the whole model. Run your scanner, hand the SARIF and the CSV to a one-file scorer, and read a number you can recompute by hand if you doubt it. No black boxes, just counts, two ratios, and a subtraction.