One Score Hides the Gap

Ask most SAST tools how good they are and you get one number, usually an F-score. It sounds precise. It hides the thing you most need to know. A single figure folds together at least three different abilities, and a tool can be excellent at one and useless at another while the average looks fine. When you are choosing a scanner, the average is exactly the wrong view. You need to see the gap.

So BenchProctor reports a triple, side by side, and never averages it.

Three different questions

A finding that crosses files and services is not one question, it is three, and each has its own answer:

flow judgment          did the tool decide the path is actually dangerous
weakness naming        did it name the right CWE
chain reconstruction   did it connect the steps into one ordered path

The weakness axis gives full credit for the exact CWE and partial credit for the correct parent class, so a near miss is not scored the same as a blind guess. A tool can recover the graph and still name the wrong weakness. It can name every CWE and never prove it traced a path. Fold those into one score and the failure disappears into the mean. Kept apart, the specific gap is right there on the scorecard, which is the only form a score is useful to a buyer in.

The axis nobody measures

One sub-score is reported entirely on its own: broken-sanitizer recall. It asks the question the field is worst at. When a sanitizer is present but does not actually neutralize the input, does the tool see through it, or does it just notice that a function named like a sanitizer was called and move on. Modeling what a control does, rather than that a control exists, is one of the most under-measured properties in static analysis, and here it gets its own line instead of being blended away.

Youden’s J, stated correctly

The flow-judgment axis uses Youden’s J, and it is worth stating precisely because it is easy to quote wrong:

J = sensitivity + specificity - 1

It is a difference measured within each class, not a raw hit rate. A tool that flags everything gets perfect sensitivity and zero specificity, so its J is 0. Flagging more does not raise it. Only being right about both the dangerous and the safe cases moves it, which is exactly the incentive a benchmark should create.

Recompute it yourself

None of this is a learned judge or an opaque aggregate. It is counts, rates, and a path check, scored against an answer key the tool never sees during the run. The scorer ships publicly, so you can grade any SARIF-emitting tool from the answer-key files alone and read the same triple we do. For the layer-by-layer detail of how the chain score is built, see how BenchProctor scores a cross-language vulnerability chain; for the single-file version, how BenchProctor scores a SAST tool.