How BenchProctor scores a cross-language vulnerability chain
A single confusion matrix cannot describe a finding that crosses files, services, and languages. Here is the three-part score BenchProctor reports for a polyglot scenario, and why it never collapses them into one number.
For a single file, the honest score is one number. Sort every result into the four buckets of a confusion matrix, compute the true-positive rate and the false-positive rate, subtract, and read Youden’s J. One number is enough because there is one question: is this file vulnerable or safe.
A cross-service chain asks three questions, so it needs three answers. Fold them into one and you hide which of the three a tool is actually bad at. So BenchProctor reports a triple, side by side, and never averages it.
Layer one: did it judge the flow
The first unit is a single declared flow: a source, a sink, and the path between them. The truth is three-valued. The flow is unsanitized, or it is effectively sanitized, or it carries a sanitizer that is present but does not actually hold. A tool should flag the first and the third and stay quiet on the second. The metric is sensitivity, specificity, and the same Youden’s J you already know, computed per flow.
One sub-metric is reported on its own: broken-sanitizer recall, the share of present-but-broken sanitizers the tool saw through. This is the axis that separates a tool that models what a sanitizer does from one that pattern-matches that a sanitizer was called, and it is one of the most under-measured properties in the field.
Layer two: did it name the weakness
The second score is the CWE, and it only counts on a flow the tool already judged correctly. That condition matters: without it, a tool could farm points by naming every CWE at every line. An exact CWE earns full credit, the correct parent class earns partial credit, and a wrong one earns nothing.
Layer three: did it connect the chain
The third unit is the whole app. Did the tool assemble the individual findings into one ordered path across the services? The bar is specific: the tool has to emit a SARIF code flow whose steps, in order, cover the true path from file to file and service to service. Naming two files in two unrelated findings is not the same as connecting them. The ordered path inside a single finding is the evidence that the tool actually traced the chain.
Partial credit here is explicit and deliberate. Finding two of five planted weaknesses is not the same as finding none, and the score says so. Reporting the entry weakness, the front door the whole chain starts at, counts for more than recovering the middle. And there is an anti-gaming rule: the chain score only counts for an app whose per-flow judgment was good enough to trust. Reconstructing a graph you cannot judge is not understanding it.
Report the triple, never average
The three answer different questions, so folding them loses information on purpose. Averaged into one figure, a tool that recovered the graph could hide the fact that it named one weakness out of four. Reported side by side, it cannot.
None of this is a learned judge or an opaque aggregate, any more than the single-file score is. It is counts, rates, and a path check anyone can recompute, scored against an answer key the tool never saw. That is what lets the number mean the same thing for every scanner, TheAuditor included.
The polyglot category is scoring in validation now. This triple joins the public release when its labels clear the same bar the standalone languages did, not before.