The vulnerability a single-service scan cannot see
BenchProctor's standalone corpus scores a tool one file at a time. The new polyglot scenario category scores it on the hardest real case: a vulnerability that exists only as a chain crossing several languages and services.
Release 2026.07.22 gave every SARIF-emitting scanner a fair, leak-resistant test of one thing: can it
analyze a file. 2,938,418 balanced cases across 11 languages and 21 framework targets, each case a
single unit of vulnerable-or-safe code with the answer key held separately. That is the right floor,
and it already separates real analyzers from pattern-matchers.
But almost no real breach lives inside one file. It lives in the seams between services, where a value read by a TypeScript frontend crosses an HTTP boundary into a Python handler, travels through a queue into a worker, and lands in a sink that a Terraform module made reachable. No single-file test can measure whether a tool follows taint across that path, because the path is the whole point.
So BenchProctor is building the case that does.
What a polyglot scenario is
A polyglot scenario is a small, idiomatic application, not a snippet wired to a label. It is composed of two to three application-tier languages plus one or two cloud or infrastructure targets. It runs to between three and ten files. It compiles under each language’s native toolchain, as a unit, with real dependency manifests. A Python or Java backend with a server-rendered TypeScript frontend, a Kubernetes manifest, and a CI workflow. A Python backend calling a Rust module, glued by a shell script, changing a cloud configuration. The shape varies from app to app on purpose, so the corpus never presents a constant silhouette a scanner can lock onto.
Inside each app are three to five planted weaknesses, not one. Some of them compose. A subset of the planted CWEs forms a compound chain: an escalation that none of the individual findings expresses on its own, reachable only by following the path from an entry weakness, through the services, to a terminal impact.
Why it is the hardest case
Here is the property that makes it hard, and it is the same one we describe publicly for compound chains: the difference between the vulnerable app and its safe twin can live in a different file, and usually a different language, than the sink itself. Read the dangerous call on its own and the two apps look identical. The only way to tell them apart is to trace the path back across the service boundaries and decide whether the control upstream actually holds.
A tool that reasons one file, or one language, at a time cannot see it. That is not a corner case. It is what most of the software your team ships actually looks like.
Same discipline, one tier up
The rules that make the standalone corpus honest still apply. The files carry no comments, no CWE tags, and no naming tells. The answer key lives outside the code the scanner reads. The vulnerable and safe variants stay balanced, and because the score subtracts the false-positive rate from the true-positive rate, flagging everything still nets zero.
The polyglot scenario category is built and scoring in validation now. It joins the public release when its labels clear the same bar every standalone language had to clear, not before. When it lands, it is the same yardstick the rest of the stack answers to, including TheAuditor, one tier closer to the code people actually run.
More on the design goal behind it: what makes a SAST test actually hard.