An answer key you cannot grep for
In a single file, anti-leakage means no comments, no CWE tags, no telltale names. When the benchmark is a whole application, the label can leak through the shape of the repo instead. Here is what a polyglot corpus has to hide that a single file never had to.
The oldest way to beat a benchmark is to read the answer without analyzing the code. In a single file, BenchProctor closes that the obvious way: no comments, no CWE tags, no category names, and nothing in the file name or an identifier that reveals the label. That much is anti-leakage by construction, and the only ground truth is a separate answer key the scanner never reads.
A polyglot scenario makes the unit of test a whole application, and an application has places to leak an answer that a single file simply does not have. If the test is a repository, the sharp question is obvious: can a scanner, or the person tuning one, skip the analysis and read the label off the structure instead? Look at which files exist, grep across them, and sort the corpus without understanding a line?
Closing that is the bar the category has to clear, and it is a harder bar than the single-file one.
The single-file rules do not cover a repo
In one file there is one surface: the bytes in front of the tool. In an application there are many files, their names, whether each one is present at all, and the relationships between them. Every one of those is a channel a careless benchmark leaks through, and none of them exists when the test is a lone snippet.
A module that appears only when it is used is a tell
Suppose a service’s authorization guard is present in the safe applications and missing from the
vulnerable ones. Now you never have to read the guard. You classify every app in the corpus with
ls, because presence itself is the label. So a service carries its guard module whether or not this
particular route calls it. The file is there either way, and its presence tells you nothing.
A name that predicts its role is a tell
Suppose the value that still carries untrusted input is always called one thing and a cleaned value
another. Then grep sorts the corpus for you, and the tool that “passes” is one that learned your
naming, not one that follows taint. So identifiers are drawn so they do not reveal whether a value
was sanitized. The same word appears on the safe side and the vulnerable side, and no name predicts
the label.
Even the markup can leak
The tell does not have to be code. If one CSS class only ever showed up on the applications carrying the planted weakness, that class is the answer key, sitting in the HTML. So a component’s vocabulary describes the element it marks up, not which end of a data flow it sits on. The class names on the safe twin and the vulnerable twin are the same names.
And at the sink, the two are the same bytes
The deepest version of this is the one the compound chain rests on. The vulnerable application and its safe twin are byte-identical at the sink. The only difference is a control several files, and usually one language, upstream. There is no local shape to memorize, because locally there is no difference to find.
Why this is the bar for going public
Hiding the answer across a whole application is harder than hiding it in one file, and that is the point. It is why the polyglot category is still scoring in validation rather than in the public release: it joins when its labels clear the same standard the eleven standalone languages did, not before. Same discipline, one tier up, and it is the same yardstick every SARIF-emitting scanner the stack relies on answers to, TheAuditor included.