Your Code Is Polyglot, So Is the Benchmark
Real systems mix languages, glue scripts, native modules, and a stack of cloud and infrastructure config. BenchProctor's corpus covers that surface, so you measure a scanner against your real attack surface instead of a sliver.
Most SAST benchmarks test one language, in one framework, on one kind of file. Your production system is nothing like that. It is a couple of application languages, a pile of glue scripts, maybe a native module, and a stack of cloud and infrastructure config that decides what is actually reachable. A tool that scores well on a Python-only benchmark has told you almost nothing about how it reads the rest of that surface.
So BenchProctor’s corpus covers the surface, not a sliver of it.
The languages
The corpus emits eleven code languages, plus a Solidity emitter for on-chain contracts:
| Tier | Languages |
|---|---|
| App and backend | Python, Go, Java, JavaScript, TypeScript, PHP, Ruby, Rust |
| Systems | C, C++ |
| Scripting | Bash |
| On-chain | Solidity |
Being honest about readiness: a language ships in a category only when it clears the same per-category floor as the others. We would rather cover less of a language than ship a category that cannot carry its weight.
The infrastructure
Code is only half of what decides whether a vulnerability is reachable. The other half is configuration, and the corpus treats it as a first-class target: infrastructure and cloud substrates across twenty-one deploy-time forms. That spans Terraform, CloudFormation, Kubernetes, Dockerfiles, GitHub Actions, GitLab CI, nginx, and Traefik, along with in-language infrastructure through CDK, CDKTF, and Pulumi. A misconfiguration in a manifest is as real as an injection in a handler, and here it is tested like one.
The weaknesses
Coverage of vulnerability classes is deliberately wide, exercised over 92 service topologies so a class shows up in more than one architectural setting rather than a single canned layout.
Why breadth is the point
A benchmark that only tests where a tool is strong flatters the tool. Breadth does the opposite: it measures the scanner against the mixed surface your team actually runs, and it makes the weak lanes visible instead of letting one strong language carry the score. That is the same discipline behind a benchmark you cannot pre-train against, and it is the surface every scanner the stack relies on answers to, TheAuditor included. More on the wider toolset at the Code Reality Labs blog.