Whole Apps, Not Snippets

A single-file test answers one question well: given these bytes, is the code vulnerable or safe. That floor matters, and it already separates real analyzers from pattern-matchers. But almost no breach your team will ever ship lives inside one file. It lives in the seams, where a value crosses an HTTP boundary from one language into another, moves through a queue, and lands in a sink that an infrastructure file made reachable. A one-file test cannot measure whether a tool follows taint across that path, because the path is the whole point.

Release v0.14.0 is BenchProctor’s answer to that gap. The corpus now ships whole applications, not snippets.

What an application is here

An application is a small, idiomatic 2026 project. It runs two or three application-tier languages plus one or two cloud and infrastructure targets. It is three to ten files. It compiles as a unit under each language’s native toolchain, with real dependency manifests, the way a repo you would actually clone does. A Python backend with a server-rendered TypeScript frontend, a Kubernetes manifest, and a CI workflow. A Go service calling into a Rust module behind a shell script that changes a cloud configuration. The shape moves from app to app on purpose, so the corpus never presents a constant silhouette a scanner can lock onto.

Inside each application are three to five planted weaknesses, not one. Some of them compose. A subset of the planted weaknesses forms a compound chain: an escalation that none of the individual findings expresses on its own, reachable only by tracing the path from an entry weakness, through the services, to a terminal impact. Read the dangerous call by itself and the vulnerable app and its safe twin look identical. The only way to tell them apart is to follow the path back across the boundaries and decide whether the control upstream actually holds.

The scale

At corpus scale the application shape is 760 applications carrying 3,030 planted weaknesses. That is not a vanity number. It is enough breadth that no short list of “the cases” exists to tune a rule against, and enough variety that a tool which scores well did so on real cross-boundary analysis rather than on a memorized silhouette.

What is done, and what is not

Being honest about scope is part of the brand, so here is the line. The application shape is live in v0.14.0, alongside the standalone single-file corpus that already covers every language. What is not done yet: deeper adversarial variants are staged behind their own quality checks and are not in this release, and full runtime and container verification of the applications is designed but has not been run. A shape joins a release when it clears the bar, not before.

For the design goal behind whole-app testing, see the vulnerability a single-service scan cannot see. This is the same yardstick every SARIF-emitting scanner the stack relies on answers to, TheAuditor included.