Java and Python are live: point a scanner at them, get a real number

When we wrote that we’d release one language at a time, only once its labels were ones we’d defend line by line, the obvious question was when. Today is the answer for the first two: Java and Python are live.

• Java: Spring (Boot 4) and Jakarta EE 11.
• Python: Flask, Django, and FastAPI.

Both are single-file standalone corpora covering every CWE category the language supports, around 210 vulnerability types, balanced exactly 50/50 vulnerable to safe. Each cleared the bar from the release plan in full. Every emitted file compiles or parses on the real toolchain, every vulnerable case carries a genuine source-to-sink taint flow, every safe twin actually neutralizes it for that sink, and the line we point at is the line the vulnerability lives on.

Three sizes, so you can trade run time for depth

Each language ships in three sizes:

• quicktest (~10k tests) covers the most prevalent CWEs at 25 vulnerable and 25 safe per type. It’s a fast first read on a new tool or a CI gate.
• normal (~40k) covers roughly all supported CWEs at 50 and 50 per type. This is the headline scoreable corpus, with enough cases per category that one or two files can’t swing a number.
• enterprise covers every CWE at the deepest sampling, for vendors who want tight confidence intervals.

Pick the one that fits your run-time budget. The answer-key format is identical across all three.

An answer key you can’t read off a filename

The point of a benchmark is that the answer is known in advance and can’t be faked. So the emitted files carry no comments, no CWE tags, no category names, and no hints in identifiers, and the file names tell a scanner nothing. A path like testcode/BenchmarkTest04417.java gives away neither the category nor the label, so a tool that pattern-matches filenames instead of analyzing code scores like a coin flip.

What you download is exactly two things per suite: the testcode and a CSV answer key (test name, category, real vulnerability, CWE). Everything we use internally to verify the labels stays internal, so the answer key can’t be reconstructed from a file you were supposed to analyze blind. The labels are locked and checked before the release ships.

Bring any SARIF, and the scorer meets your tool where it is

Scoring is one standard-library Python file, no dependencies:

your-tool scan ./Benchmarks/normal/java/spring --format sarif -o results.sarif
python scripts/score_sarif.py results.sarif Benchmarks/normal/java/spring/expectedresults-*.csv

A finding counts as a real detection when it lands on a vulnerable file and carries that file’s CWE. Most tools don’t put a bare CWE number in the SARIF ruleId, so the scorer recovers it from wherever your tool actually writes it: the ruleId, the result’s or rule’s properties/tags (e.g. external/cwe/cwe-089), or CWE taxa. If your tool emits no CWE at all, add --match-mode filename and any finding on a vulnerable file counts. That rewards over-flagging, so the CWE-aware default is the honest setting.

You get a true-positive rate, a false-positive rate, and one number that subtracts them, Youden’s J, reported both category-averaged (every vulnerability class weighted equally) and flat-aggregate. +100% is perfect, 0% is guessing, and negative means the tool is inverted.

Why the number stays honest over time

Every quarter the corpus rotates. The test code changes completely while everything scoring-relevant stays fixed: CWE identity, the difficulty mix, the 50/50 balance, and the language and framework coverage. Last quarter’s score stays comparable, and a model trained on last quarter’s files learns nothing about this quarter’s. The benchmark stays a measurement, not a leaderboard to overfit.

What’s not here yet, on purpose

This release is single-file standalone. Cross-file CWE chains, polyglot microservice scenarios, and adversarial / SAST-evasion cases are on the roadmap and ship when they clear the same bar. The remaining languages (Go, Rust, TypeScript, JavaScript, PHP, Ruby, Bash) follow the same way. We’d still rather ship two languages whose every label we’d defend than nine we’d have to qualify.

Java and Python are live now. Point your scanner at them and find out what your accuracy actually is.