Engineering notes on SAST benchmarking: methodology, scoring, and coverage for measuring static analysis tools.https://blog.benchproctor.com/en-ushttps://blog.benchproctor.com/java-and-python-are-live/https://blog.benchproctor.com/java-and-python-are-live/BenchProctor release 2026.07.22 is live with 2,938,418 balanced benchmark cases across 11 languages and 21 framework targets. Score compatible SARIF 2.1.0 output.Wed, 22 Jul 2026 00:00:00 GMTannouncementreleaseBenchProctorhttps://blog.benchproctor.com/an-honest-score-you-cannot-train-against/https://blog.benchproctor.com/an-honest-score-you-cannot-train-against/BenchProctor provides a versioned corpus, public answer key, and scorer for interpreting compatible SARIF results.Mon, 20 Jul 2026 00:00:00 GMTecosystembenchmarksastBenchProctorhttps://blog.benchproctor.com/releases-go-continuous/https://blog.benchproctor.com/releases-go-continuous/A score is meaningful only with its scanner and benchmark versions. BenchProctor uses dated, periodic, event-driven releases.Wed, 15 Jul 2026 00:00:00 GMTreleasesmethodologyBenchProctorhttps://blog.benchproctor.com/a-benchmark-you-cannot-pretrain-against/https://blog.benchproctor.com/a-benchmark-you-cannot-pretrain-against/A frozen benchmark can reward memorization. Here is how 2,938,418 balanced cases and versioned releases support a more useful test.Thu, 09 Jul 2026 00:00:00 GMTbenchmarkingmethodologyBenchProctorhttps://blog.benchproctor.com/how-to-measure-sast-false-positive-rate/https://blog.benchproctor.com/how-to-measure-sast-false-positive-rate/Most quoted SAST accuracy numbers have no ground truth behind them, so they cannot be checked. Here is what a defensible false-positive measurement actually requires.Sat, 04 Jul 2026 00:00:00 GMTsastmethodologyBenchProctorhttps://blog.benchproctor.com/reading-a-sast-scorecard/https://blog.benchproctor.com/reading-a-sast-scorecard/A single SAST score can hide as much as it shows. Here is how to read recall, false positives, and the traps that make a good tool look bad.Sun, 28 Jun 2026 00:00:00 GMTscoringmethodologyBenchProctorhttps://blog.benchproctor.com/the-2026-2-corpus-versioned-and-rotated/https://blog.benchproctor.com/the-2026-2-corpus-versioned-and-rotated/A historical note on the 2026.2 corpus and the value of versioned benchmark releases.Fri, 26 Jun 2026 00:00:00 GMTmethodologyrotationBenchProctorhttps://blog.benchproctor.com/bring-any-scanner-sarif/https://blog.benchproctor.com/bring-any-scanner-sarif/Scoring against BenchProctor is one standard-library Python file with no dependencies. The catch most benchmarks trip on is that tools don't report CWEs the same way, so the scorer recovers the CWE from wherever your tool actually writes it, with no per-tool adapter, and grades every tool on the same honest footing.Wed, 24 Jun 2026 00:00:00 GMTsarifscoringinteroperabilityBenchProctorhttps://blog.benchproctor.com/what-makes-a-sast-test-hard/https://blog.benchproctor.com/what-makes-a-sast-test-hard/A benchmark only means something if a pattern-matcher can't ace it. Here's what's inside a corpus designed to be hard: real framework idioms instead of toy snippets, sanitizers that are present but broken, taint that travels several steps, and a strict 50/50 split that makes 'flag everything' score zero.Tue, 16 Jun 2026 00:00:00 GMTmethodologysastbenchmarkingBenchProctorhttps://blog.benchproctor.com/wrong-answer-key-punishes-right-tools/https://blog.benchproctor.com/wrong-answer-key-punishes-right-tools/A SAST benchmark with a mislabeled safe file measures the wrong thing. Here is why public answer keys and transparent scoring matter.Thu, 11 Jun 2026 00:00:00 GMTmethodologycorrectnessbenchmarkingBenchProctorhttps://blog.benchproctor.com/benchmarking-in-the-agent-stack/https://blog.benchproctor.com/benchmarking-in-the-agent-stack/BenchProctor is an open benchmark corpus with a public scorer for compatible SARIF 2.1.0 output.Tue, 09 Jun 2026 00:00:00 GMTecosystembenchmarkBenchProctorhttps://blog.benchproctor.com/java-first-release-plan/https://blog.benchproctor.com/java-first-release-plan/A look back at the staged verification plan that led to all 11 standalone languages shipping in release 2026.07.22.Sat, 30 May 2026 00:00:00 GMTreleaseroadmapBenchProctorhttps://blog.benchproctor.com/how-benchproctor-scores-sast-tools/https://blog.benchproctor.com/how-benchproctor-scores-sast-tools/The whole scoring model is a confusion matrix and one subtraction. Here's how true-positive and false-positive rates become a single number, why we average per category, and how the benchmark checks itself.Thu, 28 May 2026 00:00:00 GMTscoringmethodologyBenchProctorhttps://blog.benchproctor.com/why-static-sast-benchmarks-rot/https://blog.benchproctor.com/why-static-sast-benchmarks-rot/A frozen benchmark can measure memorization as much as analysis. Versioned releases help preserve comparability while introducing fresh cases.Sun, 24 May 2026 00:00:00 GMTmethodologybenchmarkingBenchProctorhttps://blog.benchproctor.com/introducing-benchproctor/https://blog.benchproctor.com/introducing-benchproctor/An introduction to BenchProctor's public SAST benchmark corpus and transparent SARIF scoring model.Wed, 20 May 2026 00:00:00 GMTannouncementmethodologyBenchProctor