Methodology

What we measure

Leak recall is the share of labelled personal data found. Over-redaction is the share of must-not-redact values hidden. Output precision is the share of distinct suggestions matching personal-data labels.

F2 weighs recall four times as much as precision: F2 = 5 × P × R / (4 × P + R), where R is leak recall and P is 1 − over-redaction. When there are no must-not-redact labels, P falls back to output precision; the row discloses this basis and the two bases are not comparable. F2 is zero when P and R are both zero.

Cost uses vendor list price in EUR excluding VAT at the price date, cache hits at the published cached-input rate, no batch discounts, every call counted including retries and failures, divided by page-calls (pages × repeats). USD prices convert at the ECB rate for that date.

How we run systems

A bare model uses the fixed benchmark harness. A system combines a model with its own rules, harness or pipeline; a rules-only floor uses no model. Every bare model receives the same prompt and parameters. OpenDisclosure runs every entry itself.

Release evaluations use three repeats and a 95% document bootstrap. EU-hosted entries appear on the ranked board; non-EU processing appears in a separate reference section. Frontiers compare entries within a class and precision basis.

What we do not publish and why

The dataset, labels, harness and hashes remain private. Publishing them would allow tuning to the test set or confirmation of hidden values. We publish aggregate measurements, reporting-floor breakdowns, methodology, dataset and scoring versions, dated prices, run support and corrections instead.

Limitations

This benchmark evaluates the text layer only, with no OCR or visual-redaction claim. The corpus is synthetic. A tuning preview is not a ranking. Official-capacity names are reported but not scored. Latency is not published.

Conflict of interest

OpenDisclosure sells redaction software using these models and maintains this benchmark. Its own entries carry a Maintainer badge and use the same evaluation contract as submissions. We publish results whatever they show, separate model and system frontiers, and retain original results when correcting errors.