Skip to benchmark content
OpenDisclosureEU Redaction BenchmarkAbout

EU Redaction Benchmark

How well do language models redact personal data in freedom-of-information documents, and what does it cost?

Preview: results on the tuning split. This is not a ranking; no frontier is claimed.

0%20%40%60%80%100%EUR 0.10EUR 1EUR 10FreeF2 score (recall-weighted)EUR per 1,000 pages · log scaleMistral Small 2603OpenDisclosure RedactionOpenDisclosure RedactionRules onlyRules only
0%20%40%60%80%100%EUR 0.10EUR 1EUR 10FreeF2 score (recall-weighted)EUR per 1,000 pages · log scaleMistral Small 2603OpenDisclosure RedactionOpenDisclosure RedactionRules onlyRules only
  • Bare model
  • System
  • Free rules floor
  • 95% CI of F2

Each point is the latest scored run for that entry, pooled over all documents; it is not an average of runs. Hover or focus a point, or a table row, for its run date and details.

NameF2RecallOver-redactionOutput precisionEUR per 1,000 pagesRun date
OpenDisclosure Redaction
SystemMaintainer
74.3%72.8%19.1%66.1%EUR 0.512026-10-14
OpenDisclosure Redaction
SystemMaintainer
73.6%71.8%18.2%64.7%EUR 0.512026-10-14
Mistral Small 2603
Bare model
73.5%73.6%26.7%58.3%EUR 0.502026-10-14
Rules only
FloorMaintainer
16.5%13.7%4.4%64.9%EUR 0.002026-10-14
Rules only
FloorMaintainer
16.3%13.5%4.1%64.4%EUR 0.002026-10-14