Cortex
Skip to benchmark report

Cortex PII Filter vs OpenMed PII SuperClinical Large 434M v1

Cortex PII Filter vs OpenMed PII SuperClinical Large 434M v1

PII redaction, benchmarked on real-world text

Personally identifiable information leaks to model providers and all systems connected to your agent, and redaction requires context-aware, real-time filtering across messy real-world scenarios. Real-world PII sits inside free-form text, in multiple languages and document formats.

Cortex PII Filter uses an in-house fine-tuned classification head to identify and redact PII before it leaks, including names, contact details, addresses, and sensitive identifiers.

What we compared

Cortex PII Filter and OpenMed PII SuperClinical Large 434M v1 were evaluated across five benchmark suites: SPY, Hebrew-SPY, a Hebrew-localized AI4Privacy set, Gretel Synthetic Finance, and AI4Privacy. Character recall is the share of annotated PII characters redacted; exact recall measures exact matches to annotated PII spans.

99.46%
PII characters successfully redacted
98.38%
PII characters successfully redacted
4 / 5
Benchmark suites where Cortex scores higher
Why context matters

PII is contextual, multilingual, and embedded in real documents

A PII filter has to identify sensitive information inside the text people actually send to AI systems: customer-support messages, medical notes, legal questions, financial records, and multilingual documents. The same token can be harmless in one sentence and sensitive in another.

Cortex PII Filter identifies sensitive spans in context and replaces those values with typed redactions. The remaining text retains the structure and information a downstream workflow needs, without exposing the underlying personal data.

  1. 01

    Incoming text

    Free-form messages, documents, notes, and records enter the filter.

  2. 02

    PII classification

    The in-house fine-tuned classification head identifies sensitive information in context.

  3. 03

    PII spans identified

    Names, contact details, addresses, and sensitive identifiers are located in the text.

  4. 04

    Typed redaction applied

    Identified values are replaced before the text reaches connected systems.

Contact and identity details

Names, emails, phone numbers, postal addresses, and account or government identifiers.

Sensitive values within workflow text

Personal details embedded in medical notes, legal questions, customer requests, and financial records.

Multilingual and Hebrew entities

Identifiers in mixed-language documents, Hebrew text, and locally formatted names and addresses.

Reported results

PII redaction results by benchmark

Cortex leads on character recall in 4 of 5 suites and exact recall in 5 of 5.

Character recall

SPYCortex +1.08 pp
Cortex PII Filter
99.46%
OpenMed PII SuperClinical Large 434M v1
98.38%
Hebrew-SPYCortex +18.94 pp
Cortex PII Filter
96.90%
OpenMed PII SuperClinical Large 434M v1
77.96%
Hebrew-localizedCortex +33.29 pp
Cortex PII Filter
99.32%
OpenMed PII SuperClinical Large 434M v1
66.03%
GretelCortex +27.33 pp
Cortex PII Filter
94.98%
OpenMed PII SuperClinical Large 434M v1
67.65%
AI4PrivacyOpenMed PII SuperClinical Large 434M v1 +19.16 pp
Cortex PII Filter
63.93%
OpenMed PII SuperClinical Large 434M v1
83.09%
Legend
  • Cortex PII Filter
  • OpenMed PII SuperClinical Large 434M v1

Five benchmark suites. Scores use a shared 0–100% scale; further right is higher. Gaps are percentage points (pp).

Exact recall

SPYCortex +82.98 pp
Cortex PII Filter
83.18%
OpenMed PII SuperClinical Large 434M v1
0.20%
Hebrew-SPYCortex +71.98 pp
Cortex PII Filter
77.79%
OpenMed PII SuperClinical Large 434M v1
5.81%
Hebrew-localizedCortex +71.74 pp
Cortex PII Filter
85.96%
OpenMed PII SuperClinical Large 434M v1
14.22%
GretelCortex +85.90 pp
Cortex PII Filter
90.88%
OpenMed PII SuperClinical Large 434M v1
4.98%
AI4PrivacyCortex +45.43 pp
Cortex PII Filter
62.39%
OpenMed PII SuperClinical Large 434M v1
16.96%
Legend
  • Cortex PII Filter
  • OpenMed PII SuperClinical Large 434M v1

Five benchmark suites. Scores use a shared 0–100% scale; further right is higher. Gaps are percentage points (pp).

What the benchmark suites cover

Five kinds of sensitive text

SPY

Tests identification of personal information in English medical consultations and legal questions, where sensitive details appear within longer requests for advice.

Hebrew-SPY

Extends SPY's medical consultations and legal questions with Hebrew translations and locally adapted personal details. The benchmark includes English reference, parallel Hebrew, and localized Hebrew versions.

Hebrew-localized AI4Privacy

An internal benchmark for evaluating PII-redaction performance in Hebrew and with Hebrew entities. It adapts AI4Privacy education, health, and psychology scenarios into Hebrew with locally adapted names, addresses, and other identifiers.

Gretel Synthetic Finance

Tests PII identification in synthetic financial documents, including bank statements, transaction records, and financial reporting formats. This evaluation uses English-language records from Gretel's multilingual financial dataset.

AI4Privacy

Tests PII identification in synthetic text covering education, health, and psychology scenarios. The evaluated validation set includes English, Dutch, French, German, Italian, and Spanish, with personal identifiers embedded in varied document and communication formats.

Hebrew evaluation

Built to evaluate Hebrew text and Hebrew entities

Hebrew PII requires evaluation beyond translation. The internal Hebrew suites include Hebrew names, addresses, identifiers, right-to-left text, and mixed Hebrew-English documents. They measure whether a filter identifies sensitive values in the language and document formats used by Hebrew-speaking organizations.

Hebrew-SPY

Our internal benchmark evaluates Hebrew text and Hebrew entities in medical consultations and legal questions.

Hebrew-localized AI4Privacy

Our internal benchmark adapts education, health, and psychology scenarios with localized Hebrew names, addresses, and identifiers.

Traceable benchmark examples

Inspect the redaction decision

These excerpts come from the benchmark gold records identified below. Redacted views replace the annotated spans with their entity labels; they show the benchmark target, not a Cortex or comparator prediction. Character offsets refer to the original record.

SPY

Source ID: spy:093be5a0235c9e15bc1c364f10ac48b9563de34c:medical_consultations:001371
Variant: en_original
Record: 1 · Characters 0–280 (end exclusive)
2 annotated spans in excerpt

Benchmark input excerpt

Text: Oh my goodness, I'm so sorry to be bothering you doctor, I hope you're having an amazing day! I'm Stephen Davidson , and I 'm filling out this form for an online medical consultation , reachable at sreynolds@hotmail.com for any follow - up q

Redaction from gold annotations

Text: Oh my goodness, I'm so sorry to be bothering you doctor, I hope you're having an amazing day! I'm [NAME] , and I 'm filling out this form for an online medical consultation , reachable at [EMAIL] for any follow - up q

Hebrew-localized AI4Privacy

Source ID: 40770A
Variant: he_localized
Record: 1 · Characters 0–259 (end exclusive)
3 annotated spans in excerpt

Benchmark input excerpt

בפלטפורמת שיתוף הווידאו לתוכן חינוכי, התפתחה שיחה תוססת בין משתמשים מאזורים שונים בבריטניה. שרשור התגובות החל כאשר yuval_cohen_0643 הביע הערכה לתוכן התובנתי של הסרטון, ולאחר מכן amit_golan_2640 הוסיף הבהרה בנושא מורכב. amit_avraham_8901 הצטרף עם שאלה להבהרה,

Redaction from gold annotations

בפלטפורמת שיתוף הווידאו לתוכן חינוכי, התפתחה שיחה תוססת בין משתמשים מאזורים שונים בבריטניה. שרשור התגובות החל כאשר [USERNAME] הביע הערכה לתוכן התובנתי של הסרטון, ולאחר מכן [USERNAME] הוסיף הבהרה בנושא מורכב. [USERNAME] הצטרף עם שאלה להבהרה,

Gretel Synthetic Finance

Source ID: 40012
Variant: en
Record: 1 · Characters 0–470 (end exclusive)
6 annotated spans in excerpt

Benchmark input excerpt

SUPPLY CHAIN MANAGEMENT AGREEMENT This Supply Chain Management Agreement (the "Agreement") is entered into as of this 1st day of March, 2021 (the "Effective Date"), by and between Cameron-Mcknight, a company organized and existing under the laws of the state of Delaware, with its head office located at 81685 Lopez Lodge, Apt. 6502, hereinafter referred to as "Cameron-Mcknight", and Jann N. Butte, an individual with a mailing address of 81685 Lopez Lodge, Apt. 6502.

Redaction from gold annotations

SUPPLY CHAIN MANAGEMENT AGREEMENT This Supply Chain Management Agreement (the "Agreement") is entered into as of this [DATE] (the "Effective Date"), by and between [COMPANY], a company organized and existing under the laws of the state of Delaware, with its head office located at [STREET_ADDRESS], hereinafter referred to as "[COMPANY]", and [NAME], an individual with a mailing address of [STREET_ADDRESS].

Methodology

How to read the results

Character recall

The percentage of annotated PII characters successfully redacted.

Exact recall

The percentage of annotated PII spans matched exactly, including their boundaries.

Comparator: OpenMed/OpenMed-PII-SuperClinical-Large-434M-v1

Download reported results (JSON)