H1: The Hidden-Identity
Linking Benchmark

The Task

A patient has several reports over time. A report may mention the same liver lesion in March and again in June, with different wording. Another report may mention a second lesion at the same site. A tracking system must decide which mentions belong together.

H1 asks for two groupings of every world's mentions:

• Category partition. Mentions of the same finding type, organ, and side go together.
• Entity partition. Mentions of the same individual lesion go together. Two lesions can share one category, so entities are finer than categories.

How It Works

The system never sees a label. The scorer reports category and entity quality separately, so a system cannot hide a weak entity result behind a strong category result.

The eight v2 families

The test split has 400 worlds, 50 per family. Each world holds a target category and one or two look-alike distractor categories.

• granularity_with_laterality_distractor. Rib or bone metastasis on one side, with the same finding on the other side. The target's own mentions switch between "rib" and "bone".
• granularity_with_organ_distractor. Rib or bone metastasis on one side, with a sideless spine or bone metastasis sharing the word "bone".
• laterality_pair_same_type. Lung metastasis on one side and on the other side.
• same_word_different_organ. A breast primary tumor, a stomach primary tumor, and a liver metastasis, all described with "mass" or "lesion".
• same_site_different_type. Lung metastasis and pneumonia or infection in the same lung, same side.
• simultaneous_distinct_with_distractor. Two liver metastases at once (same key, two entities), plus an abdominal lymph node metastasis.
• descriptor_pair_with_distractor. Two breast primary tumors with the same key, described with or without a distinguishing descriptor, plus a rib bone metastasis.
• episode_reuse_with_distractor. A liver metastasis resolves, then a new lesion reuses the same key, plus a rib bone metastasis.

Source: DATA_CARD.md and the contracts/ folder in the repository.

Metrics

• Category B-cubed F1 and Entity B-cubed F1 are co-primary. We never combine them into one score.
• Split and merge rates. The share of true same-group pairs the system separated (split), and the share of true different-group pairs it joined (merge). Each is reported for categories and for entities.
• Coverage. The unresolved rate: the share of mentions a system left without a group. Unresolved mentions stay in the denominator.
• Uncertainty. Every interval is a 95% CI from a world bootstrap (10,000 draws, seed 42). Paired contrasts resample within each family and weight families equally.

In v2 the same one-group baseline scores a category F1 of 0.676, and the one-group-per-mention baseline scores 0.344. Trivial baselines sit beside every model result below for this reason.

Results on the v2 Test Split

All rows use the same 400 test worlds. GLM answered 342 of them. GLM spent its whole token budget on reasoning in all three tries for the other 58. Those worlds stay in the denominator and score as unresolved singletons. This earns partial credit, not zero: mean category F1 on the 58 failed GLM worlds is 0.326. The rows labelled "author reference baseline, unranked" come from the linker the benchmark authors wrote. Do not compare a model against them as if they were an independent entrant.

Cohort: 400 v2 test worlds, 50 per family. Source: results/v2_test_results.json (systems.*.overall). Means are over worlds, and CIs come from the bootstrap in the same file. A model's category-partition row and entity-partition row are two separate outputs from one model, so each row shows all columns for that output.

Paired contrasts against the author baselines

Source: results/v2_test_results.json, contrasts (mean difference and 95% CI, paired world bootstrap, equal-family weights, 10,000 draws).

What the numbers support

• Neither model beats the compatibility linker on categories. Category F1 is 0.925 for DeepSeek and 0.850 for GLM, against 0.947 for the linker. Both paired differences are negative and both intervals exclude zero.
• Both models reduce same-key entity merges. Entity merge is 0.088 (DeepSeek) and 0.089 (GLM) on their entity partitions, against 0.176 for exact key and for the compatibility linker.
• The merge gain does not become an entity F1 gain for GLM. The 58 unanswered worlds count as unresolved mentions, which lowers its entity F1 relative to exact key. DeepSeek's entity partition does beat exact key on entity F1.

Scope note: these are results for two models, one prompt, temperature 0, on synthetic oracle-frame worlds. Each world got one run, except that GLM had up to three attempts per world. They say nothing about extraction from real reports.

Run It Yourself

A clean machine can install the package, run a smoke test, score a model on the development set, and build a test submission. Each step is one command.

git clone https://github.com/Sher110106/findingframe-h1 && cd findingframe-h1
python -m venv .venv && source .venv/bin/activate
pip install -e ".[openai,test]"
pytest -q
h1bench smoke
h1bench baselines --split dev

export OPENAI_API_KEY=...
h1bench run --split dev --base-url https://api.openai.com/v1 --model gpt-4o-mini
h1bench score --split dev predictions_dev_gpt-4o-mini.jsonl

h1bench run --split test --base-url https://api.openai.com/v1 --model gpt-4o-mini
h1bench make-submission predictions_test_gpt-4o-mini.jsonl --model "gpt-4o-mini"

h1bench run calls any OpenAI-compatible endpoint and writes a predictions file named predictions_<split>_<model>.jsonl (change it with --out). h1bench score scores the development set locally, because its labels are public. h1bench make-submission checks the test predictions and writes submission.json.

The repository README lists the full options for each command.

Submitting a Model

Test labels stay private. If they were public, systems could tune to the test set. The maintainer scores test submissions instead.

• Open a pull request that adds submissions/<team>-<model>/submission.json.
• The maintainer checks the schema and scores the file privately with the frozen test labels.
• The maintainer posts aggregate and per-family scores. Per-world scores stay private.
• Each model version gets one scored submission every 7 days.

Limits and Roadmap

• v3 (next). Worlds will render report text. Families and renderer styles will be held out. The linker stays unranked.
• H1-U (later). Uncertainty and abstention: a system may decline to link, and the scorer will report coverage and calibration apart from both F1 scores.

We promise no dates.