Language Models Struggle with Staging in a Nasopharyngeal Cancer Benchmark

Nasopharyngeal carcinoma is a cancer of the upper throat behind the nose. Its care involves decisions made with different evidence at different times, a sequence that isolated clinical questions can miss. ^[1]

The finding

Scoring decisions across cancer care phases

Earlier medical benchmarks, the authors argue, score isolated encounters. They also test each question in a single wording. ^[1] This one follows each patient across six phases and asks what the clinician should do at that moment, using only the records that existed then. It also re-asks the same question in three levels of detail, to see how much the model leans on the prompt.

The result is a benchmark the authors call NPC-RubricBench. The preprint has not been peer reviewed. All numbers below are the authors' own, and every score depends on their rubrics and their automated judge.

Omissions need their own checks

A single accuracy number, the authors argue, can mask where a model is weak. A model strong at one phase can be weak at the next. A response can also fail by leaving something out, which looks different from a response that says something wrong. These need different review: checking completeness in one case, checking stated conclusions in the other. ^[1]

The numbers are not clinical error rates, and they are shares of rubric items or units, not of patients. The authors say the cases were selected for coverage and the rates should not be read as estimates of patient harm. The models were also tested in Chinese on one cancer from two hospitals. The reusable lesson is the design: judge a model against the question being asked at that point in the patient's care, with prompts at several levels of detail.

The evidence

Records and rubrics at each decision point

The cases. Records came from two hospitals: Sun Yat-sen University Cancer Center (patients diagnosed 2014 to 2024) and Nanfang Hospital (2022 to 2024). The final benchmark holds 405 patients and 941 "patient-phase units". Each unit is one patient at one decision point, such as just before treatment starts. Later information, including the eventual diagnosis, stage, treatment and outcome, was withheld from earlier units. ^[1], Section 4.1 Cases were chosen to cover important clinical scenarios, not to mirror routine admissions. ^[1]

The rubrics. Three radiation oncologists wrote reusable criteria templates for each phase. Patient facts from the decision-time record were then filled in "with model assistance", and the specialists reviewed every generated criterion. ^[1], Section 4.3 The result is 17,688 patient-specific criteria, spread over five dimensions: decision quality, use of the available evidence, judging the patient's current status, avoiding claims the record does not support (faithfulness), and staying within safety limits. A failed check on the first three dimensions is labelled either an omission (required content left out) or an error (a stated conclusion that conflicts with the record).

The grader. An open-weight model, GPT-OSS-120B, scored responses one criterion at a time. ^[1], Supplement S4.1; ^[2] Three physicians independently re-graded 108 responses from 12 cases, without seeing which model wrote them or what the judge said.

The models and prompts. Thirty models were tested, 22 general-purpose and eight medically adapted. ^[1], Supplement Prompts were written in Chinese. The main condition, called Focused, states the phase goal ("determine the patient's clinical stage"). The Broad version asks only for an assessment and next step. The Elaborated version lists the parts of the answer expected. The patient evidence and rubric stayed fixed across all three. ^[1], Section 4.2

Staging errors and prompt sensitivity

Performance was uneven across phases, strongest at diagnosis and weakest at staging. On a 0 to 100 scale, overall decision quality ranged from 29.8 to 85.6 across models. Model-level decision-quality means ranged from 56.8 to 94.7 in diagnosis and from 8.9 to 56.0 in staging. All 30 models scored above their own average in diagnosis and below it in staging. Treatment decisions scored higher than staging decisions in all 30 models, so the pattern is not a steady decline across the course of care. ^[1], Section 2.2 The authors suggest diagnosis was easier because definitive pathology evidence was already in the record. That is their interpretation. ^[1]

The failure type changed by phase. These rates are shares of applicable rubric items, not shares of patients or responses. The denominator is the preset number of applicable rubric opportunities for that phase and dimension, with items judged Met or Partial counted alongside failures, pooled across all 30 models. ^[1], Section 2.4, Figure 4b. Pooling all 30 models, explicit decision errors dominated staging, at 48.2%. Omitted decisions dominated work-up (33.4%) and follow-up (30.4%). ^[1], Section 2.4 Follow-up also had the highest rate of omitted evidence use, 31.8%. Omissions outnumbered errors in 29 of 30 models. For individual models, the overall omission rate was 5.2% for GPT-5.1 and 42.5% for HuatuoGPT-o1-7B. Each model's rate pools its applicable rubric opportunities across phases and across the decision quality, evidence use and status judgement dimensions. ^[1], Section 2.4 The abstract says medically adapted models in this set had higher omission rates than general-purpose ones. ^[1] The group-level figures are not given in the main text.

Wording mattered. Making the request broader lowered mean decision quality in all 30 models (median change of -10.48 points). The detailed prompt raised it in 28 (median change of +2.17). But the detailed prompt lowered mean faithfulness in 28 models and mean safety adherence in 19. ^[1], Section 2.5 In one example, a model told to list specific items added a negative hepatitis B DNA result the record did not contain.

Averages can hide violations. Mean faithfulness and safety scores were high, but a unit counts as a violation if any single check fails. One model scored a mean safety score of 83.8 in treatment while failing at least one safety check in 81.4% of units. That figure is a share of patient-phase units, a different denominator from the rubric-item rates above. ^[1], Section 2.3

How far to trust the grader. Exact agreement between the judge and the three experts averaged 80.12%, against 84.02% among the experts themselves. ^[1], Section 2.6 Agreement was lowest on judgements of the patient's current status, at 68.95%, close to the 69.33% among the experts. This check covers 12 hand-picked cases and three of the 30 models. Model averages tracked each other across the two hospitals, but the authors say that shows internal stability, not transfer to other settings. ^[1]

Future directions

Results are for one cancer type. Each decision point was scored on its own rather than in a running conversation; fixed rubrics cannot cover every acceptable answer. ^[1]

A multi-turn version could feed a model's earlier answers into its later ones. The authors say the framework is meant to support other cancers or chronic diseases, where the same design could also be tested. They also call for a larger audit of the grader by independent experts. The finding that omissions dominate several phases also suggests testing checklists that prompt a model for missing steps. That idea is untested here, and the elaborated prompts already show that listing items can cost faithfulness.