The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context

Diyi Yang; Sanmi Koyejo; Yanzhe Zhang

REVIEW 2 major objections 6 minor 13 references

The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context

T0 review · 2 major / 6 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read State-of-the-art models look robust to irrelevant context in aggregate, but the stability is illusory: meaningless pseudo-words flip individual answers by tens of points in both directions, and the affected questions are largely model-speci

desk verdict Careful empirical study showing two-sided per-example flips under irrelevant context; the single unvalidated LLM grader is the main soft spot, but independent local-model results and hand-checkable examples make the claim hold. read the letter →

arxiv 2607.12963 v2 pith:UF4XPNZ4 submitted 2026-07-14 cs.CL

Yanzhe Zhang , Sanmi Koyejo , Diyi Yang This is my paper · ORCID

classification cs.CL

keywords LLMrobustnesstask-irrelevantcontextper-exampleinstabilityaggregateaccuracypredictionflipsworst-taildegradationmodelspecificityevaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that state-of-the-art language models are not robust to task-irrelevant context in the way their aggregate accuracy suggests. Prepending a block of meaningless pseudo-words to a benchmark question leaves overall accuracy nearly unchanged, yet individual questions swing sharply in both directions — some become much harder, others much easier — so the near-zero mean is a cancellation, not a sign of stability. The swings are large enough to matter in deployment: per-example instability reaches 13.6 percentage points and worst-tail degradation reaches 53.2 after noise adjustment. Crucially, the questions that flip are not a property of the data: different models flip different questions, with cross-model Pearson correlation 0.00 and tail overlap barely above random, which makes the failures hard to anticipate. That is why the paper argues for per-example reliability evaluation instead of headline accuracy.

What carries the argument

The load-bearing objects are the two metrics from Section 2: Instability (INS), the mean absolute per-question accuracy change after adding context, and Worst-tail Degradation (WTD), the average accuracy loss in the worst-affected 10% of questions. Because both are estimated from 20 sampled responses per question per condition, the paper adds a bootstrap noise-floor correction — resampling only the no-context trials — to estimate the sampling-induced positive bias and subtract it. The argument runs on the contrast between the near-zero mean change and the large tail magnitudes, plus the near-zero cross-model correlation of per-question changes, which rules out a purely data-driven explanatio

What would settle it

Re-run MMLU-Pro with five times the sample budget — 100 no-context and 100 context responses per question, graded by two independent graders — and recompute the noise-adjusted INS, WTD, and cross-model correlations. If the adjusted metrics collapse toward zero, or the cross-model Pearson correlation rises well above 0.1 once attenuation is corrected, then the measured instability is mostly sampling noise; if the numbers hold, the two-sided flips and model-specificity are confirmed.

Watch

Extended reading notes

Core claim

Central claim: context-induced instability is real and two-sided. Prepending task-irrelevant text changes a question's chance of being answered correctly; the changes are large on a small tail of questions while averaging to near zero. The paper defines Instability (INS), mean absolute per-example accuracy change, and Worst-tail Degradation (WTD), average drop in the worst 10% of questions, then subtracts a bootstrap noise floor. Across four benchmarks and eight models, INS reaches 13.6% and WTD 53.2% while mean accuracy moves only a few points. Per-example shifts are uncorrelated across models (Pearson r = 0.00 on MMLU-Pro), so affected questions belong to the model, not the data.

Load-bearing premise

Every per-question change score rests on 40 sampled answers (20 per condition) judged by a single automated grader, and the bootstrap correction removes only the mean null bias, not per-example noise; if residual noise is larger than assumed, the reported magnitudes and the model-specificity result shrink.

Editorial extensions

If this is right

Evaluation that reports only aggregate accuracy keeps missing tail failures: on the worst 10% of questions a model can lose more than half its accuracy while the mean moves by under a point.
Because the affected questions differ sharply between models, a context-robustness fix validated on one model cannot be assumed to transfer to another.
The effect appears even with meaningless pseudo-word context, so it is not retrieval: the model is not being misled by real information in the context.
More test-time compute — higher reasoning effort or chain-of-thought prompting — shrinks but does not eliminate instability, so paying for extra tokens is only a partial mitigation.
A model's baseline confidence on a question predicts how much that question's answer will move under context, but not whether the move helps or hurts; question ambiguity predicts neither.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If model specificity holds under realistic long-context deployments, per-example flips could serve as a cheap behavioral fingerprint: models with similar averages that flip different questions are functionally distinct in ways standard leaderboards miss.
The uncertainty–magnitude link suggests an operational early-warning: flag high-uncertainty questions whenever long context is added, since those are the answers most likely to move — but the flag is directionless, so it cannot tell a deployer whether the move will be good or bad.
A natural next experiment the paper leaves implicit: test whether flips under irrelevant context correlate with flips under innocuous prompt rephrasing; if they do, prompt sensitivity and context sensitivity are one underlying instability, and mitigation work can be unified.
Targeting worst-tail degradation directly as a training objective, rather than average loss, could be a more direct route to context reliability than scaling test-time reasoning — a hypothesis the paper's checkpoint results make testable.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper makes one point that matters: aggregate accuracy can be flat while individual predictions flip in both directions. The authors measure this carefully across eight models and four benchmarks, and their statistical hygiene is genuinely above the norm for this literature. Bootstrap noise floors, split-sample WTD, split-half reliability, disjoint-fold correlation estimators—they clearly thought about what could fake their numbers and built controls for it. The model-specificity result is the new thing: per-example flips are essentially uncorrelated across models (r=0.00 on MMLU-Pro), which means you can't predict which examples will flip by looking at the data. I found the development-stage finding—SFT/DPO increase instability even as accuracy improves—surprising and worth follow-up.

The soft spot is the grader. All API-model measurements rest on a single automated judge (gpt-5.4-mini, low reasoning), with no human validation or agreement stats. Split-half reliability shows the judge is reproducible across trial splits, but reproducibility isn't validity. If the judge grades systematically differently when responses happen after long pseudo-word context, that could inflate INS/WTD and, because the same judge sees all models, interact with model-specific response styles to fake model-specificity. The authors give no direct evidence against that. That said, I don't think it sinks the paper. The local-model experiments on MMLU measure correctness with option probabilities rather than the LLM judge, and they show the same kind of two-sided flips. The qualitative examples in the appendix contain simple arithmetic errors (11+9=19) that a human reader can verify. And the effects are large—WTD above 50% for Mistral—so the grader would need to be systematically and heavily biased to produce them. Still, a revision should add a human-annotated subset or a second grader, and show agreement. That's the difference between 'likely true' and 'nailed.'

The mathematical 'gap' between INS and |mean Δ| is guaranteed by the triangle inequality; the contribution is the empirical magnitude and the model-specificity, so the circularity charge is a distraction.

This is a solid empirical paper, well above the threshold for a serious referee. I'd send it to review and expect minor-to-moderate revisions focused on the grader question. It will be cited.

Referee Report

2 major / 6 minor

Summary. The paper studies whether LLM accuracy on QA benchmarks is robust to task-irrelevant context. Using 10k-token pseudo-word contexts prepended to questions from MMLU-Pro, GPQA, HLE, and SimpleQA, it estimates per-example correctness change Δ_i from 20 repeated baseline trials and 20 repeated context-condition trials for eight models. Two metrics are introduced: Instability (INS), the average |Δ_i|, and Worst-tail Degradation (WTD), the mean loss in the worst 10% of examples, both adjusted by a bootstrap noise floor computed from no-context trials alone (Section 2). The central finding is that aggregate accuracy change Δ is small (mostly ±2%) while INS reaches 13.6% and WTD 53.2%; per-example changes concentrate in the tails (88.3% of total absolute change in the two outer deciles), are two-sided, and are largely model-specific (mean cross-model Pearson r ≈ 0.00; Jaccard 0.09 vs. the 0.05 random baseline). Question ambiguity is uncorrelated with change; baseline uncertainty correlates with change magnitude but not direction (split-sample ρ ≈ 0.21). Ablations cover context type, context length, test-time compute, and model development stage.

Significance. If the measurement is valid, the paper is a solid empirical contribution: it turns a diffuse worry about distracting context into a quantified, per-example phenomenon with two simple metrics, and the model-specificity result is practically important (failures cannot be predicted from the data alone). The statistical apparatus is well above the norm: bootstrap noise floors from 1000 null experiments, split-sample WTD with disjoint selection/measurement trials, split-half reliability with attenuation ceilings and Spearman-Brown correction, disjoint-fold correlation estimators, a homogeneous-shift null simulation, and released code/data. These strengthen the reproducibility of the per-example signal. The principal caveat is that all labels come from one unvalidated LLM judge, so the magnitudes and the model-specificity finding are conditional on grader validity; the phenomenon itself is corroborated by the worked examples in Appendix I. With a grader-validation study added, the paper would be a strong candidate for publication.

major comments (2)

[§3 (Experiment Details); Appendix D (Table 7); Appendix H.2] Every quantitative claim in the paper — Table 1 INS/WTD magnitudes, the 88.3% tail concentration (§3.1), cross-model r≈0.00 and Jaccard 0.09 (§3.2), and Table 3's correlations — is computed from binary correctness labels produced by a single automated judge, gpt-5.4-mini at low reasoning. No human validation, inter-annotator agreement, or comparison with the datasets' official/exact-match grading is reported, and Table 2's taxonomy is itself LLM-annotated. The split-half reliability (Table 7) establishes reproducibility of the judge's labels, not validity: a judge that systematically mis-grades responses whose style changes under long contexts would yield reproducible, context-correlated artifacts. Because one judge grades all models, judge errors interacting with model-specific response styles could also manufacture the appearance of model specificity. The required assumption — grading
[§3.3, Figure 6; Table 4 (right)] The development-stage conclusions ('instability emerges during pretraining'; 'SFT substantially increases instability'; 'post-training amplifies INS/WTD') do not control for the room-to-change confound the paper itself flags in §3.1: INS and WTD are bounded by baseline accuracy. As the OLMo checkpoints' AVG rises from 25.4% to 47.4%, part of the WTD increase (17.7→22.1) could be mechanical. Likewise, gemma-4-12B → gemma-4-12B-it shows AVG 58.5→78.5 with WTD 24.5→40.7; higher AVG means more probability mass available for degradation, making the 'amplification' reading ambiguous. A baseline-accuracy-stratified or log-odds-scale version of Figure 6/Table 4 is needed to separate genuine growth in instability from the mechanical effect of increasing room to change. The SFT/DPO comparisons, where AVG changes little while INS/WTD jump, are the least affected and do support the claim.

minor comments (6)

[Table 1 vs. Table 6] The headline INS/WTD values are noise-adjusted, but Table 1 reports them without uncertainty. Table 6 gives CIs only for the unadjusted estimates and the floors. Since the floor is an average over 1000 bootstrap trials its own variance is small, but the paper should state explicitly that adjusted CIs are obtained by shifting the Table 6 CIs by the floor, or report adjusted CIs directly, so readers can judge whether e.g. gpt-5.5's HLE WTD of 4.3 (observed 16.9, floor 12.6) is distinguishable from zero.
[§2; §3.3] INS/WTD are conditional on the particular fixed set of M=10 contexts and K=10; the bootstrap null resamples trials but not contexts, so context-draw variability is not part of the noise floor. A short sensitivity analysis varying M and K, or an explicit statement that the magnitudes are specific to the chosen context sample, would clarify the scope of the headline numbers.
[§3.1, Table 2; Appendix C] The qualitative error taxonomy is generated by another LLM (gpt-5.5) without human validation. The claim that improved and degraded examples follow different error distributions should be flagged as exploratory, or the annotation should be validated on a sample.
[§1; §2] Framing: the fact that INS can be large while Δ≈0 is a mathematical consequence of averaging absolute values (triangle inequality), so the abstract's 'masks' framing should make explicit that the contribution is the empirical magnitude, two-sidedness, reproducibility, and model-specificity of the flips, not the existence of a gap between INS and |Δ|.
[Tables 1 and 4 headers; §3 experiment details] Presentation: the header 'A VG∆INS ↓ WTD↓' appears to be a formatting artifact (should read 'AVG | Δ | INS | WTD'). Also, the sentence in §3 that contexts are 'prepended to the question as an extra user message between the question and the system message' is ambiguous about the final message order; Appendix H.1 should specify the exact sequence (system → context → question?) for reproducibility.
[Table 3] Table 3 pools n=1690 questions across four benchmarks, whereas §3.2 reports per-dataset correlations. Pooling mixes datasets with very different characteristics (e.g., HLE floor effects); reporting the Table 3 correlations per dataset would make the 'consistently uncorrelated' claim easier to verify.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the core INS/WTD results are empirical measurements supported by noise corrections and split-half reliability, not consequences of the metric definitions alone.

full rationale

The paper's derivation chain is: define latent correctness probabilities s_ij; estimate them by repeated sampling; define per-example change Δ_i, INS = mean |Δ_i|, and WTD as worst-tail degradation; apply a bootstrap noise-floor correction; then report adjusted magnitudes, tail concentration, cross-model correlations, and correlates. No load-bearing step is equivalent to its inputs by construction. It is true that INS ≥ |mean Δ_i| follows from the triangle inequality, but the paper does not present that inequality as a derived empirical finding; the empirical content is that the measured INS values are large (up to 13.6%), that per-example changes concentrate in the tails, that the effect is two-sided, and that affected examples are model-specific. Those claims are supported by the noise-corrected estimates, split-sample WTD (Appendix A), and randomized split-half reliability (Appendix D), not by the definitions themselves. The bootstrap adjustment is explicitly described as a noise-floor correction rather than an unbiased estimator, and the split-sample estimator separates selection trials from measurement trials, so no fitted quantity is renamed as a prediction. The single automated grader concern is a measurement-validity issue—grader bias correlated with context-induced response-style changes could affect magnitudes and model-specificity—but that is not circularity: the estimates could be inaccurate without the derivation reducing to its inputs. No self-citation is load-bearing; citations to prior work are contextual background rather than the justification for the central claim. Therefore no specific circular step can be exhibited, and the appropriate score is 0.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

This is a measurement paper, so the ledger is dominated by experimental hyperparameters and measurement premises rather than fitted model parameters. The reported magnitudes (INS up to 13.6%, WTD up to 53.2%) all pass through per-example Δ_i estimates, the choice of K=10, M=10, 10k-token contexts, and the gpt-5.4-mini grader. No new physical or theoretical entities are postulated; INS and WTD are new descriptive statistics, not entities with independent evidence requirements.

free parameters (6)

K (WTD tail percentile) = 10
Hand-chosen tail size; WTD magnitudes and the headline 'worst 10%' framing scale with this choice. The paper does not sweep K.
M (number of contexts per example) = 10
Per-example change is averaged over 10 fixed pseudo-word contexts; the specific context set shapes which examples flip and the INS/WTD magnitudes, yet is treated as representative of the broad class 'task-irrelevant context.'
Sampling trials (p, q) = p=20 baseline, q=2 per context
Per-example Δ_i noise scales inversely with trial count; q=2 per context is small. The bootstrap noise floors, split-half reliabilities, and attenuation corrections all depend on these counts.
Default context length = 10k tokens
Context length strongly modulates INS/WTD and changes which questions are affected (r=0.08 between 10-token and 10k-token contexts, Section 3.3), so the headline magnitudes are specific to the 10k-token configuration.
Pseudo-word vocabulary parameters = Zipf frequency threshold 1.5, vocabulary size 10000, word length 3–8 chars
Defines the task-irrelevant context distribution and the 'meaningless' claim; the English-frequency filter is the operational definition of task-irrelevance.
Automated grader = gpt-5.4-mini (low reasoning)
Every binary correctness label — and hence every Δ_i, INS, and WTD — passes through one model's extraction and grading judgment. Grader noise or context-dependent grading drift would propagate into all metrics.

assumptions (5)

domain assumption Pseudo-word contexts (random lowercase strings filtered by English frequency) are task-irrelevant and contain no task-relevant evidence.
Section 2: filtering 'minimizes recognizable lexical or factual content.' If these strings carry hidden tokenizer, distributional, or instruction-following cues that interact with questions, the effect is not purely one of task-irrelevant content. Partially mitigated by naturalistic context ablations (TOK, WEB, SQA) in Section 3.3.
domain assumption Binary correctness averaged over repeated samples is a valid estimate of a latent per-example correctness probability s_ij, and the automated grader is unbiased.
Section 2, Score Estimation. The grader extraction rules (final committed answer only) are applied by gpt-5.4-mini; whether grading behavior is symmetric across baseline and long-context conditions is not directly validated.
standard math Under the null hypothesis, context-condition trials are exchangeable with no-context trials, so resampling no-context samples yields a valid noise floor for INS and WTD.
Section 2, Noise Correction. The authors themselves note the adjustment 'should be interpreted as a noise-floor correction rather than an unbiased estimator'; the split-sample WTD (Appendix A) is the supporting check.
domain assumption The evaluation configuration (context prepended as an extra user message, no-reasoning or lowest reasoning effort) is the appropriate lens for measuring context-induced instability.
Section 3 and Figure 5 (Right) show reasoning effort strongly modulates INS/WTD; the main results therefore characterize a specific compute regime, and qualitative conclusions could shift under higher reasoning settings.
standard math Per-example change vectors Δ_i are stable model-level traits, so cross-model correlations and Jaccard overlaps are interpretable after correcting for attenuation.
Section 3.2 and Appendix D: corrected reliability is ≥0.75 on MMLU-Pro, but raw split-half reliability is as low as 0.22–0.35 on HLE/SimpleQA, so the model-specificity conclusion is stronger on multiple-choice benchmarks than on open-ended ones.

how reviews work

0 comments

Cite this review

Pith. "Pith review of The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context." pith.science (2026). https://pith.science/paper/UF4XPNZ4

@misc{pith2026260712963,
  author       = {Pith},
  title        = {Pith review of: The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UF4XPNZ4}},
  note         = {Machine review of arXiv:2607.12963}
}

read the original abstract

As large language models (LLMs) grow more capable, they are increasingly deployed in context-rich settings where task inputs are often accompanied by long, partially irrelevant context. In a controlled setting, we find that state-of-the-art models often appear robust to task-irrelevant context at the aggregate level: prepending it to benchmark questions causes little change in overall accuracy. This aggregate stability, however, masks significant per-example instability. Even semantically meaningless pseudo-words, formed by randomly combining characters, can markedly shift model predictions on a small fraction of examples, degrading performance on some while improving it on others. This two-sided effect holds consistently across a wide range of models and datasets, yet the affected examples are largely model-specific. We further show that this instability is modulated by context type, context length, test-time compute, and model development stage. Together, our findings reveal context-induced tail risks concealed by aggregate accuracy, motivating per-example reliability evaluation of language models.

Figures

Figures reproduced from arXiv: 2607.12963 by the authors.

**Figure 1.** Context-induced instability is a two-sided effect. We show two MMLU-Pro examples where task-irrelevant context respectively improves and degrades gpt-5.4’s responses by affecting commonsense and calculation. Full examples are in Appendix I. absolute change in performance across all examples, capturing how sensitive a model is to irrelevant context regardless of whether the change helps or hurts. Worst-tail Degradati… view at source ↗

**Figure 2.** Per-example performance change concentrates in the tails (MMLU-Pro). For each model, we rank its examples from most improved to most degraded by the per-example change ∆i , and keep the two 10% tails: the most-improved examples (top panel) and the most-degraded examples (bottom panel). Each tail is split into 2%-wide percentile bins (x-axis), and each bar reports the mean ∆i in that bin for one model. Solid bars sho… view at source ↗

**Figure 3.** Per-example changes are uncorrelated across models. We report Pearson r of perexample performance change for all model pairs on MMLU-Pro and SimpleQA. Raw Split-sample Feature Target Probability Logit Probability Logit Ambiguity Signed −0.01 [−0.06, +0.04] −0.01 [−0.05, +0.04] −0.01 [−0.05, +0.03] −0.01 [−0.05, +0.04] Squared +0.02 [−0.03, +0.06] +0.02 [−0.03, +0.06] +0.02 [−0.02, +0.05] +0.02 [−0.02, +0.05] Uncert… view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Context-induced instability holds across context types. We show INS, WTD, and the correlation of per-example performance change between different context types. SQA refers to sequential question answering, WEB refers to random webpages, TOK refers to random tokens, and…

**Figure 5.** Figure 5: Scaling effects of context length and reasoning effort. We vary context length (10, 100, 1k, and 10k) and reasoning effort (none, low, medium, high) on MMLU-Pro. constructed by randomly sampling tokens from the gpt-oss-120b tokenizer vocabulary. Second, Random Webpages…

**Figure 6.** Figure 6: Context-induced instability emerges across the training pipeline. We track INS and WTD on MMLU for Olmo-3-7B-Instruct at checkpoints (I) after each training stage, (II) during pretraining, and (III) during midtraining. AVG ∆ INS↓ WTD↓ AVG ∆ INS↓ WTD↓ Qwen3.5-9B-Base 72…

**Figure 7.** Figure 7: The correlation of per-example performance change between all models on GPQA and [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]

**Figure 8.** Figure 8: (Top) The correlation of per-example performance change between Olmo-3 checkpoints [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 2 linked inside Pith

[1]

Did it effectively answer a different problem than posed — misreading the question, or adding/relaxing a constraint? -> A
[2]

Is the fault a false factual claim — a wrong recalled fact/entity/attribute, or invented supporting detail — rather than a reasoning error? -> B
[3]

Is the method or reasoning itself wrong — a wrong/overgeneralized rule or concept, or an invalid inference chaining otherwise-right facts and rules (not merely a mechanical slip)? -> C. 16 Response Tokens Reasoning Tokens Model A VG∆Baseline Context Baseline Context gpt-oss-120b (low)77.4−0.6 188 194 160 154 gpt-oss-120b (mid)79.5 +0.4 201 202 443 405 gpt...
[4]

Was the method right but slipped in execution — an arithmetic/numeric error, a bookkeeping/representation slip, a miscount, or picking the wrong final option after a correct derivation? -> D
[5]

Covariance

Otherwise (none clearly applies, or too little information to tell) -> E. ## Output format First output the label line: the code, an em dash, and its short name — e.g.`B — Wrong Fact`. Then, on the next line, give a brief one-sentence explanation of why this code applies to the specific failure. Do not list multiple codes or write anything beyond the labe...
[8]

Extract the single option the response finally commits to
[9]

Finding the final answer: - A response may mention, weigh, or even initially state several options before settling on one

Decide whether that option matches the ground truth. Finding the final answer: - A response may mention, weigh, or even initially state several options before settling on one. Extract only the option it ultimately commits to — normally its concluding statement — and ignore options it stated earlier but then moved away from. - Use'None'only when the respon...
[10]

Extract the final exact answer the response commits to

Show all 13 references

[11]

Finding the final answer: - A response may explore, revise, or initially state different answers before settling on one

Decide whether that answer matches the ground truth. Finding the final answer: - A response may explore, revise, or initially state different answers before settling on one. Extract only the answer it ultimately commits to — normally its concluding statement — and ignore answe...

2018
[12]

**Paris Saint-Germain**
[13]

Response (with context, retrieved the wrong fact) Napoli came second in Group C of the 2018–19 UEFA Champions League

**Red Star Belgrade** 26 Liverpool finished second behind PSG and advanced to the knockout stage. Response (with context, retrieved the wrong fact) Napoli came second in Group C of the 2018–19 UEFA Champions League. Brief explanation: In Group C, Paris Saint-Germain finished f...

2018
[2021]

URLhttps://openreview.net/forum?id=d7KBjmI3GmQ. C.-P. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, and B. Ginsburg. RULER: What’s the real context size of your long-context language models? InFirst Conference on Language Modeling, 2024. URLhttps://openreview.net/fo...

2024 arXiv
[2024]

URLhttps://openreview.net/forum?id=Ti67584b98. M. T. Ribeiro, T. Wu, C. Guestrin, and S. Singh. Beyond accuracy: Behavioral testing of NLP models with CheckList. In D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, editors,Proceedings of the 58th Annual Meeting of the Assoc...

2020 arXiv

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.

[1] [1]

Did it effectively answer a different problem than posed — misreading the question, or adding/relaxing a constraint? -> A

[2] [2]

Is the fault a false factual claim — a wrong recalled fact/entity/attribute, or invented supporting detail — rather than a reasoning error? -> B

[3] [3]

Is the method or reasoning itself wrong — a wrong/overgeneralized rule or concept, or an invalid inference chaining otherwise-right facts and rules (not merely a mechanical slip)? -> C. 16 Response Tokens Reasoning Tokens Model A VG∆Baseline Context Baseline Context gpt-oss-120b (low)77.4−0.6 188 194 160 154 gpt-oss-120b (mid)79.5 +0.4 201 202 443 405 gpt...

[4] [4]

Was the method right but slipped in execution — an arithmetic/numeric error, a bookkeeping/representation slip, a miscount, or picking the wrong final option after a correct derivation? -> D

[5] [5]

Covariance

Otherwise (none clearly applies, or too little information to tell) -> E. ## Output format First output the label line: the code, an em dash, and its short name — e.g.`B — Wrong Fact`. Then, on the next line, give a brief one-sentence explanation of why this code applies to the specific failure. Do not list multiple codes or write anything beyond the labe...

[6] [8]

Extract the single option the response finally commits to

[7] [9]

Finding the final answer: - A response may mention, weigh, or even initially state several options before settling on one

Decide whether that option matches the ground truth. Finding the final answer: - A response may mention, weigh, or even initially state several options before settling on one. Extract only the option it ultimately commits to — normally its concluding statement — and ignore options it stated earlier but then moved away from. - Use'None'only when the respon...

[8] [10]

Extract the final exact answer the response commits to