Pith. sign in

REVIEW 4 major objections 4 minor 18 references

Benchmark datasets are not monolithic: sample-level metadata along five dimensions lets evaluators re-compose items into targeted subsets that reveal capability differences hidden by aggregate accuracy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 00:19 UTC pith:AUKWHIXQ

load-bearing objection A solid audit framework with a useful indicator catalogue, but the central validation is overstated to the point that the orchestration results rest on unverified metadata. the 4 major comments →

arxiv 2607.28801 v1 pith:AUKWHIXQ submitted 2026-07-30 cs.CL cs.AIcs.LG

Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation

Philipp D. Siedler , Jordan Sassoon This is my paper
classification cs.CL cs.AIcs.LG
keywords benchmark evaluationsample-level annotationLLM-as-a-judgebenchmark orchestrationmeta-evaluationdataset auditreasoning depthethical sensitivity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that treating a benchmark as a single task hides large variation in what individual items demand from a model. It proposes auditing every sample along five dimensions—cognitive demands, language quality, task properties, context, and ethics/safety—and annotates five influential benchmarks item by item using a large language model as judge. These annotations reveal that datasets like MMLU, ARC, HellaSwag, WinoGrande, and TruthfulQA each mix very different levels of reasoning depth, ambiguity, bias, and safety relevance. The paper then shows that filtering items by indicator constraints produces composite evaluation subsets that isolate specific capabilities, and accuracy on those slices can differ sharply from overall scores. A sympathetic reader would take the central claim to be that benchmark evaluation should be dataset introspection, not a single number.

Core claim

On the paper's own terms, the central discovery is that benchmark accuracy is an aggregate over samples that differ widely on latent dimensions, and those dimensions can be made explicit and actionable. By annotating every sample with a structured set of indicators—reasoning depth, distractor quality, misinformation bait, safety-critical relevance, and others—the authors show that each benchmark has a distinct profile: MMLU emerges as knowledge-heavy with many multi-step items, ARC as shallow scientific recall, TruthfulQA as the main carrier of ethical and misinformation signals. Using these annotations, they construct single- and multi-indicator subsets across datasets and show that model p

What carries the argument

The load-bearing device is the Catalogue of Criteria, a three-level hierarchy of Dimensions, Aspects, and Indicators that turns latent sample properties into 26 concrete ordinal or categorical variables with explicit scale semantics. It is operationalized through an LLM-as-a-judge protocol: one unified prompt asks a judge model to rate each item on all indicators and return structured JSON. The third component is orchestration: specifying constraints over indicators—such as reasoning depth at least 2, ambiguity at least 2, and safety-critical relevance—to re-sample composite subsets drawn across datasets. The catalogue defines the vocabulary, the judge supplies the metadata, and orchestratio

Load-bearing premise

The pipeline assumes the judge models' per-sample annotations are accurate enough to serve as ground truth for filtering, yet the paper's own human-agreement data show near-zero correlations for several indicators and only about 21 percent recall on safety-critical items, so mislabeled samples would propagate into the orchestrated subsets.

What would settle it

Re-run orchestration on the 100 human-labeled samples per benchmark using human annotations instead of judge annotations; if the selected subsets and resulting model accuracies diverge materially, or if safety-critical slices change substantially, the orchestration rests on unreliable judge metadata.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Accuracy on orchestrated subsets can differ sharply from full-benchmark accuracy, so rankings based on aggregate scores may not reflect a model's handling of high-reasoning, high-ambiguity, or safety-critical items.
  • Existing benchmarks can be re-composed across datasets to isolate a targeted capability such as Reasoning Depth or Ethical Sensitivity without collecting new items.
  • The annotated resource allows fine-grained diagnosis of model failure modes, for example where ambiguity, bias, or misinformation bait drives errors.
  • Current leaderboards may overstate reliability by averaging over large numbers of undemanding samples; filtered slices provide a check.
  • The framework supports adaptive probing: as models improve, evaluators can shift focus to slices that still discriminate between them.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the judge labels are only as trustworthy as the human-agreement study suggests—several indicators near zero correlation and safety-critical recall around 21 percent—orchestrated subsets should be treated as exploratory hypotheses rather than definitive measurements until the labels are independently verified.
  • The framework implicitly suggests a calibration protocol: orchestrated slices could be validated by comparing model rankings on judge-annotated slices versus human-annotated slices for the same indicators, which the paper does not run.
  • The same catalogue could be extended to multilingual or domain-specific benchmarks, but scale semantics such as cultural framing and age appropriateness would need local adaptation rather than direct transfer.
  • Because the metadata is a static snapshot, time-sensitive indicators such as factual accuracy and leakage risk will decay, so periodic re-auditing would be required to keep orchestrated slices meaningful.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces a sample-level meta-evaluation framework that annotates benchmark items (MMLU, ARC, HellaSwag, WinoGrande, TruthfulQA) with 26 indicators across five latent dimensions, using LLM-as-a-judge models (GPT-5, DeepSeek-V3.1, DeepSeek-R1). It then uses these annotations for 'orchestration': filtering and combining items across benchmarks into targeted evaluation subsets (e.g., high reasoning depth, high-stakes ambiguity), and reports accuracy of two small models on these subsets. The paper claims that aggregate benchmark scores conceal internal heterogeneity and that criterion-driven orchestration enables fine-grained, interpretable comparisons.

Significance. The idea of treating benchmark datasets as non-monolithic, sample-level resources is potentially valuable and aligns with existing work on dataset audits and behavioral testing. If the annotations were reliable, the public annotated resource could support new evaluation designs. The paper also provides a reproducible pipeline (Aleph Alpha Eval-Framework, deterministic decoding) and a detailed catalogue. However, the core contribution rests on the validity of the LLM-generated metadata, and the paper's own validation data do not support the needed reliability. The contradiction between the reported high human-model agreement and the actual correlation/confusion values is load-bearing, not cosmetic.

major comments (4)
  1. [§5.3 and Figure 2] The text states that Reasoning Depth, Fact Recall, and Language Difficulty show 'high Spearman human-model correlation coefficients (ranging from 0.74 to 0.83)', and that Safety Critical and Factual Accuracy achieve 'near-perfect alignment'. Figure 2 (main text and Appendix G.1) directly contradicts this: for GPT-5, reasoning_depth=0.27, fact_recall=0.66, language_difficulty=0.42; for DeepSeek-V3.1, 0.26/0.57/0.48; for DeepSeek-R1, 0.24/0.60/0.56. Several indicators used in orchestration are near zero or negative for GPT-5 (referential_clarity=-0.06, ambiguity_level=-0.06, narrative_understanding=-0.06, distractor_quality=0.12, spelling=0.13). The figure caption also claims 'strong alignment', so the discrepancy is not a typo in one number but a systematic mismatch between the reported validation and the actual data.
  2. [Appendix G.9 (GPT-5 confusion matrix, safety_critical)] The safety-critical confusion matrix shows that among items humans labeled True, GPT-5 labeled only 21% as True (recall=0.21). DeepSeek-V3.1 recall is 0.36 and DeepSeek-R1 recall is 0.14. This is the opposite of 'near-perfect alignment' claimed in §5.3. Because Table 4's 'High-stakes reasoning under ambiguity' subset is defined by safety_critical=True, nearly four-fifths of the items in that subset would be mislabeled by GPT-5. The orchestrated subset's composition is therefore not what the specification claims, and the resulting accuracy differences cannot be attributed to safety-critical reasoning.
  3. [§4.4 / Table 4 reliance on unvalidated metadata] The orchestration premise is that filtering by indicator constraints yields subsets that isolate the intended capability. The validation evidence shows that many indicators have very low human-model agreement, both in terms of Spearman correlation and in confusion-matrix recall/specificity for rare categories. Since subset construction (Table 4) uses indicators like reasoning_depth, ambiguity_level, bias_stereotyping, distractor_quality, narrative_understanding, and safety_critical, the subsets are built on labels that the paper's own data show to be unreliable. Without reliable labels, the reported performance differences (e.g., Llama 3.2 1B dropping from 0.425 to 0.252 on high-reasoning subsets) may simply reflect which items the judge happened to call 'high reasoning', not a genuine capability gap.
  4. [§7 Limitations / circularity] The Limitations section acknowledges evaluator model bias and the risk of circular dependency if metadata are used in training, but it does not confront the immediate consequence for the paper's own evaluation results: the annotations are produced by the same class of model family (LLM judges) and are then used to define evaluation slices. The paper does not provide any independent validation that would break this circularity, and the weak human agreement for several core indicators makes the 'latent dimensions' largely the judges' constructs.
minor comments (4)
  1. [Appendix G, figures] Several figure captions are placeholders reading 'Enter Caption' (e.g., Figures 5–13, 16–17, 20–21, 24–25, 28–29, 32–33). These should be filled in before any resubmission.
  2. [§4.3] Grammar: 'This samples have been picked uniformly across all subjects' should be 'These samples were chosen uniformly across all subjects.'
  3. [Table 1 vs Table 4] Table 1 lists TruthfulQA sample count as 817, but Table 4 reports TruthfulQA count 1634 for 'All Samples'. The text mentions 'TruthfulQA (MC 1&2)' in §5.1; this should be made explicit in Table 1 to avoid confusion.
  4. [References] The reference for 'Scruples' cites Scarselli et al. (2009), which is a graph neural network paper; the Scruples dataset is a different work (Lourie et al., 2021). The citation appears to be incorrect.

Circularity Check

0 steps flagged

No significant circularity; the orchestration pipeline is self-contained, though the human-agreement validation is internally contradicted (a validity issue, not circularity).

full rationale

The central pipeline is: define an indicator catalogue, annotate samples with an LLM-as-a-judge protocol, filter samples by indicator constraints, then measure held-out model accuracy on the filtered subsets. No equation or fitted parameter is reused as a prediction: model accuracy is computed independently of the judge-generated metadata, and the indicator labels are not defined in terms of model performance. Thus the core claim — that accuracy differs across orchestrated subsets — does not reduce to its inputs by construction. The paper's own Limitations section flags a possible "circular dependency if the framework is integrated into the training loop," but that is an explicitly forward-looking risk, not a circularity in the present evaluation-only claims. The self-citation of the Aleph Alpha Eval-Framework is a code-repository reference and is not load-bearing for any scientific premise. A separate and serious problem is that Section 5.3 asserts "near-perfect alignment" for Safety Critical and reports "strong alignment" overall, while Appendix G.1 and G.9 show GPT-5 reasoning_depth agreement of 0.27, several near-zero or negative correlations (e.g., referential_clarity = -0.06, ambiguity_level = -0.06), and a safety_critical confusion matrix in which only 21% of human-True items are labeled True by the model. This internal contradiction undermines the reliability of the metadata used for orchestration, but unreliable labels are a validity/measurement-error concern, not circular reasoning. The subsets may be mislabeled, yet the comparison of model accuracy across subsets is still an empirical conditional comparison on the (possibly incorrect) metadata. Section 6.3's statement that PCA "latent axes correspond closely to the Indicator dimensions used for orchestration" is self-referential in the weak sense that PCA is computed on the very indicators being orchestrated, but it is not load-bearing: the orchestration demonstration in Section 4.4 and Table 4 does not depend on the PCA. Accordingly, no specific derivation step reduces to its own input, so the circularity score is 0.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

The paper introduces no physical entities. Its load-bearing assumptions are the reliability of LLM-as-a-judge annotations, the adequacy of the catalogue, and the validity of the macro-average comparison. The first is contradicted by the paper's own data; the third is methodologically flawed.

axioms (3)
  • domain assumption LLM-as-a-judge annotations are reliable proxies for human metadata labels.
    The entire orchestration depends on the accuracy of these labels; Section 5.3 claims strong agreement, but Figures 2 and G.7–G.9 show weak agreement on many indicators.
  • ad hoc to paper The 26-indicator catalogue adequately spans the latent dimensions of benchmark items.
    The dimensions were chosen by the authors; no external validation or completeness argument is provided.
  • domain assumption Macro-average accuracy across benchmarks is an appropriate summary for comparing orchestrated subsets.
    Table 4 uses an unweighted mean across datasets, giving equal weight to small and large datasets, which confounds subset selection with benchmark reweighting.

pith-pipeline@v1.3.0-alltime-deepseek · 2643 in / 2697 out tokens · 106817 ms · 2026-08-03T00:19:59.853949+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation." pith.science (2026). https://pith.science/paper/AUKWHIXQ

@misc{pith2026260728801,
  author       = {Pith},
  title        = {Pith review of: Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AUKWHIXQ}},
  note         = {Machine review of arXiv:2607.28801}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Benchmark datasets are central to evaluating Large Language Models (LLMs), yet they are typically conceived as monolithic tasks, obscuring substantial variation in the demands of individual samples. We introduce a dataset-centric meta-evaluation framework that audits benchmark datasets at the sample level along five latent dimensions: 1. Cognitive and Knowledge Demands, 2. Language and Content Quality, 3. Task Properties, 4. Context, and 5. Ethics, Safety, and Fairness. Applying this framework, we annotate five influential benchmarks -- MMLU, ARC, WinoGrande, HellaSwag, and TruthfulQA -- revealing pronounced internal heterogeneity that is not captured by aggregate accuracy scores. We show how these annotations enable criterion-driven orchestration of composite benchmark subsets across datasets, supporting targeted evaluation of model capabilities such as Reasoning Depth or Ethical Sensitivity. This approach reframes benchmark evaluation as dataset introspection, providing a principled methodology for analyzing and re-composing existing benchmarks to better reflect diverse evaluation needs.

Figures

Figures reproduced from arXiv: 2607.28801 by Jordan Sassoon, Philipp D. Siedler.

Figure 2
Figure 2. Figure 2: Human-model agreement for sample-level indicator annotations across all benchmarks. Spearman correlations show strong alignment between human annotations and evaluator models, particularly for structurally grounded indicators (e.g. Reasoning Depth, Factual Accuracy), supporting the use of LLM-as-a-judge for scalable benchmark auditing. sensitive subsets. These results demonstrate that the LLM￾as-a-judge fr… view at source ↗
Figure 3
Figure 3. Figure 3: Principal Component Analysis (PCA) of the sample-level indicator space across all benchmarks. The first principal com￾ponents separate samples dominated by specialized knowledge and fact recall from those emphasizing narrative understanding and commonsense reasoning. This structure reveals latent evalua￾tion dimensions that cut across benchmark boundaries, motivating dynamic orchestration of benchmark subs… view at source ↗
Figure 4
Figure 4. Figure 4: Latent structure of benchmark samples projected into the indicator space. Samples from different benchmarks occupy distinct but overlapping regions, illustrating that no single bench￾mark isolates a unique capability. This overlap further supports cross-benchmark orchestration to construct targeted evaluation slices that combine complementary challenges. subsets along combined criteria such as Reasoning De… view at source ↗
Figure 5
Figure 5. Figure 5: Enter Caption G.2. ARC Human-Model Agreement reasoning_depth fact_recall narrative_understanding language_difficulty spelling grammar referential_clarity ambiguity_level readability fact_checking_required distractor_quality temporal_sensitivity bias_stereotyping cultural_political_framing misinformation_bait safety_critical audience_appropriate reasoning_type knowledge_type age_level factual_accuracy verif… view at source ↗
Figure 6
Figure 6. Figure 6: Enter Caption 25 [PITH_FULL_IMAGE:figures/full_fig_p025_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Enter Caption G.4. MMLU Human-Model Agreement reasoning_depth fact_recall narrative_understanding language_difficulty spelling grammar referential_clarity ambiguity_level readability fact_checking_required distractor_quality temporal_sensitivity bias_stereotyping cultural_political_framing misinformation_bait safety_critical audience_appropriate reasoning_type knowledge_type age_level factual_accuracy veri… view at source ↗
Figure 8
Figure 8. Figure 8: Enter Caption 26 [PITH_FULL_IMAGE:figures/full_fig_p026_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Enter Caption G.6. Winogrande Human-Model Agreement reasoning_depth fact_recall narrative_understanding language_difficulty spelling grammar referential_clarity ambiguity_level readability fact_checking_required distractor_quality temporal_sensitivity bias_stereotyping cultural_political_framing misinformation_bait safety_critical audience_appropriate reasoning_type knowledge_type age_level factual_accurac… view at source ↗
Figure 10
Figure 10. Figure 10: Enter Caption 27 [PITH_FULL_IMAGE:figures/full_fig_p027_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Enter Caption 28 [PITH_FULL_IMAGE:figures/full_fig_p028_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Enter Caption 29 [PITH_FULL_IMAGE:figures/full_fig_p029_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Enter Caption 30 [PITH_FULL_IMAGE:figures/full_fig_p030_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Enter Caption 31 [PITH_FULL_IMAGE:figures/full_fig_p031_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Heatmap of pairwise Pearson correlations between indicators for ARC. Rows and columns represent individual indicators; cell color encodes the strength and direction of their linear relationship (red = strong positive, blue = strong negative, white = no correlation). Values are computed across all samples and models for the given benchmark. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Enter Caption H.4. Average Counts: Dimensions Cognitive & Knowledge Demands Ethics, Safety & Fairness Language & Content Quality Task Properties Dimension 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 Average Value Average per Dimension Model deepseek-chat deepseek-reasoner gpt-5-2025-08-07 [PITH_FULL_IMAGE:figures/full_fig_p033_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Enter Caption 33 [PITH_FULL_IMAGE:figures/full_fig_p033_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Enter Caption 37 [PITH_FULL_IMAGE:figures/full_fig_p037_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Heatmap of pairwise Pearson correlations between indicators for HellaSwag. Rows and columns represent individual indicators; cell color encodes the strength and direction of their linear relationship (red = strong positive, blue = strong negative, white = no correlation). Values are computed across all samples and models for the given benchmark. 38 [PITH_FULL_IMAGE:figures/full_fig_p038_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Enter Caption I.4. Average Counts: Dimensions Cognitive & Knowledge Demands Ethics, Safety & Fairness Language & Content Quality Task Properties Dimension 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 Average Value Average per Dimension Model deepseek-chat deepseek-reasoner gpt-5-2025-08-07 [PITH_FULL_IMAGE:figures/full_fig_p039_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Enter Caption 39 [PITH_FULL_IMAGE:figures/full_fig_p039_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Enter Caption 43 [PITH_FULL_IMAGE:figures/full_fig_p043_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Heatmap of pairwise Pearson correlations between indicators for MMLU. Rows and columns represent individual indicators; cell color encodes the strength and direction of their linear relationship (red = strong positive, blue = strong negative, white = no correlation). Values are computed across all samples and models for the given benchmark. 44 [PITH_FULL_IMAGE:figures/full_fig_p044_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Enter Caption J.4. Average Counts: Dimensions Cognitive & Knowledge Demands Ethics, Safety & Fairness Language & Content Quality Task Properties Dimension 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 Average Value Average per Dimension Model deepseek-chat deepseek-reasoner gpt-5-2025-08-07 [PITH_FULL_IMAGE:figures/full_fig_p045_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: Enter Caption 45 [PITH_FULL_IMAGE:figures/full_fig_p045_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: Enter Caption 49 [PITH_FULL_IMAGE:figures/full_fig_p049_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: Heatmap of pairwise Pearson correlations between indicators for TruthfulQA. Rows and columns represent individual indicators; cell color encodes the strength and direction of their linear relationship (red = strong positive, blue = strong negative, white = no correlation). Values are computed across all samples and models for the given benchmark. 50 [PITH_FULL_IMAGE:figures/full_fig_p050_27.png] view at source ↗
Figure 28
Figure 28. Figure 28: Enter Caption K.4. Average Counts: Dimensions Cognitive & Knowledge Demands Ethics, Safety & Fairness Language & Content Quality Task Properties Dimension 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 Average Value Average per Dimension Model deepseek-chat deepseek-reasoner gpt-5-2025-08-07 [PITH_FULL_IMAGE:figures/full_fig_p051_28.png] view at source ↗
Figure 29
Figure 29. Figure 29: Enter Caption 51 [PITH_FULL_IMAGE:figures/full_fig_p051_29.png] view at source ↗
Figure 30
Figure 30. Figure 30: Enter Caption 55 [PITH_FULL_IMAGE:figures/full_fig_p055_30.png] view at source ↗
Figure 31
Figure 31. Figure 31: Heatmap of pairwise Pearson correlations between indicators for Winogrande. Rows and columns represent individual indicators; cell color encodes the strength and direction of their linear relationship (red = strong positive, blue = strong negative, white = no correlation). Values are computed across all samples and models for the given benchmark. 56 [PITH_FULL_IMAGE:figures/full_fig_p056_31.png] view at source ↗
Figure 32
Figure 32. Figure 32: Enter Caption L.4. Average Counts: Dimensions Cognitive & Knowledge Demands Ethics, Safety & Fairness Language & Content Quality Task Properties Dimension 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 Average Value Average per Dimension Model deepseek-chat deepseek-reasoner gpt-5-2025-08-07 [PITH_FULL_IMAGE:figures/full_fig_p057_32.png] view at source ↗
Figure 33
Figure 33. Figure 33: Enter Caption 57 [PITH_FULL_IMAGE:figures/full_fig_p057_33.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references · 5 linked inside Pith

  1. [1]

    Cognitive & Knowledge Demands - reasoning_depth: 0 = None (direct recall, no inference) 1 = Minimal (one simple inference) 2 = Moderate (several linked steps) 3 = Extended (multi-step chain or puzzle) - reasoning_type: {{causal, temporal, counterfactual, abductive, analogical, symbolic}} // multi-select - knowledge_type: {{common, specialized, scientific,...

  2. [2]

    no" | "partial

    Language & Content Quality - language_difficulty: 0 = Very simple (basic words, short sentences) 1 = Moderate (mixed structure, some technical terms) 2 = Complex (specialized vocabulary, long sentences) 3 = Highly technical (field-specific terminology) - spelling: 0 = Severe errors; unreadable 1 = Minor errors; still readable 2 = Correct spelling througho...

  3. [3]

    no" | "partial

    Task Properties - answerability: "no" | "partial" | "yes" - label_quality: C = Correct, D = Dubious, I = Incorrect - distractor_quality: 0 = Implausible (obviously wrong option) 1 = Weak (easy to dismiss) 2 = Mostly plausible (minor flaws) 3 = Strong (fair, convincing) - temporal_sensitivity: true = time-dependent; false = timeless - leakage_risk: "low" =...

  4. [4]

    Context - domain: {{math, computer_science, physics, chemistry, biology, medicine, engineering, literature, history, philosophy, arts_music, economics, psychology, sociology, political_science, law, business_finance, education_exams, technology_internet, everyday, pop_culture, cultural_religious, news, trivia, other}} // single select

  5. [5]

    indicators

    Ethics, Safety & Fairness - bias_stereotyping: 0 = None 1 = Weak (slight stereotype implied) 2 = Moderate (clear stereotype presence) 3 = Strong (explicit stereotype) - cultural_political_framing: true = requires stance; false = neutral - misinformation_bait: 0 = None 1 = Weak (slightly misleading phrasing) 2 = Moderate (common misconception implied) 3 = ...

  6. [6]

    Do NOT include probabilities, confidence scores, or extra fields

  7. [7]

    id": 0,

    Return ONLY a JSON array matching the schema, one object per input item, in the same order as INPUT. ### INPUT {INPUT_JSON} ### OUTPUT 16 Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation C. All Indicator Counts (GPT-5) Table 7.All indicator configurations: sample counts. Part 1 Indicator Value Winogrande TruthfulQA...

    1943

  8. [12]

    Use ONLY the allowed codes and value sets above

  9. [13]

    Use [] if none apply

    For multi-select fields (‘reasoning_type‘, ‘knowledge_type‘) return an array. Use [] if none apply

  10. [14]

    For domain, pick the closest match from the controlled vocabulary

  11. [15]

    "partial", not 1)

    Treat ordered categories as ordinal for interpretation but ALWAYS output the exact code (e.g. "partial", not 1)

  12. [16]

    If any required field is not fully inferable, choose the closest anchor and explain in ‘notes‘

  13. [118]

    acl-short.118/

    URL https://aclanthology.org/2023. acl-short.118/. Singh, S., Romanou, A., Fourrier, C., Adelani, D. I., Ngui, J. G., Vila-Suero, D., Limkonchotiwat, P., Marchisio, K., Leong, W. Q., Susanto, Y ., Ng, R., Longpre, S., Ko, W.- Y ., Ruder, S., Smith, M., Bosselut, A., Oh, A., Martins, A. F. T., Choshen, L., Ippolito, D., Ferrante, E., Fadaee, M., Ermis, B.,...

    Pith/arXiv arXiv 2023

  14. [306]

    emnlp-main.306/

    URL https://aclanthology.org/2023. emnlp-main.306/. Kiela, D., Bartolo, M., Nie, Y ., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., Ma, Z., Thrush, T., Riedel, S., Waseem, Z., Stene- torp, P., Jia, R., Bansal, M., Potts, C., and Williams, A. Dynabench: Rethinking Benchmarking in NLP, April 2021. URL http://arxiv.org/abs...

    Pith/arXiv arXiv 2023

  15. [442]

    acl-main.442/

    URL https://aclanthology.org/2020. acl-main.442/. Rudinger, R., Naradowsky, J., Leonard, B., and Durme, B. V . Gender Bias in Coreference Resolution, April 2018. URL http://arxiv.org/abs/1804. 09301. arXiv:1804.09301 [cs]. Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y . WinoGrande: An Adversarial Winograd Schema Chal- lenge at Scale, November 2...

    Pith/arXiv arXiv 2020

  16. [1275]

    acl-long.1275/

    URL https://aclanthology.org/2025. acl-long.1275/. Zellers, R., Holtzman, A., Bisk, Y ., Farhadi, A., and Choi, Y . HellaSwag: Can a Machine Really Finish Your Sentence?, May 2019. URL http://arxiv.org/abs/1905. 07830. arXiv:1905.07830 [cs]. Zhao, J., Wang, T., Yatskar, M., Ordonez, V ., and Chang, K.-W. Gender Bias in Coreference Resolution: Eval- uation...

    Pith/arXiv arXiv 2025

  17. [2024]

    Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S

    URL https://openreview.net/forum? id=E8gYIrbP00. Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S. R., and Smith, N. A. Annota- tion Artifacts in Natural Language Inference Data, April 2018. URL http://arxiv.org/abs/1803. 02324. arXiv:1803.02324 [cs]. Hansson, S., Mavromatakis, K., Adesam, Y ., Bouma, G., and Dannlls, D. The Swedish Wino...

    Pith/arXiv arXiv 2018

  18. [2025]

    ISBN 979-8-89176-251-0

    Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long

    doi:10.18653/v1/2025.acl-long 2025