Pith.

Artificial Intelligence · arXiv:2611.16300 v1 · submitted 12 Nov 2026 · 36 references

SAGE-QA: Semantic-Aware, Grounded Evaluation for Closed-Context Question Answering on SQuAD 2.0

Chandra N Sekharan, Vishakan Umapathy, Abhishek Phadke

The paper says

“Closed-context question answering (QA) requires a system to generate an answer strictly supported by a supplied passage, or to abstain when the passage lacks adequate evidence.”

more of the abstract

SQuAD 2.0 is a widely used benchmark for this setting, scored by Exact Match (EM) i.e., character-level agreement with human reference spans after lexical normalization. For generative large language models (LLMs), however, an EM failure may reflect a valid paraphrase or an alternative passage-supported answer rather than a hallucination. The best prior leaderboard EM is approximately 91%, but evaluating generative models requires separating lexical divergence from factual error. We introduce SAGE-QA (Semantic-Aware, Grounded Evaluation for Question Answering), a multi-tiered framework combining reference-based Semantic Equivalence Scoring (SES) and passage-based Re-Validation Scoring (RVS). On a fixed set of 10,000 SQuAD 2.0 validation examples, we compute EM and decision-matrix F1 from TP, TN, FP, and FN counts for zero-shot closed-weight frontier models and for open-weight models, untuned and fine-tuned. Zero-shot, Claude Opus 4.6 and Gemini 3.1 Pro reach 85.21 and 86.66 EM, outperforming untuned Llama 3 8B (56.88), gpt-oss-20b (66.28), and Llama 3.3 70B (71.39). Parameter-efficient fine-tuning adds 21.19 to 34.06 EM points, raising these models to 90.94, 87.47, and 94.24 EM, respectively. Auditing the frontier models with SAGE-QA and an external LLM judge yields dataset-wide SES scores of 92.81 for Claude Opus 4.6 and 91.67 for Gemini 3.1 Pro, and hybrid RVS scores, which combine benchmark credit with passage-based judge assessments, of 97.96 and 98.02. Fine-tuned open-weight models perform best, with fine-tuned Llama 3.3 70B surpassing the previous best known EM.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 13 Nov 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. SAGE-QA's diagnostic framework is coherent and the paper is unusually honest about its limitations, but the headline claim of 'surpassing previous best EM' is unsupported because it compares scores on different test sets. This line is wrong

4 major objections · 8 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 36 references say.

The paper says, at Abstract, §V, §VII

The review did not quote this passage.

Pith says: Objection 1 of 4 · major

The central claim that fine-tuned Llama 3.3 70B 'surpass[es] the previous best known EM' (abstract, §V, conclusion) is not supported by the evidence. The 94.24 EM is measured on the authors' 10,000-example subset of the SQuAD 2.0 validation set (§IV.A), while the 90.94 EM from IE-Net is on the official hidden test set (Table I caption).

more of objection 1

The paper provides no argument or statistical test that these two sets are comparable. The authors acknowledge the subset is a fixed sample from the validation split (§IV.A) but never address the comparability issue. This is a load-bearing error because the headline result rests on this comparison. The claim should be restricted to 'best known EM on this validation subset' or supported by a statistically grounded projection to the test set.

The paper says, at §IV.A, Table IV

The review did not quote this passage.

Pith says: Objection 2 of 4 · major

The paper does not report confidence intervals or variability estimates for any of the EM or F1 scores. Since the evaluation is on a 10,000-example subset (not the full validation set), and the claim of surpassing prior results depends on a 3.3-point gap, the absence of uncertainty quantification is a significant omission. The authors should provide bootstrap confidence intervals or at least standard errors for the reported metrics.

The paper says, at §VI.B

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.69):

“Limitations of our Research Our semantic-equivalence and passage-support assessments rely on one external LLM judge (gpt-6-luna) and there- fore we interpret SES and RVS as judge-based diagnostic scores rather than independently verified measures of answer correctness.”

Pith says: Objection 3 of 4 · major

The limitations section acknowledges that SES and RVS rely on a single external LLM judge (gpt-6-luna) and are not independently verified. However, the paper does not discuss the potential for judge bias or calibration issues in the context of the specific rubric used.

more of objection 3

The judge is from the same family (OpenAI) as some of the evaluated models? Actually gpt-6-luna is an OpenAI model, and the frontier models include Claude and Gemini, so direct self-evaluation is avoided. But the paper should at least mention that the judge's scores are not validated against human judgments, and that the rubric bands (0.95-0.99, etc.) are arbitrary. The authors do state this in §VI.B, but the discussion is brief and could be expanded.

The paper says, at §V.B, Table VI

The review did not quote this passage.

Pith says: Objection 4 of 4 · major

The score distributions show that a large fraction of RVS scores fall in the 1.00 band (e.g., 825/888 for Claude on F_ans^N). This suggests the judge may be lenient or the rubric may not be sufficiently discriminative. The paper should discuss whether the judge's high acceptance rate on EM failures is meaningful or could reflect a bias toward accepting any plausible answer.

more of objection 4

The authors note that 'scores below threshold can reflect incomplete answers as well as lack of passage support' (§VI.B), but the high rate of full credit (1.00) on F_ans^N cases (825/888 for Claude) warrants further scrutiny.

8 minor objections

The paper says, at Abstract

Pith says: minor objection 1 of 8

The abstract states 'fine-tuning open-weight models perform the best to surpass previous best known EM benchmarks.' This is grammatically awkward and could be rephrased for clarity.

The paper says, at §I

Pith says: minor objection 2 of 8

The sentence 'We explore this question because QA systems needs to satisfy the information need' has a subject-verb agreement error ('needs' should be 'need').

show 6 more

The paper says, at §III, Eq. (11) and (15)

Pith says: minor objection 3 of 8

The derivation of SES_all and RVS_all is clear, but the notation could be simplified. The use of '100/N' multiplied by sums of scores in [0,1] is equivalent to a percentage. Consider stating explicitly that SES_all = EM + (|C_S|/N) * SES_cases, which is already given but could be highlighted.

The paper says, at Table IV

Pith says: minor objection 4 of 8

The table caption says 'UNIFIED BASELINE RESULTS AND CATEGORY ERROR RATES ON 10,000 SQUAD 2.0 VALIDATION EXAMPLES.' It would be helpful to note that the FN column for fine-tuned models shows '—' for the F_∅^N / F_ans^N split because those breakdowns were not computed. This is fine, but a footnote would clarify.

The paper says, at §V.A

Pith says: minor objection 5 of 8

The sentence 'Their EM comparison therefore reflects a trade-off between reference matching and abstention' is a good observation. However, the phrase 'reference matching' could be misinterpreted; consider 'lexical matching' for consistency.

The paper says, at §V.B

Pith says: minor objection 6 of 8

The sentence 'These hybrid scores combine automatic benchmark credit with judge-assigned credit; their proximity does not establish equal answer correctness' is an important caveat. It might be better placed earlier in the section.

The paper says, at References

Pith says: minor objection 7 of 8

Reference [5] is to 'gpt-6-luna' model documentation. Since this is a proprietary model, the citation should include a URL or DOI if available. Similarly, references [27], [28], [31] are to model announcements or documentation; ensure these are publicly accessible.

The paper says, at General

Pith says: minor objection 8 of 8

The paper uses 'benchmark' as a verb in several places (e.g., 'benchmark unanswerable questions'). This is acceptable but could be replaced with 'labeled as unanswerable in the benchmark' for clarity.

What would settle it

Re-run the fine-tuned Llama 3.3 70B evaluation on the official SQuAD 2.0 test set via the leaderboard submission process. If the resulting EM is within the 95% confidence interval of the 94.24 observed on the validation subset (accounting for sampling variance), the cross-set comparison is validated. If the test-set EM is significantly lower (e.g., below 92), the claim of surpassing previous best is unsupported by the current data.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 13 Nov 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 7 s · 17,259 tokens read, 795 written
  3. Editor wrote the verdict above 6 s · 18,285 tokens read, 489 written
  4. Referee major revision: 4 major objections, 8 minor 12 s · 17,737 tokens read, 1,721 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

The paper's diagnostic framework (SES and RVS) is a legitimate contribution: it formalizes a two-stage judge-based audit of EM failures, with clear equations and routing logic. The authors are unusually explicit about what the judge scores do and do not mean (see §VI.B and multiple inline caveats). That level of candor is rare and should be credited.

The soft spot is the one the stress-test note flags. The abstract, §V, and conclusion claim that fine-tuned Llama 3.3 70B 'surpass[es] the previous best known EM' of ~90.94. That 90.94 is from the official hidden test set; the 94.24 is on the authors' 10,000-example validation subset. The paper never argues that the subset is representative of the test set, and offers no confidence interval or statistical test. The gap could reflect dataset difficulty. The authors acknowledge the subset is a fixed sample (§IV.A) but do not address the cross-set comparison in their limitations. This is not a minor oversight—it is the paper's headline result, and it is unsupported as written.

Other concerns are real but secondary. The judge (gpt-6-luna) is uncalibrated against humans, which the authors acknowledge. The prompt was selected via pilot performance on the same dataset, raising potential overfitting. No artifacts are released. These are all acknowledged or can be fixed.

Who this is for: researchers working on LLM evaluation methodology, especially those interested in separating lexical divergence from factual error in closed-context QA. The diagnostic framework itself is worth engaging with.

Recommendation: the paper deserves a serious referee, but the referee should insist that the cross-set comparison be either removed or properly supported. If the authors restrict the claim to their validation subset, the paper is solid and publishable after minor revision.

The strongest objection

The paper's central claim that fine-tuned Llama 3.3 70B 'surpass[es] the previous best known EM' (abstract, §V, conclusion) rests on comparing its 94.24 EM on the authors' 10,000-example validation subset (Table IV) against the 90.939 EM of IE-Net on the official hidden test set (Table I). The authors acknowledge the subset is a fixed sample from the validation split (§IV.A), but never argue that it is statistically representative of the hidden test set, nor do they provide a confidence interval or a matched evaluation. Because the example sets differ, the observed 3.3-point gap could reflect dataset difficulty rather than model superiority. The paper itself notes that the official leaderboard uses a hidden test set (Table I caption), and §IV.A defines the evaluation set as a subset of the validation data. No cross-set calibration or statistical test is offered. This is not a subtle methodological nuance; it is a direct mismatch between the evidence and the claim. The reader's verdict correctly identifies this as the weakest assumption. The paper's own limitations (§VI.B) do not address this comparison, focusing instead on judge calibration and diagnostic scope. The claim appears in the abstract, §V, and the conclusion, making it load-bearing for the paper's headline result. If the authors restricted the claim to 'best known EM on this validation subset' or provided a statistically grounded projection to the test set, the concern would be resolved. As written, the conclusion is unsupported.

The reader's verdict identifies the same load-bearing concern: the cross-set comparison between the 10k validation subset and the official test set is invalid. The paper's strongest claim depends on this comparison. The reader's weakest_assumption field states: 'The load-bearing premise is that EM measured on the authors' 10,000-example subset of the SQuAD 2.0 validation split is comparable to the 90.94 EM reported on the official hidden test set (Table I).' I agree fully. The paper does not address this in its limitations; it only discusses judge calibration and diagnostic scope. The concrete test I propose directly settles whether the concern lands. The verdict should remain CONDITIONAL because the diagnostic framework (SAGE-QA) and within-subset comparisons are coherent and potentially useful, but the headline claim requires either restriction to the subset or a validated projection. The paper's own sentences (b11, b8, Table IV, Table I) make the comparison explicit. No other concern is as load-bearing: the judge calibration issue is acknowledged (§VI.B) and does not affect the EM-based claim; the prompt selection is described (§IV.B) and the GEPA prompt is fixed before evaluation. The cross-set comparison is the single unresolved threat to the central claim.

The referee's report

If the central claim were properly supported, the paper would demonstrate a practical way to separate lexical divergence from factual error in closed-context QA evaluation, and show that fine-tuned open-weight models can outperform prior span-extraction systems. The SAGE-QA framework itself is a sensible contribution: combining lexical matching with judge-based semantic and passage-support assessments addresses a genuine limitation of EM for generative models. The paper is clearly written, the framework is well-defined, and the authors acknowledge several limitations (single judge, no human adjudication, diagnostics only on frontier cohort). However, the headline claim of surpassing previous best EM is not supported by the data presented, as discussed below.

Pith's reading of the paper (a machine's summary, not the authors')

The paper introduces SAGE-QA, a multi-tiered evaluation framework for closed-context question answering that supplements the standard Exact Match (EM) metric with two LLM-as-a-judge diagnostics: Semantic Equivalence Scoring (SES), which checks whether a non-matching answer preserves the meaning of a human reference, and Re-Validation Scoring (RVS), which checks whether the answer is grounded in the passage. The authors apply this framework to eight model configurations on a 10,000-example subset of SQuAD 2.0, reporting that fine-tuned open-weight models, particularly Llama 3.3 70B at 94.24 EM, surpass the previous best known EM on the leaderboard. The paper argues that lexical matching alone conflates valid paraphrases with factual errors, and that SES and RVS provide a more nuanced view of model behavior, though the authors acknowledge these are judge-based diagnostics, not independently verified correctness measures.

What carries the argument. The central mechanism is the SAGE-QA pipeline, which partitions the 10,000 evaluation examples into four lexical outcome sets (TP, TN, FP, FN) and then routes the eligible lexical failures (answered non-matches and false positives) to two separate LLM-as-a-judge rubrics: Semantic Equivalence Scoring (SES), which compares the model answer to the human reference answers, and Re-Validation Scoring (RVS), which assesses whether the answer is grounded in the passage and directly addresses the question. Both diagnostics use the same external judge model (gpt-6-luna) with a graded 0-100 scale, and the results are aggregated into hybrid dataset-wide scores that combine automatic credit for exact matches with judge-assigned credit on the eligible failures.

Core claim. The paper claims that fine-tuned open-weight generative models, evaluated on a 10,000-example subset of SQuAD 2.0, achieve an Exact Match of 94.24, which exceeds the approximately 90.94 EM reported on the official hidden test set. It further claims that the SAGE-QA framework, by routing EM failures to an external LLM judge for semantic equivalence (SES) and passage support (RVS), reveals that a large fraction of benchmark false positives and non-matching answers are judged as semantically correct and passage-supported. Specifically, on the zero-shot frontier models, RVS accepts over 90% of benchmark false positives, and SES reverses the EM ranking between Claude Opus 4.6 and Gemini 3.1 Pro. The paper positions SAGE-QA as a diagnostic tool that makes visible the disagreement between lexical matching and semantic/passage-based evaluation, without independently resolving the correctness of the answers.

The premise it rests on. The claim that fine-tuned Llama 3.3 70B surpasses the previous best known EM depends on comparing a score measured on a 10,000-example validation subset (94.24) to a score reported on the official hidden test set (90.939), and the paper does not demonstrate that the subset is representative of the test set distribution.

If the paper is right (Pith's inferences, not the authors')

If SAGE-QA is adopted, the community may begin reporting both EM and judge-based diagnostic scores, leading to a more nuanced understanding of model capabilities beyond lexical overlap.

The finding that fine-tuned open-weight models can surpass the previous best EM on a subset suggests that parameter-efficient fine-tuning (LoRA/QLoRA) is a viable path to state-of-the-art closed-context QA performance, potentially reducing reliance on closed-weight frontier models.

The high RVS acceptance rate for benchmark false positives (over 90%) challenges the assumption that all answers to unanswerable questions are hallucinations, and may prompt a re-examination of the SQuAD 2.0 answerability labels.

The reversal of the EM ranking under SES (Claude surpassing Gemini) demonstrates that evaluation criteria can change model comparisons, implying that leaderboard rankings are not robust to the choice of metric.

Because the paper's best EM (94.24) is measured on a 10,000-example validation subset rather than the official hidden test set, the claim of 'surpassing previous best known EM' is not directly supported; a fair comparison would require evaluation on the official test set or a careful analysis of subset representativeness.

The heavy reliance on a single LLM judge (gpt-6-luna) for both SES and RVS means the diagnostic scores are not independent assessments; a more robust evaluation would involve multiple judges or human adjudication, which the authors acknowledge as future work.

The paper's framework could be extended to other closed-context QA benchmarks (e.g., Natural Questions, TriviaQA) to test whether the pattern of high RVS acceptance for FP holds across different datasets and passage styles.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Closed-context question answering (QA) requires a system to generate an answer strictly supported by a supplied passage, or to abstain when the passage lacks adequate evidence. SQuAD 2.0 is a widely used benchmark for this setting, scored by Exact Match (EM) i.e., character-level agreement with human reference spans after lexical normalization. For generative large language models (LLMs), however, an EM failure may reflect a valid paraphrase or an alternative passage-supported answer rather than a hallucination. The best prior leaderboard EM is approximately 91%, but evaluating generative models requires separating lexical divergence from factual error. We introduce SAGE-QA (Semantic-Aware, Grounded Evaluation for Question Answering), a multi-tiered framework combining reference-based Semantic Equivalence Scoring (SES) and passage-based Re-Validation Scoring (RVS). On a fixed set of 10,000 SQuAD 2.0 validation examples, we compute EM and decision-matrix F1 from TP, TN, FP, and FN counts for zero-shot closed-weight frontier models and for open-weight models, untuned and fine-tuned. Zero-shot, Claude Opus 4.6 and Gemini 3.1 Pro reach 85.21 and 86.66 EM, outperforming untuned Llama 3 8B (56.88), gpt-oss-20b (66.28), and Llama 3.3 70B (71.39). Parameter-efficient fine-tuning adds 21.19 to 34.06 EM points, raising these models to 90.94, 87.47, and 94.24 EM, respectively. Auditing the frontier models with SAGE-QA and an external LLM judge yields dataset-wide SES scores of 92.81 for Claude Opus 4.6 and 91.67 for Gemini 3.1 Pro, and hybrid RVS scores, which combine benchmark credit with passage-based judge assessments, of 97.96 and 98.02. Fine-tuned open-weight models perform best, with fine-tuned Llama 3.3 70B surpassing the previous best known EM.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-13

The authors' main claim, as the classifier picked it out 0.98

“Fine-tuned open-weight models perform best, with fine-tuned Llama 3.3 70B surpassing the previous best known EM.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.08 · states limitation 0.16 · human participants 0.05 · controlled comparison 0.74 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.97 · mentions code or data availability 0.04

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Closed-context question answering (QA) requires a system to generate an answer strictly supported by a supplied passage, or to abstain when the passage lacks adequate evidence. background 1.00 0.04 0.02 asserted 1.00
SQuAD 2.0 is a widely used benchmark for this setting, scored by Exact Match (EM) i.e., character-level agreement with human reference spans after lexical normalization. background 1.00 0.04 0.02 asserted 1.00
For generative large language models (LLMs), however, an EM failure may reflect a valid paraphrase or an alternative passage-supported answer rather than a hallucination. gap 0.51 0.08 0.04 hedged possibility 0.97
The best prior leaderboard EM is approximately 91%, but evaluating generative models requires separating lexical divergence from factual error. background 0.58 0.09 0.04 asserted 0.99
We introduce SAGE-QA (Semantic-Aware, Grounded Evaluation for Question Answering), a multi-tiered framework combining reference-based Semantic Equivalence Scoring (SES) and passage-based Re-Validation Scoring (RVS). contribution 0.84 0.89 0.07 asserted 1.00
On a fixed set of 10,000 SQuAD 2.0 validation examples, we compute EM and decision-matrix F1 from TP, TN, FP, and FN counts for zero-shot closed-weight frontier models and for open-weight models, untuned and fine-tuned. method 0.99 0.96 0.04 asserted 0.97
Zero-shot, Claude Opus 4.6 and Gemini 3.1 Pro reach 85.21 and 86.66 EM, outperforming untuned Llama 3 8B (56.88), gpt-oss-20b (66.28), and Llama 3.3 70B (71.39). result 1.00 0.91 0.08 asserted 1.00
Parameter-efficient fine-tuning adds 21.19 to 34.06 EM points, raising these models to 90.94, 87.47, and 94.24 EM, respectively. result 1.00 0.93 0.13 asserted 1.00
Auditing the frontier models with SAGE-QA and an external LLM judge yields dataset-wide SES scores of 92.81 for Claude Opus 4.6 and 91.67 for Gemini 3.1 Pro, and hybrid RVS scores, which combine benchmark credit with passage-based judge assessments, of 97.96 and 98.02. result 1.00 0.94 0.14 asserted 1.00
Fine-tuned open-weight models perform best, with fine-tuned Llama 3.3 70B surpassing the previous best known EM. result 0.97 0.90 0.66 asserted 1.00

What kind of paper the classifier reports this is: method or tool 0.99 · evidence from observation or experiment 0.81

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-13: 1 request, 8,171 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (3)
Figure 1 of the paper
Figure 1. SAGE-QA pipeline. SES and RVS overlap on answered FN cases;
Figure 2 of the paper
Figure 2. Zero-shot frontier outcome matrices. The upper row contains an
Figure 3 of the paper
Figure 3. Zero-shot untuned open-weight outcome matrices. Without fine-tuning,

Figures are the authors', reproduced from the paper.

References (36)
  1. [1] P. Rajpurkar, R. Jia, and P. Liang, “Know what you don’t know: Unanswerable questions for SQuAD,” inProc. ACL, 2018, pp. 784– on Pith
  2. [2] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “SQuAD: 100,000+ questions for machine comprehension of text,” inProc. EMNLP, 2016, pp. 2383–2392. doi: 10.18653/v1/D16-1264 on Pithdoi:10.18653/v1/d16-1264
  3. [3] Stanford NLP Group, “The Stanford Question Answering Dataset: SQuAD 2.0 leaderboard.” Online leaderboard on Pith
  4. [4] A. Panickssery, S. R. Bowman, and S. Feng, “LLM evaluators recognize and favor their own generations,” inAdvances in Neural Information Processing Systems, vol. 37, pp. 68772–68802, 2024. arXiv:2404.13076 on PithPith's reviewarXiv
  5. [5] OpenAI, “gpt-6-luna,”OpenAI Platform Model Documentation, 2026. Model documentation on Pith
  6. [6] M. Seo, A. Kembhavi, A. Farhadi, and H. Hajishirzi, “Bidirectional attention flow for machine comprehension,” inProc. ICLR, 2017. arXiv:1611.01603 on PithPith's reviewarXiv
  7. [7] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” inProc. NAACL-HLT, 2019, pp. 4171–4186. doi: 10.18653/v1/N19-1423 on Pithdoi:10.18653/v1/n19-1423
  8. [8] Y . Liu et al., “RoBERTa: A robustly optimized BERT pretraining approach,” 2019. arXiv:1907.11692 on PithPith's reviewarXiv
  9. [9] E. Kamalloo, N. Dziri, C. L. A. Clarke, and D. Rafiei, “Evaluating open-domain question answering in the era of large language models,” inProc. ACL, 2023. arXiv:2305.06984 on PithPith's reviewarXiv
  10. [10] J. Bulian, C. Buck, W. Gajewski, B. Börschinger, and T. Schuster, “Tomayto, Tomahto. Beyond token-level answer equivalence for ques- tion answering evaluation,” inProc. EMNLP, 2022, pp. 291–305. doi: 10.18653/v1/2022.emnlp-main.20 on Pithdoi:10.18653/v1/2022.emnlp-main.20
  11. [11] V . Adlakha et al., “Evaluating correctness and faithfulness of instruction- following models for question answering,”Transactions of the Associ- ation for Computational Linguistics, vol. 12, pp. 681–699, 2024. doi: 10.1162/tacl_a_00667 on Pithdoi:10.1162/tacl_a_00667
  12. [12] Hugging Face, “rajpurkar/squad_v2.” Dataset repository on Pith
  13. [13] T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artzi, “BERTScore: Evaluating text generation with BERT,” inProc. ICLR, on Pith
  14. [14] Y . Liu et al., “G-Eval: NLG evaluation using GPT-4 with better human alignment,” inProc. EMNLP, 2023. arXiv:2303.16634 on PithPith's reviewarXiv
  15. [15] L. Zheng et al., “Judging LLM-as-a-judge with MT-Bench and Chatbot Arena,” inAdvances in Neural Information Processing Systems, vol. 36, on Pith
  16. [16] C. G. Northcutt, A. Athalye, and J. Mueller, “Pervasive label errors in test sets destabilize machine learning benchmarks,” inProc. NeurIPS Datasets and Benchmarks, 2021. arXiv:2103.14749 on PithPith's reviewarXiv
  17. [17] S. Gururangan et al., “Annotation artifacts in natural language inference data,” inProc. NAACL-HLT, 2018, pp. 107–112. doi: 10.18653/v1/ N18-2017 on Pithdoi:10.18653/v1/
  18. [18] H. Bahak, F. Taheri, Z. Zojaji, and A. Kazemi, “Evaluating ChatGPT as a question answering system: A comprehensive analysis and comparison with existing models,” 2023. arXiv:2312.07592 on PithPith's reviewarXiv
  19. [19] O. Khattab et al., “DSPy: Compiling declarative language model calls into self-improving pipelines,” inProc. ICLR, 2024. arXiv:2310.03714 on PithPith's reviewarXiv
  20. [20] K. Opsahl-Ong et al., “Optimizing instructions and demonstrations for multi-stage language model programs,” inProc. EMNLP, 2024, pp. 9340–9366. arXiv:2406.11695 on PithPith's reviewarXiv
show 16 more
  1. [21] L. A. Agrawal et al., “GEPA: Reflective prompt evolution can outper- form reinforcement learning,” 2025. arXiv:2507.19457 on PithPith's reviewarXiv
  2. [22] OpenAI, “GPT-5 prompt migration and improvement using the new optimizer,”OpenAI Cookbook, 2025. Online resource on Pith
  3. [23] E. J. Hu et al., “LoRA: Low-rank adaptation of large language models,” inProc. ICLR, 2022. arXiv:2106.09685 on PithPith's reviewarXiv
  4. [24] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “QLoRA: Efficient finetuning of quantized LLMs,” inAdvances in Neural Infor- mation Processing Systems, vol. 36, 2023. arXiv:2305.14314 on PithPith's reviewarXiv
  5. [25] P. Wang et al., “Large language models are not fair evaluators,” inProc. ACL, 2024, pp. 9440–9450. doi: 10.18653/v1/2024.acl-long.511 on Pithdoi:10.18653/v1/2024.acl-long.511
  6. [26] Hugging Face, “SQuAD v2 metric,”Evaluate. Metric documentation on Pith
  7. [27] Anthropic, “Introducing Claude Opus 4.6,” 2026. Model announcement on Pith
  8. [28] Google Cloud, “Gemini 3.1 Pro,”Vertex AI Generative AI Documenta- tion, 2026. Model documentation on Pith
  9. [29] Meta, “Meta-Llama-3-8B-Instruct,” 2024. Model card on Pith
  10. [30] Meta, “Llama-3.3-70B-Instruct,” 2024. Model card on Pith
  11. [31] OpenAI, “gpt-oss: Open-weight models,” 2025. Official repository on Pith
  12. [32] Anthropic, “Message Batches API,”Claude API Documentation. Batch- processing documentation on Pith
  13. [33] Google, “Batch API,”Gemini API Documentation. Batch API documen- tation on Pith
  14. [34] Unsloth AI, “Fine-tuning LLMs guide.” Online documentation on Pith
  15. [35] OpenAI, “Batch API,”OpenAI Platform Documentation. Batch API documentation on Pith
  16. [789] doi: 10.18653/v1/P18-2124 on Pithdoi:10.18653/v1/p18-2124

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment