Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

This paper claims that evaluating generated questions by first diagnosing specific error types, then scoring with that evidence, improves alignment with human judgments and reduces inflated scores for flawed questions.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 10:17 UTC pith:N2YLC6JS

load-bearing objection ErrEval is a solid, well-tested plug-in for error-aware QG evaluation, but the ablation gap means the paper overclaims what the diagnostic labels contribute. the 3 major comments →

arxiv 2601.10406 v1 pith:N2YLC6JS submitted 2026-01-15 cs.AI

ErrEval: Error-Aware Evaluation for Question Generation through Explicit Diagnostics

Weiping Fu , Bifan Wei , Jingyi Hao , Yushun Zhang , Jian Zhang , Jiaxin Wang , Bo Li , Yu He
show 2 more authors
Lingling Zhang Jun Liu
This is my paper
classification cs.AI
keywords question generation evaluationerror-aware evaluationexplicit diagnosticserror taxonomyLLM-based evaluatorsanswerabilityoverestimation reductionmulti-label error identification
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Automatic question generators often output questions that look fluent but contain hidden defects — factual hallucinations, questions that do not match the given answer, or questions unanswerable from the passage. Standard LLM-based evaluators score these holistically and tend to overestimate quality. ErrEval argues that evaluation should be split into two stages: first detect and categorize concrete errors, then feed those error labels to the LLM as evidence before it assigns a score. Across three human-annotated benchmarks and several LLM evaluators, the paper reports that this two-stage procedure raises correlation with human ratings, with the largest gains on task-oriented dimensions such as answerability, and lowers the rate at which bad questions are scored as good.

Core claim

ErrEval's central claim is that explicit error diagnosis is not an add-on but a necessary intermediate for reliable question-generation evaluation. It defines an 11-type taxonomy of errors across structural, linguistic, and content-related categories, and trains a lightweight multi-label Error Identifier to label each generated question. Only the error labels relevant to the evaluation dimension being scored — selected through a fixed dimension–error mapping — are injected into the evaluator prompt, together with their definitions. The paper reports that across four LLM evaluators and three human-annotated datasets, this diagnostic-first procedure improves agreement with human scores compare

What carries the argument

The load-bearing mechanism is a three-part pipeline: (1) a hand-built taxonomy of 11 error types grouped into structural, linguistic, and content-related categories; (2) a lightweight multi-label Error Identifier, trained with an iterative refinement loop in which a Verifier module filters its predictions and expands the training set; and (3) a dimension–error mapping that selects only the error labels relevant to the evaluation dimension being scored. The selected labels, with their definitions, are appended to the LLM evaluator's prompt as explicit evidence, turning holistic scoring into diagnosis-then-scoring.

Load-bearing premise

The framework assumes the Error Identifier's training labels — originally synthesized by an LLM generating intentionally erroneous questions and filtered only by agreement among three LLMs — are representative of the errors real question generators make; if those synthetic errors differ systematically from real ones, the diagnostic evidence is miscalibrated and the evaluation gains could disappear.

What would settle it

Have expert annotators label the errors in 1,000 questions produced by current question-generation systems, run ErrEval's Error Identifier on them, and compare against a zero-shot LLM annotator; if the EI does not beat that baseline, or if feeding its labels into an LLM evaluator does not improve correlation with human scores over vanilla prompting on this new sample, the paper's central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • ErrEval consistently raises Pearson correlation with human judgments across all four LLM evaluators tested; average relative gain is about 12–13% on one benchmark and 3–5% on another.
  • The largest improvements occur on relational dimensions — answerability, answer consistency, consistency — where the evaluator must check the question against the passage and answer.
  • ErrEval improves binary answerability classification on a reading-comprehension subset, with accuracy gains of up to several points for open-source evaluators.
  • Overestimation of low-quality questions drops: on some dimensions vanilla prompting scored over 80% of low-quality questions as high quality; with ErrEval that rate falls by about 8–12 percentage points.
  • Because the Error Identifier is plug-and-play, existing LLM evaluators can adopt the diagnostic stage without retraining.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Our inference: if the gain comes from grounding the evaluator in concrete evidence, the same diagnose-then-score recipe should transfer to other generation tasks — summarization, dialogue, data-to-text — provided a task-specific error taxonomy is built; a cheap test is to reuse an off-the-shelf taxonomy and see whether the improvement persists.
  • Our inference: the dimension–error mapping makes evaluation auditable: a user can inspect which error type triggered a low score and override mistaken diagnoses, which opaque holistic scoring cannot offer.
  • Our inference: the Error Identifier doubles as an error-analysis tool; the distribution of errors it predicts for a given question generator could guide targeted retraining, not just scoring.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ErrEval, a two-stage framework for question generation evaluation. A lightweight RoBERTa-based Error Identifier predicts 11 error types (structural, linguistic, content-related) from the (passage, answer, question) triple; the predicted labels are filtered through a dimension-error mapping and injected into LLM evaluator prompts to guide dimension-specific scoring. The Error Identifier is initialized on 1,800 LLM-synthesized, multi-model-filtered examples and refined iteratively with pseudo-labeled real QG outputs. Experiments on QGEval, SimQG, and SQuAD 2.0 with four LLM evaluators report improved Pearson correlation with human judgments and improved answerability classification relative to vanilla CoT baselines, alongside analyses linking EI accuracy to evaluation quality and reduced overestimation of low-quality questions.

Significance. If the causal claim is established, ErrEval is a practically useful, modular contribution: it is plug-and-play for existing LLM evaluators, provides interpretable error diagnostics, and directly targets the well-known overestimation problem in QG evaluation. The paper's strengths are its breadth (three benchmarks, four LLMs, both open and closed models), the iterative training pipeline, the overestimation analysis, and the release of code and resources. The main weakness is attribution: the experimental design does not isolate the effect of the diagnostic labels from the effect of a longer, more structured prompt, and no uncertainty quantification is provided. The synthetic seed-data concern is real but partially mitigated by the human-annotated development set and iterative refinement; it should be tested explicitly rather than assumed.

major comments (3)
  1. [§4.2 / Figure 10] The headline comparison does not isolate the diagnostic labels. Relative to the vanilla prompt, the error-aware prompt (Figure 10, right) adds: (i) a task description saying there are 'pre-identified errors', (ii) an explicit 'Consider Error Labels' step, and (iii) an extra justification step. Every ErrEval condition in Tables 2–4 uses this enriched prompt, while the vanilla condition uses the shorter prompt. Figure 4 only varies the EI checkpoint (Iter0–Iter4), so it shows a correlation between label quality and evaluation quality, not that the label content—rather than prompt structure—causes the gain. Add matched-prompt ablated conditions with identical instructions and either no labels, constant 'No Error' labels, random labels, or the actual EI labels. Without such controls, the abstract claim that 'explicit diagnostics' improve alignment is not established.
  2. [Tables 2–4] No uncertainty quantification is provided. The claims of 'consistent improvement' rest on point estimates, and several per-dimension deltas are negative (Table 2: Qwen3 Relevance −1.7, GPT-4o Relevance −0.5; Table 3: Qwen3 Fluency −0.2, Claude-3.5 Relevance-large −1.1). Without bootstrap confidence intervals, paired significance tests, or inter-run variance, these gains could be noise. Please add uncertainty estimates at least for the average Pearson correlations and the SQuAD F1/accuracy columns, and discuss the negative per-dimension cases explicitly.
  3. [§3.1 / §3.2] The EI's initial supervision is fully synthetic: GPT-4o generates questions with 'the same errors as the example' and three LLMs filter by agreement, with no human verification at initialization. Real QG outputs enter only through iterative pseudo-labeling by the EI and Verifier, plus a small manually verified subset. The human-annotated development set is reassuring (Micro F1 81.2 for RoBERTa-large at Iteration 3), but it does not by itself establish that the model's error distribution matches real QG errors on the evaluation benchmarks. Please report EI performance separately on synthetic vs real-QG examples, or include human labels on real outputs in the initial training data. Without this, the downstream gains could be specific to the synthetic error distribution rather than to genuine error diagnosis.
minor comments (5)
  1. [§3.2 / Appendix F] The uncertainty and inconsistency thresholds are said to be 'reported in Appendix F', but Appendix F does not actually list them. Please provide the exact threshold values and, ideally, a brief sensitivity analysis over them.
  2. [Figure 4] 'Mirco F1' should be 'Micro F1'. The legend and axes should be labeled more clearly (e.g., 'Micro F1 (%)' for the left scale and 'Pearson correlation (%)' for the right).
  3. [Table 4] The column headers 'P M', 'R M', 'F1 M' are unclear; use proper subscripts (e.g., P_M, R_M, F1_M) and define them in the caption.
  4. [Appendix E / Figure 10] The term 'COT' is used inconsistently; standardize to 'CoT' throughout the paper and figures.
  5. [Appendix A / Table 6] Mapping 'No Error' as a type in every dimension reads oddly. Clarify that it is a null label indicating 'no applicable error' rather than a positive error type.

Circularity Check

0 steps flagged

No significant circularity: the central claim is tested against external human judgments, not derived from the paper's own equations.

full rationale

The core claim—that injecting explicit error diagnostics improves alignment with human judgments—is not forced by construction. The Error Identifier is trained on LLM-synthesized error labels and an unlabeled QG-model output pool (Sections 3.1 and 3.2), and the target variable in every main experiment is external human annotation from QGEval, SimQG, or SQuAD 2.0 (Section 4.1), not an equation defined by the paper. The error taxonomy and the error-dimension mapping (Section 3.3, Table 6) are hand-specified semantic bridges; they are not fitted to the human scores being predicted. The closest concern is that the taxonomy is 'Guided by prior error analysis and commonly adopted evaluation dimensions' citing the authors' own QGEval benchmark (Fu et al., 2024b), and QGEval is also one of the three evaluation benchmarks. However, consistent gains are reported on SimQG and SQuAD 2.0, which are external to the authors, so the central claim does not reduce to a self-citation chain. The absence of a prompt-matched control (e.g., labels removed or randomized) and the paper's own Limitations statement that diagnostic signals are 'appended ... without enforcing explicit constraints' and that evaluators 'may occasionally overlook these signals' are experimental-attribution concerns rather than evidence that a fitted input is being renamed as a prediction. No load-bearing step reduces to the paper's own equations, and no fitted parameter is relabeled as an independent prediction. Accordingly, no circular step meets the required evidentiary standard.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The framework rests on a hand-built taxonomy, a hand-built dimension-error mapping, LLM-generated training labels, and human-judgment benchmarks. The main fitted choices are the agreement threshold (0.8), the iterative-refinement selection thresholds, and the checkpoint iteration (Iteration 3). No new physical entities are introduced.

free parameters (3)
  • Multi-model agreement confidence threshold = 0.8
    Section 3.1: initialization samples are retained only if at least two of three LLMs assign confidence >= 0.8. Chosen by hand; affects training data quality and EI performance.
  • Uncertainty/inconsistency selection thresholds = reported in Appendix F (not in provided text)
    Section 3.2: reliable vs unreliable sample split during iterative refinement uses empirically chosen thresholds. Not visible in the main text; affects the training data expansion.
  • Training iteration checkpoint = Iteration 3
    Section 4.1 and Appendix F: adopted because EI performance degrades at Iteration 4. A data-dependent model-selection choice that determines all downstream evaluation results.
axioms (5)
  • ad hoc to paper The 11-type error taxonomy (Table 1) is complete and appropriate for QG errors.
    Section 3.3: the taxonomy is manually designed, guided by QGEval dimensions, with no external validation of completeness or inter-annotator agreement.
  • ad hoc to paper The dimension-error mapping (Table 6) correctly links each error type to affected evaluation dimensions.
    Appendix A: the mapping is authored by hand and used to choose which error labels to inject; if wrong, error-aware prompts can mislead the evaluator.
  • domain assumption LLM-synthesized error labels filtered by multi-model agreement are reliable enough to train an Error Identifier that transfers to real QG outputs.
    Section 3.1: the initial 1,800 samples are generated by GPT-4o and filtered by GPT-4o/Claude-3.5/Gemini-2.0-pro, with no human verification except a 140-sample development set.
  • domain assumption Human judgments in QGEval, SimQG, and SQuAD 2.0 are a valid gold standard for question quality.
    Section 4.1: correlation with human scores is the primary metric; if the human labels are noisy or biased, the measured improvements are not meaningful.
  • domain assumption The Verifier can reliably distinguish correct from incorrect EI predictions.
    Section 3.2 and Table 8: verifier F1 is around 87% at Iteration 3; imperfect filtering can inject noisy labels into the training set.

pith-pipeline@v1.3.0-alltime-deepseek · 22451 in / 15049 out tokens · 162260 ms · 2026-08-03T10:17:57.529159+00:00 · methodology

0 comments
read the original abstract

Automatic Question Generation (QG) often produces outputs with critical defects, such as factual hallucinations and answer mismatches. However, existing evaluation methods, including LLM-based evaluators, mainly adopt a black-box and holistic paradigm without explicit error modeling, leading to the neglect of such defects and overestimation of question quality. To address this issue, we propose ErrEval, a flexible and Error-aware Evaluation framework that enhances QG evaluation through explicit error diagnostics. Specifically, ErrEval reformulates evaluation as a two-stage process of error diagnosis followed by informed scoring. At the first stage, a lightweight plug-and-play Error Identifier detects and categorizes common errors across structural, linguistic, and content-related aspects. These diagnostic signals are then incorporated as explicit evidence to guide LLM evaluators toward more fine-grained and grounded judgments. Extensive experiments on three benchmarks demonstrate the effectiveness of ErrEval, showing that incorporating explicit diagnostics improves alignment with human judgments. Further analyses confirm that ErrEval effectively mitigates the overestimation of low-quality questions.

Figures

Figures reproduced from arXiv: 2601.10406 by Bifan Wei, Bo Li, Jian Zhang, Jiaxin Wang, Jingyi Hao, Jun Liu, Lingling Zhang, Weiping Fu, Yu He, Yushun Zhang.

Figure 1
Figure 1. Figure 1: Comparison between vanilla CoT and error [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Pilot experiment - Pearson correlation coef [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Framework of ErrEval. Given a passage (p [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Effect of EI accuracy on evaluation result [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The overestimation rates of Vanilla Prompt [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Prompt template used for error simulation in [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: The distribution of eleven error types. All models are implemented using the Hugging Face Transformers framework and trained on a sin￾gle NVIDIA A800 GPU. We set the maximum input length to 512 tokens and use a learning rate of 2e-5. For RoBERTa-base, we use a batch size of 32, while for RoBERTa-large, we use a batch size of 16. The maximum number of training epochs is set to 20, with early stopping enable… view at source ↗
Figure 7
Figure 7. Figure 7: Prompt template used for filtering data in [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Distribution of score changes (∆ = ErrEval− Vanilla) on vanilla-correct samples. scores change after applying ErrEval [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Prompt templates used for LLM-based evaluation. [PITH_FULL_IMAGE:figures/full_fig_p017_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Case 1: a case of EI correctly identifying errors and guiding the evaluator to assign an accurate score. [PITH_FULL_IMAGE:figures/full_fig_p017_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Case 2: a case of EI correctly identifying No Error and guiding the evaluator to assign an accurate score. [PITH_FULL_IMAGE:figures/full_fig_p018_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Case 3: a case where EI incorrectly identifies an error, but the evaluator disregards it and assigns an [PITH_FULL_IMAGE:figures/full_fig_p019_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Case 4: a case where EI incorrectly identifies an error, and the evaluator follows it and assigns an [PITH_FULL_IMAGE:figures/full_fig_p020_14.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Towards Efficient and Robust Linguistic Emotion Diagnosis for Mental Health via Multi-Agent Instruction Refinement

    cs.AI 2026-01 reject novelty 4.0

    A multi-agent prompt-rewriting loop is claimed to improve LLM emotion diagnosis accuracy, but its evaluation appears to optimize on the test set and lacks replication details.

Reference graph

Works this paper leans on

36 extracted references · 2 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Factual Error

    Incorrect labels: One or more labels are wrong (e.g., Given "Factual Error", but it should be "Spell Error")

  2. [2]

    InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13274–13292, Miami, Florida, USA

    Learning personalized alignment for evalu- ating open-ended text generation. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13274–13292, Miami, Florida, USA. Association for Computational Linguistics. Xiaoqiang Wang, Bang Liu, Siliang Tang, and Lingfei Wu. 2022. QRelScore: Better evaluating gener- ated ques...

    Pith/arXiv arXiv 2024

  3. [3]

    If any of the above happens, do not accept

    Missing labels: One or more correct labels are not included. If any of the above happens, do not accept. You should return a confidence score between 0.0 and 1.0, representing how likely it is that the labels are completely correct. Please consider all three cases carefully and score accordingly: - Score 1.0 if the labels are perfectly correct (no missing...

  4. [4]

    InProceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 8989–9010, Toronto, Canada

    Synthesize, prompt and transfer: Zero-shot conversational question generation with pre-trained language model. InProceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 8989–9010, Toronto, Canada. Association for Computational Lin- guistics. Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian ...

    2020

  5. [5]

    Vague: Too broad, unclear or ambiguous

  6. [6]

    Factual Error: Contradicts passage facts

  7. [7]

    Incomplete

    Over-prediction: One or more labels are added but should not be.(e.g., only "Incomplete" is correct, but also given "Factual Error"). Special case: If the question is correct and valid without any errors, then no error labels should be present. Predicting any error labels make the case Not Accept

  8. [8]

    Unnecessary Copy from Passage: Verbose and overquotes passage

  9. [9]

    Incomplete: Ends abruptly, unfinished

  10. [10]

    Not A Question: Not an interrogative sentence

  11. [11]

    Spell Error: Misspelled words

  12. [12]

    Grammar Error: Incorrect grammar

  13. [15]

    Information Not Mentioned: Asks about non-existent info

  14. [17]

    Off Target Answer: Doesn’t match answer

  15. [18]

    Off Topic: Irrelevant to passage. Passage {{PASSAGE}} Answer {{ANSWER}} Question {{QUESTION}} Error Labels {{ERROR}} Accept Confidence (Only return one score, no explanations, comments, or extra text): Figure 7: Prompt template used for filtering data in initialization. issues, thereby producing more precise and inter- pretable evaluation results. F Imple...

  16. [21]

    Justify the Score: Provide a clear and concise explanation, referencing the passage, answer, and question to support your reasoning. COT Prompt Input Passage: {passage} Answer: {answer} Generated Question: {question} Error-aware Prompt [Task Description] You will be given a passage, an answer, a generated question, and a list of pre-identified errors rele...

  17. [23]

    Verify {Dimension Name}: {Dimension Evaluation Requirement}

  18. [25]

    have more wealth than half of all Americans combined

    Justify the Score: Provide a clear and concise explanation, referencing the question, answer, passage and the reasoning behind your rating. COT Prompt with Error Information Input Passage: {passage} Answer: {answer} Generated Question: {question} Relevant Error Labels: {error types and their definitions} Output Score: [Your score] Reason: [Your reason her...

    2014

  19. [28]

    [Input] Passage: {passage} Answer: {answer} Generated Question: {question} [Output] score: 3 reason: The generated question can be answered definitively based on the given passage

    Justify the Score: Provide a clear and concise explanation, referencing the question, passage and the reasoning behind your rating. [Input] Passage: {passage} Answer: {answer} Generated Question: {question} [Output] score: 3 reason: The generated question can be answered definitively based on the given passage. The passage states that the top 400 richest ...

  20. [30]

    Verify Answerability: Determine if the question can be explicitly answered or reasonably inferred from the passage

  21. [32]

    How many percent of Americans are not rich?

    Justify the Score: Provide a clear and concise explanation, referencing the question, passage and the reasoning behind your rating. [Input] Passage: {passage} Answer: {answer} Generated Question: {question} Relevant Error Labels: - Information Not Mentioned: Asks for information not present in the passage. [Output] score: 1 reason: The generated question ...

  22. [35]

    Are Fitness and Gael both based in the United States?

    Justify the Score: Provide a clear and concise explanation, referencing the passage, answer, and question to support your reasoning. [Input] Passage: {passage} Answer: {answer} Generated Question: {question} [Output] score: 1 reason: The generated question "Are Fitness and Gael both based in the United States?" does not align with the provided answer ""no...

  23. [39]

    Are Fitness and Gael both based in the United States?

    Justify the Score: Provide a clear and concise explanation, referencing the question, answer, passage and the reasoning behind your rating. [Input] Passage: {passage} Answer: {answer} Generated Question: {question} Relevant Error Labels: - No Error: The question is clear, relevant, and answerable without any issues. [Output] score: 3 reason: The generated...

  24. [40]

    Verify Answer Consistecy: Evaluate whether the generated question aligns with the provided answer and determine if the answer fully, partially, or fails to address it

  25. [42]

    Antigone

    Justify the Score: Provide a clear and concise explanation, referencing the passage, answer, and question to support your reasoning. [Input] Passage: {passage} Answer: {answer} Generated Question: {question} [Output] score: 3 reason: The generated question aligns perfectly with the provided answer, as the answer "Antigone" is the name of the play by Sopho...

  26. [44]

    Verify Answer Consistency: Evaluate whether the generated question aligns with the provided answer and determine if the answer fully, partially, or fails to address it

  27. [46]

    What is the name of Sophocles' play about civil disobedience?

    Justify the Score: Provide a clear and concise explanation, referencing the question, answer, passage and the reasoning behind your rating. [Input] Passage: {passage} Answer: {answer} Generated Question: {question} Relevant Error Labels: - Off Target Answer: Does not align with the provided answer. [Output] score: 3 reason: The generated question "What is...

  28. [47]

    Verify Consistency: Check whether all factual claims made in the question are consistent with the information presented in the passage

  29. [49]

    Which genus has more species, Butea or Pinus?

    Justify the Score: Provide a clear and concise explanation, referencing the question, passage and the reasoning behind your rating. [Input] Passage: {passage} Answer: {answer} Generated Question: {question} [Output] score: 3 reason: The generated question "Which genus has more species, Butea or Pinus?" is entirely consistent with the passage. The passage ...

  30. [50]

    Consider Error Labels (if provided): Review the error labels related to this dimension, but note that they may not be fully accurate and should only be used as a reference

  31. [51]

    Verify consistency: Check whether all factual claims made in the question are consistent with the information presented in the passage

  32. [52]

    Assign a Score: Rate the question according to the scoring criteria

  33. [53]

    [Input] Passage: {passage} Answer: {answer} Generated Question: {question} Relevant Error Labels: - Factual Error: Includes incorrect facts that contradict the passage

    Justify the Score: Provide a clear and concise explanation, referencing the question, passage and the reasoning behind your rating. [Input] Passage: {passage} Answer: {answer} Generated Question: {question} Relevant Error Labels: - Factual Error: Includes incorrect facts that contradict the passage. [Output] Score: 1 Reason: The generated question contain...

  34. [2023]

    In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5967–5994, Singapore

    INSTRUCTSCORE: Towards explainable text generation evaluation with automatic feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5967–5994, Singapore. Association for Computa- tional Linguistics. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- phe...

    2023

  35. [2024]

    InProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24, pages 8038–8047

    A survey on neural question generation: Meth- ods, applications, and prospects. InProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24, pages 8038–8047. International Joint Conferences on Artificial Intelli- gence Organization. Survey Track. Dongfu Jiang, Yishan Li, Ge Zhang, Wenhao Huang, Bill Yuchen Lin, an...

    Pith/arXiv arXiv 2023

  36. [2025]

    What is the cause of

    JudgeLM: Fine-tuned large language models are scalable judges. InThe Thirteenth International Conference on Learning Representations. A Error-Dimension Mapping We design a comprehensive taxonomy of error types presented in generated questions, and map them with different evaluation dimensions. The full mappings between error types and common evalua- tion ...

    2022