{"id":"c980c150-4502-405b-ab8c-760594c7317b","arxiv_id":"2504.18413","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"In long-form QA, LLM-based evaluators correlate with human judgments better than ROUGE or BERTScore, but they are biased by answer length, question type, self-reinforcement, and rare-word usage, and fine-grained prompting reduces some of these problems.","lead":"This paper compares automatic metrics, from string-overlap scores to large language model judges, against human ratings of long-form question answers. It finds that LLM judges align better with humans, but are biased by answer length, question type, and their own outputs, and that fine-grained prompts reduce some of these biases.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Human reference labels are unvalidated: no inter-annotator agreement or annotator count is reported, so the headline correlation gains (42.0 to 55.0) rest on an unmeasured ground truth.","rationale":"Without a validated human target, the paper's central comparison is uninterpretable: a metric can only align with human judgments if those judgments are themselves reliable. The authors report 4,158 ratings but no annotator count, no training details, no adjudication protocol, and no agreement coefficient. This is especially important because the two largest effects, LLM-based metrics over deterministic metrics and fine-grained over coarse, are correlation differences of about 10 to 40 percentage points on samples of 343 to 1,386 answers drawn from only 50 to 200 questions; unmodeled measurement error in the reference could easily move coefficients by that amount. Other issues, such as prompt-order sensitivity in Section 4 and the uncontrolled self-preference analysis in Section 3.4.3, are real but secondary: they affect specific attributions, whereas the human-label gap affects every quantitative claim in the paper. The reader's weakest_assumption identifies the same issue, so I agree with the CONDITIONAL verdict. The qualitative direction is plausible and the released code and data make the check feasible, but the headline numbers should not be treated as firm until the human reference is validated.","tokens_in":17839,"tokens_out":10448,"duration_ms":100626,"concrete_test":"Select a random subsample of at least 100 QA pairs spanning the three datasets (for example, 50 ASQA, 25 ANTIQUE, and 25 WikiEval), have two or more independent annotators re-rate correctness and informativeness using the same interface and instructions, and compute per-dimension Cohen's kappa or Krippendorff's alpha. If agreement is below roughly 0.4, the human reference is too unreliable to support the reported correlations; re-estimate the Table 1 and Table 2 coefficients against adjudicated or averaged labels and check whether fine-grained GPT-4o still beats coarse GPT-4o and the best deterministic metric by a comparable margin. If agreement exceeds roughly 0.6 to 0.7, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Every quantitative conclusion in Sections 3.2 through 3.4 is a correlation or agreement with human ratings of correctness and informativeness, yet Section 3.1.3 never reports how many annotators produced the 2,079 labels, whether each answer was rated once or multiply, how disagreements were resolved, or any agreement statistic such as Cohen's kappa or Krippendorff's alpha. The Conclusion acknowledges the single-point scale as a limitation but does not address annotator reliability. If the human labels are noisy, the Spearman and Kendall coefficients in Tables 1 and 2 are attenuated estimates of alignment with an unreliable reference; if annotator leniency or strictness is correlated with answer length, model identity, or question type, the bias analyses in Sections 3.3 and 3.4 can be confounded rather than revealed. The headline claim that fine-grained GPT-4o raises correlation from 42.0 to 55.0 therefore presupposes a stable, valid human target that the paper does not demonstrate. This is a load-bearing gap, not a stylistic preference.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical meta-evaluation of automatic metrics for long-form question answering (LFQA). Using answers generated by seven LLMs on ASQA, ANTIQUE, and WikiEval, the authors collect human ratings of correctness and informativeness for 2,079 QA pairs and compare these with deterministic metrics (Rouge-L, Exact Match, Disambig-F1, BERTScore, RAGAS answer relevance) and LLM-based judges (GPT-4o, Claude-3.5, Gemini-2.0, in coarse-grained and fine-grained variants). The main findings are that LLM-based judges correlate more strongly with human ratings than deterministic metrics, that deterministic metrics are sensitive to answer length and question type, that LLM judges exhibit self-preference and other biases, and that fine-grained prompt design improves LLM-judge agreement with humans, e.g., GPT-4o improves from 42.0 to 55.0 Spearman correlation on ASQA. The paper also proposes decomposing prompts into task, data, output, and criteria components. Code and data are released.","tokens_in":18021,"tokens_out":6057,"duration_ms":56965,"significance":"If the results are reliable, this is a practically useful contribution: it provides a moderate-sized human-annotated LFQA evaluation resource (2,079 QA pairs, 4,158 ratings), compares a broad range of metrics on three diverse datasets, and offers concrete evidence on the value of fine-grained LLM-based evaluation. The analysis of length, question-type, and self-preference biases is relevant to anyone using LLM judges. Credit is due for the explicit release of code and data and for acknowledging limitations such as the single-point rating scale. However, the quantitative conclusions currently rest on a human ground truth whose reliability is not reported, and several aggregate numbers in the main tables are internally inconsistent. These issues must be resolved before the headline claims can be accepted.","major_comments":[{"comment":"The human annotations are the reference for every correlation and bias analysis in Sections 3.2 through 3.4, but the manuscript reports no annotator count, no redundancy in labeling, no adjudication procedure, and no inter-annotator agreement statistic such as Cohen's kappa or Krippendorff's alpha. Without this information, the reported Spearman and Kendall coefficients are correlations against an unvalidated target, and any annotator leniency, drift, or answer-length-related bias is confounded with metric behavior. The Conclusion acknowledges the single-point scale as a limitation but not annotator reliability. To make the central claims load-bearing, the authors should report the annotation protocol in detail or, if the data cannot be recovered, substantially temper the quantitative claims and provide an uncertainty analysis.","section":"3.1.3"},{"comment":"The Average rows in Tables 1 and 2 are not the means of the model-level rows printed in the same tables. For example, in Table 2 the ANTIQUE CG average is printed as 83.0, but the mean of the seven listed values (36.7, 53.2, 74.9, 69.3, 65.6, 73.0, 54.3) is 61.0; the printed ASQA GPT-4o average of 42.0 differs from the mean of 40.5; and the ASQA EM average of 41.4 differs from the mean of 42.9. If the averages are computed through a nonlinear transformation such as Fisher z-transformation and back-transformation, this must be stated and justified; otherwise the reader cannot interpret the aggregates. This issue directly affects the headline claim in Section 3.2 that fine-grained GPT-4o improves from 42.0 to 55.0, since both numbers come from these average rows.","section":"Tables 1 and 2"},{"comment":"The manuscript repeatedly states that LLM-based metrics show \"significantly higher consistency\" with human judgments, but no significance tests, confidence intervals, or effect-size uncertainties are reported. The per-model sample sizes differ across datasets (about 50 answers per model for ASQA and WikiEval versus about 200 per model for ANTIQUE), and many of the reported differences, for instance the 42.0 versus 33.0 difference between GPT-4o and Claude-3.5 on ASQA, could be within sampling noise. The authors should provide confidence intervals or at least a significance test for the main correlation comparisons, or soften the language accordingly.","section":"3.2"},{"comment":"The self-reinforcement analysis concludes that LLM-based evaluators \"assign significantly higher scores to their own outputs,\" but the evidence is win-rate comparisons from Figure 6 without a statistical test and without a control for genuine answer-quality differences. If GPT-4o, Claude-3.5, and Gemini-2.0 simply produce better answers to most ASQA questions, the same win-rate pattern would be observed even with a perfectly unbiased judge. The claim of a \"clear evaluation bias\" therefore needs a stronger design, such as comparing judges on outputs of matched quality, or at least an inter-judge consistency analysis that accounts for model strength. As written, the bias conclusion is not fully supported.","section":"3.4.3"}],"minor_comments":[{"comment":"There is a typo in \"abstract lonng-form summarization\" (should be \"long-form\").","section":"2.1"},{"comment":"The phrase \"fatoid QA\" should be \"factoid QA.\"","section":"3.2"},{"comment":"The sentence \"A high-quality response should should be informative enough\" contains a duplicated \"should.\"","section":"3.1.3"},{"comment":"Table 5 is difficult to parse: the header \"Kendall (%) LLMs Prompts\" mixes the three default LLM-judge columns with the nine prompt-condition columns, and the caption says \"Variation of correlation coefficients\" but the table appears to show absolute correlations. Please restructure the table into clear blocks and define exactly what each column reports.","section":"Table 5"},{"comment":"Figure 6 is visually cluttered because the three win-rate matrices are overlaid and the row/column labels are partially repeated. Separate subplots or a table with model labels would make the self-reinforcement comparison readable.","section":"Figure 6"},{"comment":"Table 3 reports temperature sensitivity for ASQA and WikiEval but not ANTIQUE; the text does not explain this omission. Please either include ANTIQUE or state why it is excluded.","section":"3.3.2"},{"comment":"The IDF analysis in Section 3.4.4 is qualitative. Please specify the exact statistic used (e.g., Spearman correlation between average IDF and metric score) and report the corresponding values, since the text uses words like \"strongly correlate\" without numerical support.","section":"3.4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful empirical contribution and a released annotation resource, but the unvalidated human ground truth and the internally inconsistent Average rows in the central tables are serious issues. I would like to see a revision that addresses these before considering publication. I do not believe this rises to outright rejection, since the errors appear fixable: the authors can report their annotation protocol if the data still exist, correct or explain the aggregations, and add uncertainty estimates or soften the significance claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read this: it's a genuinely useful empirical study, and its headline numbers are not to be trusted as printed. The per-model correlations in Tables 1 and 2 show a clear pattern: fine-grained GPT-4o correlates with human judgments much better than ROUGE, BERTScore, or exact match on ASQA and ANTIQUE. That is real, new evidence for a practical claim that matters.\n\nThe paper does several things well. It collects 2,079 human-annotated QA pairs across seven different generators and three datasets, which is a non-trivial resource. The robustness analysis (prompt perturbation, temperature) and the fairness analyses (length, question type, self-reinforcement, IDF) are genuinely new measurements beyond the prior meta-evaluation by Xu et al. The prompt-component ablation is also useful: it shows all four components matter and that order matters. These are the kind of results that people building LLM judges will want to cite.\n\nBut the soft spots are real. The average rows in Tables 1 and 2 do not match the means of the rows above them. For example, on ASQA, the mean of the seven RL Spearman values is 17.8, not the printed 11.4; the mean of the EM values is 42.9, not 41.4. The abstract's flagship '42.0 to 55.0' improvement is one of those average rows, so the headline is on shaky arithmetic.\n\nSecond, and more load-bearing, the human annotations are the reference for every correlation and bias analysis, but the paper reports no inter-annotator agreement, no annotator count, and doesn't say whether answers were scored once or multiple times. If those labels are noisy or systematically lenient/strict in ways that correlate with answer length or model identity, the bias analyses could be confounded. The conclusion acknowledges the single-point scale but doesn't address annotator reliability.\n\nThe self-reinforcement analysis in Section 3.4.3 also lacks a quality control: it shows GPT-4o gives its own outputs higher win rates, but without knowing whether GPT-4o's outputs actually are better, the 'bias' is just a preference, not an error. The paper does note rankings stay consistent, which softens this.\n\nOverall, the central direction—fine-grained LLM judges align better than deterministic metrics, and they carry biases—holds up in the per-model data. But the specific numbers need re-verification and the annotation methodology needs to be reported. I'd send this to a careful referee who can recompute the tables, but I'd also want the authors to provide the annotation details before treating any of these numbers as reliable.","headline":"Useful meta-evaluation with a clear practical message, but the headline averages are miscalculated and the human ground truth is unreported.","tokens_in":18562,"tokens_out":3299,"would_cite":true,"duration_ms":29878,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-grained GPT-4o judging aligns with human scores of long-form answers far better than ROUGE or BERTScore, a 2,079-answer study finds, while LLM judges show their own biases.","keywords":["long-form question answering","LLM-as-a-judge","automatic evaluation","fine-grained evaluation","human evaluation","evaluation bias","meta-evaluation","correlation analysis"],"falsifier":"Re-run the meta-evaluation on a fresh sample of long-form answers labeled by multiple independent annotators with measured inter-annotator agreement; if fine-grained GPT-4o's Spearman correlation with the multi-annotator gold scores does not significantly exceed ROUGE-L and BERTScore, the paper's central conclusion fails. A cheaper check on the released data is to estimate label noise from a small re-annotation subset and test whether the reported 55.0 versus 42.0 fine-grained gain survives noise correction.","tokens_in":1693,"feed_emoji":"📊","tokens_out":1492,"duration_ms":52665,"temperature":0.7,"pith_summary":"The paper tests whether automatic metrics can stand in for human judgment when grading long-form answers. It collects 5,236 answers from seven LLMs across three datasets and has humans rate 2,079 of them for correctness and informativeness. It finds that LLM-based judges, especially GPT-4o with fine-grained criteria, align with human ratings far better than ROUGE, Exact Match, and BERTScore, which are unstable across question types and penalize longer answers. The paper also documents biases in LLM judges, including self-preference, length, and prompt sensitivity, and shows that fine-grained prompts and rank-based comparison mitigate them. If right, LFQA evaluation should move from lexical overlap to structured LLM judging while treating scores with caution.","feed_headline":"Fine-grained LLM judges beat ROUGE at grading long answers","feed_subtitle":"Structured GPT-4o prompting tracks human ratings of long-form answers far better than lexical overlap, with caveats.","key_machinery":"The meta-evaluation protocol: 5,236 generated answers from seven LLMs across ASQA, ANTIQUE, and WikiEval, with 2,079 human-rated on correctness and informativeness, are compared against seven automatic metrics using Spearman and Kendall correlations and pairwise win-rate agreement. The improvement mechanism is fine-grained prompting, which decomposes the evaluation prompt into four components (task, data, output requirements, criteria) and supplies explicit per-criterion rubrics; this raises GPT-4o's Spearman correlation from 42.0 to 55.0 and is the central lever the paper identifies for making LLM judges align with humans.","core_discovery":"Fine-grained evaluation with GPT-4o improves its Spearman correlation with human ratings on ASQA from 42.0 to 55.0, far exceeding deterministic metrics like Rouge-L (11.4), Exact Match (41.4), and Disambig-F1 (23.0). The authors argue that providing more detailed instructions, decomposing the judge prompt into task, data, output format, and explicit 1 to 5 rubrics for accuracy and informativeness, is the decisive factor in making LLM-based LFQA evaluation usable. At the same time, they show that LLM judges systematically favor longer answers, give higher scores to their own outputs, shift with prompt wording and temperature, and that no single prompt configuration is optimal for all evaluated models.","pith_inferences":["The paper's self-preference and length-bias results imply that single-judge LFQA leaderboards may systematically rank verbose answers from the judge's own model family higher; a direct test would be to measure rank changes when the judge is held out of the generation pool.","Since fine-grained rubrics raise consistency, a natural next step is to combine the rubric decomposition with claim-level atomic evaluation, splitting answers into verifiable units before scoring, a hybrid the paper does not test but its criteria-decomposition finding directly suggests.","The absence of inter-annotator agreement data means the true ceiling for any automatic metric is unknown; re-annotating a subset of the released 2,079 answers with multiple judges per item would recalibrate all reported correlations and quantify label noise."],"forward_implications":["Deterministic metrics such as ROUGE-L and BERTScore are unreliable as primary LFQA measures, especially for non-factoid and open-ended answers, so results built on them should be interpreted with caution.","LLM judges with fine-grained, multi-criteria instructions can approximate human consistency on correctness and informativeness, making large-scale LFQA evaluation feasible without per-item human annotation.","Because LLM judges favor longer answers and their own outputs, practitioners should rank models rather than trust raw scores, and should avoid letting a model grade its own responses.","Judge prompt structure matters: a four-component prompt (task, data, output, criteria) generally improves consistency, but no single prompt works best for all LLMs, so prompt selection affects leaderboard conclusions.","Judge temperature can change model rankings on tightly scored datasets, so LFQA evaluations should fix a low temperature and report the setting."],"supporting_citations":[{"why":"Provides the ASQA dataset of ambiguous factoid questions with long-form answers, the primary testbed for the correlation analysis.","marker":"[33]"},{"why":"Provides the ANTIQUE dataset of open-ended, non-factoid questions used to test metric behavior across question types.","marker":"[16]"},{"why":"Defines ROUGE, the longest-common-subsequence deterministic metric that the study compares against human judgments.","marker":"[23]"},{"why":"Defines BERTScore, the embedding-based semantic similarity metric that the study evaluates alongside deterministic and LLM-based judges.","marker":"[46]"},{"why":"Supplies the answer-relevance metric from RAGAS and the WikiEval factoid dataset used in the testbed.","marker":"[9]"},{"why":"Supplies the LLM-EVAL prompt-driven GPT-4 judging framework that the study adopts for both coarse-grained and fine-grained evaluations.","marker":"[24]"},{"why":"Provides the two-aspect human annotation scheme (correctness and informativeness) that the authors adapt for their human evaluation.","marker":"[47]"},{"why":"Establishes the win-rate and agreement meta-evaluation methodology for comparing evaluators against human preferences.","marker":"[49]"}],"fun_headline_variants":["Fine-grained prompts make LLM judges match humans better","LLM grading of long answers improves with detailed rubrics","GPT-4o outdoes lexical metrics when prompted with rubrics","Detailed instructions boost LLM-based long-answer evaluation"],"cache_read_input_tokens":20736,"weakest_assumption_plain":"The 2,079 human ratings of correctness and informativeness are treated as reliable ground truth, but the paper reports no inter-annotator agreement, no annotator count, and no validation of the single-point 1 to 5 rating scale, so any noise or bias in those labels propagates into every correlation and bias claim.","fun_headline_variants_meta":{"raw":{"variants":["Fine-grained prompts make LLM judges match humans better","LLM grading of long answers improves with detailed rubrics","GPT-4o outdoes lexical metrics when prompted with rubrics","Detailed instructions boost LLM-based long-answer evaluation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000263,"raw_usage":{"total_tokens":1599,"prompt_tokens":945,"completion_tokens":654,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":587}},"tokens_in":561,"tokens_out":654,"duration_ms":6997,"temperature":1.0,"reasoning_tokens":587,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:17:03.281300+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the meta-evaluation on a fresh sample of long-form answers labeled by multiple independent annotators with measured inter-annotator agreement; if fine-grained GPT-4o's Spearman correlation with the multi-annotator gold scores does not significantly exceed ROUGE-L and BERTScore, the paper's central conclusion fails. A cheaper check on the released data is to estimate label noise from a small re-annotation subset and test whether the reported 55.0 versus 42.0 fine-grained gain survives noise correction.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ANTIQUE dataset of open-ended, non-factoid questions used to test metric behavior across question types."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the win-rate and agreement meta-evaluation methodology for comparing evaluators against human preferences."}],"review_version":1}