{"id":"793856a6-1189-4fc8-b2ce-fa4c7a1f0865","arxiv_id":"2506.15215","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"MinosEval improves open-ended QA evaluation by sorting questions into factoid and non-factoid and applying tailored scoring, outperforming baselines on four datasets.","lead":"MinosEval is a two-step method for grading open-ended AI answers: it first decides whether a question is fact-seeking or creative, then applies different scoring strategies to each type. The authors report that it matches human quality rankings better than existing automatic evaluators on four question-answering datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gold rankings behind the two self-built datasets are a three-person consensus with no inter-annotator agreement; if that gold is noisy, every reported correlation gain is unstable.","rationale":"The reader's weakest assumption is exactly this: gold ranking reliability. I agree. The central claim is empirically grounded; no machine-checked proof or parameter-free derivation is offered. The evaluation protocol's only human ground truth is a small consensus, and missing inter-annotator agreement is not a stylistic omission — it controls whether the dependent variable is well-defined. Other concerns (baseline parity, TREC-DL underperformance, unmeasured interpretability) are secondary because they can be addressed without invalidating the comparison, whereas a noisy gold standard undermines all correlation numbers. I recommend keeping the verdict CONDITIONAL: the method is plausible and the tables are internally consistent, but the self-built dataset annotations must be shown reliable (or released for independent re-annotation) before the claim 'better aligns with human annotations' is accepted. If the proposed check fails, the verdict should move toward REJECT or UNVERDICTED for the empirical claim.","tokens_in":25369,"tokens_out":4446,"duration_ms":46023,"concrete_test":"Obtain the pre-consensus rankings for AlignBench_Minos and GaokaoBench_Minos from the three human annotators (or re-annotate a random subset of, say, 100 samples with at least three independent judges) and compute pairwise Kendall's tau and Krippendorff's alpha before any discussion. If mean pairwise tau < 0.6 or alpha < 0.5, recompute Table 2 against a majority-vote or independently re-annotated gold; if MinosEval no longer leads listwise/pairwise baselines, the headline empirical claim is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim — that MinosEval ranks open-ended answers more like humans than existing evaluators — is measured against gold rankings on AlignBench_Minos and GaokaoBench_Minos produced by one PhD student, two Master's students, and GPT-4o (Section 4.1). The paper reports no inter-annotator agreement, and the only reliability signal given is that 98.39% of GPT-4o's initial rankings were modified by humans. That statistic shows GPT-4o alone was not trusted, but it does not establish that the post-discussion consensus is a stable target. With three annotators and discussion-based consensus, the resulting gold rankings can reflect one team's idiosyncratic preferences, and all baseline comparisons in Table 2 inherit that subjectivity. The gains on the self-built datasets (e.g., Spearman 54.89 vs 51.46 for plain listwise on AlignBench_Minos) are point estimates without variance or reliability bounds, so we cannot tell whether they exceed the noise in the gold standard. This concern is load-bearing because the method's advantage is most visible on these larger self-built datasets, and the paper provides no formal verification or independent benchmark to substitute for the gold rankings.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MinosEval, a two-stage LLM-based evaluation framework for open-ended QA. In the first stage, an LLM classifies each question as factoid or non-factoid. For factoid questions, the method extracts key points from the reference answer and computes an average NLI entailment-minus-contradiction score for each candidate response against those key points (Eqs. 2-3). For non-factoid questions, it generates five quality-graded silver instances and uses an LLM ranker to produce a listwise ordering informed by those instances (Eq. 4). The method is evaluated on four datasets: two sampled English datasets (ANTIQUE_S5, TREC-DL-NF_S5) and two newly constructed Chinese datasets (AlignBench_Minos, GaokaoBench_Minos) with six candidate responses per question and consensus gold rankings. The paper reports Kendall's tau, Spearman's rho, and RBO against the gold rankings, together with ablations, robustness checks, cost comparisons, and error analyses.","tokens_in":25575,"tokens_out":5598,"duration_ms":55410,"significance":"The central idea—that factoid and non-factoid open-ended questions require different evaluation protocols—is well motivated, and MinosEval gives a concrete, fully automated operationalization. The public release of the datasets, code, and prompt templates is a practical strength. The ablation study is useful, particularly the comparisons of key-point scoring against direct NLI, the directionality analysis for premise/hypothesis, and the sweep over different LLM and NLI backends. I do not find a formal circularity in the evaluation: the headline correlations are computed against human annotations, and no constants are fitted to the test data. The more concrete version of that concern is that GPT-4o participated in producing both the gold rankings and the evaluation artifacts (key points and silver instances), so the risk of model-family self-confirmation is real. The empirical support is weakened by the absence of inter-annotator agreement for the gold rankings and by the lack of variance or significance estimates on the two self-built datasets, so the reported gains should be treated as promising but not yet fully established.","major_comments":[{"comment":"The gold rankings for AlignBench_Minos and GaokaoBench_Minos are described as a consensus among one PhD student, two Master's students, and GPT-4o, reached after independent annotation and discussion, with 98.39% of GPT-4o's initial rankings modified by humans. No inter-annotator agreement is reported. This is load-bearing because every headline comparison in Table 2 is a correlation against this single consensus ranking. The 98.39% statistic shows that GPT-4o alone was not trusted, but it does not quantify agreement among the three human annotators or between the human consensus and GPT-4o. Without a reliability estimate (e.g., pairwise Kendall's tau among annotators or Krippendorff's alpha), it is impossible to tell whether the margins in Table 2 (e.g., Spearman 54.89 vs 51.46 on AlignBench_Minos) exceed the noise in the gold standard. Please report human-human and human-GPT-4o agreement and, if feasible, recompute the main comparisons against a human-only gold ranking or a version that excludes GPT-4o's contribution.","section":"4.1"},{"comment":"Tables 2 and B5 report only point estimates for each method on AlignBench_Minos and GaokaoBench_Minos, with no confidence intervals, bootstrap errors, or significance tests. On GaokaoBench_Minos the gap between MinosEval and plain listwise is small (Spearman 65.77 vs 64.66), and several RBO differences are under one point. Because the gold ranking is itself a single consensus, the reader cannot tell whether these differences are meaningful. Please provide variability estimates across items (e.g., bootstrap over questions), report paired significance tests for Kendall and Spearman, or explain why such tests are not applicable.","section":"Tables 2 and B5"},{"comment":"Section 4.2 states that 'GPT-4 was used as the base model' for all LLM-based methods in the main experiments, while Section 3.2.1 and Table 6 specify GPT-4o for fact detection and Table C2 lists GPT-4o-2024-08-06 for the MinosEval instantiations. If the main experiments used GPT-4o for MinosEval but GPT-4 for the baselines, the comparisons in Tables 2 and 3 are not fully controlled. If GPT-4 was used for all methods, the descriptions of the fact-detection, key-point, and silver-instance modules are inconsistent with the implementation details. Please clarify the exact model used for each module in the main experiments and, if the models differ, rerun the baselines with the same model or report the per-model comparison explicitly.","section":"4.2 vs 3.2.1 and Table C2"}],"minor_comments":[{"comment":"In Table B3, the RBO(p=0.9) value for DeepSeekV2.5+mDeBERTa† is 52.28, far outside the range of the surrounding rows (all near 86), which appears to be a typographical or transcription error and should be corrected.","section":"Table B3"},{"comment":"Table B5 has typos in the header and surrounding text: 'Fctoid / Non-Factoid' and 'GaokaoBnech_Minos', and the cell formatting makes several values difficult to read; please clean up the table and run-on cells.","section":"Table B5"},{"comment":"In Section 2.2, 'BERTScore (Zhang et al.) easuring semantic similarity' contains a typo ('easuring' should be 'measuring'); the same section uses 'golden standard' where 'gold standard' is more conventional.","section":"2.2"},{"comment":"The term 'sliver' appears instead of 'silver' in Section 3.2.3 and in the 'SliverInstances' label in Figure 2; this should be made consistent throughout, including in the prompts shown in the appendix.","section":"3.2.3 and Figure 2"},{"comment":"The typo 'AlginBench_Minos' appears in Sections 4.5, 4.7, and Figure 3; please standardize the dataset name to AlignBench_Minos.","section":"4.5"}],"recommendation":"major_revision","confidential_remarks":"The paper is a plausible contribution to LLM-as-a-judge evaluation, and the two-stage factoid/non-factoid design is worthwhile. The main obstacle is empirical: the gold datasets lack inter-annotator reliability evidence, and the self-built dataset comparisons lack variance or significance estimates. These issues are fixable within the manuscript's scope, so I recommend major revision rather than rejection. The model-identity inconsistency in Section 4.2 must also be resolved before the controlled comparison can be trusted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What should you know about this paper: it proposes MinosEval, a two-stage evaluator that first classifies open-ended questions as factoid or non-factoid, then applies key-point scoring with an NLI model for factoid questions and instance-aware listwise ranking (with LLM-generated silver quality anchors) for non-factoid ones. The genuinely new pieces are the routing itself and the silver-anchor listwise step; the components are known from FActScore, RefChecker, and LINKAGE. They also release two self-built datasets with six candidate responses per question, which fills a real gap.\n\nThe paper does a lot right. The baseline selection is fair and broad, including specialized judges like PROMETHEUS 2 and AUTO-J. The gains on the larger self-built datasets are consistent across metrics, and the ablation convincingly shows why routing matters: adaptive key-point scoring alone collapses on non-factoid questions, while instance-aware listwise ranking generalizes but benefits from the split. The few-shot fact detection at over 90% accuracy is a practical plus. Cost analysis, error analysis, and the code/data release are all there.\n\nThe soft spots are real but not disqualifying. The biggest one is the gold rankings on AlignBench_Minos and GaokaoBench_Minos: three annotators plus GPT-4o, discussion-based consensus, and no reported inter-annotator agreement. The 98.39% modification rate for GPT-4o's initial rankings tells us GPT-4o alone wasn't trusted, but it doesn't establish that the consensus is a stable target. Because the method's largest wins appear on exactly these datasets, and Table 2 reports no variance, I'd want IAA and bootstrapped confidence intervals before leaning hard on those margins. That said, this is a weakness in evidence strength, not a fatal flaw: all baselines are measured against the same gold, so the relative ranking is still informative, just noisier than claimed.\n\nAlso worth flagging: Section 4.2 says GPT-4 was the base model for all LLM methods, while Section 3.2, Table C1, and the ablation use GPT-4o. That looks like a leftover from an earlier draft, but it needs fixing. The underperformance on TREC-DL-NF_S5 on Kendall and Spearman vs plain listwise is a caveat, not a kill shot; the paper's explanation about 2-4 candidates and misordering sensitivity is plausible but deserves more than a paragraph. Finally, the interpretability claim is qualitative — a small human check of whether key points and silver instances actually help would substantiate it.\n\nThere is no formal circularity: the evaluation is against human annotations, and the silver instances are generated by the same model family used for ranking, which is a mild self-confirmation risk but not a load-bearing one. Overall, this is a solid, honest empirical paper aimed at the LLM-evaluation subfield. It deserves a serious referee, and I would accept it conditionally, with the main requests being inter-annotator agreement, variance estimates on the self-built datasets, and a clarified base-model statement.","headline":"A sensible two-stage routing idea for open-ended QA evaluation with solid experiments, but the gold rankings on the self-built datasets need reliability evidence before I'd fully trust the margin of victory.","tokens_in":26124,"tokens_out":2726,"would_cite":true,"duration_ms":28987,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that an evaluation pipeline which classifies open-ended questions as factoid or non-factoid before scoring or ranking answers aligns better with human-annotated quality orders than one-size-fits-all automatic judges.","keywords":["open-ended QA evaluation","factoid vs non-factoid","LLM-as-a-judge","key-point scoring","natural language inference","listwise ranking","silver instances","interpretable evaluation"],"falsifier":"Take a new set of open-ended questions with high-quality human rankings from a large, independent pool of annotators with measured agreement; if MinosEval's correlation with those rankings is not stronger than plain listwise ranking when the factual/non-factual split is done automatically, the central claim fails. A second falsifier: if removing the fact/non-fact classifier (feeding all questions to one strategy) does not lower performance on at least one dataset, the distinction is not load-bearing.","tokens_in":25165,"feed_emoji":"⚖️","tokens_out":8924,"duration_ms":79789,"temperature":0.7,"pith_summary":"MinosEval is a two-stage method for grading open-ended answers. It first classifies each question as factoid (an answer with a definite factual core) or non-factoid (creative or subjective). For factoid questions it extracts key points from the reference answer and scores each model response by how strongly it entails those points using an NLI model. For non-factoid questions it generates five quality-graded silver answers and asks an LLM to do listwise ranking against those anchors. Across four datasets the resulting orderings align more closely with human annotations than do lexical metrics, embedding similarity, and standard pointwise/pairwise/listwise LLM judges, and the key points and silver examples make the verdicts interpretable.","feed_headline":"Splitting questions by type makes LLM judges align better with humans","feed_subtitle":"It sorts questions into factoid and non-factoid, then scores or ranks answers by type, matching human agreement better.","key_machinery":"The engine of MinosEval is the two-branch scoring design. The factoid branch, adaptive key-point scoring, extracts key points $K$ from the question and reference answer and scores each response $r_i$ by $\\frac{1}{|K|}\\sum_{k_j \\in K} \\mathrm{NLI}(r_i, k_j)$, where $\\mathrm{NLI}(r_i, k_j) = s_{e,ij} - s_{c,ij}$ is entailment probability minus contradiction probability from a multilingual NLI model (mDeBERTa-v3-base-mnli-xnli). The non-factoid branch, instance-aware listwise ranking, generates five silver instances $I_i$ at quality levels excellent, good, fair, poor, and bad, then hands question, reference, instances, and candidate list to an LLM ranker that outputs an ordered list $R_{nf} = A_{nf}(q_i, a_i, I_i, R_i)$. The third component is the LLM-based fact detection module that routes each sample to one branch; the paper shows that this routing is what makes the tailored scoring strategies possible, and that using either branch on all questions degrades performance.","core_discovery":"On its own terms, MinosEval claims that the factoid/non-factoid distinction is the missing ingredient in open-ended QA evaluation. The pipeline first runs an LLM-based fact-detection module (over 90% accuracy with five demonstrations) that assigns each question to a factoid cluster or a non-factoid cluster. For factoid questions it computes, for every candidate response, the average NLI score across key points extracted from the reference answer, where the NLI score is entailment probability minus contradiction probability; the responses are then sorted by that score. For non-factoid questions it generates five silver answer instances spanning excellent to bad quality and asks an LLM ranker to order the candidates with those instances as anchors. Across four datasets the resulting rankings show higher Kendall's Tau, Spearman's Rho, and Rank-Biased Overlap against human-annotated gold rankings than BLEU, ROUGE, BERTScore, BEM, PEDANTS, pointwise, pairwise, listwise, LINKAGE, FActScore, RefChecker, PROMETHEUS 2, and AUTO-J. The paper also reports that the gap widens when the question split is done by hand rather than by the automatic module, and that on small candidate sets with the automatic module it does not always beat the strongest baselines.","pith_inferences":["A testable extension: the same key points used for ranking could be fed back to an LLM as fine-grained revision hints, turning the evaluator into a grader that tells the generator which facts it missed.","The factoid/non-factoid boundary is likely a spectrum; a soft weighting between the two scoring strategies might handle borderline questions that the paper's hard split misclassifies.","Because the paper finds instance-aware listwise ranking also works reasonably on factoid questions, a cheaper coarse version could drop NLI key-point scoring for all but the most fact-critical evaluations.","The observed reclassification of many 'non-factoid' samples into factoid suggests existing open-ended QA benchmarks may have been evaluated under a mismatched scoring model; re-running past comparisons with the split could change reported model orderings."],"forward_implications":["Factoid answers can be graded transparently: the extracted key points show exactly which required facts a response entailed and which it contradicted.","Non-factoid answers are ranked with explicit quality anchors (excellent, good, fair, poor, bad silver instances), making the LLM judge's ordering easier to audit.","Automatic fact detection reaches over 90% accuracy with five demonstrations, so the full pipeline needs no manual question labeling.","On the larger self-built datasets the gap over pairwise/listwise baselines is largest, suggesting the benefit grows as candidate answers become more numerous and more similar in quality.","Benchmark results on 'non-factoid' datasets that actually contain factoid questions should be read with caution, since a one-size-fits-all judge may rank those subsets poorly."],"supporting_citations":[{"why":"Supplies the factoid-question definition used to motivate the factoid/non-factoid split.","marker":"Jiang et al. (2019)"},{"why":"Source of the ANTIQUE dataset and the prior non-factoid QA framing that the paper partially reclassifies.","marker":"Hashemi et al. (2020)"},{"why":"Provides the LINKAGE baseline and the S5 sampling procedure used to build the English evaluation sets.","marker":"Yang et al. (2024b)"},{"why":"Inspires using NLI entailment scores between a response and decomposed factual key points.","marker":"Bohnet et al. (2022)"},{"why":"Source of AlignBench, the base for the paper's larger Chinese factoid/non-factoid dataset.","marker":"Liu et al. (2023b)"},{"why":"Source of GaokaoBench, the base for the paper's second Chinese evaluation dataset.","marker":"Zhang et al. (2023)"}],"fun_headline_variants":["Type-aware LLM eval beats generic scoring on open-ended QA","Splitting by factoid vs non-factoid boosts LLM judge alignment","MinosEval: tailor LLM scoring by question type for human agreement","Why LLM evaluators improve when they split questions by type","Factoid detection plus tailored ranking improves LLM QA evaluation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the consensus human rankings used as ground truth are the correct quality order; they were produced by a small team with no measured agreement and no independent validation.","fun_headline_variants_meta":{"raw":{"variants":["Type-aware LLM eval beats generic scoring on open-ended QA","Splitting by factoid vs non-factoid boosts LLM judge alignment","MinosEval: tailor LLM scoring by question type for human agreement","Why LLM evaluators improve when they split questions by type","Factoid detection plus tailored ranking improves LLM QA evaluation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000774,"raw_usage":{"total_tokens":3479,"prompt_tokens":1053,"completion_tokens":2426,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":669,"completion_tokens_details":{"reasoning_tokens":2334}},"tokens_in":669,"tokens_out":2426,"duration_ms":19843,"temperature":1.0,"reasoning_tokens":2334,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:40:31.263072+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a new set of open-ended questions with high-quality human rankings from a large, independent pool of annotators with measured agreement; if MinosEval's correlation with those rankings is not stronger than plain listwise ranking when the factual/non-factual split is done automatically, the central claim fails. A second falsifier: if removing the fact/non-fact classifier (feeding all questions to one strategy) does not lower performance on at least one dataset, the distinction is not load-bearing.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the factoid-question definition used to motivate the factoid/non-factoid split."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the ANTIQUE dataset and the prior non-factoid QA framing that the paper partially reclassifies."}],"review_version":2}