{"id":"7e85a7fb-01c7-4fe5-8501-633e337077ea","arxiv_id":"2412.05206","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"ConQRet is a new benchmark plus LLM judge prompts for evaluating retrieval-augmented argumentation, with fine-grained document-level scoring that tracks injected irrelevant context better than single-score baselines.","lead":"This paper introduces ConQRet, a benchmark of 98 controversial topics with human-written arguments on both sides, each grounded in real web sources. It then compares several LLM-based judges for scoring retrieval quality, argument quality, and groundedness, finding that fine-grained document-level prompts track injected noise better than single-score metrics.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Context-relevance validation rests on ProCon citation qrels and cross-topic distractors, so the claimed precision tracking in Table 7 may not transfer to realistic same-topic misleading sources.","rationale":"The paper makes a genuinely useful contribution: a new benchmark for retrieval-augmented argumentation and a systematic comparison of LLM judge prompts. The argument-quality validation on the external ArgQuality corpus is a real strength: a listwise self-consistent GPT-4o judge reaches alpha=0.48 against expert annotators, exceeding the crowd/expert agreement reported by Wachsmuth et al. This result is independent of the qrels issue and gives solid support to the quality-evaluation aspect of the claim. The context-relevance aspect, however, is the load-bearing part for the 'retrieval' in RAArg, and it is validated almost entirely against citation-derived qrels and easy cross-topic distractors. Section 6.4's admission that the judges cannot distinguish finer levels of irrelevant evidence corroborates the concern that the convergence is coarse. The reader's weakest assumption identified the same qrels issue, and the evidence presented here reinforces it by showing that the reported precision errors are quantitatively substantial and the distractor type is not representative of realistic retrieval errors. The appropriate response is to keep the benchmark and the quality-judge findings, but to re-validate context relevance on human judgments and harder distractors before asserting that the fine-grained judge tracks true precision. This is exactly a conditional acceptance posture, so the reader's CONDITIONAL verdict stands unchanged.","tokens_in":24360,"tokens_out":10243,"duration_ms":110120,"concrete_test":"Sample 10 ConQRet topics (~150 documents) and collect per-stance binary relevance labels from two independent annotators with adjudication. Recompute the Table 7 and Table 16 precision and monotonicity analysis using these human labels as ground truth instead of the Section 4.3 citation qrels, and replace injected cross-topic distractors with same-topic documents that were retrieved by the BM25+reranker but not cited by ProCon authors. If the judge's precision deviates from human-labeled precision by more than 0.1 absolute, or if monotonic degradation disappears with same-topic distractors, the claim that the judge provides precision estimates close to true relevance is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the fine-grained LLM judge 'provides precision estimates close to the True Precision' is validated in Tables 7, 15-17 against qrels constructed in Section 4.3: a document is relevant iff it appears in a ProCon topic's source list, and irrelevant documents are randomly sampled from other topics. This creates two problems. First, the labels are citation-derived, not independent human relevance judgments; a source omitted by ProCon authors counts as irrelevant, and a cited but off-topic source counts as relevant, so the judge is validated against a noisy proxy. Second, the injected noise is cross-topic documents, which are trivially distinguishable by surface-level topic overlap; monotonic degradation under such noise does not demonstrate sensitivity to realistic retrieval errors, which are typically same-topic documents that are misleading, off-stance, or only partially relevant. Section 6.4 itself states that the judges 'are not able to distinguish reliably between finer levels of non-relevant evidence,' consistent with a coarse validation. The quantitative claim is also weak on its own terms: Table 7 shows absolute errors up to +0.123 at 10% irrelevant content and relative errors of 36-76% at 10-50% irrelevant content, so 'close' is not established by the reported numbers. The paper also does not specify how the judge's ratings are thresholded into binary precision, which further undermines the reproducibility of the estimate. The context-relevance component of the central claim therefore depends on an unproven equivalence between citation labels and true relevance, and on the representativeness of cross-topic distractors.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ConQRet, a new benchmark for retrieval-augmented argumentation built from ProCon.org expert-written pro/con arguments and their cited web sources. It proposes and compares several LLM-as-a-judge prompts that produce fine-grained scores for context relevance, argument groundedness, and 15 argument-quality dimensions, and it validates these judges on the ArgQuality corpus and on ConQRet. The headline results are that a listwise prompt with self-consistency achieves high agreement with expert annotations on ArgQuality, and that a Listwise+RAG Fine-Grained judge tracks a 'True Precision' measure as irrelevant documents are injected. The paper also includes sensitivity analyses for hallucinated sentences and a metric-consistency study.","tokens_in":24538,"tokens_out":7323,"duration_ms":72380,"significance":"The ConQRet resource is potentially valuable: it provides long, human-authored, stance-conditioned arguments paired with real web evidence, an area where existing datasets are short or lack grounding. The ArgQuality validation (Table 5) is the strongest contribution: Listwise+SC with GPT-4o reaches alpha=0.48 for overall quality, above the reported crowd/expert alpha=0.43, and the sensitivity analyses of fine-grained vs. single-score metrics are informative. If the retrieval-validation issues were addressed, the benchmark and judge prompts would be a useful step for automated evaluation of RAG argumentation. In its current form, however, the central quantitative claim about approximating true precision is not supported by the reported validation.","major_comments":[{"comment":"The reference 'True Precision' is constructed from qrels in which a document is relevant only if it appears in a ProCon topic's source list, and irrelevant documents are randomly sampled from other topics. This makes the validation dependent on ProCon authors' citation decisions and makes the injected negatives easy to reject by surface topic mismatch. The reported monotonic degradation and precision tracking therefore do not establish sensitivity to realistic retrieval errors, which are typically same-topic, off-stance, or partially relevant. Section 6.4's own concession that the judges cannot reliably distinguish finer levels of non-relevant evidence is consistent with this concern. I recommend adding a human relevance judgment subset and including same-topic misleading negatives.","section":"Section 4.3, Section 6.2, Table 7"},{"comment":"The 'close to True Precision' claim is not supported by the reported absolute errors of 0.043, 0.123, 0.106, 0.074, and 0.021 at 0%, 10%, 20%, 50%, and 70% irrelevant content. The relative errors at 10%, 20%, and 50% are approximately 36%, 43%, and 76%, respectively. The paper does not report confidence intervals or multiple runs, so the closeness claim is quantitatively unsubstantiated. Please report uncertainty and relative error, and either improve the estimate or narrow the claim.","section":"Section 6.2, Table 7"},{"comment":"The conversion from document-level LLM ratings to binary 'LLM Predicted Precision' is not described. No threshold, scale, or aggregation rule is given, and the Listwise+RAG Fine-Grained prompt in Figure 11 does not define a numeric relevance scale. Without this specification, the headline precision comparison cannot be reproduced or interpreted.","section":"Section 5.2.2, Table 7"},{"comment":"The paper's consistency analysis concludes that 'prompt-based LLM Judges are not able to reliably evaluate argument groundedness and overall quality' and that for all metrics the judges cannot distinguish finer levels of non-relevant evidence or hallucinations. This directly conflicts with the abstract's statement that the judges provide 'better and more interpretable assessments' than prior metrics and can 'enable rapid progress in computational argumentation.' The claims should be revised to match the scope of what is validated: coarse-grained context-relevance sensitivity and argument-quality agreement on ArgQuality, not reliable fine-grained groundedness or quality evaluation.","section":"Section 6.4, Abstract, Section 7"},{"comment":"The ConQRet argument-quality validation rests on 25 arguments, with no stated number of annotators and no inter-annotator agreement reported. In addition, the first 14 dimensions were annotated by humans on a binary scale while the LLM judges rate on a 0-5 scale, so the Krippendorff alpha values in Table 6 aggregate judgments made on different scales. The 'highest agreement' claim on ConQRet should be tempered or supported by reporting the annotation setup, reliability, and scale reconciliation.","section":"Appendix C, Table 6"}],"minor_comments":[{"comment":"There are missing spaces in phrases such as 'evaluation ofRAArg', 'Groundingof Arguments', and 'Stance ConditionedArguments'; please correct these typographical errors.","section":"Abstract, Table 1"},{"comment":"Appendix D.3 uses 'seperately' for 'separately', and Table 13's caption contains 'between between experts'; these should be fixed.","section":"Appendix D.3, Table 13"},{"comment":"The Limitations section lists only model choice and context-memory issues; it does not mention the citation-derived qrels limitation or the fine-grained resolution limitation acknowledged in Section 6.4. Please add these limitations explicitly.","section":"Section 8"},{"comment":"The paper states that code and documents 'will be released'; for a benchmark paper, please provide a stable version, license, and data documentation in the final version.","section":"Section 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is best suited to an NLP evaluation or benchmark venue. The ConQRet resource and the ArgQuality listwise judge results are of interest, and I do not see evidence of methodological dishonesty. My main concern for the editor is that the abstract overstates what the experiments show; if the authors revise the abstract and conclusions, add the missing binarization rule, and provide at least a small human relevance judgment sample or an external human-qrels validation, the paper could become acceptable. The small ConQRet human annotation sample in Appendix C (25 arguments, no stated IAA) is also a concern that should be addressed in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this is a solid, useful benchmark paper with an honest limitations section, but the abstract and Section 6.2 overclaim what the fine-grained judge actually delivers. The stress-test note is right: the \"close to true precision\" claim rests on qrels built from ProCon citation lists and cross-topic distractors, and the numbers are not that close (e.g., +0.123 at 10% noise, 36% relative error at 10%, 76% at 50%). Section 6.4 says plainly that the judges cannot reliably distinguish finer levels of non-relevant evidence. So the right claim is \"coarse-level noise detection,\" not fine-grained precision tracking.\n\nWhat is genuinely new: ConQRet is a real resource. 98 topics, ~6.5k web documents, long human expert arguments paired with the actual sources they cite. That fills a gap; prior argumentation datasets are mostly short, ungrounded, or not retrieval-conditioned. The comparison of listwise, self-consistent, and fine-grained LLM judge prompts is systematic, and the finding that document-level fine-grained scores degrade monotonically under injected noise while single-score direct metrics do not is practically useful.\n\nSoft spots, in order of importance: (1) Context-relevance validation uses automatically constructed qrels from ProCon source citations as ground truth, not independent human relevance judgments. That is a noisy proxy. (2) Injected irrelevant documents are sampled from other topics, which are easy to spot by surface relevance; realistic same-topic misleading documents are not tested. (3) The \"close\" claim in Table 7 does not survive the reported absolute and relative errors. (4) The paper never specifies how document scores are thresholded into binary precision, which hurts reproducibility. (5) The dataset and code are promised but not shipped with a commit hash. The 25-argument human annotation on ConQRet is fine for a sanity check, not for strong conclusions about the quality dimension.\n\nThe paper's own limitation section concedes that groundedness and overall-quality judgments are unreliable, which directly contradicts the abstract's broad claim that the judges provide \"better and more interpretable assessments\" across the board. That mismatch should be fixed before publication.\n\nWho it is for: people building RAG evaluation, LLM-as-a-judge pipelines, and computational argumentation systems. The benchmark and prompt ablation deserve a serious referee; I would send it out, but with the expectation of revision to align claims with evidence, tighten the qrels validation, and specify the thresholding. I would bring it to reading group and would cite the benchmark once it is actually released.","headline":"A useful new benchmark and a careful prompt comparison, but the precision-tracking claim is overstated and rests on citation-derived qrels and easy distractors.","tokens_in":25188,"tokens_out":2882,"would_cite":true,"duration_ms":30731,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that fine-grained LLM judges, validated on a new benchmark of expert-written arguments grounded in web sources, can evaluate retrieval-augmented argumentation as reliably as human annotators.","keywords":["Retrieval Augmented Argumentation","LLM-as-a-Judge","ConQRet benchmark","argument quality evaluation","context relevance","groundedness","fine-grained evaluation","controversial topics"],"falsifier":"Have human annotators judge relevance on a sample of ConQRet documents, including non-cited but topically relevant pages, and recompute the Listwise+RAG Fine-Grained precision errors from Table 7; if the errors grow beyond the reported 0.02 to 0.12 or monotonic decrease breaks, the claim that the judge tracks true relevance fails.","tokens_in":24088,"feed_emoji":"⚖️","tokens_out":8742,"duration_ms":78129,"temperature":0.7,"pith_summary":"The authors set out to show that automated, fine-grained evaluation can replace much of the human judging of retrieval-augmented argumentation (RAArg) on controversial topics. They introduce ConQRet, a benchmark of long human-written pro and con arguments paired with the real web sources they cite, and use it to test several LLM-based judges. Their central finding is that a listwise judge asked to score each retrieved document plus 15 argument-quality dimensions agrees with expert annotators at least as well as crowd workers do, and tracks true retrieval precision closely as irrelevant context is added. Single-score metrics, by contrast, fail to decrease reliably when irrelevant documents are mixed in. The authors also report that groundedness and overall-quality judgments are less reliable, especially at fine-grained levels of hallucinated content.","feed_headline":"LLM judge out-scores crowd workers on argument quality","feed_subtitle":"Document-level scores beat single-grade metrics and cut the cost of evaluating retrieval-augmented argumentation.","key_machinery":"The load-bearing piece is a benchmark and a prompt design. ConQRet is built from a popular debate portal's expert-written arguments and the 6,500 source documents cited on those pages, with relevance labels generated from the citations rather than from human annotation. The central evaluation device is the Listwise+RAG Fine-Grained judge: one prompt that asks the LLM to rate context relevance document by document, answer relevance, answer groundedness, and 15 argument-quality dimensions (local and global acceptability, relevance, sufficiency, cogency, clarity, reasonableness, and others) in a single inference call. Validation uses Krippendorff's alpha against expert and crowd ratings for quality, and precision against citation-derived qrels for relevance, with irrelevant documents injected at 0, 10, 20, 50, and 70 percent to test sensitivity.","core_discovery":"On the authors' terms, the discovery is that evaluation of retrieval-augmented argumentation can be decomposed into per-document context relevance, answer groundedness, and fine-grained argument quality, and that a single listwise LLM prompt can produce all of these in one call. Validated on the new ConQRet benchmark, the Listwise+RAG Fine-Grained judge gives precision estimates close to the true precision derived from qrels, with absolute errors around 0.02 to 0.12 across 0-70 percent irrelevant context, and it decreases more consistently than the Direct single-score metric as irrelevant documents are added. The same judge family, with self-consistency, reaches inter-annotator agreement with expert annotators on argument quality that is on par with or higher than crowd workers'. However, the paper's own stress tests show that groundedness and overall quality are not reliably distinguishable at fine granularities, and all judges struggle to separate fine levels of irrelevant evidence or hallucinations.","pith_inferences":["A likely consequence the authors leave implicit is that their judge design transfers to other long-document RAG settings, such as evidence-based fact-checking or policy briefs, wherever per-document relevance is the bottleneck.","Because the qrels come only from cited sources, ConQRet's relevance labels probably miss documents that are relevant but never cited; adding human relevance judgments on a subset would reveal how much this inflates the reported precision agreement.","A testable extension is to replace random cross-topic distractors with near-neighbour documents that are topically related but uncited; the judge's monotonicity under those harder negatives would show whether the benchmark's distractors are too easy."],"forward_implications":["A fine-grained judge can compute retrieval, groundedness, and argument-quality metrics in one inference call, lowering the cost of evaluating RAArg systems.","Document-level relevance scores give interpretable signals for fixing retrieval, instead of one opaque score for the whole context.","ConQRet's citation-derived qrels offer a path to building evaluation sets for long-document RAG without large-scale manual relevance annotation.","The reported insensitivity of all judges to fine-grained hallucination differences means groundedness evaluation is not yet safe to fully automate.","The same judge design should transfer to other retrieval-augmented generation tasks that need long, complex outputs."],"supporting_citations":[{"why":"Provides the 15 argument-quality dimensions and ArgQuality corpus used to validate the listwise judge against expert annotations.","marker":"Wachsmuth et al. (2017b)"},{"why":"Supplies the pointwise LLM annotator baseline that the listwise judge is compared against and improves on.","marker":"Mirzakhmedova et al. (2024)"},{"why":"Defines the RAG triad metrics and the Direct single-score prompt used as the main baseline for context relevance.","marker":"TruEra (2024)"},{"why":"Provides the G-Eval stepwise evaluation method adapted into one of the judge prompt variants.","marker":"Liu et al. (2023)"},{"why":"Provides the LLM-rubric method adapted into the Query-Rubric and RAG-Rubric judge variants.","marker":"Hashemi et al. (2024)"},{"why":"Specifies the listwise reranking paradigm used for the reference retriever in the RAArg pipeline.","marker":"Sun et al. (2023)"},{"why":"Sets out the retrieve-and-read architecture that the reference RAArg implementation follows.","marker":"Lewis et al. (2020)"}],"fun_headline_variants":["LLM judge matches crowd workers on argument quality","Fine-grained LLM judges outperform single-score metrics","One listwise LLM prompt grades relevance, groundedness, quality","New benchmark ConQRet tests LLM judges on argumentation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole validation rests on treating a ProCon author's cited sources as the complete set of relevant documents and random documents from other topics as irrelevant; if citations are incomplete or distractors are too easy, the judge's precision and monotonicity may not transfer to real retrieval.","fun_headline_variants_meta":{"raw":{"variants":["LLM judge matches crowd workers on argument quality","Fine-grained LLM judges outperform single-score metrics","One listwise LLM prompt grades relevance, groundedness, quality","New benchmark ConQRet tests LLM judges on argumentation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000745,"raw_usage":{"total_tokens":3345,"prompt_tokens":990,"completion_tokens":2355,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":2289}},"tokens_in":606,"tokens_out":2355,"duration_ms":16948,"temperature":1.0,"reasoning_tokens":2289,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:49:42.819714+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human annotators judge relevance on a sample of ConQRet documents, including non-cited but topically relevant pages, and recompute the Listwise+RAG Fine-Grained precision errors from Table 7; if the errors grow beyond the reported 0.02 to 0.12 or monotonic decrease breaks, the claim that the judge tracks true relevance fails.","supporting_citations":[],"review_version":1}