{"id":"ff5ed592-a1ef-4a3d-b507-7bbb5cfe47e4","arxiv_id":"2412.11567","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Concatenating the exact 'obligation' sentences that RePASs extracts yields a near-perfect score (0.947), exposing the metric's vulnerability; a verify-and-refine system with the same oracle scores a more credible 0.639.","lead":"This paper shows that a reference-free AI judge for regulatory answers, RePASs, can be fooled by simply repeating the sentences it is designed to reward, scoring above human experts. The authors then build a more honest system that picks and refines candidate answers and still beats the challenge baseline.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"VRR's score is produced by optimizing the same RePASs metric that evaluates it; the paper's own NOC attack shows RePASs is gameable, so the coherence/quality claim is unverified.","rationale":"The paper's central contribution is an honest demonstration that RePASs can be deceived by concatenating extracted obligations, and that claim is well supported by the NOC experiments. The VRR system, however, is presented as a more plausible and effective non-adversarial alternative, and its reported improvement is obtained by optimizing the same RePASs metric used for evaluation. Since the paper itself shows RePASs is easily gamed, the validity of RePASs as a verifier inside VRR is a load-bearing assumption that is not tested. The reader's weakest assumption about public-test hyperparameter tuning is a legitimate generalization concern, but it is secondary: even on the reported test set, the VRR result does not establish genuine answer quality without an independent check. The paper's Limitations section acknowledges the verifier dependence, which strengthens the case that this is the central soft spot. My concern does not overturn the reader's conditional verdict; it clarifies that the condition should include independent validation of VRR's outputs, not merely confirmation that the tuned hyperparameters transfer to the hidden test set.","tokens_in":7743,"tokens_out":4303,"duration_ms":41488,"concrete_test":"Run a blind human evaluation on a random sample of 50 hidden-test questions, comparing VRR, baseline, LOC, and NOC outputs. Have at least three annotators rate each answer for correctness, coherence, and faithfulness to the retrieved passages. If VRR is not rated significantly better than the baseline on these dimensions, its RePASs improvement is largely an artifact of optimizing the evaluation metric. As a complementary check, rerun VRR using an independent NLI model (not the one in RePASs) as the verifier and see whether the RePASs gain and human preference persist.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing insecure condition is the validity of RePASs as a proxy for answer quality inside VRR. VRR's verification step selects the candidate answer with the highest RePASs score, and its refinement step removes sentences whose contradiction score exceeds a dataset-wide average and inserts obligations that raise obligation coverage (Sections 4.5.1-4.5.2). Thus the reported 0.639 (Table 5) and the +0.056 over baseline are measured with the same metric being optimized. The paper's NOC system (Section 4.3) shows RePASs can be trivially gamed by concatenating extracted obligations, so optimizing RePASs does not guarantee semantically good answers. The paper asserts VRR produces 'readable, coherent answers' (Abstract, Section 5) but provides no human evaluation or independent metric. The Limitations section concedes VRR requires an accurate verifier 'such as RePASs, which is not always available,' but does not test whether RePASs is accurate enough on this task. If RePASs is not a valid verifier, the VRR result is another form of metric gaming and the coherence claim is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes three answer-generation systems submitted to the RIRAG-2025 shared task: NOC, which concatenates obligation sentences extracted by the RePASs metric's own LegalBERT component and achieves a RePASs score of 0.947, surpassing human experts; LOC, which rewrites obligations with an LLM and performs below baseline; and VRR, which selects among LLM-generated candidates using RePASs and then iteratively removes contradictions and inserts missing obligations, achieving 0.639 on the hidden test set. The retrieval backbone combines rank fusion of BM25, voyage-law-2, and voyage-finance-2 with a reranker. The paper's stated goal is to show that RePASs can be adversarially gamed and to propose a more plausible non-adversarial system.","tokens_in":8037,"tokens_out":3827,"duration_ms":30645,"significance":"The NOC result is a valuable, reproducible demonstration that the RePASs metric is gameable by a trivial concatenation of extracted obligations; the released code strengthens this contribution. However, the central positive claim—that VRR produces readable, coherent answers that score well—is not independently substantiated: VRR optimizes the same RePASs metric that is used for evaluation, and the paper's own NOC attack shows that RePASs does not track answer quality. The paper is transparent about this in its Limitations section but does not provide the human evaluation or independent metric needed to support the coherence claim.","major_comments":[{"comment":"VRR's reported RePASs score (0.639 in Table 5) is a fitted value rather than an independent assessment of answer quality. The verification step (Section 4.5.1) selects the candidate answer with the highest RePASs, and the refinement step (Section 4.5.2) removes sentences with above-average contradiction scores and inserts obligations to increase obligation coverage, both measured using the same NLI and obligation-extraction components that constitute RePASs. The paper then reports RePASs as the evaluation metric. The Limitations section correctly notes that VRR requires an accurate verifier 'such as RePASs,' but it never validates whether RePASs is accurate enough on this task. Given the NOC attack in Section 4.3 shows that RePASs can be trivially deceived by verbatim obligation concatenation, the claim that VRR generates 'readable, coherent answers' (Abstract, Section 5) is unsupported without a human evaluation or an independent, non-gameable metric.","section":"Sections 4.5.1, 4.5.2, Table 5; Limitations"},{"comment":"Several hyperparameters are selected by trying a few combinations on the public test set: the rank-fusion weights a=0.25 and b=0.2 (Table 2), the filtering threshold 0.90 and max drop 0.10 (Table 3), and the reranking top-N=50 (Figure 1). The hidden test set results in Table 5 are the only reported evaluation, and they assume these public-test choices generalize. The paper provides no cross-validation on the development set or confidence intervals to assess the risk of overfitting to the public test set. If the public test set is not representative of the hidden test set, the relative gains of the proposed systems may not transfer.","section":"Sections 3.3 and 4.2"},{"comment":"The reported VRR RePASs values are inconsistent across tables. Table 5 lists the leaderboard score as 0.639 and the authors' own recomputation as 0.646 (in brackets), Table 6 reports a final VRR RePASs of 0.646, and Table 7 reports 0.647. The paper attributes differences to 'using different GPUs,' but it is unclear whether Tables 6 and 7 are computed on the public test set or the hidden test set. Please clarify the evaluation split for each table and reconcile the final VRR score.","section":"Tables 5, 6, and 7"}],"minor_comments":[{"comment":"The retry limit K=3 for LOC is introduced without a sensitivity analysis or justification; a brief ablation or rationale would help the reader understand its impact.","section":"Section 4.4"},{"comment":"The phrase 'three of the best 3 text embedding models' is grammatically awkward; consider rewriting as 'three of the best text embedding models.'","section":"Section 3.1"},{"comment":"The abstract describes the NOC score as 'dubiously high' while Section 1 calls it a 'perfect score'; these characterizations are consistent in spirit but could be unified to avoid confusion.","section":"Abstract and Section 1"},{"comment":"The prompts are said to have been improved using GPT-4o and selected according to the authors' opinion on 2-3 sample questions; this selection procedure is informal and should be mentioned in the main text or accompanied by a note on its potential bias.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The paper is a shared-task system description with a useful adversarial finding, but its main positive contribution (VRR) rests on a metric-optimization loop that is not independently validated. The requested changes are feasible within the paper's scope: add a human evaluation or clearly frame VRR as a RePASs optimizer rather than a quality improvement, and address the public-test hyperparameter selection issue. The paper fits the journal's scope if reframed as a critical analysis of reference-free evaluation metrics rather than as a purely positive system description."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing worth knowing about this paper is that it gives a working, reproducible demonstration that RePASs—a reference-free RAG metric—can be trivially gamed. The NOC system concatenates obligations extracted by RePASs's own LegalBERT component, scores 0.947, above the human experts' 0.859, and the answers are verbatim excerpts from retrieved passages. That is a real result, and it lands. The paper is honest about the exploit, and the evidence is direct. Code is public.\n\nVRR is a reasonable engineering contribution: generate several candidate answers, pick the one with the best RePASs, then iteratively remove contradictions and insert missing obligations. The ablations show each stage helps. For a shared task system description, the analysis is above average.\n\nThe main soft spot is circularity. VRR uses RePASs as the selection signal, as the refinement reward, and then as the evaluation metric. The reported 0.639 is therefore to some degree a fitted value, not an independent assessment. The paper acknowledges this in the Limitations section, noting that VRR needs an accurate verifier, but it never tests whether RePASs is accurate enough on this task. The NOC result cuts both ways: if RePASs can be gamed, then optimizing it does not guarantee semantic quality. The claim that VRR produces 'readable, coherent answers' has no human evaluation behind it. Also, hyperparameters (fusion weights, thresholds, rerank N) were chosen on the public test set without cross-validation; the hidden test results assume those choices generalize.\n\nNone of this kills the paper. The NOC demonstration is the payload, and it holds. The VRR part is a system description, not a new evaluation methodology, and its limitations are stated. A reader who cares about metric robustness in RAG will get real value here, especially the clear contrast between the gamed near-perfect score and the 'plausible but still optimized' score of VRR.\n\nI would send this to peer review. A serious referee should ask for an independent check of VRR's coherence claim and for a robustness analysis around the public-test tuning, but the central cautionary result deserves to be in the literature. The paper is a good citizen: it shows a flaw in a metric that shared task organizers and metric designers should take seriously.","headline":"This shared task paper earns its place with a simple, reproducible demonstration that RePASs can be gamed by concatenating its own extracted obligations; the VRR system is a solid engineering contribution but its score is partly fitted to the metric it evaluates with.","tokens_in":8507,"tokens_out":1995,"would_cite":true,"duration_ms":18684,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By feeding RePASs its own extracted obligation sentences back as answers, the NOC system scores 0.947, above human experts, despite copying retrieved text.","keywords":["RePASs","retrieval-augmented generation","answer generation","regulatory NLP","obligation extraction","adversarial evaluation","rank fusion","reference-free evaluation"],"falsifier":"Run a human usefulness study where annotators compare NOC's verbatim answers with human-written answers on the same regulatory questions; if NOC answers are judged as useful as human answers, the claim that RePASs is deceived by verbatim copying would collapse. Alternatively, modify RePASs to remove or ignore its obligation-extraction component and re-run NOC: a score close to baseline would confirm the metric, not the answer content, was the source of the 0.947.","tokens_in":7592,"feed_emoji":"⚠️","tokens_out":4508,"duration_ms":39434,"temperature":0.7,"pith_summary":"This paper tries to establish two things about RIRAG-2025, a shared task on answering regulatory questions from retrieved passages. First, a system called NOC can achieve a near-perfect RePASs score (0.947), higher than human experts, simply by concatenating obligation sentences extracted from the retrieved passages and outputting them verbatim as the answer. Second, a more principled system called VRR, which generates multiple candidate answers, selects the best one with RePASs, and then iteratively removes contradictions and inserts missing obligations, produces coherent answers with a more plausible score (0.639). The authors use these results to argue that reference-free model-based metrics like RePASs are vulnerable to adversarial exploitation and that super-human scores on such metrics do not necessarily indicate useful answers.","feed_headline":"Concatenated obligations beat human experts on RAG metric","feed_subtitle":"Copying the metric's own extracted sentences scores 0.947, warning that reference-free RAG evaluation can be gamed.","key_machinery":"The central mechanism is the obligation-extraction and entailment pipeline inside RePASs: a fine-tuned LegalBERT model identifies important 'obligation' sentences in the retrieved passages, and an NLI model measures whether each answer sentence entails or contradicts those obligation sentences. NOC exploits this by using the exact same extracted obligations as the answer, so every answer sentence trivially entails the obligations it was extracted from, yielding near-perfect scores. VRR uses the same RePASs pipeline as a verifier: it selects the best among several candidate answers and then iteratively refines the chosen answer by adding missing obligations and deleting sentences that cause contradictions.","core_discovery":"The central discovery is that RePASs, the task's reference-free evaluation metric, can be tricked by exploiting its own obligation-extraction component. NOC takes the obligation sentences that RePASs extracts from the retrieved passages and returns them verbatim as the generated answer, achieving a RePASs score of 0.947, above the reported human expert score of 0.859. The paper also shows that when answers are rewritten to be readable and directly responsive, as in the LOC system, the score drops below the baseline, demonstrating that RePASs is sensitive to answer style and rewards verbatim copying. The VRR system instead uses RePASs as a verifier: it generates several LLM answers, picks the one with the highest RePASs score, then alternates between removing contradictory sentences and adding uncovered obligations, reaching 0.639 while keeping answers coherent.","pith_inferences":["A similar concatenation attack may transfer to other reference-free RAG metrics that reward entailment, lexical overlap, or extracted key sentences; a natural test is to apply NOC-style answers to such metrics and see whether scores inflate.","The hidden-test-set hyperparameters were chosen by trying a few combinations on the public test set, so the relative gains of VRR and the retrieval pipeline may not generalize if the two test sets differ in question or passage distribution.","The authors' suspicion that other super-human systems also trick RePASs could be tested directly by checking whether their top-scoring answers are largely verbatim excerpts from the retrieved passages.","If future RAG evaluations pair reference-free metrics with a readability or directness check, the incentive to use verbatim-extraction exploits like NOC would be reduced."],"forward_implications":["Reference-free model-based metrics like RePASs can be gamed by returning extracted sentences verbatim, so leaderboard scores based on such metrics should be treated with caution unless answers are checked for readability and directness.","Super-human scores in RAG evaluation may indicate metric exploitation rather than genuinely better answers, a concern the paper explicitly raises about the top-scoring systems.","Generating readable answers while maximizing a model-based metric is hard: rewriting obligations (LOC) hurts the score, while iterative verification and refinement (VRR) improves it substantially over the baseline.","Combining BM25 with two domain-specific neural retrievers and a reranker improves passage retrieval over the single models and the baseline fusion approach."],"supporting_citations":[{"why":"Defines RePASs, provides the baseline and human-expert scores, and describes the synthetic dataset used to fine-tune the obligation-extraction model.","marker":"Gokhan et al., 2024"},{"why":"Supplies the LegalBERT model reused as the obligation-extraction component that NOC exploits.","marker":"Chalkidis et al., 2020"},{"why":"Provides the rank-fusion method that the paper extends from two retrievers to three, motivating the retrieval backbone.","marker":"Wang et al., 2021"},{"why":"Defines BM25, one of the three fused retrieval models used in all systems.","marker":"Robertson et al., 1994"},{"why":"Inspires the self-consistency-style verification step in VRR that generates multiple answers and selects the best one.","marker":"Wang et al., 2023"},{"why":"Motivates the iterative refinement loop in VRR, where the selected answer is repeatedly improved with self-feedback.","marker":"Madaan et al., 2024"},{"why":"Frames the retrieval-augmented generation task that RIRAG-2025 instantiates.","marker":"Lewis et al., 2020"}],"fun_headline_variants":["Concatenated obligations beat human experts on RAG metric","RePASs metric tricked by copying extracted obligations","RAG eval loophole: verbatim copying scores 0.947","Honest RAG answers score lower than metric gaming","Copying metric's obligations beats human expert on RePASs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains assume the public test set used to tune retrieval and preprocessing hyperparameters is representative of the hidden test set, since the choices were made by trying a few combinations without cross-validation or confidence intervals.","fun_headline_variants_meta":{"raw":{"variants":["Concatenated obligations beat human experts on RAG metric","RePASs metric tricked by copying extracted obligations","RAG eval loophole: verbatim copying scores 0.947","Honest RAG answers score lower than metric gaming","Copying metric's obligations beats human expert on RePASs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000275,"raw_usage":{"total_tokens":1611,"prompt_tokens":882,"completion_tokens":729,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":644}},"tokens_in":498,"tokens_out":729,"duration_ms":6954,"temperature":1.0,"reasoning_tokens":644,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:47:23.722420+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a human usefulness study where annotators compare NOC's verbatim answers with human-written answers on the same regulatory questions; if NOC answers are judged as useful as human answers, the claim that RePASs is deceived by verbatim copying would collapse. Alternatively, modify RePASs to remove or ignore its obligation-extraction component and re-run NOC: a score close to baseline would confirm the metric, not the answer content, was the source of the 0.947.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the iterative refinement loop in VRR, where the selected answer is repeatedly improved with self-feedback."},{"cited_title":"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\\","cited_arxiv_id":null,"evidence_quote":"Frames the retrieval-augmented generation task that RIRAG-2025 instantiates."}],"review_version":1}