{"id":"23dcb848-c1c8-4e7f-851e-659aeacd3e78","arxiv_id":"2506.00694","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"An automated LLM-based evaluator finds that eight LLMs rarely hallucinate factors in legal argument generation but often omit relevant factors and usually fail to abstain when no common ground exists.","lead":"The paper builds an automated pipeline that uses GPT-4.1 to check whether LLM-generated legal arguments cite only the factors present in the input cases, measuring hallucination, factor completeness, and the ability to abstain when no argument is possible. Across eight LLMs, most models were faithful but incomplete, and nearly all failed to follow the instruction to abstain when cases shared no factors.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The factor-level metrics depend on unvalidated GPT-4.1 extraction, with no human gold standard or released artifacts, so the reported AccH/RecU rankings could be extraction artifacts rather than true behavioral differences.","rationale":"The reader's weakest assumption identifies exactly the load-bearing issue: the automated pipeline's validity is not established because the extractor is the only measurement instrument for the factor-level metrics. My reading confirms this is the most consequential soft spot. The paper is otherwise a well-scoped empirical study: the task definition is clear, the metrics are explicitly defined, deterministic settings are used, and the limitation section honestly flags the extractor dependency. However, the absence of extractor validation and of released artifacts means the quantitative conclusions, especially the model rankings in RecU and AccH, are not yet independently verifiable. The proposed test would settle the matter by providing an inter-annotator gold standard and a direct recomputation. Since this concern matches the reader's conditional verdict and does not, on its own, demonstrate the central claims are false, the appropriate disposition remains conditional pending validation.","tokens_in":15763,"tokens_out":5646,"duration_ms":57500,"concrete_test":"Take a stratified random sample of 50 generated arguments across all eight models and Tests 1-3, and have two independent legal annotators extract every factor asserted as present in each case, explicitly including paraphrases and correctly handling negated mentions. Compute GPT-4.1 extraction precision and recall against this human gold standard. Then recompute AccH and RecU using the human-corrected factor sets and compare model rankings with the paper's Tables 2 and 3. If the rank order and the >90% hallucination-accuracy claim survive, the extractor-bias concern is retired; if they shift, the paper must release corrected estimates and the raw annotation data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claims about hallucination avoidance and factor utilization are computed from factor sets extracted by GPT-4.1 (Section 3.4.1), yet this extraction step is validated only by informal 'spot checks' (Section 3.5) and the paper itself concedes that metric accuracy 'inherently depends on the performance of the external LLM used for factor extraction' (Section 8). The risk is concrete, not generic. The eight models under test produce very different output styles: smaller models may paraphrase factors ('the plaintiff had agreed not to disclose') instead of emitting factor codes ('F4 Agreed-not-to-disclose'). If GPT-4.1 extracts code-like mentions more reliably than paraphrases, RecU is systematically depressed for paraphrasing models, and the ranking plus the 'often fail to utilize factors' conclusion can be an artifact of extraction rather than of legal reasoning. A second specific failure mode is misattribution: when an argument says 'F5 is not present in the current case,' an extractor that records the mention against the current case would simultaneously inflate NH and depress NU, exactly the two quantities the headline metrics depend on. No precision/recall numbers against human annotation are reported, no error taxonomy for extraction is given, and neither the 90 triples, the generated outputs, nor the extraction outputs are released. This means the model rankings in Tables 2 and 3 cannot currently be independently checked. The abstention ratio, based on the required output phrase, is less exposed to extractor bias, but the paper's first two headline claims rest on an unmeasured instrument.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents an automated pipeline for evaluating LLM-generated 3-ply, factor-based legal arguments in the U.S. trade secret domain. A synthetically generated set of 90 case triples (30 per mode) is given to eight LLMs under three conditions: standard argument generation (Test 1), generation with swapped precedent roles (Test 2), and a non-arguable abstention condition (Test 3). An external LLM (GPT-4.1) extracts factor mentions attributed to each case from the generated arguments, and three metrics are computed: hallucination accuracy (AccH), factor utilization recall (RecU), and abstention ratio (RatioAbstain). The main findings are that most models exceed 90% AccH on Tests 1 and 2, RecU ranges from roughly 42% to 85%, and most models rarely abstain in Test 3 despite explicit instructions, with only GPT-4o abstaining in 86.67% of triples.","tokens_in":15996,"tokens_out":12532,"duration_ms":102952,"significance":"The paper's metrics are simple, transparent, and well matched to the factor-based CBR tradition: AccH, RecU, and RatioAbstain separately quantify three distinct failure modes, and the controlled synthetic dataset makes the ground truth unambiguous. The abstention finding is practically important: nearly all of the evaluated models, including reasonably capable ones, generate spurious arguments when explicitly instructed to stop, which bears directly on deployment risk in legal settings. The qualitative error analysis with concrete output excerpts independently corroborates the abstention failure. The paper ships no code, data, or human-validated extractor evaluation, however, so the quantitative rankings are currently the raw output of an unvalidated measurement instrument; the significance of the paper as a measurement methodology therefore rests on extractor validation that is currently missing.","major_comments":[{"comment":"The central methodological claim that the GPT-4.1-based extraction pipeline measures factor mentions reliably enough to rank the eight models is not supported by the evidence presented. Section 3.5 reports only informal 'spot checks' of extractor accuracy, Section 8 concedes that 'the accuracy of our automated metrics inherently depends on the performance of the external LLM used for factor extraction,' and no precision/recall figures against human annotation, no extraction error taxonomy, and no released outputs are provided. This is load-bearing because the headline results (AccH and RecU in Tables 2 and 3) are computed from F_Ext, the extraction of model assertions, and the eight models produce widely differing output styles: a model that paraphrases factors ('the plaintiff had agreed not to disclose') rather than emitting factor codes ('F4 Agreed-not-to-disclose') may have its mentions missed, which would systematically depress RecU; and a model that writes 'F5 is not present in the current case' may be recorded as mentioning F5 for the current case, which would simultaneously inflate NH and depress NU. Without a human-annotated validation sample stratified by model, the reported rankings cannot be distinguished from extraction artifacts. The authors should validate the extractor on a sample of outputs and report per-model extraction precision and recall, or release the data and outputs so that readers can do so.","section":"§3.4.1, §3.5, §8"},{"comment":"No uncertainty quantification is reported anywhere, and for the abstention test the sample is only 30 triples per model. RatioAbstain in Table 4 is a binomial proportion with n=30, for which the standard error reaches about 9 percentage points; accordingly Llama-3-70B (3.33%), DeepSeek (23.33%), and Qwen (56.67%) are not all pairwise distinguishable at conventional significance levels, and the statement that several models 'failed to abstain in any of the test instances' should be accompanied by a confidence interval. The same issue affects the factor-level metrics in Tables 2 and 3, since they are computed over roughly 36 ground-truth factor slots per triple and the paper gives no measure of dispersion across the 30 triples. I request confidence intervals or an exact test for the proportions, and per-triple distributions for AccH and RecU.","section":"§5.3, Table 4"},{"comment":"The abstention criterion appears to be exact-phrase matching. The prompt quoted in Appendix A instructs models to say 'No common factor between the input current case and the TSC1/TSC2,' whereas Section 4.2 states the phrase is e.g. 'Cannot generate argument due to lack of common factors' and Section 6 describes the required phrase as 'No common factor between the current case and the TSC1/TSC2'; these three formulations are inconsistent, and the exact-match requirement means that a model that recognizes the impossibility but phrases the abstention slightly differently is scored as a failure. This operationalization conflates recognition of task impossibility with verbatim instruction following, and Section 6's 'Incorrect Abstention Phrase' category shows that such near-misses occur. Please state precisely which phrase was required, and report the near-miss abstention rate separately from the exact-match RatioAbstain.","section":"§3.4.4, §6, Appendix A"},{"comment":"The claim in Section 5.3 that GPT-4o's 'spurious arguments were largely free of hallucinated factors' is not supported by the aggregate Test 3 AccH of 99.16%, because that figure includes the 86.67% of triples in which the model abstained; an abstention produces an empty extraction set and therefore trivially contributes NH=0 to the numerator. The faithfulness of the spurious arguments should be assessed by conditioning on the non-abstained outputs only, as the qualitative example in Box 3 does.","section":"§5.3, Table 2"}],"minor_comments":[{"comment":"The phrase 'of utomost importance' contains a typo; it should be 'of utmost importance.'","section":"§1 (Introduction)"},{"comment":"The Llama-4-Maverick row renders as '96.9698.1594.35' with no separators between the three test columns; this formatting error must be fixed.","section":"Table 2"},{"comment":"The word 'controlls' should be 'controls.'","section":"§4.1"},{"comment":"Test 2 is described in Section 4.3 as 'Also using the Arguable triples' with swapped order, while Section 4.1 says the separate 'Reordered' triples are used for Test 2; please clarify which dataset was actually used and whether the outcome labels were swapped.","section":"§4.1 vs §4.3"},{"comment":"The layout of Table 1 makes it difficult to determine which outcome label belongs to which case in each mode; please reformat so each case's outcome is unambiguous.","section":"Table 1"},{"comment":"Please state whether any generated outputs were truncated by the 500-token limit (or the 5,000-token limit for reasoning models), since truncation would mechanically depress RecU.","section":"§4.4"},{"comment":"RecU counts every correct per-case mention of a ground-truth factor as utilization regardless of argumentative role; since the three plies only require shared factors for analogy and differing factors for distinction, please justify why all ground-truth factors are the appropriate recall target.","section":"§3.4.3"},{"comment":"The claim that the pipeline enables 'reproducible' evaluation would be strengthened by releasing the 90 triples, the generated outputs, and the extraction outputs; please add a data and artifact availability statement.","section":"§3.5"}],"recommendation":"major_revision","confidential_remarks":"This workshop submission's contribution is a measurement methodology, and the fit with the workshop scope is good. The principal gap is the unvalidated GPT-4.1 extractor and the absence of released artifacts; this is fixable but central, so I would require a human-validated extraction study (or released outputs permitting independent validation) before acceptance rather than merely acknowledging the issue in the limitations. The self-citation to the pilot work [3] is appropriate. The qualitative error analysis partially mitigates the extractor concern for the abstention finding, but not for the RecU ranking, which is the most interesting quantitative claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a focused, well-written evaluation paper that introduces a cheap automated pipeline for measuring hallucination, factor utilization, and abstention in factor-based legal argument generation. The metrics are simple, and the empirical findings — high hallucination accuracy, mediocre factor recall, and very poor abstention in most models — are plausible and consistent with the broader LLM behavior literature. The paper is worth reading for anyone building legal-AI evaluation benchmarks.\n\nWhat's new: the pipeline itself, the three metrics (AccH, RecU, RatioAbstain), and the non-arguable abstention probe. The prior pilot work used human evaluation; here they replace it with GPT-4.1 as an extractor and validate with spot checks. The eight-model comparison is a new measurement, and the qualitative error analysis is genuinely useful — the examples of models reasoning aloud about the abstention rule and then violating it are striking.\n\nThe soft spots are real but proportionate. The entire measurement chain for AccH and RecU depends on GPT-4.1 extracting factor mentions accurately. The paper only cites informal spot checks, and Section 8 concedes the dependence. The stress-test worry is concrete: smaller models may paraphrase factors rather than emit codes, and GPT-4.1's extraction reliability could differ by output style, systematically depressing RecU for paraphrasing models. Misattribution of negated mentions is another plausible failure mode. No precision/recall numbers against human annotation, no error taxonomy, and no released outputs or code, so Tables 2 and 3 cannot be independently checked. That said, the abstention ratio is computed from the model's exact output phrase, so it is much less exposed to extractor bias. And the qualitative examples do support the broad story that models hallucinate rarely but omit factors and fail to abstain. So I would not call the findings fabricated — just not fully pinned down.\n\nThe citation pattern looks appropriate; self-citations to the human-evaluation pilot are relevant, not self-promotional. No circularity, no fitted parameters.\n\nBottom line: this is a solid workshop-to-conference-quality empirical contribution. It deserves a serious referee. The reviewer should ask for (a) a human-annotated validation set for the extractor, with precision/recall per model output style, (b) error bars or per-instance scores for the 30-triple cells, and (c) release of the generated outputs and extraction outputs. With those, the paper would be much stronger. I'd accept it conditionally; I'd also cite it if I were working in this area.","headline":"A useful, honest pipeline paper for legal-argument evaluation; the headline numbers rest on an unvalidated LLM extractor, so treat rankings as indicative until the extractor is measured.","tokens_in":16580,"tokens_out":1950,"would_cite":true,"duration_ms":18243,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that current LLMs are mostly faithful when generating factor-based legal arguments but often omit available factors and, when instructed to abstain because no shared factors exist, most generate spurious arguments anyway.","keywords":["LLM evaluation","legal argument generation","hallucination","factor-based reasoning","abstention","automated metrics","instruction following","case-based reasoning"],"falsifier":"Have legal experts manually annotate the factor mentions in the same 90 generated argument triples and compare their annotation to the GPT-4.1 extraction; if the human-extracted sets diverge systematically from the automatic sets on certain models or phrasings, the reported AccH, RecU, and model rankings would change, showing the scores are extraction artifacts rather than true behavior differences.","tokens_in":15522,"feed_emoji":"⚖️","tokens_out":2574,"duration_ms":28133,"temperature":0.7,"pith_summary":"This paper introduces an automated pipeline for evaluating how faithfully large language models generate structured 3-ply case-based legal arguments. It defines hallucination as citing a factor not present in the input case materials, factor utilization as how many of the available ground-truth factors the model mentions, and abstention as the model's willingness to stop when the cases share no common factors. Applying the pipeline to eight LLMs on three tests, the paper argues that models are largely faithful on viable argument-generation tasks but systematically incomplete, and that most models fail the abstention test despite explicit instructions to stop. The interest lies in showing that factual accuracy and instruction following can be measured separately and automatically, and that the second is the weaker behavior.","feed_headline":"Most LLMs argue on when told to abstain, study finds","feed_subtitle":"New automated metrics show models avoid hallucination but omit factors and rarely stop when they should.","key_machinery":"The pipeline is built on factor-encoded case representation, where each case is a fixed set of 26 trade-secret factors with plaintiff or defendant valence, taken from the HYPO/CATO tradition of case-based reasoning. An external evaluator LLM (GPT-4.1) extracts the factor sets the test model asserted for each case in the triple, and three metrics compare those sets to ground truth: Hallucination Accuracy (1 minus normalized hallucinated factors), Factor Utilization Recall (normalized correct factor mentions), and Abstention Ratio (successful abstentions over abstention-required triples). These metrics carry the argument because they turn qualitative fidelity judgments into reproducible scores that rank the eight models.","core_discovery":"The paper's central empirical claim is that current LLMs, when given factor-encoded case triples and asked to generate a plaintiff-defendant-rebuttal argument, achieve high Hallucination Accuracy (over 90% on Tests 1 and 2) while Factor Utilization Recall varies widely, from roughly 40% to 85%, with most models far below completeness. Its sharper claim concerns Test 3: when the current case shares no factors with either precedent, most models fail to follow the instruction to abstain, with several scoring 0% Abstention Ratio, and instead generate spurious arguments even though those arguments are often still built from factors genuinely present in the input. The paper presents this as evidence that reliable legal deployment is blocked less by hallucination than by incomplete reasoning and weak adherence to negative constraints.","pith_inferences":["The reported hallucination accuracy may be partly an artifact of the extractor: if the GPT-4.1-based evaluator misses factors the test model mentioned in prose without the factor-number label, the metric would overstate faithfulness, so a human-annotation comparison would likely shift the numbers.","The abstention failure is plausibly a general negative-instruction-following weakness rather than a legal-reasoning failure, which means Test 3 could be repurposed as a cheap probe for that behavior in any domain with structured inputs and a clear stop condition.","Because factor misattribution (citing a real factor in the wrong case) is counted as a hallucination by the current metric, a case-attribution-aware variant of AccH would separate invented factors from misplaced real ones, giving a finer diagnostic.","Extending the pipeline from synthetic factor-encoded triples to real legal documents would require a robust factor-extraction step from raw text; if that step becomes reliable, the same three metrics could evaluate arguments on authentic case law."],"forward_implications":["If the pipeline works as claimed, faithfulness, completeness, and abstention in factor-based legal argument generation can be measured at scale without manual annotation, enabling large model comparisons.","The observed gap between high hallucination accuracy and low factor recall implies that LLMs' legal arguments are often factually safe but shallow, missing supporting and distinguishing factors a competent advocate would use.","The abstention results imply that strong generative performance does not transfer to instruction following under negative constraints, so legal deployment needs explicit abstention training or evaluation rather than assuming it follows from general capability.","The three metrics give future work a concrete, automated target: improving factor utilization and abstention can be verified directly by these scores, not just by human reading."],"supporting_citations":[{"why":"The pilot human-evaluation study of LLM-generated 3-ply arguments that this paper builds upon and automates.","marker":"[3]"},{"why":"Establishes the factor-based model of legal argument and the 3-ply structure used in the task.","marker":"[13]"},{"why":"Provides the foundational factor vocabulary and legal analytics framework from which the 26-factor representation is derived.","marker":"[30]"},{"why":"Supplies the working definition of hallucination in natural language generation that the hallucination metric operationalizes.","marker":"[5]"},{"why":"Frames abstention as an instruction-following capability and motivates the abstention test design.","marker":"[28]"}],"fun_headline_variants":["LLMs stumble on knowing when to stay silent","Legal LLMs argue even when instructed to abstain","Factor recall lags as LLMs pass hallucination test","Automated pipeline shows LLMs rarely know when to stop"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The validity of every score depends on the external GPT-4.1 extractor reading each generated argument and producing the exact set of factor mentions, without silently missing factor references or inserting factors that were never actually named.","fun_headline_variants_meta":{"raw":{"variants":["LLMs stumble on knowing when to stay silent","Legal LLMs argue even when instructed to abstain","Factor recall lags as LLMs pass hallucination test","Automated pipeline shows LLMs rarely know when to stop"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000139,"raw_usage":{"total_tokens":1208,"prompt_tokens":1050,"completion_tokens":158,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":666,"completion_tokens_details":{"reasoning_tokens":94}},"tokens_in":666,"tokens_out":158,"duration_ms":2004,"temperature":1.0,"reasoning_tokens":94,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:59:37.105608+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have legal experts manually annotate the factor mentions in the same 90 generated argument triples and compare their annotation to the GPT-4.1 extraction; if the human-extracted sets diverge systematically from the automatic sets on certain models or phrasings, the reported AccH, RecU, and model rankings would change, showing the scores are extraction artifacts rather than true behavior differences.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The pilot human-evaluation study of LLM-generated 3-ply arguments that this paper builds upon and automates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the factor-based model of legal argument and the 3-ply structure used in the task."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the working definition of hallucination in natural language generation that the hallucination metric operationalizes."}],"review_version":1}