{"id":"fc471ce2-d1e2-4165-b1ee-89900cb0bf53","arxiv_id":"2508.19365","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"State-of-the-art LLMs with retrieval answer simplified boolean questions about state unemployment insurance law with at best 0.69 F1, well short of reliable end-to-end code simplification.","lead":"This paper introduces LaborBench, a 3,700-question benchmark built from U.S. Department of Labor tables of state unemployment insurance laws, and StateCodes, an 8.7 GB corpus of U.S. state statutes and regulations. It evaluates five LLMs with retrieval and finds the best F1 is 0.69, far below the accuracy needed for end-to-end statutory simplification.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Temporal/completeness alignment of StateCodes with CSUIL 2023 is the load-bearing risk: missing or stale state sections would make RAG failures corpus artifacts rather than statutory misinterpretation.","rationale":"The paper's central empirical contribution is a negative result: current RAG systems are far from reliable for statutory understanding. That result is supported by a large evaluation matrix across retrievers and generators, and the resource contribution is substantial. The reader's conditional verdict identifies the same weakest point: the gold labels from DOL CSUIL 2023 are credible, but the StateCodes corpus must be complete and temporally aligned for the RAG failure rate to be attributable to model error. I found no internal inconsistency in the evaluation pipeline, but the unverified corpus alignment is genuinely load-bearing because the headline claim is about misinterpretation, not about a resource being imperfect. The proposed check would settle whether the low F1 is a model-level limitation or partly a data-alignment artifact. Since this is exactly the condition the reader already imposed, the verdict remains conditional without change.","tokens_in":17518,"tokens_out":5826,"duration_ms":70299,"concrete_test":"Stratify-sample 100 LaborBench boolean questions across the jurisdictions shown and omitted in Figure 3. For each, locate the controlling statute/regulation from official state sources as of CSUIL 2023's effective date, and check (a) whether the identical section exists in StateCodes, (b) whether the text is substantively identical, and (c) whether the DOL gold answer is derivable from the official text. Then recompute the overall RAG F1 on the subset with confirmed corpus alignment, and separately on the subset excluding any jurisdictions absent from StateCodes. If F1 shifts materially (e.g., by more than 0.05) or the fraction of missing/stale sections exceeds about 5%, the headline error-rate claim is confounded; if F1 is stable, the concern is resolved. Also report the number of boolean questions used and per-jurisdiction coverage counts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline error-rate claim (best F1=0.691; models 'misinterpret statutory and regulatory requirements over one third of the time') depends on the RAG pipeline being evaluated against the same legal text the gold labels describe. Section 3.3 asserts that StateCodes 'dates to 2023, corresponding to LaborBench's source data (CSUIL 2023)', but the corpus is scraped from Justia and the paper provides no per-state snapshot verification. Footnote 17 concedes that 'a small number of states could not be structured,' and Figure 3 plots only 37 jurisdictions with no explanation of which are excluded or how many boolean questions the experiments actually cover. If a state's relevant UI section is missing, mis-scraped, or from a different year, the retriever cannot surface the correct law, so a model failure is a corpus artifact, not evidence of misinterpretation. The same gap undermines the retrieval-side numbers: recall@5=0.93 is measured against a corpus whose completeness and temporal alignment are unverified. Since the central claim is a negative result about model comprehension, the corpus side must be independently validated before the headline error rate can be interpreted as a statement about model reasoning.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LaborBench, a QA benchmark derived from the U.S. Department of Labor's 2023 Comparison of State Unemployment Insurance Laws (CSUIL), and StateCodes, a large corpus of U.S. state statutes and regulations scraped from Justia. The authors construct boolean QA tasks from CSUIL tables, augment them with retrieval from a UI-filtered subset of StateCodes, and evaluate five LLMs with several dense and sparse retrievers. The central finding is that even with RAG, the best configuration reaches F1=0.691, leading the authors to conclude that current systems are far from reliable end-to-end statutory simplification tools. The paper also reports that RAG substantially improves over no-retrieval baselines, while chain-of-thought citation elicitation provides only marginal gains, and it analyzes state-level variation in performance.","tokens_in":17830,"tokens_out":4791,"duration_ms":55301,"significance":"If the benchmark construction and corpus are sound, this is a valuable contribution: LaborBench operationalizes a unique expert-generated legal comparison task, and StateCodes is substantially larger and more structured than prior state-law corpora. The paper also ships the dataset and scraper, reports bootstrap standard errors, applies multiple-testing corrections, and includes a concrete error example. The headline negative result about LLM reliability is policy-relevant. However, the interpretation of the benchmark results depends critically on the temporal and jurisdictional completeness of StateCodes relative to CSUIL 2023; the current manuscript does not yet establish that alignment, and some reporting gaps in the state-level analyses need to be closed before the central claim is fully supported.","major_comments":[{"comment":"The corpus-side alignment is load-bearing for the paper's central claim. Section 3.3 asserts that StateCodes 'dates to 2023, corresponding to LaborBench's source data (CSUIL 2023),' but there is no per-state snapshot verification, and footnote 17 concedes that 'a small number of states' could not be structured. Figure 3 plots only 37 jurisdictions without stating which are excluded or how many boolean questions the experiments cover. If a relevant UI section is missing, stale, or mis-scraped, a RAG failure is a corpus artifact rather than evidence of statutory misinterpretation. Please provide a per-state coverage audit, list excluded jurisdictions, report the number of questions per state, and re-run or qualify the headline results for states with verified coverage.","section":"§3.3, footnote 17, Figure 3"},{"comment":"The construction of the boolean supplement is not fully specified. The text says 'For simplicity, we use one-hot encodings for categorical answers,' but Table 1 shows entries like 'Any time', '1 employee...', and 'Over $100...' being converted to a single boolean 'differs from FUTA' column. What is the exact rule that maps these heterogeneous string entries to True/False? If a table entry is blank, is it coded as False or missing? The distinction matters because the boolean experiments are the sole basis for the accuracy/F1 results. Please state the conversion rule explicitly and provide a validation check against a sample of the original CSUIL entries.","section":"§3.2, Table 1"},{"comment":"The state-by-state analysis is not fully auditable. Figure 3 plots 37 jurisdictions even though CSUIL tables typically contain 53 rows (50 states plus D.C., Puerto Rico, and the Virgin Islands). No explanation is given for the missing jurisdictions, and the correlation with population (r=-0.40, p=0.01) does not report the number of observations or whether territories are included. Since the 'knowledge gain from RAG' sorting and the population correlation depend on which states are retained, please report the exact N, the exclusions, and a robustness check using only the jurisdictions with confirmed StateCodes coverage.","section":"§5.3, Figure 3, §5.3.1"}],"minor_comments":[{"comment":"The abstract reports 'F1 of 0.67' for the best system, while Table 4 shows the best configuration at 0.691 (Gemini retriever + Deepseek). Please reconcile these numbers and use consistent rounding.","section":"Abstract and Section 1"},{"comment":"The phrase 'models misinterpret statutory and regulatory requirements over one third of the time' is based on F1, not accuracy or error rate. F1 is a harmonic mean of precision and recall, not a misclassification rate. Please rephrase to avoid conflating F1 with an error rate.","section":"Section 5.1"},{"comment":"The manual citation audit uses only 30 outputs. The reported recall@5 of 0.93 and citation accuracy of 0.73 would benefit from confidence intervals or at least an explicit statement that the sample is small. This is secondary to the main result, but the paper draws a positive conclusion about retriever usefulness from it.","section":"Section 5.2.3"},{"comment":"The chunk size (1000 tokens), overlap (200 tokens), and top-k (k=5) are fixed without sensitivity analysis. Since these are free parameters, a sentence acknowledging this limitation would be helpful.","section":"Section 4.2"},{"comment":"The statement that 'no covariates (of 2,970) have statistically significant predictive power in bivariate tests' is followed by 'similar results in regression analyses' without reporting the regression specification. Please include details or remove the claim.","section":"Section 5.3.1"},{"comment":"Minor wording issues: §3.1.1 contains 'justf over 2 columns' (likely 'just over'), and Figure 1 shows a truncated URL 'https://law.just...'. Please copyedit.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"This is a promising dataset/benchmark paper with a clear negative result about LLM reliability. The main risk is not circularity—the gold labels come from an external expert source and the models are fixed—but rather corpus verification. I would ask the authors to add a per-state coverage and timestamp audit, clarify the boolean conversion rule, and report the exact jurisdiction sample in the state-level analyses. These are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me give you the short version. This paper builds two genuinely useful resources—StateCodes, the first large structured corpus of US state statutes and regulations, and LaborBench, a 3,700-question QA benchmark derived from the Department of Labor's annual CSUIL state unemployment insurance tables—and then shows that current RAG systems top out around 0.69 F1 on a simplified boolean version of the task. That negative result is credible and worth knowing.\n\nThe evaluation is honest: bootstrap standard errors, FDR correction, concrete error examples, and a transparent two-step JSON extraction pipeline. The central claim, that LLM-based RAG is far from reliable for end-to-end statutory simplification, holds up as a direction.\n\nThe soft spot is the corpus. StateCodes is scraped from Justia, and the paper asserts it dates to 2023 to match CSUIL 2023, but I don't see per-state verification of completeness or snapshot date. Footnote 17 admits a few states could not be structured, and Figure 3 only plots 37 jurisdictions without explaining the missing ones. So some RAG failures could be corpus artifacts—missing or stale sections—rather than model misinterpretation. That is a real validity threat to the headline number, though probably not fatal: even with a perfect corpus, 0.69 F1 is still far from the touted end-to-end claims. But the authors should report per-state coverage and temporal alignment before anyone uses the error rate for policy conclusions.\n\nThe manual citation audit is based on only 30 outputs and the benchmark lacks citation gold labels; the authors note DOL may share them prospectively. That is a minor-to-moderate weakness, not a deal-breaker.\n\nMy read: this is a solid resources paper with a believable baseline result. It deserves a serious referee. I'd want the revision to add per-state statistics and a sensitivity analysis on the missing states before treating the exact F1 as settled.","headline":"A solid resources-and-baseline paper with a credible negative result, but the corpus alignment is a real soft spot that should be fixed before the headline F1 is read as pure model reasoning.","tokens_in":18244,"tokens_out":3661,"would_cite":true,"duration_ms":35717,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Even with retrieval, AI errs on a third of state-law questions","keywords":["statutory simplification","legal benchmark","retrieval-augmented generation","state statutes","state regulations","unemployment insurance law","boolean question answering","large language models"],"falsifier":"Take a random sample of boolean questions from the benchmark, have an expert locate the controlling state statute and the comparable federal model statute, and adjudicate the official table's answer against the statutory text; if the tables disagree with the statutes on more than a small fraction of the sample, the reported accuracy numbers conflate label error with model error.","tokens_in":17461,"feed_emoji":"⚖️","tokens_out":8362,"duration_ms":85583,"temperature":0.7,"pith_summary":"The paper sets out to test whether large language models can do what advocates of AI-driven regulatory reform claim: read, simplify, and compare state statutes and regulations end to end. To make that test possible, it builds LaborBench, a 3,700-question benchmark derived from a federal agency's annually updated, lawyer-compiled comparison of state unemployment-insurance laws, and StateCodes, a structured corpus covering most U.S. state statutes and regulations. Benchmarking five LLMs and four retrievers on the boolean subset, it finds that the best configuration reaches F1 0.691: even with retrieval and yes/no answers, models get statutory requirements wrong more than a third of the time. Retrieval raises performance substantially, while chain-of-thought prompting and citation elicitation add little. The paper's conclusion is that these systems are useful as research aids for simplification, not as end-to-end replacement pipelines.","feed_headline":"Even with retrieval, AI errs on a third of state-law questions","feed_subtitle":"A 3,700-question benchmark of unemployment-insurance law shows retrieval helps, but not enough.","key_machinery":"LaborBench is the measurement device: roughly 3,700 question-answer items plus 1,272 supplemental boolean questions created by turning each numeric, categorical, or boolean cell of the federal agency's 86 state-comparison tables into a question asked of every jurisdiction. StateCodes is the testbed: a 2023 structured corpus of 1.8 million statute sections (3.0 billion characters) from 50 states and 2.2 million regulation sections (1.8 billion characters) from 45 states, chunked inside section boundaries and indexed. The evaluation pipeline is a RAG setup: sparse or dense retrieval selects the top five sections per query, and a two-step prompt first asks the model to reason, then forces a sch","core_discovery":"The central claim is that current retrieval-augmented language models are not reliable enough for end-to-end statutory simplification, even under the most forgiving conditions the authors could construct. Each question is boolean, the corpus is filtered to the relevant unemployment-insurance provisions, and the top five statute sections are handed to the model as context; nonetheless the best retriever-generator pair across all configurations scores F1 0.691, and more than a third of statutory or regulatory requirements are misinterpreted. The paper also shows where the failures come from: retrieving relevant sections moves F1 from 0.481 to 0.665, but precision improves far less than recall,","pith_inferences":["The paper evaluates only a UI-filtered subset of the corpus; applying the same systems to a full, unfiltered state code should be expected to produce lower accuracy, since retrieval has more distractors.","A natural extension the paper leaves implicit: require the model to quote the controlling statutory sentence verbatim and then check that quote against the retrieved section; this would directly test the citation-accuracy bottleneck it identifies.","Because the gold labels come from a single 2023 snapshot, the benchmark could be replayed against earlier and later annual tables to track whether model progress outpaces statutory change.","The negative population correlation is intriguing, but with no robust institutional covariates and multiple-testing adjustments, it should not be read as a causal claim about smaller states."],"forward_implications":["Governments and vendors advertising AI code simplification should be held to a benchmark like this; on the easiest subset, the best system still misses more than a third of statutory distinctions.","Improving retrieval is the clearest lever: adding RAG raises F1 by roughly 0.18, while chain-of-thought and citation prompts add only a few points.","Per-state scores vary widely, from F1 0.52 to 0.81, so any simplification program should expect uneven gains across jurisdictions and should audit the weakest ones.","A retriever that finds the right statute 93 percent of the time can still be paired with a generator that cites correctly only 73 percent of the time; 'verifiable' legal-research outputs need independent citation checking.","The StateCodes corpus gives researchers a common substrate for 50-state retrieval and reasoning tasks well beyond unemployment insurance."],"supporting_citations":[{"why":"Provides the lawyer-compiled annual state unemployment-insurance comparison tables that serve as the benchmark's gold answers and question templates.","marker":"[40]"},{"why":"Defines retrieval-augmented generation, the method whose end-to-end performance the paper tests.","marker":"[23]"},{"why":"Supplies the dense passage retrieval technique used by the embedding-based retrievers in the experiments.","marker":"[20]"},{"why":"A prior single-jurisdiction statute retrieval dataset that the paper compares against as an existing benchmark it extends.","marker":"[42]"},{"why":"Prior state-code corpus that lacks structured sections and regulations, motivating the construction of StateCodes.","marker":"[15]"},{"why":"Prior legal RAG benchmark restricted to private-law documents, against which StateCodes' statutory and regulatory scope is contrasted.","marker":"[36]"},{"why":"Provides the state-policy covariates used in the exploratory analysis of why state-level RAG gains differ.","marker":"[12]"}],"fun_headline_variants":["AI legal simplification: 34% of rules still misread","Retrieval boosts AI law answers, but 1 in 3 still wrong","Best RAG pipeline on state law scores F1 0.691","Even with top statutes, AI botches a third of legal points","State-law AI check: retrieval helps, but not enough"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The evaluation assumes that the federal agency's 2023 comparison tables are the correct answer key and that the scraped 2023 state codes contain every controlling provision for every state; if a state's statute is missing, stale, or mis-extracted, measured errors may reflect data defects rather than model reasoning.","fun_headline_variants_meta":{"raw":{"variants":["AI legal simplification: 34% of rules still misread","Retrieval boosts AI law answers, but 1 in 3 still wrong","Best RAG pipeline on state law scores F1 0.691","Even with top statutes, AI botches a third of legal points","State-law AI check: retrieval helps, but not enough"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000297,"raw_usage":{"total_tokens":1588,"prompt_tokens":803,"completion_tokens":785,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":693}},"tokens_in":547,"tokens_out":785,"duration_ms":8770,"temperature":1.0,"reasoning_tokens":693,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:46:46.534809+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of boolean questions from the benchmark, have an expert locate the controlling state statute and the comparable federal model statute, and adjudicate the official table's answer against the statutory text; if the tables disagree with the statutes on more than a small fraction of the sample, the reported accuracy numbers conflate label error with model error.","supporting_citations":[{"cited_title":"Robertson, Steve Walker, Susan Jones, Micheline M","cited_arxiv_id":null,"evidence_quote":"Provides the lawyer-compiled annual state unemployment-insurance comparison tables that serve as the benchmark's gold answers and question templates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the dense passage retrieval technique used by the embedding-based retrievers in the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the state-policy covariates used in the exploratory analysis of why state-level RAG gains differ."}],"review_version":1}