{"id":"f2aafe6a-1845-416d-9387-eaddd13fd1cb","arxiv_id":"2412.08972","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"RuleArena evaluates LLMs on realistic rule-guided reasoning and finds that even o1-preview solves only about half of the easiest problems and near zero of the hardest.","lead":"This paper introduces RuleArena, a benchmark of 816 problems on airline baggage fees, NBA transactions, and US tax rules, and shows that leading LLMs solve only a small fraction correctly. It finds that models omit relevant rules, confuse similar regulations, and make arithmetic errors even when the correct rule is used.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The NBA evaluation may invert the meaning of model answers: the prompt tells models to output 'Answer: False' when there is no rule violation, while the annotated ground truth stores True for legal transactions; if parsed directly, correct legal judgments are marked wrong and the NBA Acc(t)…","rationale":"The reader's verdict is CONDITIONAL and identifies ground-truth validity as the weakest assumption. I agree with that broad concern, but I find a more specific and more decisive weakness: the NBA answer-format inconsistency can invert the very labels the accuracy numbers are built on. This is load-bearing because the paper's headline 'LLMs are poor at rule-guided reasoning' is supported by the low Acc(t) values in Table 3; if the NBA portion of that table is computed under an inverted mapping, the numbers for one of the three domains are not measuring what they claim. The concern is concrete and testable: the prompt and annotation schema in the appendix provide enough information to verify the mapping without any new experiments. Other concerns—e.g., GPT-4o parser accuracy for rule-usage matrices and the fidelity of airline/tax scripts—affect secondary metrics or could shift numbers, but they do not threaten the validity of the final-answer accuracy in the same direct way. The NBA simplification of Team Salary is also an admitted divergence from the real CBA, and it weakens the 'real-world' framing, but it is disclosed and can be discussed as a limitation; the answer-format issue is not disclosed as such. Since the paper's verdict should remain conditional on fixing this evaluation issue (plus the reader's other conditions), I recommend UNCHANGED: the reader's CONDITIONAL verdict is the right call, with this concern added as a required check.","tokens_in":31095,"tokens_out":7592,"duration_ms":74499,"concrete_test":"Clone the RuleArena repository (linked in the paper), locate the NBA evaluation script, and trace how the model's final 'Answer: True/False' line is converted into the predicted boolean compared with the annotated 'answer' field. Run a unit test with two fabricated model outputs—'Answer: False.' and 'Answer: True. Illegal Operation: A. Problematic Team: B.'—against a ground-truth item whose annotated answer is True (legal). The correct semantics should score 'Answer: False' as correct; if the script scores 'Answer: True' as correct, the labels are inverted. Then recompute the NBA Acc(t) column of Table 3 under the corrected mapping and compare with the published numbers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that LLMs are poor at rule-guided reasoning depends on accurate accuracy numbers across all three domains. In the NBA domain, there is an internal inconsistency between the annotation schema and the evaluation prompt. Appendix B.2 defines the annotated 'answer' field as True if all transactions are allowed and False otherwise. Appendix E's NBA prompt instructs the model to conclude with 'Answer: False.' if there is no violation to the rules and 'Answer: True. Illegal Operation: X. Problematic Team: Y.' if a team in an operation violates the rules. These two conventions are complements: a legal transaction is annotated True but should be answered 'False'. The paper does not specify any mapping that reconciles this. If the final 'Answer:' token is parsed as a boolean and compared directly with the annotated label, then a model that correctly determines a transaction is legal will output 'Answer: False' and be scored incorrect, while a model that falsely flags a legal trade as illegal will be scored correct. This would invalidate NBA Acc(t) in Table 3 and any conclusion drawn from it, including the statement that o1-preview solves only about 50-60% of Level 1 problems if that figure refers to NBA. Section 3.2's instruction that 'If LLM thinks the transaction is legit, it should generate Yes' further contradicts the prompt, so the intended semantics are ambiguous. This is a concrete, resolvable evaluation bug rather than a speculation about model behavior.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RuleArena, a benchmark for evaluating LLMs' rule-guided reasoning in three real-world domains: airline baggage fees, NBA transactions, and US tax regulations. It contains 95 rules and 816 test problems, with annotations and ground-truth solutions derived from official policies and human annotators. The authors evaluate several LLMs (Llama-3.1 70B/405B, Qwen-2.5 72B, Claude-3.5 Sonnet, GPT-4o, o1-preview) under 0-shot and 1-shot chain-of-thought prompting, and report problem-level and rule-level metrics: precision, recall, rule-application correctness, and final-answer accuracy. The central finding is that current LLMs, including strong proprietary models, perform poorly on these tasks, especially on Level 2 and 3 problems, and that external tools (Python execution) provide only partial improvement. The paper also analyzes failure modes: missed rules, confusion between similar rules, and arithmetic errors.","tokens_in":31378,"tokens_out":4261,"duration_ms":40686,"significance":"If the results are valid, RuleArena is a valuable addition to instruction-following benchmarks because it tests logical and computational reasoning over long, realistic rule texts rather than simple style constraints. The fine-grained rule-usage metrics (problem-wise and rule-wise recall/precision/correctness) are a useful diagnostic beyond final-answer accuracy. The benchmark is derived from official public sources (American Airlines, NBA CBA, IRS), and the authors provide code and data, which supports reproducibility. However, the validity of the NBA results is questionable, and the rule-parsing pipeline is not validated, so the quantitative claims must be interpreted with caution.","major_comments":[{"comment":"The NBA answer convention is internally inconsistent. Appendix B.2 defines the annotated 'answer' field as True if all transactions are allowed and False otherwise, while the prompt in Appendix E instructs the model to end with 'Answer: False.' if there is no violation and 'Answer: True. Illegal Operation: X...' if a violation exists. These conventions are complements: a legal transaction is annotated True but the correct model answer is 'False'. Section 3.2 adds further ambiguity by saying the model should generate 'Yes' for a legitimate transaction. The paper never specifies how the final answer token is mapped to the annotation for scoring. If the 'Answer:' token is compared directly with the binary annotation, then NBA Acc(t) in Table 3 is inverted: correct judgments are marked wrong and incorrect judgments are marked correct. This would invalidate all NBA accuracy numbers, the o1-preview '50~60%' Level-1 claim (Section 4.2.1) insofar as it relies on NBA, and the cross-domain comparison. The authors must either fix the annotation/prompt mismatch, describe the exact parsing protocol, and re-run the evaluation, or clearly report which transformation was applied.","section":"Appendix B.2 / Appendix E / Table 3"},{"comment":"The NBA ground truths are based on a stated simplification: 'unifying different types of team salary (defined in different rules and calculated in different ways) into one simple Team Salary.' This simplification can change whether a transaction is legal under the actual 2023 NBA CBA, because salary-cap rules often depend on specific salary definitions (e.g., 'salary' vs. 'cap hit' vs. 'tax salary'). The paper provides no argument or empirical check that the simplification preserves legality for the 216 annotated cases. Since the benchmark claims to evaluate real-world rule-following, the NBA domain's validity depends on this axiom. The authors should either revert to the actual salary definitions in the rule texts and annotations, or provide a per-case verification that the simplification does not alter any ground-truth verdict.","section":"Appendix B.2"},{"comment":"All rule-usage metrics (R(t), AC(t), P(t), and the rule-wise versions) depend on the GPT-4o structured-output parser that decides, for each response, whether each rule was applied and whether it was applied correctly. The paper reports no validation of this parser: no human agreement, no per-domain accuracy, no error analysis. Given that the parser is itself an LLM and is used on outputs from the same families of models under evaluation, systematic parser bias could distort the fine-grained conclusions in Tables 3-7 (e.g., the low recall of non-essential rules, the low precision in NBA). The authors should report a human-annotated subset with parser agreement, and ideally release the parsing prompts and intermediate outputs for reproducibility.","section":"Section 4.1 / Appendix C / Limitations"}],"minor_comments":[{"comment":"The NBA prompt template contains a typo: 'desribes' should be 'describes'.","section":"Appendix E"},{"comment":"The sentence 'If LLM thinks the transaction is legit, it should generate Yes' is inconsistent with the Appendix E prompt (which instructs 'Answer: False.' for no violation). Please reconcile these instructions, as this confusion likely contributed to the scoring ambiguity in the major comment above.","section":"Section 3.2"},{"comment":"Figure 1 states '95 commonly used and moderately complex rules,' which matches the sum in Table 1 (10+54+31), but the figure caption also says '816 test problems,' which is consistent with Table 2; no issue here, but please ensure the final version renders these numbers clearly.","section":"Figure 1"},{"comment":"The notation 'AC(r)' is used both as 'rule application correctness' and as a column header in Table 4; the text could clarify that the 'Ac(r)' in the table is the same quantity, and the inconsistent capitalization should be harmonized.","section":"Section 4.2.2 / Table 4"}],"recommendation":"major_revision","confidential_remarks":"The NBA answer-inversion issue is a concrete, easily checkable bug: the authors can run a script to see whether their evaluation compares the model's 'Answer:' token directly with the annotated boolean. If so, the NBA Acc(t) numbers are the complement of the true accuracy, which would materially change the paper's quantitative claims. Even if the airline and tax results are unaffected, the NBA section and the aggregate claims (e.g., 'o1-preview can solve only about 50-60% of Level 1 problems') need to be recomputed. The NBA salary simplification also warrants a stronger justification or a validation study. The benchmark's overall direction is solid, and I would be willing to review a revised version. One additional thought for the editor: the authors should be encouraged to release the exact evaluation script (including the NBA answer parser) with the benchmark, as this would resolve the ambiguity definitively and is standard practice for benchmark papers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"RuleArena is a genuinely useful benchmark with one landmine in the middle of it. The airline and tax domains look solid: rules are authentic, ground truths are computed from executable scripts, and the metric suite separating rule selection (precision/recall) from rule application (AC) is a real step beyond end-accuracy-only evaluations. The finding that LLMs can recall the right rules yet still fail on computation, and the distractor experiment, are worth taking seriously. Public data and code help.\n\nThe NBA domain, however, appears to have an evaluation inversion that poisons Table 3's NBA Acc(t). Appendix B.2 defines annotated 'answer' as True iff all transactions are allowed (so True = legal/no violation). Appendix E's prompt instructs models to conclude 'Answer: False.' if there is no violation, and 'Answer: True. Illegal Operation: X...' if a team violates. Those are complements. A model that correctly decides the transaction is legal will emit 'Answer: False', which, parsed directly, is scored wrong against the True label; a model that falsely flags a legal trade will be scored right. Section 3.2's instruction to generate 'Yes' for legit transactions contradicts the prompt further. The paper specifies no mapping between these schemas. This isn't a subtle modeling concern; it means NBA Acc, and any conclusions drawn from it—including the 'o1 can solve only 50-60% of Level 1' headline if it covers NBA—are not trustworthy as reported.\n\nThe other weaknesses are minor by comparison. The GPT-4o parser for rule-usage matrices is unvalidated; a human-annotation sanity check on a sample would be cheap. The NBA salary simplification (unifying different salary types into one 'Team Salary') is acknowledged but its effect on whether transactions are legal is not analyzed. No variance across runs is reported. All fixable.\n\nSo: the central qualitative claim—LLMs struggle with long, interdependent real-world rules—survives on the airline and tax domains, and probably on NBA after a corrected parser. The stress-test concern is a real bug, not a manufactured one. I'd send this to serious peer review, with the requirement that the authors resolve the NBA label semantics before the numbers go to print. If they fix that, this becomes a standard testbed.","headline":"Useful multi-domain rule-following benchmark with an NBA scoring inversion that must be fixed before its accuracy numbers can be trusted.","tokens_in":31900,"tokens_out":2208,"would_cite":true,"duration_ms":23440,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RuleArena asks whether LLMs can follow real-world rules and finds that even the strongest models fail most hard cases.","keywords":["rule-guided reasoning","LLM benchmark","rule following","chain-of-thought","NBA collective bargaining agreement","federal income tax","airline baggage fees","rule-usage matrix"],"falsifier":"Take a random sample of NBA problems and re-annotate them using the complete, unsimplified salary definitions from the 2023 collective bargaining agreement instead of the unified 'Team Salary'. If a nontrivial share of the legality labels flips, or if re-parsing the same model responses with a different extraction method changes the rule-usage matrices, then the reported recall-accuracy relationship and headline numbers rest on the simplification. A smaller check: run the airline and tax scripts against the official fee schedules and IRS instructions on a random sample and count discrepancies.","tokens_in":30888,"feed_emoji":"📋","tokens_out":10136,"duration_ms":63865,"temperature":0.7,"pith_summary":"The paper argues that rule-guided reasoning—working out which of a set of long, real-world rules apply, in what order, and then computing the result—is a skill current large language models do not reliably have. To show this, it builds RuleArena: 816 test problems backed by 95 actual rules from airline baggage fees, NBA transactions under the collective bargaining agreement, and U.S. federal income tax. Each problem comes with a ground-truth answer and a fine-grained record of which rules should be used, so a model's failure can be attributed to missing a rule, confusing similar rules, or making an arithmetic error. The headline result is low final accuracy: the strongest evaluated models, including o1-preview with chain-of-thought prompting, solve only about half to 60 percent of the easiest problems and almost none of the hardest, even though they succeed at most individual rule applications. The intended consequence is that endpoint accuracy on multi-rule tasks is the right yardstick for deployment, and that current benchmarks overstate how well models follow instructions.","feed_headline":"Even top LLMs fail most real-world rule tasks, new benchmark finds","feed_subtitle":"An 816-case test spanning airline fees, NBA trades, and tax law shows GPT-4o and o1-preview still fall far short.","key_machinery":"The rule-usage matrix $U \\in \\{0,1\\}^{N \\times M}$ is the central object: for each of $N$ problems and each of $M$ domain rules, it records whether the model invoked that rule. From it the paper defines problem-wise recall $R(t)$, precision $P(t)$, application correctness $AC(t)$, and final accuracy $Acc(t)$, plus the corresponding rule-wise versions, so rule selection and rule application are scored separately. The three-level difficulty ladder—number of checked bags, number of teams and transactions, and number of tax forms—is what allows the paper to show that accuracy degrades as the rule-set grows. The controlled distractor experiment, which adds nullified tax forms versus meaningless padding, isolates the effect of irrelevant rules from mere context length.","core_discovery":"RuleArena's construction turns real regulations into checkable problems. Airline and tax rules are converted into executable scripts that generate ground-truth totals, while NBA problems are written by human annotators who list the relevant rules and identify the violating team and transaction for each scenario. The benchmark then evaluates six LLMs under zero-shot and one-shot chain-of-thought prompting and parses their free-text responses with GPT-4o into a rule-usage matrix, from which it computes problem-wise and rule-wise recall, precision, and application correctness. The paper's central claim is that rule recall is the main bottleneck: problem-wise recall correlates almost linearly with final accuracy, whereas application correctness is high and precision is perfect in two domains; a single missed conditional rule or one arithmetic slip typically destroys the final answer. On NBA problems, precision is also low because similar rules with different applicability conditions—different types of salary-cap exceptions, for example—are routinely confused. These observations are offered as evidence that current LLMs are not yet reliable enough to follow official regulations end to end.","pith_inferences":["(Editor's inference) The benchmark's NBA ground truth could be a stress point: the authors simplified all team salary types into a single 'Team Salary', which can change whether a trade is legal, so reporting accuracy on the full, unsimplified collective bargaining agreement would be a direct follow-up test.","(Editor's inference) The distractor result suggests that in production, a retrieval or filtering layer that hands the LLM only plausibly relevant rules may be more valuable than training or prompting changes, because irrelevant rules measurably hurt performance.","(Editor's inference) Because the rule-usage parser is itself GPT-4o, the fine-grained recall and correctness numbers should be re-measured with an open parser to see how much of the reported analysis depends on that choice.","(Editor's inference) The same problem template could be applied to other regulated domains, such as insurance payouts or visa and immigration rules, to benchmark models for compliance-critical deployment."],"forward_implications":["If RuleArena's numbers hold, any single-pass LLM system that applies multi-rule regulations (baggage fees, tax preparation, trade compliance) will produce incorrect answers on a large fraction of realistic inputs, making end-to-end deployment unsafe without verification.","Because rule recall tracks final accuracy almost linearly, the paper implies that better rule retrieval and rule selection—not more parametric knowledge or better arithmetic—is the main lever for improving rule-guided reasoning.","The similar-rule confusion in the NBA domain implies that model and prompt designs need explicit disambiguation mechanisms for near-identical rules with different conditions.","Tool augmentation with an external Python interpreter roughly doubles accuracy on easier airline problems but leaves most hard problems wrong, so offloading arithmetic is a partial fix at best.","The counterintuitive finding that a level-1 in-context example can lower NBA accuracy at levels 2 and 3 implies that few-shot prompting must match the difficulty distribution of the target task."],"supporting_citations":[{"why":"Supplies the chain-of-thought prompting strategy used for every model in the main evaluation.","marker":"Wei et al., 2022"},{"why":"Supplies the zero-shot chain-of-thought variant used in the 0-shot setting.","marker":"Kojima et al., 2022"},{"why":"Identifies GPT-4o, both a model under evaluation and the parser that converts responses into rule-usage matrices.","marker":"OpenAI, 2024a"},{"why":"Identifies o1-preview, the reasoning model whose roughly 50 to 60 percent Level 1 accuracy anchors the headline.","marker":"OpenAI, 2024b"},{"why":"Identifies Claude-3.5 Sonnet, one of the strongest models the conclusion singles out as failing the hardest tasks.","marker":"Anthropic, 2024"},{"why":"Defines the Llama-3.1 70B and 405B models that provide the open-model lower end of the comparison.","marker":"Dubey et al., 2024"},{"why":"Defines Qwen2.5 72B, another open model in the comparison.","marker":"Qwen Team, 2024"}],"fun_headline_variants":["LLMs fall short on real-world rule puzzles, new benchmark shows","RuleArena: Top AI models struggle with airline, NBA, tax rules","New benchmark reveals LLMs' weak spot: following complex regulations","Even advanced LLMs miss rules and botch math in real-world scenarios","Benchmark finds LLMs often confuse similar rules, fail at arithmetic"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the ground-truth labels are correct: the scripts must faithfully encode the airline and tax rules, the NBA annotators must correctly judge legality under the actual collective bargaining agreement (which the authors simplified by merging salary types into one 'Team Salary'), and GPT-4o must accurately parse model responses into rule-usage matrices.","fun_headline_variants_meta":{"raw":{"variants":["LLMs fall short on real-world rule puzzles, new benchmark shows","RuleArena: Top AI models struggle with airline, NBA, tax rules","New benchmark reveals LLMs' weak spot: following complex regulations","Even advanced LLMs miss rules and botch math in real-world scenarios","Benchmark finds LLMs often confuse similar rules, fail at arithmetic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000274,"raw_usage":{"total_tokens":1671,"prompt_tokens":1010,"completion_tokens":661,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":626,"completion_tokens_details":{"reasoning_tokens":568}},"tokens_in":626,"tokens_out":661,"duration_ms":7030,"temperature":1.0,"reasoning_tokens":568,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:20:58.772214+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of NBA problems and re-annotate them using the complete, unsimplified salary definitions from the 2023 collective bargaining agreement instead of the unified 'Team Salary'. If a nontrivial share of the legality labels flips, or if re-parsing the same model responses with a different extraction method changes the rule-usage matrices, then the reported recall-accuracy relationship and headline numbers rest on the simplification. A smaller check: run the airline and tax scripts against the official fee schedules and IRS instructions on a random sample and count discrepancies.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Identifies Claude-3.5 Sonnet, one of the strongest models the conclusion singles out as failing the hardest tasks."}],"review_version":1}