Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarios

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read RuleArena asks whether LLMs can follow real-world rules and finds that even the strongest models fail most hard cases.

desk verdict Useful multi-domain rule-following benchmark with an NBA scoring inversion that must be fixed before its accuracy numbers can be trusted. read the letter →

arxiv 2412.08972 v2 pith:WBE6NTQY submitted 2024-12-12 cs.CL cs.AI

Ruiwen Zhou , Wenyue Hua , Liangming Pan , Sitao Cheng , Xiaobao Wu , En Yu , William Yang Wang This is my paper · ORCID
classification cs.CLcs.AI
keywords rule-guidedreasoningLLMbenchmarkrulefollowingchain-of-thoughtNBAcollectivebargainingagreementfederalincometaxairlinebaggagefeesrule-usagematrix
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that rule-guided reasoning—working out which of a set of long, real-world rules apply, in what order, and then computing the result—is a skill current large language models do not reliably have. To show this, it builds RuleArena: 816 test problems backed by 95 actual rules from airline baggage fees, NBA transactions under the collective bargaining agreement, and U.S. federal income tax. Each problem comes with a ground-truth answer and a fine-grained record of which rules should be used, so a model's failure can be attributed to missing a rule, confusing similar rules, or making an arithmetic error. The headline result is low final accuracy: the strongest evaluated models, including o1-preview with chain-of-thought prompting, solve only about half to 60 percent of the easiest problems and almost none of the hardest, even though they succeed at most individual rule applications. The intended consequence is that endpoint accuracy on multi-rule tasks is the right yardstick for deployment, and that current benchmarks overstate how well models follow instructions.

What carries the argument

The rule-usage matrix $U \in \{0,1\}^{N \times M}$ is the central object: for each of $N$ problems and each of $M$ domain rules, it records whether the model invoked that rule. From it the paper defines problem-wise recall $R(t)$, precision $P(t)$, application correctness $AC(t)$, and final accuracy $Acc(t)$, plus the corresponding rule-wise versions, so rule selection and rule application are scored separately. The three-level difficulty ladder—number of checked bags, number of teams and transactions, and number of tax forms—is what allows the paper to show that accuracy degrades as the rule-set grows. The controlled distractor experiment, which adds nullified tax forms versus meaningless padding, isolates the effect of irrelevant rules from mere context length.

What would settle it

Take a random sample of NBA problems and re-annotate them using the complete, unsimplified salary definitions from the 2023 collective bargaining agreement instead of the unified 'Team Salary'. If a nontrivial share of the legality labels flips, or if re-parsing the same model responses with a different extraction method changes the rule-usage matrices, then the reported recall-accuracy relationship and headline numbers rest on the simplification. A smaller check: run the airline and tax scripts against the official fee schedules and IRS instructions on a random sample and count discrepancies.

Watch

Extended reading notes

Core claim

RuleArena's construction turns real regulations into checkable problems. Airline and tax rules are converted into executable scripts that generate ground-truth totals, while NBA problems are written by human annotators who list the relevant rules and identify the violating team and transaction for each scenario. The benchmark then evaluates six LLMs under zero-shot and one-shot chain-of-thought prompting and parses their free-text responses with GPT-4o into a rule-usage matrix, from which it computes problem-wise and rule-wise recall, precision, and application correctness. The paper's central claim is that rule recall is the main bottleneck: problem-wise recall correlates almost linearly with final accuracy, whereas application correctness is high and precision is perfect in two domains; a single missed conditional rule or one arithmetic slip typically destroys the final answer. On NBA problems, precision is also low because similar rules with different applicability conditions—different types of salary-cap exceptions, for example—are routinely confused. These observations are offered as evidence that current LLMs are not yet reliable enough to follow official regulations end to end.

Load-bearing premise

The load-bearing premise is that the ground-truth labels are correct: the scripts must faithfully encode the airline and tax rules, the NBA annotators must correctly judge legality under the actual collective bargaining agreement (which the authors simplified by merging salary types into one 'Team Salary'), and GPT-4o must accurately parse model responses into rule-usage matrices.

Editorial extensions

If this is right

  • If RuleArena's numbers hold, any single-pass LLM system that applies multi-rule regulations (baggage fees, tax preparation, trade compliance) will produce incorrect answers on a large fraction of realistic inputs, making end-to-end deployment unsafe without verification.
  • Because rule recall tracks final accuracy almost linearly, the paper implies that better rule retrieval and rule selection—not more parametric knowledge or better arithmetic—is the main lever for improving rule-guided reasoning.
  • The similar-rule confusion in the NBA domain implies that model and prompt designs need explicit disambiguation mechanisms for near-identical rules with different conditions.
  • Tool augmentation with an external Python interpreter roughly doubles accuracy on easier airline problems but leaves most hard problems wrong, so offloading arithmetic is a partial fix at best.
  • The counterintuitive finding that a level-1 in-context example can lower NBA accuracy at levels 2 and 3 implies that few-shot prompting must match the difficulty distribution of the target task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • (Editor's inference) The benchmark's NBA ground truth could be a stress point: the authors simplified all team salary types into a single 'Team Salary', which can change whether a trade is legal, so reporting accuracy on the full, unsimplified collective bargaining agreement would be a direct follow-up test.
  • (Editor's inference) The distractor result suggests that in production, a retrieval or filtering layer that hands the LLM only plausibly relevant rules may be more valuable than training or prompting changes, because irrelevant rules measurably hurt performance.
  • (Editor's inference) Because the rule-usage parser is itself GPT-4o, the fine-grained recall and correctness numbers should be re-measured with an open parser to see how much of the reported analysis depends on that choice.
  • (Editor's inference) The same problem template could be applied to other regulated domains, such as insurance payouts or visa and immigration rules, to benchmark models for compliance-critical deployment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces RuleArena, a benchmark for evaluating LLMs' rule-guided reasoning in three real-world domains: airline baggage fees, NBA transactions, and US tax regulations. It contains 95 rules and 816 test problems, with annotations and ground-truth solutions derived from official policies and human annotators. The authors evaluate several LLMs (Llama-3.1 70B/405B, Qwen-2.5 72B, Claude-3.5 Sonnet, GPT-4o, o1-preview) under 0-shot and 1-shot chain-of-thought prompting, and report problem-level and rule-level metrics: precision, recall, rule-application correctness, and final-answer accuracy. The central finding is that current LLMs, including strong proprietary models, perform poorly on these tasks, especially on Level 2 and 3 problems, and that external tools (Python execution) provide only partial improvement. The paper also analyzes failure modes: missed rules, confusion between similar rules, and arithmetic errors.

Significance. If the results are valid, RuleArena is a valuable addition to instruction-following benchmarks because it tests logical and computational reasoning over long, realistic rule texts rather than simple style constraints. The fine-grained rule-usage metrics (problem-wise and rule-wise recall/precision/correctness) are a useful diagnostic beyond final-answer accuracy. The benchmark is derived from official public sources (American Airlines, NBA CBA, IRS), and the authors provide code and data, which supports reproducibility. However, the validity of the NBA results is questionable, and the rule-parsing pipeline is not validated, so the quantitative claims must be interpreted with caution.

major comments (3)
  1. [Appendix B.2 / Appendix E / Table 3] The NBA answer convention is internally inconsistent. Appendix B.2 defines the annotated 'answer' field as True if all transactions are allowed and False otherwise, while the prompt in Appendix E instructs the model to end with 'Answer: False.' if there is no violation and 'Answer: True. Illegal Operation: X...' if a violation exists. These conventions are complements: a legal transaction is annotated True but the correct model answer is 'False'. Section 3.2 adds further ambiguity by saying the model should generate 'Yes' for a legitimate transaction. The paper never specifies how the final answer token is mapped to the annotation for scoring. If the 'Answer:' token is compared directly with the binary annotation, then NBA Acc(t) in Table 3 is inverted: correct judgments are marked wrong and incorrect judgments are marked correct. This would invalidate all NBA accuracy numbers, the o1-preview '50~60%' Level-1 claim (Section 4.2.1) insofar as it relies on NBA, and the cross-domain comparison. The authors must either fix the annotation/prompt mismatch, describe the exact parsing protocol, and re-run the evaluation, or clearly report which transformation was applied.
  2. [Appendix B.2] The NBA ground truths are based on a stated simplification: 'unifying different types of team salary (defined in different rules and calculated in different ways) into one simple Team Salary.' This simplification can change whether a transaction is legal under the actual 2023 NBA CBA, because salary-cap rules often depend on specific salary definitions (e.g., 'salary' vs. 'cap hit' vs. 'tax salary'). The paper provides no argument or empirical check that the simplification preserves legality for the 216 annotated cases. Since the benchmark claims to evaluate real-world rule-following, the NBA domain's validity depends on this axiom. The authors should either revert to the actual salary definitions in the rule texts and annotations, or provide a per-case verification that the simplification does not alter any ground-truth verdict.
  3. [Section 4.1 / Appendix C / Limitations] All rule-usage metrics (R(t), AC(t), P(t), and the rule-wise versions) depend on the GPT-4o structured-output parser that decides, for each response, whether each rule was applied and whether it was applied correctly. The paper reports no validation of this parser: no human agreement, no per-domain accuracy, no error analysis. Given that the parser is itself an LLM and is used on outputs from the same families of models under evaluation, systematic parser bias could distort the fine-grained conclusions in Tables 3-7 (e.g., the low recall of non-essential rules, the low precision in NBA). The authors should report a human-annotated subset with parser agreement, and ideally release the parsing prompts and intermediate outputs for reproducibility.
minor comments (4)
  1. [Appendix E] The NBA prompt template contains a typo: 'desribes' should be 'describes'.
  2. [Section 3.2] The sentence 'If LLM thinks the transaction is legit, it should generate Yes' is inconsistent with the Appendix E prompt (which instructs 'Answer: False.' for no violation). Please reconcile these instructions, as this confusion likely contributed to the scoring ambiguity in the major comment above.
  3. [Figure 1] Figure 1 states '95 commonly used and moderately complex rules,' which matches the sum in Table 1 (10+54+31), but the figure caption also says '816 test problems,' which is consistent with Table 2; no issue here, but please ensure the final version renders these numbers clearly.
  4. [Section 4.2.2 / Table 4] The notation 'AC(r)' is used both as 'rule application correctness' and as a column header in Table 4; the text could clarify that the 'Ac(r)' in the table is the same quantity, and the inconsistent capitalization should be harmonized.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: RuleArena is an externally anchored benchmark evaluation, not a derivation from its own inputs.

full rationale

RuleArena is an empirical evaluation benchmark, so the central claim that current LLMs perform poorly at rule-guided reasoning is a measured observation rather than a derived theorem. Ground-truth answers are anchored outside the model outputs: airline and tax labels are computed by rule-based scripts translated from American Airlines and IRS policies (Section 3.2), and NBA labels come from annotators familiar with the 2023 CBA, with the annotation schema in Appendix B.2. Equation (4) compares model answers against these fixed external labels, and Table 3 simply reports those comparisons. No parameter is fitted to any evaluated model, and no reported accuracy is equivalent by construction to the rules or prompts that are given as inputs. The one self-referential element is the use of GPT-4o's structured output mode to parse rule-usage matrices, including for GPT-4o itself (Section 4.1). This is disclosed in the Limitations section as an LLM-as-a-judge style choice with potential bias or inaccuracies; it is a measurement instrument that could affect rule-level metrics, but it does not force the accuracy values or the qualitative conclusion. Citations to the authors' prior work (e.g., Logic-LM, TRAD, Antileak-Bench) appear in related-work or future-directions contexts and are not load-bearing for the benchmark's ground truth. The NBA boolean-convention mismatch noted in review (Appendix B.2 stores True for legal transactions while Appendix E asks models to conclude 'Answer: False.' when no violation exists) is a concrete potential evaluation-correctness bug that could threaten NBA Acc(t) if parsed directly, but it is not circularity: the labels remain anchored to external rules and human annotation. No circular step is established.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The benchmark rests on several domain assumptions. The official policies are assumed to be accurately transcribed and faithfully encoded into scripts; the NBA annotation is assumed to be correct despite a stated simplification of salary rules; and the GPT-4o parser is assumed to reliably recover rule usage from model outputs. No free parameters are fitted to data, and no new entities are postulated.

assumptions (5)
  • domain assumption The transcribed American Airlines, NBA CBA, and IRS rules accurately represent the real-world policies.
    The benchmark's ground truths and prompts are derived from these sources; any transcription error propagates to all results (Section 3.1, Appendix B.1).
  • domain assumption The rule-based scripts used to compute ground-truth answers correctly implement the official regulations.
    For airline and tax, the authors state they convert regulations into scripts and execute them to get answers; if the scripts have bugs, Acc(t) is affected (Section 3.2).
  • ad hoc to paper NBA rules can be simplified by unifying all team salary types into one 'Team Salary' without changing the legality of transactions.
    Appendix B.2 explicitly introduces this simplification to balance difficulty; it is a modeling choice that may diverge from the actual CBA.
  • domain assumption GPT-4o's structured parsing of LLM responses into rule-usage matrices is reliable.
    All rule-wise metrics and rule recall/precision/correctness depend on the parser; the paper acknowledges reliance on GPT-4o and does not provide human validation (Section 4.1, Limitations).
  • domain assumption Human annotators' relevant-rule lists and legality judgments for NBA problems are accurate and complete.
    The NBA ground truth and relevant_rules are produced by annotators; errors would bias both accuracy and rule-wise metrics (Appendix B.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarios." pith.science (2026). https://pith.science/paper/WBE6NTQY

@misc{pith2026241208972,
  author       = {Pith},
  title        = {Pith review of: RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarios},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WBE6NTQY}},
  note         = {Machine review of arXiv:2412.08972}
}
read the original abstract

This paper introduces RuleArena, a novel and challenging benchmark designed to evaluate the ability of large language models (LLMs) to follow complex, real-world rules in reasoning. Covering three practical domains -- airline baggage fees, NBA transactions, and tax regulations -- RuleArena assesses LLMs' proficiency in handling intricate natural language instructions that demand long-context understanding, logical reasoning, and accurate mathematical computation. Two key attributes distinguish RuleArena from traditional rule-based reasoning benchmarks: (1) it extends beyond standard first-order logic representations, and (2) it is grounded in authentic, practical scenarios, providing insights into the suitability and reliability of LLMs for real-world applications. Our findings reveal several notable limitations in LLMs: (1) they struggle to identify and apply the appropriate rules, frequently becoming confused by similar but distinct regulations, (2) they cannot consistently perform accurate mathematical computations, even when they correctly identify the relevant rules, and (3) in general, they perform poorly in the benchmark. We also observe a significant performance boost when LLMs are provided with external tools for oracle math and logic operations. These results highlight significant challenges and promising research directions in advancing LLMs' rule-guided reasoning capabilities in real-life applications. Our codes and data are publicly available on https://github.com/skyriver-2000/RuleArena.

Figures

Figures reproduced from arXiv: 2412.08972 by the authors.

Figure 1
Figure 1. Overview of RULEARENA. RULEARENA contains 95 commonly used and moderately complex rules and 816 test problems from three representative real-world scenarios - airline luggage fees, NBA transactions, and taxation policies. LLMs are given a the task instruction, the reference rules in this scenario, and a user instance, and required to conduct reasoning and computation for the user input under the guidance of referenc… view at source ↗
Figure 2
Figure 2. Rule-wise metrics of rules in airline domain. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Failure Case Studies. Existing LLMs commonly fail due to inadequate rule recall, inappropriate usage of [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Rule-wise metrics of rules in airline domain. [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: Rule-wise metrics of rules in NBA domain. [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 6
Figure 6. Figure 6: Rule-wise metrics of rules in tax domain. [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Correlation between problem-wise metrics and accuracy. The correlation is the most obvious and almost [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]
Figure 8
Figure 8. Figure 8: The effect of distractive rules and context length. The “Standard” mode refers to the default setting of [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge

    cs.CL 2024-12 conditional novelty 6.0 of 10

    AntiLeakBench automatically constructs QA benchmarks from knowledge updated after each model's cutoff, and its experiments suggest that pre-cutoff evaluation overstates LLM ability.

Reference graph

Works this paper leans on

51 extracted references · 5 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel - Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. https://arxiv.org/abs/1905.13319 Mathqa: Towards interpretable math word problem solving with operation-based formalisms . In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human...

    arXiv 2019

  4. [4]

    Anthropic. 2024. https://docs.anthropic.com/en/docs/resources/model-card The claude 3 model family: Opus, sonnet, haiku . Claude-3 Model Card

    work page 2024

  5. [5]

    Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2023. https://arxiv.org/abs/2211.12588 Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks . Transactions on Machine Learning Research

    arXiv 2023

  6. [7]

    Yihan Chen, Benfeng Xu, Quan Wang, Yi Liu, and Zhendong Mao. 2024 b . https://arxiv.org/abs/2401.00690 Benchmarking large language models on controllable generation under diversified instructions . In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17808--17816

    work page Pith review arXiv 2024

  7. [8]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . arXiv preprint arXiv:2110.14168

    arXiv 2021

  8. [9]

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. 2022. https://arxiv.org/abs/2301.00234 A survey on in-context learning . arXiv preprint arXiv:2301.00234

    arXiv 2022

Show all 51 references
  1. [10]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al - Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Z...

    2024 arXiv

  2. [11]

    Lizhou Fan, Wenyue Hua, Lingyao Li, Haoyang Ling, and Yongfeng Zhang. 2023. https://arxiv.org/abs/2312.14890 Nphardeval: Dynamic benchmark on reasoning ability of large language models via complexity classes . arXiv preprint arXiv:2312.14890

    2023 arXiv

  3. [12]

    Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. 2023. https://arxiv.org/abs/2301.12726 Specializing smaller language models towards multi-step reasoning . In International Conference on Machine Learning, pages 10421--10430. PMLR

    2023 arXiv

  4. [13]

    Nitish Gupta, Kevin Lin, Dan Roth, Sameer Singh, and Matt Gardner. 2020. https://arxiv.org/abs/1912.04971 Neural module networks for reasoning over text . In Proceedings of the 8th International Conference on Learning Representations ( ICLR )

    2020 arXiv

  5. [14]

    Joty, Alexander R

    Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Luke Benson, Lucy Sun, Ekaterina Zubova, Yujie Qiao, Matthew Burtell, David Peng, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Shafiq R. Joty...

    2022 arXiv

  6. [15]

    Qianyu He, Jie Zeng, Wenhao Huang, Lina Chen, Jin Xiao, Qianxi He, Xunzhe Zhou, Jiaqing Liang, and Yanghua Xiao. 2024. https://arxiv.org/abs/2309.09150 Can large language models understand real-world complex instructions? In Proceedings of the AAAI Conference on Artificial Int...

    2024 arXiv

  7. [16]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. https://arxiv.org/abs/2103.03874 Measuring mathematical problem solving with the MATH dataset . In Proceedings of the Neural Information Processing System...

    2021 arXiv

  8. [17]

    Wenyue Hua, Kaijie Zhu, Lingyao Li, Lizhou Fan, Shuhang Lin, Mingyu Jin, Haochen Xue, Zelong Li, JinDong Wang, and Yongfeng Zhang. 2024. https://arxiv.org/abs/2406.02787 Disentangling logic: The role of context in large language model reasoning capabilities . arXiv preprint ar...

    2024 arXiv

  9. [18]

    Cheonsu Jeong. 2024. https://arxiv.org/abs/2401.02981 Fine-tuning and utilization methods of domain-specific llms . arXiv preprint arXiv:2401.02981

    2024 arXiv

  10. [19]

    Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. 2024. https://arxiv.org/abs/2310.20410 Followbench: A multi-level fine-grained constraints following benchmark for large language models . In Proceedings ...

    2024 arXiv

  11. [20]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. https://arxiv.org/abs/2205.11916 Large language models are zero-shot reasoners . In Proceedings of the 36th Advances in Neural Information Processing Systems (NeurIPS)

    2022 arXiv

  12. [22]

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. https://github.com/tatsu-lab/alpaca_eval Alpacaeval: An automatic evaluator of instruction-following models

    2023

  13. [23]

    Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. https://arxiv.org/abs/1705.04146 Program induction by rationale generation: Learning to solve and explain algebraic word problems . In Proceedings of the 55th Annual Meeting of the Association for Computational Ling...

    2017 arXiv

  14. [24]

    Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Zhuoer Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, et al. 2023. https://arxiv.org/abs/2311.18743 Alignbench: Benchmarking chinese alignment of large language models . arXiv preprint arXiv:2311.18743

    2023 arXiv

  15. [25]

    Tenenbaum, and Jiajun Wu

    Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. 2019. https://arxiv.org/abs/1904.12584 The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision . In Proceedings of the 7th International Conference on Le...

    2019 arXiv

  16. [27]

    OpenAI. 2023. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . arXiv preprint arXiv:2303.08774

    2023 arXiv

  17. [28]

    OpenAI. 2024 a . https://openai.com/index/hello-gpt-4o Hello gpt-4o . OpenAI Blogs

    2024

  18. [29]

    OpenAI. 2024 b . https://arxiv.org/abs/2311.04235 Learning to reason with llms . OpenAI Blogs

    2024 arXiv

  19. [30]

    Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. 2023. https://arxiv.org/abs/2305.12295 Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning . In Findings of the Association for Computational Linguistics: EMNLP 2023 , p...

    2023 arXiv

  20. [31]

    Qwen Team . 2024. https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  21. [32]

    Jiao Sun, Yufei Tian, Wangchunshu Zhou, Nan Xu, Qian Hu, Rahul Gupta, John Frederick Wieting, Nanyun Peng, and Xuezhe Ma. 2023. https://arxiv.org/abs/2310.14542 Evaluating large language models on controlled generation tasks . arXiv preprint arXiv:2310.14542

    2023 arXiv

  22. [33]

    Wangtao Sun, Chenxiang Zhang, Xueyou Zhang, Ziyang Huang, Haotian Xu, Pei Chen, Shizhu He, Jun Zhao, and Kang Liu. 2024. https://arxiv.org/abs/2407.08440 Beyond instruction following: Evaluating rule following of large language models . arXiv preprint arXiv:2407.08440

    2024 arXiv

  23. [34]

    Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. 2021. https://arxiv.org/abs/2012.13048 Proofwriter: Generating implications, proofs, and abductive statements over natural language . In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021 , pages 3621--3634

    2021 arXiv

  24. [35]

    Xiangru Tang, Yiming Zong, Jason Phang, Yilun Zhao, Wangchunshu Zhou, Arman Cohan, and Mark Gerstein. 2023. https://arxiv.org/abs/2309.08963 Struc-bench: Are large language models really good at generating complex structured data? arXiv preprint arXiv:2309.08963

    2023 arXiv

  25. [36]

    Google Gemini Team. 2023. https://arxiv.org/abs/2312.11805 Gemini: A family of highly capable multimodal models . arXiv preprint arXiv:2312.11805

    2023 arXiv

  26. [37]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. https://arxiv.org/abs/2302.13971 Llama:...

    2023 arXiv

  27. [38]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. https://arxiv.org/abs/2212.10509 Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions . In Proceedings of the 61st Annual Meeting of the Association...

    2023 arXiv

  28. [39]

    Siyuan Wang, Zhongyu Wei, Yejin Choi, and Xiang Ren. 2024. https://arxiv.org/abs/2408.13654 Symbolic working memory enhances language models for complex rule application . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing ( EMNLP ) , pag...

    2024 arXiv

  29. [40]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. https://arxiv.org/abs/2201.11903 Chain-of-thought prompting elicits reasoning in large language models . In Proceedings of the 36th Advances in Neural In...

    2022 arXiv

  30. [41]

    Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. 2023. https://arxiv.org/abs/2303.03846 Larger language models do in-context learning differently . arXiv preprint arXiv:2303.03846

    2023 arXiv

  31. [42]

    Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxin Xu, Yiming Liu, Jie Tang, Hongning Wang, and Minlie Huang. 2024. https://arxiv.org/abs/2407.03978 Benchmarking complex instruction-following with multiple constraints ...

    2024 arXiv

  32. [43]

    Xiaobao Wu, Liangming Pan, William Yang Wang, and Anh Tuan Luu. 2024 a . https://aclanthology.org/2024.emnlp-main.843 AKEW : Assessing knowledge editing in the wild . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15118--15133

    2024

  33. [44]

    Xiaobao Wu, Liangming Pan, Yuxi Xie, Ruiwen Zhou, Shuai Zhao, Yubo Ma, Mingzhe Du, Rui Mao, Anh Tuan Luu, and William Yang Wang. 2024 b . Antileak-bench: Preventing data contamination by automatically constructing benchmarks with updated real-world knowledge. arXiv preprint ar...

    2024 arXiv

  34. [45]

    Congying Xia, Chen Xing, Jiangshu Du, Xinyi Yang, Yihao Feng, Ran Xu, Wenpeng Yin, and Caiming Xiong. 2024. https://arxiv.org/abs/2402.18667 Fofo: A benchmark to evaluate llms' format-following capability . arXiv preprint arXiv:2402.18667

    2024 arXiv

  35. [46]

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. https://arxiv.org/abs/2304.12244 Wizardlm: Empowering large language models to follow complex instructions . arXiv preprint arXiv:2304.12244

    2023 arXiv

  36. [47]

    Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li, and William Wang. 2024. https://arxiv.org/abs/2402.11436 Pride and prejudice: LLM amplifies self-bias in self-refinement . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (...

    2024 arXiv

  37. [48]

    Shaokun Zhang, Xiaobo Xia, Zhaoqing Wang, Ling-Hao Chen, Jiale Liu, Qingyun Wu, and Tongliang Liu. 2023. https://arxiv.org/abs/2310.10873 Ideal: Influence-driven selective annotations empower in-context learners in large language models . arXiv preprint arXiv:2310.10873

    2023 arXiv

  38. [49]

    Xiang Zhang and Dujian Ding. 2024. https://arxiv.org/abs/2410.14198 Supervised chain of thought . arXiv preprint arXiv:2410.14198

    2024 arXiv

  39. [50]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei - Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. https://arxiv.org/abs/2306.05685 Judging llm-as-a-judge with mt-bench and chatbot arena ....

    2023 arXiv

  40. [51]

    Wanjun Zhong, Siyuan Wang, Duyu Tang, Zenan Xu, Daya Guo, Yining Chen, Jiahai Wang, Jian Yin, Ming Zhou, and Nan Duan. 2022. https://arxiv.org/abs/2104.06598 Analytical reasoning of text . In Findings of the Association for Computational Linguistics: NAACL 2022 , pages 2306--2319

    2022 arXiv

  41. [52]

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. https://arxiv.org/abs/2311.07911 Instruction-following evaluation for large language models . arXiv preprint arXiv:2311.07911

    2023 arXiv

  42. [53]

    Ruiwen Zhou, Yingxuan Yang, Muning Wen, Ying Wen, Wenhao Wang, Chunling Xi, Guoqiang Xu, Yong Yu, and Weinan Zhang. 2024. https://arxiv.org/abs/2403.06221 TRAD : Enhancing llm agents with step-wise thought retrieval and aligned decision . In Proceedings of the 47th Internation...

    2024 arXiv

  43. [54]

    Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. 2023. https://arxiv.org/abs/2309.17167 Dyval: Graph-informed dynamic evaluation of large language models . arXiv preprint arXiv:2309.17167

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.