Pith. sign in

REVIEW 3 major objections 5 minor 196 references

On real credit-card agreements, language models fail less at arithmetic than at rules, conditions, and contractual exceptions.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 15:59 UTC pith:VJMZUZAV

load-bearing objection Solid new consumer-finance literacy benchmark with clean PoT/CoT results; the contractual-error story is directionally right but over-claimed from a 50-example single-model audit. the 3 major comments →

arxiv 2607.26952 v1 pith:VJMZUZAV submitted 2026-07-29 cs.CL

Credit Cards, Confusion, Computation, and Consequences: What Can We Uncover About Language Model Reasoning?

Arnav Hiray , Agam Shah , Caleb Lu , Meghaj Tarte , Harsit Mittal , Sudheer Chava This is my paper
classification cs.CL
keywords financial literacynumerical reasoningcredit card agreementsProgram-of-ThoughtChain-of-Thoughterror analysisconditional logicLLM evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces CreditCardQA, a 1,800-question benchmark built from real credit-card agreements so that models must reason about fees, interest, minimum payments, and penalties the way consumers actually encounter them. The authors evaluate open- and closed-source language and reasoning models with both step-by-step verbal reasoning and program-style reasoning. Program-style prompting improves accuracy for every model, helps weaker models most, and shrinks the gap between open and closed systems. Error audits and a difficulty regression show that wrong answers usually come from applying the wrong formula, skipping an if-condition, or misreading agreement language—not from botching the arithmetic once the right inputs are chosen. Comparisons, branching logic, and monetary constraints are the hardest structural features, and mistakes concentrate in edge cases such as late fees and small balances that hit financially vulnerable users hardest. The practical claim is that personal-finance literacy is a high-stakes reasoning setting where contractual interpretation, not calculator skill, is the bottleneck.

Core claim

On CreditCardQA, model failures arise less from arithmetic than from misapplied financial rules, missed conditions, and misunderstandings of contractual terms. Program-of-Thought consistently beats Chain-of-Thought (gains of roughly 0.9 to 6.2 points at a 5% error tolerance), narrows open- versus closed-source gaps, and helps weaker baseline models most; comparisons, conditional logic, and monetary constraints are the structural features that most reduce correctness.

What carries the argument

CreditCardQA: 1,800 grounded question–answer pairs from real credit-card agreements, with operator-program annotations, first- versus third-person variants, and a logistic-regression difficulty model over structural operators (comparisons, branches, money inputs) plus agreement-level effects.

Load-bearing premise

The detailed map of why models fail rests mainly on a manual audit of fifty wrong answers from one strong open model, plus a regression fit on that same model’s outputs.

What would settle it

Re-run the same error taxonomy and difficulty regression on a multi-model sample of incorrect answers with independent annotators; if arithmetic or retrieval errors dominate, or if comparisons and conditions no longer predict failure, the central failure story does not hold.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Program-style prompting should be preferred over pure verbal chain-of-thought for consumer finance Q&A.
  • Benchmarks and product evals should stress comparison, branching, and fee/penalty edge cases rather than only multi-step arithmetic length.
  • First-person consumer phrasing can raise accuracy relative to textbook-style third-person questions.
  • Simple retrieval over agreements can drop accuracy when chunks omit cross-referenced conditions, so full-document or structure-aware context still matters.
  • Errors concentrated on late fees and small balances imply higher downside risk for lower-income users of AI finance helpers.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same rule-and-exception brittleness likely appears in other consumer contracts (mortgages, insurance, leases) that mix caps, floors, and nested ifs.
  • Training or test-time methods that force explicit enumeration of triggering conditions before any arithmetic may close more of the gap than larger base models alone.
  • Agreement-level difficulty clusters suggest issuers could rewrite disclosures for machine readability as a consumer-protection lever.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces CreditCardQA (also called CREDITQA/FINLITQA), a 1,800-question numerical-reasoning benchmark built from real CFPB and related credit-card agreements, with third- and first-person variants and operator-program annotations. It evaluates 11 open- and closed-source LLMs/LRMs under CoT and PoT, reporting consistent PoT gains (about +0.9 to +6.2 points at ±5% tolerance in Table 2), competitive open-weight performance (notably GPT-OSS-120B), and a narrowing of open/closed and LRM/LLM gaps under PoT. Error analysis on 50 incorrect GPT-OSS-120B answers (Table 3) attributes failures mainly to formula/rule substitution and missed conditions rather than arithmetic; a logistic regression on structural, linguistic, and input features (§6, Eq. 1, Appendix I) identifies comparisons, branching, monetary inputs, and third-person framing as difficulty drivers, with discussion of equity-relevant edge cases.

Significance. If the measured results hold, the paper fills a clear gap: prior financial QA benchmarks emphasize professional/market tasks, whereas this one targets consumer financial literacy grounded in real contractual text. Strengths include a carefully constructed, agreement-sourced dataset with DSL solution programs, a multi-model CoT/PoT comparison with two numeric tolerances, and a difficulty analysis that goes beyond step count. The finding that PoT helps weaker models and that comparison/conditional structure hurts accuracy is useful for both evaluation design and practical prompting. The equity-facing interpretation is suggestive rather than fully established, but the benchmark itself is a concrete, reusable contribution for high-stakes personal-finance reasoning.

major comments (3)
  1. [§5 Error Analysis (RQ2), Table 3] §5 and Table 3: The central qualitative claim that failures arise less from arithmetic than from misapplied rules and missed conditions rests on a manual audit of 50 incorrect GPT-OSS-120B answers (non-mutually exclusive labels, no multi-annotator agreement reported). This is too thin to support abstract/conclusion-level generalization across the 11-model suite or closed models. Either expand the audit (multiple models, agreement stats) or clearly scope the claim to this focused open-weight audit and treat cross-model prevalence as a hypothesis.
  2. [§6–7, Abstract] §6–7 and the abstract: The equity narrative (errors concentrated in late-payment/small-balance cases affecting lower-income users) is only weakly tied to the regression and the 50-example audit. Agreement-level γ effects and hardship indicators are discussed, but there is no systematic stratification of error rates by card type (subprime/secured vs premium) or by penalty/small-balance question subsets with statistical tests. Either add that analysis or tone down causal/social claims to match the evidence.
  3. [§2, Table 1, Dataset Release] Naming and construction consistency: the manuscript alternates CreditCardQA, CREDITQA, CREDITCARDQA, and FINLITQA (§2 opening vs title/abstract/tables). This is load-bearing for a benchmark paper because release, leaderboard, and citation identity depend on a single canonical name and clear dev/test protocol (800/1000). Unify naming and state exactly what is released vs held out.
minor comments (5)
  1. [§4] Table 2 header typo: "desmonstrates" in the CoT vs PoT discussion; also fix "formodels" spacing in the "Who benefits most" paragraph.
  2. [Appendix B] Appendix B is titled "Additional Related Works on LLM Reasoning" but contains the question-generation pipeline; renumber/retitle to avoid collision with Appendix A.
  3. [Appendix J / §4] RAG evaluation (Appendix J) uses only 140 questions and one model; briefly caveat external validity in the main text when citing it, or move the negative RAG result fully to appendix without over-claiming.
  4. [§6.1, Eq. (1)] Eq. (1) writes the logit in a hard-to-parse stacked form; present a standard logit equation and define γ_c(i) and third-person coding more explicitly near the equation.
  5. [Table 1, §2.3] Flesch–Kincaid: text says "low" scores corresponding to college level while reporting Min/Avg 9.2/11.3; clarify directionality (higher grade level = harder) for non-specialist readers.

Circularity Check

0 steps flagged

No significant circularity: empirical benchmark evaluation against human gold labels, not a self-encoding derivation.

full rationale

CreditCardQA is an external evaluation paper. Questions and operator programs are human-authored from real CFPB agreements; model accuracy under CoT/PoT is scored against those held-out gold answers (Table 2), not against quantities fitted from the same model outputs. The error taxonomy (Table 3) is a post-hoc manual audit of 50 failures, and the logistic regression (Eq. 1, Appendix I) relates annotated structural features to observed correctness—standard association analysis, not a prediction forced by construction. Self-citations (e.g., Hiray et al., Shah & Chava) supply related prior work and do not underwrite uniqueness theorems or force the main claims. Methodology is anchored in external precedents (ConvFinQA, DocMath-Eval). No self-definitional loop, fitted-input-as-prediction, or load-bearing self-citation chain appears in the derivation of the reported results.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 1 invented entities

Empirical NLP benchmark paper. Load-bearing background is standard LLM eval practice plus domain assumptions that CFPB-style agreements and annotator-written operator programs are a valid proxy for financial-literacy numerical reasoning. No new physical entities; free choices are eval thresholds, sampling for error/RAG studies, and regression feature design.

free parameters (4)
  • relative_error_tolerances = 0.2% and 5%
    Correctness uses ±0.2% and ±5% relative error bands chosen by the authors; rankings are said to be stable but the headline ~80% figures depend on the looser band.
  • error_audit_sample_size_and_model = n=50, GPT-OSS-120B
    Failure-mode percentages in Table 3 come from 50 incorrect GPT-OSS-120B items only; taxonomy prevalence is sample-dependent.
  • RAG_retrieval_hyperparameters = 900/160/30+30/0.55/8
    Chunk size 900, overlap 160, top-30 BM25 + top-30 dense, fuse weight 0.55, keep top-8; these hand choices drive the RAG ablation.
  • logistic_regression_feature_set
    Difficulty conclusions depend on the chosen structural, linguistic, and input indicators and agreement fixed effects in Eq. 1 / Table 5.
axioms (5)
  • domain assumption Operator-program gold solutions in the ConvFinQA-style DSL uniquely determine the intended numerical/categorical answer for each question.
    §2.2 and Appendix B require annotators to supply DSL programs; evaluation treats those outputs as ground truth.
  • domain assumption Markdown conversions of PDF agreements preserve all contractually material numeric terms and conditions needed for the questions.
    §2.1: Marker conversion plus manual verification; questions are generated only from Markdown.
  • ad hoc to paper Relative numeric tolerance matching plus limited manual review of categorical outputs is a valid accuracy metric for consumer financial answers.
    §3 and Appendix G define extraction, ±0.2%/±5% bands, and manual overrides for unit/category mismatches.
  • domain assumption Standard CoT and PoT prompting with temperature 0 adequately represents model reasoning skill on this task.
    §3/Appendix F follow prior finance-reasoning eval practice (Tang et al.; Zhao et al.).
  • standard math Logistic regression with agreement fixed effects identifies causal-ish difficulty factors from observational question features.
    §6 Eq. 1; coefficients interpreted as odds-ratio effects holding other features fixed.
invented entities (1)
  • CreditCardQA / CREDITQA dataset no independent evidence
    purpose: Provide 1,800 grounded financial-literacy numerical QA pairs with operator annotations, person framing, and train/leaderboard split.
    Core contribution; existence is evidenced by construction pipeline and promised HF/GitHub release, but independent reuse is not yet shown in this preprint.

pith-pipeline@v1.2.0-daily-grok45 · 28782 in / 3582 out tokens · 78053 ms · 2026-07-30T15:59:59.751081+00:00 · methodology

0 comments
read the original abstract

We introduce CreditCardQA, the first financial literacy benchmark for numerical reasoning derived from real credit card agreements. The dataset contains 1,800 questions, including first-person variants that reflect how consumers naturally ask about fees, interest, and payments. We evaluate a range of large language and reasoning models under Chain-of-Thought (CoT) and Program-of-Thought (PoT) prompting. Overall, PoT yields consistent performance gains, particularly for models with weaker baseline reasoning, and narrows gaps between open- and closed-source systems. Through error analysis, we show that failures arise less from arithmetic and more from misapplied financial rules, missed conditions, and misunderstandings of contractual terms. We further analyze question difficulty and find that comparisons, conditional logic, and monetary constraints are especially challenging. We also find that errors often arise in edge cases such as late-payment penalties or small-balance scenarios that are more likely to affect lower-income or financially vulnerable individuals.

Figures

Figures reproduced from arXiv: 2607.26952 by Agam Shah, Arnav Hiray, Caleb Lu, Harsit Mittal, Meghaj Tarte, Sudheer Chava.

Figure 1
Figure 1. Figure 1: CREDITCARDQA construction pipeline: CFPB credit card agreements → 280 extracted terms → 1,800 generated questions → LLM evaluation with CoT/PoT prompting. 2.1 Credit Card Agreements A credit card agreement is a legally binding document that specifies the terms and conditions governing a credit card account, including interest rates, fees, payment obligations, penalties, and consumer rights. These agreement… view at source ↗
Figure 2
Figure 2. Figure 2: Simplified process for answering CreditCardQA questions from credit card agree [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Chain-of-Thought (COT) prompt template. • What are the inputs required to compute P(A|T, B, Qn), and how do previous questions Q0, Q1, ..., Qn−1 factor into this computation? • What is the difference between Type I and Type II conversations? Testing Question Samples: • Use the ”train.json” provided as apart of the dataset and choose two conversations to test using GPT: one Type I and one Type II. Ensure th… view at source ↗
Figure 4
Figure 4. Figure 4: Program-of-Thought (PoT) prompt template. [PITH_FULL_IMAGE:figures/full_fig_p023_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

196 extracted references · 9 canonical work pages

  1. [1]

    Numerical Claim Detection in Finance: A New Financial Dataset, Weak-Supervision Model, and Market Analysis

    Shah, Agam and Hiray, Arnav and Shah, Pratvi and Banerjee, Arkaprabha and Singh, Anushka and Eidnani, Dheeraj Deepak and Chava, Sahasra and Chaudhury, Bhaskar and Chava, Sudheer. Numerical Claim Detection in Finance: A New Financial Dataset, Weak-Supervision Model, and Market Analysis. Proceedings of the Seventh Fact Extraction and VERification Workshop (...

    work page doi:10.18653/v1/2024.fever-1.21 2024

  2. [2]

    2024 , eprint=

    Alignment faking in large language models , author=. 2024 , eprint=

    2024

  3. [3]

    2025 , url=

    Kaixuan Huang and Jiacheng Guo and Zihao Li and Xiang Ji and Jiawei Ge and Wenzhe Li and Yingqing Guo and Tianle Cai and Hui Yuan and Runzhe Wang and Yue Wu and Ming Yin and Shange Tang and Yangsibo Huang and Chi Jin and Xinyun Chen and Chiyuan Zhang and Mengdi Wang , booktitle=. 2025 , url=

    2025

  4. [4]

    2025 , eprint=

    Benchmarking LLMs' Mathematical Reasoning with Unseen Random Variables Questions , author=. 2025 , eprint=

    2025

  5. [5]

    Lauter and Swarat Chaudhuri and Dawn Song , booktitle=

    Kaiyu Yang and Gabriel Poesia and Jingxuan He and Wenda Li and Kristin E. Lauter and Swarat Chaudhuri and Dawn Song , booktitle=. Position: Formal Mathematical Reasoning. 2025 , url=

    2025

  6. [6]

    Transactions on Machine Learning Research , issn=

    Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks , author=. Transactions on Machine Learning Research , issn=. 2023 , url=

    2023

  7. [7]

    2025 , eprint=

    Large Language Models Often Know When They Are Being Evaluated , author=. 2025 , eprint=

    2025

  8. [10]

    Nguyen and Quang Pham and Nghi D

    Dung Manh Nguyen and Thang Chau Phan and Nam Le Hai and Tien-Thong Doan and Nam V. Nguyen and Quang Pham and Nghi D. Q. Bui , booktitle=. Code. 2025 , url=

    2025

  9. [11]

    The 4th Workshop on Mathematical Reasoning and AI at NeurIPS'24 , year=

    DafnyBench: A Benchmark for Formal Software Verification , author=. The 4th Workshop on Mathematical Reasoning and AI at NeurIPS'24 , year=

  10. [12]

    Aryan Gulati and Brando Miranda and Eric Chen and Emily Xia and Kai Fronsdal and Bruno de Moraes Dumont and Sanmi Koyejo , booktitle=. Putnam-. 2024 , url=

    2024

  11. [13]

    F in QA : A Dataset of Numerical Reasoning over Financial Data

    Chen, Zhiyu and Chen, Wenhu and Smiley, Charese and Shah, Sameena and Borova, Iana and Langdon, Dylan and Moussa, Reema and Beane, Matt and Huang, Ting-Hao and Routledge, Bryan and Wang, William Yang. F in QA : A Dataset of Numerical Reasoning over Financial Data. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021...

    doi:10.18653/v1/2021.emnlp-main.300 2021

  12. [16]

    2023 , eprint=

    Zero is Not Hero Yet: Benchmarking Zero-Shot Performance of LLMs for Financial Tasks , author=. 2023 , eprint=

    2023

  13. [17]

    http://www.nber.org/papers/w17108

    Lusardi, Annamaria and Mitchell, Olivia S. Financial Literacy and Retirement Planning in the United States. 2011. doi:10.3386/w17108 , URL = "http://www.nber.org/papers/w17108", abstract =

    work page doi:10.3386/w17108 2011

  14. [18]

    http://www.nber.org/papers/w17103

    Lusardi, Annamaria. Americans' Financial Capability. 2011. doi:10.3386/w17103 , URL = "http://www.nber.org/papers/w17103", abstract =

    work page doi:10.3386/w17103 2011

  15. [19]

    The Review of Financial Studies , volume =

    Meta Brown and John Grigsby and Wilbert van der Klaauw and Jaya Wen and Basit Zafar , title =. The Review of Financial Studies , volume =. 2016 , doi =

    2016

  16. [20]

    2025 , month =

    Quarterly Report on Household Debt and Credit , institution =. 2025 , month =

    2025

  17. [21]

    Credit Card Agreements Database , year =

  18. [22]

    Flesch–Kincaid readability tests --- Wikipedia , The Free Encyclopedia

    Wikipedia contributors. Flesch–Kincaid readability tests --- Wikipedia , The Free Encyclopedia. 2025

    2025

  19. [23]

    2016 , month =

    Daniel Schorn , title =. 2016 , month =

    2016

  20. [24]

    and Korkou, Eleni D

    Giannikos, Christos L. and Korkou, Eleni D. , title =. International Journal of Financial Studies , volume =. 2025 , publisher =. doi:10.3390/ijfs13010022 , url =

    work page doi:10.3390/ijfs13010022 2025

  21. [25]

    2024 , month =

    Action needed to address gaps in financial literacy among students , howpublished =. 2024 , month =

    2024

  22. [26]

    Can AI help with your personal finances? , url =

    Oudom Hean and Utsha Saha and Binita Saha and , doi =. Can AI help with your personal finances? , url =. 2025 , bdsk-url-1 =. https://doi.org/10.1080/00036846.2025.2450384 , journal =

    arXiv 2025

  23. [27]

    Optimized Financial Planning: Integrating Individual and Cooperative Budgeting Models with LLM Recommendations , url =

    de Zarz. Optimized Financial Planning: Integrating Individual and Cooperative Budgeting Models with LLM Recommendations , url =. AI , number =. 2024 , bdsk-url-1 =. doi:10.3390/ai5010006 , issn =

    work page doi:10.3390/ai5010006 2024

  24. [29]

    2023 , eprint=

    FinanceBench: A New Benchmark for Financial Question Answering , author=. 2023 , eprint=

    2023

  25. [30]

    TAT - QA : A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance

    Zhu, Fengbin and Lei, Wenqiang and Huang, Youcheng and Wang, Chao and Zhang, Shuo and Lv, Jiancheng and Feng, Fuli and Chua, Tat-Seng. TAT - QA : A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conf...

    doi:10.18653/v1/2021.acl-long.254 2021

  26. [32]

    2024 , organization =

    The Personal Finance Index (P-Fin Index) , author =. 2024 , organization =

    2024

  27. [33]

    2024 , month = apr, organization =

    Can you answer these 3 questions about your finances? The majority of US adults cannot , author =. 2024 , month = apr, organization =

    2024

  28. [34]

    Market Share by Credit Card Issuer , year =

  29. [35]

    D oc M ath-Eval: Evaluating Math Reasoning Capabilities of LLM s in Understanding Long and Specialized Documents

    Zhao, Yilun and Long, Yitao and Liu, Hongjun and Kamoi, Ryo and Nan, Linyong and Chen, Lyuhao and Liu, Yixin and Tang, Xiangru and Zhang, Rui and Cohan, Arman. D oc M ath-Eval: Evaluating Math Reasoning Capabilities of LLM s in Understanding Long and Specialized Documents. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguis...

    doi:10.18653/v1/2024.acl-long.852 2024

  30. [36]

    2022 , organization =

    Do financial literacy interventions actually work better than I think they do? (and thoughts about meta-analyses) , author =. 2022 , organization =

    2022

  31. [37]

    2024 , month = nov, journal =

    Gen Z, Millennials Are Using AI for Personal Finance Advice, Report Finds , author =. 2024 , month = nov, journal =

    2024

  32. [38]

    2025 , month = may, organization =

    When Does Financial Education Work Best? Evidence-Based Insights , author =. 2025 , month = may, organization =

    2025

  33. [39]

    2025 , month = nov, journal =

    More than Half of People Use AI as a `Financial Adviser' , author =. 2025 , month = nov, journal =

    2025

  34. [40]

    http://www.nber.org/papers/w34255

    Chatterji, Aaron and Cunningham, Thomas and Deming, David J and Hitzig, Zoe and Ong, Christopher and Shan, Carl Yan and Wadman, Kevin. How People Use ChatGPT. 2025. doi:10.3386/w34255 , URL = "http://www.nber.org/papers/w34255", abstract =

    doi:10.3386/w34255 2025

  35. [41]

    Redesigning Financial Education to Engage and Entertain Audiences Is Delivering Results , year =

  36. [42]

    http://www.nber.org/papers/w27057

    Kaiser, Tim and Lusardi, Annamaria and Menkhoff, Lukas and Urban, Carly J. Financial Education Affects Financial Knowledge and Downstream Behaviors. 2020. doi:10.3386/w27057 , URL = "http://www.nber.org/papers/w27057", abstract =

    work page doi:10.3386/w27057 2020

  37. [43]

    Kilts Center at Chicago Booth Marketing Data Center Paper , year=

    Gender-Based Sorting in the Credit Card Market , author=. Kilts Center at Chicago Booth Marketing Data Center Paper , year=

  38. [44]

    2025 , eprint=

    MultiFinBen: A Multilingual, Multimodal, and Difficulty-Aware Benchmark for Financial LLM Evaluation , author=. 2025 , eprint=

    2025

  39. [45]

    F in DV er: Explainable Claim Verification over Long and Hybrid-content Financial Documents

    Zhao, Yilun and Long, Yitao and Jiang, Tintin and Wang, Chengye and Chen, Weiyuan and Liu, Hongjun and Tang, Xiangru and Zhang, Yiming and Zhao, Chen and Cohan, Arman. F in DV er: Explainable Claim Verification over Long and Hybrid-content Financial Documents. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. do...

    doi:10.18653/v1/2024.emnlp-main.818 2024

  40. [47]

    Financial Language Model Evaluation ( FL a ME )

    Matlin, Glenn and Okamoto, Mika and Pardawala, Huzaifa and Yang, Yang and Chava, Sudheer. Financial Language Model Evaluation ( FL a ME ). Findings of the Association for Computational Linguistics: ACL 2025. 2025. doi:10.18653/v1/2025.findings-acl.1164

    work page doi:10.18653/v1/2025.findings-acl.1164 2025

  41. [48]

    F inance R easoning: Benchmarking Financial Numerical Reasoning More Credible, Comprehensive and Challenging

    Tang, Zichen and E, Haihong and Ma, Ziyan and He, Haoyang and Liu, Jiacheng and Yang, Zhongjun and Rong, Zihua and Li, Rongjin and Ji, Kun and Huang, Qing and Hu, Xinyang and Liu, Yang and Zheng, Qianhe. F inance R easoning: Benchmarking Financial Numerical Reasoning More Credible, Comprehensive and Challenging. Proceedings of the 63rd Annual Meeting of t...

    doi:10.18653/v1/2025.acl-long.766 2025

  42. [49]

    Wang and Kaylie Hausknecht and Jonah Brenner and Danxian Liu and Nianli Peng and Corey Wang and Michael Brenner , booktitle=

    Jingxuan Fan and Sarah Martinson and Erik Y. Wang and Kaylie Hausknecht and Jonah Brenner and Danxian Liu and Nianli Peng and Corey Wang and Michael Brenner , booktitle=. 2024 , url=

    2024

  43. [50]

    FOLIO : Natural Language Reasoning with First-Order Logic

    Han, Simeng and Schoelkopf, Hailey and Zhao, Yilun and Qi, Zhenting and Riddell, Martin and Zhou, Wenfei and Coady, James and Peng, David and Qiao, Yujie and Benson, Luke and Sun, Lucy and Wardle-Solano, Alexander and Szab \'o , Hannah and Zubova, Ekaterina and Burtell, Matthew and Fan, Jonathan and Liu, Yixin and Wong, Brian and Sailor, Malcolm and Ni, A...

    2024

  44. [51]

    The Thirteenth International Conference on Learning Representations , year=

    Large Language Models Meet Symbolic Provers for Logical Reasoning Evaluation , author=. The Thirteenth International Conference on Learning Representations , year=

  45. [52]

    The 4th Workshop on Mathematical Reasoning and AI at NeurIPS'24 , year=

    How Transformers Reason: A Case Study on a Synthetic Propositional Logic Problem , author=. The 4th Workshop on Mathematical Reasoning and AI at NeurIPS'24 , year=

  46. [53]

    We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?

    Qiao, Runqi and Tan, Qiuna and Dong, Guanting and MinhuiWu, MinhuiWu and Sun, Chong and Song, Xiaoshuai and Wang, Jiapeng and GongQue, Zhuoma and Lei, Shanglin and Zhang, YiFan and Wei, Zhe and Zhang, Miaoxuan and Qiao, Runfeng and Zong, Xiao and Xu, Yida and Yang, Peiqing and Bao, Zhimin and Diao, Muxi and Li, Chen and Zhang, Honggang. We-Math: Does Your...

    2025

  47. [54]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  48. [55]

    Publications Manual , year = "1983", publisher =

    1983

  49. [56]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    arXiv 1981

  50. [57]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

  51. [58]

    Dan Gusfield , title =. 1997

    1997

  52. [59]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    2015

  53. [60]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

  54. [61]

    2024 , eprint=

    Evaluating Mathematical Reasoning of Large Language Models: A Focus on Error Identification and Correction , author=. 2024 , eprint=

    2024

  55. [62]

    2025 , eprint=

    FLAMES: Improving LLM Math Reasoning via a Fine-Grained Analysis of the Data Synthesis Pipeline , author=. 2025 , eprint=

    2025

  56. [63]

    2025 , eprint=

    A Toolbox, Not a Hammer -- Multi-TAG: Scaling Math Reasoning with Multi-Tool Aggregation , author=. 2025 , eprint=

    2025

  57. [64]

    2025 , eprint=

    Constructive Circuit Amplification: Improving Math Reasoning in LLMs via Targeted Sub-Network Updates , author=. 2025 , eprint=

    2025

  58. [65]

    2025 , eprint=

    Structured Reasoning with Tree-of-Thoughts for Bengali Math Word Problems , author=. 2025 , eprint=

    2025

  59. [66]

    2025 , eprint=

    DivLogicEval: A Framework for Benchmarking Logical Reasoning Evaluation in Large Language Models , author=. 2025 , eprint=

    2025

  60. [67]

    2025 , eprint=

    Dissecting Logical Reasoning in LLMs: A Fine-Grained Evaluation and Supervision Study , author=. 2025 , eprint=

    2025

  61. [68]

    2025 , url=

    Anonymous , booktitle=. 2025 , url=

    2025

  62. [69]

    2025 , eprint=

    TabDSR: Decompose, Sanitize, and Reason for Complex Numerical Reasoning in Tabular Data , author=. 2025 , eprint=

    2025

  63. [70]

    2025 , eprint=

    NUMINA: A Natural Understanding Benchmark for Multi-dimensional Intelligence and Numerical Reasoning Abilities , author=. 2025 , eprint=

    2025

  64. [71]

    2025 , eprint=

    MDToC: Metacognitive Dynamic Tree of Concepts for Boosting Mathematical Problem-Solving of Large Language Models , author=. 2025 , eprint=

    2025

  65. [72]

    2025 , eprint=

    Can Large Reasoning Models Improve Accuracy on Mathematical Tasks Using Flawed Thinking? , author=. 2025 , eprint=

    2025

  66. [73]

    2025 , eprint=

    DAG-Math: Graph-Guided Mathematical Reasoning in LLMs , author=. 2025 , eprint=

    2025

  67. [74]

    2025 , eprint=

    Ax-Prover: A Deep Reasoning Agentic Framework for Theorem Proving in Mathematics and Quantum Physics , author=. 2025 , eprint=

    2025

  68. [75]

    2025 , eprint=

    Population-Evolve: a Parallel Sampling and Evolutionary Method for LLM Math Reasoning , author=. 2025 , eprint=

    2025

  69. [76]

    2025 , eprint=

    Do Math Reasoning LLMs Help Predict the Impact of Public Transit Events? , author=. 2025 , eprint=

    2025

  70. [77]

    2025 , eprint=

    Rectify Evaluation Preference: Improving LLMs' Critique on Math Reasoning via Perplexity-aware Reinforcement Learning , author=. 2025 , eprint=

    2025

  71. [78]

    ArXiv , year=

    Fill in the Blank: Exploring and Enhancing LLM Capabilities for Backward Reasoning in Math Word Problems , author=. ArXiv , year=

  72. [79]

    ArXiv , year=

    MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems , author=. ArXiv , year=

  73. [80]

    Conference on Empirical Methods in Natural Language Processing , year=

    Fewer is More: Boosting Math Reasoning with Reinforced Context Pruning , author=. Conference on Empirical Methods in Natural Language Processing , year=

  74. [81]

    ArXiv , year=

    Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verification , author=. ArXiv , year=

  75. [82]

    ArXiv , year=

    Scheherazade: Evaluating Chain-of-Thought Math Reasoning in LLMs with Chain-of-Problems , author=. ArXiv , year=

  76. [83]

    ArXiv , year=

    Arithmetic Reasoning with LLM: Prolog Generation & Permutation , author=. ArXiv , year=

  77. [84]

    ArXiv , year=

    Confucius3-Math: A Lightweight High-Performance Reasoning LLM for Chinese K-12 Mathematics Learning , author=. ArXiv , year=

  78. [85]

    ArXiv , year=

    AgenticMath: Enhancing LLM Reasoning via Agentic-based Math Data Generation , author=. ArXiv , year=

  79. [86]

    ArXiv , year=

    LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts , author=. ArXiv , year=

  80. [87]

    ArXiv , year=

    From Ambiguity to Verdict: A Semiotic-Grounded Multi-Perspective Agent for LLM Logical Reasoning , author=. ArXiv , year=

Showing first 80 references.