Pith. sign in

REVIEW 1 major objections 7 minor 32 references

MIRA-Math: A Benchmark for Minimal Information Requesting and Mathematical Reasoning

T0 review · 1 major / 7 minor · reviewed 2026-07-09 · glm-5.2

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Asking the right question and using the answer are separate skills

desk verdict Controlled benchmark isolating information-requesting from mathematical integration in LLMs; responder channel noise is the main open question. read the letter →

arxiv 2607.07391 v1 pith:DSQXG63K submitted 2026-07-08 cs.AI

Charbel Al Bateh , Samer Saab Jr This is my paper
classification cs.AI
keywords mathematicalreasoninginformationacquisitionbenchmarkpartialobservabilitylargelanguagemodelsdiagnosticevaluationinteractiveproblemsolving
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces MIRA-Math, a benchmark designed to isolate a specific capability that standard mathematical reasoning evaluations overlook: can a model recognize that a problem is missing a necessary fact, ask for that exact fact in natural language, and then correctly integrate it into a final exact answer? Each of the 2,310 instances is generated from a fully determined mathematical state (spanning algebra, probability, linear systems, signal processing, Markov chains, circuits, and more), but the solver receives a deliberately incomplete view with exactly one atomic fact removed. A fixed, constrained responder model holds only the missing fact and must either offer it (if the solver's request semantically matches) or decline. The benchmark's deterministic components cover instance generation, hint specification, validation, and final-answer checking; the one non-deterministic component is the LLM-mediated semantic matching between the solver's free-form request and the held fact. The central experimental finding is that request success and final-answer accuracy are separable: models that ask for the right fact often fail the downstream computation, and models that compute well may fail to elicit the needed hint in the first place. The paper demonstrates this with a trace-level decomposition showing, for example, that one model acquires the canonical hint in 78.7% of instances but converts only 24.8% to correct final answers, while another model has the highest request hit rate but lower final accuracy than a competitor.

What carries the argument

The benchmark's key machinery is the global-well-posed / local-underdetermined construction: each instance has a unique answer given all facts, but the solver's private view is ambiguous until the single missing atomic hint is supplied. The fixed constrained responder channel—offering only a structured offer-or-decline response—prevents the information holder from solving, explaining, or volunteering extra information, isolating the solver's requesting precision and downstream computation as the only variables under test.

What would settle it

If a deterministic matching baseline (e.g., exact keyword or template matching against the canonical hint specification) were applied to the same transcripts and produced substantially different offer/decline decisions than the LLM responder, then a significant fraction of measured 'request-acquisition failures' would be attributable to responder noise rather than solver inability, undermining the benchmark's diagnostic decomposition.

Watch

Extended reading notes

Core claim

The paper's central discovery is that information acquisition and mathematical integration are independently measurable failure modes. By decomposing each transcript into three binary outcomes—whether the canonical hint was acquired, whether the final answer was correct, and whether the hint was acquired but the answer was wrong—the authors show that a single aggregate accuracy score can mask fundamentally different weaknesses: a model may fail because it cannot localize which slot is missing, because it phrases its request too vaguely for the responder to match, or because it receives the correct value but botches the arithmetic. The Type A / Type B split (fixed missing slot vs. variable,实例

Load-bearing premise

The benchmark assumes that the fixed LLM responder (gpt-4o-mini) is a sufficiently stable and neutral arbiter of whether a solver's natural-language request matches the held atomic fact. If the responder systematically misjudges requests—declining valid ones due to phrasing variation or accepting noncanonical ones—then the request metrics conflate solver requesting ability with responder matching noise, and the paper does not quantify responder error rates or provide a purely

Editorial extensions

If this is right

  • Model developers can use the trace-level decomposition to diagnose whether failures stem from communication precision, slot localization, or mathematical computation, enabling targeted improvement of each subskill rather than treating math reasoning as monolithic.
  • The finding that four-shot prompting can hurt accuracy—by inducing exemplar anchoring where the solver imitates demonstrated request forms instead of adapting to the current instance—suggests that prompt design for information-requesting tasks requires instance-specific adaptation rather than fixed demonstrations.
  • The benchmark's deterministic generation and verification pipeline, combined with the release of generators, verifiers, and raw transcripts, makes it possible to re-evaluate models as APIs change, turning a one-time leaderboard into a regression suite for the specific capability of asking for missing information.
  • Families where the request is easy but the computation is brittle (e.g., Chinese-remainder reconstruction, phase retrieval) vs. families where slot localization is the bottleneck (e.g., birth-death chains, linear systems with a missing coefficient) provide a per-family diagnostic map that can guide capability-specific training or architectural choices.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the responder's semantic matching is a significant source of noise, then the request hit rate and first-request success metrics conflate solver ability with responder reliability. A deterministic matching baseline (e.g., keyword or template-based) would let one quantify how much of the measured 'request-acquisition failure' is actually responder error, and whether different solver models are di
  • The separability of requesting and computing suggests a natural architectural implication: a system that decouples the information-acquisition module from the mathematical-computation module could be optimized for each subskill independently, potentially outperforming a monolithic model that must do both in a single forward pass.
  • The benchmark currently requires exactly one missing fact per instance. If the same decomposition were extended to multi-hint instances, one could study whether models can prioritize which missing fact to request first—a capability closer to real-world information gathering where multiple unknowns exist and request budgets are limited.
  • The Type A / Type B distinction (fixed vs. variable missing slot) could be generalized into a spectrum of slot-localization difficulty, predicting that models with stronger self-monitoring or working-memory-like mechanisms would show a steeper advantage on Type B families.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 7 minor

Summary. This paper introduces MIRA-Math, a benchmark of 2,310 instances across 22 typed mathematical families designed to isolate a specific capability: recognizing that a problem is missing one necessary atomic fact, requesting that fact in natural language from a constrained information-holder LLM, and then integrating the returned fact into an exact final answer. The benchmark's deterministic components (instance generation, hint specifications, validation, and final-answer verification) are carefully separated from its non-deterministic component (an LLM-mediated responder that performs semantic matching between solver requests and held atomic facts). The central empirical claim is that request success and final-answer accuracy are separable capabilities, supported by a trace-level decomposition (Table 3) showing that models may acquire the canonical hint yet fail downstream computation, or fail to acquire it at all. Six solver models are evaluated under zero-shot and four-shot regimes, with full per-family breakdowns and released transcripts.

Significance. The benchmark addresses a genuine gap in the evaluation literature: existing math benchmarks assume full information, while interactive benchmarks conflate information acquisition with many other capabilities. The construction is methodologically careful—deterministic generators with family-specific nondegeneracy checks, exact rational arithmetic, and exact verifiers—and the trace-level decomposition is computed from released transcripts without additional model calls. The release of generators, verifiers, prompts, run metadata, and raw transcripts is a significant strength that supports reproducibility. The Type A/Type B distinction (fixed vs. variable missing slots) is a useful diagnostic axis. The separability finding, while perhaps not surprising in the abstract, is made concrete and quantitatively supported by the trace-level data.

major comments (1)
  1. §3.2, §8, Tables 11–34: The LLM-mediated responder channel is the benchmark's weakest link, and the paper acknowledges this (§8: 'request metrics can still reflect occasional responder matching errors'). However, the concern is most acute in specific families where ALL six solver models show ~0% first-request success across both prompt regimes. In moment_problem (Tables 12, 16, 20, 24, 28, 32), the responder holds a specific equation (e.g., 'q1 + 4q2 = 5'), and the T4 transcript (Appendix D) shows the solver asking for 'the 2nd moment'—a semantically reasonable request that is declined. In graph_path_sums, all models also show near-zero first-request success. The paper does not provide any deterministic matching baseline, responder error rate estimate, or human audit of responder decisions to distinguish 'models genuinely cannot formulate the request' from 'responder systematicallyreject
minor comments (7)
  1. Table 5 notes that when gpt-4o-mini is the solver, Agents A and B are the same model. The paper does not discuss whether this self-interaction creates any systematic bias (e.g., shared vocabulary or phrasing patterns that could inflate or deflate matching rates). A brief remark would be helpful.
  2. Table 22 (gemini-2.5-flash four-shot effect): accuracy drops from 0.490 (ZS) to 0.328 (4S), with poly_interpolation showing a -0.660 delta. The paper frames this as a 'prompt-sensitivity hypothesis' (§6). While the honesty is appreciated, the magnitude is large enough that a brief discussion of whether this could be an artifact of the specific four-shot examples chosen (rather than a general phenomenon) would strengthen the analysis.
  3. §4.1: The 20/50 split (20 instances per difficulty for Type A, 50 for Type B) is stated but not justified. Why not equal counts? A one-sentence rationale would help.
  4. Appendix C, family A2 (crt_reconstruction) and B4 (discrete_tomography): these families 'do not scale with difficulty' and are always run at d=1. This means the difficulty-by-family breakdown (Table 8) is uninformative for these families. Consider noting this explicitly in the table or caption.
  5. Table 3 caption: 'Acc. given canonical hint' is defined as Correct / (Hint but wrong + Correct), but this is not stated explicitly in the caption or in §3.4 where the decomposition is introduced. Adding the formula would improve clarity.
  6. §3.4: The token estimate ('mean transcript word-count × 1.3') is approximate. The basis for the 1.3 multiplier is not explained. A brief justification or citation would be appropriate.
  7. The paper uses 'gpt-5.1' and 'Gemma-4' as model aliases. If these are not yet widely established names, a footnote or table noting the exact checkpoint versions would aid reproducibility beyond what Table 5 already provides.

Simulated Author's Rebuttal

5 responses · 0 unresolved

We thank the referee for the careful reading and the constructive recommendation. The referee's single major comment concerns the LLM-mediated responder channel—specifically, whether near-zero first-request success in families like moment_problem and graph_path_sums reflects genuine solver inability or systematic responder rejection of semantically reasonable requests. We agree this distinction is important and will address it with three additions: (1) a deterministic keyword/regex matching baseline run alongside the LLM responder, (2) a human audit of responder decisions on the two flagged families, and (3) an explicit responder error-rate estimate. We also clarify below which aspects of the referee's concern we can fully resolve and which remain inherent to the benchmark's design.

read point-by-point responses

  1. Referee: §3.2, §8, Tables 11–34: The LLM-mediated responder channel is the benchmark's weakest link... In moment_problem, the responder holds a specific equation (e.g., 'q1 + 4q2 = 5'), and the T4 transcript shows the solver asking for 'the 2nd moment'—a semantically reasonable request that is declined. In graph_path_sums, all models also show near-zero first-request success. The paper does not provide any deterministic matching baseline, responder error rate estimate, or human audit of responder decisions to distinguish 'models genuinely cannot formulate the request' from 'responder systematically rejects reasonable requests.'

    Authors: The referee is correct that the responder channel is the benchmark's weakest link, and we agree that the current manuscript does not provide sufficient evidence to disentangle solver failure from responder matching failure in the families flagged. We will address this in the revision with three concrete additions, described below. We also agree with the referee's specific observation about the T4 transcript: the solver asking for 'the 2nd moment' when the responder holds the equation 'q1 + 4q2 = 5' is a semantically reasonable request that the responder declines, and this is a genuine responder matching limitation rather than a solver failure. We acknowledge this explicitly in the revision. revision: yes

  2. Referee: Deterministic matching baseline

    Authors: We will add a deterministic keyword/regex matching baseline that runs alongside the LLM responder on all 2,310 instances. For each family, the deterministic matcher checks whether the solver's request contains family-specific canonical keywords (e.g., 'second moment' or 'q1 + 4q2' for moment_problem; 's3' or 'e01 + e20' for graph_path_sums). This baseline provides a lower bound on what a non-LLM matcher would accept, making it possible to compare LLM-responder acceptance rates against a transparent, auditable rule. Where the deterministic matcher accepts requests that the LLM responder declines, this will be flagged as a potential responder error. We will report per-family acceptance rates under both matching conditions in a new appendix table. We expect this to confirm that in moment_problem and graph_path_sums, the LLM responder is systematically stricter than a reasonable deterministic matcher, inflating the apparent solver failure rate in those families. revision: yes

  3. Referee: Responder error rate estimate

    Authors: We will compute and report an estimated responder error rate. Specifically, for each family, we will compare the LLM responder's offer/decline decisions against the deterministic matcher's decisions on the same transcripts. Cases where the deterministic matcher would accept but the LLM responder declines will be classified as potential false negatives; cases where the LLM responder offers a non-canonical constraint will be classified as potential false positives. We will report the per-family false-negative rate, which is the more consequential direction for the referee's concern. This will be added to the appendix alongside the deterministic baseline table. We note that this estimate is itself a lower bound on the true error rate, since the deterministic matcher may also be imperfect—but it provides a transparent, reproducible reference point that the current manuscript lacks. revision: yes

  4. Referee: Human audit of responder decisions in flagged families

    Authors: We will conduct a human audit of all responder decisions in the two families the referee specifically flags: moment_problem and graph_path_sums. For each transcript in these families across all six solver models and both prompt regimes, one of the authors will classify each responder decline as (a) correct decline—the request does not identify the held atomic fact, (b) borderline—the request is semantically related but imprecise, or (c) likely error—the request reasonably identifies the held fact but was declined. We will report the counts and percentages of each category. Based on preliminary inspection of the T4 transcript and similar moment_problem traces, we expect category (c) to be non-trivial in this family, which supports the referee's concern. For graph_path_sums, the issue appears to be different: the held fact is a path-sum equation (e.g., 'e01 + e20 = s3'), and solvers tend to ask for individual edge weights rather than the path-sum equation, which is a genuine solver formulation gap rather than a responder error. The human audit will make this distinction explicit. revision: yes

  5. Referee: Discussion of implications for benchmark interpretation

    Authors: We will add a subsection to §8 (Limitations) explicitly discussing the implications of the responder matching analysis for benchmark interpretation. Specifically, we will state that: (1) request metrics in families with high responder false-negative rates (e.g., moment_problem) should be interpreted as upper bounds on solver failure rather than precise measurements, since some declines reflect responder limitations; (2) the final-answer accuracy metric remains unaffected by responder matching errors, since it is computed by deterministic verifiers; and (3) the separability finding (Table 3) is robust to responder errors in one direction—if anything, responder false negatives inflate the 'no canonical hint' category, meaning the true integration-failure rate is at least as high as reported. We believe this honest accounting strengthens the paper's credibility without undermining its central claims. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the benchmark is constructed from independent mathematical generators with deterministic validation, and the central separability claim is an empirical observation from external model evaluations, not a definitional identity.

full rationale

The paper's derivation chain is self-contained and non-circular. The benchmark instances are generated deterministically from 22 typed mathematical families with exact verifiers (Section 4.2), independent of any evaluation results. The central empirical claim — that request success and final-answer accuracy are separable capabilities — is supported by independently measured quantities in Table 3: the trace-level indicators C_i (canonical hint acquired) and F_i (final answer correct) are computed from separate audit steps (offline hint matching and exact-answer verification respectively). The separability finding arises from observing that these independently measured quantities diverge across models (e.g., gpt-4o-mini ZS: 78.7% hint acquisition vs 24.8% accuracy), which is an empirical observation, not a definitional identity. No parameters are fitted to data and then 'predicted.' The responder model (gpt-4o-mini) is a fixed external model, not a construct of the authors. The paper does not invoke any self-cited uniqueness theorem, ansatz, or prior result by the same authors as load-bearing for its central claims. The concern about LLM-mediated responder matching noise (raised in Section 8 and by the skeptic) is a validity/correctness risk, not a circularity issue — it concerns whether the measurements are accurate, not whether the conclusion is forced by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The benchmark introduces no new mathematical entities, physical objects, or postulated constructs. All mathematical families (Bayes' rule, CRT, Markov chains, etc.) are standard. The 'constrained information-holder' is an LLM with a structured-output prompt, not a new architectural component. The free parameters are design choices for benchmark construction, not fitted scientific constants.

free parameters (4)
  • Instances per family (Type A: 20, Type B: 50) = 20/50 per difficulty level
    Chosen to balance coverage and cost; not derived from a power analysis or statistical criterion.
  • Request budget B = 3d = 3/6/9 for difficulty 1/2/3
    Scales linearly with difficulty; the specific multiplier (3) is a design choice without stated justification.
  • Responder model = gpt-4o-mini
    Fixed across all experiments; chosen for cost and availability, not derived from a selection criterion.
  • Four-shot prompt examples = Not specified in detail
    The paper states examples cover 'valid minimal requests and final-answer formatting' but does not describe which families are exemplified, which affects the prompt-anchoring results.
assumptions (4)
  • domain assumption Each mathematical instance has a unique answer given the full latent state (global well-posedness, Eq. 1).
    Invoked in Section 3.1; enforced by family-specific generators via rank/uniqueness tests. Reasonable for synthetic math families.
  • domain assumption The solver's private view alone is underdetermined (local underdetermination, Eq. 2).
    Invoked in Section 3.1; enforced by construction. Reasonable given the generator design.
  • ad hoc to paper The fixed LLM responder (gpt-4o-mini) provides a sufficiently stable semantic matching channel that request metrics reflect solver ability rather than responder noise.
    Invoked implicitly throughout Section 5-6; the paper acknowledges this is not deterministic (Section 3.2, Section 8) but does not validate it empirically. No responder reliability measurement or gold-standard comparison is provided.
  • domain assumption The 22 mathematical families are representative of the target capability (minimal information requesting in mathematical reasoning).
    Invoked in Section 4.1; the families span algebra, probability, linear systems, etc. Coverage is broad but not derived from a systematic taxonomy of mathematical reasoning tasks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MIRA-Math: A Benchmark for Minimal Information Requesting and Mathematical Reasoning." pith.science (2026). https://pith.science/paper/DSQXG63K

@misc{pith2026260707391,
  author       = {Pith},
  title        = {Pith review of: MIRA-Math: A Benchmark for Minimal Information Requesting and Mathematical Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DSQXG63K}},
  note         = {Machine review of arXiv:2607.07391}
}
read the original abstract

Mathematical reasoning benchmarks typically provide all facts needed to solve each problem, while interactive benchmarks often mix reasoning with tools, retrieval, and long-horizon dialogue. We introduce MIRA-Math, a benchmark for a narrower diagnostic capability: solving mathematical problems whose full latent state has a unique answer, but whose solver-facing view is missing exactly one necessary atomic fact. The solver must request the missing information in natural language under a strict budget and then integrate the returned fact into an exact final answer. A fixed constrained LLM responder sees only the dataset-provided atomic fact and must either offer the quoted fact when the request matches it, or decline otherwise. Thus, instance generation, typed hint specifications, validation, and final-answer verification are deterministic, while request metrics are measured under a fixed LLM-mediated responder channel. MIRA-Math contains 2{,}310 generated instances from 22 typed mathematical families spanning algebra, probability, linear systems, discrete structures, signal processing, Markov chains, circuits, interpolation, and numerical boundary-value problems. Experiments across frontier and small models show that request success and final-answer accuracy are separable: models may ask for the right fact yet fail the downstream computation, or fail before obtaining the canonical hint. We release generators, verifiers, prompts, run metadata, and dataset documentation to support reproducible evaluation of minimal information requesting in mathematical reasoning.

Figures

Figures reproduced from arXiv: 2607.07391 by the authors.

Figure 1
Figure 1. Overview of the MIRA-Math protocol, illustrated with a successful circuit missing resistance trace. Agent A receives a private view with one atomic fact hidden, asks for the missing slot in natural language, and receives ei￾ther a structured offer or decline from a fixed constrained information-holder that sees only its private constraint. After receiving the hint, Agent A must still compute the final answer and pas… view at source ↗
Figure 2
Figure 2. Overall final-answer accuracy by solver model and prompting regime. Bars are grouped by solver model, with separate zero-shot (ZS) and four-shot (4S) results. The full numeric counterpart, including request hit rate, first-request success, average requests, declinations, and token estimates, is retained in Appendix B.1. Request acquisition versus mathematical integration. The transcript decompo￾sition confirms that … view at source ↗
Figure 3
Figure 3. Type A versus Type B final-answer accuracy by solver model and prompting regime. Type A families have a fixed missing-information slot, whereas Type B families require instance-level localization of the missing slot. The full numeric tables for Type A/Type B accuracy and request quality are retained in Ap￾pendix B.1. Several families make this distinction visible. In fixed-slot families such as bayes missing prior, … view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Overall final-answer accuracy by difficulty level, solver model, and prompting regime. Bars are grouped by solver model and split by difficulty level D1–D3 for zero-shot and four-shot prompting. The full numeric difficulty tables, including average request counts and t…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

32 extracted references · 32 canonical work pages

  1. [1]

    Journal of Data-centric Machine Learning Research , year=

    Oala, Luis and Maskey, Manil and Bat-Leah, Lilith and Parrish, Alicia and G. Journal of Data-centric Machine Learning Research , year=

    work page

  2. [2]

    Journal of Data-centric Machine Learning Research , year=

    Building Better Datasets: Seven Recommendations for Responsible Design from Dataset Creators , author=. Journal of Data-centric Machine Learning Research , year=

    work page

  3. [3]

    Journal of Data-centric Machine Learning Research , year=

    Rethinking Symbolic Regression Datasets and Benchmarks for Scientific Discovery , author=. Journal of Data-centric Machine Learning Research , year=

    work page

  4. [4]

    Journal of Data-centric Machine Learning Research , year=

    LabelBench: A Comprehensive Framework for Benchmarking Adaptive Label-Efficient Learning , author=. Journal of Data-centric Machine Learning Research , year=

    work page

  5. [5]

    2024 , url=

    Zhang, Jingyang and Yang, Jingkang and Wang, Pengyun and Wang, Haoqi and Lin, Yueqian and Zhang, Haoran and Sun, Yiyou and Du, Xuefeng and Li, Yixuan and Liu, Ziwei and Chen, Yiran and Li, Hai , journal=. 2024 , url=

    work page 2024

  6. [6]

    Journal of Data-centric Machine Learning Research , year=

    TopoBench: A Framework for Benchmarking Topological Deep Learning , author=. Journal of Data-centric Machine Learning Research , year=

    work page

  7. [7]

    Bsharat, Sondos Mahmoud and Ranjan, Mukul and Myrzakhan, Aidar and Liu, Jiacheng and Guo, Bowei and Tang, Shengkun and Liu, Zhuang and Li, Yuanzhi and Shen, Zhiqiang , journal=. Mobile-. 2026 , url=

    work page 2026

  8. [8]

    2026 , url=

    Somasekharan, Nithin and Yue, Ling and Cao, Yadi and LI, Weichao and Emami, Patrick and Bhargav, Pochinapeddi Sai and Acharya, Anurag and Xie, Xingyu and Pan, Shaowu , journal=. 2026 , url=

    work page 2026

Show all 32 references
  1. [9]

    Communications of the ACM , volume=

    Datasheets for Datasets , author=. Communications of the ACM , volume=. 2021 , publisher=

    2021

  2. [10]

    Proceedings of the Conference on Fairness, Accountability, and Transparency , pages=

    Model Cards for Model Reporting , author=. Proceedings of the Conference on Fairness, Accountability, and Transparency , pages=. 2019 , doi=

    2019

  3. [11]

    arXiv preprint arXiv:2311.12983 , year=

    Mialon, Gr. arXiv preprint arXiv:2311.12983 , year=

    arXiv

  4. [12]

    arXiv preprint arXiv:2308.03688 , year=

    AgentBench: Evaluating LLMs as Agents , author=. arXiv preprint arXiv:2308.03688 , year=

    arXiv

  5. [13]

    arXiv preprint arXiv:2307.16789 , year=

    ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs , author=. arXiv preprint arXiv:2307.16789 , year=

    arXiv

  6. [14]

    arXiv preprint arXiv:2110.14168 , year=

    Training Verifiers to Solve Math Word Problems , author=. arXiv preprint arXiv:2110.14168 , year=

    arXiv

  7. [15]

    Measuring Mathematical Problem Solving with the

    Hendrycks, Dan and Burns, Collin and Kadavath, Saurav and Arora, Akul and Basart, Steven and Tang, Eric and Song, Dawn and Steinhardt, Jacob , journal=. Measuring Mathematical Problem Solving with the. 2021 , doi=

    2021

  8. [16]

    arXiv preprint arXiv:2505.23281 , year=

    MathArena: Evaluating LLMs on Uncontaminated Math Competitions , author=. arXiv preprint arXiv:2505.23281 , year=

    arXiv

  9. [17]

    arXiv preprint arXiv:2503.22674 , year=

    QuestBench: Can LLMs Ask the Right Question to Acquire Information in Reasoning Tasks? , author=. arXiv preprint arXiv:2503.22674 , year=

  10. [18]

    2024 , doi=

    Gan, Yujian and Li, Changling and Xie, Jinxia and Wen, Luou and Purver, Matthew and Poesio, Massimo , journal=. 2024 , doi=

    2024

  11. [19]

    Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , pages=

    AmbigQA: Answering Ambiguous Open-domain Questions , author=. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , pages=. 2020 , publisher=. doi:10.18653/v1/2020.emnlp-main.466 , url=

    2020 doi

  12. [20]

    arXiv preprint arXiv:1605.06676 , year=

    Learning to Communicate with Deep Multi-Agent Reinforcement Learning , author=. arXiv preprint arXiv:1605.06676 , year=

    arXiv

  13. [21]

    arXiv preprint arXiv:1605.07736 , year=

    Learning Multiagent Communication with Backpropagation , author=. arXiv preprint arXiv:1605.07736 , year=

    arXiv

  14. [22]

    Proceedings of the 36th International Conference on Machine Learning , pages=

    Das, Abhishek and Gervet, Th. Proceedings of the 36th International Conference on Machine Learning , pages=. 2019 , volume=

    2019

  15. [23]

    Proceedings of the 37th International Conference on Machine Learning , pages=

    Learning Efficient Multi-agent Communication: An Information Bottleneck Approach , author=. Proceedings of the 37th International Conference on Machine Learning , pages=. 2020 , volume=

    2020

  16. [24]

    arXiv preprint arXiv:2308.08155 , year=

    AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation , author=. arXiv preprint arXiv:2308.08155 , year=

    arXiv

  17. [25]

    2023 , doi=

    Li, Guohao and Hammoud, Hasan Abed Al Kader and Itani, Hani and Khizbullin, Dmitrii and Ghanem, Bernard , journal=. 2023 , doi=

    2023

  18. [26]

    arXiv preprint arXiv:2308.10848 , year=

    AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors , author=. arXiv preprint arXiv:2308.10848 , year=

    arXiv

  19. [27]

    International Conference on Learning Representations , year=

    Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systems , author=. International Conference on Learning Representations , year=

  20. [28]

    arXiv preprint arXiv:2508.11252 , year=

    Beyond Solving Math Quiz: Evaluating the Ability of Large Reasoning Models to Ask for Information in Incomplete Mathematical Problems , author=. arXiv preprint arXiv:2508.11252 , year=

    arXiv

  21. [29]

    Proceedings of the International Conference on Machine Learning , year=

    From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information? , author=. Proceedings of the International Conference on Machine Learning , year=

  22. [30]

    arXiv preprint arXiv:2602.11199 , year=

    When and What to Ask: AskBench and Rubric-Guided RLVR for LLM Clarification , author=. arXiv preprint arXiv:2602.11199 , year=

    arXiv

  23. [31]

    Proceedings of the International Conference on Learning Representations , year=

    Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questions , author=. Proceedings of the International Conference on Learning Representations , year=

  24. [32]

    arXiv preprint arXiv:2307.13854 , year=

    WebArena: A Realistic Web Environment for Building Autonomous Agents , author=. arXiv preprint arXiv:2307.13854 , year=

    arXiv

Pith tools

Reviewed July 9, 2026 · model on record in the stance chip above.