Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

MM-Agent: LLM as Agents for Real-world Mathematical Modeling Problem

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An autonomous LLM agent, MM-Agent, can take an open-ended real-world mathematical modeling problem from raw description to finished, competition-grade report, outperforming award-winning human teams by 11.88% on the authors' MM-Bench.

desk verdict Useful benchmark and agent framework, but the 'beats human experts' number rests on a same-family LLM judge and won't survive scrutiny until re-scored independently. read the letter →

arxiv 2505.14148 v1 pith:ZO4S5FHQ submitted 2025-05-20 cs.AI

Fan Liu , Zherui Yang , Cancheng Liu , Tianrui Song , Xiaofeng Gao , Hao Liu This is my paper
classification cs.AI
keywords LLMagentsmathematicalmodelingMM-BenchhierarchicallibraryMCM/ICMcontestsactor-criticoptimizationopen-endedproblemsolvingbenchmarkevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Mathematical modeling, turning an open-ended real-world situation into assumptions, equations, code, and a written analysis, has resisted automation because the problem formulation itself is open. The paper argues that an LLM-based agent can do this end to end by decomposing the work into four stages: problem analysis, model formulation, computational solving, and report generation. To support the claim, the authors build MM-Bench, a benchmark of 111 problems drawn from the MCM/ICM competitions (2000 to 2025), and evaluate MM-Agent against existing LLM agents and award-winning human solutions. On a 32-problem test set, MM-Agent scores above the human benchmark by 11.88% overall while using roughly $0.88 of API cost and about 15 minutes per task on GPT-4o. If the results hold, an autonomous agent can perform real-world mathematical modeling at expert-competition level, shifting the bottleneck from constructing models to reliably evaluating open-ended work.

What carries the argument

The central object is the four-stage workflow plus the Hierarchical Mathematical Modeling Library (HMML). HMML is a three-level tree of modeling domains (such as operations research and optimization), subdomains (such as programming theory), and method nodes that each store a method name, its core idea, and typical applications. At retrieval time a depth-first traversal scores each node by the embedding similarity between the subtask and the method, blended with its parent's score, and returns the top-K methods; an actor-critic loop then iteratively proposes, critiques, and revises the modeling scheme. The task coordinator additionally builds a directed dependency graph of subtasks and a memory that passes intermediate models, code, and results between stages. This machinery is what lets the agent abstract an unstructured scenario into a formal model rather than merely solving a pre-formulated problem.

What would settle it

Take the 32-problem MM-Bench test set, remove authorship labels from human and MM-Agent reports, and have a fresh panel of at least three contest-experienced judges score them on the four rubric dimensions. If the blinded overall margin is no longer positive and statistically significant, especially on Modeling Rigorousness where the paper reports human-human agreement of only 0.4813, then the claim of beating award-winning human teams is not supported.

Watch

Extended reading notes

Core claim

The central claim is that open-ended mathematical modeling can be reduced to a four-stage expert-inspired pipeline that an LLM can execute autonomously. In the authors' telling, the decisive ingredient is not a bigger model but structure: the agent first analyzes the problem and decomposes it into a dependency graph of subtasks; it then retrieves candidate modeling methods from a hierarchical library (HMML) of 98 schemas organized into domains, subdomains, and method nodes; an actor-critic loop refines the modeling scheme; a code-writing module generates and debugs the computation; and a reporting module compiles a structured LaTeX report. On MM-Bench the pipeline outperforms repurposed data-science and research agents and award-winning human teams on all four evaluation dimensions, with 2021–2024 and 2025 results consistent across two backbones, GPT-4o and DeepSeek-R1-671B. The same system, operated under official contest rules as a copilot, helped two undergraduate teams reach the Finalist tier, the top 2.0% of 27,456 teams, in MCM/ICM 2025.

Load-bearing premise

The paper's headline result, that an agent beats expert humans, rests on subjective scores from a four-criteria rubric assigned by GPT-4o and a small panel of contest-experienced humans, not on verifiable ground truth; if those judges are systematically friendlier to agent-produced reports, the reported advantage shrinks or disappears.

Editorial extensions

If this is right

  • MM-Agent produces complete competition-grade modeling reports in roughly 15 minutes and about $0.88 per task on GPT-4o, so the per-problem cost of expert-level modeling drops below one dollar.
  • Performance is similar on problems from 2021–2024 and from 2025, which the authors read as evidence that the results come from genuine modeling rather than memorized contest solutions.
  • Acting as a copilot under official MCM/ICM rules, MM-Agent helped two undergraduate teams reach Finalist, the top 2.0% of 27,456 teams, in MCM/ICM 2025.
  • On well-defined OPTIBENCH optimization problems, MM-Agent also beats GPT-4o in zero-shot settings and raises code pass rate to 99.3%, indicating the pipeline generalizes beyond open-ended modeling.
  • Stronger backbones help: MM-Agent on DeepSeek-R1-671B posts overall scores of 8.85 on 2021–2024 and 8.92 on 2025, above its GPT-4o scores, suggesting the architecture composes with model improvements.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the rubric is subjective, with human-human agreement as low as 0.4813 on Modeling Rigorousness, a natural next step is a double-blind study in which fresh contest-experienced judges score human and MM-Agent reports without knowing the author; the size of the reported 11.88% gain under those conditions would reveal how much is modeling quality versus style.
  • The library-retrieval design is not tied to competition problems; the same abstraction-aware retrieval could be tested on other open-ended engineering or policy tasks where the hard step is choosing a modeling paradigm.
  • If the four-stage structure matters mostly by compensating for weak base-model reasoning, then as LLMs improve the gap between a raw LLM and MM-Agent should narrow; tracking that gap across model generations would test the architecture's specific contribution.
  • MM-Bench should be refreshed periodically with new contest problems, as the paper itself recommends, so that public competition data does not eventually contaminate LLM pretraining; a public annual update would preserve the benchmark's value for measuring progress.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. This paper formalizes LLM-based real-world mathematical modeling as an agentic task, introduces MM-Bench (111 MCM/ICM problems from 2000 to 2025 across 10 domains), and proposes MM-Agent, a four-stage pipeline (problem analysis, mathematical modeling with hierarchical retrieval and actor-critic refinement, computational solving, and report generation) built around a tri-level Hierarchical Mathematical Modeling Library (HMML). On a 32-problem subset of MM-Bench, the paper reports that MM-Agent outperforms baseline agents and award-winning human solutions by 11.88% in overall score under a GPT-4o-based rubric, at roughly 15 minutes and $0.88 per task with GPT-4o, with similar results under DeepSeek-R1. The system also assisted two undergraduate teams in achieving the Finalist Award (top 2.0%) in MCM/ICM 2025. Additional experiments include ablations, cost and runtime analysis, human expert evaluation, and zero-shot results on the ground-truth OPTIBENCH dataset for well-posed optimization problems.

Significance. If the claims held up, this would be a notable advance: the first systematic benchmark for open-ended LLM mathematical modeling, a reusable agent framework with a structured modeling library, and evidence that an autonomous pipeline can approach competition-level modeling. The public code and demo, the use of real competition problems with manual verification of extracted elements, and the ground-truth OPTIBENCH experiments are concrete strengths. However, the headline result, that MM-Agent significantly outperforms award-winning human teams, depends entirely on a same-family LLM judge: GPT-4o generates the agent's reports and GPT-4o scores them. The paper's own Appendix A concedes judge-bias risk, and Appendix E.4 reports model-human agreement as low as 0.5068 on AE and 0.5692 on RBA, with human-human agreement of only 0.4813 on MR. The human evaluation in Appendix E.3 does not even include the human-team condition. The central claim therefore needs independent, blinded human rescoring and cross-family judge validation before it can be credited; the current evidence supports a more modest claim of strong performance on the proposed benchmark.

major comments (5)
  1. [§4.1, Table 1, Appendix D, Figure 12] The 11.88% improvement over human experts is computed from scores assigned by GPT-4o, the same model family that generates MM-Agent's solutions. The Practicality and Scientificity prompt (Figure 12, criteria 3.1/3.2) explicitly rewards 'innovation', 'goes beyond standard machine learning', and 'novel frameworks', so a systematic stylistic self-preference would directly inflate the reported gap. The paper's own Appendix A states that 'bias still exists in both human and LLM annotators', and Appendix E.4 reports model-human agreement of 0.5068 (AE) and 0.5692 (RBA). Given that the overall gap is only 8.85 vs. 7.91 on a 10-point scale, a judge bias of about 0.5 points would erase the claimed advantage. The manuscript must provide counter-evidence: blinded rescoring of both agent and human reports by contest-experienced human judges, scores from at least one different LLM family, and per-criterion confidence intervals or significance tests.
  2. [§4.1] The construction of the 32-problem test set is underspecified: the text only says the authors 'select a subset' from the past five years 'ensuring diversity across problem types and domains', without a sampling protocol, a list of selected problems, or stratification details. This undermines reproducibility and weakens the claimed 'temporal consistency' between the 2021-2024 and 2025 splits. In addition, the 'Human Team' baseline scores are not official MCM/ICM scores but re-scores by the same GPT-4o judge; reporting the official award level for each selected problem would provide an independent calibration anchor.
  3. [Appendix E.3, Figure 6] The human expert evaluation covers only Agent Laboratory, DS-Agent, ResearchAgent, and MM-Agent; it does not include human team solutions. Consequently, the statement that MM-Agent 'significantly outperforms human experts' is not corroborated by human evaluation in any direct way. Either include award-winning human reports in the human evaluation, or restrict the human-outperformance claim to the LLM-judged comparison and clearly say so.
  4. [§4.3, Figure 4, Table 6] Ablation results are presented only as line charts without numerical values, sample sizes, or error bars. With 32 problems and inter-annotator agreement as low as 0.4813 on MR (Table 6), the observed differences between MM-Agent and its ablated variants cannot be distinguished from annotator noise. Report the actual scores, per-condition standard deviations, and paired significance tests (e.g., bootstrap or Wilcoxon signed-rank) for the ablations and for the main comparisons in Table 1.
  5. [Abstract, §4.2] The Finalist Award result is presented as evidence of practical effectiveness, but MM-Agent acted as a copilot assisting two human teams, not as an autonomous agent. This supports a human-AI collaboration claim, not the autonomous outperformance claim made in the abstract. Please separate these claims explicitly, and if possible report what the teams achieved without the aid of MM-Agent, or state that no controlled comparison was performed.
minor comments (6)
  1. [§3.3.1] The heading 'Problem Undersanding' contains a typo and should be 'Problem Understanding'.
  2. [Appendix D] The heading 'Evalaution' contains a typo and should be 'Evaluation'.
  3. [Figure 4] The axis labels contain uninterpretable glyph strings such as '/uni00000024/uni00000028/...'; the figure needs proper axis and legend labels.
  4. [§4.3, Appendix D] The hyperparameters w, K, n_r, n_c, and tasknum are introduced in §3.3.2 but their experimental values are never reported; please provide them for reproducibility.
  5. [§2] The claim that this is the first work on LLMs for real-world mathematical modeling should be qualified in light of the cited optimization-modeling papers (NL4OPT, OPTIMUS, OptiBench), which address a related but narrower formulation; a brief comparison would clarify the novelty.
  6. [Appendix E.4] The text says 'We observe consistently high agreement in the four metrics', but Table 6 reports human-human agreement of only 0.4813 for MR; this apparent contradiction should be resolved.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found; the main empirical claims rest on benchmark evaluation, and the same-family LLM judge issue is a validity limitation, not a circular construction.

full rationale

The paper makes no formal derivation chain; its central claims are empirical evaluations of an agent system on a curated benchmark. The headline 11.88% improvement over human expert solutions is an arithmetic comparison of rubric scores reported in Table 1, not a parameter fitted to the data that is then renamed as a prediction. The rubric scores are produced by GPT-4o-based automatic scoring and human expert review; MM-Agent uses GPT-4o as a backbone, and Appendix A explicitly concedes that 'bias still exists in both human and LLM annotators, particularly due to the inherent subjectivity in evaluating different modeling solutions.' That is a genuine threat to external validity, but it is not circularity in the sense required here: the score is not defined in terms of the agent's output, no equation reduces to another by construction, and the result is not imported from a self-citation. The HMML is a manually structured taxonomy of standard modeling methods, used for retrieval and evaluated by ablation, so it is not a renamed version of the target result. MLE-Solver, mGTE, and other cited tools are external resources, and no load-bearing claim depends on a citation by the present authors. Accordingly, under the strict reduction standard, no circular step is identifiable.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central claim rests on three domain assumptions about benchmark validity and judge impartiality, plus five hand-set design hyperparameters. No new physical or mathematical entities are introduced.

free parameters (5)
  • omega (retrieval weight)
    Hyperparameter balancing node similarity and parent similarity in HMML retrieval, used in S(D,N)=omega*Sim(D,N)+(1-omega)*Sim(D,N_parent). Chosen by hand, no sensitivity analysis reported.
  • K (top-K retrieved methods)
    Number of method nodes returned by HMML retrieval for each subtask. Set by hand without ablation.
  • n_r (max actor-critic iterations)
    Maximum refinement rounds in hierarchical actor-critic modeling. Chosen by hand.
  • n_c (max code repair iterations)
    Maximum debugging loop iterations in MLE-Solver. Chosen by hand.
  • tasknum (number of subtasks)
    Number of decompositions for each problem. Determined by prompt, not systematically tuned or reported per problem.
assumptions (3)
  • domain assumption MCM/ICM contest problems and award-winning reports are valid representatives of real-world mathematical modeling tasks and expert solutions.
    The benchmark and human baseline are built entirely from MCM/ICM. If contest performance is not a good proxy, the claims about real-world modeling weaken. Invoked throughout Sections 3.1 and 4.1.
  • domain assumption The four-criteria rubric (AE, MR, PS, RBA) operationalized by GPT-4o and a small human panel measures modeling quality.
    The rubric is subjective, human-human agreement is as low as 0.4813 on MR in Table 6, and no calibration against official judges is reported. Sections 3.1 and Appendix E.4.
  • domain assumption LLM evaluator scores are not biased by the generating model family or report formatting.
    GPT-4o evaluates reports produced by GPT-4o-based agents, and self-preference is not controlled. Section 4.1 and Appendix A.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MM-Agent: LLM as Agents for Real-world Mathematical Modeling Problem." pith.science (2026). https://pith.science/paper/ZO4S5FHQ

@misc{pith2026250514148,
  author       = {Pith},
  title        = {Pith review of: MM-Agent: LLM as Agents for Real-world Mathematical Modeling Problem},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZO4S5FHQ}},
  note         = {Machine review of arXiv:2505.14148}
}
abstract

Mathematical modeling is a cornerstone of scientific discovery and engineering practice, enabling the translation of real-world problems into formal systems across domains such as physics, biology, and economics. Unlike mathematical reasoning, which assumes a predefined formulation, modeling requires open-ended problem analysis, abstraction, and principled formalization. While Large Language Models (LLMs) have shown strong reasoning capabilities, they fall short in rigorous model construction, limiting their utility in real-world problem-solving. To this end, we formalize the task of LLM-powered real-world mathematical modeling, where agents must analyze problems, construct domain-appropriate formulations, and generate complete end-to-end solutions. We introduce MM-Bench, a curated benchmark of 111 problems from the Mathematical Contest in Modeling (MCM/ICM), spanning the years 2000 to 2025 and across ten diverse domains such as physics, biology, and economics. To tackle this task, we propose MM-Agent, an expert-inspired framework that decomposes mathematical modeling into four stages: open-ended problem analysis, structured model formulation, computational problem solving, and report generation. Experiments on MM-Bench show that MM-Agent significantly outperforms baseline agents, achieving an 11.88\% improvement over human expert solutions while requiring only 15 minutes and \$0.88 per task using GPT-4o. Furthermore, under official MCM/ICM protocols, MM-Agent assisted two undergraduate teams in winning the Finalist Award (\textbf{top 2.0\% among 27,456 teams}) in MCM/ICM 2025, demonstrating its practical effectiveness as a modeling copilot. Our code is available at https://github.com/usail-hkust/LLM-MM-Agent

Figures

Figures reproduced from arXiv: 2505.14148 by the authors.

Figure 1
Figure 1. Traditional well-defined mathematics problem vs LLM-powered open-ended mathematical [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The structure of HMML is organized in three levels: [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Overview of the MM-Agent framework. The workflow consists of four sequential phases: [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (32 more)
Figure 4
Figure 4. Figure 4: Ablation study of the effect of the problem analysis [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Illustrations of problem domain and types. [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: Human Evaluation Results. The experiments evaluated by human experts are summarized in [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]
Figure 7
Figure 7. Figure 7: The workflow of the problem analysis phase in MM-Agent. Mathematical modeling [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]
Figure 8
Figure 8. Figure 8: The workflow of the mathematical modeling phase and computational solving and solution [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 1
Figure 1. Figure 1: task1 [PITH_FULL_IMAGE:figures/full_fig_p023_1.png]
Figure 10
Figure 10. Figure 10: The prompt used for evaluating Analysis of Agent. [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]
Figure 12
Figure 12. Figure 12: The prompt used for evaluating Practicality and Scientificity of Agent. [PITH_FULL_IMAGE:figures/full_fig_p027_12.png]
Figure 13
Figure 13. Figure 13: The prompt used for evaluating Result and Bias Analysis of Agent. [PITH_FULL_IMAGE:figures/full_fig_p028_13.png]
Figure 14
Figure 14. Figure 14: The prompt used to describe the dataset. [PITH_FULL_IMAGE:figures/full_fig_p029_14.png]
Figure 15
Figure 15. Figure 15: The Problem Template prompt. Problem Understanding Prompt # Mathematical Modeling Problem: {modeling_problem} — You are tasked with analyzing a mathematical modeling problem with a focus on the underlying concepts, logical reasoning, and assumptions that inform the so…
Figure 16
Figure 16. Figure 16: The prompt used in the Problem Understanding step. It guides the agent to perform a [PITH_FULL_IMAGE:figures/full_fig_p030_16.png]
Figure 17
Figure 17. Figure 17: The prompt used for criticizing problem analysis in the Problem Understanding step. [PITH_FULL_IMAGE:figures/full_fig_p031_17.png]
Figure 18
Figure 18. Figure 18: The prompt used for improving problem analysis in the Problem Understanding step. [PITH_FULL_IMAGE:figures/full_fig_p031_18.png]
Figure 19
Figure 19. Figure 19: The prompt used in the Problem Decomposition step. It instructs the agent to transform a [PITH_FULL_IMAGE:figures/full_fig_p032_19.png]
Figure 20
Figure 20. Figure 20: The prompt used for refining and improving task descriptions. It guides the agent to [PITH_FULL_IMAGE:figures/full_fig_p032_20.png]
Figure 21
Figure 21. Figure 21: The prompt used in the Task Dependency Analysis step. It guides the agent to identify and [PITH_FULL_IMAGE:figures/full_fig_p033_21.png]
Figure 22
Figure 22. Figure 22: The prompt used for constructing the Task Dependency Graph. It instructs the agent [PITH_FULL_IMAGE:figures/full_fig_p034_22.png]
Figure 23
Figure 23. Figure 23: The prompt used for constructing the model formulas. It instructs the agent to derive [PITH_FULL_IMAGE:figures/full_fig_p035_23.png]
Figure 24
Figure 24. Figure 24: The prompt used for criticizing the model formulas. It guides the agent to identify [PITH_FULL_IMAGE:figures/full_fig_p036_24.png]
Figure 25
Figure 25. Figure 25: The prompt used for improving the model formulas. [PITH_FULL_IMAGE:figures/full_fig_p037_25.png]
Figure 26
Figure 26. Figure 26: The prompt used for improving the model formulas. It instructs the agent to revise existing [PITH_FULL_IMAGE:figures/full_fig_p038_26.png]
Figure 27
Figure 27. Figure 27: The prompt used for generating code. It instructs the agent to produce fully executable [PITH_FULL_IMAGE:figures/full_fig_p039_27.png]
Figure 28
Figure 28. Figure 28: The prompt used for debugging code. It instructs the agent to identify and fix execution [PITH_FULL_IMAGE:figures/full_fig_p039_28.png]
Figure 29
Figure 29. Figure 29: The prompt used for extracting the structure of code. [PITH_FULL_IMAGE:figures/full_fig_p041_29.png]
Figure 30
Figure 30. Figure 30: The prompt used for extracting the structure of code. It guides the agent to parse and [PITH_FULL_IMAGE:figures/full_fig_p041_30.png]
Figure 31
Figure 31. Figure 31: The prompt used for formulating the solution. It guides the agent to synthesize modeling [PITH_FULL_IMAGE:figures/full_fig_p043_31.png]
Figure 32
Figure 32. Figure 32: The prompt used for generating chart creation guidelines. It instructs the agent to [PITH_FULL_IMAGE:figures/full_fig_p044_32.png]
Figure 33
Figure 33. Figure 33: The prompt used for creating a paper chapter. It guides the agent to transform a structured [PITH_FULL_IMAGE:figures/full_fig_p045_33.png]
Figure 34
Figure 34. Figure 34: The prompt used for creating a paper chapter with preceding content. It guides the agent [PITH_FULL_IMAGE:figures/full_fig_p045_34.png]
Figure 35
Figure 35. Figure 35: The prompt used for creating paper notations. It instructs the agent to extract and format [PITH_FULL_IMAGE:figures/full_fig_p046_35.png]
Figure 36
Figure 36. Figure 36: The prompt used for creating paper meta information. It instructs the agent to generate a [PITH_FULL_IMAGE:figures/full_fig_p047_36.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Retrieve, Schedule, Reflect: LLM Agents for Chip QoR Optimization

    cs.AR 2026-03 conditional novelty 6.0 of 10

    An LLM scheduler using tree-structured retrieval and Pareto-based reflection automates chip QoR optimization, matching human experts and beating an RL baseline on eight designs.

Reference graph

Works this paper leans on

90 extracted references · 50 canonical work pages · cited by 1 Pith paper

  1. [1]

    Courier Corporation, 2000

    Edward A Bender.An introduction to mathematical modeling. Courier Corporation, 2000

    2000

  2. [2]

    Academic press, 2013

    Mark Meerschaert.Mathematical modeling. Academic press, 2013

    2013

  3. [3]

    Trinh, Yuhuai Wu, Quoc V

    Trieu H. Trinh, Yuhuai Wu, Quoc V . Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations.Nat., 625(7995):476–482, 2024

    2024

  4. [4]

    Optibench meets resocratic: Measure and improve LLMs for optimization modeling

    Zhicheng Yang, Yiwei Wang, Yinya Huang, Zhijiang Guo, Wei Shi, Xiongwei Han, Liang Feng, Linqi Song, Xiaodan Liang, and Jing Tang. Optibench meets resocratic: Measure and improve LLMs for optimization modeling. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  5. [5]

    Paperbench: Evaluating ai’s ability to replicate ai research.arXiv preprint arXiv:2504.01848, 2025

    Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al. Paperbench: Evaluating ai’s ability to replicate ai research.arXiv preprint arXiv:2504.01848, 2025

    arXiv 2025

  6. [6]

    Gpt-4o system card, 2024

    OpenAI. Gpt-4o system card, 2024

    2024

  7. [7]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai D...

    arXiv 2025

  8. [8]

    Narasimhan, and Yuan Cao

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InICLR, 2023

    2023

Show all 90 references
  1. [9]

    Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors

    Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, Yujia Qin, Xin Cong, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Agentverse: Facilitating multi-agent collaboration and exploring emergent behavio...

    2024

  2. [10]

    CAMEL: communica- tive agents for "mind" exploration of large language model society

    Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. CAMEL: communica- tive agents for "mind" exploration of large language model society. InNeurIPS, 2023

    2023

  3. [11]

    Chatdev: Communicative agents for software development

    Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. Chatdev: Communicative agents for software development. InACL (1), pages 15174–15186, 2024

    2024

  4. [12]

    Autogen: Enabling next-gen LLM applications via multi-agent conversation framework.CoRR, abs/2308.08155, 2023

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen LLM applications via multi-agent conversation framework.CoRR, abs/2308.08155, 2023

    2023 arXiv

  5. [13]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. Swe-bench: Can language models resolve real-world github issues? InICLR, 2024

    2024

  6. [14]

    Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brenna...

    2024 arXiv

  7. [15]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. InNeurIPS, 2024. 10

    2024

  8. [16]

    Large language model-based agents for software engineering: A survey.CoRR, abs/2409.02977, 2024

    Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, and Yiling Lou. Large language model-based agents for software engineering: A survey.CoRR, abs/2409.02977, 2024

    2024 arXiv

  9. [17]

    AL, Andrew Ahn, Nic Becker, Stephanie Carroll, Nico Christie, Manuel Cortes, Arda Demirci, Melissa Du, Frankie Li, Shuying Luo, Peter Y

    Altera. AL, Andrew Ahn, Nic Becker, Stephanie Carroll, Nico Christie, Manuel Cortes, Arda Demirci, Melissa Du, Frankie Li, Shuying Luo, Peter Y . Wang, Mathew Willows, Feitong Yang, and Guangyu Robert Yang. Project sid: Many-agent simulations toward AI civilization.CoRR, abs/2...

    2024 arXiv

  10. [18]

    Chessgpt: Bridging policy learning and language modeling

    Xidong Feng, Yicheng Luo, Ziyan Wang, Hongrui Tang, Mengyue Yang, Kun Shao, David Mguni, Yali Du, and Jun Wang. Chessgpt: Bridging policy learning and language modeling. InNeurIPS, 2023

    2023

  11. [19]

    V oyager: An open-ended embodied agent with large language models.Trans

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. V oyager: An open-ended embodied agent with large language models.Trans. Mach. Learn. Res., 2024, 2024

    2024

  12. [20]

    Villageragent: A graph-based multi-agent framework for coordinating complex task dependencies in minecraft

    Yubo Dong, Xukun Zhu, Zhengzhe Pan, Linchao Zhu, and Yi Yang. Villageragent: A graph-based multi-agent framework for coordinating complex task dependencies in minecraft. InACL (Findings), pages 16290–16314, 2024

    2024

  13. [21]

    Language agents with reinforcement learning for strategic play in the werewolf game

    Zelai Xu, Chao Yu, Fei Fang, Yu Wang, and Yi Wu. Language agents with reinforcement learning for strategic play in the werewolf game. InICML, 2024

    2024

  14. [22]

    Tekin, Gaowen Liu, Ramana Kompella, and Ling Liu

    Sihao Hu, Tiansheng Huang, Fatih Ilhan, Selim F. Tekin, Gaowen Liu, Ramana Kompella, and Ling Liu. A survey on large language model-based game agents.CoRR, abs/2404.02039, 2024

    2024 arXiv

  15. [23]

    O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S

    Joon Sung Park, Joseph C. O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. InUIST, pages 2:1–2:22, 2023

    2023

  16. [24]

    Zou, Aaron Shaw, Benjamin Mako Hill, Carrie J

    Joon Sung Park, Carolyn Q. Zou, Aaron Shaw, Benjamin Mako Hill, Carrie J. Cai, Meredith Ringel Morris, Robb Willer, Percy Liang, and Michael S. Bernstein. Generative agent simulations of 1,000 people.CoRR, abs/2411.10109, 2024

    2024 arXiv

  17. [25]

    Jimenez, Farshad Khorrami, Prashanth Krishnamurthy, Brendan Dolan- Gavitt, Muhammad Shafique, Karthik Narasimhan, Ramesh Karri, and Ofir Press

    Talor Abramovich, Meet Udeshi, Minghao Shao, Kilian Lieret, Haoran Xi, Kimberly Milner, Sofija Jancheska, John Yang, Carlos E. Jimenez, Farshad Khorrami, Prashanth Krishnamurthy, Brendan Dolan- Gavitt, Muhammad Shafique, Karthik Narasimhan, Ramesh Karri, and Ofir Press. Enigma...

    2024 arXiv

  18. [26]

    LLM agents can autonomously hack websites.CoRR, abs/2402.06664, 2024

    Richard Fang, Rohan Bindu, Akul Gupta, Qiusi Zhan, and Daniel Kang. LLM agents can autonomously hack websites.CoRR, abs/2402.06664, 2024

    2024 arXiv

  19. [27]

    CYBERSECEV AL 3: Advancing the evaluation of cybersecurity risks and capabilities in large language models.CoRR, abs/2408.01605, 2024

    Shengye Wan, Cyrus Nikolaidis, Daniel Song, David Molnar, James Crnkovich, Jayson Grace, Manish Bhatt, Sahana Chennabasappa, Spencer Whitman, Stephanie Ding, Vlad Ionescu, Yue Li, and Joshua Saxe. CYBERSECEV AL 3: Advancing the evaluation of cybersecurity risks and capabilitie...

    2024 arXiv

  20. [28]

    Joshi, Kyle Jeffrey, Rosario Jauregui Ruano, Jasmine Hsu, Keerthana Gopalakrishnan, Byron David, Andy Zeng, and Chuyuan Kelly Fu

    Brian Ichter, Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, Dmitry Kalashnikov, Sergey Levine, Yao Lu, Carolina Parada, Kanishka Rao, Pierre Sermanet, Alexander Toshev, Vincent Vanho...

    2022

  21. [29]

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, Jam...

    arXiv

  22. [30]

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J. Joshi, Ryan Julian, Dmitry Kalash...

    2023

  23. [31]

    Tran, Radu Soricut, Anikait Singh, Jaspiar Singh, Pierre Sermanet, Pannag R

    Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, Quan Vuong, Vincent Vanhoucke, Huong T. Tran, Radu Soricut, Anikait Singh, Jaspiar Singh, Pierre Sermanet, Pannag R. Sanketi, Grecia Salazar, Michael S. ...

    2023

  24. [32]

    Zhao, Samuel Schmidgall, Anton Deguet, Marin Kobilarov, Chelsea Finn, and Axel Krieger

    Ji Woong Kim, Tony Z. Zhao, Samuel Schmidgall, Anton Deguet, Marin Kobilarov, Chelsea Finn, and Axel Krieger. Surgical robot transformer (SRT): imitation learning for surgical tasks. InCoRL, volume 270, pages 130–144, 2024

    2024

  25. [33]

    Ds-agent: Automated data science by empowering large language models with case-based reasoning

    Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, and Jun Wang. Ds-agent: Automated data science by empowering large language models with case-based reasoning. InICML, 2024

    2024

  26. [34]

    Data interpreter: An LLM agent for data science.CoRR, abs/2402.18679, 2024

    Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Danyang Li, Jiaqi Chen, Jiayi Zhang, Jinlin Wang, Li Zhang, Lingyao Zhang, Min Yang, Mingchen Zhuge, Taicheng Guo, Tuo Zhou, Wei Tao, Wenyi Wang, Xiangru Tang, Xiangtao Lu, Xiawu Zheng, Xinbing Liang, Yaying Fei, Yuhe...

    2024 arXiv

  27. [35]

    Wang, Ruoxi Sun, Pengcheng Yin, Caiming Xiong, Ansong Ni, Qian Liu, Victor Zhong, Lu Chen, Kai Yu, and Tao Yu

    Ruisheng Cao, Fangyu Lei, Haoyuan Wu, Jixuan Chen, Yeqiao Fu, Hongcheng Gao, Xinzhuang Xiong, Hanchong Zhang, Wenjing Hu, Yuchen Mao, Tianbao Xie, Hongshen Xu, Danyang Zhang, Sida I. Wang, Ruoxi Sun, Pengcheng Yin, Caiming Xiong, Ansong Ni, Qian Liu, Victor Zhong, Lu Chen, Kai...

    2024

  28. [36]

    Mahadi Hassan, R

    Md. Mahadi Hassan, R. Alexander Knipper, and Shubhra Kanti Karmaker Santu. Chatgpt as your personal data scientist.CoRR, abs/2305.13657, 2023

    2023 arXiv

  29. [37]

    Sara Mahdavi, Sushant Prakash, Anupam Pathak, Christopher Semturs, Shwetak N

    Daniel McDuff, Mike Schaekermann, Tao Tu, Anil Palepu, Amy Wang, Jake Garrison, Karan Singhal, Yash Sharma, Shekoofeh Azizi, Kavita Kulkarni, Le Hou, Yong Cheng, Yun Liu, S. Sara Mahdavi, Sushant Prakash, Anupam Pathak, Christopher Semturs, Shwetak N. Patel, Dale R. Webster, E...

    2023 arXiv

  30. [38]

    Agentclinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments.CoRR, abs/2405.07960, 2024

    Samuel Schmidgall, Rojin Ziaei, Carl Harris, Eduardo Pontes Reis, Jeffrey Jopling, and Michael Moor. Agentclinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments.CoRR, abs/2405.07960, 2024

    2024 arXiv

  31. [39]

    Sara Mahdavi, Christopher Semturs, Juraj Gottweis, Joelle K

    Tao Tu, Anil Palepu, Mike Schaekermann, Khaled Saab, Jan Freyberg, Ryutaro Tanno, Amy Wang, Brenna Li, Mohamed Amin, Nenad Tomasev, Shekoofeh Azizi, Karan Singhal, Yong Cheng, Le Hou, Albert Webson, Kavita Kulkarni, S. Sara Mahdavi, Christopher Semturs, Juraj Gottweis, Joelle ...

    2024 arXiv

  32. [40]

    Llm-based agentic systems in medicine and healthcare.Nature Machine Intelligence, 6(12):1418–1420, 2024

    Jianing Qiu, Kyle Lam, Guohao Li, Amish Acharya, Tien Yin Wong, Ara Darzi, Wu Yuan, and Eric J Topol. Llm-based agentic systems in medicine and healthcare.Nature Machine Intelligence, 6(12):1418–1420, 2024

    2024

  33. [41]

    Mind2web: Towards a generalist agent for the web

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samual Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. InNeurIPS, 2023

    2023

  34. [42]

    Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust

    Izzeddin Gur, Hiroki Furuta, Austin V . Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and program synthesis. InICLR, 2024

    2024

  35. [43]

    Webvoyager: Building an end-to-end web agent with large multimodal models

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. InACL (1), pages 6864–6890, 2024

    2024

  36. [44]

    Agent Q: advanced reasoning and learning for autonomous AI agents.CoRR, abs/2408.07199, 2024

    Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent Q: advanced reasoning and learning for autonomous AI agents.CoRR, abs/2408.07199, 2024. 12

    2024 arXiv

  37. [45]

    The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search

    Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search. arXiv preprint arXiv:2504.08066, 2025

    2025 arXiv

  38. [46]

    Agent laboratory: Using LLM agents as research assistants.CoRR, abs/2501.04227, 2025

    Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Zicheng Liu, and Emad Barsoum. Agent laboratory: Using LLM agents as research assistants.CoRR, abs/2501.04227, 2025

    2025 arXiv

  39. [47]

    Foerster, Jeff Clune, and David Ha

    Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob N. Foerster, Jeff Clune, and David Ha. The AI scientist: Towards fully automated open-ended scientific discovery.CoRR, abs/2408.06292, 2024

    2024 arXiv

  40. [48]

    Researchagent: Iterative research idea generation over scientific literature with large language models.CoRR, abs/2404.07738, 2024

    Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. Researchagent: Iterative research idea generation over scientific literature with large language models.CoRR, abs/2404.07738, 2024

    2024 arXiv

  41. [49]

    Foerster, Yoram Bachrach, William Yang Wang, and Roberta Raileanu

    Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vincent Moens, Amar Budhiraja, Despoina Magka, Vladislav V orotilov, Gaurav Chaurasia, Dieuwke Hupkes, Ricardo Sil- veira Cabral, Tatiana Shavrina, Jakob N. Foerster, Yoram Bachrach, William Yang W...

    2025 arXiv

  42. [50]

    Benchmarking large language models as AI research agents.CoRR, abs/2310.03302, 2023

    Qian Huang, Jian V ora, Percy Liang, and Jure Leskovec. Benchmarking large language models as AI research agents.CoRR, abs/2310.03302, 2023

    2023 arXiv

  43. [51]

    Empowering biomedical discovery with ai agents

    Shanghua Gao, Ada Fang, Yepeng Huang, Valentina Giunchiglia, Ayush Noori, Jonathan Richard Schwarz, Yasha Ektefaie, Jovana Kondic, and Marinka Zitnik. Empowering biomedical discovery with ai agents. Cell, 187(22):6125–6151, 2024

    2024

  44. [52]

    Organa: a robotic assistant for automated chemistry experimentation and characterization.Matter, 8(2), 2025

    Kourosh Darvish, Marta Skreta, Yuchi Zhao, Naruki Yoshikawa, Sagnik Som, Miroslav Bogdanovic, Yang Cao, Han Hao, Haoping Xu, Alán Aspuru-Guzik, et al. Organa: a robotic assistant for automated chemistry experimentation and characterization.Matter, 8(2), 2025

    2025

  45. [53]

    Automating traffic model enhancement with AI research agent.CoRR, abs/2409.16876, 2024

    Xusen Guo, Xinxi Yang, Mingxing Peng, Hongliang Lu, Meixin Zhu, and Hai Yang. Automating traffic model enhancement with AI research agent.CoRR, abs/2409.16876, 2024

    2024

  46. [54]

    Measuring mathematical problem solving with the MATH dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. InNeurIPS Datasets and Benchmarks, 2021

    2021

  47. [56]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V . Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. InNeurIPS, 2022

    2022

  48. [57]

    minif2f: a cross-system benchmark for formal olympiad-level mathematics

    Kunhao Zheng, Jesse Michael Han, and Stanislas Polu. minif2f: a cross-system benchmark for formal olympiad-level mathematics. InICLR, 2022

    2022

  49. [58]

    Ayers, and Stanislas Polu

    Jesse Michael Han, Jason Rute, Yuhuai Wu, Edward W. Ayers, and Stanislas Polu. Proof artifact co-training for theorem proving with language models. InICLR, 2022

    2022

  50. [59]

    Learning to prove theorems via interacting with proof assistants

    Kaiyu Yang and Jia Deng. Learning to prove theorems via interacting with proof assistants. InICML, volume 97 ofProceedings of Machine Learning Research, pages 6984–6994, 2019

    2019

  51. [60]

    Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.Trans. Mach. Learn. Res., 2023, 2023

    2023

  52. [61]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  53. [62]

    Chain-of-experts: When llms meet complex operations research problems

    Ziyang Xiao, Dongxiang Zhang, Yangjun Wu, Lilin Xu, Yuan Jessica Wang, Xiongwei Han, Xiaojin Fu, Tao Zhong, Jia Zeng, Mingli Song, and Gang Chen. Chain-of-experts: When llms meet complex operations research problems. InThe Twelfth International Conference on Learning Represent...

    2024

  54. [63]

    Optimus: Scalable optimization modeling with (MI)LP solvers and large language models

    Ali AhmadiTeshnizi, Wenzhi Gao, and Madeleine Udell. Optimus: Scalable optimization modeling with (MI)LP solvers and large language models. InForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024

    2024

  55. [64]

    Nl4opt competition: Formulating optimization problems based on their natural language descriptions

    Rindranirina Ramamonjison, Timothy Yu, Raymond Li, Haley Li, Giuseppe Carenini, Bissan Ghaddar, Shiqi He, Mahdi Mostajabdaveh, Amin Banitalebi-Dehkordi, Zirui Zhou, et al. Nl4opt competition: Formulating optimization problems based on their natural language descriptions. InNeu...

    2022

  56. [65]

    Highlights from comap’s 2024 mcm and icm, 2024

    COMAP. Highlights from comap’s 2024 mcm and icm, 2024

    2024

  57. [66]

    Mcm/icm contest rules, registration and instructions, 2025

    COMAP. Mcm/icm contest rules, registration and instructions, 2025

    2025

  58. [67]

    mgte: Generalized long-context text representation and reranking models for multilingual text retrieval

    Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, et al. mgte: Generalized long-context text representation and reranking models for multilingual text retrieval. InProceedings of the 2024 Conference on Empi...

    2024

  59. [68]

    the player who seemed to have the advantage are often attributed to “momentum

    Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Aleksander Madry, and Lilian Weng. MLE-bench: Evaluating machine learning agents on machine learning engineering. InICLR, 2025. 14 MM-A...

    2025

  60. [69]

    Depth of Thinking: Evaluate whether the analysis demonstrates a comprehensive understanding of the underlying problem. Does it go beyond surface-level observations? Are the assumptions, limitations, and potential implications of the results carefully considered? Assess whether...

  61. [70]

    Novelty of Perspective: Analyze the originality of the approach taken in the analysis. Does it introduce new insights or merely rehash well-established methods or solutions? Are alternative perspectives or unconventional techniques explored, or is the analysis constrained by a...

  62. [71]

    Critical Evaluation of Results: Consider the extent to which the analysis critically engages with the results. Are the conclusions drawn from the analysis well-supported by the mathematical findings, or do they overlook key uncertainties or counterexamples? Does the analysis a...

  63. [72]

    Rigor and Precision: Assess the level of rigor applied in the analysis. Are the steps logically consistent and mathematically sound, or are there overlooked errors, gaps, or assumptions that undermine the conclusions? Does the analysis exhibit a clear, methodical approach, or ...

  64. [73]

    Contextual Awareness: Evaluate how well the analysis situates itself within the broader landscape of mathematical modeling in this area. Does it consider previous work or developments in the field? Is there any indication of awareness of real-world implications, practical cons...

  65. [74]

    Clearly outline how earlier tasks influence or constrain later ones

    **Identify Task Dependencies:** For each task, determine which preceding tasks provide necessary input, data, or conditions for its execution. Clearly outline how earlier tasks influence or constrain later ones

  66. [75]

    This includes: - *Data Dependency:* When one task produces outputs that are required as inputs for another task

    **Describe Dependency Types:** Specify the nature of the dependencies between tasks. This includes: - *Data Dependency:* When one task produces outputs that are required as inputs for another task. - *Methodological Dependency:* When a later task builds upon a theoretical fram...

  67. [76]

    ‘json {{

    **Ensure Completeness:** Verify that all tasks in the decomposition are accounted for in the dependency analysis and that no essential dependencies are missing. ## Output Format: Respond as comprehensively and in as much detail as possible. Do not format your response in Markd...

  68. [77]

    Accuracy and Rigor: 35 - Formula Integrity: Evaluate whether the mathematical models and the corresponding formulas are mathemati- cally sound and consistent with the underlying assumptions of the problem. Are the formulas properly derived, free from logical errors, and reflec...

  69. [78]

    Innovation and Insight: - Novelty of Approach: Critique the originality of the modeling approach. Does the model present a new or unconventional way of solving the problem, or does it simply rely on established methodologies without offering new insights? - Consider whether an...

  70. [79]

    Other Files

    Applicable: - Real-World Relevance: Evaluate the model’s practical applicability. How well does it apply to real-world problems, and to what extent does it provide actionable insights for decision-making or problem-solving in the field? Critique the analysis without offering a...

  71. [80]

    **It should be tested to verify it works in the intended environment**

    **Ensure the code is executable** and will successfully run without errors, producing the expected results. **It should be tested to verify it works in the intended environment**

  72. [81]

    Other Files

    **Reuse files from "Other Files" whenever possible** instead of redoing tasks that have already been completed by other agents

  73. [82]

    **All data processing steps must save the processed results to local files (CSV , JSON, or pickle) for easy access by other agents.** 38

  74. [83]

    **The output should be as detailed as possible**, including intermediate results and final outputs

  75. [84]

    Other Files

    **Ensure transparency** by logging key computation steps and providing clear outputs. ## Implementation Guidelines: - **Prioritize using files from "Other Files" before processing raw data** to avoid redundant computation. - Follow the provided **modeling formulas** and **mode...

  76. [85]

    Well-structured and easy to read

  77. [86]

    Properly typeset for LaTeX documents. 45

  78. [87]

    Adaptive in size and position to fit neatly into any document

  79. [88]

    Truncate and wrap long formulas, symbols and text in the table for better readability. <paper> {previous_chapters} </paper> Exmple of Table Format: “‘latex \begin{table}[H] \centering \renewcommand{\arraystretch}{1.3} \begin{tabular}{>{\raggedright\arraybackslash}p{3cm}> {\rag...

  80. [89]

    A concise, descriptive title that reflects the paper’s main focus

  81. [90]

    A comprehensive and detailed summary highlighting key findings and methodology

  82. [91]

    ‘Json {{

    4-6 relevant keywords that capture the paper’s main themes Returns the Legal JSON Format: “‘Json {{ "title": "A clear, concise title", "summary": "A well-structured summary covering the following information: \n- Restatement and Clarification of the Problem: Describe the probl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.