Pith. sign in

REVIEW 5 major objections 4 minor 80 references

MARCO: Meta-Reflection with Cross-Referencing for Code Reasoning

T0 review · 5 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MARCO claims that an LLM code-reasoning agent can become progressively smarter during inference by accumulating condensed takeaways from past problems and borrowing proposal-feedback lessons from peer agents.

desk verdict MARCO is a plausible test-time learning recipe for code reasoning, but the current experiments don't separate its mechanisms from the extra compute and context it receives, so the headline claim remains unproven. read the letter →

arxiv 2505.17481 v1 pith:3ICD3UEN submitted 2025-05-23 cs.CL

Yusheng Zhao , Xiao Luo , Weizhi Zhang , Wei Ju , Zhiping Xiao , Philip S. Yu , Ming Zhang This is my paper
classification cs.CL
keywords codereasoningLLMagentmeta-reflectioncross-referencingknowledgeaccumulationmulti-agentlessonsharinginductive/deductive/abductiveinference-timeself-improvement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether an LLM can become progressively better at code reasoning over a sequence of problems, and proposes MARCO, a framework that adds two things to the usual propose-and-reflect loop. First, meta-reflection writes a concise takeaway after each solved problem and stores it in a knowledge bank, so later problems are solved with guidance drawn from earlier ones. Second, cross-referencing lets each agent see the solutions and interpreter feedback of peer agents working on the same problem, so mistakes made by one agent can be avoided by others. Across eight datasets covering inductive, deductive, and abductive code reasoning, MARCO outperforms chain-of-thought, chain-of-code, and a reflection-based baseline on several LLM backbones. The paper is trying to show that inference-time self-improvement, without weight updates, is a practical route to cumulative gains in code reasoning.

What carries the argument

The mechanism is a two-channel memory at inference time. Inter-problem, meta-reflection summarizes each problem's reasoning path into a key takeaway, which is added to a knowledge bank and periodically condensed by a knowledge condenser into a short set of transferable sentences. Intra-problem, cross-referencing passes the extracted code and interpreter feedback of each peer agent, forming the lesson pair $\langle B^i_{j',t-1}, \hat{Y}^i_{j',t-1}\rangle$, to the current agent. The prompt template then combines the condensed knowledge $\hat{K}^i$, the agent's own chat history, and peers' lessons, so future generations are conditioned on accumulated experience instead of only on the current problem.

What would settle it

Run MARCO against a baseline given an equally large prompt containing the same number of extra tokens, such as irrelevant summaries and random peer lessons, with the same iteration budget; if accuracy matches MARCO, the mechanisms are not the cause of the improvement.

Watch

Extended reading notes

Core claim

The central claim is that an LLM agent's code-reasoning ability can evolve during inference: with each problem it solves, the agent gets smarter, and the improvement accumulates. MARCO operationalizes this so that the next solution depends not only on the current reasoning path but also on condensed cross-problem knowledge and on the proposal-feedback lessons of peer agents. The evidence includes accuracy gains over CoT, CoC, and RHDA on ListFunction, MiniARC, RobustFill, DeepCoder, CRUXEval, and LiveCodeBench, larger relative gains for weaker backbones, and an ablation showing that removing meta-reflection, the knowledge condenser, or cross-referencing each degrades performance. The paper also reports that the improvement over the baseline grows from the first half to the second half of the problem sequence, which it reads as evidence that the agent is genuinely accumulating knowledge.

Load-bearing premise

The reported gains are attributed to the meta-reflection and cross-referencing mechanisms, but the experiments do not control for the fact that MARCO agents receive a much larger prompt, with accumulated knowledge plus peers' lessons, and more tokens than the baselines, so the improvements may come from extra context rather than from learning.

Editorial extensions

If this is right

  • An agent using MARCO should show growing accuracy as it works through a sequence of code-reasoning problems, with larger gains in the second half of the sequence than the first.
  • Removing meta-reflection, the knowledge condenser, or cross-referencing should each lower accuracy, so all three components carry part of the gain.
  • Two proposal-feedback iterations are enough; spending more iterations reflecting on the same problem does not improve results.
  • The benefits should be larger on harder sub-tasks such as abduction and on weaker backbones, where accumulated knowledge has more room to help.
  • Condensing the knowledge bank every 8 to 10 problems performs better than shorter or longer periods; too short gives non-generalizable summaries, and too long makes the bank stale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mechanism, rather than just extra prompt tokens, is what drives the gains, MARCO points toward a general inference-time memory layer: any LLM could accumulate condensed takeaways while working through a batch of tasks, with no weight update. The paper does not test this framing.
  • The cross-referencing component is a form of social learning among agents; the same proposal-feedback sharing could be tried on non-code reasoning, such as math word problems or planning, where feedback comes from a verifier rather than a Python interpreter.
  • A direct control would be to give baselines the same number of extra context tokens, such as random takeaways and random peer lessons, with the same iteration budget; if the gap closes, the cumulative improvement is largely a context and compute effect.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper introduces MARCO, an inference-time framework that lets an LLM code-reasoning agent accumulate cross-problem experience through meta-reflection (summarizing past reasoning paths into a condensed knowledge bank) and share intra-problem lessons with peer agents through cross-referencing. The method is evaluated on eight datasets covering inductive, deductive, and abductive code reasoning, with three LLM backbones, against CoT, CoC, and RHDA baselines; the reported tables show consistent gains. Ablations, hyperparameter analyses, and a first-half/second-half comparison are used to argue that the gains come from knowledge accumulation and lesson sharing.

Significance. If the reported gains are causal, MARCO would be a useful step beyond static prompting: it is an inexpensive, training-free way to make a code-reasoning agent improve across a sequence of problems, with results spanning three reasoning subtasks and several model families. The paper is also transparent about the components that need to be tuned, and the qualitative examples in Figure 5 give some evidence that the summarized takeaways are semantically meaningful. The central causal claim, however, is not yet established: the experimental design does not control for MARCO's much larger context, multi-agent sampling, and compute budget, and no variance or code is provided. These are correctable experimental gaps rather than internal inconsistencies.

major comments (5)
  1. [Section 4.2, Tables 1-2] The reported gains are confounded by inference budget. MARCO uses T=2 iterations (Section 4.4), M peer agents in Eq. (7), and a growing condensed knowledge bank K-hat (Eqs. 4-5), while CoT, CoC, and RHDA receive only the problem text; the manuscript never reports M, total token usage, or a budget-matched baseline. Please add controls that give the baselines the same number of samples and the same total reasoning tokens, and report repeated-run variance and significance tests.
  2. [Section 4.3, Table 3] The ablations do not isolate the mechanisms. MARCOv1, MARCOv2, and MARCOv3 each remove one component but keep all other extra context and multi-agent budget, and MARCOv2 additionally changes the prompt size (full knowledge bank instead of condensed), so the performance drops could be caused by context length rather than by the removed component. A prompt-budget-controlled comparison, e.g., replacing the condensed knowledge with random or irrelevant takeaways of the same length, is needed.
  3. [Section 4.4] The hyperparameters T and T_c are selected on ListFunction and then applied to all other datasets, and the main results in Tables 1-2 include ListFunction itself; this creates a risk of selection-on-the-benchmark. Please provide a nested or held-out hyperparameter analysis, or show that the ranking of MARCO versus baselines is stable for a range of T and T_c on the other datasets.
  4. [Section 4.5, Figure 4 (right)] The first-half/second-half comparison is relative to CoT only and does not rule out a generic 'more context and compute' effect. A same-budget baseline whose prompt grows with random or stale takeaways would also likely show larger absolute gains in the second half as the MARCO pipeline warms up. Please report absolute per-half accuracies and add such a control before claiming that MARCO 'becomes progressively smarter.'
  5. [Equation (5) and Section 4.5] The condensation step is assumed to preserve transferable knowledge, but the paper provides no quantitative evidence of fidelity; the ablation of the condenser (MARCOv2) is confounded by context size as noted above. Since the knowledge bank is the mechanism behind cumulative improvement, please evaluate condensation fidelity directly, e.g., by comparing condensed versus full-bank prompts at equal token budgets or by measuring information retention.
minor comments (4)
  1. [Section 4.1] The reference 'Appendix??' is unresolved; the DSL details for RobustFill and DeepCoder need to be filled in.
  2. [Section 4.2, first paragraph] 'Transductive reasoning' appears where the paper's own taxonomy says 'abductive reasoning'; please correct this terminology.
  3. [Equations (2) and (7)] The notation is inconsistent: Eq. (2) conditions on full previous paths and full peer paths, while Eq. (7) uses only condensed knowledge and peer lessons; please clarify which information is actually included in the prompts.
  4. [Figures 4 and 5] The axes in Figure 4 left/middle are not labeled with the variable names, and Figure 5 relies on color markers (blue/green/orange) that may not be legible in grayscale; please add labels and use distinct markers.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: MARCO's reported improvements are empirical outcomes on held-out target pairs, not quantities defined into existence by the method's equations.

full rationale

MARCO is an empirical prompting framework, not a formal derivation, and its central claim is that injecting accumulated meta-reflections and peer lessons improves code reasoning. The claimed improvements are measured on held-out input-output pairs within each problem (Section 4.1: half of the pairs are visible, the other half are not), and the cross-problem improvement in Section 4.5 is evaluated on later problems whose answers are not used to construct the condensed knowledge bank. Equations 3-7 define the method's inference procedure, but they do not define the evaluation metric or the reported accuracy; a wrong answer is still wrong regardless of prompt context. The only substantive concern is that baselines receive less prompt context and compute than MARCO, so the gains are not cleanly causally attributed to meta-reflection and cross-referencing specifically. That is a threat to experimental validity, not circularity: the numbers could have come out the other way, and no equation or fitted parameter forces the reported improvements. Hyperparameters are chosen on ListFunction and reused, which risks overfitting but does not make the benchmark result equivalent to the hyperparameter choice by construction. No self-citation chain is load-bearing; the cited prior work is used for baselines and background, not to define away the target result. Accordingly, there are no circular steps to report.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The method introduces no fitted mathematical parameters in a derivation sense, but four design choices act as free parameters (T, Tc, M, prompts). The key assumptions are about the reliability of LLM in-context learning, the interpreter oracle, the transferability of condensed knowledge, and the fairness of the baseline comparison.

free parameters (4)
  • Number of refinement iterations T = 2
    Set to 2 based on hyperparameter analysis on ListFunction (Figure 4); halting at a plateau is a hand-picked budget.
  • Condensation period Tc = 8 to 10
    Chosen from hyperparameter analysis on ListFunction (Figure 4); the paper reports the range but does not fix a single value or show sensitivity across datasets.
  • Number of peer agents M = not stated
    The framework uses M agents exchanging lessons, but the paper never reports M, its cost, or an ablation over M.
  • Meta-reflection and condensation prompt templates = not disclosed
    The SUMMARIZE and CONDENSE prompts (Eqs. 3 and 5) are central to the method but are only described in prose; different phrasings could change the knowledge quality.
assumptions (5)
  • domain assumption LLM agents respond to in-context knowledge and peer feedback in a way that improves their generated solutions.
    The whole method relies on the model using condensed takeaways and peer lessons constructively (Eqs. 6-7).
  • domain assumption The Python interpreter provides correct binary feedback on whether a proposed function matches the visible examples.
    Feedback B is used both for iterative refinement and for generating lessons; if the interpreter signal is noisy or misused, the method degrades (Section 3.1).
  • domain assumption The datasets and the held-out input-output splits measure transferable code reasoning rather than memorization.
    Accuracy on the hidden pairs is the sole evidence for cumulative improvement (Section 4.1).
  • ad hoc to paper The condensation step preserves transferable knowledge while removing problem-specific details.
    The condenser (Eq. 5) is a prompt-based summarizer whose reliability is assumed; the paper shows ablation support but no analysis of knowledge loss.
  • domain assumption The baseline implementations are comparably optimized and are not disadvantaged by missing the extra context MARCO receives.
    CoT, CoC, and RHDA are evaluated without a knowledge bank or peer lessons, so the comparison assumes MARCO's improvements are not just an artifact of more context (Section 4.2).
invented entities (1)
  • Knowledge bank with condensed takeaways
    purpose: Stores and condenses cross-problem experiences to be injected into future prompts
    It is an internal memory structure with no external falsifiable handle; its value is shown only by in-paper ablations, so there is no independent evidence it transfers to other settings.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MARCO: Meta-Reflection with Cross-Referencing for Code Reasoning." pith.science (2026). https://pith.science/paper/3ICD3UEN

@misc{pith2026250517481,
  author       = {Pith},
  title        = {Pith review of: MARCO: Meta-Reflection with Cross-Referencing for Code Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3ICD3UEN}},
  note         = {Machine review of arXiv:2505.17481}
}
read the original abstract

The ability to reason is one of the most fundamental capabilities of large language models (LLMs), enabling a wide range of downstream tasks through sophisticated problem-solving. A critical aspect of this is code reasoning, which involves logical reasoning with formal languages (i.e., programming code). In this paper, we enhance this capability of LLMs by exploring the following question: how can an LLM agent become progressively smarter in code reasoning with each solution it proposes, thereby achieving substantial cumulative improvement? Most existing research takes a static perspective, focusing on isolated problem-solving using frozen LLMs. In contrast, we adopt a cognitive-evolving perspective and propose a novel framework named Meta-Reflection with Cross-Referencing (MARCO) that enables the LLM to evolve dynamically during inference through self-improvement. From the perspective of human cognitive development, we leverage both knowledge accumulation and lesson sharing. In particular, to accumulate knowledge during problem-solving, we propose meta-reflection that reflects on the reasoning paths of the current problem to obtain knowledge and experience for future consideration. Moreover, to effectively utilize the lessons from other agents, we propose cross-referencing that incorporates the solution and feedback from other agents into the current problem-solving process. We conduct experiments across various datasets in code reasoning, and the results demonstrate the effectiveness of MARCO.

Figures

Figures reproduced from arXiv: 2505.17481 by the authors.

Figure 1
Figure 1. We adopt a cognitive-evolving perspective and propose [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Existing methods adopt a static perspective, and the LLM agents do not improve during the [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The overall framework of the proposed MARCO, which includes meta-reflection and cross￾referencing. Meta-reflection summarizes previous problem-solving experiences into transferable knowledge accumulated for future usage. Cross-referencing enables the LLM agent to learn from the lessons of its peer agents so as to improve the current problem-solving process. solving the problem correctly. Therefore, in this paper, we… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Left and middle: performance under different iterations and condensation periods in terms of accuracy and problem accuracy on the ListFunction dataset. Right: the comparison of absolute improvements of MARCO and the baseline in both the first half and the second half o…
Figure 5
Figure 5. Figure 5: We present examples of the summarized reasoning experiences using meta-reflection on [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

80 extracted references · 24 canonical work pages

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Iot-llm: Enhancing real-world iot task reasoning with large language models.arXiv preprint arXiv:2410.02429, 2024

    Tuo An, Yunjiao Zhou, Han Zou, and Jianfei Yang. Iot-llm: Enhancing real-world iot task reasoning with large language models.arXiv preprint arXiv:2410.02429, 2024

    arXiv 2024

  3. [3]

    Deepcoder: Learning to write programs.arXiv preprint arXiv:1611.01989, 2016

    Matej Balog, Alexander L Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. Deepcoder: Learning to write programs.arXiv preprint arXiv:1611.01989, 2016

    arXiv 2016

  4. [4]

    How well can large language models reflect? a human evaluation of llm-generated reflections for motivational interviewing dialogues

    Erkan Bas, ar, Xin Sun, Iris Hendrickx, Jan de Wit, Tibor Bosse, Gert-Jan De Bruijn, Jos A Bosch, and Emiel Krahmer. How well can large language models reflect? a human evaluation of llm-generated reflections for motivational interviewing dialogues. InProceedings of the 31st International Conference on Computational Linguistics, pages 1964–1982, 2025

    work page 1964

  5. [5]

    Graph of thoughts: Solving elaborate problems with large language models

    Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17682–17690, 2024

    2024

  6. [6]

    Demystifying chains, trees, and graphs of thoughts.arXiv preprint arXiv:2401.14295, 2024

    Maciej Besta, Florim Memedi, Zhenyu Zhang, Robert Gerstenberger, Guangyuan Piao, Nils Blach, Piotr Nyczyk, Marcin Copik, Grzegorz Kwa´sniewski, Jürgen Müller, et al. Demystifying chains, trees, and graphs of thoughts.arXiv preprint arXiv:2401.14295, 2024

    arXiv 2024

  7. [7]

    Forest-of-thought: Scaling test-time compute for enhancing llm reasoning.arXiv preprint arXiv:2412.09078, 2024

    Zhenni Bi, Kai Han, Chuanjian Liu, Yehui Tang, and Yunhe Wang. Forest-of-thought: Scaling test-time compute for enhancing llm reasoning.arXiv preprint arXiv:2412.09078, 2024

    arXiv 2024

  8. [8]

    Reflective multi-agent collaboration based on large language models.Advances in Neural Information Processing Systems, 37:138595–138631, 2024

    Xiaohe Bo, Zeyu Zhang, Quanyu Dai, Xueyang Feng, Lei Wang, Rui Li, Xu Chen, and Ji-Rong Wen. Reflective multi-agent collaboration based on large language models.Advances in Neural Information Processing Systems, 37:138595–138631, 2024

    2024

Show all 80 references
  1. [9]

    Probabilistic tree-of-thought reasoning for answering knowledge-intensive complex questions

    Shulin Cao, Jiajie Zhang, Jiaxin Shi, Xin Lv, Zijun Yao, Qi Tian, Juanzi Li, and Lei Hou. Probabilistic tree-of-thought reasoning for answering knowledge-intensive complex questions. arXiv preprint arXiv:2311.13982, 2023

    2023 arXiv

  2. [10]

    A survey on evaluating large language models in code generation tasks.arXiv preprint arXiv:2408.16498, 2024

    Liguo Chen, Qi Guo, Hongrui Jia, Zhengran Zeng, Xin Wang, Yijiang Xu, Jian Wu, Yidong Wang, Qing Gao, Jindong Wang, et al. A survey on evaluating large language models in code generation tasks.arXiv preprint arXiv:2408.16498, 2024

    2024 arXiv

  3. [11]

    Llm contextbridge: A hybrid approach for intent and dialogue understanding in ivsr

    Changwoo Chun, Daniel Rim, and Juhee Park. Llm contextbridge: A hybrid approach for intent and dialogue understanding in ivsr. InProceedings of the 31st International Conference on Computational Linguistics: Industry Track, pages 794–806, 2025

    2025

  4. [12]

    Robustfill: Neural program learning under noisy i/o

    Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. Robustfill: Neural program learning under noisy i/o. InInternational conference on machine learning, pages 990–998. PMLR, 2017

    2017

  5. [13]

    An interactive query generation assistant using llm-based prompt modification and user feedback.arXiv preprint arXiv:2311.11226, 2023

    Kaustubh D Dhole, Ramraj Chandradevan, and Eugene Agichtein. An interactive query generation assistant using llm-based prompt modification and user feedback.arXiv preprint arXiv:2311.11226, 2023

    2023 arXiv

  6. [14]

    Dynamic parallel tree search for efficient llm reasoning.arXiv preprint arXiv:2502.16235, 2025

    Yifu Ding, Wentao Jiang, Shunyu Liu, Yongcheng Jing, Jinyang Guo, Yingjie Wang, Jing Zhang, Zengmao Wang, Ziwei Liu, Bo Du, et al. Dynamic parallel tree search for efficient llm reasoning.arXiv preprint arXiv:2502.16235, 2025

    2025 arXiv

  7. [15]

    Evaluating large language models in class-level code generation

    Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. Evaluating large language models in class-level code generation. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering,...

    2024

  8. [16]

    Llm-based test-driven interactive code generation: User study and empirical evaluation.IEEE Transactions on Software Engineering, 2024

    Sarah Fakhoury, Aaditya Naik, Georgios Sakkas, Saikat Chakraborty, and Shuvendu K Lahiri. Llm-based test-driven interactive code generation: User study and empirical evaluation.IEEE Transactions on Software Engineering, 2024. 10

    2024

  9. [17]

    Towards llm-driven dialogue state tracking.arXiv preprint arXiv:2310.14970, 2023

    Yujie Feng, Zexin Lu, Bo Liu, Liming Zhan, and Xiao-Ming Wu. Towards llm-driven dialogue state tracking.arXiv preprint arXiv:2310.14970, 2023

    2023 arXiv

  10. [18]

    Interpretable contrastive monte carlo tree search reasoning.arXiv preprint arXiv:2410.01707, 2024

    Zitian Gao, Boye Niu, Xuzheng He, Haotian Xu, Hongzhang Liu, Aiwei Liu, Xuming Hu, and Lijie Wen. Interpretable contrastive monte carlo tree search reasoning.arXiv preprint arXiv:2410.01707, 2024

    2024 arXiv

  11. [19]

    Clipsyntel: clip and llm synergy for multimodal question summarization in healthcare

    Akash Ghosh, Arkadeep Acharya, Raghav Jain, Sriparna Saha, Aman Chadha, and Setu Sinha. Clipsyntel: clip and llm synergy for multimodal question summarization in healthcare. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 22031–22039, 2024

    2024

  12. [20]

    Cruxeval: A benchmark for code reasoning, understanding and execution.arXiv preprint arXiv:2401.03065, 2024

    Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. Cruxeval: A benchmark for code reasoning, understanding and execution.arXiv preprint arXiv:2401.03065, 2024

    2024 arXiv

  13. [21]

    Llm-based code generation method for golang compiler testing

    Qiuhan Gu. Llm-based code generation method for golang compiler testing. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 2201–2203, 2023

    2023

  14. [22]

    Longcoder: A long-range pre-trained language model for code completion

    Daya Guo, Canwen Xu, Nan Duan, Jian Yin, and Julian McAuley. Longcoder: A long-range pre-trained language model for code completion. InInternational Conference on Machine Learning, pages 12098–12107. PMLR, 2023

    2023

  15. [23]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  16. [24]

    Reasoning with language model is planning with world model.arXiv preprint arXiv:2305.14992, 2023

    Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhit- ing Hu. Reasoning with language model is planning with world model.arXiv preprint arXiv:2305.14992, 2023

    2023 arXiv

  17. [25]

    Thought graph: Generating thought process for biological reasoning

    Chi-Yang Hsu, Kyle Cox, Jiawei Xu, Zhen Tan, Tianhua Zhai, Mengzhou Hu, Dexter Pratt, Tianlong Chen, Ziniu Hu, and Ying Ding. Thought graph: Generating thought process for biological reasoning. InCompanion Proceedings of the ACM Web Conference 2024, pages 537–540, 2024

    2024

  18. [26]

    Unlocking the potential of user feedback: Leveraging large language model as user simulators to enhance dialogue system

    Zhiyuan Hu, Yue Feng, Anh Tuan Luu, Bryan Hooi, and Aldo Lipani. Unlocking the potential of user feedback: Leveraging large language model as user simulators to enhance dialogue system. InProceedings of the 32nd ACM International Conference on Information and Knowledge Managem...

    2023

  19. [27]

    Large language models cannot self-correct reasoning yet.arXiv preprint arXiv:2310.01798, 2023

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet.arXiv preprint arXiv:2310.01798, 2023

    2023 arXiv

  20. [28]

    Knowledge-aware code generation with large language models

    Tao Huang, Zhihong Sun, Zhi Jin, Ge Li, and Chen Lyu. Knowledge-aware code generation with large language models. InProceedings of the 32nd IEEE/ACM International Conference on Program Comprehension, pages 52–63, 2024

    2024

  21. [29]

    Language models for code completion: A practical evaluation

    Maliheh Izadi, Jonathan Katzy, Tim Van Dam, Marc Otten, Razvan Mihai Popescu, and Arie Van Deursen. Language models for code completion: A practical evaluation. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–13, 2024

    2024

  22. [30]

    Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974, 2024

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  23. [31]

    Cotran: An llm-based code translator using reinforcement learning with feedback from compiler and symbolic execution

    Prithwish Jana, Piyush Jha, Haoyang Ju, Gautham Kishore, Aryan Mahajan, and Vijay Ganesh. Cotran: An llm-based code translator using reinforcement learning with feedback from compiler and symbolic execution. InECAI 2024, pages 4011–4018. IOS Press, 2024. 11

    2024

  24. [32]

    Towards mitigating hallucination in large language models via self-reflection.arXiv preprint arXiv:2310.06271, 2023

    Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. Towards mitigating hallucination in large language models via self-reflection.arXiv preprint arXiv:2310.06271, 2023

    2023 arXiv

  25. [33]

    Self-planning code generation with large language models.ACM Transactions on Software Engineering and Methodology, 33(7):1–30, 2024

    Xue Jiang, Yihong Dong, Lecheng Wang, Zheng Fang, Qiwei Shang, Ge Li, Zhi Jin, and Wenpin Jiao. Self-planning code generation with large language models.ACM Transactions on Software Engineering and Methodology, 33(7):1–30, 2024

    2024

  26. [34]

    Playgrounds for abstraction and reasoning

    Subin Kim, Prin Phunyaphibarn, Donghyun Ahn, and Sundong Kim. Playgrounds for abstraction and reasoning. InNeurIPS 2022 Workshop on Neuro Causal and Symbolic AI (nCSI), 2022

    2022

  27. [35]

    Training language models to self-correct via reinforcement learning.arXiv preprint arXiv:2409.12917, 2024

    Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning.arXiv preprint arXiv:2409.12917, 2024

    2024 arXiv

  28. [36]

    Supporting self- reflection at scale with large language models: Insights from randomized field experiments in classrooms

    Harsh Kumar, Ruiwei Xiao, Benjamin Lawson, Ilya Musabirov, Jiakai Shi, Xinyuan Wang, Huayin Luo, Joseph Jay Williams, Anna N Rafferty, John Stamper, et al. Supporting self- reflection at scale with large language models: Insights from randomized field experiments in classrooms...

    2024

  29. [37]

    A survey on enhancing reinforcement learning in complex environments: Insights from human and llm feedback.arXiv preprint arXiv:2411.13410, 2024

    Alireza Rashidi Laleh and Majid Nili Ahmadabadi. A survey on enhancing reinforcement learning in complex environments: Insights from human and llm feedback.arXiv preprint arXiv:2411.13410, 2024

    2024 arXiv

  30. [38]

    Boosting logical reasoning in large language models through a new framework: The graph of thought.arXiv preprint arXiv:2308.08614, 2023

    Bin Lei, Chunhua Liao, Caiwen Ding, et al. Boosting logical reasoning in large language models through a new framework: The graph of thought.arXiv preprint arXiv:2308.08614, 2023

    2023 arXiv

  31. [39]

    Chain of code: Reasoning with a language model-augmented code emulator.arXiv preprint arXiv:2312.04474, 2023

    Chengshu Li, Jacky Liang, Andy Zeng, Xinyun Chen, Karol Hausman, Dorsa Sadigh, Sergey Levine, Li Fei-Fei, Fei Xia, and Brian Ichter. Chain of code: Reasoning with a language model-augmented code emulator.arXiv preprint arXiv:2312.04474, 2023

    2023 arXiv

  32. [40]

    Large language model-aware in-context learning for code generation.ACM Transactions on Software Engineering and Methodology, 2023

    Jia Li, Chongyang Tao, Jia Li, Ge Li, Zhi Jin, Huangzhao Zhang, Zheng Fang, and Fang Liu. Large language model-aware in-context learning for code generation.ACM Transactions on Software Engineering and Methodology, 2023

    2023

  33. [41]

    Dora: Dynamic optimization prompt for continuous reflection of llm-based agent

    Kun Li, Tingzhang Zhao, Wei Zhou, and Songlin Hu. Dora: Dynamic optimization prompt for continuous reflection of llm-based agent. InProceedings of the 31st International Conference on Computational Linguistics, pages 7546–7557, 2025

    2025

  34. [42]

    Selective reflection-tuning: Student-selected data recycling for llm instruction-tuning

    Ming Li, Lichang Chen, Jiuhai Chen, Shwai He, Jiuxiang Gu, and Tianyi Zhou. Selective reflection-tuning: Student-selected data recycling for llm instruction-tuning. InFindings of the Association for Computational Linguistics ACL 2024, pages 16189–16211, 2024

    2024

  35. [43]

    Enhancing chain-of-thought reasoning in large language models through text style diversity and prompt fusion

    Weichen Li and Weimin Pan. Enhancing chain-of-thought reasoning in large language models through text style diversity and prompt fusion. InThird International Conference on Electronic Information Engineering, Big Data, and Computer Technology (EIBDCT 2024), volume 13181, pages...

    2024

  36. [44]

    Cctest: Testing and repairing code completion systems

    Zongjie Li, Chaozheng Wang, Zhibo Liu, Haoxuan Wang, Dong Chen, Shuai Wang, and Cuiyun Gao. Cctest: Testing and repairing code completion systems. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 1238–1250. IEEE, 2023

    2023

  37. [45]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  38. [46]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36:21558–21572, 2023. 12

    2023

  39. [47]

    Integration of llms and the physical world: Research and application

    Xiaoyu Luo, Daping Liu, Fan Dang, and Hanjiang Luo. Integration of llms and the physical world: Research and application. InProceedings of the ACM Turing Award Celebration Conference-China 2024, pages 1–5, 2024

    2024

  40. [48]

    Llm-a*: Large language model enhanced incremental heuristic search on path planning.arXiv preprint arXiv:2407.02511, 2024

    Silin Meng, Yiwei Wang, Cheng-Fu Yang, Nanyun Peng, and Kai-Wei Chang. Llm-a*: Large language model enhanced incremental heuristic search on path planning.arXiv preprint arXiv:2407.02511, 2024

    2024 arXiv

  41. [49]

    Enhancing reasoning capabilities of llms via principled synthetic logic corpus.Advances in Neural Information Processing Systems, 37:73572–73604, 2024

    Terufumi Morishita, Gaku Morio, Atsuki Yamaguchi, and Yasuhiro Sogawa. Enhancing reasoning capabilities of llms via principled synthetic logic corpus.Advances in Neural Information Processing Systems, 37:73572–73604, 2024

    2024

  42. [50]

    Ros-llm: A ros framework for embodied ai with task feedback and structured reasoning.arXiv preprint arXiv:2406.19741, 2024

    Christopher E Mower, Yuhui Wan, Hongzhan Yu, Antoine Grosnit, Jonas Gonzalez-Billandon, Matthieu Zimmer, Jinlong Wang, Xinyu Zhang, Yao Zhao, Anbang Zhai, et al. Ros-llm: A ros framework for embodied ai with task feedback and structured reasoning.arXiv preprint arXiv:2406.19741, 2024

    2024 arXiv

  43. [51]

    Adaptive graph of thoughts: Test-time adaptive reasoning unifying chain, tree, and graph structures.arXiv preprint arXiv:2502.05078, 2025

    Tushar Pandey, Ara Ghukasyan, Oktay Goktas, and Santosh Kumar Radha. Adaptive graph of thoughts: Test-time adaptive reasoning unifying chain, tree, and graph structures.arXiv preprint arXiv:2502.05078, 2025

    2025 arXiv

  44. [52]

    Llms can learn self- restraint through iterative self-reflection.arXiv preprint arXiv:2405.13022, 2024

    Alexandre Piché, Aristides Milios, Dzmitry Bahdanau, and Chris Pal. Llms can learn self- restraint through iterative self-reflection.arXiv preprint arXiv:2405.13022, 2024

    2024 arXiv

  45. [53]

    Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement.arXiv preprint arXiv:2310.08559, 2023

    Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang, Yoon Kim, Yejin Choi, Nouha Dziri, et al. Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement.arXiv preprint arXiv...

    arXiv

  46. [54]

    A tree-of-thoughts to broaden multi-step reasoning across languages

    Leonardo Ranaldi, Giulia Pucci, Federico Ranaldi, Elena Sofia Ruzzetti, and Fabio Massimo Zanzotto. A tree-of-thoughts to broaden multi-step reasoning across languages. InFindings of the Association for Computational Linguistics: NAACL 2024, pages 1229–1241, 2024

    2024

  47. [55]

    Collaborative quest completion with llm-driven non-player characters in minecraft.arXiv preprint arXiv:2407.03460, 2024

    Sudha Rao, Weijia Xu, Michael Xu, Jorge Leandro, Ken Lobb, Gabriel DesGarennes, Chris Brockett, and Bill Dolan. Collaborative quest completion with llm-driven non-player characters in minecraft.arXiv preprint arXiv:2407.03460, 2024

    2024 arXiv

  48. [56]

    Self-reflection in llm agents: Effects on problem-solving performance.arXiv preprint arXiv:2405.06682, 2024

    Matthew Renze and Erhan Guven. Self-reflection in llm agents: Effects on problem-solving performance.arXiv preprint arXiv:2405.06682, 2024

    2024 arXiv

  49. [57]

    PhD thesis, Massachusetts Institute of Technology, 2020

    Joshua Stewart Rule.The child as hacker: building more human-like models of learning. PhD thesis, Massachusetts Institute of Technology, 2020

    2020

  50. [58]

    Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36:8634–8652, 2023

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36:8634–8652, 2023

    2023

  51. [59]

    Learning to summarize from llm-generated feedback.arXiv preprint arXiv:2410.13116, 2024

    Hwanjun Song, Taewon Yun, Yuho Lee, Jihwan Oh, Gihun Lee, Jason Cai, and Hang Su. Learning to summarize from llm-generated feedback.arXiv preprint arXiv:2410.13116, 2024

    2024 arXiv

  52. [60]

    Determlr: Augmenting llm-based logical reasoning from indeterminacy to determinacy

    Hongda Sun, Weikai Xu, Wei Liu, Jian Luan, Bin Wang, Shuo Shang, Ji-Rong Wen, and Rui Yan. Determlr: Augmenting llm-based logical reasoning from indeterminacy to determinacy. arXiv preprint arXiv:2310.18659, 2023

    2023 arXiv

  53. [61]

    Debugbench: Evaluating debugging capability of large language models.arXiv preprint arXiv:2401.04621, 2024

    Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Yinxu Pan, Yesai Wu, Haotian Hui, Weichuan Liu, Zhiyuan Liu, et al. Debugbench: Evaluating debugging capability of large language models.arXiv preprint arXiv:2401.04621, 2024

    2024 arXiv

  54. [62]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  55. [63]

    Dynamic self-consistency: Leveraging reasoning paths for efficient llm sampling.arXiv preprint arXiv:2408.17017, 2024

    Guangya Wan, Yuqi Wu, Jie Chen, and Sheng Li. Dynamic self-consistency: Leveraging reasoning paths for efficient llm sampling.arXiv preprint arXiv:2408.17017, 2024. 13

    2024 arXiv

  56. [64]

    Llm-empowered state representation for reinforcement learning.arXiv preprint arXiv:2407.13237, 2024

    Boyuan Wang, Yun Qu, Yuhang Jiang, Jianzhun Shao, Chang Liu, Wenming Yang, and Xi- angyang Ji. Llm-empowered state representation for reinforcement learning.arXiv preprint arXiv:2407.13237, 2024

    2024 arXiv

  57. [65]

    Towards self-improvement of llms via mcts: Leveraging stepwise knowledge with curriculum preference learning.arXiv preprint arXiv:2410.06508, 2024

    Xiyao Wang, Linfeng Song, Ye Tian, Dian Yu, Baolin Peng, Haitao Mi, Furong Huang, and Dong Yu. Towards self-improvement of llms via mcts: Leveraging stepwise knowledge with curriculum preference learning.arXiv preprint arXiv:2410.06508, 2024

    arXiv

  58. [66]

    Unveiling factual recall behaviors of large language models through knowledge neurons.arXiv preprint arXiv:2408.03247, 2024

    Yifei Wang, Yuheng Chen, Wanting Wen, Yu Sheng, Linjing Li, and Daniel Dajun Zeng. Unveiling factual recall behaviors of large language models through knowledge neurons.arXiv preprint arXiv:2408.03247, 2024

    2024 arXiv

  59. [67]

    Codet5+: Open code large language models for code understanding and generation.arXiv preprint arXiv:2305.07922, 2023

    Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi. Codet5+: Open code large language models for code understanding and generation.arXiv preprint arXiv:2305.07922, 2023

    2023 arXiv

  60. [68]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  61. [69]

    Mindmap: Knowledge graph prompting sparks graph of thoughts in large language models.arXiv preprint arXiv:2308.09729, 2023

    Yilin Wen, Zifeng Wang, and Jimeng Sun. Mindmap: Knowledge graph prompting sparks graph of thoughts in large language models.arXiv preprint arXiv:2308.09729, 2023

    2023 arXiv

  62. [70]

    Beyond chain-of-thought: A survey of chain-of-x paradigms for llms.arXiv preprint arXiv:2404.15676, 2024

    Yu Xia, Rui Wang, Xu Liu, Mingyan Li, Tong Yu, Xiang Chen, Julian McAuley, and Shuai Li. Beyond chain-of-thought: A survey of chain-of-x paradigms for llms.arXiv preprint arXiv:2404.15676, 2024

    2024 arXiv

  63. [71]

    Jamplate: exploring llm-enhanced templates for idea reflection

    Xiaotong Xu, Jiayu Yin, Catherine Gu, Jenny Mar, Sydney Zhang, Jane L E, and Steven P Dow. Jamplate: exploring llm-enhanced templates for idea reflection. InProceedings of the 29th International Conference on Intelligent User Interfaces, pages 907–921, 2024

    2024

  64. [72]

    Softcot: Soft chain-of-thought for efficient reasoning with llms.arXiv preprint arXiv:2502.12134, 2025

    Yige Xu, Xu Guo, Zhiwei Zeng, and Chunyan Miao. Softcot: Soft chain-of-thought for efficient reasoning with llms.arXiv preprint arXiv:2502.12134, 2025

    2025 arXiv

  65. [73]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  66. [74]

    Tree of thoughts: Deliberate problem solving with large language models.Ad- vances in neural information processing systems, 36:11809–11822, 2023

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models.Ad- vances in neural information processing systems, 36:11809–11822, 2023

    2023

  67. [75]

    Rest-mcts*: Llm self-training via process reward guided tree search.Advances in Neural Information Processing Systems, 37:64735–64772, 2024

    Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mcts*: Llm self-training via process reward guided tree search.Advances in Neural Information Processing Systems, 37:64735–64772, 2024

    2024

  68. [76]

    Chain of preference optimization: Improving chain-of-thought reasoning in llms.Advances in Neural Information Processing Systems, 37:333–356, 2024

    Xuan Zhang, Chao Du, Tianyu Pang, Qian Liu, Wei Gao, and Min Lin. Chain of preference optimization: Improving chain-of-thought reasoning in llms.Advances in Neural Information Processing Systems, 37:333–356, 2024

    2024

  69. [77]

    Evaluating large language models through role-guide and self-reflection: A comparative study

    Lili Zhao, Yang Wang, Qi Liu, Mengyun Wang, Wei Chen, Zhichao Sheng, and Shijin Wang. Evaluating large language models through role-guide and self-reflection: A comparative study. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  70. [78]

    Unveiling the magic of code reasoning through hypothesis decompo- sition and amendment.arXiv preprint arXiv:2502.13170, 2025

    Yuze Zhao, Tianyun Ji, Wenjun Feng, Zhenya Huang, Qi Liu, Zhiding Liu, Yixiao Ma, Kai Zhang, and Enhong Chen. Unveiling the magic of code reasoning through hypothesis decompo- sition and amendment.arXiv preprint arXiv:2502.13170, 2025

    2025 arXiv

  71. [79]

    Can llm replace stack overflow? a study on robustness and reliability of large language model code generation

    Li Zhong and Zilong Wang. Can llm replace stack overflow? a study on robustness and reliability of large language model code generation. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 21841–21849, 2024

    2024

  72. [80]

    Debug like a human: A large language model debugger via verifying runtime execution step-by-step.arXiv preprint arXiv:2402.16906, 2024

    Li Zhong, Zilong Wang, and Jingbo Shang. Debug like a human: A large language model debugger via verifying runtime execution step-by-step.arXiv preprint arXiv:2402.16906, 2024. 14

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.