Pith. sign in

REVIEW 4 major objections 5 minor 13 references

DSMentor: Enhancing Data Science Agents with Curriculum Learning and Online Knowledge Accumulation

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that ordering data-science tasks from easy to hard and letting the agent reuse its own earlier solved attempts improves LLM data-science agents.

desk verdict A plausible inference-time memory plus curriculum recipe whose headline gains are inflated by a transductive evaluation that bundles memory with ordering. read the letter →

arxiv 2505.14163 v1 pith:23KNAMF2 submitted 2025-05-20 cs.AI

He Wang , Alexander Hanbo Li , Yiqun Hu , Sheng Zhang , Hideo Kobayashi , Jiani Zhang , Henry Zhu , Chung-Wei Hang
show 1 more author
Patrick Ng
This is my paper
classification cs.AI
keywords datascienceagentscurriculumlearninginference-timeoptimizationonlinelong-termmemoryLLMcodegenerationcausalreasoningretrieval-augmentedprompting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that how a large language model agent sequences data-science problems during inference matters as much as the model or prompt. DSMentor uses a mentor agent to rate each problem's difficulty from its description alone and then orders the problems from easy to hard. While solving them, a student agent keeps every previous attempt, the question, the generated code, and a correct/incorrect tag, in a growing memory and retrieves the most similar past examples when facing the next problem. On the DSEval and QRData benchmarks, this recipe improves pass rates over vanilla program-of-thoughts agents and prior data-science agents, with a reported 8.8-point gain on causal-reasoning questions. If the effect is real, an agent can get better at data science without additional training, fine-tuning, or a larger model.

What carries the argument

The load-bearing mechanism is the pair formed by a mentor-generated easy-to-hard curriculum and an online long-term memory. The memory is $M_i=\{(p_k,c_k,t_k)\}_{k=1}^{i-1}$: for each earlier problem, its description $p_k$, the code $c_k$ the student generated, and the environment tag $t_k\in\{\text{Correct},\text{Incorrect}\}$. Retrieval for problem $i$ selects the top-$K$ entries by cosine similarity $\cos(E(p_i),E(p_k))$ using a fixed embedding model, and the retrieved examples are presented in increasing similarity before code generation. The curriculum is produced by the mentor LLM, which rates difficulty from the problem text alone using written scale guidelines. This mechanism converts a static problem set into a sequential learning experience in which each attempt potentially improves the next.

What would settle it

Run DSMentor with the long-term memory either emptied before each question or filled only from a disjoint set of data-science problems; if the pass-rate advantage over vanilla program-of-thoughts agents disappears, the reported gains are caused by retrieving the benchmark's own earlier questions rather than by curriculum ordering.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that an LLM agent's capability on data-science tasks can be grown during inference by combining two simple mechanisms: a mentor-generated curriculum and an accumulating memory. The mentor assigns difficulty levels using only problem statements, building a curriculum that starts with filtering and counting tasks and moves toward multi-step analysis; the student solves problems in that order and, for each new problem, retrieves the top-K prior problem, code, and outcome entries ranked by embedding similarity and ordered from least to most similar. Because the environment's evaluation tag is appended to memory after every attempt, the agent's later attempts are conditioned on its own earlier successes and failures. Across DSEval and QRData, this framework outperformed vanilla program-of-thoughts agents with the same base model and several prior data-science agents, and its edge was largest on causal-reasoning problems.

Load-bearing premise

The weakest point is the evaluation protocol: the benchmark's own problems are solved in sequence and every earlier question, code, and pass/fail tag is stored and retrieved for later problems, so the measured gains assume that this reuse of the benchmark itself is a fair test of curriculum learning.

Editorial extensions

If this is right

  • The same base model improves without any weight updates, so inference-time ordering and memory are a complementary lever to model scale.
  • The easy-to-hard ordering with increasing-similarity retrieval beat random and hard-to-easy orders in the reported ablations, so task order itself carries part of the gain.
  • Because the biggest reported improvement was on causal reasoning, curriculum accumulation may be especially useful for multi-step reasoning where simpler statistical tasks build the needed concepts.
  • Including failed attempts in memory helped more on small datasets, implying that error cases act as useful learning signals when the memory pool is small.
  • The framework transfers across two base LLMs, Claude-3.5-Sonnet and Llama-3.1-70b, so the mechanism is not tied to a single model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer that the protocol as reported is transductive: memory is filled from the benchmark's own ordered questions, so the gains may partly reflect test-time retrieval of the test set itself; the paper does not report a run on a disjoint, unseen collection of problems.
  • We infer a practical extension: use the mentor to order a backlog of real analyst tasks and let the memory persist across tasks; if transductive leakage is not driving the results, this could improve live workloads where a stream of related requests arrives.
  • We infer that the difficulty signal may not need a sophisticated judge; in one LeetCode ablation, pass-rate-based difficulty performed nearly as well as the LLM mentor, so a simple proxy could replace the mentor in production.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes DSMentor, an inference-time curriculum learning framework for data science agents. A Mentor agent estimates the difficulty of each problem in a dataset and orders them from easy to hard; a Student agent solves the problems in that order, storing each solved problem (description, generated code, and pass/fail tag) in a growing memory, and before solving a problem retrieves the top-K most similar previous problems from that memory. Experiments on DSEval and QRData with Claude-3.5-Sonnet and Llama-3.1-70B report pass-rate improvements over Program-of-Thoughts (PoT) baselines and prior agents, with a notable 8.8% gain on causal reasoning questions in QRData.

Significance. If the reported gains are real, the paper contributes a simple and practical method that improves data science agent performance at inference time without fine-tuning, and it highlights the role of problem ordering and online memory. The framework is clearly described, and the ablation comparing easy-to-hard, hard-to-easy, and random curricula (Tables 5 and 6) is a reasonable attempt to isolate the ordering effect. However, the transductive evaluation protocol and the absence of statistical validation mean that the evidence for the central claim is currently incomplete.

major comments (4)
  1. [§3.2 and §4.2] The long-term memory M_i in §3.2 stores all previously solved benchmark problems (p_1,...,p_{i-1}) with their generated code and pass/fail tags, so when solving problem i the agent sees up to i-1 other test-set questions. The PoT baselines in §4.2 and the published baselines in Tables 1 and 2 solve each question independently with no access to other benchmark problems. The comparison therefore conflates two factors: the growing same-benchmark memory and the curriculum ordering. The random-order rows in Tables 5 and 6 share the same memory mechanism and thus control only the ordering variable; they already exceed the PoT baselines (e.g., Claude-3.5-Sonnet on DSEval-Exercise: Random 0.775 vs PoT 0.745; on DSEval-Kaggle: 0.678 vs 0.641). This suggests that much of the headline gain over PoT baselines is retrieval augmentation from test-set examples rather than the easy-to-hard curriculum. To support the central claim, the paper should provide an inductive evaluation where memory is built only from problems outside the evaluation set, or a 'memory without curriculum' baseline that reveals the retrieval-only contribution.
  2. [§4.4.2, Tables 5 and 6] The claim that easy-to-hard outperforms random and hard-to-easy curricula rests on between-condition differences that are small (e.g., Llama-3.1-70b on DSEval-SO: 0.837 vs 0.803; Claude-3.5-Sonnet on DSEval-LeetCode: 0.892 vs 0.850) and no error bars or significance tests are reported. With three runs for easy-to-hard/hard-to-easy and five for random, and no variance information, these differences may be within run-to-run noise. On DSEval-Kaggle, hard-to-easy even exceeds easy-to-hard for both models (Llama: 0.583 vs 0.577; Claude: 0.680 vs 0.678). Please report per-run results and perform paired significance tests (or provide confidence intervals) before concluding that curriculum ordering is responsible for the improvement.
  3. [§4.1 and §4.4.3] The number of retrieved examples K is a per-dataset hyperparameter whose values are only given in the (missing) appendix, and Figure 4 shows that performance is sensitive to K, with some curves peaking and then declining. Selecting K on the evaluation data can inflate the reported pass rates and makes it difficult to assess the framework's robustness. Please report the chosen K for each dataset/model and provide results with a fixed K, or a sensitivity analysis that separates default from tuned settings.
  4. [§4.1] The manuscript states that certain DSEval evaluation issues were addressed 'per the appendix of the supplementary material,' but the appendix is not visible in the submission. Because the pass rates in Tables 1 and 2 are compared against published baselines that used the original evaluation, the modified evaluation makes the absolute numbers non-comparable. The appendix or a complete description of the modifications must be included for the results to be verifiable.
minor comments (5)
  1. [§4.3] 'shwon' should be 'shown' in the paragraph discussing online long-term memory.
  2. [Table 2] The header 'Multiple Choice/Numerical' is ambiguous; clarify whether the entries such as '0.460/0.540' are rates for each response type or a column split.
  3. [§4.4.3] The phrase 'incorrectly answered question with failed attempts' should be 'incorrectly answered questions with failed attempts'.
  4. [Figure 4] The text mentions performance trends for both models but does not describe the curves or legend; add a caption that identifies the datasets and whether the lines represent individual runs or averages.
  5. [§2 Related Work] The paper cites Reflexion (Shinn et al., 2024) as inspiration but does not discuss the relationship between the growing memory and Reflexion's episodic memory; a brief comparison would clarify the novelty of the memory mechanism.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported pass rates are measured, the curriculum effect is ablated with matched memory access, and no derived quantity reduces to a fitted input or self-citation chain.

full rationale

I walked the derivation chain and found no step where a prediction or claimed result is equivalent to its inputs by construction. The central quantities are measured pass rates on DSEval and QRData, obtained by executing generated code against ground truth; no reported number is computed from a fitted parameter. The curriculum ordering is produced by a Mentor LLM from problem descriptions (Section 3.1, Step 1) and is not fitted to the reported pass rates. The long-term memory (Section 3.2) stores previous problems, generated code, and evaluation tags, and retrieval is by cosine similarity; this is the intervention being tested, not a self-definitional target. Crucially, the easy-to-hard ordering is ablated against random and hard-to-easy curricula with the same memory mechanism (Tables 5 and 6), so the curriculum claim is not forced by construction. The comparison against memory-less PoT baselines conflates retrieval augmentation with curriculum, but that is an evaluation-protocol concern about external validity, not circularity: it does not make the measured gain equivalent to an input. No load-bearing self-citation chain appears; the cited baselines and prior curriculum-learning works are external and not used to define the reported improvements. The only design choices selected on evaluation data, such as the number K of retrieved examples, are hyperparameters and do not by construction force the headline improvements. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The method relies on assumptions about transferability of knowledge between tasks, the validity of the transductive benchmark protocol, and the usefulness of embeddings and failed examples. The only explicit free parameter is the retrieval count K, which is tuned per dataset.

free parameters (1)
  • K (number of retrieved examples) = 5 for DSEval-LeetCode, 15 for DSEval-SO (Table 3); varied up to 20 in Figure 4
    The number of memories retrieved per problem is a tunable hyperparameter. The paper selects values per dataset and shows performance varies with K in Figure 4, indicating tuning on the evaluation sets.
assumptions (4)
  • domain assumption Transductive use of benchmark tasks as memory is a valid evaluation protocol.
    Section 3.2 defines M_i from all previous problems in the same dataset, and Section 4.1 evaluates on DSEval/QRData as a single ordered sequence. This assumes that solving earlier test questions and retaining their code is legitimate and externally valid.
  • domain assumption LLM difficulty assessment from problem text alone correlates with true difficulty.
    Section 3.1 Step 1 relies on the Mentor agent's difficulty scoring. The ablation in Section 4.4.1 shows problem-based difficulty is competitive with pass-rate and reference-code metrics, providing partial support.
  • domain assumption Cosine similarity between problem descriptions is a good proxy for transferable knowledge.
    Section 3.2 Step 3 uses cos(E(p_i), E(p_k)) for retrieval. The paper gives no analysis of retrieval quality or whether semantically similar descriptions correspond to useful code examples.
  • domain assumption Including incorrectly solved examples with failed code helps the student.
    Section 3.2 stores both correct and incorrect attempts, and Table 7 supports the benefit empirically. The mechanism is not analyzed, and the risk of reinforcing errors is not discussed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DSMentor: Enhancing Data Science Agents with Curriculum Learning and Online Knowledge Accumulation." pith.science (2026). https://pith.science/paper/23KNAMF2

@misc{pith2026250514163,
  author       = {Pith},
  title        = {Pith review of: DSMentor: Enhancing Data Science Agents with Curriculum Learning and Online Knowledge Accumulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/23KNAMF2}},
  note         = {Machine review of arXiv:2505.14163}
}
read the original abstract

Large language model (LLM) agents have shown promising performance in generating code for solving complex data science problems. Recent studies primarily focus on enhancing in-context learning through improved search, sampling, and planning techniques, while overlooking the importance of the order in which problems are tackled during inference. In this work, we develop a novel inference-time optimization framework, referred to as DSMentor, which leverages curriculum learning -- a strategy that introduces simpler task first and progressively moves to more complex ones as the learner improves -- to enhance LLM agent performance in challenging data science tasks. Our mentor-guided framework organizes data science tasks in order of increasing difficulty and incorporates a growing long-term memory to retain prior experiences, guiding the agent's learning progression and enabling more effective utilization of accumulated knowledge. We evaluate DSMentor through extensive experiments on DSEval and QRData benchmarks. Experiments show that DSMentor using Claude-3.5-Sonnet improves the pass rate by up to 5.2% on DSEval and QRData compared to baseline agents. Furthermore, DSMentor demonstrates stronger causal reasoning ability, improving the pass rate by 8.8% on the causality problems compared to GPT-4 using Program-of-Thoughts prompts. Our work underscores the importance of developing effective strategies for accumulating and utilizing knowledge during inference, mirroring the human learning process and opening new avenues for improving LLM performance through curriculum-based inference optimization.

Figures

Figures reproduced from arXiv: 2505.14163 by the authors.

Figure 1
Figure 1. A motivating example illustrating how the agent can learn from previously solved easier [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Our framework DSMentor. Here, the Mentor agent assesses the difficulty of each problem and generates a curriculum accordingly. Once the curriculum are determined, the Student agent—responsible for answering questions through code generation—retrieves relevant examples from an accumulated online long-term memory. After the environment evaluates the generated code, the Student agent will append the question, its outpu… view at source ↗
Figure 3
Figure 3. Examples of determining difficulties during the curriculum-generation stage. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Performance of DSMentor models across different datasets on DSEval and QRData, with [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

13 extracted references · 5 canonical work pages

  1. [3]

    Liying Cheng, Xingxuan Li, and Lidong Bing

    ISSN 2835-8856. Liying Cheng, Xingxuan Li, and Lidong Bing. Is GPT-4 a good data analyst? InFindings of the Association for Computational Linguistics: EMNLP 2023, pp. 9496–9514,

    work page 2023

  2. [5]

    Data interpreter: An LLM agent for data science

    Sirui Hong, Yizhang Lin, Bangbang Liu, Binhao Wu, Danyang Li, Jiaqi Chen, Jiayi Zhang, Jinlin Wang, Lingyao Zhang, Mingchen Zhuge, et al. Data interpreter: An LLM agent for data science. arXiv preprint arXiv:2402.18679, 2024a. Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zi...

    arXiv

  3. [6]

    Are llms capable of data-based statistical and causal reasoning? benchmarking advanced quantitative reasoning with data.arXiv preprint arXiv:2402.17644, 2024a

    Xiao Liu, Zirui Wu, Xueqing Wu, Pan Lu, Kai-Wei Chang, and Yansong Feng. Are llms capable of data-based statistical and causal reasoning? benchmarking advanced quantitative reasoning with data.arXiv preprint arXiv:2402.17644, 2024a. Yinpeng Liu, Jiawei Liu, Xiang Shi, Qikai Cheng, and Wei Lu. Let’s learn step by step: Enhancing in-context learning ability...

    arXiv

  4. [10]

    Haoran Sun, Lixin Liu, Junjie Li, Fengyu Wang, Baohua Dong, Ran Lin, and Ruohui Huang

    URLhttps://github.com/shroominic/ codeinterpreter-api. Haoran Sun, Lixin Liu, Junjie Li, Fengyu Wang, Baohua Dong, Ran Lin, and Ruohui Huang. Conifer: Improving complex constrained instruction-following ability of large language models. arXiv preprint arXiv:2404.02823,

    arXiv

  5. [13]

    Benchmarking data science agents.arXiv preprint arXiv:2402.17168, 2024b

    Yuge Zhang, Qiyang Jiang, Xingyu Han, Nan Chen, Yuqing Yang, and Kan Ren. Benchmarking data science agents.arXiv preprint arXiv:2402.17168, 2024b. 13

    arXiv

  6. [2009]

    Generating Multidimensional Clusters With Support Lines

    Soham Biswas, Sukanta Bose, and Bhaskar Mukherjee. Is ChatGPT a good data scientist? a pre- liminary study.arXiv preprint arXiv:2301.10327,

    work page Pith review arXiv

  7. [2017]

    Plug and play language models: A simple approach to controlled text generation.arXiv preprint arXiv:1912.02164,

    Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosin- ski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation.arXiv preprint arXiv:1912.02164,

    arXiv 1912

  8. [2018]

    Competence-based curriculum learning for neural machine translation

    Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnab ´as Pocz ´os, and Tom Mitchell. Competence-based curriculum learning for neural machine translation. InProceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 1162–1172,

    work page 2019

Show all 13 references
  1. [2019]

    Automatic curriculum learning for deep rl: A short survey

    R´emy Portelas, C ´edric Colas, Lilian Weng, Katja Hofmann, and Pierre-Yves Oudeyer. Automatic curriculum learning for deep rl: A short survey. InIJCAI 2020-International Joint Conference on Artificial Intelligence,

    2020

  2. [2021]

    Schmidt, Z

    D. Schmidt, Z. Jiang, and Y . Wu. Aide: Human-level performance in data science competitions. weco.ai, 2024.https://www.weco.ai/blog/technical-report. Lucy Xiaoyang Shi, Yunfan Jiang, Jake Grigsby, Linxi ”Jim” Fan, and Yuke Zhu. Cross-episodic curriculum for transformer agents...

    2024

  3. [2022]

    Mlcopilot: Unleashing the power of large language models in solving machine learning tasks

    Lei Zhang, Yuge Zhang, Kan Ren, Dongsheng Li, and Yuqing Yang. Mlcopilot: Unleashing the power of large language models in solving machine learning tasks. InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: L...

    arXiv

  4. [2023]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al

    URLhttps://github.com/chapyter/chapyter. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.033...

    arXiv

  5. [2024]

    Haotian Wang, Xiyuan Du, Weijiang Yu, Qianglong Chen, Kun Zhu, Zheng Chu, Lian Yan, and Yi Guan

    ISSN 2835-8856. Haotian Wang, Xiyuan Du, Weijiang Yu, Qianglong Chen, Kun Zhu, Zheng Chu, Lian Yan, and Yi Guan. Apollo’s oracle: Retrieval-augmented reasoning in multi-agent debates.arXiv preprint arXiv:2312.04854,

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.