pith. sign in

arxiv: 2502.00955 · v2 · submitted 2025-02-02 · 💻 cs.CL

Efficient Multi-Agent System Training with Data Influence-Oriented Tree Search

Wentao Shi , Zichun Yu , Fuli Feng , Xiangnan He , Chenyan Xiong This is my paper

Pith reviewed 2026-05-23 04:01 UTC · model grok-4.3

classification 💻 cs.CL
keywords multi-agent systemsinfluence scoresMonte Carlo Tree Searchsynthetic data generationLLM trainingdata selectionnon-differentiable metrics
0
0 comments X

The pith

Influence scores guide tree search to select synthetic data that improves multi-agent LLM training more effectively than Q-values.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes Data Influence-oriented Tree Search (DITS) as a framework for generating synthetic data to train large language model based multi-agent systems. It replaces sole reliance on Q-values from Monte Carlo Tree Search with influence scores that directly target data points most likely to enhance model performance after inclusion. Influence scores for non-differentiable metrics are estimated using only forward inference passes, which cuts computational cost. Experiments on eight multi-agent datasets confirm performance gains, and the work finds that directing more inference effort toward influence estimation rather than Q-value computation produces stronger training results. This approach resolves a mismatch between the objective of data synthesis and the signals traditionally used to guide it.

Core claim

The paper establishes that incorporating influence scores to steer both the tree expansion in search and the final selection of synthetic trajectories yields data that produces larger gains in multi-agent system capability than Q-value guidance alone, with dedicated inference-only estimators derived for non-differentiable evaluation metrics that avoid gradient computation.

What carries the argument

Data Influence-oriented Tree Search (DITS), a tree-search procedure that substitutes influence scores for Q-values in both node selection and data retention decisions.

If this is right

  • Synthetic data generated under DITS produces measurable gains in multi-agent system performance on held-out tasks.
  • Inference-only computation of influence scores lowers the cost of each search iteration relative to methods that require gradients.
  • Shifting inference budget toward influence estimation and away from Q-value estimation improves training outcomes.
  • The framework remains effective across eight distinct multi-agent datasets without task-specific tuning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same influence-score substitution might be tested in single-agent or non-LLM search-based data synthesis pipelines.
  • Hybrid scoring that blends limited Q-value information with influence scores could be evaluated as a low-cost middle ground.
  • Resource allocation rules derived here may apply to other inference-heavy loops such as self-play or evolutionary training.

Load-bearing premise

Influence scores estimated from inference-only runs on non-differentiable metrics correctly rank data points by their expected contribution to downstream multi-agent performance.

What would settle it

Training runs in which data chosen by standard Q-value MCTS produces equal or higher downstream multi-agent task scores than data chosen by DITS on the same eight datasets.

Figures

Figures reproduced from arXiv: 2502.00955 by Chenyan Xiong, Fuli Feng, Wentao Shi, Xiangnan He, Zichun Yu.

Figure 1
Figure 1. Figure 1: (a) The scatter plot and density plots of Q-values and influence scores for synthetic data. The top 30% of the data se￾lected using DITS is highlighted in red. (b) Performance trends with different data synthesis budgets (Tokens). tools and environments to accomplish various tasks (Chen et al., 2023; Yao et al., 2023). Nevertheless, individual agents often face significant limitations when confronted with … view at source ↗
Figure 2
Figure 2. Figure 2: Overview of our method. (a) illustrates the traversal of a cyclic agent network in topological order. We introduce virtual agents to distinguish the same agent in the traversal. (b) showcases the application of MCTS to generate synthetic multi-agent training data, where the color of each agent represents the magnitude of the node’s Q-value. (c) depicts the computation process of influence scores for a non-… view at source ↗
Figure 3
Figure 3. Figure 3: The scatter plot and density plots of Q-values and influence scores for the synthetic data. The top 30% of the data selected by DITS is highlighted in red. data selection. For a fair comparison, we set the selection ratio as 50% for all methods. As shown in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The effect of hyperparameter selection ratio α on the performance of DITS on the 2WMH QA and TrivalQA datasets. 5.3. Synthesis Time Scaling In this study, we empirically demonstrate that increasing the synthesis budget during the data synthesis phase enhances model performance, as shown in [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: The distribution of synthetic data influence scores across different iterations on the HotpotQA and MMLU datasets, with the mean of the distribution highlighted by a red dashed line. To gain deeper insights into the iterative data synthesis and training process, we analyzed the distribution of influence scores for synthetic data across different iterations on the HotpotQA and MMLU datasets, as shown in [P… view at source ↗
Figure 5
Figure 5. Figure 5: The relative performance improvement of DITS-iSFT￾DPO across all datasets at different iterations. The best perfor￾mance of each dataset is set as 1.0. conduct experiments on two Information Exchange tasks: 2WMH QA and Trival QA datasets and present the results in [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: The scatter plot and density plots of Q-values and influence scores for synthetic data. The top 30% of the data selected by DITS is highlighted in red. B. Training Details The hyperparameters we used are shown in [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
read the original abstract

Monte Carlo Tree Search (MCTS) based methods provide promising approaches for generating synthetic data to enhance the self-training of Large Language Model (LLM) based multi-agent systems (MAS). These methods leverage Q-values to estimate individual agent contributions. However, relying solely on Q-values to identify informative data may misalign with the data synthesis objective, as the focus should be on selecting data that best enhances model training. To address this discrepancy, we propose Data Influence-oriented Tree Search (DITS), a novel framework that incorporates influence scores to guide both tree search and data selection. By leveraging influence scores, we effectively identify the most impactful data for system improvement, thereby enhancing model performance. Furthermore, we derive influence score estimation methods tailored for non-differentiable metrics, significantly reducing computational overhead by utilizing inference computations. Extensive experiments on eight multi-agent datasets demonstrate the robustness and effectiveness of the proposed methods. Notably, our findings reveal that allocating more inference resources to estimate influence scores, rather than Q-values, during data synthesis can more effectively and efficiently enhance model training.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper proposes Data Influence-oriented Tree Search (DITS), a framework that replaces or augments Q-value guidance in MCTS-based synthetic data generation for LLM multi-agent systems with influence scores. The central claim is that influence scores better align data synthesis with the objective of improving downstream model training; the authors derive inference-only estimators for these scores on non-differentiable metrics to lower compute cost, and report that experiments on eight multi-agent datasets show robustness, with the key empirical finding that allocating more inference budget to influence estimation than to Q-values yields more effective training gains.

Significance. If the influence estimator is shown to rank data points by actual contribution to MAS performance gains more reliably than Q-values, the approach could improve the efficiency of self-training pipelines by redirecting inference compute toward higher-impact data selection. The inference-only formulation for non-differentiable metrics would be a practical strength if it avoids gradient requirements while preserving ranking quality.

major comments (3)
  1. [Abstract / §3] Abstract and §3 (method): the claim that the inference-only influence estimator 'accurately identify[s] the most impactful data' requires explicit validation that its scores correlate with downstream performance deltas; without the estimator definition, any bias analysis (e.g., over-weighting easy trajectories or ignoring agent-interaction effects), or ablation showing larger gains than Q-value baselines, the realignment argument remains unverified.
  2. [§4] §4 (experiments): the statement that 'allocating more inference resources to estimate influence scores, rather than Q-values' is more effective needs quantitative support—specific tables or figures must report performance deltas, compute budgets, and statistical significance across the eight datasets; absent these numbers the efficiency claim cannot be assessed.
  3. [§3.2] §3.2 (influence estimation): the derivation for non-differentiable metrics must be checked for whether it reduces to a fitted or heuristic quantity; if the estimator introduces systematic ranking errors relative to true leave-one-out impact, the superiority over Q-value guidance does not follow.
minor comments (2)
  1. [Abstract] Abstract: the eight datasets are not named and no numerical results (accuracy, win rates, or relative gains) are supplied, making it impossible to gauge effect sizes from the summary alone.
  2. [§2 / §3] Notation: 'influence scores' and 'Q-values' should be defined with explicit formulas at first use to allow readers to compare the two guidance signals directly.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their thoughtful review and the opportunity to clarify our contributions. We address each major comment below.

read point-by-point responses

  1. Referee: [Abstract / §3] Abstract and §3 (method): the claim that the inference-only influence estimator 'accurately identify[s] the most impactful data' requires explicit validation that its scores correlate with downstream performance deltas; without the estimator definition, any bias analysis (e.g., over-weighting easy trajectories or ignoring agent-interaction effects), or ablation showing larger gains than Q-value baselines, the realignment argument remains unverified.

    Authors: The estimator definition is provided in §3.2, where we derive the inference-only method for non-differentiable metrics. In §4, we present ablations on eight datasets comparing DITS to Q-value guided MCTS, showing consistent larger gains in model performance. While we do not include explicit correlation coefficients between influence scores and performance deltas, the empirical results across multiple datasets support that influence scores better identify impactful data. We can include additional bias analysis in a revision. revision: partial

  2. Referee: [§4] §4 (experiments): the statement that 'allocating more inference resources to estimate influence scores, rather than Q-values' is more effective needs quantitative support—specific tables or figures must report performance deltas, compute budgets, and statistical significance across the eight datasets; absent these numbers the efficiency claim cannot be assessed.

    Authors: Section §4 includes tables reporting performance on the eight datasets under different resource allocations, with performance deltas relative to baselines. Compute budgets are discussed in terms of inference calls for influence vs Q-value estimation. We report mean improvements and note consistency across datasets as evidence of robustness; statistical significance testing can be added if the referee deems it necessary. revision: partial

  3. Referee: [§3.2] §3.2 (influence estimation): the derivation for non-differentiable metrics must be checked for whether it reduces to a fitted or heuristic quantity; if the estimator introduces systematic ranking errors relative to true leave-one-out impact, the superiority over Q-value guidance does not follow.

    Authors: The derivation in §3.2 starts from the influence function concept and adapts it to an inference-only estimator by approximating the impact on the loss without requiring differentiability or gradients. It is not a simple heuristic but follows from the definition of data influence. Empirical validation through superior performance over Q-value methods on the datasets indicates it does not introduce detrimental ranking errors. Full leave-one-out computation is computationally prohibitive, which is why the approximation is used. revision: no

Circularity Check

0 steps flagged

No significant circularity; derivation relies on empirical validation rather than self-referential reduction

full rationale

The paper proposes DITS as a framework that replaces Q-value guidance with influence scores for tree search and data selection in MAS training. It describes deriving influence estimation methods for non-differentiable metrics that use only inference computations. Central claims are validated through experiments on eight datasets showing improved performance when allocating inference resources to influence scores. No equations, self-definitional constructs, fitted parameters renamed as predictions, or load-bearing self-citations appear in the abstract or described chain. The result does not reduce to its inputs by construction and remains self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only input supplies no explicit free parameters, axioms, or invented entities; ledger is therefore empty.

pith-pipeline@v0.9.0 · 5719 in / 1130 out tokens · 77930 ms · 2026-05-23T04:01:33.947682+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 23 canonical work pages · 7 internal anchors

  1. [1]

    Training data attribution via approximate unrolled differentiation

    Bae, J., Lin, W., Lorraine, J., and Grosse, R. Training data attribution via approximate unrolled differentiation. CoRR, abs/2405.12186,

    work page arXiv

  2. [2]

    D., Richardson, K., Sabharwal, A., Schoenick, C., Tafjord, O., and Clark, P

    Bhakthavatsalam, S., Khashabi, D., Khot, T., Mishra, B. D., Richardson, K., Sabharwal, A., Schoenick, C., Tafjord, O., and Clark, P. Think you have solved direct-answer question answering? try arc-da, the direct-answer AI2 reasoning challenge. CoRR, abs/2102.03315,

    work page arXiv

  3. [3]

    Fireact: Toward language agent fine-tuning

    Chen, B., Shu, C., Shareghi, E., Collier, N., Narasimhan, K., and Yao, S. Fireact: Toward language agent fine-tuning. CoRR, abs/2310.05915,

    work page arXiv

  4. [4]

    Beyond natural language: Llms leveraging alternative formats for enhanced reasoning and communication

    Chen, W., Yuan, C., Yuan, J., Su, Y ., Qian, C., Yang, C., Xie, R., Liu, Z., and Sun, M. Beyond natural language: Llms leveraging alternative formats for enhanced reasoning and communication. In EMNLP (Findings), pp. 10626–10641. Association for Computational Linguistics, 2024a. Chen, W., Yuan, J., Qian, C., Yang, C., Liu, Z., and Sun, M. Optima: Optimizi...

    work page arXiv

  5. [5]

    rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking

    URL https://arxiv.org/abs/2501.04519. Guo, T., Chen, X., Wang, Y ., Chang, R., Pei, S., Chawla, N. V ., Wiest, O., and Zhang, X. Large language model based multi-agents: A survey of progress and challenges. In IJCAI, pp. 8048–8057. ijcai.org,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines

    Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., San- thanam, K., Vardhamanan, S., Haq, S., Sharma, A., Joshi, T. T., Moazam, H., Miller, H., Zaharia, M., and Potts, C. Dspy: Compiling declarative language model calls into self-improving pipelines. CoRR, abs/2310.03714,

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    Crafting papers on machine learning

    Langley, P. Crafting papers on machine learning. In Langley, P. (ed.),Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207–1216, Stan- ford, CA,

    work page 2000

  8. [8]

    More agents is all you need

    Li, J., Zhang, Q., Yu, Y ., Fu, Q., and Ye, D. More agents is all you need. CoRR, abs/2402.05120, 2024a. Li, S., Dong, S., Luan, K., Di, X., and Ding, C. Enhancing reasoning through process supervision with monte carlo tree search,

    work page arXiv

  9. [9]

    Li, X., Yu, Z., and Xiong, C

    URL https://arxiv.org/abs/ 2501.01478. Li, X., Yu, Z., and Xiong, C. Montessori-instruct: Gener- ate influential training data tailored for student learning. CoRR, abs/2410.14208, 2024b. Liang, T., He, Z., Jiao, W., Wang, X., Wang, Y ., Wang, R., Yang, Y ., Shi, S., and Tu, Z. Encouraging divergent thinking in large language models through multi-agent deb...

    work page arXiv

  10. [10]

    Motwani, S

    URL https://arxiv.org/abs/ 2501.05790. Motwani, S. R., Smith, C., Das, R. J., Rybchuk, M., Torr, P. H. S., Laptev, I., Pizzati, F., Clark, R., and de Witt, C. S. MALT: improving reasoning with multi-agent LLM training. CoRR, abs/2412.01928,

    work page arXiv

  11. [11]

    GPT-4 Technical Report

    OpenAI. GPT-4 technical report. CoRR, abs/2303.08774,

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [12]

    Y ., Yuan, W., Cho, K., He, H., Sukhbaatar, S., and Weston, J

    Pang, R. Y ., Yuan, W., Cho, K., He, H., Sukhbaatar, S., and Weston, J. Iterative reasoning preference optimization. CoRR, abs/2404.19733,

    work page arXiv

  13. [13]

    L., Yang, F., and Yang, M

    Qi, Z., Ma, M., Xu, J., Zhang, L. L., Yang, F., and Yang, M. Mutual reasoning makes smaller llms stronger problem- solvers. CoRR, abs/2408.06195,

    work page arXiv

  14. [14]

    Itera- tive experience refinement of software-developing agents

    Qian, C., Li, J., Dang, Y ., Liu, W., Wang, Y ., Xie, Z., Chen, W., Yang, C., Zhang, Y ., Liu, Z., and Sun, M. Itera- tive experience refinement of software-developing agents. CoRR, abs/2405.04219, 2024a. Qian, C., Liu, W., Liu, H., Chen, N., Dang, Y ., Li, J., Yang, C., Chen, W., Su, Y ., Cong, X., Xu, J., Li, D., Liu, Z., and Sun, M. Chatdev: Communicat...

    work page arXiv

  15. [15]

    B., Finn, C., and Niekum, S

    Rafailov, R., Chittepu, Y ., Park, R., Sikchi, H., Hejna, J., Knox, W. B., Finn, C., and Niekum, S. Scaling laws for reward model overoptimization in direct alignment algorithms. CoRR, abs/2406.02900,

    work page arXiv

  16. [16]

    Large language models are learnable planners for long-term recommendation

    Shi, W., He, X., Zhang, Y ., Gao, C., Li, X., Zhang, J., Wang, Q., and Feng, F. Large language models are learnable planners for long-term recommendation. In SIGIR, pp. 1893–1903. ACM, 2024a. Shi, W., Yuan, M., Wu, J., Wang, Q., and Feng, F. Direct multi-turn preference optimization for language agents. In EMNLP, pp. 2312–2324. Association for Computa- ti...

    work page arXiv 1903

  17. [17]

    Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters

    Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. CoRR, abs/2408.03314,

    work page internal anchor Pith review Pith/arXiv arXiv

  18. [18]

    Multi-Agent Collaboration Mechanisms: A Survey of LLMs

    URL https://arxiv.org/abs/2501.06322. Wang, J., Wang, J., Athiwaratkun, B., Zhang, C., and Zou, J. Mixture-of-agents enhances large language model capabilities. CoRR, abs/2406.04692, 2024a. Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y ., Zhao, W. X., Wei, Z., and Wen, J. A survey on large language model ...

    work page internal anchor Pith review Pith/arXiv arXiv

  19. [19]

    AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation

    Wu, Q., Bansal, G., Zhang, J., Wu, Y ., Zhang, S., Zhu, E., Li, B., Jiang, L., Zhang, X., and Wang, C. Autogen: Enabling next-gen LLM applications via multi-agent conversation framework. CoRR, abs/2308.08155,

    work page internal anchor Pith review Pith/arXiv arXiv

  20. [20]

    URL https://arxiv.org/abs/ 2408.00724. Xi, Z., Chen, W., Guo, X., He, W., Ding, Y ., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., Zheng, R., Fan, X., Wang, X., Xiong, L., Zhou, Y ., Wang, W., Jiang, C., Zou, Y ., Liu, X., Yin, Z., Dou, S., Weng, R., Cheng, W., Zhang, Q., Qin, W., Zheng, Y ., Qiu, X., Huang, X., and Gui, T. The rise and potential of l...

    work page internal anchor Pith review Pith/arXiv arXiv

  21. [21]

    P., Kawaguchi, K., and Shieh, M

    Xie, Y ., Goyal, A., Zheng, W., Kan, M., Lillicrap, T. P., Kawaguchi, K., and Shieh, M. Monte carlo tree search boosts reasoning via iterative preference learning. CoRR, abs/2405.00451,

    work page arXiv

  22. [22]

    Large language model-brained gui agents: A survey, 2024a

    Zhang, C., He, S., Qian, J., Li, B., Li, L., Qin, S., Kang, Y ., Ma, M., Liu, G., Lin, Q., Rajmohan, S., Zhang, D., and Zhang, Q. Large language model-brained gui agents: A survey, 2024a. 11 Efficient Multi-Agent System Training with Data Influence-Oriented Tree Search Zhang, D., Huang, X., Zhou, D., Li, Y ., and Ouyang, W. Accessing GPT-4 level mathemati...

    work page arXiv

  23. [23]

    "Film"",

    Case study to demonstrate the data selected by Q-value and influence score on 2WMH QA dataset. Question Which film has the director who was born later, Eyes Of The Forest or Stardust On The Sage? Q-value Select Alice: [ [""Film"", ""Eyes of the Forest""], [""Director"", ""Lambert Hillyer""], [""Birth Date"", ""July 8, 1893""], [""Death Date"", ""July 5, 1...

    work page 1969