Pith. sign in

REVIEW 3 major objections 4 minor 51 references

A frozen model's hidden states carry enough signal to decide when it should defer, clarify, use tools, or abstain, cutting large-model usage by up to 90.7 percent.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 02:34 UTC pith:IKMXN3HX

load-bearing objection A genuinely useful latent-control paper with a clean two-head decomposition and broad multi-family evaluation, but the Resolution Head's judge-derived labels and the TriviaQA train/eval overlap need to be fixed before the headline numbers can be trusted. the 3 major comments →

arxiv 2607.14277 v1 pith:IKMXN3HX submitted 2026-07-15 cs.CL

Multi-Head Latent Control: A Unified Interface for LLM Agent Decision Making

Amirhosein Ghasemabadi , Ruichen Chen , Bahador Rashidi , Di Niu This is my paper
classification cs.CL
keywords latent controlLLM agentsmodel routingcapability predictiontool-use decisionshidden statesfrozen modelscost-quality tradeoff
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that deployment-time control for LLM agents does not need external prompt routing or fine-tuning: the hidden-state trajectory of a frozen model already contains enough signal to decide whether the current model can handle an instance or should defer to a stronger model, and whether it should ask for clarification, call a tool, or abstain. It introduces two lightweight heads, a Capability Head and a Resolution Head, trained only on hidden-state traces from the same frozen backbone. If this is right, agent systems can route between a small local model and a large paid model at per-step granularity and keep most of the large model's quality at much lower cost. The paper reports reductions in paid large-model usage of up to 90.7 percent on a long-horizon Android-agent benchmark and 27-53 percent average cost reduction across benchmarks, plus better tool-use decisions with up to +158.9 percent relative score gain and 65.5 percent fewer missed-needed tool calls.

Core claim

The central claim is that a frozen language or vision-language model's latent generation process is a usable control surface. The paper defines Multi-Head Latent Control as two small heads attached to the frozen backbone: a Capability Head that reads the final-layer hidden-state trace and emits a scalar adequacy score, and a Resolution Head that reads a middle-layer trace and scores three interventions (request information, use tools, abstain), with direct answering represented implicitly. Both heads are trained on hidden-state traces collected from the same frozen model, supervised by adequacy labels from an LLM judge and by resolution labels derived from a tool-calling benchmark. The empir

What carries the argument

The central object is the pair of latent control heads. Each head takes a hidden-state trajectory from the frozen backbone, compresses it to a fixed-size representation through a projection encoder, and maps it to a control score: the Capability Head outputs a scalar adequacy score from the final-layer trace, and the Resolution Head outputs a three-way score vector from a middle-layer trace. The inference-time policy sits on top of these scores: defer to the stronger model when the adequacy score falls below a fixed threshold, and trigger an intervention only when the top resolution score exceeds a second threshold. The layer choices and thresholds are empirical; the load-bearing idea is tha

Load-bearing premise

Both control heads are trained on labels produced by an external LLM judge rather than on verified ground truth; if that judge is noisy or systematically disagrees with what actually makes a response correct, the learned control policy is distorted and the reported cost and tool-use gains will not transfer.

What would settle it

Retrain both heads on the same benchmarks using verified ground-truth correctness labels or human annotations instead of the LLM judge's scores, then compare routed cost, score, and resolution accuracy. If the head's performance collapses, or if a sampled check shows the judge's labels agree poorly with ground truth, the claim that latent traces reliably carry deployment control signals is not supported.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A frozen model can be given control behavior without fine-tuning, so a new backbone generation can be deployed with control heads retrained in about a day on a single GPU using roughly 16 GB of memory.
  • Per-step routing in long-horizon agent tasks avoids the large model except when adequacy dips, as in the Android-agent run that cut paid API cost by 90.7 percent while improving success.
  • Early handoff is feasible: a Capability Head trained on 200-token prefixes recovers most of the full-trajectory routing benefit, letting failing trajectories be stopped before completion.
  • Tool escalation decisions improve even for strong models: on a web-search test, a 32B model's missed-needed web calls fell from 328 to 222 with the control head.
  • The same latent substrate supports four resolution behaviors (defer, clarify, use tools, abstain) with one architecture, so control is a single interface rather than per-task fine-tuning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the adequacy signal transfers across backbone families, a control head trained on a diverse mixture of models and tasks could serve as a universal handoff layer, removing the need to recollect traces for each new model.
  • Editorial inference: the same latent-reading mechanism could be applied inside a single generation to abort or revise an action, not just at completion or a fixed prefix; the paper stops at the 200-token prefix regime, leaving mid-trajectory correction untested.
  • Editorial inference: the reported cost savings assume the small model runs locally for free; deployments with two paid APIs would need to re-tune the deferral threshold against actual relative prices, and the headline 27-53 percent savings would shrink.
  • Editorial inference: the resolution action space of info, tool, and abstain could be extended to other interventions such as safety refusal or switching tool sets; the paper notes this direction but does not test it, so it remains an open extension.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper proposes Multi-Head Latent Control, a post hoc control layer for frozen LLMs/VLMs. A Capability Head reads final-layer hidden-state traces from a frozen primary model and predicts a scalar adequacy score; a Resolution Head reads a middle-layer trace and predicts one of info/tool/cant actions (direct answering is implicit). Both heads are trained with the backbone frozen, using supervision generated by an external LLM judge. At inference, capability scores route instances to a stronger fallback model; resolution scores trigger clarification, tool use, or abstention. Experiments report routing cost-quality tradeoffs on six benchmarks plus AndroidWorld, resolution decisions on WHEN2CALL, web-tool decisions on TriviaQA, and prefix-time capability prediction.

Significance. If the results hold, the contribution is significant: it demonstrates that frozen-model latent traces can support lightweight, transferable control decisions, with large API cost reductions and modest quality loss, and it offers a practical alternative to prompt routers and fine-tuned judge models. The paper includes useful ablations (layer choice, prefix length, training-mixture breadth), compares against prompt-level self-switching and token-confidence baselines, releases code, and evaluates across three model families. The main caveat is that both supervision signals come from a single unvalidated LLM judge, and one evaluation task overlaps the training mixture; these threaten the external validity of the headline numbers and need to be addressed before the claims can be accepted at face value.

major comments (3)
  1. [§4.3 / App. A.1] TriviaQA is listed in Appendix A.1 as a Capability Head training source, yet Table 4 evaluates web-tool decision quality on TriviaQA. The reported improvements (e.g., +158.9% relative score for Qwen3-VL-4B) may partly reflect the head having seen these instances during training rather than learning transferable adequacy/tool-escalation decisions. Please either remove TriviaQA from the training mixture and retrain, or evaluate on a disjoint held-out split and report the overlap statistics.
  2. [§3.4.1–3.4.2, App. A.2] Both control heads are supervised entirely by the Qwen3-VL-30B-A3B judge: the Capability Head's scalar y_cap is the judge's adequacy score, and the Resolution Head's 'gold' labels are converted from WHEN2CALL's preference-oriented data by the same judge. No agreement analysis against human labels, verified ground truth, or actual m2 improvement is reported. Table 3's F1/accuracy is computed against the same judge-derived label construction used for training, so the evaluation can be self-consistent without being correct. Please add judge–human agreement (e.g., Cohen's kappa on a labeled sample) and/or re-evaluate routing using labels based on whether the fallback model actually improves the outcome.
  3. [§4.2, Table 3] The text says latent control 'consistently improves' resolution decisions, but Table 3 shows a decrease for Qwen-VL-2B-Thk (F1 55.1→54.5, Acc 73.6→69.5), and the Resolution Head row appears to omit Gemma-2B-Thk entirely. This makes the 'consistently' claim unsupported as presented. Please correct the table and qualify the claim, or explain the discrepancy.
minor comments (4)
  1. [§3.3] Typo: 'stornger' should be 'stronger'.
  2. [Table 1 / Table 2] All scores are point estimates without variance or significance testing. For AndroidWorld, the routed success (0.60) is only 0.02 above the always-large baseline (0.58); a sentence on run-to-run variability would help.
  3. [App. D] Gemma API costs are estimated using Qwen3-VL pricing as a proxy. This is a reasonable approximation but should be stated in the main text next to the cost tables, not only in the appendix.
  4. [§4.2] The comparison against MNM 4B/8B When2Call-SFT baselines is informative but not apples-to-apples, since those are full fine-tuned models. Please make this explicit in the text near Table 3.

Circularity Check

2 steps flagged

Central routing results are externally evaluated, but the TriviaQA tool-use gain is fitted on the evaluation benchmark and the Resolution Head is scored against the same judge-derived labels used as its training target.

specific steps
  1. fitted input called prediction [Appendix A.1 (training mixture) / §4.3, Table 4]
    "For each prompt, we query the frozen backbone model, record its generated output and aligned hidden-state trajectory, and score the generation against the task reference using an external LLM-based evaluator. The resulting scalar score in [0,1] is used as the supervision target for the Capability Head. ... Training sources. The mixture includes the following datasets. ... TriviaQA ... Finally, we study TriviaQA as a concrete external-action setting for web-search decision quality."

    The Capability Head that drives the web-search escalation policy is trained on adequacy labels computed for TriviaQA prompts, and TriviaQA is explicitly part of the training mixture. The web-search evaluation then reports the same head's +158.9% score gain and 65.5% fewer missed calls on TriviaQA. The head has been fitted to the very benchmark used to claim the predictive gain, so the TriviaQA result is partly memorization of which questions are hard rather than a transfer prediction. This is a fitted-input-called-prediction setup for that headline claim.

  2. other [Appendix A.2 / §3.4.2 / §4.2, Table 3]
    "We therefore apply an external judge LLM offline to derive a gold resolution decision label for each example in the action space A={info,tool,cant}... Table 3 reports F1 score and accuracy."

    The Resolution Head is trained with BCE loss against judge-derived labels (Eq. 4), and its F1/accuracy in Table 3 is computed against those same judge-derived 'gold' labels. The paper calls the recovered decisions 'correct resolution decision(s),' but correctness is defined by the judge's preference conversion, not by verified ground truth. The evaluation is therefore self-consistent with the training target: the reported improvement measures agreement with the judge's annotation policy, and no judge-agreement analysis is provided to show the labels correspond to actual tool-use necessity. This makes the WHEN2CALL result a measure of fit to the labeling procedure rather than an externally validated prediction.

full rationale

The paper is not globally circular: the central routed-execution claims (AndroidWorld 90.7% cost reduction, 27-53% average savings on the six-benchmark suite) are evaluated on external task scores and measured token/cost usage, not on the Capability Head's own training labels. The head is a supervised regressor on hidden-state traces, so the routing curve in Fig. 2 is an empirical tradeoff, and the author-chosen 0.8/0.5 thresholds are disclosed. The encoder reuse from the authors' prior [12] is a minor self-citation and is not load-bearing (no uniqueness theorem or ansatz is imported). The two flagged issues are localized: (1) TriviaQA is in the Capability Head training mixture and is then used as the web-search evaluation benchmark, inflating the +158.9% claim by construction; (2) the Resolution Head's 'gold' labels are judge-derived, and its F1/accuracy is measured against those same judge-derived labels, with no judge-agreement analysis, so the WHEN2CALL improvement is self-consistent with the labeling policy rather than validated against independent ground truth. These are partial 'prediction reduces to fit' issues in two headline numbers, while the main routing findings remain externally grounded, supporting a moderate score rather than a high one.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 2 invented entities

The method rests on three empirical premises: hidden states carry adequacy/resolution information, the judge's labels are valid supervision, and the cost model approximates reality. The thresholds and layer choices are hand-selected operating points. No new physical entities are postulated; the two heads are trainable components with in-paper benchmark evidence.

free parameters (7)
  • Capability threshold τcap = 0.8
    Fixed operating point (§3.3); head hands off to m2 only when pcap < 0.8. Threshold chosen by authors; Figure 2 shows a tradeoff curve over 0.5–0.9.
  • Resolution threshold τres = 0.5
    Intervention triggered only when top resolution score exceeds 0.5 (§3.3). No validation protocol described.
  • Head layer selections = final layer (Capability), middle layer (Resolution)
    Selected empirically from layer ablations in Appendix C.1/C.2.
  • Judge model for all labels = Qwen3-VL-30B-A3B
    Used to score Capability training targets (§3.4.1) and derive WHEN2CALL gold labels (App. A.2); no judge-agreement analysis.
  • Capability training mixture composition = 120K examples across ~15 datasets
    Mixture breadth deliberately chosen (App. A.1); proportions and inclusion of TriviaQA affect transfer and in-domain eval.
  • Prefix length 200 for prefix-time variants = 200 tokens
    Chosen for the prefix-time experiments (Tables 5–6); other lengths in App. C.4.
  • Loss reweighting weights wi = unspecified
    Weighted MSE to compensate for training imbalance (§3.4.1); exact weights not given.
axioms (5)
  • domain assumption Hidden states of a frozen model encode information about the adequacy of its (possibly incorrect) output for the current instance.
    The entire method reads control signals from latent traces (§3.4: 'such information may still be encoded in the hidden-state trajectory').
  • domain assumption The LLM judge's scalar adequacy scores are valid supervision targets for 'can this model solve this instance?'.
    All Capability labels come from an external judge (§3.4.1), with no agreement analysis against human labels.
  • domain assumption Judge-derived WHEN2CALL labels are valid gold resolution decisions (info/tool/cant).
    Appendix A.2: the released WHEN2CALL labels are preference-oriented, so gold labels are derived by a judge; if the judge disagrees with the benchmark's definition, the head learns a distorted target.
  • domain assumption Estimated API cost is a valid proxy (Gemma cost approximated with Qwen pricing; m1 assumed free/local).
    Appendix D: Gemma has no comparable public pricing, so Qwen3-VL pricing is used as a proxy; all routed cost savings assume the small model runs locally.
  • domain assumption The fixed encoder architecture (from Gnosis [12]) extracts features sufficient for both heads.
    §3.2: encoder is kept fixed from [12]; no evidence is given that this specific encoder is optimal for resolution decisions.
invented entities (2)
  • Capability Head independent evidence
    purpose: Predicts a scalar adequacy score pcap from final-layer hidden traces for routing/deferral decisions.
    Evaluated empirically via ROC-AUC/ECE and end-to-end routed cost-quality; no external falsifiable handle outside the paper's own benchmarks.
  • Resolution Head independent evidence
    purpose: Predicts info/tool/cant scores from a middle-layer trace for intervention decisions.
    Evaluated on WHEN2CALL F1/accuracy; same caveat as above.

pith-pipeline@v1.3.0-alltime-deepseek · 20288 in / 18781 out tokens · 190548 ms · 2026-08-02T02:34:54.515684+00:00 · methodology

0 comments
read the original abstract

Large language models are increasingly deployed as agents, but reliable agentic behavior requires more than next-token prediction. At inference time, it is preferred that an agent can decide whether to proceed with its current reasoning, defer to a stronger model, request additional information, invoke external tools, or abstain under the given setup. Existing approaches address these decisions through prompt-level routing, external orchestration, or task-specific fine-tuning, which primarily rely on input-side signals, and are often costly and difficult to maintain as model backbones evolve. We ask whether such control decisions can be inferred directly from a model's latent generation process. We introduce Multi-Head Latent Control, a lightweight layer that reads hidden-state trajectories from a frozen LLM or VLM to produce deployment-time control signals. A Capability Head predicts whether the current model can solve the instance or should defer to a stronger collaborator, while a Resolution Head predicts appropriate resolution decision Clarification, Tool Use, Abstention, or Direct Answering. Both heads are trained only on latent traces from the same frozen LLM backbone, enabling post hoc adaptation without modifying the model. Across language and vision-language settings, Multi-Head Latent Control consistently improves the quality-cost tradeoff of multi-model systems, enabling early handoff from partial generations and more accurate intervention decisions. In routed execution (small + large model), it reduces large-model usage by up to 90.7 percent on AndroidWorld and 27-53 percent on average across benchmarks, while retaining most of large-model performance. Additionally, the learned control signals improve tool-use decision quality, yielding up to +158 percent relative score gain and 65.5 percent fewer missed-required tool calls.

Figures

Figures reproduced from arXiv: 2607.14277 by Amirhosein Ghasemabadi, Bahador Rashidi, Di Niu, Ruichen Chen.

Figure 1
Figure 1. Figure 1: Multi-head latent control as an intrinsic control interface for frozen foundation models. (a) From the latent trajectory of a frozen primary model m1, a Capability Head decides whether to remain with m1 or hand off to a stronger model m2, while a Resolution Head selects the appropriate resolution decision: request more information, invoke tools, abstain, or proceed directly. (b) Repeated use of these contr… view at source ↗
Figure 2
Figure 2. Figure 2: Overall cost–performance tradeoff for two routed systems: [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Composition of the 120K training mixture for the Capability Head. The mixture intention [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Capability Head scores versus model token confidence on Qwen3.5-9B over ScreenSpot￾Pro. The left panel shows the distribution of Capability Head scores for correct and incorrect predictions. The right panel shows the corresponding distribution of the model’s mean token probability over the last 100 response tokens. Token confidence exhibits substantial overlap between correct and incorrect cases, making it… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

51 extracted references · 20 linked inside Pith

  1. [1]

    Azaria and T

    A. Azaria and T. Mitchell. The internal state of an llm knows when it’s lying. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 967–976, 2023

    2023

  2. [2]

    S. Bai, Y . Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, W. Ge, Z. Guo, Q. Huang, J. Huang, F. Huang, B. Hui, S. Jiang, Z. Li, M. Li, M. Li, K. Li, Z. Lin, J. Lin, X. Liu, J. Liu, C. Liu, Y . Liu, D. Liu, S. Liu, D. Lu, R. Luo, C. Lv, R. Men, L. Meng, X. Ren, X. Ren, S. Song, Y . Sun, J. Tang, J. Tu, J. Wan, P. Wang, P. Wang,...

    Pith/arXiv arXiv 2025

  3. [3]

    Burns, H

    C. Burns, H. Ye, D. Klein, and J. Steinhardt. Discovering latent knowledge in language models without supervision.arXiv preprint arXiv:2212.03827, 2022

    Pith/arXiv arXiv 2022

  4. [4]

    T. Cai, Y . Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, and T. Dao. Medusa: Simple llm inference acceleration framework with multiple decoding heads.arXiv preprint arXiv:2401.10774, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    C. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre, and J. Jumper. Accelerating large language model decoding with speculative sampling.arXiv preprint arXiv:2302.01318, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    L. Chen, M. Zaharia, and J. Zou. Frugalgpt: How to use large language models while reducing cost and improving performance.arXiv preprint arXiv:2305.05176, 2023

    Pith/arXiv arXiv 2023

  7. [7]

    S. Chen, W. Jiang, B. Lin, J. Kwok, and Y . Zhang. Routerdc: Query-based router by dual contrastive learning for assembling large language models.Advances in Neural Information Processing Systems, 37:66305–66328, 2024

    2024

  8. [8]

    Cheng, W

    X. Cheng, W. Zhang, S. Zhang, J. Yang, X. Guan, X. Wu, X. Li, G. Zhang, J. Liu, Y . Mai, et al. Simplevqa: Multimodal factuality evaluation for multimodal large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4637–4646, 2025

    2025

  9. [9]

    Dekoninck, M

    J. Dekoninck, M. Baader, and M. Vechev. A unified approach to routing and cascading for llms. arXiv preprint arXiv:2410.10347, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    Y . Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch. Improving factuality and reasoning in language models through multiagent debate. InForty-first international conference on machine learning, 2024

    2024

  11. [11]

    open-r1-multimodal

    EvolvingLMMs-Lab. open-r1-multimodal. https://github.com/EvolvingLMMs-Lab/ open-r1-multimodal, 2025. GitHub repository; includes the multimodal-open-r1-8k-verified dataset

    2025

  12. [12]

    Ghasemabadi and D

    A. Ghasemabadi and D. Niu. Can llms predict their own failures? self-awareness via internal circuits.arXiv preprint arXiv:2512.20578, 2025

    arXiv 2025

  13. [13]

    Gemma 4 model overview

    Google. Gemma 4 model overview. https://ai.google.dev/gemma/docs/core, 2026. Official model documentation

    2026

  14. [14]

    Goyal, T

    Y . Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017. 10

    2017

  15. [15]

    S. Hong, M. Zhuge, J. Chen, X. Zheng, Y . Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, et al. Metagpt: Meta programming for a multi-agent collaborative framework. InThe twelfth international conference on learning representations, 2023

    2023

  16. [16]

    Hsiao, F

    Y .-C. Hsiao, F. Zubach, G. Baechler, S. Sunkara, V . C ˘arbune, J. Lin, M. Wang, Y . Zhu, and J. Chen. Screenqa: Large-scale question-answer pairs over mobile app screenshots. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), ...

    2025

  17. [17]

    Joshi, E

    M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, 2017

    2017

  18. [18]

    Kembhavi, M

    A. Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Hajishirzi, and A. Farhadi. A diagram is worth a dozen images. InEuropean conference on computer vision, pages 235–251. Springer, 2016

    2016

  19. [19]

    Leviathan, M

    Y . Leviathan, M. Kalman, and Y . Matias. Fast inference from transformers via speculative decoding. InInternational Conference on Machine Learning, pages 19274–19286. PMLR, 2023

    2023

  20. [20]

    K. Li, Z. Meng, H. Lin, Z. Luo, Y . Tian, J. Ma, Z. Huang, and T.-S. Chua. Screenspot-pro: Gui grounding for professional high-resolution computer use. InProceedings of the 33rd ACM International Conference on Multimedia, pages 8778–8786, 2025

    2025

  21. [21]

    W. Li, D. Li, K. Dong, C. Zhang, H. Zhang, W. Liu, Y . Wang, R. Tang, and Y . Liu. Adaptive tool use in large language models with meta-cognition trigger. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13346–13370, 2025

    2025

  22. [22]

    B. Liao, Y . Xu, H. Dong, J. Li, C. Monz, S. Savarese, D. Sahoo, and C. Xiong. Reward-guided speculative decoding for efficient llm reasoning.arXiv preprint arXiv:2501.19324, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in neural information processing systems, 35:2507–2521, 2022

    2022

  24. [24]

    P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023

    Pith/arXiv arXiv 2023

  25. [25]

    Masry, X

    A. Masry, X. L. Do, J. Q. Tan, S. Joty, and E. Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. InFindings of the association for computational linguistics: ACL 2022, pages 2263–2279, 2022

    2022

  26. [26]

    Mathew, D

    M. Mathew, D. Karatzas, and C. Jawahar. Docvqa: A dataset for vqa on document images. InProceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021

    2021

  27. [27]

    Mathew, V

    M. Mathew, V . Bagal, R. Tito, D. Karatzas, E. Valveny, and C. Jawahar. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1697–1706, 2022

    2022

  28. [28]

    Narasimhan, W

    H. Narasimhan, W. Jitkrittum, A. S. Rawat, S. Kim, N. Gupta, A. K. Menon, and S. Kumar. Faster cascades via speculative decoding.arXiv preprint arXiv:2405.19261, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    I. Ong, A. Almahairi, V . Wu, W.-L. Chiang, T. Wu, J. E. Gonzalez, M. W. Kadous, and I. Stoica. Routellm: Learning to route llms with preference data.arXiv preprint arXiv:2406.18665, 2024

    Pith/arXiv arXiv 2024

  30. [30]

    S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez. Gorilla: Large language model connected with massive apis.Advances in Neural Information Processing Systems, 37:126544–126565, 2024. 11

    2024

  31. [31]

    Prabhakar, Z

    A. Prabhakar, Z. Liu, M. Zhu, J. Zhang, T. Awalgaonkar, S. Wang, Z. Liu, H. Chen, T. Hoang, J. C. Niebles, et al. Apigen-mt: Agentic pipeline for multi-turn data generation via simulated agent-human interplay.arXiv preprint arXiv:2504.03601, 2025

    Pith/arXiv arXiv 2025

  32. [32]

    Y . Qin, S. Liang, Y . Ye, K. Zhu, L. Yan, Y . Lu, Y . Lin, X. Cong, X. Tang, B. Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis.arXiv preprint arXiv:2307.16789, 2023

    Pith/arXiv arXiv 2023

  33. [33]

    Qwen3.5: Towards native multimodal agents, February 2026

    Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https: //qwen.ai/blog?id=qwen3.5

    2026

  34. [34]

    Rawles, S

    C. Rawles, S. Clinckemaillie, Y . Chang, J. Waltz, G. Lau, M. Fair, A. Li, W. Bishop, W. Li, F. Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents.arXiv preprint arXiv:2405.14573, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    H. Ross, A. S. Mahabaleshwarkar, and Y . Suhara. When2call: When (not) to call tools. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3391–3409, 2025

    2025

  36. [36]

    Schick, J

    T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom. Toolformer: Language models can teach themselves to use tools. Advances in neural information processing systems, 36:68539–68551, 2023

    2023

  37. [37]

    Schwenk, A

    D. Schwenk, A. Khandelwal, C. Clark, K. Marino, and R. Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. InEuropean conference on computer vision, pages 146–162. Springer, 2022

    2022

  38. [38]

    Z. Shen, H. Lang, B. Wang, Y . Kim, and D. Sontag. Learning to decode collaboratively with multiple language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12974–12990, 2024

    2024

  39. [39]

    H. Su, S. Diao, X. Lu, M. Liu, J. Xu, X. Dong, Y . Fu, P. Belcak, H. Ye, H. Yin, et al. Toolorchestra: Elevating intelligence via efficient model and tool orchestration.arXiv preprint arXiv:2511.21689, 2025

    arXiv 2025

  40. [40]

    J. Wang, J. Wang, B. Athiwaratkun, C. Zhang, and J. Zou. Mixture-of-agents enhances large language model capabilities.arXiv preprint arXiv:2406.04692, 2024

    Pith/arXiv arXiv 2024

  41. [41]

    Y . Wang, X. Ma, G. Zhang, Y . Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. Advances in Neural Information Processing Systems, 37:95266–95290, 2024

    2024

  42. [42]

    Z. Wang, M. Xia, L. He, H. Chen, Y . Liu, R. Zhu, K. Liang, X. Wu, H. Liu, S. Malladi, et al. Charxiv: Charting gaps in realistic chart understanding in multimodal llms.Advances in Neural Information Processing Systems, 37:113569–113697, 2024

    2024

  43. [43]

    Wiedmann, O

    L. Wiedmann, O. Zohar, A. Mahla, X. Wang, R. Li, T. Frere, L. von Werra, A. R. Gosthipaty, and A. Marafioti. Finevision: Open data is all you need.arXiv preprint arXiv:2510.17269, 2025

    Pith/arXiv arXiv 2025

  44. [44]

    H. Xu, Z. Zhu, L. Pan, Z. Wang, S. Zhu, D. Ma, R. Cao, L. Chen, and K. Yu. Reducing tool hallucination via reliability alignment.arXiv preprint arXiv:2412.04141, 2024

    Pith/arXiv arXiv 2024

  45. [45]

    H. Xu, Z. Wang, Z. Zhu, L. Pan, X. Chen, S. Fan, L. Chen, and K. Yu. Alignment for efficient tool calling of large language models. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 17787–17803, 2025

    2025

  46. [46]

    H. Xu, X. Zhang, H. Liu, J. Wang, Z. Zhu, S. Zhou, X. Hu, F. Gao, J. Cao, Z. Wang, et al. Mobile-agent-v3. 5: Multi-platform fundamental gui agents.arXiv preprint arXiv:2602.16855, 2026

    arXiv 2026

  47. [47]

    Y . Xu, Z. Wang, J. Wang, D. Lu, T. Xie, A. Saha, D. Sahoo, T. Yu, and C. Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction.arXiv preprint arXiv:2412.04454, 2024. 12

    Pith/arXiv arXiv 2024

  48. [48]

    Q. Yu, Z. Zhang, R. Zhu, Y . Yuan, X. Zuo, Y . Yue, W. Dai, T. Fan, G. Liu, L. Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    Pith/arXiv arXiv 2025

  49. [49]

    Zhang, Y

    A. Zhang, Y . Chen, J. Pan, C. Zhao, A. Panda, J. Li, and H. He. Reasoning models know when they’re right: Probing hidden states for self-verification.arXiv preprint arXiv:2504.05419, 2025

    Pith/arXiv arXiv 2025

  50. [50]

    Zhang, D

    R. Zhang, D. Jiang, Y . Zhang, H. Lin, Z. Guo, P. Qiu, A. Zhou, P. Lu, K.-W. Chang, Y . Qiao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? InEuropean Conference on Computer Vision, pages 169–186. Springer, 2024

    2024

  51. [51]

    Zheng, Z

    L. Zheng, Z. Huang, Z. Xue, X. Wang, B. An, and S. Yan. Agentstudio: A toolkit for building general virtual agents.arXiv preprint arXiv:2403.17918, 2024. 13 $0 $5 $10 $15 $20 $25 $30 Overall Estimated Cost (USD) 0.66 0.67 0.68 0.69 0.70 0.71 0.72 0.73Overall Score M1 M2 thr=0.50 thr=0.60 thr=0.70 thr=0.80 thr=0.90 Overall Cost Performance Tradeoff: qwen35...

    Pith/arXiv arXiv 2024