A modular 8B agent with episodic visual memory and RL-trained retrieval reaches 91.4% cross-turn image recall over 20 turns, outperforming 32B all-context baselines with ~1.8× lower latency.
Multi-modal agent tuning: Building a vlm-driven agent for efficient tool usage.arXiv preprint arXiv:2412.15606
4 Pith papers cite this work. Polarity classification is still indexing.
4
Pith papers citing it
abstract
The advancement of large language models (LLMs) prompts the development of multi-modal agents, which are used as a controller to call external tools, providing a feasible way to solve practical tasks. In this paper, we propose a multi-modal agent tuning method that automatically generates multi-modal tool-usage data and tunes a vision-language model (VLM) as the controller for powerful tool-usage reasoning. To preserve the data quality, we prompt the GPT-4o mini model to generate queries, files, and trajectories, followed by query-file and trajectory verifiers. Based on the data synthesis pipeline, we collect the MM-Traj dataset that contains 20K tasks with trajectories of tool usage. Then, we develop the T3-Agent via \underline{T}rajectory \underline{T}uning on VLMs for \underline{T}ool usage using MM-Traj. Evaluations on the GTA and GAIA benchmarks show that the T3-Agent consistently achieves improvements on two popular VLMs: MiniCPM-V-8.5B and {Qwen2-VL-7B}, which outperforms untrained VLMs by $20\%$, showing the effectiveness of the proposed data synthesis pipeline, leading to high-quality data for tool-usage capabilities.
citation-role summary
background 1
citation-polarity summary
roles
background 1polarities
support 1representative citing papers
ReGRPO augments group-relative policy optimization with a reflective data engine that generates ErrorType-Evidence-FixPlan triplets from near-miss tool actions to improve recovery in multimodal agents.
Multi-agent VLM frameworks outperform single VLMs for automated coding of on-screen collaborative learning behaviors using the ICAP framework.
TRACE uses an evidence bank to score tool-augmented LLM agents on efficiency, hallucination, and adaptivity without ground-truth trajectories.
citing papers explorer
-
Cognitive-structured Multimodal Agent for Multimodal Understanding, Generation, and Editing
A modular 8B agent with episodic visual memory and RL-trained retrieval reaches 91.4% cross-turn image recall over 20 turns, outperforming 32B all-context baselines with ~1.8× lower latency.
-
ReGRPO: Reflection-Augmented Policy Optimization for Tool-Using Agents
ReGRPO augments group-relative policy optimization with a reflective data engine that generates ErrorType-Evidence-FixPlan triplets from near-miss tool actions to improve recovery in multimodal agents.
-
Single-agent vs. Multi-agents for Automated Video Analysis of On-Screen Collaborative Learning Behaviors
Multi-agent VLM frameworks outperform single VLMs for automated coding of on-screen collaborative learning behaviors using the ICAP framework.
-
Beyond the Final Answer: Evaluating the Reasoning Trajectories of Tool-Augmented Agents
TRACE uses an evidence bank to score tool-augmented LLM agents on efficiency, hallucination, and adaptivity without ground-truth trajectories.