Pith. sign in

REVIEW 10 cited by

MLLM-Tool: A Multimodal Large Language Model For Tool Agent Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.10727 v3 pith:GKDRKGKI submitted 2024-01-19 cs.CV

Chenyu Wang , Weixin Luo , Sixun Dong , Xiaohua Xuan , Zhengxin Li , Lin Ma , Shenghua Gao This is my paper · ORCID
classification cs.CV
keywords llmsmllm-toolmulti-modallanguagetooltoolsagentdataset
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Recently, the astonishing performance of large language models (LLMs) in natural language comprehension and generation tasks triggered lots of exploration of using them as central controllers to build agent systems. Multiple studies focus on bridging the LLMs to external tools to extend the application scenarios. However, the current LLMs' ability to perceive tool use is limited to a single text query, which may result in ambiguity in understanding the users' real intentions. LLMs are expected to eliminate that by perceiving the information in the visual- or auditory-grounded instructions. Therefore, in this paper, we propose MLLM-Tool, a system incorporating open-source LLMs and multi-modal encoders so that the learned LLMs can be conscious of multi-modal input instruction and then select the function-matched tool correctly. To facilitate the evaluation of the model's capability, we collect a dataset featuring multi-modal input tools from HuggingFace. Another essential feature of our dataset is that it also contains multiple potential choices for the same instruction due to the existence of identical functions and synonymous functions, which provides more potential solutions for the same query. The experiments reveal that our MLLM-Tool is capable of recommending appropriate tools for multi-modal instructions. Codes and data are available at https://github.com/MLLM-Tool/MLLM-Tool.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMs

    cs.CV 2025-04 conditional novelty 7.0 of 10

    All-Angles Bench, a 2,132-question benchmark across 90 real scenes, shows current MLLMs score around 60% on multi-view understanding while humans score 82%, with the largest gaps in camera pose estimation and cross-vi...

  2. MMAT-1M: A Large Reasoning Dataset for Multimodal Agent Tuning

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A new one-million-sample multimodal agent tuning dataset with GPT-4o-generated rationales, reflection, and tool/RAG calls is shown to improve fine-tuned models, though training/eval benchmark overlap is not addressed.

  3. MassTool: A Multi-Task Search-Based Tool Retrieval Framework for Large Language Models

    cs.IR 2025-07 conditional novelty 6.0 of 10

    A multi-task retriever that combines tool-usage detection with query-centered graph and search-based modules improves tool retrieval accuracy over prior baselines.

  4. MusiXQA: Advancing Visual Music Understanding in Multimodal Large Language Models

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A synthetic music sheet QA dataset and a LoRA-fine-tuned Phi-3 model show large accuracy gains on OMR and chord tasks, but only within the synthetic distribution.

  5. How Well Can Modern LLMs Act as Agent Cores in Radiology Environments?

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Large language models complete only 29-67% of simulated radiology agent tasks, and prompting tricks and a simulated tool builder do not close the gap on complex workflows.

  6. MageBench: Bridging Large Multimodal Models to Agents

    cs.CV 2024-12 conditional novelty 6.0 of 10

    MageBench introduces a 483-scenario benchmark showing current large multimodal models are far weaker than humans at agent tasks requiring continuous visual feedback and planning.

  7. Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuning

    cs.CV 2025-04 conditional novelty 5.0 of 10

    SPORT builds a multimodal tool agent by generating synthetic tasks, sampling candidate tool calls, and tuning the agent on AI-verified step preferences, improving GTA and GAIA accuracy over the MAT baseline.

  8. SurgVisAgent: Multimodal Agentic Model for Versatile Surgical Visual Enhancement

    cs.CV 2025-07 conditional novelty 4.0 of 10

    SurgVisAgent combines a ResNet distortion-prior with GPT-4o chain-of-thought reasoning to classify distortion type and severity and invoke the appropriate enhancement model, outperforming single-task models on a synth...

  9. Large language models for artificial general intelligence (AGI): A survey of foundational principles and approaches

    cs.AI 2025-01 conditional novelty 3.0 of 10

    This survey argues that embodiment, symbol grounding, causality, and memory are the foundational principles needed to make large language models achieve artificial general intelligence.

  10. Visual Large Language Models for Generalized and Specialized Applications

    cs.CV 2025-01 conditional novelty 3.0 of 10

    This paper reviews and taxonomizes VLLM applications into vision-to-text, vision-to-action, and text-to-vision, adding ethics and future-work discussion.

Pith tools