Pith. sign in

REVIEW 5 major objections 5 minor 2 cited by

TAIJI: MCP-based Multi-Modal Data Analytics on Data Lakes

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read Decomposing a data-lake query into modality-specific operators and dispatching each to a dedicated MCP server running a small specialized LLM beats a single unified large LLM on accuracy and latency, while update machinery keeps data and…

desk verdict MCP-based multi-modal analytics: a plausible vision whose only experiment validates a specialized vision model, not the architecture. read the letter →

arxiv 2505.11270 v1 pith:5RWAWQ36 submitted 2025-05-16 cs.DB cs.AI

classification cs.DBcs.AI
keywords multi-modaldataanalyticslakesModelContextProtocolsemanticoperatorsNL2Operatorqueryplanningmachineunlearningfreshness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TAIJI sets out to solve multi-modal analytics over data lakes—tables, JSON, text, and images—by replacing the single unified LLM with a modular architecture organized around the Model Context Protocol (MCP). The central claim is that one size does not fit all: a host LLM translates natural-language intent into a hierarchy of semantic operators, and each operator is executed by a small MCP server specialized for one modality, which is both more accurate and cheaper than running everything through a giant model. On a furniture benchmark combining relational rows with images, the prototype reports higher accuracy on one of three queries, higher recall on two of three queries, and cuts end-to-end latency by 43% on average. The paper also proposes update machinery—query-driven data augmentation and machine unlearning—to keep both the lake and the models fresh. The practical pay-off is a scalable template for federating many small models over heterogeneous data, provided the initial query-translation step can be trusted.

What carries the argument

The central mechanism is the MCP client–host–server split: a host-side LLM agent interprets a natural-language query, maps it through an NL2Operator translator into a hierarchy of semantic operators, and dispatches each sub-plan to an MCP server, where a specialized foundation model processes one data modality over its own data source. The semantic operator hierarchy is what makes cross-modal joins and filters over structured, semi-structured, and unstructured data expressible without forcing one model to know everything. Around this core sit a sampling-based cost optimizer that chooses the lowest-latency execution plan, an iterative feedback loop that lets a server refine its query when initial results are sparse, a filter-aware vector index for embedding search under metadata constraints, and an augmentor that refreshes the data lake and the models.

What would settle it

Run the NL2Operator translator in isolation on a held-out set of ambiguous multi-modal queries with expert-labeled operator assignments; if its assignment accuracy is no better than a generic LLM doing direct translation, then the end-to-end gains in the paper cannot be attributed to the MCP orchestration and may come from the specialized image model alone.

Watch

Extended reading notes

Core claim

The core discovery the paper argues for is that a multi-modal query over a data lake should be decomposed into modality-specific semantic operators and distributed to dedicated MCP servers, each running a tailored LLM, rather than processed end-to-end by a single unified LLM. On the paper's own terms, this one-size-does-not-fit-all principle is what delivers higher accuracy, lower inference overhead, and natural scalability, because the central LLM only translates intent and plans, while small specialized models do the heavy per-modality work. The strongest supporting evidence is a three-query experiment on a 3,000-listing furniture dataset with relational and image tables: the MCP prototype reaches 85% accuracy versus 65% for the unified baseline on the hardest image-predicate query, improves recall on two of three queries, and reduces latency by 43% on average, with the gap widening as the intermediate result size grows.

Load-bearing premise

The whole pipeline depends on the NL2Operator translator reliably mapping ambiguous natural-language queries to the correct semantic operators; the paper reports no standalone accuracy measurement for that step, so if it misroutes a query, every downstream filter, join, and image match will be wrong regardless of the specialized models' quality.

Editorial extensions

If this is right

  • A data-lake service could answer multi-modal queries with several small specialized models instead of one trillion-parameter LLM, with lower latency and comparable or better accuracy on image-plus-table workloads.
  • The semantic operator hierarchy gives analytics systems a uniform way to express cross-modal joins and filters across tables, documents, graphs, images, audio, and video without inventing a new query language.
  • Sampling-based cost estimation produces deterministic query plans that adapt to the data distribution, avoiding the unpredictability of LLM-only planning.
  • Feedback-driven iterative retrieval lets a server refine its own query when initial results are sparse, which should raise recall on ambiguous predicates.
  • Machine unlearning plus query-driven fine-tuning gives a data lake a path to remove obsolete or sensitive knowledge without full retraining, keeping model knowledge aligned with a changing lake.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the modular-delegation result generalizes, the same client–host–server split could be applied to audio, video, and sensor streams, where no single model is uniformly best; the paper sketches this possibility but does not test it.
  • The 43% latency gain is measured against one unified baseline with a fixed query plan; varying the plan choices and comparing against other monolithic multi-modal models would clarify whether the gain comes from model specialization or from MCP orchestration.
  • The likely bottleneck is the NL2Operator translator, so adding a clarification dialogue or human-in-the-loop disambiguation before planning could improve end-to-end accuracy more than swapping server models.
  • A standardized multi-modal data-lake benchmark, which the paper itself calls for, would let the field isolate the architecture's contribution from the choice of specific server models.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes TAIJI, an architecture for multi-modal data analytics over data lakes built on the Model Context Protocol (MCP). The design assigns each data modality to a dedicated MCP server hosting a specialized LLM, with a client-side NL2Operator translator, a cost-based query planner, an iterative operator optimizer, embedding-based indexing, and an augmentor that refreshes both data and model knowledge. A preliminary experiment on a Craigslist furniture dataset compares a prototype (GPT-4.1 client, Qwen2.5-VL-7B image server, PostgreSQL for tables) against GPT-4.1 with PostgreSQL assistance, reporting improved precision/recall on some queries and about 43% lower average latency. The paper positions itself as a vision plus preliminary prototype, and the conclusion states that full system implementation and more experiments remain as future work.

Significance. If the architecture delivered what is claimed, the MCP-based delegation of modality-specific tasks to specialized small models would be a meaningful direction for multi-modal data analytics, potentially improving both accuracy and inference cost relative to monolithic unified LLMs. The paper identifies a timely problem, gives a clear system decomposition, and provides an initial prototype on a real dataset. It also honestly discusses challenges such as security, hybrid deployment, and cost-aware model management. However, the current evidence supports only a narrow feasibility claim about one vision component, and the central architectural claims about translation, planning, indexing, and freshness are not yet empirically validated.

major comments (5)
  1. [Section 4.1, Table 1] The evaluation fixes the query plan by design, stating that 'for simplicity, the query plan is fixed by performing a filter on the furniture table and then conducting matches on images.' This bypasses the NL2Operator translator, query planner, and iterative operator optimizer, all of which are core novel components of the proposed architecture. Consequently, the experiment only demonstrates that Qwen2.5-VL-7B outperforms GPT-4.1 on three image-classification predicates over the Craigslist dataset; it does not validate the MCP-based architecture's distinctive mechanisms.
  2. [Section 4.2] No error bars, confidence intervals, or multiple-run statistics are reported for precision, recall, or latency. With only three queries and stochastic LLM inference, the reported differences (e.g., Q2 accuracy 85% vs. 65%; average latency improvement 43%) may fall within run-to-run variance. The paper should either provide repeated trials with variance estimates or temper the quantitative claims accordingly.
  3. [Section 3.1.1] NL2Operator is the load-bearing component that maps ambiguous natural-language queries to the semantic operator hierarchy, but no evaluation of translation accuracy is reported. If this mapping fails, downstream filtering, image matching, and cross-modal joins are wrong regardless of how capable the specialized models are. The manuscript needs an evaluation of operator-mapping correctness on a held-out query set, including out-of-scope or ambiguous queries.
  4. [Section 4.1, Baseline] The baseline also uses PostgreSQL for table filtering, so the comparison isolates the choice of image-analysis model (Qwen2.5-VL-7B vs. GPT-4.1) rather than the MCP-based architecture itself. The text calls the baseline 'pure GPT-4.1' while simultaneously not using a pure model; this inconsistency makes the architectural contribution indistinguishable from a simple model swap. An ablation with a pure unified-LLM baseline, and an ablation where GPT-4.1 performs image analysis within the MCP framework, would be needed to separate the claims.
  5. [Conclusions, last paragraph] The paper explicitly states, 'In the future, we will finish implementing the whole system and conduct more experiments.' The current prototype covers only query translation, table filtering, and image analysis; the embedding indexing (Section 3.2.1), data augmentor (Section 3.3.1), model refreshing with unlearning (Section 3.3.2), and the cost-based planner (Section 3.1.2) are neither implemented nor evaluated. This undercuts the broader claims of scalability, freshness, and efficiency that form the paper's stated contributions, and the manuscript should present at least component-level evidence or clearly frame the paper as a position paper rather than a systems paper.
minor comments (5)
  1. [Section 4.1, Evaluation Metric] The text says 'to compute recall and precision' but the metric definition earlier in Section 2.1 lists recall, precision, and latency; please ensure the terms accuracy, precision, and recall are used consistently, since Figure 6 is labeled 'Precision' while the text also reports 'accuracy.'
  2. [Throughout] There are several misspellings and typos, including 'Knowlege' in Section 2.1, 'fasion' in Section 4.1, 'hugh' for 'huge' in Section 5, 'subcription' in Section 3.3.2, and 'excel' for 'excellent' in Section 4.2. A copyediting pass is needed.
  3. [Figures 4-8] Figures 4, 5, 6, 7, and 8 are referenced in the text, but the figures themselves are not visible in the manuscript text provided; please ensure they are embedded and readable, and that each has a self-contained caption explaining what is plotted.
  4. [Section 3.3.1, Redundancy Elimination] The description mentions MinHash, Sentence-BERT, HNSW, and FAISS but gives no parameter values or implementation details; adding a concrete algorithmic specification or pseudocode would improve reproducibility.
  5. [References] Reference [17] duplicates the GPT-4 technical report already cited as [2]; please consolidate or disambiguate these references.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TAIJI is an architecture paper whose preliminary comparison is an external benchmark against GPT-4.1, with no fit-by-construction or self-citation chain.

full rationale

TAIJI contains no formal derivation whose output equals its input, no fitted parameters renamed as predictions, and no load-bearing self-citation. The central claim, that per-modality MCP servers with tailored models improve accuracy and reduce inference overhead, is tested directly by comparing Qwen2.5-VL-7B plus PostgreSQL against GPT-4.1 on a fixed three-query workload over the Craigslist furniture dataset. The paper explicitly fixes the query plan in Section 4.1, so the NL2Operator translator, planner, and optimizer are not exercised; this is an evidence or completeness gap about an untested component, not circularity. Cited works such as MCP, Qwen2.5-VL, ThalamusDB, and Palimpzest are external and do not embed TAIJI's results. The two references involving author Ju Fan ([6] and [13]) are background on NL2SQL and are not used to justify the architecture or to forbid alternative designs. There is thus no instance in which a prediction reduces by construction to its inputs, and the honest finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No numeric constants are fitted; the paper gives no equations or derived predictions. The load-bearing premises are empirical and design assumptions about specialized models, NL2Operator reliability, MCP overhead, and update safety. No new physical or conceptual entities beyond the system architecture itself are introduced.

assumptions (4)
  • domain assumption A specialized smaller LLM per data modality is more accurate and cheaper than one unified large LLM for multi-modal analytics.
    Stated in Sections 1 and 3.2.2; the only supporting evidence is one image model (Qwen2.5-VL-7B) versus GPT-4.1 on three queries.
  • domain assumption An LLM agent can reliably translate natural language queries into the semantic operator hierarchy.
    Section 3.1.1 asserts this without presenting translation accuracy results; the entire execution pipeline depends on it.
  • domain assumption MCP provides a scalable, low-overhead execution abstraction for dispatching sub-tasks to servers.
    Sections 1 and 3 assume MCP is an efficient standard interface; no protocol overhead is measured in the experiments.
  • domain assumption Data and model knowledge can be refreshed by web ingestion, fine-tuning, and machine unlearning without degrading other capabilities.
    Section 3.3 describes the update mechanism qualitatively; no experiments or formal guarantees are provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TAIJI: MCP-based Multi-Modal Data Analytics on Data Lakes." pith.science (2026). https://pith.science/paper/5RWAWQ36

@misc{pith2026250511270,
  author       = {Pith},
  title        = {Pith review of: TAIJI: MCP-based Multi-Modal Data Analytics on Data Lakes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5RWAWQ36}},
  note         = {Machine review of arXiv:2505.11270}
}
read the original abstract

The variety of data in data lakes presents significant challenges for data analytics, as data scientists must simultaneously analyze multi-modal data, including structured, semi-structured, and unstructured data. While Large Language Models (LLMs) have demonstrated promising capabilities, they still remain inadequate for multi-modal data analytics in terms of accuracy, efficiency, and freshness. First, current natural language (NL) or SQL-like query languages may struggle to precisely and comprehensively capture users' analytical intent. Second, relying on a single unified LLM to process diverse data modalities often leads to substantial inference overhead. Third, data stored in data lakes may be incomplete or outdated, making it essential to integrate external open-domain knowledge to generate timely and relevant analytics results. In this paper, we envision a new multi-modal data analytics system. Specifically, we propose a novel architecture built upon the Model Context Protocol (MCP), an emerging paradigm that enables LLMs to collaborate with knowledgeable agents. First, we define a semantic operator hierarchy tailored for querying multi-modal data in data lakes and develop an AI-agent-powered NL2Operator translator to bridge user intent and analytical execution. Next, we introduce an MCP-based execution framework, in which each MCP server hosts specialized foundation models optimized for specific data modalities. This design enhances both accuracy and efficiency, while supporting high scalability through modular deployment. Finally, we propose a updating mechanism by harnessing the deep research and machine unlearning techniques to refresh the data lakes and LLM knowledges, with the goal of balancing the data freshness and inference efficiency.

Figures

Figures reproduced from arXiv: 2505.11270 by the authors.

Figure 1
Figure 1. An Illustrate Example on MCP-based Multi-Modal Data Analytics. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An Overview of TAIJI. can either process the query directly or delegate it to specialized sub-servers (e.g., for JSON, HTML, XML). This hierarchical design offers three advantages: (1) Scalability and flexibility, by modularizing the processing across different servers; (2) High concurrency support, through distributed workload balancing; (3) Simplified translation, by mapping NL to high-level modality-specific oper… view at source ↗
Figure 3
Figure 3. An illustrative example of iterative operator optimization in TA [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Embedding of Multi-Modal Data. 3.2.2 Multi-Modal MCP Servers Multi-Modal MCP Servers deal with heterogeneous data ecosystems by unifying structured (e.g., relational databases), semi-structured (e.g., JSON logs, XML), and unstructured data (e.g., text, images, sensor s…
Figure 5
Figure 5. Figure 5: Data Augmentation and Model Refreshing in TA [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Precision Evaluation [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces

    cs.AI 2026-08 conditional novelty 6.0 of 10

    A new heterogeneous-workspace benchmark for data agents, built from transformed Text-to-SQL tasks, with complete-table deterministic evaluation and a best measured accuracy of 66.34%.

  2. FDABench: A Benchmark for Data Agents on Analytical Queries over Heterogeneous Data

    cs.DB 2025-09 conditional novelty 6.0 of 10

    FDABench is a 2,007-task benchmark for AI agents that must combine structured databases with documents, web, image, video, and audio sources to answer analytical queries.

Reference graph

Works this paper leans on

26 extracted references · 19 canonical work pages · cited by 2 Pith papers

  1. [1]

    https://platform.openai.com/docs/guides/ text-generation

    Introduction of openai text generation apis. https://platform.openai.com/docs/guides/ text-generation. 11

  2. [3]

    Azizi, K

    I. Azizi, K. Echihabi, and T. Palpanas. Graph-based vector search: An experimental evaluation of the state-of-the-art. Proc. ACM Manag. Data, 3(1):43:1–43:31, 2025

  3. [4]

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y . Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y . Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin. Qwen2.5-VL Technical Report. arXiv preprint arXiv:2502.13923, 2025

  4. [5]

    Chourasia and N

    R. Chourasia and N. Shah. Forget unlearning: Towards true data-deletion in machine learning. volume 202, pages 6028–6073. PMLR, 2023

  5. [6]

    J. Fan, Z. Gu, S. Zhang, Y . Zhang, Z. Chen, L. Cao, G. Li, S. Madden, X. Du, and N. Tang. Combining small language models and large language models for zero-shot nl2sql. Proceedings of the VLDB Endowment , 17(11):2750–2763, 2024

  6. [7]

    A. I. for AI. Science parse: An open-source library for extracting structured data from scientific pdfs, 2018. https: //github.com/allenai/science-parse

  7. [8]

    X. Hou, Y . Zhao, S. Wang, and H. Wang. Model context protocol (mcp): Landscape, security threats, and future research directions. arXiv preprint arXiv:2503.23278, 2025

  8. [9]

    Jo and I

    S. Jo and I. Trummer. Thalamusdb: Approximate query processing on multi-modal data. Proceedings of the ACM on Management of Data, 2(3):1–26, 2024

Show all 26 references
  1. [10]

    Lewis, E

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W.-t. Yih, T. Rockt¨aschel, S. Riedel, and D. Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing Systems, volume 33, p...

  2. [11]

    C. Liu, M. Russo, M. Cafarella, L. Cao, P. B. Chen, Z. Chen, M. Franklin, T. Kraska, S. Madden, R. Shahout, et al. Palimpzest: Optimizing ai-powered analytics with declarative query processing. In Proceedings of the Conference on Innovative Database Research (CIDR), 2025

  3. [12]

    C. Liu, M. Russo, M. Cafarella, L. Cao, P. B. Chen, Z. Chen, M. Franklin, T. Kraska, S. Madden, and G. Vitagliano. A declarative system for optimizing ai workloads. arXiv preprint arXiv:2405.14696, 2024

  4. [13]

    X. Liu, S. Shen, B. Li, P. Ma, R. Jiang, Y . Zhang, J. Fan, G. Li, N. Tang, and Y . Luo. A survey of nl2sql with large language models: Where are we, and where are we going? arXiv preprint arXiv:2408.05109, 2024

  5. [14]

    P. Lopez. Grobid: Generation of bibliographic data, 2020. https://github.com/kermitt2/grobid

  6. [15]

    Lu and I

    J. Lu and I. Holubov ´a. Multi-model databases: a new journey to handle the variety of data. ACM Computing Surveys (CSUR), 52(3):1–38, 2019

  7. [16]

    Nargesian, E

    F. Nargesian, E. Zhu, R. J. Miller, K. Q. Pu, and P. C. Arocena. Data lake management: challenges and opportunities. Proceedings of the VLDB Endowment, 12(12):1986–1989, 2019

  8. [17]

    OpenAI and e. a. Achiam. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774, 2023

  9. [18]

    S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez. Gorilla: Large language model connected with massive apis. Advances in Neural Information Processing Systems, 37:126544–126565, 2024

  10. [19]

    M. C. Protocol. Model context protocol servers, 2025. https://github.com/modelcontextprotocol/ servers

  11. [20]

    Reimers and I

    N. Reimers and I. Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pages 3982–3992, 2019

  12. [21]

    Schick, J

    T. Schick, J. Dwivedi-Yu, R. Dess `ı, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36:68539–68551, 2023. 12

  13. [22]

    Shuster, M

    K. Shuster, M. Komeili, L. Adolphs, S. Roller, A. Szlam, and J. Weston. Language models that seek for knowledge: Modular search and generation for dialogue and prompt completion. InFindings of the Association for Computational Linguistics: EMNLP 2022, pages 373–393, 2022

  14. [23]

    Urban and C

    M. Urban and C. Binnig. Caesura: Language models as multi-modal query planners. arXiv preprint arXiv:2308.03424, 2023

  15. [24]

    Urban and C

    M. Urban and C. Binnig. Eleet: Efficient learned query execution over text and tables. Proc. VLDB Endow, 17:13, 2024

  16. [25]

    Wang and G

    J. Wang and G. Li. Aop: Automated and interactive llm pipeline orchestration for answering complex queries. CIDR, 2025

  17. [26]

    J. Wang, G. Li, and J. Feng. idatalake: An llm-powered analytics system on data lakes. Data Engineering, page 57

  18. [27]

    Zhang, Z

    Y . Zhang, Z. Hu, Y . Bai, J. Wu, Q. Wang, and F. Feng. Recommendation unlearning via influence function. arXiv preprint arXiv:2307.02147, 2023. 13

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.