REVIEW 5 major objections 5 minor 2 cited by
TAIJI: MCP-based Multi-Modal Data Analytics on Data Lakes
T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Decomposing a data-lake query into modality-specific operators and dispatching each to a dedicated MCP server running a small specialized LLM beats a single unified large LLM on accuracy and latency, while update machinery keeps data and…
desk verdict MCP-based multi-modal analytics: a plausible vision whose only experiment validates a specialized vision model, not the architecture. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the MCP client–host–server split: a host-side LLM agent interprets a natural-language query, maps it through an NL2Operator translator into a hierarchy of semantic operators, and dispatches each sub-plan to an MCP server, where a specialized foundation model processes one data modality over its own data source. The semantic operator hierarchy is what makes cross-modal joins and filters over structured, semi-structured, and unstructured data expressible without forcing one model to know everything. Around this core sit a sampling-based cost optimizer that chooses the lowest-latency execution plan, an iterative feedback loop that lets a server refine its query when initial results are sparse, a filter-aware vector index for embedding search under metadata constraints, and an augmentor that refreshes the data lake and the models.
What would settle it
Run the NL2Operator translator in isolation on a held-out set of ambiguous multi-modal queries with expert-labeled operator assignments; if its assignment accuracy is no better than a generic LLM doing direct translation, then the end-to-end gains in the paper cannot be attributed to the MCP orchestration and may come from the specialized image model alone.
Extended reading notes
Core claim
The core discovery the paper argues for is that a multi-modal query over a data lake should be decomposed into modality-specific semantic operators and distributed to dedicated MCP servers, each running a tailored LLM, rather than processed end-to-end by a single unified LLM. On the paper's own terms, this one-size-does-not-fit-all principle is what delivers higher accuracy, lower inference overhead, and natural scalability, because the central LLM only translates intent and plans, while small specialized models do the heavy per-modality work. The strongest supporting evidence is a three-query experiment on a 3,000-listing furniture dataset with relational and image tables: the MCP prototype reaches 85% accuracy versus 65% for the unified baseline on the hardest image-predicate query, improves recall on two of three queries, and reduces latency by 43% on average, with the gap widening as the intermediate result size grows.
Load-bearing premise
The whole pipeline depends on the NL2Operator translator reliably mapping ambiguous natural-language queries to the correct semantic operators; the paper reports no standalone accuracy measurement for that step, so if it misroutes a query, every downstream filter, join, and image match will be wrong regardless of the specialized models' quality.
Editorial extensions
If this is right
- A data-lake service could answer multi-modal queries with several small specialized models instead of one trillion-parameter LLM, with lower latency and comparable or better accuracy on image-plus-table workloads.
- The semantic operator hierarchy gives analytics systems a uniform way to express cross-modal joins and filters across tables, documents, graphs, images, audio, and video without inventing a new query language.
- Sampling-based cost estimation produces deterministic query plans that adapt to the data distribution, avoiding the unpredictability of LLM-only planning.
- Feedback-driven iterative retrieval lets a server refine its own query when initial results are sparse, which should raise recall on ambiguous predicates.
- Machine unlearning plus query-driven fine-tuning gives a data lake a path to remove obsolete or sensitive knowledge without full retraining, keeping model knowledge aligned with a changing lake.
Reading between the lines
- If the modular-delegation result generalizes, the same client–host–server split could be applied to audio, video, and sensor streams, where no single model is uniformly best; the paper sketches this possibility but does not test it.
- The 43% latency gain is measured against one unified baseline with a fixed query plan; varying the plan choices and comparing against other monolithic multi-modal models would clarify whether the gain comes from model specialization or from MCP orchestration.
- The likely bottleneck is the NL2Operator translator, so adding a clarification dialogue or human-in-the-loop disambiguation before planning could improve end-to-end accuracy more than swapping server models.
- A standardized multi-modal data-lake benchmark, which the paper itself calls for, would let the field isolate the architecture's contribution from the choice of specific server models.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes TAIJI, an architecture for multi-modal data analytics over data lakes built on the Model Context Protocol (MCP). The design assigns each data modality to a dedicated MCP server hosting a specialized LLM, with a client-side NL2Operator translator, a cost-based query planner, an iterative operator optimizer, embedding-based indexing, and an augmentor that refreshes both data and model knowledge. A preliminary experiment on a Craigslist furniture dataset compares a prototype (GPT-4.1 client, Qwen2.5-VL-7B image server, PostgreSQL for tables) against GPT-4.1 with PostgreSQL assistance, reporting improved precision/recall on some queries and about 43% lower average latency. The paper positions itself as a vision plus preliminary prototype, and the conclusion states that full system implementation and more experiments remain as future work.
Significance. If the architecture delivered what is claimed, the MCP-based delegation of modality-specific tasks to specialized small models would be a meaningful direction for multi-modal data analytics, potentially improving both accuracy and inference cost relative to monolithic unified LLMs. The paper identifies a timely problem, gives a clear system decomposition, and provides an initial prototype on a real dataset. It also honestly discusses challenges such as security, hybrid deployment, and cost-aware model management. However, the current evidence supports only a narrow feasibility claim about one vision component, and the central architectural claims about translation, planning, indexing, and freshness are not yet empirically validated.
major comments (5)
- [Section 4.1, Table 1] The evaluation fixes the query plan by design, stating that 'for simplicity, the query plan is fixed by performing a filter on the furniture table and then conducting matches on images.' This bypasses the NL2Operator translator, query planner, and iterative operator optimizer, all of which are core novel components of the proposed architecture. Consequently, the experiment only demonstrates that Qwen2.5-VL-7B outperforms GPT-4.1 on three image-classification predicates over the Craigslist dataset; it does not validate the MCP-based architecture's distinctive mechanisms.
- [Section 4.2] No error bars, confidence intervals, or multiple-run statistics are reported for precision, recall, or latency. With only three queries and stochastic LLM inference, the reported differences (e.g., Q2 accuracy 85% vs. 65%; average latency improvement 43%) may fall within run-to-run variance. The paper should either provide repeated trials with variance estimates or temper the quantitative claims accordingly.
- [Section 3.1.1] NL2Operator is the load-bearing component that maps ambiguous natural-language queries to the semantic operator hierarchy, but no evaluation of translation accuracy is reported. If this mapping fails, downstream filtering, image matching, and cross-modal joins are wrong regardless of how capable the specialized models are. The manuscript needs an evaluation of operator-mapping correctness on a held-out query set, including out-of-scope or ambiguous queries.
- [Section 4.1, Baseline] The baseline also uses PostgreSQL for table filtering, so the comparison isolates the choice of image-analysis model (Qwen2.5-VL-7B vs. GPT-4.1) rather than the MCP-based architecture itself. The text calls the baseline 'pure GPT-4.1' while simultaneously not using a pure model; this inconsistency makes the architectural contribution indistinguishable from a simple model swap. An ablation with a pure unified-LLM baseline, and an ablation where GPT-4.1 performs image analysis within the MCP framework, would be needed to separate the claims.
- [Conclusions, last paragraph] The paper explicitly states, 'In the future, we will finish implementing the whole system and conduct more experiments.' The current prototype covers only query translation, table filtering, and image analysis; the embedding indexing (Section 3.2.1), data augmentor (Section 3.3.1), model refreshing with unlearning (Section 3.3.2), and the cost-based planner (Section 3.1.2) are neither implemented nor evaluated. This undercuts the broader claims of scalability, freshness, and efficiency that form the paper's stated contributions, and the manuscript should present at least component-level evidence or clearly frame the paper as a position paper rather than a systems paper.
minor comments (5)
- [Section 4.1, Evaluation Metric] The text says 'to compute recall and precision' but the metric definition earlier in Section 2.1 lists recall, precision, and latency; please ensure the terms accuracy, precision, and recall are used consistently, since Figure 6 is labeled 'Precision' while the text also reports 'accuracy.'
- [Throughout] There are several misspellings and typos, including 'Knowlege' in Section 2.1, 'fasion' in Section 4.1, 'hugh' for 'huge' in Section 5, 'subcription' in Section 3.3.2, and 'excel' for 'excellent' in Section 4.2. A copyediting pass is needed.
- [Figures 4-8] Figures 4, 5, 6, 7, and 8 are referenced in the text, but the figures themselves are not visible in the manuscript text provided; please ensure they are embedded and readable, and that each has a self-contained caption explaining what is plotted.
- [Section 3.3.1, Redundancy Elimination] The description mentions MinHash, Sentence-BERT, HNSW, and FAISS but gives no parameter values or implementation details; adding a concrete algorithmic specification or pseudocode would improve reproducibility.
- [References] Reference [17] duplicates the GPT-4 technical report already cited as [2]; please consolidate or disambiguate these references.
Circularity Check
No significant circularity: TAIJI is an architecture paper whose preliminary comparison is an external benchmark against GPT-4.1, with no fit-by-construction or self-citation chain.
full rationale
TAIJI contains no formal derivation whose output equals its input, no fitted parameters renamed as predictions, and no load-bearing self-citation. The central claim, that per-modality MCP servers with tailored models improve accuracy and reduce inference overhead, is tested directly by comparing Qwen2.5-VL-7B plus PostgreSQL against GPT-4.1 on a fixed three-query workload over the Craigslist furniture dataset. The paper explicitly fixes the query plan in Section 4.1, so the NL2Operator translator, planner, and optimizer are not exercised; this is an evidence or completeness gap about an untested component, not circularity. Cited works such as MCP, Qwen2.5-VL, ThalamusDB, and Palimpzest are external and do not embed TAIJI's results. The two references involving author Ju Fan ([6] and [13]) are background on NL2SQL and are not used to justify the architecture or to forbid alternative designs. There is thus no instance in which a prediction reduces by construction to its inputs, and the honest finding is no significant circularity.
Assumptions & free parameters
assumptions (4)
- domain assumption A specialized smaller LLM per data modality is more accurate and cheaper than one unified large LLM for multi-modal analytics.
- domain assumption An LLM agent can reliably translate natural language queries into the semantic operator hierarchy.
- domain assumption MCP provides a scalable, low-overhead execution abstraction for dispatching sub-tasks to servers.
- domain assumption Data and model knowledge can be refreshed by web ingestion, fine-tuning, and machine unlearning without degrading other capabilities.
Cite this review
Pith. "Pith review of TAIJI: MCP-based Multi-Modal Data Analytics on Data Lakes." pith.science (2026). https://pith.science/paper/5RWAWQ36
@misc{pith2026250511270,
author = {Pith},
title = {Pith review of: TAIJI: MCP-based Multi-Modal Data Analytics on Data Lakes},
year = {2026},
howpublished = {\url{https://pith.science/paper/5RWAWQ36}},
note = {Machine review of arXiv:2505.11270}
}
read the original abstract
The variety of data in data lakes presents significant challenges for data analytics, as data scientists must simultaneously analyze multi-modal data, including structured, semi-structured, and unstructured data. While Large Language Models (LLMs) have demonstrated promising capabilities, they still remain inadequate for multi-modal data analytics in terms of accuracy, efficiency, and freshness. First, current natural language (NL) or SQL-like query languages may struggle to precisely and comprehensively capture users' analytical intent. Second, relying on a single unified LLM to process diverse data modalities often leads to substantial inference overhead. Third, data stored in data lakes may be incomplete or outdated, making it essential to integrate external open-domain knowledge to generate timely and relevant analytics results. In this paper, we envision a new multi-modal data analytics system. Specifically, we propose a novel architecture built upon the Model Context Protocol (MCP), an emerging paradigm that enables LLMs to collaborate with knowledgeable agents. First, we define a semantic operator hierarchy tailored for querying multi-modal data in data lakes and develop an AI-agent-powered NL2Operator translator to bridge user intent and analytical execution. Next, we introduce an MCP-based execution framework, in which each MCP server hosts specialized foundation models optimized for specific data modalities. This design enhances both accuracy and efficiency, while supporting high scalability through modular deployment. Finally, we propose a updating mechanism by harnessing the deep research and machine unlearning techniques to refresh the data lakes and LLM knowledges, with the goal of balancing the data freshness and inference efficiency.
Figures
Figures from the paper (3 more)
Forward citations
Cited by 2 Pith papers
-
DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces
A new heterogeneous-workspace benchmark for data agents, built from transformed Text-to-SQL tasks, with complete-table deterministic evaluation and a best measured accuracy of 66.34%.
-
FDABench: A Benchmark for Data Agents on Analytical Queries over Heterogeneous Data
FDABench is a 2,007-task benchmark for AI agents that must combine structured databases with documents, web, image, video, and audio sources to answer analytical queries.
Reference graph
Works this paper leans on
-
[1]
https://platform.openai.com/docs/guides/ text-generation
Introduction of openai text generation apis. https://platform.openai.com/docs/guides/ text-generation. 11
- [3]
-
[4]
S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y . Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y . Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin. Qwen2.5-VL Technical Report. arXiv preprint arXiv:2502.13923, 2025
arXiv 2025
-
[5]
R. Chourasia and N. Shah. Forget unlearning: Towards true data-deletion in machine learning. volume 202, pages 6028–6073. PMLR, 2023
work page 2023
-
[6]
J. Fan, Z. Gu, S. Zhang, Y . Zhang, Z. Chen, L. Cao, G. Li, S. Madden, X. Du, and N. Tang. Combining small language models and large language models for zero-shot nl2sql. Proceedings of the VLDB Endowment , 17(11):2750–2763, 2024
work page 2024
-
[7]
A. I. for AI. Science parse: An open-source library for extracting structured data from scientific pdfs, 2018. https: //github.com/allenai/science-parse
work page 2018
-
[8]
X. Hou, Y . Zhao, S. Wang, and H. Wang. Model context protocol (mcp): Landscape, security threats, and future research directions. arXiv preprint arXiv:2503.23278, 2025
arXiv 2025
- [9]
Show all 26 references
-
[10]
Lewis, E
P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W.-t. Yih, T. Rockt¨aschel, S. Riedel, and D. Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing Systems, volume 33, p...
2020
-
[11]
C. Liu, M. Russo, M. Cafarella, L. Cao, P. B. Chen, Z. Chen, M. Franklin, T. Kraska, S. Madden, R. Shahout, et al. Palimpzest: Optimizing ai-powered analytics with declarative query processing. In Proceedings of the Conference on Innovative Database Research (CIDR), 2025
2025
-
[12]
C. Liu, M. Russo, M. Cafarella, L. Cao, P. B. Chen, Z. Chen, M. Franklin, T. Kraska, S. Madden, and G. Vitagliano. A declarative system for optimizing ai workloads. arXiv preprint arXiv:2405.14696, 2024
2024 arXiv
-
[13]
X. Liu, S. Shen, B. Li, P. Ma, R. Jiang, Y . Zhang, J. Fan, G. Li, N. Tang, and Y . Luo. A survey of nl2sql with large language models: Where are we, and where are we going? arXiv preprint arXiv:2408.05109, 2024
2024
-
[14]
P. Lopez. Grobid: Generation of bibliographic data, 2020. https://github.com/kermitt2/grobid
2020
-
[15]
Lu and I
J. Lu and I. Holubov ´a. Multi-model databases: a new journey to handle the variety of data. ACM Computing Surveys (CSUR), 52(3):1–38, 2019
2019
-
[16]
Nargesian, E
F. Nargesian, E. Zhu, R. J. Miller, K. Q. Pu, and P. C. Arocena. Data lake management: challenges and opportunities. Proceedings of the VLDB Endowment, 12(12):1986–1989, 2019
1986
-
[17]
OpenAI and e. a. Achiam. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774, 2023
2023 arXiv
-
[18]
S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez. Gorilla: Large language model connected with massive apis. Advances in Neural Information Processing Systems, 37:126544–126565, 2024
2024
-
[19]
M. C. Protocol. Model context protocol servers, 2025. https://github.com/modelcontextprotocol/ servers
2025
-
[20]
Reimers and I
N. Reimers and I. Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pages 3982–3992, 2019
2019
-
[21]
Schick, J
T. Schick, J. Dwivedi-Yu, R. Dess `ı, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36:68539–68551, 2023. 12
2023
-
[22]
Shuster, M
K. Shuster, M. Komeili, L. Adolphs, S. Roller, A. Szlam, and J. Weston. Language models that seek for knowledge: Modular search and generation for dialogue and prompt completion. InFindings of the Association for Computational Linguistics: EMNLP 2022, pages 373–393, 2022
2022
-
[23]
Urban and C
M. Urban and C. Binnig. Caesura: Language models as multi-modal query planners. arXiv preprint arXiv:2308.03424, 2023
2023 arXiv
-
[24]
Urban and C
M. Urban and C. Binnig. Eleet: Efficient learned query execution over text and tables. Proc. VLDB Endow, 17:13, 2024
2024
-
[25]
Wang and G
J. Wang and G. Li. Aop: Automated and interactive llm pipeline orchestration for answering complex queries. CIDR, 2025
2025
-
[26]
J. Wang, G. Li, and J. Feng. idatalake: An llm-powered analytics system on data lakes. Data Engineering, page 57
-
[27]
Zhang, Z
Y . Zhang, Z. Hu, Y . Bai, J. Wu, Q. Wang, and F. Feng. Recommendation unlearning via influence function. arXiv preprint arXiv:2307.02147, 2023. 13
2023 arXiv
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.