Pith. sign in

REVIEW 3 major objections 5 minor 38 references

Do Data Agents Need Semantic Metadata? A Comparative Study in Agentic Data Retrieval

T0 review · 3 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that for autonomous agents, retrieving usable data still depends on semantic metadata: an agent searching a structured metadata corpus achieved 65.7% higher precision for FAIR-compliant datasets than an otherwise identical

desk verdict First solid head-to-head data on semantic vs open-web dataset retrieval for agents, but the 'indispensable' conclusion outruns the design: the semantic advantage is bundled with quality filtering and dataset-specific ranking. read the letter →

arxiv 2605.28787 v2 pith:YGKTUBGV submitted 2026-05-27 cs.IR cs.AI

Shiyu Chen , Tarfah Alrashed , Alon Halevy , Natasha Noy This is my paper · ORCID
classification cs.IRcs.AI
keywords semanticmetadataschema.orgagenticdataretrievalFAIRprinciplesLLM-as-a-judgedatasetdiscoveryautonomousagentsprecision
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether autonomous data-retrieval agents still need semantic metadata, or whether large language models can reliably find usable data on the open web. To answer it, the authors built two nearly identical agents: one queries a general web search index, the other queries a curated index of about 90 million dataset records annotated with semantic markup. Using an LLM-as-a-judge evaluation mapped to the FAIR principles, they find the metadata-based agent retrieves machine-actionable, FAIR-compliant datasets with 65.7% higher precision. The open-web agent answers 40% more questions but frequently fails the 'last mile', returning prose-heavy pages and portal landing pages instead of actual data. The paper concludes that while unstructured retrieval supports broad exploratory tasks, structured metadata remains the foundation for reliable, execution-oriented autonomous workflows.

What carries the argument

The central mechanism is the structured corpus created by schema.org/Dataset markup: a filtered index of roughly 90 million dataset records, each carrying explicit machine-readable fields about the data. This lets the agent land directly on metadata-rich registry pages with direct download links or APIs. The evaluation machinery is a composite FAIR metric that requires a relevance score of 2, an accessibility level of 6 (machine-readable), and a DATA_REGISTRY page type, scored by an LLM judge that follows a chain-of-thought protocol on frozen Markdown snapshots of retrieved pages.

What would settle it

Run both agents against the same dataset index with schema.org annotations stripped out, so the only difference is the presence of semantic metadata while ranking and tool logic are held identical—if the precision gap vanishes or shrinks dramatically, the claim that metadata is the decisive factor fails.

Watch

Extended reading notes

Core claim

The central claim is that semantic metadata still matters for agentic data retrieval when the goal is action, not just discovery. The Semantic Agent, searching a corpus of schema.org-annotated dataset records, achieved 46.4% precision on a composite FAIR metric (relevance score 2, accessibility level 'machine-readable', and page type DATA_REGISTRY) versus 28.0% for the Baseline Agent that searches the open web—a 65.7% relative improvement. The Semantic Agent also returned machine-readable pages in 71.4% of its retrievals, compared to 48.7% for the Baseline, and reduced narrative-heavy pages, presentation-bound data, and non-data pages by 46.6% to 86.6%. The Baseline Agent had higher coverage

Load-bearing premise

The conclusion assumes the measured precision gap is caused by the presence of semantic metadata, not by differences in how the two live search engines rank their results, which the paper acknowledges are black boxes.

Editorial extensions

If this is right

  • Autonomous data workflows that need execution-ready payloads should prioritize metadata-indexed dataset endpoints over general web search.
  • A hybrid architecture—query the semantic layer first, fall back to the open web only on empty results—can combine precision with broad coverage.
  • The 'fail-fast' behavior of structured retrieval (returning nothing rather than a probabilistic guess) may be preferable for agents where false positives are costlier than empty states.
  • Investment in semantic metadata publication remains valuable for making datasets usable by machines, not just findable by humans.
  • LLM-as-a-judge evaluation mapped to FAIR principles can serve as a scalable proxy for assessing agent data retrieval quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extension: The 40% coverage gap suggests the main practical cost of semantic metadata is its limited reach; if schema.org adoption grows, the fallback-to-open-web step may become unnecessary for most queries.
  • Extension: The composite FAIR metric is strict—a relevant dataset served as a plain CSV without registry metadata would not count—so the true precision gap for practical workflows could be smaller or larger depending on whether the task needs full provenance or just a download link.
  • Extension: The advantage may generalize to other structured metadata standards (such as DCAT) if an equivalent filtered index exists, but the paper only tests the schema.org-based ecosystem.
  • Extension: As LLMs improve at extracting data from prose, the open-web agent's 'last mile' failures may shrink, but ranking noise and portal-trapping appear to be independent bottlenecks that better text extraction alone may not solve.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper reports a comparative study of two agentic data-retrieval systems built on the same ADK/Gemini 2.5 Pro stack: a Baseline Agent searching the general Google Search index and a Semantic Agent searching Google Dataset Search, a corpus of roughly 90 million schema.org/Dataset-tagged metadata records. Using 58 English keyword queries from NTCIR-16 Data Search 2, the authors evaluate retrieved pages with an LLM-as-a-judge pipeline mapped to three FAIR-inspired dimensions: relevance, data accessibility, and dataset page type. They report that the Semantic Agent achieves higher precision on machine-readable and registry-type results and a 65.7% relative improvement on a composite 'agentic FAIRness' metric (46.4% vs. 28.0%, p<0.01), while the Baseline Agent answers more queries (56 vs. 40). The paper concludes that structured semantic ecosystems are 'the indispensable foundation' for reliable, execution-oriented autonomous workflows.

Significance. The question is timely and practically important, and the paper has real strengths: it evaluates on a public benchmark query set rather than self-authored queries, uses live web-scale indices, publishes the full evaluator prompts, validates the LLM judge against human gold labels (κ≈0.73–0.78), and is transparent about scraping failures and ranking black boxes. If the causal claim were established, the result would be of broad interest to the Semantic Web, data-discovery, and agent-systems communities. However, the significance is currently limited because the experiment compares two complete ecosystems, not the presence versus absence of semantic metadata. The Semantic Agent differs from the Baseline in corpus curation, a quality classifier, a dataset-specific index/ranker, and a structured tool payload, in addition to the metadata itself. The composite FAIR metric is also defined in a way that partially encodes the Semantic Agent's natural output type. The paper's central conclusion is therefore stronger than the design supports. The authors' own §7 acknowledges that the evaluation measures end-to-end utility rather than isolating the 'algorithmic delta.' With additional analysis

major comments (3)
  1. [§3.1, §3.2, §7, §8] The central claim — that semantic metadata is 'indispensable' — is not established by the experimental design. The Semantic Agent differs from the Baseline in at least four bundled factors: (1) corpus inclusion restricted to schema.org/Dataset pages, (2) a machine-learned quality filter removing ~80% of invalid annotations (§3.1), (3) a dataset-specific index and ranking system (§3.2), and (4) a structured metadata payload returned to the agent, versus synthesized snippets. Any of factors (2)–(4) could produce the measured 65.7% precision advantage. §7 concedes that the ranking delta is not isolated and that the evaluation measures end-to-end utility. The conclusion in §8 that structured ecosystems are 'the indispensable foundation' causal wording is therefore not supported. I would like to see either (a) a decisive control, e.g., using a general web index restricted to the same metadata
  2. [§5.4] The composite 'FAIR-compliant' definition is problematic for the paper's main precision metric. A dataset is called FAIR-compliant only if it is simultaneously score-2 relevant, Accessibility Level 6, and DATA_REGISTRY. This excludes RAW_DATA pages — direct machine-readable files or APIs — which are arguably the most directly actionable. Since the Semantic Agent operates over a registry-indexed corpus, requiring DATA_REGISTRY partially builds the Semantic Agent's advantage into the metric. The 65.7% relative improvement is the headline result, so this definition needs justification. At minimum, report the same comparison with an alternative composite that treats RAW_DATA as compliant, and show that the conclusion is robust to the choice of page-type threshold.
  3. [§3.1, §5.4] The Baseline Agent is given query expansion ('dataset') while the Semantic Agent is not, and the authors describe this as 'redundant' for the Semantic Agent. I agree this asymmetry is unlikely to favor the Baseline, and it is not the main threat to validity. Nevertheless, for a paper whose goal is an apples-to-apples comparison, the asymmetry should be justified empirically or at least controlled by running the Baseline both with and without query expansion. Without such a control, the magnitude of the precision gap cannot be cleanly attributed to the retrieval environment.
minor comments (5)
  1. [§5.4] The p<0.01 for dataset-level precision is reported without specifying the statistical test. Please state the test (e.g., Fisher's exact test or a bootstrap) and, ideally, a confidence interval for the precision difference.
  2. [§4.3] There is a typo: 'we mapped the autorater's classifications to progressive ordinal scales for relevance andaccessibilityandand derived thescalefor...' should be 'and accessibility and derived the scale for...'.
  3. [§4.3] When 31% of pages are routed to human evaluation, it would be clearer to state explicitly how these human labels are combined with the LLM labels in the final dataset and whether any weighting is applied, since the two annotation paths may have different distributions.
  4. [§4.2/§5.1] The rubric says an unreachable page receives Relevance -1, but §4.3 says 'non-ordinal Unreachable categories' are excluded before kappa computation. Please clarify how unreachable pages enter the reported precision and accessibility distributions.
  5. [Abstract/§5.2] The abstract reports '44.9% higher precision for metadata-rich registries' and '46.6% higher precision for pages with machine-readable downloads.' In the body these appear as relative increases over the Baseline; please make 'relative' explicit in the abstract to avoid misreading as absolute percentage-point gains.

Circularity Check

1 steps flagged · score 6.0 of 10

FAIR-compliant precision is partially self-definitional: the composite metric requires DATA_REGISTRY pages, which is the defining property of the Semantic Agent's corpus.

  1. self definitional [§5.4 (Agentic FAIRness), with §3.1 (Semantic Agent setup) and Table 4 (DATA_REGISTRY definition)]
    "The agent queries the Google Dataset Search index via a metadata search tool. Representing the structured ecosystem, this curated corpus aggregates all pages with schema.org/Dataset markup. ... We define a dataset as fully “FAIR-compliant” if it achieves a perfect composite score across three criteria: a relevance score of 2 (Highly Relevant), Dataset Accessibility at Level 6 (Machine-Readable), and Dataset Page Type of DATA_REGISTRY."

    The paper's headline quantitative result—65.7% higher 'FAIR-compliant' precision—uses DATA_REGISTRY as a required component. DATA_REGISTRY is defined in Table 4 as 'Dataset landing pages with interoperable metadata records (e.g., DOIs, data dictionaries, provenance)', which is essentially the property used to build the Semantic Agent's corpus: pages with schema.org/Dataset markup. Thus the Semantic Agent's higher DATA_REGISTRY rate (88.4% vs 61.0%) and the composite precision advantage are substantially built into the evaluation definition rather than discovered empirically. The conclusion that structured metadata is 'indispensable' partly restates the input selection criterion as the outcome. The relevance and accessibility components are not definitionally guaranteed, so the circularity

full rationale

The study is not globally circular: it uses an external NTCIR-16 query set, publishes the judge prompts, validates the LLM judges against human labels, and does not fit parameters to the outcome. However, the central quantitative claim reduces partly by construction. The composite 'FAIR-compliant' metric requires the retrieved page to be DATA_REGISTRY, a page type characterized by interoperable metadata records; the Semantic Agent's corpus is definitionally composed of pages carrying schema.org/Dataset markup. Therefore the 65.7% precision advantage is, to a substantial degree, the treatment definition renamed as the evaluation outcome. Confounds such as the proprietary ranking black box, the quality-filter classifier, and end-to-end versus ablative measurement are genuine validity threats but are not circularity, and the paper's own §7 limitation statement acknowledges them. Self-citations to prior Google Dataset Search work [4,5,22,29] describe and motivate the corpus but are not an unverified uniqueness or ansatz chain, so they do not raise the score further. Because relevance and accessibility are independent criteria that could in principle have favored the Baseline, the circularity is partial rather than total, yielding a score of 6.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No fitted parameters: all numbers are measured from system outputs. The load-bearing assumptions are the representativeness of the 58-query sample, the validity of the LLM judge, the author-defined FAIR composite, the comparability of the two live search environments, and the adequacy of scraped Markdown for judging page content. No invented entities are introduced.

assumptions (5)
  • domain assumption NTCIR-16 Data Search 2 English keyword queries (N=58) are a representative sample of real data discovery intent.
    §4.1: only 58 queries from the IR subtask; no justification of sufficiency for web-scale claims.
  • domain assumption Gemini 2.5 Pro LLM-as-a-judge reliably approximates human FAIR assessment.
    §4.3: reported linear-weighted kappa 0.73–0.78 against an author-built gold set; moderate agreement is treated as ground truth.
  • ad hoc to paper A dataset is FAIR-compliant iff relevance=2, accessibility level 6, and page type DATA_REGISTRY.
    §5.4: composite metric is defined by the authors for this evaluation, not derived from the FAIR principles.
  • domain assumption Differences between Google Search and Google Dataset Search are attributable to semantic metadata, not to ranking or index engineering.
    §3.2/§7: the authors acknowledge ranking mechanisms are black boxes; the end-to-end comparison cannot isolate the metadata variable.
  • domain assumption Serialized Markdown snapshots of pages are sufficient to judge data accessibility and page type.
    §4.3/§7: scraping failed for 31% of pages, so those were labeled manually by the authors; content representation may affect LLM judgment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do Data Agents Need Semantic Metadata? A Comparative Study in Agentic Data Retrieval." pith.science (2026). https://pith.science/paper/YGKTUBGV

@misc{pith2026260528787,
  author       = {Pith},
  title        = {Pith review of: Do Data Agents Need Semantic Metadata? A Comparative Study in Agentic Data Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YGKTUBGV}},
  note         = {Machine review of arXiv:2605.28787}
}
abstract

In the era of autonomous agents, machine-actionable data is critical for data-driven workflows. For more than a decade, semantic metadata like schema$.$org has anchored the FAIR principles (Findable, Accessible, Interoperable, and Reusable) for machine-actionable data and enabled discovery tools like Google Dataset Search. However, the rise of Large Language Models (LLMs) capable of navigating the unstructured web raises a fundamental question: Is semantic metadata still necessary for agentic data discovery, or can agents reliably retrieve actionable data directly from the web? We present a comparative analysis of agentic data retrieval across two distinct environments: a Baseline Agent searching billions of open-web documents, and a Semantic Agent leveraging a corpus of 90 million datasets using schema$.$org. We deploy an "LLM-as-a-judge" evaluation pipeline, mapped directly to the FAIR principles, to assess the semantic relevance, data accessibility, and computational utility of the retrieved data. Our results reveal a clear divergence. The Semantic Agent excels at retrieving actionable data, achieving a 44.9% higher precision for metadata-rich registries and a 46.6% higher precision for pages with machine-readable downloads among its returned results. Conversely, the Baseline Agent frequently suffers "Last-Mile Utility" failures, retrieving prose-heavy pages (20.1% of results) and portal landing pages (8.5%) rather than actual data pages. While the Baseline Agent achieves higher coverage by answering 40% more questions, the Semantic Agent delivers greater accuracy, achieving 65.7% higher overall precision in retrieving FAIR-compliant datasets. We conclude that while unstructured retrieval supports broad exploratory tasks, structured ecosystems remain the indispensable foundation for reliable, execution-oriented autonomous workflows.

Figures

Figures reproduced from arXiv: 2605.28787 by the authors.

Figure 1
Figure 1. Comparative System Architecture. Similar agent logic is evaluated across un￾structured Baseline Agent and Semantic Agent dataset search environments. Both feed a unified, FAIR-aligned evaluation of relevance, accessibility, and utility. 3.1 Agentic Framework and Setup To ensure experimental parity, we contrast a Semantic Agent against a Base￾line Agent using identical underlying architectures ( [PITH_FULL_IMAGE:fig… view at source ↗
Figure 2
Figure 2. Comparison of the agent results by relevance scores. semantic metadata, the agent avoided non-computational roadblocks, achieving relative reductions of 46.6% in Narrative/Unstructured Data (data embedded in narrative prose), 62.9% in Presentation-Bounded Data (pages with only charts or interactive dashboards, without metadata), and 76.3% in Non-Data (false positive pages lacking data entirely) [PITH_FULL_IMAGE:fig… view at source ↗
Figure 3
Figure 3. Comparison of the agent results by data accessibility levels 5.3 Dataset Page Type The distribution of retrieved dataset type pages varied between the two systems ( [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Comparison of the agent results by dataset page type 5.4 Agentic FAIRness To evaluate an agent’s capacity to identify machine-actionable registry entries that ensure metadata compliance and direct data accessibility. We define a dataset as fully “FAIR-compliant” if it …

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 5 linked inside Pith

  1. [1]

    Aghzal, M., Stein, G.J., Yao, Z.: Why Do LLM-based Web Agents Fail? A Hierar- chical Planning Perspective (2026)

    2026

  2. [2]

    W3C recommendation, World Wide Web Consortium (W3C) (2024), https://www.w3.org/TR/vocab- dcat-3/

    Albertoni, R., Browning, D., Cox, S.J.D., Gonzalez Beltran, A., Perego, A., Win- stanley, P.: Data catalog vocabulary (DCAT) - version 3. W3C recommendation, World Wide Web Consortium (W3C) (2024), https://www.w3.org/TR/vocab- dcat-3/

    2024

  3. [3]

    Data Intelligence6(2), 457–487 (2024)

    Albertoni, R., Browning, D., Cox, S.J.D., Gonzalez-Beltran, A.N., Perego, A., Winstanley, P.: The W3C data catalog vocabulary, version 2: Ratio- nale, design principles, and uptake. Data Intelligence6(2), 457–487 (2024). https://doi.org/10.1162/dint_a_00241

    doi:10.1162/dint_a_00241 2024

  4. [4]

    In: The Semantic Web – ISWC 2021: 20th International Semantic Web Conference, ISWC 2021, Virtual Event, October 24–28, 2021, Proceedings

    Alrashed, T., Paparas, D., Benjelloun, O., Sheng, Y., Noy, N.: Dataset or Not? A Study on the Veracity of Semantic Markup for Dataset Pages. In: The Semantic Web – ISWC 2021: 20th International Semantic Web Conference, ISWC 2021, Virtual Event, October 24–28, 2021, Proceedings. p. 338–356. Springer-Verlag, Berlin, Heidelberg (2021), https://doi.org/10.100...

    doi:10.1007/978-3-030-88361-4_20 2021

  5. [5]

    In: International Semantic Web Conference (ISWC-2020), In-Use Track (2020), https://arxiv.org/abs/2006.06894

    Benjelloun, O., Chen, S., Noy, N.: Google Dataset Search by the numbers. In: International Semantic Web Conference (ISWC-2020), In-Use Track (2020), https://arxiv.org/abs/2006.06894

    arXiv 2020

  6. [6]

    The VLDB Journal29(1), 251–272 (Aug 2019), https://doi.org/10.1007/s00778-019-00564-x

    Chapman,A.,Simperl,E.,Koesten,L.,Konstantinidis,G.,Ibáñez,L.D.,Kacprzak, E., Groth, P.: Dataset Search: A Survey. The VLDB Journal29(1), 251–272 (Aug 2019), https://doi.org/10.1007/s00778-019-00564-x

    doi:10.1007/s00778-019-00564-x 2019

  7. [7]

    Chezelles, T.L.S.D., Gasse, M., Drouin, A., Caccia, M., Boisvert, L., Thakkar, M., Marty, T., Assouel, R., Shayegan, S.O., Jang, L.K., Lù, X.H., Yoran, O., Kong, D., Xu, F.F., Reddy, S., Cappart, Q., Neubig, G., Salakhutdinov, R., Chapados, N., Lacoste, A.: The BrowserGym Ecosystem for Web Agent Research (2025), https://arxiv.org/abs/2412.05467

    arXiv 2025

  8. [8]

    (eds.) Proceed- ings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Net- works for NLP

    Chiang, C.H., Lee, H.y.: Do Metadata and Appearance of the Retrieved Web- pages Affect LLM’s Reasoning in Retrieval-Augmented Generation? In: Belinkov, Y., Kim, N., Jumelet, J., Mohebbi, H., Mueller, A., Chen, H. (eds.) Proceed- ings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Net- works for NLP. pp. 389–406. Association for Computa...

    doi:10.18653/v1/2024.blackboxnlp-1.24 2024

Show all 38 references
  1. [9]

    Edge, D., Trinh, H., Cheng, N., Bradley, J., Chao, A., Mody, A., Tru- itt, S., Metropolitansky, D., Ness, R.O., Larson, J.: From Local to Global: A Graph RAG Approach to Query-Focused Summarization (2025), https://arxiv.org/abs/2404.16130

    2025 arXiv

  2. [10]

    Google: Agent Development Kit (ADK) (2026), https://adk.dev/, accessed: 2026- 04-22 Do Agents Need Semantic Metadata? 17

    2026

  3. [11]

    Guha, R.V., Brickley, D., Macbeth, S.: Schema.org: evolution of struc- tured data on the web. Commun. ACM59(2), 44–51 (Jan 2016), https://doi.org/10.1145/2844544

    2016 doi

  4. [12]

    In: Bouamor, H., Pino, J., Bali, K

    Gur, I., Nachum, O., Miao, Y., Safdari, M., Huang, A., Chowdhery, A., Narang, S., Fiedel, N., Faust, A.: Understanding HTML with Large Language Models. In: Bouamor, H., Pino, J., Bali, K. (eds.) Findings of the Association for Com- putational Linguistics: EMNLP 2023. pp. 2803–...

    2023

  5. [13]

    In: Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., Sabato, S

    Humphreys, P.C., Raposo, D., Pohlen, T., Thornton, G., Chhaparia, R., Mul- dal, A., Abramson, J., Georgiev, P., Santoro, A., Lillicrap, T.: A data- driven approach for learning to control computers. In: Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., Sabato, S. ...

    2022

  6. [14]

    In: Bouamor, H., Pino, J., Bali, K

    Jiang, J., Zhou, K., Dong, Z., Ye, K., Zhao, X., Wen, J.R.: Struct- GPT: A General Framework for Large Language Model to Reason over Structured Data. In: Bouamor, H., Pino, J., Bali, K. (eds.) Proceed- ings of the 2023 Conference on Empirical Methods in Natural Lan- guage Proc...

    2023 doi

  7. [15]

    Kato, M.P., Ohshima, H., Liu, Y.H., Chen, H.L.: NTCIR-16 Data Search 2 (2022)

    2022

  8. [16]

    In: Proceedings of the 34th In- ternational Conference on Neural Information Processing Systems

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.t., Rocktäschel, T., Riedel, S., Kiela, D.: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In: Proceedings of the 34th In- ternational Conference on Neura...

    2020

  9. [17]

    Li, X., Lyu, T., Yang, Y., Shan, L., Yang, S., Zhang, L., Huang, Z., Liu, Q., Li, Y.: Escaping the Context Bottleneck: Active Context Curation for LLM Agents via Reinforcement Learning (2026)

    2026

  10. [18]

    In: Kim, B., Yue, Y., Chaudhuri, S., Fragkiadaki, K., Khan, M., Sun, Y

    Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang, K., Zhang, S., Deng, X., Zeng, A., Du, Z., Zhang, C., Shen, S., Zhang, T., Su, Y., Sun, H., Huang, M., Dong, Y., Tang, J.: AgentBench: Evaluating LLMs as Agents. In: Kim, B., Yue, Y., Chaudh...

    2024

  11. [19]

    In: Bouamor, H., Pino, J., Bali, K

    Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., Zhu, C.: G-eval: NLG Evaluation using GPT-4 with Better Human Alignment. In: Bouamor, H., Pino, J., Bali, K. (eds.) Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. pp. 2511–2522. Association f...

    2023

  12. [20]

    Information Services and Use37(1), 49–56 (2017)

    Mons, B., Neylon, C., Velterop, J., Dumontier, M., da Silva Santos, L.O.B., Wilkin- son, M.D.: Cloudy, increasingly FAIR; revisiting the FAIR Data guiding principles for the European Open Science Cloud. Information Services and Use37(1), 49–56 (2017)

    2017

  13. [21]

    Chen et al

    Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., Jiang, X., Cobbe, K., Eloundou, T., Krueger, G., But- 18 S. Chen et al. ton, K., Knight, M., Chess, B., Schulman, J.: WebGPT: Browser-assisted question- answering ...

    2022 arXiv

  14. [22]

    Noy, N., Burgess, M., Brickley, D.: Google Dataset Search: Building a Search En- gineforDatasetsinanOpenWebEcosystem.In:TheWorldWideWebConference. p. 1365–1375. WWW ’19, Association for Computing Machinery, New York, NY, USA (2019), https://doi.org/10.1145/3308558.3313685

    2019

  15. [23]

    In: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

    Overwijk, A., Xiong, C., Callan, J.: ClueWeb22: 10 Billion Web Docu- ments with Rich Information. In: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. p. 3360–3362. SIGIR ’22, Association for Computing Machin- ery...

    2022

  16. [24]

    IEEE Transactions on Knowledge and Data Engineering36(7), 3580–3599 (2024)

    Pan, S., Luo, L., Wang, Y., Chen, C., Wang, J., Wu, X.: Unify- ing Large Language Models and Knowledge Graphs: A Roadmap. IEEE Transactions on Knowledge and Data Engineering36(7), 3580–3599 (2024). https://doi.org/10.1109/TKDE.2024.3352100

    2024

  17. [25]

    In: Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology

    Park, J.S., O’Brien, J., Cai, C.J., Morris, M.R., Liang, P., Bernstein, M.S.: Generative agents: Interactive simulacra of human behavior. In: Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. UIST ’23, Association for Computing Machinery, ...

    2023

  18. [26]

    Patil, S.G., Zhang, T., Wang, X., Gonzalez, J.E.: Gorilla: Large Language Model ConnectedwithMassiveAPIs.AdvancesinNeuralInformationProcessingSystems 37, 126544–126565 (2024)

    2024

  19. [27]

    In: The Twelfth International Conference on Learning Represen- tations

    Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L., Lu, Y., Lin, Y., Cong, X., Tang, X., Qian, B., et al.: ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. In: The Twelfth International Conference on Learning Represen- tations. ICLR ’24, https://openrevi...

  20. [28]

    In: Proceedings of the 37th International Conference on Neural Information Processing Systems

    Schick, T., Dwivedi-Yu, J., Dessí, R., Raileanu, R., Lomeli, M., Hambro, E., Zettle- moyer, L., Cancedda, N., Scialom, T.: Toolformer: Language Models Can Teach Themselves to Use Tools. In: Proceedings of the 37th International Conference on Neural Information Processing Syste...

  21. [29]

    Harvard Data Science Review (Special Issue 4) (apr 2 2024), https://hdsr.mitpress.mit.edu/pub/psnc8zsr

    Sostek, K., Russell, D.M., Goyal, N., Alrashed, T., Dugall, S., Noy, N.: Discov- ering Datasets on the Web Scale: Challenges and Recommendations for Google Dataset Search. Harvard Data Science Review (Special Issue 4) (apr 2 2024), https://hdsr.mitpress.mit.edu/pub/psnc8zsr

    2024

  22. [30]

    In: Proceedings of the ACM on Web Conference 2025

    Tan, J., Dou, Z., Wang, W., Wang, M., Chen, W., Wen, J.R.: HtmlRAG: HTML is Better Than Plain Text for Modeling Retrieved Knowledge in RAG Systems. In: Proceedings of the ACM on Web Conference 2025. p. 1733–1746. WWW ’25, Association for Computing Machinery, New York, NY, USA ...

    2025

  23. [31]

    Scientific Data8(1), 192 (2021), https://doi.org/10.1038/s41597-021-00981-0

    Tedersoo, L., Küngas, R., Oras, E., Köster, K., Eenmaa, H., Leijen, Ä., Pedaste, M., Raju, M., Astapova, A., Lukner, H., Kogermann, K., Sepp, T.: Data shar- ing practices and data availability upon request differ across scientific disciplines. Scientific Data8(1), 192 (2021), ...

    2021 doi

  24. [32]

    arXiv:2406.06519 (2024) Do Agents Need Semantic Metadata? 19

    Upadhyay, S., Pradeep, R., Thakur, N., Craswell, N., Lin, J.: UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor. arXiv:2406.06519 (2024) Do Agents Need Semantic Metadata? 19

    2024 arXiv

  25. [33]

    In: Advances in Neural Information Processing Systems

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., Zhou, D.: Chain- of-Thought Prompting Elicits Reasoning in Large Language Models. In: Advances in Neural Information Processing Systems. vol. 35, pp. 24824–24837 (2022)

    2022

  26. [34]

    Scientific Data3(1), 1–9 (2016), https://doi.org/10.1038/sdata.2016.18

    Wilkinson, M.D., Dumontier, M., Aalbersberg, I.J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.W., da Silva Santos, L.B., Bourne, P.E., et al.: The FAIR Guiding Principles for Scientific Data Management and Stewardship. Scientific Data3(1), 1–9 (2016), https://d...

    2016 doi

  27. [35]

    In: The Eleventh International Conference on Learning Representations (2023), https://openreview.net/forum?id=WE_vluYUL-X

    Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K.R., Cao, Y.: ReAct: Synergizing Reasoning and Acting in Language Models. In: The Eleventh International Conference on Learning Representations (2023), https://openreview.net/forum?id=WE_vluYUL-X

    2023

  28. [36]

    In: Proceedings of the 37th International Con- ference on Neural Information Processing Systems

    Zheng, L., Chiang, W.L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E.P., Zhang, H., Gonzalez, J.E., Stoica, I.: Judging LLM-as-a-judge with MT-bench and Chatbot Arena. In: Proceedings of the 37th International Con- ference on Neural Information ...

    2023

  29. [37]

    (eds.) International Conference on Learning Representations

    Zhou, S., Xu, F.F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., Alon, U., Neubig, G.: WebArena: A Realistic Web Environment for BuildingAutonomousAgents.In:Kim,B.,Yue,Y.,Chaudhuri,S.,Fragkiadaki,K., Khan, M., Sun, Y. (eds.) International Con...

    2024

  30. [38]

    PLoS one15(9), e0239283 (2020), https://doi.org/10.1371/journal.pone.0239283

    Zuiderwijk, A., Shinde, R., Jeng, W.: What Drives and Inhibits Researchers to Share and Use Open Research Data? A Systematic Literature Review to Analyze Factors Influencing Open Research Data Adoption. PLoS one15(9), e0239283 (2020), https://doi.org/10.1371/journal.pone.0239283

    2020 doi

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.