ESOFinder: an LLM-powered tool to help users navigate ESO documentation

A. Barnes; C. Reinero; J. Pritchard; M. Marsset; M. Rejkuba; M. Romaniello; M. Vioque; M. Wittkowski; P. S\'anchez-S\'aez

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

ESOFinder uses a local open-source LLM with multi-step retrieval to answer ESO documentation queries and link to sources.

2026-06-30 04:04 UTC pith:YT4UOUDL

load-bearing objection ESOFinder is a standard RAG chatbot for ESO docs with no metrics or comparisons to back its reliability claims. the 1 major comments →

arxiv 2606.30029 v1 pith:YT4UOUDL submitted 2026-06-29 astro-ph.IM

ESOFinder: an LLM-powered tool to help users navigate ESO documentation

P. S\'anchez-S\'aez , C. Reinero , M. Vioque , M. Wittkowski , M. Rejkuba , M. Romaniello , A. Barnes , J. Pritchard

show 1 more author

M. Marsset

This is my paper

classification astro-ph.IM

keywords ESOLLMRAGchatbotdocumentation navigationastronomy toolsdata processingproposal preparation

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

ESOFinder is an in-house chatbot that uses large language models and retrieval augmented generation to help users find information in ESO's extensive documentation. The tool covers the full observing lifecycle from proposal preparation to data processing by integrating manuals, knowledge bases, and web resources. It runs on open-source models locally to ensure privacy and includes a multi-step process to verify sources and reduce incorrect answers. This addresses the challenge of navigating over 3500 links and 100 manuals for astronomers. The system is currently being tested internally to refine its coverage and accuracy.

Core claim

ESOFinder integrates public information from instrument manuals, phase 1/2/3 documentation, data reduction pipeline manuals, the ESO Knowledge Base, and key web resources to provide concise, context-aware, and reference-linked answers to user queries about proposal or observation preparation, data retrieval, and data processing. Its multi-step architecture allows verification of retrieved documents and generated answers, reducing the risk of hallucinations and improving the reliability of responses compared to commercial tools.

What carries the argument

The multi-step retrieval augmented generation architecture that verifies retrieved documents and generated answers before responding.

Load-bearing premise

The multi-step architecture allows verification of retrieved documents and generated answers, thereby reducing the risk of hallucinations and improving reliability.

What would settle it

Quantitative evaluation metrics from the ongoing internal tests showing lower hallucination rates or higher accuracy compared to commercial LLM tools without the multi-step verification.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Supports users in proposal or observation preparation, data retrieval, and data processing.
Ensures data privacy and transparency through local open-source implementation.
Allows for ongoing improvements based on internal testing, such as adding more documentation.
Provides an interface for navigating ESO's documentation ecosystem for both staff and community astronomers.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The approach could be extended to other astronomical observatories facing similar documentation challenges.
Integration with existing ESO workflows might streamline daily tasks for astronomers beyond just answering queries.
Future enhancements in retrieval strategies could further improve accuracy for complex questions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

ESOFinder is a standard RAG chatbot for ESO docs with no metrics or comparisons to back its reliability claims.

read the letter

The paper reports the development of ESOFinder, a local open-source LLM tool that uses retrieval-augmented generation on ESO manuals, pipelines, and web resources to answer user questions about proposals, observations, and data handling.

It does one thing straightforwardly: it assembles more than 3500 links and 100 manuals into a single, privacy-controlled interface that runs on local hardware. That setup gives ESO staff and users a contained alternative to commercial chatbots, which is a practical step for an observatory with complex documentation.

The main weakness is the absence of any evaluation. The abstract states that the multi-step architecture reduces hallucination risk relative to commercial tools, yet the manuscript supplies no hallucination rates, no side-by-side tests against GPT-4 or similar models, no user accuracy scores, and no ablation of the verification steps. Internal testing is described only as planned future work. Without those data the reliability advantage remains an assertion.

The work is a tool-description report rather than a methods advance. ESO users or staff building comparable internal systems may find the architecture choices and data sources worth noting for their own implementations. Outside that narrow group the paper adds little that is new or measurable.

I would not bring this to a reading group. It does not yet merit peer review because the central performance claim lacks supporting evidence; a short software note could be appropriate once quantitative results are added.

Referee Report

1 major / 0 minor

Summary. The manuscript describes ESOFinder, an in-house LLM and RAG-powered chatbot that integrates ESO documentation (instrument manuals, phase 1/2/3 docs, pipelines, knowledge base, and >3500 web links) to answer queries on proposal preparation, observations, data retrieval, and processing. It runs on local open-source models for privacy and control, employs a multi-step architecture for document and answer verification, and states that this reduces hallucination risk relative to commercial tools. Internal testing at ESO is underway to assess performance, with plans for future improvements.

Significance. If the multi-step verification architecture demonstrably lowers hallucination rates, ESOFinder could meaningfully improve documentation navigation efficiency for ESO users across the observing lifecycle. The local open-source implementation and emphasis on reference-linked outputs are strengths for transparency and data privacy. The work provides a concrete example of applying RAG to a large, domain-specific knowledge base in astronomy.

major comments (1)

[Abstract] Abstract: The central claim that the multi-step architecture 'reducing the risk of hallucinations and improving the reliability of responses compared to commercial tools' is stated without any supporting quantitative data (hallucination rates, accuracy metrics, side-by-side comparisons to GPT-4 or similar, ablation studies of verification steps, or user-study results). The manuscript defers all such evaluation to future internal testing, leaving the reliability advantage as an assertion rather than a demonstrated result.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their constructive review of our manuscript on ESOFinder. We address the single major comment below and agree that revisions are needed to ensure claims are supported by the presented content.

read point-by-point responses

Referee: [Abstract] Abstract: The central claim that the multi-step architecture 'reducing the risk of hallucinations and improving the reliability of responses compared to commercial tools' is stated without any supporting quantitative data (hallucination rates, accuracy metrics, side-by-side comparisons to GPT-4 or similar, ablation studies of verification steps, or user-study results). The manuscript defers all such evaluation to future internal testing, leaving the reliability advantage as an assertion rather than a demonstrated result.

Authors: We agree that the abstract asserts a reliability advantage without quantitative support, as all performance evaluation is described as future work. The multi-step architecture is presented as enabling document and answer verification, but no metrics or comparisons are provided in the current manuscript. We will revise the abstract to remove the comparative claim and instead describe the architecture's design intent without asserting demonstrated superiority over commercial tools. This will align the text with the manuscript's actual content. revision: yes

Circularity Check

0 steps flagged

No circularity: descriptive tool report with no derivations or self-referential claims

full rationale

The manuscript is a tool-description paper introducing ESOFinder, an LLM+RAG chatbot. It contains no equations, fitted parameters, predictions, or derivation chains of any kind. The multi-step architecture is presented as an engineering design choice whose benefit (reduced hallucination risk) is asserted without supporting metrics or comparisons; this is an evidentiary gap rather than a circular reduction. No self-citations, uniqueness theorems, or ansatzes are invoked to justify core claims. The paper is therefore self-contained as a software report and exhibits none of the enumerated circularity patterns.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No mathematical model, free parameters, or invented physical entities; the work rests on standard assumptions that LLMs can be prompted for factual retrieval and that the chosen document collection is representative.

pith-pipeline@v0.9.1-grok · 5838 in / 1032 out tokens · 37960 ms · 2026-06-30T04:04:52.851765+00:00 · methodology

0 comments

read the original abstract

The large amount and diversity of documentation available for users of the European Southern Observatory (ESO), spanning the full observing lifecycle from proposal preparation and observation planning to data reduction and archival access, makes it increasingly challenging for the astronomical community to efficiently find relevant information. To address this, we have developed ESOFinder, an in-house chatbot powered by Large Language Models (LLMs) and Retrieval Augmented Generation (RAG). ESOFinder integrates public information from instrument manuals, phase 1/2/3 documentation, data reduction pipeline manuals, the ESO Knowledge Base, and key web resources (spanning more than 3500 links and over 100 manuals) to provide concise, context-aware, and reference-linked answers to user queries about proposal or observation preparation, data retrieval, and data processing. Built on open-source LLMs, running on a local server, ESOFinder ensures data privacy, transparency, and complete control over the knowledge base. Its multi-step architecture allows verification of retrieved documents and generated answers, reducing the risk of hallucinations and improving the reliability of responses compared to commercial tools. The current version of ESOFinder is being tested internally at ESO to evaluate its performance, assess its integration with internal workflows, and identify limitations in coverage and accuracy. These tests will guide further improvements, including the incorporation of additional documentation, and enhanced retrieval strategies. Ultimately, ESOFinder aims to become an interface for users to navigate ESO's complex documentation ecosystem and to support both staff and community astronomers in their daily tasks.

Figures

Figures reproduced from arXiv: 2606.30029 by A. Barnes, C. Reinero, J. Pritchard, M. Marsset, M. Rejkuba, M. Romaniello, M. Vioque, M. Wittkowski, P. S\'anchez-S\'aez.

**Figure 1.** Figure 1: ESOFinder RAG strategy summary, including semantic and keyword-based document retrieval and reranking [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

**Figure 2.** Figure 2: ESOFinder agentic query-answer loop summary. The yellow arrows indicate steps that make use of the [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Figure 3: Summary of the third feedback exercise. The left panel shows the distribution of the five-star overall rating, [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

11 extracted references · 5 canonical work pages · 5 internal anchors

[1]

Attention is All you Need , url =

Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, ukasz and Polosukhin, Illia , booktitle =. Attention is All you Need , url =
[2]

GPT-4 Technical Report

GPT-4 Technical Report. arXiv e-prints , keywords =. doi:10.48550/arXiv.2303.08774 , archivePrefix =. 2303.08774 , primaryClass =

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2303.08774
[3]

Gemini: A Family of Highly Capable Multimodal Models

Gemini: A Family of Highly Capable Multimodal Models. arXiv e-prints , keywords =. doi:10.48550/arXiv.2312.11805 , archivePrefix =. 2312.11805 , primaryClass =

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2312.11805
[4]

Anthropic , year =. The
[5]

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv e-prints , keywords =. doi:10.48550/arXiv.2005.11401 , archivePrefix =. 2005.11401 , primaryClass =

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2005.11401 2005
[6]

Nomic Embed: Training a Reproducible Long Context Text Embedder

Nomic Embed: Training a Reproducible Long Context Text Embedder. arXiv e-prints , keywords =. doi:10.48550/arXiv.2402.01613 , archivePrefix =. 2402.01613 , primaryClass =

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2402.01613
[7]

Robertson, S. E. and Walker, S. Some Simple Effective Approximations to the 2-Poisson Model for Probabilistic Weighted Retrieval. SIGIR '94. 1994

1994
[8]

Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval , year=

The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries , author=. Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval , year=
[9]

and Raghavan, Prabhakar and Schütze, Hinrich , biburl =

Manning, Christopher D. and Raghavan, Prabhakar and Schütze, Hinrich , biburl =. Introduction to Information Retrieval , url =
[10]

2023 , url=

BGE-Reranker-Base , author=. 2023 , url=

2023
[11]

Efficient Memory Management for Large Language Model Serving with PagedAttention

Efficient Memory Management for Large Language Model Serving with PagedAttention. arXiv e-prints , keywords =. doi:10.48550/arXiv.2309.06180 , archivePrefix =. 2309.06180 , primaryClass =

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2309.06180

[1] [1]

Attention is All you Need , url =

Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, ukasz and Polosukhin, Illia , booktitle =. Attention is All you Need , url =

[2] [2]

GPT-4 Technical Report

GPT-4 Technical Report. arXiv e-prints , keywords =. doi:10.48550/arXiv.2303.08774 , archivePrefix =. 2303.08774 , primaryClass =

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2303.08774

[3] [3]

Gemini: A Family of Highly Capable Multimodal Models

Gemini: A Family of Highly Capable Multimodal Models. arXiv e-prints , keywords =. doi:10.48550/arXiv.2312.11805 , archivePrefix =. 2312.11805 , primaryClass =

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2312.11805

[4] [4]

Anthropic , year =. The

[5] [5]

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv e-prints , keywords =. doi:10.48550/arXiv.2005.11401 , archivePrefix =. 2005.11401 , primaryClass =

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2005.11401 2005

[6] [6]

Nomic Embed: Training a Reproducible Long Context Text Embedder

Nomic Embed: Training a Reproducible Long Context Text Embedder. arXiv e-prints , keywords =. doi:10.48550/arXiv.2402.01613 , archivePrefix =. 2402.01613 , primaryClass =

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2402.01613

[7] [7]

Robertson, S. E. and Walker, S. Some Simple Effective Approximations to the 2-Poisson Model for Probabilistic Weighted Retrieval. SIGIR '94. 1994

1994

[8] [8]

Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval , year=

The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries , author=. Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval , year=

[9] [9]

and Raghavan, Prabhakar and Schütze, Hinrich , biburl =

Manning, Christopher D. and Raghavan, Prabhakar and Schütze, Hinrich , biburl =. Introduction to Information Retrieval , url =

[10] [10]

2023 , url=

BGE-Reranker-Base , author=. 2023 , url=

2023

[11] [11]

Efficient Memory Management for Large Language Model Serving with PagedAttention

Efficient Memory Management for Large Language Model Serving with PagedAttention. arXiv e-prints , keywords =. doi:10.48550/arXiv.2309.06180 , archivePrefix =. 2309.06180 , primaryClass =

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2309.06180