Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Two retrieval systems—one tree-based, one graph-based—pull expert-level answers out of the LHCb corpus by respecting document structure and cross-paper relationships instead of matching isolated chunks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 22:57 UTC pith:7QW2NT7F

load-bearing objection A solid proof-of-concept for tree-structured retrieval over physics papers, with an honest but under-powered evaluation—worth engaging, but the central comparison needs full human grading before it's established. the 3 major comments →

arxiv 2509.06855 v1 pith:7QW2NT7F submitted 2025-09-08 hep-ex

Seeing the Forest Through the Trees: Knowledge Retrieval for Streamlining Particle Physics Analysis

James McGreivy , Blaise Delaney , Anja Beck , Mike Williams This is my paper
classification hep-ex
keywords Retrieval Augmented Generationknowledge graphLHCbparticle physics literaturesystematic uncertaintiesLLM-as-judgedocument treeCERN open data
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that standard chunk-based Retrieval Augmented Generation (RAG) is too fragmented for particle-physics literature, and that exploiting the hierarchy of papers with SciTreeRAG, plus a corpus-level knowledge graph with SciGraphRAG, gives LLMs more focused, contextually rich information. If right, newcomers and non-experts could query decades of LHCb publications for practical guidance, such as which decay channels to use and which systematic uncertainties will dominate, without first becoming domain experts. The claim is demonstrated on 834 LHCb papers with a 56-question evaluation derived from the Heavy Flavour Averaging Group report, where the tree-based method reduces 'poor' answers from 25% to 10% and raises 'satisfactory' or 'good' answers above 50%. The graph system is presented as a proof-of-concept on two vetted expert queries rather than a full benchmark.

Core claim

On the paper's own terms, the central discovery is that the logical skeleton of a scientific paper—its section and paragraph hierarchy—can be exploited to make retrieval-augmented generation less fragmented and more focused. The authors build a tree for each LHCb article, summarize nodes recursively, and diffuse embeddings down the tree, then retrieve by a greedy best-first traversal so that semantically similar leaves are only selected if their parent sections are also topically relevant. On a 56-question benchmark derived from the heavy-flavour averages report, SciTreeRAG reduces 'poor' answers from 25% to 10% and raises 'satisfactory' or 'good' answers from roughly 42% to above 50% compar

What carries the argument

The paper tree is the central object for local retrieval: roots are abstracts, internal nodes are sections and subsections carrying LLM-generated summaries, and leaves are paragraphs, figure captions, table captions, and equations. Embeddings are refined by a recursive attention-weighted diffusion step, and retrieval proceeds by a greedy best-first traversal of the tree, so candidate chunks are filtered by topical section relevance as well as semantic similarity. The knowledge graph is the central object for global retrieval: five entity types (paper, observable, decay, uncertainty_source, method) connected by four relations (determines, measured_with, affects, estimates), constructed per ar

Load-bearing premise

The claimed improvement rests on trusting LLM-generated questions, rubrics, and grades; if the judge LLM systematically prefers its own style of answer, the measured gap could be self-consistency rather than better retrieval.

What would settle it

Have two independent human physicists blind-grade all 168 anonymized answers (56 queries times three retrieval systems) against the same rubrics; if the 10%-versus-25% 'poor' rate gap disappears under human grading, the reported improvement is an artifact of the LLM judge rather than a genuine retrieval gain.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If the reported numbers hold, a collaboration can index thousands of legacy papers at a one-time cost of tens of dollars in LLM calls, making routine corpus structuring feasible for future experiments.
  • A queryable graph with provenance returns answers tied to specific arXiv identifiers, letting researchers vet analysis choices—like which decay channel to use—before committing to a strategy.
  • A standardized schema focused on measurements and uncertainties, agreed across LHC experiments, would make knowledge graphs human-understandable and enable shared retrieval tooling.
  • This lowers the barrier to CERN open data, allowing non-specialists to propose and assess analyses without first absorbing hundreds of papers.
  • The tree method alone reduces context fragmentation, which should make any downstream LLM answer more coherent and more traceable to the source literature.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The tree representation should transfer to any corpus with regular hierarchical sectioning—other LHC experiments, biomedical reviews, or technical standards—since it only assumes a consistent document skeleton.
  • The authors' own caveat that the graph system 'frequently fails due to poor mapping of the query onto the schema' suggests the bottleneck is natural-language-to-graph translation, not graph construction; improving that translation may matter more than further canonicalization.
  • A blind human re-grade of the full 168-response evaluation set would be the quickest external check of the reported improvement; until then the numbers should be treated as indicative rather than decisive.
  • Combining tree-level local retrieval with graph-level global retrieval in one orchestrated pipeline is a natural next step, since each addresses a limitation the other leaves open.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces two retrieval-augmented generation (RAG) systems for the LHCb publication corpus. SciTreeRAG builds a hierarchical tree from each paper's LaTeX structure, uses recursive LLM summarization and attention-weighted embedding diffusion, and retrieves leaf nodes via greedy best-first traversal. SciGraphRAG constructs a knowledge graph of observables, decays, uncertainty sources, methods, and papers, then answers natural-language queries by translating them to Cypher. The main quantitative claim is that SciTreeRAG, especially with diffusion, outperforms standard chunk-based RAG: on a 56-question evaluation set built from the HFLAV report, SciTreeRAG receives 'poor' ratings 10% of the time versus 25% for BaseRAG, and 'satisfactory' or 'good' ratings over 50% versus about 42%. SciGraphRAG is presented only as illustrative proof-of-concept on two hand-vetted queries because the authors state it still requires development for reliable automated KG construction and NL-to-Cypher translation.

Significance. If substantiated, the work would be a useful proof-of-concept for domain-specific knowledge retrieval in experimental particle physics. The strengths include: (i) using an external benchmark (HFLAV report) deliberately excluded from the retrieval corpus, which reduces memorization and tests genuine retrieval; (ii) leveraging the regular hierarchical structure of experimental papers in a principled way; (iii) transparent reporting of computational costs and graph-canonicalization statistics; and (iv) a clearly scoped knowledge-graph schema with provenance tracking. The main quantitative comparison, however, rests entirely on LLM-as-judge evaluation with only partial human validation, and no statistical uncertainty is reported. Because the queries, rubrics, answers, and judge are all produced by overlapping LLM generations, the magnitude and even the existence of the claimed improvement are not yet firmly established. The contribution is incremental but appropriate for a workshop-level venue, provided the evaluation concerns are addressed.

major comments (3)
  1. [App. B.1, Fig. 2] The central claim that SciTreeRAG outperforms BaseRAG rests solely on LLM-as-judge grades. The paper states human validation was performed only on 'a subset' without reporting the subset size, selection protocol, or inter-annotator agreement. Since the queries and rubrics are LLM-generated (App. A.2) and the judge is also an LLM, the observed 15-point gap in 'poor' ratings (25% to 10%) could reflect judge style preference rather than retrieval quality. Please provide human grading of the full 56-query set, or at least a statistically justified random sample with inter-annotator agreement, and report per-context-window grade distributions.
  2. [Sec. 4, Fig. 2] No statistical uncertainty or significance testing is reported. With 56 queries per system (or 168 if each context-window size is counted separately), a difference of 15 percentage points may be within binomial noise. Please report bootstrap confidence intervals or a paired significance test across the 56 queries, and clarify exactly how the aggregation over the three context-window sizes was performed. The statement 'averaged across multiple context window sizes' is not sufficient without showing the per-window results.
  3. [App. C.2, Eqs. (1)-(2)] The diffusion-enhanced embeddings that define the best-performing configuration depend on free parameters λ and τ. The manuscript does not report the values used or any sensitivity analysis. Without this, the improvement of 'SciTreeRAG with diffusion' over 'without diffusion' is not yet robustly established. Please report the chosen values and show the grade distribution for a range of λ and τ to demonstrate that the central comparison does not hinge on a particular parameter choice.
minor comments (5)
  1. [General] Capitalization is inconsistent: 'SCITREERAG', 'SciTreeRAG', 'SCIGRAPHRAG', 'GraphRAG' are used interchangeably. Please unify the notation for the two systems.
  2. [App. A.2] 'HFLA V' appears with an erroneous space; use 'HFLAV' consistently. Also, the HFLAV report is from 2024; please cite it with its full arXiv reference in the appendix where the dataset is introduced.
  3. [Fig. 2] The figure lacks error bars, sample sizes, and a precise definition of the grade categories in the caption. It would help to state whether percentages are over 56 questions or over 168 context-window evaluations, and to include a table with per-window and per-system counts.
  4. [Sec. 4 / App. B.2] The SciGraphRAG examples in App. B.2 are explicitly hand-selected success cases. The main text should make this even clearer, perhaps by stating that no systematic evaluation of SciGraphRAG is claimed, to avoid the impression that it outperforms the other systems in general.
  5. [Sec. 4] The promise that 'code will be made public and linked in the camera-ready version' is not yet fulfilled. Please include an anonymized repository link or at least a clear statement of availability in the revised manuscript.

Circularity Check

0 steps flagged

No significant circularity: the retrieval systems are not fitted to the evaluation outcomes, the benchmark is external to the corpus, and the only self-citation is a non-load-bearing preprocessing tool.

full rationale

The paper's central derivation is empirical rather than definitional. SCITREERAG's tree construction, embedding diffusion, and best-first retrieval are fixed procedures with parameters (lambda, tau) that are not tuned to maximize the evaluation grades. The HFLAV evaluation dataset is explicitly external to the LHCb retrieval corpus (App. A.2), so the test measures retrieval rather than memorization of the judge's inputs. The LLM-as-judge protocol (App. B.1) is a potential validity limitation, and the authors acknowledge this: 'since this human-expert validation was not conducted across the entire evaluation dataset, these results should be interpreted with appropriate caution.' However, this is a measurement-validity concern, not circularity: the judge's grades are an output of the evaluation, not an input to the construction of either retrieval system. No equation or definition in the paper reduces the claimed improvement to the evaluation inputs. The only self-citation (the expand-latex-macros Python library, McGreivy 2025) is used for preprocessing LaTeX sources and is not load-bearing for the central claim. Therefore no significant circularity is present.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The systems introduce no new physical entities. They do, however, depend on several unreported hyperparameters (lambda, tau, clustering weights, thresholds), on the assumption that LLM summaries and extractions are faithful, and on an evaluation built from LLM-generated queries and judgments. These are the main costs the reader pays for the central claim.

free parameters (5)
  • Diffusion parameter lambda = not reported
    Controls how much child embeddings are mixed into parent node embeddings in Eq. (1); value must be chosen by hand and is not stated.
  • Attention temperature tau = not reported
    Controls sharpness of child-weighting in Eq. (2); value not stated.
  • Entity semantic similarity threshold = 0.5
    In the gamma example (App. B.2), observables are kept when semantic similarity to the query exceeds 0.5; a hand-chosen cut.
  • TF-IDF/embedding combination weights = not specified
    Entity clustering uses a weighted sum of TF-IDF and embedding cosine similarities (App. D.2.2); the weights are not given.
  • Agglomerative clustering thresholds = dynamic
    Thresholds are adjusted iteratively until cluster sizes stay small (App. D.2.2); the protocol is configurable and not pinned down.
axioms (5)
  • domain assumption LHCb articles share a uniform hierarchical structure that can be parsed into semantically meaningful trees.
    Section 2 and App. C.1 rely on section and subsection structure as the semantic backbone for retrieval.
  • domain assumption LLM-generated section summaries preserve the semantic content needed for retrieval.
    Recursive summarization (App. C.2) is used to build node summaries; summary hallucinations could bias retrieval.
  • domain assumption LLMs can extract accurate per-article knowledge graphs from abstracts and uncertainty sections.
    Section 3 and App. D.2 assume GPT-5 mini produces correct entity and relationship extraction.
  • domain assumption HFLAV is a valid external ground truth for this retrieval task.
    App. A.2 motivates using HFLAV because it is outside the corpus and contains verifiable cross-document synthesis.
  • domain assumption LLM-as-judge grading with LLM-generated rubrics measures answer quality.
    App. B.1 uses an evaluator LLM with human validation only on a subset; this is a methodological premise for the main quantitative claim.

pith-pipeline@v1.3.0-alltime-deepseek · 14168 in / 9177 out tokens · 85366 ms · 2026-08-04T22:57:35.929587+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Seeing the Forest Through the Trees: Knowledge Retrieval for Streamlining Particle Physics Analysis." pith.science (2026). https://pith.science/paper/7QW2NT7F

@misc{pith2026250906855,
  author       = {Pith},
  title        = {Pith review of: Seeing the Forest Through the Trees: Knowledge Retrieval for Streamlining Particle Physics Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7QW2NT7F}},
  note         = {Machine review of arXiv:2509.06855}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Generative Large Language Models (LLMs) are a promising approach to structuring knowledge contained within the corpora of research literature produced by large-scale and long-running scientific collaborations. Within experimental particle physics, such structured knowledge bases could expedite methodological and editorial review. Complementarily, within the broader scientific community, generative LLM systems grounded in published work could make for reliable companions allowing non-experts to analyze open-access data. Techniques such as Retrieval Augmented Generation (RAG) rely on semantically matching localized text chunks, but struggle to maintain coherent context when relevant information spans multiple segments, leading to a fragmented representation devoid of global cross-document information. Here, we utilize the hierarchical organization of experimental physics articles to build a tree representation of the corpus, and present the SciTreeRAG system that uses this structure to create contexts that are more focused and contextually rich than standard RAG. Additionally, we develop methods for using LLMs to transform the unstructured corpus into a structured knowledge graph representation. We then implement SciGraphRAG, a retrieval system that leverages this knowledge graph to access global cross-document relationships eluding standard RAG, thereby encapsulating domain-specific connections and expertise. We demonstrate proof-of-concept implementations using the corpus of the LHCb experiment at CERN.

Figures

Figures reproduced from arXiv: 2509.06855 by Anja Beck, Blaise Delaney, James McGreivy, Mike Williams.

Figure 1
Figure 1. Figure 1: Visualization of knowledge subgraphs for Query (left) [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of response quality grades across [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Example per-paper knowledge graphs showing entity types and relationships. Purple [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. CLVisc Agent for autonomous relativistic hydrodynamics studies

    nucl-th 2026-07 conditional novelty 6.0

    An LLM agent autonomously created a CLVisc skill and ran two hydrodynamic studies, finding that the high-temperature branch of η/s dominates flow suppression and that PGCM-uniform 16O decouples ellipticity from size.

  2. Are We Ready for AI-Driven Discovery? AI Verification Before the Next Fundamental Physics Breakthrough

    physics.data-an 2026-07 accept novelty 4.0

    Verification of ML in fundamental physics is essential precisely when models enter statistical modeling, inference, or hypothesis testing, and is bounded by unavoidable inductive bias, sample complexity, and experimen...

Reference graph

Works this paper leans on

13 extracted references · 7 canonical work pages · cited by 2 Pith papers · 1 internal anchor

  1. [1]

    B 0 s →D ∓ s K ±

    This algorithm im- plements a best-first search strategy across a forest of hierarchical document trees to retrieve the most relevant content for a given query. Starting with all tree roots in a boundary set, the algorithm iteratively selects the node with highest similarity to the query embedding, then either expands it by adding its children to the boun...

    work page 2028

  2. [4]

    Javier Duarte, Gaia Grosso, Raghav Kansal, and Pietro Vischia

    URL https://cds.cern.ch/ record/2935252. Javier Duarte, Gaia Grosso, Raghav Kansal, and Pietro Vischia. Bites of foundation models for science: Llms for experiments in fundamental physics. CERN Workshop, June

    work page arXiv

  3. [7]

    Document ranking with a pretrained sequence-to- sequence model.arXiv preprint arXiv:2003.06713,

    Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. Document ranking with a pretrained sequence-to- sequence model.arXiv preprint arXiv:2003.06713,

    Pith/arXiv arXiv 2003

  4. [8]

    LHCb Collaboration

    URLhttps://api.semanticscholar.org/CorpusID:268987666. LHCb Collaboration. LHCb Publications. https://lbfence.cern.ch/alcm/public/analysis, 2010-today. Accessed: 2025-08-28. INSPIRE Collaboration. Inspire-hep: High energy physics literature database,

    work page 2010

  5. [9]

    Version 2.0.0 (April 24, 2025); retrieved 2025-08-27. Sw. Banerjee, E. Ben-Haim, F. Bernlochner, E. Bertholet, M. Bona, A. Bozek, C. Bozzi, J. Brodzicka, V . Chobanova, M. Chrzaszcz, U. Egede, M. Gersabeck, P. Goldenzweig, N. Gharbi, L. Grillo, K. Hayasaka, T. Humair, D. Johnson, T. Kuhr, O. Leroy, A. Lusiani, H. L. Ma, M. Margoni, R. Mizuk, P. Naik, T. N...

    work page 2025

  6. [10]

    org/abs/2411.18639

    URL https://arxiv. org/abs/2411.18639. A Evaluation Datasets A.1 LHCb Corpus For this work a corpus of 834 LHCb publications LHCb Collaboration [2010-today] was assembled by querying the INSPIRE-HEP INSPIRE Collaboration

    Pith/arXiv arXiv 2010

  7. [11]

    literature database API. This col- lection comprises published and peer-reviewed measurement papers, detector-performance papers, conference contributions, review papers, and theoretical papers published by the LHCb collaboration between December 2009 and August

    work page 2009

  8. [12]

    Averages ofb- hadron, c-hadron, and τ-lepton properties as of 2023

    For each publication, the raw LATEX source was downloaded and merged into a monolithic source file using the latexpand command-line toolkit Moy [2023]. This document was further processed to remove all extraneous LATEX content, comments, bibliography entries, and collaboration author lists. All LHCb collaboration and user-defined latex macros were expande...

    arXiv 2023

  9. [2019]

    A PubMedBERT-based Classifier with Data Augmentation Strategy for Detecting Medication Mentions in Tweets

    Qing Han, Shubo Tian, and Jinfeng Zhang. A pubmedbert-based classifier with data augmentation strategy for detecting medication mentions in tweets.arXiv preprint arXiv:2112.02998,

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [2020]

    Passage re-ranking with bert.arXiv preprint arXiv:1901.04085,

    Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with bert.arXiv preprint arXiv:1901.04085,

    Pith/arXiv arXiv 1901

  11. [2021]

    Retrieval-augmented generation in biomedicine: A survey of technologies, datasets, and clinical applications.arXiv preprint arXiv:2505.01146,

    Jiawei He, Boya Zhang, Hossein Rouhizadeh, Yingjian Chen, Rui Yang, Jin Lu, Xudong Chen, Nan Liu, Irene Li, and Douglas Teodoro. Retrieval-augmented generation in biomedicine: A survey of technologies, datasets, and clinical applications.arXiv preprint arXiv:2505.01146,

    arXiv

  12. [2024]

    5 Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al

    URL https://indico.cern.ch/event/1543967/. 5 Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented genera- tion for knowledge-intensive nlp tasks.Advances in neural information processing systems, 33: 9459–9474,

    work page arXiv

  13. [2025]

    Scibert: A pretrained language model for scientific text

    Iz Beltagy, Kyle Lo, and Arman Cohan. Scibert: A pretrained language model for scientific text. arXiv preprint arXiv:1903.10676,

    Pith/arXiv arXiv 1903