pith. sign in

arxiv: 2606.11199 · v1 · pith:Y4EYQUXCnew · submitted 2026-04-21 · 💻 cs.CL · cs.AI· cs.IR· cs.LG

NightFeats @ MMU-RAGent NeurIPS 2025: A Context-Optimized Multi-Agent RAG System for the Text-to-Text Track

Quentin Fever , Naziha Aslam This is my paper

Pith reviewed 2026-07-05 07:03 UTC · model glm-5.2

classification 💻 cs.CL cs.AIcs.IRcs.LG
keywords retrieval-augmented generationmulti-agent systemscitation traceabilitytemporal rerankingcontradiction reconciliationhuman preference evaluationRAG-ArenaNeurIPS 2025
0
0 comments X

The pith

Citation-First RAG Beats Proprietary LLMs in Live User Preferences

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper presents NightFeats, a multi-agent retrieval-augmented generation system that decomposes knowledge synthesis into three phases—retrieval, curation, and composition—each governed by explicit intermediate data models and handoff contracts between agents. The central claim is that architectural transparency, verifiable evidence grounding, and citation traceability are better aligned with real user preferences than systems that optimize for automatic similarity metrics like ROUGE-L or BERTScore. The system introduces three core mechanisms: a temporal-semantic reranking formula that fuses semantic relevance with document recency, a bounded contradiction reconciliation loop that caps iterative refinement at two sub-retrieval cycles to prevent recursive drift, and citation-preserving composition that enforces source traceability as a structural invariant rather than a generation artifact. At the NeurIPS 2025 MMU-RAGent competition, NightFeats was awarded Best Dynamic Evaluation in the text-to-text track via live pairwise user preferences on the RAG-Arena platform, where it was preferred over proprietary baselines including Claude-SonnetV2 and Nova-Pro. The authors argue this outcome validates their design thesis: users value factual traceability and temporal currency even when standard automatic metrics penalize citation-annotated outputs for reduced lexical overlap with reference answers.

Core claim

The paper's central contribution is a three-agent RAG pipeline (ResearchAgent, GeneratorAgent, WriterAgent) where each stage outputs a standardized intermediate representation carrying provenance metadata, contradiction annotations, and normalized citations. The key architectural primitives are: (1) a temporal-semantic fusion score that combines a semantic reranker score with a recency decay term weighted at w=0.3, discounting stale but topically similar documents; (2) a bounded contradiction reconciliation loop that classifies detected contradictions by severity and triggers at most two targeted sub-retrieval cycles, achieving factual coherence without unbounded iterative drift; and (3) a c

What carries the argument

The system coordinates three asynchronously operating agents built on the OpenAI Agents SDK with GPT-5. The ResearchAgent executes a seven-step retrieval pipeline: query expansion (1-3 subqueries), retrieval via Jina Reader over Brave Search (Recent mode) or FineWeb corpus (Historical mode), semantic chunking (700 tokens, 100 overlap), embedding with text-embedding-3-small, temporal-emantic reranking via Voyage AI using the fusion formula s_i_tilde = (1-w)*s_i + w*1/(1+d_i/365) with w=0.3, adaptive cutoff using tau = max(tau_min, mu - alpha*sigma) with alpha=1.0 and tau_min=0.2, and deduplication pruning above 0.85 cosine similarity. The GeneratorAgent extracts atomic cited facts, clusters

Load-bearing premise

The claim that NightFeats surpasses proprietary baselines on human preference evaluations rests on the RAG-Arena live pairwise comparison results, but the paper provides no quantitative data on win rates, sample sizes, or statistical significance for that evaluation, and Table 1 shows 'n/a' for NightFeats's LLM-as-a-Judge scores. The weakest assumption is that the competition's dynamic evaluation award constitutes sufficient evidence to confirm the general superiority of this

What would settle it

A controlled study with reported sample sizes and statistical tests showing that NightFeats does not significantly outperform proprietary baselines on human preference or LLM-as-a-Judge evaluations when queried on identical question sets outside the competition environment. Additionally, if the temporal-semantic reranking weight (w=0.3) were ablated and shown to have no measurable effect on user preference, or if the bounded contradiction reconciliation loop (capped at 2 iterations) were shown to produce no improvement in factual coherence over a single-pass pipeline, the core architectural

read the original abstract

We present NightFeats, a structured multi-agent retrieval-augmented generation (RAG) system submitted to the MMU-RAGent competition at NeurIPS 2025, where it was awarded Best Dynamic Evaluation in the text-to-text track. Rather than targeting benchmark maximization, this work proposes a principled pipeline that decomposes knowledge synthesis into three coordinated phases: retrieval, curation, and composition, each governed by explicit intermediate representations and handoff contracts. Inspired by Agentic Context Engineering (ACE), the system introduces temporal-semantic reranking, bounded contradiction reconciliation, and citation-preserving composition as core architectural primitives. Competition results show that NightFeats surpasses proprietary baselines including Claude-SonnetV2 and Nova-Pro on LLM-as-a-Judge and Human Likert evaluations, confirming that architectural transparency and verifiable evidence grounding are better aligned with human preferences than systems optimizing narrowly for automatic similarity metrics.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 6 minor

Summary. The paper presents NightFeats, a multi-agent RAG system submitted to the MMU-RAGent NeurIPS 2025 competition. The architecture decomposes knowledge synthesis into three phases—retrieval (ResearchAgent), curation (GeneratorAgent), and composition (WriterAgent)—governed by structured intermediate representations and handoff contracts. Key design primitives include temporal-semantic reranking, bounded contradiction reconciliation (capped at two iterations), and citation-preserving composition. The system was awarded 'Best Dynamic Evaluation' in the text-to-text track.

Significance. The architectural design is clearly articulated and the system addresses genuine limitations of conventional RAG pipelines (provenance drift, retrieval myopia, unbounded iterative drift). The design principles P1–P4 are well-motivated and grounded in prior work (ACE, Reflexion, Self-RAG, CRAG). The temporal-semantic fusion formula (§3.1, Eq. for s_i_tilde) and adaptive cutoff (Eq. for tau) are concrete and reproducible with stated hyperparameters. The bounded curation loop (P3) is a practical contribution. However, the central empirical claim of surpassing baselines is not supported by data within the paper itself.

major comments (3)
  1. The abstract states that NightFeats 'surpasses proprietary baselines including Claude-SonnetV2 and Nova-Pro on LLM-as-a-Judge and Human Likert evaluations.' However, Table 1 shows 'n/a' for both LLM-as-a-Judge columns (Sem. Sim. and Fact Cov.) for NightFeats, and no Human Likert scores are reported for any system anywhere in the paper. No RAG-Arena win rates, sample sizes, or statistical significance values are provided. The entire comparative claim rests on the competition award designation, which is an external judgment whose underlying data is not visible to the reader. The abstract and conclusion must be revised to accurately reflect what is and is not demonstrated in the paper, or the supporting quantitative data from the competition must be included.
  2. Section 1 states: 'This paper presents NightFeats as a system contribution: a reproducible, modular architecture for multi-agent RAG reasoning, not a claim of state-of-the-art performance on any single metric.' This directly conflicts with the abstract's assertion of surpassing baselines on LLM-as-a-Judge and Human Likert evaluations. The framing is internally inconsistent. The authors should reconcile this: either the paper makes a comparative performance claim (requiring data) or it is purely a system description (requiring revised abstract/conclusion language).
  3. Table 1 shows NightFeats with ROUGE-L=0.159 and BERTScore=0.840, both below the two baselines (Claude-SonnetV2: 0.181/0.850; Nova-Pro: 0.200/0.851). The paper attributes the ROUGE-L gap to citation annotations reducing lexical overlap, which is plausible, but provides no evidence (e.g., ablation with citations removed) to support this explanation. Without any metric where NightFeats outperforms baselines quantitatively, the claim of superiority rests entirely on the undisclosed RAG-Arena results. At minimum, the paper should present whatever RAG-Arena data is available or clearly state that the comparative claim is based solely on the external competition award.
minor comments (6)
  1. §3.1: The temporal-semantic reranking formula uses notation s_i_tilde for the fused score but the variable is not typeset as an equation with a number. Consider formatting as a numbered equation to improve cross-referencing.
  2. §3.1: The chunk size (700 tokens, 100 overlap) and deduplication threshold (0.85 cosine similarity) are stated in prose. Consolidating all hyperparameters (w, alpha, tau_min, chunk_size, chunk_overlap, deduplication_threshold, max_curation_iterations) in a table would improve reproducibility.
  3. §3.2: The three contradiction severity levels (minor, important, critical) are mentioned but not formally defined. A brief characterization of each level and the corresponding resolution strategy would strengthen the description.
  4. Figure 1 is referenced but the actual figure content is not visible in the manuscript text. Ensure the figure is properly rendered and that it captures the temporal routing flow described in the text.
  5. §5: The footnote on Table 1 (†) states LLM-as-a-Judge scores 'are not separately reported in [8].' Clarify whether this means the scores do not exist or are reported only in aggregate form in the competition report.
  6. References: Several arXiv preprints are cited (e.g., [1], [4]). Where published versions exist, prefer the published reference. Ensure all author names are complete.

Simulated Author's Rebuttal

3 responses · 1 unresolved

We thank the referee for a careful and fair reading. All three major comments identify a genuine gap between the claims in our abstract/conclusion and the quantitative evidence presented in the paper. We agree with the referee's assessment and will revise accordingly. Below we address each comment in turn.

read point-by-point responses

  1. Referee: The abstract states that NightFeats 'surpasses proprietary baselines including Claude-SonnetV2 and Nova-Pro on LLM-as-a-Judge and Human Likert evaluations.' However, Table 1 shows 'n/a' for both LLM-as-a-Judge columns for NightFeats, and no Human Likert scores are reported for any system anywhere in the paper. No RAG-Arena win rates, sample sizes, or statistical significance values are provided. The entire comparative claim rests on the competition award designation, whose underlying data is not visible to the reader.

    Authors: The referee is correct. The abstract and conclusion overstate what the paper itself demonstrates. Table 1 reports 'n/a' for NightFeats on both LLM-as-a-Judge columns, and no Human Likert scores or RAG-Arena win rates appear anywhere in the manuscript. The claim that NightFeats 'surpasses proprietary baselines on LLM-as-a-Judge and Human Likert evaluations' is not supported by data within the paper. We will revise the abstract and conclusion to accurately state that NightFeats was awarded Best Dynamic Evaluation in the text-to-text track via the RAG-Arena live preference protocol, and that the underlying quantitative data (win rates, sample sizes, significance tests) were not separately reported to us by the competition organizers and are therefore not included in this paper. We will remove the language asserting superiority on LLM-as-a-Judge and Human Likert evaluations unless we can obtain and include the supporting data. We are currently requesting the detailed RAG-Arena results from the competition organizers; if these data become available, we will incorporate them into a revised Table 1 or a supplementary table. If they do not, the revised paper will clearly state that the comparative claim rests solely on the external competition award. revision: yes

  2. Referee: Section 1 states: 'This paper presents NightFeats as a system contribution: a reproducible, modular architecture for multi-agent RAG reasoning, not a claim of state-of-the-art performance on any single metric.' This directly conflicts with the abstract's assertion of surpassing baselines on LLM-as-a-Judge and Human Likert evaluations. The framing is internally inconsistent.

    Authors: We agree that the framing is internally inconsistent. Section 1 positions the paper as a system description without performance claims, while the abstract makes a comparative superiority claim. This is a drafting error on our part. In the revision, we will reconcile the two: the paper will be consistently framed as a system contribution that documents the architecture as deployed in the MMU-RAGent competition, notes the competition award as an external evaluation outcome, and does not claim quantitative superiority on any metric reported within the paper itself. The abstract will be brought into alignment with the more cautious framing already present in Section 1. revision: yes

  3. Referee: Table 1 shows NightFeats with ROUGE-L=0.159 and BERTScore=0.840, both below the two baselines. The paper attributes the ROUGE-L gap to citation annotations reducing lexical overlap but provides no evidence (e.g., ablation with citations removed). Without any metric where NightFeats outperforms baselines quantitatively, the claim of superiority rests entirely on undisclosed RAG-Arena results. At minimum, the paper should present whatever RAG-Arena data is available or clearly state that the comparative claim is based solely on the external competition award.

    Authors: The referee is correct on all counts. NightFeats does not outperform either baseline on any metric reported in Table 1. Our explanation for the ROUGE-L gap—citation annotations reducing lexical overlap—is plausible but currently unsupported by evidence in the paper. We will address this in two ways. First, we will add an ablation experiment in which the WriterAgent's output is post-processed to strip citation annotations before computing ROUGE-L, allowing a direct comparison of citation-annotated versus citation-free outputs under the same retrieval and curation pipeline. This will either confirm or refute our hypothesis. Second, we will revise the results section to state plainly that NightFeats does not outperform baselines on any automatic metric reported in the paper, and that the Best Dynamic Evaluation award reflects the competition's live RAG-Arena preference protocol, whose detailed results are not included in this manuscript. We will add an explicit statement to that effect in both Section 5 and the conclusion. If RAG-Arena data become available from the organizers before revision, we will include them; otherwise, we will not claim quantitative superiority. revision: yes

standing simulated objections not resolved
  • The detailed RAG-Arena win rates, sample sizes, and significance values are controlled by the MMU-RAGent competition organizers and were not provided to us in disaggregated form. If the organizers do not release these data, we cannot include them in the paper. In that case, the revised paper will state that the comparative outcome is based solely on the external competition award and that the underlying data are not available to the authors.

Circularity Check

0 steps flagged

No circularity detected; the paper's derivation chain is self-contained with external citations and no self-referential definitions or fits.

full rationale

The paper describes a system architecture (multi-agent RAG pipeline) inspired by external work (ACE [1], Reflexion [2], etc.), none of which are self-citations — the authors (Fever, Aslam) do not appear on any reference. The architectural components (temporal-semantic reranking, bounded contradiction reconciliation, citation-preserving composition) are design choices, not derived results that reduce to their inputs. The evaluation references an external competition [8] with different authors. The central comparative claim ('surpasses baselines on LLM-as-a-Judge and Human Likert') is unsupported by the paper's own Table 1 (which shows n/a for those metrics), but this is an evidence/claim-vs-data problem, not circularity — the claim does not reduce to a fitted input or a self-citation by construction. No equation or definition in the paper is circular. The score of 1 reflects the absence of circularity with a minor note that the competition report [8] is cited to describe the competition's own evaluation protocol, but this is standard practice for competition papers and not load-bearing in a circular sense.

Axiom & Free-Parameter Ledger

7 free parameters · 3 axioms · 0 invented entities

The ledger reflects hand-set hyperparameters and domain assumptions. No new physical or mathematical entities are invented.

free parameters (7)
  • w (temporal weight) = 0.3
    Hand-set parameter controlling the fusion of semantic and temporal scores in the reranking formula.
  • alpha (cutoff threshold) = 1.0
    Hand-set parameter for the adaptive cutoff formula.
  • tau_min (minimum cutoff) = 0.2
    Hand-set minimum threshold for the adaptive cutoff.
  • chunk_size = 700
    Hand-set token limit for semantic chunking.
  • chunk_overlap = 100
    Hand-set overlap for semantic chunking.
  • deduplication_threshold = 0.85
    Hand-set cosine similarity threshold for deduplication.
  • max_curation_iterations = 2
    Hand-set limit on contradiction reconciliation cycles.
axioms (3)
  • domain assumption GPT-5 and the OpenAI Agents SDK are sufficient substrates for implementing the described agent coordination.
    The system architecture is built entirely on this stack (Section 3).
  • domain assumption The MMU-RAGent competition's RAG-Arena protocol is a valid measure of real-world user preference.
    The central claim relies on the competition results being a meaningful signal (Section 5).
  • domain assumption Citation-annotated outputs systematically reduce lexical overlap with reference answers.
    Used to explain the low ROUGE-L score (Section 5).

pith-pipeline@v1.1.0-glm · 11151 in / 2185 out tokens · 103867 ms · 2026-07-05T07:03:50.694371+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

7 extracted references · 7 canonical work pages · 5 internal anchors

  1. [1]

    Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models

    Q. Zhang, C. Hu, S. Upasani, B. Ma, F. Hong, V. Kamanuru, J. Rainton, C. Wu, M. Ji, H. Li, U. Thakker, J. Zou, and K. Olukotun, "Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models," arXiv:2510.04618,

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    Reflexion: Language Agents with Verbal Reinforcement Learning

    N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao, "Reflexion: Language Agents with Verbal Reinforcement Learning," arXiv:2303.11366,

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    RAG Meets Temporal Graphs: Time-Sensitive Modeling and Retrieval for Evolving Knowledge,

    J. Han, A. Cheung, Y. Wei, Z. Yu, X. Wang, B. Zhu, and Y. Yang, "RAG Meets Temporal Graphs: Time-Sensitive Modeling and Retrieval for Evolving Knowledge," arXiv:2510.13590,

    work page arXiv

  4. [4]

    Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection

    A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi, "Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection," arXiv:2310.11511,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    Active Retrieval Augmented Generation

    Z. Jiang, F. F. Xu, L. Gao, Z. Sun, Q. Liu, J. Dwivedi-Yu, Y. Yang, J. Callan, and G. Neubig, "Active Retrieval Augmented Generation," arXiv:2305.06983,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    Corrective Retrieval Augmented Generation

    S.-Q. Yan, J.-C. Gu, Y. Zhu, and Z.-H. Ling, "Corrective Retrieval Augmented Generation," arXiv:2401.15884,

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    MMU-RAGent: Massive Multi-Modal User-Centric Retrieval Augmented Generation Benchmark,

    L. Q. Chan, T. Wang, S. Wang, Z. Zhang, A. Gomez, P. Chandrahasan, A. Tang, L. Yan, Z. Qiu, M. Ziyadi, T. Wu, M. Diab, A. Asai, and C. Xiong, "MMU-RAGent: Massive Multi-Modal User-Centric Retrieval Augmented Generation Benchmark," Proceedings of Machine Learning Research, NeurIPS 2025 Competition Track,

    work page 2025