Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

TeaRAG claims that agentic retrieval-augmented generation can cut output tokens by roughly 60 percent while improving answer accuracy, by compressing retrieved content into knowledge-triplet graphs and training the model to prefer shorter r

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 23:28 UTC pith:762NPQJW

load-bearing objection Solid agentic RAG system with a clever hybrid retrieval and a process reward that is partly self-fulfilling; the reward proxy needs validation. the 3 major comments →

arxiv 2511.05385 v2 pith:762NPQJW submitted 2025-11-07 cs.IR cs.AI

TeaRAG: A Token-Efficient Agentic Retrieval-Augmented Generation Framework

Chao Zhang , Yuhao Wang , Derong Xu , Haoxin Zhang , Yuanjie Lyu , Yuhao Chen , Shuochen Liu , Tong Xu
show 4 more authors
Xiangyu Zhao Yan Gao Yao Hu Enhong Chen
This is my paper
classification cs.IR cs.AI
keywords agentic RAGtoken efficiencyknowledge graphPersonalized PageRankprocess rewarddirect preference optimizationmulti-hop question answeringreasoning compression
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

TeaRAG argues that agentic retrieval-augmented generation can be made substantially cheaper without sacrificing answer quality. The central claim is that two inefficiencies—verbose retrieved chunks and unnecessary multi-step reasoning—can be attacked together: replace much of the retrieved text with knowledge triplets filtered by a knowledge association graph, and train the model to prefer reasoning paths whose intermediate steps yield high knowledge-matching scores per step. Across six question-answering benchmarks, TeaRAG reports average Exact Match gains of 4% and 2% on two instruction-tuned LLMs while reducing output tokens by 61% and 59%. A reader should care because token waste from overthinking and redundant retrieval is the main practical cost of agentic RAG, and this work makes that cost the object of optimization.

Core claim

The core discovery is that retrieval content and reasoning steps are jointly compressible. For each reasoning step, TeaRAG identifies key entities, generates a subquery, retrieves both document chunks and knowledge triplets, and builds a Knowledge Association Graph in which chunks, triplets, entities, and the subquery are linked by co-occurrence and similarity edges. Personalized PageRank over this graph selects the highest-ranked chunks and triplets, so concise triplet facts replace verbose chunk text while co-occurrence acts as a relevance filter. Separately, IP-DPO defines a process reward that scores subquery generation, retrieved context, and summaries by how well they match golden evid

What carries the argument

Knowledge Association Graph (KAG): a heterogeneous graph built per reasoning step from a subquery node, chunk nodes, triplet nodes, and entity nodes, with co-occurrence edges (weight 1) encoding structural links such as a triplet's connection to its source chunk and relevance edges weighted by retriever similarity to the subquery. Personalized PageRank (PPR), a graph-ranking algorithm biased toward query-relevant seeds, is run on this graph to rank all content nodes, and the top-k chunks and triplets become the compressed retrieved context. Iterative Process-aware DPO (IP-DPO) supplies the training signal: a reward made of outcome, format, and process terms, where the process term measures r

Load-bearing premise

The load-bearing premise is that a reranker-based similarity score between each intermediate step and the golden evidence reliably measures whether the model actually acquired the knowledge it needed; if that score is a poor proxy for multi-hop evidence coverage, preference pairs become mislabeled and shorter paths can be rewarded for omitting genuinely necessary reasoning.

What would settle it

A direct check: construct or use a multi-hop QA set whose golden evidence sentences are paraphrased or lexically distant from the subqueries the model generates, so reranker similarity understates true coverage; if TeaRAG's trained model then shows accuracy falling while its process reward stays high, that confirms the reward is rewarding omission rather than efficiency.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Agentic RAG inference cost can drop by more than half on typical QA workloads while maintaining or improving answer quality, making iterative retrieval practical for latency- and token-sensitive applications.
  • Multi-hop questions no longer require a fixed chain of many retrievals; the same task can often be solved in one or two steps when retrieved context is dense enough.
  • Training agentic RAG no longer needs online reinforcement learning with live retrieval calls; offline sampling combined with iterative DPO achieves the efficiency gains.
  • Co-occurrence between chunks and triplets is a usable relevance signal that filters noise better than either retrieval mode alone.
  • Process-level supervision prevents the 'overthinking' failure mode where models keep searching after the evidence is already in hand.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • An extension implicit in this design is that the process reward's knowledge-matching scheme could generalize to other agentic tool-use tasks where 'golden evidence' is available, such as code repair or scientific lookup, potentially reducing overthinking beyond RAG.
  • If the reranker similarity is a faithful evidence-coverage measure, the same pair-construction recipe could be applied to long-form generation and summarization, where per-step information gain rather than step count is the binding constraint.
  • A testable consequence of the co-occurrence filter is that TeaRAG should degrade gracefully as corpus redundancy drops; in a corpus of nearly disjoint documents, chunk–triplet co-occurrence would carry less signal and the PPR filter would resemble plain reranking. This is an editorial inference, not tested in the paper.
  • The paper notes that graph retrieval and semantic retrieval can run in parallel, so the reported inference-time reduction understates the potential latency benefit if that parallelization is implemented; the authors leave this as future work.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes TeaRAG, a token-efficient agentic RAG framework with two main components: (1) a hybrid chunk-plus-triplet retrieval method that builds a Knowledge Association Graph (KAG) from semantic similarity and co-occurrence and uses Personalized PageRank (PPR) to filter the retrieved content, increasing information density per retrieval; and (2) a two-stage training pipeline—supervised fine-tuning on MuSiQue-derived reasoning paths and an Iterative Process-aware Direct Preference Optimization (IP-DPO)—whose process reward combines reranker-based knowledge matching (Eqs. 9–11) with a step-count normalization (Eq. 12) intended to penalize overthinking. Experiments on six QA datasets with Llama3-8B-Instruct and Qwen2.5-14B-Instruct report average EM gains of 4% and 2% (relative) over strong Search-R1+R baselines while reducing output tokens by 61% and 59%, along with ablations of retrieval mode, reward type, and training iterations.

Significance. If the results hold, TeaRAG is a useful contribution to agentic RAG: it targets a real bottleneck—token overhead—and combines graph-enhanced retrieval with an offline DPO-style training that avoids the high resource costs of PPO/GRPO. Strengths include the release of code, construction and use of a large-scale Wikipedia knowledge graph (51M entities, 130M triplets), evaluation on both in-domain and out-of-domain datasets, and an ablation showing that KAG+PPR improves over naive concatenation at lower token counts. The finding that process-aware rewards stabilize iterative DPO while outcome-only rewards collapse (Table 8) is also interesting. However, the step-reduction and much of the token saving are directly incentivized by the IP-DPO reward design—Eq. (12) divides accumulated evidence by k, and single-hop chosen responses scale outcome by 1/k—so the efficiency gains are partly a restatement of the objective rather than an emergent property. The central claims are thus plausible but need additional validation of the reward proxy and its calibration.

major comments (3)
  1. [§4.3.2, Eqs. (12)–(14)] The process reward normalizes ΣM_q—an unnormalized sum over l golden-evidence pieces—by the number of steps k. For a perfect 1-step path on an l-hop question, the score can be as large as l, while a perfect l-step path scores about 1. The fixed thresholds 0.7/0.8 in Eq. (14) are therefore not calibrated across datasets or hop counts; a 1-step path can pass the process criterion even when it omits genuinely required reasoning, and a correct 2-step path can be ranked below an over-compressed 1-step path. This biases DPO pair construction. The paper does not report the distribution of process rewards or a sensitivity analysis for these thresholds. Please normalize by l (or by max(l,k)), report per-dataset score distributions, and ablate the k-normalization separately from the knowledge-matching term to show that the step reduction is not purely an artifact of the chosen normalizer.
  2. [§4.3.2, Eqs. (9)–(11)] The knowledge-matching reward uses BGE-reranker-v2 similarity between golden evidence and subqueries/contexts/summaries as a proxy for 'knowledge acquisition.' This proxy is not validated. A reranker trained for relevance ranking may assign high similarity to a subquery that merely mentions the evidence without the model actually using it, or low similarity to a semantically equivalent paraphrase. Mislabeled preference pairs from Eq. (14) would make the measured token savings non-transferable to harder or noisier domains. I recommend reporting the correlation between r_process and r_outcome on the sampled paths, a manual error analysis of high/low process-reward paths, or a comparison with an alternative similarity function. At minimum, the paper should explicitly discuss this limitation.
  3. [Abstract; §5.3.2; §4.3.2] The headline 61%/59% token reduction is dominated by the reduction in reasoning steps, and that reduction is directly incentivized by the IP-DPO reward: Eq. (12) divides accumulated evidence by k, and single-hop chosen responses set r_outcome = r_outcome/k (Sec. 4.3.2). Observing fewer steps after training with this reward is therefore partly a restatement of the reward design. The paper's framing that IP-DPO 'curbs overthinking' would be strengthened by an experiment that removes the k-normalization while keeping the knowledge-matching term, to measure the accuracy–efficiency trade-off attributable to the reward signal rather than to the penalty itself. Also, the abstract's 4%/2% EM improvements are relative gains (+1.75 and +0.94 absolute points in Table 6) and should be labeled as such.
minor comments (5)
  1. [Table 6] SelfRAG* reports identical scores (EM 33.90, F1 41.86) under both the Llama3-8B and Qwen2.5-14B sections. Since SelfRAG is not re-trained on these bases, the duplication is misleading; label it as a fixed baseline and clarify which base model it uses.
  2. [Abstract; Table 6] The EM improvements of '4% and 2%' are relative gains, not absolute point increases. Please state this explicitly to avoid ambiguity.
  3. [Fig. 3] The figure contains stray text ('Judge？') and an ungrammatical sentence; clean up the caption and labels.
  4. [References] Several references are incomplete, using '[n. d.]' or missing venue/year (e.g., [6], [23], [32], [48], [57], [81]). Please update.
  5. [Table 3; §5.1.1] NQ has both a development and a test split; clarify which split is used for the NQ results in Table 6.

Circularity Check

1 steps flagged

IP-DPO's k-normalized reward makes the headline step/token reduction partly true by construction; retrieval-side compression and accuracy gains remain independent.

specific steps
  1. fitted input called prediction [Section 4.3.2, Eqs. (12)-(14); reported in Section 5.3.1 and Table 8]
    "To promote conciseness and penalize inefficiency, we normalize the summed memory scores by the number of steps, k. The final rewards are calculated as follows: r_q(P_k)=ΣM_q/k, r_c(P_k)=ΣM_c/k, r_s(P_k)=ΣM_s/k ... For single-hop questions ... we directly scale the outcome reward by dividing it by the length of the reasoning path k, setting r_outcome = r_outcome/k and r_process = 0 ... r_format =1 ∧ r_outcome =1, if q is single-hop ... Process rewards reduce reasoning steps and increase reasoning efficiency."

    The paper's headline claim that TeaRAG reduces reasoning steps and output tokens is partially encoded in the training objective itself. Eq. (12) divides every knowledge-matching reward by k, so any two reasoning paths with equal evidence coverage receive a higher process reward when the path is shorter. For single-hop questions, the outcome reward is divided by k, and Eq. (14) accepts a chosen response only when r_outcome=1 after this division, which is achievable only by a perfect 1-step path. IP-DPO then builds preference pairs that favor shorter paths by construction; the later observation of fewer steps (Fig. 6, Table 8) is therefore a restatement of the reward rather than an independent empirical discovery. The 61%/59% output-token reductions are largely downstream of this built-in st

full rationale

The derivation chain has two independent components. The KAG+PPR retrieval compression is not circular: it is an offline knowledge-graph construction followed by Personalized PageRank selection, and it is evaluated with ablations (Table 7) against semantic-only, graph-only, and concatenated hybrid retrieval. The accuracy improvements from IP-DPO are also not forced: the process reward depends on an external BGE-reranker-v2 similarity to golden evidence, and the reported EM/F1 gains require the model to produce correct answers, which is not entailed by the reward design. However, the central token-efficiency claim is partly by construction. Eq. (12) normalizes the process reward by the number of steps k, and single-hop chosen responses require r_outcome/k = 1, so the DPO preference pairs are explicitly designed to prefer shorter reasoning paths. The paper then presents this engineered preference as a finding ('Process rewards reduce reasoning steps and increase reasoning efficiency') and uses it to support the 61%/59% token-reduction claims. That is a fitted objective renamed as an empirical result, which is the circular component. The unvalidated reranker proxy and fixed 0.7/0.8 thresholds are correctness/transferability concerns, not circularity. No load-bearing self-citation chain was found. Overall score 5 reflects partial circularity of the step/token reduction claim while the accuracy and retrieval-density contributions retain independent content.

Axiom & Free-Parameter Ledger

7 free parameters · 6 axioms · 0 invented entities

TeaRAG is an empirical ML paper: it contributes a pipeline, not a mathematical derivation. The central claim rests on several hand-set hyperparameters and domain assumptions—most importantly that triplet extraction preserves facts, that co-occurrence is a reliable relevance signal, and that a reranker similarity is a valid proxy for knowledge acquisition in the training reward. These are not free parameters in the physics sense but are choices that materially affect the headline token-efficiency numbers.

free parameters (7)
  • tau (triplet-subquery edge threshold) = 0.2
    Eq. 2; chosen by hand, controls how much triplet-query similarity counts; no sensitivity analysis reported.
  • alpha (PPR teleport/balance) = 0.5
    Eq. 3; hand-set; ablation Fig. 12 shows robustness over 0.1-0.7 but best in 0.3-0.7.
  • top-k counts k_d, k_t, k_f = 5 chunks, 10 triplets, 5 final items
    Implementation details; hard-coded retrieval budget; no sensitivity study of k_f beyond Figs. 10-11.
  • process reward weights = 0.1 / 0.3 / 0.3 / 0.3
    Eq. 13; hand-assigned balancing consistency and knowledge-matching; no sensitivity analysis.
  • DPO chosen/rejected thresholds = r_process>=0.7 or >=0.8; r_outcome<=0.3; hard-rejection margin 0.3
    Eq. 14; chosen by hand to define positive/negative pairs; affects learned preference signal.
  • DPO beta and SFT weight eta = beta=0.5; eta in [0.25,0.5,1] decreasing
    Implementation details; standard hyperparameters, but eta schedule is tuned during IP-DPO rounds.
  • max reasoning steps k_max = 5
    Implementation details; caps the reasoning path and can suppress genuinely needed longer chains.
axioms (6)
  • domain assumption Qwen2.5-14B-extracted triplets are an accurate, lossless-enough compression of chunk content for QA.
    Section 4.1: KG construction uses Qwen2.5-14B to extract triplets from each chunk; if extraction drops facts needed downstream, PPR filtering over triplets will lose the answer.
  • domain assumption Co-occurrence between a chunk and a triplet is a strong relevance signal.
    Section 4.2.4: co-occurrence edges are assigned weight 1 and form the structural backbone of the KAG; if co-occurrence is coincidental or noisy, PPR will amplify irrelevant content.
  • domain assumption BGE-reranker-v2 similarity is a valid proxy for whether golden evidence was acquired in subquery/context/summary.
    Eqs. 9-11 compute memory vectors with sim_theta; the entire process reward and DPO pair construction depend on these similarities being meaningful.
  • domain assumption Normalizing summed memory scores by k yields a valid conciseness objective that does not discard necessary reasoning.
    Eq. 12 divides by k; this assumes each additional step should be penalized linearly and that evidence coverage is comparable across path lengths.
  • standard math Personalized PageRank converges and its top-k_f stationary-distribution nodes are the most relevant context items.
    Eq. 3: standard PPR; convergence is assumed with alpha=0.5 and N=200 iterations.
  • domain assumption MuSiQue's golden decomposition and supporting evidence transfer to training an agent for NQ, HotpotQA, PopQA, 2Wiki, and Bamboogle.
    Section 4.3.1: SFT data is synthesized from MuSiQue; if this transfer is weak, SFT may teach the wrong reasoning format.

pith-pipeline@v1.3.0-alltime-deepseek · 33991 in / 15030 out tokens · 127229 ms · 2026-08-03T23:28:18.939539+00:00 · methodology

0 comments
read the original abstract

Retrieval-Augmented Generation (RAG) utilizes external knowledge to augment Large Language Models' (LLMs) reliability. For flexibility, agentic RAG employs autonomous, multi-round retrieval and reasoning to resolve queries. Although recent agentic RAG has improved via reinforcement learning, they often incur substantial token overhead from search and reasoning processes. This trade-off prioritizes accuracy over efficiency. To address this issue, this work proposes TeaRAG, a token-efficient agentic RAG framework capable of compressing both retrieval content and reasoning steps. 1) First, the retrieved content is compressed by augmenting chunk-based semantic retrieval with a graph retrieval using concise triplets. A knowledge association graph is then built from semantic similarity and co-occurrence. Finally, Personalized PageRank is leveraged to highlight key knowledge within this graph, reducing the number of tokens per retrieval. 2) Besides, to reduce reasoning steps, Iterative Process-aware Direct Preference Optimization (IP-DPO) is proposed. Specifically, our reward function evaluates the knowledge sufficiency by a knowledge matching mechanism, while penalizing excessive reasoning steps. This design can produce high-quality preference-pair datasets, supporting iterative DPO to improve reasoning conciseness. Across six datasets, TeaRAG improves the average Exact Match by 4% and 2% while reducing output tokens by 61% and 59% on Llama3-8B-Instruct and Qwen2.5-14B-Instruct, respectively. Code is available at https://github.com/Applied-Machine-Learning-Lab/TeaRAG.

Figures

Figures reproduced from arXiv: 2511.05385 by Chao Zhang, Derong Xu, Enhong Chen, Haoxin Zhang, Shuochen Liu, Tong Xu, Xiangyu Zhao, Yan Gao, Yao Hu, Yuanjie Lyu, Yuhao Chen, Yuhao Wang.

Figure 1
Figure 1. Figure 1: 𝑇𝑖 denotes the thinking tokens at the i-th step, 𝑅𝑖 denotes the retrieved context at the i-th step, and 𝑂 represents the final output. (a) illustrates Search-R1, a representative agentic RAG method optimized based on the final outcome. (b) shows our proposed method TeaRAG, which achieves a token-efficient agentic RAG by optimizing the retrieved content length with high-density triplets and controlling the … view at source ↗
Figure 2
Figure 2. Figure 2: (a) shows the token usage. (b) shows the distribution of reasoning steps. (c) shows the F1 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The overall pipeline of TeaRAG. Based on an offline-built knowledge graph and chunk corpus [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: (a) shows the structure of a reasoning step. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The overall training framework for TeaRAG follows a two-stage paradigm. First, we conduct [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of reasoning step distributions between TeaRAG and other agentic RAG baselines. [PITH_FULL_IMAGE:figures/full_fig_p020_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Comparison of output token usage between TeaRAG and other agentic RAG baselines. The [PITH_FULL_IMAGE:figures/full_fig_p021_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Performance of TeaRAG with Llama3-8B-Instruct as the LLM changes across IP-DPO iterations. [PITH_FULL_IMAGE:figures/full_fig_p023_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Performance of TeaRAG with Qwen2.5-14B-Instruct as the LLM changes across IP-DPO [PITH_FULL_IMAGE:figures/full_fig_p023_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Performance of TeaRAG-8B and Search-R1-base-7B+R across varying numbers of input [PITH_FULL_IMAGE:figures/full_fig_p024_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Performance of TeaRAG-14B and Search-R1-base-14B+R across varying numbers of input [PITH_FULL_IMAGE:figures/full_fig_p025_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Performance and the average number of content tokens per retrieval on six QA benchmarks [PITH_FULL_IMAGE:figures/full_fig_p026_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Performance of TeaRAG-8B with Search-R1-base-7B+R across varying generation tempera [PITH_FULL_IMAGE:figures/full_fig_p026_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments

    cs.AI 2026-03 unverdicted novelty 7.0

    PERMA is a new benchmark using temporally ordered events, text variability, and linguistic alignment to evaluate LLM memory agents on persona consistency beyond simple retrieval.

  2. HyGRL: Adaptive Hybrid Graph Reasoning for Multi-Entity Questions

    cs.AI 2026-07 conditional novelty 5.0

    HyGRL trains a small policy to walk a mixed knowledge-graph/text-chunk graph, reporting higher multi-hop QA accuracy than graph-RAG baselines at lower token cost.

  3. AgenticRAG: Agentic Retrieval for Enterprise Knowledge Bases

    cs.AI 2026-05 unverdicted novelty 5.0

    AgenticRAG equips an LLM with iterative retrieval and navigation tools, delivering 49.6% recall@1 on BRIGHT, 0.96 factuality on WixQA, and 92% correctness on FinanceBench.

Reference graph

Works this paper leans on

88 extracted references · 1 canonical work pages · cited by 3 Pith papers

  1. [1]

    Nicholas Alonso and Beren Millidge. 2024. Mixture-of-PageRanks: Replacing Long-Context with Real-Time, Sparse GraphRAG. arXiv preprint arXiv:2412.06078 (2024)

    Pith/arXiv arXiv 2024

  2. [2]

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In ICLR

    2023

  3. [3]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. NeurIPS 33 (2020), 1877–1901

    2020

  4. [4]

    Nikhil Chandak, Shashwat Goel, and Ameya Prabhu. 2025. Incorrect Baseline Evaluations Call into Question Recent LLM-RL Claims. https://safe-lip-9a8.notion.site/Incorrect-Baseline-Evaluations-Call-into-Question-Recent-LLM-RL- Claims-2012f1fbf0ee8094ab8ded1953c15a37?pvs=4. Notion Blog

    2025

  5. [5]

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2023. BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. arXiv:2309.07597 [cs.CL]

    Pith/arXiv arXiv 2023

  6. [6]

    Liyi Chen, Panrong Tong, Zhongming Jin, Ying Sun, Jieping Ye, and Hui Xiong. [n. d.]. Plan-on-Graph: Self-Correcting Adaptive Planning of Large Language Model on Knowledge Graphs. In NeurIPS

  7. [7]

    Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. 2024. Do not think that much for 2+ 3=? on the overthinking of o1-like llms. arXiv preprint arXiv:2412.21187 (2024)

    Pith/arXiv arXiv 2024

  8. [8]

    Alejandro Cuadron, Dacheng Li, Wenjie Ma, Xingyao Wang, Yichuan Wang, Siyuan Zhuang, Shu Liu, Luis Gaspar Schroeder, Tian Xia, Huanzhi Mao, et al. 2025. The danger of overthinking: Examining the reasoning-action dilemma in agentic tasks. arXiv preprint arXiv:2502.08235 (2025)

    Pith/arXiv arXiv 2025

  9. [9]

    Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. 2024. From local to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404.16130 (2024)

    Pith/arXiv arXiv 2024

  10. [10]

    Jinyuan Fang, Zaiqiao Meng, and Craig Macdonald. 2025. KiRAG: Knowledge-Driven Iterative Retriever for Enhancing Retrieval-Augmented Generation. arXiv preprint arXiv:2502.18397 (2025)

    Pith/arXiv arXiv 2025

  11. [11]

    Jiaxuan Gao, Wei Fu, Minyang Xie, Shusheng Xu, Chuyi He, Zhiyu Mei, Banghua Zhu, and Yi Wu. 2025. Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RL. arXiv preprint arXiv:2508.07976 (2025)

    arXiv 2025

  12. [12]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997 2 (2023)

    Pith/arXiv arXiv 2023

  13. [13]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al . 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    Pith/arXiv arXiv 2024

  14. [14]

    Zirui Guo, Lianghao Xia, Yanhua Yu, Tu Ao, and Chao Huang. 2024. LightRAG: Simple and Fast Retrieval-Augmented Generation. arXiv preprint arXiv:2410.05779 (2024)

    Pith/arXiv arXiv 2024

  15. [15]

    Bernal Jiménez Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. 2024. Hipporag: Neurobiologically inspired long-term memory for large language models. In NeurIPS. , V ol. 1, No. 1, Article . Publication date: November 2018. 30 Zhang et al

    2024

  16. [16]

    Bernal Jiménez Gutiérrez, Yiheng Shu, Weijian Qi, Sizhe Zhou, and Yu Su. 2025. From rag to memory: Non-parametric continual learning for large language models. arXiv preprint arXiv:2502.14802 (2025)

    Pith/arXiv arXiv 2025

  17. [17]

    Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh Chawla, Thomas Laurent, Yann LeCun, Xavier Bresson, and Bryan Hooi

  18. [18]

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps. In COLING. 6609–6625

    2020

  19. [19]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al

  20. [20]

    Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. 2025. Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models. arXiv preprint arXiv:2501.03262 (2025)

    Pith/arXiv arXiv 2025

  21. [21]

    Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A Smith, Yejin Choi, and Hanna Hajishirzi. 2024. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback. NeurIPS 37 (2024), 36602–36633

    2024

  22. [22]

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720 (2024)

    Pith/arXiv arXiv 2024

  23. [23]

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. [n. d.]. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. In EMNLP

  24. [24]

    Pengcheng Jiang, Lang Cao, Ruike Zhu, Minhao Jiang, Yunyi Zhang, Jimeng Sun, and Jiawei Han. 2025. RAS: Retrieval-And-Structuring for Knowledge-Intensive LLM Generation. arXiv preprint arXiv:2502.10996 (2025)

    arXiv 2025

  25. [25]

    Pengcheng Jiang, Xueqiang Xu, Jiacheng Lin, Jinfeng Xiao, Zifeng Wang, Jimeng Sun, and Jiawei Han. 2025. s3: You Don’t Need That Much Data to Train a Search Agent via RL. arXiv preprint arXiv:2505.14146 (2025)

    arXiv 2025

  26. [26]

    Bowen Jin, Jinsung Yoon, Priyanka Kargupta, Sercan O Arik, and Jiawei Han. 2025. An Empirical Study on Reinforce- ment Learning for Reasoning-Search Interleaved LLM Agents. arXiv preprint arXiv:2505.15117 (2025)

    Pith/arXiv arXiv 2025

  27. [27]

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516 (2025)

    Pith/arXiv arXiv 2025

  28. [28]

    Jiajie Jin, Yutao Zhu, Xinyu Yang, Chenghao Zhang, and Zhicheng Dou. 2024. Flashrag: A modular toolkit for efficient retrieval-augmented generation research. arXiv preprint arXiv:2405.13576 (2024)

    Pith/arXiv arXiv 2024

  29. [29]

    Adam Tauman Kalai, Ofir Nachum, Santosh S Vempala, and Edwin Zhang. 2025. Why language models hallucinate. arXiv preprint arXiv:2509.04664 (2025)

    Pith/arXiv arXiv 2025

  30. [30]

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering.. In EMNLP. 6769–6781

    2020

  31. [31]

    Saeed Khaki, JinJin Li, Lan Ma, Liu Yang, and Prathap Ramachandra. 2024. RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models. In Findings of NAACL. 1665–1680

    2024

  32. [32]

    Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. [n. d.]. Understanding the Effects of RLHF on LLM Generalisation and Diversity. InThe Twelfth International Conference on Learning Representations

  33. [33]

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. TACL 7 (2019), 453–466

    2019

  34. [34]

    Hanyu Lai, Xiao Liu, Hao Yu, Yifan Xu, Iat Long Iong, Shuntian Yao, Aohan Zeng, Zhengxiao Du, Yuxiao Dong, and Jie Tang. 2025. WebGLM: Towards an Efficient and Reliable Web-Enhanced Question-Answering System. ACM Trans. Inf. Syst. 43, 5, Article 122 (July 2025), 43 pages. doi:10.1145/3729421

    doi:10.1145/3729421 2025

  35. [35]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. NeurIPS 33 (2020), 9459–9474

    2020

  36. [36]

    Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. 2025. Search-o1: Agentic search-enhanced large reasoning models. arXiv preprint arXiv:2501.05366 (2025)

    Pith/arXiv arXiv 2025

  37. [37]

    Xiaopeng Li, Lixin Su, Pengyue Jia, Suqi Cheng, Junfeng Wang, Dawei Yin, and Xiangyu Zhao. 2025. Agent4Ranking: Semantic Robust Ranking via Personalized Query Rewriting Using Multi-agent LLMs. ACM Trans. Inf. Syst. (July 2025). doi:10.1145/3749099 Just Accepted

    work page doi:10.1145/3749099 2025

  38. [38]

    Lei Liang, Zhongpu Bo, Zhengke Gui, Zhongshu Zhu, Ling Zhong, Peilong Zhao, Mengshu Sun, Zhiqiang Zhang, Jun Zhou, Wenguang Chen, et al. 2025. Kag: Boosting llms in professional domains via knowledge augmented generation. In Companion Proceedings of the ACM on Web Conference 2025. 334–343

    2025

  39. [39]

    Haoran Luo, Guanting Chen, Qika Lin, Yikai Guo, Fangzhi Xu, Zemin Kuang, Meina Song, Xiaobao Wu, Yifan Zhu, Luu Anh Tuan, et al. 2025. Graph-R1: Towards Agentic GraphRAG Framework via End-to-end Reinforcement Learning. arXiv preprint arXiv:2507.21892 (2025). , V ol. 1, No. 1, Article . Publication date: November 2018. TeaRAG : A Token-Efficient Agentic Re...

    Pith/arXiv arXiv 2025

  40. [40]

    Yuanjie Lyu, Zihan Niu, Zheyong Xie, Chao Zhang, Tong Xu, Yang Wang, and Enhong Chen. 2024. Retrieve-Plan- Generation: An Iterative Planning and Answering Framework for Knowledge-Intensive LLM Generation. InEMNLP. 4683–4702

    2024

  41. [41]

    Shengjie Ma, Chengjin Xu, Xuhui Jiang, Muzhi Li, Huaren Qu, Cehao Yang, Jiaxin Mao, and Jian Guo. 2024. Think-on- graph 2.0: Deep and faithful large language model reasoning with knowledge-guided retrieval augmented generation. arXiv preprint arXiv:2407.10805 (2024)

    Pith/arXiv arXiv 2024

  42. [42]

    Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023. Query rewriting in retrieval-augmented large language models. In EMNLP. 5303–5315

    2023

  43. [43]

    Iain Mackie, Ivan Sekulic, Shubham Chatterjee, Jeffrey Dalton, and Fabio Crestani. 2023. GRM: generative relevance modeling using relevance-aware sample estimation for document retrieval. arXiv preprint arXiv:2306.09938 (2023)

    Pith/arXiv arXiv 2023

  44. [44]

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories. InACL. 9802–9822

    2023

  45. [45]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. NeurIPS 35 (2022), 27730–27744

    2022

  46. [46]

    Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. 2024. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402.13228 (2024)

    Pith/arXiv arXiv 2024

  47. [47]

    Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason Weston. 2024. Iterative reasoning preference optimization. NeurIPS 37 (2024), 116617–116637

    2024

  48. [48]

    Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. [n. d.]. Measuring and Narrowing the Compositionality Gap in Language Models. In EMNLP

  49. [49]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. NeurIPS 36 (2023), 53728–53741

    2023

  50. [50]

    Bhaskarjit Sarmah, Dhagash Mehta, Benika Hall, Rohan Rao, Sunil Patel, and Stefano Pasquali. 2024. Hybridrag: Inte- grating knowledge graphs and vector retrieval augmented generation for efficient information extraction. In Proceedings of the 5th ACM International Conference on AI in Finance. 608–616

    2024

  51. [51]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)

    Pith/arXiv arXiv 2017

  52. [52]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 (2024)

    Pith/arXiv arXiv 2024

  53. [53]

    Yaorui Shi, Sihang Li, Chang Wu, Zhiyuan Liu, Junfeng Fang, Hengxing Cai, An Zhang, and Xiang Wang. 2025. Search and Refine During Think: Autonomous Retrieval-Augmented Reasoning of LLMs. arXiv preprint arXiv:2505.11277 (2025)

    arXiv 2025

  54. [54]

    Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. 2025. R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning. arXiv preprint arXiv:2503.05592 (2025)

    Pith/arXiv arXiv 2025

  55. [55]

    Huatong Song, Jinhao Jiang, Wenqing Tian, Zhipeng Chen, Yuhuan Wu, Jiahao Zhao, Yingqian Min, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. 2025. R1-Searcher++: Incentivizing the Dynamic Knowledge Acquisition of LLMs via Reinforcement Learning. arXiv preprint arXiv:2505.17005 (2025)

    Pith/arXiv arXiv 2025

  56. [56]

    Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Fei Huang, and Yan Zhang. 2025. Zerosearch: Incentivize the search capability of llms without searching. arXiv preprint arXiv:2505.04588 (2025)

    Pith/arXiv arXiv 2025

  57. [57]

    Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Lionel Ni, Heung-Yeung Shum, and Jian Guo. [n. d.]. Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph. In ICLR

  58. [58]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. MuSiQue: Multihop Questions via Single-hop Question Composition. TACL 10 (2022), 539–554

    2022

  59. [59]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions. In ACL

    2023

  60. [60]

    Songjun Tu, Jiahao Lin, Xiangyu Tian, Qichao Zhang, Linjing Li, Yuqian Fu, Nan Xu, Wei He, Xiangyuan Lan, Dongmei Jiang, et al. 2025. Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigation. arXiv preprint arXiv:2503.12854 (2025)

    Pith/arXiv arXiv 2025

  61. [61]

    Prakhar Verma, Sukruta Prakash Midigeshi, Gaurav Sinha, Arno Solin, Nagarajan Natarajan, and Amit Sharma. 2024. Plan* rag: Efficient test-time planning for retrieval augmented generation. arXiv preprint arXiv:2410.20753 (2024)

    Pith/arXiv arXiv 2024

  62. [62]

    Hongru Wang, Cheng Qian, Wanjun Zhong, Xiusi Chen, Jiahao Qiu, Shijue Huang, Bowen Jin, Mengdi Wang, Kam- Fai Wong, and Heng Ji. 2025. Acting Less is Reasoning More! Teaching Model to Act Efficiently. arXiv preprint arXiv:2504.14870 (2025). , V ol. 1, No. 1, Article . Publication date: November 2018. 32 Zhang et al

    Pith/arXiv arXiv 2025

  63. [63]

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2024. A survey on large language model based autonomous agents. FCS 18, 6 (2024), 186345

    2024

  64. [64]

    Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei

  65. [65]

    Shuting Wang, Xin Yu, Mang Wang, Weipeng Chen, Yutao Zhu, and Zhicheng Dou. 2025. RichRAG: Crafting Rich Responses for Multi-faceted Queries in Retrieval-Augmented Generation. In COLING. 11317–11333

    2025

  66. [66]

    Ziliang Wang, Xuhui Zheng, Kang An, Cijun Ouyang, Jialu Cai, Yuhang Wang, and Yichao Wu. 2025. StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimization. arXiv preprint arXiv:2505.15107 (2025)

    Pith/arXiv arXiv 2025

  67. [67]

    arXiv preprint arXiv:2212.03533 (2022)

    Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533 (2022)

    Pith/arXiv arXiv 2022

  68. [68]

    Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, et al . 2025. A minimalist approach to llm reasoning: from rejection sampling to reinforce. arXiv preprint arXiv:2504.11343 (2025)

    Pith/arXiv arXiv 2025

  69. [69]

    Derong Xu, Wei Chen, Wenjun Peng, Chao Zhang, Tong Xu, Xiangyu Zhao, Xian Wu, Yefeng Zheng, Yang Wang, and Enhong Chen. 2024. Large language models for generative information extraction: A survey.FCS 18, 6 (2024), 186357

    2024

  70. [70]

    Jinbo Wen, Cheng Su, Jiawen Kang, Jiangtian Nie, Yang Zhang, Jianhang Tang, Dusit Niyato, and Chau Yuen. 2025. HybridRAG-based LLM Agents for Low-Carbon Optimization in Low-Altitude Economy Networks. arXiv preprint arXiv:2506.15947 (2025)

    Pith/arXiv arXiv 2025

  71. [71]

    Derong Xu, Tong Xu, Shiwei Wu, Jingbo Zhou, and Enhong Chen. 2022. Relation-enhanced negative sampling for multimodal knowledge graph completion. In Proceedings of the 30th ACM international conference on multimedia. 3857–3866

    2022

  72. [72]

    Derong Xu, Ziheng Zhang, Zhenxi Lin, Xian Wu, Zhihong Zhu, Tong Xu, Xiangyu Zhao, Yefeng Zheng, and Enhong Chen. 2024. Multi-perspective Improvement of Knowledge Graph Completion with Large Language Models. In LREC/COLING

    2024

  73. [73]

    Derong Xu, Pengyue Jia, Xiaopeng Li, Yingyi Zhang, Maolin Wang, Qidong Liu, Xiangyu Zhao, Yichao Wang, Huifeng Guo, Ruiming Tang, et al. 2025. Align-GRAG: Reasoning-Guided Dual Alignment for Graph Retrieval-Augmented Generation. arXiv preprint arXiv:2505.16237 (2025)

    arXiv 2025

  74. [74]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2.5 Technical Report. arXiv e-prints (2024), arXiv–2412

    2024

  75. [75]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. InEMNLP. 2369–2380

    2018

  76. [76]

    Fangyuan Xu, Weijia Shi, and Eunsol Choi. 2024. Recomp: Improving retrieval-augmented lms with compression and selective augmentation. (2024)

    2024

  77. [77]

    Chuanyue Yu, Kuo Zhao, Yuhan Li, Heng Chang, Mingjian Feng, Xiangzhe Jiang, Yufei Sun, Jia Li, Yuzhi Zhang, Jianxin Li, et al. 2025. GraphRAG-R1: Graph Retrieval-Augmented Generation with Process-Constrained Reinforcement Learning. arXiv preprint arXiv:2507.23581 (2025)

    arXiv 2025

  78. [78]

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al . 2025. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476 (2025)

    Pith/arXiv arXiv 2025

  79. [79]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In ICLR

    2023

  80. [80]

    Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou

Showing first 80 references.