Pith. sign in

REVIEW 4 major objections 5 minor 48 references

Checking a fact's self-consistency before writing it to agent memory cuts hallucinated facts from 50% to 34% on real conversations and to 1.2% on synthetic data.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 04:00 UTC pith:XWAUJB7U

load-bearing objection A well-framed idea with an honest limitations section, but the real-conversation benchmarks test contradiction detection against pre-seeded answers rather than write-time verification, so the headline contamination reductions are likely overstated. the 4 major comments →

arxiv 2607.22962 v1 pith:XWAUJB7U submitted 2026-07-25 cs.AI cs.CL

ConsistencyGate: Preventing Memory Contamination in LLM Agents via Self-Consistency Admission Control

Yan Zhang , Shibo Li This is my paper
classification cs.AI cs.CL
keywords LLM agentsmemory contaminationwrite-time admissionself-consistencyhallucinationmemory gatelong-horizon reasoningsoft support scoring
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper targets memory contamination in long-horizon LLM agents: a hallucinated fact written once becomes a false premise for every later reasoning step, so write-time errors compound over a trajectory. Existing memory systems manage retrieval and capacity but are blind to correctness, because a false fact is as relevant, novel, and recent as the true one. ConsistencyGate is a write-time admission control that, before storing each candidate fact, queries the same LLM K times for a soft 0–1 support score against the source context and admits the fact only when the average exceeds a threshold (default τ=0.7, K=5). Across four backbones it reduces contamination on every benchmark relative to write-everything—from 50% to 34.1% and 36.7% on real conversations, to 1.2% on a synthetic near-oracle corpus—with the cost concentrated on facts stated implicitly rather than explicitly. The paper also offers a single-forward-pass log-probability variant and an adaptive threshold for deployments where per-model tuning is infeasible.

Core claim

Memory contamination is a write-time admission problem, not a retrieval or capacity problem: a fact written once is read many times, and utility-based criteria are blind to correctness because a hallucinated paraphrase has nearly identical novelty and relevance to the true fact. ConsistencyGate operationalizes the missing correctness criterion: before committing a candidate fact m extracted from context c, the LLM is queried K=5 times for a soft support score, and m is admitted only when the average support is at least 0.7. On two real long-conversation benchmarks with planted single-detail corruptions this reduces contamination from 50% to 34.1% and 36.7%, and on a synthetic near-oracle cor

What carries the argument

The consistency score p̂(m|c) = (1/K) Σ s_k, where each s_k is an LLM-provided soft support rating of candidate fact m against source context c (1.0 fully supported, 0.5 ambiguous, 0.0 contradicted), compared to threshold τ. It carries the entire argument: the gate has no independent access to truth, so correctness is proxied by inter-sample agreement between the writer and the verifier, both the same LLM. Two efficiency variants extend it: a binary yes/no prompt read from token log-probabilities in one forward pass, and an adaptive threshold that scales with memory occupancy when τ cannot be tuned. The mechanism is what makes write-time correctness tractable without fine-tuning or external

Load-bearing premise

The load-bearing premise is that a fact's average soft support across LLM samples tracks its factual correctness against the source context; if the model is consistently overconfident about a plausible but wrong detail, the gate admits it and the whole mechanism collapses.

What would settle it

A focused test would build a corpus where every corrupted fact is plausible and consistent with the surrounding context, then run the gate at its recommended threshold; if contamination stays near the 50% baseline, the support-score premise fails in exactly the cases that matter.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the gate works as reported, memory contamination stops compounding: a rejected hallucinated fact never becomes a false premise, so the benefit grows with trajectory length.
  • Utility- and recency-based memory management must be complemented by write-time correctness filtering; no retrieval or eviction policy can undo a stored falsehood.
  • Admission recall is the main trade-off: on real conversations 42% of correct facts are rejected when they are stated implicitly, so deployments must either accept a quieter memory or provide distributed/multi-hop evidence to the verifier.
  • The cheap single-pass variant widens the operating envelope: structured document or tool-output memory can be gated at roughly 25ms per fact, making write-time verification affordable at scale.
  • Clean-memory regression is small (−1.5% QA F1 on clean LoCoMo), so the gate does not sacrifice normal operation to guard against contamination.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Across-corpus comparison suggests the gate's signal is strongest where source context is unambiguous and self-contained; a testable extension would deliberately corrupt facts so they remain consistent with surrounding context, where the paper's own false positives predict the gate's signal inverts.
  • The authors' pre-seeding of correct facts in real benchmarks means the reported QA F1 understates the gate's downstream benefit in actual deployments, where rejected contaminated facts would otherwise corrupt retrieval; the contamination-rate and precision numbers are the honest evidence.
  • One could couple the gate with a retrieval step that gathers multiple relevant chunks before scoring, which would target the 42% implicit-fact rejection without abandoning self-consistency.
  • The near-perfect synthetic numbers should be read as an upper bound, not a deployment expectation: the gate is a self-consistency filter, not a fact-checker, and practitioners should not present gated memory as verified.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces ConsistencyGate, a write-time admission gate for LLM agent memory that scores candidate facts by querying the LLM K times for a soft support score and admitting the fact only when the average exceeds a threshold. The gate is model-agnostic and requires no fine-tuning; a single-pass log-probability variant is also proposed. The authors construct three benchmarks: MemContam (synthetic, controlled), LoCoMo-Contam and MSC-Contam (real conversations with planted single-detail corruptions). Across four LLM backbones, they report that ConsistencyGate reduces memory contamination relative to WriteAll on all benchmarks, with the largest reductions on MemContam (e.g., 50% to 1.2% for Qwen2.5-32B) and more modest reductions on the real-conversation benchmarks (50% to 34.1% on LoCoMo-Contam, 50% to 36.7% on MSC-Contam). The paper also analyzes failure modes, ablates K and tau, and releases code and benchmarks.

Significance. If the reported reductions hold in deployment, ConsistencyGate would be a simple, practical addition to memory-augmented LLM agents, addressing a failure mode (write-time contamination) that is distinct from retrieval and capacity. The paper is transparent about limitations: MemContam is explicitly presented as a near-oracle upper bound, the real benchmarks are small, and the gate's recall on LoCoMo-Contam is only 0.58. The central premise—that self-consistency against the source context tracks factual correctness—is plausible and the experiments are internally consistent. However, the real-benchmark protocol pre-seeds correct facts into memory before probing, which fundamentally changes the task from verification against the source turn to contradiction detection against a gold answer key; this is a serious threat to the external validity of the headline deployment-facing numbers. The paper also tunes its recommended hyperparameters on MemContam and then reports MemContam results with those hyperparameters, introducing a form of test-set feedback. These issues require major revision.

major comments (4)
  1. [§5.2, Appendix E.2] The evaluation protocol pre-seeds the correct probe facts directly into memory before presenting the probe turn. The gate's 'source context' is then retrieved memory containing the correct answer, so a corrupted probe is a direct string/meaning contradiction of a known entity, not a test of whether the fact is supported by the source turn. The paper acknowledges this design choice only for QA F1 ('primary signals on this benchmark are the contamination rate and admission precision'), but the same pre-seeding makes the contamination-rate metric a contradiction-detection task, not a write-time verification task. The reported 34.1%/36.7% contamination rates are therefore likely to be optimistically biased relative to the natural setting where a newly extracted fact is verified only against the current context. I recommend re-running the real-conversation benchmarks without pre-seeding (or w
  2. [§6.5, Table 4] The recommended configuration (tau=0.7, K=5) is selected by ablation on MemContam (Table 4) and then used to report MemContam's headline results in Tables 1 and 2. This is circular: the 1.2% contamination figure is the result of optimizing the threshold on the same data used for evaluation. The paper should either report MemContam results with a threshold fixed a priori (e.g., tau=0.5) or use a proper development/test split for the synthetic benchmark. The cross-model comparisons in Table 2 are also affected because three models use the adaptive schedule and one uses fixed tau=0.7, making the 'near-oracle' claim less clean than it appears.
  3. [§5.2, Table 1] The real-conversation benchmarks are very small: 50 probe pairs in LoCoMo-Contam and 41 in MSC-Contam, yielding roughly 100 and 82 binary admission decisions for the contamination-rate metric. No error bars, confidence intervals, or significance tests are reported. With 100 Bernoulli trials, the standard error of a 34% rate is about 4.8 percentage points, so the gap between WriteAll (50%) and ConsistencyGate (34.1%) is substantial but the paper should quantify its uncertainty. A bootstrap or exact binomial CI would strengthen the claim and clarify whether the improvement is consistent across the 10 LoCoMo conversations (the table suggests high variance, e.g., Qwen's recall is 0.58 but the range is 0.52–0.68 across backbones).
  4. [§7, Appendix D] The paper's own false positives reveal a structural limitation that the real-benchmark protocol does not exercise: when a corruption is semantically plausible and consistent with the stored facts, the gate admits it with score 1.0 (e.g., 'Jon and Gina attended a dance class together' for a date corruption on the dance class fact). In the pre-seeded protocol, most corrupted probes directly contradict a pre-seeded correct fact and are therefore easy to detect. The harder case—a corruption that is plausible given memory but not directly contradicted—is exactly the case the benchmarks under-sample. The paper's recommendation to use the gate mainly for document-grounded writes (Section 7) conflicts with the abstract's claim that the gate prevents contamination in 'LLM agents' broadly. Please reconcile the scope of the claim with the evidence, or add an evaluation that includes distractors con
minor comments (5)
  1. [Table 3] The row labelled 'ConsistencyGate (K=5, default)' uses the adaptive threshold, while the following row 'Fixed tau=0.7' is the recommended configuration. This is confusing because the main results in Table 1 call the fixed-tau setting 'ConsistencyGate'. Clarify the labels, e.g., 'ConsistencyGate (adaptive)' and 'ConsistencyGate (fixed tau=0.7)'.
  2. [§6.2] The phrase 'reduces contamination by 31.8% and 26.6%' is ambiguous: these are relative reductions (from 50% to 34.1% and 36.7%). State this explicitly to avoid misreading as absolute percentage points.
  3. [Appendix D] In Table 6, the 'Admitted' column uses check/cross symbols that are visually identical to the 'Correct?' column. Use explicit 'Yes/No' strings to disambiguate the admission decision from the correctness label.
  4. [§6.1] The Random baseline is matched to ConsistencyGate's empirical admission rate, which is a good design, but the paper should state the exact admission rates for each baseline on each benchmark (e.g., p=0.6 is the overall rate, but the actual per-benchmark rate may differ). This would help assess whether any residual contamination reduction is due to selectivity alone.
  5. [§4.3] Equation (5) defines the adaptive threshold but does not specify how tau_min and tau_max are set in practice. The paper uses tau_min=0.4, tau_max=0.8 for some cross-model runs; a brief rule-of-thumb or sensitivity result would be useful.

Circularity Check

1 steps flagged

One localized fitted-hyperparameter issue: the MemContam headline results use a threshold tuned on the same MemContam pool, so that row is an optimized fit rather than an independent prediction; the real-conversation claims are not circular.

specific steps
  1. fitted input called prediction [§6.5 (Table 4) and Table 1; Appendix E.1]
    "Raising τ from 0.3 to 0.7 sharply improves precision on the three more sensitive backbones (Qwen 0.931→0.988, Gemma 0.966→0.984, Scout 0.860→0.997), and τ=0.9 matches τ=0.7 within noise, identifying τ=0.7 as the natural decision boundary; ... Recommended configuration: fixed τ=0.7, K=5. ... All 200 samples are used as a single evaluation pool; there is no train split."

    The headline MemContam row (ρ=1.2%, QA F1=0.840 in Table 1) is produced with the configuration τ=0.7, K=5 that the paper selects from a full hyperparameter sweep over the same 200-sample MemContam pool, whose gold labels are known. Because there is no held-out split, τ was chosen to optimize exactly the contamination-rate metric later reported as evidence for the method. The MemContam result is therefore a selected optimum rather than an independent prediction; only the latency trade-off (K) and the real-benchmark rows retain predictive content. The paper discloses τ as 'recommended,' but the fitted-input nature of the MemContam number remains.

full rationale

No other load-bearing circularity was found. ConsistencyGate's p-hat (Eq. 3) is an LLM support score, not defined in terms of the contamination labels, and the admission rule is a thresholded average of that score; the evaluation labels come from external benchmark construction (planted single-detail corruptions with gold labels), not from the gate's own definition. The pre-seeding of correct facts in LoCoMo/MSC is disclosed by the authors and affects external validity of those benchmarks, but it is not an equation-level circularity because the gate can still admit corrupted probes and reject correct ones. There is no self-citation chain: A-MAC (Zhang et al. 2026) is not authored by the present authors, and the self-consistency prior is cited to Wang et al. (2023). The one genuine circular element is the MemContam hyperparameter tuning, which is localized and does not by itself invalidate the real-conversation reductions reported with the same threshold.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central mechanism depends on four domain assumptions: self-consistency as a proxy for correctness, verifier instruction-following, benchmark representativeness, and retrieval propagation. The only tuned quantities are the threshold tau, sample count K, capacity C, and the random baseline probability p. No new entities are introduced.

free parameters (4)
  • admission threshold tau = 0.7 (fixed) or tau_min=0.4, tau_max=0.8 (adaptive)
    Recommended value selected from ablation on MemContam (Section 6.5, Table 4); used in main results on the same benchmark.
  • number of samples K = 5 (default), with K=1 and K=10 considered
    Default K=5 chosen based on ablation showing marginal gains beyond K=1 (Section 6.5); hyperparameter tuned on MemContam.
  • memory capacity C = 500
    Set high enough not to bind in any run; affects adaptive threshold schedule only.
  • Random baseline admission probability p = 0.6
    Matched to ConsistencyGate's empirical admission rate at tau=0.7; used to control for selectivity (Section 6.1).
axioms (4)
  • domain assumption Correct facts are more consistently supported by the source context than corrupted facts across K LLM samples
    Core premise of self-consistency admission, inherited from self-consistency literature (Wang et al., 2023); Section 2, Eq. (3). The paper's false-positive examples (Appendix D) show this can fail.
  • domain assumption The LLM verifier follows the instruction to judge support only from the provided context and ignores outside knowledge
    Prompts in Appendix C say 'Do not use outside knowledge'; there is no mechanism preventing the model from using its priors, and Section 7 notes it may vote yes for plausible-but-wrong claims requiring world knowledge.
  • domain assumption Planted single-detail corruptions in LoCoMo/MSC are representative of natural memory-contamination errors
    Benchmarks in Section 5 use one-detail corruptions (number, entity, negation); real hallucinations may be more subtle or distributed across turns, which the paper itself acknowledges in Section 7 (implicit facts).
  • domain assumption External memory retrieval surfaces both correct and corrupted facts to downstream reasoning, causing contamination to propagate
    Section 3.2 asserts oracle retriever propagates contamination; no quantitative model of retrieval is given, so the cascade claim (Figure 2) is only demonstrated on MemContam.

pith-pipeline@v1.3.0-alltime-deepseek · 16020 in / 13064 out tokens · 110553 ms · 2026-08-01T04:00:29.504848+00:00 · methodology

0 comments
read the original abstract

LLM agents that operate over many turns accumulate facts in an external memory store and reuse them as premises for downstream reasoning. A hallucinated fact written at one step therefore persists as a false premise for every subsequent step, a failure mode we call memory contamination. Existing memory management addresses retrieval and capacity but not write-time correctness; this admission problem cannot be solved by utility- or recency-based criteria, and uncontrolled contamination compounds across long trajectories. We propose ConsistencyGate, a write-time admission gate that, before committing a candidate fact m extracted from context c, queries the LLM K times for a soft support score and admits m only when the average exceeds a threshold. The mechanism is model-agnostic, requires no fine-tuning, and reduces to a single forward pass in a log-probability variant for latency-sensitive deployments. To measure the effect on natural data, we construct two real-conversation benchmarks (LoCoMo-Contam and MSC-Contam) by planting controlled single-detail corruptions in long-term conversations from LoCoMo and MSC, and complement them with a structured synthetic corpus (MemContam) that isolates a near-oracle upper bound. Across four LLM backbones, ConsistencyGate reduces contamination on every benchmark relative to a write-everything baseline, with the cost concentrated on facts that are stated only implicitly in the source context. We release all three benchmarks together with the gate implementation.

Figures

Figures reproduced from arXiv: 2607.22962 by Shibo Li, Yan Zhang.

Figure 1
Figure 1. Figure 1: Memory contamination and the ConsistencyGate mitigation. (Left) A baseline agent writes every extracted fact to memory, including hallucinated ones (red), which then propagate into downstream answers. (Right) ConsistencyGate samples a verification score K times before each write; facts with low average support are rejected and memory stays clean. (Bottom) Contamination rate and downstream QA accuracy versu… view at source ↗
Figure 2
Figure 2. Figure 2: Contamination rate and QA F1 vs. interaction turns on [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

48 extracted references · 12 linked inside Pith

  1. [1]

    Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology , publisher =

    Generative Agents: Interactive Simulacra of Human Behavior , author=. Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology , publisher =

  2. [3]

    The Eleventh International Conference on Learning Representations , year=

    Self-Consistency Improves Chain of Thought Reasoning in Language Models , author=. The Eleventh International Conference on Learning Representations , year=

  3. [4]

    SelfCheck

    Potsawee Manakul and Adian Liusie and Mark Gales , booktitle=. SelfCheck. 2023 , url=

    2023

  4. [5]

    ACM Computing Surveys , volume=

    Survey of Hallucination in Natural Language Generation , author=. ACM Computing Surveys , volume=

  5. [6]

    Retrieval-Augmented Generation for Knowledge-Intensive

    Lewis, Patrick and Perez, Ethan and Piktus, Aleksandra and Petroni, Fabio and Karpukhin, Vladimir and Goyal, Naman and K\". Retrieval-Augmented Generation for Knowledge-Intensive. Advances in Neural Information Processing Systems , publisher =

  6. [7]

    Adaptive Memory Admission Control for

    Guilin Zhang and Wei Jiang and Xiejiashan Wang and Aisha Behr and Kai Zhao and Jeffrey Friedman and Xu Chu and Amine Anoun , journal=. Adaptive Memory Admission Control for

  7. [8]

    Evaluating Very Long-Term Conversational Memory of

    Maharana, Adyasha and Lee, Dong-Ho and Tulyakov, Sergey and Bansal, Mohit and Barbieri, Francesco and Fang, Yuwei , booktitle=. Evaluating Very Long-Term Conversational Memory of

  8. [9]

    Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics , volume=

    Beyond Goldfish Memory: Long-Term Open-Domain Conversation , author=. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics , volume=

  9. [10]

    Advances in Neural Information Processing Systems , publisher=

    Reflexion: Language Agents with Verbal Reinforcement Learning , author=. Advances in Neural Information Processing Systems , publisher=

  10. [11]

    Transactions on Machine Learning Research , year=

    Voyager: An Open-Ended Embodied Agent with Large Language Models , author=. Transactions on Machine Learning Research , year=

  11. [12]

    Min, Sewon and Krishna, Kalpesh and Lyu, Xinxi and Lewis, Mike and Yih, Wen-tau and Koh, Pang and Iyyer, Mohit and Zettlemoyer, Luke and Hajishirzi, Hannaneh , booktitle=

  12. [13]

    Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , publisher=

    On Faithfulness and Factuality in Abstractive Summarization , author=. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , publisher=

  13. [15]

    and Xu, Yan and Fung, Pascale , booktitle=

    Bang, Yejin and Cahyawijaya, Samuel and Lee, Nayeon and Dai, Wenliang and Su, Dan and Wilie, Bryan and Lovenia, Holy and Ji, Ziwei and Yu, Tiezheng and Chung, Willy and Do, Quyet V. and Xu, Yan and Fung, Pascale , booktitle=. A Multitask, Multilingual, Multimodal Evaluation of

  14. [16]

    2023 , howpublished=

    LLM Powered Autonomous Agents , author=. 2023 , howpublished=

    2023

  15. [17]

    Reiichiro Nakano and Jacob Hilton and Suchir Balaji and Jeff Wu and Long Ouyang and Christina Kim and Christopher Hesse and Shantanu Jain and Vineet Kosaraju and William Saunders and Xu Jiang and Karl Cobbe and Tyna Eloundou and Gretchen Krueger and Kevin Button and Matthew Knight and Benjamin Chess and John Schulman , journal=. Web

  16. [19]

    Zhong, Wanjun and Guo, Lianghong and Gao, Qiqi and Ye, He and Wang, Yanlin , journal=. Memory

  17. [20]

    Chenxu Hu and Jie Fu and Chenzhuang Du and Simian Luo and Junbo Zhao and Hang Zhao , journal=. Chat

  18. [21]

    Efficient Memory Management for Large Language Model Serving with

    Kwon, Woosuk and Li, Zhuohan and Zhuang, Siyuan and Sheng, Ying and Zheng, Lianmin and Yu, Cody Hao and Gonzalez, Joseph and Zhang, Hao and Stoica, Ion , booktitle=. Efficient Memory Management for Large Language Model Serving with

  19. [22]

    2024 , url=

    Asai, Akari and Wu, Zeqiu and Wang, Yizhong and Sil, Avirup and Hajishirzi, Hannaneh , booktitle=. 2024 , url=

    2024

  20. [24]

    and Lao, Ni and Lee, Hongrae and Juan, Da-Cheng and Guu, Kelvin , booktitle=

    Gao, Luyu and Dai, Zhuyun and Pasupat, Panupong and Chen, Anthony and Chaganty, Arun Tejasvi and Fan, Yicheng and Zhao, Vincent Y. and Lao, Ni and Lee, Hongrae and Juan, Da-Cheng and Guu, Kelvin , booktitle=. 2023 , url=

    2023

  21. [26]

    2023 , url=

    Madaan, Aman and Tandon, Niket and Gupta, Prakhar and Hallinan, Skyler and Gao, Luyu and Wiegreffe, Sarah and Alon, Uri and Dziri, Nouha and Prabhumoye, Shrimai and Yang, Yiming and Gupta, Shashank and Majumder, Bodhisattwa Prasad and Hermann, Katherine and Welleck, Sean and Yazdanbakhsh, Amir and Clark, Peter , booktitle=. 2023 , url=

    2023

  22. [28]

    Self-RAG : Learning to retrieve, generate, and critique through self-reflection

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG : Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2310.11511

    Pith/arXiv arXiv 2024

  23. [29]

    Do, Yan Xu, and Pascale Fung

    Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. A multitask, multilingual, multimodal evaluation of C hat GPT on reasoning, hallucination, and interactivity. In Proceedings of the 13th International Joint Conference on Natural Language P...

    2023

  24. [30]

    Chain-of-verification reduces hallucination in large language models

    Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. Chain-of-verification reduces hallucination in large language models. arXiv preprint arXiv:2309.11495, 2023

    Pith/arXiv arXiv 2023

  25. [31]

    Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu

    Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y. Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. RARR : Researching and revising what language models say, using language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023. URL https://arxiv.o...

    Pith/arXiv arXiv 2023

  26. [32]

    Retrieval-augmented generation for large language models: A survey

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2024

    Pith/arXiv arXiv 2024

  27. [33]

    Chat DB : Augmenting LLM s with databases as their symbolic memory

    Chenxu Hu, Jie Fu, Chenzhuang Du, Simian Luo, Junbo Zhao, and Hang Zhao. Chat DB : Augmenting LLM s with databases as their symbolic memory. arXiv preprint arXiv:2306.03901, 2023

    Pith/arXiv arXiv 2023

  28. [34]

    Survey of hallucination in natural language generation

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55 0 (12): 0 1--38, 2023

    2023

  29. [35]

    Language models (mostly) know what they know

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec,...

    Pith/arXiv arXiv 2022

  30. [36]

    Efficient memory management for large language model serving with PagedAttention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention . In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611--626. Association for Computing Machinery, 2023

    2023

  31. [37]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K\" u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\" a schel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459--9474. Curran Ass...

    2020

  32. [38]

    Self-Refine : Iterative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-Refine : Iterative refinement with self-feedback. In Advances in Neural Information Processing S...

    Pith/arXiv arXiv 2023

  33. [39]

    Evaluating very long-term conversational memory of LLM agents

    Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. Evaluating very long-term conversational memory of LLM agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, volume 1, pages 13851--13870, 2024

    2024

  34. [40]

    Selfcheck GPT : Zero-resource black-box hallucination detection for generative large language models

    Potsawee Manakul, Adian Liusie, and Mark Gales. Selfcheck GPT : Zero-resource black-box hallucination detection for generative large language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https://openreview.net/forum?id=RwzFNbJ3Ez

    2023

  35. [41]

    On faithfulness and factuality in abstractive summarization

    Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906--1919. Association for Computational Linguistics, 2020

    1906

  36. [42]

    FA ct S core: Fine-grained atomic evaluation of factual precision in long form text generation

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FA ct S core: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076--12100. Association for Comp...

    2023

  37. [43]

    Web GPT : Browser-assisted question-answering with human feedback

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Web GPT : Browser-assisted question-answering with human feedback. arXiv preprint ar...

    Pith/arXiv arXiv 2022

  38. [44]

    Patil, Ion Stoica, and Joseph E

    Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. Memgpt: Towards llms as operating systems. arXiv preprint arXiv:2310.08560, 2024

    Pith/arXiv arXiv 2024

  39. [45]

    Generative agents: Interactive simulacra of human behavior

    Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. Association for Computing Machinery, 2023

    2023

  40. [46]

    Reflexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, pages 8634--8652. Curran Associates, Inc., 2023

    2023

  41. [47]

    Voyager: An open-ended embodied agent with large language models

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. Transactions on Machine Learning Research, 2024

    2024

  42. [48]

    Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=1PL1NIMMrw

    2023

  43. [49]

    Llm powered autonomous agents

    Lilian Weng. Llm powered autonomous agents. https://lilianweng.github.io/posts/2023-06-23-agent/, 2023

    2023

  44. [50]

    Beyond goldfish memory: Long-term open-domain conversation

    Jing Xu, Arthur Szlam, and Jason Weston. Beyond goldfish memory: Long-term open-domain conversation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, volume 1, pages 5180--5197, 2022

    2022

  45. [51]

    Corrective retrieval augmented generation

    Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. Corrective retrieval augmented generation. arXiv preprint arXiv:2401.15884, 2024

    Pith/arXiv arXiv 2024

  46. [52]

    Adaptive memory admission control for LLM agents

    Guilin Zhang, Wei Jiang, Xiejiashan Wang, Aisha Behr, Kai Zhao, Jeffrey Friedman, Xu Chu, and Amine Anoun. Adaptive memory admission control for LLM agents. arXiv preprint arXiv:2603.04549, 2026

    arXiv 2026

  47. [53]

    Memory B ank: Enhancing large language models with long-term memory

    Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. Memory B ank: Enhancing large language models with long-term memory. arXiv preprint arXiv:2305.10250, 2023

    Pith/arXiv arXiv 2023

  48. [54]

    Agents: An open-source framework for autonomous language agents

    Wangchunshu Zhou, Yuchen Eleanor Jiang, Long Li, Jialong Wu, Tiannan Wang, Shi Qiu, Jintian Zhang, Jing Chen, Ruipu Wu, Shuai Wang, Shiding Zhu, Jiyu Chen, Wentao Zhang, Xiangru Tang, Ningyu Zhang, Huajun Chen, Peng Cui, and Mrinmaya Sachan. Agents: An open-source framework for autonomous language agents. arXiv preprint arXiv:2309.07870, 2023

    Pith/arXiv arXiv 2023