Pith. sign in

REVIEW 3 major objections 7 minor 37 references

Self-graded memory in LLM agents systematically overvalues wrong episodes, compounding through reuse, and can be corrected by a de-correlated, answer-free signal.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 01:43 UTC pith:Z2O75LUU

load-bearing objection Echo Gap is a real, well-measured phenomenon and LUCID looks effective; the EIA 'necessary' claim is proved only for a narrow linear update family, which the abstract overstates. the 3 major comments →

arxiv 2608.00017 v1 pith:Z2O75LUU submitted 2026-06-29 cs.AI cs.CE

Memory Reward Inflation in Self-Improving LLM Agents

Mohammad Asadolahi , Amir Amini , Samira Talebi , Amirfarhad Farhadi , Azadeh Zamanifar This is my paper
classification cs.AI cs.CE
keywords agentmemoryrewardstoredagentsinflationscoreself-improving
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Many AI agents try to improve from experience without retraining. They keep a memory of past questions and answers, give each answer a score, and later reuse the most similar memory when a new task arrives. This paper looks at what happens when the score is written by the AI itself, because in the real world there is often no gold answer to check against. The authors find that AI self-grading systematically overvalues wrong answers: on factual question banks, models marked 31-54% of their own wrong answers as correct. Worse, the wrong answers that get the highest scores are the ones the agent later retrieves and trusts most, so errors compound instead of canceling out. The authors call this the Echo Gap.

They prove a formal condition, the Error-Independence Assumption, for a signal to repair this: a verifier must track truth and must not make errors that echo the original self-grade bias. Parametric re-graders, even much stronger models, fail this test because their errors are correlated with the first grader. The paper's mitigation, LUCID, uses cheap answer-free signals - like SQL failing to execute or returning only NULLs - to demote inflated memories. On the BIRD text-to-SQL benchmark, LUCID raises execution accuracy to 56.9% versus 54.0% for the plain memory agent and 52.4% for no memory.

Core claim

The strongest claim is that the Error-Independence Assumption (EIA) is a necessary condition for correcting self-grade reward inflation in label-free memory agents: 'EIA is moreover a necessary condition for de-inflation, not merely a description of a good verifier' (Section 3.3). If correct, a usable de-inflation signal must track truth and decorrelate its error from the self-grade bias, and the recoverable payoff is the closed-form function of Section 3.3.

Load-bearing premise

The proof of EIA necessity (Proposition 2) assumes de-inflation takes the specific form Q'_i = Q_i - α(Q_i - V_i) - a linear pull of the stored score toward the verifier (Section 3.3, Appendix B). If a different correction mechanism could reduce inflation without satisfying EIA, the 'necessary condition' claim fails. The paper restricts to this update family without arguing why all reasonable de-inflations must be of this form.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper studies self-improving LLM agents that write episodes to an external memory with LLM-assigned utility scores. It defines the 'Echo Gap' as frequency-correlated reward inflation in label-free memory loops, reports leniency of self-graders across model families on factual banks, and claims that correcting the inflation requires a verifier satisfying the Error-Independence Assumption (EIA): high truth-tracking and low correlation between verifier error and self-grade bias. The paper further claims EIA is a necessary condition for de-inflation (Proposition 2), analyzes a mean-field write-back model with a corrupted attractor, and proposes LUCID, an answer-free hard-demotion algorithm. On BIRD text-to-SQL, LUCID is reported to improve execution accuracy over a Memento-style self-graded agent and a no-memory control. The paper includes code, data, and per-episode traces.

Significance. If the results hold, the Echo Gap is a useful characterization of a real failure mode in label-free memory-based agents, and the practical LUCID signal (execution failure, degeneracy, literal-grounding checks) is a transferable, low-cost mitigation. The bank-level verifier study is well designed: it compares multiple re-grading strategies, ensembles, and an information-retrieval channel, and the contrast between parametric re-graders and the retrieval-based verifier on both EIA axes is compelling. The paper's strengths include released artifacts, reproducible evaluation with pinned model versions, official BIRD execution-accuracy scoring, and explicit attention to causal controls (dose-response, budget-matched random pruning). However, the theoretical framing overreaches: the necessity claim for EIA is not supported for the deployed algorithm, and the attractor 'prediction' is an in-sample consistency check rather than a falsifiable out-of-sample test.

major comments (3)
  1. [§3.3, Proposition 2 (Appendix B), §6.3] The headline claim that EIA is a necessary condition for de-inflation is not supported by the proof. Proposition 2 analyzes only the linear-pull family Q'_i = Q_i − α(Q_i − V_i) and only the variance of the inflation, Var(b'). The deployed LUCID algorithm uses hard demotion of flagged memories, whose benefit is governed by the precision condition ρ > ρ* (Proposition 1), not by EIA. A concrete counterexample to necessity: a detector that flags 90% of wrong memories and 10% of correct ones (equal base rates) has precision 0.9 but Corr(ν,b)≈0.32, so it fails any plausible EIA decorrelation threshold yet, by the paper's own Proposition 1, improves the bank. Thus EIA is not necessary for the actual algorithm. The authors should either restrict the necessity claim to the linear-pull family and variance objective, or prove it for a broader class of demotion operators. As stated, the abstract's
  2. [§G, §5.2] The claimed confirmation of the corrupted-attractor prediction is not out-of-sample. The parameters (κ, ℓ, s, e0) in Section G are measured from the same BIRD traces that produce the observed trusted-bank corruption 0.42. Plugging these fitted parameters into Theorem 2 and obtaining p* = 0.45 is a consistency check of the model, not an independent prediction. The one-shot value 0.28 and the closed-loop value 0.45 are both functions of the same fitted parameters, so their comparison does not provide independent evidence of compounding; the only independent evidence for coupling is the measured positive κ, which is already a model assumption. The phrase 'independently measured' in Section G is therefore misleading, and the language about a 'falsifiable prediction' should be softened accordingly.
  3. [§3.4] The Echo Gap is defined by two conditions: positive self-grade bias on wrong memories (Equation 5) and positive covariance between bias and reuse among wrong memories (Equation 6). The empirical section reports leniency (Table 2) but gives no numerical estimate of Cov(b_i, n_i | U_i=0) or Corr(b_i, n_i | U_i=0) on the factual banks. The text asserts this correlation exists but provides no table, figure, or statistic. The BIRD coupling κ in Section G is measured under similarity-only retrieval and corresponds to the trust channel, not the score-ranked reuse covariance. Without a reported estimate, the bank-level Echo Gap is only half-demonstrated. The authors should report the conditional covariance/correlation for the factual banks.
minor comments (7)
  1. [Abstract] Typo: 'baseed' should be 'based'.
  2. [Introduction] Typo: 'do do not have a gold label' should be 'do not have a gold label'.
  3. [§5.2] Typo: 'demonestrate' should be 'demonstrate'.
  4. [§3.3 / §F] The precision threshold ρ* = hr/(gr+hr) is reported as 0.5, which implicitly assumes gr = hr. This assumption should be stated explicitly, since gr and hr are not estimated.
  5. [§6.4 / Table 5] The claim that LUCID delivers a 'consistent' gain rests on only two seeds. The per-seed confidence intervals are useful, but the word 'consistent' should be qualified as 'across the two tested seeds'.
  6. [§3.3] The statement that the answer-free signal is 'decorrelated by construction' (Section 3.3) is not a proof; residual correlation can arise through the planner's behavior. This is not load-bearing, since the precision condition is what drives LUCID, but the phrasing invites misunderstanding.
  7. [References] Reference [23] is listed as 'Anonymous'; if the journal requires author names for arXiv preprints, this should be resolved.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

No new physical entities are postulated. The paper introduces named concepts (Echo Gap, EIA, LUCID) but no new forces, particles, or conserved quantities with independent falsifiable handles.

free parameters (4)
  • memory-to-behavior coupling κ = 0.38
    Measured on BIRD traces by OLS slope of agent error on retrieved-set corruption (Section G); used in Theorem 2 to predict the corrupted attractor.
  • self-grader leniency ℓ = 0.76
    Measured on BIRD as Pr[self-grade=1 | U=0] (Section G); input to mean-field attractor.
  • self-grader sensitivity s = 0.90
    Measured as Pr[self-grade=1 | U=1]; input to mean-field attractor.
  • clean-bank error e0 = 0.32
    Measured error rate when retrieved set is clean; input to mean-field attractor.
axioms (5)
  • domain assumption The memory-loop reward model: stored utility score Qi acts as a reward signal for future retrieval and trust.
    Section 3.1 states this framing; it is the lens of the paper.
  • ad hoc to paper De-inflation is modeled as a linear pull toward the verifier: Q'_i = Qi - α(Qi - Vi).
    Used in Proposition 2 to prove EIA necessity; restricts the space of corrections, so necessity is conditional on this update family.
  • domain assumption Trust weight τ is non-decreasing in stored score and strictly increasing on {0, b}.
    Theorem 1(ii) trust channel relies on this property of the planner's conditioning behavior.
  • domain assumption Softmax retrieval in the retrieval channel: π(i) ∝ e^{Qi/T}.
    Theorem 1(i) models score-ranked retrieval via softmax; top-k is the T→0 limit.
  • ad hoc to paper Mean-field write-back model: p_{t+1} = q(p_t) with q(p)=eℓ/(eℓ+(1-e)s), e=e0+κp.
    Theorem 2's stylized dynamics; chosen for measurability.

pith-pipeline@v1.3.0-alltime-deepseek · 21700 in / 13975 out tokens · 123258 ms · 2026-08-04T01:43:10.283804+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Memory Reward Inflation in Self-Improving LLM Agents." pith.science (2026). https://pith.science/paper/Z2O75LUU

@misc{pith2026260800017,
  author       = {Pith},
  title        = {Pith review of: Memory Reward Inflation in Self-Improving LLM Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z2O75LUU}},
  note         = {Machine review of arXiv:2608.00017}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Self-improving LLM agents increasingly learn from experience without updating any weights. Each episode is stored in an external memory, scored, and retrieved for similar future tasks to shape later behavior. Viewed through a reward lens, the stored score is a proxy reward for an implicit, non-parametric policy. Each retrieved episode then becomes a policy-improvement step whose reliability hinges on how that score is produced. In deployment, ground-truth labels are unavailable, so the stored reward is at best an LLM assessment. This substitution creates a failure mode, the *Echo Gap*, across the memory-based self-improving agents and model families studied. Incorrect episodes receive inflated rewards; thus, the agent preferentially reuses the very mistakes it has most confident in. Because the error compounds through memory rather than averaging out and the confirming judge's errors remain correlated with the original self-grading bias, so it cannot identify which memories are overvalued. The missing property is formalized as the *Error-Independence Assumption* (EIA), which we prove is a *necessary* condition for correcting the inflation, not merely a description of a good verifier: a usable signal must track truth *and* decorrelate its error from the memory bias, and the recoverable payoff is a closed-form function of exactly those two quantities. We further show the inflation compounds not only when retrieval ranks by the stored score but also under plain similarity retrieval which is the regime the deployed agent uses. Finally, the answer-free de-inflation algorithm LUCID delivers a consistent end-to-end gain on the BIRD text-to-SQL benchmark. It raises execution accuracy to $56.9\%$, above both a Memento-style self-graded agent ($54.0\%$, a $+2.9$-point mean gain across seeds) and a memory-less agent of identical architecture ($52.4\%$).

Figures

Figures reproduced from arXiv: 2608.00017 by Amir Amini, Amirfarhad Farhadi, Azadeh Zamanifar, Mohammad Asadolahi, Samira Talebi.

Figure 1
Figure 1. Figure 1: Conceptual mechanism of the Echo Gap in self improving memory agents and LUCID [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Measuring the memory-to-behavior coupling on BIRD. Agent error rate rises monotonically [PITH_FULL_IMAGE:figures/full_fig_p024_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

37 extracted references · 19 linked inside Pith

  1. [1]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto.Reinforcement Learning: An Introduction. MIT Press, second edition, 2018

    2018

  2. [2]

    Reflexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. InAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  3. [3]

    Self-refine: Iterative refinement with self-feedback

    Aman Madaan et al. Self-refine: Iterative refinement with self-feedback. InAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  4. [4]

    Memento: Fine-tuning LLM agents without fine-tuning LLMs.arXiv preprint arXiv:2508.16153, 2025

    Huichi Zhou et al. Memento: Fine-tuning LLM agents without fine-tuning LLMs.arXiv preprint arXiv:2508.16153, 2025. 24

    Pith/arXiv arXiv 2025

  5. [5]

    Christiano, Jan Leike, Tom B

    Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. InAdvances in Neural Information Processing Systems (NeurIPS), 2017

    2017

  6. [6]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedba...

    2022

  7. [7]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-Bench and chatbot arena. InAdvances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2023. arXiv:2306.05685

    Pith/arXiv arXiv 2023

  8. [8]

    Bowman, and Shi Feng

    Jane Pan, He He, Samuel R. Bowman, and Shi Feng. Spontaneous reward hacking in iterative self-refinement.arXiv preprint arXiv:2407.04549, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    ReAct: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  10. [10]

    Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. InAdvances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  11. [11]

    Retrieval-augmented generation for knowledge-intensive NLP tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. InAdvances in Neural Information Processing Systems (NeurIPS), 2020

    2020

  12. [12]

    O’Brien, Carrie J

    Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. InACM Symposium on User Interface Software and Technology (UIST), 2023

    2023

  13. [13]

    ReasoningBank: Scaling agent self-evolving with reasoning memory.arXiv preprint arXiv:2509.25140, 2025

    Siru Ouyang, Jun Yan, I-Hung Hsu, Yanfei Chen, Ke Jiang, et al. ReasoningBank: Scaling agent self-evolving with reasoning memory.arXiv preprint arXiv:2509.25140, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    MemRL: Self-evolving agents via runtime reinforcement learning on episodic memory.arXiv preprint arXiv:2601.03192, 2026

    Shengtao Zhang, Jiaqian Wang, Ruiwen Zhou, et al. MemRL: Self-evolving agents via runtime reinforcement learning on episodic memory.arXiv preprint arXiv:2601.03192, 2026

    Pith/arXiv arXiv 2026

  15. [15]

    Memory-R1: Enhancing large language model agents to manage and utilize memories via reinforcement learning.arXiv preprint arXiv:2508.19828, 2025

    Sikuan Yan, Xiufeng Yang, Zuchao Huang, et al. Memory-R1: Enhancing large language model agents to manage and utilize memories via reinforcement learning.arXiv preprint arXiv:2508.19828, 2025

    Pith/arXiv arXiv 2025

  16. [16]

    Mem-α: Learning memory construction via reinforcement learning.arXiv preprint arXiv:2509.25911, 2025

    Yu Wang, Ryuichi Takanobu, Zhiqi Liang, et al. Mem-α: Learning memory construction via reinforcement learning.arXiv preprint arXiv:2509.25911, 2025

    Pith/arXiv arXiv 2025

  17. [17]

    Agentic memory: Learning unified long-term and short- term memory management for large language model agents.arXiv preprint arXiv:2601.01885, 2026

    Yi Yu, Liuyi Yao, Yuexiang Xie, et al. Agentic memory: Learning unified long-term and short- term memory management for large language model agents.arXiv preprint arXiv:2601.01885, 2026. 25

    Pith/arXiv arXiv 2026

  18. [18]

    Learning when to remember: Risk-sensitive contextual bandits for abstention- aware memory retrieval in LLM-based coding agents.arXiv preprint arXiv:2604.27283, 2026

    Mehmet Iscan. Learning when to remember: Risk-sensitive contextual bandits for abstention- aware memory retrieval in LLM-based coding agents.arXiv preprint arXiv:2604.27283, 2026

    Pith/arXiv arXiv 2026

  19. [19]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. InInternational Conference on Learning Representations (ICLR), 2023. arXiv:2203.11171

    Pith/arXiv arXiv 2023

  20. [20]

    Process reward models that think.arXiv preprint arXiv:2504.16828, 2025

    Muhammad Khalifa et al. Process reward models that think.arXiv preprint arXiv:2504.16828, 2025

    arXiv 2025

  21. [21]

    Self-preference bias in LLM-as-a-judge

    Koki Wataoka, Tsubasa Takahashi, and Ryokan Ri. Self-preference bias in LLM-as-a-judge. arXiv preprint arXiv:2410.21819, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    Beyond the surface: Measuring self-preference in LLM judgments.arXiv preprint arXiv:2506.02592, 2025

    Zhi-Yuan Chen et al. Beyond the surface: Measuring self-preference in LLM judgments.arXiv preprint arXiv:2506.02592, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    Nine judges, two effective votes: Correlated errors undermine LLM evaluation panels.arXiv preprint arXiv:2605.29800, 2026

    Anonymous. Nine judges, two effective votes: Correlated errors undermine LLM evaluation panels.arXiv preprint arXiv:2605.29800, 2026

    Pith/arXiv arXiv 2026

  24. [24]

    QuickCheck: A lightweight tool for random testing of Haskell programs

    Koen Claessen and John Hughes. QuickCheck: A lightweight tool for random testing of Haskell programs. InInternational Conference on Functional Programming (ICFP), pages 268–279, 2000

    2000

  25. [25]

    Finding and understanding bugs in C compilers

    Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. Finding and understanding bugs in C compilers. InProgramming Language Design and Implementation (PLDI), pages 283–294, 2011

    2011

  26. [26]

    Metamorphic testing: A new approach for generating next test cases

    Tsong Yueh Chen, Shing Chi Cheung, and Siu Ming Yiu. Metamorphic testing: A new approach for generating next test cases. Technical Report HKUST-CS98-01, Department of Computer Science, Hong Kong University of Science and Technology, 1998

    1998

  27. [27]

    Sanchez, and Antonio Ruiz-Cortés

    Sergio Segura, Gordon Fraser, Ana B. Sanchez, and Antonio Ruiz-Cortés. A survey on metamorphic testing.IEEE Transactions on Software Engineering, 42(9):805–824, 2016

    2016

  28. [28]

    CodeT: Code generation with generated tests

    Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. CodeT: Code generation with generated tests. InInternational Conference on Learning Representations (ICLR), 2023. arXiv:2207.10397

    Pith/arXiv arXiv 2023

  29. [29]

    SEDM: Scalable self-evolving distributed memory for agents.arXiv preprint arXiv:2509.09498, 2025

    Haoran Xu et al. SEDM: Scalable self-evolving distributed memory for agents.arXiv preprint arXiv:2509.09498, 2025

    arXiv 2025

  30. [30]

    A-MemGuard: A proactive defense framework for LLM-based agent memory.arXiv preprint arXiv:2510.02373, 2025

    Qianshan Wei, Tengchao Yang, Yaochen Wang, et al. A-MemGuard: A proactive defense framework for LLM-based agent memory.arXiv preprint arXiv:2510.02373, 2025

    arXiv 2025

  31. [31]

    MemMA: Coordinating the memory cycle through multi-agent reasoning and in-situ self-evolution.arXiv preprint arXiv:2603.18718, 2026

    Minhua Lin, Zhiwei Zhang, Hanqing Lu, et al. MemMA: Coordinating the memory cycle through multi-agent reasoning and in-situ self-evolution.arXiv preprint arXiv:2603.18718, 2026

    arXiv 2026

  32. [32]

    Useful memories become faulty when continuously updated by LLMs.arXiv preprint arXiv:2605.12978, 2026

    Dylan Zhang et al. Useful memories become faulty when continuously updated by LLMs.arXiv preprint arXiv:2605.12978, 2026

    Pith/arXiv arXiv 2026

  33. [33]

    Large language models cannot self-correct reasoning yet

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. InInternational Conference on Learning Representations (ICLR), 2024. arXiv:2310.01798. 26

    Pith/arXiv arXiv 2024

  34. [34]

    Estimating the accuracies of multiple classifiers without labeled data.Artificial Intelligence and Statistics (AISTATS), 2015

    Ariel Jaffe, Boaz Nadler, and Yuval Kluger. Estimating the accuracies of multiple classifiers without labeled data.Artificial Intelligence and Statistics (AISTATS), 2015. arXiv:1407.7644

    Pith/arXiv arXiv 2015

  35. [35]

    The logic of NTQR evaluations of noisy AI agents.arXiv preprint arXiv:2312.05392, 2023

    Andrés Corrada-Emmanuel. The logic of NTQR evaluations of noisy AI agents.arXiv preprint arXiv:2312.05392, 2023

    Pith/arXiv arXiv 2023

  36. [36]

    Can LLM already serve as a database interface? a BIg bench for large-scale database grounded text-to-SQLs

    Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, et al. Can LLM already serve as a database interface? a BIg bench for large-scale database grounded text-to-SQLs. In Advances in Neural Information Processing Systems (NeurIPS), 2023. arXiv:2305.03111

    Pith/arXiv arXiv 2023

  37. [37]

    SimCSE: Simple contrastive learning of sentence embeddings

    Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. InEmpirical Methods in Natural Language Processing (EMNLP), 2021. 27

    2021