Pith. sign in

REVIEW 4 major objections 5 minor 53 references

Continuously Steering LLMs Sensitivity to Contextual Knowledge with Proxy Models

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CSKS claims that adding a scaled contrastive difference between two small proxy models' output distributions to a frozen large LLM's logits dials its sensitivity to context continuously in both directions.

desk verdict A useful same-family decoding-time dial for context-vs-parametric trust, but the black-box cross-tokenizer claim is undefined and the paper has a numbers inconsistency. read the letter →

arxiv 2508.19720 v3 pith:HGFKYUN4 submitted 2025-08-27 cs.CL

Yilin Wang , Heng Wang , Yuyang Bai , Minnan Luo This is my paper
classification cs.CL
keywords knowledgeconflictscontextualfaithfulnessdecoding-timecontrolproxymodelscontrastivedecodingretrieval-augmentedgenerationsensitivitysteeringLLMevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that an LLM's sensitivity to contradictory knowledge in its input context can be dialed up or down continuously, in both directions, without touching the large model's weights. The recipe is to fine-tune two small models: one trained to follow context, one trained to stick to its internal knowledge. At every decoding step, the difference between their output distributions is added to the large model's logits, scaled by a single hyperparameter. If correct, this gives retrieval-augmented systems a lightweight, black-box-compatible knob for deciding whether to trust fresh context or stored parametric knowledge, and it provides a graded evaluation method for measuring that trust.

What carries the argument

The central object is a contrastive proxy pair: P, a small LM fine-tuned on ECQA-derived examples to follow the contextual answer, and N, fine-tuned to keep the parametric answer; both are roughly a tenth of the target's size and share its tokenizer. The steering identity is X_t ~ softmax[DL + (DP − DN) α], applied autoregressively at every step; DP − DN encodes a 'context-versus-memory' direction, and α is the dial. The paper's evaluation machinery is a difficulty-ranked Sensitivity Score that weights each conflict by perturbation degree, contextual detail, and corpus frequency rather than reporting plain accuracy.

What would settle it

Feed the target model a question whose parametric answer is known and a context asserting the opposite answer, then sweep α from −2 to +2. The paper's claim predicts a monotone transition from the parametric answer to the contextual answer; a flat or non-monotone curve would falsify it. A sharper check: perform the same sweep when the proxy pair and target do not share a tokenizer and only top-five logprobs are available—if the steering direction disappears there, the black-box claim fails.

Watch

Extended reading notes

Core claim

The decoding rule is X_t ~ softmax[DL + (DP − DN) α]: at each step, the large model's logits DL are shifted by the scaled difference between the output distributions of a context-faithful proxy P and a parametric-faithful proxy N, all conditioned on the same query, context, and previously generated prefix. Positive α moves generation toward the contextual answer; negative α moves it back toward parametric knowledge. The paper shows monotone, continuous control of its sensitivity score on Llama-3-70B, Qwen2.5-72B, and Gemma-2-27B, extends the same steering to GPT-3.5-Turbo through top-five logprobs, and validates on DynamicQA real-world conflicts. Extreme α degrades MMLU and multi-hop reasoni

Load-bearing premise

The method assumes that the difference between two small, fine-tuned proxy models' output distributions, computed on the target model's own generated prefix, remains a reliable steering direction when applied to a much larger model—including, in the black-box case, a model with a different tokenizer and only five exposed token probabilities.

Editorial extensions

If this is right

  • RAG systems can adjust how much retrieved passages override model memory per deployment, without retraining the generator.
  • Models that resist updating outdated facts can be pushed toward context-faithful answering with positive α, which the paper shows improves accuracy on DynamicQA conflict partitions.
  • Negative α gives a knob for reducing context influence, useful when the supplied context is noisy, outdated, or untrusted.
  • Because only output distributions are modified, the approach extends to closed models that expose logprobs for a few tokens, as demonstrated on GPT-3.5-Turbo.
  • The difficulty-ranked Sensitivity Score lets future work measure context adherence more granularly than binary accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the proxies are trained once and the target weights stay frozen, the same proxy pair should in principle steer any target model sharing its tokenizer; the per-request cost is two extra small-model forward passes, suggesting the method could be offered as a cheap API-level control on top of RAG services.
  • The monotone negative-α direction implies a testable use: suppressing context influence could act as a lightweight defense against prompt-injected or misleading context, at the cost of the general-capability degradation the paper documents for extreme α.
  • The paper's own limitation—manual α calibration—invites an automatic scheduler that adapts α per query, for example from the disagreement between DL and DP − DN or from the target model's confidence.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes CSKS, a decoding-time framework that steers a target LLM's sensitivity to contextual knowledge by adding a scaled difference of output distributions from two fine-tuned small "proxy" models: a positive proxy P trained to follow context and a negative proxy N trained to rely on parametric knowledge. At each generation step the adjusted distribution is softmax[DL + (DP - DN) * alpha], with alpha controlling the direction and strength of steering. The authors introduce a synthetic benchmark with difficulty dimensions (perturbation degree, contextual detail, popularity) and a Sensitivity Score, and evaluate on Llama-3-70B, Qwen2.5-72B, Gemma-2-27b, and GPT-3.5-Turbo. They report continuous bidirectional control, superiority over CAD/COIECD/prompting baselines, proxy-size scalability, and trade-off experiments on MMLU and 2WikiMultiHopQA.

Significance. If the claims hold, CSKS is a practically attractive control knob for RAG systems: it avoids modifying the target model's weights, requires only fine-tuning small proxies, and offers a single continuous alpha to trade context-following against parametric reliance. The empirical package is unusually broad for a short paper: two synthetic datasets, a real-world conflict benchmark, proxy-size ablation, cross-family Gemini results, and general-capability trade-offs. The core same-family evidence (Llama and Qwen proxies steering same-family targets) is credible and consistent with the proposed mechanism. However, the paper's headline black-box/cross-family claim rests on a GPT-3.5 experiment whose steering signal is not well-defined across tokenizers, and the main controllability figure is internally inconsistent with Table 1. These issues are load-bearing for the advertised advantages, though fixable in revision.

major comments (4)
  1. [Sec. 2.1 vs. Sec. 3.5, Eq. (1), Table 3] The black-box/cross-family experiment is not well-defined. Sec. 2.1 states that proxies and target need only share vocabulary/tokenization, and all open-model experiments use same-family proxies accordingly. Yet Sec. 3.5 applies LLaMA-3-8B and Qwen2.5-7B proxies to GPT-3.5-Turbo, which uses a different tokenizer. The paper never specifies how DP and DN are computed for GPT-3.5's top-five tokens: are the five tokens string-matched to proxy-vocabulary entries? Is the prefix re-tokenized with the proxy tokenizer? Are proxy log-probs summed over subword pieces? Without such a mapping, Eq. (1) is undefined across vocabularies and Table 3 cannot be reproduced or falsified. The Appendix H Gemma-2-27b experiment uses Gemma-2-2b, which shares a tokenizer, so it does not supply cross-family or black-box evidence. This undermines a central advertised advantage.
  2. [Sec. 3.3, Figure 3, Table 1] The reported controllability numbers are internally inconsistent. The text says "increasing α values (α > 0) produce a monotonic enhancement of sensitivity score from 4.32 to 39.80 for LLaMA on MuSiQue," but Table 1 lists Origin at 38.13 and CSKS at 66.72 for the same setup. If α=0 is the Origin baseline, a positive-α trajectory starting at 4.32 is impossible; if 4.32 corresponds to a different α anchor, that anchor is not identified. Since continuous control is the paper's central contribution, this discrepancy must be resolved with a clear description of the axis, the α values, and the relationship to Table 1.
  3. [Sec. 3.3, Table 1, Appendix C] The α value used for CSKS in the main results is not stated. Appendix C says "we do not search for an optimal parameter but just setting α the to same as CAD," but the CAD hyperparameter value is not given anywhere. Because Eq. (1) is a one-parameter family, the headline comparison in Table 1 depends entirely on this unspecified value. Please report the exact α (or range) used for each experiment and justify the choice.
  4. [Sec. 3.4, Figure 4] The DynamicQA evaluation lacks a precise scoring protocol. The paper reports accuracy as a function of α for Static, Temporal, and Disputable conflicts, but does not state how the "correct" answer is determined in each partition, whether the context always contains the target answer, or whether the metric measures context-following rather than factual correctness. Since the synthetic evaluation deliberately defines 'correct' as the context answer, the DynamicQA protocol must be specified to show that the real-world result is not merely re-labeling context-following as accuracy.
minor comments (5)
  1. [Eq. (1)] The notation "softmax[DL + (DP − DN) α]" is ambiguous; it should be written as softmax(DL + α(DP − DN)) to make clear that α scales the difference term.
  2. [Sec. 2.1, Sec. 5] Typos: "contect" in Sec. 2.1 and "Conlusion" in Sec. 5. The sentence "just setting α the to same as CAD" in Appendix C is also ungrammatical and should be rewritten.
  3. [Sec. 3.3, Figure 3] The figure is difficult to read in the submitted version because axis labels and data-point annotations are garbled (appearing as "/uni..." strings). Please provide a clean vector figure with legible tick labels and values.
  4. [Table 2] The text says "particularly Humanities (-4.10%)" but the table shows a 1.03-point drop at α=-2.0 and a 4.13-point drop at α=+2.0 relative to the α=0 row. Please clarify which delta is being cited and report it consistently.
  5. [Sec. 3.5, black-box setting] The statement "CSKS only reweights the five tokens" implies that the adjusted distribution is not a full softmax over the vocabulary and that sampling is restricted to the API's top-five tokens. This is a different decoding rule from Eq. (1) and should be described precisely, including how ties and off-top-five context answers are handled.

Circularity Check

1 steps flagged · score 5.0 of 10

Synthetic sensitivity metric and steering direction are aligned by construction; real-world DynamicQA provides partial independent grounding.

  1. self definitional [Sec. 2.1 Eq. (1); Sec. 3.2; Appendix A]
    "˜Xt ∼ softmax[DL + (DP − DN ) α] ... Correct options correspond to the answers in context, and the incorrect options correspond to the original answers to the question. ... The P model was fine-tuned on the selected answers and their associated explanations. The N model was fine-tuned on the original correct answers and their explanations."

    The steering signal is the log-probability difference between a model fine-tuned to emit the constructed 'contextual answer' (P) and a model fine-tuned to emit the original answer (N). The synthetic evaluation's 'correct' option is exactly that contextual answer, and the Sensitivity Score sums difficulty weights over such context-answer responses. Therefore DP−DN is, by training construction, a positive logit shift toward the tokens the metric counts as correct; increasing α monotonically raises the score. The headline 'continuous and precise control' curves are a definitional consequence of this alignment rather than an independent test. DynamicQA and MMLU/2Wiki results are not built this way, so they provide independent support; hence partial, not total, circularity.

full rationale

The main circular element is the alignment between the proxy-training objective and the synthetic evaluation metric. P is fine-tuned to output the context answer, N is fine-tuned to output the parametric answer, and then the Sensitivity Score defines the context answer as correct. Thus the steering direction of the method and the measured 'sensitivity' are the same quantity by construction. This makes the synthetic Table 1 and Figure 3 partly a consistency check rather than a fully independent validation. However, the paper also evaluates on DynamicQA, where the context answer is the real-world updated answer and the proxies were not trained on that benchmark; the monotonic α-control there is genuine empirical evidence. The MMLU and 2WikiMultiHopQA trade-off experiments also provide independent grounding. The black-box GPT-3.5 experiment has an undefined cross-tokenizer mapping for DP−DN (the paper itself requires shared vocabulary in Sec. 2.1 but applies LLaMA/Qwen proxies to GPT-3.5-Turbo); this is a reproducibility/correctness concern, not a circularity concern, so it does not increase the circularity score. There is no load-bearing self-citation or imported uniqueness theorem. Overall, the central claim retains independent content, but the synthetic 'prediction' partially reduces to the training definition, warranting a moderate score of 5.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new physical or conceptual entities are introduced; the P/N proxy models are existing architectures, and the Sensitivity Score is an evaluation metric rather than an entity. The method's reliance is on hand-chosen hyperparameters and empirical transfer assumptions, not on invented objects.

free parameters (2)
  • alpha (steering knob) = Ranges -2.0 to +2.0 in experiments; value used in Table 1 not stated
    Scales the proxy distribution difference; the central control lever, set by hand and swept to demonstrate continuous control.
  • Proxy fine-tuning configuration = Not fully specified in main text (Appendix A)
    Includes the P/N answer-selection rule (least versus most related wrong option), context length choices, and LoRA/learning-rate settings, all of which the results depend on.
assumptions (4)
  • ad hoc to paper Logit-space additive shift of the form DL + alpha*(DP - DN) preserves coherence and monotonically modifies contextual sensitivity.
    Core methodological premise; no derivation or proof is given, only empirical alpha sweeps.
  • domain assumption Small proxy models fine-tuned on ECQA generalize to MuSiQue, PopQA, and DynamicQA conflict distributions.
    Fine-tuning domain differs from evaluation domains, but transfer is assumed and only partially validated across model families.
  • domain assumption GPT-4o-mini generated synthetic data faithfully represents knowledge conflicts and difficulty.
    The evaluation pipeline relies on the generator for perturbation, context, and difficulty ranking; no human validation is reported.
  • domain assumption The target model's previous tokens X<t can be fed to proxy models as conditioning, even though they come from a different distribution.
    Proxies are trained on their own generations but conditioned on the target's prefix at inference, a covariate-shift risk.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Continuously Steering LLMs Sensitivity to Contextual Knowledge with Proxy Models." pith.science (2026). https://pith.science/paper/HGFKYUN4

@misc{pith2026250819720,
  author       = {Pith},
  title        = {Pith review of: Continuously Steering LLMs Sensitivity to Contextual Knowledge with Proxy Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HGFKYUN4}},
  note         = {Machine review of arXiv:2508.19720}
}
read the original abstract

In Large Language Models (LLMs) generation, there exist knowledge conflicts and scenarios where parametric knowledge contradicts knowledge provided in the context. Previous works studied tuning, decoding algorithms, or locating and editing context-aware neurons to adapt LLMs to be faithful to new contextual knowledge. However, they are usually inefficient or ineffective for large models, not workable for black-box models, or unable to continuously adjust LLMs' sensitivity to the knowledge provided in the context. To mitigate these problems, we propose CSKS (Continuously Steering Knowledge Sensitivity), a simple framework that can steer LLMs' sensitivity to contextual knowledge continuously at a lightweight cost. Specifically, we tune two small LMs (i.e. proxy models) and use the difference in their output distributions to shift the original distribution of an LLM without modifying the LLM weights. In the evaluation process, we not only design synthetic data and fine-grained metrics to measure models' sensitivity to contextual knowledge but also use a real conflict dataset to validate CSKS's practical efficacy. Extensive experiments demonstrate that our framework achieves continuous and precise control over LLMs' sensitivity to contextual knowledge, enabling both increased sensitivity and reduced sensitivity, thereby allowing LLMs to prioritize either contextual or parametric knowledge as needed flexibly. Our data and code are available at https://github.com/OliveJuiceLin/CSKS.

Figures

Figures reproduced from arXiv: 2508.19720 by the authors.

Figure 1
Figure 1. (left) The pipeline we use to build the proxy models, where each box represents a processing step. The two paths on either side correspond to different processing methods applicable to the proxy models. Details are shown in Appendix A. (right) When confronted with conflicting contexts, the proxy models function together as a guiding "steering wheel", assisting the large model in aligning more closely with the contex… view at source ↗
Figure 2
Figure 2. Performance of models of different sizes un [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The performance of LLaMA and Qwen controlled bidirectionally, demonstrating the continuous adjustment [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Accuracy of Qwen2.5 steered by CSKS on the DynamicQA dataset as a function of the control parameter α. Results are shown overall and broken down by con￾flict partition type, demonstrating CSKS’s effectiveness and continuous controllability in handling diverse real-worl…
Figure 5
Figure 5. Figure 5: The performance of CSKS under varying proxy model sizes on MuSiQue and PopQA respectively. Smaller proxy models (0.5B, 1.5B) have a marginal yet increasing effect on the 72B target model’s sensitivity score. Remarkably, the 3B proxy model matches the 7B model in sensit…
Figure 6
Figure 6. Figure 6: The pipeline to get the data used to finetune our [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: The accuracy of the LLaMA-3-70B-Instruct [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 9
Figure 9. Figure 9: Sensitivity score variation with alpha values [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 8
Figure 8. Figure 8: The effects of different methods on the LLaMA [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 10
Figure 10. Figure 10: Comparative performance (Accuracy %) of CSKS and baseline methods (Raw Model, Prompt, CAD, COIECD) on the DynamicQA dataset. Results are shown for Overall Accuracy and broken down by conflict parti￾tion types: Static, Temporal, and Disputable. CSKS con￾sistently outpe…
Figure 11
Figure 11. Figure 11: The prompt we use to ask gpt to make a slight [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 13
Figure 13. Figure 13: The prompt we use to ask gpt to generate a [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: The prompt we use to ask gpt to generate a [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 22 canonical work pages

  1. [1]

    Shourya Aggarwal, Divyanshu Mandowara, Vishwajeet Agrawal, Dinesh Khandelwal, Parag Singla, and Dinesh Garg. 2021. https://doi.org/10.18653/v1/2021.acl-long.238 E xplanations for C ommonsense QA : N ew D ataset and M odels . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conferen...

    work page doi:10.18653/v1/2021.acl-long.238 2021

  2. [2]

    Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. 2023. https://openreview.net/forum?id=ETKGuby0hcs Discovering latent knowledge in language models without supervision . In The Eleventh International Conference on Learning Representations

    work page 2023

  3. [3]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  4. [4]

    Shangbin Feng, Taylor Sorensen, Yuhan Liu, Jillian Fisher, Chan Young Park, Yejin Choi, and Yulia Tsvetkov. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.240 Modular pluralism: Pluralistic alignment via multi- LLM collaboration . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4151--4171, Miami, Florida...

    doi:10.18653/v1/2024.emnlp-main.240 2024

  5. [5]

    Zorik Gekhman, Jonathan Herzig, Roee Aharoni, Chen Elkind, and Idan Szpektor. 2023. https://openreview.net/forum?id=rRwPzcSFeL Trueteacher: Learning factual consistency evaluation with large language models . In The 2023 Conference on Empirical Methods in Natural Language Processing

    work page 2023

  6. [6]

    Anshita Gupta, Debanjan Mondal, Akshay Sheshadri, Wenlong Zhao, Xiang Li, Sarah Wiegreffe, and Niket Tandon. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.511 Editing common sense in transformers . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8214--8232, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.emnlp-main.511 2023

  7. [7]

    Chi Han, Jialiang Xu, Manling Li, Yi Fung, Chenkai Sun, Nan Jiang, Tarek Abdelzaher, and Heng Ji. 2024. https://doi.org/10.18653/v1/2024.acl-long.864 Word embeddings are steers for language models . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16410--16430, Bangkok, Thailand. Ass...

    doi:10.18653/v1/2024.acl-long.864 2024

  8. [8]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR)

    2021

Show all 53 references
  1. [9]

    Li, and Jacob Andreas

    Evan Hernandez, Belinda Z. Li, and Jacob Andreas. 2024. https://openreview.net/forum?id=ADtL6fgNRv Inspecting and editing knowledge representations in language models . In First Conference on Language Modeling

    2024

  2. [10]

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. https://doi.org/10.18653/v1/2020.coling-main.580 Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps . In Proceedings of the 28th International Conference on Computational Li...

    2020 doi

  3. [11]

    Zhuoran Jin, Pengfei Cao, Yubo Chen, Kang Liu, Xiaojian Jiang, Jiexin Xu, Qiuxia Li, and Jun Zhao. 2024. Tug-of-war between knowledge: Exploring and resolving knowledge conflicts in retrieval-augmented language models. arXiv preprint arXiv:2402.14409

    2024 arXiv

  4. [12]

    Evgenii Kortukov, Alexander Rubinstein, Elisa Nguyen, and Seong Joon Oh. 2024. https://openreview.net/forum?id=xm8zYRfrqE Studying large language model behaviors under context-memory conflicts with real documents . In First Conference on Language Modeling

    2024

  5. [13]

    Kyungjae Lee, Wookje Han, Seung-won Hwang, Hwaran Lee, Joonsuk Park, and Sang-Woo Lee. 2022. https://doi.org/10.18653/v1/2022.findings-acl.37 Plug-and-play adaptation for continuously-updated QA . In Findings of the Association for Computational Linguistics: ACL 2022, pages 43...

    2022 doi

  6. [14]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K\" u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\" a schel, Sebastian Riedel, and Douwe Kiela. 2020. https://proceedings.neurips.cc/paper_files/paper/2020/file/6b493230205f78...

    2020

  7. [15]

    Kenneth Li, Oam Patel, Fernanda Vi \'e gas, Hanspeter Pfister, and Martin Wattenberg. 2023 a . https://openreview.net/forum?id=aLLuYpn83y Inference-time intervention: Eliciting truthful answers from a language model . In Thirty-seventh Conference on Neural Information Processi...

    2023

  8. [16]

    Sha Li, Chi Han, Pengfei Yu, Carl Edwards, Manling Li, Xingyao Wang, Yi Fung, Charles Yu, Joel Tetreault, Eduard Hovy, and Heng Ji. 2023 b . https://doi.org/10.18653/v1/2023.findings-emnlp.799 Defining a new NLP playground . In Findings of the Association for Computational Lin...

    2023 doi

  9. [17]

    Adam Liska, Tomas Kocisky, Elena Gribovskaya, Tayfun Terzi, Eren Sezener, Devang Agrawal, D’Autume Cyprien De Masson, Tim Scholtes, Manzil Zaheer, Susannah Young, et al. 2022. Streamingqa: A benchmark for adaptation to new knowledge over time in question answering models. In I...

    2022

  10. [18]

    Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia Tsvetkov, Yejin Choi, and Noah A. Smith. 2024 a . https://openreview.net/forum?id=dribhnhm1i Tuning language models by proxy . In First Conference on Language Modeling

    2024

  11. [19]

    Smith, and Yejin Choi

    Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. 2021. https://doi.org/10.18653/v1/2021.acl-long.522 DE xperts: Decoding-time controlled text generation with experts and anti-experts . In Proceedings of the 59th Annual ...

    2021 doi

  12. [20]

    Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. 2024 b . https://openreview.net/forum?id=u2vAyMeLMm Infini-gram: Scaling unbounded n-gram language models to a trillion tokens . In First Conference on Language Modeling

    2024

  13. [21]

    Kelvin Luu, Daniel Khashabi, Suchin Gururangan, Karishma Mandyam, and Noah A. Smith. 2022. https://doi.org/10.18653/v1/2022.naacl-main.435 Time waits for no one! analysis and challenges of temporal misalignment . In Proceedings of the 2022 Conference of the North American Chap...

    2022 doi

  14. [22]

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. https://doi.org/10.18653/v1/2023.acl-long.546 When not to trust language models: Investigating effectiveness of parametric and non-parametric memories . In Proceedings of the 6...

    2023 doi

  15. [23]

    Sara Vera Marjanovic, Haeun Yu, Pepa Atanasova, Maria Maistro, Christina Lioma, and Isabelle Augenstein. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.838 DYNAMICQA : Tracing internal knowledge conflicts in language models . In Findings of the Association for Computati...

    2024 doi

  16. [24]

    Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. 2023. https://openreview.net/forum?id=MkbcAHIYgyS Mass-editing memory in a transformer . In The Eleventh International Conference on Learning Representations

    2023

  17. [25]

    Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, KaShun Shum, Randy Zhong, Juntong Song, and Tong Zhang. 2024. https://doi.org/10.18653/v1/2024.acl-long.585 RAGT ruth: A hallucination corpus for developing trustworthy retrieval-augmented language models . In Proceedings of the 62n...

    2024 doi

  18. [26]

    OpenAI. 2024. Introducing gpt-4o: our fastest and most affordable flagship model. https://platform.openai.com/docs/guides/vision. Accessed: 2024-05-26

    2024

  19. [27]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, an...

    2022

  20. [28]

    Fabio Petroni, Tim Rockt \"a schel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. https://doi.org/10.18653/v1/D19-1250 Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language P...

    2019 doi

  21. [29]

    Cheng Qian, Xinran Zhao, and Tongshuang Wu. 2024. https://openreview.net/forum?id=Pvn1dKreZW ''merge conflicts!''' exploring the impacts of external knowledge distractors to parametric knowledge graphs . In First Conference on Language Modeling

    2024

  22. [30]

    Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. https://doi.org/10.1162/tacl_a_00605 In-context retrieval-augmented language models . Transactions of the Association for Computational Linguistics, 11:1316--1331

    2023 doi

  23. [31]

    Dan Shi, Renren Jin, Tianhao Shen, Weilong Dong, Xinwei Wu, and Deyi Xiong. 2024 a . https://openreview.net/forum?id=ZfXRAqbBKX IRCAN : Mitigating knowledge conflicts in LLM generation via identifying and reweighting context-aware neurons . In The Thirty-eighth Annual Conferen...

    2024

  24. [32]

    Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Wen-tau Yih. 2024 b . https://doi.org/10.18653/v1/2024.naacl-short.69 Trusting your evidence: Hallucinate less with context-aware decoding . In Proceedings of the 2024 Conference of the North America...

    2024 doi

  25. [33]

    Smith, Luke Zettlemoyer, Wen tau Yih, and Mike Lewis

    Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Xi Victoria Lin, Noah A. Smith, Luke Zettlemoyer, Wen tau Yih, and Mike Lewis. 2024 c . https://openreview.net/forum?id=LXVswInHOo In-context pretraining: Language modeling beyond document boundaries . In The Twe...

    2024

  26. [34]

    Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2024 d . https://doi.org/10.18653/v1/2024.naacl-long.463 REPLUG : Retrieval-augmented black-box language models . In Proceedings of the 2024 Conference of the ...

    2024 doi

  27. [35]

    Hexiang Tan, Fei Sun, Wanli Yang, Yuanzhuo Wang, Qi Cao, and Xueqi Cheng. 2024 a . https://doi.org/10.18653/v1/2024.acl-long.337 Blinded by generated contexts: How language models merge generated and retrieved contexts when knowledge conflicts? In Proceedings of the 62nd Annua...

    2024 doi

  28. [36]

    Xue Tan, Hao Luan, Mingyu Luo, Xiaoyan Sun, Ping Chen, and Jun Dai. 2024 b . Knowledge database or poison base? detecting rag poisoning attack through llm activations. arXiv preprint arXiv:2411.18948

    2024 arXiv

  29. [37]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118

    2024 arXiv

  30. [38]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. https://doi.org/10.1162/tacl_a_00475 ♫ M u S i Q ue: Multihop questions via single-hop question composition . Transactions of the Association for Computational Linguistics, 10:539--554

    2022 doi

  31. [39]

    Han Wang, Archiki Prasad, Elias Stengel-Eskin, and Mohit Bansal. 2024 a . Adacad: Adaptively decoding to balance conflicts between contextual and parametric knowledge. arXiv preprint arXiv:2409.07394

    2024 arXiv

  32. [40]

    Yike Wang, Shangbin Feng, Heng Wang, Weijia Shi, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. 2024 b . https://openreview.net/forum?id=ptvV5HGTNN Resolving knowledge conflicts in large language models . In First Conference on Language Modeling

    2024

  33. [41]

    Kevin Wu, Eric Wu, and James Zou. 2024 a . https://api.semanticscholar.org/CorpusID:269157310 Clasheval: Quantifying the tug-of-war between an llm's internal prior and external evidence . In Neural Information Processing Systems

    2024

  34. [42]

    Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D Manning, and Christopher Potts. 2024 b . https://openreview.net/forum?id=fykjplMc0V Re FT : Representation finetuning for language models . In The Thirty-eighth Annual Conference on Neural Inf...

    2024

  35. [43]

    Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. 2023. https://api.semanticscholar.org/CorpusID:263610324 Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts . In International Conference on Learning Representations

    2023

  36. [44]

    Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. 2024. https://openreview.net/forum?id=auKAUJZMO6 Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts . In The Twelfth International Conference on Learning Representations

    2024

  37. [45]

    Rongwu Xu, Zehan Qi, Zhijiang Guo, Cunxiang Wang, Hongru Wang, Yue Zhang, and Wei Xu. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.486 Knowledge conflicts for LLM s: A survey . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, page...

    2024 doi

  38. [46]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  39. [47]

    Xiaowei Yuan, Zhao Yang, Yequan Wang, Shengping Liu, Jun Zhao, and Kang Liu. 2024. https://doi.org/10.18653/v1/2024.findings-acl.234 Discerning and resolving knowledge conflicts through adaptive decoding with contextual information-entropy constraint . In Findings of the Assoc...

    2024 doi

  40. [48]

    Shenglai Zeng, Jiankun Zhang, Bingheng Li, Yuping Lin, Tianqi Zheng, Dante Everaert, Hanqing Lu, Hui Liu, Hui Liu, Yue Xing, Monica Xiao Cheng, and Jiliang Tang. 2025. https://aclanthology.org/2025.naacl-long.151/ Towards knowledge checking in retrieval-augmented generation: A...

    2025

  41. [49]

    Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. https://doi.org/10.18653/v1/2020.acl-demos.30 DIALOGPT : Large-scale generative pre-training for conversational response generation . In Proceeding...

    2020 doi

  42. [50]

    Yu Zhao, Alessio Devoto, Giwon Hong, Xiaotang Du, Aryo Pradipta Gema, Hongru Wang, Xuanli He, Kam-Fai Wong, and Pasquale Minervini. 2024. https://arxiv.org/abs/2410.15999 Steering knowledge selection behaviours in llms via sae-based representation engineering . Preprint, arXiv...

    2024 arXiv

  43. [51]

    Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.968 Context-faithful prompting for large language models . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14544--14556, Singapore. As...

    2023 doi

  44. [52]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  45. [53]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.