Pith. sign in

REVIEW 5 major objections 6 minor 35 references

Towards a Principled Evaluation of Knowledge Editors

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Knowledge-editor rankings flip when the scoring method or edit batch size changes; benchmark winners can be artifacts of evaluation design.

desk verdict A useful meta-evaluation showing that scoring protocol changes editor rankings, with a large MEMIT/CounterFact effect that carries the claim; thinner on error bars and small reversals. read the letter →

arxiv 2507.05937 v1 pith:MEG7ACBF submitted 2025-07-08 cs.CL

Sebastian Pohl , Max Ploner , Alan Akbik This is my paper
classification cs.CL
keywords knowledgeeditingmodelevaluationmethodologyCounterFactMEMITstringmatchingeditbatchsizebenchmarkrobustness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the choice of evaluation design—the scoring metric, the evaluation methodology, and the number of edits applied in a batch—can change which knowledge editor is ranked best. The authors test four editors (MEMIT, LoRA, an in-context editor, and a retrieval-based context editor) on four benchmarks (zsRE, CounterFact, MQuAKE, RippleEdits) under three scoring schemes: greedy token matching, multiple-choice ranking, and exact string matching in generated text. They report order reversals, most notably MEMIT leading CounterFact's multiple-choice score but trailing under generation-based scoring, and they show that exact string matching produces false positives as generated text grows longer. They also add general language-understanding tasks to measure editing side effects and show that edit batch size affects both editing accuracy and how much the editor disrupts the model. A careful reader would care because benchmark rankings are only meaningful if they are robust to these design choices.

What carries the argument

The machinery is a comparative evaluation matrix: four editors (parameter-updating MEMIT, LoRA fine-tuning, in-context prepending, and a context-retriever that prepends the four nearest edits) crossed with four knowledge editing datasets (zsRE, CounterFact, MQuAKE, RippleEdits) and three scoring methods (argmax token accuracy, multiple-choice ranking by sequence log-likelihood, and exact string matching of generated aliases). The paper adds two further instruments: sweeping the edit batch size (1, 16, 64, 512, 2048) on 2048-example subsamples, and a manual rating exercise on 200 generated answers that distinguishes true positives from false positives and false negatives produced by the string-matcher, plus an LLM-as-a-judge comparison.

What would settle it

Re-run the full evaluation with five random 2048-example subsamples and seeds for each editor, dataset, and scoring method. If argmax, multiple-choice, and generation all yield the same editor ordering in every subsample (e.g., MEMIT never drops below the other editors on CounterFact generation), the paper's claim that method choice reverses rankings would be directly contradicted; conversely, overlapping confidence intervals for the current single-run scores would show the reversals are not established.

Watch

Extended reading notes

Core claim

The central claim is that reported rankings of knowledge editors are not stable under evaluation design choices. Using the three scoring methods (argmax greedy decoding, multiple-choice sequence likelihood ranking, and exact string matching in generated text), the paper shows that on CounterFact, MEMIT scores 0.866 by multiple choice but only 0.462 by generation on GPT-J, falling behind the in-context and context-retriever editors; LoRA similarly moves up or down by method. The paper further claims that the string-matching method, preferred by newer datasets, produces false positives: in a manual review of 200 generated answers, 150 were 'late success' cases where a match appeared only in the second half of the generated text, and longer generation lengths increased false positives, particularly for the context-retriever. Finally, varying the edit batch size from 1 to 2048 shows that in-context editors lose accuracy when edits overflow the context window, while MEMIT is more robust; on the general language-understanding tasks lambada and hellaswag, MEMIT is least disruptive and LoRA most disruptive, with LoRA perplexity reaching millions on lambada at large batch sizes.

Load-bearing premise

The central ranking comparisons treat one run of each editor on one 2048-example subsample of each dataset as a stable accuracy estimate, so the reported rank reversals could be comparable to run-to-run or sampling noise.

Editorial extensions

If this is right

  • Existing benchmark comparisons built on a single scoring method, especially CounterFact's multiple-choice, should be re-examined, since the method, not just the editor, can determine the winner.
  • Generation-based evaluation should report results across generation lengths, because longer outputs inflate accuracy through false positive string matches.
  • Knowledge editing studies should sweep edit batch sizes rather than fixing one, since both editing accuracy and side effects shift with batch size.
  • Side-effect monitoring on tasks like lambada and hellaswag should be standard in editing papers; they differentiated editors where most other control tasks did not.
  • LLM-as-a-judge scoring is a promising replacement for exact string matching and deserves larger-scale validation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If evaluation design choices flip rankings at the scales tested here, then published claims that one editing method is 'better' than another may be method-specific; a fair comparison should report a matrix of metrics and batch sizes rather than a single leaderboard number.
  • The false-positive pattern suggests the generate metric's generation-length hyperparameter is effectively a free parameter that can be tuned to favor one editor; benchmark designers should either fix it by task difficulty or use human or LLM verification.
  • A testable extension: run the same four editors on the same datasets with several random subsamples and seeds to quantify whether the rank reversals (e.g., MEMIT 0.866 versus LoRA 0.688 on CounterFact multiple-choice) exceed sampling noise; the paper's single-run tables do not settle that.
  • In-context and retrieval editors gaining at large batch sizes on RippleEdits hints that retrieval-based editing can degrade gracefully toward the unedited model, which might be reframed as a feature (less disruption) rather than only a loss of editing efficacy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper investigates whether the evaluation methodology used for knowledge editing benchmarks materially changes the measured ranking of editors. It compares four editors (MEMIT, LoRA, in-context, context-retriever) on four datasets (zsRE, CounterFact, MQuAKE, RippleEdits) under three scoring methods (argmax, multiple-choice, generate-with-string-matching), on GPT-J and GPT2-XL. It further evaluates editors across edit batch sizes and measures side effects on LM Evaluation Harness tasks. The central claims are that evaluation design choices can reverse editor rankings, that CounterFact's multiple-choice scoring favors MEMIT, that string-matching evaluation produces false positives at longer generation lengths, and that MEMIT tends to be less disruptive on general language tasks.

Significance. If the empirical claims hold, the paper makes a useful methodological contribution by demonstrating that knowledge-editing conclusions are not metric-independent and by quantifying the fragility of exact string matching. The release of a unified evaluation framework and the integration with LM Evaluation Harness are practical strengths, and the manual audit of string-matching false positives is a valuable, falsifiable contribution. The main limitation is that the central ranking comparisons are presented without measures of uncertainty, so the paper currently supports an existential claim (large effects such as MEMIT's CounterFact reversal are visible) more strongly than its detailed comparative narrative.

major comments (5)
  1. [§3.1, §3.2, Tables 1 and 2] All accuracy numbers come from a single random 2048-example subsample per dataset and a single run, with no seeds, confidence intervals, or repeated subsamples reported. Because the paper's central claim is that evaluation choices change editor rankings, the rank-reversal comparisons need uncertainty quantification. For example, the MQuAKE argmax reversal on GPT-J between in-context (0.345) and context-retriever (0.330) is a difference of 0.015, which is within binomial sampling noise for n=2048, whereas the CounterFact MEMIT reversal (0.866 multiple-choice versus 0.462 generate) is enormous. Without error bars or paired bootstrap intervals, a reader cannot determine which of the claimed reversals are method effects and which are sampling artifacts.
  2. [§3.1, §4.2] LoRA hyperparameters were tuned specifically for edit batch size 16, but LoRA is then compared at other batch sizes and on side-effect tasks with the same hyperparameters. The paper acknowledges this in §3.1 and §4.2, but the conclusions that LoRA is 'the most disruptive editor' on lambada/hellaswag and that its batch-size performance drops are confounded by hyperparameter mismatch. A fair comparison requires per-batch-size tuning or a sensitivity analysis showing that the conclusions are robust to the tuning choice.
  3. [§3.2, Figures 5 and 6] The manual false-positive analysis is based on only 200 examples, with 150 drawn from the 'late success' class and 50 from the early-success class, and the paper reports no inter-rater agreement or sampling variance. The claim that the context-retriever benefits from longer generation because of a larger false positive rate is based on small counts in a nonrandom subsample and lacks uncertainty quantification; the authors should report confidence intervals and, ideally, multiple raters.
  4. [§3.2, Table 3] The LLM-as-a-judge comparison is presented as promising because Qwen2.5-32B 'slightly outperformed' exact matching, but with 200 examples and no significance test the differences are not shown to be meaningful; for MQuAKE, exact match and Qwen both score 0.897. The claims about judge model superiority need confidence intervals or a paired test.
  5. [§4.2, Tables 6 and 7] The conclusion that MEMIT is 'the least disruptive' editor on general language tasks rests on small differences in lambada and hellaswag, while most other tasks show no clear trends. As with the knowledge-editing tables, these numbers are single-run values with no repeated subsamples, so the 'least disruptive' claim is not statistically quantified.
minor comments (6)
  1. [§2.1 footnote] The footnotes refer to resources as 'model editing' and 'paper results' rather than providing actual URLs or repository identifiers; these placeholders should be replaced before publication.
  2. [§2.2] There is a typo: 'Contreiver' should be 'Contriever' (the model of Izacard et al., 2022).
  3. [§3.1] The description of example selection for manual evaluation says an equal number of examples was drawn from each dataset for both classes, but the final counts are 150 late-success and 50 early-success examples; the sampling scheme should be stated more precisely.
  4. [§3.2] The sentence 'LoRA out performes MEMIT' contains a typo ('out performes' should be 'outperforms').
  5. [§4.2] The phrase 'the context-editor' should be 'the context-retriever' when discussing the effect of increasing batch size on retrieval-based editing.
  6. [General] Figure captions are very compressed; for example, Figure 1 does not state that all curves correspond to a single subsample, and Figures 5 and 6 would benefit from stating the sample sizes and selection criteria in the captions rather than only in the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are direct measurements against external knowledge-editing datasets, LM Evaluation Harness tasks, and human-rater ground truth, with no fitted quantity reused to define any conclusion.

full rationale

The paper is an empirical meta-evaluation rather than a derivation chain. Its central claims — that evaluation methodology, metric choice, and edit batch size change editor rankings, and that exact string matching produces false positives — are supported by accuracy measurements on four external datasets (zsRE, CounterFact, MQuAKE, RippleEdits) whose targets are fixed by the dataset creators, and by manual annotations from human raters that constitute an external ground truth. No parameter is fitted and then renamed as a prediction: the only tuned hyperparameters are LoRA's, tuned once at batch size 16, and the paper explicitly discloses this confound (Section 3.2, 'Unlike for other editors, however, we specifically tuned the LoRA hyper-parameters to the edit batch size of 16'), so the rank-comparison evidence does not reduce to the tuning choice. The false-positive finding is not self-definitional: correctness is judged by human raters ('whether the first answer given by the model correctly answers the prompt'), an independent criterion from the exact-substring matcher being critiqued, and the oversampling of late-success examples is corrected by projection to true dataset proportions (Figure 6). The reference list contains no self-citations by Pohl, Ploner, or Akbik, so there is no self-citation chain carrying a load-bearing premise. The acknowledged Limitations section ('experiments... are limited to a subset of published Model Editors and are conducted only on two small, less powerful Language Models') qualifies generality but does not indicate circularity. Weaknesses such as single-run, single-subsample tables without error bars are statistical-robustness concerns that belong to correctness risk, not to circularity, since the observed conclusions are point measurements rather than quantities defined by the paper's own assumptions.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper is empirical; its 'free parameters' are evaluation and hyperparameter choices rather than fitted physical constants. The most consequential choices are LoRA hyperparameters tuned to batch size 16, the retrieval count of 4, and the generated-token length. The claims rest on domain assumptions that the selected datasets, models, and control tasks represent knowledge editing. No invented entities are introduced.

free parameters (3)
  • LoRA hyperparameters = rank 8, alpha 32, 20 epochs, learning rate 5e-3 for GPT2-XL and 1e-3 for GPT-J
    Tuned by briefly exploring a range to optimize performance on edit batch size 16; this favors LoRA at that batch size and affects later comparisons at other batch sizes (Section 3.1, Section 3.2).
  • Context-retriever k = 4 nearest neighbors
    Chosen because MQuAKE examples depend on at most four edits for 4-hop reasoning, not tuned per dataset (Section 2.2).
  • Generation length = 20 tokens for headline tables, 64-token sweep, 24 tokens for manual and LLM judge comparison
    The generate method's false-positive rate depends on this hand-chosen length; the paper shows accuracy changes with length (Section 3.1, Section 3.2, Table 3).
assumptions (4)
  • domain assumption The four benchmarks (zsre, CounterFact, MQuAKE, RippleEdits) validly operationalize knowledge editing success.
    All ranking and false-positive claims are measured through these datasets; their validity is taken from prior work without re-validation (Section 2.1).
  • domain assumption GPT-J and GPT2-XL are representative models for knowledge editing.
    They are chosen because MEMIT hyperparameters are published only for these models; the authors acknowledge that results may not transfer to larger instruction-tuned models (Section 3.1, Limitations).
  • domain assumption lambada and hellaswag are suitable controls for post-edit capability loss.
    The side-effect conclusions rely mainly on these two tasks; most other control tasks show no clear trend (Section 4.2).
  • domain assumption Human ratings of generated answers are correct ground truth.
    The false-positive analysis and LLM-as-a-judge comparison use these ratings as ground truth, but no inter-annotator agreement is reported (Section 3.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards a Principled Evaluation of Knowledge Editors." pith.science (2026). https://pith.science/paper/MEG7ACBF

@misc{pith2026250705937,
  author       = {Pith},
  title        = {Pith review of: Towards a Principled Evaluation of Knowledge Editors},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MEG7ACBF}},
  note         = {Machine review of arXiv:2507.05937}
}
read the original abstract

Model editing has been gaining increasing attention over the past few years. For Knowledge Editing in particular, more challenging evaluation datasets have recently been released. These datasets use different methodologies to score the success of editors. Yet, it remains under-explored how robust these methodologies are and whether they unfairly favor some editors. Moreover, the disruptive impact of these editors on overall model capabilities remains a constant blind spot. We address both of these problems and show that choosing different metrics and evaluation methodologies as well as different edit batch sizes can lead to a different ranking of knowledge editors. Crucially we demonstrate this effect also on general language understanding tasks evaluated alongside the knowledge editing tasks. Further we include a manual assessment of the string matching based evaluation method for knowledge editing that is favored by recently released datasets, revealing a tendency to produce false positive matches.

Figures

Figures reproduced from arXiv: 2507.05937 by the authors.

Figure 1
Figure 1. Accuracies for different Model Editors and [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Average number of unique n ≤ 5-grams per generated 64 answer tokens for different datasets and editors. Alcide De Gasperi worked in the Italian Parliament for over 30 years. He was a mem￾ber of the Christian Democratic Party and was Prime Minister of Italy from 1948 to 1953. He was also Presi￾dent of the European Parliament from 1958 to 1959. Alcide De Gasperi was born in Rome on April 26, 1881. He was the [PITH_FU… view at source ↗
Figure 3
Figure 3. Prompt (in italcis) and generated answer (matched substring marked in bold). remains unclear. Despite the naïve matching scheme (exact sub￾strings), the rate of false negatives is relatively small (assuming at least 10 tokens are generated). At least partially, this may be due to the relatively aggressive matching. For example, [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Prompt (in italcis) and generated answer (matched substring marked in bold). The initial answer (“Lithium”) may be considered correct but is ignored by the exact matching al￾gorithm since it is capitalized, but the expected answer is not. Only later is the answer deeme…
Figure 5
Figure 5. Figure 5: True Positives, True Negatives, False Positives, and False Negatives for each Editor, Dataset, and Generate [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: True Positives, True Negatives, False Positives and False Negatives for each dataset and generate length [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Accuracies on Knowledge Editing datasets for [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: LM Evaluation Harness results for selected [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: True Positives, True Negatives, False Positives and False Negatives for each editor, dataset and generate [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

35 extracted references · 3 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Costa-jussà

    Magdalena Biesialska, Katarzyna Biesialska, and Marta R. Costa-jussà. 2020. https://doi.org/10.18653/v1/2020.coling-main.574 Continual lifelong learning in natural language processing: A survey . In Proceedings of the 28th International Conference on Computational Linguistics. International Committee on Computational Linguistics

    doi:10.18653/v1/2020.coling-main.574 2020

  4. [4]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    arXiv 2020

  5. [5]

    Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. https://arxiv.org/abs/2104.08164 Editing factual knowledge in language models . Preprint, arXiv:2104.08164

    arXiv 2021

  6. [6]

    Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. 2023. https://arxiv.org/abs/2307.12976 Evaluating the ripple effects of knowledge editing in language models . Preprint, arXiv:2307.12976

    arXiv 2023

  7. [7]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2023. https...

    doi:10.5281/zenodo.10256836 2023

  8. [8]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. https://arxiv.org/abs/2312.10997 Retrieval-augmented generation for large language models: A survey . Preprint, arXiv:2312.10997

    arXiv 2024

Show all 35 references
  1. [9]

    Yihuai Hong, Lei Yu, Haiqin Yang, Shauli Ravfogel, and Mor Geva. 2024. https://arxiv.org/abs/2406.11614 Intrinsic evaluation of unlearning using parametric knowledge traces . Preprint, arXiv:2406.11614

    2024 arXiv

  2. [10]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. https://arxiv.org/abs/2106.09685 Lora: Low-rank adaptation of large language models . Preprint, arXiv:2106.09685

    2021 arXiv

  3. [11]

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023. https://arxiv.org/abs/2212.04089 Editing models with task arithmetic . Preprint, arXiv:2212.04089

    2023 arXiv

  4. [12]

    Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. https://arxiv.org/abs/2112.09118 Unsupervised dense information retrieval with contrastive learning . Preprint, arXiv:2112.09118

    2022 arXiv

  5. [13]

    Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. 2022. https://arxiv.org/abs/2210.01504 Knowledge unlearning for mitigating privacy risks in language models . Preprint, arXiv:2210.01504

    2022 arXiv

  6. [14]

    Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017. https://doi.org/10.1073/pnas.1...

    2017 doi

  7. [15]

    Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. https://doi.org/10.18653/v1/K17-1034 Zero-shot relation extraction via reading comprehension . In Proceedings of the 21st Conference on Computational Natural Language Learning ( C o NLL 2017) , pages 333--342, Va...

    2017 doi

  8. [16]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2021. https://arxiv.org/abs/2005.11401 Retrieval-augmented generation for knowledge-int...

    2021 arXiv

  9. [17]

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2023 a . https://arxiv.org/abs/2202.05262 Locating and editing factual associations in gpt . Preprint, arXiv:2202.05262

    2023 arXiv

  10. [18]

    Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2023 b . https://arxiv.org/abs/2210.07229 Mass-editing memory in a transformer . Preprint, arXiv:2210.07229

    2023 arXiv

  11. [19]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. https://arxiv.org/abs/1609.07843 Pointer sentinel mixture models . Preprint, arXiv:1609.07843

    2016 arXiv

  12. [20]

    Mistral. 2024. Mistral-7b-instruct-v0.3. https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3. Accessed: 2025-06-15

    2024

  13. [21]

    Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. 2022 a . https://arxiv.org/abs/2110.11309 Fast model editing at scale . Preprint, arXiv:2110.11309

    2022 arXiv

  14. [22]

    Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. 2022 b . https://proceedings.mlr.press/v162/mitchell22a.html Memory-based model editing at scale . In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proc...

    2022

  15. [23]

    Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI : A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computat...

    2020

  16. [24]

    Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. https://arxiv.org/abs/1606.06031 The lambada dataset: Word prediction requiring a broad discourse context . Prepri...

    2016 arXiv

  17. [25]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf

    2019

  18. [26]

    Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitriy Pyrkin, Sergei Popov, and Artem Babenko. 2020. https://arxiv.org/abs/2004.00345 Editable neural networks . Preprint, arXiv:2004.00345

    2020 arXiv

  19. [27]

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. https://doi.org/10.18653/v1/N19-1421 C ommonsense QA : A question answering challenge targeting commonsense knowledge . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associa...

    2019 doi

  20. [28]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. https://doi.org/10.18653/v1/W18-5446 GLUE : A multi-task benchmark and analysis platform for natural language understanding . In Proceedings of the 2018 EMNLP Workshop B lackbox NLP : A...

    2018 doi

  21. [29]

    Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model . https://github.com/kingoflolz/mesh-transformer-jax

    2021

  22. [30]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng...

    2024 arXiv

  23. [31]

    Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. 2023. https://arxiv.org/abs/2305.13172 Editing large language models: Problems, methods, and opportunities . Preprint, arXiv:2305.13172

    2023 arXiv

  24. [32]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

    2019

  25. [33]

    Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, Siyuan Cheng, Ziwen Xu, Xin Xu, Jia-Chen Gu, Yong Jiang, Pengjun Xie, Fei Huang, Lei Liang, Zhiqiang Zhang, Xiaowei Zhu, Jun Zhou, and Huajun Chen. ...

    2024 arXiv

  26. [34]

    Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. 2023. https://arxiv.org/abs/2305.12740 Can we edit factual knowledge by in-context learning? Preprint, arXiv:2305.12740

    2023 arXiv

  27. [35]

    Manning, Christopher Potts, and Danqi Chen

    Zexuan Zhong, Zhengxuan Wu, Christopher D. Manning, Christopher Potts, and Danqi Chen. 2024. https://arxiv.org/abs/2305.14795 Mquake: Assessing knowledge editing in language models via multi-hop questions . Preprint, arXiv:2305.14795

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.