Pith. sign in

REVIEW 4 major objections 5 minor 19 references

Quantifying Cross-Modality Memorization in Vision-Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Vision-language models transfer facts across modalities, but the transfer is uneven and asymmetric.

desk verdict A solid empirical study with a new synthetic benchmark, but the headline asymmetry claim is confounded by uncalibrated probe difficulty and single-run evidence. read the letter →

arxiv 2506.05198 v1 pith:JPGXYXKP submitted 2025-06-05 cs.CV cs.LG

Yuxin Wen , Yangsibo Huang , Tom Goldstein , Ravi Kumar , Badih Ghazi , Chiyuan Zhang This is my paper
classification cs.CVcs.LG
keywords cross-modalmemorizationvision-languagemodelsfactualknowledgetransfermembershipinferencemachineunlearningmulti-hopreasoningsyntheticpersonadatasetprivacyleakage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper studies how vision-language models (VLMs) memorize factual knowledge when trained on only one modality — either a photo of a person or a textual description — and then are tested on the other. Using a synthetic dataset of 100 personas with paired images and descriptions, it finds that facts learned in one modality do transfer to the other, but accuracy in the target modality lags behind accuracy in the source modality. The gap is asymmetric: training on images transfers to text more effectively than training on text transfers to images, and this asymmetry persists in larger models, in machine unlearning attempts, and in multi-hop reasoning. The paper also shows that VLMs unintentionally memorize irrelevant visual private information, such as a 'favorite number' overlaid on an image, detectable through membership inference. These findings matter for building trustworthy multimodal assistants and for privacy protection when models are trained on personal photos and messages.

What carries the argument

The central object is a synthetic persona dataset, a set of 'canaries' with full experimental control: 100 unique personas, each with a fixed name, a favorite number (acting as surrogate PII), a set of visual attributes, 100 training images and 100 training textual descriptions plus one held-out image and description for testing. The evaluation protocol trains a VLM on one modality via LoRA fine-tuning, then measures name-recall accuracy on held-out inputs of both modalities, comparing source accuracy with target accuracy to quantify the cross-modal transfer gap. For the privacy experiment, the machinery is a membership-inference attack: for each test image, the model's loss is computed with the ground-truth favorite number overlaid versus a random one, and the AUC of the resulting ROC curve measures unintentional memorization.

What would settle it

Run the same training and evaluation protocol with a matched-difficulty control: have human annotators (or a strong probe model) achieve equal accuracy on name-from-image and name-from-description for the same persona set, and if the VLM still shows an asymmetric gap then the asymmetry is real. Conversely, if the probes are equally difficult and a VLM trained on a single image variant achieves equal source and target accuracy at any epoch, the central claim of an irreducible cross-modal gap would be disproven.

Watch

Extended reading notes

Core claim

The paper's central claim is that cross-modal factual memorization in VLMs is real but incomplete and systematically imbalanced. In controlled fine-tuning experiments, a model trained on descriptions of a person can often name them from a held-out description, but struggles more when shown a photo; a model trained on photos shows better transfer to text. This asymmetry — image-to-text transfer stronger than text-to-image — holds across model sizes within the Gemma-3 family, meaning scaling does not fix the transfer gap. The paper further shows that cross-modal unlearning is incomplete: removing a fact in the modality it was learned leaves the other modality largely intact. In multi-hop settings, models trained on description-to-name and name-to-favorite-number fail to answer description-to-favorite-number, a cross-modal variant of the 'two-hop curse.' Finally, by injecting a three-digit 'favorite number' into training images, the paper demonstrates unintentional memorization of task-irrelevant visual PII, with membership-inference AUC reaching about 70% after 100 epochs even though direct extraction succeeds almost never.

Load-bearing premise

The claim that the accuracy gap reflects a property of cross-modal memorization assumes that the image and text test sets are equally sensitive probes of the same underlying fact; if recognizing a person from a photo is intrinsically easier or harder than from a description, the measured asymmetry could be an artifact of task difficulty rather than a genuine transfer asymmetry.

Editorial extensions

If this is right

  • Applications like personal assistants that learn from a user's photos or text messages will be less reliable when asked to recall facts in the modality they were not trained on, and this shortfall cannot be cured simply by scaling the model.
  • Privacy-removal procedures (machine unlearning) must be applied to every modality separately; unlearning a fact from text does not remove it from the model's visual memory of the same person.
  • Augmenting training data with in-distribution synthetic image-text pairs improves cross-modal transfer, whereas out-of-distribution data (e.g., generic COCO captioning) does not, suggesting that data diversity must match the target distribution to help.
  • VLMs can memorize task-irrelevant visual PII, so data filtering and redaction pipelines should scrub incidental sensitive content from images even when that content is unrelated to the training objective.
  • The cross-modal multi-hop curse implies that compositional knowledge (A→B, B→C) does not automatically yield A→C across modalities, which limits reasoning in multimodal knowledge-intensive tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The asymmetric transfer (image→text better than text→image) might reflect an asymmetry in information content: a photo is a single concrete rendering, whereas a textual description is an abstract, lossy summary; a controlled experiment equalizing information content would separate this from a genuinely modality-specific learning bias.
  • The membership-inference signal at 70% AUC, despite near-zero direct extraction, suggests that current alignment or safety training does not prevent latent memorization of visual PII; testable extensions include probing whether the signal survives different unlearning methods or stronger data augmentation.
  • The finding that OOD augmentation fails to improve transfer implies that generic web-scale multimodal pretraining may not automatically make downstream fine-tuning more cross-modally robust; future work could test whether mixing on-distribution synthetic pairs at scale closes the gap.
  • The multi-hop failure being more severe for description-trained models hints that image-based training may produce more disentangled or visual-anchored representations; a representational similarity analysis between the two training regimes could directly test this hypothesis.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces a synthetic persona dataset of 100 fictional individuals, each with 100 training image variants, 100 training description variants, and one held-out test image and test description. The authors fine-tune Gemma-3 vision-language models with LoRA on either the image or the description alone, then evaluate name-recall accuracy on held-out inputs in both modalities. They report that factual knowledge transfers across modalities but with a source-target accuracy gap, that the gap is asymmetric (image-to-text transfer is more effective than text-to-image), and that the gap persists across model sizes, a re-labeling-based unlearning procedure, and a multi-hop favorite-number task. They also propose an in-distribution image-caption augmentation that improves transferability, and they report a membership-inference experiment showing that a VLM can unintentionally memorize a task-irrelevant favorite number overlaid on training images.

Significance. If the empirical claims hold, this is a useful and timely contribution to the study of memorization and privacy in multimodal models. The controlled canary-style dataset, the held-out evaluation protocol, and the systematic variation of training modality, model size, and training diversity are genuine strengths. The paper makes falsifiable predictions (e.g., the direction of the cross-modal gap, the ineffectiveness of OOD augmentation, the presence of membership-inference signal for visual PII) that other groups can test. The main weakness is that the central asymmetry claim is not yet identified because the two modality probes are not calibrated against each other.

major comments (4)
  1. [§3.3, Figures 2–4] The paper's central claim that cross-modal transfer is asymmetric (image-to-text more effective than text-to-image) rests on comparing the target accuracy of a description-trained model on held-out images (around 0.3 in Figure 3a) with the target accuracy of an image-trained model on held-out descriptions (around 0.4 in Figure 3b). These two probes are not calibrated against each other. Recognizing a name from a single photograph may be intrinsically harder than recognizing it from a verbose, deliberately disambiguating description, regardless of training modality. Without a human baseline, a matched-difficulty subset, or a mixed-modality training control that equates per-probe difficulty, the observed gap does not identify an asymmetry in transfer; it may simply reflect task difficulty. Because the asymmetry is invoked in the abstract, in the conclusion, and in Takeaways 1 and 3, this is a load-bearing confound. I ask the authors to add a probe-difficulty calibration (e.g., human accuracy on the same test items, or an internal control where a model trained on both modalities is measured on both probes) and to qualify the asymmetry claim until such calibration is provided.
  2. [§3.3–§4, Figures 2–8 and Table 1] All experimental curves and tables are reported as single-run point estimates. With 100 personas, the standard error of a reported 0.30 accuracy is about 4.6 percentage points, so a 10-point cross-modal gap is roughly two standard errors. Claims that the transferability rate is 'nearly unchanged' across model sizes (Takeaway 3) and that unlearning effects are modality-dependent (Takeaway 5) depend on the shapes and slopes of these curves. I request that the authors run at least three seeds per configuration, report means with error bars or confidence intervals, and, where they assert that a slope or rate is unchanged, provide a quantitative fit with confidence intervals rather than visual inspection.
  3. [§3.4, Figure 5] The ID augmentation is described as adding 'held-out synthetic persona images and descriptions' to the training set. This wording is ambiguous and potentially problematic: the dataset design reserves one distinct test image and one test description per persona, and if those reserved test items are used as augmentation, the mitigation result in Figure 5 is invalid by construction. Please clarify whether the augmented pairs come from the unused training-pool variants or from the actual test items, state how many pairs are added, and specify the auxiliary captioning objective used for these pairs. If the augmented data are from the training pool, the 'held-out' terminology should be changed.
  4. [§3.5, Figure 6] The unlearning procedure is not machine unlearning in the usual sense; it is continued training on the same inputs with completely re-assigned names. The conclusion that unlearning in a cross-modal setting does not transfer may be an artifact of this overwriting procedure rather than a property of forgetting: for example, the image model may require more epochs to overwrite a visually learned association, producing the observed asymmetry in relative accuracy drop. I recommend either adopting a more standard unlearning method (e.g., gradient ascent on the forget set, with the safeguards mentioned in the text) or explicitly re-framing the section as 'continued training with re-labeled data' and adjusting the strength of Takeaway 5 accordingly.
minor comments (5)
  1. [§3.1] There is a typo in the opening paragraph: 'or or their textual descriptions' should read 'or their textual descriptions'.
  2. [§3.2] Please report the chance accuracy for the name-recall task (1/100) as a reference point; the target accuracies around 0.3–0.4 are clearly above chance, but the reader cannot assess effect sizes without this baseline.
  3. [§3.3, Figure 4] The term 'transferability rate' is not defined quantitatively; the text infers a slope from a few points without any fitted line. Please either define the rate and report its estimated value with uncertainty, or use a different expression such as 'the accuracy curves remain roughly parallel.'
  4. [Table 1] The row and column structure of Table 1 is difficult to parse because the Training Data columns (Base Data, Multi-Hop Data) are not visually separated from the Test Type columns. Reformating with subheaders or separate panels for Desc⇒Name and Image⇒Name base data would greatly improve readability.
  5. [§4, Figure 8] There is a typo in the title of Figure 7 ('Unintential Memorization'), and the AUC of about 0.7, while above chance, is a modest signal; please state explicitly that this is a single-run estimate and report a confidence interval.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical measurements with direct interventions, not quantities derived from their own definitions or from load-bearing self-citation.

full rationale

This paper is an empirical measurement study, not a derivation from a fitted model. The central quantity—cross-modal memorization and transfer—is defined operationally: the model is trained on one modality (descriptions or images) and tested on held-out inputs in the other modality, with accuracy compared between source and target. Nothing in this procedure defines the target asymmetry in terms of the training inputs or evaluation protocol; the asymmetry is an observed difference in measured accuracies. The claimed gap could indeed reflect unequal probe difficulty rather than asymmetric transfer, but that is a threat to construct validity, not circularity. Under the review rules, uncalibrated probes are a correctness concern, not a self-referential derivation. No parameter is fitted to a subset of the data and then reported as a prediction: all reported numbers are direct accuracies or relative drops from controlled training runs. The unlearning experiment is explicitly a proxy ('we adopt a practically simpler workaround') and is not presented as a theoretically forced result. The multi-hop results are direct evaluations of held-out combinations. Citations to prior work by the authors (e.g., canary injection, diffusion memorization) are contextual and none is load-bearing: the paper does not invoke a uniqueness theorem or a prior 'cosh rescaling' ansatz to justify its conclusions. The membership inference result in Section 4 is a standard in-member/out-member loss comparison, not a renamed fit. Appendix E reports a null effect from prompt engineering and does not smuggle in any assumption. Overall, the derivation chain is self-contained against external benchmarks, and no step reduces by construction to its own inputs. Score 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper's central claim is qualitative and empirical; no fitted constants are used to derive it. The listed axioms are domain assumptions about the validity of synthetic data, LoRA fine-tuning, accuracy as a memorization measure, and membership inference as a memorization signal. The 'favorite number' is a surrogate for PII, not an invented physical entity; the synthetic persona dataset is an experimental artifact, not a postulated entity.

free parameters (4)
  • LoRA configuration = r=32, alpha=32, dropout=0.05
    Hand-chosen for all fine-tuning; affects model capacity to memorize and therefore the measured source/target accuracies.
  • Optimizer hyperparameters = lr=2e-4, batch=16
    Hand-chosen standard values; influence memorization dynamics but are not fitted to the observed gap.
  • Training epochs = 1,5,10,25,50,75,100
    Experimental schedule; the 'transferability rate' interpretation is based on how accuracy evolves over these epochs.
  • Dataset size = 100 personas, 100 variants per modality
    Chosen dataset scale; the test set is only 100 samples per modality, which limits precision of reported accuracies.
assumptions (4)
  • domain assumption Fine-tuning Gemma-3 with LoRA is a representative proxy for vision-language model adaptation in practical applications.
    All experiments use Gemma-3 with LoRA rather than full fine-tuning of a variety of architectures; Section 3.2.
  • domain assumption Synthetic personas generated by Imagen 3 and Gemini 2.0 capture the relevant structure of real user-specific multimodal knowledge.
    The central measurements are all made on this synthetic dataset; Section 3.1.
  • domain assumption Recognition accuracy on held-out images and descriptions is a valid operationalization of factual memorization.
    Used throughout Section 3.3; the model is asked for the person's name and accuracy is reported as memorization.
  • domain assumption A loss-based membership inference AUC indicates memorization of the injected number.
    Section 4 interprets increasing AUC as unintentional memorization without a control model trained without the overlay.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Quantifying Cross-Modality Memorization in Vision-Language Models." pith.science (2026). https://pith.science/paper/JPGXYXKP

@misc{pith2026250605198,
  author       = {Pith},
  title        = {Pith review of: Quantifying Cross-Modality Memorization in Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JPGXYXKP}},
  note         = {Machine review of arXiv:2506.05198}
}
read the original abstract

Understanding what and how neural networks memorize during training is crucial, both from the perspective of unintentional memorization of potentially sensitive information and from the standpoint of effective knowledge acquisition for real-world, knowledge-intensive tasks. While previous studies primarily investigate memorization within a single modality, such as text memorization in large language models or image memorization in diffusion models, unified multimodal models are becoming increasingly prevalent in practical applications. In this work, we focus on the unique characteristics of cross-modality memorization and conduct a systematic study centered on vision-language models. To facilitate controlled experiments, we first introduce a synthetic persona dataset comprising diverse synthetic person images and textual descriptions. We quantify factual knowledge memorization and cross-modal transferability by training models on a single modality and evaluating their performance in the other. Our results reveal that facts learned in one modality transfer to the other, but a significant gap exists between recalling information in the source and target modalities. Furthermore, we observe that this gap exists across various scenarios, including more capable models, machine unlearning, and the multi-hop case. At the end, we propose a baseline method to mitigate this challenge. We hope our study can inspire future research on developing more robust multimodal learning techniques to enhance cross-modal transferability.

Figures

Figures reproduced from arXiv: 2506.05198 by the authors.

Figure 1
Figure 1. Illustration of Our Data Generation Pipeline and Evaluation Pipeline. However, while prior work has investigated memorization within individual modalities—such as the regurgitation of specific training text sequences by LLMs [Carlini et al., 2019, 2021, 2022] and the reproduction of specific training images by diffusion models [Somepalli et al., 2023b, Carlini et al., 2023, Wen et al., 2024a]—these studies primarily… view at source ↗
Figure 2
Figure 2. Training with Single Description/Image. 0 20 40 60 80 100 Training Epoch 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Testing Type Description Image (a) Training on Descriptions 0 20 40 60 80 100 Training Epoch 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy (b) Training on Images 0.0 0.2 0.4 0.6 0.8 1.0 Source Accuracy 0.0 0.2 0.4 0.6 0.8 1.0 Target Accuracy Training Type Description Image (c) Cross-Modality Accuracy [PITH_FULL_IMAGE:figure… view at source ↗
Figure 3
Figure 3. Training with Image/Description Variants. 3.2 Experimental Setup We fine-tune the latest open-source vision-language model, Gemma-3-4b [Team et al., 2025], to conduct all our experiments. During fine-tuning, we utilize LoRA [Hu et al., 2022] with a rank of r = 32, a scaling factor of α = 32, and a dropout probability of 0.05. We use AdamW [Loshchilov and Hutter, 2017] with a learning rate of 2 × 10−4 and a batch siz… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Training with Different Model Sizes. Although cross-modal knowledge transfer occurs in both training scenarios, there remains a significant performance gap between the training modality and the testing modality. This gap is more evident in Figure 2c, where the plot com…
Figure 5
Figure 5. Figure 5: Training with Augmentation Mitigation. transferability, we train three models from the Gemma3 family: Gemma3-4B-IT, Gemma3-12B-IT, and Gemma3-27B-IT. As shown in [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Unlearning cross Modalities. 3.5 Cross-Modal Unlearning Machine unlearning [Bourtoule et al., 2021] has gained popularity as a method for removing specific user data without requiring the model to be retrained from scratch. Consequently, it is essential to investigate …
Figure 7
Figure 7. Figure 7: Illustration of Unintential Memorization. At the end, we share a phenomenon observed in our experiments, which we refer to as unintentional memorization: the model inadvertently memorizes the injected PII from the image, despite it being entirely unrelated to the curre…
Figure 8
Figure 8. Figure 8: Membership Inference Results for Un￾intentional Memorization We present the results in [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Testing with Different Prompt Strategies. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 1 canonical work pages

  1. [1]

    Physics of language models: Part 3.2, knowledge manipulation

    Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.2, knowledge manipulation. arXiv preprint arXiv:2309.14402,

    arXiv

  2. [4]

    ISBN 978-1-939133-37-3

    USENIX Association. ISBN 978-1-939133-37-3. URL https://www.usenix. org/conference/usenixsecurity23/presentation/carlini. Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning llms on new knowledge encourage hallucinations?arXiv preprint arXiv:2405.05904,

    arXiv

  3. [6]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt

    URLhttp://jmlr.org/papers/v24/23-0569.html. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300,

    arXiv 2009

  4. [9]

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick

    URLhttps://proceedings.mlr.press/v162/kandpal22a.html. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. InComputer vision– ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13, pages 740–75...

    work page 2014

  5. [12]

    Scalable extraction of training data from (production) language models.arXiv preprint arXiv:2311.17035,

    Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models.arXiv preprint arXiv:2311.17035,

    arXiv

  6. [13]

    Gpqa: A graduate-level google-proof q&a benchmark, nov.arXiv preprint arXiv:2311.12022,

    D Rein, BL Hou, AC Stickland, J Petty, RY Pang, J Dirani, J Michael, and SR Bowman. Gpqa: A graduate-level google-proof q&a benchmark, nov.arXiv preprint arXiv:2311.12022,

    arXiv

  7. [14]

    Understanding and mitigating copying in diffusion models

    Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Gold- stein. Understanding and mitigating copying in diffusion models. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neu- ral Information Processing Systems, volume 36, pages 47783–47803. Curran Associates, Inc., 2023a. URL https://proce...

    arXiv 2023

  8. [15]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805,

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805,

    arXiv

Show all 19 references
  1. [16]

    Gemma 3 technical report.arXiv preprint arXiv:2503.19786,

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report.arXiv preprint arXiv:2503.19786,

    arXiv

  2. [17]

    Measuring short-form factuality in large language models

    Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368,

    arXiv

  3. [18]

    Detecting, explaining, and mitigating memo- rization in diffusion models

    Yuxin Wen, Yuchen Liu, Chen Chen, and Lingjuan Lyu. Detecting, explaining, and mitigating memo- rization in diffusion models. InThe Twelfth International Conference on Learning Representations, 2024a. URLhttps://openreview.net/forum?id=84n3UwkH7b. Yuxin Wen, Leo Marchyok, Sang...

    1905 arXiv

  4. [19]

    Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang

    URL https://proceedings.neurips.cc/paper_files/paper/2023/file/ 7bc4f74e35bcfe8cfe43b0a860786d6a-Paper-Conference.pdf. Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors.arXiv preprint arXiv:2309.03882,

    2023 arXiv

  5. [2014]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101,

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101,

    arXiv

  6. [2017]

    Tofu: A task of fictitious unlearning for llms.arXiv preprint arXiv:2401.06121,

    Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter. Tofu: A task of fictitious unlearning for llms.arXiv preprint arXiv:2401.06121,

    arXiv

  7. [2021]

    Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974,

    12 Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974,

    arXiv

  8. [2022]

    Training data leakage analysis in language models.arXiv preprint arXiv:2101.05405,

    Huseyin A Inan, Osman Ramadan, Lukas Wutschitz, Daniel Jones, Victor Rühle, James Withers, and Robert Sim. Training data leakage analysis in language models.arXiv preprint arXiv:2101.05405,

    arXiv

  9. [2023]

    Imagen 3.arXiv preprint arXiv:2408.07009,

    Jason Baldridge, Jakob Bauer, Mukul Bhutani, Nicole Brichtova, Andrew Bunner, Lluis Castrejon, Kelvin Chan, Yichang Chen, Sander Dieleman, Yuqing Du, et al. Imagen 3.arXiv preprint arXiv:2408.07009,

  10. [2024]

    Fantastic copyrighted beasts and how (not) to generate them.arXiv preprint arXiv:2406.14526,

    Luxi He, Yangsibo Huang, Weijia Shi, Tinghao Xie, Haotian Liu, Yue Wang, Luke Zettlemoyer, Chiyuan Zhang, Danqi Chen, and Peter Henderson. Fantastic copyrighted beasts and how (not) to generate them.arXiv preprint arXiv:2406.14526,

    arXiv

  11. [2025]

    a is b" fail to learn

    URLhttps://openreview.net/forum?id=HVblmL5Rws. Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Llms trained on" a is b" fail to learn" b is a".arXiv preprint arXiv:2309.12288,

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.