REVIEW 3 cited by
Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenon
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Memorization in language models is typically treated as a homogenous phenomenon, neglecting the specifics of the memorized data. We instead model memorization as the effect of a set of complex factors that describe each sample and relate it to the model and corpus. To build intuition around these factors, we break memorization down into a taxonomy: recitation of highly duplicated sequences, reconstruction of inherently predictable sequences, and recollection of sequences that are neither. We demonstrate the usefulness of our taxonomy by using it to construct a predictive model for memorization. By analyzing dependencies and inspecting the weights of the predictive model, we find that different factors influence the likelihood of memorization differently depending on the taxonomic category.
Forward citations
Cited by 3 Pith papers
-
Phantoms and Disclosures: A Statistical Framework for Auditing Privacy in Synthetic Data
A model-agnostic audit detects synthetic data disclosures via feature-match and membership-inference tests that separate true from phantom leaks and give empirical differential-privacy lower bounds.
-
Low-Perplexity LLM-Generated Sequences and Where To Find Them
Only about 40% of low-perplexity 6-token spans generated by Pythia-6.9B can be exactly matched to The Pile, and the authors categorize matched and unmatched spans into four classes.
-
Enhancing SLMs for Sustainable Code Optimization in Radio-Astronomy
A 7-billion-parameter coding model with multi-sampling matches a 32-billion-parameter model on repository-level completion under the same time budget, and compiler-error feedback improves pass rates across all models tested.
Discussion (0). Sign in to comment.