WikiMem, a Wikidata-derived canary dataset and a calibrated NLL-ranking metric, identifies which human-fact associations an LLM has memorized, with higher rates for famous people and larger models.
Targeting the Benchmark: On Methodology in Current Natural Language Processing Research
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
It has become a common pattern in our field: One group introduces a language task, exemplified by a dataset, which they argue is challenging enough to serve as a benchmark. They also provide a baseline model for it, which then soon is improved upon by other groups. Often, research efforts then move on, and the pattern repeats itself. What is typically left implicit is the argumentation for why this constitutes progress, and progress towards what. In this paper, we try to step back for a moment from this pattern and work out possible argumentations and their parts.
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
What Should LLMs Forget? Quantifying Personal Data in LLMs for Right-to-Be-Forgotten Requests
WikiMem, a Wikidata-derived canary dataset and a calibrated NLL-ranking metric, identifies which human-fact associations an LLM has memorized, with higher rates for famous people and larger models.