REVIEW 17 cited by
Dated Data: Tracing Knowledge Cutoffs in Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Dated Data: Tracing Knowledge Cutoffs in Large Language Models
read the original abstract
Released Large Language Models (LLMs) are often paired with a claimed knowledge cutoff date, or the dates at which training data was gathered. Such information is crucial for applications where the LLM must provide up to date information. However, this statement only scratches the surface: do all resources in the training data share the same knowledge cutoff date? Does the model's demonstrated knowledge for these subsets closely align to their cutoff dates? In this work, we define the notion of an effective cutoff. This is distinct from the LLM designer reported cutoff and applies separately to sub-resources and topics. We propose a simple approach to estimate effective cutoffs on the resource-level temporal alignment of an LLM by probing across versions of the data. Using this analysis, we find that effective cutoffs often differ from reported cutoffs. To understand the root cause of this observation, we conduct a direct large-scale analysis on open pre-training datasets. Our analysis reveals two reasons for these inconsistencies: (1) temporal biases of CommonCrawl data due to non-trivial amounts of old data in new dumps and (2) complications in LLM deduplication schemes involving semantic duplicates and lexical near-duplicates. Overall, our results show that knowledge cutoffs are not as simple as they have seemed and that care must be taken both by LLM dataset curators as well as practitioners who seek to use information from these models.
Forward citations
Cited by 17 Pith papers
-
Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters
When forecasters are barred from reading post-cutoff text, retrieval still improves Brier score on 8 of 9 LLMs, but only on markets Reddit had discussed beforehand; on speculative topics retrieval makes forecasts worse.
-
MeMo: Memory as a Model
MeMo encodes new knowledge into a separate memory model for frozen LLMs, achieving strong performance on BrowseComp-Plus, NarrativeQA, and MuSiQue while capturing cross-document relationships and remaining robust to r...
-
Library Hallucinations in LLM-Generated Code: A Risk Analysis Grounded in Developer Queries
A study of seven LLMs finds that realistic prompt variations such as one-character misspellings trigger library hallucinations in up to 26% of cases, fabricated names in up to 99%, and time-based prompts in up to 85%,...
-
AISPA: User-Centric System Prompt Auditing for Large Language Model Applications
A human-in-the-loop audit of system prompts from 88 commercial AI products finds protective instructions nearly universal yet incomplete, with ~40% of products containing at least one user-harmful directive.
-
Agentic Persona Generation with Critique-Refinement: An Industrial Evaluation
PerGent, an agentic critique-refinement system for persona generation, reaches 96.9% expert approval in an industrial evaluation at Kinaxis and reproduces more pre-LLM expert content than single-shot baselines.
-
Greener Than Humans? Environmental Attitudes in Large Language Models
Many LLMs exhibit stronger environmental cognition, affect, and behavioral recommendations than human survey averages and shift with persona prompts.
-
The Geometry of Forgetting: Temporal Knowledge Drift as an Independent Axis in LLM Representations
Temporal knowledge drift is encoded as a geometrically orthogonal direction in LLM residual streams, independent of correctness and uncertainty.
-
Benchmarking Open-Weight Foundation Models for Global AI Technical Governance
Open-weight frontier models fabricate ~72% of AI-governance numeric answers, almost never refuse, and show inverted North/South accuracy driven largely by a proportional ±10% scoring rule and sparse high-value indicators.
-
Developer Experience with AI Coding Agents: HTTP Behavioral Signatures in Documentation Portals
AI coding agents and assistants fetch documentation in one or two HTTP requests with identifiable fingerprints, undermining standard web analytics.
-
Knowledge is Not Enough: Injecting RL Skills for Continual Adaptation
PaST extracts a domain-agnostic skill vector from RL training and linearly injects it into SFT-adapted LLMs to improve knowledge use on QA and tool-use tasks.
-
CacheClip: Accelerating RAG with Effective KV Cache Reuse
CacheClip accelerates RAG prefill by up to 3.33x via auxiliary-model-guided selective KV recomputation while retaining 85-91% of full-attention quality on NIAH and LongBench.
-
MeMo: Memory as a Model
MeMo encodes new knowledge into a separate memory model that integrates with frozen LLMs, showing strong performance on QA benchmarks while avoiding catastrophic forgetting and working without access to model weights.
-
Teaching Large Language Models When Not to Know: Learning Temporal Critique for Ex-Ante Reasoning
TCFT trains LLMs on temporal critique tasks to reduce post-cutoff knowledge leakage by 37-42 percentage points over prompting and standard SFT on Qwen models.
-
Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
LLM agent progress depends on externalizing cognitive functions into memory, skills, protocols, and harness engineering that coordinates them reliably.
-
Developer Experience with AI Coding Agents: HTTP Behavioral Signatures in Documentation Portals
AI coding agents produce identifiable HTTP behavioral signatures and compress multi-page navigation into one or two requests, rendering standard engagement metrics unreliable.
-
ZoFia: Zero-Shot Fake News Detection with Entity-Guided Retrieval and Multi-LLM Interaction
ZoFia is a zero-shot fake news detection framework that uses hierarchical entity salience retrieval followed by multi-LLM adversarial debate to improve robustness over single-model approaches.
-
Advancing Multi-Agent RAG Systems with Minimalist Reinforcement Learning
Mujica-MyGo decomposes multi-turn RAG interactions via multi-agent workflows and applies minimalist policy gradient optimization to improve performance on QA benchmarks while avoiding long-context problems.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.