LACUNA is a new testbed that injects PII into predefined model parameters to benchmark the localization precision of LLM unlearning methods, revealing that SOTA approaches are imprecise despite strong output performance.
super hub Mixed citations
Title resolution pending
Mixed citation behavior. Most common role is background (56%).
147
Pith papers citing it
Background
56% of classified citations
abstract
We introduce Olmo 3, a family of state-of-the-art, fully-open language models at the 7B and 32B parameter scales. Olmo 3 model construction targets long-context reasoning, function calling, coding, instruction following, general chat, and knowledge recall. This release includes the entire model flow, i.e., the full lifecycle of the family of models, including every stage, checkpoint, data point, and dependency used to build it. Our flagship model, Olmo 3 Think 32B, is the strongest fully-open thinking model released to-date.
hub tools
citation-role summary
background 13
dataset 2
method 2
baseline 1
citation-polarity summary
claims ledger
- abstract We introduce Olmo 3, a family of state-of-the-art, fully-open language models at the 7B and 32B parameter scales. Olmo 3 model construction targets long-context reasoning, function calling, coding, instruction following, general chat, and knowledge recall. This release includes the entire model flow, i.e., the full lifecycle of the family of models, including every stage, checkpoint, data point, and dependency used to build it. Our flagship model, Olmo 3 Think 32B, is the strongest fully-open thinking model released to-date.
co-cited works
representative citing papers
CoT transformers simulate any Word RAM algorithm with poly-logarithmic overhead in three architectures, improving on quadratic TM overhead.
Sumi is an openly released 7B parameter uniform diffusion language model pretrained from scratch on 1.5T tokens that matches autoregressive models on several benchmarks.
Introduces BonaFide benchmark of 3,066 ground-truth labeled CoTs showing most faithfulness metrics perform near chance with biases and poor scaling to longer chains.
BoLT is a benchmark of surrogate models fitted to real LLM experiment data that enables evaluation of Bayesian and black-box optimization methods on multi-fidelity, multi-objective, high-dimensional LLM tasks.
Persona vectors form within the first 0.22% of LLM pretraining and remain effective for steering post-trained models, with continued refinement and transfer to other models.
LLMs lack temporal awareness of medical knowledge, showing gradual performance decline on up-to-date facts, much lower accuracy on historical knowledge (25-54% relative), and inconsistent year-to-year predictions.
LLM popularity judgments align more closely with pretraining data exposure counts than with Wikipedia popularity, with stronger effects in pairwise comparisons and larger models.
LLMs collapse advice into a single supportive persona; Inverse-Process Distillation restores human-like persona diversity, yet raters still prefer the collapsed default.
Privileged-context on-policy self-distillation degrades thinking models' long-budget accuracy by suppressing forking and self-correction behaviors, while helping instruction-tuned models.
STEB is a new benchmark of 96 datasets in 7 languages for evaluating style text embeddings on authorship, detection, and linguistic probing tasks.
MultiHashFormer enables hash-based autoregression in LMs by encoding tokens as multi-hash signatures, outperforming standard Transformers at 100M-3B scales while keeping parameter count constant for multilingual expansion.
Hybrid models outperform transformers on semantic state tracking tasks but underperform on syntactic bracket matching and n-gram copying at the token level.
Large Language Gibbs uses LLM next-token conditionals as MCMC transition operators for iterative resampling of structured variables, aiming to produce a stationary distribution that compromises across all local conditionals.
Authors demonstrate functional memorization in code LLMs via counterfactual midtraining comparison on functional equivalence metrics beyond textual overlap.
ModSleuth reconstructs dependency graphs from public artifacts for four LLM releases, recovering 1,060 source-verified dependencies and exposing license issues, train-evaluation coupling, and documentation gaps.
A finetuned Qwen3-235B model organism achieves comparable train-time harmfulness to controls while sustaining a ~15 percentage point compliance gap across 700 RL steps by framing compliance as context-specific.
WhiFlash introduces token-level cross-paradigm routing between autoregressive and diffusion drafting models, with cache optimizations, to raise acceptance lengths and deliver up to 69.6% throughput gains over EAGLE-3.
OPD updates occupy a relaxed off-principal regime and rapidly lock into a low-dimensional subspace that is functionally sufficient for its performance, distinct from SFT and RLVR trajectories.
LLMs detect fabricated statistics in isolation but, in multi-source synthesis, weight sources by analytical writing register rather than numeric validity.
DistIL applies distributional DAgger with forward cross-entropy to achieve monotonic policy improvement and better Pass@N from rich feedback in RL for reasoning tasks.
LoopMoE is a looped MoE language model that outperforms matched vanilla MoE on 8 of 9 downstream benchmarks at 3B scale and continues to outperform at 9B scale under strictly controlled budgets.
IndoBias is a dual-track culturally grounded benchmark revealing strong LLM bias in Indonesian prototypical sentences and higher ideology/religion bias in local languages, with Common Crawl pretraining adding more bias than curated sources.
Subliminal learning is steering vector distillation: a student fine-tuned on a steered teacher's outputs learns to imitate the steering vector.
citing papers explorer
-
A Human-Centric Framework for Data Attribution in Large Language Models
Introduces a parameter-driven framework for data attribution in LLMs that enables negotiation among creators, users, and intermediaries to meet stakeholder goals within the data economy.
-
Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RL
PULSE exploits BF16-invisible sparsity in weight updates to enable over 100x lower communication in distributed RL post-training via compute-visible sparsification.
-
MoCo: A One-Stop Shop for Model Collaboration Research
MoCo supplies a unified library of 26 collaboration strategies and benchmarks demonstrating average outperformance over single models in 61 percent of (model, data) pairs.
-
SUPERNOVA: Eliciting General Reasoning in LLMs with Reinforcement Learning on Natural Instructions
Curating 25K natural-instruction examples for RLVR yields a 64.4pp relative gain on BBEH for Qwen3-0.6B and generalizes across scales and model families.
-
Procedural Memory Distillation: Online Reflection for Self-Improving Language Models
PMD extracts and distills cross-episode procedural knowledge from RL rollouts into LLM policies at three abstraction levels, yielding 3.8-13.6% gains over SDPO on SCIKNOWEVAL and LIVECODEBENCH via co-evolution.
-
Watermarking for Proprietary Dataset Protection
Watermark-based dataset inference achieves membership detection performance comparable to loss-based methods when subset exposure is high, under alternate assumptions.
-
Push Puppet Networks: Structured Bayesian Pruning Algorithm for Language Model Compression
Push puppet networks learn a hierarchical penalty function during training to enable structured pruning of LLMs to arbitrary target sizes with reported speedups over SparseGPT and Wanda at high pruning ratios.
-
Aloe-Vision: Robust Vision-Language Models for Healthcare
Releases open medical LVLMs trained on a quality-filtered multimodal dataset, introduces CareQA-Vision benchmark from exams, reports performance gains over baselines, and flags adversarial vulnerabilities.
-
Randomized YaRN Improves Length Generalization for Long-Context Reasoning
Randomized YaRN improves LLM reasoning performance on 16K-128K contexts when trained only on <8K data by randomizing YaRN positional encodings during short-context training.
-
Neurosymbolic Clinical Trial Matching via LLM-Driven Abduction and Logical Verification
αNeSy-CTM is a neurosymbolic CTM system using LLM-driven abduction plus logical verification that reports up to 30% relative improvement over zero-shot LLM baselines.
-
From Drift to Coherence: Stabilizing Beliefs in LLMs
In multiple-choice QA, LLM beliefs drift early under repeated sampling but self-stabilize; seed-answer prompting and a self-consistency loss reduce drift while preserving accuracy.
-
Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signal
A new pipeline uses interpretability to characterize concepts in preference data and shape rewards via feature or data interventions during LM post-training.
-
A Predictive Law for On-Policy Self-Distillation From World Feedback
A linear relationship between initial student-self-teacher performance gap and OPSD improvement provides a predictive law across contexts and model families.
-
Efficient Pre-Training of LLMs through Truncated SVD Layers
TSVD framework maintains low-rank orthonormal weights during LLM pretraining via truncated SVD, adaptive spectral rank selection, and caching to reduce compute while matching baseline performance.
-
CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generations
CroCo applies English-reward-ranked self-generations for contrastive preference tuning that improves two LLMs on structured and open-ended tasks across 14 languages without language-specific annotations.
-
Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replay
Self-generated replay from language models nearly eliminates catastrophic forgetting during finetuning except when models are pretrained close to saturation.
-
Seed Bank, Co-op, Stoop Swap: Metaphors for Governing Language Model Data for Creative Writing
Workshops with over 100 creative writers produced metaphors and four themes for language model governance that favor consent-driven, smaller open models encoding community values.
-
Before the Last Token: Diagnosing Final-Token Safety Probe Failures
Final-token safety probes miss jailbreaks whose unsafe evidence appears earlier in prefill; a clean-trained PCA-HMM trajectory model recovers many of those misses without catastrophic false positives.
-
A Recipe for Long-Context Reasoning in Large Language Models via On-Policy Optimization and Distillation
Combines GRPO with teacher-guided on-policy distillation and introduces LongBlocks dataset to yield more stable long-context reasoning than either method alone.
-
Mid-Training with Self-Generated Data Improves Reinforcement Learning in Language Models
Mid-training LLMs on self-generated diverse reasoning paths improves subsequent RL performance on mathematical benchmarks and OOD tasks.
-
When LLMs Stop Following Steps: A Diagnostic Study of Procedural Execution in Language Models
Across 15 LLMs, first-answer accuracy on arithmetic procedural execution falls from 63% at 5 steps to 20% at 95 steps, with under-execution increasing.
-
Multilinguality at the Edge: Developing Language Models for the Global South
A survey of 232 papers frames the 'last mile' challenge of deploying multilingual language models under edge hardware constraints for Global South communities.
-
Hyperloop Transformers
A looped Transformer with matrix-valued hyper-connections matches depth-matched baselines at ~50% fewer parameters, including under post-training quantization.
-
Heterogeneity in Formal Linguistic Competence of Language Models: Is Data the Real Bottleneck?
Injecting 1% synthetic data targeting specific constructions during pre-training of GPT-2 Small boosts performance on 8 of 9 weakest BLiMP paradigms (e.g., only_npi_scope from 20.9% to 69.4%), while aggregate performance holds or improves, with one resistant case.
-
Your Model Diversity, Not Method, Determines Reasoning Strategy
The optimal reasoning strategy for LLMs depends on the model's diversity profile rather than the exploration method itself.
-
Long-Context Reasoning Through Proxy-Based Chain-of-Thought Tuning
ProxyCoT transfers CoT reasoning from proxy short contexts to full long contexts through RL/distillation followed by SFT, outperforming baselines with lower overhead and generalizing out-of-domain.
-
Apriel-1.5-OpenReasoner: RL Post-Training for General-Purpose and Efficient Reasoning
Apriel-1.5-OpenReasoner uses RL post-training with adaptive sampling and difficulty-aware penalties to boost reasoning accuracy on AIME, GPQA, MMLU-Pro and LiveCodeBench while producing shorter traces and generalizing beyond its 16K training budget.
-
On the Utility and Factual Reliability of Pruned Mixture-of-Experts Models in the Biomedical Domain
Moderate pruning of MoE models preserves in-domain biomedical utility and reliability but both degrade rapidly in cross-domain settings and at extreme pruning ratios.
-
NebulaExp-8B: An Empirical Post-Training Pipeline via Full-Scale Ablation Research
NebulaExp reports an empirical post-training pipeline on Qwen3-8B that raises instruct scores from 55.01 to 61.85 and reasoning scores from 73.88 to 75.17 via curated data, SFT, GRPO RL, and OPD/MOPD distillation.
-
ZONOS2 Technical Report
ZONOS2 8B is a scaled MoE TTS model with 900M active parameters trained on 6M hours of data that reports competitive SOTA results on naturalness, speaker similarity, WER, and a new ZTTS1-Eval benchmark while releasing weights and code.
-
Do Large Language Models Always Tell The Same Stories?
LLM stories show higher narrative similarity than human ones across 10 models, with frontier models converging on generic narratives and standard mitigations failing to increase diversity.
-
Neutrality Bites: Gender Representation in AI-Generated Animal Stories
LLMs exhibit masculine bias when assigning gender to animal characters in generated stories, with neutrality often resulting in erasure of feminine perspectives.
-
What Am I Missing? Question-Answering as Hidden State Probing
Question generation produces a hidden-state signal that predicts final correctness before the answer is produced, yet gating interventions based on that signal do not reliably improve trajectories.
-
It Takes Two: Complementary Self-Distillation for Contextual Integrity in LLMs
SELFCI uses complementary self-distillation with two reverse KL divergences to align LLMs to contextual integrity while preserving utility, outperforming RL baselines like GRPO in agentic settings.
-
Reflections and New Directions for Human-Centered Large Language Models
Model developers must address human concerns, preferences, values, and goals with rigor at every stage of the LLM pipeline rather than only in post-training.
-
Mellum2 Technical Report
Mellum 2 is a 12B MoE model with 2.5B active parameters, trained on 10.6T tokens with MoE, GQA, SWA, and MTP, then post-trained into Instruct and Thinking variants, claimed competitive with 4B-14B models at 2.5B compute.
-
Developing an ESG-Oriented Large Language Model through ESG Practices
ESG-adapted versions of Qwen-3-4B using LoRA and IRM outperform the base model and Llama-3/Gemma-3 baselines on generative ESG question-answering tasks.
-
Reinforcement Learning from Human Feedback
An expository book that systematically presents RLHF methods, from reward modeling to direct alignment algorithms, aimed at readers with quantitative backgrounds.
- DRIFT: Difficulty Routing Self-DIstillation with Rhythm-Gated Exploration and Success BuFfer Training
- DataComp-VLM: Improved Open Datasets for Vision-Language Models
- LMs as Task-Specific Knowledge Bases: An Interpretability Analysis
- Language Models Compare Quantities Using Number-specific and Unit-specific Heuristics
- Early Data Exposure Improves Robustness to Subsequent Fine-Tuning
- Estimating Tail Risks in Language Model Output Distributions
- Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR
- Large Language Models Align with the Human Brain during Creative Thinking
- On the Limits of Layer Pruning for Generative Reasoning in Large Language Models