REVIEW 16 cited by
Designing and Interpreting Probes with Control Tasks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Probes, supervised models trained to predict properties (like parts-of-speech) from representations (like ELMo), have achieved high accuracy on a range of linguistic tasks. But does this mean that the representations encode linguistic structure or just that the probe has learned the linguistic task? In this paper, we propose control tasks, which associate word types with random outputs, to complement linguistic tasks. By construction, these tasks can only be learned by the probe itself. So a good probe, (one that reflects the representation), should be selective, achieving high linguistic task accuracy and low control task accuracy. The selectivity of a probe puts linguistic task accuracy in context with the probe's capacity to memorize from word types. We construct control tasks for English part-of-speech tagging and dependency edge prediction, and show that popular probes on ELMo representations are not selective. We also find that dropout, commonly used to control probe complexity, is ineffective for improving selectivity of MLPs, but that other forms of regularization are effective. Finally, we find that while probes on the first layer of ELMo yield slightly better part-of-speech tagging accuracy than the second, probes on the second layer are substantially more selective, which raises the question of which layer better represents parts-of-speech.
Forward citations
Cited by 16 Pith papers
-
Robust Control under Stationary Ambiguity
Policies trained under stationary latent ambiguity, implemented by refreshing the latent parameter, preserve robustness to regime shifts better than policies trained under a fixed latent draw.
-
Verbalizable Representations Form a Global Workspace in Language Models
Language models represent their current reasoning in a small, readable set of verbalizable vectors (the J-space) that functions like a global workspace.
-
When Does Reward Teach State? A Hidden-Automaton Instrument and a Group-Language Warning Signal
High reward in sparse RL does not imply latent-state recovery; a hidden-DFA instrument separates perception from planning gaps and flags group-language structure as a pre-training warning.
-
Two Stages of Folding: Convergent Mechanisms in AI Protein Folding Trunks
Causal interventions show ESMFold's folding trunk first transfers sequence chemistry into its pairwise representation (blocks 0–7), then builds pairwise spatial features that control output geometry (blocks 25+).
-
TeaMatch: Teachable Cross-Modal Representation Learning for 2D-3D Matching
A training-only regularizer makes cross-modal 2D-3D matching representations recoverable by weak probes, improving robustness and pose accuracy on 7-Scenes and RGB-D Scenes V2.
-
Mechanistic Interpretability of Cognitive Complexity in LLMs via Linear Probing using Bloom's Taxonomy
Linear probes on LLM residual streams classify Bloom's Taxonomy levels with high accuracy, but the result may reflect prompt lexico-semantic cues rather than a general cognitive-complexity representation.
-
Quantitative Analysis of Proxy Tasks for Anomalous Sound Detection
Better proxy-task performance does not generally improve anomalous sound detection; only source separation showed a strong, consistent positive correlation.
-
Tracking World States with Language Models: State-Based Evaluation Using Chess
A model-agnostic chess evaluation metric measures state-tracking fidelity by comparing legal-move sets of predicted and true positions, showing GPT-4o's reconstruction quality degrades over longer games.
-
What Does it Mean for a Neural Network to Learn a "World Model"?
Defines a world model as a simple commutative-diagram factorization through an intermediate representation, with conditions that the model be learned and emergent rather than inherited from input or output.
-
METAGENE-1: Metagenomic Foundation Model for Pandemic Monitoring
A 7B transformer pretrained on 1.5T base pairs of wastewater metagenomic reads achieves strong pathogen detection and embedding scores, though some new benchmarks are partly in-distribution.
-
Decoding Task Progress from VLA Representations
Task progress is linearly decodable from the residual stream of π0.5 and can serve as a label-free out-of-distribution detector for deployed robot policies.
-
LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformers
LazyDiT learns small gates that decide when to reuse cached layer outputs, cutting diffusion transformer compute by up to half while matching or beating DDIM quality.
-
Numerical Pruning for Efficient Autoregressive Models
A Newton's method derived importance score plus a constrained least-squares compensation prunes LLaMA and LlamaGen models with reported gains over LLM-Pruner, SliceGPT, and FLAP.
-
Emergent Stack Representations in Modeling Counter Languages Using Transformers
A small transformer trained on counter languages encodes the current stack depth in its final-layer activations, recoverable by simple probing classifiers.
-
Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey
A survey maps the field of MLLM explainability and interpretability into data, model, and training and inference perspectives.
-
Towards Transparent AI: A Survey on Explainable Large Language Models
A review that groups LLM explainability methods by transformer architecture and discusses their evaluation and applications.
Discussion (0). Continue with ORCID to comment.