REVIEW 31 cited by
Datasheets for Datasets
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Datasheets for Datasets
read the original abstract
The machine learning community currently has no standardized process for documenting datasets, which can lead to severe consequences in high-stakes domains. To address this gap, we propose datasheets for datasets. In the electronics industry, every component, no matter how simple or complex, is accompanied with a datasheet that describes its operating characteristics, test results, recommended uses, and other information. By analogy, we propose that every dataset be accompanied with a datasheet that documents its motivation, composition, collection process, recommended uses, and so on. Datasheets for datasets will facilitate better communication between dataset creators and dataset consumers, and encourage the machine learning community to prioritize transparency and accountability.
Forward citations
Cited by 31 Pith papers
-
PDEAgent-Bench: A Multi-Metric, Multi-Library Benchmark for PDE Solver Generation
PDEAgent-Bench is the first multi-metric, multi-library benchmark for AI-generated PDE solvers, evaluating executability, numerical accuracy, and efficiency across DOLFINx, Firedrake, and deal.II.
-
#PraCegoVer: A Large Dataset for Image Captioning in Portuguese
The paper introduces #PraCegoVer, the first large-scale image captioning dataset in Portuguese sourced from Instagram posts with single user-generated captions per image.
-
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
The Pile is a newly constructed 825 GiB dataset from 22 diverse sources that enables language models to achieve better performance on academic, professional, and cross-domain tasks than models trained on Common Crawl ...
-
GraphInfer-Bench: Benchmarking LLM's Inference Capability on Graphs
Presents GraphInfer-Bench to demonstrate that no evaluated LLM-based method family closes the performance gap on graph inference tasks requiring multi-node reasoning, with plain GNNs matching or exceeding them.
-
Toward Calibrated, Fair, and accurate Deepfake Detection
Face-Feature Tuning is a label-free logit remapping method that reduces FPR/TPR gaps across groups in deepfake detection while preserving overall accuracy.
-
CIRCLED: A Multi-turn CIR Dataset with Consistent Dialogues across Domains
CIRCLED is a multi-turn CIR dataset of 22,608 sessions generated from existing single-turn datasets via CIReVL pipeline and curated with filters for consistency, scale, and generality across domains.
-
Positional Failures in Long-Context LLMs: A Blind Spot in Reasoning Benchmarks
Audits reveal no reasoning benchmark controls position/filler/length jointly; CRE shows LLMs drop up to 88pp on middle-position tasks at 64K context, with diagnostic probe supporting positional cause.
-
Mechanism Plausibility in Generative Agent-Based Modeling
Introduces the Mechanism Plausibility Scale to distinguish generative sufficiency from mechanistic plausibility in LLM-based agent-based models.
-
TRAVELFRAUDBENCH: A Configurable Evaluation Framework for GNN Fraud Ring Detection in Travel Networks
TravelFraudBench is a new configurable benchmark for GNN-based fraud ring detection in travel networks, simulating star, clique, and chain topologies and showing GraphSAGE outperforming MLP baselines on AUC and ring recovery.
-
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
Imagen achieves state-of-the-art photorealistic text-to-image generation by scaling a text-only pretrained T5 language model within a diffusion framework, reaching FID 7.27 on COCO without training on it.
-
The Decomposition Is the Fingerprint: Per-Component Identity for Agent Skills
A per-component SimHash fingerprint supplies structural identity for AI agent skills, recovering family membership under paraphrase and refactoring with AUC 0.974 while localizing changes.
-
Frankenstein in the Pipeline: Computational Epistemicide in Facial Recognition
Facial recognition enacts computational epistemicide by progressively reducing faces to standardized numerical vectors, rendering reformist ethical AI insufficient and requiring abolition of vectorized identity as a b...
-
TeraGram: A Structured Longitudinal Dataset of the Telegram Messenger
A large-scale longitudinal dataset of public Telegram content is introduced to enable studies of engagement patterns and network evolution without algorithmic curation.
-
To Build or Not to Build? Factors that Lead to Non-Development or Abandonment of AI Systems
A scoping review and empirical analysis produce a six-category taxonomy of factors driving AI non-development and abandonment, showing that practical issues like resource limits and organizational dynamics often outwe...
-
A Human-Centric Framework for Data Attribution in Large Language Models
Introduces a parameter-driven framework for data attribution in LLMs that enables negotiation among creators, users, and intermediaries to meet stakeholder goals within the data economy.
-
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
RLHF-aligned language models show increasing resistance to red teaming with scale up to 52B parameters, unlike prompted or rejection-sampled models, supported by a released dataset of 38,961 attacks.
-
Ethical and social risks of harm from Language Models
The authors provide a detailed taxonomy of 21 risks associated with language models, covering discrimination, information leaks, misinformation, malicious applications, interaction harms, and societal impacts like job...
-
Deduplicating Training Data Makes Language Models Better
Deduplicating training datasets reduces language model verbatim memorization by 10x, improves training efficiency, and enables more accurate evaluation by cutting train-test overlap.
-
Measuring Coding Challenge Competence With APPS
APPS benchmark shows models like GPT-Neo pass roughly 20% of test cases on introductory problems, indicating machine learning is beginning to learn basic coding.
-
Aligning AI With Shared Human Values
Introduces ETHICS benchmark showing current language models have promising but incomplete ability to predict basic human ethical judgments on text scenarios.
-
SkillCenter: A Large-Scale Source-Grounded Skill Library for Autonomous AI Agents
SkillCenter provides 216,938 source-grounded skills across 24 domains via an automated LLM-based pipeline, with a downstream evaluation showing skills help agents only when the task exceeds the model's knowledge and r...
-
Benchmarks for Vision-Language Models in Urban Perception Should Be Reliability-Aware and Negotiated
Benchmarks for VLMs in urban perception should incorporate reliability metrics and negotiable labels, as model-human agreement co-varies with human reliability and distributional mismatches appear in a Montreal street...
-
TRACE: A taxonomy-grounded synthetic dataset for teaching-program generation and session interpretation in Applied Behavior Analysis
TRACE is a taxonomy-grounded synthetic instruction-tuning dataset with 2,999 examples for ABA teaching-program generation and multi-session behavioral interpretation, released with code, provenance, and stratified splits.
-
Mechanism Plausibility in Generative Agent-Based Modeling
Introduces the Mechanism Plausibility Scale, a four-level framework separating generative sufficiency from mechanistic plausibility in LLM-based agent-based models.
-
Evaluating Structured Documentation as a Tool for Reflexivity in Dataset Development
Structured dataset documentation shows little engagement with major reflexivity themes from FAccT literature, leading to a new codebook and extended datasheet questions.
-
Benchmarked Yet Not Measured -- Generative AI Should be Evaluated Against Real-World Utility
Generative AI evaluation should prioritize real-world utility through stakeholder goals and longitudinal human outcome measurements instead of static benchmark performance.
-
Reproducibility in Machine Learning for Health
Systematic evaluation of over 100 ML4H papers finds poorer reproducibility than other ML fields, driven by limited data and code access, and offers recommendations to data providers, publishers, and researchers.
-
Benchmarked Yet Not Measured -- Generative AI Should be Evaluated Against Real-World Utility
Generative AI evaluation must shift from static benchmark scores to measuring sustained improvements in human capabilities within specific deployment contexts.
-
Mapping the Stochastic Penal Colony
Content moderation operates as a stochastic penal colony that banishes users through the constant threat of account suspension, shown via auto-ethnographic case studies of Twitter, OpenAI DALL-E 2, and Pinterest.
-
Toward Reusability of AI Models Using Dynamic Updates of AI Documentation
A data-driven methodology is proposed to dynamically update AI model cards from community practices in the Hugging Face repository to improve model reusability.
-
LLMs in Qualitative Research: Opportunities, Limitations, and Practical Considerations
The paper outlines opportunities, limitations, and practical parameters for integrating LLMs into qualitative research while aligning with epistemological commitments like reflexivity and interpretive judgment.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.