REVIEW 2 cited by
A Statistical Turing Test for Generative Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
A Statistical Turing Test for Generative Models
read the original abstract
The emergence of human-like abilities of AI systems for content generation in domains such as text, audio, and vision has prompted the development of classifiers to determine whether content originated from a human or a machine. Implicit in these efforts is an assumption that the generation properties of a human are different from that of the machine. In this work, we provide a framework in the language of statistical pattern recognition that quantifies the difference between the distributions of human and machine-generated content conditioned on an evaluation context. We describe current methods in the context of the framework and demonstrate how to use the framework to evaluate the progression of generative models towards human-like capabilities, among many axes of analysis.
Forward citations
Cited by 2 Pith papers
-
Testing for LLM response differences: the case of a composite null consisting of semantically irrelevant query perturbations
A new hypothesis test for binary LLM responses treats semantically equivalent query perturbations as an unknown null set and gives asymptotic validity and consistency guarantees under a uniformity assumption.
-
When prompt perturbations break your A/B test: A valid statistical test for generative surveying
Standard hypothesis tests fail for generative surveys under realistic prompt perturbations; a permutation test is valid and practical guidance on budget allocation is given.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.