Pith. sign in

REVIEW 2 major objections 5 minor 12 references

Asking a language model to answer in JSON instead of prose changes which answer it picks, compressing the field's diversity by about 0.22 bits per model and pushing the most distinctive models toward the crowd.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 15:16 UTC pith:HSGNBVH3

load-bearing objection A careful, honest measurement that a prompt-level JSON request compresses answer-choice diversity, with the main caveat being exactly the one the authors flag: one clause per register leaves the JSON-token confound open, so the headline generalization outruns the design. the 2 major comments →

arxiv 2607.18476 v1 pith:HSGNBVH3 submitted 2026-07-20 cs.CL cs.AIcs.LG

Structured Output Collapses Answer Diversity Across 44 Language Models

Tapan Parikh This is my paper
classification cs.CL cs.AIcs.LG
keywords answer diversitystructured outputJSON promptingoutput formatmode collapseanswer-choice surprisalregister effectslanguage model conformity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether the surface through which software actually consumes language models — structured output such as JSON — is served by a different, more homogeneous model than the chat surface on which models are evaluated. Re-running a 31-prompt, 44-model one-word census with the answer requested in JSON, with no schema enforcement or constrained decoding, the field's answer-choice surprisal falls from 1.80 to 1.58 bits, and the modal answer to an unconstrained prompt rises from 41% to 64% of the pool. The models that lose the most distinctiveness are precisely the most distinctive ones, while the conformist floor barely moves. The effect is a sharpener rather than a re-indexer: the plain-chat modal answer survives in 28 of 31 categories. The upshot is that diversity metrics measured in chat overstate the diversity of the model that an agent or pipeline actually talks to.

Core claim

The central discovery is that the register in which an answer is requested changes which answer a model chooses: a simple appended clause — 'Reply with JSON only, in the form {"word": "<your answer>"}' — compresses the field of answers even though nothing in the decoder or sampler is altered. Measured within each format column, mean answer-choice surprisal drops by 0.22 bits (p = 0.0002), and on the unconstrained prompt the modal answer's share rises from 41% to 64% while distinct answers fall from 52 to 36. The compression is progressive: only six of 44 models move individually at a 10% false-discovery rate, all toward the mode, led by the strongest explorer, whose distinctiveness halves; t

What carries the argument

The instrument is the answer-choice surprisal metric: a leave-one-out measure, in bits, of how unlikely a model's answers are under the pooled answers of every other model, computed by exact match on normalized one-word replies. The paper's manipulation is a single appended format clause on a frozen 31-prompt, 44-model battery, scored within each format column so a column's convergence is an internal property rather than a comparison against chat. The five clauses (JSON, XML, YAML, CSV, and a bracket wrapper) and the response_format enforcement column isolate the register request as the only manipulated variable.

Load-bearing premise

The paper identifies the entire JSON register with a single fixed clause — 'Reply with JSON only, in the form {"word": "<your answer>"}' — so if the observed compression comes from that exact phrasing rather than from JSON as a register, the central generalization to all structured-output pipelines fails.

What would settle it

A clause-paraphrase battery: re-run the 31-prompt battery with several paraphrases of the JSON request (e.g., 'Output a JSON object containing your answer', 'Respond in JSON', 'Put your answer in a JSON field called word') and check whether the ~0.22-bit compression persists. If only the original sentence compresses, the register-gradient claim is a wording effect.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Diversity numbers collected in chat overstate the answer diversity that deployed systems see, because software consumes models through structured output and that surface is measurably more collapsed.
  • Decoding-side fixes are not the lever: enforcing a JSON schema at the decoder compresses only 0.03 bits beyond the request, so the collapse lives in the model's response to the register, not in the sampler.
  • The compression is progressive: the most distinctive models lose the most, so any downstream diversity budget should expect the tail to be the first casualty.
  • Because a model's defaults are register-indexed, a model's 'personality' is not a fixed trait; the same model can give an offbeat answer in chat and a crowd answer in JSON, or vice versa.
  • Format compliance falls out of the battery as a standing capability measure: older models cannot speak some registers at all, giving a dated, trackable record of when models acquired each output format.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the register gradient reflects tool-use post-training, then models released with stronger tool-use tuning should show larger JSON/XML compression; a per-release public instrument could test this directly.
  • A clause-paraphrase control is the natural next step: the paper's 'JSON register' is currently identified with one fixed sentence, so the whole claim should be read as about that template family until paraphrases confirm it.
  • Downstream tasks that consume structured outputs and are sensitive to answer diversity — surveys, recommendations, brainstorming, LLM-judge panels — may systematically under-sample minority answers; prompting for variety or sampling multiple registers could counteract this.
  • The surprisal/self-distinctness dissociation implies the collapse is positional mass relocation rather than rescaled sampling, so API-level temperature adjustments alone are unlikely to restore diversity.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. Parikh re-runs the One-Word Census on 44 models, appending to each of 31 prompts a request to reply in one of five serialization formats (JSON, XML, YAML, CSV, or square brackets). Using within-column answer-choice surprisal, the paper reports that the JSON clause lowers field-mean surprisal from 1.80 to 1.58 bits (Δ = −0.22, p = .0002), moves six models individually toward the mode, leaves 28/31 modal answers unchanged, shifts 53% of stable chat defaults, and shows a format gradient, with decoder-level enforcement adding only −0.03 bits. The paper concludes that the structured-output surface is served by a more homogeneous model than the chat surface and attributes the mechanism primarily to tool-use post-training.

Significance. If the result generalizes, it is important for any pipeline that consumes model outputs through JSON or other serialization: diversity measured on chat leaderboards may substantially overstate deployed diversity. The study is unusually careful: within-column scoring prevents the format shift from masquerading as convergence; permutation tests are run on both exchangeable units; compliance conditioning is checked on a common subset; split-half reliability is used against a regression-to-the-mean account; a panel-free JSD check distinguishes stranding from divergence; and an echo-guard sweep probes the brackets reversal. The code and raw data are released. The internal, same-clause effect is credible. What remains unestablished is the external generalization from the specific clauses tested to 'structured output' as a class.

major comments (2)
  1. [§3.2, §5] The paper's central generalization — that the 'structured-output surface' is more homogeneous — rests on one clause per format. The authors acknowledge in §5 that 'Each format is probed with a single clause wording, so we cannot separate the register from the particular phrasing that invokes it.' The gradient rules out shared word-slot/fill-in framing and clause length, but not the literal format-name token: 'JSON' and 'XML' may cue an API/tool-use distribution in pretraining, while 'YAML'/'CSV' may cue data files and 'brackets' a game. The default-acquisition (§4.4) and enforcement (§4.6) results use the same JSON clause and inherit the confound. Because the title, abstract, and Discussion generalize to structured output as a class, this is load-bearing. Add a clause-paraphrase control (different phrasings, or a schema-only instruction without the token 'JSON') or scope the claims to th
  2. [§4.6] The claim that decoder-level enforcement 'compresses no further than the request' is supported only by a point estimate (−0.03 bits) with no uncertainty quantification. With 36 models and 4 samples per cell, this difference may be within noise, and the abstract's statement that 'the collapse lives in the model's response to the register, not the decoder' is stronger than the evidence reported. Provide a confidence interval, a non-inferiority/equivalence test, or at minimum an explicit error-bar analysis for the −0.03 bit difference.
minor comments (5)
  1. [§4.5] The initial gradient reports compliance-conditioned means on different subsets (JSON n=43, XML n=41, YAML n=37, CSV n=39, brackets n=43). The common-subset analysis (n=34) reproduces the pattern; to avoid confusion, present the common-subset numbers as the primary comparison and the full-subset numbers as a robustness check.
  2. [§4.6] Define 'gateway coercion' for response_format and state whether enforcement was applied via a consistent provider API parameter across the 36 models.
  3. [Figure 1] The blue/amber distinction is likely not accessible to colorblind readers; add shape or label encoding in addition to color.
  4. [§3.2] The YAML clause formatting with backticks is ambiguous; use a display equation or code display for the exact clause.
  5. [§4.1] The split-half reliability and regression-to-the-mean null are reported without the sample size of the split-half; give the number of categories/models used in that calculation.

Circularity Check

0 steps flagged

No circular derivation: the JSON-vs-chat surprisal delta is new measured data, not a re-statement of the cited census; score 2 reflects only the substantial same-author baseline citation.

full rationale

The central claim—that requesting JSON lowers answer-choice surprisal relative to plain chat—rests on a new empirical contrast: the format columns (JSON, XML, YAML, CSV, brackets) are fresh runs (5,456 calls per column, §3.2) scored within each column against that column's own pooled answers (§3.1). The headline Δ-surprisal (JSON minus plain) is not a fitted parameter and is not defined in terms of the conclusion; no equation reduces 'JSON compression' to an input. Permutation tests on both exchangeable units (§4.5), compliance conditioning, the self-distinctness proxy (§4.3), the panel-free Jensen–Shannon check, and the split-half regression-to-the-mean null are all independent controls rather than circular re-statements. The companion census [7] is cited for the frozen instrument—prompts, 44-model panel, plain-chat transcripts, normalization/junk guard, and reliability statistics—but these are inputs, not outputs: the census contains no JSON/XML/YAML/CSV/brackets columns and no Δ-surprisal results, so the central finding is not entailed by the self-citation. The §5 limitation ('Each format is probed with a single clause wording, so we cannot separate the register from the particular phrasing that invokes it') is a genuine external-validity confound and the paper correctly proposes a clause-paraphrase control; however, it does not make any reported quantity equal to an input by construction. The AI-collaboration note is a transparency disclosure, not a derivation step. Overall, no circular step is present; score 2 reflects only the presence of a same-author baseline citation that is not load-bearing in the forbidden sense.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The central claims rest on the imported One-Word Census instrument, the control assumption that only the clause changes, and a set of measurement choices (token normalization, compliance cutoff, add-one smoothing, n=20 re-sample). No parameter is fitted to produce the headline delta; the strongest unstated assumption is that one clause wording per format represents the entire register.

free parameters (3)
  • Add-one smoothing pseudocount = 1
    Chosen for the leave-one-out surprisal metric and imported from the census; it shifts absolute bits but not format-to-format deltas.
  • Compliance conditioning cutoff = 90% wrapper compliance
    Used to restrict formats to models that can produce the wrapper; robustness is verified on the n=34 all-format-compliant subset.
  • Default re-sample size = n=20
    Fixed sample to distinguish stable defaults from lucky streaks; the four-of-four flag plus 20-run check supports the default claims.
axioms (5)
  • domain assumption The 31-prompt, 44-model One-Word Census instrument is a valid measure of answer-choice conformity.
    Central claim builds on this instrument from companion paper [7]; if the instrument is biased, the format effect estimate inherits that bias.
  • domain assumption The only difference between format columns and plain chat is the appended clause.
    The experiment relies on this control; §3.2 says all other conditions are identical.
  • domain assumption Requested temperature 1.0 is honored well enough that self-distinctness is a valid cooling proxy.
    §5 acknowledges providers may not honor temperature; §4.3 uses self-distinctness to argue the collapse is positional rather than a sampling change.
  • domain assumption Normalized exact-match token is the right granularity for answer identity.
    §3.1 and §B specify the normalization and junk guard; no embeddings or judges are used, and morphological variants are merged only for bare plurals.
  • domain assumption One clause wording per format represents the register.
    §5 explicitly flags that a clause-paraphrase column is the natural control; the gradient rules out some wording confounds but not all phrasings.

pith-pipeline@v1.3.0-alltime-deepseek · 9446 in / 13035 out tokens · 122967 ms · 2026-08-01T15:16:52.207585+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Structured Output Collapses Answer Diversity Across 44 Language Models." pith.science (2026). https://pith.science/paper/HSGNBVH3

@misc{pith2026260718476,
  author       = {Pith},
  title        = {Pith review of: Structured Output Collapses Answer Diversity Across 44 Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HSGNBVH3}},
  note         = {Machine review of arXiv:2607.18476}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

When a language model must choose one answer from a large space of equally valid options, a format clause -- "Reply with JSON only" -- changes which answer it chooses. We re-run the One-Word Census (arXiv:2607.12796): 31 wide-answer-space category prompts asked of 44 models, now with the reply requested in JSON -- no schema enforcement, no constrained decoding, only the request. Convergence deepens sharply: on the unconstrained "Pick a word" prompt the modal answer rises from 41% to 64% of the pool and distinct answers fall from 52 to 36; mean answer-choice surprisal drops from 1.80 to 1.58 bits. The tax is progressive: six of 44 models move individually (BH-FDR q=.10), all toward the mode, led by the most distinctive models, while the conformist floor is immobile. It is a sharpener, not a re-indexer -- the plain-chat modal answer survives in 28 of 31 categories. Defaults are register-indexed: a within-run re-sample (n=20) finds JSON shifts 53% of a model's stable chat defaults, mostly back to the crowd, and installs defaults absent from chat (Claude Fable 5 answers "cerulean" for colour 0% of the time in chat, 100% in JSON). Full-battery controls reveal a register gradient: compression is significant and specific to the answer-delivery formats models are trained to speak (JSON -0.22 bits, p=.0002; XML -0.19, p=.002), absent for YAML and CSV, and reversed for an arbitrary bracket wrapper (+0.13, p=.009) -- weighing the mechanism toward tool-use post-training. Enforcing the schema at the decoder (response_format) compresses no further than the request (-0.03 bits): the collapse lives in the model's response to the register, not the decoder. Structured output is how software consumes language models, and that surface is served by a measurably more homogeneous model than the chat surface on which models are evaluated, compared, and chosen.

Figures

Figures reproduced from arXiv: 2607.18476 by Tapan Parikh.

Figure 1
Figure 1. Figure 1: Answer-choice surprisal for all 44 models, open chat (gray dot) versus JSON (colored dot), sorted by the open-chat score. Blue marks a model that becomes more generic under JSON (31 of 44); amber marks one whose relative score rises (13 — mostly register-invariant models the collapsing field strands, §4.5). The distinctive models at the top slide farthest toward the conformist floor, which is itself immobi… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

12 extracted references · 8 linked inside Pith

  1. [1]

    Artificial Hivemind: The open-ended homogeneity of language models (and beyond).Advances in Neural Information Processing Systems (NeurIPS), 2025

    Liwei Jiang, Yuanjun Chai, Margaret Li, Mickel Liu, Raymond Fok, Nouha Dziri, Yulia Tsvetkov, Maarten Sap, Alon Albalak, and Yejin Choi. Artificial Hivemind: The open-ended homogeneity of language models (and beyond).Advances in Neural Information Processing Systems (NeurIPS), 2025. arXiv:2510.22954

    arXiv 2025

  2. [2]

    Understanding the effects of RLHF on LLM generalisation and diversity

    Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, et al. Understanding the effects of RLHF on LLM generalisation and diversity. InInternational Conference on Learning Representations (ICLR), 2024. arXiv:2310.06452

    Pith/arXiv arXiv 2024

  3. [3]

    let me speak freely

    Will Kurt. Say what you mean: A response to “let me speak freely”.https://blog.dottxt. ai/say-what-you-mean.html, 2024. 10

    2024

  4. [4]

    The format tax, 2026

    Ivan Yee Lee, Loris D’Antoni, and Taylor Berg-Kirkpatrick. The format tax, 2026. arXiv:2604.03616

    Pith/arXiv arXiv 2026

  5. [5]

    Constraint tax in open-weight llms: An empirical study of tool calling suppression under structured output constraints, 2026

    Fangzheng Li, Aimin Zhang, and Chen Lv. Constraint tax in open-weight llms: An empirical study of tool calling suppression under structured output constraints, 2026. arXiv:2606.25605

    Pith/arXiv arXiv 2026

  6. [6]

    Automata-based steering of large language models for diverse structured generation, 2025

    Xiaokun Luan, Zeming Wei, Yihao Zhang, and Meng Sun. Automata-based steering of large language models for diverse structured generation, 2025. arXiv:2511.11018

    arXiv 2025

  7. [7]

    The one-word census: Answer-choice conformity across 44 language models,

    Tapan Parikh. The one-word census: Answer-choice conformity across 44 language models,

  8. [8]

    The constraint tax: Measuring validity-correctness tradeoffs in structured outputs for small language models, 2026

    Jaideep Ray. The constraint tax: Measuring validity-correctness tradeoffs in structured outputs for small language models, 2026. arXiv:2605.26128

    Pith/arXiv arXiv 2026

  9. [9]

    Quantifying language models’ sensitivity to spurious features in prompt design, or: How i learned to start worrying about prompt formatting

    Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models’ sensitivity to spurious features in prompt design, or: How i learned to start worrying about prompt formatting. InICLR, 2024. arXiv:2310.11324

    Pith/arXiv arXiv 2024

  10. [10]

    Let me speak freely? a study on the impact of format restrictions on perfor- mance of large language models

    Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh-Yen Lin, Hung-yi Lee, and Yun-Nung Chen. Let me speak freely? a study on the impact of format restrictions on perfor- mance of large language models. InProceedings of EMNLP 2024: Industry Track, 2024. arXiv:2408.02442

    Pith/arXiv arXiv 2024

  11. [11]

    Name a[n]X

    Yiming Zhang, Harshita Diddee, Susan Holm, Hanchen Liu, Xinyue Liu, Vinay Samuel, Barry Wang, and Daphne Ippolito. NoveltyBench: Evaluating language models for humanlike diversity. InConference on Language Modeling (COLM), 2025. arXiv:2504.05228. A Prompts The battery is 31 single-turn prompts, frozen before data collection — the census stimulus unchanged...

    Pith/arXiv arXiv 2025

  12. [2026]

    arXiv:2607.12796; data and explorer athttps://github.com/tap2k/modelun

    Pith/arXiv arXiv