Pith. sign in

REVIEW 3 major objections 23 references

Apparent cross-model value differences mix genuine disagreement with how sharply each model commits, and the access client further reshapes those measured values.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 13:31 UTC pith:FIGMMWT5

load-bearing objection Two real measurement confounds for LLM value audits—commitment sharpness and the deployment client—with careful design and honest scope limits. the 3 major comments →

arxiv 2607.10202 v1 pith:FIGMMWT5 submitted 2026-07-11 cs.LG

Two Confounds in Cross-Model Value Comparison: Response Determinism and the Access Harness

Hong-In Won , Jinseok Jang , Hyoseop Kim This is my paper
classification cs.LG
keywords language modelsvalue dispositionsresponse determinismaccess harnesscross-model comparisonforced-choice dilemmasalignment auditvalue elicitation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Cross-model comparisons treat single forced-choice answers on value dilemmas as evidence that language models hold distinct values. The paper shows this practice silently mixes two quantities: whether models lean opposite ways (central tendency) and how firmly each commits to its side (response determinism). The authors introduce no-rule dilemmas measured with counterbalanced repeated draws, an extremity index, and a decomposition that splits raw distance into direction flips versus same-side extremity. Across nine models, determinism varies widely once clients are matched, and correcting for it shrinks most apparent individuation while a few cross-family disagreements survive a strict test. Separately, the deployment client itself is a value-shaping layer: one subscription CLI moves a profile by 0.31, flips items, and inflates the flagship's apparent softness relative to the raw API, while an agent system prompt causally forces compliance on a model that otherwise refuses the forced choice. Audits that rank by single-draw distance therefore rank a confounded quantity and systematically under-flag non-committal models.

Core claim

Under single-draw measurement, an apparent cross-model value distance conflates a genuine difference in value direction with a difference in response determinism. A separation protocol of no-rule dilemmas, counterbalanced repeated forced choices, an extremity index, and a flip-versus-same-side decomposition separates the two; applied to nine models it shows client-matched determinism spans roughly 0.66–0.95 among engaging models and that correction shrinks apparent individuation while a few cross-family direction disagreements survive. A second orthogonal confound is the access harness: the same model through a subscription CLI versus raw API can shift mean choice probability by 0.31, flip i

What carries the argument

The determinism-corrected decomposition: for each item and model pair, classify divergence as a direction-flip (opposite sides of 0.5: genuine disagreement) or same-direction-more-extreme (labeled determinism), then report the genuine share of raw profile distance. Supported by an extremity index (mean |P−0.5|×2) and client-matched re-collection through raw APIs.

Load-bearing premise

The residual same-side distance is labeled a difference in commitment sharpness even though it could instead be a genuinely lukewarm value leaning on the same side.

What would settle it

Re-collect a denser multi-provider sample fully path-matched on raw APIs with a larger dilemma pool; if direction-flip components collapse near zero for all pairs, or if client-induced profile shifts and refusals disappear under controlled system prompts, the two-confound account fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 0 minor

Summary. The paper argues that single-draw cross-model value comparisons conflate a genuine difference in value direction (central tendency) with a difference in response determinism (commitment sharpness). It introduces a separation protocol—no-rule binary dilemmas, N=40 label-counterbalanced draws, an extremity index, and a flip-vs-same-side decomposition—and applies it to nine models across five providers. Determinism varies substantially (client-matched extremity roughly 0.66–0.95 among engaging models); correcting for the labeled same-side component shrinks apparent individuation, while a few cross-family direction-flips survive a ±0.1 dead-band. A second, orthogonal confound is the access harness: re-collection through raw APIs shows one subscription CLI shifts mean |ΔP| by ~0.31, flips four of eighteen items, and inflates the flagship’s apparent softness (0.34 CLI vs 0.66 raw API), while another provider’s client is clean. A white-box control establishes that an agent system prompt causally eliminates forced-choice refusal and shifts the profile (ΔP≈0.16), though it does not fully reproduce the proprietary CLI shift. The alignment implication is that single-draw audits rank a determinism- and harness-inflated quantity and can under-flag non-committal models.

Significance. If the results hold, this is a useful methods contribution to a growing practice of reading LM dilemma choices as individuated values. The dual-confound diagnosis is concrete and actionable: the flip/magnitude decomposition is a simple, reproducible correction that prior order-randomization work does not supply, and the harness result identifies a previously under-accounted value-shaping layer outside the model weights. Strengths include the counterbalanced repeated design, the letter-invariance non-engagement check, the robust-flip dead-band with bootstrap intervals, client-matched re-collection, the English language re-run, and especially the white-box system-prompt ablation that cleanly establishes compliance causation. Limitations (magnitude-label ambiguity, sparse per-provider sampling, 18-item pool, incomplete full path-matching) are largely stated by the authors. The paper does not need to settle whether models are individuated; the instrument that makes that judgment measurable is the contribution.

major comments (3)
  1. §4.2 explicitly labels the same-side residual as “determinism” and correctly notes it may instead be graded lukewarm ambivalence, with load-bearing claims resting on the direction-flip component. Nonetheless the abstract, Fig. 1, the §4.3 extremity table, and the narrative “determinism shares” still present magnitude/extremity as primary quantitative findings (e.g., 0.66–0.95; genuine-share percentages). Please demote quantitative magnitude shares more consistently to labeled/secondary status in the abstract and main figures, and lead with robust flip counts and client-matched orderings so readers cannot over-read the residual as proven pure determinism.
  2. §6.1–6.2 and Limitations: the clean demonstration that provider family is confounded with access client is for Anthropic (claude -p vs raw API; ΔP 0.17–0.31) versus OpenAI (codex clean). Gemini and Grok are asserted client-robust because direct API is the raw path, but are not tested against a second client; DeepSeek was measured through a harness client. The broader claim that “the harness is a value-shaping layer” is supported by the Anthropic effect and the white-box control, but the frozen-benchmark “provider family confounded with client” framing should be tightened to the pairs actually path-matched, and the incomplete full nine-model path-match should be stated earlier in §6 rather than only in Limitations.
  3. §4.4–4.5 and Limitations (item-population uncertainty): genuine-share and robust-flip claims rest on 18 items; bootstrap intervals are over responses (N=40), not over the item sample. The worked third-provider replication is a post-hoc n=5 slice. For the claim that “a few cross-family disagreements survive a strict test,” please add at least a leave-one-item-out or item-bootstrap sensitivity on the leading robust-flip pairs (e.g., Fable5–Gemini, Grok pairs), or clearly mark those counts as descriptive of this item set rather than as population estimates of value individuation.

Circularity Check

0 steps flagged

Empirical measurement paper with operational definitions and observed re-collections; no derivation reduces to its inputs by construction.

full rationale

The paper is a methods/measurement contribution, not a first-principles derivation. Extremity is defined operationally as mean |P(v1)−0.5|×2 from N=40 counterbalanced draws and then measured across models; the flip-vs-same-side split is a post-hoc classification of those observed P values relative to 0.5, not a fitted parameter re-presented as a prediction. The authors explicitly label the same-side residual rather than claiming a proven separation from lukewarm ambivalence, and load-bearing claims rest on direction-flips that survive a ±0.1 dead-band. The harness confound is established by re-collecting the same models through raw APIs versus deployment clients and by a white-box system-prompt control that eliminates refusals and shifts ΔP; these are experimental contrasts, not self-definitional identities. No self-citations appear among the load-bearing references, no uniqueness theorem is imported, and no ansatz is smuggled in via prior author work. Related work (order randomization, within-model decidedness, variance-components individuality) is acknowledged as adjacent rather than used as a circular premise. The instrument is self-contained against the reported measurements; residual scope limits (item set, provider density, labeled magnitude residual) are caveats of construct and sampling, not circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 2 invented entities

The paper is an empirical methods contribution. Its central claims rest on a small set of measurement definitions and design choices rather than free physical constants or new ontological entities. The main interpretive risk is the labeled attribution of same-side residuals to ‘determinism’ and the assumption that the chosen dilemma set and forced-letter format track the value construct used in alignment audits.

free parameters (3)
  • N draws per model-item = 40
    Fixed at 40; determines sampling interval (~±0.08 at P=0.5) and therefore which flips are called robust.
  • dead-band around 0.5 for robust flips = ±0.1
    ±0.1 (or margin exceeding sampling interval) chosen to demote near-boundary flips; directly affects which cross-family disagreements ‘survive’.
  • extremity index formula = mean |P-0.5|×2
    mean_i |P(v1)−0.5|×2; a conventional rescaling, not fitted, but the sole scalar used to claim substantial determinism variation.
axioms (4)
  • domain assumption No-rule, surface-symmetric forced binary dilemmas with counterbalanced labels yield a P(v1) that separates content-tracking value leaning from position-lock non-engagement.
    Stated in §3.1–3.2; underpins both the determinism index and the non-engagement detection for DeepSeek.
  • ad hoc to paper Opposite sides of 0.5 constitute genuine value-direction disagreement; same-side residual distance is attributable to (labeled) determinism rather than graded ambivalence.
    §4.2 decomposition rule; authors explicitly note the magnitude component is a label, not a proven separation.
  • domain assumption Holding model weights and backend fixed, profile differences between subscription CLI and raw API are caused by the access harness (system prompt and related client layers).
    §6 design; white-box control confirms system-prompt sufficiency for compliance and a partial shift, but not the full proprietary CLI shift.
  • domain assumption Schwartz basic values / Moral Foundations axes operationalized via the 18 workplace dilemmas are an adequate construct for the value dispositions compared in alignment audits.
    Construct grounding paragraph and appendix mapping; stimulus scope is acknowledged as benign workplace trade-offs only.
invented entities (2)
  • Determinism-corrected flip/magnitude decomposition of pairwise value distance no independent evidence
    purpose: Split apparent cross-model distance into direction-flip (genuine) vs same-side-more-extreme (labeled determinism).
    Core methodological object; no independent prior operationalization as a between-model distance confound.
  • Access harness as value-shaping layer independent evidence
    purpose: Name the deployment client (CLI/system prompt) as a distinct confound that moves measured value profiles.
    Observed large client-specific ΔP and causal system-prompt control; the entity is an interpretive framing of measured client effects rather than a new physical object.

pith-pipeline@v1.1.0-grok45 · 25313 in / 3356 out tokens · 33537 ms · 2026-07-14T13:31:25.219290+00:00 · methodology

0 comments
read the original abstract

Cross-model comparisons read divergence in value dispositions as evidence that language models hold individuated values. Under single-draw measurement this conflates two quantities: a difference in central tendency (a genuine value difference) and a difference in response determinism (how sharply a model commits to a forced choice). We introduce a separation protocol -- no-rule value dilemmas with counterbalanced, repeated forced-choice measurement and a determinism index -- and a determinism-corrected decomposition that splits an apparent cross-model distance into a direction-flip component (genuine disagreement) and a same-side-more-extreme component we label determinism. Across nine models, determinism varies substantially (0.66-0.95 among engaging models); whether it is a per-model trait or tracks provider and scale is a question our method makes measurable but our sample leaves open. Correcting for determinism shrinks apparent individuation, while a few cross-family disagreements survive a strict test. We then isolate a second confound: the access harness serving each model. Re-collecting the same models through raw provider APIs, we find the deployment client shifts a model's value profile substantially and client-specifically: one subscription CLI moves a profile by 0.31, flips four of eighteen items, and inflates the flagship's apparent softness (0.34 via CLI vs 0.66 via raw API), whereas another provider's client is clean, confounding provider family with access client. The harness is a value-shaping layer: a base model that refuses one-in-ten forced choices is made compliant by an agent system prompt, established causally in a white-box control. An audit ranking models by single-draw value distance thus ranks a determinism-inflated quantity, confounded further by the client used. We contribute the decomposition and identify the deployment harness as a distinct value confound.

Figures

Figures reproduced from arXiv: 2607.10202 by Hong-In Won, Hyoseop Kim, Jinseok Jang.

Figure 1
Figure 1. Figure 1: Value-choice extremity (mean |P(v1) − 0.5| × 2 over 18 dilemmas) by model; as-deployed clients vs. client-matched raw API for the four Anthropic models. claude -p systematically compresses extremity relative to the underlying model, most severely for Opus (0.34 → 0.66) and Sonnet (0.48 → 0.79); DeepSeek is excluded from the ranking as position-locked/non-engaging. Here v1 is the designated value side of ea… view at source ↗
Figure 2
Figure 2. Figure 2: Decomposing cross-model disagreement into genuine direction-flips (opposite quadrants) [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The access harness as a value-shaping layer. The same model (fixed weights), called [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The apparent determinism gap is a client-harness artifact, not a model-family one: mean [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 20 linked inside Pith

  1. [1]

    Questioning the survey responses of large language models, 2023

    Ricardo Dominguez-Olmedo, Moritz Hardt, and Celestine Mendler-D¨ unner. Questioning the survey responses of large language models, 2023. arXiv:2306.07951

    Pith/arXiv arXiv 2023

  2. [2]

    Nino Scherrer, Claudia Shi, Amir Feder, and David M. Blei. Evaluating the moral beliefs encoded in LLMs, 2023. arXiv:2307.14324

    Pith/arXiv arXiv 2023

  3. [3]

    ValueBench: Towards comprehensively evaluating value orientations and understanding of large language models,

    Yuanyi Ren, Haoran Ye, Hanjun Fang, Xin Zhang, and Guojie Song. ValueBench: Towards comprehensively evaluating value orientations and understanding of large language models,

  4. [4]

    Value portrait: Assessing language models’ values through psychometrically and ecologically valid items, 2025

    Jongwook Han, Dongmin Choi, Woojung Song, Eun-Ju Lee, and Yohan Jo. Value portrait: Assessing language models’ values through psychometrically and ecologically valid items, 2025. arXiv:2505.01015

    Pith/arXiv arXiv 2025

  5. [5]

    DailyDilemmas: Revealing value preferences of LLMs with quandaries of daily life, 2024

    Yu Ying Chiu, Liwei Jiang, and Yejin Choi. DailyDilemmas: Revealing value preferences of LLMs with quandaries of daily life, 2024. arXiv:2410.02683

    Pith/arXiv arXiv 2024

  6. [6]

    Every act has its price: Compressed moral composition in frontier LLMs, 2026

    Weijia Zhang, Ruiqi Chen, Yunze Xiao, and Weihao Xuan. Every act has its price: Compressed moral composition in frontier LLMs, 2026. arXiv:2606.11232

    Pith/arXiv arXiv 2026

  7. [7]

    Kaufman, and Mark Dredze

    Jen-tse Huang, Jiantong Qin, Xueli Qiu, Sharon Levy, Michelle R. Kaufman, and Mark Dredze. Knowing but not doing: Convergent morality and divergent action in LLMs, 2026. arXiv:2601.07972

    arXiv 2026

  8. [8]

    deterministic

    Berk Atil, Sarp Aykent, Alexa Chittams, Lisheng Fu, Rebecca J. Passonneau, Evan Radcliffe, Guru Rajan Rajagopal, Adam Sloan, Tomasz Tudrej, Ferhan Ture, Zhe Wu, Lixinyu Xu, and Breck Baldwin. Non-determinism of “deterministic” LLM settings, 2024. arXiv:2408.04667

    Pith/arXiv arXiv 2024

  9. [9]

    Are large language models consistent over value-laden questions?, 2024

    Jared Moore, Tanvi Deshpande, and Diyi Yang. Are large language models consistent over value-laden questions?, 2024. arXiv:2407.02996

    Pith/arXiv arXiv 2024

  10. [10]

    Lee, Yeongheon Lee, and Hyunsoo Cho

    Bruce W. Lee, Yeongheon Lee, and Hyunsoo Cho. Inertia in moral and value judgments of large language models, 2024. arXiv:2408.09049

    Pith/arXiv arXiv 2024

  11. [11]

    Dorit Hadar-Shoval, Kfir Asraf, Yonathan Mizrachi, Yuval Haber, and Zohar Elyoseph. Assessing the alignment of large language models with human values for mental health integration: Cross-sectional study using Schwartz’s theory of basic values.JMIR Mental Health, 2024. PMC11040439. 22

    2024

  12. [12]

    Valentin Kriegmair and Dirk U. Wulff. Machine individuality: Separating genuine idiosyncrasy from response bias in large language models, 2026. arXiv:2604.16755

    Pith/arXiv arXiv 2026

  13. [13]

    Measuring and controlling instruction (In)stability in language model dialogs

    Kenneth Li et al. Measuring and controlling instruction (In)stability in language model dialogs. InConference on Language Modeling (COLM), 2024. arXiv:2402.10962; COLM 2024

    Pith/arXiv arXiv 2024

  14. [14]

    Examining identity drift in conversations of LLM agents, 2024

    Junhyuk Kim et al. Examining identity drift in conversations of LLM agents, 2024. arXiv:2412.00804

    Pith/arXiv arXiv 2024

  15. [15]

    Towards understanding sycophancy in language models, 2023

    Mrinank Sharma et al. Towards understanding sycophancy in language models, 2023. arXiv:2310.13548; ICLR 2024

    Pith/arXiv arXiv 2023

  16. [16]

    Sycophancy to subterfuge: Investigating reward-tampering in large language models, 2024

    Carson Denison et al. Sycophancy to subterfuge: Investigating reward-tampering in large language models, 2024. arXiv:2406.10162

    Pith/arXiv arXiv 2024

  17. [17]

    Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting, 2024

    Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting, 2024. arXiv:2310.11324

    Pith/arXiv arXiv 2024

  18. [18]

    Yunbei Zhang, Janet Wang, Yingqiang Ge, Weijie Xu, Jihun Hamm, and Chandan K. Reddy. Stop comparing LLM agents without disclosing the harness, 2026. arXiv:2605.23950

    Pith/arXiv arXiv 2026

  19. [19]

    Harness-Bench: Measuring harness effects across models in realistic agent workflows, 2026

    Yilun Yao, Xinyu Tan, Chao-Hsuan Liu, Yaoming Li, Zhengyang Wang, Wenhan Yu, Zhewen Tan, Yuxuan Tian, Guangxiang Zhao, Lin Sun, Xiangzheng Zhang, and Tong Yang. Harness-Bench: Measuring harness effects across models in realistic agent workflows, 2026. arXiv:2605.27922

    Pith/arXiv arXiv 2026

  20. [20]

    Political compass or spinning arrow? towards more meaningful evaluations for values and opinions in large language models, 2024

    Paul R¨ ottger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Rose Kirk, Hinrich Sch¨ utze, and Dirk Hovy. Political compass or spinning arrow? towards more meaningful evaluations for values and opinions in large language models, 2024. arXiv:2402.16786

    Pith/arXiv arXiv 2024

  21. [21]

    Mind the gap: How elicitation protocols shape the stated-revealed preference gap in language models, 2026

    Pranav Mahajan, Ihor Kendiukhov, Syed Hussain, and Lydia Nottingham. Mind the gap: How elicitation protocols shape the stated-revealed preference gap in language models, 2026. arXiv:2601.21975

    Pith/arXiv arXiv 2026

  22. [22]

    When choices become risks: Safety failures of large language models under multiple-choice constraints, 2026

    Yuheng Chen, Zhiyu Wu, Bowen Cheng, and Tetsuro Takahashi. When choices become risks: Safety failures of large language models under multiple-choice constraints, 2026. arXiv:2604.16916

    Pith/arXiv arXiv 2026

  23. [23]

    The non-determinism of small LLMs: Evidence of low answer consistency in repetition trials of standard multiple-choice benchmarks, 2025

    Claudio Pinhanez, Paulo Cavalin, Cassia Sanctos, Marcelo Grave, and Yago Primerano. The non-determinism of small LLMs: Evidence of low answer consistency in repetition trials of standard multiple-choice benchmarks, 2025. arXiv:2509.09705. 23

    Pith/arXiv arXiv 2025