Pith. sign in

REVIEW 3 major objections 5 minor 18 references

Prompt robustness in LLMs is task-dependent: subjective belief and value questions are less stable under prompt changes than objective questions.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 05:51 UTC pith:SNSR72GV

load-bearing objection Solid empirical comparison showing subjective survey items are less prompt-stable than objective MCQs, with large type × perturbation interactions; useful methodological warning, moderate novelty. the 3 major comments →

arxiv 2607.05554 v1 pith:SNSR72GV submitted 2026-07-06 cs.CL cs.AI

Prompt Robustness Is Task-Dependent: Comparing Objective and Belief-Style Questions in LLM Evaluation

Sadia Kamal , Arefa Patwary , Anthony Marchiafava , Atriya Sen , Sagnik Ray Choudhury This is my paper
classification cs.CL cs.AI
keywords prompt robustnesslarge language modelssurvey evaluationobjective vs subjective questionsanswer consistencyprompt sensitivityvalue measurementoption order
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Survey-style evaluations of large language models often treat a single prompted answer as evidence of the model's values, politics, or beliefs. This paper tests whether that practice is reliable by comparing answer consistency on objective questions that have fixed correct answers against subjective questions that ask for opinions or values. Across four instruction-tuned model families and six datasets, subjective items produce lower consistency under meaning-preserving prompt variants such as rewording, formatting, label changes, and especially option reordering. Statistical models show significant effects of model, dataset type, and prompt category, plus a large interaction between dataset type and prompt category. The finding matters because a single survey score can no longer be treated as a stable model trait once the prompt form is allowed to vary.

Core claim

Prompt robustness depends jointly on question type, the kind of prompt change, and the model. Subjective survey-style questions yield systematically lower answer consistency than objective multiple-choice questions under the same families of meaning-preserving perturbations, and the gap is largest for answer-presentation changes such as option order.

What carries the argument

Answer consistency ratio for each item (the share of prompt variants that match the majority answer), analyzed with binomial generalized estimating equations that treat repeated variants of the same item as clustered data.

Load-bearing premise

The curated prompt changes are assumed to keep the intended task meaning the same for both factual and survey items, so any answer flip can be counted as a robustness failure rather than a legitimate reinterpretation.

What would settle it

A replication on the same or expanded datasets that finds no significant dataset-type main effect and no dataset-type by prompt-category interaction in the binomial GEE models would falsify the central claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A single prompted answer to a political or value question should not be treated as a stable model belief unless it survives controlled prompt variation.
  • Survey-style LLM evaluations need to report consistency across multiple prompt families, not only a final score.
  • Option-order and other answer-presentation perturbations should be treated as higher-priority robustness checks for subjective items than pure lexical paraphrases.
  • Robustness scores must be reported as a joint function of model, dataset, and prompt category rather than as a single global model property.
  • Objective and subjective tasks cannot be assumed to share the same sensitivity profile when designing or interpreting evaluations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Political and value benchmarks may need default protocols that randomize or balance option order and labels before any score is published.
  • The larger instability on subjective items may partly reflect models treating surface cues (order, labels, framing) as legitimate stance signals rather than pure noise.
  • Open-ended, non-forced-choice formats could either shrink or enlarge the objective-subjective gap and should be tested as a natural extension.
  • Model cards that claim value alignment should include multi-prompt consistency ranges rather than single-prompt point estimates.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that prompt robustness in LLMs is task-dependent: subjective/belief-style survey items are less answer-consistent under meaning-preserving prompt variants than objective multiple-choice items. Four instruction-tuned model families are evaluated on three Type-I datasets (MMLU, ARC, CulturalBench-Easy) and three Type-II datasets (Political Compass Test, ValueBench, WVS). A seven-category perturbation taxonomy (paraphrase, spelling noise, lexical substitution, logical equivalence, label substitution, format variation, option shuffling) is applied under temperature-0 forced-choice decoding with answer normalization. Consistency is defined per item as C_i = max_y n_i(y)/N_i. Binomial GEE models with item clustering yield significant main effects of model, dataset, dataset type, and prompt category, plus large dataset-type × prompt-category and model × prompt-category interactions. Option order produces the largest Type-I vs Type-II gap (0.485 vs 0.328).

Significance. If the result holds, it supplies a concrete, statistically supported caution against treating single-prompt survey responses as stable measures of LLM values or beliefs. The unified design across objective and subjective tasks, the explicit separation of semantic vs surface vs answer-presentation perturbations, and the clustered GEE analysis are genuine strengths relative to prior single-prompt or single-domain sensitivity studies. The work is useful for evaluation methodology and for any paper that reports political-compass or value-survey scores for LLMs. It does not claim a new training method or a closed-form theory; its contribution is empirical and methodological.

major comments (3)
  1. §3.2–3.3 and Table 1: The central causal reading of the Type-I/Type-II gap rests on the claim that every perturbation “preserves the intended task” equally for factual and survey items. No independent human validation (e.g., annotator agreement that paraphrase/logical-equivalent variants leave survey-item meaning fixed) is reported. Without that check, lower consistency on Type-II items under option-order or format changes (Table 2) can be read either as robustness failure or as legitimate re-interpretation of ambiguous survey wording. A short validation study or explicit limitation that the gap is descriptive rather than purely causal would make the claim load-bearing-safe.
  2. §3.1 and Limitations: Models are reported only at family level (Gemma, Llama, Mistral, Qwen) with no checkpoint IDs, parameter sizes, or instruction-tuning recipes. Because the model × prompt interaction is large (Tables 3–4, Appendix Table 6), family-level aggregation leaves open whether the robustness ranking is driven by size, post-training, or particular checkpoints. Exact model identifiers are needed for reproducibility and for interpreting the model-level claims in RQ3.
  3. §3.4 Eq. (1) and GEE setup: Consistency is defined relative to the majority answer among the observed variants for that item. When N_i is small or the answer distribution is multimodal (plausible for subjective items under option shuffling), the majority label itself is noisy and can inflate or deflate C_i. The paper should report the distribution of N_i per category, sensitivity of results to alternative anchors (e.g., a fixed canonical prompt), and the working correlation structure used in the GEE.
minor comments (5)
  1. Figure 1 heatmaps are informative but the color scale and numeric overlays become hard to read at small size; consider a supplementary table of the same means.
  2. Table 6 (Appendix) appears to reuse the same χ² values for Gemma as the within-dataset Table 5; clarify whether Gemma is the reference or whether the numbers are intentionally identical.
  3. §6 Limitations correctly notes forced-choice and deterministic decoding; a one-sentence pointer to how open-ended or temperature > 0 settings might change the gap would help readers.
  4. Minor wording: “Ismithdeen et al.” appears consistently; verify the author spelling against the cited Promptception paper.
  5. CulturalBench-Easy is justified as Type-I because of fixed keys; a brief note on how many items were retained after any filtering would aid replication.

Circularity Check

0 steps flagged

No circularity: purely empirical consistency measurement and GEE estimation; no derivation, prediction, or uniqueness claim reduces to its own inputs by construction.

full rationale

The paper reports an observational comparison of answer consistency under curated prompt perturbations across six external datasets and four model families. Consistency is defined directly from observed answer frequencies (Eq. 1: Ci = max_y n_i(y)/N_i) and is not fitted to force any Type-I/Type-II gap. Binomial GEE models estimate coefficients from the resulting binary match-to-majority data; the reported Wald tests (Tables 3–5) are ordinary statistical summaries of those data, not predictions derived from parameters that already encode the claimed effect. Perturbation taxonomy (Table 1, §3.2) and answer-normalization rules are design choices, not self-definitional identities that make the main result true by construction. No uniqueness theorem, ansatz, or load-bearing self-citation is invoked to forbid alternatives or to rename a known pattern as a new derivation. The work is therefore self-contained against external benchmarks and exhibits none of the six circularity patterns.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 3 invented entities

This is an empirical evaluation paper, not a theory derivation. Load-bearing content is mostly domain assumptions about what counts as a meaning-preserving prompt change and how consistency should be scored for items without gold labels. There are essentially no fitted free parameters that the central claim depends on beyond standard statistical estimation; the claim rests on experimental design choices and the curated perturbation taxonomy.

free parameters (2)
  • temperature / decoding policy = 0
    Fixed at temperature 0 (deterministic decoding) by design choice; not fitted to maximize the Type-I/Type-II gap, but the numerical consistency values depend on this choice.
  • GEE working correlation / clustering structure
    Binomial GEE with item-level clustering is a modeling choice that affects standard errors and Wald tests; the paper does not report sensitivity to alternative correlation structures.
axioms (5)
  • domain assumption A meaning-preserving prompt variant for a subjective survey item is well-defined by the same operations used for objective MCQ items (paraphrase, lexical substitution, logical equivalence, etc.).
    Invoked in §3.2–3.3 when the same perturbation taxonomy is applied to both Type-I and Type-II datasets; without this, answer flips on surveys need not count as robustness failures.
  • domain assumption Answer consistency against the majority response is a valid robustness metric for items that have no external gold answer.
    Defined in §3.4 equation (1); for Type-II items this replaces accuracy and is the quantity fed to the GEE.
  • domain assumption CulturalBench-Easy items are objective knowledge questions with fixed keys, not subjective preference items.
    Stated in §3.1 to justify placing CulturalBench in Type-I; misclassification would blur the dataset-type contrast.
  • domain assumption Forced-choice, single-label extraction with canonical remapping of labels/symbols does not itself create the Type-I vs Type-II robustness gap.
    §3.3 keeps a common answer format so groups differ mainly in content; if extraction artifacts dominate, the interaction claim weakens.
  • standard math Binomial GEE with Wald tests is an appropriate model for clustered binary consistency outcomes across related prompt variants.
    Cited Liang & Zeger 1986; used for all significance claims in §4.
invented entities (3)
  • Type-I vs Type-II dataset partition (objective MCQ vs subjective survey) independent evidence
    purpose: Primary experimental factor for testing whether prompt robustness depends on question type.
    The partition is a paper-level design construct built from existing datasets; independent evidence for the labels is the datasets’ original answer keys vs opinion scales, but the binary grouping is the authors’ analysis frame.
  • Seven-category prompt perturbation taxonomy (Table 1) no independent evidence
    purpose: Unified set of operations applied to both objective and subjective items to measure category-specific sensitivity.
    Curated from prior literature and adapted; not a physical entity, but a constructed experimental instrument whose coverage choices affect which interactions can be found.
  • Per-item consistency ratio C_i = max_y n_i(y) / N_i independent evidence
    purpose: Primary robustness metric feeding descriptive tables and GEE response coding.
    Standard majority-agreement style metric; defined in §3.4. Independent of any new physical postulate.

pith-pipeline@v1.1.0-grok45 · 14183 in / 3546 out tokens · 33267 ms · 2026-07-11T05:51:58.889214+00:00 · methodology

0 comments
read the original abstract

Survey-style evaluations of large language models often treat a prompted response as a measure of a model's values or beliefs. This assumption is particularly fragile when responses are read as evidence of political values, social attitudes, or beliefs. We ask whether prompt robustness differs between objective questions with fixed answers and subjective questions that ask for opinions or values. We evaluate four instruction-tuned model families on three objective datasets (MMLU, ARC, and CulturalBench) and three subjective datasets (Political Compass Test, ValueBench, and World Values Survey). For each question/statement, we apply multiple types of prompt changes, such as variations in wording, framing, and format, and measure whether the model gives the same answer across variants. Using a binomial generalized estimating equation, we find significant effects of model, dataset, prompt category, and their interactions. The dataset type effect is also significant, and the interaction between dataset type and prompt category is large. These results show that prompt robustness depends on the question type, the prompt change, and the model.

Figures

Figures reproduced from arXiv: 2607.05554 by Anthony Marchiafava, Arefa Patwary, Atriya Sen, Sadia Kamal, Sagnik Ray Choudhury.

Figure 1
Figure 1. Figure 1: Dataset-level mean consistency across perturbation categories for all four models. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references · 1 linked inside Pith

  1. [1]

    POSIX : A Prompt Sensitivity Index For Large Language Models

    Chatterjee, Anwoy and Renduchintala, H S V N S Kowndinya and Bhatia, Sumit and Chakraborty, Tanmoy. POSIX : A Prompt Sensitivity Index For Large Language Models. Findings of the Association for Computational Linguistics: EMNLP 2024. 2024. doi:10.18653/v1/2024.findings-emnlp.852

    doi:10.18653/v1/2024.findings-emnlp.852 2024

  2. [2]

    Beyond Prompt Brittleness: Evaluating the Reliability and Consistency of Political Worldviews in LLM s

    Ceron, Tanise and Falk, Neele and Bari \'c , Ana and Nikolaev, Dmitry and Pad \'o , Sebastian. Beyond Prompt Brittleness: Evaluating the Reliability and Consistency of Political Worldviews in LLM s. Transactions of the Association for Computational Linguistics. 2024

    2024

  3. [3]

    2025 , eprint=

    On Robustness and Reliability of Benchmark-Based Evaluation of LLMs , author=. 2025 , eprint=

    2025

  4. [4]

    2025 , eprint=

    Prompt Perturbations Reveal Human-Like Biases in Large Language Model Survey Responses , author=. 2025 , eprint=

    2025

  5. [5]

    2024 , isbn =

    Lunardi, Riccardo and La Barbera, David and Roitero, Kevin , title =. 2024 , isbn =. doi:10.1145/3627673.3680002 , pages =

    doi:10.1145/3627673.3680002 2024

  6. [6]

    Chatterjee, Anwoy and Renduchintala, H. S. V. N. S. Kowndinya and Bhatia, Sumit and Chakraborty, Tanmoy , booktitle =. 2024 , publisher =

    2024

  7. [7]

    International Conference on Learning Representations , year =

    Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting , author =. International Conference on Learning Representations , year =

  8. [8]

    Findings of the Association for Computational Linguistics: EMNLP 2025 , pages =

    Promptception: How Sensitive Are Large Multimodal Models to Prompts? , author =. Findings of the Association for Computational Linguistics: EMNLP 2025 , pages =. 2025 , publisher =

    2025

  9. [9]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics , pages =

    Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models , author =. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics , pages =. 2024 , publisher =

    2024

  10. [10]

    2024 , publisher =

    Wright, Dustin and Arora, Arnav and Borenstein, Nadav and Yadav, Srishti and Belongie, Serge and Augenstein, Isabelle , booktitle =. 2024 , publisher =

    2024

  11. [11]

    Advances in Neural Information Processing Systems , volume =

    Questioning the Survey Responses of Large Language Models , author =. Advances in Neural Information Processing Systems , volume =

  12. [12]

    2025 , eprint =

    Prompt Perturbations Reveal Human-Like Biases in Large Language Model Survey Responses , author =. 2025 , eprint =

    2025

  13. [13]

    Biometrika , volume =

    Longitudinal Data Analysis Using Generalized Linear Models , author =. Biometrika , volume =. 1986 , doi =

    1986

  14. [14]

    International Conference on Learning Representations , year =

    Measuring Massive Multitask Language Understanding , author =. International Conference on Learning Representations , year =

  15. [15]

    Think you have Solved Question Answering? Try

    Clark, Peter and Cowhey, Isaac and Etzioni, Oren and Khot, Tushar and Sabharwal, Ashish and Schoenick, Carissa and Tafjord, Oyvind , journal =. Think you have Solved Question Answering? Try

  16. [16]

    CulturalBench: a Robust, Diverse and Challenging Benchmark on Measuring the Lack of Cultural Knowledge of

    Chiu, Yu Ying and Jiang, Liwei and Lin, Bill Yuchen and Park, Chan Young and Li, Shuyue Stella and Ravi, Sahithya and Bhatia, Mehar and Antoniak, Maria and Tsvetkov, Yulia and Shwartz, Vered and Choi, Yejin , year =. CulturalBench: a Robust, Diverse and Challenging Benchmark on Measuring the Lack of Cultural Knowledge of. 2410.02677 , archivePrefix =

    Pith/arXiv arXiv

  17. [17]

    2024 , eprint =

    ValueBench: Towards Comprehensively Evaluating Value Orientations and Understanding of Large Language Models , author =. 2024 , eprint =

    2024

  18. [18]

    2022 , publisher =

    World Values Survey Wave 7 (2017-2022) Cross-National Data-Set , author =. 2022 , publisher =

    2017