{"id":"286a6e11-c948-4535-88d8-6704d255ed71","arxiv_id":"2506.10095","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LLM outputs drift measurably when prompts are reworded without changing meaning, instruction-tuned models drift less, and the new PBSS score quantifies this drift using embedding distance.","lead":"The paper measures how much an AI chatbot's answers change when the same question is reworded without changing meaning, a drift it calls prompt variance. It reports that newer, instruction-tuned models drift less, and offers a simple embedding score, PBSS, for flagging unstable responses before deployment.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PBSS conflates prompt-form drift with decoding noise: no same-prompt resampling baseline supports Section 5.9's 'this drift is not noise' claim.","rationale":"The reader identified the same load-bearing gap: no same-prompt, same-temperature repeated-generation baseline. This is the correct central concern because the paper's headline contribution depends on Section 5.9's assertion that drift is structured and model-specific rather than incidental sampling variability. At T=1.3, decoding is stochastic, and a single generation per prompt cannot separate prompt-form effects from sampling entropy. The paper's repeated claim that patterns persist across temperatures is not a substitute: entropy itself is a property that persists across temperature levels, and the model ordering in the CDFs aligns with known entropy and response-stereotypy differences between small legacy models and instruction-tuned systems. The three-encoder consistency is real evidence that the measured differences are not an artifact of one embedding model, but it does not address the sampling-noise confound because all SBERT variants respond similarly to text-level stochastic variation. Secondary issues—post-hoc model tiers tested on the same data, the placeholder code link, and inconsistent count statements—reinforce the unsupported status but are not needed for the verdict. The proposed same-prompt resampling test would settle whether the central claim survives; absent that test, REJECT remains the appropriate verdict.","tokens_in":28292,"tokens_out":2967,"duration_ms":35949,"concrete_test":"For each of the five core models and each task, generate N=30 samples for a single identical prompt (e.g., variant V1) at T=1.3, and compute PBSS across same-prompt output pairs using all three SBERT encoders. Then compute the same metric across the 15 paraphrase variants. If the median or 90th percentile of same-prompt PBSS lies within the interquartile range of cross-prompt PBSS for the legacy models (GPT-2, GPT-Neo), sampling noise alone can explain the reported drift and the 'not noise' claim fails. Report the effect size and distribution overlap for each model and model tier.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that PBSS reveals structured, model-specific sensitivity to the token form of instructions—requires that measured output divergence be attributed to the paraphrase rather than to decoding stochasticity. Appendix B.1 samples each prompt exactly once per temperature (T=0.2 and T=1.3); no same-prompt repeated-generation condition is reported anywhere. At T=1.3, the PBSS score D(pi,pj)=1−cos(SBERT(f(pi)), SBERT(f(pj))) conflates prompt-induced drift with sampling noise. The observed ordering—GPT-2/GPT-Neo showing high drift, instruction-tuned models low drift—is exactly the ordering of sampling entropy and response stereotypy; the same CDF separation would likely appear if prompts were held fixed and repeated generations were compared. Section 5.9 argues that 'temperature instability' is absent, but comparing T=0.2 vs T=1.3 only shows that the model ordering is stable across entropy levels; it does not show that within T=1.3 the cross-prompt drift exceeds same-prompt sampling drift. The t-SNE clusters by model in Figures 3–4 also do not distinguish prompt sensitivity from model-specific output style or entropy. Because the load-bearing inference is not separable from decoding stochasticity in the current data, the signature claim that drift is not noise is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces Prompt-Based Semantic Shift (PBSS), a cosine-distance measure between SBERT embeddings of a model's outputs to semantically equivalent paraphrased prompts. The authors evaluate five to ten LLMs on ten constrained tasks with fifteen prompt variants per prompt set at two temperatures, and report model-specific drift patterns, a behavioral phase boundary between instruction-tuned and legacy models, and robustness across three SBERT encoders. They propose PBSS as a lightweight QoS screening diagnostic for high-stakes deployment.","tokens_in":28479,"tokens_out":3703,"duration_ms":44719,"significance":"If the central claim holds, PBSS would be a cheap, black-box diagnostic for surface-form sensitivity, with potential value for clinical and legal deployment screening. The paper has strengths: it releases code, prompts, and outputs; it reports Kruskal-Wallis statistics across three encoders and two temperatures; and the descriptive ordering (instruction-tuned models more stable than legacy models under rephrasing) is plausible from the CDFs. However, the signature inference that measured drift is 'not noise' (Section 5.9) currently lacks a same-prompt repeated-generation control, so the paper's main contribution is not yet established.","major_comments":[{"comment":"The claim that 'this drift is not noise' is not supported by the experimental design. PBSS compares two different prompts, each sampled once per temperature; at T=1.3, sampling noise alone can shift sentence embeddings substantially, and the paper reports no same-prompt repeated-generation baseline anywhere. Without such a baseline, the observed cross-prompt divergence cannot be separated from decoding stochasticity, and the small-vs-large model ordering is confounded with sampling entropy and response stereotypy. Please add a condition where the same prompt is generated multiple times at T=1.3 and compare the within-prompt embedding distance to the cross-prompt PBSS distribution.","section":"§5.9 and Appendix B.1"},{"comment":"The 'behavioral phase boundary' between alignment-aware and legacy architectures is not established because model scale and alignment stage are fully confounded. GPT-3.5 is both the largest and most heavily instruction-tuned model, while GPT-2 and GPT-Neo are both the smallest and least aligned. The model tiers in Table 1 are also assigned post hoc after observing the CDF ordering, and the cutoffs (e.g., SmolLM-360M as Small, Phi-2 2.7B as Medium, MythoMax-13B as Large) appear arbitrary. A matched comparison—same architecture with and without instruction tuning, or a regression with parameter count as a covariate—is needed before the drift patterns can be attributed to alignment stage rather than scale.","section":"§5.5 and §5.7"},{"comment":"The semantic-equivalence validation uses all-mpnet-base-v2, which is also one of the three encoders used to compute PBSS. Because prompt variants are selected or filtered for high similarity under this encoder, the measured drift for that encoder is not independent of the selection criterion. The authors should repeat the prompt validation with an independent embedding model or with human ratings, and confirm that the PBSS findings survive when the validation encoder is excluded from the analysis.","section":"§4.2"},{"comment":"The interpretation of PBSS as capturing 'tone, structure, or rhetorical differences' is asserted rather than validated. Cosine distance in SBERT space is sensitive to many output properties, including length, lexical overlap, and topic. The paper provides no evidence that the measured distances align with the claimed behavioral axis, nor any comparison of PBSS to lexical diversity measures, reference-free text similarity metrics, or human judgments of rhetorical stability. Adding such a validation would strengthen the claim that PBSS measures behavioral drift rather than generic output variability.","section":"§3.2 and Appendix A"}],"minor_comments":[{"comment":"The statement 'The 750 unique instructions per model yield 300 distinct prompt pairs' is numerically inconsistent with the stated design: 50 prompt sets (10 tasks x 5 origins) each contribute C(15,2)=105 pairs, giving 5,250 pairs. Please clarify the counting or correct the number.","section":"§5.9"},{"comment":"Several figure captions contain the typo 'T emp' instead of 'Temp', and the manuscript inconsistently uses 'S-BERT', 'SBERT', and 'S-BERT variants'. Please standardize notation throughout.","section":"Figures 5, 6, 25, 26"},{"comment":"The main text focuses on five core models while Appendix C introduces five additional models and 'alt case' figures; please clarify in the main text which analyses are primary and which are supplementary, and unify the model naming (e.g., 'Mistral-7B-instruct-v0.2' vs 'Mistral-7B-Instruct-v0.2').","section":"Appendix C"},{"comment":"The Kruskal-Wallis results report H statistics and p-values but no effect sizes; for the extremely large H values, a rank-based effect size such as epsilon-squared would help readers judge the practical magnitude of the between-group differences.","section":"Table 3"},{"comment":"The sentence 'This has significant implications...' overstates the direct clinical utility of an embedding-distance diagnostic; please temper the claim or provide a specific validation scenario in a clinical NLP setting.","section":"§6.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of cs.CL, and the code/data release is a positive feature. The GitHub link is currently a placeholder ('our github link'); a working repository should be provided before publication. The citation list is adequate, but some citations are used loosely (e.g., Zhou et al. 2024 for 'larger models become less predictable'); please verify that the cited sources support the specific claims. The central fix—adding a same-prompt repeated-generation baseline—is feasible and should be prioritized. No concerns about novelty disclosure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's descriptive core is probably right: across ten constrained tasks, older small models show wider embedding-space divergence under paraphrased prompts, while newer instruction-tuned models are tighter. That ordering is consistent across three SBERT encoders and two temperatures. The evaluation is a real systematic effort—multi-task, multi-model, multi-encoder—and the prompt-sanity layer (manual review, syntax filtering, semantic thresholding) is good practice. PBSS itself is simple cosine distance over SBERT embeddings, so the novelty is in the application, not the metric, but the application is legitimate.\n\nThe load-bearing claim is not supported. Section 5.9 says 'this drift is not noise,' but there is no same-prompt repeated-generation condition anywhere in the appendices. Each prompt is generated once per temperature. At T=1.3, sampling entropy alone can move sentence embeddings substantially. The observed ordering—GPT-2/GPT-Neo high drift, instruction-tuned low drift—is exactly the ordering of sampling entropy and response stereotypy. Comparing T=0.2 vs T=1.3 only shows the model ordering is stable across entropy levels; it does not show that cross-prompt drift at T=1.3 exceeds same-prompt sampling drift. That missing baseline is the difference between a quality-of-service diagnostic and a description of decode noise.\n\nThe causal talk about tokenization and decoding is also asserted rather than tested: models differ jointly in scale, training data, and alignment, and no tokenizer or decoding variable is manipulated. The Kruskal-Wallis tiers are post hoc, defined after looking at PBSS distributions and tested on the same data; 'Combined' encoder rows pool three correlated encoders and overstate significance. The code link is a placeholder and the paper's own counts disagree (4,500 vs 'over 13,000' prompt-response cases per model). These are fixable, but they make the paper rougher than it should be.\n\nOne thing I'd push back on: the shared-encoder circularity is real but not fatal. all-mpnet-base-v2 validates prompt equivalence, and it is also one of the three encoders used to measure output drift—but the validation is about the prompt set, not the model outputs. It is worth flagging, not disqualifying.\n\nWho this is for: LLM operators and evaluation researchers thinking about paraphrase stability as a QoS screen. It deserves a serious referee because the descriptive pattern and the evaluation infrastructure are worth engaging. The referee should require a same-prompt resampling baseline, published prompts and code, pre-registered or clearly descriptive tiers, and a restrained interpretation. My verdict is skeptical, not dismissive.","headline":"A useful descriptive sweep undermined by a missing control: the claim that PBSS drift is 'not noise' is unsupported because no same-prompt repeated-generation baseline exists.","tokens_in":29111,"tokens_out":3031,"would_cite":false,"duration_ms":37579,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs drift under rewording even when meaning is fixed.","keywords":["prompt variance","behavioral drift","semantic embeddings","quality of service","LLM evaluation stability","token-level sensitivity","instruction-tuned models","paraphrase robustness"],"falsifier":"Run each of the 15 paraphrase prompts through the same model many times at the same temperature and compute the PBSS distance between repeated outputs of the identical prompt; if that repeat-distance distribution overlaps the between-paraphrase distance distribution, the reported drift could be sampling noise rather than prompt-induced sensitivity.","tokens_in":27957,"feed_emoji":"🔀","tokens_out":6643,"duration_ms":66007,"temperature":0.7,"pith_summary":"This paper tries to establish that large language models systematically change their behavior when a prompt is reworded in ways that keep its meaning identical—what it calls prompt variance—and that this drift is structured and model-specific rather than random noise. It proposes a diagnostic, Prompt-Based Semantic Shift (PBSS), that measures how far apart the model's answers to semantically equivalent prompts are in embedding space. Across ten tasks, five to ten models, and three embedding encoders, the paper finds a sharp divide: instruction-tuned models stay stable under rephrasing, while older smaller models drift widely. If true, this gives deployers a cheap, black-box way to screen models for behavioral reliability before putting them in high-stakes settings such as clinical or legal workflows, where a rephrased question could change the answer.","feed_headline":"LLMs drift under rewording even when meaning is fixed","feed_subtitle":"Embedding distances expose which models stay stable when prompts are rephrased, a cheap deployment screen","key_machinery":"The load-bearing object is Prompt-Based Semantic Shift (PBSS), defined as the cosine distance between sentence-embedding vectors of a model's outputs to two paraphrased prompts that share intent. Pairwise PBSS scores are assembled into a drift matrix and a cumulative distribution function, with z-score heatmaps isolating outlier prompt variants; the steepness and location of the CDF becomes the model's stability fingerprint. The mechanism that carries the argument is the comparison of these fingerprints across models, temperatures, tasks, and three independent embedding encoders, which the paper argues isolates model-internal sensitivity to token-level surface form.","core_discovery":"On the paper's own terms, the central discovery is that surface-level rephrasing—different token sequences that preserve intent—produces measurable, consistent response shifts in LLMs, and the size of that shift is a stable property of the model, not of the prompt set or the embedding encoder. Grouping models by parameter scale and alignment stage, PBSS cumulative distributions separate cleanly into behavioral tiers: GPT-3.5, LLaMA-2, and Mistral cluster tightly with low drift, while GPT-2 and GPT-Neo show broad dispersion, a divide the paper calls a phase boundary between alignment-aware and legacy architectures. The claim that this is structure rather than noise rests on three consistency observations: the same model rankings appear across three different sentence-embedding encoders, the CDF shapes survive the change in decoding temperature from 0.2 to 1.3, and Kruskal–Wallis tests return highly significant differences between model tiers.","pith_inferences":["The 'phase boundary' may partly be a decoding-entropy effect: the legacy small models in the sample are also highly stochastic samplers at temperature 1.3, so a same-prompt repetition baseline would be needed to separate prompt sensitivity from sampling variance; building that baseline is a natural next experiment.","If the drift rankings are stable across embedding encoders, the same experimental design could be used as a cheap calibration probe for testing whether alignment techniques monotonically reduce prompt sensitivity, without needing downstream task labels.","A testable extension: check whether PBSS drift on benign rephrasing predicts refusal-rate variance or jailbreak success on the same prompts; a positive correlation would turn the diagnostic into a security screening tool.","The paper's encoder-agnostic 'semantic resonance' suggests PBSS might measure something about the model's hidden representations rather than its output distribution; probing internal activations for the same paraphrases could reveal whether the drift originates in tokenization or in later decoding decisions."],"forward_implications":["PBSS can act as a pre-deployment quality-of-service screen: a model whose PBSS drift is high on benign rephrasings can be flagged before it reaches clinical, legal, or financial workflows.","Model rankings by drift are consistent across encoder choices and temperatures, so the diagnostic does not depend on a particular embedding model to expose the behavioral tiers.","The observed phase boundary implies that instruction-tuned models have acquired rhetorical consistency as a side effect of alignment, making drift resistance a property that could be tracked over successive fine-tuning generations.","Because jailbreaking is reframed as an extreme case of prompt-induced drift, high-PBSS prompt regions become candidate early-warning zones for jailbreak susceptibility, a direction the paper explicitly opens.","For safety-sensitive users, the method gives a concrete number—mean pairwise drift per model—that can be monitored over model updates to detect regressions in behavioral stability."],"supporting_citations":[{"why":"Supplies the sentence-embedding encoder used to project outputs and define PBSS cosine distances.","marker":"(Reimers and Gurevych, 2019)"},{"why":"Provides a contrastive embedding model used as an additional encoder variant in the robustness checks.","marker":"(Gao et al., 2021)"},{"why":"Frames prompts as capability triggers, motivating the paper's focus on prompt-form sensitivity.","marker":"(Reynolds and McDonell, 2021)"},{"why":"Questions whether models truly parse instruction semantics, supporting the idea that surface form can drive behavior.","marker":"(Webson and Pavlick, 2022)"},{"why":"Documents that larger, more instructable models can become less reliable, a baseline the paper's tier ordering extends.","marker":"(Zhou et al., 2024)"},{"why":"Defines the instruction-tuning-with-human-feedback alignment stage that anchors the paper's low-drift tier.","marker":"(Ouyang et al., 2022)"}],"fun_headline_variants":["Rephrasing LLMs: same meaning, different outputs","Token-level rewording exposes stable LLM drift","Model drift under rephrasing is a stable fingerprint","LLM stability under rephrasing: a phase boundary","Same intent, different tokens: LLMs drift predictably"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's structural claims stand on the assumption that the measured output divergence is caused by the prompt paraphrase and not by the model's own random sampling, since identical prompts are never repeated to measure a noise floor.","fun_headline_variants_meta":{"raw":{"variants":["Rephrasing LLMs: same meaning, different outputs","Token-level rewording exposes stable LLM drift","Model drift under rephrasing is a stable fingerprint","LLM stability under rephrasing: a phase boundary","Same intent, different tokens: LLMs drift predictably"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000289,"raw_usage":{"total_tokens":1640,"prompt_tokens":838,"completion_tokens":802,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":722}},"tokens_in":454,"tokens_out":802,"duration_ms":8689,"temperature":1.0,"reasoning_tokens":722,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:36:48.101688+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run each of the 15 paraphrase prompts through the same model many times at the same temperature and compute the PBSS distance between repeated outputs of the identical prompt; if that repeat-distance distribution overlaps the between-paraphrase distance distribution, the reported drift could be sampling noise rather than prompt-induced sensitivity.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Questions whether models truly parse instruction semantics, supporting the idea that surface form can drive behavior."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents that larger, more instructable models can become less reliable, a baseline the paper's tier ordering extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the instruction-tuning-with-human-feedback alignment stage that anchors the paper's low-drift tier."}],"review_version":1}