{"id":"a565283a-bc95-4c0c-bf36-47209f826284","arxiv_id":"2504.12180","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Minor prompt rewording produces statistically significant shifts in GPT-4o mini's Spanish sentiment labels, yet overall agreement between prompts stays between 92% and 98%.","lead":"Researchers asked GPT-4o mini to classify 100,000 Spanish-language YouTube comments about four Latin American presidents as positive, negative, or neutral, using ten slightly different prompts. They found that most small wording changes caused statistically significant label differences, and that unstructured prompts produced more off-topic responses.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The chi-square test is applied to marginal label counts, not to per-comment agreement, so with N=100,000 the reported p-values cannot support 'major differences' while the coincidence matrix shows 92–98% agreement; a paired test and effect size are needed.","rationale":"The reader's conditional verdict identifies the same core weakness: the chi-square test ignores the paired structure of the data and the extremely small p-values are driven by the large sample size rather than by the practical magnitude of the differences. My reading adds a sharper formulation: the chi-square test is not merely misusing independence; it is testing marginal label distributions, whereas the paper's stated claim is about per-comment classification differences. The coincidence matrix already provides the correct paired information, showing 92–98% agreement, which directly undercuts the phrase 'major differences.' The paper has genuine strengths: a large empirical corpus, a reproducible setup, and a useful cautionary message about LLM sensitivity to prompt wording. The fix is statistical rather than conceptual: use paired tests and report effect sizes. With that revision, the conclusion should be softened but the paper remains acceptable. Therefore I do not recommend changing the reader's conditional verdict.","tokens_in":17731,"tokens_out":7557,"duration_ms":82094,"concrete_test":"For each pair of prompts, form the paired contingency table of labels (positive/negative/neutral, treating 'inconsistent' separately) across the 100,000 comments and run the Stuart-Maxwell test of marginal homogeneity (or McNemar's test on collapsed positive-vs-rest for each class), together with Cohen's kappa, Gwet's AC1, and the asymmetric disagreement rate with bootstrap confidence intervals. Then compare these results with Table 6. If most pairs remain significant but kappa is at least 0.92, the conclusion should be revised to 'statistically detectable but small differences'; if some pairs that Table 6 marks significant are not significant under the paired test, the list of sensitive prompt pairs shrinks accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing weakness is in the confirmatory analysis ('Análisis confirmatorio'). The paper states that the chi-square test checks whether one prompt classifies each comment similarly to another prompt, but the test is computed on the aggregate label counts (positive/negative/neutral/inconsistent) for each prompt. A marginal chi-square cannot detect per-comment disagreement: two prompts can have identical marginal distributions while assigning opposite labels to every comment. The 100,000 observations are paired (the same comments are labeled by all prompts), yet the procedure explicitly assumes independence ('Este procedimiento asume que cada resultado de calificación solicitado por un prompt es independiente de otro'). This assumption is false and is not innocuous: at N=100,000, even a 1–2% aggregate shift yields p-values far below 0.05, so the test chiefly measures sample size, not the magnitude of prompt sensitivity. The paper's own coincidence matrix (Table 4) is the relevant paired statistic, with 0.92–0.98 exact agreement, but it is never turned into an inferential test or an effect size. Thus 'major differences' conflates statistical significance with practical significance; the evidence supports at most small, statistically detectable instabilities.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper tests the hypothesis that subtle variations in prompt wording (lexical, syntactic, modal-pragmatic, and unstructured) do not produce significant differences in sentiment polarity classification when using GPT-4o mini. The authors collected 100,000 Spanish YouTube comments about four Latin American presidents and classified each comment ten times, once per prompt, at temperature zero. Exploratory analyses include PCA of prompt embeddings, a pairwise coincidence matrix of exact label agreement, and Levenshtein distances between prompts. Confirmatory analysis uses chi-square tests on the aggregate label counts for each prompt. The reported coincidence matrix shows 92–98% exact agreement between prompt pairs, yet the chi-square tests reject the null hypothesis of equal distributions for nearly all pairs, with only prompts 1 and 7 not significantly different. The paper concludes that subtle prompt changes significantly affect classifications and that unstructured prompts increase the frequency of inconsistent (hallucinated) outputs.","tokens_in":17926,"tokens_out":4278,"duration_ms":43169,"significance":"If the central claim were properly supported, the paper would make a useful contribution to computational social science and applied NLP by quantifying prompt sensitivity in a large, non-English, real-world corpus. Strengths include the large Spanish dataset, several linguistically motivated prompt manipulations, a controlled temperature setting, and a public code repository. The exploratory coincidence matrix and the documentation of inconsistent outputs are valuable descriptive contributions. However, the confirmatory statistical analysis is flawed in a load-bearing way: the chi-square test ignores the paired structure of the data, and the reported p-values conflate statistical significance with practical significance. The paper's main conclusion therefore needs to be re-argued with appropriate paired analyses and effect sizes before the claim of 'significant variations' is acceptable.","major_comments":[{"comment":"The chi-square test is applied to aggregate label counts (as reported in Table 3), not to per-comment agreement, and the procedure explicitly states that 'cada resultado de calificación solicitado por un prompt es independiente de otro.' This independence assumption is false because the same 100,000 comments are classified by every prompt, making the observations paired. A marginal chi-square test cannot detect per-comment disagreement—two prompts could have identical marginal distributions while assigning opposite labels to every comment. The authors should use a paired test (e.g., the Stuart-Maxwell test or McNemar-type tests for multinomial categories) or model per-comment agreement directly. As written, the confirmatory analysis does not support the claim that the prompts produce significantly different classifications.","section":"Análisis confirmatorio, Eq. (1); Tables 6 and 7"},{"comment":"The extremely small p-values are driven primarily by the sample size of 100,000 and do not by themselves establish that the differences are practically meaningful. The coincidence matrix (Table 4) shows exact agreement between 0.92 and 0.98, meaning that 2–8% of comments are classified differently across prompt pairs. No effect size, such as Cohen's kappa, the proportion of disagreements, or the magnitude of marginal shifts, is reported. The conclusion that 'cambios sutiles ... sí producen variaciones significativas' is therefore overstated; the evidence supports at most small, statistically detectable differences. The authors should report an effect size and interpret the magnitude of the differences in light of the 92–98% agreement.","section":"Resultados, Tables 4, 6, and 7"},{"comment":"There is an internal inconsistency in the definition of the categories used in the chi-square test. The confirmatory section defines the events i as 'positivo, negativo y neutral' (three categories), but the subsequent description of the chi-square analysis states that the categorical results were transformed to numeric values 1, 2, 3, and 4, with 4 representing 'inconsistente.' This means the test actually uses four categories, including responses that were not requested by the prompt. The authors should specify whether inconsistent responses were excluded or included, and, if included, justify treating these non-requested outputs as a category in the marginal comparison. This ambiguity affects the reproducibility and validity of the reported p-values in Tables 6 and 7.","section":"Prueba Chi-cuadrado vs. Análisis confirmatorio"}],"minor_comments":[{"comment":"The text cites OpenAI 2023d as the source for the sentiment analysis prompt used as prompt 1, but the reference list entry for OpenAI 2023d is titled 'Meeting minutes,' which appears to be a different document; the reference should be corrected or clarified.","section":"Introduction, reference to OpenAI 2023d"},{"comment":"The paper cites two different works by 'Zhang et al. 2023': one on sentiment analysis in the era of LLMs (Zhang, Deng, Liu, Pan, and Bing) and one on meta prompting (Zhang, Yuan, and Yao). The prompt design section refers to 'Zhang et al. (2023)' for the base prompt 2 without disambiguating which of these references is meant; please specify the intended citation.","section":"References, Zhang et al. 2023"},{"comment":"The PCA is performed on only ten prompt embeddings, and the resulting two-dimensional plot is described in terms of proximity and clustering. This is reasonable as an exploratory visualization, but the interpretation of 'closeness' would benefit from reporting the variance explained by the principal components, since with ten points the projection may be visually misleading.","section":"Análisis exploratorio, PCA"},{"comment":"The text references Table 3 as showing the percentage of each category per prompt, but the table entries are not described in the body; a short narrative summary of the ranges of the percentages (e.g., the largest margin differences between prompts) would help readers interpret the subsequent chi-square results.","section":"Resultados, Table 3"},{"comment":"The chi-square statistic is written with a typo in the formula: 'χ2=∑!\"#1(%!&’!)2’!' should be a standard sum over categories of (O_i - E_i)^2 / E_i; please correct the notation.","section":"Prueba Chi-cuadrado"},{"comment":"The statement that unstructured prompts (prompts 9 and 10) produce 'more than 1000' inconsistent responses would be easier to interpret if the counts were reported in a table, along with the total number of responses per prompt, rather than only in prose.","section":"Discussion"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and socially relevant question, and the descriptive findings (especially the coincidence matrix and the increase in inconsistent outputs for unstructured prompts) are worth publishing after a rigorous re-analysis. The main concern is that the chi-square test as applied is inappropriate for paired data and that the practical magnitude of the differences is not quantified. The authors have the per-comment labels, so a paired analysis and effect sizes are feasible within the scope of the current dataset. I recommend major revision rather than rejection, provided the confirmatory analysis is redone and the conclusions are calibrated to the actual effect sizes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Luis—quick take on arXiv:2504.12180. The paper is a large-scale demonstration of prompt sensitivity for GPT-4o mini in Spanish sentiment classification. That is not a new phenomenon, but the dataset (100k comments about four Latin American presidents), the prompt design, and the open code make it a useful addition to the empirical record. The coincidence matrix and the counts of inconsistent responses are transparent and reproducible, and the observation that unstructured prompts produce more off-label responses is a concrete practical warning.\n\nThe weak spot is the confirmatory analysis. The chi-square test is applied to the marginal label counts for each prompt, not to per-comment agreement. The same 100k comments are classified under every prompt, so the observations are paired, and a marginal test cannot detect disagreement where prompts swap labels on individual comments. With N=100,000, tiny differences become 'significant'; that is why almost every pair rejects the null while the paper's own coincidence matrix shows 92–98% exact agreement. The conclusion that subtle prompt changes produce 'major differences' is not supported. The evidence supports small, systematic instabilities. The authors should report a paired test (McNemar or Stuart-Maxwell), a chance-corrected agreement measure like Cohen's kappa, and an effect size. The single non-rejected pair (prompts 1 and 7) is interesting, but with this framing it is just one non-rejection, not demonstrated equivalence.\n\nA couple of smaller items: labeling format violations as 'hallucinations' is a stretch; some are just refusals or translations. And the trust discussion in the back half is more of a literature review than a direct payoff from the data. Those are minor.\n\nOverall: the paper is honest and reproducible, and the Spanish-language corpus is genuinely useful. It just overclaims in the title and the conclusion. I would send it to peer review and ask for a reworked statistical section and a toned-down claim. I'd probably not cite it in my own work—the phenomenon is already documented—but I would point a student to it as a clean example of why paired data matter.","headline":"Large, reproducible Spanish-language demonstration of prompt sensitivity undercut by a paired-data chi-square and an overclaim of 'major differences'.","tokens_in":18490,"tokens_out":3770,"would_cite":false,"duration_ms":36351,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Subtle prompt rewording shifts GPT-4o mini's sentiment classifications: ten variants over 100,000 Spanish comments produced statistically different label distributions except for one near-identical pair.","keywords":["large language models","prompt engineering","sentiment analysis","polarity classification","GPT-4o mini","robustness","trust","Spanish-language data"],"falsifier":"Apply a paired or effect-size analysis to the paper's own 10 by 100,000 classification table: if McNemar-style paired comparisons or measures such as Cohen's kappa show that pairwise disagreements are negligible in size despite p-values below 0.05, then the claim that prompt tweaks produce significant classification differences would be weakened to 'statistically detectable but practically trivial'.","tokens_in":17545,"feed_emoji":"🤖","tokens_out":8270,"duration_ms":78754,"temperature":0.7,"pith_summary":"The paper asks how far a slightly reworded instruction can change what a large language model reports about the same texts. It classified 100,000 Spanish YouTube comments about four Latin American presidents as positive, negative, or neutral with GPT-4o mini, using ten zero-shot prompts built from two seed prompts and varied by sentence order, synonym choice, modal verbs, and grammatical breakdown. The tested hypothesis—that subtle prompt changes cause no significant differences—fails: chi-square comparisons reject equal label distributions for almost every prompt pair, the only exception being the first seed prompt and its modal variation. The paper reads this as evidence that LLM sentiment classification is not robust to non-semantic prompt variation, and that trust in these models cannot rest on technical performance alone.","feed_headline":"Rewording a prompt changes ChatGPT's sentiment verdicts","feed_subtitle":"Only one of ten prompt variants gave statistically matching results on 100,000 Spanish comments.","key_machinery":"The argument is carried by a controlled perturbation design: ten zero-shot prompts in two groups, each built from a different seed prompt, with four linguistic perturbation types per seed—inter-sentence reordering, lexical-semantic substitution, modal-verb shift, and telegraphic desestructuring. The confirmatory instrument is the chi-square test, which compares each prompt's marginal counts of positive, negative, neutral, and inconsistent labels against those of every other prompt, using the other prompt as the expected distribution. Supporting exploratory tools are PCA on prompt embeddings, a pairwise coincidence matrix, and word-level Levenshtein distance, which together show that structurally closer prompts tend to classify more alike without guaranteeing identical outcomes.","core_discovery":"The paper's central discovery is that GPT-4o mini does not treat semantically similar instructions as equivalent. Holding model and temperature constant, the same 100,000 comments were classified under ten prompts; pairwise coincidence rates were high, between 0.92 and 0.98, yet chi-square tests of the label distributions rejected equal distributions for all pairs except prompts 1 and 7. Unstructured prompts produced the most off-scheme outputs—mixed categories, unsolicited explanations, and non-Spanish answers—which the paper counts as inconsistent and links to hallucination. The conclusion is that changes in lexical, syntactic, modal, or structural prompt form produce statistically significant variations in polarity classification, and therefore that the model's robustness to prompt perturbation is limited.","pith_inferences":["The high coincidence rates between 0.92 and 0.98 mean the statistically significant differences may be small in practical size; a paired effect-size analysis would clarify how much of the conclusion survives.","The same 100,000 comments could be given human ground-truth labels to identify which prompt's flips are actual errors; the paper names this as future work, but the data already permit it.","Running the same ten-prompt battery on other models and in other languages would show whether this fragility is specific to GPT-4o mini on Spanish or a general property of LLMs.","Because 'inconsistent' responses were added as a fourth category, the chi-square result depends partly on off-scheme outputs; reanalyzing with only the three instructed categories would isolate the effect of the inconsistent responses."],"forward_implications":["A single prompt cannot be treated as a neutral measurement instrument for LLM sentiment analysis; another wording can shift the aggregate label distribution.","Social scientists using LLM classification should report their prompt and test its variants, because prompt-dependent results limit comparability across studies.","Telegraphic or grammatically broken prompts increase off-scheme outputs, so prompt clarity is a data-quality issue, not just a usability preference.","If robustness is defined as invariance to non-semantic prompt changes, the paper's results show that this criterion is not met for GPT-4o mini on Spanish sentiment classification."],"supporting_citations":[{"why":"Supplies the official sentiment-analysis seed prompt for Group A and the API batch pattern used to run the 100,000 comments.","marker":"OpenAI 2023d"},{"why":"Supplies the second seed prompt for Group B and motivates the study with evidence that semantically similar prompts can give different LLM outputs.","marker":"Zhang et al. 2023"},{"why":"Defines robustness as resistance to non-semantic prompt variation, the criterion the experiment is designed to test.","marker":"Bolton et al. 2024"},{"why":"Provides the prompt-components framework (instruction, context, input, output indicator) used to categorize the variations.","marker":"DAIR.AI 2023"},{"why":"Supplies the batch API code that sends each comment through the model with each prompt, the execution mechanism of the experiment.","marker":"Guzman 2024"},{"why":"Provides the probability-density framework behind the chi-square comparison of label distributions.","marker":"Papoulis 2002"},{"why":"Supplies the experimental design template in which one variable, the prompt, is manipulated while temperature and model stay fixed.","marker":"Creswell y Creswell 2018"}],"fun_headline_variants":[],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the 100,000 classifications made under each prompt are independent of the classifications of the same comments under the other prompts; because the data are paired and the sample is large, even small practical disagreements produce very small p-values.","fun_headline_variants_meta":{"error":"DeepSeek 429: {\"error\":{\"message\":\"Too many requests. Your current concurrency is 148, which exceeds your concurrency limit of 117 based on your remaining balance. Please top up your balance to restore your concurrency.\",\"type\":\"rate_limit_error\",\"param\":null,\"code\":\"invalid_request_error\"}}"},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:35:08.911588+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Apply a paired or effect-size analysis to the paper's own 10 by 100,000 classification table: if McNemar-style paired comparisons or measures such as Cohen's kappa show that pairwise disagreements are negligible in size despite p-values below 0.05, then the claim that prompt tweaks produce significant classification differences would be weakened to 'statistically detectable but practically trivial'.","supporting_citations":[],"review_version":1}