Trusting CHATGPT: how minor tweaks in the prompts lead to major differences in sentiment classification

Andres Felipe Mican-Castiblanco; Jaime Andres Pavlich-Mariscal; Jaime E. Cuellar; Juan Guillermo Torres-Hurtado; Oscar Moreno-Martinez; Paula Sofia Torres-Rodriguez

REVIEW 3 major objections 6 minor 59 references

Trusting CHATGPT: how minor tweaks in the prompts lead to major differences in sentiment classification

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Subtle prompt rewording shifts GPT-4o mini's sentiment classifications: ten variants over 100,000 Spanish comments produced statistically different label distributions except for one near-identical pair.

desk verdict Large, reproducible Spanish-language demonstration of prompt sensitivity undercut by a paired-data chi-square and an overclaim of 'major differences'. read the letter →

arxiv 2504.12180 v1 pith:WJDVGMGD submitted 2025-04-16 cs.CL cs.AI

Jaime E. Cuellar , Oscar Moreno-Martinez , Paula Sofia Torres-Rodriguez , Jaime Andres Pavlich-Mariscal , Andres Felipe Mican-Castiblanco , Juan Guillermo Torres-Hurtado This is my paper · ORCID

classification cs.CLcs.AI

keywords largelanguagemodelspromptengineeringsentimentanalysispolarityclassificationGPT-4ominirobustnesstrustSpanish-languagedata

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks how far a slightly reworded instruction can change what a large language model reports about the same texts. It classified 100,000 Spanish YouTube comments about four Latin American presidents as positive, negative, or neutral with GPT-4o mini, using ten zero-shot prompts built from two seed prompts and varied by sentence order, synonym choice, modal verbs, and grammatical breakdown. The tested hypothesis—that subtle prompt changes cause no significant differences—fails: chi-square comparisons reject equal label distributions for almost every prompt pair, the only exception being the first seed prompt and its modal variation. The paper reads this as evidence that LLM sentiment classification is not robust to non-semantic prompt variation, and that trust in these models cannot rest on technical performance alone.

What carries the argument

The argument is carried by a controlled perturbation design: ten zero-shot prompts in two groups, each built from a different seed prompt, with four linguistic perturbation types per seed—inter-sentence reordering, lexical-semantic substitution, modal-verb shift, and telegraphic desestructuring. The confirmatory instrument is the chi-square test, which compares each prompt's marginal counts of positive, negative, neutral, and inconsistent labels against those of every other prompt, using the other prompt as the expected distribution. Supporting exploratory tools are PCA on prompt embeddings, a pairwise coincidence matrix, and word-level Levenshtein distance, which together show that structurally closer prompts tend to classify more alike without guaranteeing identical outcomes.

What would settle it

Apply a paired or effect-size analysis to the paper's own 10 by 100,000 classification table: if McNemar-style paired comparisons or measures such as Cohen's kappa show that pairwise disagreements are negligible in size despite p-values below 0.05, then the claim that prompt tweaks produce significant classification differences would be weakened to 'statistically detectable but practically trivial'.

Watch

Extended reading notes

Core claim

The paper's central discovery is that GPT-4o mini does not treat semantically similar instructions as equivalent. Holding model and temperature constant, the same 100,000 comments were classified under ten prompts; pairwise coincidence rates were high, between 0.92 and 0.98, yet chi-square tests of the label distributions rejected equal distributions for all pairs except prompts 1 and 7. Unstructured prompts produced the most off-scheme outputs—mixed categories, unsolicited explanations, and non-Spanish answers—which the paper counts as inconsistent and links to hallucination. The conclusion is that changes in lexical, syntactic, modal, or structural prompt form produce statistically significant variations in polarity classification, and therefore that the model's robustness to prompt perturbation is limited.

Load-bearing premise

The load-bearing assumption is that the 100,000 classifications made under each prompt are independent of the classifications of the same comments under the other prompts; because the data are paired and the sample is large, even small practical disagreements produce very small p-values.

Editorial extensions

If this is right

A single prompt cannot be treated as a neutral measurement instrument for LLM sentiment analysis; another wording can shift the aggregate label distribution.
Social scientists using LLM classification should report their prompt and test its variants, because prompt-dependent results limit comparability across studies.
Telegraphic or grammatically broken prompts increase off-scheme outputs, so prompt clarity is a data-quality issue, not just a usability preference.
If robustness is defined as invariance to non-semantic prompt changes, the paper's results show that this criterion is not met for GPT-4o mini on Spanish sentiment classification.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The high coincidence rates between 0.92 and 0.98 mean the statistically significant differences may be small in practical size; a paired effect-size analysis would clarify how much of the conclusion survives.
The same 100,000 comments could be given human ground-truth labels to identify which prompt's flips are actual errors; the paper names this as future work, but the data already permit it.
Running the same ten-prompt battery on other models and in other languages would show whether this fragility is specific to GPT-4o mini on Spanish or a general property of LLMs.
Because 'inconsistent' responses were added as a fourth category, the chi-square result depends partly on off-scheme outputs; reanalyzing with only the three instructed categories would isolate the effect of the inconsistent responses.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Luis—quick take on arXiv:2504.12180. The paper is a large-scale demonstration of prompt sensitivity for GPT-4o mini in Spanish sentiment classification. That is not a new phenomenon, but the dataset (100k comments about four Latin American presidents), the prompt design, and the open code make it a useful addition to the empirical record. The coincidence matrix and the counts of inconsistent responses are transparent and reproducible, and the observation that unstructured prompts produce more off-label responses is a concrete practical warning.

The weak spot is the confirmatory analysis. The chi-square test is applied to the marginal label counts for each prompt, not to per-comment agreement. The same 100k comments are classified under every prompt, so the observations are paired, and a marginal test cannot detect disagreement where prompts swap labels on individual comments. With N=100,000, tiny differences become 'significant'; that is why almost every pair rejects the null while the paper's own coincidence matrix shows 92–98% exact agreement. The conclusion that subtle prompt changes produce 'major differences' is not supported. The evidence supports small, systematic instabilities. The authors should report a paired test (McNemar or Stuart-Maxwell), a chance-corrected agreement measure like Cohen's kappa, and an effect size. The single non-rejected pair (prompts 1 and 7) is interesting, but with this framing it is just one non-rejection, not demonstrated equivalence.

A couple of smaller items: labeling format violations as 'hallucinations' is a stretch; some are just refusals or translations. And the trust discussion in the back half is more of a literature review than a direct payoff from the data. Those are minor.

Overall: the paper is honest and reproducible, and the Spanish-language corpus is genuinely useful. It just overclaims in the title and the conclusion. I would send it to peer review and ask for a reworked statistical section and a toned-down claim. I'd probably not cite it in my own work—the phenomenon is already documented—but I would point a student to it as a clean example of why paired data matter.

Referee Report

3 major / 6 minor

Summary. The paper tests the hypothesis that subtle variations in prompt wording (lexical, syntactic, modal-pragmatic, and unstructured) do not produce significant differences in sentiment polarity classification when using GPT-4o mini. The authors collected 100,000 Spanish YouTube comments about four Latin American presidents and classified each comment ten times, once per prompt, at temperature zero. Exploratory analyses include PCA of prompt embeddings, a pairwise coincidence matrix of exact label agreement, and Levenshtein distances between prompts. Confirmatory analysis uses chi-square tests on the aggregate label counts for each prompt. The reported coincidence matrix shows 92–98% exact agreement between prompt pairs, yet the chi-square tests reject the null hypothesis of equal distributions for nearly all pairs, with only prompts 1 and 7 not significantly different. The paper concludes that subtle prompt changes significantly affect classifications and that unstructured prompts increase the frequency of inconsistent (hallucinated) outputs.

Significance. If the central claim were properly supported, the paper would make a useful contribution to computational social science and applied NLP by quantifying prompt sensitivity in a large, non-English, real-world corpus. Strengths include the large Spanish dataset, several linguistically motivated prompt manipulations, a controlled temperature setting, and a public code repository. The exploratory coincidence matrix and the documentation of inconsistent outputs are valuable descriptive contributions. However, the confirmatory statistical analysis is flawed in a load-bearing way: the chi-square test ignores the paired structure of the data, and the reported p-values conflate statistical significance with practical significance. The paper's main conclusion therefore needs to be re-argued with appropriate paired analyses and effect sizes before the claim of 'significant variations' is acceptable.

major comments (3)

[Análisis confirmatorio, Eq. (1); Tables 6 and 7] The chi-square test is applied to aggregate label counts (as reported in Table 3), not to per-comment agreement, and the procedure explicitly states that 'cada resultado de calificación solicitado por un prompt es independiente de otro.' This independence assumption is false because the same 100,000 comments are classified by every prompt, making the observations paired. A marginal chi-square test cannot detect per-comment disagreement—two prompts could have identical marginal distributions while assigning opposite labels to every comment. The authors should use a paired test (e.g., the Stuart-Maxwell test or McNemar-type tests for multinomial categories) or model per-comment agreement directly. As written, the confirmatory analysis does not support the claim that the prompts produce significantly different classifications.
[Resultados, Tables 4, 6, and 7] The extremely small p-values are driven primarily by the sample size of 100,000 and do not by themselves establish that the differences are practically meaningful. The coincidence matrix (Table 4) shows exact agreement between 0.92 and 0.98, meaning that 2–8% of comments are classified differently across prompt pairs. No effect size, such as Cohen's kappa, the proportion of disagreements, or the magnitude of marginal shifts, is reported. The conclusion that 'cambios sutiles ... sí producen variaciones significativas' is therefore overstated; the evidence supports at most small, statistically detectable differences. The authors should report an effect size and interpret the magnitude of the differences in light of the 92–98% agreement.
[Prueba Chi-cuadrado vs. Análisis confirmatorio] There is an internal inconsistency in the definition of the categories used in the chi-square test. The confirmatory section defines the events i as 'positivo, negativo y neutral' (three categories), but the subsequent description of the chi-square analysis states that the categorical results were transformed to numeric values 1, 2, 3, and 4, with 4 representing 'inconsistente.' This means the test actually uses four categories, including responses that were not requested by the prompt. The authors should specify whether inconsistent responses were excluded or included, and, if included, justify treating these non-requested outputs as a category in the marginal comparison. This ambiguity affects the reproducibility and validity of the reported p-values in Tables 6 and 7.

minor comments (6)

[Introduction, reference to OpenAI 2023d] The text cites OpenAI 2023d as the source for the sentiment analysis prompt used as prompt 1, but the reference list entry for OpenAI 2023d is titled 'Meeting minutes,' which appears to be a different document; the reference should be corrected or clarified.
[References, Zhang et al. 2023] The paper cites two different works by 'Zhang et al. 2023': one on sentiment analysis in the era of LLMs (Zhang, Deng, Liu, Pan, and Bing) and one on meta prompting (Zhang, Yuan, and Yao). The prompt design section refers to 'Zhang et al. (2023)' for the base prompt 2 without disambiguating which of these references is meant; please specify the intended citation.
[Análisis exploratorio, PCA] The PCA is performed on only ten prompt embeddings, and the resulting two-dimensional plot is described in terms of proximity and clustering. This is reasonable as an exploratory visualization, but the interpretation of 'closeness' would benefit from reporting the variance explained by the principal components, since with ten points the projection may be visually misleading.
[Resultados, Table 3] The text references Table 3 as showing the percentage of each category per prompt, but the table entries are not described in the body; a short narrative summary of the ranges of the percentages (e.g., the largest margin differences between prompts) would help readers interpret the subsequent chi-square results.
[Prueba Chi-cuadrado] The chi-square statistic is written with a typo in the formula: 'χ2=∑!"#1(%!&’!)2’!' should be a standard sum over categories of (O_i - E_i)^2 / E_i; please correct the notation.
[Discussion] The statement that unstructured prompts (prompts 9 and 10) produce 'more than 1000' inconsistent responses would be easier to interpret if the counts were reported in a table, along with the total number of responses per prompt, rather than only in prose.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the study directly compares observed label distributions across prompt variants, with no fitted parameter renamed as prediction and no load-bearing self-citation.

full rationale

The paper's central claim is an empirical finding: ten zero-shot prompts are applied to the same 100,000 Spanish comments and the resulting label distributions are compared. The confirmatory chi-square analysis uses only the directly observed classification counts and expected counts derived from another prompt's observed counts; no parameter is fitted to a subset and then 'predicted' on the same subset. The base prompts are attributed to OpenAI and Zhang et al. (2023), neither of which overlaps with the present authors, so the conclusion does not depend on a self-citation chain. Exploratory tools (PCA, coincidence matrix, Levenshtein distance) and the chi-square test are standard descriptive and inferential statistics applied to the experimental output, not derivations whose conclusions are already contained in the inputs. The paired-observations issue raised in the reader's take is a statistical appropriateness concern about whether a marginal chi-square is ideal when the same comments are labeled by every prompt; it is a substantive methodological caveat about effect size and test choice, not circularity in the derivation. No step satisfies the bar of 'Eq. X = Eq. Y by construction' or 'fitted parameter renamed as prediction,' so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted; the study is entirely empirical. The main assumptions are statistical and operational, not mathematical axioms.

assumptions (3)

domain assumption The chi-square test provides a valid comparison of classification distributions across prompts.
The paper uses this test in the confirmatory analysis but treats paired observations as independent, which may invalidate the test.
domain assumption Temperature set to zero yields deterministic outputs from GPT-4o mini.
The authors assume that setting temperature to 0 makes repeated runs of the same prompt identical, but OpenAI does not guarantee strict determinism, and they did not run repetitions to check.
domain assumption The embeddings, PCA, and Levenshtein distances reflect meaningful prompt similarity.
These exploratory measures are used to interpret prompt grouping, but their validity as proxies for model behavior is assumed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Trusting CHATGPT: how minor tweaks in the prompts lead to major differences in sentiment classification." pith.science (2026). https://pith.science/paper/WJDVGMGD

@misc{pith2026250412180,
  author       = {Pith},
  title        = {Pith review of: Trusting CHATGPT: how minor tweaks in the prompts lead to major differences in sentiment classification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WJDVGMGD}},
  note         = {Machine review of arXiv:2504.12180}
}

read the original abstract

One fundamental question for the social sciences today is: how much can we trust highly complex predictive models like ChatGPT? This study tests the hypothesis that subtle changes in the structure of prompts do not produce significant variations in the classification results of sentiment polarity analysis generated by the Large Language Model GPT-4o mini. Using a dataset of 100.000 comments in Spanish on four Latin American presidents, the model classified the comments as positive, negative, or neutral on 10 occasions, varying the prompts slightly each time. The experimental methodology included exploratory and confirmatory analyses to identify significant discrepancies among classifications. The results reveal that even minor modifications to prompts such as lexical, syntactic, or modal changes, or even their lack of structure impact the classifications. In certain cases, the model produced inconsistent responses, such as mixing categories, providing unsolicited explanations, or using languages other than Spanish. Statistical analysis using Chi-square tests confirmed significant differences in most comparisons between prompts, except in one case where linguistic structures were highly similar. These findings challenge the robustness and trust of Large Language Models for classification tasks, highlighting their vulnerability to variations in instructions. Moreover, it was evident that the lack of structured grammar in prompts increases the frequency of hallucinations. The discussion underscores that trust in Large Language Models is based not only on technical performance but also on the social and institutional relationships underpinning their use.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 36 canonical work pages

[1]

Autores Cuellar, Jaime E

1 ¿Confiar en ChatGPT? Cuando una variación sutil en el prompt puede modificar significativamente los resultados. Autores Cuellar, Jaime E. jaimecuellar@javeriana.edu.co Facultad de Comunicación y Lenguaje Pontificia Universidad Javeriana (Colombia) https://orcid.org/0009-0004-0858-2823 Moreno-Martínez, Óscar. Ph.D. morenoo@javeriana.edu.co Facultad de Co...

work page 2023
[2]

En este sentido, se buscó poner a prueba la capacidad del modelo para interpretar correctamente el mensaje subyacente, con formas lingüísticas desestructuradas

Por último, los prompts 9 y 10 fueron diseñados sin palabras gramaticales ni puntuación para evaluar cómo el modelo maneja instrucciones cargadas semánticamente, pero fragmentadas gramaticalmente. En este sentido, se buscó poner a prueba la capacidad del modelo para interpretar correctamente el mensaje subyacente, con formas lingüísticas desestructuradas....

work page 2021
[4]

(2024) emplearon el análisis de sentimientos como base para desarrollar una métrica del impacto de los rumores en redes sociales en términos de daño

De manera similar, Li et al. (2024) emplearon el análisis de sentimientos como base para desarrollar una métrica del impacto de los rumores en redes sociales en términos de daño. En el ámbito de la salud, Braig et al. (2023) exploraron cómo el análisis de sentimientos en Twitter puede ofrecer información clave para gestionar la pandemia del COVID-19. En e...

work page 2024
[6]

(País) + (nombre del Presidente) + ‘presidente’

Cuando decimos canales periodísticos locales nos referimos a medios del país donde el presidente es gobernante. 10 seleccionados se generó a partir de palabras clave organizadas de la siguiente manera: “(País) + (nombre del Presidente) + ‘presidente’ ”. Esta conjugación de palabras permitió focalizar videos y comentarios específicamente sobre los cuatro l...

work page 2023
[8]

El código utilizado fue construido a partir de la documentación de OpenAI para realizar análisis de sentimientos (Guzman 2024; OpenAI 2023a)

Procesamiento de datos Luego de contar con la base de datos construida y los prompts diseñados, se instruyó a GPT-4o mini, a través de su API y del lenguaje Python, para que clasificara los 100.000 comentarios en positivos, negativos o neutrales con los diez prompts. El código utilizado fue construido a partir de la documentación de OpenAI para realizar a...

work page 2024
[10]

Sin embargo, su contraposición en el grupo B, los prompts 2 y 8, que son el otro prompt de referencia y su variación modal respectivamente, no tienen tal cercanía

Su posición sugiere que estos prompts podrían generar resultados similares en la clasificación realizada por el modelo. Sin embargo, su contraposición en el grupo B, los prompts 2 y 8, que son el otro prompt de referencia y su variación modal respectivamente, no tienen tal cercanía. 19 Otros prompts con gran cercanía son los 2 y 6 del grupo B. El prompt 2...

work page 2023
[11]

Lo anterior proporciona un marco inicial para conocer, contrastar y analizar los resultados de la prueba de Chi-cuadrado

Nuevamente, su contraposición en el grupo A, los prompts 1 y 5 (el prompt de referencia de OpenAI y su variación léxico-semántica) no tienen tal cercanía. Lo anterior proporciona un marco inicial para conocer, contrastar y analizar los resultados de la prueba de Chi-cuadrado. Este análisis preliminar permite identificar patrones y posibles similitudes en ...

work page 2023
[13]

También, se observa que los prompts desestructurados, sin palabras gramaticales ni puntuación, no fueron los que mayor difirieron con el resto, como pasó en el análisis anterior

Asimismo, la distancia de palabras entre el prompt 2 y el 6 es de tan solo 1 palabra, a pesar de que su coincidencia en la Tabla 4 es de 0,96. También, se observa que los prompts desestructurados, sin palabras gramaticales ni puntuación, no fueron los que mayor difirieron con el resto, como pasó en el análisis anterior. Finalmente, la correlación entre es...

work page 2002

Show all 59 references

[14]

Otro hallazgo está en la robustez del LLM

Esto sugiere que la falta de oraciones bien formadas en los prompts y el uso de signos de puntuación puede generar una mayor cantidad de alucinaciones. Otro hallazgo está en la robustez del LLM. El experimento muestra que, ante un análisis de sentimientos con solo tres opcione...

2024
[16]

Nunca vi a el karma de actuar de manera tan instantánea

Se emplea el análisis de sentimientos como fuente principal en el código, ya que esta herramienta, ampliamente adoptada en los últimos años, facilita la clasificación de datos y permite una comparación más clara y práctica entre los resultados. No obstante, el objetivo de este...

2022
[17]

ChatGPT and the Future of Medical Writing

“ChatGPT and the Future of Medical Writing.” Radiology 307 (2): e223312. https://doi.org/10.1148/radiol.223312 Bolton, William James, Rafael Poyiadzi, Edward R. Morrell, Gabriela van Bergen Gonzalez Bueno, y Lea Goetz

doi
[18]

RAmBLA: A Framework for Evaluating the Reliability of LLMs as Assistants in the Biomedical Domain

“RAmBLA: A Framework for Evaluating the Reliability of LLMs as Assistants in the Biomedical Domain.” Cornell University https://doi.org/10.48550/arxiv.2403.14578 Borji, Ali. “A categorical archive of ChatGPT failures.” arXiv. https://doi.org/10.48550/arXiv.2302.03494 Braig, Ni...

doi arXiv
[19]

Machine Learning Techniques for Sentiment Analysis of COVID-19-Related Twitter Data

“Machine Learning Techniques for Sentiment Analysis of COVID-19-Related Twitter Data.” IEEE Access 11: 14778–14803. DOI: 10.1109/ACCESS.2023.3242234 Brown, Tom, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Giris...

2023
[20]

Language models are few-shot learners

“Language models are few-shot learners.” In Proceedings of the 34th International Conference on Neural Information Processing Systems (NIPS '20). Curran Associates Inc., Red Hook, New York, USA, Article 159: 1877–1901. Chandra, Rohitash, and Ritij Saini

1901
[21]

The Nexus between Information Disorder and Terrorism: A Mix of Machine Learning Approach and Content Analysis on 39 Terror Attacks

“The Nexus between Information Disorder and Terrorism: A Mix of Machine Learning Approach and Content Analysis on 39 Terror Attacks.” Dynamics of Asymmetric Conflict 15 (3): 190–209. https://doi.org/10.1080/17467586.2022.2055097 28 Coeckelbergh, Mark

2022
[24]

New York: Routledge

The Routledge Social Science Handbook of AI. New York: Routledge. https://doi.org/10.4324/9780429198533 Ermakova, Tatiana, Max Henke, and Fabian Benjamin

doi
[25]

Commercial Sentiment Analysis Solutions: A Comparative Study

“Commercial Sentiment Analysis Solutions: A Comparative Study.” International Conference on Web Information Systems and Technologies. DOI: 10.5220/0010709400003058i Fundación Karisma

doi
[26]

ACOSO, SOLEDAD Y DESPRESTIGIO: Un estudio sobre las formas, las rutas de atención y el impacto de las violencias digitales contra las candidatas al Congreso colombiano en 2022

“ACOSO, SOLEDAD Y DESPRESTIGIO: Un estudio sobre las formas, las rutas de atención y el impacto de las violencias digitales contra las candidatas al Congreso colombiano en 2022.” https://web.karisma.org.co/acoso-soledad-y-desprestigio/ Giray, Louie

2022
[27]

Prompt Engineering with ChatGPT: A Guide for Academic Writers

“Prompt Engineering with ChatGPT: A Guide for Academic Writers.” Annals of Biomedical Engineering 51 (12): 2629–2633. https://doi.org/10.1007/s10439-023-03272-4 29 Goldberg, Sanford C

doi
[31]

TrustLLM: Trustworthiness in Large Language Models

“TrustLLM: Trustworthiness in Large Language Models.” Cornell University. https://doi.org/10.48550/arxiv.2401.05561 Jacobson, Daniel, Greg Brail, and Dan Woods

doi arXiv
[32]

A brief history of APIs

“A brief history of APIs.” In Handbook of Computational Social Science, Volume 2: Data Science, Statistical Modelling, and Machine Learning Methods, edited by Uwe Engel, Anabel Quan-Haase, Sunny Liu, and Lars Lyberg. London: Routledge. https://doi.org/10.4324/9781003025245 Kha...

doi
[33]

Analyzing European Migrant-Related Twitter Deliberations

“Analyzing European Migrant-Related Twitter Deliberations.” In Companion Proceedings of the Web Conference 2021, 166–170. New York: Association for Computing Machinery. https://doi.org/10.1145/3442442.3453459 Koubaa, Anis, Wadii Boulila, Lahouari Ghouti, Ayyub Alzahem, and Sha...

2021
[34]

Exploring ChatGPT Capabilities and Limitations: A Survey

“Exploring ChatGPT Capabilities and Limitations: A Survey.” IEEE Access. https://doi.org/10.1109/ACCESS.2023.3326474 Krause, Lea, and Piek T.J.M. Vossen

2023
[36]

The Language of Prompting: What Linguistic Properties Make a Prompt Successful?

“The Language of Prompting: What Linguistic Properties Make a Prompt Successful?.” https://doi.org/10.48550/arXiv.2311.01967 Li, Hao, Wu Yang, Wei Wang, and Huanran Wang

doi arXiv
[37]

Vasarhelyi

https://doi.org/10.1186/s13677-024-00596-x Li, Huaxia, Haoyun Gao, Chengzhang Wu, and Miklos A. Vasarhelyi

doi
[38]

Extracting financial data from unstructured sources: leveraging large language models

“Extracting financial data from unstructured sources: leveraging large language models.” Social Science Research Network. Forthcoming in the Journal of Information Systems https://doi.org/10.2139/ssrn.4567607 31 Liu, Jiacheng, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ro...

doi
[39]

Generated Knowledge Prompting for Commonsense Reasoning

“Generated Knowledge Prompting for Commonsense Reasoning.” https://doi.org/10.48550/arXiv.2110.08387 Logan IV, Robert L., Ivana Balažević, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel

doi arXiv
[40]

Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models

“Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models.” https://doi.org/10.48550/arXiv.2106.13353 Long, Jieyi

doi arXiv
[41]

Large Language Model Guided Tree-of-Thought

“Large Language Model Guided Tree-of-Thought.” https://doi.org/10.48550/arXiv.2305.08291 Loomba, Sankalp, Madhavi Dave, Harshal Arolkar, y Sachin Sharma

doi arXiv
[42]

Sentiment Analysis Using Dictionary-Based Lexicon Approach: Analysis on the Opinion of Indian Community for the Topic of Cryptocurrency

“Sentiment Analysis Using Dictionary-Based Lexicon Approach: Analysis on the Opinion of Indian Community for the Topic of Cryptocurrency.” Annals of Data Science 11 (6): 2019–2034. https://doi.org/10.1007/s40745-023-00496-y Majeed, Abdul, and Seong Hwang

2019 doi
[43]

https://doi.org/10.3390/bdcc5030043 Latour, Bruno

doi
[44]

Reliability Issues of LLMs: ChatGPT a Case Study

“Reliability Issues of LLMs: ChatGPT a Case Study.” IEEE Reliability Magazine : 1–11. https://doi.org/10.1109/MRL.2024.3420849 Marvin, Ggaliwango, Nakayiza Hellen, Daudi Jjingo, and Joyce Nakatumba-Nabende

2024
[45]

Where Did the News Come From? Detection of News Agency Releases in Historical Newspapers

“Where Did the News Come From? Detection of News Agency Releases in Historical Newspapers.” Master’s thesis, École Polytechnique Fédérale de Lausanne. https://doi.org/10.5281/zenodo.8333933 Mitchell, Ryan

doi
[46]

Toxic Bias: Perspective API Misreads German as More Toxic

“Toxic Bias: Perspective API Misreads German as More Toxic.” Cornell University. https://doi.org/10.48550/arxiv.2312.12651 Nowotny, Helga

doi arXiv
[47]

From Twitter to Aso-Rock: A Sentiment Analysis Framework for Understanding Nigeria 2023 Presidential Election

“From Twitter to Aso-Rock: A Sentiment Analysis Framework for Understanding Nigeria 2023 Presidential Election.” Heliyon 9 (5): e16085. https://doi.org/10.1016/j.heliyon.2023.e16085 OpenAI. 2023a. “Batch API.” OpenAI Platform. https://platform.openai.com/docs/guides/batch/rate...

2023 doi
[48]

Negligent Algorithmic Discrimination

“Negligent Algorithmic Discrimination.” Law and Contemporary Problems 84 (3): 19–33. https://doi.org/10.2139/ssrn.3765778 Papoulis, Athanasios

doi
[49]

A Computational Look at Oral History Archives

“A Computational Look at Oral History Archives.” Journal on Computing and Cultural Heritage 15 (1): 1–16. https://doi.org/10.1145/3477605 Reynolds, Laria, and Kyle McDonell

doi
[50]

Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm

“Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm.” In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, 1–7. Rogers, Richard, and Xiaoke Zhang

2021
[51]

33 Rolin, Kristina

https://doi.org/10.1177/20563051241254379. 33 Rolin, Kristina

doi
[52]

Trust in artificial agents

“Trust in artificial agents.” The Routledge handbook of trust and philosophy, edited by Judith Simon. New York: Routledge, 298-312. https://doi.org/10.4324/9781315542294 Saucedo, Ricardo

doi
[53]

Natural Language Processing for Cultural Heritage Domains: NLP for Cultural Heritage Domains

“Natural Language Processing for Cultural Heritage Domains: NLP for Cultural Heritage Domains.” Language and Linguistics Compass 4 (9): 750–768. https://doi.org/10.1111/j.1749-818x.2010.00230.x Sultana, Jabeen, Nasreen Sultana, Kusum Yadav, y Fayez AlFayez

2010 arXiv
[55]

Self-Consistency Improves Chain of Thought Reasoning in Language Models

“Self-Consistency Improves Chain of Thought Reasoning in Language Models.” https://doi.org/10.48550/arXiv.2203.11171 Wei, Jason, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le

doi arXiv
[56]

Finetuned Language Models Are Zero-Shot Learners

“Finetuned Language Models Are Zero-Shot Learners.” https://doi.org/10.48550/arXiv.2109.01652 34 Wei, Jason, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou

doi arXiv
[57]

Sentiment Analysis Using Deep Learning Architectures: A Review

“Sentiment Analysis Using Deep Learning Architectures: A Review.” Artificial Intelligence Review 53 (6): 4335-4385. https://doi.org/10.1007/s10462-019-09794-5 Zamfirescu-Pereira J.D., Richmond Wong, Bjoern Hartmann, and Qian Yang

doi
[58]

Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts

“Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts.” In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems,

2023
[59]

https://doi.org/10.1145/3544548.3581388 Zanotti, Giacomo, Mattia Petrolo, Daniele Chiffi, and Viola Schiaffonati

ACM, New York, NY, USA. https://doi.org/10.1145/3544548.3581388 Zanotti, Giacomo, Mattia Petrolo, Daniele Chiffi, and Viola Schiaffonati
[60]

Keep Trusting! A Plea for the Notion of Trustworthy AI

“Keep Trusting! A Plea for the Notion of Trustworthy AI.” AI & Society 39 (6): 2691–2702. https://doi.org/10.1007/s00146-023-01789-9 Zhang, Wenxuan, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing

doi
[61]

Sentiment Analysis in the Era of Large Language Models: A Reality Check

“Sentiment Analysis in the Era of Large Language Models: A Reality Check.” arXiv, preprint, arXiv:2305.15005. https://doi.org/10.48550/arXiv.2305.15005 Zhang, Yifan, Yang Yuan, and Andrew Chi-Chih Yao

doi arXiv
[62]

Meta Prompting for AGI Systems

“Meta Prompting for AGI Systems.” https://doi.org/10.48550/arXiv.2311.11482 Anexo

doi arXiv
[68]

Las diferencias son mayores cuando se comparan los prompts basados en el oficial de OpenAI con los basados en Zhang et al. (2023). Por ejemplo, si se compara el prompt 1 con el 2, los dos de base, hay 55 palabras de distancia. En cambio, si se compara el 1 con el 7 hay una dis...

2023
[2009]

Defining Trust and E-trust: From Old Theories to New Problems

“Defining Trust and E-trust: From Old Theories to New Problems.” International Journal of Technology and Human Interaction 5 (2): 23–35. http://dx.doi.org/10.4018/jthi.2009040102 Wang, Xuezhi, Jason Wei, Dale Schuurmans, Quoc Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, ...

doi
[2010]

Toward a Model of Trust and E-trust Processes Using Object-oriented Methodologies

“Toward a Model of Trust and E-trust Processes Using Object-oriented Methodologies.” ETHICOMP 2010 Proceedings. Universitat Rovira i Virgili. Tarragona, Spain, April, 14–16. Grodzinsky, Frances, Keith Miller, and Marty J. Wolf

2010
[2012]

Can we trust robots?

“Can we trust robots?.” Ethics and information technology 14: 53-60. https://doi.org/10.1007/s10676-011-9279-1 Cook, Karen S., and Jessica J. Santana

doi
[2018]

Prediction of Sentiment Analysis on Educational Data Based on Deep Learning Approach

“Prediction of Sentiment Analysis on Educational Data Based on Deep Learning Approach.” In Proceedings of the 2018 21st Saudi Computer Society National Computer Conference (NCC), 1–5. https://doi.org/10.1109/NCG.2018.8593108 Taddeo, Mariarosaria

2018
[2020]

Su impacto en diferentes áreas del conocimiento ha sido inmediato

que reciben instrucciones en forma de prompt y pueden generar respuestas en texto. Su impacto en diferentes áreas del conocimiento ha sido inmediato. En la medicina, por ejemplo, se han usado para proporcionar información de salud personalizada a los pacientes y para brindar s...

2023
[2021]

Biden vs Trump: Modelling US General Elections Using BERT Language Model

“Biden vs Trump: Modelling US General Elections Using BERT Language Model.” IEEE Access. https://doi.org/10.1109/ACCESS.2021.3111035 Chen, Xingyu Ken, Loo Seng Neo, Anais Ang, Jing Yi Lee, Gabriel Ong, and Majeed Khader

2021
[2022]

A pesar del reciente auge de los LLM como ChatGPT, ya se han documentado varias limitaciones del modelo

y para analizar comentarios de redes sociales chinas con respecto a, por ejemplo, la guerra entre Ucrania y Rusia (Rogers y Zhang 2024). A pesar del reciente auge de los LLM como ChatGPT, ya se han documentado varias limitaciones del modelo. Borji (2023) construye una tipologí...

2023
[2023]

verdad científica

y recomendaciones para la construcción de prompts entre las que se incluye cuidar su especificidad y su claridad, estructurar el tipo de input, especificar el formato del resultado deseado, utilizar delimitadores, descomponer oraciones complejas, entre otras. De manera complem...

2024
[2024]

Trust as reliance in computer artifacts means that we expect an object to do something to help us attain our goals

como con una relación social de atributo relacional (Cook y Santana 2020). La confianza en estos modelos puede darse a partir de que confiamos en la 25 institución que los crea y mantiene (Cook, Hardin y Levi 2005), por experiencias sociales y tecnológicas previas, pero tambié...

2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Autores Cuellar, Jaime E

1 ¿Confiar en ChatGPT? Cuando una variación sutil en el prompt puede modificar significativamente los resultados. Autores Cuellar, Jaime E. jaimecuellar@javeriana.edu.co Facultad de Comunicación y Lenguaje Pontificia Universidad Javeriana (Colombia) https://orcid.org/0009-0004-0858-2823 Moreno-Martínez, Óscar. Ph.D. morenoo@javeriana.edu.co Facultad de Co...

work page 2023

[2] [2]

En este sentido, se buscó poner a prueba la capacidad del modelo para interpretar correctamente el mensaje subyacente, con formas lingüísticas desestructuradas

Por último, los prompts 9 y 10 fueron diseñados sin palabras gramaticales ni puntuación para evaluar cómo el modelo maneja instrucciones cargadas semánticamente, pero fragmentadas gramaticalmente. En este sentido, se buscó poner a prueba la capacidad del modelo para interpretar correctamente el mensaje subyacente, con formas lingüísticas desestructuradas....

work page 2021

[3] [4]

(2024) emplearon el análisis de sentimientos como base para desarrollar una métrica del impacto de los rumores en redes sociales en términos de daño

De manera similar, Li et al. (2024) emplearon el análisis de sentimientos como base para desarrollar una métrica del impacto de los rumores en redes sociales en términos de daño. En el ámbito de la salud, Braig et al. (2023) exploraron cómo el análisis de sentimientos en Twitter puede ofrecer información clave para gestionar la pandemia del COVID-19. En e...

work page 2024

[4] [6]

(País) + (nombre del Presidente) + ‘presidente’

Cuando decimos canales periodísticos locales nos referimos a medios del país donde el presidente es gobernante. 10 seleccionados se generó a partir de palabras clave organizadas de la siguiente manera: “(País) + (nombre del Presidente) + ‘presidente’ ”. Esta conjugación de palabras permitió focalizar videos y comentarios específicamente sobre los cuatro l...

work page 2023

[5] [8]

El código utilizado fue construido a partir de la documentación de OpenAI para realizar análisis de sentimientos (Guzman 2024; OpenAI 2023a)

Procesamiento de datos Luego de contar con la base de datos construida y los prompts diseñados, se instruyó a GPT-4o mini, a través de su API y del lenguaje Python, para que clasificara los 100.000 comentarios en positivos, negativos o neutrales con los diez prompts. El código utilizado fue construido a partir de la documentación de OpenAI para realizar a...

work page 2024

[6] [10]

Sin embargo, su contraposición en el grupo B, los prompts 2 y 8, que son el otro prompt de referencia y su variación modal respectivamente, no tienen tal cercanía

Su posición sugiere que estos prompts podrían generar resultados similares en la clasificación realizada por el modelo. Sin embargo, su contraposición en el grupo B, los prompts 2 y 8, que son el otro prompt de referencia y su variación modal respectivamente, no tienen tal cercanía. 19 Otros prompts con gran cercanía son los 2 y 6 del grupo B. El prompt 2...

work page 2023

[7] [11]

Lo anterior proporciona un marco inicial para conocer, contrastar y analizar los resultados de la prueba de Chi-cuadrado

Nuevamente, su contraposición en el grupo A, los prompts 1 y 5 (el prompt de referencia de OpenAI y su variación léxico-semántica) no tienen tal cercanía. Lo anterior proporciona un marco inicial para conocer, contrastar y analizar los resultados de la prueba de Chi-cuadrado. Este análisis preliminar permite identificar patrones y posibles similitudes en ...

work page 2023

[8] [13]

También, se observa que los prompts desestructurados, sin palabras gramaticales ni puntuación, no fueron los que mayor difirieron con el resto, como pasó en el análisis anterior

Asimismo, la distancia de palabras entre el prompt 2 y el 6 es de tan solo 1 palabra, a pesar de que su coincidencia en la Tabla 4 es de 0,96. También, se observa que los prompts desestructurados, sin palabras gramaticales ni puntuación, no fueron los que mayor difirieron con el resto, como pasó en el análisis anterior. Finalmente, la correlación entre es...

work page 2002