Computation and Language · arXiv:2610.22187 v1 · submitted 15 Oct 2026 · 94 references
Change the Number, Change the Answer: Assessing Rating Scale Sensitivity in LLM-Based Text Scoring
The paper says
“Large language models (LLMs) are increasingly used in computational social science to assign scalar scores to social and psychological constructs. However, studies use different rating scales, and whether their scores remain comparable across scale designs is understudied.”
more of the abstract
To address this gap, we define and systematically investigate scale sensitivity, the variation in LLM-generated scores caused by rating scale design. We distinguish numerical sensitivity, changes in normalized scores, from semantic sensitivity, changes in score interpretation. To assess both, our experimental design crosses 2 formats (continuous vs. ordinal), 4 ranges (3, 5, 11, and 21 points), and 2 signs (signed vs. unsigned) across 3 tasks (political leaning, sentiment, and toxicity), 2 prompting strategies (zero-shot and rubric-based), and 5 LLMs. We find that changing one design element shifts normalized scores by about 0.10 on average and changes the category of roughly 20% of texts. Sensitivity is greatest for toxicity, the only unipolar construct examined. Also, rubric-based prompting does not reduce sensitivity compared with zero-shot prompting. We provide practical guidance for designing more reliable LLM-based text scoring and highlight the need for robustness checks across alternative scales.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 16 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. The practical conclusion holds—scale design changes LLM scores—but the 0.10 headline mixes real scale effects with a discretization floor; the paper deserves refereeing with a request for a simulation baseline. This line is wrong
2 major objections · 7 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 94 references say.
The paper says, at §3.5.3 (Eq. 2), §4.1, Appendix D
The review did not quote this passage.
Pith says: Objection 1 of 2 · major
SSMAE conflates genuine scale-induced judgment change with the mechanical effect of output-grid coarseness, and this attribution gap is not acknowledged in the Limitations (b180–b185). The paper's formal definition of scale sensitivity is output-level, and the output-level measurements are not in question; the problem is that the abstract and conclusion (b5, b23, b165) present the 0.10 shift as a property of LLM judgment, which requires a decomposition that Eq.
more of objection 1
(2) cannot provide. Appendix D itself documents that model outputs concentrate on midpoints and whole numbers (the vertical steps in Figure 8), so the judgment-to-number map is coarse even in continuous conditions. Under the null that the model's judgment is identical across formats and the ordinal output is just the continuous judgment rounded to the nearest grid point, a uniform latent on [0,1] gives a mean absolute rounding difference of 0.125 on a 3-point grid and 0.0125 on a 21-point grid. The floor is therefore 86% of the observed 3-point format effect (0.125 of 0.146, §4.1) but only 16% of the observed 21-point effect; the data are consistent with the coarse-scale effect being almost entirely mechanical, and the headline 0.10, which averages over conditions with very different floors, cannot be read as the amount by which the scale changes the model's judgment of a text. The claim that finer scales reduce numerical format sensitivity (b129) must survive floor removal, and the paper does not provide the decomposition needed to check this. I recommend a concrete fix: for each format and range pair, take the continuous-condition outputs as the empirical latent proxy, round them to the coarser grid, and compute the implied floor SSMAE and SSswitch, reporting the residual as the genuine component; at minimum, report within-pair rank correlations to separate monotone quantization from re-ranking. The semantic findings are more robust to this critique, since at 21 points the rounding floor for SSswitch is about 2% while the observed category-switch rate stays near 18–20%.
The paper says, at §4.1 (b120–b127), §4.2, Table 6
The review did not quote this passage.
Pith says: Objection 2 of 2 · major
The comparative claims across design elements are not on a common scale. Sign minimal pairs have identical normalized grids on both sides, so their mechanical floor is exactly zero, while format and range pairs carry positive floors. The conclusions that the magnitudes are "broadly similar" across elements (b126) and that sign has the smallest numerical sensitivity (0.085 vs. 0.098 and 0.098, b120–b124) are therefore not established: after floor removal, the ordering could reverse.
more of objection 2
The sign-based results—the toxicity-by-sign moderation (+0.080 on SSswitch, Table 6) and the zero-anchoring effect documented in Appendix D and Figure 9—are the cleanest evidence in the paper precisely because they are floor-free; the paper should say so explicitly and foreground these findings, while re-expressing the format and range results with floor-removed estimates.
7 minor objections
The paper says, at Abstract
Pith says: minor objection 1 of 7
Missing space in "systematically investigatescale sensitivity".
The paper says, at §5
Pith says: minor objection 2 of 7
"Therefore, We recommend unsigned scales" capitalizes "We" mid-sentence; the sentence should continue with lowercase "we".
show 5 more
The paper says, at Figure 3
Pith says: minor objection 3 of 7
The caption reads "Continuous scales (b) Ordinal scales (b)"; the second "(b)" appears to be duplicated.
The paper says, at Figure 8
Pith says: minor objection 4 of 7
The 16-panel cumulative-distribution figure with light-to-dark lines for scale lengths is hard to parse at print size; a shared legend or a small-multiple redesign would help.
The paper says, at §3.5.5
Pith says: minor objection 5 of 7
The claim that "every 95% confidence interval lies within 0.01 of its estimate" is stronger than what is reported; state the maximum half-width or the relevant percentile so the claim is verifiable.
The paper says, at Table 2
Pith says: minor objection 6 of 7
Unsigned 3- and 5-point scales run from 1 to N while unsigned 11- and 21-point scales run from 0 to N (footnote 2); this means unsigned range pairs such as [1,3] vs. [0,10] vary the low anchor as well as the number of points, and this minor confound is worth acknowledging when interpreting range effects on unsigned scales.
The paper says, at Table 5
Pith says: minor objection 7 of 7
The open models run at temperature 1.0 with a single generation per item-condition (as far as reported), and the item bootstrap does not capture generation stochasticity, so the reported intervals likely understate total uncertainty; a same-condition repeated-prompt control, or seed-controlled generation, would bound this noise floor.
What would settle it
Decompose SSMAE into mechanical and genuine components from the paper's own paired outputs. For each format minimal pair (continuous vs ordinal at range r), compute the floor B = mean_i |z_cont,i − round_r(z_cont,i)|, where round_r maps to the nearest grid point {0, 1/(r−1), …, 1}, and compare B with the observed mean_i |z_cont,i − z_ord,i|; for range pairs, round the finer condition's outputs to the coarser grid; for sign pairs B = 0 by construction. If observed SSMAE exceeds B by less than about 0.02 for the 3- and 5-point format pairs and for coarse-vs-fine range pairs, then the headline 0.10 (b5) is largely mechanical, and the format/range magnitudes and their ranking against sign (b120–b124) must be re-reported as rebaselined (genuine) estimates; if the excess is large, the concern is resolved. Also overlay B as a dashed line in Figure 3.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 16 Oct 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 2 major objections, 7 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
What you should know: this is a careful, honest measurement paper, and its central practical claim survives scrutiny. Changing one rating scale element shifts normalized LLM scores by about 0.10 and switches the category of roughly one in five texts. Those are real scale-design effects. What is oversold is the causal gloss that they reflect changes in LLM judgment rather than partly mechanical consequences of the output grid.
What is new and good: nobody has systematically crossed format, range, and sign for LLM scalar scoring with minimal pairs, and the numerical/semantic distinction is genuinely useful. The design is solid: 16 conditions, 40 minimal pairs, 1,200 estimates per metric, item bootstrap with 2,000 resamples, WLS moderation, and a variance decomposition that sums to 100% with correct degrees of freedom. The paper also states its own limitations up front (b180–b185) and refuses to treat human labels as ground truth (b359–b361). The toxicity-by-sign interaction is a clean and interesting finding, and the reasoning-level and human-agreement appendices add value beyond the headline.
The soft spots are real but not fatal. The main one is unacknowledged: SSMAE mixes genuine scale-induced shift with a mechanical rounding floor. Appendix D shows outputs pile up on midpoints and whole numbers. For a 3-point ordinal scale, a uniform latent score rounded to the grid differs from the continuous value by E|u−round(u)| = 0.125, against an observed 3-point format effect of 0.146. So most of that particular number could be discretization, not a change in the model's judgment. This also undercuts the cross-element ranking: sign minimal pairs have identical grids and zero floor, while format and range pairs carry positive floors, so comparing the 0.098/0.098/0.085 averages in b120–b124 is comparing quantities of different kinds. The toxicity × sign moderation is not affected by this problem and is the paper's strongest specific result. Two smaller issues: the rubric null result rests on one authored rubric instance per task, and no code, output data, or commit hash is provided, so replication means regenerating millions of API calls.
Even with the mechanical floor, the practical takeaway stands: scores from studies using different scale designs are not interchangeable, and robustness checks are warranted. That is a useful message for anyone doing LLM-based measurement.
Bottom line: worth a serious referee. The revision should add a simulation baseline that separates the rounding floor from genuine judgment shift, and ideally release the outputs. I would cite this paper and would bring it to a reading group.
The strongest objection
The central claim (b5, b23) — a one-element scale change shifts normalized scores by about 0.10 and recategorizes one in five texts — is presented as a property of LLM judgment. That reading requires Eq. 1's affine normalization and Eq. 2's SSMAE to measure judgment change, not output-constraint artifacts. The paper's own Appendix D reports that outputs concentrate on midpoints and whole numbers (vertical steps in Figure 8), so the judgment-to-number map is coarse even in continuous conditions; SSMAE therefore mixes genuine scale-induced shift with a mechanical floor: the deterministic distance between an unrounded score and its nearest grid point, which appears even for a perfectly stable latent judgment. No baseline or simulation separates the two, and the Limitations (b180–b185) do not flag this attribution gap. Quantitatively, for a uniform latent the 3-point rounding floor is E|u − round(u)| = 0.125, the largest any distribution can produce; the observed 3-point format effect is 0.146 (Section 4.1), so the data are consistent with that effect being almost entirely mechanical. At 21 points the floor is 0.0125 versus an observed 0.078, so a genuine component exists at fine granularity; where between these extremes the split lies is an empirical question the paper leaves open. The comparative claims inherit the problem: sign minimal pairs have identical grids on both sides (mechanical floor exactly zero), while format and range pairs carry positive floors, so the numerical ranking format ≈ range (0.098) > sign (0.085) in b120–b124 compares quantities of different kinds; the paper's own semantic ordering (sign largest, 20.3%) already suggests the genuine ranking may differ. The toxicity × sign moderation (Table 6, +0.080 SSswitch) is mechanical-free and survives.
The reader's verdict is CONDITIONAL with Eq. 1's linear comparability as the weakest assumption; my stress-test agrees that this is the single most load-bearing concern and that it lands in a bounded way. The paper is a well-executed measurement study: the 2×4×2 design with 40 minimal pairs isolates design elements, item bootstrap with 2,000 resamples is honestly reported, the variance decomposition (Table 7) sums to 100% with correct degrees of freedom, and Appendix F documents the regression weighting. I credit these. My pass refines the reader's concern in three ways. First, it localizes the failure: affine normalization (Eq. 1) is defensible as a descriptive rescaling; the load-bearing gap is the missing mechanical floor, since a coarse ordinal grid guarantees nonzero SSMAE even for a perfectly stable latent judgment. Second, it bounds the stakes honestly: the 0.125 three-point rounding figure is the maximum possible floor for any output distribution (uniform achieves it), so the observed 0.146 is at most 17% above a worst-case mechanical account — the data are consistent with the 3-point format effect being almost entirely mechanical, and also with it being largely genuine; the paper provides no way to tell, and its own output distributions (Appendix D, Figure 8) make the true floor an empirical quantity rather than a known one. Third, it spells out a consequence the reader implied but did not state: sign pairs have identical grids on both sides, so their floor is exactly zero, whereas format and range pairs have positive floors; the numerical ranking format ≈ range > sign (b120–b124) therefore mixes quantities on different scales, and the semantic metric (sign first at 20.3%) already hints that the genuine ranking differs. Per the reviewing rule I note that the paper's Limitations (b180–b185) do not mention this measurement-level caveat, so it remains unresolved rather than acknowledged. The concern does not overturn the qualitative claims: the toxicity × sign moderation (Table 6) is mechanical-free and is the paper's cleanest result; the rubric null (b7) is conditional on one rubric instance per task, a scope limit the reader already recorded; and the practical guidance to run robustness checks across scales is reinforced, not weakened, by a mechanical reading. The single check that settles the issue is the mechanical-floor decomposition described in concrete_test, computable entirely from the paper's reported design. Since the reader's verdict already encodes exactly this conditionality, I recommend no change: UNCHANGED.
The referee's report
The question of cross-scale comparability is important and understudied for LLM-based measurement, and this paper attacks it with an unusually thorough design. Strengths I want to credit explicitly: the minimal-pair logic isolates single design elements; the sign manipulation is a clean identification because signed and unsigned grids coincide after normalization (zero mechanical floor); uncertainty is quantified with item-level bootstrap (2,000 resamples); the variance decomposition (Table 7) has correct degrees of freedom and sums to 100%; supplementary analyses on reasoning effort and human agreement broaden the scope; and the code and prompts are linked. The RQ3 null result (rubrics do not stabilize scores across scales) is a useful negative finding with a falsifiable prediction. If the attribution issues raised below are addressed, the paper's practical guidance is well supported, and the paper would establish rating scale design as a first-order, reportable measurement decision for LLM-based text scoring.
Pith's reading of the paper (a machine's summary, not the authors')
The paper asks whether the numbers on a rating scale change what an LLM's scores mean. It shows that switching a scale's format (continuous vs. ordinal), range (3, 5, 11, or 21 points), or sign (signed vs. unsigned) moves the average normalized score by about $0.10$ of the full scale and puts roughly one in five texts into a different low/middle/high category. The effect is largest for toxicity, the only unipolar construct tested, and most sensitive to whether the scale contains negative numbers. Adding a detailed scoring rubric does not reduce the sensitivity, and the choice of model matters far less than the scale itself. The authors conclude that any LLM-based score should be read together with its scale design, and that scores produced on different scales should not be pooled or compared as if they were the same measurement.
What carries the argument. The load-bearing object is the controlled minimal-pair comparison: two rating scales that differ in exactly one design element while the prompt, task, model, and text are held fixed. Scores are put on a common footing by affine normalization to $[0,1]$ (Equation 1), numerical sensitivity is the mean absolute difference between normalized scores (Equation 2), and semantic sensitivity is the fraction of texts whose normalized score crosses the $1/3$ or $2/3$ boundaries into a different low/middle/high category (Equations 3 and 4). The 16 scale conditions are arranged so that each element can be isolated: 8 minimal pairs differ only in format, 24 only in range, and 8 only in sign, yielding 40 pairs per task, prompting strategy, and model, and 1,200 sensitivity estimates in total. Because the prompt is identical except for the scale specification, any measured difference is attributed to the scale itself.
Core claim. The central claim is that rating scale design is a genuine source of variation in LLM-generated scalar scores, not a neutral implementation detail. Crossing format, range, and sign into 16 scale conditions and comparing only minimal pairs that differ in exactly one element, the paper finds mean absolute differences of $0.098$ (format), $0.098$ (range), and $0.085$ (sign) in scores normalized to $[0,1]$, with 18.2% to 20.3% of texts switching low/middle/high category under a single-element change. Sensitivity is strongest for toxicity, where signed scales push far more texts into the high-toxicity category because the models treat 0 as 'not toxic' rather than the scale's lowest endpoint. Rubric-based prompting leaves sensitivity essentially unchanged (within 0.007 of zero-shot), which the authors read as evidence that the effect comes from how models map judgments onto numerical scales rather than from underspecified evaluation criteria. The variance decomposition reinforces the point: scale design explains 40.3% of the numerical variation, and the scale-by-task interaction explains 16.0% of the semantic variation, while model and prompting strategy together explain only a few percent.
The premise it rests on. The headline result assumes that scores from different scales are comparable after a simple linear rescaling to $[0,1]$; if an LLM's judgment-to-number mapping is nonlinear — and the paper's own figures show outputs bunching at zero, midpoints, and whole numbers — then a shift of $0.10$ on one scale may not represent the same change in judgment as a shift of $0.10$ on another, and the unified headline is not a common yardstick.
If the paper is right (Pith's inferences, not the authors')
Scores from studies that use different scale designs are not interchangeable: two studies measuring the same construct can disagree by about $0.10$ of the normalized scale on average, so comparison across studies requires robustness checks.
A scale change re-categorizes roughly one in five texts as low, middle, or high, which means category shares and aggregate statistics are not directly comparable across studies.
For unipolar constructs such as toxicity, signed scales that run through zero systematically inflate the high category, so an unsigned scale whose lowest endpoint means absence of the attribute is the safer measurement choice.
Rubric-based prompting does not stabilize scores across scales, so elaborating the prompt is not a remedy; the paper recommends reporting scale design and testing findings under at least one alternative scale.
Even at the highest reasoning settings tested, numerical sensitivity falls by only $0.006$ to $0.016$, so scale sensitivity does not disappear with more reasoning effort.
Because the measured shifts may reflect how models anchor on scale features such as zero, the midpoint, and whole numbers, a post-hoc calibration that maps each scale's raw outputs through that model's own response distribution could recover a stable latent judgment; this is a testable extension the paper does not attempt.
If input-stage interventions like rubrics do not help, output-stage corrections (for example, scoring each text on several scales and averaging, or estimating a per-model transfer function between scale pairs) become the natural next experiments.
The toxicity result suggests a concrete mechanism: models may treat 0 as a semantic anchor meaning 'not toxic' rather than as an arithmetic endpoint, which would predict that signed scales bias any unipolar construct, not just toxicity.
The equal-thirds category boundaries are an analytic choice, so the 'one in five texts' figure is a property of this discretization; a substantive theory that placed different boundaries would change the semantic sensitivity numbers.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Large language models (LLMs) are increasingly used in computational social science to assign scalar scores to social and psychological constructs. However, studies use different rating scales, and whether their scores remain comparable across scale designs is understudied. To address this gap, we define and systematically investigate scale sensitivity, the variation in LLM-generated scores caused by rating scale design. We distinguish numerical sensitivity, changes in normalized scores, from semantic sensitivity, changes in score interpretation. To assess both, our experimental design crosses 2 formats (continuous vs. ordinal), 4 ranges (3, 5, 11, and 21 points), and 2 signs (signed vs. unsigned) across 3 tasks (political leaning, sentiment, and toxicity), 2 prompting strategies (zero-shot and rubric-based), and 5 LLMs. We find that changing one design element shifts normalized scores by about 0.10 on average and changes the category of roughly 20% of texts. Sensitivity is greatest for toxicity, the only unipolar construct examined. Also, rubric-based prompting does not reduce sensitivity compared with zero-shot prompting. We provide practical guidance for designing more reliable LLM-based text scoring and highlight the need for robustness checks across alternative scales.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-16
The authors' main claim, as the classifier picked it out 0.97
“We find that changing one design element shifts normalized scores by about 0.10 on average and changes the category of roughly 20% of texts.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.10 · states limitation 0.58 · human participants 0.08 · controlled comparison 0.34 · randomized assignment 0.03 · states quantitative result 0.99 · compares with prior results 0.40 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Large language models (LLMs) are increasingly used in computational social science to assign scalar scores to social and psychological constructs. | background 1.00 | 0.04 | 0.02 | asserted 0.99 |
| However, studies use different rating scales, and whether their scores remain comparable across scale designs is understudied. | gap 0.99 | 0.04 | 0.02 | asserted 0.96 |
| To address this gap, we define and systematically investigate scale sensitivity, the variation in LLM-generated scores caused by rating scale design. | contribution 0.99 | 0.82 | 0.04 | asserted 0.93 |
| We distinguish numerical sensitivity, changes in normalized scores, from semantic sensitivity, changes in score interpretation. | contribution 0.92 | 0.71 | 0.04 | asserted 0.97 |
| To assess both, our experimental design crosses 2 formats (continuous vs. ordinal), 4 ranges (3, 5, 11, and 21 points), and 2 signs (signed vs. unsigned) across 3 tasks (political leaning, sentiment, and toxicity), 2 prompting strategies (zero-shot and rubric-based), and 5 LLMs. | method 1.00 | 0.91 | 0.02 | asserted 0.94 |
| We find that changing one design element shifts normalized scores by about 0.10 on average and changes the category of roughly 20% of texts. | result 1.00 | 0.98 | 0.68 | asserted 1.00 |
| Sensitivity is greatest for toxicity, the only unipolar construct examined. | result 1.00 | 0.93 | 0.14 | asserted 1.00 |
| Also, rubric-based prompting does not reduce sensitivity compared with zero-shot prompting. | result 1.00 | 0.94 | 0.11 | asserted 1.00 |
| We provide practical guidance for designing more reliable LLM-based text scoring and highlight the need for robustness checks across alternative scales. | future work 0.73 | 0.68 | 0.19 | asserted 0.93 |
What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 0.97
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-16: 1 request, 6,888 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (11)
Figures are the authors', reproduced from the paper.
References (94)
- [1] Alfred V. Aho and Jeffrey D. Ullman , title =. 1972 on Pith
- [2] Publications Manual , year = "1983", publisher = on Pith
- [3] Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243 on Pithdoi:10.1145/322234.322243
- [4] Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of on Pith
- [5] Dan Gusfield , title =. 1997 on Pith
- [6] Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url = on Pith
- [7] Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages = on Pith
- [8] Leung, Shing-On , journal=. A comparison of psychometric properties and normality in 4-, 5-, 6-, and 11-point. 2011 , publisher= on Pith
- [9] Treiblmaier, Horst and Filzmoser, Peter , title =. Proceedings of the International Conference on Information Systems (ICIS) , year = on Pith
- [10] Cai, MY and Lin, Y and Zhang, Wen-Jun , booktitle=. Study of the optimal number of rating bars in the on Pith
- [11] Comparison of the quality of assessments using continuous and discrete ordinal rating scales , author=. Biometrical Journal: Journal of Mathematical Methods in Biosciences , volume=. 2000 , publisher= on Pith
- [12] Chang, Lei , journal=. A psychometric evaluation of 4-point and 6-point. 1994 , publisher= on Pith
- [13] Kusmaryono, Imam and Wijayanti, Dyana and Maharani, Hevy Risqi , journal=. Number of response options, reliability, validity, and potential bias in the use of the. 2022 , publisher= on Pith
- [14] Xu, Meng Lin and Leung, Shing On , journal=. Effects of varying numbers of. 2018 , publisher= on Pith
- [15] Comparison of rating scales: sensitivity, replicates and relative measurement , author=. Journal of Sensory Studies , volume=. 1986 , publisher= on Pith
- [16] Rathje, Steve and Mirea, Dan-Mircea and Sucholutsky, Ilia and Marjieh, Raja and Robertson, Claire E and Van Bavel, Jay J , journal=. 2024 , publisher= on Pith
- [17] Sclar, Melanie and Choi, Yejin and Tsvetkov, Yulia and Suhr, Alane , title =. The Twelfth International Conference on Learning Representations , year = on Pith
- [18] Licht, Hauke and Sarkar, Rupak and Wu, Patrick Y and Goel, Pranav and Stoehr, Niklas and Ash, Elliott and Hoyle, Alexander Miserlis , booktitle=. Measuring scalar constructs in social science with on Pith
- [19] Gilardi, Fabrizio and Alizadeh, Meysam and Kubli, Ma. Proceedings of the National Academy of Sciences , volume=. 2023 , publisher= on Pith
- [20] Large language models as a substitute for human experts in annotating political text , author=. Research & Politics , volume=. 2024 , publisher= on Pith
show 74 more
- [21] T. arXiv preprint arXiv:2304.06588 , year= on PithPith's reviewarXiv
- [22] Li, Lingyao and Fan, Lizhou and Atreja, Shubham and Hemphill, Libby , journal=. ``HOT''. 2024 , publisher= on Pith
- [23] O'Hagan, Sean and Schein, Aaron , journal=. Measurement in the age of on Pith
- [24] Positioning political texts with large language models by asking and averaging , author=. Political Analysis , volume=. 2025 , publisher= on Pith
- [25] Koh, Hyukhun and Kim, Dohyung and Lee, Minwoo and Jung, Kyomin , journal=. Can on Pith
- [26] How to train your stochastic parrot: Large language models for political texts , author=. Political Science Research and Methods , volume=. 2025 , publisher= on Pith
- [27] Demystifying hateful content: Leveraging large multimodal models for hateful meme detection with explainable decisions , author=. Proceedings of the International AAAI Conference on Web and Social Media , volume= on Pith
- [28] Prompting large language models for topic modeling , author=. 2023 IEEE International Conference on Big Data (BigData) , pages=. 2023 , organization= on Pith
- [29] Promptmtopic: Unsupervised multimodal topic modeling of memes using large language models , author=. Proceedings of the 31st ACM International Conference on Multimedia , pages= on Pith
- [30] Accuracy of a large language model in distinguishing anti-and pro-vaccination messages on social media: The case of human papillomavirus vaccination , author=. Preventive Medicine Reports , volume=. 2024 , publisher= on Pith
- [31] Kuang, Xianwen and Liu, Jun and Zhang, Haiyang and Schweighofer, Simon , journal=. Towards algorithmic framing analysis: expanding the scope by using. 2025 , publisher= on Pith
- [32] Movva, Rajiv and Koh, Pang Wei and Pierson, Emma , journal=. Annotation alignment: Comparing on Pith
- [33] Kristensen-McLachlan, Ross Deans and Canavan, Miceal and K. Are chatbots reliable text annotators?. PNAS Nexus , year =. doi:10.1093/pnasnexus/pgaf069 , url = on Pithdoi:10.1093/pnasnexus/pgaf069
- [34] A Crisis of Civility? Modeling Incivility and Its Effects in Political Discourse Online , author=. Proceedings of the International AAAI Conference on Web and Social Media , volume= on Pith
- [35] Incivility in congressional tweets , author=. American Politics Research , volume=. 2022 , publisher= on Pith
- [36] Yu, Xudong and Wojcieszak, Magdalena and Casas, Andreu , journal=. Partisanship on social media: In-party love among. 2024 , publisher= on Pith
- [37] Out-group animosity drives engagement on social media , author=. Proceedings of the National Academy of Sciences , volume=. 2021 , publisher= on Pith
- [38] Toxic Bias: Perspective API misreads German as more toxic , author=. Proceedings of the International AAAI Conference on Web and Social Media , volume= on Pith
- [39] Moralization in social networks and the emergence of violence during protests , author=. Nature Human Behaviour , volume=. 2018 , publisher= on Pith
- [40] How social learning amplifies moral outrage expression in online social networks , author=. Science Advances , volume=. 2021 , publisher= on Pith
- [41] Negativity drives online news consumption , author=. Nature Human Behaviour , volume=. 2023 , publisher= on Pith
- [42] Sentiment analysis of news articles: a lexicon based approach , author=. 2019 2nd International Conference on Computing, Mathematics and Engineering Technologies (iCoMET) , pages=. 2019 , organization= on Pith
- [43] Sentiment analysis of comments in social media. , author=. International Journal of Electrical & Computer Engineering (2088-8708) , volume= on Pith
- [44] Uncertainty detection in natural language: A probabilistic model , author=. Proceedings of the 6th International Conference on Web Intelligence, Mining and Semantics , pages= on Pith
- [45] A language model-based generative classifier for sentence-level discourse parsing , author=. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages= on Pith
- [46] Partisan stereotypes and policy attitudes , author=. Journal of Communication , volume=. 2012 , publisher= on Pith
- [47] Informing the electorate? How party cues and policy information affect public opinion about initiatives , author=. American Journal of Political Science , volume=. 2014 , publisher= on Pith
- [48] Emotion: The science of sentiment , author=. 2002 , publisher= on Pith
- [49] Defining and detecting toxicity on social media: context and knowledge are key , author=. Neurocomputing , volume=. 2022 , publisher= on Pith
- [50] Pandita, Swati and Garg, Ketika and Zhang, Jiajin and Mobbs, Dean , title =. Trends in Cognitive Sciences , year = on Pith
- [51] Jo, Claire Wonjeong and Weso. Harmful. arXiv preprint arXiv:2411.05854 , year =. doi:10.48550/arXiv.2411.05854 , url = on PithPith's reviewarXivdoi:10.48550/arxiv.2411.05854
- [52] Wu, Hoi Yan and Leung, Sing Oi , title =. Journal of Social Service Research , year = on Pith
- [53] Proceedings of the iConference 2021 , author =. 2021 , location = on Pith
- [54] Hutto, Clayton and Gilbert, Eric , title =. Proceedings of the International AAAI Conference on Web and Social Media (ICWSM) , year = on Pith
- [55] A comprehensive survey of prompt engineering techniques in large language models , author=. TechRxiv , year= on Pith
- [56] A survey on uncertainty quantification of large language models: Taxonomy, open research challenges, and future directions , author=. ACM Computing Surveys , year= on Pith
- [57] Achiam, Josh and Adler, Steven and Agarwal, Sandhini and Ahmad, Lama and Akkaya, Ilge and Aleman, Florencia Leoni and Almeida, Diogo and Altenschmidt, Janko and Altman, Sam and Anadkat, Shyamal and others , journal= on Pith
- [58] Liu, Aixin and Mei, Aoxue and Lin, Bangcai and Xue, Bing and Wang, Bingxuan and Xu, Bingzheng and Wu, Bochao and Zhang, Bowei and Lin, Chaofan and Dong, Chen and others , journal= on Pith
- [59] Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year= on PithPith's reviewarXiv
- [60] Mistral , year=. Mistral Small 3.1 , url= on Pith
- [61] Google , year=. Gemini 3.1 Flash-Lite: Built for intelligence at scale , url= on Pith
- [62] Li, Dawei and Jiang, Bohan and Huang, Liangjie and Beigi, Alimohammad and Zhao, Chengshuai and Tan, Zhen and Bhattacharjee, Amrita and Jiang, Yuxuan and Chen, Canyu and Wu, Tianhao and others , booktitle=. From generation to judgment: Opportunities and challenges of on Pith
- [63] JudgeLM: Fine-tuned Large Language Models are Scalable Judges , author=. The Thirteenth International Conference on Learning Representations , year= on Pith
- [64] He, Xingwei and Lin, Zhenghao and Gong, Yeyun and Jin, A-Long and Zhang, Hang and Lin, Chen and Jiao, Jian and Yiu, Siu Ming and Duan, Nan and Chen, Weizhu , booktitle= on Pith
- [65] Wan, Mengting and Safavi, Tara and Jauhar, Sujay Kumar and Kim, Yujin and Counts, Scott and Neville, Jennifer and Suri, Siddharth and Shah, Chirag and White, Ryen W and Yang, Longqi and others , booktitle= on Pith
- [66] Language Models are Few-Shot Learners , author=. Advances in Neural Information Processing Systems , volume= on Pith
- [67] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models , author=. Advances in Neural Information Processing Systems , volume= on Pith
- [68] Mistralai , year=. mistralai/Mistral-Small-3.2-24B-Instruct-2506 , url= on Pith
- [69] OpenAI , year=. Introducing on Pith
- [70] Google , year=. Gemini 3 Flash: frontier intelligence built for speed , url= on Pith
- [71] Difar: Enhancing multimodal misinformation detection with diverse, factual, and relevant rationales , author=. arXiv preprint arXiv:2508.10444 , year= on PithPith's reviewarXiv
- [72] Large language models are better reasoners with self-verification , author=. Findings of the Association for Computational Linguistics: EMNLP 2023 , pages= on Pith
- [73] Sharma, Manasi and Zhang, Chen Bo Calvin and Bandi, Chaithanya and Wang, Clinton and Aich, Ankit and Nghiem, Huy and Rabbani, Tahseen and Htet, Ye and Jang, Brian and Basu, Sumana and others , journal= on Pith
- [74] Eltanbouly, Sohaila and Albatarni, Salam and Elsayed, Tamer , booktitle= on Pith
- [75] Li, Gaotang and Mishra, Bhavana Dalvi and Wang, Zifeng and Yan, Jun and Chen, Yanfei and Li, Chun-Liang and Le, Long T and Han, Rujun and Lee, George and Tong, Hanghang and others , journal= on Pith
- [76] He, Jia and Rungta, Mukund and Koleczek, David and Sekhon, Arshdeep and Wang, Franklin X and Hasan, Sadid , journal=. Does prompt formatting have any impact on on Pith
- [77] The method of constructing an attitude scale , author=. Scaling , pages=. 2017 , publisher= on Pith
- [78] Reliability and validity testing of a new scale for measuring attitudes toward learning statistics with technology. , author=. Acta Didactica Napocensia , volume=. 2011 , publisher= on Pith
- [79] Guan, Bryan and Roosta, Tanya and Passban, Peyman and Rezagholizadeh, Mehdi , journal=. The order effect: investigating prompt sensitivity to input order in on Pith
- [80] Zhuo, Jingming and Zhang, Songyang and Fang, Xinyu and Duan, Haodong and Lin, Dahua and Chen, Kai , booktitle= on Pith
- [81] Hashemi, Helia and Eisner, Jason and Rosset, Corby and Van Durme, Benjamin and Kedzie, Chris , booktitle= on Pith
- [82] 2026 , month = aug, url = on Pith
- [83] 2026 , month = jul, url = on Pith
- [84] Schwarz, Norbert and Kn. Rating Scales: Numeric Values May Change the Meaning of Scale Labels , journal =. 1991 , volume =. doi:10.1086/269282 , url = on Pithdoi:10.1086/269282
- [85] Tourangeau, Roger and Couper, Mick P. and Conrad, Frederick , title =. Public Opinion Quarterly , year =. doi:10.1093/poq/nfl046 , url = on Pithdoi:10.1093/poq/nfl046
- [86] Chochlakis, Georgios and Maruthu Pandiyan, Niyantha and Lerman, Kristina and Narayanan, Shrikanth , title =. arXiv preprint arXiv:2409.06173 , year =. doi:10.48550/arXiv.2409.06173 , url = on PithPith's reviewarXivdoi:10.48550/arxiv.2409.06173
- [87] Dhuliawala, Shehzaad and Komeili, Mojtaba and Xu, Jing and Raileanu, Roberta and Li, Xian and Celikyilmaz, Asli and Weston, Jason , title =. Findings of the Association for Computational Linguistics: ACL 2024 , year =. doi:10.18653/v1/2024.findings-acl.212 , url = on Pithdoi:10.18653/v1/2024.findings-acl.212
- [88] Treier, Shawn and Hillygus, D. Sunshine , title =. Public Opinion Quarterly , year =. doi:10.1093/poq/nfp067 , url = on Pithdoi:10.1093/poq/nfp067
- [89] Feldman, Stanley and Johnston, Christopher , title =. Political Psychology , year =. doi:10.1111/pops.12055 , url = on Pithdoi:10.1111/pops.12055
- [90] Volkens, Andrea and Burst, Tobias and Krause, Werner and Lehmann, Pola and Matthie. The. 2021 , edition = on Pith
- [91] Pang, Bo and Lee, Lillian , title =. Foundations and Trends in Information Retrieval , year =. doi:10.1561/1500000011 , url = on Pithdoi:10.1561/1500000011
- [92] n.d. , howpublished = on Pith
- [93] Borkan, Daniel and Dixon, Lucas and Sorensen, Jeffrey and Thain, Nithum and Vasserman, Lucy , title =. Companion Proceedings of the 2019 World Wide Web Conference , year =. doi:10.1145/3308560.3317593 , url = on Pithdoi:10.1145/3308560.3317593
- [94] Liusie, Adian and Manakul, Potsawee and Gales, Mark. LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models. Proceedings of the 18th Conference of the E uropean Chapter of the A ssociation for C omputational L inguistics (... on Pithdoi:10.18653/v1/2024.eacl-long.8
Cited by 1 Pith paper
Reviewed papers on Pith that cite this one. The full citation record
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment