Pith.

Computation and Language · arXiv:2610.22187 v1 · submitted 15 Oct 2026 · 94 references

Change the Number, Change the Answer: Assessing Rating Scale Sensitivity in LLM-Based Text Scoring

Wonjeong Jo, Gyuwan Kim, Ming Shan Hee

The paper says

“Large language models (LLMs) are increasingly used in computational social science to assign scalar scores to social and psychological constructs. However, studies use different rating scales, and whether their scores remain comparable across scale designs is understudied.”

more of the abstract

To address this gap, we define and systematically investigate scale sensitivity, the variation in LLM-generated scores caused by rating scale design. We distinguish numerical sensitivity, changes in normalized scores, from semantic sensitivity, changes in score interpretation. To assess both, our experimental design crosses 2 formats (continuous vs. ordinal), 4 ranges (3, 5, 11, and 21 points), and 2 signs (signed vs. unsigned) across 3 tasks (political leaning, sentiment, and toxicity), 2 prompting strategies (zero-shot and rubric-based), and 5 LLMs. We find that changing one design element shifts normalized scores by about 0.10 on average and changes the category of roughly 20% of texts. Sensitivity is greatest for toxicity, the only unipolar construct examined. Also, rubric-based prompting does not reduce sensitivity compared with zero-shot prompting. We provide practical guidance for designing more reliable LLM-based text scoring and highlight the need for robustness checks across alternative scales.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 16 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. The practical conclusion holds—scale design changes LLM scores—but the 0.10 headline mixes real scale effects with a discretization floor; the paper deserves refereeing with a request for a simulation baseline. This line is wrong

2 major objections · 7 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 94 references say.

The paper says, at §3.5.3 (Eq. 2), §4.1, Appendix D

The review did not quote this passage.

Pith says: Objection 1 of 2 · major

SSMAE conflates genuine scale-induced judgment change with the mechanical effect of output-grid coarseness, and this attribution gap is not acknowledged in the Limitations (b180–b185). The paper's formal definition of scale sensitivity is output-level, and the output-level measurements are not in question; the problem is that the abstract and conclusion (b5, b23, b165) present the 0.10 shift as a property of LLM judgment, which requires a decomposition that Eq.

more of objection 1

(2) cannot provide. Appendix D itself documents that model outputs concentrate on midpoints and whole numbers (the vertical steps in Figure 8), so the judgment-to-number map is coarse even in continuous conditions. Under the null that the model's judgment is identical across formats and the ordinal output is just the continuous judgment rounded to the nearest grid point, a uniform latent on [0,1] gives a mean absolute rounding difference of 0.125 on a 3-point grid and 0.0125 on a 21-point grid. The floor is therefore 86% of the observed 3-point format effect (0.125 of 0.146, §4.1) but only 16% of the observed 21-point effect; the data are consistent with the coarse-scale effect being almost entirely mechanical, and the headline 0.10, which averages over conditions with very different floors, cannot be read as the amount by which the scale changes the model's judgment of a text. The claim that finer scales reduce numerical format sensitivity (b129) must survive floor removal, and the paper does not provide the decomposition needed to check this. I recommend a concrete fix: for each format and range pair, take the continuous-condition outputs as the empirical latent proxy, round them to the coarser grid, and compute the implied floor SSMAE and SSswitch, reporting the residual as the genuine component; at minimum, report within-pair rank correlations to separate monotone quantization from re-ranking. The semantic findings are more robust to this critique, since at 21 points the rounding floor for SSswitch is about 2% while the observed category-switch rate stays near 18–20%.

The paper says, at §4.1 (b120–b127), §4.2, Table 6

The review did not quote this passage.

Pith says: Objection 2 of 2 · major

The comparative claims across design elements are not on a common scale. Sign minimal pairs have identical normalized grids on both sides, so their mechanical floor is exactly zero, while format and range pairs carry positive floors. The conclusions that the magnitudes are "broadly similar" across elements (b126) and that sign has the smallest numerical sensitivity (0.085 vs. 0.098 and 0.098, b120–b124) are therefore not established: after floor removal, the ordering could reverse.

more of objection 2

The sign-based results—the toxicity-by-sign moderation (+0.080 on SSswitch, Table 6) and the zero-anchoring effect documented in Appendix D and Figure 9—are the cleanest evidence in the paper precisely because they are floor-free; the paper should say so explicitly and foreground these findings, while re-expressing the format and range results with floor-removed estimates.

7 minor objections

The paper says, at Abstract

Pith says: minor objection 1 of 7

Missing space in "systematically investigatescale sensitivity".

The paper says, at §5

Pith says: minor objection 2 of 7

"Therefore, We recommend unsigned scales" capitalizes "We" mid-sentence; the sentence should continue with lowercase "we".

show 5 more

The paper says, at Figure 3

Pith says: minor objection 3 of 7

The caption reads "Continuous scales (b) Ordinal scales (b)"; the second "(b)" appears to be duplicated.

The paper says, at Figure 8

Pith says: minor objection 4 of 7

The 16-panel cumulative-distribution figure with light-to-dark lines for scale lengths is hard to parse at print size; a shared legend or a small-multiple redesign would help.

The paper says, at §3.5.5

Pith says: minor objection 5 of 7

The claim that "every 95% confidence interval lies within 0.01 of its estimate" is stronger than what is reported; state the maximum half-width or the relevant percentile so the claim is verifiable.

The paper says, at Table 2

Pith says: minor objection 6 of 7

Unsigned 3- and 5-point scales run from 1 to N while unsigned 11- and 21-point scales run from 0 to N (footnote 2); this means unsigned range pairs such as [1,3] vs. [0,10] vary the low anchor as well as the number of points, and this minor confound is worth acknowledging when interpreting range effects on unsigned scales.

The paper says, at Table 5

Pith says: minor objection 7 of 7

The open models run at temperature 1.0 with a single generation per item-condition (as far as reported), and the item bootstrap does not capture generation stochasticity, so the reported intervals likely understate total uncertainty; a same-condition repeated-prompt control, or seed-controlled generation, would bound this noise floor.

What would settle it

Decompose SSMAE into mechanical and genuine components from the paper's own paired outputs. For each format minimal pair (continuous vs ordinal at range r), compute the floor B = mean_i |z_cont,i − round_r(z_cont,i)|, where round_r maps to the nearest grid point {0, 1/(r−1), …, 1}, and compare B with the observed mean_i |z_cont,i − z_ord,i|; for range pairs, round the finer condition's outputs to the coarser grid; for sign pairs B = 0 by construction. If observed SSMAE exceeds B by less than about 0.02 for the 3- and 5-point format pairs and for coarse-vs-fine range pairs, then the headline 0.10 (b5) is largely mechanical, and the format/range magnitudes and their ranking against sign (b120–b124) must be re-reported as rebaselined (genuine) estimates; if the excess is large, the concern is resolved. Also overlay B as a dashed line in Figure 3.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 16 Oct 2026

  1. Reader holds up, with conditions confidence high
  2. Skeptic agrees 3 min 3 s · 24,037 tokens read, 22,763 written
  3. Editor wrote the verdict above 29 s · 25,096 tokens read, 3,206 written
  4. Referee major revision: 2 major objections, 7 minor 3 min 8 s · 24,314 tokens read, 25,344 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

What you should know: this is a careful, honest measurement paper, and its central practical claim survives scrutiny. Changing one rating scale element shifts normalized LLM scores by about 0.10 and switches the category of roughly one in five texts. Those are real scale-design effects. What is oversold is the causal gloss that they reflect changes in LLM judgment rather than partly mechanical consequences of the output grid.

What is new and good: nobody has systematically crossed format, range, and sign for LLM scalar scoring with minimal pairs, and the numerical/semantic distinction is genuinely useful. The design is solid: 16 conditions, 40 minimal pairs, 1,200 estimates per metric, item bootstrap with 2,000 resamples, WLS moderation, and a variance decomposition that sums to 100% with correct degrees of freedom. The paper also states its own limitations up front (b180–b185) and refuses to treat human labels as ground truth (b359–b361). The toxicity-by-sign interaction is a clean and interesting finding, and the reasoning-level and human-agreement appendices add value beyond the headline.

The soft spots are real but not fatal. The main one is unacknowledged: SSMAE mixes genuine scale-induced shift with a mechanical rounding floor. Appendix D shows outputs pile up on midpoints and whole numbers. For a 3-point ordinal scale, a uniform latent score rounded to the grid differs from the continuous value by E|u−round(u)| = 0.125, against an observed 3-point format effect of 0.146. So most of that particular number could be discretization, not a change in the model's judgment. This also undercuts the cross-element ranking: sign minimal pairs have identical grids and zero floor, while format and range pairs carry positive floors, so comparing the 0.098/0.098/0.085 averages in b120–b124 is comparing quantities of different kinds. The toxicity × sign moderation is not affected by this problem and is the paper's strongest specific result. Two smaller issues: the rubric null result rests on one authored rubric instance per task, and no code, output data, or commit hash is provided, so replication means regenerating millions of API calls.

Even with the mechanical floor, the practical takeaway stands: scores from studies using different scale designs are not interchangeable, and robustness checks are warranted. That is a useful message for anyone doing LLM-based measurement.

Bottom line: worth a serious referee. The revision should add a simulation baseline that separates the rounding floor from genuine judgment shift, and ideally release the outputs. I would cite this paper and would bring it to a reading group.

The strongest objection

The central claim (b5, b23) — a one-element scale change shifts normalized scores by about 0.10 and recategorizes one in five texts — is presented as a property of LLM judgment. That reading requires Eq. 1's affine normalization and Eq. 2's SSMAE to measure judgment change, not output-constraint artifacts. The paper's own Appendix D reports that outputs concentrate on midpoints and whole numbers (vertical steps in Figure 8), so the judgment-to-number map is coarse even in continuous conditions; SSMAE therefore mixes genuine scale-induced shift with a mechanical floor: the deterministic distance between an unrounded score and its nearest grid point, which appears even for a perfectly stable latent judgment. No baseline or simulation separates the two, and the Limitations (b180–b185) do not flag this attribution gap. Quantitatively, for a uniform latent the 3-point rounding floor is E|u − round(u)| = 0.125, the largest any distribution can produce; the observed 3-point format effect is 0.146 (Section 4.1), so the data are consistent with that effect being almost entirely mechanical. At 21 points the floor is 0.0125 versus an observed 0.078, so a genuine component exists at fine granularity; where between these extremes the split lies is an empirical question the paper leaves open. The comparative claims inherit the problem: sign minimal pairs have identical grids on both sides (mechanical floor exactly zero), while format and range pairs carry positive floors, so the numerical ranking format ≈ range (0.098) > sign (0.085) in b120–b124 compares quantities of different kinds; the paper's own semantic ordering (sign largest, 20.3%) already suggests the genuine ranking may differ. The toxicity × sign moderation (Table 6, +0.080 SSswitch) is mechanical-free and survives.

The reader's verdict is CONDITIONAL with Eq. 1's linear comparability as the weakest assumption; my stress-test agrees that this is the single most load-bearing concern and that it lands in a bounded way. The paper is a well-executed measurement study: the 2×4×2 design with 40 minimal pairs isolates design elements, item bootstrap with 2,000 resamples is honestly reported, the variance decomposition (Table 7) sums to 100% with correct degrees of freedom, and Appendix F documents the regression weighting. I credit these. My pass refines the reader's concern in three ways. First, it localizes the failure: affine normalization (Eq. 1) is defensible as a descriptive rescaling; the load-bearing gap is the missing mechanical floor, since a coarse ordinal grid guarantees nonzero SSMAE even for a perfectly stable latent judgment. Second, it bounds the stakes honestly: the 0.125 three-point rounding figure is the maximum possible floor for any output distribution (uniform achieves it), so the observed 0.146 is at most 17% above a worst-case mechanical account — the data are consistent with the 3-point format effect being almost entirely mechanical, and also with it being largely genuine; the paper provides no way to tell, and its own output distributions (Appendix D, Figure 8) make the true floor an empirical quantity rather than a known one. Third, it spells out a consequence the reader implied but did not state: sign pairs have identical grids on both sides, so their floor is exactly zero, whereas format and range pairs have positive floors; the numerical ranking format ≈ range > sign (b120–b124) therefore mixes quantities on different scales, and the semantic metric (sign first at 20.3%) already hints that the genuine ranking differs. Per the reviewing rule I note that the paper's Limitations (b180–b185) do not mention this measurement-level caveat, so it remains unresolved rather than acknowledged. The concern does not overturn the qualitative claims: the toxicity × sign moderation (Table 6) is mechanical-free and is the paper's cleanest result; the rubric null (b7) is conditional on one rubric instance per task, a scope limit the reader already recorded; and the practical guidance to run robustness checks across scales is reinforced, not weakened, by a mechanical reading. The single check that settles the issue is the mechanical-floor decomposition described in concrete_test, computable entirely from the paper's reported design. Since the reader's verdict already encodes exactly this conditionality, I recommend no change: UNCHANGED.

The referee's report

The question of cross-scale comparability is important and understudied for LLM-based measurement, and this paper attacks it with an unusually thorough design. Strengths I want to credit explicitly: the minimal-pair logic isolates single design elements; the sign manipulation is a clean identification because signed and unsigned grids coincide after normalization (zero mechanical floor); uncertainty is quantified with item-level bootstrap (2,000 resamples); the variance decomposition (Table 7) has correct degrees of freedom and sums to 100%; supplementary analyses on reasoning effort and human agreement broaden the scope; and the code and prompts are linked. The RQ3 null result (rubrics do not stabilize scores across scales) is a useful negative finding with a falsifiable prediction. If the attribution issues raised below are addressed, the paper's practical guidance is well supported, and the paper would establish rating scale design as a first-order, reportable measurement decision for LLM-based text scoring.

Pith's reading of the paper (a machine's summary, not the authors')

The paper asks whether the numbers on a rating scale change what an LLM's scores mean. It shows that switching a scale's format (continuous vs. ordinal), range (3, 5, 11, or 21 points), or sign (signed vs. unsigned) moves the average normalized score by about $0.10$ of the full scale and puts roughly one in five texts into a different low/middle/high category. The effect is largest for toxicity, the only unipolar construct tested, and most sensitive to whether the scale contains negative numbers. Adding a detailed scoring rubric does not reduce the sensitivity, and the choice of model matters far less than the scale itself. The authors conclude that any LLM-based score should be read together with its scale design, and that scores produced on different scales should not be pooled or compared as if they were the same measurement.

What carries the argument. The load-bearing object is the controlled minimal-pair comparison: two rating scales that differ in exactly one design element while the prompt, task, model, and text are held fixed. Scores are put on a common footing by affine normalization to $[0,1]$ (Equation 1), numerical sensitivity is the mean absolute difference between normalized scores (Equation 2), and semantic sensitivity is the fraction of texts whose normalized score crosses the $1/3$ or $2/3$ boundaries into a different low/middle/high category (Equations 3 and 4). The 16 scale conditions are arranged so that each element can be isolated: 8 minimal pairs differ only in format, 24 only in range, and 8 only in sign, yielding 40 pairs per task, prompting strategy, and model, and 1,200 sensitivity estimates in total. Because the prompt is identical except for the scale specification, any measured difference is attributed to the scale itself.

Core claim. The central claim is that rating scale design is a genuine source of variation in LLM-generated scalar scores, not a neutral implementation detail. Crossing format, range, and sign into 16 scale conditions and comparing only minimal pairs that differ in exactly one element, the paper finds mean absolute differences of $0.098$ (format), $0.098$ (range), and $0.085$ (sign) in scores normalized to $[0,1]$, with 18.2% to 20.3% of texts switching low/middle/high category under a single-element change. Sensitivity is strongest for toxicity, where signed scales push far more texts into the high-toxicity category because the models treat 0 as 'not toxic' rather than the scale's lowest endpoint. Rubric-based prompting leaves sensitivity essentially unchanged (within 0.007 of zero-shot), which the authors read as evidence that the effect comes from how models map judgments onto numerical scales rather than from underspecified evaluation criteria. The variance decomposition reinforces the point: scale design explains 40.3% of the numerical variation, and the scale-by-task interaction explains 16.0% of the semantic variation, while model and prompting strategy together explain only a few percent.

The premise it rests on. The headline result assumes that scores from different scales are comparable after a simple linear rescaling to $[0,1]$; if an LLM's judgment-to-number mapping is nonlinear — and the paper's own figures show outputs bunching at zero, midpoints, and whole numbers — then a shift of $0.10$ on one scale may not represent the same change in judgment as a shift of $0.10$ on another, and the unified headline is not a common yardstick.

If the paper is right (Pith's inferences, not the authors')

Scores from studies that use different scale designs are not interchangeable: two studies measuring the same construct can disagree by about $0.10$ of the normalized scale on average, so comparison across studies requires robustness checks.

A scale change re-categorizes roughly one in five texts as low, middle, or high, which means category shares and aggregate statistics are not directly comparable across studies.

For unipolar constructs such as toxicity, signed scales that run through zero systematically inflate the high category, so an unsigned scale whose lowest endpoint means absence of the attribute is the safer measurement choice.

Rubric-based prompting does not stabilize scores across scales, so elaborating the prompt is not a remedy; the paper recommends reporting scale design and testing findings under at least one alternative scale.

Even at the highest reasoning settings tested, numerical sensitivity falls by only $0.006$ to $0.016$, so scale sensitivity does not disappear with more reasoning effort.

Because the measured shifts may reflect how models anchor on scale features such as zero, the midpoint, and whole numbers, a post-hoc calibration that maps each scale's raw outputs through that model's own response distribution could recover a stable latent judgment; this is a testable extension the paper does not attempt.

If input-stage interventions like rubrics do not help, output-stage corrections (for example, scoring each text on several scales and averaging, or estimating a per-model transfer function between scale pairs) become the natural next experiments.

The toxicity result suggests a concrete mechanism: models may treat 0 as a semantic anchor meaning 'not toxic' rather than as an arithmetic endpoint, which would predict that signed scales bias any unipolar construct, not just toxicity.

The equal-thirds category boundaries are an analytic choice, so the 'one in five texts' figure is a property of this discretization; a substantive theory that placed different boundaries would change the semantic sensitivity numbers.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Large language models (LLMs) are increasingly used in computational social science to assign scalar scores to social and psychological constructs. However, studies use different rating scales, and whether their scores remain comparable across scale designs is understudied. To address this gap, we define and systematically investigate scale sensitivity, the variation in LLM-generated scores caused by rating scale design. We distinguish numerical sensitivity, changes in normalized scores, from semantic sensitivity, changes in score interpretation. To assess both, our experimental design crosses 2 formats (continuous vs. ordinal), 4 ranges (3, 5, 11, and 21 points), and 2 signs (signed vs. unsigned) across 3 tasks (political leaning, sentiment, and toxicity), 2 prompting strategies (zero-shot and rubric-based), and 5 LLMs. We find that changing one design element shifts normalized scores by about 0.10 on average and changes the category of roughly 20% of texts. Sensitivity is greatest for toxicity, the only unipolar construct examined. Also, rubric-based prompting does not reduce sensitivity compared with zero-shot prompting. We provide practical guidance for designing more reliable LLM-based text scoring and highlight the need for robustness checks across alternative scales.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-16

The authors' main claim, as the classifier picked it out 0.97

“We find that changing one design element shifts normalized scores by about 0.10 on average and changes the category of roughly 20% of texts.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.10 · states limitation 0.58 · human participants 0.08 · controlled comparison 0.34 · randomized assignment 0.03 · states quantitative result 0.99 · compares with prior results 0.40 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Large language models (LLMs) are increasingly used in computational social science to assign scalar scores to social and psychological constructs. background 1.00 0.04 0.02 asserted 0.99
However, studies use different rating scales, and whether their scores remain comparable across scale designs is understudied. gap 0.99 0.04 0.02 asserted 0.96
To address this gap, we define and systematically investigate scale sensitivity, the variation in LLM-generated scores caused by rating scale design. contribution 0.99 0.82 0.04 asserted 0.93
We distinguish numerical sensitivity, changes in normalized scores, from semantic sensitivity, changes in score interpretation. contribution 0.92 0.71 0.04 asserted 0.97
To assess both, our experimental design crosses 2 formats (continuous vs. ordinal), 4 ranges (3, 5, 11, and 21 points), and 2 signs (signed vs. unsigned) across 3 tasks (political leaning, sentiment, and toxicity), 2 prompting strategies (zero-shot and rubric-based), and 5 LLMs. method 1.00 0.91 0.02 asserted 0.94
We find that changing one design element shifts normalized scores by about 0.10 on average and changes the category of roughly 20% of texts. result 1.00 0.98 0.68 asserted 1.00
Sensitivity is greatest for toxicity, the only unipolar construct examined. result 1.00 0.93 0.14 asserted 1.00
Also, rubric-based prompting does not reduce sensitivity compared with zero-shot prompting. result 1.00 0.94 0.11 asserted 1.00
We provide practical guidance for designing more reliable LLM-based text scoring and highlight the need for robustness checks across alternative scales. future work 0.73 0.68 0.19 asserted 0.93

What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 0.97

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-16: 1 request, 6,888 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (11)
Figure 1 of the paper
Figure 1. Experimental design for evaluating scale sensitivity. topic or framing analysis (Wang et al., 2023; Kuang et al., 2025). Many constructs in the social sciences are bet￾ter understood when measured along a continuum rather than as discrete categories (Licht et al., 2025). For example, concepts such as ideological extrem￾ity, toxicity severity, and moral emotions vary in de￾gree rather than kind. To incorporate these nuances and gradations, a growing number of researchers prompt LLMs to generate scalar scores on numer￾ical scales, treating model outputs as continuous or ordinal…
Figure 2 of the paper
Figure 2. Scale sensitivity by rating scale design ele
Figure 3 of the paper
Figure 3. (a) Format sensitivity on scales with different
Figure 4 of the paper
Figure 4. Scale sensitivity across tasks, prompting strategies, and models (RQ2–RQ4), averaged across the remaining
Figure 5 of the paper
Figure 5. Share of the variation in scale sensitivity ex
Figure 6 of the paper
Figure 6. Zero-shot prompt for political leaning. B.1 Zero-Shot Prompt
Figure 7 of the paper
Figure 7. Lines that the rubric-based prompt adds to the
Figure 8 of the paper
Figure 8. Cumulative distribution of normalized scores
Figure 9 of the paper
Figure 9. Share of texts in the low, middle, and high
Figure 10 of the paper
Figure 10. Scale sensitivity at three reasoning levels for the three open models. The black line is the average over
Figure 11 of the paper
Figure 11. Agreement between LLM scores and human labels for every model, prompting strategy, and rating scale

Figures are the authors', reproduced from the paper.

References (94)
  1. [1] Alfred V. Aho and Jeffrey D. Ullman , title =. 1972 on Pith
  2. [2] Publications Manual , year = "1983", publisher = on Pith
  3. [3] Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243 on Pithdoi:10.1145/322234.322243
  4. [4] Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of on Pith
  5. [5] Dan Gusfield , title =. 1997 on Pith
  6. [6] Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url = on Pith
  7. [7] Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages = on Pith
  8. [8] Leung, Shing-On , journal=. A comparison of psychometric properties and normality in 4-, 5-, 6-, and 11-point. 2011 , publisher= on Pith
  9. [9] Treiblmaier, Horst and Filzmoser, Peter , title =. Proceedings of the International Conference on Information Systems (ICIS) , year = on Pith
  10. [10] Cai, MY and Lin, Y and Zhang, Wen-Jun , booktitle=. Study of the optimal number of rating bars in the on Pith
  11. [11] Comparison of the quality of assessments using continuous and discrete ordinal rating scales , author=. Biometrical Journal: Journal of Mathematical Methods in Biosciences , volume=. 2000 , publisher= on Pith
  12. [12] Chang, Lei , journal=. A psychometric evaluation of 4-point and 6-point. 1994 , publisher= on Pith
  13. [13] Kusmaryono, Imam and Wijayanti, Dyana and Maharani, Hevy Risqi , journal=. Number of response options, reliability, validity, and potential bias in the use of the. 2022 , publisher= on Pith
  14. [14] Xu, Meng Lin and Leung, Shing On , journal=. Effects of varying numbers of. 2018 , publisher= on Pith
  15. [15] Comparison of rating scales: sensitivity, replicates and relative measurement , author=. Journal of Sensory Studies , volume=. 1986 , publisher= on Pith
  16. [16] Rathje, Steve and Mirea, Dan-Mircea and Sucholutsky, Ilia and Marjieh, Raja and Robertson, Claire E and Van Bavel, Jay J , journal=. 2024 , publisher= on Pith
  17. [17] Sclar, Melanie and Choi, Yejin and Tsvetkov, Yulia and Suhr, Alane , title =. The Twelfth International Conference on Learning Representations , year = on Pith
  18. [18] Licht, Hauke and Sarkar, Rupak and Wu, Patrick Y and Goel, Pranav and Stoehr, Niklas and Ash, Elliott and Hoyle, Alexander Miserlis , booktitle=. Measuring scalar constructs in social science with on Pith
  19. [19] Gilardi, Fabrizio and Alizadeh, Meysam and Kubli, Ma. Proceedings of the National Academy of Sciences , volume=. 2023 , publisher= on Pith
  20. [20] Large language models as a substitute for human experts in annotating political text , author=. Research & Politics , volume=. 2024 , publisher= on Pith
show 74 more
  1. [21] T. arXiv preprint arXiv:2304.06588 , year= on PithPith's reviewarXiv
  2. [22] Li, Lingyao and Fan, Lizhou and Atreja, Shubham and Hemphill, Libby , journal=. ``HOT''. 2024 , publisher= on Pith
  3. [23] O'Hagan, Sean and Schein, Aaron , journal=. Measurement in the age of on Pith
  4. [24] Positioning political texts with large language models by asking and averaging , author=. Political Analysis , volume=. 2025 , publisher= on Pith
  5. [25] Koh, Hyukhun and Kim, Dohyung and Lee, Minwoo and Jung, Kyomin , journal=. Can on Pith
  6. [26] How to train your stochastic parrot: Large language models for political texts , author=. Political Science Research and Methods , volume=. 2025 , publisher= on Pith
  7. [27] Demystifying hateful content: Leveraging large multimodal models for hateful meme detection with explainable decisions , author=. Proceedings of the International AAAI Conference on Web and Social Media , volume= on Pith
  8. [28] Prompting large language models for topic modeling , author=. 2023 IEEE International Conference on Big Data (BigData) , pages=. 2023 , organization= on Pith
  9. [29] Promptmtopic: Unsupervised multimodal topic modeling of memes using large language models , author=. Proceedings of the 31st ACM International Conference on Multimedia , pages= on Pith
  10. [30] Accuracy of a large language model in distinguishing anti-and pro-vaccination messages on social media: The case of human papillomavirus vaccination , author=. Preventive Medicine Reports , volume=. 2024 , publisher= on Pith
  11. [31] Kuang, Xianwen and Liu, Jun and Zhang, Haiyang and Schweighofer, Simon , journal=. Towards algorithmic framing analysis: expanding the scope by using. 2025 , publisher= on Pith
  12. [32] Movva, Rajiv and Koh, Pang Wei and Pierson, Emma , journal=. Annotation alignment: Comparing on Pith
  13. [33] Kristensen-McLachlan, Ross Deans and Canavan, Miceal and K. Are chatbots reliable text annotators?. PNAS Nexus , year =. doi:10.1093/pnasnexus/pgaf069 , url = on Pithdoi:10.1093/pnasnexus/pgaf069
  14. [34] A Crisis of Civility? Modeling Incivility and Its Effects in Political Discourse Online , author=. Proceedings of the International AAAI Conference on Web and Social Media , volume= on Pith
  15. [35] Incivility in congressional tweets , author=. American Politics Research , volume=. 2022 , publisher= on Pith
  16. [36] Yu, Xudong and Wojcieszak, Magdalena and Casas, Andreu , journal=. Partisanship on social media: In-party love among. 2024 , publisher= on Pith
  17. [37] Out-group animosity drives engagement on social media , author=. Proceedings of the National Academy of Sciences , volume=. 2021 , publisher= on Pith
  18. [38] Toxic Bias: Perspective API misreads German as more toxic , author=. Proceedings of the International AAAI Conference on Web and Social Media , volume= on Pith
  19. [39] Moralization in social networks and the emergence of violence during protests , author=. Nature Human Behaviour , volume=. 2018 , publisher= on Pith
  20. [40] How social learning amplifies moral outrage expression in online social networks , author=. Science Advances , volume=. 2021 , publisher= on Pith
  21. [41] Negativity drives online news consumption , author=. Nature Human Behaviour , volume=. 2023 , publisher= on Pith
  22. [42] Sentiment analysis of news articles: a lexicon based approach , author=. 2019 2nd International Conference on Computing, Mathematics and Engineering Technologies (iCoMET) , pages=. 2019 , organization= on Pith
  23. [43] Sentiment analysis of comments in social media. , author=. International Journal of Electrical & Computer Engineering (2088-8708) , volume= on Pith
  24. [44] Uncertainty detection in natural language: A probabilistic model , author=. Proceedings of the 6th International Conference on Web Intelligence, Mining and Semantics , pages= on Pith
  25. [45] A language model-based generative classifier for sentence-level discourse parsing , author=. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages= on Pith
  26. [46] Partisan stereotypes and policy attitudes , author=. Journal of Communication , volume=. 2012 , publisher= on Pith
  27. [47] Informing the electorate? How party cues and policy information affect public opinion about initiatives , author=. American Journal of Political Science , volume=. 2014 , publisher= on Pith
  28. [48] Emotion: The science of sentiment , author=. 2002 , publisher= on Pith
  29. [49] Defining and detecting toxicity on social media: context and knowledge are key , author=. Neurocomputing , volume=. 2022 , publisher= on Pith
  30. [50] Pandita, Swati and Garg, Ketika and Zhang, Jiajin and Mobbs, Dean , title =. Trends in Cognitive Sciences , year = on Pith
  31. [51] Jo, Claire Wonjeong and Weso. Harmful. arXiv preprint arXiv:2411.05854 , year =. doi:10.48550/arXiv.2411.05854 , url = on PithPith's reviewarXivdoi:10.48550/arxiv.2411.05854
  32. [52] Wu, Hoi Yan and Leung, Sing Oi , title =. Journal of Social Service Research , year = on Pith
  33. [53] Proceedings of the iConference 2021 , author =. 2021 , location = on Pith
  34. [54] Hutto, Clayton and Gilbert, Eric , title =. Proceedings of the International AAAI Conference on Web and Social Media (ICWSM) , year = on Pith
  35. [55] A comprehensive survey of prompt engineering techniques in large language models , author=. TechRxiv , year= on Pith
  36. [56] A survey on uncertainty quantification of large language models: Taxonomy, open research challenges, and future directions , author=. ACM Computing Surveys , year= on Pith
  37. [57] Achiam, Josh and Adler, Steven and Agarwal, Sandhini and Ahmad, Lama and Akkaya, Ilge and Aleman, Florencia Leoni and Almeida, Diogo and Altenschmidt, Janko and Altman, Sam and Anadkat, Shyamal and others , journal= on Pith
  38. [58] Liu, Aixin and Mei, Aoxue and Lin, Bangcai and Xue, Bing and Wang, Bingxuan and Xu, Bingzheng and Wu, Bochao and Zhang, Bowei and Lin, Chaofan and Dong, Chen and others , journal= on Pith
  39. [59] Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year= on PithPith's reviewarXiv
  40. [60] Mistral , year=. Mistral Small 3.1 , url= on Pith
  41. [61] Google , year=. Gemini 3.1 Flash-Lite: Built for intelligence at scale , url= on Pith
  42. [62] Li, Dawei and Jiang, Bohan and Huang, Liangjie and Beigi, Alimohammad and Zhao, Chengshuai and Tan, Zhen and Bhattacharjee, Amrita and Jiang, Yuxuan and Chen, Canyu and Wu, Tianhao and others , booktitle=. From generation to judgment: Opportunities and challenges of on Pith
  43. [63] JudgeLM: Fine-tuned Large Language Models are Scalable Judges , author=. The Thirteenth International Conference on Learning Representations , year= on Pith
  44. [64] He, Xingwei and Lin, Zhenghao and Gong, Yeyun and Jin, A-Long and Zhang, Hang and Lin, Chen and Jiao, Jian and Yiu, Siu Ming and Duan, Nan and Chen, Weizhu , booktitle= on Pith
  45. [65] Wan, Mengting and Safavi, Tara and Jauhar, Sujay Kumar and Kim, Yujin and Counts, Scott and Neville, Jennifer and Suri, Siddharth and Shah, Chirag and White, Ryen W and Yang, Longqi and others , booktitle= on Pith
  46. [66] Language Models are Few-Shot Learners , author=. Advances in Neural Information Processing Systems , volume= on Pith
  47. [67] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models , author=. Advances in Neural Information Processing Systems , volume= on Pith
  48. [68] Mistralai , year=. mistralai/Mistral-Small-3.2-24B-Instruct-2506 , url= on Pith
  49. [69] OpenAI , year=. Introducing on Pith
  50. [70] Google , year=. Gemini 3 Flash: frontier intelligence built for speed , url= on Pith
  51. [71] Difar: Enhancing multimodal misinformation detection with diverse, factual, and relevant rationales , author=. arXiv preprint arXiv:2508.10444 , year= on PithPith's reviewarXiv
  52. [72] Large language models are better reasoners with self-verification , author=. Findings of the Association for Computational Linguistics: EMNLP 2023 , pages= on Pith
  53. [73] Sharma, Manasi and Zhang, Chen Bo Calvin and Bandi, Chaithanya and Wang, Clinton and Aich, Ankit and Nghiem, Huy and Rabbani, Tahseen and Htet, Ye and Jang, Brian and Basu, Sumana and others , journal= on Pith
  54. [74] Eltanbouly, Sohaila and Albatarni, Salam and Elsayed, Tamer , booktitle= on Pith
  55. [75] Li, Gaotang and Mishra, Bhavana Dalvi and Wang, Zifeng and Yan, Jun and Chen, Yanfei and Li, Chun-Liang and Le, Long T and Han, Rujun and Lee, George and Tong, Hanghang and others , journal= on Pith
  56. [76] He, Jia and Rungta, Mukund and Koleczek, David and Sekhon, Arshdeep and Wang, Franklin X and Hasan, Sadid , journal=. Does prompt formatting have any impact on on Pith
  57. [77] The method of constructing an attitude scale , author=. Scaling , pages=. 2017 , publisher= on Pith
  58. [78] Reliability and validity testing of a new scale for measuring attitudes toward learning statistics with technology. , author=. Acta Didactica Napocensia , volume=. 2011 , publisher= on Pith
  59. [79] Guan, Bryan and Roosta, Tanya and Passban, Peyman and Rezagholizadeh, Mehdi , journal=. The order effect: investigating prompt sensitivity to input order in on Pith
  60. [80] Zhuo, Jingming and Zhang, Songyang and Fang, Xinyu and Duan, Haodong and Lin, Dahua and Chen, Kai , booktitle= on Pith
  61. [81] Hashemi, Helia and Eisner, Jason and Rosset, Corby and Van Durme, Benjamin and Kedzie, Chris , booktitle= on Pith
  62. [82] 2026 , month = aug, url = on Pith
  63. [83] 2026 , month = jul, url = on Pith
  64. [84] Schwarz, Norbert and Kn. Rating Scales: Numeric Values May Change the Meaning of Scale Labels , journal =. 1991 , volume =. doi:10.1086/269282 , url = on Pithdoi:10.1086/269282
  65. [85] Tourangeau, Roger and Couper, Mick P. and Conrad, Frederick , title =. Public Opinion Quarterly , year =. doi:10.1093/poq/nfl046 , url = on Pithdoi:10.1093/poq/nfl046
  66. [86] Chochlakis, Georgios and Maruthu Pandiyan, Niyantha and Lerman, Kristina and Narayanan, Shrikanth , title =. arXiv preprint arXiv:2409.06173 , year =. doi:10.48550/arXiv.2409.06173 , url = on PithPith's reviewarXivdoi:10.48550/arxiv.2409.06173
  67. [87] Dhuliawala, Shehzaad and Komeili, Mojtaba and Xu, Jing and Raileanu, Roberta and Li, Xian and Celikyilmaz, Asli and Weston, Jason , title =. Findings of the Association for Computational Linguistics: ACL 2024 , year =. doi:10.18653/v1/2024.findings-acl.212 , url = on Pithdoi:10.18653/v1/2024.findings-acl.212
  68. [88] Treier, Shawn and Hillygus, D. Sunshine , title =. Public Opinion Quarterly , year =. doi:10.1093/poq/nfp067 , url = on Pithdoi:10.1093/poq/nfp067
  69. [89] Feldman, Stanley and Johnston, Christopher , title =. Political Psychology , year =. doi:10.1111/pops.12055 , url = on Pithdoi:10.1111/pops.12055
  70. [90] Volkens, Andrea and Burst, Tobias and Krause, Werner and Lehmann, Pola and Matthie. The. 2021 , edition = on Pith
  71. [91] Pang, Bo and Lee, Lillian , title =. Foundations and Trends in Information Retrieval , year =. doi:10.1561/1500000011 , url = on Pithdoi:10.1561/1500000011
  72. [92] n.d. , howpublished = on Pith
  73. [93] Borkan, Daniel and Dixon, Lucas and Sorensen, Jeffrey and Thain, Nithum and Vasserman, Lucy , title =. Companion Proceedings of the 2019 World Wide Web Conference , year =. doi:10.1145/3308560.3317593 , url = on Pithdoi:10.1145/3308560.3317593
  74. [94] Liusie, Adian and Manakul, Potsawee and Gales, Mark. LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models. Proceedings of the 18th Conference of the E uropean Chapter of the A ssociation for C omputational L inguistics (... on Pithdoi:10.18653/v1/2024.eacl-long.8
Cited by 1 Pith paper

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment