Pith. sign in

REVIEW 4 major objections 5 minor 99 references

Algorithmic Tradeoffs, Applied NLP, and the State-of-the-Art Fallacy

T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that newer “state of the art” NLP methods did not outperform older, more interpretable methods on a corpus of 238,892 college admissions essays, and that treating computer-science benchmarks as the best tool for social sci

desk verdict A useful concept in search of a fair test: the state-of-the-art fallacy is worth naming, but Study 1's comparison is too unequal to carry the headline claim yet. read the letter →

arxiv 2509.08199 v1 pith:55WDFCJF submitted 2025-09-10 cs.CY stat.AP

AJ Alvero , Ruohong Dong , Klint Kanopka , David Lang This is my paper
classification cs.CYstat.AP
keywords state-of-the-artfallacycomputationalsociologynaturallanguageprocessingtextanalysisinterpretabilitylargemodelssiliconsamplingcollegeadmissionsessays
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the “state-of-the-art fallacy” is real: methods anointed by computer-science benchmarks do not automatically deliver better results for social-scientific text analysis. Revisiting 238,892 college admissions essays, it shows that an older, interpretable pipeline—correlated topic modeling plus LIWC features fed into a cross-validated linear model—matched or beat word embeddings, neural topic models, XGBoost, and a CNN at predicting SAT scores. A second study finds that LLM-generated essays, the current state of the art, are demographically misaligned out of the box, writing more like applicants with college-educated parents and private-school backgrounds. If true, the paper gives sociologists and other applied researchers a reason to choose methods by theoretical and empirical fit rather than by hype, and to test many models against a strong baseline.

What carries the argument

The analytic engine is a comparison of methodological regimes on one corpus. The older regime is correlated topic modeling (a 2006 topic model that allows topics to be correlated) plus LIWC2015 dictionary features, scored by a 10-fold cross-validated linear regression on adjusted R² and RMSE. The newer regimes are word-embedding averages (GloVe/Word2Vec) and neural topic modeling (BERTopic) fed into flexible predictors (XGBoost, 1D-CNN), plus sequence models. In Study 2 the mechanism is a transfer test: LIWC-feature classifiers trained on human essays are applied to LLM-generated essays to measure which demographic groups the synthetic text resembles.

What would settle it

Run a replication on the same or comparable essay corpus in which BERTopic, a fine-tuned transformer, and the CNN/LSTM are given documented hyperparameter searches, multiple seeds, and the same 10-fold evaluation; if any newer method exceeds the CTM+LIWC adjusted R² of 0.526 by a meaningful margin, the paper's central empirical claim fails.

Watch

Extended reading notes

Core claim

The central claim is that newer is not better by default. In Study 1, the older CTM plus LIWC linear model had adjusted R² = 0.486 (RMSE 124.87) for predicting SAT scores, and combining CTM with LIWC raised R² to 0.526; every newer alternative the authors implemented—GloVe averaging with XGBoost, Word2Vec-based CNNs, prompt-specific topic models, and BERTopic—performed no better, and the weakest (BERTopic with XGBoost) had adjusted R² = 0.058. In Study 2, classifiers trained to predict applicant demographics from human essays labeled GPT-3.5, GPT-4o, Mistral Small, and Claude 3 Haiku essays as mostly written by continuing-generation, private-school applicants (74–90% for parental education,

Load-bearing premise

The load-bearing premise is that the newer methods in Study 1 were implemented with enough tuning to be fair representatives of those methods; the authors concede they may have been on the cusp of trying the right model, so if the implementations were under-tuned, the conclusion that older methods win collapses.

Editorial extensions

If this is right

  • Social-science text analysts should not assume the newest NLP model is the best predictor; a transparent baseline can match or beat it.
  • The “state-of-the-art fallacy” gives a vocabulary and a demonstration for resisting benchmark-driven method selection in sociology.
  • LLM-generated text used for silicon sampling can introduce demographic confounds; its outputs should be checked for alignment before substituting for human data.
  • Applied researchers should run multiple methods and report baselines, since the extra cost of comparison is justified by the range of results (adjusted R² from 0.058 to 0.526 in Study 1).
  • Computer-science benchmarks are not portable evidence of suitability for sociological questions; model selection should be justified theoretically.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the finding generalizes beyond essays, it suggests that “state of the art” should be reported with a domain qualifier: a model can be state of the art on generic NLP benchmarks and still underperform a topic model on a social-science prediction task.
  • A testable extension is to vary hyperparameter effort: the paper’s own limitation concedes the newer models may have been under-tuned, so a replication with exhaustive tuning and multiple seeds would separate the fallacy from implementation artifact.
  • The Study 2 alignment results imply a concrete research practice: before using LLM-generated text as a stand-in for human essays, researchers should train a demographic classifier on human text and compare predicted group distributions, essentially a bias audit.
  • The same fallacy logic likely applies to other applied fields such as political science, psychology, and education, where prediction benchmarks are imported from computer science; the paper’s argument is framed sociologically but its evidence structure is portable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces the 'state-of-the-art fallacy': the belief that the NLP method currently labeled state of the art by computer scientists will be the best tool for sociological research. It argues that social scientists should choose methods based on the research question rather than on benchmark prestige, and it presents two empirical demonstrations using 238,892 college admissions essays. Study 1 compares the authors' prior CTM + linear regression baseline with newer methods (GloVe + XGBoost, BERTopic, prompt-specific topic models, a 1D-CNN, and an unreported LSTM), concluding that the older, interpretable baseline performs as well as or better than the alternatives. Study 2 uses LLM-generated essays from GPT-3.5, GPT-4o, Mistral Small, and Claude 3 Haiku to show that a classifier trained on human essays tends to label LLM essays as coming from continuing-generation, private-school, and below-median-GPA applicants. The paper concludes with a call for methodological pluralism and caution about adopting 'silicon sampling' without validity checks.

Significance. The conceptual contribution is timely and valuable: it articulates a real risk in computational social science—that benchmark-driven method choice can be misaligned with social-scientific goals—and it names a useful fallacy that many researchers will recognize. The authors deserve credit for conducting a large-scale empirical comparison on a substantial corpus, for testing four different LLMs, and for explicitly acknowledging in the Limitations that the comparison is not exhaustive and that a future or better-tuned model could change the results. That said, the empirical evidence currently falls short of the headline claim. The central assertion that 'newer techniques did not outperform prior results in meaningful ways' depends on the fairness and representativeness of the alternative implementations, and several load-bearing aspects of the comparison are either uncontrolled or internally inconsistent. If these issues are addressed, the paper could be a strong contribution; as it stands, the empirical demonstration supports a considerably weaker claim.

major comments (4)
  1. [§Data / Study 1] The baseline's feature set was constructed using outcome-informed feature selection: 'because topics sum to unity, they removed the topic that was least correlated with each dependent variable (SAT score; household income)' (Data section). If this selection was performed on the full sample before cross-validation, it leaks outcome information into the baseline's features and inflates its out-of-sample R²/RMSE. The alternative methods (GloVe, BERTopic, CNN, LSTM) were not given an equivalent outcome-informed reduction, nor a nested version of it. This makes the comparison unequal and directly undermines the conclusion that 'CTM and linear modeling outperformed a variety of both NLP and prediction methods' (Discussion). The authors should either re-run the analysis with nested cross-validation or empirically demonstrate that the dropped-topic choice is invariant to the training fold.
  2. [§Study 1: Methods / Model Selection] The alternative methods appear substantially under-tuned relative to what is typically expected for these models. BERTopic is reported only as an R²=0.058 result, with no hyperparameters (e.g., minimum topic size, embedding model, clustering parameters); the LSTM configuration is not described at all; and the XGBoost grid is extremely narrow: 'varying the learning rate[-5,.01], tree depth [5,15]' with only '50 trees'. For 300-dimensional embeddings, 50 boosting rounds is likely far too few. The paper's own Limitations concede: 'It is quite possible that we were just on the cusp of trying the right model.' Because the headline claim rests entirely on these implementations being fair, the current results support only the weaker claim that these particular implementations did not outperform the baseline, not that state-of-the-art methods generally fail to do so.
  3. [§Study 1: Results / Figure 2] The reported metrics are internally inconsistent. The text states that 'XGBoost and CTM generated a higher R2 but lower RMSE (0.495 vs. 0.486 and 125 vs 124 respectively), meaning it had slightly higher average error than out-of-sample linear modeling.' If both R² and RMSE are computed on the same test set and the dependent variable is identical, a higher R² necessarily implies a lower RMSE. The pair (0.495, 125) versus (0.486, 124) cannot both be true on the same held-out data. The authors need to clarify whether these metrics were computed on the same folds/test sets, and if so, correct the reported values. This is not a cosmetic issue: the conclusion that the XGBoost advantage 'is not meaningful' depends on which metric is actually larger.
  4. [§Study 2: Results / Table 2] The demographic-alignment percentages in Study 2 are based on predictions from a classifier trained on human essays, not on direct comparisons of the LIWC features between human-written and LLM-generated text. With classifier accuracies of only 63–70% (Table 5), the statement that 'essays written by LLMs were much more likely to also come from those with college educated parents' conflates the classifier's decision boundary and prior with properties of the text itself. For example, a classifier with a strong default toward the majority class could produce high predicted probabilities for continuing-generation status even if the underlying feature distributions are similar. Because the authors have the LIWC features for the AI essays, a direct distributional comparison would be more informative and would support the claimed bias-and-confounding conclusion more convincingly.
minor comments (5)
  1. [§Study 1: Methods] The XGBoost hyperparameter description 'learning rate[-5,.01]' appears typographically garbled; please clarify the intended range (and the sign convention).
  2. [Table 2] In the GPA Median row for Claude, the value '93&' should read '93%'.
  3. [Study 1: Results] There is a duplicated 'somewhat' in 'The weaker performance of the embedding approaches was somewhat somewhat surprising.'
  4. [Discussion] The sentence beginning 'The pressure and influence of computational hype, but there are a few ways...' is grammatically incomplete.
  5. [Limitations] The phrase 'text analysisa lalinear regression' appears to be a typo; likely 'text analysis, like linear regression'.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the benchmark comparison and the alignment classifier are not self-referential; the flagged concerns are validity/effort issues, not constructional equivalence.

full rationale

The paper's derivation chain does not reduce to its own inputs. Study 1 is a benchmark comparison: the baseline R2=0.486 and RMSE=124.87 are imported from the authors' prior published analysis ('These are the benchmarks we will use to compare state-of-the-art methods from the time that the original paper was written'), not fitted and renamed as predictions inside this paper. The flagged outcome-informed topic removal ('because topics sum to unity, they removed the topic that was least correlated with each dependent variable') cannot be the source of the baseline's edge in a linear model: with an intercept, dropping any one of a set of variables that sums to a constant spans the same column space, so the fitted values and R2 are invariant to which topic is dropped. Study 2's 'alignment' is measured by a classifier trained on human essays and then applied to LLM-generated essays ('The trained models were then given the AI generated essays to label'); the LLM texts are new inputs, not training targets, so the predicted labels are a measurement rather than a fitted parameter passed off as a finding. The paper explicitly concedes the alternative models may have been under-tuned ('It is quite possible that we were just on the cusp of trying the "right" model, or a future advance will outperform any model presented here'), but under-tuning is an effort-equivalence/validity threat, not a constructional equivalence. The self-citations to prior AI-essay work ([92], [93]) are corroborative and not load-bearing; the current analyses are computed in this paper. Therefore no circular step is identifiable.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The paper's central comparison depends on the trustworthiness of the prior study's feature outputs, the comparability of evaluation metrics, the transfer of a weak classifier to out-of-distribution text, and the representativeness of the chosen modern methods. These assumptions are not verified with shipped code, data, or formal checks.

free parameters (4)
  • Number of CTM topics (K) = 70
    Set in the prior (Author) study using ldatuning; defines the baseline feature representation and is taken as fixed in the comparison.
  • CNN architecture hyperparameters = 24 filters; k ∈ {3,5,10,20,50,100,150,200}; λ=0.05; dropout=0.2
    Selected by validation; the final architecture (484,200 parameters) is a choice that affects the CNN's performance relative to the baseline.
  • XGBoost hyperparameters = learning rate ∈ [-5,.01], tree depth ∈ [5,15], 50 trees (final values not reported)
    Tuned via grid search and early stopping; unreported final values make the comparison hard to audit.
  • BERTopic parameters = not reported
    No embedding model, cluster count, or threshold specified; the very low R² (0.058) may reflect configuration rather than inherent method quality.
assumptions (4)
  • domain assumption Validity of the prior study's computed features
    Study 1 reuses CTM topic loadings and LIWC features from '(Author)' computed in 2020; the paper does not recompute them.
  • domain assumption Comparability of evaluation protocols
    Original baseline used 10-fold cross-validation; the newer models appear to use a single validation/test split; the paper does not establish metric comparability.
  • domain assumption Classifier transfer to out-of-distribution LLM text
    Study 2 interprets classifier predictions on LLM essays as evidence of demographic style; this assumes no OOD artifacts dominate.
  • domain assumption Representativeness of the chosen 'state of the art' methods
    The paper selects GloVe averaging, BERTopic, XGBoost, 1D-CNN, and an unreported LSTM as stand-ins for SOTA; if these are not representative, the fallacy demonstration is weaker.
invented entities (1)
  • 'State-of-the-art fallacy' concept
    purpose: Labels the belief that the method deemed best by computer-science benchmarks is best for all sociological questions
    A heuristic framing, not a measurable entity; its value is conceptual and cannot be falsified independent of the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Algorithmic Tradeoffs, Applied NLP, and the State-of-the-Art Fallacy." pith.science (2026). https://pith.science/paper/55WDFCJF

@misc{pith2026250908199,
  author       = {Pith},
  title        = {Pith review of: Algorithmic Tradeoffs, Applied NLP, and the State-of-the-Art Fallacy},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/55WDFCJF}},
  note         = {Machine review of arXiv:2509.08199}
}
read the original abstract

Computational sociology is growing in popularity, yet the analytic tools employed differ widely in power, transparency, and interpretability. In computer science, methods gain popularity after surpassing benchmarks of predictive accuracy, becoming the "state of the art." Computer scientists favor novelty and innovation for different reasons, but prioritizing technical prestige over methodological fit could unintentionally limit the scope of sociological inquiry. To illustrate, we focus on computational text analysis and revisit a prior study of college admissions essays, comparing analyses with both older and newer methods. These methods vary in flexibility and opacity, allowing us to compare performance across distinct methodological regimes. We find that newer techniques did not outperform prior results in meaningful ways. We also find that using the current state of the art, generative AI and large language models, could introduce bias and confounding that is difficult to extricate. We therefore argue that sociological inquiry benefits from methodological pluralism that aligns analytic choices with theoretical and empirical questions. While we frame this sociologically, scholars in other disciplines may confront what we call the "state-of-the-art fallacy", the belief that the tool computer scientists deem to be the best will work across topics, domains, and questions.

Figures

Figures reproduced from arXiv: 2509.08199 by the authors.

Figure 1
Figure 1. Conceptual diagram of the tradeoffs between interpretability and flexibility in NLP methods used in [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. CTM was as good or better than other methods tested. CTM and XGBoost had a higher [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

99 extracted references · 64 canonical work pages

  1. [1]

    Springer, 2009

    Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman.The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009

    2009

  2. [2]

    Springer, 2013

    Gareth James, Daniela Witten, Trevor Hastie, Robert Tibshirani, et al.An introduction to statistical learning, volume 112. Springer, 2013

    2013

  3. [3]

    Comment: The essential tension between parsimony and accuracy.Sociological Methodology, 28(1):231– 236, 1998

    Yu Xie. Comment: The essential tension between parsimony and accuracy.Sociological Methodology, 28(1):231– 236, 1998

    1998

  4. [4]

    Machine translation: Mining text for social theory.Annual review of sociology, 42(1):21–50, 2016

    James A Evans and Pedro Aceves. Machine translation: Mining text for social theory.Annual review of sociology, 42(1):21–50, 2016

    2016

  5. [5]

    Oxford University Press, 2024

    Christian Borch and Juan Pablo Pardo-Guerra.The Oxford Handbook of the Sociology of Machine Learning. Oxford University Press, 2024

    2024

  6. [6]

    The rise of machine learning in the academic social sciences.AI & SOCIETY, 39(2):799–801, 2024

    Charles Rahal, Mark Verhagen, and David Kirk. The rise of machine learning in the academic social sciences.AI & SOCIETY, 39(2):799–801, 2024

    2024

  7. [7]

    Incorporating machine learning into sociological model-building.Sociological Methodology, 54(2):217–268, 2024

    Mark D Verhagen. Incorporating machine learning into sociological model-building.Sociological Methodology, 54(2):217–268, 2024

    2024

  8. [8]

    Researcher reasoning meets computational capacity: Machine learning for social science.Social science research, 108:102807, 2022

    Ian Lundberg, Jennie E Brand, and Nanum Jeon. Researcher reasoning meets computational capacity: Machine learning for social science.Social science research, 108:102807, 2022

    2022

Show all 99 references
  1. [9]

    Ai competitions as infrastructures of power in medical imaging.Information, Communication & Society, pages 1–22, 2024

    Dieuwertje Luitse, Tobias Blanke, and Thomas Poell. Ai competitions as infrastructures of power in medical imaging.Information, Communication & Society, pages 1–22, 2024

    2024

  2. [10]

    Koch and David Peterson

    Bernard J. Koch and David Peterson. From protoscience to epistemic monoculture: How benchmarking set the stage for the deep learning revolution, 2024

    2024

  3. [11]

    50 years of data science.Journal of Computational and Graphical Statistics, 26(4):745–766, 2017

    David Donoho. 50 years of data science.Journal of Computational and Graphical Statistics, 26(4):745–766, 2017

    2017

  4. [12]

    Bernard Koch, Emily Denton, Alex Hanna, and Jacob G. Foster. Reduced, reused and recycled: The life of a dataset in machine learning research, 2021

    2021

  5. [13]

    The file drawer problem and tolerance for null results.Psychological bulletin, 86(3):638, 1979

    Robert Rosenthal. The file drawer problem and tolerance for null results.Psychological bulletin, 86(3):638, 1979

    1979

  6. [14]

    Troubling trends in machine learning scholarship: Some ml papers suffer from flaws that could mislead the public and stymie future research.Queue, 17(1):45–77, 2019

    Zachary C Lipton and Jacob Steinhardt. Troubling trends in machine learning scholarship: Some ml papers suffer from flaws that could mislead the public and stymie future research.Queue, 17(1):45–77, 2019

    2019

  7. [15]

    Oxford University Press, 2024

    Dustin S Stoltz and Marshall A Taylor.Mapping Texts: Computational Text Analysis for the Social Sciences. Oxford University Press, 2024

    2024

  8. [16]

    Princeton University Press, 2022

    Justin Grimmer, Margaret E Roberts, and Brandon M Stewart.Text as data: A new framework for machine learning and the social sciences. Princeton University Press, 2022

    2022

  9. [17]

    What good is a regression? inference to the best explanation and the practice of political science research.The Journal of Politics, 87(4):000–000, 2025

    Arthur Spirling and Brandon M Stewart. What good is a regression? inference to the best explanation and the practice of political science research.The Journal of Politics, 87(4):000–000, 2025

    2025

  10. [18]

    Transcending general linear reality.Sociological theory, pages 169–186, 1988

    Andrew Abbott. Transcending general linear reality.Sociological theory, pages 169–186, 1988

    1988

  11. [19]

    Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.Nature machine intelligence, 1(5):206–215, 2019

    Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.Nature machine intelligence, 1(5):206–215, 2019

    2019

  12. [20]

    Measuring the predictability of life outcomes with a scientific mass collaboration.Proceedings of the National Academy of Sciences, 117(15):8398–8403, 2020

    Matthew J Salganik, Ian Lundberg, Alexander T Kindel, Caitlin E Ahearn, Khaled Al-Ghoneim, Abdullah Almaatouq, Drew M Altschul, Jennie E Brand, Nicole Bohme Carnegie, Ryan James Compton, et al. Measuring the predictability of life outcomes with a scientific mass collaboration....

    2020

  13. [21]

    Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty.Proceedings of the National Academy of Sciences, 119(44):e2203150119, 2022

    Nate Breznau, Eike Mark Rinke, Alexander Wuttke, Hung HV Nguyen, Muna Adem, Jule Adriaans, Amalia Alvarez-Benjumea, Henrik K Andersen, Daniel Auer, Flavio Azevedo, et al. Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty.Pro...

    2022

  14. [22]

    Text mining for social science–the state and the future of computational text analysis in sociology

    Ana Macanovic. Text mining for social science–the state and the future of computational text analysis in sociology. Social Science Research, 108:102784, 2022

    2022

  15. [23]

    Predictability hypotheses: A metatheoretical and methodological introduction

    Austin van Loon. Predictability hypotheses: A metatheoretical and methodological introduction. InThe Oxford Handbook of the Sociology of Machine Learning. Oxford University Press, 2025

    2025

  16. [24]

    Three families of automated text analysis.Social Science Research, 108:102798, 2022

    Austin van Loon. Three families of automated text analysis.Social Science Research, 108:102798, 2022. 18 State-of-the-Art Fallacy

    2022

  17. [25]

    Computational grounded theory: A methodological framework.Sociological Methods & Research, 49(1):3–42, 2020

    Laura K Nelson. Computational grounded theory: A methodological framework.Sociological Methods & Research, 49(1):3–42, 2020

    2020

  18. [26]

    Wright Mills.The Sociological Imagination

    C. Wright Mills.The Sociological Imagination. Oxford University Press, New York, 1959

    1959

  19. [27]

    Wendy Nelson Espeland and Mitchell L. Stevens. A sociology of quantification.European Journal of Sociology, 49(3):401–436, 2008

    2008

  20. [28]

    The coming crisis of empirical sociology.Sociology, 41(5):885–899, 2007

    Mike Savage and Roger Burrows. The coming crisis of empirical sociology.Sociology, 41(5):885–899, 2007

    2007

  21. [29]

    Critical questions for big data: Provocations for a cultural, technological, and scholarly phenomenon.Information, communication & society, 15(5):662–679, 2012

    danah boyd and Kate Crawford. Critical questions for big data: Provocations for a cultural, technological, and scholarly phenomenon.Information, communication & society, 15(5):662–679, 2012

    2012

  22. [30]

    Harvard University Press, 1999

    Karin Knorr Cetina.Epistemic cultures: How the sciences make knowledge. Harvard University Press, 1999

    1999

  23. [31]

    From ends to means: The promise of computational text analysis for theoretically driven sociological research.Sociological Methods & Research, 51(4):1469–1483, 2022

    Bart Bonikowski and Laura K Nelson. From ends to means: The promise of computational text analysis for theoretically driven sociological research.Sociological Methods & Research, 51(4):1469–1483, 2022

    2022

  24. [32]

    Computational social science and sociology.Annual review of sociology, 46(1):61–81, 2020

    Achim Edelmann, Tom Wolff, Danielle Montagne, and Christopher A Bail. Computational social science and sociology.Annual review of sociology, 46(1):61–81, 2020

    2020

  25. [33]

    What is your estimand? defining the target quantity connects statistical evidence to theory.American Sociological Review, 86(3):532–565, 2021

    Ian Lundberg, Rebecca Johnson, and Brandon M Stewart. What is your estimand? defining the target quantity connects statistical evidence to theory.American Sociological Review, 86(3):532–565, 2021

    2021

  26. [34]

    Harvard University Press, 1991

    Pierre Bourdieu.Language and symbolic power. Harvard University Press, 1991

    1991

  27. [35]

    Sociology in the era of big data: The ascent of forensic social science.The American Sociologist, 47:12–35, 2016

    Daniel A McFarland, Kevin Lewis, and Amir Goldberg. Sociology in the era of big data: The ascent of forensic social science.The American Sociologist, 47:12–35, 2016

    2016

  28. [36]

    Data and its (dis) contents: A survey of dataset development and use in machine learning research.Patterns, 2(11), 2021

    Amandalynne Paullada, Inioluwa Deborah Raji, Emily M Bender, Emily Denton, and Alex Hanna. Data and its (dis) contents: A survey of dataset development and use in machine learning research.Patterns, 2(11), 2021

    2021

  29. [37]

    A synopsis of linguistic theory, 1930-1955.Studies in linguistic analysis, pages 10–32, 1957

    John Firth. A synopsis of linguistic theory, 1930-1955.Studies in linguistic analysis, pages 10–32, 1957

    1930

  30. [38]

    The geometry of culture: Analyzing the meanings of class through word embeddings.American Sociological Review, 84(5):905–949, 2019

    Austin C Kozlowski, Matt Taddy, and James A Evans. The geometry of culture: Analyzing the meanings of class through word embeddings.American Sociological Review, 84(5):905–949, 2019

    2019

  31. [39]

    Meaning in hyperspace: Word embeddings as tools for cultural measurement.Annual Review of Sociology, 51, 2025

    Andrei Boutyline and Alina Arseniev-Koehler. Meaning in hyperspace: Word embeddings as tools for cultural measurement.Annual Review of Sociology, 51, 2025

    2025

  32. [40]

    Theoretical foundations and limits of word embeddings: what types of meaning can they capture?Sociological Methods & Research, 53(4):1753–1793, 2024

    Alina Arseniev-Koehler. Theoretical foundations and limits of word embeddings: what types of meaning can they capture?Sociological Methods & Research, 53(4):1753–1793, 2024

    2024

  33. [41]

    Distributed representations of words and phrases and their compositionality.Advances in neural information processing systems, 26, 2013

    Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality.Advances in neural information processing systems, 26, 2013

    2013

  34. [42]

    Glove: Global vectors for word representation

    Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. InProceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014

    2014

  35. [43]

    How the machine ‘thinks’: Understanding opacity in machine learning algorithms.Big data & society, 3(1):2053951715622512, 2016

    Jenna Burrell. How the machine ‘thinks’: Understanding opacity in machine learning algorithms.Big data & society, 3(1):2053951715622512, 2016

    2016

  36. [44]

    Toward a sociology of machine learning explainability: Human–machine interaction in deep neural network-based automated trading.Big Data & Society, 9(2):20539517221111361, 2022

    Christian Borch and Bo Hee Min. Toward a sociology of machine learning explainability: Human–machine interaction in deep neural network-based automated trading.Big Data & Society, 9(2):20539517221111361, 2022

    2022

  37. [45]

    A threatening tone: Homicide, racial threat narratives, and the historical growth of incarceration in the united states, 1926–2016.Social Forces, page soad022, 2023

    Scott W Duxbury. A threatening tone: Homicide, racial threat narratives, and the historical growth of incarceration in the united states, 1926–2016.Social Forces, page soad022, 2023

    1926

  38. [46]

    Intersecting the academic gender gap: The education of lesbian, gay, and bisexual america

    Joel Mittleman. Intersecting the academic gender gap: The education of lesbian, gay, and bisexual america. American Sociological Review, 87(2):303–335, 2022

    2022

  39. [47]

    From codebooks to promptbooks: Extracting information from text with generative large language models.Sociological Methods & Research, page 00491241251336794

    Oscar Stuhler, Cat Dang Ton, and Etienne Ollion. From codebooks to promptbooks: Extracting information from text with generative large language models.Sociological Methods & Research, page 00491241251336794

  40. [48]

    The augmented social scientist: Using sequential transfer learning to annotate millions of texts with human-level accuracy.Sociological Methods & Research, 53(3):1167–1200, 2024

    Salomé Do, Étienne Ollion, and Rubing Shen. The augmented social scientist: Using sequential transfer learning to annotate millions of texts with human-level accuracy.Sociological Methods & Research, 53(3):1167–1200, 2024

    2024

  41. [49]

    A primer on deep learning for causal inference.Sociological Methods & Research, page 00491241241234866

    Bernard J Koch, Tim Sainburg, Pablo Geraldo Bastías, Song Jiang, Yizhou Sun, and Jacob G Foster. A primer on deep learning for causal inference.Sociological Methods & Research, page 00491241241234866

  42. [50]

    The mixed subjects design: Treating large language models as potentially informative observations.Sociological Methods & Research, page 00491241251326865

    David Broska, Michael Howes, and Austin van Loon. The mixed subjects design: Treating large language models as potentially informative observations.Sociological Methods & Research, page 00491241251326865. 19 State-of-the-Art Fallacy

  43. [51]

    Language models in sociological research: An application to classifying large administrative data and measuring religiosity.Sociological Methodology, 52(1):30–52, 2022

    Jeffrey L Jensen, Daniel Karell, Cole Tanigawa-Lau, Nizar Habash, Mai Oudah, and Dhia Fairus Shofia Fani. Language models in sociological research: An application to classifying large administrative data and measuring religiosity.Sociological Methodology, 52(1):30–52, 2022

    2022

  44. [52]

    Automated hate speech detection and the problem of offensive language

    Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. Automated hate speech detection and the problem of offensive language. InProceedings of the international AAAI conference on web and social media, volume 11, pages 512–515, 2017

    2017

  45. [53]

    Stoltz, and John Levi Martin

    Oscar Stuhler, Dustin S. Stoltz, and John Levi Martin. Meaning and Machines. InThe Oxford Handbook of the Sociology of Machine Learning. Oxford University Press, 2023

    2023

  46. [54]

    Berts are generative in-context learners.Advances in Neural Information Processing Systems, 37:2558–2589, 2024

    David Samuel. Berts are generative in-context learners.Advances in Neural Information Processing Systems, 37:2558–2589, 2024

    2024

  47. [55]

    Large language models struggle to describe the haystack without human help: A social science-inspired evaluation of topic models

    Zongxia Li, Lorena Calvo-Bartolomé, Alexander Miserlis Hoyle, Paiheng Xu, Daniel Kofi Stephens, Juan Fran- cisco Fung, Alden Dima, and Jordan Lee Boyd-Graber. Large language models struggle to describe the haystack without human help: A social science-inspired evaluation of to...

    2025

  48. [56]

    Simple rules outperform machine learning for personnel selection: insights from the 3rd annual siop machine learning competition.Discover Artificial Intelligence, 3(1):2, 2023

    Jason L Harman and Jaelle Scheuerman. Simple rules outperform machine learning for personnel selection: insights from the 3rd annual siop machine learning competition.Discover Artificial Intelligence, 3(1):2, 2023

    2023

  49. [57]

    low-resource

    Zhiying Jiang, Matthew Yang, Mikhail Tsirlin, Raphael Tang, Yiqin Dai, and Jimmy Lin. “low-resource” text classification: A parameter-free classification method with compressors. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors,Findings of the Association for Co...

    2023

  50. [58]

    Harper Business, 2019

    Mary L Gray and Siddharth Suri.Ghost work: How to stop Silicon Valley from building a new global underclass. Harper Business, 2019

    2019

  51. [59]

    Sociolectal analysis of pretrained language models

    Sheng Zhang, Xin Zhang, Weiming Zhang, and Anders Søgaard. Sociolectal analysis of pretrained language models. InProceedings of the 2021 conference on empirical methods in natural language processing, pages 4581–4588, 2021

    2021

  52. [60]

    Propaganda bias

    Hannah C Waight, Brandon M Stewart, Margaret E Roberts, Solomon Messing, Eddie Yang, Joshua A Tucker, and Yin Yuan. Propaganda bias. 2024

    2024

  53. [61]

    Facets of specialization and its relation to career success: An analysis of us sociology, 1980 to 2015.American Sociological Review, 86(6):1164–1192, 2021

    Raphael H Heiberger, Sebastian Munoz-Najar Galvez, and Daniel A McFarland. Facets of specialization and its relation to career success: An analysis of us sociology, 1980 to 2015.American Sociological Review, 86(6):1164–1192, 2021

    1980

  54. [62]

    A status-based model of market competition.American journal of sociology, 98(4):829–872, 1993

    Joel M Podolny. A status-based model of market competition.American journal of sociology, 98(4):829–872, 1993

    1993

  55. [63]

    Princeton University Press, 1996

    Theodore M Porter.Trust in Numbers: The Pursuit of Objectivity in Science and Public Life. Princeton University Press, 1996

    1996

  56. [64]

    Uncovering sociological effect heterogeneity using tree-based machine learning.Sociological Methodology, 51(2):189–223, 2021

    Jennie E Brand, Jiahui Xu, Bernard Koch, and Pablo Geraldo. Uncovering sociological effect heterogeneity using tree-based machine learning.Sociological Methodology, 51(2):189–223, 2021

    2021

  57. [65]

    Correlated topic models.Advances in neural information processing systems, 18:147, 2006

    David Blei and John Lafferty. Correlated topic models.Advances in neural information processing systems, 18:147, 2006

    2006

  58. [66]

    The development and psychometric properties of liwc2015

    James W Pennebaker, Ryan L Boyd, Kayla Jordan, and Kate Blackburn. The development and psychometric properties of liwc2015. Technical report, 2015

    2015

  59. [67]

    Latent dirichlet allocation.Journal of machine Learning research, 3(Jan):993–1022, 2003

    David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichlet allocation.Journal of machine Learning research, 3(Jan):993–1022, 2003

    2003

  60. [68]

    Select number of topics for lda model.Available on: https://cran

    Murzintcev Nikita. Select number of topics for lda model.Available on: https://cran. rproject. org/web/packages/ldatuning/vignettes/topics. html.[March 3rd, 2019], 2016

    2019

  61. [69]

    A density-based method for adaptive lda model selection.Neurocomputing, 72(7-9):1775–1781, 2009

    Juan Cao, Tian Xia, Jintao Li, Yongdong Zhang, and Sheng Tang. A density-based method for adaptive lda model selection.Neurocomputing, 72(7-9):1775–1781, 2009

    2009

  62. [70]

    Finding scientific topics.Proceedings of the National academy of Sciences, 101(suppl 1):5228–5235, 2004

    Thomas L Griffiths and Mark Steyvers. Finding scientific topics.Proceedings of the National academy of Sciences, 101(suppl 1):5228–5235, 2004

    2004

  63. [71]

    On finding the natural number of topics with latent dirichlet allocation: Some observations

    Rajkumar Arun, Venkatasubramaniyan Suresh, CE Veni Madhavan, and MN Narasimha Murthy. On finding the natural number of topics with latent dirichlet allocation: Some observations. InPacific-Asia conference on knowledge discovery and data mining, pages 391–402. Springer, 2010. 2...

    2010

  64. [72]

    Accurate and effective latent concept modeling for ad hoc information retrieval.Document numérique, 17(1):61–84, 2014

    Romain Deveaud, Eric SanJuan, and Patrice Bellot. Accurate and effective latent concept modeling for ad hoc information retrieval.Document numérique, 17(1):61–84, 2014

    2014

  65. [73]

    Social causation or social construction of suicide? an investigation into the social organization of official rates.American Sociological Review, pages 80–100, 1986

    Bernice A Pescosolido and Robert Mendelsohn. Social causation or social construction of suicide? an investigation into the social organization of official rates.American Sociological Review, pages 80–100, 1986

    1986

  66. [74]

    When small words foretell academic success: The case of college admissions essays.PloS one, 9(12):e115844, 2014

    James W Pennebaker, Cindy K Chung, Joey Frazee, Gary M Lavergne, and David I Beaver. When small words foretell academic success: The case of college admissions essays.PloS one, 9(12):e115844, 2014

    2014

  67. [75]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human langua...

    2019

  68. [76]

    Bertopic: Neural topic modeling with a class-based tf-idf procedure.arXiv preprint arXiv:2203.05794, 2022

    Maarten Grootendorst. Bertopic: Neural topic modeling with a class-based tf-idf procedure.arXiv preprint arXiv:2203.05794, 2022

    2022 arXiv

  69. [77]

    Xgboost: A scalable tree boosting system

    Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. InProceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785–794, 2016

    2016

  70. [78]

    Convolutional neural networks for sentence classification.arXiv preprint arXiv:1408.5882, 2014

    Yoon Kim. Convolutional neural networks for sentence classification.arXiv preprint arXiv:1408.5882, 2014

    2014 arXiv

  71. [79]

    David Picard. Torch. manual_seed (3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision.arXiv preprint arXiv:2109.08203, 2021

    2021 arXiv

  72. [80]

    Can generative ai improve social science?Proceedings of the National Academy of Sciences, 121(21):e2314021121, 2024

    Christopher A Bail. Can generative ai improve social science?Proceedings of the National Academy of Sciences, 121(21):e2314021121, 2024

    2024

  73. [81]

    Simulating subjects: The promise and peril of ai stand-ins for social agents and interactions

    Austin C Kozlowski and James Evans. Simulating subjects: The promise and peril of ai stand-ins for social agents and interactions. Apr 2025

    2025

  74. [82]

    Out of one, many: Using language models to simulate human samples.Political Analysis, 31(3):337–351, 2023

    Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. Out of one, many: Using language models to simulate human samples.Political Analysis, 31(3):337–351, 2023

    2023

  75. [83]

    In silico sociology: forecasting covid-19 polarization with large language models.arXiv preprint arXiv:2407.11190, 2024

    Austin C Kozlowski, Hyunku Kwon, and James A Evans. In silico sociology: forecasting covid-19 polarization with large language models.arXiv preprint arXiv:2407.11190, 2024

    2024 arXiv

  76. [84]

    Ai-augmented surveys: Leveraging large language models for opinion prediction in nationally representative surveys.arXiv preprint arXiv:2305.09620, 2023

    Junsol Kim and Byungkyu Lee. Ai-augmented surveys: Leveraging large language models for opinion prediction in nationally representative surveys.arXiv preprint arXiv:2305.09620, 2023

    2023 arXiv

  77. [85]

    Whose opinions do language models reflect? InInternational Conference on Machine Learning, pages 29971–30004

    Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? InInternational Conference on Machine Learning, pages 29971–30004. PMLR, 2023

    2023

  78. [86]

    Cultural bias and cultural alignment of large language models.PNAS nexus, 3(9):pgae346, 2024

    Yan Tao, Olga Viberg, Ryan S Baker, and René F Kizilcec. Cultural bias and cultural alignment of large language models.PNAS nexus, 3(9):pgae346, 2024

    2024

  79. [87]

    How to make causal inferences using texts.Science Advances, 8(42):eabg2652, 2022

    Naoki Egami, Christian J Fong, Justin Grimmer, Margaret E Roberts, and Brandon M Stewart. How to make causal inferences using texts.Science Advances, 8(42):eabg2652, 2022

    2022

  80. [88]

    they have black in their blood

    Marissa E Thompson, Sam Trejo, and AJ Alvero. “they have black in their blood”: Exploring how genetic ancestry tests affect racial appraisals and classifications. 2023

    2023

  81. [89]

    Generative ai meets open-ended survey responses: Research participant use of ai and homogenization.Sociological Methods & Research, page 00491241251327130, 2025

    Simone Zhang, Janet Xu, and AJ Alvero. Generative ai meets open-ended survey responses: Research participant use of ai and homogenization.Sociological Methods & Research, page 00491241251327130, 2025

    2025

  82. [90]

    Stereotypical gender associations in language have decreased over time.Sociological Science, 7:1–35, 2020

    Jason J Jones, Mohammad Ruhul Amin, Jessica Kim, and Steven Skiena. Stereotypical gender associations in language have decreased over time.Sociological Science, 7:1–35, 2020

    2020

  83. [91]

    Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning.Advances in Neural Information Processing Systems, 36:15614–15638, 2023

    2023

  84. [92]

    Large language models, social demography, and hegemony: comparing authorship in human and synthetic text.Journal of Big Data, 11(1):138, 2024

    AJ Alvero, Jinsook Lee, Alejandra Regla-Vargas, René F Kizilcec, Thorsten Joachims, and Anthony Lising Antonio. Large language models, social demography, and hegemony: comparing authorship in human and synthetic text.Journal of Big Data, 11(1):138, 2024

    2024

  85. [93]

    Poor alignment and steerability of large language models: Evidence from college admission essays

    Jinsook Lee, AJ Alvero, Thorsten Joachims, and René Kizilcec. Poor alignment and steerability of large language models: Evidence from college admission essays. 2025

    2025

  86. [94]

    Concept class analysis: A method for identifying cultural schemas in texts

    Marshall A Taylor and Dustin S Stoltz. Concept class analysis: A method for identifying cultural schemas in texts. Sociological Science, 7:544–569, 2020. 21 State-of-the-Art Fallacy

    2020

  87. [95]

    Structural topic models for open-ended survey responses

    Margaret E Roberts, Brandon M Stewart, Dustin Tingley, Christopher Lucas, Jetson Leder-Luis, Shana Kushner Gadarian, Bethany Albertson, and David G Rand. Structural topic models for open-ended survey responses. American journal of political science, 58(4):1064–1082, 2014

    2014

  88. [96]

    Embedding regression: Models for context-specific description and inference.American Political Science Review, 117(4):1255–1274, 2023

    Pedro L Rodriguez, Arthur Spirling, and Brandon M Stewart. Embedding regression: Models for context-specific description and inference.American Political Science Review, 117(4):1255–1274, 2023

    2023

  89. [97]

    Model uncertainty in sociological research: An application to religion and economic growth

    Cristobal Young. Model uncertainty in sociological research: An application to religion and economic growth. American Sociological Review, 74(3):380–397, 2009

    2009

  90. [98]

    We ran 9 billion regressions: Eliminating false positives through computational model robustness.Sociological Methodology, 48(1):1–33, 2018

    John Muñoz and Cristobal Young. We ran 9 billion regressions: Eliminating false positives through computational model robustness.Sociological Methodology, 48(1):1–33, 2018

    2018

  91. [99]

    Coping with plenitude: A computational approach to selecting the right algorithm.Sociological Methods & Research, 52(4):1838–1882, 2023

    Ramina Sotoudeh and Paul DiMaggio. Coping with plenitude: A computational approach to selecting the right algorithm.Sociological Methods & Research, 52(4):1838–1882, 2023. 22 State-of-the-Art Fallacy Appendix 23 Univ ersity Course offered? Course(s) NLP/T ext Analysis? Harv ar...

    2023

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.