pith. sign in

arxiv: 2605.19745 · v1 · pith:3CSQJP5Xnew · submitted 2026-05-19 · 📊 stat.OT

Making Uncertainty Visible: Multiverse Analysis for Robust Computational Social Science

Maximilian Linde , Jun Sun , Paul Balluff , Danica Radovanovi\'c , Chung-hong Chan This is my paper

Pith reviewed 2026-05-20 01:31 UTC · model grok-4.3

classification 📊 stat.OT
keywords multiverse analysisresearcher degrees of freedomcomputational social sciencerobustnesstransparencyBayesian analysisnetwork modelingmachine learning
0
0 comments X

The pith

Multiverse analysis reveals how computational social science findings shift across different methodological choices.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Computational social science often relies on methods like Bayesian modeling, network generation, and machine learning that involve many discretionary decisions during data processing and analysis. These choices, known as researcher degrees of freedom, can produce different results even when applied to the same data and question. The paper tests this idea by running multiverse analyses on three published studies, systematically varying plausible combinations of those decisions. The results show that empirical conclusions can change substantially depending on the path taken, and that some combinations simply fail to run. Making these variations and failures visible improves the ability to judge how robust any single reported finding really is.

Core claim

Through three case studies the paper shows that multiverse analysis applied to Bayesian analysis, network generative modeling, and machine learning with or without large language models exposes how published computational social science results vary with alternative but defensible methodological decision combinations and also identifies combinations that lead to computational failure.

What carries the argument

Multiverse analysis, the systematic evaluation of results across all reasonable combinations of researcher choices in data preparation, modeling, and inference.

If this is right

  • Single-analysis reports in computational social science likely understate the uncertainty attached to their conclusions.
  • Computational failures that arise from certain methodological paths should be reported rather than omitted.
  • Defensible decision combinations can be identified by consulting prior literature and domain experts before running the multiverse.
  • Fair communication of multiverse results requires showing the full distribution of outcomes rather than highlighting only the preferred path.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same multiverse approach could be extended to other computational fields such as computational linguistics or digital humanities where similar discretionary choices exist.
  • Journals could require a brief multiverse summary as a standard robustness check for papers using flexible computational methods.
  • Pre-registering the space of decisions before seeing the data would reduce the risk that the multiverse itself is shaped by post-hoc preferences.

Load-bearing premise

The specific sets of decision combinations examined in the case studies are representative of the full range of plausible choices other analysts could reasonably make on the same data and questions.

What would settle it

A re-analysis of one of the three original datasets using a new set of methodological choices that produces results lying entirely outside the range of outcomes reported in the corresponding multiverse analysis.

read the original abstract

Through case studies, we demonstrate how multiverse analysis can strengthen the robustness and transparency of computational social science findings against alternative methodological decisions. We conduct multiverse analyses of three published social science studies that use the following computational methods: Bayesian analysis, network generative modeling, and machine learning with or without large language models. These methods are applied frequently in computational social science studies, yet entail a greater degree of arbitrariness in terms of methodological choices, or "researcher degrees of freedom." Our multiverse analyses reveal how the empirical findings in these studies vary as a function of various plausible decision combinations. Our three case studies also expose an often-ignored motivation for conducting multiverse analysis: Showing which methodological combinations lead to computational failure. These failed cases are usually not communicated in the published reports, even though these sophisticated computational methods have a much higher likelihood of failure. We end our paper with suggestions on how to find defensible decision combinations for multiverse analysis of computational social science studies and how to communicate multiverse analysis findings fairly.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The paper claims that multiverse analysis strengthens robustness and transparency in computational social science by revealing how empirical findings vary across plausible methodological decisions and by exposing computational failures. It supports this via three case studies applying the approach to published work using Bayesian analysis, network generative modeling, and machine learning (with or without LLMs), and closes with suggestions for selecting defensible decision combinations and communicating results fairly.

Significance. If the examined decision sets prove representative, the work usefully highlights an under-communicated motivation for multiverse analysis—the documentation of failure modes in complex computational pipelines—and offers practical guidance for a field with high researcher degrees of freedom. The empirical demonstrations on real published studies are a concrete strength.

major comments (1)
  1. [Case studies (Sections 3–5) and concluding suggestions] The central claim that the three case studies demonstrate strengthened robustness against alternative methodological decisions depends on the examined combinations being representative of plausible researcher degrees of freedom. The manuscript presents specific choices for each case study (Bayesian priors/models, network parameters, ML/LLM pipelines) as plausible but supplies no systematic elicitation, literature survey, or external validation showing coverage of the space other analysts would reasonably explore. This assumption is load-bearing for the robustness conclusion.
minor comments (2)
  1. [Abstract] The abstract states that findings 'vary as a function of various plausible decision combinations' but does not report the total number of combinations tested per case study; adding these counts would clarify the scale of each multiverse.
  2. [Results presentation] Notation for decision grids or failure rates could be standardized across the three case studies to ease comparison.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the detailed and constructive report. The major comment raises an important point about the scope of our case studies, which we address directly below.

read point-by-point responses

  1. Referee: The central claim that the three case studies demonstrate strengthened robustness against alternative methodological decisions depends on the examined combinations being representative of plausible researcher degrees of freedom. The manuscript presents specific choices for each case study (Bayesian priors/models, network parameters, ML/LLM pipelines) as plausible but supplies no systematic elicitation, literature survey, or external validation showing coverage of the space other analysts would reasonably explore. This assumption is load-bearing for the robustness conclusion.

    Authors: We agree that the manuscript does not contain a systematic literature survey or external validation exercise to demonstrate that the chosen decision sets exhaustively cover the space of plausible researcher degrees of freedom. Our intent was not to claim such exhaustive coverage. Instead, the case studies were designed to illustrate the practical application of multiverse analysis to real published CSS work, showing both variation in results and the occurrence of previously unreported computational failures. The specific decisions were selected from methodological variations explicitly discussed or implied in the original papers and from standard practices in each subfield (e.g., common prior families in Bayesian modeling, typical parameter ranges in network models). We will revise the manuscript to (a) add a dedicated subsection in each case study explaining the rationale and sources for the included decisions, (b) explicitly state that the multiverses are illustrative rather than exhaustive, and (c) qualify the robustness claims to refer to the explored decision space. These changes will prevent over-interpretation while preserving the core demonstration that multiverse analysis can surface sensitivity and failure modes that single-analysis reporting conceals. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical demonstrations without fitted predictions or load-bearing self-citations

full rationale

The paper consists of three case studies applying multiverse analysis to existing published studies in Bayesian modeling, network generation, and ML/LLM pipelines. No equations, parameters, or first-principles derivations are present that could reduce a claimed prediction to a fitted input or self-referential definition. The central demonstration—that varying methodological choices reveals sensitivity—is grounded in direct computation on the chosen decision sets rather than any circular renaming, imported uniqueness theorem, or self-citation chain. The representativeness assumption noted by the skeptic is a potential external-validity concern but does not constitute circularity under the specified criteria, as the paper does not claim deductive necessity or statistical forcing from its own inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the domain assumption that the chosen methodological alternatives adequately sample the space of reasonable researcher decisions; no free parameters or new entities are introduced.

axioms (1)
  • domain assumption Alternative methodological decisions constitute researcher degrees of freedom that should be systematically explored for robustness
    Invoked throughout the description of the three case studies and the motivation for multiverse analysis.

pith-pipeline@v0.9.0 · 5717 in / 1224 out tokens · 35550 ms · 2026-05-20T01:31:15.373149+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

125 extracted references · 125 canonical work pages

  1. [1]

    American Economic Review , year = 2015, volume = 105, number = 5, month = may, pages =

    Athey, Susan and Imbens, Guido , title =. American Economic Review , year = 2015, volume = 105, number = 5, month = may, pages =. doi:10.1257/aer.p20151020 , url =

    work page doi:10.1257/aer.p20151020 2015

  2. [2]

    doi:10.31222/osf.io/4yzeh_v1 , url =

    Short, Cassie and Breznau, Nate and Bruntsch, Maria and Burkhardt, Micha and Busch, Niko and Cesnaite, Elena and Frank, Maximilian and Gießing, Carsten and Krähmer, Daniel and Kristanto, Daniel and Lonsdorf, Tina B and Neuendorf, Claudia and Nguyen, Hung Hoang Viet and Rausch, Manuel and Schmalz, Xenia and Schneck, Andreas and Tabakci, Cem and Hildebrandt...

    work page doi:10.31222/osf.io/4yzeh_v1

  3. [3]

    The Review of Economics and Statistics , volume =

    Lonnie Magee , title =. The Review of Economics and Statistics , volume =. 1990 , month =. doi:10.2307/2109761 , url =

    work page doi:10.2307/2109761 1990

  4. [4]

    Leamer , title =

    Edward E. Leamer , title =. The American Economic Review , volume =. 1983 , publisher =

    work page 1983

  5. [5]

    We Ran 9 Billion Regressions: Eliminating False Positives through Computational Model Robustness , journal =

    Mu\. We Ran 9 Billion Regressions: Eliminating False Positives through Computational Model Robustness , journal =. doi:10.1177/0081175018777988 , url =

    work page doi:10.1177/0081175018777988

  6. [6]

    Quality & Quantity , year = 2023, volume = 58, number = 2, month = jun, pages =

    Cantone, Giulio Giacomo and Tomaselli, Venera , title =. Quality & Quantity , year = 2023, volume = 58, number = 2, month = jun, pages =. doi:10.1007/s11135-023-01698-5 , url =

    work page doi:10.1007/s11135-023-01698-5 2023

  7. [7]

    International Journal of Epidemiology , year = 2020, volume = 50, number = 1, month = nov, pages =

    Klau, Simon and Hoffmann, Sabine and Patel, Chirag J and Ioannidis, John PA and Boulesteix, Anne-Laure , title =. International Journal of Epidemiology , year = 2020, volume = 50, number = 1, month = nov, pages =. doi:10.1093/ije/dyaa164 , url =

    work page doi:10.1093/ije/dyaa164 2020

  8. [8]

    Sociological Methods & Research , year = 2016, volume = 46, number = 1, month = jul, pages =

    Young, Cristobal and Holsteen, Katherine , title =. Sociological Methods & Research , year = 2016, volume = 46, number = 1, month = jul, pages =. doi:10.1177/0049124115610347 , url =

    work page doi:10.1177/0049124115610347 2016

  9. [9]

    Navigating Child Online Protection in

    Radovanovi\'. Navigating Child Online Protection in. Digital Society , year = 2026, volume = 5, number = 1, month = Jan, issn =. doi:10.1007/s44206-025-00244-0 , url =

    work page doi:10.1007/s44206-025-00244-0 2026

  10. [10]

    Political Communication , year = 2016, volume = 34, number = 3, month = oct, pages =

    Scharkow, Michael and Bachl, Marko , title =. Political Communication , year = 2016, volume = 34, number = 3, month = oct, pages =. doi:10.1080/10584609.2016.1235640 , url =

    work page doi:10.1080/10584609.2016.1235640 2016

  11. [11]

    The International Journal of Press/Politics , year =

    Christoph Ivanusch and Paul Balluff , title =. The International Journal of Press/Politics , year =

    work page

  12. [12]

    and Klingler, Jonathan D

    Ramey, Adam J. and Klingler, Jonathan D. and Hollibaugh, Gary E. , title =. Political Science Research and Methods , year = 2016, volume = 7, number = 1, month = mar, pages =. doi:10.1017/psrm.2016.12 , url =

    work page doi:10.1017/psrm.2016.12 2016

  13. [13]

    and Wagner, Claudia , title =

    Birkenmaier, Lukas and Lechner, Clemens M. and Wagner, Claudia , title =. Communication Methods and Measures , year = 2023, volume = 18, number = 3, month = nov, pages =. doi:10.1080/19312458.2023.2285765 , url =

    work page doi:10.1080/19312458.2023.2285765 2023

  14. [14]

    Can Large Language Models Transform Computational Social Science?

    Ziems, Caleb and Held, William and Shaikh, Omar and Chen, Jiaao and Zhang, Zhehao and Yang, Diyi. Can Large Language Models Transform Computational Social Science?. Computational Linguistics. 2024. doi:10.1162/coli_a_00502

    work page doi:10.1162/coli_a_00502 2024

  15. [15]

    doi:10.48550/ARXIV.2401.00284 , url =

    Weber, Maximilian and Reichardt, Merle , title =. doi:10.48550/ARXIV.2401.00284 , url =

    work page doi:10.48550/arxiv.2401.00284

  16. [16]

    Introduction to Exponential-family Random Graph Models with ergm , howpublished =

    work page

  17. [17]

    Jones, James Holland and Ready, Elspeth and Hazel, Ashley , title =

    work page

  18. [18]

    Proceedings of the National Academy of Sciences , year = 2025, volume = 122, number = 43, month = oct, issn =

    Auspurg, Katrin , title =. Proceedings of the National Academy of Sciences , year = 2025, volume = 122, number = 43, month = oct, issn =. doi:10.1073/pnas.2521917122 , url =

    work page doi:10.1073/pnas.2521917122 2025

  19. [19]

    Proceedings of the National Academy of Sciences , year = 2025, volume = 122, number = 25, month = jun, issn =

    Ganslmeier, Michael and Vlandas, Tim , title =. Proceedings of the National Academy of Sciences , year = 2025, volume = 122, number = 25, month = jun, issn =. doi:10.1073/pnas.2414926122 , url =

    work page doi:10.1073/pnas.2414926122 2025

  20. [20]

    Proceedings of the National Academy of Sciences , year = 2014, volume = 111, number = 29, month = jul, pages =

    Editorial Expression of Concern: Experimental evidence of massivescale emotional contagion through social networks , author =. Proceedings of the National Academy of Sciences , year = 2014, volume = 111, number = 29, month = jul, pages =. doi:10.1073/pnas.1412469111 , url =

    work page doi:10.1073/pnas.1412469111 2014

  21. [21]

    Kramer, Adam D. I. and Guillory, Jamie E. and Hancock, Jeffrey T. , title =. Proceedings of the National Academy of Sciences , year = 2014, volume = 111, number = 24, month = jun, pages =. doi:10.1073/pnas.1320040111 , url =

    work page doi:10.1073/pnas.1320040111 2014

  22. [22]

    doi:10.1109/mis.2014.80 , url =

    Strohmaier, Markus and Wagner, Claudia , title =. doi:10.1109/mis.2014.80 , url =

    work page doi:10.1109/mis.2014.80 2014

  23. [23]

    Journal of the American Statistical Association , volume=

    Goodness of fit of social network models , author=. Journal of the American Statistical Association , volume=. 2008 , publisher=

    work page 2008

  24. [24]

    EPJ Data Science , volume=

    Computational reproducibility in computational social science , author=. EPJ Data Science , volume=. 2024 , publisher=

    work page 2024

  25. [25]

    Child development , volume=

    Relational aggression, gender, and social-psychological adjustment , author=. Child development , volume=. 1995 , publisher=

    work page 1995

  26. [26]

    Kracking

    “Kracking” the missing data problem: applying krackhardt's cognitive social structures to school-based social networks , author=. Sociology of Education , volume=. 2008 , publisher=

    work page 2008

  27. [27]

    Social networks , volume=

    An introduction to exponential random graph (p*) models for social networks , author=. Social networks , volume=. 2007 , publisher=

    work page 2007

  28. [28]

    Rijnhart, Judith J. M. and Twisk, Jos W. R. and Deeg, Dorly J. H. and Heymans, Martijn W. , title =. Prevention Science , year = 2021, volume = 23, number = 5, month = jul, pages =. doi:10.1007/s11121-021-01280-1 , url =

    work page doi:10.1007/s11121-021-01280-1 2021

  29. [29]

    and Carpenter, Catherine M

    Mullin, Hollie A. and Carpenter, Catherine M. and Cwiek, Andrew P. and Lan, Gloria and Chase, Spencer O. and Carter, Emily E. and Vervoordt, Samantha M. and Rabinowitz, Amanda and Venkatesan, Umesh and Hillary, Frank G. , title =. Network Neuroscience , year = 2025, volume = 9, number = 3, pages =. doi:10.1162/netn_a_00459 , url =

    work page doi:10.1162/netn_a_00459 2025

  30. [30]

    and Wagner, Claudia , title =

    TeBlunthuis, Nathan and Hase, Valerie and Chan, Chung-Hong , title =. Communication Methods and Measures , year = 2024, month = jan, pages =. doi:10.1080/19312458.2023.2293713 , url =

    work page doi:10.1080/19312458.2023.2293713 2024

  31. [31]

    Political Analysis , year = 2020, volume = 29, number = 4, month =

    Fong, Christian and Tyler, Matthew , title =. Political Analysis , year = 2020, volume = 29, number = 4, month =. doi:10.1017/pan.2020.38 , url =

    work page doi:10.1017/pan.2020.38 2020

  32. [32]

    Journalism & Mass Communication Quarterly , year = 2025, month = oct, issn =

    Linde, Maximilian and Chan, Chung-hong and Balluff, Paul , title =. Journalism & Mass Communication Quarterly , year = 2025, month = oct, issn =. doi:10.1177/10776990251378122 , url =

    work page doi:10.1177/10776990251378122 2025

  33. [33]

    and Hovy, Dirk , title =

    Baumann, Joachim and Röttger, Paul and Urman, Aleksandra and Wendsjö, Albert and Plaza-del-Arco, Flor Miriam and Gruber, Johannes B. and Hovy, Dirk , title =. doi:10.48550/ARXIV.2509.08825 , url =

    work page doi:10.48550/arxiv.2509.08825

  34. [34]

    and Fariss, Christopher J

    Bond, Robert M. and Fariss, Christopher J. and Jones, Jason J. and Kramer, Adam D. I. and Marlow, Cameron and Settle, Jaime E. and Fowler, James H. , title =. Nature , year = 2012, volume = 489, number = 7415, month = sep, pages =. doi:10.1038/nature11421 , url =

    work page doi:10.1038/nature11421 2012

  35. [35]

    Four best practices for measuring news sentiment using ‘off-the-shelf’ dictionaries: a large-scale p-hacking experiment , url=

    Chan, Chung-hong and Bajjalieh, Joseph and Auvil, Loretta and Wessler, Hartmut and Althaus, Scott and Welbers, Kasper and. Four best practices for measuring news sentiment using ‘off-the-shelf’ dictionaries: a large-scale p-hacking experiment , url=. doi:10.31235/osf.io/np5wa , journal=

    work page doi:10.31235/osf.io/np5wa

  36. [36]

    2025 , publisher=

    Multiverse analysis: Computational methods for robust results , author=. 2025 , publisher=

    work page 2025

  37. [37]

    , title =

    Jung, Kiju and Shavitt, Sharon and Viswanathan, Madhu and Hilbe, Joseph M. , title =. Proceedings of the National Academy of Sciences , year = 2014, volume = 111, number = 24, month = jun, pages =. doi:10.1073/pnas.1402786111 , url =

    work page doi:10.1073/pnas.1402786111 2014

  38. [38]

    Lazer, David M. J. and Pentland, Alex and Watts, Duncan J. and Aral, Sinan and Athey, Susan and Contractor, Noshir and Freelon, Deen and Gonzalez-Bailon, Sandra and King, Gary and Margetts, Helen and Nelson, Alondra and Salganik, Matthew J. and Strohmaier, Markus and Vespignani, Alessandro and Wagner, Claudia , title =. Science , year = 2020, volume = 369...

    work page doi:10.1126/science.aaz8170 2020

  39. [39]

    Social networks , volume=

    Cognitive social structures , author=. Social networks , volume=. 1987 , publisher=

    work page 1987

  40. [40]

    Infant and child development , volume=

    Understanding children's prosocial behaviour and classroom affiliative relationships: A social network analysis , author=. Infant and child development , volume=. 2023 , publisher=

    work page 2023

  41. [41]

    Journal of Statistical Software , volume=

    ergm: A package to fit, simulate and diagnose exponential-family models for networks , author=. Journal of Statistical Software , volume=. 2008 , doi=

    work page 2008

  42. [43]

    and Liu, Yang and Jansen, Yvonne and Dragicevic, Pierre and Chevalier, Fanny and Kay, Matthew , title =

    Hall, Brian D. and Liu, Yang and Jansen, Yvonne and Dragicevic, Pierre and Chevalier, Fanny and Kay, Matthew , title =. Computer Graphics Forum , year = 2022, volume = 41, number = 1, month = feb, pages =. doi:10.1111/cgf.14443 , url =

    work page doi:10.1111/cgf.14443 2022

  43. [44]

    and Nelson, Leif D

    Simonsohn, Uri and Simmons, Joseph P. and Nelson, Leif D. , title =. Nature Human Behaviour , year = 2020, volume = 4, number = 11, month = jul, pages =. doi:10.1038/s41562-020-0912-z , url =

    work page doi:10.1038/s41562-020-0912-z 2020

  44. [45]

    , title =

    Harder, Jenna A. , title =. Perspectives on Psychological Science , year = 2020, volume = 15, number = 5, month = jun, pages =. doi:10.1177/1745691620917678 , url =

    work page doi:10.1177/1745691620917678 2020

  45. [46]

    and Brandes, Ulrik , title =

    Schoch, David and Valente, Thomas W. and Brandes, Ulrik , title =. Social Networks , year = 2017, volume = 50, month = jul, pages =. doi:10.1016/j.socnet.2017.03.010 , url =

    work page doi:10.1016/j.socnet.2017.03.010 2017

  46. [47]

    Social Networks , year = 2019, volume = 59, month = oct, pages =

    Stoltenberg, Daniela and Maier, Daniel and Waldherr, Annie , title =. Social Networks , year = 2019, volume = 59, month = oct, pages =. doi:10.1016/j.socnet.2019.07.001 , url =

    work page doi:10.1016/j.socnet.2019.07.001 2019

  47. [48]

    2022 , journal =

    Observing Many Researchers Using the Same Data and Hypothesis Reveals a Hidden Universe of Uncertainty , author =. 2022 , journal =

    work page 2022

  48. [49]

    2021 , journal =

    A Traveler's Guide to the Multiverse: Promises, Pitfalls, and a Framework for the Evaluation of Analytic Decisions , author =. 2021 , journal =

    work page 2021

  49. [50]

    2012 , journal =

    Why We (Usually) Don't Have to Worry about Multiple Comparisons , author =. 2012 , journal =

    work page 2012

  50. [51]

    2014 , journal =

    The Statistical Crisis in Science , author =. 2014 , journal =

    work page 2014

  51. [52]

    2015 , journal =

    Assessment of Vibration of Effects Due to Model Specification Can Demonstrate the Instability of Observational Associations , author =. 2015 , journal =

    work page 2015

  52. [53]

    doi:10.48550/arXiv.2109.08203 , pubstate =

    Picard, David , year =. doi:10.48550/arXiv.2109.08203 , pubstate =. 2109.08203 , eprinttype =

    work page doi:10.48550/arxiv.2109.08203

  53. [54]

    2023 , journal =

    If You Have Choices, Why Not Choose (and Share) All of Them? A Multiverse Approach to Understanding News Engagement on Social Media , author =. 2023 , journal =

    work page 2023

  54. [55]

    Proceedings of the 2023

    Sarma, Abhraneel and Kale, Alex and Moon, Michael Jongho and Taback, Nathan and Chevalier, Fanny and Hullman, Jessica and Kay, Matthew , year =. Proceedings of the 2023. doi:10.1145/3544548.3580726 , isbn =

    work page doi:10.1145/3544548.3580726 2023

  55. [56]

    2018 , journal =

    Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results , author =. 2018 , journal =

    work page 2018

  56. [57]

    2011 , journal =

    False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant , author =. 2011 , journal =

    work page 2011

  57. [58]

    2016 , journal =

    Increasing Transparency through a Multiverse Analysis , author =. 2016 , journal =

    work page 2016

  58. [59]

    Increasing the Transparency of Research Papers with Explorable Multiverse Analyses , booktitle =

    Dragicevic, Pierre and Jansen, Yvonne and Sarma, Abhraneel and Kay, Matthew and Chevalier, Fanny , year =. Increasing the Transparency of Research Papers with Explorable Multiverse Analyses , booktitle =. doi:10.1145/3290605.3300295 , isbn =

    work page doi:10.1145/3290605.3300295

  59. [60]

    IEEE Transactions on Visualization and Computer Graphics , volume =

    Liu, Yang and Kale, Alex and Althoff, Tim and Heer, Jeffrey , year =. IEEE Transactions on Visualization and Computer Graphics , volume =

    work page

  60. [61]

    2021 , journal =

    Same Data, Different Conclusions: Radical Dispersion in Empirical Results When Independent Analysts Operationalize and Test the Same Hypothesis , author =. 2021 , journal =

    work page 2021

  61. [62]

    Supporting

    Riha, Anna Elisabeth and Siccha, Nikolas and Oulasvirta, Antti and Vehtari, Aki , year =. Supporting. doi:10.48550/arXiv.2404.01688 , pubstate =. 2404.01688 , eprinttype =

    work page doi:10.48550/arxiv.2404.01688

  62. [63]

    Political Analysis , author=

    Measuring Politicians’ Public Personality Traits Using Computational Text Analysis: A Multimethod Feasibility Study for Agency and Communion , DOI=. Political Analysis , author=. 2025 , pages=

    work page 2025

  63. [64]

    Chan, Chung-hong and Rauchfleisch, Adrian , year = 2023, journal =

    work page 2023

  64. [65]

    doi:10.17605/OSF.IO/UZCA3 , urldate =

    Puschmann, Cornelius and Haim, Mario , year = 2020, month = jan, publisher =. doi:10.17605/OSF.IO/UZCA3 , urldate =

    work page doi:10.17605/osf.io/uzca3 2020

  65. [66]

    China statistical yearbook 2019 , author =

    work page 2019

  66. [67]

    Journal of Statistical Software , volume =

    B. Journal of Statistical Software , volume =

    work page

  67. [68]

    Advanced

    B. Advanced. The R Journal , volume =

    work page

  68. [69]

    International news flow theory revisited through a space--time interaction model:

    Grasland, Claude , year = 2020, journal =. International news flow theory revisited through a space--time interaction model:

    work page 2020

  69. [70]

    Journal of Communication , volume =

    Systemic determinants of international news coverage: a comparison of 38 countries , author =. Journal of Communication , volume =

    work page

  70. [71]

    Proceedings of the National Academy of Sciences , volume =

    The preregistration revolution , author =. Proceedings of the National Academy of Sciences , volume =. doi:10.1073/pnas.1708274114 , urldate =

    work page doi:10.1073/pnas.1708274114

  71. [72]

    Evidence-Based Toxicology , volume =

    The benefits of preregistration and registered reports , author =. Evidence-Based Toxicology , volume =. doi:10.1080/2833373X.2024.2376046 , urldate =

    work page doi:10.1080/2833373x.2024.2376046 2024

  72. [73]

    Trends in Cognitive Sciences , volume =

    Preregistration is hard, and worthwhile , author =. Trends in Cognitive Sciences , volume =. doi:10.1016/j.tics.2019.07.009 , urldate =

    work page doi:10.1016/j.tics.2019.07.009 2019

  73. [74]

    , year = 2013, journal =

    Chambers, Christopher D. , year = 2013, journal =. Registered reports: a new publishing initiative at

    work page 2013

  74. [75]

    and Vianello, Michelangelo and Hasselman, Fred and Adams, Byron G

    Klein, Richard A. and Vianello, Michelangelo and Hasselman, Fred and Adams, Byron G. and AdamsJr., Reginald B. and Alper, Sinan and Aveyard, Mark and Axt, Jordan R. and Babalola, Mayowa T. and Bahn. Many labs 2:. Advances in Methods and Practices in Psychological Science , volume =. doi:10.1177/2515245918810225 , urldate =

    work page doi:10.1177/2515245918810225

  75. [76]

    and Atherton, Olivia E

    Ebersole, Charles R. and Atherton, Olivia E. and Belanger, Aimee L. and Skulborstad, Hayley M. and Allen, Jill M. and Banks, Jonathan B. and Baranski, Erica and Bernstein, Michael J. and Bonfiglio, Diane B. V. and Boucher, Leanne and Brown, Elizabeth R. and Budiman, Nancy I. and Cairo, Athena H. and Capaldi, Colin A. and Chartier, Christopher R. and Chung...

    work page doi:10.1016/j.jesp.2015.10.012 2015

  76. [77]

    and Uhlmann, E

    Silberzahn, R. and Uhlmann, E. L. and Martin, D. P. and Anselmi, P. and Aust, F. and Awtrey, E. and Bahn. Many analysts, one data set:. Advances in Methods and Practices in Psychological Science , volume =. doi:10.1177/2515245917747646 , urldate =

    work page doi:10.1177/2515245917747646

  77. [78]

    What's a multiverse good for anyway? , author =

    work page

  78. [79]

    Statistical rethinking: a

    McElreath, Richard , year = 2020, edition =. Statistical rethinking: a

    work page 2020

  79. [80]

    Acta Psychologica , volume =

    Specification curve analysis shows that social media use is linked to poor mental health, especially among girls , author =. Acta Psychologica , volume =. doi:10.1016/j.actpsy.2022.103512 , urldate =

    work page doi:10.1016/j.actpsy.2022.103512 2022

  80. [81]

    Nature Human Behaviour , volume =

    The association between adolescent well-being and digital technology use , author =. Nature Human Behaviour , volume =. doi:10.1038/s41562-018-0506-1 , urldate =

    work page doi:10.1038/s41562-018-0506-1

Showing first 80 references.