REVIEW 2 major objections 7 minor 5 references
Experiments in Social Media
T0 review · 2 major / 7 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A 61-million-person Facebook experiment may have flipped two Vermont races.
desk verdict A clear, well-written ethics commentary whose Vermont 'could have flipped an election' example is honest speculation, not evidence; fine as a position piece, thin as a research contribution. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the 2010 Facebook voter-mobilization experiment itself: 61 million US users aged 18 and over on 2 November 2010, randomly assigned to a social message with friends' 'I voted' thumbnails, a plain 'Today is Election Day' message, or a control, with an estimated 340,000 additional votes, about 0.5% of turnout, as the treatment effect. The argument works by multiplying a small average effect by a huge population and then intersecting it with the razor-thin margins of Vermont House seats; the demographic pivot is Facebook's skew toward women aged 18 to 29, which the paper uses to convert 'more turnout' into 'more Democratic turnout.' The central mechanism is scale-amplification of small effects, together with the electoral-risk idea that review boards should weigh societal-level consequences rather than only effects on individual subjects.
What would settle it
A re-analysis of Facebook's actual 2010 Vermont user data, matched to the state voter file and precinct returns, would settle the claim: if Vermont's Facebook users were not younger and more female than the voting population, or if younger women did not vote disproportionately for the Democratic female candidates, then the hypothesized one-or-two-vote shift has no empirical basis.
Extended reading notes
Core claim
The paper's central claim is that a randomized experiment run for scientific purposes can change who wins an election when the treatment effect, although tiny on average, is multiplied across tens of millions of users and lands in a jurisdiction with a one-vote margin. It takes the 2010 Facebook voter-participation experiment as its case: 61 million US Facebook users were randomly assigned to see an 'I voted' social message, an informational message, or nothing, and the analysis attributes roughly 340,000 additional votes to the intervention. Because Facebook's user base is not a demographic mirror of the electorate, the author argues, an ostensibly neutral get-out-the-vote nudge was not politically neutral; in the Windsor-Orange 1 and Rutland 5-4 Vermont House districts, decided by one vote each, one or two extra Democratic votes from a younger, more female Facebook population could have flipped both seats. The author is explicit that the election cannot be rerun and the counterfactual cannot be observed, so the claim is a reasoned possibility supported by the closeness of those races and the demographics of Facebook, not a proven causal fact.
Load-bearing premise
The claim that the experiment flipped the two Vermont races rests on two unverified demographic assumptions: that Vermont's 2010 Facebook users were younger and more female than its voters, and that those younger women voted for the female Democratic candidates; the paper offers no data for either.
Editorial extensions
If this is right
- A research intervention designed purely to increase turnout can alter election outcomes, so neutral intent and random assignment do not make an experiment politically neutral.
- Ethics review of social-media experiments should include societal-level risk, such as changing an election, and not only risks to individual subjects.
- Receiving or handing over already-collected experimental data should itself require consent or an institutional review board waiver, closing the route used in the emotional-contagion data analysis.
- Subjects who were experimented on without consent should be informed directly after the study, not merely through journal publication or press coverage.
- If public confidence is not protected by such safeguards, the resulting backlash could block legitimate large-scale behavioural research on social media.
Reading between the lines
- If the paper's single-vote argument is correct, the same logic extends beyond formal experiments to ordinary algorithmic feed ranking: any platform-driven change in message visibility near an election is a de facto experiment, whether or not it is labelled as research, and could be held to the same neutrality standard.
- An implementable safeguard the paper leaves implicit is pre-registering a demographic-balance and partisan-neutrality check for any voter-facing experiment, with a stopping rule if the projected effect could exceed the tightest expected margin in affected districts.
- The demographic assumption is historically testable: re-analysis of Facebook's 2010 user logs matched to Vermont voter records and precinct returns would settle whether the Facebook-using electorate was actually younger, more female, and more Democratic than the voters who decided those two races.
- Across all hundreds of races held on 2 November 2010, the expected number of elections flipped by the experiment is a computable quantity from the distribution of vote margins and the estimated treatment effect; the paper stops at two examples, but the general expected-flip count would make the scale of the risk concrete.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper is a short ethics commentary on large-scale social-media experiments. It recounts the 2010 Facebook 61-million-user get-out-the-vote experiment, argues that such interventions can affect election outcomes, and uses two Vermont House of Representatives races decided by a single vote as a case study. It then reviews the UCSD IRB waiver for that experiment, contrasts it with the Facebook-Cornell emotional contagion study, and proposes three recommendations: consider societal-level risks in ethics review, require IRB approval before transferring previously collected data, and inform subjects directly after experiments when consent was waived.
Significance. If the Vermont example were quantitatively established, the paper would make an important point: a randomized, nonpartisan get-out-the-vote experiment can alter electoral outcomes, and IRB review of social-media experiments should therefore consider aggregate societal effects. The paper is valuable for its close reading of the original IRB materials and for the clarity of its three recommendations. Its strengths are that it is transparent about the speculative status of the Vermont inference (using 'Suppose', 'could', and 'might') and that it grounds its discussion in primary sources. However, as it stands the central case study is not an established empirical finding, and the recommendations, while reasonable, are not derived from the case study in a quantitative way.
major comments (2)
- [A case study in voter participation] The assertion that the 2010 Facebook experiment could have supplied the decisive votes in Windsor-Orange 1 and Rutland 5-4 is not supported by the cited evidence. The paper jumps from the national estimate of 340,000 extra votes to 'This might easily have got one or two extra votes for the Democrats' without computing how many treated Facebook users lived in those two districts, what fraction of them were induced to vote by the intervention, or how their votes would have been split between the Democratic and Republican candidates. The two 'Suppose' assumptions are explicitly hypothetical and are backed only by 2014 national Pew demographics, not by 2010 Vermont or district-level data; if induced voters in those districts were not disproportionately Democratic, the expected Democratic gain could be zero or negative even if the assumptions held statewide. The paper should either provide this calculation and supporting data or explicitly label the Vermont passage as an unquantified motivational hypothetical, not as evidence that the experiment changed any election.
- [Ethical approval] The rhetorical question 'Shouldn't changing the outcome of some of the elections be considered an important risk?' rests on the unsupported claim that the experiment could plausibly have changed specific outcomes. The paper should separate the ethical argument from the empirical claim: the risk that a large get-out-the-vote experiment could change a close election is a legitimate IRB consideration even if the Vermont races are not shown to have been affected. As written, the paper's ethical critique is entangled with the unverified case study, making it seem that the validity of the critique depends on the truth of the Vermont speculation.
minor comments (7)
- [Abstract and first paragraph] 'safe guards' should be one word: 'safeguards'.
- [A case study in voter participation] 'difference groups' should be 'different groups'.
- [A case study in voter participation] The statement that the 340,000 additional votes is 'around 0.5% of the total number of votes cast' is arithmetically inaccurate: 340,000 is about 0.4% of the roughly 90 million votes cast in the 2010 U.S. midterm elections, or about 0.56% of the 61 million Facebook users, not 0.5% of votes cast.
- [Related experiments] The name appears as 'Inda Verma' in the text and as 'Inder Verma' in reference 7; the spelling should be consistent.
- [Recommendations] 'hand over date to a third party' should be 'hand over data to a third party'.
- [Recommendations] 'demographical balance' should be 'demographic balance'.
- [Recommendations] 'a follow up after the experiment' should be 'a follow-up after the experiment'.
Circularity Check
No circularity: the paper's argument is an ethical commentary built on external studies and explicit hypothetical assumptions, not on a self-referential derivation.
full rationale
This manuscript is a commentary/ethics essay rather than a derivation or empirical study. Its central claims about the 2010 Facebook voter-mobilization experiment are taken from the external, published Bond et al. (2012) paper (reference 1), and the Vermont election outcomes are taken from external news reports (references 2 and 3). The paper does not fit any parameter, define any quantity in terms of its own conclusions, or cite its own prior work as the load-bearing premise. The Vermont illustration is explicitly conditional: it begins with 'Suppose, for a moment, that Facebook had a younger and more female demographic in Vermont in 2010 than the voting population of Vermont itself' and 'Now suppose younger women in Vermont are more likely to vote for a female Democratic candidate than for a male Republican.' These are stated as assumptions used for a hypothetical illustration, not as outputs derived from the paper's own analysis. The subsequent 'one or two extra votes' is presented as a plausible consequence of those assumptions and the known closeness of the races, not as a measured or fitted prediction. Even if the Vermont example is under-supported or speculative, that is a correctness or evidentiary concern, not circular reasoning. The recommendations for ethical oversight do not rely on any quantity produced by the paper itself. No equation is derived, no parameter is fitted, and no self-citation supplies the conclusion. Accordingly, the circularity score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption Facebook's user base in Vermont in 2010 was younger and more female than the state's voting population.
- domain assumption Younger women in Vermont were more likely to vote for the female Democratic candidates than for the male Republicans.
- domain assumption The results of Bond et al. 2012 (61 million person experiment, 340,000 additional votes) are accurate.
Cite this review
Pith. "Pith review of Experiments in Social Media." pith.science (2026). https://pith.science/paper/DTKTY2SF
@misc{pith2026190809097,
author = {Pith},
title = {Pith review of: Experiments in Social Media},
year = {2026},
howpublished = {\url{https://pith.science/paper/DTKTY2SF}},
note = {Machine review of arXiv:1908.09097}
}
read the original abstract
Social media platforms like Facebook and Twitter permit experiments to be performed at minimal cost on populations of a size that scientists might previously have dreamt about. For instance, one experiment on Facebook involved over 60 million subjects. Such large scale experiments introduce new challenges as even small effects when multiplied by a large population can have a significant impact. Recent revelations about the use of social media to manipulate voting behaviour compound such concerns. It is believed that the psychometric data used by Cambridge Analytica to target US voters was collected by Dr Aleksandr Kogan from Cambridge University using a personality quiz on Facebook. There is a real risk that researchers wanting to collect data and run experiments on social media platforms in the future will face a public backlash that hinders such studies from being conducted. We suggest that stronger safe guards are put in place to help prevent this, and ensure the public retain confidence in scientists using social media for behavioural and other studies.
Reference graph
Works this paper leans on
-
[1]
Experiments in Social Media Author: Toby Walsh 1 Abstract. Social media platforms like Facebook and Twitter permit experiments to be performed at minimal cost on populations of a size that scientists might previously have dreamt about. For instance, one experiment on Facebook involved over 60 million subjects. Such large scale experiments introduce new ch...
work page 2010
-
[4]
https://doi.org/10.1073/pnas.1320040111 7.Inder Verma. Editorial Expression of Concern: Experimental evidence of massive-scale emotional contagion through social networks. PNAS July 22,
-
[5]
111 (29) 10779; https:// doi.org/10.1073/pnas.1412469111 Acknowledgments: The author acknowledges support from the European Research Council (AMPLify Advanced Grant 670077) and the Asian Office of Aerospace Research & Development (Grant FA2386-15-1-4016). The author thanks Dr James Fowler for providing copies of the documentation submitted to the IRB at U...
-
[2012]
However, the authors argue that the experiment “was consistent with Facebook’ s Data Use Policy, to which all users agree prior to creating an account on Facebook, constituting informed consent for this research” (6). Another aspect of the news feed experiment different to the midterm voter participation experiment was that it was less obvious to the subj...
work page 2012
-
[2014]
https://www.motherjones.com/politics/2014/10/can-voting-facebook- button-improve-voter-turnout/ 6.Adam D. I. Kramer, Jamie E. Guillory, and Jeffrey T. Hancock. Experimental evidence of massive-scale emotional contagion through social networks. PNAS June 17,
work page 2014
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.