Integrative Experiments Identify How Punishment Impacts Welfare in Public Goods Games

Abdullah Almaatouq; David G. Rand; Duncan J. Watts; Mohammed Alsobay

REVIEW 2 major objections 5 minor 1 cited by

Integrative Experiments Identify How Punishment Impacts Welfare in Public Goods Games

T0 review · 2 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Punishment consistently raises contributions to public goods, but across 360 game configurations its effect on welfare swings from a 43% gain to a 44% loss, and a model trained on those designs predicts outcomes in new experiments better th

desk verdict A solid, high-scope punishment experiment with genuine out-of-sample prediction; needs two caution flags on platform confound and the expert-comparison claim before its headline statements are taken at face value. read the letter →

arxiv 2508.17151 v1 pith:LSWNCJUC submitted 2025-08-23 econ.GN cs.GTcs.LGq-fin.EC

Mohammed Alsobay , David G. Rand , Duncan J. Watts , Abdullah Almaatouq This is my paper · ORCID

classification econ.GNcs.GTcs.LGq-fin.EC

keywords publicgoodsgamespunishmentcooperationsocialdilemmasintegrativeexperimentsmachinelearningpredictionefficiencycommunication

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Punishment reliably raises how much people contribute to a public good, but whether that helps or hurts the group's overall earnings is highly context-dependent: across 360 sampled game configurations, punishment improved welfare by as much as 43% and reduced it by as much as 44%. The paper argues this spread is systematic, not noise, because a statistical model trained on design parameters and control-condition efficiency predicted punishment outcomes in 20 brand-new experiments with out-of-sample R² of 0.53, while averaged predictions from domain experts and laypeople were close to no better than guessing. Communication was the single most important design feature, followed by contribution framing (opt-out vs. opt-in), contribution type, game length, visibility of others' earnings, and the presence of rewards. Most of these features combine through interactions rather than as independent add-ons—for example, longer games only help punishment when the group can communicate. The upshot is a shift from asking whether punishment 'works' to asking under which specific conditions it works.

What carries the argument

The central object is a 14-dimensional design space of public goods games, with each condition run as a matched pair (punishment enabled vs. disabled). The carrying mechanism is an elastic-net regression with pairwise feature interactions that takes the 13 design parameters plus the no-punishment control efficiency as inputs and predicts the punishment treatment efficiency; it is interpreted with permutation importance and SHAP values, and validated on pre-registered out-of-sample conditions. The paper also introduces 'normalized efficiency' as a scale-free outcome so game configurations with different multipliers and group sizes can be compared.

What would settle it

Run the same trained models on a fresh, pre-registered wave of randomly sampled conditions drawn from the same 14-dimensional space, with participants from a different recruitment source and sessions run at different times; if out-of-sample R² falls to zero or below the human benchmark, the claim that the design parameters are the systematic drivers fails.

Watch

Extended reading notes

Core claim

The central claim is that punishment's welfare effect is governed by specific, discoverable combinations of the game's design parameters. Averaged across all conditions, punishment hurt efficiency (from 0.71 to 0.63 in the learning wave), but this average masks large and statistically significant variation, with some configurations seeing welfare increase by roughly 43% and others fall by roughly 44%. The authors show that a model can predict these outcomes out-of-sample, which implies the heterogeneity is predictable, and that the dominant predictor is whether groups can communicate, with opt-out contribution framing, all-or-nothing vs. variable contributions, game length, and visibility of

Load-bearing premise

The prediction result assumes that the 13 game-design parameters plus the no-punishment baseline exhaust the systematic causes of punishment effectiveness, so if hidden factors—like which recruitment panel participants came from, when the session ran, or what was said in chat—vary with the design, the model's out-of-sample accuracy could be partly an artifact of those correlates.

Editorial extensions

If this is right

Policy and design decisions about introducing peer punishment should be made condition-by-condition, not on the average effect, because the same mechanism that raises contributions can either roughly double or roughly halve welfare.
Communication is not just another factor; it is the dominant lever, three times more important than any other parameter, and it changes whether longer interactions help or hurt.
Rarely studied features like the opt-out default matter more than the punishment technology's cost and impact, redirecting research attention from punishment mechanics to defaults, framing, and information structure.
The two-wave design plus out-of-sample prediction provides a template for turning heterogeneous, seemingly contradictory experimental results into quantitative predictions in other contested domains.
Because most important features interact, single-factor studies and meta-analyses that pool across unspecified procedures will keep producing conflicting answers unless the design space is sampled systematically.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the model's success generalizes beyond online Western participant pools, the same design space could be used to map where punishment-based institutions will backfire before they are deployed in field settings.
Because communication dominates, an untested extension is whether the content of chat matters—whether norm-setting and coordination, rather than mere presence of a chat window, is what drives the interaction with game length.
The finding that punishment-technology parameters barely predict outcomes suggests that efforts to tune the severity or cost of sanctions may matter far less than the social and informational context in which they are embedded—an inference the paper only partially endorses.
A direct test of the communication mechanism: add a condition where communication is enabled but restricted to non-task chat; if communication's importance persists, the mechanism is not coordination but something like group identity.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Bottom line: this is a serious paper and the first to simultaneously vary 14 PGG parameters at scale and show out-of-sample prediction of punishment effectiveness. The two-wave design is a real strength: Wave 1 Sobol-sampled learning data, Wave 2 pre-registered validation with 8–12 trials per condition, open code and data. The E-Net R² of 0.53 on held-out conditions is real evidence that the heterogeneity is systematic, not noise, and the feature-importance finding that communication dominates is interesting and consistent with prior single-factor work. No circularity: models are fit on Wave 1 and tested on Wave 2; using control efficiency as an input is legitimate, not leakage of the target.

Main soft spots are three. First, the learning wave mixed 604 MTurk participants with 3,014 Prolific, and the MTurk sessions are the earliest in the Sobol sequence, so design parameters and platform/time are correlated. That makes the attribution of predictive success to specific design features—especially the "communication is three times more important" claim—less clean. The validation wave is Prolific-only and still predicts well, so the central claim survives, but the authors should re-estimate without the MTurk games or include platform as a feature to show feature importances are stable. Second, the abstract says the models "outperformed" domain experts; the bootstrapped CI for the expert difference in RMSE is [-0.10, 3.71], which includes zero, so the wording overstates the evidence. Third, the single reward-ring game (efficiency 27) is excluded as an outlier; that is defensible but post-hoc, and it should be shown as a sensitivity rather than quietly filtered.

None of these break the main argument. The paper delivers a useful quantitative map of when punishment helps or hurts welfare, and it demonstrates the integrative-experiment-plus-ML recipe cleanly. The authors even concede in the Discussion that their prediction performance is likely optimistic because population variation is unexplored, which is the right honest framing.

This deserves a serious referee. I'd send it to review and ask for the platform-robustness check and a softer abstract before final acceptance. It's the kind of paper I'd assign in a reading group on experimental methods and prediction.

Referee Report

2 major / 5 minor

Summary. The paper reports a large-scale integrative experiment in which 14 public-good-game (PGG) design parameters are systematically varied, using 320 Sobol-sampled conditions in a learning wave and 40 pre-registered validation conditions with 8–12 replicates each (360 conditions total, 147,618 decisions). The central finding is that punishment reliably increases contributions but has highly heterogeneous effects on normalized efficiency, ranging from a 43% improvement to a 44% reduction across settings. The authors train several machine-learning models on the learning wave to predict treatment-group efficiency from the 13 design parameters plus control-group efficiency, and the elastic-net model achieves an out-of-sample R² of 0.53 on the 20 validation experiments, outperforming 553 human forecasters. Feature-importance analyses identify communication as the dominant predictor (about three times the next feature), with important interactions involving game length, contribution framing, contribution type, and peer-outcome visibility. The paper argues that heterogeneity in punishment effectiveness is systematic and predictable, rather than noise, and that the debate should shift from whether punishment works to the conditions under which it helps or harms welfare.

Significance. If the findings are robust, this is a substantial contribution: it is among the largest single experimental studies of punishment, combines a space-filling design with a pre-registered out-of-sample validation phase, provides open data and code, and demonstrates that a simple regularized model can outperform human experts at predicting the welfare consequences of punishment in new settings. The design's two-wave structure and the use of permutation tests and I² statistics are strengths. The paper also makes the useful methodological point that single-factor studies cannot quantify relative feature importance or detect higher-order interactions. However, the load-bearing inference that heterogeneity is attributable to design parameters, and the specific claim that communication is the dominant determinant, rest on an assumption of uniform protocols across learning and validation waves that is contradicted by the paper's own platform-switch description (SI S5.2). Because this assumption is central to the interpretation of both the out-of-sample R² and the feature-importance rankings, it must be addressed with a targeted robustness analysis before the paper's primary claims can be ful

major comments (2)

[SI S5.2 and Results ("Heterogeneous effects...")] The paper claims in Results that "because all experiments were conducted under identical protocols, recruitment procedures, and interfaces, this heterogeneity can only be attributed to differences in game parameters." This is contradicted by SI S5.2: the learning wave mixed 604 MTurk participants (7 early experiments) with 3,014 Prolific participants, while the validation wave was entirely Prolific. Because the learning conditions were generated sequentially by a Sobol sequence, the 7 MTurk experiments are the earliest Sobol points, so platform/phase is correlated with design coordinates. The prediction model (Methods, Model Construction) is trained on D_learning, which includes these 7 MTurk pairs, and then validated on Prolific-only data. If MTurk participants differ in attention, comprehension, or behavioral norms, the model could learn spurious associations between design parameters
[Results ("Predicting punishment effectiveness...") and SI S9.3] The headline out-of-sample R² of 0.53 is computed on 20 validation experiments. The paper provides bootstrap 95% CIs for RMSE (SI Table S8; E-Net RMSE CI [3.35, 5.69], ratio to baseline [0.53, 0.90]), but not for R². Since R² is a nonlinear transformation of the same residuals, the RMSE CI does not directly convey the uncertainty of R². Given the small number of validation conditions, the R² estimate could be imprecise; a bootstrap CI for R² (or the ratio of mean squared errors) should be reported. This is load-bearing because the abstract and discussion repeatedly emphasize R²=0.53 as the quantitative evidence that the heterogeneity is systematic and predictable.

minor comments (5)

[Abstract / Introduction] The abstract says "sampling 360 experimental conditions" but the Introduction says 320 learning conditions and 40 validation conditions; after filtering, the learning wave is described as 335 games / 150 matched pairs. Please make the count of conditions vs. games vs. matched pairs consistent and explicit.
[Methods, Model Construction] The text says "13 other PGG design parameters are used as inputs" because reward impact is dropped; this is fine but should be stated in the main text (Table 1 lists 14 parameters) so readers know why reward impact is excluded. Also specify which of the 13 are binary vs. continuous.
[SI S1] The paper honestly reports that the model hyperparameters were not pre-registered in advance of collecting the training data; the "stricter pre-registration strategy" note is appropriate, but the main text's characterization of the two-preregistration procedure should be softened to avoid implying the entire modeling pipeline was pre-registered.
[SI S5.2] Please report how many of the 150 matched learning pairs involve MTurk participants, and their design-parameter coordinates, so readers can assess the degree of correlation between platform and the Sobol sequence. This is directly relevant to the first major comment.
[Discussion (penultimate paragraph)] The caveat that the prediction exercise "likely represents an optimistic assessment of performance" is a useful limitation statement, but it is positioned as a general point about population variation. It should be moved closer to the results or methods and explicitly connected to the platform switch in the learning wave.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the out-of-sample prediction is genuinely held out.

full rationale

The paper's central derivation is a two-wave integrative experiment: models are trained on Wave 1 learning experiments using design parameters plus control-condition efficiency as inputs and treatment-condition efficiency as the target, then applied to Wave 2 configurations not used in training. The target variable never enters the fitting of the model's constants for the validation predictions; control efficiency is used as a legitimate input feature that mirrors the decision-maker scenario defined by the paper, not as a stand-in for the outcome. The reported out-of-sample R² = 0.53 is evaluated on the held-out Wave 2 treatment outcomes, and the human-forecaster comparison provides humans with the same inputs, so the model-vs-human claim is not constructed from the answer. Feature importance (communication, framing, game length, etc.) is derived from permutation and SHAP analyses applied to the fitted model, not from the validation labels. Self-citations to the integrative-experiment framework and the Empirica platform are methodological inputs and do not carry the empirical result; no uniqueness theorem or ansatz is imported to force the conclusions. The manuscript's own SI (S5.2) notes a platform transition in the learning wave (MTurk to Prolific), which is a potential confound for attributing predictive success specifically to design parameters, but that is an external-validity/correctness concern, not a circularity concern. No equation or fitted parameter is re-labelled as a prediction, and no claimed result reduces by definition to its inputs.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The paper's central claim is empirical and does not rest on a fitted constant from a theoretical derivation. The entries above are the assumptions and design choices that the out-of-sample prediction and heterogeneity claims depend on. The main conceptual input is that the 14-dimensional parameterization plus baseline efficiency is a sufficient description of the cooperative context; the main unmodeled risk is participant-platform variation. Model hyperparameters are listed as free parameters because they are fitted to Wave 1 and used for the headline E-Net predictor, though they are standard ML choices rather than scientific constants.

free parameters (4)

E-Net alpha = 0.07
Regularization strength selected by Bayesian optimization with 10-fold cross-validation on Wave 1; the headline predictor uses it.
E-Net l1_ratio = 0.15
Mixing parameter for L1/L2 regularization selected by the same cross-validated optimization.
k-means cluster count = 20
Chosen to enable SE estimation and visualization in the single-observation learning wave; affects the reported +43%/-44% extreme estimates in Fig 2D and Table S6.
Dropout exclusion threshold = 18% of intended group size
Pre-registered but ambiguously described; implementation excludes games missing more than 18% at game start, changing the sample from 366 to 335 learning games and 470 to 417 validation games.

assumptions (6)

domain assumption The 14 PGG design parameters plus the control-condition efficiency are sufficient to predict punishment-enabled efficiency; unspecified protocol differences are negligible.
Model Construction defines the prediction task as using design parameters and control efficiency to predict treatment efficiency. If important hidden moderators vary across the 360 conditions, the out-of-sample R² would not reflect generalizable design effects.
domain assumption Wave 1 single-observation-per-condition estimates are adequate for training predictive models, and Wave 2's 8-12 trials estimate condition means precisely enough to evaluate them.
Authors cite Kenny & Judd (2019), Moerbeek & Teerenstra (2021), and Baribault et al. (2018) for the breadth-over-precision sampling rationale; the validation results depend on this.
domain assumption Normalized efficiency is a valid cross-condition welfare measure, and full-defection and full-cooperation baselines appropriately scale earnings across different MPCRs.
Defined in Materials and Methods; all heterogeneity analysis and the 43%/44% effect ranges use it.
standard math Standard statistical machinery, including mixed-effects linear models, Cochran's Q, I², permutation tests, cross-validated hyperparameter selection, and bootstrap, is correctly applied.
Used throughout Results and Methods; no formal proof or code-level audit was possible in this review.
domain assumption Treatment and control arms of each configuration are comparable despite being played by different participants recruited in batches; random assignment and identical protocols make the only systematic difference the availability of punishment.
The causal interpretation of punishment effects in Figures 2B-2F requires this exchangeability, but the MTurk/Prolific split in Wave 1 partially threatens it.
domain assumption Outlier exclusions, specifically multiplier=1 games, one reward-ring game, and >18% dropout games, do not bias the central heterogeneity and prediction results.
S2.2 describes content-based and recruitment filters; Table S2 shows the dropout-criterion alternative is negligible, but the reward-ring exclusion is not reanalyzed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Integrative Experiments Identify How Punishment Impacts Welfare in Public Goods Games." pith.science (2026). https://pith.science/paper/LSWNCJUC

@misc{pith2026250817151,
  author       = {Pith},
  title        = {Pith review of: Integrative Experiments Identify How Punishment Impacts Welfare in Public Goods Games},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LSWNCJUC}},
  note         = {Machine review of arXiv:2508.17151}
}

read the original abstract

Punishment as a mechanism for promoting cooperation has been studied extensively for more than two decades, but its effectiveness remains a matter of dispute. Here, we examine how punishment's impact varies across cooperative settings through a large-scale integrative experiment. We vary 14 parameters that characterize public goods games, sampling 360 experimental conditions and collecting 147,618 decisions from 7,100 participants. Our results reveal striking heterogeneity in punishment effectiveness: while punishment consistently increases contributions, its impact on payoffs (i.e., efficiency) ranges from dramatically enhancing welfare (up to 43% improvement) to severely undermining it (up to 44% reduction) depending on the cooperative context. To characterize these patterns, we developed models that outperformed human forecasters (laypeople and domain experts) in predicting punishment outcomes in new experiments. Communication emerged as the most predictive feature, followed by contribution framing (opt-out vs. opt-in), contribution type (variable vs. all-or-nothing), game length (number of rounds), peer outcome visibility (whether participants can see others' earnings), and the availability of a reward mechanism. Interestingly, however, most of these features interact to influence punishment effectiveness rather than operating independently. For example, the extent to which longer games increase the effectiveness of punishment depends on whether groups can communicate. Together, our results refocus the debate over punishment from whether or not it "works" to the specific conditions under which it does and does not work. More broadly, our study demonstrates how integrative experiments can be combined with machine learning to uncover generalizable patterns, potentially involving interactions between multiple features, and help generate novel explanations in complex social phenomena.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Choose Your Agent: Tradeoffs in Adopting AI Advisors, Coaches, and Delegates in Multi-Party Negotiation
cs.GT 2026-02 conditional novelty 6.0 of 10

Users prefer an AI Advisor but gain most with a Delegate, because human editing filters out the AI's best proposals.

Reference graph

Works this paper leans on

9 extracted references · 6 canonical work pages · cited by 1 Pith paper

[1]

P., Paton, N., Watts, D

Almaatouq, A., Becker, J., Houghton, J. P., Paton, N., Watts, D. J., & Whiting, M. E. (2021). Empirica: a virtual lab for high-throughput macro-level experiments. Behavior Research Methods , 53 (5), 2158–2171

work page 2021
[2]

Breiman, L. (2001). Random Forests. Machine Learning , 45 (1), 5–32

work page 2001
[3]

(2016, August 13)

Chen, T., & Guestrin, C. (2016, August 13). XGBoost: A Scalable Tree Boosting System. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . KDD ’16: The 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco California USA. https://doi.org/10.1145/2939672.2939785

arXiv 2016
[4]

DellaVigna, S., Pope, D., & Vivalt, E. (2019). Predict science to improve science. Science , 366 (6464), 428–429

work page 2019
[5]

Head, T., Kumar, M., Nahrstaedt, H., Louppe, G., & Shcherbatyi, I. (2021). scikit-optimize/scikit-optimize . Zenodo. https://doi.org/10.5281/ZENODO.5565057

doi:10.5281/zenodo.5565057 2021
[6]

M., & Lee, S.-I

Lundberg, S. M., & Lee, S.-I. (2017). A unified approach to interpreting model predictions. Proceedings of the 31st International Conference on Neural Information Processing Systems , 4768–4777

work page 2017
[7]

Brucher, M., Perrot, M., & Duchesnay, E. (2011). Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research: JMLR , abs/1201.0490 (85), 2825–2830

arXiv 2011
[8]

E., Haberland, M., Reddy, T., Cournapeau, D., & Van Mulbregt P

Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., & Van Mulbregt P. (2020). SciPy 1.0: fundamental algorithms for scientific computing in Python. Nature Methods , 17 (3), 261–272

work page 2020

Show all 9 references

[9]

Zou, H., & Hastie, T. (2005). Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society. Series B, Statistical Methodology , 67 (2), 301–320. 36

2005

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

P., Paton, N., Watts, D

Almaatouq, A., Becker, J., Houghton, J. P., Paton, N., Watts, D. J., & Whiting, M. E. (2021). Empirica: a virtual lab for high-throughput macro-level experiments. Behavior Research Methods , 53 (5), 2158–2171

work page 2021

[2] [2]

Breiman, L. (2001). Random Forests. Machine Learning , 45 (1), 5–32

work page 2001

[3] [3]

(2016, August 13)

Chen, T., & Guestrin, C. (2016, August 13). XGBoost: A Scalable Tree Boosting System. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . KDD ’16: The 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco California USA. https://doi.org/10.1145/2939672.2939785

arXiv 2016

[4] [4]

DellaVigna, S., Pope, D., & Vivalt, E. (2019). Predict science to improve science. Science , 366 (6464), 428–429

work page 2019

[5] [5]

Head, T., Kumar, M., Nahrstaedt, H., Louppe, G., & Shcherbatyi, I. (2021). scikit-optimize/scikit-optimize . Zenodo. https://doi.org/10.5281/ZENODO.5565057

doi:10.5281/zenodo.5565057 2021

[6] [6]

M., & Lee, S.-I

Lundberg, S. M., & Lee, S.-I. (2017). A unified approach to interpreting model predictions. Proceedings of the 31st International Conference on Neural Information Processing Systems , 4768–4777

work page 2017

[7] [7]

Brucher, M., Perrot, M., & Duchesnay, E. (2011). Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research: JMLR , abs/1201.0490 (85), 2825–2830

arXiv 2011

[8] [8]

E., Haberland, M., Reddy, T., Cournapeau, D., & Van Mulbregt P

Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., & Van Mulbregt P. (2020). SciPy 1.0: fundamental algorithms for scientific computing in Python. Nature Methods , 17 (3), 261–272

work page 2020