REVIEW 4 major objections 6 minor 1 cited by
Impact of Outreach on Physics Student Development: Quantitative Results from a National Survey
T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read The first national survey of undergraduate physics students who facilitate outreach finds that participation is strongly associated with confidence explaining physics, career skill development, growth mindset, and sense of belonging.
desk verdict First national survey of outreach participation in physics, but the headline mindset claim leans on a factor whose construct validity isn't established. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The survey instrument and its empirically derived seven-factor structure carry the argument. Likert items were designed to measure constructs of physics identity (interest and motivation, competence beliefs, recognition, and belonging), persistence, confidence, self-efficacy, and mindset; principal component analysis with Varimax rotation and Kaiser normalization reduced the items to seven factors, including a two-item mindset factor and a belonging factor. Logistic and linear regressions then related these factors to participation in outreach, respondent characteristics, and institutional characteristics. What matters is that the factors are treated as measurements of the theoretical constructs, so the associations in the results inherit their meaning from that measurement assumption.
What would settle it
A concrete test would be a longitudinal or randomized study that measures growth mindset and belonging before and after students begin facilitating outreach, with a matched control group of nonparticipants; if mindset and belonging do not change following participation, the reported associations would be explained by selection rather than by outreach.
Extended reading notes
Core claim
The central claim is that student facilitation of informal physics outreach is meaningfully tied to undergraduate student development, not just to audience impact. Students who participated in outreach were more likely to agree they can explain physics and astronomy ideas to people inside and outside the discipline, and large majorities reported using and feeling prepared for career skills such as communication, networking, creativity, design, and leadership during outreach. Multiple regression models controlling for demographics and institutional type showed that participation in outreach was positively associated with growth mindset and with sense of belonging at $p<0.01$. The paper interprets this as evidence that outreach may contribute to persistence-relevant motivational beliefs, while acknowledging that the reverse selection explanation remains possible.
Load-bearing premise
The argument depends on the assumption that the seven factors produced by factor analysis, especially the two-item growth-mindset factor, actually measure the psychological constructs they are named after; if the measurement is off, the reported links between outreach and mindset or belonging are not interpretable.
Editorial extensions
If this is right
- Students who participate in outreach report higher confidence in explaining physics to both lay and expert audiences, which suggests outreach serves as a communication-skills venue beyond the classroom.
- The results provide departments a justification for developing or expanding outreach programs as co-curricular career-skill development, since students reported practicing communication, teamwork, leadership, and creativity at high rates.
- The link between outreach participation and growth mindset, if causal, gives departments a concrete, low-capital activity that may support resilience and persistence alongside coursework.
- No significant demographic differences in participation were found by gender, race, or international status, so outreach appears accessible across groups, though first-year students participate less.
Reading between the lines
- Inference: If the outreach-to-mindset direction is causal, outreach may function like other social-belonging interventions, and the asymmetry noted in the paper could be tested with a randomized encouragement design where students are assigned to facilitate outreach and mindset is measured before and after.
- Inference: The two-item growth-mindset factor likely conflates mindset with feedback-seeking attitude, so future work should re-validate the factor with the full item set before building interventions on this specific result.
- Inference: The null demographic differences, if replicated, suggest outreach could be an equitable engagement channel; a longitudinal comparison of outreach participants and nonparticipants matched on motivation would test whether selection explains the belonging result.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports the development, distribution, and quantitative analysis of a national survey of undergraduate physics students recruited through the Society of Physics Students, focusing on the relationship between student participation in informal physics outreach programs and constructs including physics identity, sense of belonging, growth mindset, self-efficacy, and career skill development. The authors describe a pilot phase, a national sample of 704 respondents, a factor-analytic reduction of survey items to seven factors, and a set of logistic and linear regression models. They report that outreach participation is associated with greater confidence in explaining physics, perceived career-skill development, and positive associations with growth mindset and sense of belonging. The paper also presents descriptive tables on motivations, career skill usage, and demographics, and discusses limitations including self-report and cross-sectional design.
Significance. If the central associations hold, this would be a valuable contribution: it is the largest national dataset to date on undergraduate physics outreach facilitators, and the study addresses an underrepresented topic in physics education research. The authors took useful steps such as grounding items in prior pilot work, distributing through a national network, and controlling for demographic and institutional variables in regressions. The paper also honestly acknowledges several limitations. However, the significance of the results depends on the construct validity of the factors and on the magnitude of the reported associations, both of which are currently under-supported: no factor loadings or reliability statistics are reported, no effect sizes or confidence intervals accompany any regression result, and the factor solution was revised after inspecting data-driven loadings. These issues must be resolved before the paper's claims can be interpreted.
major comments (4)
- [Section II.C and Table II] The text states that the Growth Mindset construct 'resulted in two factors with two components loading together and one not loading with any other components,' but Table II lists three items under 'Mindset,' including the feedback item. If the feedback item did not load, the reported factor is not the hypothesized growth-mindset construct, and the regression result in Section III.A linking outreach participation to this factor may reflect openness to feedback rather than growth mindset. Please report rotated loadings for all items, Cronbach's alpha or omega for each factor, and a sensitivity analysis of the Section III.A result with the feedback item removed or reassigned.
- [Section III.A and Fig. 1] The abstract's 'strong association' claim is not supported by the reported statistics. All regression results are given only as p-values; no standardized coefficients, odds ratios, or confidence intervals are reported. A p<0.01 result at this sample size could correspond to a trivially small effect, so 'strongly positive' (Section III.A) and 'strong association' (Abstract) are unjustified. Please report effect sizes with confidence intervals for the outreach coefficients in all models, and adjust Figure 1 to encode effect size or at least report it in the text.
- [Section II.D] The manuscript reports over 400 hypothesis tests at a nominal alpha of 0.01 without any multiple-comparison correction. At alpha=0.01, one expects roughly 4 false positives by chance, and the large number of tests makes individual p<0.01 results weak evidence. Please apply a correction (e.g., Benjamini-Hochberg) or clearly label the analysis as exploratory; in either case, the 'rigorous alpha' statement in Section II.D should be revised.
- [Title and Section IV] The title and Discussion use causal language ('Impact'), but the design is a cross-sectional, self-reported survey. The authors acknowledge in Section III.A that regression does not provide causal direction, yet they later 'cautiously favor' a directional interpretation based on the p<0.01 vs p<0.05 asymmetry in Figure 1. This asymmetry is not a valid basis for causal inference. Please temper the causal framing throughout (including the title) or provide a clear causal identification strategy.
minor comments (6)
- [Section III.A] The text includes 'LQBTQIA+,' which appears to be a typo for 'LGBTQIA+'.
- [Table VI] The blank entries in Table VI are unexplained in the caption; a note should indicate which questions were not administered.
- [Table III] The Mann-Whitney U test is reported only as p<0.01; adding the test statistic or an effect size would aid interpretation.
- [Section II.C] The method is described as 'confirmatory factor analysis,' but the implementation is principal component analysis with Varimax rotation; the terminology should be aligned with the actual procedure.
- [References] Reference [61] is a bare URL; a full citation with access date is needed.
- [Figure 1] Figure 1 is described as showing 'statistically significant relationships' but omits the reverse-direction result mentioned in the text (p<0.05); clarify whether the figure includes all tested models.
Circularity Check
No significant circularity: the survey regressions are empirical associations, not predictions derived from their own inputs; the factor-analysis concerns are measurement-validity issues, not circular reductions.
full rationale
This paper makes no first-principles prediction; its headline results are descriptive regressions and self-report comparisons computed from a national survey. Participation in outreach and the outcome factors are measured from separate survey items, and no outcome variable is constructed from the participation variable by definition. The factor analysis in Section II.C is exploratory in that the final seven-factor solution was selected after inspecting loadings, and the Mindset factor in Table II appears to retain a feedback item that the text says did not load with the other two mindset items; that is a genuine construct-validity and potential overfitting concern, but it is not a circular reduction, because no fitted parameter or factor loading is reused as the target result. The authors' prior single-institution pilot studies are cited for context and survey development, but the central associations are computed from the new national dataset rather than imported from those citations. No equation is equivalent to its input by construction, and no fitted value is renamed as a prediction, so the paper does not meet the standard for circularity.
Assumptions & free parameters
free parameters (2)
- Factor solution (7 factors with item assignments) =
Table II: 7 factors, e.g., Mindset includes 3 items, Competence includes 4 items
- p-value significance threshold =
0.01
assumptions (4)
- domain assumption Self-reported Likert responses validly measure the intended constructs (motivation, belonging, competence, confidence, recognition, self-efficacy, mindset).
- domain assumption The SPS membership list is an acceptable sampling frame for US undergraduate physics students.
- domain assumption The regression models control for the confounders needed to interpret the participation-construct associations.
- domain assumption Ordinal Likert items can be treated as continuous after normalization to 0-1 and subjected to PCA with Varimax rotation.
Cite this review
Pith. "Pith review of Impact of Outreach on Physics Student Development: Quantitative Results from a National Survey." pith.science (2026). https://pith.science/paper/LFKTLD7E
@misc{pith2026250509874,
author = {Pith},
title = {Pith review of: Impact of Outreach on Physics Student Development: Quantitative Results from a National Survey},
year = {2026},
howpublished = {\url{https://pith.science/paper/LFKTLD7E}},
note = {Machine review of arXiv:2505.09874}
}
read the original abstract
This work reports on results from the first national study of the impact of student facilitation of informal physics outreach programs on their physics identity, sense of belonging, and essential career skill development. Drawing on results from studies at a single institution with a well-developed physics outreach program, a national survey was developed and distributed through the Society of Physics Students' network to more than five thousand individuals. Responses to closed-form questions on the survey were analyzed descriptively and using multiple regression analysis to evaluate the relationship between student participation in informal physics outreach programs and the constructs of interest. Results show a strong association between students engaging in outreach with their confidence in communicating their physics knowledge, the development of key career skills, and direct connections with growth mindset and sense of belonging. These results may be useful to physics and other STEM departments around the United States who are seeking to elevate and broaden the learning experience of students.
Figures
Forward citations
Cited by 1 Pith paper
-
Impact of Outreach on Physics Student Development: Qualitative Results from a National Survey
A national survey of physics students who facilitate outreach shows self-reported growth in belonging, identity, resilience, communication skills, and a growth mindset.
Reference graph
Works this paper leans on
-
[1]
K. L. Lewis, J. G. Stout, N. D. Finkelstein, S. J. Pollock, A. Miyake, G. L. Cohen, and T. A. Ito, Psychology of Women Quarterly 41, 420 (2017)
work page 2017
-
[2]
D. S. Yeager, G. M. Walton, S. T. Brady, E. N. Akcinar, D. Paunesku, L. Keane, D. Kamentz, G. Ritter, A. L. Duckworth, R. Urstein, et al. , Proceedings of the Na- tional Academy of Sciences 113, E3341 (2016)
2016
-
[3]
James, E
M. James, E. Bertschinger, B. Beckford, T. Dobbins, S. Fries-Britt, S. Gates, J. Isler, M. Ong, A. Richard- son, Q. Williams, et al. , American Institute of Physics (2020)
2020
-
[4]
J. L. Burnette, J. Billingsley, G. C. Banks, L. E. Knouse, C. L. Hoyt, J. M. Pollack, and S. Simon, Psychological bulletin 149, 174 (2023)
2023
-
[5]
C. L. Williams, Q. Hirschi, C. S. Hulleman, and J. Roksa, International Journal of Community Well-Being , 1 (2021)
2021
-
[6]
Dobbins, Bulletin of the American Physical Society (2020)
T. Dobbins, Bulletin of the American Physical Society (2020)
2020
-
[7]
K. L. Lewis, J. G. Stout, S. J. Pollock, N. D. Finkelstein, and T. A. Ito, Phys. Rev. Phys. Educ. Res. 12, 020110 (2016)
2016
-
[8]
G. M. Walton and G. L. Cohen, Journal of personality and social psychology 92, 82 (2007)
2007
Show all 75 references
-
[9]
C. S. Dweck, Psychological review 124, 689 (2017)
2017
-
[10]
D. S. Yeager, C. J. Bryan, J. J. Gross, J. S. Murray, D. Krettek Cobb, P. HF Santos, H. Gravelding, M. John- son, and J. P. Jamieson, Nature 607, 512 (2022)
2022
-
[11]
C. A. Hecht, A. G. Latham, R. E. Buskirk, D. R. Hansen, and D. S. Yeager, CBE—Life Sciences Education21, ar82 (2022)
2022
-
[12]
Hazari, D
Z. Hazari, D. Chari, G. Potvin, and E. Brewe, Journal of Research in Science Teaching 57, 1583 (2020)
2020
-
[13]
Hazari, G
Z. Hazari, G. Sonnert, P. M. Sadler, and M.-C. Shanahan, Journal of Research in Science Teaching 47, 978 (2010), https://onlinelibrary.wiley.com/doi/pdf/10.1002/tea.20363
2010 doi
-
[14]
Heron and L
P. Heron and L. McNeil, American Physical Society and the American Association of Physics Teachers, Joint Task Force on Undergraduate Physics Programs (2016)
2016
-
[15]
Smith, C
M. Smith, C. Fracchiolla, S. Fleming, A. Dominguez, A. Lau, S. Greco, D. Lincoln, E. Katifori, W. Ratcliff, M. Longobardi, et al. , arXiv preprint arXiv:2112.10623 (2021)
2021 arXiv
-
[16]
Rethman, J
C. Rethman, J. Perry, J. P. Donaldson, D. Choi, and T. Erukhimova, Physical Review Physics Education Re- search 17, 020110 (2021)
2021
-
[17]
Randolph, J
J. Randolph, J. Perry, J. P. Donaldson, C. Rethman, and T. Erukhimova, Physical Review Physics Education Research 18, 020123 (2022)
2022
-
[18]
Hinko and N
K. Hinko and N. D. Finkelstein, AIP Conference Pro- ceedings 1513, 178 (2012)
2012
-
[19]
Graur, American Journal of Physics 86, 725 (2018)
O. Graur, American Journal of Physics 86, 725 (2018)
2018
-
[20]
A. I. Leshner, Outreach training needed (2007)
2007
-
[21]
Hinko and N
K. Hinko and N. D. Finkelstein, in AIP Conference Pro- ceedings, Vol. 1513 (American Institute of Physics, 2013) pp. 178–181
2013
-
[22]
Fracchiolla, S
C. Fracchiolla, S. Hyater-Adams, N. Finkelstein, and K. Hinko, in Physics Education Research Conference 2016, PER Conference (Sacramento, CA, 2016) pp. 124– 127
2016
-
[23]
N. R. Council, D. of Behavioral, S. Sciences, B. on Sci- ence Education, and C. on Successful Out-of School STEM Learning, Identifying and supporting produc- tive STEM programs in out-of-school settings (National Academies Press, 2015)
2015
-
[24]
Editorial, Nature 542, 391 (2017)
2017
-
[25]
N. R. Council et al. , Learning science in informal envi- ronments: People, places, and pursuits (2009)
2009
-
[26]
Rainie, C
L. Rainie, C. Funk, M. Anderson, and D. Page, Pew Re- search Center (2015)
2015
-
[27]
J. C. Besley, A. Dudo, and S. Yuan, Public Understand- ing of Science 27, 708 (2018)
2018
-
[28]
S. E. Brownell, J. V. Price, and L. Steinman, Journal of undergraduate neuroscience education 12, E6 (2013)
2013
-
[29]
Prefontaine, C
B. Prefontaine, C. Fracchiolla, M. Vasquez, and K. Hinko, in Proceedings of the 2018 Physics Education Research Conference, Washington, DC (AIP, 2018)
2018
-
[30]
Perry, J
J. Perry, J. P. Donaldson, and T. Erukhimova, in Pro- ceedings of PER Conf. 2021, virtual conference , Vol. 10 (2021)
2021
-
[31]
N. D. Finkelstein and L. Mayhew, in AIP Conference Proceedings, Vol. 1064 (American Institute of Physics,
-
[32]
K. A. Hinko, P. Madigan, E. Miller, and N. D. Finkel- stein, Phys. Rev. Phys. Educ. Res. 12, 010111 (2016)
2016
-
[33]
Fracchiolla, B
C. Fracchiolla, B. Prefontaine, and K. Hinko, Phys. Rev. Phys. Educ. Res. 16, 020115 (2020)
2020
-
[34]
Hazari, P
Z. Hazari, P. M. Sadler, and G. Sonnert, Journal of Col- lege Science Teaching 42, 82 (2013)
2013
-
[35]
Sauncy, K
T. Sauncy, K. Redmond, and R. Czujko, AIP Conference Proceedings 1697, 120014 (2015), https://aip.scitation.org/doi/pdf/10.1063/1.4937719
2015 doi
-
[36]
Bunshaft, J
A. Bunshaft, J. Curtis-Fink, A. Gerstein, D. Boyington, T. Edwards, and C. Jacobson, STEM connector’s STEM Innovation Task Force, available at: www. STEMconnec- tor. org (2015)
2015
-
[37]
G. Quan, B. Gutmann, J. Corbo, B. Pollard, and C. Turpen, in Physics Education Research Conference 2019 (2019) pp. 482–487
2019
-
[38]
Perez, J
T. Perez, J. Cromley, and A. Kaplan, Journal of Educa- tional Psychology , 315 (2014)
2014
-
[39]
Sawtelle, E
V. Sawtelle, E. Brewe, and L. H. Kramer, Jour- nal of Research in Science Teaching 49, 1096 (2012), https://onlinelibrary.wiley.com/doi/pdf/10.1002/tea.21050
2012 doi
-
[40]
J. P. Zwolak, R. Dou, E. A. Williams, and E. Brewe, Phys. Rev. Phys. Educ. Res. 13, 010113 (2017)
2017
-
[41]
Thiry, S
H. Thiry, S. L. Laursen, and A.-B. Hunter, The Journal of Higher Education 82, 357 (2011)
2011
-
[42]
Z. Y. Kalender, E. Marshman, C. D. Schunn, T. J. Nokes- Malach, and C. Singh, Phys. Rev. Phys. Educ. Res. 15, 020119 (2019)
2019
-
[43]
Hyater-Adams, C
S. Hyater-Adams, C. Fracchiolla, N. Finkelstein, and K. Hinko, Phys. Rev. Phys. Educ. Res.14, 010132 (2018)
2018
-
[44]
Hyater-Adams, C
S. Hyater-Adams, C. Fracchiolla, T. Williams, N. Finkel- stein, and K. Hinko, Physical Review Physics Education Research 15, 020115 (2019)
2019
-
[45]
E. W. Close, J. Conn, and H. G. Close, Phys. Rev. Phys. Educ. Res. 12, 010109 (2016)
2016
-
[46]
P. W. Irving and E. C. Sayre, Phys. Rev. ST Phys. Educ. Res. 11, 020120 (2015)
2015
-
[47]
D. F. Feldon, J. Peugh, B. E. Timmerman, M. A. Maher, M. Hurst, D. Strickland, J. A. Gilmore, and C. Stiegelmeyer, Science 333, 1037 (2011)
2011
-
[48]
Drane, M
D. Drane, M. Micari, and G. Light, Educational Research and Evaluation 20, 210 (2014)
2014
-
[49]
Otero, S
V. Otero, S. Pollock, and N. Finkelstein, American Jour- nal of Physics 78, 1218 (2010)
2010
-
[50]
Garrett, T
C. Garrett, T. L. Erukhimova, J. D. Perry, and J. P. Don- aldson, in Proceedings of the Physics Education Research Conference (PERC) (2023) pp. 108–113
2023
-
[51]
J. D. Perry, T. L. Erukhimova, C. Garrett, T. Sauncy, J. P. Donaldson, S. White, J. Tyler, and R. L. Ivie, in Physics Education Research Conference 2024, PER Con- ference (Boston, MA, 2024) pp. 318–323
2024
-
[52]
Lave and E
J. Lave and E. Wenger, Situated learning: Legitimate pe- ripheral participation (Cambridge university press, 1991)
1991
-
[53]
Mezirow, An overview on transformative learning, in Contemporary theories of learning: learning theorists in their own words , edited by K
J. Mezirow, An overview on transformative learning, in Contemporary theories of learning: learning theorists in their own words , edited by K. Illeris (Routledge, Abing- don, 2009) pp. 90–105
2009
-
[54]
Kegan, in Contemporary theories of learning (Rout- ledge, 2018) pp
R. Kegan, in Contemporary theories of learning (Rout- ledge, 2018) pp. 29–45
2018
-
[55]
Kaplan and J
A. Kaplan and J. K. Garner, Developmental Psychology 53, 2036 (2017)
2017
-
[56]
T. A. Brown, M. T. Moore, et al., Handbook of structural equation modeling 361, 379 (2012)
2012
-
[57]
Brown, JALT Testing & Evaluation SIG Newsletter 13 (2009)
J. Brown, JALT Testing & Evaluation SIG Newsletter 13 (2009)
2009
-
[58]
Weber, Studien zum Physik- und Chemielernen (2022)
J. Weber, Studien zum Physik- und Chemielernen (2022)
2022
-
[59]
Vazquez-Abad, L
J. Vazquez-Abad, L. Winer, and J.-R. Derome, McGill Journal of Education/Revue des sciences de l’´ education de McGill 32 (1997)
1997
-
[60]
Riegle-Crumb, C
C. Riegle-Crumb, C. Moore, and A. Ramos-Wada, Sci- ence Education 95, 458 (2011)
2011
-
[61]
https://www.aps.org/learning- center/statistics/diversity
-
[62]
Cheng, G
H. Cheng, G. Potvin, R. Khatri, L. H. Kramer, R. M. Lock, and Z. Hazari, in Physics Education Research Con- ference 2018 (2018)
2018
-
[63]
C. Good, A. Rattan, and C. S. Dweck, Journal of per- sonality and social psychology 102, 700 (2012)
2012
-
[64]
Master, S
A. Master, S. Cheryan, and A. N. Meltzoff, Journal of educational psychology 108, 424 (2016)
2016
-
[65]
Bedford, Research papers in education 32, 424 (2017)
S. Bedford, Research papers in education 32, 424 (2017)
2017
-
[66]
J. L. Burnette, C. L. Hoyt, V. M. Russell, B. Lawson, C. S. Dweck, and E. Finkel, Social Psychological and Per- sonality Science 11, 107 (2020)
2020
-
[67]
Marshman, Z
E. Marshman, Z. Y. Kalender, C. Schunn, T. Nokes- Malach, and C. Singh, Canadian Journal of Physics 96, 391 (2018)
2018
-
[68]
Z. Y. Kalender, E. Marshman, C. D. Schunn, T. J. Nokes- Malach, and C. Singh, Physical Review Physics Educa- tion Research 18, 010116 (2022)
2022
-
[69]
S. Kim, J. Yun, B. Schneider, M. Broda, C. Klager, and I.-C. Chen, Journal of Economic Behavior & Organiza- tion 195, 219 (2022)
2022
-
[70]
E. M. Marshman, Z. Y. Kalender, T. Nokes-Malach, C. Schunn, and C. Singh, Physical review physics edu- cation research 14, 020123 (2018)
2018
-
[71]
J. M. Nissen and J. T. Shemwell, Physical Review Physics Education Research 12, 020105 (2016)
2016
-
[72]
Z. Y. Kalender, E. Marshman, T. J. Nokes-Malach, C. Schunn, and C. Singh, in Proceedings of the 2017 Physics Education Research Conference (2017) pp. 204– 207
2017
-
[73]
Wilson, R
D. Wilson, R. Bates, E. P. Scott, S. M. Painter, and J. Shaffer, Journal of Women and Minorities in Science and Engineering 21 (2015)
2015
-
[74]
R. S. Barthelemy, M. Swirtz, S. Garmon, E. H. Simmons, K. Reeves, M. L. Falk, W. Deconinck, E. A. Long, and T. J. Atherton, Physical Review Physics Education Re- search 18, 010124 (2022)
2022
-
[75]
L. D. Conlin, E. Kuo, and N. R. Hallinen, Physical Re- view Physics Education Research 15, 020104 (2019)
2019
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.