REVIEW 4 major objections 5 minor 59 references
Impact of Outreach on Physics Student Development: Qualitative Results from a National Survey
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Students who facilitate physics outreach report growth in identity, belonging, resilience, skills, and mindset in a national survey.
desk verdict National qualitative dataset and network-coding method are genuinely useful; the causal verbs in the abstract outrun what retrospective self-reports can support. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central analytical object is a co-occurrence network of 49 qualitative codes applied to 621 open-ended responses from 239 students. Edges are drawn only between codes whose co-occurrence is significant at p < 0.001, code size reflects eigenvector centrality, and a Girvan-Newman clustering algorithm with Q = 0.476 partitions the codes into six macro-themes. This network does the argument's work by turning individual quotes into a map of relationships, allowing the paper to claim that the impacts are interrelated and that audience dialog connects the other clusters.
What would settle it
A longitudinal study that measures physics identity, belonging, mindset, and skill confidence before and after students begin facilitating outreach, compared with matched students who did not participate, would settle whether outreach causes the reported growth; if the two groups' trajectories match, the central claim is falsified.
Extended reading notes
Core claim
The paper's central claim is that facilitating informal physics outreach programs produces developmental effects on the students doing the facilitating: their self-reports describe enhanced resilience and belonging, growth in physics identity, development of career skills such as communication and design, and a growth mindset that carries from outreach back into coursework. The finding rests on a national rather than single-institution sample, which the authors argue extends prior results by showing the impacts are shared across institution types. The network analysis identifies six core themes of student experience, community participation, resilience, transformation, audience dialog, disciplinary development, and disciplinary connectedness, and shows that the first four are tightly interconnected. The authors single out audience dialog as the central feature: interactions with an authentic public link the other clusters and supply the recognition and competence feedback that drive identity and belonging.
Load-bearing premise
The load-bearing premise is that students' retrospective, self-reported accounts of how outreach changed them are accurate evidence of real change caused by outreach, because the survey has no baseline measurement and no comparison group.
Editorial extensions
If this is right
- Outreach facilitation benefits appear across a national sample of institutions, so they are not limited to departments with mature outreach programs.
- The four interconnected clusters show that resilience, belonging, identity, and transformation are linked through audience interaction, making audience dialog a central feature of the developmental impact.
- Students report developing career-relevant skills, including communication, design, teamwork, and networking, through outreach, including skills the formal physics curriculum typically underemphasizes.
- The directional growth-mindset code that runs from outreach to discipline suggests outreach helps students carry a more malleable view of ability back into their coursework.
Reading between the lines
- The authors leave implicit that departments could use outreach facilitation as a low-cost retention lever, since belonging and identity are known to predict persistence in physics majors.
- The network structure suggests audience interaction, not outreach activity in general, is the likely active ingredient; a testable prediction is that programs with more direct audience engagement produce larger developmental gains.
- The appearance of only the outreach-to-discipline growth-mindset code supports a causal story that outreach flexes students' beliefs about ability, a hypothesis that could be tested with pre-post mindset measures.
- A useful extension would be to code the open-ended responses separately by institution type and program maturity to see whether the six-cluster structure is stable or sensitive to local context.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript analyzes open-ended responses from 239 students who reported facilitating physics outreach, drawn from a national Society of Physics Students survey. Using a deductive coding scheme with 49 codes, five coders, and inter-rater reliability κ ≥ 0.90, the authors construct a code co-occurrence network and apply Girvan-Newman clustering (Q = 0.476) to identify six clusters: community participation, resilience, transformation, audience dialog, disciplinary development, and disciplinary connectedness. The paper's central claim, stated in the abstract and conclusion, is that facilitating informal physics outreach enhanced students' resilience and belonging, grew their physics identity, developed career skills, and cultivated a growth mindset, with the first four clusters highly interconnected.
Significance. The national scope of the dataset is a genuine strength: it extends prior single-institution work and provides a diverse sample across institution types, genders, and racial/ethnic groups. The coding process is rigorous and well documented, and the network-clustering approach is appropriate for exploring interrelationships among qualitative codes. If the findings are reframed as students' retrospective self-reported perceptions rather than measured developmental effects, the study offers useful evidence that outreach facilitation is associated with positive narratives about identity, belonging, and skills. The grouping of codes into interpretable clusters and the supporting verbatim quotes are valuable resources for physics education research.
major comments (4)
- [Abstract and II. Methods, third prompt] The abstract and conclusion state that outreach 'enhanced' resilience and belonging, 'grew' physics identity, and 'cultivated' a growth mindset. The only direct evidence is retrospective answers to open-ended prompts, one of which presupposes change: 'How has your perception of yourself as a physicist changed through your participation in physics/astronomy outreach?' With no baseline measure, no comparison group of students who did not facilitate outreach, and no longitudinal data, these causal verbs outrun the design. Students who already felt confident and connected may have been more likely to join outreach and to respond to the survey. I recommend rewriting all causal claims as students' self-reported perceived changes and adding an explicit statement that the design cannot establish actual developmental effects.
- [II. Methods (Mindset codes) and IV. Discussion] The codebook divides growth and fixed mindset into directional subcodes ('outreach to discipline,' 'discipline to outreach,' 'ambiguous') before analysis. The Discussion then treats the appearance of only 'growth mindset - outreach to discipline' in Fig. 1 as evidence that outreach caused a growth mindset, asserting that this 'signals a likely explanation for the direction of the effect from our previous study.' This is circular: the direction is assigned by coders using the codebook, not discovered by the network analysis. The absence of the other directional codes may reflect coding decisions or low base rates rather than a true one-way effect. Please present this result as a coding-derived hypothesis that requires independent validation, not as directional evidence.
- [V. Limitations] The Limitations section contains only two sentences, acknowledging self-report and potential selection bias, but the manuscript's headline claims do not consistently observe this limitation. It also omits the demand characteristic introduced by the change-presupposing prompt and the absence of any comparison group. This mismatch is load-bearing because the abstract and conclusion make causal claims. The limitations should be expanded to explain what the design can and cannot support, and the conclusions and abstract should be calibrated to those limits.
- [III.B. Connections and IV. Discussion] The text repeatedly refers to 'statistically significant links' between clusters and uses language such as 'led to,' 'reinforced,' and 'as a result' when describing connections among codes. Because the network is built from co-occurrence correlations among codes within the same set of 239 responses, these edges show that codes co-occur in text, not that one experience causally produced another. I recommend relabeling these as co-occurrence relationships and systematically avoiding causal wording when interpreting edges.
minor comments (5)
- [III.A, audience dialog cluster] In the quote about explaining to children, 'I need to understand it well to that it makes sense' appears to be a typo for 'so that it makes sense.'
- [III.A, community participation cluster] The quote 'She is is now pursuing a physics major in her first year of undergrad' contains a duplicated 'is.'
- [Table I] The demographic table reports that 47% of respondents have a physical or mental disability, which is unexpectedly high relative to national undergraduate populations; please clarify the category definition or discuss this figure.
- [IV. Discussion] The sentence 'Figures 1 clearly indicates...' should be 'Figure 1 clearly indicates...' for grammatical agreement.
- [Fig. 1] The network map should be legible at publication size; the current version may be difficult for readers to parse, especially the code labels.
Circularity Check
No significant circularity: the network themes are data-driven, and the causal wording stems from acknowledged self-report design rather than a definitional or fitted-input reduction.
full rationale
The paper's central claims are qualitative descriptions of respondents' self-reported experiences, not quantities derived from equations or from the same fitted parameters they are claimed to predict. The deductive codebook includes categories from external frameworks (Hazari et al.'s physics identity model, DSMRI, situated learning, transformative learning, Dweck's mindset) and from the authors' prior work, but the key result—the six-cluster structure and the interconnections among clusters—was produced by an unconstrained co-occurrence network with Girvan-Newman clustering, with cluster names assigned after the clustering, so the themes are not fixed by the codebook alone. The directional mindset code ('growth mindset - outreach to discipline') is defined in terms of students' own attributions, and the Discussion's directional inference is a reading of which of the three pre-defined directionality codes reached the network; the alternative codes existed and could have appeared, so the observation is not equivalent to the code definition by construction. The abstract's causal verbs ('enhanced,' 'grew,' 'cultivated') outrun the study design, and the Limitations section acknowledges the self-report and selection-bias concerns; but that is a validity limitation, not a circularity. No load-bearing step reduces to a self-citation or to a fitted parameter renamed as a prediction.
Assumptions & free parameters
free parameters (2)
- Significance threshold p < 0.001 =
0.001
- Modularity threshold Q > 0.30 =
0.30 (achieved Q=0.476)
assumptions (7)
- domain assumption Hazari et al.'s physics identity framework (interest, performance/competence, recognition, belonging) is a valid lens for student development.
- domain assumption Dynamic Systems Model of Role Identity (DSMRI) is a valid lens for context-specific self-perceptions.
- domain assumption Situated learning theory (legitimate peripheral participation) describes student integration into the physics community.
- domain assumption Transformative learning theory describes changes in frames of reference from outreach.
- domain assumption Students' retrospective self-reports accurately reflect their development.
- domain assumption Co-occurrence of codes within a response represents conceptual relationship.
- domain assumption SPS network respondents are representative enough of U.S. physics undergraduates for generalization.
Cite this review
Pith. "Pith review of Impact of Outreach on Physics Student Development: Qualitative Results from a National Survey." pith.science (2026). https://pith.science/paper/CWULRQ2P
@misc{pith2026250622700,
author = {Pith},
title = {Pith review of: Impact of Outreach on Physics Student Development: Qualitative Results from a National Survey},
year = {2026},
howpublished = {\url{https://pith.science/paper/CWULRQ2P}},
note = {Machine review of arXiv:2506.22700}
}
read the original abstract
The role of student experiences in physics beyond the classroom which support their development has been the subject of exciting research in recent years. Results, typically from small studies at single institutions, have illustrated that facilitating informal physics experiences for non-scientists can enhance student disciplinary identity, learning, sense of belonging, and more. However, it is essential to examine whether these impacts are the sole provenance of institutions with well-developed outreach programs or if they may be shared by institutions anywhere. This work reports on the analysis and findings of responses to three open-ended questions presented to students who indicated they had engaged in facilitating outreach programs as part of a national survey distributed through the Society of Physics Students network in spring 2023. Employing a network analysis with Girvan-Newman clusters revealed six core themes of student experiences: community participation, resilience, transformation, audience dialog, disciplinary development, and disciplinary connectedness. The first four of these clusters were observed to be highly interconnected, providing evidence that the impacts and experiences within them are interrelated with other clusters, particularly interactions with the audience, which is a central feature of informal physics programs. In particular, student experiences highlighted that facilitating informal physics programs enhanced their resilience and belonging, grew their physics identity, provided opportunities to develop essential career skills, and cultivated a growth mindset.
Figures
Reference graph
Works this paper leans on
-
[1]
P. Bell, B. Lewenstein, A. W. Shouse, M. A. Feder, et al. , Learning science in informal environments: Peo- ple, places, and pursuits , Vol. 140 (National Academies Press Washington, DC, 2009)
work page 2009
-
[2]
N. R. Council, D. of Behavioral, S. Sciences, B. on Sci- ence Education, and C. on Successful Out-of School STEM Learning, Identifying and supporting produc- tive STEM programs in out-of-school settings (National Academies Press, 2015)
work page 2015
-
[3]
A. A. Bergerson, B. K. Hotchkins, and C. Furse, Jour- nal of College Student Retention: Research, Theory & Practice 16, 165 (2014)
work page 2014
-
[4]
A. K. Hansen, S. Siyahhan, J. Kisiel, J. Marshall, J. M. Nation, E. Price, and M. Shin, Connected Science Learn- ing 7, 2 (2025)
work page 2025
- [5]
-
[6]
C. Fracchiolla, B. Prefontaine, and K. Hinko, Phys. Rev. Phys. Educ. Res. 16, 020115 (2020)
work page 2020
- [7]
-
[8]
C. Garrett, T. L. Erukhimova, J. D. Perry, and J. P. Don- aldson, in Proceedings of the Physics Education Research Conference (PERC) (2023) pp. 108–113
work page 2023
Show all 59 references
-
[9]
Hinko and N
K. Hinko and N. D. Finkelstein, AIP Conference Pro- ceedings 1513, 178 (2012)
2012
-
[10]
Perry, J
J. Perry, J. P. Donaldson, and T. Erukhimova, in Pro- ceedings of PER Conf. 2021, virtual conference , Vol. 10 (2021)
2021
-
[11]
Randolph, J
J. Randolph, J. Perry, J. P. Donaldson, C. Rethman, and T. Erukhimova, Physical Review Physics Education Research 18, 020123 (2022)
2022
-
[12]
Rethman, J
C. Rethman, J. Perry, J. Donaldson, D. Choi, and T. Erukhimova, accepted to Phys. Rev. Phys. Educ. Res. (2021)
2021
-
[13]
R. C. Tillinghast, D. C. Appel, C. Winsor, and M. Man- souri, in 2020 IEEE Integrated STEM Education Confer- ence (ISEC) (IEEE, 2020) pp. 1–20
2020
-
[14]
J. D. Perry, T. L. Erukhimova, C. Garrett, T. Sauncy, J. P. Donaldson, S. White, J. Tyler, and R. L. Ivie, in Physics Education Research Conference 2024, PER Con- ference (Boston, MA, 2024) pp. 318–323
2024
-
[15]
Heron and L
P. Heron and L. McNeil, American Physical Society and the American Association of Physics Teachers, Joint Task Force on Undergraduate Physics Programs (2016)
2016
-
[16]
McNeil and P
L. McNeil and P. Heron, Physics Today 70, 38 (2017)
2017
-
[17]
Mulvey and J
P. Mulvey and J. Pold, American Institute of Physics (2020)
2020
-
[18]
Pold and P
J. Pold and P. Mulvey, AIP Statistical Research Center (2015)
2015
-
[19]
Sarkar, T
M. Sarkar, T. Overton, C. Thompson, and G. Rayner, International Journal of Innovation in Science and Math- ematics Education 24 (2016)
2016
-
[20]
A. I. Leshner, Outreach training needed (2007)
2007
-
[21]
Nature 542, 391 (2017)
2017
-
[22]
K. M. Foster, K. B. Bergin, A. F. McKenna, D. L. Mil- lard, L. C. Perez, J. T. Prival, D. Y. Rainey, H. M. Se- vian, E. A. VanderPutten, and J. E. Hamos, Science329, 906 (2010)
2010
-
[23]
https://www.aip.org/statistics
-
[24]
N. D. Finkelstein and L. Mayhew, in AIP Conference Proceedings, Vol. 1064 (American Institute of Physics,
-
[25]
Hinko, J
K. Hinko, J. Seneca, and N. Finkelstein, in Physics Edu- cation Research Conference 2014, PER Conference (Min- neapolis, MN, 2014) pp. 115–118
2014
-
[26]
J. D. Perry, T. Sauncy, S. White, R. L. Ivie, J. Tyler, and T. Erukhimova, Impact of outreach on physics student development: Quantitative results from a national survey (2025), arXiv:2505.09874 [physics.ed-ph]
2025 arXiv
-
[27]
K. L. Lewis, J. G. Stout, S. J. Pollock, N. D. Finkelstein, and T. A. Ito, Phys. Rev. Phys. Educ. Res. 12, 020110 (2016)
2016
-
[28]
D. S. Yeager, C. Romero, D. Paunesku, C. S. Hulle- man, B. Schneider, C. Hinojosa, H. Y. Lee, J. O’Brien, K. Flint, A. Roberts, et al. , Journal of educational psy- chology 108, 374 (2016)
2016
-
[29]
James, E
M. James, E. Bertschinger, B. Beckford, T. Dobbins, S. Fries-Britt, S. Gates, J. Isler, M. Ong, A. Richard- son, Q. Williams, et al. , American Institute of Physics (2020)
2020
-
[30]
C. L. Williams, Q. Hirschi, C. S. Hulleman, and J. Roksa, International Journal of Community Well-Being , 1 (2021)
2021
-
[31]
Dobbins, Bulletin of the American Physical Society (2020)
T. Dobbins, Bulletin of the American Physical Society (2020)
2020
-
[32]
G. M. Walton and G. L. Cohen, Journal of personality and social psychology 92, 82 (2007)
2007
-
[33]
Hazari, D
Z. Hazari, D. Chari, G. Potvin, and E. Brewe, Journal of Research in Science Teaching 57, 1583 (2020)
2020
-
[34]
Kaplan and J
A. Kaplan and J. K. Garner, Developmental Psychology 53, 2036 (2017)
2017
-
[35]
just another physics major
which described learning through legitimate periph- eral participation, transformative learning theory [36, 37] describing changes to one’s frames of reference, and a list of skills identified as essential for 21 st century physics careers [15]. In total, 49 codes were used in...
-
[36]
Lave and E
J. Lave and E. Wenger, Situated learning: Legitimate pe- ripheral participation (Cambridge university press, 1991)
1991
-
[37]
Mezirow, An overview on transformative learning, in Contemporary theories of learning: learning theorists in their own words , edited by K
J. Mezirow, An overview on transformative learning, in Contemporary theories of learning: learning theorists in their own words , edited by K. Illeris (Routledge, Abing- don, 2009) pp. 90–105
2009
-
[38]
Kegan, in Contemporary theories of learning (Rout- ledge, 2018) pp
R. Kegan, in Contemporary theories of learning (Rout- ledge, 2018) pp. 29–45
2018
-
[39]
L. S. Vygotsky and M. Cole, Mind in society: Develop- ment of higher psychological processes (Harvard univer- sity press, 1978)
1978
-
[40]
C. S. Dweck, Institute for Advanced Study (2014)
2014
-
[41]
Segev, Semantic network analysis in social sciences (Routledge London, 2022)
E. Segev, Semantic network analysis in social sciences (Routledge London, 2022)
2022
-
[42]
J. P. Donaldson and A. Allen-Handy, Educational Psy- chology Review 32, 545 (2019)
2019
-
[43]
S. P. Borgatti, M. G. Everett, and L. C. Freeman, Har- vard, MA: analytic technologies 6 (2002)
2002
-
[44]
Kadushin, Understanding social networks : theories, concepts, and findings
C. Kadushin, Understanding social networks : theories, concepts, and findings. Charles Kadushin (Oxford Uni- versity Press, 2012)
2012
-
[45]
Girvan and M
M. Girvan and M. E. Newman, Proceedings of the na- tional academy of sciences 99, 7821 (2002). 12
2002
-
[46]
Zahiri, J
M. Zahiri, J. Mohammadzadeh, and S. Harifi, Journal of Ambient Intelligence and Humanized Computing , 1 (2023)
2023
-
[47]
C. A. Hecht, A. G. Latham, R. E. Buskirk, D. R. Hansen, and D. S. Yeager, CBE—Life Sciences Education21, ar82 (2022)
2022
-
[48]
G. M. Walton, M. C. Murphy, C. Logel, D. S. Yeager, J. P. Goyer, S. T. Brady, K. T. Emerson, D. Paunesku, O. Fotuhi, A. Blodorn, et al. , Science 380, 499 (2023)
2023
-
[49]
D. S. Yeager, C. J. Bryan, J. J. Gross, J. S. Murray, D. Krettek Cobb, P. HF Santos, H. Gravelding, M. John- son, and J. P. Jamieson, Nature 607, 512 (2022)
2022
-
[50]
J. L. Burnette, J. Billingsley, G. C. Banks, L. E. Knouse, C. L. Hoyt, J. M. Pollack, and S. Simon, Psychological bulletin 149, 174 (2023)
2023
-
[51]
D. S. Yeager, G. M. Walton, S. T. Brady, E. N. Akcinar, D. Paunesku, L. Keane, D. Kamentz, G. Ritter, A. L. Duckworth, R. Urstein, et al. , Proceedings of the Na- tional Academy of Sciences 113, E3341 (2016)
2016
-
[52]
E. W. Close, J. Conn, and H. G. Close, Phys. Rev. Phys. Educ. Res. 12, 010109 (2016)
2016
-
[53]
Perez, J
T. Perez, J. Cromley, and A. Kaplan, Journal of Educa- tional Psychology , 315 (2014)
2014
-
[54]
Hazari, G
Z. Hazari, G. Sonnert, P. M. Sadler, and M.-C. Shanahan, Journal of Research in Science Teaching 47, 978 (2010), https://onlinelibrary.wiley.com/doi/pdf/10.1002/tea.20363
2010 doi
-
[55]
Chakraverty, International Journal of Doctoral Stud- ies 15, 329 (2020)
D. Chakraverty, International Journal of Doctoral Stud- ies 15, 329 (2020)
2020
-
[56]
B. A. Beesley, N. G. Vece, and Z. Johnson-Ulrich, Psi Chi Journal of Psychological Research 29 (2024)
2024
-
[57]
R. Ivie, S. White, and R. Y. Chu, Physical Review Physics Education Research 12, 020109 (2016)
2016
-
[58]
K. A. Hinko, P. Madigan, E. Miller, and N. D. Finkel- stein, Phys. Rev. Phys. Educ. Res. 12, 010111 (2016)
2016
-
[59]
Felten and L
P. Felten and L. M. Lambert, Relationship-rich educa- tion: How human connections drive success in college (Jhu Press, 2020)
2020
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.