pith. sign in

arxiv: 2508.08417 · v1 · pith:JDHB3VDJnew · submitted 2025-08-11 · ⚛️ physics.ed-ph

Evaluating recognition and recall formats of social network surveys in physics education research

Meagan Sundstrom , Justin Gambrell , Adrienne L. Traxler , Eric Brewe This is my paper

Pith reviewed 2026-05-21 22:49 UTC · model grok-4.3

classification ⚛️ physics.ed-ph
keywords social network analysissurvey formatrecognition and recallphysics education researchstudent interactionsname order effectsintroductory physicsnetwork data collection
0
0 comments X

The pith

Recognition surveys lead students to name more peer interactions than recall surveys in physics courses, with little name order bias.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tests two common ways of collecting social network data from students in introductory physics classes. Recognition surveys give students a full roster of names to pick from, while recall surveys ask them to type names from memory without help. Direct comparison shows students report more interactions when the roster is provided. Separate checks across many courses find that the order of names on the roster rarely causes students to favor earlier names over later ones. The results guide choices about which survey format to use when mapping student interactions.

Core claim

When the same prompt is given in both formats to 65 students, the recognition version produces reports of more peer interactions than the recall version. In a separate set of 54 recognition surveys drawn from 27 courses, the large majority show no statistically significant name order effects, indicating that roster position does not systematically skew selections.

What carries the argument

Direct comparison of recognition (roster selection) versus recall (open-response) formats for eliciting lists of peer interactions.

If this is right

  • Network studies using recognition surveys will typically measure denser student interaction graphs than those using recall surveys.
  • Roster order can be treated as a minor concern when designing recognition surveys for most introductory physics classes.
  • Researchers can choose the recognition format when the goal is to capture a fuller set of reported interactions.
  • Comparisons of network properties across studies must account for the survey format used to avoid attributing density differences to pedagogy instead of method.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Standardizing on one format across projects would make it easier to compare student network structures between different physics courses or institutions.
  • The extra interactions captured by recognition may include weaker or less salient ties that recall surveys miss, which could affect interpretations of social support or collaboration.
  • Testing the same formats in laboratory or upper-division courses would show whether the pattern holds outside large introductory classes.

Load-bearing premise

The samples of 65 students and 54 surveys from 27 courses are representative of typical introductory physics settings and differences are due to survey format rather than course structure or demographics.

What would settle it

A replication study using larger samples across multiple institutions that finds no reliable difference in the number of reported interactions between the two formats would falsify the central claim about recognition yielding more reports.

Figures

Figures reproduced from arXiv: 2508.08417 by Adrienne L. Traxler, Eric Brewe, Justin Gambrell, Meagan Sundstrom.

Figure 2
Figure 2. Figure 2: FIG. 2: Interaction networks from each survey only using [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: FIG. 3: Correlation coefficients measuring the relationship between students’ position on the alphabetized course roster and the [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
read the original abstract

An increasing number of studies in physics education research use social network analysis to quantify interactions among students. These studies typically gather data through online surveys using one of two different survey formats: recognition, where students select peers' names from a provided course roster, and recall, where students type their peers' names from memory as an open response. These survey formats, however, may be subject to two possible systematic errors. First, students may report more peers' names on a recognition survey than a recall survey because the course roster facilitates their memory of their interactions, whereas they may only remember a subset of their interactions on the recall format. Second, recognition surveys may be subject to name order effects, where students are more likely to select peers' names that appear early on in the roster than those that appear later on (e.g., due to survey fatigue). Here we report the results of two methodological studies of these possible errors in the context of introductory physics courses: one directly comparing 65 student responses to recognition and recall versions of the same network survey prompt, and the other measuring name order effects on 54 recognition surveys from 27 different courses. We find that students may report more peer interactions on a recognition survey than a recall survey and that most recognition surveys are not subject to significant name order effects. These results help to inform survey design for future network studies in physics education research.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript evaluates two potential systematic biases in social network surveys used in physics education research: (1) whether recognition formats (selecting names from a provided roster) elicit more reported peer interactions than recall formats (open-ended name entry from memory), and (2) whether recognition surveys exhibit name-order effects. It reports results from a direct within-student comparison involving 65 responses and a separate analysis of 54 recognition surveys drawn from 27 introductory physics courses, concluding that students may report more interactions on recognition surveys while most recognition surveys show no significant name-order effects.

Significance. If the empirical comparisons prove robust after fuller methodological documentation, the work supplies practical, field-specific guidance for survey design in an expanding area of PER that relies on social network data. It offers concrete evidence on reporting volume differences and order-effect prevalence that can help standardize data collection and improve the reliability of interaction metrics across studies.

major comments (2)
  1. [Direct comparison study] The direct-comparison component (65 student responses) does not describe randomization of format order, whether the same students completed both versions, cohort matching, or any statistical controls for course-level variables such as roster length, class size, or demographics. Without these details the observed difference cannot be confidently attributed to survey format rather than contextual factors.
  2. [Name order effects analysis] The name-order analysis (54 surveys from 27 courses) provides no information on the statistical tests, effect-size thresholds, p-value criteria, or adjustments for multiple comparisons and roster length used to classify a survey as showing 'significant' order effects. This information is required to evaluate the strength of the claim that 'most' surveys are unaffected.
minor comments (1)
  1. [Abstract] The abstract appropriately uses cautious language ('may report more', 'most ... are not subject'), but the manuscript should explicitly discuss the representativeness of the 65-student and 27-course samples for typical introductory physics settings.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their detailed review and constructive comments on our manuscript. We have carefully considered each point and made revisions to the manuscript to provide the requested methodological details. Below we respond point by point to the major comments.

read point-by-point responses

  1. Referee: The direct-comparison component (65 student responses) does not describe randomization of format order, whether the same students completed both versions, cohort matching, or any statistical controls for course-level variables such as roster length, class size, or demographics. Without these details the observed difference cannot be confidently attributed to survey format rather than contextual factors.

    Authors: We agree that these details are important for interpreting the results. The study was designed as a within-subjects comparison in which the same students completed both survey formats during the same class period, with the order of presentation randomized to mitigate order effects. We have added a description of this procedure, including the randomization method, to the Methods section of the revised manuscript. Regarding controls, the comparison was conducted within a single course to control for class size and roster length, and we have now included basic demographic information and noted the absence of additional statistical controls as a limitation. We believe these additions allow for a more confident attribution to survey format. revision: yes

  2. Referee: The name-order analysis (54 surveys from 27 courses) provides no information on the statistical tests, effect-size thresholds, p-value criteria, or adjustments for multiple comparisons and roster length used to classify a survey as showing 'significant' order effects. This information is required to evaluate the strength of the claim that 'most' surveys are unaffected.

    Authors: We appreciate this feedback and have revised the manuscript to include a detailed description of our statistical methods. Specifically, we used a chi-squared goodness-of-fit test for each survey to compare the observed distribution of selections across name positions against a uniform distribution, with significance set at p < 0.05. We applied a Bonferroni correction for multiple comparisons across the 54 surveys and report effect sizes using Cramer's V. Roster length was addressed by dividing the roster into quartiles for analysis. These details have been added to the Methods and Results sections, strengthening the basis for our conclusion that most surveys did not exhibit significant name-order effects. revision: yes

Circularity Check

0 steps flagged

No circularity: purely empirical comparison of survey formats

full rationale

The paper presents an observational methodological study that directly compares student responses to recognition versus recall network surveys and measures name-order effects on recognition surveys. All central claims rest on collected data (65 student responses and 54 surveys from 27 courses) rather than any derivation, fitted parameter renamed as a prediction, or self-referential definition. No equations, ansatzes, or uniqueness theorems appear; the attribution of differences to survey format follows from the study design itself and does not reduce to prior quantities defined by the authors. This is a standard self-contained empirical evaluation with no load-bearing self-citation chains or constructed equivalences.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The claims rest on student survey responses collected in introductory physics courses and on standard statistical comparisons; no new free parameters, invented entities, or ad-hoc axioms are introduced beyond routine assumptions of survey methodology.

axioms (1)
  • domain assumption Standard statistical assumptions hold for comparing paired survey responses and for testing name-order effects.
    Conclusions about more reported interactions and lack of significant order effects rely on these background statistical procedures.

pith-pipeline@v0.9.0 · 5785 in / 1296 out tokens · 55791 ms · 2026-05-21T22:49:59.770830+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

29 extracted references · 29 canonical work pages

  1. [1]

    To what extent does the number of peers students report interacting with vary between a recognition survey and a recall survey? (Study 1)

    work page

  2. [2]

    How do peer interaction networks measured with recog- nition and recall surveys compare to one another? (Study 1)

    work page

  3. [3]

    Please list any stu- dents in your physics class that you had a meaningful inter- action with about the course material this week

    To what extent is the number of times a student is se- lected by their peers on a recognition survey correlated with their position on the provided roster? (Study 2) II. METHODS A. Study 1 We collected data from an introductory, calculus-based me- chanics course at a large, private, Ph.D.-granting institution in the northeastern United States with an R1 d...

    work page 2025

  4. [4]

    It is possible that our findings are mostly representative of high performers, who are more likely to fill out educational surveys than low per- formers [22]

    First, there was a low percentage of enrolled students with matched responses, limiting generalizability. It is possible that our findings are mostly representative of high performers, who are more likely to fill out educational surveys than low per- formers [22]. Future research should conduct similar compar- isons of recognition and recall surveys in ot...

    work page

  5. [5]

    The roles of engagement: Network analysis in physics education research

    Eric Brewe. The roles of engagement: Network analysis in physics education research. In Getting Started in PER , vol- ume 2. American Association of Physics Teachers, College Park, MD, July 2018. 5

    work page 2018

  6. [6]

    Remy Dou and Justyna P. Zwolak. Practitioner’s guide to so- cial network analysis: Examining physics anxiety in an active- learning setting. Physical Review Physics Education Research , 15(2):020105, 2019

    work page 2019

  7. [7]

    Networks and learning: A view from physics

    Adrienne Traxler. Networks and learning: A view from physics. Journal of Learning Analytics, 9(1):111–119, 2022

    work page 2022

  8. [8]

    Talking and learning physics: Predicting future grades from network measures and Force Concept Inventory pretest scores

    Jesper Bruun and Eric Brewe. Talking and learning physics: Predicting future grades from network measures and Force Concept Inventory pretest scores. Physical Review Special Topics-Physics Education Research, 9(2):020109, 2013

    work page 2013

  9. [9]

    Williams, Justyna P

    Eric A. Williams, Justyna P. Zwolak, Remy Dou, and Eric Brewe. Linking engagement and performance: The social net- work analysis perspective. Physical Review Physics Education Research, 15(2):020150, 2019

    work page 2019

  10. [10]

    How academic achievement spreads: The role of distinct social networks in academic performance diffusion

    Sofia Dokuka, Diliara Valeeva, and Maria Yudkevich. How academic achievement spreads: The role of distinct social networks in academic performance diffusion. PLoS One , 15(7):e0236737, 2020

    work page 2020

  11. [11]

    Heim, and N

    Meagan Sundstrom, Andy Schang, Ashley B. Heim, and N. G. Holmes. Understanding interaction network formation across instructional contexts in remote physics courses. Physical Re- view Physics Education Research, 18:020141, 2022

    work page 2022

  12. [12]

    Charac- terizing active learning environments in physics using network analysis and classroom observations

    Kelley Commeford, Eric Brewe, and Adrienne Traxler. Charac- terizing active learning environments in physics using network analysis and classroom observations. Physical Review Physics Education Research, 17(2):020136, 2021

    work page 2021

  13. [13]

    Bahrick, Phyllis O

    Harry P. Bahrick, Phyllis O. Bahrick, and Roy P. Wittlinger. Fifty years of memory for names and faces: A cross-sectional approach. Journal of Experimental Psychology: General , 104(1):54, 1975

    work page 1975

  14. [14]

    Social access and the clustering of personal connections

    Muriel Hammer. Social access and the clustering of personal connections. Social Networks, 2(4):305–325, 1980

    work page 1980

  15. [15]

    Explorations into the meaning of social net- work interview data

    Muriel Hammer. Explorations into the meaning of social net- work interview data. Social Networks, 6(4):341–371, 1984

    work page 1984

  16. [16]

    Experiments in the measurement of the size of social networks

    Seymour Sudman. Experiments in the measurement of the size of social networks. Social Networks, 7(2):127–151, 1985

    work page 1985

  17. [17]

    Experiments in measuring neighbor and rel- ative social networks

    Seymour Sudman. Experiments in measuring neighbor and rel- ative social networks. Social Networks, 10(1):93–108, 1988

    work page 1988

  18. [18]

    V . Hlebec. Recall versus recognition: Comparison of the two alternative procedures for collecting social network data. InDe- velopments in Statistics and Methodology: Proceedings of the International Conference on Methodology and Statistics , pages 121–128, Bled, Slovenia, 1993

    work page 1993

  19. [19]

    Devon D. Brewer. Patterns in the recall of persons in a student community. Social Networks, 15(4):335–359, 1993

    work page 1993

  20. [20]

    Franz J. Neyer. Free recall or recognition in collecting ego- centered networks: The role of survey techniques. Journal of Social and Personal Relationships, 14(3):305–316, 1997

    work page 1997

  21. [21]

    Devon D. Brewer. Forgetting in the recall-based elicitation of personal and social networks. Social Networks , 22(1):29–43, 2000

    work page 2000

  22. [22]

    Franc ¸ois Poulin and Thomas J. Dishion. Methodological issues in the use of peer sociometric nominations with middle school youth. Social Development, 17(4):908–921, 2008

    work page 2008

  23. [23]

    Peter E. L. Marks, Antonius H. N. Cillessen, and Ben Bab- cock. Relations between alphabetized name order and nomina- tion counts in peer nomination measures. Social Development, 25(4):866–874, 2016

    work page 2016

  24. [24]

    Nolin, and James A

    Shuyin Liu, David A. Nolin, and James A. Kitts. Name order effects in measuring adolescent social networks using rosters. Social Networks, 76:68–78, 2024

    work page 2024

  25. [25]

    Peer Instruction: A User’s Manual

    Eric Mazur. Peer Instruction: A User’s Manual. Prentice Hall, 1997

    work page 1997

  26. [26]

    Nissen, Manher Jariwala, Eleanor W

    Jayson M. Nissen, Manher Jariwala, Eleanor W. Close, and Ben Van Dusen. Participation and performance on paper- and computer-based low-stakes assessments. International Journal of STEM Education, 5:1–17, 2018

    work page 2018

  27. [27]

    McDermott

    Lillian C. McDermott. Tutorials in Introductory Physics. Pren- tice Hall, 2002

    work page 2002

  28. [28]

    Investigative science learning environment–a science process approach to learning physics

    Eugenia Etkina and Alan Van Heuvelen. Investigative science learning environment–a science process approach to learning physics. Research-based Reform of University Physics, 1:1–48, 2007

    work page 2007

  29. [29]

    Beichner, Jeffery M

    Robert J. Beichner, Jeffery M. Saul, David S. Abbott, Jeanne J. Morse, Duane Deardorff, Rhett J. Allain, Scott W. Bonham, Melissa H. Dancy, and John S. Risley. The student-centered ac- tivities for large enrollment undergraduate programs (SCALE- UP) project. Research-based Reform of University Physics , 1:2–39, 2007

    work page 2007