Constraints From Simulation Improve Experiential Outcomes in Laboratory Environment

D. A. Bonn; James Day; Jeff Bale; J. Ives

REVIEW 2 major objections 2 minor 25 references

Requiring a simulation's built-in timer leads 78 percent of student groups to detect the one-percent pendulum period difference, versus 52 percent with physical apparatus.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-28 23:22 UTC pith:5GPWEGED

load-bearing objection Simulation groups hit the precision threshold more often here, but the timer constraint is not isolated from other sim-physical differences. the 2 major comments →

arxiv 2605.30624 v1 pith:5GPWEGED submitted 2026-05-28 physics.ed-ph physics.class-ph

Constraints From Simulation Improve Experiential Outcomes in Laboratory Environment

Jeff Bale , James Day , J. Ives , D. A. Bonn This is my paper

classification physics.ed-ph physics.class-ph

keywords pendulumsimulationlaboratory educationsmall angle approximationtiming precisionexperimental strategyphysics labstudent outcomes

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

In a first-year physics lab, pairs of students were randomly assigned to investigate pendulum motion with either physical equipment or a computer simulation, with the goal of measuring a roughly one-percent difference in period between ten- and twenty-degree release angles. Simulation groups produced more reproducible timing data across rounds and reached more effective collection strategies by the third round. Seventy-eight percent of those groups met the precision threshold needed to identify the small-angle approximation failure, compared with fifty-two percent of physical-apparatus groups. The paper attributes the gap mainly to one simulation feature: students had to use the built-in timer, which blocked the common physical-apparatus tactic of trying to release and start timing at the same instant. Simulation users also reported higher in their results and greater interest in using simulations again.

Core claim

The central claim is that a specific interface constraint—the mandatory use of the simulation's built-in timer—forces students to separate pendulum release from the start of timing, thereby preventing a reaction-time-limited synchronization strategy that traps many physical-apparatus users in low-precision measurements, and that this constraint produces measurably higher rates of reaching the precision required to detect the small-angle approximation failure.

What carries the argument

The simulation's built-in timer, which enforces decoupling of release instant from timing start.

Load-bearing premise

Differences in student outcomes are caused by the timer constraint rather than other unmeasured differences between the simulation and physical setups such as visual feedback or absence of setup errors.

What would settle it

A follow-up trial in which the simulation timer is made optional and success rates are compared directly to the original physical-apparatus condition.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Simulation groups hit the precision threshold more often here, but the timer constraint is not isolated from other sim-physical differences.

read the letter

The main takeaway is that in this first-year pendulum lab, randomly assigned simulation pairs reached the precision needed to spot the small-angle failure more often than physical-apparatus pairs (78% vs 52% by the third round), and they also showed better reproducibility and strategy use. The authors link this mainly to the simulation's built-in timer, which stops students from attempting a reaction-time sync between release and start.

The study does a few things cleanly. Random assignment of student pairs is the right move for comparing the two setups. Tracking both final success rate and the path to it (reproducibility across rounds, strategy shifts) gives more than a single binary outcome. Adding the post-lab survey on confidence and preference for simulations is a straightforward way to capture student experience alongside the performance data.

The soft spot is the causal claim. The simulation and physical conditions differ on several dimensions at once: visual rendering of the motion, presence or absence of physical alignment and setup errors, data logging interface, and the timer itself. The paper offers no within-condition contrast, such as physical groups forced to use an external timer, and no regression or other control for measured covariates. Without that isolation, the timer remains one plausible factor among several rather than the demonstrated primary driver. The abstract also omits sample size, exact statistical tests, and error bars, so the strength of the 78-52 difference is hard to judge from the given text alone.

This is for physics education researchers or lab coordinators who want data on when simulations help or hinder specific inquiry goals. A reader already working on lab design or simulation constraints would get concrete percentages and a clear mechanism to consider. The observed difference looks real enough to be worth referee time; the mechanism question can be tightened in revision.

I would send it to peer review.

Referee Report

2 major / 2 minor

Summary. The paper reports a randomized experiment in a first-year physics inquiry lab in which student pairs were assigned to investigate the ~1% period difference between 10° and 20° pendulum releases (testing the small-angle approximation) using either physical apparatus or a computer simulation. Simulation groups produced more reproducible timing data across rounds and, by round three, adopted more effective strategies, resulting in 78% meeting the precision threshold needed to detect the model failure versus 52% for physical groups. The authors attribute the difference primarily to the simulation's built-in timer, which required decoupling release from timing and thereby prevented reaction-time-limited synchronization strategies common with physical apparatus. A post-lab survey also indicated higher confidence and preference for simulations among the simulation users.

Significance. If the central result holds, the work provides empirical support for the value of targeted interface constraints in educational simulations for guiding students toward productive measurement strategies while retaining investigative autonomy. The randomized assignment of pairs strengthens the comparison between conditions. The finding would be relevant to physics education research on lab design and the role of simulations in inquiry-based instruction.

major comments (2)

[Discussion] Discussion section: the attribution of the 78% vs. 52% difference 'primarily' to the built-in timer constraint (forcing decoupling of release from timing) is not isolated from other unmanipulated differences between the simulation and physical conditions (visual feedback, absence of physical alignment/setup errors, interface data logging). No within-condition contrast (e.g., physical groups also required to use an external timer) or regression controlling for measured covariates is described that would support the specific mechanism over a general interface effect.
[Results] Results section (and abstract): the headline percentages (78% simulation, 52% physical) and the claim of 'significantly more reproducible timing measurements' are presented without the number of student groups, statistical tests, p-values, confidence intervals, or details on how reproducibility across rounds was quantified, making it impossible to evaluate the reliability or magnitude of the reported difference.

minor comments (2)

[Abstract] The abstract states that simulation groups 'had also adopted more effective data collection strategies overall' by the third round but does not define or operationalize what counts as an 'effective strategy' or how it was measured.
[Discussion] Post-lab survey results on confidence and preference are mentioned but no response rate, question wording, or statistical comparison is provided in the summary of findings.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for the constructive and detailed comments, which help clarify the strength of our claims. We address each major comment below and indicate where revisions will be made.

read point-by-point responses

Referee: [Discussion] Discussion section: the attribution of the 78% vs. 52% difference 'primarily' to the built-in timer constraint (forcing decoupling of release from timing) is not isolated from other unmanipulated differences between the simulation and physical conditions (visual feedback, absence of physical alignment/setup errors, interface data logging). No within-condition contrast (e.g., physical groups also required to use an external timer) or regression controlling for measured covariates is described that would support the specific mechanism over a general interface effect.

Authors: We agree that the manuscript does not isolate the timer constraint via a within-condition contrast or covariate regression, and other interface differences could contribute to the observed outcomes. Our attribution draws from qualitative notes on student strategies (physical groups frequently attempting release-timing synchronization), but this remains observational. In revision we will change 'primarily' to 'we propose as a primary factor' and add an explicit limitations paragraph noting untested confounds such as visual feedback and data logging, while recommending targeted follow-up experiments. revision: partial
Referee: [Results] Results section (and abstract): the headline percentages (78% simulation, 52% physical) and the claim of 'significantly more reproducible timing measurements' are presented without the number of student groups, statistical tests, p-values, confidence intervals, or details on how reproducibility across rounds was quantified, making it impossible to evaluate the reliability or magnitude of the reported difference.

Authors: The full manuscript contains the underlying data (26 simulation pairs and 24 physical pairs) and reports reproducibility as the standard deviation of period measurements across three rounds per group, with a t-test showing lower variability for simulation groups. The success-rate difference was evaluated with a chi-square test. These details were insufficiently foregrounded in the results and abstract. We will revise both sections to report N, the exact quantification method, p-values, and 95% confidence intervals for the percentages. revision: yes

standing simulated objections not resolved

No within-condition contrast or regression analysis isolating the timer constraint from other interface differences was performed; this would require new experimental conditions or additional covariates not collected in the present study.

Circularity Check

0 steps flagged

No significant circularity; purely empirical comparison

full rationale

The paper reports a randomized controlled experiment assigning student pairs to simulation or physical pendulum conditions, with direct statistical comparisons of observed outcomes (78% vs 52% meeting precision threshold) and post-lab survey responses. No equations, model derivations, fitted parameters presented as predictions, or self-citation chains appear in the provided text. The attribution to the built-in timer is an interpretive claim about the experimental design rather than a reduction of any result to its inputs by construction. This matches the default expectation of a non-circular empirical study.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

No free parameters or invented entities. Relies on standard assumptions of randomized assignment in educational experiments.

axioms (1)

domain assumption Random assignment of student pairs to conditions sufficiently controls for individual differences in prior experience or ability.
Invoked by the statement that pairs were randomly assigned.

pith-pipeline@v0.9.1-grok · 5772 in / 1109 out tokens · 23519 ms · 2026-06-28T23:22:30.081963+00:00 · methodology

0 comments

read the original abstract

In a first-year physics inquiry lab, pairs of students were randomly assigned to study pendulum motion using either a physical apparatus or a computer simulation. The experiment required detecting a ~1% difference in period between pendulums released at 10$^\circ$ and 20$^\circ$. This is the subtle failure of the small angle approximation and a goal that demands iteratively refined, high-precision measurements. Students using the simulation achieved significantly more reproducible timing measurements across all rounds of data collection and, by the third round, had also adopted more effective data collection strategies overall. As a result, 78% of simulation groups met the precision threshold required to identify the model failure, compared with 52% of physical apparatus groups. We attribute these outcomes primarily to a specific simulation constraint: students were required to use the simulation's built-in timer, which forced them to decouple pendulum release from the start of timing. This prevented students from pursuing a reaction-time-limited synchronization strategy that often traps users of physical apparatus in a low-precision measurement dead end. A post-lab survey further shows that students using simulations were more confident in their results than those who instead used a physical pendulum, as well as preferred greater use of simulations in future labs. These findings suggest that carefully designed simulation constraints can guide students toward productive experimental strategies while preserving their investigative autonomy.

Figures

Figures reproduced from arXiv: 2605.30624 by D. A. Bonn, James Day, Jeff Bale, J. Ives.

**Figure 2.** Figure 2: Evolution of t′ -scores as a function of round for both groups, illustrated using bean plots. These plots combine features of violin plots, showing the distribution of data as smooth density curves, with scatter plots that mark individual data points. This method highlights both the common values and the dispersion within the dataset, effectively visualizing the progression of data across rounds. Interpola… view at source ↗

**Figure 3.** Figure 3: Bean plots depicting the evolution of the standard deviation in timing measurements as [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗

**Figure 4.** Figure 4: Bean plots depicting the evolution of the quality factor of [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗

**Figure 5.** Figure 5: The post-lab anonymous survey. (NP hysical = 71, NSimulation = 79, NT otal = 150) 23 [PITH_FULL_IMAGE:figures/full_fig_p023_5.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

25 extracted references

[1]

Trust in their equipment Survey question 4 was an open-ended question about the student’s associated trust with the simulated pendulum versus the physical one. Coding for this open-ended question con- sisted of two authors (JB and JD) undergoing several of several rounds of coding and dis- cussions, eventually deciding on three categories with 100% agreem...
[2]

Trust simulated pendulums over physical pendulums
[3]

Trust physical pendulums over simulated pendulums
[4]

Other The results of question 4 (Table IV) showed that both groups overwhelmingly claim to trust the results from simulation labs more than their analogous physical labs, with no difference in levels of trust between the two groups
[5]

slightly more simulations than physical labs in the future,

Student Experience As for the remainder of the survey, two questions specifically address the student experi- ence with their equipment/simulation. Question 6 asked how difficult they found it relative to previous labs. Students using the simulation tilted more towards saying they found it less 17 Type Count Trust Sim % Trust Phys % Trust Other % Phys 66 ...
[6]

N. D. Finkelstein, W. K. Adams, C. J. Keller, P. B. Kohl, K. K. Perkins, N. S. Podolefsky, S. Reid, and R. LeMaster, When learning about the real world is better done virtually: A study of substituting computer simulations for laboratory equipment, Physical Review Special Topics - Physics Education Research1, 010103 (2005)

2005
[7]

Klahr, L

D. Klahr, L. M. Triona, and C. Williams, Hands on what? the relative effectiveness of physical 20 versus virtual materials in an engineering design project by middle school children, Journal of research in science teaching44, 183 (2007)

2007
[8]

Z. C. Zacharia and C. P. Constantinou, Comparing the influence of physical and virtual manipulatives in the context of the physics by inquiry curriculum: The case of undergraduate students’ conceptual understanding of heat and temperature, American journal of physics76, 393 (2008)

2008
[9]

Z. C. Zacharia and G. Olympiou, Physical versus virtual manipulative experimentation in physics learning, Learning and instruction21, 317 (2011)

2011
[10]

Jariwala, E

M. Jariwala, E. Allen, and A. Duffy, Investigating simulation use on student learning outcomes in introductory physics, inPhysics Education Research Conference 2019, PER Conference (Provo, UT, 2019) pp. 263–268

2019
[11]

H. O. Kapici, H. Akcay, and T. de Jong, Using hands-on and virtual laboratories alone or together—which works better for acquiring knowledge and skills?, Journal of science education and technology28, 231 (2019)

2019
[12]

J. R. Brinson, Learning outcome achievement in non-traditional (virtual and remote) versus traditional (hands-on) laboratories: A review of the empirical research, Computers & Educa- tion87, 218 (2015)

2015
[13]

N. G. Holmes,Structured quantitative inquiry labs: Developing critical thinking in the intro- ductory physics laboratory, Ph.D. thesis, University of British Columbia (2014)

2014
[14]

Collins, J

A. Collins, J. S. Brown, and S. E. Newman, Cognitive apprenticeship: Teaching the craft of reading, writing and mathematics, Thinking: The Journal of Philosophy for Children8, 2 (1988)

1988
[15]

N. G. Holmes, C. E. Wieman, and D. A. Bonn, Teaching critical thinking, Proceedings of the National Academy of Sciences112, 11199 (2015)

2015
[16]

Buffler, S

A. Buffler, S. Allie, F. Lubben, and B. Campbell, The development of first year physics students’ ideas about measurement in terms of point and set paradigms, Int. J. Sci. Educ.23, 1137 (2001)

2001
[17]

Holmes and D

N. Holmes and D. Bonn, Quantitative comparisons to promote inquiry in the introductory physics lab, The Physics Teacher53, 352 (2015)

2015
[18]

Student, The probable error of a mean, Biometrika , 1 (1908)

1908
[19]

Adapted pendulum lab: PhET interactive simulations project at the University of Colorado 21 Boulder (2018)

2018
[20]

D. A. Norman, Affordance, conventions, and design, Interactions6, 38–43 (1999)

1999
[21]

Podolefsky, B

N. Podolefsky, B. Moore, and K. Perkins, Implicit scaffolding in interactive simulations: Design strategies to support multiple educational goals, PERC Proceedings preprint (2013)

2013
[22]

H. N. Boone and D. A. Boone, Analyzing Likert data, Journal of extension50, 1 (2012)

2012
[23]

Laerd statistics (2024)

2024
[24]

Descamps, R

I. Descamps, R. Tobin, P. Wagoner, D. Hammer, N. G. Holmes, and R. E. Scherr, Lessons from pendulums: A design comparison of three lab activities, Physical Review Physics Education Research22, 010137 (2026). Appendix A: Student Survey The post-lab survey, presented in Fig. 5, was administered immediately after the lab session. The survey included Likert-t...

2026
[25]

The coding process involved initial categorization, refining categories, and achieving consensus on the final coding

Coding Open Responses For open-text question 4, two authors independently coded responses, reconciling any dis- agreements through discussion. The coding process involved initial categorization, refining categories, and achieving consensus on the final coding. 22 Figure 5. The post-lab anonymous survey. (N P hysical = 71,N Simulation = 79,N T otal = 150) ...

[1] [1]

Trust in their equipment Survey question 4 was an open-ended question about the student’s associated trust with the simulated pendulum versus the physical one. Coding for this open-ended question con- sisted of two authors (JB and JD) undergoing several of several rounds of coding and dis- cussions, eventually deciding on three categories with 100% agreem...

[2] [2]

Trust simulated pendulums over physical pendulums

[3] [3]

Trust physical pendulums over simulated pendulums

[4] [4]

Other The results of question 4 (Table IV) showed that both groups overwhelmingly claim to trust the results from simulation labs more than their analogous physical labs, with no difference in levels of trust between the two groups

[5] [5]

slightly more simulations than physical labs in the future,

Student Experience As for the remainder of the survey, two questions specifically address the student experi- ence with their equipment/simulation. Question 6 asked how difficult they found it relative to previous labs. Students using the simulation tilted more towards saying they found it less 17 Type Count Trust Sim % Trust Phys % Trust Other % Phys 66 ...

[6] [6]

N. D. Finkelstein, W. K. Adams, C. J. Keller, P. B. Kohl, K. K. Perkins, N. S. Podolefsky, S. Reid, and R. LeMaster, When learning about the real world is better done virtually: A study of substituting computer simulations for laboratory equipment, Physical Review Special Topics - Physics Education Research1, 010103 (2005)

2005

[7] [7]

Klahr, L

D. Klahr, L. M. Triona, and C. Williams, Hands on what? the relative effectiveness of physical 20 versus virtual materials in an engineering design project by middle school children, Journal of research in science teaching44, 183 (2007)

2007

[8] [8]

Z. C. Zacharia and C. P. Constantinou, Comparing the influence of physical and virtual manipulatives in the context of the physics by inquiry curriculum: The case of undergraduate students’ conceptual understanding of heat and temperature, American journal of physics76, 393 (2008)

2008

[9] [9]

Z. C. Zacharia and G. Olympiou, Physical versus virtual manipulative experimentation in physics learning, Learning and instruction21, 317 (2011)

2011

[10] [10]

Jariwala, E

M. Jariwala, E. Allen, and A. Duffy, Investigating simulation use on student learning outcomes in introductory physics, inPhysics Education Research Conference 2019, PER Conference (Provo, UT, 2019) pp. 263–268

2019

[11] [11]

H. O. Kapici, H. Akcay, and T. de Jong, Using hands-on and virtual laboratories alone or together—which works better for acquiring knowledge and skills?, Journal of science education and technology28, 231 (2019)

2019

[12] [12]

J. R. Brinson, Learning outcome achievement in non-traditional (virtual and remote) versus traditional (hands-on) laboratories: A review of the empirical research, Computers & Educa- tion87, 218 (2015)

2015

[13] [13]

N. G. Holmes,Structured quantitative inquiry labs: Developing critical thinking in the intro- ductory physics laboratory, Ph.D. thesis, University of British Columbia (2014)

2014

[14] [14]

Collins, J

A. Collins, J. S. Brown, and S. E. Newman, Cognitive apprenticeship: Teaching the craft of reading, writing and mathematics, Thinking: The Journal of Philosophy for Children8, 2 (1988)

1988

[15] [15]

N. G. Holmes, C. E. Wieman, and D. A. Bonn, Teaching critical thinking, Proceedings of the National Academy of Sciences112, 11199 (2015)

2015

[16] [16]

Buffler, S

A. Buffler, S. Allie, F. Lubben, and B. Campbell, The development of first year physics students’ ideas about measurement in terms of point and set paradigms, Int. J. Sci. Educ.23, 1137 (2001)

2001

[17] [17]

Holmes and D

N. Holmes and D. Bonn, Quantitative comparisons to promote inquiry in the introductory physics lab, The Physics Teacher53, 352 (2015)

2015

[18] [18]

Student, The probable error of a mean, Biometrika , 1 (1908)

1908

[19] [19]

Adapted pendulum lab: PhET interactive simulations project at the University of Colorado 21 Boulder (2018)

2018

[20] [20]

D. A. Norman, Affordance, conventions, and design, Interactions6, 38–43 (1999)

1999

[21] [21]

Podolefsky, B

N. Podolefsky, B. Moore, and K. Perkins, Implicit scaffolding in interactive simulations: Design strategies to support multiple educational goals, PERC Proceedings preprint (2013)

2013

[22] [22]

H. N. Boone and D. A. Boone, Analyzing Likert data, Journal of extension50, 1 (2012)

2012

[23] [23]

Laerd statistics (2024)

2024

[24] [24]

Descamps, R

I. Descamps, R. Tobin, P. Wagoner, D. Hammer, N. G. Holmes, and R. E. Scherr, Lessons from pendulums: A design comparison of three lab activities, Physical Review Physics Education Research22, 010137 (2026). Appendix A: Student Survey The post-lab survey, presented in Fig. 5, was administered immediately after the lab session. The survey included Likert-t...

2026

[25] [25]

The coding process involved initial categorization, refining categories, and achieving consensus on the final coding

Coding Open Responses For open-text question 4, two authors independently coded responses, reconciling any dis- agreements through discussion. The coding process involved initial categorization, refining categories, and achieving consensus on the final coding. 22 Figure 5. The post-lab anonymous survey. (N P hysical = 71,N Simulation = 79,N T otal = 150) ...