Pith. sign in

REVIEW 3 major objections 5 minor 78 references

A Tool-Invariant Framework for Teaching and Assessing Computational Methods in the Age of Agentic AI

T0 review · 3 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read What students must know stays constant across tool changes; only the weight on verification moves, so assessment must move from submissions to live defense.

desk verdict Solid, honest framework paper that makes a credible case for verification-centered teaching and oral-defense assessment under agentic AI; the load-bearing white-box dose is untested but explicitly flagged as a bet. read the letter →

arxiv 2607.15518 v1 pith:KM7ZUOQR submitted 2026-07-17 physics.ed-ph physics.comp-ph

Larry Engelhardt This is my paper · ORCID
classification physics.ed-phphysics.comp-ph PACS 01.40.Ha
keywords tool-invariantframeworkagenticAIcomputationalphysicseducationverificationoralassessmentwhite-box/black-boxsensemakingcollapse
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that what students must learn to use computational methods has stayed constant across tool transitions: inputs and outputs, conceptual understanding, terminology, sensemaking, and tool operation. Agentic AI is the latest and largest offloading yet, shifting the weight almost entirely onto verification, because AI-generated simulations are bespoke artifacts that no community has validated. The traditional homework artifact—code, plots, report—no longer certifies the student, since AI can produce it on demand. The paper's response is a two-instrument assessment: AI-free in-class coding quizzes for a brief white-box phase, paired with ten-minute oral defenses of comment-stripped, AI-assisted work, gated on a verification dimension. The product of a computational physics course becomes the student's ability to explain and defend computational artifacts in the language of physics.

What carries the argument

The five-pillar framework: competent use of any computational method requires (1) specifying inputs and outputs, (2) conceptual understanding of the method, (3) precise terminology, (4) sensemaking to judge whether outputs are right, and (5) operating the current tool. The argument runs through a Specify–Predict–Delegate–Verify–Interpret loop and the 'principle of validation authority'—delegation is safe only where the delegator retains the ability to validate outputs. That principle distinguishes opaque but socially validated library code from opaque, bespoke, agent-generated simulations, and it is what makes verification newly load-bearing.

What would settle it

A controlled experiment comparing two cohorts on the same simulation task—one that completes the hand-coded Euler white-box exercise, one that jumps straight to agent-generated code—would settle the central claim: if both groups detect injected errors at equal rates, the framework's core assumption fails.

Watch

Extended reading notes

Core claim

The paper claims that agentic AI is the latest step in a centuries-long migration of mechanical work from human to tool, and that what students must know has not changed: inputs and outputs, conceptual understanding, terminology, sensemaking, and tool operation. What changes is weight: with AI writing code, sensemaking and verification become load-bearing, and the unsupervised artifact no longer certifies the student. Because each AI-generated simulation is a bespoke artifact with a population of one, the verification burden that library ecosystems once amortized across the community now lands on each student, for every artifact, every time. The constructive consequence is a two-instrument a

Load-bearing premise

A single brief hand-coded encounter with each method leaves enough conceptual residue for students to verify agent-generated simulations later

Editorial extensions

If this is right

  • If the framework holds, computational physics courses will shift from teaching code authorship to teaching specification, prediction, and verification as the core skills.
  • Unsupervised computational assignments can no longer certify student understanding; courses must move to supervised formats such as proctored quizzes and oral defenses.
  • A brief white-box phase—one hand-coded Euler integrator per method—is enough to leave the conceptual residue needed for later verification and diagnosis of agent errors.
  • A ten-minute oral defense with a verification gate can reliably distinguish genuine understanding from performed command, and is feasible in the small classes where the subject lives.
  • AI-free coding quizzes remain the only valid measure of coding fluency, which the paper regards as a permanent but small learning goal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 'proxy collapse' generalizes beyond physics: any discipline that relies on artifacts—code, essays, designs—faces the same assessment problem once AI can generate those artifacts on demand.
  • The validation-authority principle suggests that as AI improves, the verification burden may not shrink as much as expected, because bespoke artifacts never acquire the community validation that libraries enjoy.
  • A concrete testable extension is whether prediction-first exercises measurably improve detection of injected errors; if not, the verification curriculum risks becoming ritual rather than genuine calibration.
  • The framework implies class size becomes an academic-integrity variable: institutions that cannot staff oral defenses may issue systematically less trustworthy credentials, an equity concern the paper makes explicit.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a tool-invariant framework for teaching and assessing computational methods in physics, organized around five pillars (inputs/outputs, conceptual understanding, terminology, sensemaking, and tool operation). It argues that as agentic AI assumes code-writing and execution, verification—not code authorship—becomes the load-bearing skill, and that unsupervised submitted artifacts no longer certify student understanding. The author proposes a two-instrument assessment model: AI-free in-class coding quizzes for the white-box phase, and comment-stripped oral defenses with a verification-gated rubric for AI-assisted project work. The paper explicitly positions itself as a framework and a structure for adaptation, honestly labels its central assumptions as untested, and lists open questions, most importantly whether a brief white-box phase suffices to build the conceptual residue needed for later verification of agent-generated simulations.

Significance. If the framework holds, it provides a timely and actionable response to the collapse of unsupervised computational assessment, and it reframes the computational physics curriculum around validation authority and disciplinary discourse rather than artifact production. The paper is strong in its conceptual organization, its synthesis of the computational-thinking and assessment literatures, and its explicit, falsifiable open questions. It is also unusually transparent: it names its central untested assumptions, reports the anecdotal status of its evidence, and discloses its own AI-assisted authorship. The main risk is that the practical recommendations will be read as validated when they are, by the author's own account, a testable design awaiting cohort data.

major comments (3)
  1. [Secs. V A, VII, VIII Q1] The white-box dose is the load-bearing empirical premise: the claim that a single hand-coded Euler encounter plus weekly quizzes leaves enough conceptual residue to verify agent-generated simulations is untested. The author's citation [56] shows manual-flying skills decay under training doses far larger than this phase; the reply that weekly quizzes re-exercise the residue is plausible but not evidence, since the quizzes assess coding fluency and injected-error diagnosis, not transfer to bespoke agent artifacts. Because the oral-defense verification gate depends on this transfer, the manuscript should either present preliminary pre/post evidence or explicitly mark the practical recommendations as conditional on a hypothesis that the first cohorts will test.
  2. [Secs. VI B and VI C] The evidence base for the defense protocol is one semester in two upper-level courses (n=8), without comment-stripping and without the formal rubric. Despite this, the text states 'ten minutes suffices' and 'genuine and performed command separate within minutes' as design facts. These are anecdotal judgments, not measurements. Because the verification gate is the core assessment innovation, the protocol's validity requires at least inter-rater reliability, gate-failure rates, and differential-impact data (the author lists these in Sec. VIII Q1 but does not provisionally answer them). Please separate 'experience so far' from 'validated practice' and soften the prescriptive language accordingly.
  3. [Secs. V B, VI B, VIII Q2] The framework claims that error-injection exercises and provided-code quizzes train the code-reading fluency that the defense walkthrough demands, and that verification is 'the teachable core.' Yet open question 2 concedes that how much code reading directing agents requires is unresolved. The assertion in Sec. V B that the exercises 'provide the supervisory practice that the agentic workflow no longer provides by itself' is therefore stronger than the evidence. A revision should state the assumed mechanism linking error-injection practice to real-agent verification and specify the measurement that would confirm or refute it.
minor comments (5)
  1. [Secs. II and III B] The headings contain typographical artifacts: 'COMPUT A TION HAS AL W A YS BEEN DELEGA TED' and 'V erify'. These should be corrected to 'COMPUTATION HAS ALWAYS BEEN DELEGATED' and 'Verify'.
  2. [Sec. VI C] The phrase 'the deeper reply is constructive alignment itself' is awkward; 'deeper point' would be clearer. As written, it is unclear whether 'reply' refers to the answer to the adversarial objection or to the assessment mechanism.
  3. [Table I] The final row 'What is new' has no entries in the antecedent columns; clarify whether this is a merged row or a deliberate empty row. The supplementary tables S1 and S2 are referenced but not included in the main text; state where they are available.
  4. [Sec. VI B] The statement 'the ten-minute session length is measured—a semester of ... stands behind it' overstates what a semester of informal experience provides. Use 'estimated from experience' rather than 'measured'.
  5. [Sec. V D] The sentence 'an end-of-semester survey drew exactly one response, itself a small lesson in instrumentation' is effective and honest, but the subsequent 'response split sharply' should be flagged even more clearly as one instructor's impression, since the only systematic instrument produced n=1.

Circularity Check

0 steps flagged · score 0.0 of 10

No material circularity: the framework is argued from external evidence and explicitly labels its one untested premise as an open question.

full rationale

This paper contains no derivation chain in which a claimed prediction is equivalent by construction to a fitted input or to a self-citation. The central claims are conceptual and are supported by external citations (Goodhart via [47], Bainbridge [38], Casner et al. [56], Kortemeyer [15], the RCT on AI tutoring [43], etc.), not by a loop into the author's own prior results. The self-citations that exist ([3], [5], [40], [41]) are contextual or infrastructural; none of them carries the load-bearing argument that verification, not code authorship, should be the assessed skill. The five pillars are explicitly presented in Table I as a repackaging of the computational-thinking literature, with the claimed novelty located in the re-weighting and in the assessment consequence, so there is no disguised renaming of a known result. The oral-defense assessment is aligned with the intended learning outcome, which the paper itself identifies as constructive alignment [53]; aligning an assessment instrument to the target competency is not circular reasoning. The one genuine vulnerability, the sufficiency of a brief white-box phase, is acknowledged by the author rather than hidden: Sec. V A calls it 'the framework's central untested assumption' and Sec. VIII lists it as open question 1. This is an unvalidated empirical premise, not a self-referential derivation, and it does not make the argument circular. Moreover, the paper explicitly forbids the one circular move it could have made: Sec. III D states that deciding what counts as correctness is a scientific judgment and that 'delegating it to the system being checked is circular.' No circular steps were identified.

Assumptions & free parameters 0 free parameters · 5 assumptions · 2 invented entities

The paper introduces no fitted constants; its burden is conceptual. The central claims rest on domain assumptions about AI artifact reliability, the sufficiency of a brief white-box phase, and the validity of oral assessment—several of which the paper explicitly marks untested. The two invented constructs (validation authority, verification-gated defense) are normative design principles, not empirically grounded entities.

assumptions (5)
  • domain assumption Five pillars (inputs/outputs, concept, terminology, sensemaking, tool operation) are necessary and sufficient for competent use of any computational method.
    Stated in Sec. III A as a universal claim spanning single-digit addition to molecular dynamics; no empirical proof is offered beyond examples.
  • domain assumption An AI-generated artifact is bespoke and unsocially validated, so the verification burden falls on the user for each artifact.
    Sec. III C argues opacity was never the problem and validation is; this is a plausible but untested characterization of current AI reliability.
  • ad hoc to paper A brief white-box phase is sufficient to establish the conceptual residue needed for later orchestration.
    Sec. V A and VIII Q1 explicitly label this the framework's central untested assumption; false if one hand-coded integrator proves too small a dose.
  • domain assumption Oral defense with live adaptive probing reliably separates genuine from performed understanding.
    Sec. VI C asserts genuine and performed command separate within minutes, based on one semester informal experience, not controlled data; oral-assessment literature supports the underlying construct.
  • domain assumption Delegation is safe exactly where the delegator can validate outputs (validation authority).
    Sec. III C introduces this as a principle; it is definitional rather than derived from data, but not circular.
invented entities (2)
  • Validation authority
    purpose: A principle that delegation is safe only where the delegator retains ability to validate outputs; used to argue verification is now the load-bearing skill.
    Introduced as a new conceptual principle in Sec. III C; no independent falsifiable handle—it is a normative teaching principle.
  • Verification-gated oral defense
    purpose: An assessment protocol in which the verification dimension must reach 'functional' level for pass; used to certify students when artifacts are AI-generated.
    Proposed as new two-instrument design in Sec. VI; no outcome data yet, first full test planned for Fall 2026.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Tool-Invariant Framework for Teaching and Assessing Computational Methods in the Age of Agentic AI." pith.science (2026). https://pith.science/paper/KM7ZUOQR

@misc{pith2026260715518,
  author       = {Pith},
  title        = {Pith review of: A Tool-Invariant Framework for Teaching and Assessing Computational Methods in the Age of Agentic AI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KM7ZUOQR}},
  note         = {Machine review of arXiv:2607.15518}
}
read the original abstract

Learning a computational method has always meant learning to operate a tool -- pencil, slide rule, calculator, or programming language. Agentic artificial intelligence, which writes, executes, and revises simulation code from natural-language specifications, is the latest and largest step in a centuries-long migration of mechanical work from human to tool. I argue that what a learner must know has remained remarkably stable: the inputs and outputs of a method, the concept of what it does, the terminology to communicate about it, the judgment to evaluate its results, and the skill of operating the current tool. This paper organizes these requirements into a tool-invariant framework spanning single-digit addition to agent-orchestrated molecular dynamics, argues that verification -- not code authorship -- is now the load-bearing skill, and draws the consequence for assessment: when artifacts can be generated on demand, the artifact no longer certifies the student. I describe a practical response, designed for the small classes where the subject lives -- AI-free in-class coding quizzes paired with oral defenses of comment-stripped, AI-assisted work -- and argue that the real product of a computational physics course is the student's ability to explain and defend computational artifacts in the language of the discipline.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

78 extracted references · 1 canonical work pages

  1. [56]

    Should students learn integration rules?,

    B. Buchberger, “Should students learn integration rules?,” ACM SIGSAM Bull. 24(1), 10–17 (1990)

    1990

  2. [1]

    What goes in, in what form, under what conditions of validity; what comes out, and what it means

    Inputs and outputs. What goes in, in what form, under what conditions of validity; what comes out, and what it means. This includes posing the problem—deciding that a computation is worth do- ing at all

  3. [2]

    Conceptual understanding of the method. Not the ability to execute the algorithm, but a working model of what it does—including its knobs (step size, tolerance, sample count) and its charac- teristic failure modes (instability, divergence, over- fitting, aliasing)

  4. [3]

    T erminology .The vocabulary of the method and of the physics, precisely used

  5. [4]

    Sensemaking. The ability to judge whether out- puts make sense—in the physics-education-research sense of building an explanation that resolves a perceived gap or conflict in one’s knowledge[21]— and to establish whether they are right (Sec. IV). Computation reliably provokes exactly such con- flicts, and that confrontation is where the learning lives.[22–24]

  6. [5]

    integrate this ODE with an adaptive Runge–Kutta method and verify energy conservation

    Operating the tool. The mechanical or inter- actional skill of actuating the computation: pen- manship, keystrokes, syntax—or, now, directing an agent. The child adding 2 + 3 exercises all five. So does the senior directing an AI agent to build a molecular dy- namics simulation. What changes between those rungs is the content of each pillar and the weight...

  7. [6]

    Inputs and out- puts Data practices and problem preparation;[8] specification-level practices[9]

  8. [7]

    Method con- cept Modeling-and-simulation practices;[8] the cognitive dimension of computa- tional literacy[10]

Show all 78 references
  1. [8]

    Terminology The social dimension of computational literacy (disciplinary discourse)[10]

  2. [9]

    Sensemaking Model-assessment and debugging practices;[8, 9] the sensemaking con- struct itself[21]

  3. [10]

    anyone can build software with AI now

    Tool operation The material dimension of computa- tional literacy;[10] programming and tool practices[8, 9] What is new The re-weighting under agentic media- tion; validation authority (Sec. III C); the assessment consequence (Sec. VI) B. The pillars in action: iterating The p...

    1995

  4. [11]

    Regardless of whether or not the agent asks good clarifying questions— today’s agents routinely do—deciding what is worth computing is the scientist’s responsibility

    Posing the problem. Regardless of whether or not the agent asks good clarifying questions— today’s agents routinely do—deciding what is worth computing is the scientist’s responsibility

  5. [12]

    The agent will implement a wrong 4 model beautifully

    Choosing the physical model and owning its assumptions. The agent will implement a wrong 4 model beautifully

  6. [13]

    The prediction , made before execution, without the tool

  7. [14]

    Deciding what counts as evidence of correctness is the scientific judgment; delegating it to the system being checked is circular

    Specifying the checks. Deciding what counts as evidence of correctness is the scientific judgment; delegating it to the system being checked is circular

  8. [15]

    The AI said so

    Final epistemic responsibility . “The AI said so” is not a justification available to a scientist. Items 1, 2, and 5 are not claims about current AI ca- pability, and do not weaken as models improve. They are constitutive of doing science. A full task-by-task progression apply...

    2026

  9. [16]

    How much white-box is enough—and does the verification curriculum actually calibrate students? These are the framework’s central untested assumptions, and both are testable in our courses now: Does the hand-coded Euler encounter improve later diagnosis of black-box failures? D...

  10. [17]

    permanent but small

    What is the fate of coding fluency as a learn- ing goal? I hold the “permanent but small” position—some direct coding, honestly assessed in AI-free conditions, indefinitely. Colleagues I re- spect hold both neighboring positions; the question is empirical, not settled. Its sha...

  11. [18]

    What preserves competence, autonomy, and relatedness in the 300-student ser- vice course?

    How do we build intrinsic motivation at scale? The defense model works partly because it is small and human. What preserves competence, autonomy, and relatedness in the 300-student ser- vice course?

  12. [19]

    How do we develop faculty? My own conver- sion required a week of in-person immersion among practicing colleagues—as the expertise literature predicts.[25] PICUP’s workshops are infrastructure built for exactly this; what is the plan for scaling it to the majority of physics f...

  13. [20]

    What is a computational physics course in 2036? Does it dissolve into every course— computation as a medium—or sharpen into the dis- cipline of specification and verification described here? The half-century of paradigm stability be- hind us is over; curricular structures shou...

  14. [21]

    Student programming in the introductory physics course: M.U.P.P.E.T.,

    E. F. Redish and J. M. Wilson, “Student programming in the introductory physics course: M.U.P.P.E.T.,” Am. J. Phys. 61(3), 222–232 (1993)

    1993

  15. [22]

    Computational physics in the introductory calculus-based course,

    R. Chabay and B. Sherwood, “Computational physics in the introductory calculus-based course,” Am. J. Phys. 76(4), 307–313 (2008)

    2008

  16. [23]

    Re- sources for integrating computation into physics courses: Exercise sets, workshops, and a supportive community,

    L. Engelhardt, M. Lopez del Puerto, and K. Roos, “Re- sources for integrating computation into physics courses: Exercise sets, workshops, and a supportive community,” in Integrating Computing into the Undergraduate Physics Curriculum, edited by M. D. Caballero (IOP Publishing,...

    2025 doi

  17. [24]

    Behringer, J

    E. Behringer, J. Burciaga, D. Dietz, A. Gavrin, J. Kozminski, V. Migenes, and D. Schroeder (AAPT Un- dergraduate Curriculum Task Force), AAPT Recommen- dations for Computational Physics in the Undergraduate Physics Curriculum (AAPT, College Park, MD, 2016), https://www.aapt.or...

    2016

  18. [25]

    Behringer and L

    E. Behringer and L. Engelhardt, “Guest Editorial: AAPT Recommendations for computational physics in the undergraduate physics curriculum, and the Partner- ship for Integrating Computation into Undergraduate Physics,” Am. J. Phys. 85(5), 325–326 (2017)

    2017

  19. [26]

    Resource Letter CP-3: Computational physics,

    T. J. Atherton, “Resource Letter CP-3: Computational physics,” Am. J. Phys. 91(1), 7–27 (2023)

    2023

  20. [27]

    Computational thinking,

    J. M. Wing, “Computational thinking,” Commun. ACM 49(3), 33–35 (2006)

    2006

  21. [28]

    Defining computational thinking for mathematics and science classrooms,

    D. Weintrop, E. Beheshti, M. Horn, K. Orton, K. Jona, L. Trouille, and U. Wilensky, “Defining computational thinking for mathematics and science classrooms,” J. Sci. Educ. Technol. 25(1), 127–147 (2016)

    2016

  22. [29]

    Development and illustration of a frame- work for computational thinking practices in introduc- tory physics,

    D. P. Weller, T. E. Bott, M. D. Caballero, and P. W. Irving, “Development and illustration of a frame- work for computational thinking practices in introduc- tory physics,” Phys. Rev. Phys. Educ. Res. 18, 020106 (2022)

    2022

  23. [30]

    Physics computational literacy: An exploratory case study using computational essays,

    T. O. B. Odden, E. Lockwood, and M. D. Caballero, “Physics computational literacy: An exploratory case study using computational essays,” Phys. Rev. Phys. Educ. Res. 15, 020152 (2019)

    2019

  24. [31]

    Could an artificial-intelligence agent pass an introductory physics course?,

    G. Kortemeyer, “Could an artificial-intelligence agent pass an introductory physics course?,” Phys. Rev. Phys. Educ. Res. 19, 010132 (2023)

    2023

  25. [32]

    How understanding large language models can inform the use of ChatGPT in physics education,

    G. Polverini and B. Gregorcic, “How understanding large language models can inform the use of ChatGPT in physics education,” Eur. J. Phys. 45, 025701 (2024)

    2024

  26. [33]

    How do we assess computation in physics?,

    H. C. Sabo, T. O. B. Odden, and M. D. Caballero, “How do we assess computation in physics?,” Phys. Teach. 64(3), 200–203 (2026)

    2026

  27. [34]

    Editorial: Focused Collection: Artificial intelligence tools in physics teaching and physics education research,

    “Editorial: Focused Collection: Artificial intelligence tools in physics teaching and physics education research,” Phys. Rev. Phys. Educ. Res. 22, 010001 (2026)

    2026

  28. [35]

    The boiling-frog problem of physics education,

    G. Kortemeyer, “The boiling-frog problem of physics education,” Phys. Teach. 64(1), 8–12 (2026), doi:10.1119/5.0296601; arXiv:2508.08842

    2026

  29. [36]

    Computing education in the era of generative AI,

    P. Denny, J. Prather, B. A. Becker, J. Finnie-Ansley, A. Hellas, J. Leinonen, A. Luxton-Reilly, B. N. Reeves, E. A. Santos, and S. Sarsa, “Computing education in the era of generative AI,” Commun. ACM 67(2), 56–67 (2024)

    2024

  30. [37]

    How undergraduate physics students use generative AI for computational modeling,

    K. H. Fredly, T. O. B. Odden, and B. M. Zwickl, “How undergraduate physics students use generative AI for computational modeling,” arXiv:2603.06342 (2026)

    2026

  31. [38]

    D. A. Grier, When Computers Were Human (Princeton University Press, Princeton, NJ, 2005)

    2005

  32. [39]

    Effects of hand-held cal- culators in precollege mathematics education: A meta- analysis,

    R. Hembree and D. J. Dessart, “Effects of hand-held cal- culators in precollege mathematics education: A meta- analysis,” J. Res. Math. Educ. 17(2), 83–99 (1986)

    1986

  33. [40]

    A meta-analysis of the effects of calcula- tors on students’ achievement and attitude levels in pre- college mathematics classes,

    A. J. Ellington, “A meta-analysis of the effects of calcula- tors on students’ achievement and attitude levels in pre- college mathematics classes,” J. Res. Math. Educ. 34(5), 433–463 (2003)

    2003

  34. [41]

    Defining sensemaking: Bringing clarity to a fragmented theoretical construct,

    T. O. B. Odden and R. S. Russ, “Defining sensemaking: Bringing clarity to a fragmented theoretical construct,” Sci. Educ. 103(1), 187–205 (2019)

    2019

  35. [42]

    How computation can facilitate sensemaking about physics: A case study,

    O. P. Sand, T. O. B. Odden, C. Lindstrøm, and M. D. Caballero, “How computation can facilitate sensemaking about physics: A case study,” in 2018 Physics Education Research Conference Proceedings (AAPT, 2019)

    2018

  36. [43]

    As- sessing mathematical sensemaking in physics through calculation-concept crossover,

    E. Kuo, M. M. Hull, A. Elby, and A. Gupta, “As- sessing mathematical sensemaking in physics through calculation-concept crossover,” Phys. Rev. Phys. Educ. Res. 16, 020109 (2020). 10

    2020

  37. [44]

    Sensemaking and scientific modeling: Intertwined pro- cesses analyzed in the context of physics problem solv- ing,

    A. Sirnoorkar, P. D. O. Bergeron, and J. T. Laverty, “Sensemaking and scientific modeling: Intertwined pro- cesses analyzed in the context of physics problem solv- ing,” Phys. Rev. Phys. Educ. Res. 19, 010118 (2023)

    2023

  38. [45]

    Collins and R

    H. Collins and R. Evans, Rethinking Expertise (Univer- sity of Chicago Press, Chicago, 2007)

    2007

  39. [46]

    White and R

    R. White and R. Gunstone, Probing Understanding (Falmer Press, London, 1992)

    1992

  40. [47]

    Teaching com- puter programming with PRIMM: A sociocultural per- spective,

    S. Sentance, J. Waite, and M. Kallia, “Teaching com- puter programming with PRIMM: A sociocultural per- spective,” Comput. Sci. Educ. 29(2–3), 136–176 (2019)

    2019

  41. [48]

    Mahajan, Street-Fighting Mathematics: The Art of Educated Guessing and Opportunistic Problem Solving (MIT Press, Cambridge, MA, 2010)

    S. Mahajan, Street-Fighting Mathematics: The Art of Educated Guessing and Opportunistic Problem Solving (MIT Press, Cambridge, MA, 2010)

    2010

  42. [49]

    Trust in automation: Design- ing for appropriate reliance,

    J. D. Lee and K. A. See, “Trust in automation: Design- ing for appropriate reliance,” Hum. Factors 46(1), 50–80 (2004)

    2004

  43. [50]

    Humphreys, Extending Ourselves: Computational Sci- ence, Empiricism, and Scientific Method (Oxford Univer- sity Press, New York, 2004)

    P. Humphreys, Extending Ourselves: Computational Sci- ence, Empiricism, and Scientific Method (Oxford Univer- sity Press, New York, 2004)

    2004

  44. [51]

    The philosophical novelty of computer simulation methods,

    P. Humphreys, “The philosophical novelty of computer simulation methods,” Synthese 169(3), 615–626 (2009)

    2009

  45. [52]

    Grounds for trust: Es- sential epistemic opacity and computational reliabilism,

    J. M. Dur´ an and N. Formanek, “Grounds for trust: Es- sential epistemic opacity and computational reliabilism,” Minds Mach. 28(4), 645–666 (2018)

    2018

  46. [53]

    Navigating the jagged technological frontier: Field experimental evi- dence of the effects of AI on knowledge worker produc- tivity and quality,

    F. Dell’Acqua, E. McFowland III, E. R. Mollick, H. Lifshitz-Assaf, K. C. Kellogg, S. Rajendran, L. Krayer, F. Candelon, and K. R. Lakhani, “Navigating the jagged technological frontier: Field experimental evi- dence of the effects of AI on knowledge worker produc- tivity and q...

    2026

  47. [54]

    W. L. Oberkampf and C. J. Roy, Verification and Val- idation in Scientific Computing (Cambridge University Press, Cambridge, 2010)

    2010

  48. [55]

    An experimental eval- uation of the assumption of independence in multiversion programming,

    J. C. Knight and N. G. Leveson, “An experimental eval- uation of the assumption of independence in multiversion programming,” IEEE Trans. Softw. Eng. SE-12(1), 96– 109 (1986)

    1986

  49. [57]

    Teaching physics (and some compu- tation) using intentionally incorrect simulations,

    A. J. Cox, W. F. Junkin III, W. Christian, M. Belloni, and F. Esquembre, “Teaching physics (and some compu- tation) using intentionally incorrect simulations,” Phys. Teach. 49(5), 273–276 (2011)

    2011

  50. [58]

    Ironies of automation,

    L. Bainbridge, “Ironies of automation,” Automatica 19(6), 775–779 (1983)

    1983

  51. [59]

    Prompt Problems: A new programming exercise for the genera- tive AI era,

    P. Denny, J. Leinonen, J. Prather, A. Luxton-Reilly, T. Amarouche, B. A. Becker, and B. N. Reeves, “Prompt Problems: A new programming exercise for the genera- tive AI era,” in Proceedings of the 55th ACM Technical Symposium on Computer Science Education (SIGCSE

  52. [61]

    Engelhardt, Offline Interactive Dashboard & Down- loader for Blackboard , v0.9.0 (2026), https://github

    L. Engelhardt, Offline Interactive Dashboard & Down- loader for Blackboard , v0.9.0 (2026), https://github. com / lengelhardt / bb - offline - interactive - dashboard

    2026

  53. [62]

    1D Quantum Dynamics Assignment,

    L. Engelhardt, “1D Quantum Dynamics Assignment,” PICUP Collection, April 2019, https://www.compadre. org / PICUP / exercises / exercise . cfm ? A = quantum _ dynamics_1D (accessed July 2026)

    2019

  54. [63]

    Simulating a Double Pendulum with a Large Language Model,

    T. Zimmerman, “Simulating a Double Pendulum with a Large Language Model,” submitted to the PICUP Col- lection (2026)

    2026

  55. [64]

    AI tutoring outperforms in-class active learning: An RCT introducing a novel research-based design in an authentic educational setting,

    G. Kestin, K. Miller, A. Klales, T. Milbourne, and G. Ponti, “AI tutoring outperforms in-class active learning: An RCT introducing a novel research-based design in an authentic educational setting,” Sci. Rep. 15, 17458 (2025)

    2025

  56. [65]

    Help-seeking: An understudied problem-solving skill in children,

    S. Nelson-Le Gall, “Help-seeking: An understudied problem-solving skill in children,” Dev. Rev. 1(3), 224– 246 (1981)

    1981

  57. [66]

    Why A.I. isn’t going to make art,

    T. Chiang, “Why A.I. isn’t going to make art,” The New Yorker (August 31, 2024), https : / / www . newyorker . com / culture / the - weekend - essay / why - ai - isnt - going - to - make - art. Chiang’s image—using Chat- GPT for assignments is like bringing a forklift into the...

    2024

  58. [67]

    Intrinsic and extrinsic mo- tivations: Classic definitions and new directions,

    R. M. Ryan and E. L. Deci, “Intrinsic and extrinsic mo- tivations: Classic definitions and new directions,” Con- temp. Educ. Psychol. 25(1), 54–67 (2000)

    2000

  59. [68]

    ‘Improving ratings’: Audit in the British University system,

    M. Strathern, “ ‘Improving ratings’: Audit in the British University system,” Eur. Rev. 5(3), 305–321 (1997)

    1997

  60. [69]

    Partners in cognition: Extending human intelligence with intelli- gent technologies,

    G. Salomon, D. N. Perkins, and T. Globerson, “Partners in cognition: Extending human intelligence with intelli- gent technologies,” Educ. Res. 20(3), 2–9 (1991)

    1991

  61. [70]

    Dimensions of oral assessment,

    G. Joughin, “Dimensions of oral assessment,” Assess. Eval. High. Educ. 23(4), 367–378 (1998)

    1998

  62. [71]

    Students’ views of oral per- formance assessment in mathematics: Straddling the ‘as- sessment of’ and ‘assessment for’ learning divide,

    P. Iannone and A. Simpson, “Students’ views of oral per- formance assessment in mathematics: Straddling the ‘as- sessment of’ and ‘assessment for’ learning divide,” As- sess. Eval. High. Educ. 40(7), 971–987 (2015)

    2015

  63. [72]

    Code interviews: Design and evaluation of a more authentic assessment for introductory programming assignments,

    S. Kannam, Y. Yang, A. Dharm, and K. Lin, “Code interviews: Design and evaluation of a more authentic assessment for introductory programming assignments,” in Proceedings of the 56th ACM Technical Symposium on Computer Science Education (SIGCSE 2025) (ACM, New York, 2025); arX...

    2025 arXiv

  64. [73]

    The quizzes use a browser-embedded Python environ- ment (Trinket), served inside a lockdown browser. The original service, https://trinket.io, sunsets in August 2026; an open-source successor maintained by members of the PICUP community—now including a variable ex- plorer and ...

    2026

  65. [74]

    Enhancing teaching through constructive alignment,

    J. Biggs, “Enhancing teaching through constructive alignment,” High. Educ. 32(3), 347–364 (1996)

    1996

  66. [75]

    Eliciting self-explanations improves under- standing,

    M. T. H. Chi, N. de Leeuw, M.-H. Chiu, and C. La- Vancher, “Eliciting self-explanations improves under- standing,” Cogn. Sci. 18(3), 439–477 (1994)

    1994

  67. [76]

    comprehension debt

    The term “comprehension debt”—the gap between the code a system contains and the code its maintainers gen- uinely understand—comes from software-engineering dis- course. The earliest documented use I am aware of is J. Gorman, “Comprehension debt: The ticking time bomb of LLM-g...

    2025

  68. [77]

    The retention of manual flying skills in the automated cockpit,

    S. M. Casner, R. W. Geven, M. P. Recker, and J. W. Schooler, “The retention of manual flying skills in the automated cockpit,” Hum. Factors 56(8), 1506–1516 (2014)

    2014

  69. [78]

    Learning mathematics in a CAS environ- ment: The genesis of a reflection about instrumenta- tion and the dialectics between technical and concep- tual work,

    M. Artigue, “Learning mathematics in a CAS environ- ment: The genesis of a reflection about instrumenta- tion and the dialectics between technical and concep- tual work,” Int. J. Comput. Math. Learn. 7(3), 245–274 (2002)

    2002

  70. [2024]

    (ACM, New York, 2024)

    2024

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.