Pith. sign in

REVIEW 3 major objections 4 minor 34 references

This registered study will build a theory of how software professionals evaluate AI-generated code from their own practices, preferences, and values.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-07-13 03:00 UTC pith:WCBAVM7B

load-bearing objection Solid registered-report protocol for a constructivist GT study of AI-code evaluation; timely and carefully designed, not yet a theory. the 3 major comments →

arxiv 2607.09434 v1 pith:WCBAVM7B submitted 2026-07-10 cs.SE

How Do Software Professionals Evaluate AI-Generated Code? (Registered Report)

Samuli M\"a\"att\"a , Hera Arif , Burak Turhan , Paul Ralph , Markus Kelanti This is my paper
classification cs.SE
keywords Generative AIsoftware engineeringprogramminggrounded theorycode evaluationladdering interviewsAI-assisted programmingtrust
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Software professionals now routinely use generative AI tools such as Copilot and ChatGPT, yet how they decide whether to trust, repair, reject, or regenerate the resulting code remains poorly understood. This registered report describes a constructivist grounded-theory study that starts from an already-collected Finnish survey and proceeds with laddering and semi-structured interviews of roughly 20–50 professionals until theoretical saturation. The goal is a cohesive theory of evaluative practices grounded in participants’ accounts of what they do, why they prefer certain approaches, and what values or goals drive those choices under productivity pressure. Readers should care because evaluation effort, over-reliance, and responsibility for AI-generated defects are becoming everyday engineering problems as tools grow more capable. The resulting theory is intended to explain how professionals construct “good enough” evaluation when careful reading conflicts with deadlines and incomplete understanding.

Core claim

The paper claims that a constructivist grounded-theory program—combining open survey responses on validation differences, productivity-versus-risk practices, and trust change with laddering and semi-structured interviews—can produce a transferable theory of how software professionals evaluate AI-generated code, rooted in their constructed meanings, attribute–consequence–value chains, and practical constraints rather than in external measures of code quality alone.

What carries the argument

Laddering interviews that elicit attribute–consequence–value chains (means–end hierarchies) from concrete evaluation practices up to the personal or organisational values that justify them; these chains, together with survey open items and semi-structured transcripts, feed constant comparison and theoretical coding until saturation.

Load-bearing premise

That self-reported survey and interview accounts from a mainly Finnish convenience sample, without systematic observation of live coding, will be rich and accurate enough to reach theoretical saturation and a transferable theory of real evaluation under pressure.

What would settle it

If iterative interviews of the planned 20–50 professionals produce only fragmented or contradictory codes with no stable categories or attribute–consequence–value patterns after repeated theoretical sampling, the claim that this design yields a cohesive grounded theory of evaluation practices would be falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This registered report proposes a constructivist grounded-theory study of how software professionals evaluate AI-generated code. The protocol combines an already-collected survey of 163 Finnish software professionals (open items on validation differences, productivity-vs-risk practices, and trust change) with laddering interviews and complementary semi-structured interviews of approximately 20–50 AI-using professionals, continuing until theoretical saturation. The stated aim is a cohesive theory grounded in participants’ accounts of evaluative practices, perceptions, and preferences, under an intentionally broad initial RQ that is allowed to evolve with the analysis.

Significance. The topic is timely and under-theorised: generative AI is shifting professional work toward reading, judging, and repairing generated code under productivity pressure, yet cohesive accounts of evaluation practices remain sparse. A well-executed constructivist GT theory would be useful to SE research and tool design. Strengths of the protocol include an explicit Charmaz-style workflow (initial/focused/theoretical coding, constant comparison, memos, theoretical sampling), dual interview modes (laddering plus semi-structured), survey open-text already in hand, and a candid positionality and reflexivity plan (§2.6). Prior work (e.g., Grounded Copilot) is used as gap motivation rather than as a forced template.

major comments (3)
  1. [§2.4 Constructivist Grounded Theory] §2.4 (and Abstract / §2.3): For a registered report, the stopping rule is too underspecified. “Theoretical saturation” is defined only as no new themes or insights emerging, with a sample band of 20–50. Please pre-specify operational criteria (e.g., consecutive interviews that add no new properties to focal categories; how memo review and dual-coder discussion will confirm saturation; whether laddering chains and semi-structured transcripts are saturated separately or jointly). Without this, Stage-2 acceptance criteria remain uncheckable.
  2. [§2.2 Interview Design and Piloting] §2.2: The laddering protocol is still too open for Stage-1 registration. Stimuli are not listed; questions are illustrative (“What would you evaluate…?”) and “must be refined by piloting”; the stimuli list “may change” if GT directions emerge. Provide an initial stimuli set (or concrete derivation rules from the survey), a pilot-ready interview guide, and decision rules for when stimuli may be revised without voiding the registered design. Also state how attribute–consequence–value chains will be coded so that means–end structure informs, rather than over-constrains, constructivist categories.
  3. [§1 Introduction; §2.1; §2.3] §1 and §2.1–2.3: The design explicitly prioritises constructed meanings over observation of coding sessions, and the initial pool is a convenience/snowball Finnish subsample (44 eligible contacts). That is coherent with constructivism, but the protocol should pre-register claim boundaries: what the theory will be entitled to say about “evaluative practices” versus reported preferences and justifications; how theoretical sampling will expand beyond Finland if categories require it; and how productivity-pressure and over-reliance claims will be grounded without behavioural data. This is load-bearing for transferability of the promised theory.
minor comments (4)
  1. [§2.1 Initial Survey] §2.1: The validation/evaluation terminology shift is explained, but the manuscript should state once, early, that participant language will be preserved in coding even if the analytic term remains “evaluation.”
  2. [Table 1] Table 1: Role percentages sum above 100% only if multiple roles were allowed; clarify whether titles were single-choice or multi-select.
  3. [§2.5 Research Philosophy] §2.5: The hybrid constructivist/critical-realist stance is welcome; a short note on how that stance will be visible in memoing and category writing would help readers interpret later theory claims.
  4. [References] References: ensure consistent treatment of arXiv preprints vs. published versions (e.g., Ferino et al., Li et al., Sarkar et al.) before camera-ready.

Circularity Check

0 steps flagged

No circularity: registered-report protocol proposes future GT theory from new interviews plus own survey text; no equations, fits, or load-bearing self-citation reductions.

full rationale

This is a registered report outlining a constructivist grounded-theory study (survey already collected + planned laddering and semi-structured interviews of 20–50 professionals until theoretical saturation). It contains no mathematical derivation chain, no fitted parameters renamed as predictions, no uniqueness theorems, and no equations that restate their inputs by construction. Prior citations (e.g., Barke et al. Grounded Copilot) are used only for gap motivation and terminology, not as the source of the future theory. The authors’ own Finnish survey open responses are legitimate primary data for the planned GT analysis, not a circular self-citation that forces the result. Positionality/reflexivity statements and planned theoretical sampling are standard methodological transparency, not circular steps. The central claim is only that the protocol can yield a theory grounded in participants’ accounts; that claim is self-contained against external benchmarks and does not reduce to its inputs. Score 0 is therefore required.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

Load-bearing commitments are methodological, not physical constants. The central future claim rests on constructivist GT epistemology, laddering as a means–end elicitation tool, theoretical saturation as a stopping rule, and the adequacy of self-report from a constrained sampling frame. No new physical entities; free parameters are design ranges (n interviews, stimulus set) chosen by the researchers rather than fitted to a quantitative model.

free parameters (3)
  • interview_sample_size_range
    Target 20–50 interviewees until theoretical saturation; range is intentionally broad and researcher-judged, not derived from a power calculation.
  • theoretical_saturation_stopping_rule
    Data collection stops when “no new themes or insights emerge”; operational threshold is analyst judgment, a free design parameter of GT studies.
  • laddering_stimuli_set
    Scenarios of AI capability (from survey or pre-interview survey) are ranked and may change as theory emerges—hand-crafted stimuli that shape which practices are elicited.
axioms (4)
  • domain assumption Constructivist grounded theory is an appropriate method: knowledge of evaluation practices is co-constructed from participants’ accounts and researcher interpretation rather than discovered as value-free fact (Charmaz; §2.4–2.5).
    Underwrites the entire analysis pipeline and the claim that interview-based theory answers the RQ.
  • domain assumption Laddering (attribute–consequence–value chains via repeated why-probes) can validly surface stable drivers of evaluation practices for AI-generated code (§2.2).
    Method imported from consumer research; assumed transferable to professional SE evaluation under productivity pressure.
  • ad hoc to paper Self-reported practices, preferences, and trust narratives are sufficient primary data for a theory of evaluation, with observation deprioritized (§1).
    Explicit design choice; if reports systematically diverge from behavior, the theory misdescribes practice.
  • domain assumption Theoretical sampling from an initial Finnish convenience/snowball survey pool (plus expansion as needed) can saturate categories relevant beyond that context (§2.1, §2.3).
    Standard GT stance against statistical generalizability, but still assumes enough diversity for a useful theory.

reviewed 2026-07-13 · how reviews work

0 comments
Cite this review

Pith. "Pith review of How Do Software Professionals Evaluate AI-Generated Code? (Registered Report)." pith.science (2026). https://pith.science/paper/WCBAVM7B

@misc{pith2026260709434,
  author       = {Pith},
  title        = {Pith review of: How Do Software Professionals Evaluate AI-Generated Code? (Registered Report)},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WCBAVM7B}},
  note         = {Machine review of arXiv:2607.09434}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Recent advances in generative AI tools have significantly changed how software professionals write, evaluate, and interact with code. Generative AI tools such as GitHub Copilot, ChatGPT, and Claude are increasingly being integrated into everyday workflows. Despite the growing adoption of and reliance on these tools, it remains unclear as to how software professionals evaluate the code they generate. To explore this topic, we will conduct a constructivist grounded theory study that incorporates a survey, semi-structured interviews, and laddering interviews. With the initial survey data collection complete, we aim to interview 20--50 software professionals iteratively until theoretical saturation is achieved. This research aims to build a theory of how software professionals evaluate AI-generated code, grounded in their accounts of evaluative practices, perceptions, and preferences.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 1 canonical work pages

  1. [1]

    2016 , isbn =

    Stol, Klaas-Jan and Ralph, Paul and Fitzgerald, Brian , title =. 2016 , isbn =. doi:10.1145/2884781.2884833 , booktitle =

    doi:10.1145/2884781.2884833 2016

  2. [2]

    Lessons Learned from an Extended Participant Observation Grounded Theory Study , year=

    Sedano, Todd and Ralph, Paul and Péraire, Cécile , booktitle=. Lessons Learned from an Extended Participant Observation Grounded Theory Study , year=

  3. [3]

    , volume =

    LADDERING THEORY, METHOD, ANALYSIS, AND INTERPRETATION. , volume =. Journal of Advertising Research , author =. 1988 , note =. doi:10.1080/00218499.1988.12467766 , abstract =

    work page doi:10.1080/00218499.1988.12467766 1988

  4. [4]

    2024 , publisher=

    Constructing Grounded Theory , author=. 2024 , publisher=

    2024

  5. [5]

    How Templated Requirements Specifications Inhibit Creativity in Software Engineering , year=

    Mohanani, Rahul and Ralph, Paul and Turhan, Burak and Mandić, Vladimir , journal=. How Templated Requirements Specifications Inhibit Creativity in Software Engineering , year=

  6. [6]

    and Polikarpova, Nadia , title =

    Barke, Shraddha and James, Michael B. and Polikarpova, Nadia , title =. 2023 , issue_date =. doi:10.1145/3586030 , journal =

    doi:10.1145/3586030 2023

  7. [7]

    Myers , title =

    Tuure Tuunanen and Juuli Lumivalo and Tero Vartiainen and Yixin Zhang and Michael D. Myers , title =. Journal of Service Research , volume =. 2024 , doi =

    2024

  8. [8]

    Comprehensive Criteria to Judge Validity and Reliability of Qualitative Research within the Realism Paradigm , volume =

    Healy, Marilyn and Perry, Chad , year =. Comprehensive Criteria to Judge Validity and Reliability of Qualitative Research within the Realism Paradigm , volume =. Qualitative Market Research: An International Journal , doi =

  9. [9]

    2020 , isbn =

    Ferdowsifard, Kasra and Ordookhanians, Allen and Peleg, Hila and Lerner, Sorin and Polikarpova, Nadia , title =. 2020 , isbn =. doi:10.1145/3379337.3415869 , booktitle =

    doi:10.1145/3379337.3415869 2020

  10. [10]

    , title =

    Vaithilingam, Priyan and Zhang, Tianyi and Glassman, Elena L. , title =. 2022 , isbn =. doi:10.1145/3491101.3519665 , booktitle =

    doi:10.1145/3491101.3519665 2022

  11. [11]

    2022 , eprint=

    What is it like to program with artificial intelligence? , author=. 2022 , eprint=

    2022

  12. [12]

    Developer Behaviors in Validating and Repairing

    Tang, Ningzhi and Chen, Meng and Ning, Zheng and Bansal, Aakash and Huang, Yu and McMillan, Collin and Li, Toby Jia-Jun , booktitle=. Developer Behaviors in Validating and Repairing. 2024 , volume=

    2024

  13. [13]

    Advait Sarkar and Xiaotong and Xu and Neil Toronto and Ian Drosos and Christian Poelitz , year=. When. 2412.15030 , archivePrefix=

    Pith/arXiv arXiv

  14. [14]

    Proceedings of the ACM CHI Conference on Human Factors in Computing Systems , year =

    Lee, Hao-Ping (Hank) and Sarkar, Advait and Tankelevitch, Lev and Drosos, Ian and Rintel, Sean and Banks, Richard and Wilson, Nicholas , title =. Proceedings of the ACM CHI Conference on Human Factors in Computing Systems , year =

  15. [15]

    and Yang, Chenyang and Myers, Brad A

    Liang, Jenny T. and Yang, Chenyang and Myers, Brad A. , title =. 2024 , isbn =. doi:10.1145/3597503.3608128 , booktitle =

    doi:10.1145/3597503.3608128 2024

  16. [16]

    1983 , issn =

    Ironies of automation , journal =. 1983 , issn =. doi:https://doi.org/10.1016/0005-1098(83)90046-8 , author =

    doi:10.1016/0005-1098(83)90046-8 1983

  17. [17]

    Ironies of Generative

    Auste Simkute and Lev Tankelevitch and Viktor Kewenig and Ava Elizabeth Scott and Abigail Sellen and Sean Rintel , year=. Ironies of Generative. 2402.11364 , archivePrefix=

    Pith/arXiv arXiv

  18. [18]

    2023 , issue_date =

    Bird, Christian and Ford, Denae and Zimmermann, Thomas and Forsgren, Nicole and Kalliamvakou, Eirini and Lowdermilk, Travis and Gazit, Idan , title =. 2023 , issue_date =. doi:10.1145/3582083 , journal =

    doi:10.1145/3582083 2023

  19. [19]

    2024 , isbn =

    Mozannar, Hussein and Bansal, Gagan and Fourney, Adam and Horvitz, Eric , title =. 2024 , isbn =. doi:10.1145/3613904.3641936 , booktitle =

    doi:10.1145/3613904.3641936 2024

  20. [20]

    2014 , issue_date =

    Maalej, Walid and Tiarks, Rebecca and Roehm, Tobias and Koschke, Rainer , title =. 2014 , issue_date =. doi:10.1145/2622669 , journal =

    doi:10.1145/2622669 2014

  21. [21]

    and Polikarpova, Nadia and Lerner, Sorin , title =

    Ferdowsi, Kasra and Huang, Ruanqianqian (Lisa) and James, Michael B. and Polikarpova, Nadia and Lerner, Sorin , title =. 2024 , isbn =. doi:10.1145/3613904.3642495 , booktitle =

    doi:10.1145/3613904.3642495 2024

  22. [22]

    2024 , isbn =

    Nam, Daye and Macvean, Andrew and Hellendoorn, Vincent and Vasilescu, Bogdan and Myers, Brad , title =. 2024 , isbn =. doi:10.1145/3597503.3639187 , booktitle =

    doi:10.1145/3597503.3639187 2024

  23. [23]

    2023 , isbn =

    Al Madi, Naser , title =. 2023 , isbn =. doi:10.1145/3551349.3560438 , booktitle =

    doi:10.1145/3551349.3560438 2023

  24. [24]

    and Denny, Paul and Becker, Brett A

    Prather, James and Reeves, Brent N. and Denny, Paul and Becker, Brett A. and Leinonen, Juho and Luxton-Reilly, Andrew and Powell, Garrett and Finnie-Ansley, James and Santos, Eddie Antonio , title =. 2023 , issue_date =. doi:10.1145/3617367 , journal =

    doi:10.1145/3617367 2023

  25. [25]

    and Martinez, Fernando and Houde, Stephanie and Muller, Michael and Weisz, Justin D

    Ross, Steven I. and Martinez, Fernando and Houde, Stephanie and Muller, Michael and Weisz, Justin D. , title =. 2023 , isbn =. doi:10.1145/3581641.3584037 , booktitle =

    doi:10.1145/3581641.3584037 2023

  26. [26]

    and Muller, Michael and Houde, Stephanie and Richards, John and Ross, Steven I

    Weisz, Justin D. and Muller, Michael and Houde, Stephanie and Richards, John and Ross, Steven I. and Martinez, Fernando and Agarwal, Mayank and Talamadupula, Kartik , title =. 2021 , isbn =. doi:10.1145/3397481.3450656 , booktitle =

    doi:10.1145/3397481.3450656 2021

  27. [27]

    2406.17325 , archivePrefix=

    Ze Shi Li and Nowshin Nawar Arony and Ahmed Musa Awon and Daniela Damian and Bowen Xu , year=. 2406.17325 , archivePrefix=

    Pith/arXiv arXiv

  28. [28]

    Walking the Tightrope of

    Samuel Ferino and Rashina Hoda and John Grundy and Christoph Treude , year=. Walking the Tightrope of. 2511.06428 , archivePrefix=

    Pith/arXiv arXiv

  29. [29]

    2026 , eprint =

    Shen, Judy Hanwen and Tamkin, Alex , title =. 2026 , eprint =

    2026

  30. [30]

    and Strauss, Anselm L

    Glaser, Barney G. and Strauss, Anselm L. , title =. 1967 , publisher =

    1967

  31. [31]

    Theoretical sensitivity : advances in the methodology of grounded theory , author =

  32. [32]

    2015 , edition =

    Basics of qualitative research : techniques and procedures for developing grounded theory , author =. 2015 , edition =

    2015

  33. [33]

    Ralph, Paul and Baltes, Sebastian and Bianculli, Domenico and Dittrich, Yvonne and Felderer, Michael and Feldt, Robert and Filieri, Antonio and Furia, Carlo and Graziotin, Daniel and He, Pinjia and Hoda, Rashina and Juristo, Natalia and Kitchenham, Barbara and Robbes, Romain and Méndez Fernández, Daniel and Molleri, Jefferson and Spinellis, Diomidis and S...

  34. [34]

    How Do Programmers Evaluate AI-Generated Code? , year =

    M\". How Do Programmers Evaluate AI-Generated Code? , year =. doi:10.1109/ESEM64174.2025.00057 , booktitle =

    doi:10.1109/esem64174.2025.00057 2025

This paper was first reviewed by grok-4.5 on July 13, 2026.