REVIEW 4 major objections 5 minor 33 references
Non-programmers Assessing AI-Generated Code: A Case Study of Business Users Analyzing Data
T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Skeptical, incentivized business professionals cannot reliably verify AI-generated data analyses on their own.
desk verdict Careful small-n study showing business non-programmers miss flaws in AI-generated analyses; the result is plausible and important, but the hand-curated stimuli and author-defined flaw labels need validation before the strong 'cannot reliably verify' claim is taken at face value. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The study uses a two-phase LLM pipeline: the model first generates and iteratively executes code until it runs without error, then produces a natural-language 'Methods and Results' explanation of its approach and output. The authors manually review and adjust these explanations to ensure they faithfully describe the code and remain accessible to non-technical readers, then present them to participants alongside a shared 27-week marketing dataset and eight realistic tasks modeled on business users' stated workflows. The second survey adds a within-subject manipulation—formatted step lists and per-step alternatives—to test why flaws are missed: AI decisions are not salient, and alternative dec
What would settle it
Rerun the eight tasks with a fresh group of marketing and sales professionals, presenting unedited LLM output for the same code without manual adjustment, and have independent data-science experts label the flaws without the authors' input. If participants catch most expert-agreed safety-critical flaws in that setting, the central claim of unreliable verification would not survive.
Extended reading notes
Core claim
On its own terms, the paper's central finding is that domain expertise and general critical thinking do not transfer reliably to the unfamiliar technical context of evaluating an AI's approach. Participants could identify valid flaws, but no single flaw was consistently detected, and some safety-critical issues were missed by all. The authors attribute the failures not to overconfidence, since distrust was explicitly primed and incentivized, but to difficulty applying domain knowledge inside a technical frame and, in some cases, low tolerance for engaging deeply with methodological detail. The second survey shows that clearly delineated steps and explicitly presented alternatives help, but i
Load-bearing premise
The study's detection rates rest on the assumption that the flaws the authors identified in each AI response are genuinely flaws, and that the manually cleaned and adjusted explanations shown to participants faithfully represent what real LLM outputs would look like.
Editorial extensions
If this is right
- Organizations should not treat an end-user's self-verification of LLM-generated analysis as sufficient for consequential business decisions.
- External verification by someone with data-science skills becomes the default safety requirement for business-facing AI analysis tools.
- Explanation design can be improved by making each AI decision a distinct, labeled step and by supplying alternatives, but this is not enough for harder flaws.
- The results raise the reliability bar for AI: adoption by non-programmers may remain unsafe unless model error rates drop dramatically.
- Wide individual differences in verification ability suggest that training, skill-building, or adaptive explanation modes may be needed rather than a single presentation strategy.
Reading between the lines
- A direct testable extension would rerun the exact tasks with unedited, raw LLM outputs and independent expert adjudication of flaws; if detection rates rise substantially, the central claim may be partly an artifact of curated stimuli.
- The paper's assumption that participants have relevant domain expertise could be probed by formally measuring marketing knowledge and testing whether it correlates with flaw detection.
- The positive effect of alternatives on several tasks hints at a 'need for cognition' moderator; measuring cognitive reflection might predict who benefits from step-and-alternative formats.
- The finding bears on the broader question of human oversight of AI: oversight cannot be assumed to emerge naturally but must be designed with the same rigor as the AI system itself.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports two surveys of marketing and sales professionals who, despite warnings, incentives, and explicit prompts to critique AI-generated analyses, frequently failed to detect flaws in natural-language explanations of LLM-generated code. The first survey (n=10) used eight realistic data-analysis tasks with author-identified flaws; participants detected some valid flaws but none consistently, and several safety-critical flaws were missed by all. The second survey (n=18) tested whether reformatting explanations into clear steps and adding alternative decisions improves detection. The authors report positive but inconsistent effects. They conclude that business non-programmers cannot reliably verify AI-generated data analyses on their own, and discuss design implications for explanations that better support critical engagement.
Significance. If the central finding holds, it is consequential for HCI and AI safety: it challenges the assumption that domain experts can catch LLM mistakes in data analysis without programming or data-science expertise. The study is thoughtfully designed in several ways: participants were primed to distrust the AI, monetarily incentivized, required to restate the AI's methods, and presented with tasks grounded in real business workflows. The qualitative analysis is plausible and yields specific behavioral categories. However, the paper's load-bearing evidence depends on an unvalidated flaw set and on author-edited explanations, and the samples are small with no inferential statistics for the intervention comparison. With additional validation and more cautious framing, this could be a useful case study for the community; in its current form, the strength of the conclusion exceeds what the evidence supports.
major comments (4)
- [Section III, 'LLM-Generated Responses'] The stimuli shown to participants were not raw LLM outputs: 'All explanations are manually reviewed and, when necessary, adjusted by the authors to ensure they faithfully and comprehensively describe the generated code.' This is a load-bearing validity issue. Detection rates in Sections IV-B and V-B measure performance on author-edited artifacts, not on actual LLM explanations. The paper provides no comparison between original and adjusted text, no check that edits did not alter the salience of flaws, and no inter-rater reliability for the editing process. Without such evidence, the conclusion in Section VIII that business professionals cannot verify AI-generated analyses does not generalize to real deployments. This can be addressed by showing a sample of original versus adjusted explanations, or by running a control condition with unedited outputs.
- [Table I and Section III, 'Categories of Flaws'] The ground-truth flaw set is author-defined with no independent validation or inter-rater reliability. Several labels are contestable. For Task 2, 'Difficult to interpret Mean Squared Error to evaluate model quality' is classified as safety-critical but is arguably a presentation issue rather than a flaw that directly causes unsound decisions. For Task 1, 'Does not add variables for lagged effect of marketing efforts' may be a defensible modeling choice rather than an error. If these are not true flaws, then participants' failure to detect them does not demonstrate a verification deficit. The authors should have independent data-science experts validate the flaw labels, and should separately report detection rates for uncontroversial safety-critical flaws (e.g., Task 3, Task 4, Task 8).
- [Section V-B and Figure 5] The second study's comparison between formatted responses and formatted-plus-alternatives is presented without any inferential statistics, effect sizes, or confidence intervals. With n=18 and only 9 participants per condition-task pair, the differences in Figure 5 could easily be sampling noise. The text says the effects are 'positive but inconsistent,' but no quantitative support is given. At minimum, the authors should report per-task counts, paired tests or exact CIs, and effect sizes. This is especially important because the second study is a stated contribution of the paper.
- [Section IV-B.1 and Figure 3] The claim that participants 'frequently failed to detect critical flaws' is based on 10 participants. While the qualitative observations are informative, the quantitative statement that 'none were consistently detected' is weak with n=10; a flaw missed by 9 of 10 is not statistically distinguishable from one missed by all. The paper should temper the generalization in Section VIII or provide additional evidence that the detection failure is robust across the participant pool.
minor comments (5)
- [Section V-B] The participant quote contains a typo: 'Many decison-makers would like' should be 'decision-makers.' Please proofread.
- [Section IV-B.5] The quote 'testing information should be use in a higher % than the training information' should be cleaned up or presented as [sic] to indicate the grammar is the participant's.
- [Throughout] The model identifier 'Llama-3.1-70B-Instruct-Turbo' should include a version/date or API reference so that results are reproducible; also clarify whether the same model was used for both coding and explanation phases and for generating alternatives.
- [Section III and Table I] The definition of 'safety-critical' in the text says a flaw could lead to poor or unsound decisions, but Table I's Task 5 example ('Determines influential factors outside of dataset exist but does not discuss them') is labeled not safety-critical. This distinction should be briefly justified so readers understand the boundary.
- [Section V-A] The sentence 'To draw a balanced comparison, we randomly sample 9 participants from the preliminary study' is confusing because the second survey recruits 18 new participants. Clarify whether the comparison is between the new 18 and a random 9 of the original 10, and what 'balanced' means here.
Circularity Check
No significant circularity: the study is an empirical measurement, not a derivation that reduces to its inputs.
full rationale
The paper makes no formal derivation or fitting-based prediction. Its central claim—that skeptical, motivated business professionals frequently fail to detect author-identified flaws in LLM-generated analyses—is an empirical result obtained from two surveys. The tasks were built to contain known flaws, and the outcome measure is participants' failure to identify those flaws; this is the intended experimental design, not a conclusion built into the input. The flaw labels are author-defined, and the AI explanations were manually reviewed and adjusted, which raises validity concerns about stimulus representativeness, but this is a measurement/ground-truth issue, not circularity: the paper does not claim to derive the flaw labels from participants' responses or to predict detection rates from the same data used to fit them. No load-bearing self-citations were found; references to prior work are external and do not constitute the argument's foundation. The dataset is sourced externally [23], tasks are derived from prior business-user workflow research [9], and participants were screened and incentivized. Therefore, the appropriate circularity score is minimal (1).
Assumptions & free parameters
assumptions (4)
- domain assumption Author-defined flaws (Table I) constitute valid shortcomings that a business user should detect.
- ad hoc to paper The natural-language explanations, manually reviewed and adjusted by the authors, faithfully and comprehensively describe the AI's code and results.
- domain assumption Screened Prolific participants are representative of business professionals who use data for decisions.
- domain assumption A survey context with warnings and a bonus simulates real-world motivation to verify AI output.
Cite this review
Pith. "Pith review of Non-programmers Assessing AI-Generated Code: A Case Study of Business Users Analyzing Data." pith.science (2026). https://pith.science/paper/QRZ2A24V
@misc{pith2026250806484,
author = {Pith},
title = {Pith review of: Non-programmers Assessing AI-Generated Code: A Case Study of Business Users Analyzing Data},
year = {2026},
howpublished = {\url{https://pith.science/paper/QRZ2A24V}},
note = {Machine review of arXiv:2508.06484}
}
abstract
Non-technical end-users increasingly rely on AI code generation to perform technical tasks like data analysis. However, large language models (LLMs) remain unreliable, and it is unclear whether end-users can effectively identify model errors $\unicode{x2014}$ especially in realistic and domain-specific scenarios. We surveyed marketing and sales professionals to assess their ability to critically evaluate LLM-generated analyses of marketing data. Participants were shown natural language explanations of the AI's code, repeatedly informed the AI often makes mistakes, and explicitly prompted to identify them. Yet, participants frequently failed to detect critical flaws that could compromise decision-making, many of which required no technical knowledge to recognize. To investigate why, we reformatted AI responses into clearly delineated steps and provided alternative approaches for each decision to support critical evaluation. While these changes had a positive effect, participants often struggled to reason through the AI's steps and alternatives. Our findings suggest that business professionals cannot reliably verify AI-generated data analyses on their own and explore reasons why to inform future designs. As non-programmers adopt code-generating AI for technical tasks, unreliable AI and insufficient human oversight poses risks of unsafe or low-quality decisions.
Figures
Reference graph
Works this paper leans on
-
[1]
Crowdgenui: Enhancing llm-based ui widget generation with a crowdsourced preference library,
Y . Liu, M. Sra, and C. Xiao, “Crowdgenui: Enhancing llm-based ui widget generation with a crowdsourced preference library,” 2024. [Online]. Available: https://arxiv.org/abs/2411.03477
arXiv 2024
-
[2]
Zero-shot prompting approaches for llm-based graphical user interface generation,
K. Kolthoff, F. Kretzer, L. Fiebig, C. Bartelt, A. Maedche, and S. P. Ponzetto, “Zero-shot prompting approaches for llm-based graphical user interface generation,” 2024. [Online]. Available: https://arxiv.org/abs/2412.11328
arXiv 2024
-
[3]
M. X. Liu, A. Sarkar, C. Negreanu, B. Zorn, J. Williams, N. Toronto, and A. D. Gordon, ““what it wants me to say”: Bridging the abstraction gap between end-user programmers and code-generating large language models,” in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems , ser. CHI ’23. New York, NY , USA: Association for Computin...
arXiv 2023
-
[4]
InsightPilot: An LLM-empowered automated data exploration system,
P. Ma, R. Ding, S. Wang, S. Han, and D. Zhang, “InsightPilot: An LLM-empowered automated data exploration system,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , Y . Feng and E. Lefever, Eds. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 346–352. [Online]. Available...
work page 2023
-
[5]
Data interpreter: An llm agent for data science,
S. Hong, Y . Lin, B. Liu, B. Liu, B. Wu, C. Zhang, C. Wei, D. Li, J. Chen, J. Zhang, J. Wang, L. Zhang, L. Zhang, M. Yang, M. Zhuge, T. Guo, T. Zhou, W. Tao, X. Tang, X. Lu, X. Zheng, X. Liang, Y . Fei, Y . Cheng, Z. Gou, Z. Xu, and C. Wu, “Data interpreter: An llm agent for data science,” 2024. [Online]. Available: https://arxiv.org/abs/2402.18679
arXiv 2024
-
[6]
Ds- agent: Automated data science by empowering large language models with case-based reasoning,
S. Guo, C. Deng, Y . Wen, H. Chen, Y . Chang, and J. Wang, “Ds- agent: Automated data science by empowering large language models with case-based reasoning,” 2024. [Online]. Available: https://arxiv.org/abs/2402.17453
arXiv 2024
-
[7]
Alchemist: Llm-aided end-user development of robot applications,
U. B. Karli, J.-T. Chen, V . N. Antony, and C.-M. Huang, “Alchemist: Llm-aided end-user development of robot applications,” in Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction, ser. HRI ’24. New York, NY , USA: Association for Computing Machinery, 2024, p. 361–370. [Online]. Available: https://doi.org/10.1145/3610977.3634969
arXiv 2024
-
[8]
Cocobo: Exploring large language models as the engine for end-user robot programming,
Y . Ge, Y . Dai, R. Shan, K. Li, Y . Hu, and X. Sun, “Cocobo: Exploring large language models as the engine for end-user robot programming,” in 2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) , 2024, pp. 89–95
work page 2024
Show all 33 references
-
[9]
What-if analysis for business users: Current practices and future opportunities,
S. Gathani, Z. Liu, P. J. Haas, and C ¸ . Demiralp, “What-if analysis for business users: Current practices and future opportunities,” Conference on Human Factors in Computing Systems , 2022
2022
-
[10]
Benchmarking data science agents,
Y . Zhang, Q. Jiang, X. XingyuHan, N. Chen, Y . Yang, and K. Ren, “Benchmarking data science agents,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), L.-W. Ku, A. Martins, and V . Srikumar, Eds. Bangkok, Thai...
2024
-
[11]
Ds-1000: A natural and reliable benchmark for data science code generation,
Y . Lai, C. Li, Y . Wang, T. Zhang, R. Zhong, L. Zettlemoyer, S. W. tau Yih, D. Fried, S. Wang, and T. Yu, “Ds-1000: A natural and reliable benchmark for data science code generation,” ArXiv, vol. abs/2211.11501, 2022
2022 arXiv
-
[12]
Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models,
P. Vaithilingam, T. Zhang, and E. L. Glassman, “Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models,” in Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems , ser. CHI EA ’22. New York, ...
2022
-
[13]
Github copilot ai pair programmer: Asset or liability?
A. Moradi Dakhel, V . Majdinasab, A. Nikanjam, F. Khomh, M. C. Desmarais, and Z. M. J. Jiang, “Github copilot ai pair programmer: Asset or liability?” J. Syst. Softw. , vol. 203, no. C, Sep. 2023. [Online]. Available: https://doi.org/10.1016/j.jss.2023.111734
2023
-
[14]
Non-expert programmers in the generative ai future,
M. Q. Feldman and C. J. Anderson, “Non-expert programmers in the generative ai future,” in Proceedings of the 3rd Annual Meeting of the Symposium on Human-Computer Interaction for Work , ser. CHIWORK ’24. New York, NY , USA: Association for Computing Machinery,
-
[15]
Pwr: Exploring the role of representations in conversational programming,
P. YM, V . Ganesan, D. K. Arumugam, M. Gupta, N. Shadagopan, T. Dixit, S. Segal, P. Kumar, M. Jain, and S. Rajamani, “Pwr: Exploring the role of representations in conversational programming,”
-
[16]
Intelliexplain: Enhancing conversational code generation for non-professional programmers,
H. Yan, T. D. Latoza, and Z. Yao, “Intelliexplain: Enhancing conversational code generation for non-professional programmers,”
-
[17]
COLDECO: An End User Spreadsheet Inspection Tool for AI-Generated Code ,
K. Ferdowsi, J. Williams, I. Drosos, A. D. Gordon, C. Negreanu, N. Polikarpova, A. Sarkar, and B. Zorn, “ COLDECO: An End User Spreadsheet Inspection Tool for AI-Generated Code ,” in 2023 IEEE Symposium on Visual Languages and Human- Centric Computing (VL/HCC) . Los Alamitos, ...
2023
-
[18]
Does the whole exceed its parts? the effect of ai explanations on complementary team performance,
G. Bansal, T. Wu, J. Zhou, R. Fok, B. Nushi, E. Kamar, M. T. Ribeiro, and D. Weld, “Does the whole exceed its parts? the effect of ai explanations on complementary team performance,” in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems , ser. CHI ’21...
2021
-
[19]
To rely or not to rely? evaluating interventions for appropriate reliance on large language models,
J. Y . Bo, S. Wan, and A. Anderson, “To rely or not to rely? evaluating interventions for appropriate reliance on large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2412.15584
2024
-
[20]
Available: https://arxiv.org/abs/2405.10250
[Online]. Available: https://arxiv.org/abs/2405.10250
-
[21]
Intercode: Standardizing and benchmarking interactive coding with execution feedback,
J. Yang, A. Prabhakar, K. R. Narasimhan, and S. Yao, “Intercode: Standardizing and benchmarking interactive coding with execution feedback,” in Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2023. [Online]. Available: https:/...
2023
-
[22]
Lever: Learning to verify language-to-code generation with execution,
A. Ni, S. Iyer, D. Radev, V . Stoyanov, W. tau Yih, S. I. Wang, and X. V . Lin, “Lever: Learning to verify language-to-code generation with execution,” 2023. [Online]. Available: https://arxiv.org/abs/2302.08468
2023 arXiv
-
[23]
Marketing mix model: In- sights and optimization,
L. Analytics, “Marketing mix model: In- sights and optimization,” 2019, accessed: 2025- 01-04. [Online]. Available: https://www.latentview.com/wp- content/uploads/2019/08/Marketing-Mix-Model.pdf
2019
-
[24]
The state of the art in end-user software engineering,
A. J. Ko, R. Abraham, L. Beckwith, A. Blackwell, M. Burnett, M. Erwig, C. Scaffidi, J. Lawrance, H. Lieberman, B. Myers, M. B. Rosson, G. Rothermel, M. Shaw, and S. Wiedenbeck, “The state of the art in end-user software engineering,” ACM Comput. Surv., vol. 43, no. 3, Apr
-
[25]
On mitigating code llm hallucinations with api documentation,
N. Jain, R. Kwiatkowski, B. Ray, M. K. Ramanathan, and V . Kumar, “On mitigating code llm hallucinations with api documentation,” 2024. [Online]. Available: https://arxiv.org/abs/2407.09726
2024 arXiv
-
[26]
Prolific,
Prolific, “Prolific,” https://www.prolific.com, 2014
2014
-
[27]
Statistical methods for research workers,
R. A. Fisher, “Statistical methods for research workers,” in Break- throughs in statistics: Methodology and distribution . Springer, 1970, pp. 66–70
1970
-
[31]
Young people’s descriptions of computational rules in role-playing games: An empirical study,
J. Good, K. Howland, and K. Nicholson, “Young people’s descriptions of computational rules in role-playing games: An empirical study,” in Proceedings of the 2010 IEEE Symposium on Visual Languages and Human-Centric Computing , ser. VLHCC ’10. USA: IEEE Computer Society, 2010, ...
2010 doi
-
[2011]
Available: https://doi.org/10.1145/1922649.1922658
[Online]. Available: https://doi.org/10.1145/1922649.1922658
-
[2021]
Available: https://doi.org/10.1145/3411764.3445717
[Online]. Available: https://doi.org/10.1145/3411764.3445717
-
[2022]
Available: https://doi.org/10.1145/3491101.3519665
[Online]. Available: https://doi.org/10.1145/3491101.3519665
-
[2023]
Available: https://arxiv.org/abs/2309.09495
[Online]. Available: https://arxiv.org/abs/2309.09495
-
[2024]
Available: https://doi.org/10.1145/3663384.3663393
[Online]. Available: https://doi.org/10.1145/3663384.3663393
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.