Pith. sign in

REVIEW 4 major objections 5 minor 33 references

Non-programmers Assessing AI-Generated Code: A Case Study of Business Users Analyzing Data

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Pith's one-line read Skeptical, incentivized business professionals cannot reliably verify AI-generated data analyses on their own.

desk verdict Careful small-n study showing business non-programmers miss flaws in AI-generated analyses; the result is plausible and important, but the hand-curated stimuli and author-defined flaw labels need validation before the strong 'cannot reliably verify' claim is taken at face value. read the letter →

arxiv 2508.06484 v1 pith:QRZ2A24V submitted 2025-08-08 cs.HC

classification cs.HC
keywords end-userprogramminglargelanguagemodelscodegenerationdataanalysisverifiabilitybusinessprofessionalshuman-AIinteractioncriticalevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that non-technical business professionals cannot be relied on to verify AI-generated data analyses. In two surveys, marketing and sales professionals were shown natural-language explanations of LLM-generated analyses, warned that the AI often errs, prompted to find mistakes, and offered a bonus for doing so, yet they frequently missed flaws that could compromise decisions, including ones requiring no technical knowledge. Restructuring responses into clear steps and offering alternative approaches improved detection of simple flaws but did not make evaluation reliable. The authors conclude that safe use of code-generating AI by non-programmers requires either substantially more reliable models or explanations designed to help users engage critically with technical context.

What carries the argument

The study uses a two-phase LLM pipeline: the model first generates and iteratively executes code until it runs without error, then produces a natural-language 'Methods and Results' explanation of its approach and output. The authors manually review and adjust these explanations to ensure they faithfully describe the code and remain accessible to non-technical readers, then present them to participants alongside a shared 27-week marketing dataset and eight realistic tasks modeled on business users' stated workflows. The second survey adds a within-subject manipulation—formatted step lists and per-step alternatives—to test why flaws are missed: AI decisions are not salient, and alternative dec

What would settle it

Rerun the eight tasks with a fresh group of marketing and sales professionals, presenting unedited LLM output for the same code without manual adjustment, and have independent data-science experts label the flaws without the authors' input. If participants catch most expert-agreed safety-critical flaws in that setting, the central claim of unreliable verification would not survive.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central finding is that domain expertise and general critical thinking do not transfer reliably to the unfamiliar technical context of evaluating an AI's approach. Participants could identify valid flaws, but no single flaw was consistently detected, and some safety-critical issues were missed by all. The authors attribute the failures not to overconfidence, since distrust was explicitly primed and incentivized, but to difficulty applying domain knowledge inside a technical frame and, in some cases, low tolerance for engaging deeply with methodological detail. The second survey shows that clearly delineated steps and explicitly presented alternatives help, but i

Load-bearing premise

The study's detection rates rest on the assumption that the flaws the authors identified in each AI response are genuinely flaws, and that the manually cleaned and adjusted explanations shown to participants faithfully represent what real LLM outputs would look like.

Editorial extensions

If this is right

  • Organizations should not treat an end-user's self-verification of LLM-generated analysis as sufficient for consequential business decisions.
  • External verification by someone with data-science skills becomes the default safety requirement for business-facing AI analysis tools.
  • Explanation design can be improved by making each AI decision a distinct, labeled step and by supplying alternatives, but this is not enough for harder flaws.
  • The results raise the reliability bar for AI: adoption by non-programmers may remain unsafe unless model error rates drop dramatically.
  • Wide individual differences in verification ability suggest that training, skill-building, or adaptive explanation modes may be needed rather than a single presentation strategy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct testable extension would rerun the exact tasks with unedited, raw LLM outputs and independent expert adjudication of flaws; if detection rates rise substantially, the central claim may be partly an artifact of curated stimuli.
  • The paper's assumption that participants have relevant domain expertise could be probed by formally measuring marketing knowledge and testing whether it correlates with flaw detection.
  • The positive effect of alternatives on several tasks hints at a 'need for cognition' moderator; measuring cognitive reflection might predict who benefits from step-and-alternative formats.
  • The finding bears on the broader question of human oversight of AI: oversight cannot be assumed to emerge naturally but must be designed with the same rigor as the AI system itself.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper reports two surveys of marketing and sales professionals who, despite warnings, incentives, and explicit prompts to critique AI-generated analyses, frequently failed to detect flaws in natural-language explanations of LLM-generated code. The first survey (n=10) used eight realistic data-analysis tasks with author-identified flaws; participants detected some valid flaws but none consistently, and several safety-critical flaws were missed by all. The second survey (n=18) tested whether reformatting explanations into clear steps and adding alternative decisions improves detection. The authors report positive but inconsistent effects. They conclude that business non-programmers cannot reliably verify AI-generated data analyses on their own, and discuss design implications for explanations that better support critical engagement.

Significance. If the central finding holds, it is consequential for HCI and AI safety: it challenges the assumption that domain experts can catch LLM mistakes in data analysis without programming or data-science expertise. The study is thoughtfully designed in several ways: participants were primed to distrust the AI, monetarily incentivized, required to restate the AI's methods, and presented with tasks grounded in real business workflows. The qualitative analysis is plausible and yields specific behavioral categories. However, the paper's load-bearing evidence depends on an unvalidated flaw set and on author-edited explanations, and the samples are small with no inferential statistics for the intervention comparison. With additional validation and more cautious framing, this could be a useful case study for the community; in its current form, the strength of the conclusion exceeds what the evidence supports.

major comments (4)
  1. [Section III, 'LLM-Generated Responses'] The stimuli shown to participants were not raw LLM outputs: 'All explanations are manually reviewed and, when necessary, adjusted by the authors to ensure they faithfully and comprehensively describe the generated code.' This is a load-bearing validity issue. Detection rates in Sections IV-B and V-B measure performance on author-edited artifacts, not on actual LLM explanations. The paper provides no comparison between original and adjusted text, no check that edits did not alter the salience of flaws, and no inter-rater reliability for the editing process. Without such evidence, the conclusion in Section VIII that business professionals cannot verify AI-generated analyses does not generalize to real deployments. This can be addressed by showing a sample of original versus adjusted explanations, or by running a control condition with unedited outputs.
  2. [Table I and Section III, 'Categories of Flaws'] The ground-truth flaw set is author-defined with no independent validation or inter-rater reliability. Several labels are contestable. For Task 2, 'Difficult to interpret Mean Squared Error to evaluate model quality' is classified as safety-critical but is arguably a presentation issue rather than a flaw that directly causes unsound decisions. For Task 1, 'Does not add variables for lagged effect of marketing efforts' may be a defensible modeling choice rather than an error. If these are not true flaws, then participants' failure to detect them does not demonstrate a verification deficit. The authors should have independent data-science experts validate the flaw labels, and should separately report detection rates for uncontroversial safety-critical flaws (e.g., Task 3, Task 4, Task 8).
  3. [Section V-B and Figure 5] The second study's comparison between formatted responses and formatted-plus-alternatives is presented without any inferential statistics, effect sizes, or confidence intervals. With n=18 and only 9 participants per condition-task pair, the differences in Figure 5 could easily be sampling noise. The text says the effects are 'positive but inconsistent,' but no quantitative support is given. At minimum, the authors should report per-task counts, paired tests or exact CIs, and effect sizes. This is especially important because the second study is a stated contribution of the paper.
  4. [Section IV-B.1 and Figure 3] The claim that participants 'frequently failed to detect critical flaws' is based on 10 participants. While the qualitative observations are informative, the quantitative statement that 'none were consistently detected' is weak with n=10; a flaw missed by 9 of 10 is not statistically distinguishable from one missed by all. The paper should temper the generalization in Section VIII or provide additional evidence that the detection failure is robust across the participant pool.
minor comments (5)
  1. [Section V-B] The participant quote contains a typo: 'Many decison-makers would like' should be 'decision-makers.' Please proofread.
  2. [Section IV-B.5] The quote 'testing information should be use in a higher % than the training information' should be cleaned up or presented as [sic] to indicate the grammar is the participant's.
  3. [Throughout] The model identifier 'Llama-3.1-70B-Instruct-Turbo' should include a version/date or API reference so that results are reproducible; also clarify whether the same model was used for both coding and explanation phases and for generating alternatives.
  4. [Section III and Table I] The definition of 'safety-critical' in the text says a flaw could lead to poor or unsound decisions, but Table I's Task 5 example ('Determines influential factors outside of dataset exist but does not discuss them') is labeled not safety-critical. This distinction should be briefly justified so readers understand the boundary.
  5. [Section V-A] The sentence 'To draw a balanced comparison, we randomly sample 9 participants from the preliminary study' is confusing because the second survey recruits 18 new participants. Clarify whether the comparison is between the new 18 and a random 9 of the original 10, and what 'balanced' means here.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the study is an empirical measurement, not a derivation that reduces to its inputs.

full rationale

The paper makes no formal derivation or fitting-based prediction. Its central claim—that skeptical, motivated business professionals frequently fail to detect author-identified flaws in LLM-generated analyses—is an empirical result obtained from two surveys. The tasks were built to contain known flaws, and the outcome measure is participants' failure to identify those flaws; this is the intended experimental design, not a conclusion built into the input. The flaw labels are author-defined, and the AI explanations were manually reviewed and adjusted, which raises validity concerns about stimulus representativeness, but this is a measurement/ground-truth issue, not circularity: the paper does not claim to derive the flaw labels from participants' responses or to predict detection rates from the same data used to fit them. No load-bearing self-citations were found; references to prior work are external and do not constitute the argument's foundation. The dataset is sourced externally [23], tasks are derived from prior business-user workflow research [9], and participants were screened and incentivized. Therefore, the appropriate circularity score is minimal (1).

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The study adds no fitted parameters or invented entities. Its conclusions depend on the validity of the flaw labels, the faithfulness of the curated explanations, and the generalizability of small Prolific samples.

assumptions (4)
  • domain assumption Author-defined flaws (Table I) constitute valid shortcomings that a business user should detect.
    Ground truth is established by the authors without inter-rater reliability or independent validation; detection rates are computed against these labels.
  • ad hoc to paper The natural-language explanations, manually reviewed and adjusted by the authors, faithfully and comprehensively describe the AI's code and results.
    Section III states 'All explanations are manually reviewed and, when necessary, adjusted by the authors to ensure they faithfully and comprehensively describe the generated code.' This curation is specific to this study and may not represent real LLM explanations.
  • domain assumption Screened Prolific participants are representative of business professionals who use data for decisions.
    The samples (n=10, n=18) are small and self-selected; the paper acknowledges limited generalizability in Section VII.
  • domain assumption A survey context with warnings and a bonus simulates real-world motivation to verify AI output.
    The authors note participants might behave differently in their usual work environments; the claim of 'cannot reliably verify' assumes transferability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Non-programmers Assessing AI-Generated Code: A Case Study of Business Users Analyzing Data." pith.science (2026). https://pith.science/paper/QRZ2A24V

@misc{pith2026250806484,
  author       = {Pith},
  title        = {Pith review of: Non-programmers Assessing AI-Generated Code: A Case Study of Business Users Analyzing Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QRZ2A24V}},
  note         = {Machine review of arXiv:2508.06484}
}
abstract

Non-technical end-users increasingly rely on AI code generation to perform technical tasks like data analysis. However, large language models (LLMs) remain unreliable, and it is unclear whether end-users can effectively identify model errors $\unicode{x2014}$ especially in realistic and domain-specific scenarios. We surveyed marketing and sales professionals to assess their ability to critically evaluate LLM-generated analyses of marketing data. Participants were shown natural language explanations of the AI's code, repeatedly informed the AI often makes mistakes, and explicitly prompted to identify them. Yet, participants frequently failed to detect critical flaws that could compromise decision-making, many of which required no technical knowledge to recognize. To investigate why, we reformatted AI responses into clearly delineated steps and provided alternative approaches for each decision to support critical evaluation. While these changes had a positive effect, participants often struggled to reason through the AI's steps and alternatives. Our findings suggest that business professionals cannot reliably verify AI-generated data analyses on their own and explore reasons why to inform future designs. As non-programmers adopt code-generating AI for technical tasks, unreliable AI and insufficient human oversight poses risks of unsafe or low-quality decisions.

Figures

Figures reproduced from arXiv: 2508.06484 by the authors.

Figure 1
Figure 1. The end-user describes a task, the AI generates and [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. When prompted to select a model for predicting sales, the LLM generates and executes the code on the left. It generates [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Total participants who identified each flaw. All flaws [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Enhancement of preliminary response in Figure 2. It [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Total number of flaws identified by participants under [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 16 canonical work pages

  1. [1]

    Crowdgenui: Enhancing llm-based ui widget generation with a crowdsourced preference library,

    Y . Liu, M. Sra, and C. Xiao, “Crowdgenui: Enhancing llm-based ui widget generation with a crowdsourced preference library,” 2024. [Online]. Available: https://arxiv.org/abs/2411.03477

  2. [2]

    Zero-shot prompting approaches for llm-based graphical user interface generation,

    K. Kolthoff, F. Kretzer, L. Fiebig, C. Bartelt, A. Maedche, and S. P. Ponzetto, “Zero-shot prompting approaches for llm-based graphical user interface generation,” 2024. [Online]. Available: https://arxiv.org/abs/2412.11328

  3. [3]

    “what it wants me to say

    M. X. Liu, A. Sarkar, C. Negreanu, B. Zorn, J. Williams, N. Toronto, and A. D. Gordon, ““what it wants me to say”: Bridging the abstraction gap between end-user programmers and code-generating large language models,” in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems , ser. CHI ’23. New York, NY , USA: Association for Computin...

  4. [4]

    InsightPilot: An LLM-empowered automated data exploration system,

    P. Ma, R. Ding, S. Wang, S. Han, and D. Zhang, “InsightPilot: An LLM-empowered automated data exploration system,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , Y . Feng and E. Lefever, Eds. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 346–352. [Online]. Available...

  5. [5]

    Data interpreter: An llm agent for data science,

    S. Hong, Y . Lin, B. Liu, B. Liu, B. Wu, C. Zhang, C. Wei, D. Li, J. Chen, J. Zhang, J. Wang, L. Zhang, L. Zhang, M. Yang, M. Zhuge, T. Guo, T. Zhou, W. Tao, X. Tang, X. Lu, X. Zheng, X. Liang, Y . Fei, Y . Cheng, Z. Gou, Z. Xu, and C. Wu, “Data interpreter: An llm agent for data science,” 2024. [Online]. Available: https://arxiv.org/abs/2402.18679

  6. [6]

    Ds- agent: Automated data science by empowering large language models with case-based reasoning,

    S. Guo, C. Deng, Y . Wen, H. Chen, Y . Chang, and J. Wang, “Ds- agent: Automated data science by empowering large language models with case-based reasoning,” 2024. [Online]. Available: https://arxiv.org/abs/2402.17453

  7. [7]

    Alchemist: Llm-aided end-user development of robot applications,

    U. B. Karli, J.-T. Chen, V . N. Antony, and C.-M. Huang, “Alchemist: Llm-aided end-user development of robot applications,” in Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction, ser. HRI ’24. New York, NY , USA: Association for Computing Machinery, 2024, p. 361–370. [Online]. Available: https://doi.org/10.1145/3610977.3634969

  8. [8]

    Cocobo: Exploring large language models as the engine for end-user robot programming,

    Y . Ge, Y . Dai, R. Shan, K. Li, Y . Hu, and X. Sun, “Cocobo: Exploring large language models as the engine for end-user robot programming,” in 2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) , 2024, pp. 89–95

Show all 33 references
  1. [9]

    What-if analysis for business users: Current practices and future opportunities,

    S. Gathani, Z. Liu, P. J. Haas, and C ¸ . Demiralp, “What-if analysis for business users: Current practices and future opportunities,” Conference on Human Factors in Computing Systems , 2022

  2. [10]

    Benchmarking data science agents,

    Y . Zhang, Q. Jiang, X. XingyuHan, N. Chen, Y . Yang, and K. Ren, “Benchmarking data science agents,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), L.-W. Ku, A. Martins, and V . Srikumar, Eds. Bangkok, Thai...

  3. [11]

    Ds-1000: A natural and reliable benchmark for data science code generation,

    Y . Lai, C. Li, Y . Wang, T. Zhang, R. Zhong, L. Zettlemoyer, S. W. tau Yih, D. Fried, S. Wang, and T. Yu, “Ds-1000: A natural and reliable benchmark for data science code generation,” ArXiv, vol. abs/2211.11501, 2022

  4. [12]

    Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models,

    P. Vaithilingam, T. Zhang, and E. L. Glassman, “Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models,” in Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems , ser. CHI EA ’22. New York, ...

  5. [13]

    Github copilot ai pair programmer: Asset or liability?

    A. Moradi Dakhel, V . Majdinasab, A. Nikanjam, F. Khomh, M. C. Desmarais, and Z. M. J. Jiang, “Github copilot ai pair programmer: Asset or liability?” J. Syst. Softw. , vol. 203, no. C, Sep. 2023. [Online]. Available: https://doi.org/10.1016/j.jss.2023.111734

  6. [14]

    Non-expert programmers in the generative ai future,

    M. Q. Feldman and C. J. Anderson, “Non-expert programmers in the generative ai future,” in Proceedings of the 3rd Annual Meeting of the Symposium on Human-Computer Interaction for Work , ser. CHIWORK ’24. New York, NY , USA: Association for Computing Machinery,

  7. [15]

    Pwr: Exploring the role of representations in conversational programming,

    P. YM, V . Ganesan, D. K. Arumugam, M. Gupta, N. Shadagopan, T. Dixit, S. Segal, P. Kumar, M. Jain, and S. Rajamani, “Pwr: Exploring the role of representations in conversational programming,”

  8. [16]

    Intelliexplain: Enhancing conversational code generation for non-professional programmers,

    H. Yan, T. D. Latoza, and Z. Yao, “Intelliexplain: Enhancing conversational code generation for non-professional programmers,”

  9. [17]

    COLDECO: An End User Spreadsheet Inspection Tool for AI-Generated Code ,

    K. Ferdowsi, J. Williams, I. Drosos, A. D. Gordon, C. Negreanu, N. Polikarpova, A. Sarkar, and B. Zorn, “ COLDECO: An End User Spreadsheet Inspection Tool for AI-Generated Code ,” in 2023 IEEE Symposium on Visual Languages and Human- Centric Computing (VL/HCC) . Los Alamitos, ...

  10. [18]

    Does the whole exceed its parts? the effect of ai explanations on complementary team performance,

    G. Bansal, T. Wu, J. Zhou, R. Fok, B. Nushi, E. Kamar, M. T. Ribeiro, and D. Weld, “Does the whole exceed its parts? the effect of ai explanations on complementary team performance,” in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems , ser. CHI ’21...

  11. [19]

    To rely or not to rely? evaluating interventions for appropriate reliance on large language models,

    J. Y . Bo, S. Wan, and A. Anderson, “To rely or not to rely? evaluating interventions for appropriate reliance on large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2412.15584

  12. [20]

    Available: https://arxiv.org/abs/2405.10250

    [Online]. Available: https://arxiv.org/abs/2405.10250

  13. [21]

    Intercode: Standardizing and benchmarking interactive coding with execution feedback,

    J. Yang, A. Prabhakar, K. R. Narasimhan, and S. Yao, “Intercode: Standardizing and benchmarking interactive coding with execution feedback,” in Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2023. [Online]. Available: https:/...

  14. [22]

    Lever: Learning to verify language-to-code generation with execution,

    A. Ni, S. Iyer, D. Radev, V . Stoyanov, W. tau Yih, S. I. Wang, and X. V . Lin, “Lever: Learning to verify language-to-code generation with execution,” 2023. [Online]. Available: https://arxiv.org/abs/2302.08468

  15. [23]

    Marketing mix model: In- sights and optimization,

    L. Analytics, “Marketing mix model: In- sights and optimization,” 2019, accessed: 2025- 01-04. [Online]. Available: https://www.latentview.com/wp- content/uploads/2019/08/Marketing-Mix-Model.pdf

  16. [24]

    The state of the art in end-user software engineering,

    A. J. Ko, R. Abraham, L. Beckwith, A. Blackwell, M. Burnett, M. Erwig, C. Scaffidi, J. Lawrance, H. Lieberman, B. Myers, M. B. Rosson, G. Rothermel, M. Shaw, and S. Wiedenbeck, “The state of the art in end-user software engineering,” ACM Comput. Surv., vol. 43, no. 3, Apr

  17. [25]

    On mitigating code llm hallucinations with api documentation,

    N. Jain, R. Kwiatkowski, B. Ray, M. K. Ramanathan, and V . Kumar, “On mitigating code llm hallucinations with api documentation,” 2024. [Online]. Available: https://arxiv.org/abs/2407.09726

  18. [26]

    Prolific,

    Prolific, “Prolific,” https://www.prolific.com, 2014

  19. [27]

    Statistical methods for research workers,

    R. A. Fisher, “Statistical methods for research workers,” in Break- throughs in statistics: Methodology and distribution . Springer, 1970, pp. 66–70

  20. [31]

    Young people’s descriptions of computational rules in role-playing games: An empirical study,

    J. Good, K. Howland, and K. Nicholson, “Young people’s descriptions of computational rules in role-playing games: An empirical study,” in Proceedings of the 2010 IEEE Symposium on Visual Languages and Human-Centric Computing , ser. VLHCC ’10. USA: IEEE Computer Society, 2010, ...

  21. [2011]

    Available: https://doi.org/10.1145/1922649.1922658

    [Online]. Available: https://doi.org/10.1145/1922649.1922658

  22. [2021]

    Available: https://doi.org/10.1145/3411764.3445717

    [Online]. Available: https://doi.org/10.1145/3411764.3445717

  23. [2022]

    Available: https://doi.org/10.1145/3491101.3519665

    [Online]. Available: https://doi.org/10.1145/3491101.3519665

  24. [2023]

    Available: https://arxiv.org/abs/2309.09495

    [Online]. Available: https://arxiv.org/abs/2309.09495

  25. [2024]

    Available: https://doi.org/10.1145/3663384.3663393

    [Online]. Available: https://doi.org/10.1145/3663384.3663393

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.