Pith. sign in

REVIEW 3 major objections 8 minor 1 cited by

Generative AI's statistical foundation forecloses formal correctness, so software assurance should be reframed as graded confidence in fitness for purpose, with each property's value, credibility, and provenance recorded.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 08:33 UTC pith:4NU5S3IT

load-bearing objection A senior voice worth hearing, but its key claim about AI's 'unsound' statistical basis is overstated; the reframing to confidence and credentials stands on its own. the 3 major comments →

arxiv 2607.04667 v2 pith:4NU5S3IT submitted 2026-07-06 cs.SE

Correctness, confidence, and context: Framing software assurance in the AI age

Mary Shaw This is my paper
classification cs.SE
keywords software correctnesssoftware assurancefitness for purposesufficient correctnesstacit contextdigital shadowssoftware credentialslimitations of AI
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that software engineering's long-standing goal of formal correctness is unattainable for modern software and especially for AI-influenced systems, because the context that actually determines fitness for purpose is mostly tacit and undocumented. Generative AI is built on a statistical basis that yields 'probably approximately correct' results, so it can support only probabilistic assurances, not rigorous proof. The paper proposes reframing assurance as confidence in fitness for purpose, and introduces 'credentials': collections of attribute-value pairs augmented with credibility and provenance. It calls for an assurance plan per project, a field guide to assurance techniques, and a cost-benefit choice among them. If the argument holds, the community should stop equating good software with proven software and instead engineer assurance deliberately.

Core claim

On the paper's own terms, the central claim is that the 'correctness' mindset that still shapes software engineering—the idea that enough information and rigor would let us reason our way to correct software—is no longer viable, and generative AI makes that impossibility explicit. Because LLMs are trained only on 'digital shadows' of reality, the tacit and implicit context that encodes design intent, domain expertise, and cultural norms is mostly lost to them; because their predictions are statistical, they cannot support the hard, rigorous assurances that formal verification promises. The paper therefore redefines the assurance task as establishing confidence that a system is fit for its pu

What carries the argument

The credential: a set of four-tuples { <attribute, value, credibility, provenance> } that replaces monolithic specifications with a graded, provenance-tagged record of each property of interest. The associated 'fitness for purpose landscape' maps the world-machine distinction, and the 'digital shadow' concept explains why AI only sees incomplete traces. The credential is what carries the argument: it makes 'confidence' a concrete, inspectable artifact and enables the proposed optimization of assurance resources.

Load-bearing premise

The argument depends on the premise that the tacit context, design intent, and cultural knowledge that shape software leave only scant traces in the digital record, so AI models trained on that record cannot recover them.

What would settle it

A controlled study in which engineers write down the unrecorded rationale behind critical design decisions, then an LLM is given only the code and documentation and asked to reconstruct that rationale; if the LLM faithfully recovers a substantial fraction of the decisions, the 'scant shadows' premise collapses. A second test: a requirement that mentions a symbol with culture-specific meaning (e.g., 'sunflower') with no explicit context, and an LLM prompt that includes the client's background; if the LLM reliably selects the culturally relevant association, then some tacit knowledge is recovera

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Each project would begin with an assurance plan that specifies the important properties and the required level of confidence for each, rather than aiming at blanket correctness.
  • Credentials make it possible to combine evidence of different rigor—formal proof, empirical tests, expert judgment—with explicit provenance and credibility, so a portfolio of assurance techniques can be compared on the same footing.
  • Assurance choices become a deliberative resource-allocation problem: either maximize confidence for available resources or compute the resources needed to reach required confidence.
  • A community-maintained 'field guide' of assurance techniques, with uniform descriptions of properties, cost, coverage, and rigor, becomes a prerequisite for principled tradeoffs.
  • AI-generated code and AI-supported tools would be subject to the same confidence/provenance regime as any other software component.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The credential model is a natural fit for 'nutrition-label' style reporting on AI systems, where users see not just a score but the source and certainty behind it; this could be developed into a standard exchange format.
  • If LLMs continue to improve, some knowledge the paper classifies as tacit may become inferable from larger corpora; the framework's boundary would then shift, suggesting the paper's inherent-limitation claim is a technological transition rather than a permanent epistemic ceiling.
  • The paper's framing implies a research agenda for cost models: measuring the marginal confidence gained per dollar spent on each assurance technique, which no existing SE literature systematically provides.
  • One could test the 'digital shadow' premise empirically by constructing a corpus of projects with known but undocumented design rationale and measuring how much of it an LLM can reconstruct from code alone.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. Shaw argues that software engineering's traditional ideal of formal correctness is not only practically unattainable but conceptually insufficient, because fitness for purpose depends on tacit and implicit context that escapes formalization and digital records. She extends this to generative AI: LLMs are trained on 'digital shadows' and their statistical, 'probably approximately correct' basis cannot support rigorous assurance. The paper proposes to replace the correctness mindset with confidence in fitness for purpose, using 'credentials'—tuples of attribute, value, credibility, provenance—to make evidence graded and comparable, and a 'field guide' to assurance techniques that would support cost-effective selection. The manuscript is a keynote narrative, explicitly leaving the field guide and optimization as open challenges.

Significance. The paper's value is its clear articulation of a shift already occurring in practice: from verification-oriented correctness to risk- and evidence-based fitness-for-purpose. The examples are effective; the credentials idea offers a compact representation for heterogeneous assurance; and the call for a comparative 'field guide' is a useful research agenda. If the strong claims about AI's inherent limitations are appropriately qualified, the paper could serve as a generative position statement. In its current form, however, the central technological premise is asserted rather than demonstrated, and that premise carries much of the argument's urgency.

major comments (3)
  1. [Abstract and §4, Figure 10] The load-bearing premise is stated in the Abstract and in §4 ('Unsound, statistical basis'): generative AI's statistical foundation 'does not permit rigorous reasoning' and 'inherently limits assurances... to probabilistic assertions.' This is not demonstrated, and as stated it is difficult to defend. PAC learning—the exact source of the phrase 'probably approximately correct' [36]—is a rigorous mathematical framework with finite-sample error bounds. Probabilistic model checking, randomized algorithms, and conformal prediction are accepted forms of rigorous probabilistic assurance. If the intended claim is narrower (e.g., that current LLM decoding does not provide such guarantees), the paper must say so explicitly and explain why PAC-style guarantees cannot be extended to LLMs. As written, the dichotomy between 'rigorous reasoning' and 'probabilistic assertions' carries the conclusion th
  2. [§3, Figure 7] The 'digital shadows' argument treats inaccessibility of tacit/implicit context as an inherent limitation of LLMs. But the paper concedes that implicit context is in principle inferable and cites Unblocked [35] as an attempt to reconstruct exactly this context. The examples (sunflower associations, cargo cults, chemical plant operators) are evocative but do not establish that assurance-relevant tacit knowledge is absent from LLM training data or cannot be recovered. The paper should recast this as an empirical/architectural limitation—e.g., measured performance on tasks requiring implicit design rationale—or give a formal account of why context cannot be represented. Without this, Figure 7's red 'lost information' is an assertion, not a result.
  3. [§5–§6] The constructive framework in §5–§6 is a promise, not yet a specification. The credential tuple is described with one example, but no semantics are given for credibility values, provenance types, or the combination operation ('propagate new values' is mentioned without rules). The field guide is a list of attributes, and the optimization is left as an open question. If these are intended as the paper's contribution, at least one instantiation—even synthetic—should show how the tuple fields and the selection mechanism would be used together. As it stands, the central reframing is supported by arguments from practice, but the allegedly enabling machinery remains a research agenda.
minor comments (8)
  1. [Figure 2 / references] The text citing Figure 2 uses '[39]', but [39] is an image credit, not the StackOverflow survey; the intended source appears to be [40]/[28].
  2. [Image credits] Image credit [43] says Figure 5/6b are 'from [4]', but [4] is the CrowdStrike RCA; the likely intended source is [6] (Evans 1917).
  3. [§3 heading] 'peoples' heads' should be 'people's heads'.
  4. [§4 heading] The term 'unsound' has a precise meaning in logic; the statistical basis of an ML model is not itself an argument. Consider 'non-deductive' or 'probabilistic' to avoid loading the term.
  5. [§5] Credentials would benefit from positioning relative to existing assurance case notations (e.g., GSN) and evidence-based software engineering; currently the link to systematic reviews [17] is the only bridge.
  6. [§2, Figure 4] The axes 'consequences of failure' and 'degree of automation' are intuitive but not defined; a reader cannot operationalize where the 'good enough vs correctness' boundary lies.
  7. [Throughout] The paper uses 'LLM' and 'machine learning' interchangeably. The statistical-basis argument applies to ML broadly, but the context/digital-shadows argument is specific to LLMs; clarify the scope of each claim.
  8. [References] Important support is given by unpublished or non-archival sources ([24], [31], [34]). For a peer-reviewed venue, accessible versions or independent corroboration would strengthen the argument.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 2 invented entities

The paper's central argument rests on domain assumptions about tacit knowledge and the limits of statistical reasoning; it proposes two new conceptual artifacts (credentials and a field guide) but provides no empirical validation. No free parameters are fitted.

axioms (3)
  • domain assumption Tacit and implicit context (design intent, cultural norms, domain expertise) is largely absent from digital records, so LLMs trained on those records cannot access it.
    This is the basis for the 'digital shadows' argument in Sections 3-4 (Figure 7). If LLMs could recover such context from data, the paper's chief limitation of AI would be overstated.
  • domain assumption Statistical / probably-approximately-correct foundations cannot support the hard assurances required for safety-critical software.
    The paper asserts this in Section 4 without formal argument; it motivates the turn to graded 'credentials' and probabilistic confidence.
  • ad hoc to paper Software engineering should be judged by cost-effective fitness for purpose rather than formal correctness.
    This reframing is a normative value choice, not a consequence of prior results; the paper acknowledges it is a challenge to the field.
invented entities (2)
  • Credentials (attribute-value-credibility-provenance tuples) no independent evidence
    purpose: To represent assurance evidence with varying degrees of rigor and provenance so designers can choose cost-effective assurance techniques.
    This is a conceptual artifact, not a physical entity or prediction; it carries no falsifiable handle outside the paper.
  • Field guide to assurance techniques no independent evidence
    purpose: An open-ended catalog of assurance methods with structured descriptions to support tradeoff analysis.
    Explicitly proposed as an open problem; no implementation or data provided.

pith-pipeline@v1.3.0-alltime-deepseek · 14601 in / 11445 out tokens · 114601 ms · 2026-08-02T08:33:16.584751+00:00 · methodology

0 comments
read the original abstract

Software engineering has a complicated relationship with "correctness". We recognize the challenges of full formal rigor as well as many required properties beyond functional correctness. Although we satisfice in practice, we are still stuck in the mindset that we could reason our way to correctness, if only we had enough information. Unfortunately for our hopes of formal rigor, our expectations are shaped by unspoken knowledge that is personal, subjective, qualitative, and largely unavailable. Generative AI has introduced a new dimension to assurance: its foundation is statistical rather than formal. Traditional software engineering establishes confidence through rigorous reasoning, domain knowledge and expert judgment. In contrast, generative AI results are sophisticated predictions, "probably approximately correct". This inherently limits assurances about the results to probabilistic assertions. Further, the nuances that guide human judgment are often tacit or implicit. This knowledge casts only scant shadows into the digital record, so that critical source of knowledge is only faintly represented in AI models. We have many approaches for developing assurances that a software system does what it's expected to do, though most of them focus on code specifications rather than system requirements, let alone the system's fitness for its purpose. We have failed to develop a systematic understanding of the relative merits of the various approaches to assurance. I hope that generative AI will finally force us to tackle this. To that end, I will challenge us to think systematically about our assurance techniques, especially the role of hidden context and the challenges of AI. We need ways to make informed, reasoned choices about cost-effective combinations of approaches to developing confidence in our systems. We call ourselves software engineers. Let's act like engineers.

Figures

Figures reproduced from arXiv: 2607.04667 by Mary Shaw.

Figure 1
Figure 1. Figure 1: This view is insufficient, however. Although SE tends to regard a system as simply a composition of code modules, real systems actually integrate com￾ponents of many types. Even for simple standalone systems, data is as important as code and less often validated. An especially poignant example occurred on July 19 2024, when Crowdstrike distributed an updated configuration file for their security software. … view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. AssumptionMiner: Extracting, Tracing, and Revising Implicit Assumptions in LLM Code Generation

    cs.SE 2026-07 conditional novelty 7.0

    AssumptionMiner extracts implicit assumptions from LLM-generated code with a cross-model ensemble F1 of 0.816 on a new 180-task benchmark, and enables assumption-guided targeted regeneration.

Reference graph

Works this paper leans on

45 extracted references · 1 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Lewis Binns. 2026. Deploying modern software architecture on a high-value, high-utilisation install base: an ASML case study. Keynote presentation, ICSA26, International Conference on Software Architecture, Amsterdam NL, June 2026

    2026

  2. [2]

    No Silver Bullet—Essence and Accident in Software Engineering

    Frederick P. Brooks. 1987. "No Silver Bullet—Essence and Accident in Software Engineering". IEEE Computer. 20 (4): 10–19. doi:10.1109/MC.1987.1663532

    arXiv 1987

  3. [3]

    Crowdstrike. 2024. Remediation and Guidance Hub: Channel File 291 Incident. https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance- hub/

    2024

  4. [4]

    Crowdstrike. 2024. External Technical Root Cause Analysis — Channel File

    2024

  5. [5]

    Cybersecurity News. 2024. CrowdStrike Outage Leads to Estimated Financial Loss of $5.4 Billion. https://cybersecuritynews.com/crowdstrike-outage-loss-5- 4-billion/

    2024

  6. [6]

    William Augustus Evans. 1917. Dr. Evans’ How to Keep Well. Wikimedia Commons File:Dr. Evans' How to keep well; (1917) (14766899081).jpg

    1917

  7. [7]

    David Garlan, Robert Allen, and John Ockerbloom. 1995. Architectural mismatch or why it's hard to build systems out of existing parts. Proc. 17th international conference on Software engineering (ICSE ‘95). 179–185. https://doi.org/10.1145/225014.225031

    arXiv 1995

  8. [8]

    Hayakawa and Alan R

    S.I. Hayakawa and Alan R. Hayakawa. 1990. Language in Thought and Action. Collins 1990. ISBN 978-0-15-648240-0

    1990

  9. [9]

    E. D. Hirsch, Jr. 1987. Cultural literacy: what every American needs to know. Houghton Mifflin 1987. ISBN 039543095X

    1987

  10. [10]

    International Conference on Software Architecture, ICSA26. 2026. IEEE

    2026

  11. [11]

    Michael Jackson. 1995. The world and the machine. Proc17th international conference on Software engineering (ICSE '95)., 283–292. https://doi.org/10.1145/225014.225041

    arXiv 1995

  12. [12]

    Christian Kästner. 2026. On risks and correctness in ML systems. Lecture, Carnegie Mellon University, March 2026

    2026

  13. [13]

    Eunsuk Kang and Mary Shaw. 2024. Tl;dr: Chill, y’all: AI Will Not Devour SE. Onward! ‘24. 303–315. https://doi.org/10.1145/3689492.3689816

    arXiv 2024

  14. [14]

    Philip Koopman. 2026. Embodied AI Safety: reimagining safety engineering for artificial intelligence in physical systems. Seminar at Carnegie Mellon April 2026

    2026

  15. [15]

    Butler W. Lampson. 1983. Hints for computer system design. SIGOPS Oper. Syst. Rev. 17, 5 (October 1983), 33–48. https://doi.org/10.1145/773379.806614

    arXiv 1983

  16. [16]

    Zeb Larson. 2026. The oral tradition that built software may not survive AI. Fast Company, 5-29-2026. https://www.fastcompany.com/91549609/the-oral- tradition-that-built-software-may-not-survive-ai

    arXiv 2026

  17. [17]

    Bridging the gap: From research to practical advice

    Claire Le Goues, Ciera Jaspan, Ipek Ozkaya, Mary Shaw, Kathryn T Stolee.2018. Bridging the gap: From research to practical advice. IEEE Softw are, Sep 2018

    2018

  18. [18]

    Nancy G. Leveson. 1986. Software safety: why, what, and how. ACM Comput. Surv. 18, 2 (June 1986), 125–163. https://doi.org/10.1145/7474.7528

    arXiv 1986

  19. [19]

    Ford rehires human engineers after AI fails to match quality checks

    Liv McMahon. Ford rehires human engineers after AI fails to match quality checks. BBC. https://www.bbc.com/news/articles/cgrkd41n2v9o

  20. [20]

    Computer Science: Reflections on the Field, Reflections from the Field

    National Research Council, 2004. Computer Science: Reflections on the Field, Reflections from the Field. Washington, DC: The National Academies Press. https://doi.org/10.17226/11106

    doi:10.17226/11106 2004

  21. [21]

    Plato. ~375BCE. The Republic, Book 6. (described elswehere)

  22. [22]

    Dilemmas in a General Theory of Planning

    Horst W.J. Rittel and Melvin M. Webber. 1973. "Dilemmas in a General Theory of Planning" . Policy Sciences. 4 (2): 155–169. doi:10.1007/bf01405730

    doi:10.1007/bf01405730 1973

  23. [23]

    Bogdan Sandu. 2025. What is Software Verification and How it’s done. TMS. https://tms-outsource.com/blog/posts/what-is-software-verification/

    2025

  24. [24]

    Chris Scaffidi and Mary Shaw. 2007. Developing Confidence in Software through Credentials and Low-Ceremony Evidence. Int’l Workshop on Living with Uncertainties (IWLU’07), co-located with the 23rd IEEE/ACM International Conference on Automated Software Engineering (ASE 2007), Atlanta, GA, November 2007. Not in digital library. Online at NON- ARCHIVAL http...

    2007

  25. [25]

    Donald A. Schön. 1983. The Reflective Practitioner_ How Professionals Think In Action. Basic Books (1983)

    1983

  26. [26]

    Mary Shaw. 1990. Prospects for an engineering discipline of software. IEEE Software, vol. 7, no. 6, pp. 15-24, Nov. 1990, doi: 10.1109/52.60586

    doi:10.1109/52.60586 1990

  27. [27]

    Mary Shaw. 1996. Truth vs Knowledge: The Difference Between What a Component Does and What We Know It Does. Proc. 8th Int'l Workshop on Software Specification and Design, March 1996. doi: 10.1109/IWSSD.1996.501165 There are two ways to deal with the possibility of Bad Things: prevention and remediation Correctness is a social construct New technology does...

    arXiv 1996

  28. [28]

    Mary Shaw. 2021. Myths and Mythconceptions: What does it mean to be a programming language, anyhow? Proc. ACM Prog Lang 4, HOPL, Article 234 (June 2021). https://doi.org/10.1145/3480947

    doi:10.1145/3480947 2021

  29. [29]

    Mary Shaw. 2025. Creative Nudges: Jus’ Sayin’. Card deck. Shaw-Weil Associates 2025

    2025

  30. [30]

    Design obligations for software, with examples from data abstraction and adaptive systems

    Mary Shaw. 2025. "Design obligations for software, with examples from data abstraction and adaptive systems." 2025 IEEE/ACM International Workshop on Designing Software (Designing). IEEE

    2025

  31. [31]

    Mary Shaw. 2026. On Correctness, Sense-making, Digital Shadows, the Entrails of Goats: how inaccessible knowledge fundamentally limits LLMs. Unpublished manuscript

    2026

  32. [32]

    Rational Choice and the Structure of the Environment

    Herbert A. Simon (1956). "Rational Choice and the Structure of the Environment" (PDF). Psychological Review. 63 (2): 129–138

    1956

  33. [33]

    Jan-Philipp Steghöfer and Markus Borg. 2026. An abstraction is worth a thousand vibes. IEEE Software, vol. 43, no. 01, pp. 13-16, Jan.-Feb. 2026, doi: 10.1109/MS.2025.3621786

    arXiv 2026

  34. [34]

    Margaret-Anne Storey. 2026. From Technical Debt to Cognitive and Intent Debt: Rethinking Software Health in the Age of AI. arXiv:2603.22106v4 [cs.SE]

    Pith/arXiv arXiv 2026

  35. [35]

    Unblocked. 2026. Unblocked: the context engine for modern engineering teams. https://getunblocked.com/

    2026

  36. [36]

    2013 Probably Approximately Correct: Nature’s Algorithms for Learning and Prospering in a Complex World

    Leslie Valiant. 2013 Probably Approximately Correct: Nature’s Algorithms for Learning and Prospering in a Complex World. Basic Books 2013

    2013

  37. [37]

    Wikipedia. 2026. Common Sunflower. https://en.wikipedia.org/wiki/Common_sunflower

    2026

  38. [38]

    Dimitris Xygalatas. 2022. What cargo cult rituals reveal about human nature. Sapiens. https://www.sapiens.org/culture/cargo-cult-rituals/ IMAGE CREDITS

    2022

  39. [39]

    Figures 1, 3, 5, 8 and 10 were drawn by Mary Shaw for this paper

  40. [40]

    Figure 2 ecosystem of web development is from [28] (by Mary Shaw) based on https://insights.stackoverflow.com/survey/2019#development-environments- and-tools

    2019

  41. [41]

    Figure 4 Assurance realms is from [28] (by Mary Shaw)

  42. [42]

    Figure 6 shadow projection of a block is Adobe Stock photo, licensed via Adobe Creative Comons

  43. [43]

    Public domain

    Figure 6b and Figure 5 a paper cup and how to fold it are from [4]. Public domain

  44. [44]

    Figure 9 is Mary Shaw’s simplification of an example in [18]

  45. [291]

    https://www.crowdstrike.com/wp-content/uploads/2024/08/Channel-File- 291-Incident-Root-Cause-Analysis-08.06.2024.pdf

    2024