Upstream and Downstream AI Safety: Both on the Same River?

Ibrahim Habli; John McDermid; Yan Jia

Upstream and Downstream AI Safety: Both on the Same River?

T0 review · 0 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that downstream safety—assessing AI in its context of use—remains essential for general-purpose AI, and that upstream frontier-model safety can feed it through a modular safety case and HAZOP-style deviation classes.

desk verdict A genuinely useful synthesis of upstream and downstream AI safety, held together by a speculative HAZOP mapping that the authors themselves flag as unproven. read the letter →

arxiv 2501.05455 v1 pith:CUNGTI6F submitted 2024-12-09 cs.CY cs.AI

John McDermid , Yan Jia , Ibrahim Habli This is my paper · ORCID

classification cs.CYcs.AI

keywords AIsafetyupstreamdownstreamgeneral-purposecaseHAZOPregulationrewardhacking

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the traditional 'downstream' approach to safety—assessing a system in its actual context of use—remains essential in the age of general-purpose AI, provided it adapts. It claims that 'upstream' safety work on frontier model capabilities is not a separate enterprise but a tributary that can flow into downstream safety, merging through the regulatory ecosystem and through a shared safety-case structure. The proposed bridge is a set of 'GPAI deviation classes' that translate known model failure modes such as reward hacking and distributional shift into the deviation language of HAZOP, so that upstream evaluations can inform context-specific hazard analysis. If this confluence works, frontier-model safety evidence could be used by domain regulators in sectors like automotive, maritime, and healthcare rather than being assessed in isolation.

What carries the argument

The central object is the 'GPAI deviation class,' an adaptation of HAZOP deviation guidewords to the failure modes of general-purpose AI. In classical HAZOP, deviations such as omission, commission, too much, and other than are used to explore how a system might depart from intended operation; the paper proposes that general GPAI failure modes such as reward hacking and distributional shift constitute classes of deviation from intent that can be made concrete in a downstream application context. The second load-bearing mechanism is the modular safety case expressed in Goal Structuring Notation (GSN), which integrates AI ethics, AI system safety, purpose-specific model safety, and general-purpose model safety arguments, allowing upstream evidence such as evals and red teaming to be imported into downstream safety cases. These two mechanisms together are what would allow the upstream and downstream streams to converge.

What would settle it

Run a concrete application study: take a GPAI model fine-tuned for a specific downstream task with known hazards, such as an LLM computing maritime voyage plans; use upstream evals to identify distributional shift or reward-hacking incidents; then perform a HAZOP-style analysis using the proposed deviation classes and check whether it surfaces hazards that the upstream evals alone did not surface, and whether those hazards are specific to the application context. If the deviation-class analysis repeatedly adds nothing beyond direct evaluations, the bridge between upstream and downstream safety is not carrying the weight the paper assigns to it.

Watch

Extended reading notes

Core claim

The paper's central claim is that the downstream model of safety is still relevant in the GPAI world, but needs to adapt, and that upstream AI safety can be seen as a tributary of downstream safety, with the two streams merging through the regulatory ecosystem. To realize this, it proposes a modular safety case in Goal Structuring Notation that integrates four sub-arguments: AI ethics, AI system safety, purpose-specific model safety, and general-purpose model safety. The conceptual key is treating general failure modes of GPAI—reward hacking, distributional shift, and the like—as 'GPAI deviation classes' that can be mapped onto HAZOP guidewords (omission, commission, too much, other than), making upstream capability evaluations usable in downstream hazard identification. The paper also identifies weight exfiltration and other broad risks as 'particular risks' analogous to classical safety engineering concerns, and suggests a dialectic regulatory process in which developers present a safety case and a red team presents a countervailing risk case.

Load-bearing premise

The proposed bridge depends on the assumption that general GPAI failure modes such as reward hacking and distributional shift can be characterized as 'GPAI deviation classes' and mapped onto HAZOP-style deviations (omission, commission, too much, other than) in a way that yields actionable downstream safety analyses; the paper itself flags this mapping as speculative and in need of further conceptual and empirical work.

Editorial extensions

If this is right

Domain-based regulators could incorporate upstream evidence on model capabilities and failure modes into context-specific hazard analyses, using adapted HAZOP methods.
Upstream safety frameworks could adopt downstream concepts like common mode and common cause failures to assess guardrails, especially 'deference' arguments where the same underlying model guards itself.
National AI regulators could take responsibility for 'particular risks' and direct GPAI use, while domain regulators handle specific applications, with knowledge flowing both ways.
A dialectic process of safety case versus risk case could provide the independent challenge that advanced AI safety claims currently lack.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct test of the paper's proposal would be a HAZOP-style exercise on a concrete GPAI application—say, an LLM used for vessel voyage planning—to see whether the deviation-class mapping yields hazard identifications that upstream evals alone would miss.
If the mapping from GPAI failure modes to HAZOP deviations turns out to be too loose, the modular safety case could still stand with the upstream and purpose-specific arguments treated as separate modules, so the regulatory confluence does not strictly depend on the deviation-class mechanism.
The 'particular risks' framing implies that national AI regulators might build cross-domain scenario libraries—for example, the correlated failure of booking systems across all transport modes—as an instrument for assessing systemic GPAI-related risks.
Treating GPAI failure modes as deviation classes suggests a research programme of injecting simulated deviations into models to generate hazard identifications, a tooling direction the paper mentions but does not develop.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This is a well-scoped position paper that gives AI safety regulators and practitioners a useful shared vocabulary, but the load-bearing bridge between upstream and downstream — the mapping of GPAI failure modes onto HAZOP deviation classes — is explicitly speculative and remains the main open question. The reader's ACCEPT verdict holds up; I'd send it to review.

What's genuinely new: the specific mappings. Treating critical capability levels as 'particular risks' in the downstream sense, and reward hacking/distributional shift as 'GPAI deviation classes' that could be interpreted through HAZOP guidewords, is a real synthesis. The modular safety case in Figure 3, integrating ethics, system, purpose-specific model, and general-purpose model arguments, is a constructive structure that regulators and domain safety engineers can react to. I also found the discussion of common-mode failures in the context of 'deference' (AI guarding AI) — and the point that models trained on the same data will share limitations — sharp and useful. The authors are careful to separate established safety engineering from their own proposals, and the concrete examples (electric vehicle motors, humanoid robots, autonomous vessels) ground the discussion.

The soft spots are real but proportionate. The stress-test note is correct: HAZOP guidewords are defined relative to a specific intended function and process variables. A general-purpose model has no context-specific design intent, so an upstream failure mode like reward hacking does not have a determinate reading as a 'deviation' until an application context is supplied. The paper's own France/Germany road sign example is not a deviation from the model's intent; it is an input-distribution difference that only becomes a hazard once an operational design domain is fixed. Footnote 22 concedes exactly this: the translation to GPAI 'would require more work at the conceptual and empirical levels.' So the 'tributary' conclusion rests on an unvalidated analogy, not a worked method. That is a limitation, not a fatal flaw, because the paper frames it as a research agenda and the modular safety case remains useful even if that particular mapping never firms up.

Minor: the authors lean on their own prior work (AMLAS, BIG argument), but this is not circular — the central thesis does not depend on those results being correct. And the 'same river' metaphor is overworked by the end, but that's a style point.

Who this is for: safety engineers crossing into frontier AI, AI governance people who need a bridge to existing assurance practice, and researchers working on safety cases for advanced AI. It deserves a serious referee. The main revision request should be to make the speculative status of the HAZOP mapping more prominent — or better, to provide one worked example of how a deviation class would be interpreted in a concrete GPAI application.

Send it to review.

Referee Report

0 major / 7 minor

Summary. This position paper distinguishes two approaches to AI safety: 'downstream' safety, which follows traditional safety engineering and assesses a system in its context of use (e.g., an autonomous vehicle in its operational design domain), and 'upstream' safety, which focuses on the development and general capabilities of general-purpose AI (GPAI) models, such as preventing reward hacking, distributional shift, and model autonomy risks. The paper compares the two frameworks, identifies analogous concepts (e.g., SOTIF vs. capability evaluation, SEooC vs. evals), and argues that each community can learn from the other. It proposes a modular safety case (Figure 3) that integrates AI ethics, system safety, purpose-specific model safety, and general-purpose model safety arguments, and it discusses how regulatory ecosystems might allow upstream safety to become a 'tributary' of downstream safety. The central conclusion is that the downstream model remains relevant in the GPAI world but must continue to adapt.

Significance. The paper makes a constructive contribution by bridging two usually disjoint communities: safety engineering and frontier AI safety. Its concrete examples (electric vehicle motor configurations, SDV perception, OpenAI o1 reward hacking) ground otherwise abstract debates. It is also honest about its limits: the proposed mapping of GPAI failure modes to HAZOP 'deviation classes' is explicitly labelled as speculative (footnote 22), and the paper largely frames its stronger claims as a research agenda rather than a demonstrated mechanism. This scoping is appropriate for a conceptual paper and makes the central message—that downstream, contextual analysis remains essential and can be enriched by upstream capability information—credible.

minor comments (7)

[Comparison and Analysis, Table 1] The table is dense and its 'Insights' column is vague; consider splitting the table or adding a short prose summary that highlights the three or four most important contrasts for readers unfamiliar with both literatures.
[The Same River or a Confluence?, HAZOP paragraph] The sentence 'Where the signs are the same, do they have the same meaning and the same sizes so that detection distances can remain the same?' is run-on and could be split into two sentences for clarity.
[Footnote 19] The footnote begins with 'Se,e:' which appears to be a typo for 'See:'.
[Figure 3] The figure contains typos ('Descrition', 'developpment') and is nearly illegible at page size; a cleaner rendering or a higher-level textual summary of the argument structure would help readers who cannot read the GSN details.
[Upstream Safety, Observations] The phrase 'one in a million operations/hours or less' should be rephrased as 'one failure per million operations or hours' to avoid ambiguity.
[References] Reference [29] is incomplete: it appears to be an arXiv preprint but lacks the arXiv ID or a full citation.
[Regulatory challenges in upstream safety assurance] The term 'vires' is used without explanation; consider replacing it with 'legal authority' or adding a brief parenthetical definition.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is a conceptual synthesis whose argument is self-contained and whose speculative mappings are explicitly flagged as research directions, not derived predictions.

full rationale

This is not a derivation paper: there are no fitted parameters, equations, or empirical predictions that could reduce to their own inputs. The central claims are conceptual, namely that downstream safety remains relevant for GPAI and that upstream analyses could in principle inform downstream HAZOP-style analyses. The load-bearing bridge, the proposed 'GPAI deviation classes' mapping onto HAZOP guidewords, is explicitly acknowledged by the authors as speculative and requiring further conceptual and empirical work (footnote 22), so it is not presented as a demonstrated result. Self-citations to AMLAS and other York safety-case work appear as contextual references and illustrative guidance, but the paper's thesis does not depend on those prior results being true in a circular way. No step in the argument equates an output to an input by definition, renames a known result as a prediction, or imports a uniqueness theorem from the authors' prior work. The paper is honest about its limitations and frames its contribution as a research agenda, so the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 2 invented entities

The paper is a conceptual synthesis rather than an empirical derivation. Its central claim rests primarily on domain assumptions about the transferability of safety engineering methods to GPAI and on the adoption of a proposed vocabulary; the paper itself acknowledges most of these as open research questions.

assumptions (3)

domain assumption Classical safety engineering techniques (HAZOP, FTA, FMEA, safety cases) remain applicable when the system under analysis includes GPAI components.
The entire transfer argument in the 'Comparison and Analysis' section assumes these methods can be adapted to GPAI; the paper itself identifies this as an open research issue (e.g. 'how to translate to GPAI would require more work at the conceptual and empirical levels').
ad hoc to paper The combination of capability evaluation and evolutionary thinking can serve as the 'underlying science' of upstream safety.
Proposed in the Upstream Safety section explicitly 'in order to prompt discussion'; no empirical or theoretical support is offered.
domain assumption Regulators with domain-specific remits and new national AI regulators will be able to collaborate effectively so that upstream and downstream safety can merge through the regulatory ecosystem.
The regulatory confluence scenario in 'Regulatory challenges' relies on this coordination, which is acknowledged as an assumption ('we assume that the national regulators will operate as a collaborative network').

invented entities (2)

GPAI deviation classes
purpose: General categories of GPAI misbehavior (e.g. reward hacking, distributional shift) that could be used as HAZOP-style guidewords for downstream safety analysis.
Introduced in 'The Same River or a Confluence?' section as speculative: 'At this stage this is speculative.' No empirical instances are provided.
Integrated modular AI safety case (Figure 3) spanning upstream and downstream assurance
purpose: A Goal Structuring Notation pattern that combines AI ethics, system safety, purpose-specific model, and general-purpose model arguments to link the two frameworks.
Proposed as a sketch, based on the authors' 'BIG Argument for AI Safety Cases' which is described as 'to appear'; not yet validated in practice.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Upstream and Downstream AI Safety: Both on the Same River?." pith.science (2026). https://pith.science/paper/CUNGTI6F

@misc{pith2026250105455,
  author       = {Pith},
  title        = {Pith review of: Upstream and Downstream AI Safety: Both on the Same River?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CUNGTI6F}},
  note         = {Machine review of arXiv:2501.05455}
}

read the original abstract

Traditional safety engineering assesses systems in their context of use, e.g. the operational design domain (road layout, speed limits, weather, etc.) for self-driving vehicles (including those using AI). We refer to this as downstream safety. In contrast, work on safety of frontier AI, e.g. large language models which can be further trained for downstream tasks, typically considers factors that are beyond specific application contexts, such as the ability of the model to evade human control, or to produce harmful content, e.g. how to make bombs. We refer to this as upstream safety. We outline the characteristics of both upstream and downstream safety frameworks then explore the extent to which the broad AI safety community can benefit from synergies between these frameworks. For example, can concepts such as common mode failures from downstream safety be used to help assess the strength of AI guardrails? Further, can the understanding of the capabilities and limitations of frontier AI be used to inform downstream safety analysis, e.g. where LLMs are fine-tuned to calculate voyage plans for autonomous vessels? The paper identifies some promising avenues to explore and outlines some challenges in achieving synergy, or a confluence, between upstream and downstream safety frameworks.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

3 extracted references · 3 linked inside Pith

[2]

incidents

Safety case to keep numbers of incidents below a prescribed level with red team validation pre-deployment. 3. Prevention of access to the model capabilities (an open research problem). The description of level 2 doesn’t use the downstream terminology for safety cases but implicitly it includes goals (what is to be demonstrated), strategies for decomposing...

arXiv 2024
[9]

Should healthcare providers do safety cases? Lessons from a cross-industry review of safety case practices

Radio Technical Commission for Aeronautics (RTCA), Software Considerations in Airborne Systems and Equipment Certification, RTCA DO-178C/EUROCAE ED- 12C, 2011. [10] International Organisation for Standardisation (ISO), ISO 14971:2019 Medical devices — Application of risk management to medical devices, 2019. [11] International Organisation for Standardisat...

arXiv 2016
[26]

A principles-based ethics assurance argument pattern for AI and autonomous systems

Porter, et al. "A principles-based ethics assurance argument pattern for AI and autonomous systems." AI and Ethics 4.2 (2024): 593-616. [27] Burr, C, and Leslie D. "Ethical assurance: a practical approach to the responsible design, development, and deployment of data-driven technologies." AI and Ethics 3.1 (2023): 73-98. [28] Hawkins R, Osborne M, Parsons...

arXiv 2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [2]

incidents

Safety case to keep numbers of incidents below a prescribed level with red team validation pre-deployment. 3. Prevention of access to the model capabilities (an open research problem). The description of level 2 doesn’t use the downstream terminology for safety cases but implicitly it includes goals (what is to be demonstrated), strategies for decomposing...

arXiv 2024

[2] [9]

Should healthcare providers do safety cases? Lessons from a cross-industry review of safety case practices

Radio Technical Commission for Aeronautics (RTCA), Software Considerations in Airborne Systems and Equipment Certification, RTCA DO-178C/EUROCAE ED- 12C, 2011. [10] International Organisation for Standardisation (ISO), ISO 14971:2019 Medical devices — Application of risk management to medical devices, 2019. [11] International Organisation for Standardisat...

arXiv 2016

[3] [26]

A principles-based ethics assurance argument pattern for AI and autonomous systems

Porter, et al. "A principles-based ethics assurance argument pattern for AI and autonomous systems." AI and Ethics 4.2 (2024): 593-616. [27] Burr, C, and Leslie D. "Ethical assurance: a practical approach to the responsible design, development, and deployment of data-driven technologies." AI and Ethics 3.1 (2023): 73-98. [28] Hawkins R, Osborne M, Parsons...

arXiv 2024