pith. sign in

arxiv: 2605.18755 · v1 · pith:UMB5S7W2new · submitted 2026-03-06 · 💻 cs.DC · cs.DB

Operational Memory Architecture for Kubernetes:Preserving Causal Context Across the Evidence Horizon

Shamsher Khan This is my paper

Pith reviewed 2026-05-21 12:20 UTC · model grok-4.3

classification 💻 cs.DC cs.DB
keywords Kubernetescausal chainsevidence preservationoperational memoryOOMKillConfigMappod lifecyclecrash loops
0
0 comments X

The pith

The Operational Memory Architecture preserves causal failure evidence in Kubernetes by capturing events into chains before the evidence horizon overwrites LastTerminationState.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Kubernetes clusters lose critical diagnostic context during pod restarts because the LastTerminationState field is overwritten shortly after each failure. This paper defines that loss point as the evidence horizon and introduces the Operational Memory Architecture to retain causal chains of events instead. OMA uses three explicit patterns to encode common failure modes and stores them in a lightweight database for later inspection. Experiments show the approach adds negligible overhead while keeping evidence available even after repeated crash loops.

Core claim

OMA encodes evidence retention and causal reconstruction as explicit architectural requirements. It captures operational events into causal chains using three patterns: P001 for OOMKill chains, P002 for ConfigMap variable misconfiguration, and P003 for ConfigMap volume mount propagation. The architecture preserves this evidence before the evidence horizon is crossed, as shown by a Go-based watcher and SQLite store that maintains mean causal edge latency below 1 ms and under 10 MB memory use.

What carries the argument

Operational Memory Architecture (OMA), which builds causal chains from operational events using three defined patterns and stores them to retain evidence past the evidence horizon.

If this is right

  • Causal failure context remains available for inspection after pod restarts in crash loops.
  • Mean latency for building causal edges stays below 1 ms under load.
  • Memory usage remains under 10 MB while processing events at roughly 2.8 per second.
  • The three patterns cover diagnostically valuable context for typical failures.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same retention approach could apply to other container orchestration systems that rotate event state.
  • Adding patterns for network or storage failures would expand coverage without changing the core store.
  • Operators could query the stored chains directly instead of reconstructing context from scattered logs.

Load-bearing premise

The three defined patterns are sufficient to encode the diagnostically valuable context across typical pod lifecycle transitions.

What would settle it

A stress test with 20 crash-looping pods that shows causal chains for OOMKill or ConfigMap events are missing after multiple restarts would falsify the preservation claim.

Figures

Figures reproduced from arXiv: 2605.18755 by Shamsher Khan.

Figure 1
Figure 1. Figure 1: The 90-second evidence horizon in Kubernetes. [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: OMA four-layer architecture. Layer 1 (Go collector) subscribes to Kubernetes API watch streams via [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: P001 causal edge graph from the AKS run ( [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
read the original abstract

Kubernetes clusters generate rich operational events during pod lifecycle transitions, yet the platform's native event retention model discards the most diagnostically valuable context. The LastTerminationState field, which records a container's last failure, is overwritten shortly after a pod restart. We define this as the evidence horizon. During high-frequency crash loops, this horizon may be crossed multiple times before inspection, permanently losing critical evidence. This paper introduces the Operational Memory Architecture (OMA) to preserve causal failure evidence before event rotation. OMA encodes evidence retention and causal reconstruction as explicit architectural requirements. It captures operational events into causal chains using three patterns: P001 (OOMKill chain), P002 (ConfigMap variable misconfiguration), and P003 (ConfigMap volume mount propagation). We implement OMA as an open-source system with a Go-based Kubernetes watcher, SQLite operational memory store, and a simple query interface. Experiments on Minikube and AKS include a 30-run latency analysis and stress tests with up to 20 crash-looping pods. Causal edges are built with mean latency below 1 ms. The collector processes ~2.8 events/sec while using under 10 MB memory, showing minimal overhead and effective evidence preservation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The manuscript claims that Kubernetes discards diagnostically valuable context when the LastTerminationState field is overwritten after pod restarts (termed the evidence horizon), and introduces the Operational Memory Architecture (OMA) to preserve causal failure evidence. OMA encodes retention and reconstruction via three explicit patterns (P001 OOMKill chain, P002 ConfigMap variable misconfiguration, P003 ConfigMap volume mount propagation), implemented as an open-source Go-based Kubernetes watcher with SQLite storage and a query interface. Experiments on Minikube and AKS report mean causal-edge latency below 1 ms, throughput of ~2.8 events/sec, and memory use under 10 MB in stress tests with up to 20 crash-looping pods.

Significance. If the preservation mechanism generalizes, OMA supplies a low-overhead, practical system for retaining causal operational context in Kubernetes, which could improve debugging of transient failures. The open-source implementation together with concrete 30-run latency measurements and stress-test results on both local and cloud platforms constitutes a tangible engineering contribution.

major comments (1)
  1. The central preservation claim rests on the assumption that the three hardcoded patterns (P001–P003) are sufficient to capture diagnostically valuable context across typical pod lifecycle transitions. The experiments and analysis address only crash-loop scenarios that map directly onto these patterns; no data or argument is supplied for other common transitions such as ImagePullBackOff, Evicted, or FailedScheduling, leaving the generalizability of the evidence-retention guarantee unsupported.
minor comments (2)
  1. The latency and throughput figures are presented without error bars, variance, or statistical significance measures, and no baseline comparison to native Kubernetes event handling is provided.
  2. Details on how causal edges are validated (e.g., ground-truth construction or manual inspection) are absent from the experimental description.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive review and for recognizing the practical engineering contribution of the open-source implementation and measurements. We address the single major comment below regarding the scope of the patterns and generalizability.

read point-by-point responses

  1. Referee: The central preservation claim rests on the assumption that the three hardcoded patterns (P001–P003) are sufficient to capture diagnostically valuable context across typical pod lifecycle transitions. The experiments and analysis address only crash-loop scenarios that map directly onto these patterns; no data or argument is supplied for other common transitions such as ImagePullBackOff, Evicted, or FailedScheduling, leaving the generalizability of the evidence-retention guarantee unsupported.

    Authors: We agree that the reported experiments and analysis are confined to crash-loop scenarios that exercise P001–P003. The manuscript presents these three patterns as concrete, representative encodings of causal chains that cross the evidence horizon (LastTerminationState overwrite), rather than an exhaustive catalog. The OMA watcher itself is pattern-agnostic: it ingests all relevant Kubernetes events and stores them in the SQLite operational memory before rotation occurs. We will revise the manuscript to (1) add an explicit Scope and Limitations subsection clarifying that P001–P003 illustrate the mechanism for termination-related failures and that other states (e.g., ImagePullBackOff, Evicted, FailedScheduling) would require additional pattern definitions, and (2) argue that the core architectural guarantee—capturing causal context prior to the evidence horizon—remains applicable once the appropriate patterns are supplied. No new experimental data for those states will be added in this revision, as that would constitute a separate study. revision: yes

Circularity Check

0 steps flagged

No circularity: implementation and measurement study with no derivations or fitted predictions

full rationale

The paper introduces OMA as an explicit architectural design implemented via a Go watcher and SQLite store, then measures its performance on crash-loop scenarios using the three author-defined patterns P001-P003. No equations, first-principles derivations, or statistical predictions appear; the patterns are presented as chosen encoding mechanisms rather than outputs derived from data or prior results. The work is therefore self-contained as an engineering artifact whose claims rest on direct implementation and empirical overhead measurements rather than any reduction to inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The central claim rests on the Kubernetes native event model discarding LastTerminationState after restart and on the sufficiency of the three patterns to capture causal context; no free parameters are fitted in the abstract, no new physical entities are postulated, and background assumptions are standard Kubernetes behavior.

axioms (1)
  • domain assumption Kubernetes LastTerminationState is overwritten shortly after pod restart, creating an evidence horizon that loses diagnostic context during crash loops.
    Stated directly in the abstract as the motivating problem; treated as given platform behavior.
invented entities (1)
  • Operational Memory Architecture (OMA) no independent evidence
    purpose: Explicit architectural requirement for evidence retention and causal reconstruction using patterns P001-P003.
    New named system introduced to address the evidence horizon; no independent falsifiable prediction outside the implementation is provided in the abstract.

pith-pipeline@v0.9.0 · 5740 in / 1411 out tokens · 40929 ms · 2026-05-21T12:20:38.929721+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

  • IndisputableMonolith/Foundation/ArrowOfTime.lean arrow_from_z unclear
    ?
    unclear

    Relation between the paper passage and the cited Recognition theorem.

    OMA encodes evidence retention and causal reconstruction as explicit architectural requirements. It captures operational events into causal chains using three patterns: P001 (OOMKill chain), P002 (ConfigMap variable misconfiguration), and P003 (ConfigMap volume mount propagation).

  • IndisputableMonolith/Foundation/ArithmeticFromLogic.lean embed_add unclear
    ?
    unclear

    Relation between the paper passage and the cited Recognition theorem.

    The causal edges OMA constructs are analogous to happened-before relationships [9]: if an OOMKillEvidence event e2 is observed for pod P within 90 seconds of an OOMKill event e1 for the same pod, then e1 → e2 in the happened-before sense.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

12 extracted references · 12 canonical work pages

  1. [1]

    Prometheus: From metrics to insight,

    Prometheus Authors, “Prometheus: From metrics to insight,” prometheus.io, 2024. [Online]. Available: https://prometheus.io

    work page 2024

  2. [2]

    Canopy: An end-to-end performance tracing and analysis system,

    J. Kaldor, J. Mace, M. Bejda, E. Gao, W. Kuropatwa, J. O’Neill, K. W. Ong, B. Schaller, P. Shan, B. Viscomi, V . Venkataraman, K. Veeraraghavan, and Y . J. Song, “Canopy: An end-to-end performance tracing and analysis system,” inProc. ACM Symp. Oper. Syst. Princ. (SOSP), Shanghai, China, 2017, pp. 34–50

    work page 2017

  3. [3]

    Gormley and Z

    C. Gormley and Z. Tong,Elasticsearch: The Definitive Guide.Sebastopol, CA, USA: O’Reilly Media, 2015

    work page 2015

  4. [4]

    OpenTelemetry specification,

    OpenTelemetry Authors, “OpenTelemetry specification,” opentelemetry.io, 2024. [Online]. Available: https://opentelemetry.io/docs/specs/otel/

    work page 2024

  5. [5]

    Kubernetes documentation,

    The Kubernetes Authors, “Kubernetes documentation,” kubernetes.io, 2024. [Online]. Available: https://kubernetes.io/docs/

    work page 2024

  6. [6]

    Drain: An online log parsing approach with fixed depth tree,

    P. He, J. Zhu, Z. Zheng, and M. R. Lyu, “Drain: An online log parsing approach with fixed depth tree,” inProc. IEEE Int. Conf. Web Services (ICWS), Honolulu, HI, USA, 2017, pp. 33–40

    work page 2017

  7. [7]

    Auditing,

    The Kubernetes Authors, “Auditing,” kubernetes.io, 2024. [Online]. Available: https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/

    work page 2024

  8. [8]

    Pearl,Causality: Models, Reasoning, and Inference,2nd ed

    J. Pearl,Causality: Models, Reasoning, and Inference,2nd ed. Cambridge, U.K.: Cambridge Univ. Press, 2009

    work page 2009

  9. [9]

    Time, clocks, and the ordering of events in a distributed system,

    L. Lamport, “Time, clocks, and the ordering of events in a distributed system,”Commun. ACM, vol. 21, no. 7, pp. 558–565, Jul. 1978

    work page 1978

  10. [10]

    CloudRCA: A root cause analysis framework for cloud computing platforms,

    W. Wang, M. Chen, J. Zhang, S. Qin, A. Qin, X. Ding, P. Chen, and Y . Kang, “CloudRCA: A root cause analysis framework for cloud computing platforms,” inProc. ACM Int. Conf. Inf. Knowl. Manage. (CIKM), Queensland, Australia, 2021, pp. 4373–4382

    work page 2021

  11. [11]

    Borg, Omega, and Kubernetes,

    B. Burns, B. Grant, D. Oppenheimer, E. Brewer, and J. Wilkes, “Borg, Omega, and Kubernetes,”ACM Queue, vol. 14, no. 1, pp. 70–93, Jan. 2016, DOI: 10.1145/2898442.2898444

    work page doi:10.1145/2898442.2898444 2016

  12. [12]

    Velero: Backup and migrate Kubernetes applications,

    VMware Tanzu, “Velero: Backup and migrate Kubernetes applications,” velero.io, 2024. [Online]. Available: https://velero.io

    work page 2024