pith. sign in

arxiv: 1907.05636 · v2 · pith:DM2MVPS4new · submitted 2019-07-12 · 💻 cs.MA · cs.AI· cs.DC· cs.SY· eess.SY

From Observability to Significance in Distributed Information Systems

Mark Burgess This is my paper

Pith reviewed 2026-05-24 22:14 UTC · model grok-4.3

classification 💻 cs.MA cs.AIcs.DCcs.SYeess.SY
keywords observabilitydistributed systemscausalityinformation transmissionmonitoringforensicssignificancejournals and logs
0
0 comments X

The pith

A promise theoretic model based on distinguishability and causality defines three distinct views of how information is transmitted and lost in distributed systems.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes a model for understanding observability in distributed information systems by using elementary distinguishability of observations and classical causality with history. It seeks three distinct views of a system across scales to describe information transmission and loss as data moves around and gets aggregated into journals and logs. This matters because monitoring and tracing processes for debugging and forensics have left measurement principles implicit in computer science. If the model holds, it allows deciding the significance of process behaviours by telling stories based on what can be observed.

Core claim

The central claim is that a simple promise theoretic model, based on distinguishability of observations and classical causality with history, can define three distinct views of a system that explain how information is transmitted and lost as it moves around the system at different scales, aggregated into journals and logs.

What carries the argument

The promise theoretic model that separates three views using elementary distinguishability and classical causality to track information flow and loss.

If this is right

  • Information transmission and loss can be modeled explicitly across multiple scales in distributed systems.
  • The aggregation of data into journals and logs can be analyzed through these three views.
  • Observability challenges in monitoring, debugging, and forensics can be addressed using distinguishability and causality.
  • Significance of behaviours can be determined by separating the views of the system.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Applying this model might improve the design of tracing tools by making the three views explicit.
  • It could connect observability in IT systems to similar concepts in other distributed processes like biological or social systems.
  • A testable extension would be to implement the model in a specific distributed application and verify if the views remain distinct.

Load-bearing premise

That a promise theoretic approach can supply the foundation to distinctly define the three views from distinguishability and causality alone.

What would settle it

A demonstration that the three views cannot be separated distinctly using only distinguishability of observations and classical causality with history.

Figures

Figures reproduced from arXiv: 1907.05636 by Mark Burgess.

Figure 1
Figure 1. Figure 1: Space and time as agent parallelism and serialism respectively. There are three kinds of story or explanation we want to be able to tell about distributed systems (figure 2): 1) The data traveller log. What a travelling data packet experiences along its journey, e.g. which software including version handled it and in what order? 2) The checkpoint visitor log, from key sign￾posts around the data processing … view at source ↗
Figure 3
Figure 3. Figure 3: A single agent, with a reference clock can be scaled into a superagent provided the agents within promise to coordinate their behaviours. Thus a collective formed from independent sources can act as a single reliably ordered source, but at a cost growing like N 2 in the number of agents. If independent agents need to coordinate their clocks, they can build on a single source of truth, by appointment to the… view at source ↗
Figure 4
Figure 4. Figure 4: Aggregation of observations from multiple sources can happen at any node in a distributed process. When causal influences come together, in this way, the confluence point becomes an effective observer of the sources that feed into it. Observers are not only human! B. Shared resource counters (kernel metrics) The consequence of lemma 6 is that scaling of ob￾servations causes not only a reduction of informat… view at source ↗
Figure 5
Figure 5. Figure 5: Causally ordered change in a process, and informa￾tion observed about the process are two distinct things. As long as the observation of the process retains the order of the process, inferences about causality can be made, regardless of whether the system itself could be reversed. You can trace the source of the Nile, but you can’t make the river flow backwards. sidered persistent or even invariant by the … view at source ↗
Figure 6
Figure 6. Figure 6: Causal order may be different from clock time. It is generated by prerequisite dependencies: either by underlying topology or by constraint. Agents can only trust directly agents that they are in scope of (in practice, their direct neighbours), as they have no calibrated information about the promises of agents. Agent A promises to listen for a DNS lookup (a query, i.e. an invitation to reply). As long as … view at source ↗
Figure 7
Figure 7. Figure 7: As data get propagated farther from their initial con￾text, their original meaning is degraded, unless all context is transported with them. Each of the rings may represent an intermediate agent that may or may only promise to forward data selectively or after distortion. becomes of largely forensic interest. System designers need to find expedient ways to compress context and filter it: what can remain lo… view at source ↗
Figure 8
Figure 8. Figure 8: An expanded view of a compressed local checkpoint log for a process. The first column is system clock timestamp which provides approximate time of day context for relating events to human scales. The next fields use interior monotonic counters that increment on SignPost events, even through concurrent coroutines. Each event points back to the preceding event, to give actual causal history. Explanations of … view at source ↗
Figure 9
Figure 9. Figure 9: The four kinds of promise that spacetime can express: i) containment, ii) succession, iii) local attributes, and iv) proximity. Although we can distinguish different sub-types of these four, it’s hypothesize that the four are necessary and sufficient for describing observable phenom￾ena. Generalization is not as in taxonomy: a con￾cept may have any number of generaliza￾tions, i.e. there is no unique typolo… view at source ↗
Figure 10
Figure 10. Figure 10: An excerpt of a map of invariants, generated by a search. Invariants are accumulated from distributed and concurrent processes and their relationships are classified by the four spacetime types. The pathways through these relationships tell different kinds of stories. The excerpt shown involves expansive reasoning: combining generalization and causality. scope Past (retarded) Future (advanced) causality … view at source ↗
Figure 12
Figure 12. Figure 12: The assessment of proximity between agents may seem to imply something about the orthogonal semantics above, but this is ambiguous. • Arriving at each new concept, we follow promises to generalize and specialize the con￾cept to find all links arising from the col￾lective generalized concept, and follow these along different story paths. In other words, we multiply the number of stories by concep￾tual asso… view at source ↗
Figure 13
Figure 13. Figure 13: The scope of knowledge about spacelike informa￾tion is accumulated as memory from past events propagated into a model of the present. XI. MODELS, SHARDING, IDEMPOTENCE, AND FORGETTING From sampling of data at the edge of a network, to actionable insight, there is a chain of reasoning to monitoring that starts with observability and ends with the deletion of irrelevant and antiquated data: 1) Data collecti… view at source ↗
read the original abstract

To understand and explain process behaviour we need to be able to see it, and decide its significance, i.e. be able to tell a story about its behaviours. This paper describes a few of the modelling challenges that underlie monitoring and observation of processes in IT, by human or by software. The topic of the observability of systems has been elevated recently in connection with computer monitoring and tracing of processes for debugging and forensics. It raises the issue of well-known principles of measurement, in bounded contexts, but these issues have been left implicit in the Computer Science literature. This paper aims to remedy this omission, by laying out a simple promise theoretic model, summarizing a long standing trail of work on the observation of distributed systems, based on elementary distinguishability of observations, and classical causality, with history. Three distinct views of a system are sought, across a number of scales, that described how information is transmitted (and lost) as it moves around the system, aggregated into journals and logs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 2 minor

Summary. The paper claims to remedy implicit measurement issues in the computer science literature on monitoring and tracing by presenting a promise-theoretic model of observability in distributed systems. Grounded in elementary distinguishability of observations and classical causality with history, the model seeks three distinct views of a system across scales that describe how information is transmitted and lost as it moves through the system and is aggregated into journals and logs.

Significance. If the model holds, it supplies an explicit conceptual framework for reasoning about observability and significance in IT processes, extending a long-standing line of work on promise theory to make measurement principles less implicit. The contribution is primarily foundational and synthetic rather than a new derivation or empirical result.

minor comments (2)
  1. [Abstract] The abstract states that three distinct views are sought but does not name or briefly characterize them; adding a short enumeration in the abstract or introduction would improve immediate clarity for readers.
  2. The manuscript is positioned as a summary of prior modeling; ensuring that any new diagrams or definitions of the three views are cross-referenced to the specific earlier publications would help readers trace the development without ambiguity.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for their positive assessment of the manuscript and for recommending acceptance. The review accurately captures the paper's intent to make measurement principles explicit in the context of distributed systems monitoring using promise theory.

Circularity Check

1 steps flagged

Core model rests on author's prior promise theory without independent external derivation

specific steps
  1. self citation load bearing [Abstract]
    "This paper aims to remedy this omission, by laying out a simple promise theoretic model, summarizing a long standing trail of work on the observation of distributed systems, based on elementary distinguishability of observations, and classical causality, with history."

    The central contribution is defined as a promise theoretic model whose justification is a summary of the author's own prior trail of work on the topic. The premise that promise theory supplies the appropriate foundation for observability thus reduces to self-developed concepts without citation to independent external benchmarks or derivations in the abstract.

full rationale

The paper explicitly frames its contribution as laying out a promise theoretic model that summarizes the author's own long-standing trail of work. This makes the foundational premise load-bearing on self-developed concepts (promise theory) rather than independent benchmarks or external derivations. No equations or new predictions are shown in the provided text that reduce by construction, but the central modeling framework itself depends on the self-referential premise flagged in the abstract. This qualifies as partial circularity per the self-citation load-bearing pattern, but the work is positioned as conceptual summary rather than a forced new result, preventing a higher score.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central claim rests on the applicability of promise theory to observability and the sufficiency of distinguishability plus causality for defining three system views; these are domain assumptions drawn from prior author work rather than derived or externally validated in the abstract.

axioms (2)
  • domain assumption Promise theory provides a suitable foundation for modeling system behaviors, observations, and significance.
    Invoked when the abstract states that the model is built on promise theory to address implicit measurement principles.
  • domain assumption Elementary distinguishability of observations combined with classical causality is sufficient to define three distinct system views across scales.
    Stated when the abstract describes seeking three views based on these elements.

pith-pipeline@v0.9.0 · 5700 in / 1344 out tokens · 22927 ms · 2026-05-24T22:14:48.557260+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

54 extracted references · 54 canonical work pages · 5 internal anchors

  1. [1]

    Hoogenboom and J

    P. Hoogenboom and J. Lepreau. Computer system perfor- mance problem detection using time series models. Pro- ceedings of the USENIX Technical Conference, (USENIX Association: Berkeley, CA), page 15, 1993

    work page 1993

  2. [2]

    Metrics for management

    C Hogan. Metrics for management. Proceedings of the Ninth Systems Administration Conference (LISA IX) (USENIX As- sociation: Berkeley, CA, page 125, 1995

    work page 1995

  3. [3]

    Hellerstein

    J.L. Hellerstein. An approach to selecting metrics for detecting performance problems in information systems. Per- formance Evaluation Review , 24:266, 1996

    work page 1996

  4. [4]

    Gonzalez Prieto and R

    A. Gonzalez Prieto and R. Stadler. Adaptive distributed monitoring with accuracy objectives. ACM SIGCOMM workshop on Internet Network Management (INM 06), Pisa, Italy, 2006

    work page 2006

  5. [5]

    Sekar, T

    R. Sekar, T. Bowen, and M. Segal. On preventing intru- sions by process behaviour monitoring. Proceedings of the workshop on intrusion detection and network monitoring , USENIX, 1999

    work page 1999

  6. [6]

    Dasgupta and S

    D. Dasgupta and S. Forest. An anomaly detection algorithm inspired by the immune system. Artifical immune systems and their applications , page 262, 1998

    work page 1998

  7. [7]

    opentracing.io, 2017

    The open tracing initiative. opentracing.io, 2017

    work page 2017

  8. [8]

    W. Louth. ObservabilityŁŁtraces and trees. Medium article (@autoletics), 2019

    work page 2019

  9. [9]

    W. Louth. ObservabilityŁŁtraces and tags. Medium article (@autoletics), 2019

    work page 2019

  10. [10]

    Seltzer and C

    M.I. Seltzer and C. Small. Self-monitoring and self-adapting operating systems. Proceedings of the Sixth workshop on Hot Topics in Operating Systems,Cape Cod, Massachusetts, USA. IEEE Computer Society Press , 1997

    work page 1997

  11. [11]

    Cradley Chen, Y

    J. Cradley Chen, Y . Endo, D. Mazieres, A. Dias, M. Seltzer, and M.D. Smith. The measured performance of personal computer operating systems. ACM transactions on computing systems and Proceedings of the 15th ACM symposium on Operating System Principles , 1995. 13The question of whether to invest in promising an expensive and late consensus over a coarse g...

    work page 1995

  12. [12]

    P. A. Porras and P. G. Neumann. EMERALD: Event mon- itoring enabling responses to anomalous live disturbances. In Proc. 20th NIST-NCSC National Information Systems Security Conference, pages 353–365, 1997

    work page 1997

  13. [13]

    H. Abdu, H. Lutfiya, and M. Bauer. A model for adaptive monitoring configurations. Proceedings of the VI IFIP/IEEE IM conference on network management , page 371, 1999

    work page 1999

  14. [14]

    A Big Data Analyzer for Large Trace Logs

    A. Balliu, D. Olivetti, O. Babaoglu, M. Marzolla, , and A. Sirbu. A big data analyzer for large trace logs. arXiv:1509.00773v1 [cs.DC], 2015

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  15. [15]

    Generating representative web workloads for network and server performance evalua- tion

    Paul Barford and Mark Crovella. Generating representative web workloads for network and server performance evalua- tion. In SIGMETRICS ’98/PERFORMANCE ’98: Proceed- ings of the 1998 ACM SIGMETRICS joint international con- ference on Measurement and modeling of computer systems , pages 151–160, New York, NY , USA, 1998. ACM Press

    work page 1998

  16. [16]

    Paxos Made Simple

    Leslie Lamport. Paxos Made Simple. SIGACT News , 32(4):51–58, December 2001

    work page 2001

  17. [17]

    In search of an un- derstandable consensus algorithm

    Diego Ongaro and John Ousterhout. In search of an un- derstandable consensus algorithm. In Proceedings of the 2014 USENIX Conference on USENIX Annual Technical Conference, USENIX ATC’14, pages 305–320, Berkeley, CA, USA, 2014. USENIX Association

    work page 2014

  18. [18]

    Bergstra and M

    J.A. Bergstra and M. Burgess. Promise Theory: Principles and Applications. χtAxis Press, 2014

    work page 2014

  19. [19]

    An approach to understanding policy based on autonomy and voluntary cooperation

    Mark Burgess. An approach to understanding policy based on autonomy and voluntary cooperation. In IFIP/IEEE 16th international workshop on distributed systems operations and management (DSOM), in LNCS 3775 , pages 97–108, 2005

    work page 2005

  20. [20]

    M. Burgess. Spacetimes with semantics (i). http://arxiv.org/abs/1411.5563, 2014

    work page internal anchor Pith review Pith/arXiv arXiv 2014

  21. [21]

    M. Burgess. Spacetimes with semantics (ii). http://arxiv.org/abs/1505.01716, 2015

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  22. [22]

    M. Burgess. Spacetimes with semantics (iii). http://arxiv.org/abs/1608.02193, 2016

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  23. [23]

    M. Burgess. A spacetime approach to generalized cognitive reasoning in multi-scale learning. https://arxiv.org/abs/1702.04638, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  24. [24]

    Burgess and A

    M. Burgess and A. Couch. On system rollback and totalized fields: An algebraic approach to system change. J. Log. Algebr. Program., 80(8):427–443, 2011

    work page 2011

  25. [25]

    Shannon and W

    C.E. Shannon and W. Weaver. The mathematical theory of communication. University of Illinois Press, Urbana, 1949

    work page 1949

  26. [26]

    Cover and J.A

    T.M. Cover and J.A. Thomas. Elements of Information Theory. (J.Wiley & Sons., New York), 1991

    work page 1991

  27. [27]

    Queueing Systems: Computer Applica- tions, volume 2

    Leonard Kleinrock. Queueing Systems: Computer Applica- tions, volume 2. John Wiley & Sons, Inc., 1976

    work page 1976

  28. [28]

    Badonnel and M

    R. Badonnel and M. Burgess. Service load balancing with autonomic servers: Reversing the decision making process. In Resilient Networks and Services, Second International Conference on Autonomous Infrastructure, Management and Security, AIMS 2008, Bremen, Germany, July 1-3, 2008, Proceedings, pages 92–104, 2008

    work page 2008

  29. [29]

    Bon ´er, D

    J. Bon ´er, D. Farley, R. Kuhn, and M. Thompson. The reactive manifesto. https://www.reactivemanifesto.org/

    work page

  30. [30]

    W.R. Ashby. Design for a brain . J. Wiley & Sons, 1952

    work page 1952

  31. [31]

    W.R. Ashby. An introduction to cybernetics . J. Wiley & Sons, 1956

    work page 1956

  32. [32]

    Time, clocks, and the ordering of events in a distributed system

    Leslie Lamport. Time, clocks, and the ordering of events in a distributed system. Commun. ACM , 21(7):558–565, July 1978

    work page 1978

  33. [33]

    Borrill, M

    P. Borrill, M. Burgess, A. Karp, and A. Kasuya. Spacetime- entangled networks (i) relativity and observability of stepwise consensus. arXiv:1807.08549 [cs.DC], 2018

    work page arXiv 2018

  34. [34]

    M. Burgess. A Treatise on Systems: Volume 2: Intentional systems with faults, errors, and flaws . in progress, 2004-

    work page 2004

  35. [35]

    Fischer, N.A

    M.J. Fischer, N.A. Lynch, and M.S. Paterson. Impossibility of distributed consensus with one faulty process. J. ACM, 32(2):374–382, April 1985

    work page 1985

  36. [36]

    M. Burgess. In Search of Certainty: the science of our information infrastructure. Xtaxis Press, 2013

    work page 2013

  37. [37]

    The new rules of sampling

    White paper. The new rules of sampling. Technical report, Honeycomb.com, 2019

    work page 2019

  38. [38]

    Cockcroft

    A. Cockcroft. Utilization is virtually useless as a metric! In Proceedings of Int. CMG Conference , pages 557–562, 2006

    work page 2006

  39. [39]

    M. Burgess. Smart Spacetime. χtAxis Press, 2019

    work page 2019

  40. [40]

    Dam and R

    M. Dam and R. Stadler. A generic protocol for network state aggregation. RVK 05, Linkping, Sweden, June 14-16 , 2005

    work page 2005

  41. [41]

    Burgess and M

    M. Burgess and M. Disney. Understanding scalability in network aggregation with continuous monitoring. In Lec- ture Notes on Computer Science, Proc. 18th IFIP/IEEE Distributed Systems: Operations and Management (DSOM 2007), volume (submitted). Springer, 2007

    work page 2007

  42. [42]

    Wuhib, M

    F. Wuhib, M. Dam, R. Stadler, and A. Clemm. Robust monitoring of network-wide aggregates through gossiping. In 10th IFIP/IEEE International Symposium on Integrated Management (IM 2007) , 2007

    work page 2007

  43. [43]

    Burgess, H

    M. Burgess, H. Haugerud, T. Reitan, and S. Straumsnes. Measuring host normality. ACM Transactions on Computing Systems, 20:125–160, 2001

    work page 2001

  44. [44]

    J. Pearl. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference . Morgen Kaufmann, San Francisco, 1988

    work page 1988

  45. [45]

    J. Pearl. Causality. Cambridge University Press, Cambridge, 2000

    work page 2000

  46. [46]

    Couch and M

    A. Couch and M. Burgess. Compass and direction in topic maps. (Oslo University College preprint) , 2009

    work page 2009

  47. [47]

    Burgess and W

    M. Burgess and W. Louth. Preserving the significance of distributed observations. unpublished, 2019

    work page 2019

  48. [48]

    Zenil, N.A

    H. Zenil, N.A. Kiani, and J. Tegn ´er. The thermodynamics of network coding and an algorithmic refinement of the principle of maximum entropy. Entropy, 21(560), 2019

    work page 2019

  49. [49]

    Koalja history package

    Aljabr Inc. Koalja history package. https://github.com/AljabrIO/ koalja-operator/tree/master/pkg/history

    work page

  50. [50]

    M. Burgess. A tiny overview of cfengine: convergent main- tenance agent. In Proceedings of the 1st International Work- shop on Multi-Agent and Robotic Systems, MARS/ICINCO , 2005

    work page 2005

  51. [51]

    L. Page, S. Brin, R. Motwani, and T. Winograd. The pagerank citation ranking: Bringing order to the web. Technical report, Stanford University, Stanford, CA , 1998

    work page 1998

  52. [52]

    Bjelland, M

    J. Bjelland, M. Burgess, G. Canright, and K. Eng-Monsen. Eigenvectors of directed graphs and importance scores: dominance, t-rank, and sink remedies. Data Mining and Knowledge Discovery, 20(1):98–151, 2010

    work page 2010

  53. [53]

    M. Burgess. A site configuration engine. Computing systems (MIT Press: Cambridge MA) , 8:309, 1995

    work page 1995

  54. [54]

    M. Burgess. Two dimensional time-series for anomaly detection and regulation in adaptive systems. Lecture Notes in Computer Science, IFIP/IEEE 13th International Work- shop on Distributed Systems: Operations and Management (DSOM 2002), 2506:169, 2002. 24

    work page 2002