Pith. sign in

REVIEW 3 major objections 4 minor 30 references

Towards Computational Provenance: Carrying Causal-State Evidence in Generated Text

T0 review · 3 major / 4 minor · reviewed 2026-08-27 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A generated text can carry an authenticated, detectable trace of which causally relevant internal state produced it, even when the final answer is unchanged.

desk verdict A clean toy demonstration of state-keyed watermarking, but the 'computational provenance' framing is larger than the experiment supports: the signal is inserted by a trusted controller, not carried by the model's own computation. read the letter →

arxiv 2608.16868 v1 pith:PDFSRGBM submitted 2026-08-17 cs.CL cs.AI

Benjamin Belay This is my paper · ORCID
classification cs.CLcs.AI
keywords computationalprovenancecausal-stateevidenceinternalstateverificationtextwatermarkingHMACreceiptschain-of-thoughtfaithfulnesslanguagemodelinterpretabilityanswer-equivalentexecutions
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that, in a controlled setting, the internal computation behind generated text can leave a detectable signature in the text itself. It builds two small models, a modular feed-forward network and a transformer, that solve an arithmetic task through a mandatory passage through two discrete intermediate states, and it arranges for two different internal paths to yield the same final answer. The state actually used is cryptographically authenticated, and that authenticated state biases the wording of a fixed-content report; a fixed detector then recovers the correct state from the wording alone. Both architectures passed all 128 matched pairs in public and sealed protected evaluations, and the causal pathway reproduced across multiple independently trained instances. The result matters because it suggests a route toward auditing language model outputs that does not rely on the model's own explanations.

What carries the argument

The load-bearing identity is modular arithmetic on the task: z3=(5z2+d) mod 16 and the final output y=z3 mod 8, so substituting z2' = (z2+8) mod 16 gives z3' = (z3+8) mod 16 while leaving y unchanged. This creates paired executions with identical answers but distinct internal paths. Around this identity the paper builds a mandatory discrete-state pathway (two separate modules or transformer encoders, with the answer head receiving only z3), an HMAC-based receipt that authenticates the observed state, and a statistical text carrier: eight template sentences with six variable word positions each, where four of eight alternatives per position are favoured depending on the verified z2 value, and a fixed scoring detector that must beat threshold T=4.041451884327381 and rank uniquely first. The same identity and carrier run in both architectures, with the transformer reusing the feed-forward model's calibrated signal and detector.

What would settle it

Train a paired version of the model in which the z3 module receives d but not z2, or in which the causal intervention on z2 has no effect on z3, while still recording z2, generating the receipts, and applying the state-dependent word bias; if the fixed detector still recovers the verified z2 from the text at 128/128 on a protected set, then the text signal alone, rather than the causally relevant state, explains the result.

Watch

Extended reading notes

Core claim

The central result is a controlled proof of concept for computational provenance: a verified, causally relevant internal state can determine a detectable pattern in generated text even when the prompt, final answer, semantic content, and sampling randomness are fixed. On the arithmetic task, replacing z2 by (z2+8) mod 16 changes z3 to (z3+8) mod 16 while leaving y=z3 mod 8 unchanged, so the same answer is reached through different z2 to z3 paths. The paper authenticates which path occurred with HMAC-protected receipts, lets the verified z2 value select which words are statistically favoured in a 24-position report, and shows that a threshold-based detector identifies the correct state in every one of 128 matched pairs, both publicly and on a sealed protected set, for a modular feed-forward network and for a transformer built from two encoder stages. A separate answer-only transformer, trained without supervision on z2 or z3, learned the task but yielded no probe-recoverable z2 state, so the authors present the result as a proof of concept rather than as evidence that large pretrained models naturally expose such states.

Load-bearing premise

The entire chain rests on a trustworthy external controller that observes the internal state, signs it with a secret HMAC key, and then uses that verified state to bias the text generator; if that instrumentation, verifier, or key is compromised, or if no comparable trusted observer exists inside a real language model, the detected text pattern is indistinguishable from an ordinary watermark attached from the outside.

Editorial extensions

If this is right

  • Two executions with the same prompt, same final answer, and same position-indexed sampling draws can still be distinguished by the statistical wording pattern, so identical output no longer implies identical internal computation.
  • The combination of causal intervention tests, authenticated receipts, and text-carried signal is what carries the provenance claim; a receipt alone or a watermark alone would not tie the text to the actual internal state.
  • The mechanism transfers across architectures: the engineered transformer passed public and protected evaluations using the same 16 state patterns, generator, detector, and thresholds as the feed-forward model.
  • Because answer-only transformers did not expose a probe-recoverable intermediate state, extending the approach to real language models will require constructing or learning provenance-ready states, not just reading them off.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the same carrier is robust to paraphrasing, editing, or different decoding strategies, which this paper does not test, computational provenance could be used to audit chain-of-thought traces by marking which reasoning steps were actually causally connected to the answer.
  • The finite-sample calibration bound (false-positive probability at most 2/513 in the registered setting) suggests the detection machinery could be packaged as a verifiable claim registry, where each state-conditioned report is a signed assertion that can be checked without retraining the detector.
  • A natural next experiment is to test whether the detector still works when the state-dependent word bias is applied without the causal pathway, for instance by making z3 independent of z2; if it does, the text signal alone, rather than the causal link, is doing the work.
  • The negative probe result hints that real LLMs may need explicit induction of discrete, causally active states, for example by training or steering the model to pass through them, before computational provenance can move beyond toy arithmetic.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Request a human review

A listed scientist reviews the paper for a fee and the review publishes here regardless of verdict. See the reviewers or get listed.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces 'computational provenance' as a protocol in which a causally relevant intermediate state of a small arithmetic model is cryptographically verified and then used to bias word choices in a constrained text generator, so that the generated text carries a detectable statistical pattern identifying the state. The authors test this in a modular feed-forward network and a small two-encoder transformer, using the identity that (z2 + 8) mod 16 propagates to (z3 + 8) mod 16 while leaving the final answer unchanged, to create answer-equivalent execution paths. They report 128/128 matched-pair success on public and sealed protected sets for each architecture, reproducibility of the causal pathway across independent seeds, and a separate negative result in which linear probes found no qualifying natural state in answer-only transformers.

Significance. As a controlled engineering demonstration, the paper is careful: the arithmetic identity is correct; the protected set is sealed before training; detector thresholds are fixed on separate calibration data; the causal relevance of z2 is tested with multiple controls; and the pipeline is transferred across architectures without recalibration. If the central claim were established, it would be a useful step toward verifiable links between internal computation and generated output. However, the significance is limited by the design: the text signal is inserted by trusted external instrumentation after the computation, and no control tests whether causal relevance contributes at all. The result is best characterized as a state-keyed watermark with cryptographic receipts, not as evidence that a model's own computation leaves provenance in generated text. The paper's own negative natural-state result underscores this gap.

major comments (3)
  1. [3.2–3.3] The 'carried' signal is inserted by the experimenter, not produced by the model's computation. Trusted instrumentation observes z2, authenticates it, and the authenticated value is then used to bias a separate text generator's word choices. The causal chain from internal state to output therefore passes through the external controller, not through the learned z2→z3→y pathway. The intervention tests in Section 3.1 show that z2 affects z3 and the answer, but they do not show that the computation itself leaves any trace in the generated text. The 128/128 results validate an externally attached, state-keyed watermark. Add a control in which the signal is keyed to a causally irrelevant quantity (for example, the auxiliary z1, which is not a model state, or a random coin) and show that causal relevance changes detection behavior; without this control, the claimed distinction from ordinary watermarking is unsupported.
  2. [5.3 and 6.2] The paper's own results and limitations make the gap in the previous comment load-bearing. In the answer-only transformers, no qualifying natural state was found at any of the 20 probed sites (best full-z2 accuracy 23.80%, best within-pair balanced accuracy 50.39%), and Section 6.2 explicitly says the statistical signal 'should not be interpreted as a trust-free or output-only proof.' Thus the only demonstrated mechanism requires a trusted external controller to inject the signal; there is no positive evidence that a model's own computation can carry provenance. The authors should either implement the irrelevant-key control and compare outcomes, or narrow the stated contribution to externally mediated state verification rather than 'computational provenance' as defined in the introduction.
  3. [4.3 and 5.2, Table 1] The full provenance pipeline is evaluated on only one designated model per architecture, despite the abstract's phrase 'across five independently trained feed-forward models and three independently trained transformers.' Table 1's robustness rows concern only the causal pathway, whereas the end-to-end provenance rows are single-model results. If the cross-model claim is intended to cover the complete provenance mechanism, the authors need full-pipeline evaluations on multiple independent models; otherwise the wording should be restricted to single designated models.
minor comments (4)
  1. [5.2] The phrase 'the same 16 authorities' is unclear; it should read 'the same 16 state-specific patterns' or 'the same 16 authority states'.
  2. [B.4] The example matched pair prints 'z2 is 4' in both outputs even though the authenticated states are 2 and 10; the accompanying note explains this, but the inline presentation is confusing and would benefit from a visual distinction between printed report fields and authenticated internal states.
  3. [B.2] The finite-sample calibration guarantee of 2/513 is stated for future null examples, not for the signal-present matched pairs; the text should say so explicitly to avoid overstating what the bound covers.
  4. [B.3] The optional phrases list is not mapped to the six word groups; indicating whether these phrases are additional variable positions would improve reproducibility.

Circularity Check

1 steps flagged · score 6.0 of 10

The end-to-end provenance result is a closed loop: the experimenter's state-conditioned word bias is detected by a scorer using the same pre-defined pattern, so the central 'carrying' claim reduces by construction to state-conditioned watermarking.

  1. self definitional [Section 3.2 and Section 3.3 (state verification and carrying into generated text)]
    "Only after these checks succeed is the authenticated value of z2 allowed to determine the statistical signal used during text generation. ... The detector then compares the 24 observed word choices with the patterns associated with each of the 16 possible values of z2. It uses a fixed statistical scoring rule, rather than a trained classifier, and gives higher scores when more of the observed words match those favoured by a candidate state."

    The 'carrying' mechanism is implemented by the controller: the authenticated z2 value selects which four of eight words in each group are favoured, and the detector recovers the state by checking which of those same pre-defined favoured subsets is over-represented. Generation and detection use the same state-to-pattern mapping, so the 128/128 end-to-end success verifies that the inserted bias survives the fixed sampling draws, not that the model's own computation left a trace in the text. The causal intervention tests show z2 affects z3 in the arithmetic head, but the text bias is applied by the controller after the computation, making the central provenance claim equivalent to its input by construction.

full rationale

No self-citation chain or imported uniqueness theorem is load-bearing here; the paper is single-authored and cites only external work. The answer-only transformer experiment is an honest negative control and does not manufacture positive evidence. However, the paper's central positive result is not an inference about naturally occurring computational traces: Section 3.2 states that the authenticated z2 value 'is allowed to determine the statistical signal used during text generation,' and Section 3.3 defines the detector as scoring the same 'patterns associated with each of the 16 possible values of z2.' The claimed provenance evidence is therefore inserted by the experimenter and read back by a matched detector; the causal relevance of z2 to the arithmetic is established separately, but the text signal is not caused by the model's learned pathway. The paper is transparent about the trust assumptions and calls the result a controlled proof of concept, which prevents a higher score, but the central demonstration still reduces, by construction, to a state-conditioned watermark rather than to evidence that computation naturally leaves a detectable trace in generated text.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central construction contributes a protocol that combines known components: supervised training through discrete states, causal intervention tests, HMAC receipts, and state-conditioned word biasing. The only numbers fitted to data are detector constants. The chain from state to text is authored by the experimenter, so the ledger's key item is the explicit trust assumption on instrumentation and keys.

free parameters (4)
  • Detector threshold T = 4.041451884327381
    Calibrated as the maximum detector score over 512 no-signal closures (Appendix B.2); a number fit to data, not derived from theory.
  • Detector margin M = 0
    Set from the largest calibrated confusion margin, which was below zero; a threshold-like constant fit to calibration data.
  • Reports per execution = 8
    Chosen as a design constant in the carrier; larger counts improve detection but are not derived from the task.
  • Favoured-word preference strength = not reported
    The probability bias favoring four of eight alternatives per state is a free design parameter in Section 3.3; the paper does not state its value, so a re-implementation must choose it.
assumptions (4)
  • domain assumption Trusted instrumentation, verifier, and secret HMAC keys are not compromised.
    Explicitly stated in Sections 3.2 and 6.2; without this trust, receipts and the state-to-signal link are not credible.
  • domain assumption Calibration examples and evaluation examples are exchangeable for the finite-sample false-positive bound.
    Appendix B.2 derives the 2/513 bound from the maximum of 512 calibration closures; exchangeability is assumed but not tested.
  • ad hoc to paper The straight-through argmax with lowest-index tie-breaking implements the intended discrete state transition.
    Appendix B.1 describes discrete selection via straight-through argmax; the causal tests assume this makes z2 and z3 the actual states consumed by later modules.
  • standard math Standard HMAC security holds under the assumed key secrecy.
    Receipt integrity relies on HMAC being a secure message authentication code (Sections 3.2, A.8).
invented entities (1)
  • Cryptographic receipt independent evidence
    purpose: Records the exact or abstract value of z2 and z3 used in an execution and authenticates it with an HMAC so a verifier can detect alteration or fabrication.
    Not a new physical entity; a software artifact whose integrity is challenged by altered, replayed, reordered, mismatched, and missing records, so it has a falsifiable operational handle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Computational Provenance: Carrying Causal-State Evidence in Generated Text." pith.science (2026). https://pith.science/paper/PDFSRGBM

@misc{pith2026260816868,
  author       = {Pith},
  title        = {Pith review of: Towards Computational Provenance: Carrying Causal-State Evidence in Generated Text},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PDFSRGBM}},
  note         = {Machine review of arXiv:2608.16868}
}
read the original abstract

A language model's output does not by itself provide verifiable evidence about the internal computation that produced it. We study computational provenance: whether generated text can carry detectable evidence of which causally relevant internal state occurred. We test a bounded form of this idea in two controlled architectures: a modular feed-forward neural network and a transformer-based model. Both architectures are trained on the same arithmetic task with a mandatory pathway through two discrete intermediate states, allowing different internal paths to produce the same answer. We deliberately switch between these paths, authenticate the state actually used, and let that verified state determine a subtle statistical pattern in the generated text that can later be detected. The feed-forward and transformer systems each passed all 128 matched pairs in both their public and separately sealed protected end-to-end evaluations, with the detector recovering the signal associated with the authenticated internal state. The required causal computation also reproduced across five independently trained feed-forward models and three independently trained transformers. In a separate answer-only transformer experiment, our linear probes did not recover a naturally learned intermediate state. These results provide a controlled proof of concept that information about a verified, causally relevant internal state can be preserved in generated text even when the answer is unchanged.

Figures

Figures reproduced from arXiv: 2608.16868 by the authors.

Figure 1
Figure 1. Overview of the experimental construction, implemented with both feed-forward and [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 29 canonical work pages

  1. [1]

    Measuring Progress on Scalable Oversight for Large Language Models

    Measuring Progress on Scalable Oversight for Large Language Models , author=. arXiv preprint arXiv:2211.03540 , year=. doi:10.48550/arXiv.2211.03540 , url=

    work page Pith review arXiv doi:10.48550/arxiv.2211.03540

  2. [2]

    Hedges: A study in meaning criteria and the logic of fuzzy concepts

    Let's Verify Step by Step , author=. arXiv preprint arXiv:2305.20050 , year=. doi:10.48550/arXiv.2305.20050 , url=

    work page Pith review arXiv doi:10.48550/arxiv.2305.20050

  3. [3]

    AI safety via debate

    AI Safety via Debate , author=. arXiv preprint arXiv:1805.00899 , year=. doi:10.48550/arXiv.1805.00899 , url=

    work page Pith review arXiv doi:10.48550/arxiv.1805.00899

  4. [4]

    Sleeper Agents: Training Deceptive

    Hubinger, Evan and Denison, Carson and Mu, Jesse and Lambert, Mike and Tong, Meg and MacDiarmid, Monte and Lanham, Tamera and others , journal=. Sleeper Agents: Training Deceptive. 2024 , doi=

    work page 2024

  5. [5]

    Large Language Models can Strategically Deceive their Users when Put Under Pressure

    Large Language Models Can Strategically Deceive Their Users When Put Under Pressure , author=. arXiv preprint arXiv:2311.07590 , year=. doi:10.48550/arXiv.2311.07590 , url=

    work page Pith review arXiv doi:10.48550/arxiv.2311.07590

  6. [6]

    The Twelfth International Conference on Learning Representations , year =

    Sparse Autoencoders Find Highly Interpretable Features in Language Models , author =. The Twelfth International Conference on Learning Representations , year =

    work page

  7. [7]

    and Ameisen, Emmanuel and Chen, James and Kishylau, Dzmitry and Pearce, Adam and Tarng, Julius and Wu, Alex and Wu, Jeff and Zhang, Yang and Ziegler, Daniel M

    Fraser-Taliente, Kit and Kantamneni, Subhash and Ong, Euan and Mossing, Dan and Lu, Christina and Bogdan, Paul C. and Ameisen, Emmanuel and Chen, James and Kishylau, Dzmitry and Pearce, Adam and Tarng, Julius and Wu, Alex and Wu, Jeff and Zhang, Yang and Ziegler, Daniel M. and Hubinger, Evan and Batson, Joshua and Lindsey, Jack and Zimmerman, Samuel and M...

    work page

  8. [8]

    Verbalizable Representations Form a Global Workspace in Language Models

    Verbalizable Representations Form a Global Workspace in Language Models , author =. arXiv preprint arXiv:2607.15495 , year =

    work page Pith review arXiv

Show all 30 references
  1. [9]

    Proceedings of the 41st International Conference on Machine Learning , series=

    Weak-to-Strong Generalization: Eliciting Strong Capabilities with Weak Supervision , author=. Proceedings of the 41st International Conference on Machine Learning , series=. 2024 , url=

    2024

  2. [10]

    Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    Attention is not Explanation , author=. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=. 2019 , publisher=. doi:10.18653/v1/N19-1357 , url=

    2019 doi

  3. [11]

    arXiv preprint arXiv:2307.15771 , year=

    The Hydra Effect: Emergent Self-Repair in Language Model Computations , author=. arXiv preprint arXiv:2307.15771 , year=. doi:10.48550/arXiv.2307.15771 , url=

    doi arXiv

  4. [12]

    Advances in Neural Information Processing Systems , volume=

    Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting , author=. Advances in Neural Information Processing Systems , volume=. 2023 , url=

    2023

  5. [13]

    arXiv preprint arXiv:2307.13702 , year=

    Measuring Faithfulness in Chain-of-Thought Reasoning , author=. arXiv preprint arXiv:2307.13702 , year=. doi:10.48550/arXiv.2307.13702 , url=

    doi arXiv

  6. [14]

    Proceedings of the 40th International Conference on Machine Learning , series=

    A Watermark for Large Language Models , author=. Proceedings of the 40th International Conference on Machine Learning , series=. 2023 , url=

    2023

  7. [15]

    Nature , volume=

    Scalable Watermarking for Identifying Large Language Model Outputs , author=. Nature , volume=. 2024 , doi=

    2024

  8. [16]

    Transactions on Machine Learning Research , year=

    Robust Distortion-Free Watermarks for Language Models , author=. Transactions on Machine Learning Research , year=

  9. [17]

    International Conference on Learning Representations , year=

    Towards Codable Watermarking for Injecting Multi-Bits Information to LLMs , author=. International Conference on Learning Representations , year=

  10. [18]

    Proceedings of the 42nd International Conference on Machine Learning , series=

    StealthInk: A Multi-bit and Stealthy Watermark for Large Language Models , author=. Proceedings of the 42nd International Conference on Machine Learning , series=. 2025 , url=

    2025

  11. [19]

    International Conference on Learning Representations , year=

    Distilling the Thought, Watermarking the Answer: A Principle Semantic Guided Watermark for Reasoning Large Language Models , author=. International Conference on Learning Representations , year=

  12. [20]

    2025 , url=

    Yu, Zhuohao and Jiang, Xingru and Gu, Weizheng and Wang, Yidong and Wen, Qingsong and Zhang, Shikun and Ye, Wei , booktitle=. 2025 , url=

    2025

  13. [21]

    Proceedings of the 43rd International Conference on Machine Learning , series=

    Echoes within the Reasoning: Stealthy and Effective Watermarking via Chain of Thought , author=. Proceedings of the 43rd International Conference on Machine Learning , series=. 2026 , url=

    2026

  14. [22]

    arXiv preprint arXiv:2605.05443 , year=

    SLAM: Structural Linguistic Activation Marking for Language Models , author=. arXiv preprint arXiv:2605.05443 , year=

    arXiv

  15. [23]

    Advances in Neural Information Processing Systems , volume=

    Causal Abstractions of Neural Networks , author=. Advances in Neural Information Processing Systems , volume=. 2021 , url=

    2021

  16. [24]

    Proceedings of the 39th International Conference on Machine Learning , series=

    Inducing Causal Structure for Interpretable Neural Networks , author=. Proceedings of the 39th International Conference on Machine Learning , series=. 2022 , url=

    2022

  17. [25]

    Advances in Neural Information Processing Systems , volume=

    SafetyNets: Verifiable Execution of Deep Neural Networks on an Untrusted Cloud , author=. Advances in Neural Information Processing Systems , volume=. 2017 , url=

    2017

  18. [26]

    International Conference on Learning Representations , year=

    Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware , author=. International Conference on Learning Representations , year=

  19. [27]

    2025 , url=

    Sun, Yifan and Li, Yuhang and Zhang, Yue and Jin, Yuchen and Zhang, Huan , booktitle=. 2025 , url=

    2025

  20. [28]

    2026 , doi=

    Guan, Yongjie , journal=. 2026 , doi=

    2026

  21. [29]

    Proceedings of the 40th International Conference on Machine Learning , series=

    A Closer Look at the Intervention Procedure of Concept Bottleneck Models , author=. Proceedings of the 40th International Conference on Machine Learning , series=. 2023 , url=

    2023

  22. [30]

    Proceedings of the 37th International Conference on Machine Learning , series=

    Concept Bottleneck Models , author=. Proceedings of the 37th International Conference on Machine Learning , series=. 2020 , url=

    2020

Pith tools

Reviewed August 27, 2026 · model on record in the stance chip above.