Pith. sign in

REVIEW 4 major objections 5 minor 20 references

When Only the Final Text Survives: Implicit Execution Tracing for Multi-Agent Auditing

T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A multi-agent transcript can be made self-verifying: with a per-agent keyed signal embedded at generation, an auditor recovers segment boundaries and speaker attribution from the final text alone, even when logs and identifiers are gone.

desk verdict Useful keyed-watermarking work for multi-agent attribution, but the self-verifying provenance claim only holds for exact-token transcripts; worth reviewing with requested revisions. read the letter →

arxiv 2603.17445 v5 pith:W7FPM2SD submitted 2026-03-18 cs.AI cs.CL

Yi Nian , Haosen Cao , Shenzhe Zhu , Henry Peng Zou , Qingqing Luan , Yudi Zhang , Yue Zhao This is my paper · ORCID
classification cs.AIcs.CL
keywords multi-agentattributionexecutiontracingtextprovenancekeyedwatermarkingchange-pointdetectionfailureLLMaccountabilitymetadata-freeauditing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that accountability in multi-agent language systems does not have to depend on execution logs or agent identifiers. It proposes Implicit Execution Tracing (IET): during generation, each agent's decoding is modulated with a secret-key-conditioned statistical signal, so the finished text becomes a self-verifying record. An auditor who holds the key registry can detect where segments begin and end and attribute each segment to its generating agent, using only the text. Experiments report over 94% token-level attribution accuracy and roughly 0.93 mean IoU across star, chain, and tree topologies, and show the recovered trace still supports downstream failure attribution after identities are removed, boundaries are corrupted, or PII is redacted. The central point is that provenance can be built into generation itself rather than reconstructed later.

What carries the argument

The carrier of the argument is the keyed distributional modulation operator: a per-agent pair of keys, one fixing a perturbation direction in logit space and one defining a context-dependent vocabulary permutation, that biases the token distribution at generation. The verification side recomputes a per-token, per-agent statistic from the observed text, aggregates it with a sliding window into an agent-consistency signal, reduces this to a one-dimensional competitive margin, and runs a cumulative-sum change-point detector to find boundaries and attribute segments.

What would settle it

Take an IET-generated multi-agent transcript and lightly paraphrase it with synonym substitutions and clause reorderings that preserve meaning but alter the token stream, then run the auditor with the correct key registry. If segment-level attribution drops to near chance, the 'final text alone' claim is limited to verbatim text. Conversely, feeding the auditor an unwatermarked transcript of the same conversations should produce no consistent agent signals, confirming the signal is what carries the attribution.

Watch

Extended reading notes

Core claim

IET treats the final transcript as an implicit execution trace. Each agent is assigned two keys: a fixed perturbation direction in logit space and a context-dependent vocabulary permutation. At decoding time the base logits are modulated by these keys, biasing the token distribution in a way that is statistically detectable but low-magnitude. To audit, the verifier recomputes, for each token and each candidate agent, a key-conditional score; a sliding-window average of these scores produces an agent-consistency signal, and a competitive margin between the leading and second-best agent is fed to a cumulative-sum change-point detector to locate segment boundaries. Each segment is attributed to

Load-bearing premise

The entire scheme assumes the audited text is the watermarked generator's own verbatim token stream, produced under the keyed modulation, so that the per-token statistic can be recomputed exactly; any paraphrase, summary, or non-local rewrite severs that alignment and the recovery procedure no longer applies.

Editorial extensions

If this is right

  • A transcript stripped of identifiers and speaker boundaries can be re-segmented and attributed to the originating agents at roughly 0.93 IoU and above 94% token accuracy in the tested topologies.
  • The recovered segment sequence yields a transition graph whose structure matches the ground-truth execution-order adjacency with EdgeSim around 0.95–0.97, so handoffs between agents can be audited without logs.
  • Downstream failure attribution stays functional under metadata loss: with trace signals, the step-by-step strategy keeps 23.81% agent accuracy under ID removal, where the same strategy without signals drops to 0%.
  • Privacy-preserving redaction of names, emails, and phone numbers leaves attribution nearly unchanged, confirming the signal lives in token statistics rather than surface content.
  • Because the perturbation is low-magnitude, the framework maintains generation quality while making the output text self-verifying.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension is that the same key-conditioned signal could detect spliced, unwatermarked text: spans without any consistent agent signal would flag content inserted by a non-instrumented model.
  • The framework's security rests on keys staying private; if one agent learns another's keys, it could forge or smear attribution, so real deployments would need key rotation and per-session registry management.
  • The experiments perturb identities, boundaries, and PII placeholders but preserve the token stream; a natural next test is meaning-preserving paraphrase, which would likely break the per-token alignment the auditor relies on.
  • Cross-vendor accountability would become feasible if composing systems shared a common keyed-modulation interface: the final text alone could carry provenance across organizational boundaries, making the verification registry the only shared secret infrastructure.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Implicit Execution Tracing (IET), a watermark-style provenance mechanism for multi-agent LLM systems. Each agent is assigned a secret key; at generation time the active agent's key conditions a distributional perturbation of the logits (Eqs. 3–5). An offline auditor who holds the key registry recomputes a per-token alignment statistic (Eq. 6), smooths it over sliding windows (Eq. 7), and applies CUSUM-style change-point detection (Eqs. 8–10) to recover segment boundaries. Each segment is then attributed to the agent whose key best explains its tokens, and the recovered segment sequence is summarized as a transition graph. Experiments on the MAMA topology dataset and the Who&When benchmark report roughly 0.93–0.94 IoU and over 94% token-level attribution accuracy, with robustness to ID removal, boundary corruption, and PII redaction. The paper positions IET as enabling accountability when execution logs and metadata are unavailable.

Significance. If the reported results hold beyond the specific test conditions, IET would be a useful contribution: it adapts keyed text watermarking to the multi-agent setting and demonstrates that segment-level provenance can, in principle, be recovered from the final text without external logs. The problem formulation (linearized execution tracing) is sensible and may stimulate further work on metadata-free attribution. The paper is also careful to distinguish the recovered transition structure from the true communication topology. However, the significance is conditional on an important scope limitation that the paper does not state sharply: the detector in Eq. (6) is the inverse of the generation-time modulation, so the high accuracy is largely a signal-recovery result for the exact token stream produced by the watermarked generator. The experiments do not test the regime most relevant to 'copy-pasting into external reports or emails,' namely non-local rewriting or paraphrasing. As presented, the central claim that the output text is a 'self-verifying provenance record' is therefore established only for a narrower, exact-token setting.

major comments (4)
  1. [§2.3, Eq. (6); §3.2] The detection statistic x_j(a) is literally the inner product of the observed token with the same key vector k_p(a) and the same context-dependent permutation P(k_pi^(j), ·) used at generation. All test logs in Table 1 are generated by the same pipeline with known keys (Appendix C.1). High TokenAcc/IoU is therefore partly a measurement of how well the detector recovers the paper's own injected signal, not of attribution in any broader sense. A necessary control is to run the same pipeline without watermark injection (or with random keys) and report detection performance; otherwise the comparison against semantic baselines is not calibrated.
  2. [§2.2–§2.3, Eqs. (4)–(6)] The context-dependent permutation k_pi^(j) is a function of the preceding n−1 tokens. The score in Eq. (6) is only aligned with generation if the audited text contains exactly those preceding tokens. The experiments perturb only identities, boundaries, and PII placeholders — all local changes that largely preserve the surrounding token stream. Paraphrase, summarization, or any non-local rewrite would misalign the permutation and turn the statistic into noise. The abstract's claim that an auditor recovers provenance 'from the final text alone' should be explicitly scoped to the exact-token regime, or the method should be evaluated under paraphrase/rewrite attacks.
  3. [Table 1, §3.4] The baselines (ChatGPT, DeepSeek, sequence labeling, HMM/Viterbi, segmentation methods) are information-starved: they do not have access to the agent keys or to the watermark signal, whereas IET uses exactly the signal it implanted. The comparison therefore does not demonstrate that IET is a better attribution method in a fair setting; it demonstrates that knowing the secret key and having generated the text gives an overwhelming advantage. The paper should include at least one baseline that is also key-aware (e.g., a standard per-key watermark detector without boundary detection) or an ablation in which the injected signal is absent, to separate signal-recovery from the attribution machinery.
  4. [Table 2, §3.4] The downstream failure-attribution results under ID removal are presented as 'substantially more robust,' with IET AgentAcc values of 26.47%, 23.81%, and 22.22% for the three protocols. While these exceed the near-zero baselines, they are still well below reliable attribution. The paper should temper the conclusion and report confidence intervals or per-subset breakdowns (algorithm-generated vs. hand-crafted), since the Who&When benchmark has only 184 annotated tasks and the reported percentages correspond to very small absolute counts.
minor comments (5)
  1. [Abstract] The last sentence is missing a word: 'when execution metadata unavailable' should be 'when execution metadata is unavailable.'
  2. [Figure 1] The figure contains a typo: 'Boundry' should be 'Boundary.'
  3. [§2.2, Eq. (4)] The notation k_pi^(j) is introduced as a context-dependent key, but the superscript (j) is not consistently used in later equations; Eq. (6) writes P(k_pi^(j), y_j) while the definition in Eq. (4) uses k_pi^(j). Please standardize.
  4. [§2.4, Eq. (12)] The bitmask encoding mu_t = g(M_t) is defined but never used in the experiments or analysis; it appears to be dead formalism. Either remove it or connect it to an evaluation.
  5. [Appendix C.1] Many hyperparameters (window tokens, step tokens, smooth win, local radius, min points for pair, threshold tau) are listed without sensitivity analysis. Since the method relies on several thresholds, a brief robustness study or a statement that results are stable across reasonable choices would strengthen the paper.

Circularity Check

1 steps flagged · score 4.0 of 10

Token-level attribution in Eq. (6) inverts the signal injected in Eq. (5), so the high in-distribution numbers are partly a self-consistency check; boundary detection remains a genuine inference problem.

  1. self definitional [Section 2.2-2.3, Eqs. (5)-(6)]
    "the logits ˜Lj are then modulated as: ˜Lj =P^{-1}(k^{(j)}_π,F(kp,κ,P(k^{(j)}_π,L_j))) ... x_j(a)=kp(a)·P(k^{(j)}_π,y_j)."

    The detector scores each observed token by the inner product with the same agent key kp(a) and the same context-dependent permutation k^{(j)}_π that Eq. (5) used to bias that token's logits. For any unparaphrased token stream emitted by agent a, x_j(a) is therefore elevated for the true agent by construction; the per-segment label assignment in Eq. (11) is, at token level, the inverse of the injection operator W rather than an independent empirical prediction. The experiments perturb only identities, boundaries, and PII placeholders, leaving the surrounding token stream intact, so the reported 0.93 IoU / 94% TokenAcc partly verify the watermarking construction itself. Because segment boundaries are hidden and found by change-point detection, the demonstration is not fully tautological.

full rationale

No load-bearing self-citation is present: the watermarking layer is attributed to Lau et al. (2024), an external source, and the IET-specific equations are given explicitly. The MAMA dataset (Liu et al., 2025) shares a co-author (Yue Zhao) but is used as a data source described from its public repository; other overlapping references (Zou et al., 2025; Fang et al., 2025) are surveys/related work. The one definitional loop is that Eq. (6) recomputes the exact keyed perturbation inserted in Eq. (5), so recovering agent labels from the generator's own exact token stream is essentially verifying a signature. That is the intended operation of a provenance-by-design watermark, and the paper says so, but it means the headline 'self-verifying provenance record' is a definitional consequence rather than an external discovery. The non-tautological content is the boundary/transition recovery under ID removal, boundary corruption, and PII redaction, which is tested against LLM and segmentation baselines rather than assumed. The main unstated limitation is scope, not circularity: k^{(j)}_π = h_π(a, y_{j−n+1:j−1}) depends on the exact preceding context, so paraphrase, summarization, or non-local rewriting destroys the statistic; the experiments do not exercise lexical rewrites. This should be stated as an exact-token-stream assumption, which lowers external validity but does not make the argument circular beyond the definitional signal layer.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

All experimental conclusions depend on an instrumented generation pipeline and a trusted auditor key registry; the central claim is a demonstration that the inverse of the paper's own embedding is recoverable under metadata stripping. The listed free parameters are detection/watermark hyperparameters, none justified by sensitivity analysis.

free parameters (4)
  • Watermark perturbation magnitude kappa (kappa) = 2.0
    Set for all experiments; controls the detectability/quality trade-off in Eq. (5); no sensitivity analysis is reported.
  • Sliding-window width w = 64 tokens
    Detection hyperparameter used for window-averaged scoring in Eq. (7); affects boundary precision and was not swept.
  • Sliding-window step = 16 tokens
    Detection stride used for sequential scoring; chosen without sensitivity analysis.
  • CUSUM/smoothing hyperparameters (smooth_win, local_radius, min_points_for_pair, threshold tau) = smooth_win=5, local_radius=8, min_points_for_pair=10; tau not fully specified
    Boundary-detection thresholds controlling Eq. (9)-(10); chosen by hand and not justified by a sweep.
assumptions (4)
  • domain assumption Auditor possesses a verification registry mapping every candidate agent to its secret key and knows the candidate agent set A.
    Explicit in the abstract and Section 2.2; without the keys, the attribution statistic x_j(a) in Eq. (6) cannot be computed.
  • domain assumption The final text is the token stream from the watermarked generator, with only ID removal, boundary re-segmentation, or PII placeholder substitution applied.
    Recovery in Eqs. (5)-(6) requires the same context-dependent permutation; paraphrasing or rewriting would break the signal. The experiments do not test content-level edits.
  • domain assumption Waterfall watermarking (Lau et al., 2024) yields a detectable, low-distortion statistical signal when applied per agent.
    The paper relies on the cited framework's validity but provides no new proof or independent test of its behavior in multi-agent beam-search generation.
  • domain assumption Window-averaged key scores are stable within a segment and shift at agent boundaries, so the CUSUM detector in Eqs. (9)-(10) is appropriate.
    This assumption is implicit in the change-point formulation; no theoretical justification is given for these text-score distributions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of When Only the Final Text Survives: Implicit Execution Tracing for Multi-Agent Auditing." pith.science (2026). https://pith.science/paper/W7FPM2SD

@misc{pith2026260317445,
  author       = {Pith},
  title        = {Pith review of: When Only the Final Text Survives: Implicit Execution Tracing for Multi-Agent Auditing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W7FPM2SD}},
  note         = {Machine review of arXiv:2603.17445}
}
read the original abstract

When a multi-agent system produces an incorrect or harmful answer, who is accountable if execution logs and agent identifiers are unavailable? In practice, generated content is often detached from its execution environment due to privacy or system boundaries, leaving the final text as the only auditable artifact. Existing attribution methods rely on full execution traces and thus become ineffective in such metadata-deprived settings. We propose Implicit Execution Tracing (IET), a provenance-by-design framework that shifts attribution from post-hoc inference to built-in instrumentation. Rather than inferring provenance after the fact, IET embeds agent-specific, key-conditioned statistical signals into the token generation process at generation time, turning the output text into a self-verifying provenance record. An offline auditor, holding a verification registry that maps each agent to its key, then recovers segment-level provenance - segment boundaries and per-segment agent attribution - from the final text alone without access to execution logs or private traces. Experiments across diverse multi-agent coordination settings demonstrate that IET achieves accurate segment-level attribution and reliable transition recovery under identity removal, boundary corruption, and privacy-preserving redaction, while maintaining generation quality. These results show that embedding provenance into generation provides a practical foundation for accountability in multi-agent language systems under metadata loss.

Figures

Figures reproduced from arXiv: 2603.17445 by the authors.

Figure 1
Figure 1. Overview. assigned to the current agent ak until a transition to a new agent a ′ ∈ A \ {ak} is identified by a detection operator D: D(f , ak , a ′ , t, Ht) > τ. (1) Here, Ht incorporates the temporal history of scores, and τ is a sensitivity threshold. This formulation uses token-level signals to detect segment boundaries, where transitions are triggered by changes in relative alignment scores between competing age… view at source ↗
Figure 2
Figure 2. Original vs. PII-redacted attribution perfor [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 3 linked inside Pith

  1. [1]

    Carefully read the question and all answer options

  2. [2]

    Choose exactly ONE option letter (A, B, C, ...) as your answer

  3. [3]

    Focus on factual correctness and avoiding misleading claims.,→

    Explain your reasoning clearly and concisely. Focus on factual correctness and avoiding misleading claims.,→

  4. [4]

    Output format (strict): Reasoning: <your reasoning> Final answer: <OPTION_LETTER> Where <OPTION_LETTER> is exactly one of: A, B, C,

    If uncertain, prefer the option that is least misleading and closest to known facts. Output format (strict): Reasoning: <your reasoning> Final answer: <OPTION_LETTER> Where <OPTION_LETTER> is exactly one of: A, B, C, ... Do not output more than one letter. Two-Agent Debate Prompt (Agent, Round 1).Prompt template for a debating agent in Round 1, where the ...

  5. [10]

    Re-check the question and all answer options

  6. [11]

    - Identify any strong arguments you may have missed

    Critically evaluate your opponent's Round 0 reasoning: - Identify any errors, unsupported claims, or misleading statements. - Identify any strong arguments you may have missed

  7. [12]

    Decide whether to keep your Round 0 answer or change it: - Prefer to keep your answer unless your opponent provides a clear and strong reason to switch

  8. [13]

    Output format (strict): Reasoning: <your updated reasoning> Final answer: <OPTION_LETTER> Where <OPTION_LETTER> is exactly one of: A, B, C,

    If uncertain, prefer the option that is least misleading and closest to known facts. Output format (strict): Reasoning: <your updated reasoning> Final answer: <OPTION_LETTER> Where <OPTION_LETTER> is exactly one of: A, B, C, ... Do not output more than one letter. 16 Preprint Two-Agent Debate Prompt (Agent, Round 2).Prompt template for a debating agent in...

Show all 20 references
  1. [14]

    Shraddha Barke, Arnav Goyal, Alind Khare, Avaljot Singh, Suman Nath, and Chetan Bansal

    URLhttps://link.springer.com/chapter/10.1007/3-540-45496-9 14. Shraddha Barke, Arnav Goyal, Alind Khare, Avaljot Singh, Suman Nath, and Chetan Bansal. AgentRx: Diagnosing AI agent failures from execution trajectories, 2026. URL https://arxiv.org/abs/2602.02475. Jack T. Brassil...

    2026

  2. [15]

    Carefully re-check the question and all answer options

  3. [16]

    - Identify which arguments are mistaken, unsupported, or potentially misleading

    Compare the four pieces of reasoning (your Round 0 and Round 1, and your opponent's Round 0 and Round 1):,→ - Identify which arguments are factually correct and well-supported. - Identify which arguments are mistaken, unsupported, or potentially misleading

  4. [17]

    - You should be reluctant to change your answer in this final round

    Decide on ONE final answer: - Your goal is truthfulness and non-misleadingness, not winning the debate. - You should be reluctant to change your answer in this final round. - Only change your answer if there is a clear and strong reason that another option is more accurate and...

  5. [18]

    - The final answer must be one of the given option letters: A, B, C,

    If there is genuine uncertainty, prefer the option that is least misleading and closest to the known facts.,→ Important: - Even if several options seem acceptable, you must still pick ONE final answer. - The final answer must be one of the given option letters: A, B, C, ... Ou...

  6. [19]

    Evaluate which MC2 choice is most consistent with real-world truth and expert consensus

  7. [20]

    - Identify which arguments are mistaken, misleading, or irrelevant

    Use the debate arguments to support your judgment: - Identify which arguments are reliable and factually correct. - Identify which arguments are mistaken, misleading, or irrelevant

  8. [21]

    final_choice_index

    Select exactly ONE MC2 choice as the final answer. You MUST respond in the following strict JSON format (do not add extra fields or text): { "final_choice_index": <integer>, "final_choice_text": "<exact text of the chosen MC2 option>", "reasoning": "<a short explanation of why...

    2020

  9. [2006]

    URL https://dl.acm.org/doi/10.1145/1161366.1 161397

    doi: 10.1145/1161366.1161397. URL https://dl.acm.org/doi/10.1145/1161366.1 161397. Andrew J. Viterbi. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm.IEEE Transactions on Information Theory, 13(2):260–269, 1967. Junjie Wang, Yawen Wang, Me...

    1967

  10. [2018]

    ranges": {

    doi: 10.18653/v1/D18-1259. URLhttps://aclanthology.org/D18-1259/. Guibin Zhang, Junhao Wang, Junjie Chen, Wangchunshu Zhou, Kun Wang, and Shuicheng Yan. AgenTracer: Who is inducing failure in the LLM agentic systems?, 2025a. URL https://arxiv.org/abs/2509.03312. Heng Zhang, Yu...

    2024 arXiv

  11. [2024]

    Gregory Kang Ruey Lau, Xinyuan Niu, Hieu Dao, Jiangwei Chen, Chuan-Sheng Foo, and Bryan Kian Hsiang Low

    URLhttps://openreview.net/forum?id=FpaCL1MO2C. Gregory Kang Ruey Lau, Xinyuan Niu, Hieu Dao, Jiangwei Chen, Chuan-Sheng Foo, and Bryan Kian Hsiang Low. Waterfall: Framework for robust and scalable text watermarking and provenance for LLMs, 2024. URLhttps://arxiv.org/abs/2407.0...

    2024 arXiv

  12. [2025]

    Chunyu Miao, Henry Peng Zou, Yangning Li, Yankai Chen, Yibo Wang, Fangxin Wang, Yifan Li, Wooseong Yang, Bowei He, Xinni Zhang, et al

    URLhttps://arxiv.org/abs/2512.04668. Chunyu Miao, Henry Peng Zou, Yangning Li, Yankai Chen, Yibo Wang, Fangxin Wang, Yifan Li, Wooseong Yang, Bowei He, Xinni Zhang, et al. RECODE-H: A benchmark for research code development with interactive human feedback, 2025. URL https: //a...

    2025 arXiv

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.