{"id":"ce62e54d-78e7-4064-8893-8a8ec2b1fa51","arxiv_id":"2603.17445","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Per-agent watermark-style signals embedded during generation allow segment-level attribution and transition recovery from final text alone.","lead":"Multi-agent AI outputs usually lose their execution history once text is copied out, making it impossible to say which agent wrote what. This paper proposes embedding hidden per-agent signals into generated tokens so an auditor with the matching keys can later recover segment boundaries and agent labels from the final text alone.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"IET's signal is only recoverable from the generator's exact token stream; paraphrase or non-local rewriting destroys the key-conditioned statistic in Eq. (6), so the 'self-verifying provenance record' claim needs an explicit exact-token scope.","rationale":"The reader's weakest assumption is exactly the load-bearing condition: the audited final text must be the watermarked generator's own token stream. I agree. The paper's controlled results are internally consistent and the high IoU/TokenAcc numbers in Table 1 are plausible evidence that the mechanism works when tokens are preserved. The PII-redaction test further shows robustness to sparse local replacements. However, none of the perturbations in §3.2 or Appendix C.3 change the lexical content in a non-local way: ID removal deletes small identity tokens, boundary corruption re-segments without reordering content, and PII redaction replaces spans with placeholders. None of these tests a paraphrase or summary, which would change the context window used by Eq. (6) for subsequent tokens. The abstract's phrase 'self-verifying provenance record' invites a broader reading than the method actually supports. The right response is not rejection — the mechanism is sound under its intended metadata-loss setting — but a scope revision and a paraphrase stress test. Since the reader already assigned CONDITIONAL, my analysis does not move the verdict; it reinforces the condition.","tokens_in":19504,"tokens_out":11418,"duration_ms":126017,"concrete_test":"Generate 100 IET-watermarked 4-agent chain traces (same configuration as Table 1), then apply three content-level perturbations: (a) LLM-based paraphrase with temperature 0.7, (b) round-trip translation, and (c) targeted synonym substitution. Run the IET recovery pipeline with the published detector settings (window=64, step=16, smooth_win=5, local_radius=8). If IoU or TokenAcc drops below the Sequence-Labeling baseline (IoU≈0.72, TokenAcc≈0.83) under any perturbation, the claim must be revised to 'exact-token metadata loss only.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — that the final text is a self-verifying provenance record from which segment-level provenance can be recovered — silently assumes that the audited text is the exact token stream emitted by the watermarked generator. The detector in Eq. (6) recomputes x_j(a) = k_p(a) · P(k_π^{(j)}, y_j), where k_π^{(j)} = h_π(a, y_{j−n+1:j−1}). This is only valid when the preceding tokens in the audited text are identical to those present at generation time. Any paraphrase, summary, or non-local rewrite changes the context for every following token, so the permutation used at generation no longer aligns with the observed token; the computed statistic becomes noise rather than an agent-specific signal. The experiments perturb only identities, boundaries, and PII placeholders — bounded, local changes that leave the surrounding token stream intact — so they do not exercise this condition. Because real-world content often gets edited or rephrased before being copied into reports or emails, the scope of the headline claim is materially narrower than the abstract suggests. This is not an internal inconsistency; it is an unstated, load-bearing scope condition.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Implicit Execution Tracing (IET), a watermark-style provenance mechanism for multi-agent LLM systems. Each agent is assigned a secret key; at generation time the active agent's key conditions a distributional perturbation of the logits (Eqs. 3–5). An offline auditor who holds the key registry recomputes a per-token alignment statistic (Eq. 6), smooths it over sliding windows (Eq. 7), and applies CUSUM-style change-point detection (Eqs. 8–10) to recover segment boundaries. Each segment is then attributed to the agent whose key best explains its tokens, and the recovered segment sequence is summarized as a transition graph. Experiments on the MAMA topology dataset and the Who&When benchmark report roughly 0.93–0.94 IoU and over 94% token-level attribution accuracy, with robustness to ID removal, boundary corruption, and PII redaction. The paper positions IET as enabling accountability when execution logs and metadata are unavailable.","tokens_in":19851,"tokens_out":3807,"duration_ms":45695,"significance":"If the reported results hold beyond the specific test conditions, IET would be a useful contribution: it adapts keyed text watermarking to the multi-agent setting and demonstrates that segment-level provenance can, in principle, be recovered from the final text without external logs. The problem formulation (linearized execution tracing) is sensible and may stimulate further work on metadata-free attribution. The paper is also careful to distinguish the recovered transition structure from the true communication topology. However, the significance is conditional on an important scope limitation that the paper does not state sharply: the detector in Eq. (6) is the inverse of the generation-time modulation, so the high accuracy is largely a signal-recovery result for the exact token stream produced by the watermarked generator. The experiments do not test the regime most relevant to 'copy-pasting into external reports or emails,' namely non-local rewriting or paraphrasing. As presented, the central claim that the output text is a 'self-verifying provenance record' is therefore established only for a narrower, exact-token setting.","major_comments":[{"comment":"The detection statistic x_j(a) is literally the inner product of the observed token with the same key vector k_p(a) and the same context-dependent permutation P(k_pi^(j), ·) used at generation. All test logs in Table 1 are generated by the same pipeline with known keys (Appendix C.1). High TokenAcc/IoU is therefore partly a measurement of how well the detector recovers the paper's own injected signal, not of attribution in any broader sense. A necessary control is to run the same pipeline without watermark injection (or with random keys) and report detection performance; otherwise the comparison against semantic baselines is not calibrated.","section":"§2.3, Eq. (6); §3.2"},{"comment":"The context-dependent permutation k_pi^(j) is a function of the preceding n−1 tokens. The score in Eq. (6) is only aligned with generation if the audited text contains exactly those preceding tokens. The experiments perturb only identities, boundaries, and PII placeholders — all local changes that largely preserve the surrounding token stream. Paraphrase, summarization, or any non-local rewrite would misalign the permutation and turn the statistic into noise. The abstract's claim that an auditor recovers provenance 'from the final text alone' should be explicitly scoped to the exact-token regime, or the method should be evaluated under paraphrase/rewrite attacks.","section":"§2.2–§2.3, Eqs. (4)–(6)"},{"comment":"The baselines (ChatGPT, DeepSeek, sequence labeling, HMM/Viterbi, segmentation methods) are information-starved: they do not have access to the agent keys or to the watermark signal, whereas IET uses exactly the signal it implanted. The comparison therefore does not demonstrate that IET is a better attribution method in a fair setting; it demonstrates that knowing the secret key and having generated the text gives an overwhelming advantage. The paper should include at least one baseline that is also key-aware (e.g., a standard per-key watermark detector without boundary detection) or an ablation in which the injected signal is absent, to separate signal-recovery from the attribution machinery.","section":"Table 1, §3.4"},{"comment":"The downstream failure-attribution results under ID removal are presented as 'substantially more robust,' with IET AgentAcc values of 26.47%, 23.81%, and 22.22% for the three protocols. While these exceed the near-zero baselines, they are still well below reliable attribution. The paper should temper the conclusion and report confidence intervals or per-subset breakdowns (algorithm-generated vs. hand-crafted), since the Who&When benchmark has only 184 annotated tasks and the reported percentages correspond to very small absolute counts.","section":"Table 2, §3.4"}],"minor_comments":[{"comment":"The last sentence is missing a word: 'when execution metadata unavailable' should be 'when execution metadata is unavailable.'","section":"Abstract"},{"comment":"The figure contains a typo: 'Boundry' should be 'Boundary.'","section":"Figure 1"},{"comment":"The notation k_pi^(j) is introduced as a context-dependent key, but the superscript (j) is not consistently used in later equations; Eq. (6) writes P(k_pi^(j), y_j) while the definition in Eq. (4) uses k_pi^(j). Please standardize.","section":"§2.2, Eq. (4)"},{"comment":"The bitmask encoding mu_t = g(M_t) is defined but never used in the experiments or analysis; it appears to be dead formalism. Either remove it or connect it to an evaluation.","section":"§2.4, Eq. (12)"},{"comment":"Many hyperparameters (window tokens, step tokens, smooth win, local radius, min points for pair, threshold tau) are listed without sensitivity analysis. Since the method relies on several thresholds, a brief robustness study or a statement that results are stable across reasonable choices would strengthen the paper.","section":"Appendix C.1"}],"recommendation":"major_revision","confidential_remarks":"The core direction is reasonable and the controlled-setting numbers are internally consistent, but the evaluation is self-referential and the headline claim is broader than what the experiments support. The authors should be asked to (1) add a no-signal control and at least one key-aware baseline, (2) explicitly scope the claim to exact-token streams or add paraphrase/rewrite robustness experiments, and (3) soften the downstream failure-attribution conclusion given the low absolute accuracy. With these changes, the paper could be acceptable; without them, the contribution is a watermark-recovery demonstration rather than a general provenance solution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the paper is more honest than its abstract. The actual contribution is a keyed watermarking scheme that lets someone with the registry map segments of an unedited token stream back to agents. The framing as 'linearized execution tracing' is new and reasonable. It builds directly on Waterfall (Lau et al. 2024), but the per-agent key assignment, sliding-window scoring, and CUSUM boundary detection are real engineering work. Controlled numbers are high: ~0.93 IoU and >94% token accuracy across star/chain/tree for 4-6 agents, and transition graph recovery is decent. The downstream failure-attribution result is modest but shows the trace is usable. Credit also for testing PII redaction; that is the right kind of robustness question.\n\nSoft spots, in rough order. The headline claim overreaches. The method recovers provenance only when the audited text is the exact token stream the watermarked generator emitted. Eq. (6) recomputes x_j(a) with a context-dependent permutation derived from the preceding tokens. Any paraphrase, summary, or non-local rewrite changes that context and destroys the signal. Their perturbations — ID removal, boundary corruption, PII placeholders — leave the token stream intact. So 'self-verifying provenance record' is really 'exact-token provenance record.' That is still useful (copy-paste of raw model output is common) but the abstract should say so.\n\nSecond, the evaluation is self-referential in a way that is partly intrinsic to watermarking and partly avoidable. Detecting your own injected signal with your own key is expected; the issue is that all baselines are information-starved (no key, no embedded signal), so the comparison does not show that IET beats a watermark-aware alternative. A stronger baseline would be: use a single Waterfall key for all agents and attempt attribution by change-point detection alone, or decode each segment with a multi-key detector. That would isolate the per-agent key contribution. Also no error bars, no released code/data, and the Who&When 'regeneration' step — how the watermarked traces were produced from the benchmark histories — is left implicit. That matters because regeneration with a different model could change the content.\n\nThe citation pattern is fine; Waterfall is properly credited. The math is simple but sound. Overall: a credible instrumentation advance with an overbroad scope claim. For people working on multi-agent attribution, watermarking, or auditability, this is worth reading. It deserves peer review, but the authors should be pushed to release artifacts, specify the regeneration protocol, add a signal-aware baseline, and test at least one paraphrase condition. If those hold, this is a citable result for multi-agent accountability work.","headline":"Useful keyed-watermarking work for multi-agent attribution, but the self-verifying provenance claim only holds for exact-token transcripts; worth reviewing with requested revisions.","tokens_in":20298,"tokens_out":3267,"would_cite":true,"duration_ms":33017,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multi-agent transcript can be made self-verifying: with a per-agent keyed signal embedded at generation, an auditor recovers segment boundaries and speaker attribution from the final text alone, even when logs and identifiers are gone.","keywords":["multi-agent attribution","execution tracing","text provenance","keyed watermarking","change-point detection","failure attribution","LLM accountability","metadata-free auditing"],"falsifier":"Take an IET-generated multi-agent transcript and lightly paraphrase it with synonym substitutions and clause reorderings that preserve meaning but alter the token stream, then run the auditor with the correct key registry. If segment-level attribution drops to near chance, the 'final text alone' claim is limited to verbatim text. Conversely, feeding the auditor an unwatermarked transcript of the same conversations should produce no consistent agent signals, confirming the signal is what carries the attribution.","tokens_in":19436,"feed_emoji":"🔑","tokens_out":5807,"duration_ms":62589,"temperature":0.7,"pith_summary":"The paper argues that accountability in multi-agent language systems does not have to depend on execution logs or agent identifiers. It proposes Implicit Execution Tracing (IET): during generation, each agent's decoding is modulated with a secret-key-conditioned statistical signal, so the finished text becomes a self-verifying record. An auditor who holds the key registry can detect where segments begin and end and attribute each segment to its generating agent, using only the text. Experiments report over 94% token-level attribution accuracy and roughly 0.93 mean IoU across star, chain, and tree topologies, and show the recovered trace still supports downstream failure attribution after identities are removed, boundaries are corrupted, or PII is redacted. The central point is that provenance can be built into generation itself rather than reconstructed later.","feed_headline":"Over 94% accuracy tracing agent segments from text alone","feed_subtitle":"An offline auditor recovers boundaries and speaker attribution even after logs and identifiers are stripped.","key_machinery":"The carrier of the argument is the keyed distributional modulation operator: a per-agent pair of keys, one fixing a perturbation direction in logit space and one defining a context-dependent vocabulary permutation, that biases the token distribution at generation. The verification side recomputes a per-token, per-agent statistic from the observed text, aggregates it with a sliding window into an agent-consistency signal, reduces this to a one-dimensional competitive margin, and runs a cumulative-sum change-point detector to find boundaries and attribute segments.","core_discovery":"IET treats the final transcript as an implicit execution trace. Each agent is assigned two keys: a fixed perturbation direction in logit space and a context-dependent vocabulary permutation. At decoding time the base logits are modulated by these keys, biasing the token distribution in a way that is statistically detectable but low-magnitude. To audit, the verifier recomputes, for each token and each candidate agent, a key-conditional score; a sliding-window average of these scores produces an agent-consistency signal, and a competitive margin between the leading and second-best agent is fed to a cumulative-sum change-point detector to locate segment boundaries. Each segment is attributed to","pith_inferences":["A direct extension is that the same key-conditioned signal could detect spliced, unwatermarked text: spans without any consistent agent signal would flag content inserted by a non-instrumented model.","The framework's security rests on keys staying private; if one agent learns another's keys, it could forge or smear attribution, so real deployments would need key rotation and per-session registry management.","The experiments perturb identities, boundaries, and PII placeholders but preserve the token stream; a natural next test is meaning-preserving paraphrase, which would likely break the per-token alignment the auditor relies on.","Cross-vendor accountability would become feasible if composing systems shared a common keyed-modulation interface: the final text alone could carry provenance across organizational boundaries, making the verification registry the only shared secret infrastructure."],"forward_implications":["A transcript stripped of identifiers and speaker boundaries can be re-segmented and attributed to the originating agents at roughly 0.93 IoU and above 94% token accuracy in the tested topologies.","The recovered segment sequence yields a transition graph whose structure matches the ground-truth execution-order adjacency with EdgeSim around 0.95–0.97, so handoffs between agents can be audited without logs.","Downstream failure attribution stays functional under metadata loss: with trace signals, the step-by-step strategy keeps 23.81% agent accuracy under ID removal, where the same strategy without signals drops to 0%.","Privacy-preserving redaction of names, emails, and phone numbers leaves attribution nearly unchanged, confirming the signal lives in token statistics rather than surface content.","Because the perturbation is low-magnitude, the framework maintains generation quality while making the output text self-verifying."],"fun_headline_variants":["IET: provenance embedded in text, agents recoverable from output alone","No execution logs? IET traces agents from final text at 94% accuracy","Self-verifying text: IET bakes agent provenance into generation","Audit without logs: IET recovers segment attribution from text","Token-level key signals turn output into audit trail"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire scheme assumes the audited text is the watermarked generator's own verbatim token stream, produced under the keyed modulation, so that the per-token statistic can be recomputed exactly; any paraphrase, summary, or non-local rewrite severs that alignment and the recovery procedure no longer applies.","fun_headline_variants_meta":{"raw":{"variants":["IET: provenance embedded in text, agents recoverable from output alone","No execution logs? IET traces agents from final text at 94% accuracy","Self-verifying text: IET bakes agent provenance into generation","Audit without logs: IET recovers segment attribution from text","Token-level key signals turn output into audit trail"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000211,"raw_usage":{"total_tokens":1252,"prompt_tokens":744,"completion_tokens":508,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":488,"completion_tokens_details":{"reasoning_tokens":416}},"tokens_in":488,"tokens_out":508,"duration_ms":6028,"temperature":1.0,"reasoning_tokens":416,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T05:45:28.766350+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take an IET-generated multi-agent transcript and lightly paraphrase it with synonym substitutions and clause reorderings that preserve meaning but alter the token stream, then run the auditor with the correct key registry. If segment-level attribution drops to near chance, the 'final text alone' claim is limited to verbatim text. Conversely, feeding the auditor an unwatermarked transcript of the same conversations should produce no consistent agent signals, confirming the signal is what carries the attribution.","supporting_citations":[],"review_version":1}