{"id":"a7070ef8-1680-484e-be0d-70167fd0e1bc","arxiv_id":"2606.10813","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"RedAct redacts agent traces to drop normalized skill transfer below the no-skill baseline on CapTraceBench while preserving audit evidence and adding detectable behavioral watermarks.","lead":"The paper introduces RedAct, a framework that redacts key procedural details from AI agent execution traces to block skill extraction while retaining evidence needed for audits. A smart generalist might read it because agent logs are becoming common in deployed systems and this work treats their release as a security interface rather than just a transparency tool.","discovery_kind":"unclear","skeptic_critique":{"model":"grok-4.3","headline":"RedAct's protection may not generalize beyond the specific reuse attacks evaluated","rationale":"The reader's weakest_assumption directly identifies the same untested robustness condition. Because the full manuscript was not supplied in the initial query, the UNVERDICTED status is retained; the concrete_test above would be the minimal addition needed to move beyond that status.","tokens_in":1761,"tokens_out":298,"duration_ms":9199,"concrete_test":"Add two new extraction attacks (one LLM-based strategy inference from redacted traces, one statistical aggregation across multiple traces) to the CapTraceBench evaluation pipeline; recompute NST for RedAct-protected traces. If NST rises above the no-skill baseline on either attack, the central claim does not hold for general reuse.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result (NST reduced below no-skill baseline while preserving audit evidence) rests on RedAct correctly localizing and redacting only 'protected key information' without removing 'verifier-critical evidence.' This distinction is performed by the framework itself; the paper evaluates only a fixed set of representative trace reuse methods. If an adversary uses a different extraction strategy (e.g., one that reconstructs procedural logic from the remaining audit evidence or from statistical patterns across multiple redacted traces), the NST reduction could fail while the reported metrics remain unchanged. No evidence is given that the redaction boundary is robust to such methods.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces CapTraceBench (75 long-horizon tasks, 154 skills across seven domains) and the RedAct framework, which localizes protected key information in agent execution traces, rewrites them to preserve verifier-critical evidence, and embeds behavioral watermarks. It reports that RedAct reduces normalized skill transfer (NST) from 44.7--67.1% on raw traces to below the no-skill baseline across representative reuse methods while retaining audit utility, and that its watermarks achieve 93.6--100.0% true detection at ≤1.9% false-alarm rate.","tokens_in":1895,"tokens_out":549,"duration_ms":12512,"significance":"If the empirical results hold under the stated conditions, the work is significant for security and accountability in agent systems: it supplies the first benchmark and concrete mechanism for treating public traces as security interfaces and demonstrates that selective redaction can decouple procedural leakage from audit evidence. The open-source release and the quantitative framing of NST are concrete strengths.","major_comments":[{"comment":"§4 (Evaluation) and §5.2 (NST results): the headline claim that NST falls below the no-skill baseline rests on RedAct's internal distinction between 'protected key information' and 'verifier-critical evidence.' The paper evaluates only the listed reuse methods; no experiments or analysis address whether an adversary could reconstruct procedural logic from the remaining audit evidence or from statistical patterns across multiple redacted traces. This directly affects whether the reported NST reduction generalizes.","section":"§4, §5.2"},{"comment":"§3.2 (Redaction procedure) and §5.1 (baseline construction): the manuscript supplies no explicit algorithmic description or pseudocode for how redaction decisions are made, how the no-skill baseline is constructed, or how NST is normalized. Without these details the quantitative claims cannot be reproduced or stress-tested against the robustness concern above.","section":"§3.2, §5.1"}],"minor_comments":[{"comment":"Table 2 and Figure 3: axis labels and caption text for NST and watermark metrics should explicitly state the number of runs and whether error bars represent standard deviation or confidence intervals.","section":"Table 2, Figure 3"},{"comment":"§2 (Related work): the discussion of prior trace-protection techniques omits recent work on watermarking LLM outputs; adding two or three citations would strengthen context.","section":"§2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive and detailed feedback. We address each major comment below and indicate the revisions we will make to strengthen the manuscript.","responses":[{"response":"We acknowledge the referee's point that the current evaluation is limited to the listed reuse methods and does not explicitly test reconstruction attacks that exploit statistical patterns across multiple redacted traces. The core design of RedAct relies on the distinction between protected key information and verifier-critical evidence (detailed in §3), and the reported NST results hold for the representative methods evaluated in §5.2. We agree that broader adversary models merit explicit discussion. In the revised manuscript we will add a dedicated limitations subsection in §5.2 that analyzes the scope of the evaluated reuse methods, discusses why cross-trace statistical reconstruction is not addressed by the current experiments, and clarifies the conditions under which the NST reduction is claimed to hold.","revision_made":"partial","referee_comment":"[§4, §5.2] §4 (Evaluation) and §5.2 (NST results): the headline claim that NST falls below the no-skill baseline rests on RedAct's internal distinction between 'protected key information' and 'verifier-critical evidence.' The paper evaluates only the listed reuse methods; no experiments or analysis address whether an adversary could reconstruct procedural logic from the remaining audit evidence or from statistical patterns across multiple redacted traces. This directly affects whether the reported NST reduction generalizes."},{"response":"We agree that the absence of explicit algorithmic descriptions hinders reproducibility. In the revised manuscript we will insert pseudocode in §3.2 that formalizes the redaction decision procedure, including localization of protected key information and the rewriting rules that preserve verifier-critical evidence. We will also add a precise description of the no-skill baseline construction and the NST normalization formula (including all constants and aggregation steps) in §5.1. These additions will directly enable the stress-testing suggested by the referee.","revision_made":"yes","referee_comment":"[§3.2, §5.1] §3.2 (Redaction procedure) and §5.1 (baseline construction): the manuscript supplies no explicit algorithmic description or pseudocode for how redaction decisions are made, how the no-skill baseline is constructed, or how NST is normalized. Without these details the quantitative claims cannot be reproduced or stress-tested against the robustness concern above."}],"tokens_in":1440,"tokens_out":516,"duration_ms":15215,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main point is that this paper builds CapTraceBench with 75 long-horizon tasks and 154 skills across seven domains, then shows RedAct can localize protected information in traces, rewrite them to keep audit evidence, and embed behavioral watermarks. On the reported reuse methods, normalized skill transfer drops from 44.7-67.1% down below the no-skill baseline, and the watermarks hit 93.6-100% true detection with at most 1.9% false alarms.\n\nThe work does something useful by treating public traces as a security interface and giving concrete numbers on leakage reduction without fully stripping diagnostic value. The benchmark itself is a fresh artifact that lets others measure this risk.\n\nThe soft spot is generalization. The redaction step has to separate protected key information from verifier-critical evidence, but the evaluation only covers a fixed set of representative reuse methods. If an adversary uses a different approach, such as pulling logic from statistical patterns across redacted traces or reconstructing from the remaining evidence, the NST reduction could disappear while the published metrics stay the same. The abstract also skips details on how localization decisions are made or how the no-skill baseline is constructed, so the full methods section will need to carry the weight.\n\nThis is for researchers working on agent accountability, trace sharing, and procedural skill protection. A reader who needs a starting benchmark or a practical redaction example would find it worth reading.\n\nSend it to peer review. The benchmark and the pipeline address a concrete, growing problem even if the attack coverage stays narrow.","headline":"RedAct introduces a new benchmark and redaction pipeline that cuts skill transfer below baseline on the tested reuse attacks while adding detectable watermarks, but the protection's boundary may not hold against other extraction strategies.","tokens_in":2362,"tokens_out":404,"would_cite":false,"duration_ms":14216,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"RedAct shows selective redaction of agent traces blocks procedural skill extraction while keeping audit evidence usable.","keywords":["agent traces","procedural skill protection","redaction","behavioral watermarks","capability leakage","trace reuse","audit evidence","security interfaces"],"falsifier":"A new trace reuse method that extracts skills from RedAct redacted traces at rates well above the no-skill baseline.","tokens_in":2665,"feed_emoji":"🔒","tokens_out":636,"duration_ms":15709,"temperature":0.7,"pith_summary":"The paper establishes that detailed execution traces from agents leak specialized procedural skills to reuse methods that recover formulas, thresholds, and strategies without needing model weights. It introduces a benchmark of long-horizon tasks and skills across domains to measure this leakage risk. The RedAct framework then localizes protected information inside traces, rewrites the traces to hide skills, and adds behavioral watermarks for later provenance checks. If the approach works, organizations could release traces publicly for debugging and accountability without handing over private capabilities. The central result is that redacted traces drop skill transfer below the level seen with no skills at all.","feed_headline":"Redaction blocks skill extraction from agent traces","feed_subtitle":"Framework rewrites traces to drop reuse success below no-skill baseline while keeping audit evidence intact.","key_machinery":"The RedAct framework, which localizes protected key information in traces, rewrites them to preserve verifier-critical evidence, and embeds behavioral watermarks.","core_discovery":"RedAct is a protected trace release framework that localizes protected key information, rewrites traces while preserving verifier-critical evidence, and embeds behavioral watermarks for downstream provenance analysis. Across representative trace reuse methods, RedAct reduces normalized skill transfer from 44.7--67.1% on raw traces to below the no-skill baseline, while preserving audit evidence. Its standalone behavioral watermarks reach 93.6--100.0% true detection with a false alarm rate of at most 1.9%.","pith_inferences":["The same localization-and-rewrite approach could be tested on other agent outputs such as decision logs or intermediate state dumps.","Organizations might add RedAct-style redaction as a standard step before any public trace sharing to meet privacy requirements.","Future work could measure how the redaction affects human auditors who rely on the preserved evidence."],"forward_implications":["Public release of agent traces for diagnosis and accountability becomes feasible without exposing private procedural skills.","Audit processes can continue using the redacted traces because verifier-critical evidence is retained.","Behavioral watermarks allow reliable detection of trace provenance even after downstream reuse attempts.","Trace reuse attacks lose effectiveness for recovering specialized skills from the released data."],"fun_headline_variants":["RedAct redacts traces to block skill extraction","Redaction cuts trace skill transfer below baseline","Protected traces hide skills but keep audit evidence","RedAct rewrites traces while embedding watermarks"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The redaction process can reliably separate protected key information from verifier-critical evidence in a way that holds against extraction methods beyond those tested.","fun_headline_variants_meta":{"raw":{"variants":["RedAct redacts traces to block skill extraction","Redaction cuts trace skill transfer below baseline","Protected traces hide skills but keep audit evidence","RedAct rewrites traces while embedding watermarks"]},"model":"grok-4.3","cost_usd":0.003449,"raw_usage":{"total_tokens":1837,"prompt_tokens":700,"num_sources_used":0,"completion_tokens":55,"cost_in_usd_ticks":34487000,"prompt_tokens_details":{"text_tokens":700,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1082,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":700,"tokens_out":55,"duration_ms":7207,"temperature":1.0,"reasoning_tokens":1082,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-27T12:49:20.151408+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A new trace reuse method that extracts skills from RedAct redacted traces at rates well above the no-skill baseline.","supporting_citations":[],"review_version":1}