{"id":"8d681f62-d6b4-4a70-a3ac-924b0638617c","arxiv_id":"2608.29028","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":8.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Multi-agent LLM handoff summaries preserve operational facts while preferentially stripping boundary metadata, causing systematic privacy leakage that typed audience-allowlists can nearly eliminate.","lead":"This paper shows that when one AI agent summarizes a conversation for another agent, the summary keeps the actionable facts but tends to drop the privacy rules that controlled how those facts could be used, causing unintended leaks. This failure is structural, not adversarial, and affects multiple models; the most effective fix is a typed, machine-readable boundary field rather than plain-language instructions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"The σb/σop decoupling is scored by one LLM judge whose prompt encodes the paper's own boundary-vs-fact distinction, and the PrivacyLens check reuses that same judge, so construct bias is not independently ruled out.","rationale":"The reader's weakest assumption was external validity: whether hand-authored scenarios and semi-synthetic PrivacyLens traces generalize to production multi-agent systems. I agree that is an open question, but I think the more load-bearing premise sits one step earlier: whether the σb/σop measurement itself is construct-independent. The paper has many strengths that should be credited: the design isolates the handoff artifact as the unit of analysis; the matched-variant downstream tests (E3, E4, E8) hold operational content constant and show a sharp L1-to-L2 explicitness transition; the raw ordinal label proportions in Table 3 and the scenario-stratified CMH tests in Table 22 show the decoupling is not an artifact of the 0.35 numeric weight; and the PrivacyLens replication demonstrates that the pattern is not unique to the authors' scenario authoring. The concern I raise is narrower than external validity: it is that the central quantitative claims all flow through one judge whose prompt instantiates the paper's own boundary-versus-fact ontology. The authors' human validation used the same ontology, so the validation checks reliability of the rubric, not validity of the construct. The PrivacyLens check, while valuable, inherits the same judge. This does not make the finding wrong; it does mean the conditional verdict is appropriate. A neutral re-scoring that separates 'meaning preservation' from the boundary/fact categorization would either confirm the decoupling as a property of the content or reveal that part of it is a property of the measurement prompt. If it confirms, the external-validity question becomes the remaining limitation; if it does not, the headline claims need substantial revision. I therefore keep the reader's CONDITIONAL verdict rather than upgrading to ACCEPT, and I do not see grounds for REJECT because the downstream leakage contrasts are largely independent of the survival-judge concern.","tokens_in":28723,"tokens_out":5411,"duration_ms":59984,"concrete_test":"Take a stratified sample of roughly 100 handoffs from E1 and E2 (spanning both models and the free_text through 25-word conditions). Have external annotators, blind to the paper's definitions, score each handoff against a list of source statements with a single neutral question: \"Did the handoff preserve the meaning of this statement?\" Do not label statements as boundary markers versus operational facts. Map the answers onto the paper's gold split and recompute σb, σop, and the decoupling. If the gap shrinks by more than about 0.1 or the low-σb/high-σop quadrant count drops by more than 30%, the effect is partly a rubric artifact. A second check connects the measurement to the downstream harm: run the E2 25-word compressed handoffs through the E3-style downstream probes and regress per-handoff final leakage on per-handoff σb while controlling for σop.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires measuring survival of boundary metadata separately from operational facts. Both σb and σop are produced by a single GPT-5.5 Thinking judge using the rubric in Appendix B.4, which tells the judge that boundary metadata are \"instructions, caveats, constraints, uncertainty markers, audience limits, ownership limits, or privacy limits\" and that \"use discretion\" should be scored weakened when the gold marker was a hard boundary. Because hard compression systematically shortens text, a judge primed to look for uncertainty markers and audience limits may classify compression-related shortening as weakened or absent even when the underlying semantic restriction is inferable or preserved in a shorter form. Human validation (κ=0.74, accuracy 0.92) does not fully close this gap: the human labels were produced by the two authors using the same rubric and the same construct definitions, so shared construct bias is not excluded. The PrivacyLens replication in Section E is strong evidence for cross-dataset reproducibility, but it uses the identical frozen judge, so it validates the behavior of the same judge/scenario pipeline rather than the independence of the construct. If the judge is biased downward on markers relative to facts, the headline numbers in Table 2, the low-σb/high-σop quadrant counts, and the CMH odds ratios in Table 22 would all be inflated. The downstream experiments (E3, E4, E8, E9) use hand-authored marker variants rather than per-handoff σb scores, so they would not detect such a bias. The risk is not that the paper is fabricated; it is that the quantitative core of summary collapse is mediated by a measurement instrument whose construct definition is supplied by the authors and never independently re-scored without that definition.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"Multi-agent LLM systems coordinate through handoff artifacts: one agent compresses an upstream interaction into a summary that another agent treats as shared state. The paper identifies and measures 'summary collapse'—the claim that this compression preferentially preserves operational facts while weakening or dropping the boundary metadata (audience limits, disclosure caveats, hedges, ownership claims) that govern how those facts may be used. The authors construct BOUND-Handoff, 36 hand-authored scenarios across seven domains crossed with four handoff surfaces and five handoff conditions, and measure boundary-marker survival (σb) and operational-fact survival (σop) with a GPT-5.5 Thinking judge validated against human labels (κ=0.74). On GPT-5-mini, DeepSeek-R1-32B, and Qwen3-32B, σb and σop are nearly uncorrelated at the handoff level and diverge under a 25-word budget (σb ≈ 0.56–0.69 versus σop ≈ 0.97–1.00), with 25–39% of markers scored absent. Downstream probes show marker explicitness causally controls leakage (vague hedges leak in 73%/50%/38% of cases; explicit constraints at most 8/48 per model under semantic detection), that prompt-only mitigation and exact-string redaction leave residual leakage, and that a gold-derived typed audience allowlist suppresses leakage almost entirely while degrading smoothly under simulated label error.","tokens_in":29057,"tokens_out":27514,"duration_ms":281625,"significance":"The paper's core idea—moving the privacy unit of analysis from the downstream agent's decision to the intermediate handoff artifact—is genuinely differentiated from the single-agent contextual-integrity benchmarks it cites, and if the measurements hold it has direct engineering consequences (typed, machine-actionable boundary fields in handoff schemas). The experimental discipline is a real strength and should be credited: three model families with held-out open replicas; a 720-cell design matrix with a documented stratified sample; scenario-level bootstrap CIs; pinned-package reproducible statistics; two leakage detectors with opposite bias directions; manual audits that revise leak counts upward when heuristics under-count; external annotators on E7 analyzed conservatively (dual-agreement floor, unresolved disputes retained as unresolved); and a cross-dataset replication on PrivacyLens that is transparent about its weaker σop grounding. The paper also reports its own controls honestly, including the negative role-label manipulation, the null three-hop full-replay result, and the E5 single-agent control.","major_comments":[{"comment":"The decoupling that anchors the paper's headline claim is measured by a single GPT-5.5 Thinking judge whose rubric embeds the authors' boundary-versus-fact construct (the Appendix B.4 definition list of 'instructions, caveats, constraints, uncertainty markers, audience limits, ownership limits, or privacy limits') and which explicitly instructs that a handoff softening a hard marker to 'use discretion' be scored 'weakened.' The reported human validation (κ=0.74, accuracy 0.92) was produced by the two authors applying that same rubric, as the Limitations section candidly concedes, and the PrivacyLens replication in Section E reuses the same frozen judge, so it establishes stability of the judge-scenario pipeline rather than independence of the construct. Because hard compression shortens every handoff, a marker-primed judge could systematically rate compression-related shortening as weakened or absent even when the governing restriction remains inferable or is present in shorter paraphrase; if so, the σb-σop gaps in Table 2, the absent mass in Table 3, and the CMH odds ratios in Table 22 would all be inflated in the same direction. This concern does not touch the leakage-based experiments (E3/E4/E8 rest on the exact/alias detector, a separately calibrated semantic judge, and manual audit, with the E8 gradient replicated on all three models), so the causal role of marker strength survives even a biased survival judge; what is at stake is the magnitude and construct purity of the decoupling itself. I request (i) an external annotation pass on a stratified E1/E2 sample using a plain-language 'is this restriction present in the handoff' question posed without the paper's construct definitions, and (ii) per-handoff behavioral grounding showing that judge-scored-absent markers predict downstream behavior on the same artifacts (the analysis requested in Major Comment 2 would supply this).","section":"4.4 / B.4; Tables 2–3"},{"comment":"The abstract's causal framing ('this handoff step is a structural source of privacy leakage') is composed from two designs that never meet on the same artifacts. E1/E2 show that compression lowers judge-scored σb while σop holds near ceiling; E3/E4/E8 show that handoffs constructed without an effective marker leak downstream (48/48 and 37/48 for no-marker; 44/48 and 38/48 for 'use discretion') while constructed explicit-marker handoffs rarely leak. The actual compressed handoffs of E2, including those with 25–39% of markers scored absent in Table 3, are never run through the downstream leakage probes, and no reported analysis regresses per-handoff leakage on per-handoff σb; the three-hop study is the only design in which generated handoff artifacts reach downstream agents, and there final leakage is non-monotonic across only 12 scenarios. As it stands, the link 'compression lowers σb' plus 'low σb leaks' is an informal inference across experiments rather than a direct measurement. Please either add a matched experiment that feeds free-text and 25-word E2 handoffs on the same scenarios into the E3/E4 downstream probes and reports leakage as a function of σb (which would also validate the survival measure behaviorally), or explicitly rephrase the abstract and Section 5 to present the two results as complementary but separately demonstrated.","section":"5.1–5.2; E2 vs. E3/E4/E8"}],"minor_comments":[{"comment":"The headline numbers 'σb≈0.80', 'σb≈0.57', and 'under 15%' mix models and detectors; attribute them explicitly (0.781 is the DeepSeek free_text value, 0.557 is the Qwen 25-word value, and the E8 GPT-5-mini L3 cell reaches 8/48 ≈ 17% under the semantic judge, above the stated 15%).","section":"Abstract, 5.2"},{"comment":"The baseline trajectory is non-monotonic in a way the main text does not acknowledge: DeepSeek-R1-32B free_text σb (0.781) is below its compressed_free_text (0.885), and GPT-5-mini's 40-word cap (0.919) is essentially unchanged from compressed_free_text (0.916) before the sharp drop at 25 words; a sentence in Section 5.1 explaining that mild caps do not reduce σb (and that DeepSeek drops markers even in uncompressed free text) would strengthen the selectivity claim.","section":"C.1–C.2, Tables 8/15"},{"comment":"With σop near ceiling, the handoff-level Pearson r has almost no variance to correlate, so the near-zero values are weakly informative; the quadrant counts (7/180 and 26/180) and the CMH odds ratios are the informative statistics, so consider presenting those as primary and describing r as auxiliary.","section":"5.1, C.4"},{"comment":"The 'Flagship' row model identity is not disclosed, but the row underpins the claim that summary collapse is not an artifact of small or distilled models; identify the model or justify withholding the identity.","section":"Table 2"},{"comment":"The survival judge is GPT-5.5 Thinking while one of the evaluated pipelines is GPT-5-mini; cross-model σb comparisons (e.g., GPT baseline 0.954 versus DeepSeek 0.880) may include an in-family evaluator preference, which should be acknowledged or tested.","section":"4.4, B.4"},{"comment":"The E9-versus-E6 comparison pools 48 downstream prompts against 288 across different handoff-generation protocols; the within-E9 no-marker controls (43/48 and 39/48 for the primary models, 36/48 for the Qwen column of Table 32) are the cleaner baselines, and the text's '36/48 no-marker control' should say which model's control is meant.","section":"5.3, Table 19, C.12"},{"comment":"The sentence that the no-handoff direct-full-marker condition 'leaks more often than the operationalized handoff' compares E5 with E3 across conditions that differ in marker operationalization; the control's clean evidence is that single-agent direct access with strong markers still leaks (20/48 GPT, 11/48 DeepSeek), so rephrase the comparison accordingly.","section":"5.2, C.8"},{"comment":"Label each table or column with the detector that produced it; in particular, the E6 counts (58/288 and 42/288) are not identified as exact/alias-only, and if they are, the paper should state that they undercount paraphrase leakage, a conservative direction for its claim.","section":"Tables 4–5, 23–26, 31–32"},{"comment":"Report per-hop σb and per-hop leakage for the three-hop study in a table; the text currently gives only the hop-1→hop-3 paired differences, and the non-monotonicity claim is otherwise uncheckable against the released data.","section":"5.3"},{"comment":"Specify which model and threshold define the calibrated semantic judge; the text describes its calibration on human-labeled E3 data but does not name the model.","section":"4.4"},{"comment":"Fix the grammar slip in the CPM paragraph, where a standalone period interrupts the sentence ('We follow Petronio (2002) specifically. whose rule-management heuristics...').","section":"2"}],"recommendation":"major_revision","confidential_remarks":"Strong submission with unusually good transparency: the released testbed, prompts, pinned requirements, and raw annotation labels make the empirical claims checkable, and the negative controls (role-label manipulation, three-hop full replay, E5) are reported honestly. The two major comments are in-scope revision work: an external annotation pass on the survival rubric and a direct compression-to-leakage experiment. One item the editor may want resolved during review is the identity of the 'flagship' model in Table 2, since the scale-control claim depends on it. The citation pattern is normal and the related-work coverage of single-agent privacy benchmarks and inter-agent extraction is accurate; the artifact-level unit of analysis is genuinely differentiated from the cited prior work. Scope fit with cs.AI is appropriate, though the paper reads as a strong EMNLP-type empirical submission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"You should know that this paper makes a clean empirical case that multi-agent LLM handoffs preferentially strip boundary metadata while preserving operational facts, and that marker explicitness causally controls downstream leakage. The core finding—that summary collapse is a structural privacy failure mode at the handoff artifact level—is likely real and important for the growing deployment of multi-agent systems.\n\nWhat it does well: the experimental design is thorough, spanning three model families, controlled conditions, human validation (κ=0.74), multiple sensitivity checks, and a causal marker-strength experiment (E8) that holds content constant while varying only the marker language. The ordinal label proportions and scenario-stratified CMH tests strengthen the case beyond the numeric σb/σop averages. The paper is also honest about its limitations, including the need for production-traces validation and the gold-label nature of the allowlist mitigation.\n\nThe soft spot is the one the stress-test note hits: the central σb and σop scores are produced by a single GPT-5.5 Thinking judge whose prompt encodes the paper's own boundary-vs-fact distinction. The human validation used the same rubric from two of the authors, so shared construct bias is not ruled out. The PrivacyLens replication uses the identical frozen judge, so while it validates reproducibility across datasets, it does not validate construct independence. The ordinal proportions and CMH tests on the \"absent vs not absent\" binary are less vulnerable to weighting choices, but they still rely on the judge's classification of what counts as a missing marker.\n\nThis concern doesn't sink the paper. The qualitative decoupling pattern—markers drop much faster than facts, vague hedges leak while explicit constraints protect—is robust across multiple experiments and models. Even if an independent judge shifted absolute σb values, the relative pattern and the L1→L2 dose-response would likely hold. But the magnitudes in Table 2 and the quadrant counts should be read as upper bounds until independently rescored.\n\nWho this is for: researchers building or auditing multi-agent pipelines where handoff compression is used (scheduling, care coordination, finance). Practitioners will get concrete design guidance (typed audience fields over prose instructions).\n\nRecommendation: This paper deserves a serious referee. Send it to peer review. The judge-bias concern should be flagged for reviewers, but it does not warrant desk rejection—the contribution is important and the evidence base is strong enough to warrant community scrutiny.","headline":"Solid evidence for a real privacy failure mode in multi-agent LLM handoffs; the core qualitative pattern is robust even if the exact magnitudes may shift under independent measurement.","tokens_in":29521,"tokens_out":1670,"would_cite":true,"duration_ms":17129,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"Multi-agent LLM handoffs suffer from 'summary collapse': operational facts survive compression while the privacy rules that limit their use are weakened or dropped, causing systematic downstream leakage.","keywords":["multi-agent LLM systems","privacy leakage","summary collapse","boundary metadata","handoff compression","contextual integrity","communication privacy management","marker strength"],"falsifier":"A large-scale study of production multi-agent systems that measures handoff artifacts with ground-truth boundary labels, finding that σ_b and σ_op are highly correlated (Pearson r > 0.7) under compression, or that vague hedges provide equivalent protection to explicit constraints in downstream leakage tests, would contradict the central claim.","tokens_in":30085,"feed_emoji":"🔒","tokens_out":6280,"duration_ms":58892,"temperature":0.7,"pith_summary":"The paper identifies a structural privacy failure in multi-agent LLM systems: when one agent compresses an upstream interaction into a handoff artifact for a downstream agent, the summary preferentially preserves the operational facts the next agent needs while weakening or dropping the boundary metadata that governed how those facts could be used. The authors call this selective loss \"summary collapse\" and measure it with two survival scores—boundary-marker survival (σ_b) and operational-fact survival (σ_op)—showing they decouple under compression, especially under tight token budgets. Controlled experiments demonstrate that the strength of the boundary marker (vague hedges vs. explicit constraints) causally determines whether leaked content appears in audience-facing outputs, and that typed, machine-readable audience allowlists suppress leakage almost entirely when correctly populated. The work moves the privacy question from a single agent's decision to the artifact passed between agents, arguing that the real leak happens upstream of the final output.","feed_headline":"Handoff compression strips privacy rules, leaving facts exposed","feed_subtitle":"Boundary metadata vanishes under token budgets, enabling downstream disclosure without consent.","key_machinery":"The key mechanism is the decoupling of boundary-marker survival (σ_b) from operational-fact survival (σ_op) during handoff compression, quantified via a human-validated LLM judge (κ=0.74). The central causal chain is marker strength: a five-level gradient from no marker to hard imperative shows a sharp drop in leakage at the transition from vague hedge to soft suggestion (L1→L2), indicating that the marker must be readable as an operational instruction, not merely privacy-sounding language. The typed audience-allowlist schema operationalizes the boundary as a machine-actionable constraint, which nearly eliminates leakage when correctly populated.","core_discovery":"The central discovery is that handoff compression in multi-agent LLM systems systematically discards the rules that constrain sensitive information while preserving the information itself, a failure mode termed summary collapse. Using a custom testbed (BOUND-HANDOFF) of 36 scenarios across seven domains, the authors show that boundary-marker survival (σ_b) and operational-fact survival (σ_op) are nearly uncorrelated at the individual-handoff level, and a substantial fraction of handoffs land in the low-σ_b, high-σ_op quadrant. The decoupling widens under hard compression (e.g., a 25-word budget reduces σ_b to approximately 0.57 while σ_op stays above 0.97). Through matched downstream tests, they prove that marker explicitness—not mere presence—is the causal determinant of leakage: vague language like \"use discretion\" leaks in 73% of GPT-5-mini cases, whereas explicit constraints reduce leakage to under 15%. A gold-derived typed audience-allowlist schema eliminates leakage on two primary models, but only when the boundary field is correct; a corrupted allowlist restores leakage to near no-marker levels.","pith_inferences":["The selective loss pattern may generalize beyond handoff compression to any LLM summarization or memory retrieval task where operational content competes with meta-constraints, suggesting a broader vulnerability in agentic systems.","The typed allowlist approach, while effective in the controlled setting, requires accurate extraction of boundaries from upstream dialogue, which is itself an open problem that could introduce new failure modes (e.g., extraction errors amplifying leakage).","Cross-cultural or multilingual deployments may require recalibrating the marker-strength gradient, as politeness conventions differ across languages."],"forward_implications":["Multi-agent system designers should encode privacy boundaries as typed fields (e.g., allowed_audiences) rather than rely on natural-language prompts, as the latter are stripped by compression.","Privacy evaluation in multi-agent systems must measure handoff artifacts, not just final outputs, because boundary loss occurs upstream and may be hidden by later information loss.","The marker-strength gradient provides a design principle: vague hedges are functionally equivalent to no marker for leakage risk; only explicit, instruction-like constraints offer meaningful protection.","Boundary extraction becomes a bottleneck: systems need automated, paraphrase-robust validation of boundary fields with confidence scoring and abstention mechanisms."],"supporting_citations":[{"why":"Establishes that LLMs violate contextual-integrity norms even under explicit privacy prompts, providing the problem context that motivates studying handoff-level leakage.","marker":"Mireshghallah et al. 2024"},{"why":"Provides the PrivacyLens dataset used for external validation; the paper reproduces the decoupling pattern on those independently constructed traces.","marker":"Shao et al. 2024"},{"why":"Introduces Communication Privacy Management theory, supplying the conceptual framework for boundary metadata and rule-based disclosure.","marker":"Petronio 1991"},{"why":"Extends CPM with rule-management heuristics, which the paper operationalizes into the four marker categories (audience, caveat, hedge, ownership).","marker":"Petronio 2002"}],"fun_headline_variants":["Summary collapse: compressed handoffs keep facts, discard usage rules","Compressing agent handoffs erases privacy boundaries, not facts","Multi-agent summaries leak by dropping boundary rules, not content","Token budgets strip handoff privacy rules while facts survive","Handoff compression severs consent metadata from sensitive facts"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The experimental scenarios, though spanning seven domains and independently replicated on PrivacyLens, are hand-authored and semi-synthetic; the load-bearing premise is that the decoupling of σ_b and σ_op and the causal role of marker strength generalize to production multi-agent systems with real users, long-lived memory, and natural boundary-marking language.","fun_headline_variants_meta":{"raw":{"variants":["Summary collapse: compressed handoffs keep facts, discard usage rules","Compressing agent handoffs erases privacy boundaries, not facts","Multi-agent summaries leak by dropping boundary rules, not content","Token budgets strip handoff privacy rules while facts survive","Handoff compression severs consent metadata from sensitive facts"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000271,"raw_usage":{"cost":0.00027104,"is_byok":false,"cost_details":{"upstream_inference_cost":0.00027104,"upstream_inference_prompt_cost":0.0001778,"upstream_inference_completions_cost":9.324e-05},"total_tokens":1603,"prompt_tokens":1270,"completion_tokens":333,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":0,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":266}},"tokens_in":1270,"tokens_out":333,"cost_source":"openrouter_usage","duration_ms":4919,"temperature":1.0,"reasoning_tokens":266,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-09-09T06:31:53.669763+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"falsifier":"A large-scale study of production multi-agent systems that measures handoff artifacts with ground-truth boundary labels, finding that σ_b and σ_op are highly correlated (Pearson r > 0.7) under compression, or that vague hedges provide equivalent protection to explicit constraints in downstream leakage tests, would contradict the central claim.","supporting_citations":[],"review_version":1}