Pith. sign in

REVIEW 3 major objections 10 minor 26 references

No multi-user agent system preserves privacy under shared use

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-07-07 17:40 UTC pith:6XEG2653

load-bearing objection New benchmark for cross-user privacy spillage in shared agentic systems; absolute violation rates are judge-sensitive but qualitative findings hold. the 3 major comments →

arxiv 2607.05318 v1 pith:6XEG2653 submitted 2026-07-06 cs.MA cs.CR

PiSAs: Benchmarking Contextual Integrity in Multi-User Agentic Systems

Shubham Gupta , Nazanin Mohammadi Sepahvand , Abhinav Kumar , Cem Subakan , Spandana Gella , Pierre-Andr\'e No\"el , Perouz Taslakian , Eugene Bagdasarian
show 1 more author
Valentina Zantedeschi
This is my paper
classification cs.MA cs.CR
keywords acrossagenticagentsmemoryprivacyshareduserscontextual
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces PiSAs, a benchmark for measuring unintentional privacy leakage when LLM agents serve multiple users who share the same agentic infrastructure. The central insight is that existing privacy benchmarks, grounded in contextual integrity theory, focus on single-user settings or independently owned agents negotiating with each other, and therefore miss a qualitatively new risk: when users share agents and memory, sensitive information spills across users through the system's own internal channels, not just through outputs to external recipients. PiSAs addresses this with dual annotations on every piece of information: whether it is appropriate to use for the current task, and which users are legitimately allowed to see it. This lets the benchmark track two distinct failure modes: appropriateness violations, where the system uses information it should not, and visibility violations, where information reaches users who should not have access to it. The authors evaluate three agent topologies (single, centralized, decentralized), three memory configurations (none, private, shared/hybrid), and three LLM backbones, and find that no configuration achieves acceptable privacy preservation. Data and agent partitioning reduces leakage but does not eliminate it, and adding memory shifts violations from communication channels into memory stores rather than removing them. The paper identifies the core bottleneck as the LLM's own judgment: even the strongest models fail to reliably filter inappropriate content or restrict information to authorized users, particularly when sensitive attributes are entangled with task completion.

Core claim

The central discovery is that in multi-user agentic systems, privacy violations are pervasive and structural rather than incidental. Partitioning information across agents and restricting communication topology reduces but does not eliminate violations, because the LLM's judgment calls remain the bottleneck. When memory is introduced to improve task performance, violations migrate from agent-to-agent communication channels into memory stores, making memory a persistent and harder-to-defend leakage surface. The hardest case for systems is not irrelevant personal information but decision-critical personal attributes that must be used in sanitized form yet are frequently propagated in their raw

What carries the argument

The benchmark's machinery is the dual annotation scheme: each attribute in a scenario carries a task-contextual appropriateness label (whether it should be used for the task) and a per-person visibility structure (which users may legitimately access it). These two orthogonal annotations, combined with evaluation at three leakage surfaces (agent-to-agent communication, memory stores, and gathered-information summaries), allow the benchmark to distinguish between appropriateness violations and visibility violations, and to detect when mitigations relocate rather than eliminate leaks. The scenarios are constructed backward from a hidden oracle solution, ensuring that a privacy-preserving answer

Load-bearing premise

The evaluation pipeline relies on LLM judges to detect whether privacy violations occurred. If these judges systematically over-count or under-count violations, for example by hallucinating matches or missing paraphrased leaks, the reported violation rates could be artifacts of the evaluation method rather than properties of the agentic systems being tested. The authors note that absolute metric values vary across judge configurations, though they claim the main findings hold

What would settle it

If an LLM-judge configuration were found that systematically reversed the relative ranking of system designs or backbones on violation rates, or if a deployed system with the same architecture achieved near-zero violations on these scenarios through a mitigation not studied in the paper, the central claim that no configuration achieves acceptable privacy would be weakened.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Multi-user agentic systems deployed in organizational settings today likely already exhibit the cross-user spillage this benchmark measures, since no studied configuration achieves acceptable privacy preservation.
  • Memory systems in multi-agent architectures function as unregulated data lakes for sensitive information, and privacy-preserving memory designs that enforce visibility constraints at the storage layer rather than relying on LLM judgment are needed.
  • The finding that broad policy guidance performs worse than no guidance at all suggests that naturalistic policy documents may license information use rather than constrain it, with implications for how organizations instruct both human and AI agents.
  • Fine-tuning on contextual integrity reasoning, rather than prompt-level instructions alone, may be necessary to close the judgment gap the paper identifies as the primary bottleneck.
  • The benchmark's design, which separates appropriateness from visibility, could be extended to cross-task leakage scenarios where information legitimately accessed for one task surfaces inappropriately in a later one.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 10 minor

Summary. The paper introduces PiSAs, a benchmark for evaluating contextual integrity (CI) violations in multi-user agentic systems where multiple users share the same infrastructure (agents, memory, communication channels). Unlike prior multi-user CI benchmarks that study independently owned agents negotiating via dialogue, PiSAs evaluates spillage through shared infrastructure. The benchmark comprises 85 manually curated workplace scenarios across three task families (JIRA allocation, meeting allocation, severity classification), each annotated with dual CI labels: task-contextual appropriateness and per-person visibility. The authors evaluate three agent topologies (single, centralized, decentralized), three memory configurations, three LLM backbones, and varying privacy-prompt levels. The central finding is that no studied configuration achieves acceptable privacy preservation: appropriateness violations remain pervasive (e.g., V_appr exceeding 77% for single-agent systems), partitioning reduces but does not eliminate leakage, and memory-based mitigations relocate violations to memory stores rather than eliminating them. The paper also finds that task-entangled personal attributes are the hardest to handle, and that strict rule specification outperforms broad policy guidance.

Significance. The paper addresses a genuine gap in the CI benchmarking literature: prior work either studies single-user settings or independently owned multi-agent systems, but does not consider the shared-infrastructure multi-user setting where spillage occurs through memory and inter-agent communication. The dual-annotation scheme (appropriateness + visibility) is well-motivated and enables measurement of qualitatively distinct failure modes. The systematic experimental design—varying topology, memory, backbone, and prompt defenses—is a strength, as is the transparent reporting of the LLM-judge pipeline and its sensitivity. The worked example (Appendix J) concretely demonstrates genuine violations, lending credibility to the evaluation. The finding that memory relocates rather than eliminates violations is a useful and actionable insight for system designers. The benchmark's system-agnostic design and the commitment to release data under Apache 2.0 are additional strengths.

major comments (3)
  1. §5, Table 2, and Appendix I, Tables 24–25: The central claim that 'no studied configuration achieves an acceptable level of privacy preservation' (§6) rests on absolute violation rates produced by the LLM-judge pipeline. The paper's own sensitivity analysis shows that replacing the verifier ensemble nearly doubles centralized V_vis from 22.9% to 42.9% and V_appr from 26.2% to 42.9% (Table 24, centralized column). While the authors state that 'qualitative trends remain consistent,' the claim of 'no acceptable configuration' is a statement about absolute levels, not just trends. The paper does not define what 'acceptable' means quantitatively, making the central claim unfalsifiable as stated. The authors should either (a) define an explicit threshold for 'acceptable' privacy preservation and show it is exceeded under all judge configurations, or (b) reframe the conclusion to acknowledge it
  2. §5 and Appendix B: The 'even partially' verification criterion (Appendix B: verifiers vote on whether each candidate 'reveals the key fact, even partially') sets a low bar that could conflate sanitized operational abstractions with raw attribute leaks. For example, an agent stating 'Eli has a medical condition affecting room choice' might be flagged as partially revealing 'documented hyperacusis diagnosis,' even though this is closer to the sanitized form ('Meetings requiring Eli should not use Delta'). The paper acknowledges sanitized forms in Appendix A and the worked example (Appendix J) shows the centralized system producing 'Eli has a medical condition making the Delta room unsuitable' — which appears to be a reasonable sanitization, yet would likely be flagged. The paper should report what fraction of flagged violations are partial/paraphrased leaks versus near-verbatim raw-attr
  3. §5.2, Figure 3: The claim that memory 'relocates violations rather than eliminating them' is supported by the observation that total visibility violations rise from 36–47% to 63–90% with Hybrid memory. However, the paper also notes that the reduction in A2A violations is 'confounded with a reduction in the number of A2A messages' (agents read from memory instead of communicating). The paper does not report whether the total number of attribute-exposure opportunities is comparable across memory conditions. If memory reduces total information flow volume, a fair comparison would normalize violations by exposure opportunities (e.g., violations per attribute-transmission event). Without this, it is unclear whether memory genuinely increases total violations or whether the higher count reflects more surfaces being audited. The authors should either add this normalization or qualify the claim.
minor comments (10)
  1. Abstract: 'PiSAsis' — missing space before 'is'.
  2. Abstract: 'We introducePiSAs' — missing space before 'PiSAs'.
  3. Table 1: The 'Acc.' column uses 'T' and 'U' for accumulation type, but the legend only explains these in the caption indirectly ('cross-task T, cross-user U or – not measured'). A reader unfamiliar with the notation may find this unclear on first reading.
  4. §3, Scenario Construction: The paper states '85 scenarios' but Table 3 reports 25+25+35=85. This is consistent, but the distribution across task families is uneven (Severity has only 3 appropriate attributes per scenario). The paper should discuss whether this imbalance affects the aggregated metrics reported in Table 2.
  5. Appendix A.3: Visibility annotations for Severity Classification are assigned using Qwen3.6-27B as an LLM judge, while JIRA and Meeting allocations use manual annotation. This inconsistency should be noted in the main text (§3), not only in the appendix, as it affects the reliability of visibility violation rates for the Severity task.
  6. Table 2: The 'Failure rate' column is defined as 'fraction of runs with at least one appropriateness violation' but the caption does not make clear whether this is per-run or per-scenario (union across K=3). Appendix D.4 clarifies it is per-run, but the main table caption should state this.
  7. Figure 3: The y-axis label 'Violation (%)' is ambiguous — it is unclear whether this is V_appr, V_vis, or both. The figure caption should specify which metric is plotted for each bar group.
  8. Appendix G, Tables 19–20: The any-K aggregation is justified well, but the paper could note that with K=3, the any-K union inflates rates relative to mean-run by approximately 1.3–1.5× (based on the tables). This quantitative characterization would help readers calibrate the absolute numbers.
  9. References: Several entries cite 2026 papers (e.g., Anthropic 2026, Park et al. 2026, Fu et al. 2026). If these are accepted/published, please verify venue and page numbers; if preprints, please mark as 'Preprint' consistently.
  10. §4.1, Single configuration: The paper states 'we simulate this system by giving an LLM unrestricted access to all available information in its context (except the ground-truth labels).' It would help to clarify whether the single agent receives all attributes in a single prompt or through simulated multi-turn interaction.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive report. The referee correctly identifies three substantive issues: (1) the central 'no acceptable configuration' claim is not tied to a defined threshold and is sensitive to judge choice; (2) the 'even partially' verification criterion may conflate sanitized abstractions with raw attribute leaks; and (3) the memory 'relocation' claim is not normalized by exposure opportunities. We agree with all three points and will revise the manuscript accordingly. Specifically, we will define an explicit acceptability threshold, add a breakdown of partial vs. near-verbatim leaks, and add a normalized violation-rate analysis for the memory comparison. No standing objections remain.

read point-by-point responses

  1. Referee: The central claim that 'no studied configuration achieves an acceptable level of privacy preservation' rests on absolute violation rates from the LLM-judge pipeline, but the sensitivity analysis shows judge replacement nearly doubles some metrics, and 'acceptable' is undefined, making the claim unfalsifiable.

    Authors: The referee is correct on both points. The term 'acceptable' is used informally and is not tied to a defined threshold, which makes the central claim unfalsifiable as stated. We also agree that the sensitivity analysis in Appendix I shows substantial absolute-level variation under different judge/verifier configurations, which complicates any claim resting on absolute rates. We will make two changes. First, we will define an explicit acceptability threshold. We propose adopting a per-attribute violation rate threshold of 5% (i.e., at most 5% of inappropriate attributes leaked, and at most 5% of hidden attributes exposed to unauthorized users), motivated by the severity of privacy harm from even single-attribute leaks in organizational settings. Under this threshold, even the most conservative judge configuration (Paid J + Paid V) yields V_appr of 25.4% for the best centralized configuration (Table 2), far exceeding 5%. The alternative judge configuration that produces the lowest centralized V_appr (Gemma J + Paid V: 22.8%) still exceeds the threshold by a factor of 4.5. No configuration approaches this bar under any judge configuration we tested. Second, we will reframe the conclusion to explicitly state the threshold and acknowledge the judge sensitivity: 'Under a 5% per-attribute violation threshold, no studied configuration achieves acceptable privacy preservation under any judge configuration tested, though absolute violation rates vary substantially across evaluation stacks.' We believe this makes the claim both falsifiable and appropriately qualified. revision: yes

  2. Referee: The 'even partially' verification criterion sets a low bar that could conflate sanitized operational abstractions with raw attribute leaks. The paper should report what fraction of flagged violations are partial/paraphrased leaks versus near-verbatim raw-attribute leaks.

    Authors: We agree this is an important distinction and that the current presentation does not adequately separate these cases. The referee's example is well-taken: a statement like 'Eli has a medical condition making the Delta room unsuitable' (which appears in the centralized system's output in Appendix J) is arguably a reasonable sanitization, yet under our 'even partially' criterion it would likely be flagged as partially revealing the raw attribute 'documented hyperacusis diagnosis.' We will address this in two ways. First, we will add a breakdown classifying flagged violations into three categories: (a) near-verbatim raw-attribute leaks (the sensitive detail is propagated essentially as-is), (b) partial/paraphrased leaks (some identifying detail from the raw attribute is present, but not the full sensitive fact), and (c) sanitized-abstraction leaks (only the operational implication is conveyed, which our current pipeline may still flag). We will produce this breakdown by running a post-hoc categorization pass on the existing flagged violations. Second, we will discuss the limitation that our verification pipeline does not distinguish sanitized abstractions from partial leaks, and note that this may inflate violation rates for task-entangled attributes where the system produces a reasonable sanitization. We note that even after accounting for this, the violation rates for clearly irrelevant personal attributes (V_appr = 19.3% averaged over systems, Table 12) — where sanitization is not applicable — remain well above any reasonable acceptability threshold, so the core finding is not solely an artifact of the partial-leak criterion. revision: yes

  3. Referee: The claim that memory 'relocates violations rather than eliminating them' is confounded by a reduction in A2A message volume. The paper does not normalize violations by exposure opportunities, making it unclear whether memory genuinely increases total violations or whether the higher count reflects more surfaces being audited.

    Authors: This is a fair concern. The current comparison is between raw violation counts (or rates) across memory conditions that differ in the number of information-transmission events. As the referee notes, Hybrid memory reduces A2A messages from ~11 to ~4 (Table 10), so comparing raw A2A violation rates across conditions is not apples-to-apples. We will add a normalized analysis. Specifically, we will compute violations per attribute-exposure opportunity, where an exposure opportunity is defined as an instance where an attribute could potentially surface at a given surface (e.g., one A2A message, one memory write event). This normalizes for the fact that memory changes both the volume and the location of information flow. We will report this alongside the raw rates. We expect that the normalized analysis will show that memory reduces per-opportunity violation rates on the A2A channel (consistent with the current observation that A2A violations drop sharply) but that the memory channel itself has a high per-opportunity violation rate, supporting the 'relocation' claim in a more controlled way. However, we will also qualify the claim if the normalized analysis shows that the total per-opportunity violation rate does not increase with memory. We agree that without this normalization, the strong form of the 'relocation' claim is not fully supported. revision: yes

Circularity Check

0 steps flagged

No significant circularity found; benchmark metrics are defined independently of evaluated systems

full rationale

The paper constructs scenarios with ground-truth annotations (appropriateness labels and visibility matrices) established during scenario construction (Appendix A), entirely independent of the agentic systems being evaluated. The evaluation pipeline (Appendix B) uses LLM judges to detect whether pre-defined attributes appear in system outputs, communications, or memory — the ground truth is not derived from system behavior. The violation metrics (V_appr, V_vis) are computed as |detected violations| / |attributes at risk|, where both numerator and denominator reference pre-defined annotations, not system outputs. The central claim — that no configuration achieves acceptable privacy preservation — is an empirical finding from running systems against this benchmark, not a result that reduces to its inputs by construction. The paper cites Mireshghallah et al. (2025) for the any-K aggregation strategy and privacy-prompting approach, but this citation is methodological rather than load-bearing: the paper provides its own ablation comparing three aggregation strategies (Appendix G, Tables 19–20) and its own sensitivity analysis across judge/verifier configurations (Appendix I, Tables 24–25), showing that while absolute values shift, qualitative trends persist. The skeptic's concern about the LLM-judge pipeline inflating rates is a measurement-validity concern (correctness risk), not circularity — the judge's criteria are defined independently of the systems under test. No step in the derivation chain reduces to its own inputs by definition or by self-citation.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

The paper introduces a benchmark and evaluation framework rather than a theoretical derivation or new physical model. There are no free parameters fitted to data, no new mathematical axioms, and no invented entities. The core 'invention' is the dataset and the dual-annotation scheme, which are evaluated empirically.

axioms (3)
  • domain assumption Contextual Integrity (CI) theory (Nissenbaum, 2004) accurately models privacy as a property of information flows rather than data in isolation.
    Invoked in Section 1 and Section 3 to define the appropriateness and visibility annotations.
  • domain assumption LLM-judges can reliably detect privacy violations and task completion when given ground-truth attributes.
    The entire evaluation framework (Appendix B) relies on LLM-judges to compute violation rates, completeness, and utility.
  • domain assumption The 85 manually curated scenarios are representative of real-world workplace privacy challenges.
    The generalizability of the findings depends on the scenarios covering the relevant failure modes, acknowledged as a limitation in Section 7.

reviewed 2026-07-07 · how reviews work

0 comments
Cite this review

Pith. "Pith review of PiSAs: Benchmarking Contextual Integrity in Multi-User Agentic Systems." pith.science (2026). https://pith.science/paper/6XEG2653

@misc{pith2026260705318,
  author       = {Pith},
  title        = {Pith review of: PiSAs: Benchmarking Contextual Integrity in Multi-User Agentic Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6XEG2653}},
  note         = {Machine review of arXiv:2607.05318}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

As LLM agents evolve from single-user assistants into shared organizational infrastructure, new privacy risks emerge: inappropriate information may not only be exposed through outputs for external recipients, but also internally across users through inter-agent messages, shared memory and agents. These data spillage risks are not captured by existing privacy benchmarks grounded in contextual integrity (CI) as they focus primarily on either single-user settings or interactions between independently owned agents. We introducePiSAs (Privacy in Shared Agentic systems), a benchmark for assessing unintentional leaks with dual CI annotations: whether an information is appropriate for the task, and which users may legitimately access it. This enables direct measurement of cross-user spillage across agentic system components and interfaces, such as outputs, inter-agent communication, and memory. PiSAsis system-agnostic and supports evaluation across different agent topologies and memory regimes. We find that, although system design improves CI compliance, results are bottlenecked by incorrect LLM judgment calls: even state-of-the-art models fail to reliably filter inappropriate content or restrict transmission to authorized users. Our findings underscore the need for privacy-preserving strategies, beyond those studied in this work.

Figures

Figures reproduced from arXiv: 2607.05318 by Abhinav Kumar, Cem Subakan, Eugene Bagdasarian, Nazanin Mohammadi Sepahvand, Perouz Taslakian, Pierre-Andr\'e No\"el, Shubham Gupta, Spandana Gella, Valentina Zantedeschi.

Figure 1
Figure 1. Figure 1: Problem setting. A team of users in a given workplace share an agentic system to complete tasks. The system may consist of LLM agents specialized to a user or to a task. Memory is another common component, to persistently store information useful for future tasks. Attributes, such as user data, reside within the system and should flow according to CI norms. In this example, the system is tasked with assign… view at source ↗
Figure 2
Figure 2. Figure 2: PiSAs is system-agnostic. (a) It allows to evaluate completeness, utility, and violations at three surfaces: memory, A2A communication and task output. In this work, we study three main agent topologies: single, (b) decentralized and (c) centralized. required substantial manual effort, but this depth is what enables us to surface and analyze failure modes specific to multi-user organizational settings. To … view at source ↗
Figure 3
Figure 3. Figure 3: Average violation rates on JIRA Allocation scenar￾ios by the Centralized system with Sonnet-4.6 as back￾bone. Results are broken down by type (appropriateness Vappr. and visibility Vvis.) and surface (where they occur). We plot violation rates across the three surfaces 7 [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Ablations on Severity Classification scenarios under the centralized system with Claude-Sonnet-4.6. in [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Privacy-preserving instructions for memory writing (left panel) and A2A communication (right panel) at [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Correlations across system performance metrics. Each panel shows a per-system, per-backbone scatter of two metrics, averaged over all tasks and scenarios. (a) Completeness vs. visibility violation rate. (b) Completeness vs. appropriateness violation rate. (c) Visibility violation rate vs. appropriateness violation rate. In panels (a) and (b), the top-left corner is the desirable region (higher completeness… view at source ↗
Figure 7
Figure 7. Figure 7: Per-scenario relationships among privacy and task metrics for Meeting Allocation. We study how appropriateness violations (Vappr) relate to completeness, utility, and visibility violations (Vvis) on a per-scenario basis across all Meeting Allocation scenarios. The plots show clear trends across both axes of variation: the backbone LLM used for orchestration and the choice of system design. Thin lines indic… view at source ↗
Figure 8
Figure 8. Figure 8: Total appropriateness Vappr. and visibility Vvis. violation rates on JIRA Allocation scenarios with High prompt-level, by the Centralized system with Sonnet-4.6 as backbone, depending on memory configuration. have no scheduling consequence. They should not be used as evidence for the schedule and should not be propagated through gathered summaries, inter-agent messages, memory, or the final response [PITH… view at source ↗
Figure 9
Figure 9. Figure 9: Three personal attribute types used in the Meeting Allocation analysis. Decision-critical personal attributes must [PITH_FULL_IMAGE:figures/full_fig_p021_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Strict rule specification shown to the system in the explicit-rubric case. [PITH_FULL_IMAGE:figures/full_fig_p023_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Policy-style rendering of the same severity-classification logic. This version preserves the intended decision [PITH_FULL_IMAGE:figures/full_fig_p023_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Four cue variants for the same attribute with a scheduling implication. [PITH_FULL_IMAGE:figures/full_fig_p026_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Cue pools used in the privacy-cue conditions. Pool A is used for sensitive raw attributes that have an appropriate [PITH_FULL_IMAGE:figures/full_fig_p026_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 26 canonical work pages · 4 internal anchors

  1. [1]

    2506.17336 , archivePrefix=

    Yubeen Bae and Minchan Kim and Jaejin Lee and Sangbum Kim and Jaehyung Kim and Yejin Choi and Niloofar Mireshghallah , year=. 2506.17336 , archivePrefix=

    work page arXiv

  2. [2]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Privaci-bench: Evaluating privacy with contextual integrity and legal compliance , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    work page

  3. [3]

    2024 , isbn =

    Bagdasarian, Eugene and Yi, Ren and Ghalebikesabi, Sahra and Kairouz, Peter and Gruteser, Marco and Oh, Sewoong and Balle, Borja and Ramage, Daniel , title =. 2024 , isbn =. doi:10.1145/3658644.3690350 , booktitle =

    work page doi:10.1145/3658644.3690350 2024

  4. [4]

    CI-Bench: Benchmarking Contextual Integrity of AI Assistants on Synthetic Data

    Zhao Cheng and Diane Wan and Matthew Abueg and Sahra Ghalebikesabi and Ren Yi and Eugene Bagdasarian and Borja Balle and Stefan Mellem and Shawn O'Banion , year=. 2409.13903 , archivePrefix=

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    2025 , note =

    Introducing. 2025 , note =

    work page 2025

  6. [6]

    2026 , note =

    Introducing. 2026 , note =

    work page 2026

  7. [7]

    2026 , volume=

    Yagoubi, Faouzi El and Badu-Marfo, Godwin and Mallah, Ranwa Al , journal=. 2026 , volume=

    work page 2026

  8. [8]

    G old C oin: Grounding Large Language Models in Privacy Laws via Contextual Integrity Theory

    Fan, Wei and Li, Haoran and Deng, Zheye and Wang, Weiqi and Song, Yangqiu. G old C oin: Grounding Large Language Models in Privacy Laws via Contextual Integrity Theory. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. doi:10.18653/v1/2024.emnlp-main.195

    work page doi:10.18653/v1/2024.emnlp-main.195 2024

  9. [9]

    CI -Work: Benchmarking Contextual Integrity in Enterprise LLM Agents

    Fu, Wenjie and Qin, Xiaoting and Zhang, Jue and Lin, Qingwei and Wutschitz, Lukas and Sim, Robert and Rajmohan, Saravan and Zhang, Dongmei. CI -Work: Benchmarking Contextual Integrity in Enterprise LLM Agents. Proceedings of the 64th Annual Meeting of the A ssociation for C omputational L inguistics (Volume 6: Industry Track). 2026. doi:10.18653/v1/2026.a...

    work page doi:10.18653/v1/2026.acl-industry.103 2026

  10. [10]

    2024 , eprint=

    Gemma 2: Improving Open Language Models at a Practical Size , author=. 2024 , eprint=

    work page 2024

  11. [11]

    2025 , eprint=

    Gemma 3 Technical Report , author=. 2025 , eprint=

    work page 2025

  12. [12]

    2024 , eprint=

    Operationalizing Contextual Integrity in Privacy-Conscious Assistants , author=. 2024 , eprint=

    work page 2024

  13. [13]

    Memory in the Age of AI Agents

    Yuyang Hu and Shichun Liu and Yanwei Yue and Guibin Zhang and Boyang Liu and Fangyi Zhu and Jiahang Lin and Honglin Guo and Shihan Dou and Zhiheng Xi and Senjie Jin and Jiejun Tan and Yanbin Yin and Jiongnan Liu and Zeyu Zhang and Zhongxiang Sun and Yutao Zhu and Hao Sun and Boci Peng and Zhenrong Cheng and Xuanbo Fan and Jiaxin Guo and Xinlei Yu and Zhen...

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [14]

    arXiv preprint arXiv:2510.15186 (2025)

    Gurusha Juneja and Jayanth Naga Sai Pasupulati and Alon Albalak and Wenyue Hua and William Yang Wang , year=. 2510.15186 , archivePrefix=

    work page arXiv

  15. [15]

    1-2-3 Check: Enhancing Contextual Privacy in LLM via Multi-Agent Reasoning

    Li, Wenkai and Sun, Liwen and Guan, Zhenxiang and Zhou, Xuhui and Sap, Maarten. 1-2-3 Check: Enhancing Contextual Privacy in LLM via Multi-Agent Reasoning. Proceedings of the The First Workshop on LLM Security (LLMSEC). 2025

    work page 2025

  16. [16]

    Niloofar Mireshghallah and Hyunwoo Kim and Xuhui Zhou and Yulia Tsvetkov and Maarten Sap and Reza Shokri and Yejin Choi , booktitle=. Can. 2024 , url=

    work page 2024

  17. [17]

    2511.14937 , archivePrefix=

    Niloofar Mireshghallah and Neal Mangaokar and Narine Kokhlikyan and Arman Zharmagambetov and Manzil Zaheer and Saeed Mahloujifar and Kamalika Chaudhuri , year=. 2511.14937 , archivePrefix=

    work page arXiv

  18. [18]

    Washington Law Review , volume =

    Helen Nissenbaum , title =. Washington Law Review , volume =

    work page

  19. [19]

    PAC - BENCH : Evaluating Multi-Agent Collaboration under Privacy Constraints

    Park, Minjun and Kim, Donghyun and Ju, Hyeonjong and Lim, Seungwon and Choi, Dongwook and Kwon, Taeyoon and Kim, Minju and Yeo, Jinyoung. PAC - BENCH : Evaluating Multi-Agent Collaboration under Privacy Constraints. Findings of the A ssociation for C omputational L inguistics: ACL 2026. 2026. doi:10.18653/v1/2026.findings-acl.1552

    work page doi:10.18653/v1/2026.findings-acl.1552 2026

  20. [20]

    2026 , eprint=

    PersistBench: When Should Long-Term Memories Be Forgotten by LLMs? , author=. 2026 , eprint=

    work page 2026

  21. [21]

    PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action , url =

    Shao, Yijia and Li, Tianshi and Shi, Weiyan and Liu, Yanchen and Yang, Diyi , booktitle =. PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action , url =. doi:10.52202/079017-2837 , editor =

    work page doi:10.52202/079017-2837

  22. [22]

    2026 , eprint=

    Agents of Chaos , author=. 2026 , eprint=

    work page 2026

  23. [23]

    AgentSocialBench: Evaluating Privacy Risks in Human-Centered Agentic Social Networks

    Prince Zizhuang Wang and Shuli Jiang , year=. 2604.01487 , archivePrefix=

    work page internal anchor Pith review Pith/arXiv arXiv

  24. [24]

    The Eleventh International Conference on Learning Representations , year=

    ReAct: Synergizing Reasoning and Acting in Language Models , author=. The Eleventh International Conference on Learning Representations , year=

    work page

  25. [25]

    Zharmagambetov, Arman and Guo, Chuan and Evtimov, Ivan and Pavlova, Maya and Salakhutdinov, Ruslan and Chaudhuri, Kamalika , booktitle =

    work page

  26. [26]

    CalBench: Evaluating Coordination-Privacy Trade-offs in Multi-Agent LLMs

    Chelsea Zou and Yiheng Yao and Selena She and Noah Goodman and Robert D. Hawkins , year=. 2605.09823 , archivePrefix=

    work page internal anchor Pith review Pith/arXiv arXiv

This paper was first reviewed by glm-5.2 on July 7, 2026.