{"id":"b8f8f68e-5711-4571-adbc-5594502e5588","arxiv_id":"2608.03272","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"In agent-based collaborative filtering, attack spread and privacy leakage grow with interaction connectivity, but the effect is asymmetric between user and item agents and differs between early and steady-state phases.","lead":"This paper tests how the connectivity settings of agent-based movie recommenders change their vulnerability to attacks: how many items a user agent rates per turn, and how much of the item catalog is shared across users. It maps where manipulation and privacy leaks spread fastest, and shows single attack-success scores hide differences between user and item agents.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-run ASR curves leave F3's slope/plateau decoupling statistically unsupported; repeated-seed confidence intervals are needed before accepting connectivity as the control knob.","rationale":"I considered the two assumptions the reader flagged — the k/rho confounds and the uncalibrated LLM judge — but the missing repeated-seed statistics is the single most load-bearing gap. The confounds are explicitly acknowledged and partly inherent to the operational definitions of k and rho, and the judge has deterministic spot-checks for canary/PII content; neither, on its own, would invalidate the central 'connectivity modulates vulnerability' claim as strongly as the absence of error bars. The paper's central refinement F3 is a claim about comparing two summary statistics (transient slope vs steady-state mean) across configurations; with one run per configuration, any apparent reversal could be sampling noise. This is not a fatal flaw, because the paper honestly lists it as a limitation, but it means the empirical characterization is not yet established at the level of confidence the policy conclusion ('single-scalar ASR is not sufficiently descriptive') requires. The paper otherwise deserves credit: adaptations are concrete, ablations cover attacker ratio, style, and LLM backbone, and the predictive metric is appropriately framed as exploratory with its over-simplification acknowledged. A conditional acceptance is the right stance; the recommended seed-level check is cheap, directly targets the load-bearing ordering, and would either rescue or downgrade F3.","tokens_in":22712,"tokens_out":4919,"duration_ms":62671,"concrete_test":"Repeat the MASLeak and CORBA/NetSafe connectivity sweeps (k=1,2,3 at rho=1; rho=0.5,1,2 at k=2) with N=10 independent seeds each, keeping prompts, judge model, and dataset fixed. For each configuration compute per-turn ASR means and 95% bootstrap CIs for transient slope (U_tr, I_tr) and steady-state mean (U_ss, I_ss). Then test the F3-defining orderings (e.g., MASLeak: U_tr k=3>k=2>k=1 but U_ss k=2>k=3>k=1; NetSafe item: I_tr k=3>k=1>k=2 with I_ss k=3>k=2>k=1). If the contradictory pairings fall within overlapping CIs or fail a paired test at alpha=0.05, F3 is not established and the temporal-regime claim should be downgraded to exploratory; if the orderings survive across seeds, the central claim is materially strengthened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central refinement (F3, §4: faster early transient slope does not always imply higher steady-state ASR; §5's claim that a single scalar ASR is insufficient) rests on comparing curve orderings across k and rho. Every plotted curve and every entry in Table 1 appears to come from a single run; §8 Limitations (4) explicitly lists 'statistical significance across repeated seeds' as future work. LLM agents are stochastic, and the non-monotonic patterns that carry the argument — e.g., MASLeak steady-state k=2 > k=3 despite k=3's faster transient, NetSafe item-side k=2 vs k=1 reversal, TOMA FPR-driven reversal, CORBA item-side rho non-monotonicity — are exactly the kind of small-gap orderings that can flip between runs. If the pairwise orderings are within run-to-run noise, F3 collapses and the practical recommendation that robustness must be reported per temporal regime loses its empirical basis. This concern is independent of the k/rho confounds: even if the axes are cleanly defined, the measured outcome differences still need repeated-seed support to be interpretable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper adapts a suite of MAS-inspired attacks (CORBA, NetSafe, MAMA, MASLeak, TOMA, MASTER) and defenses (G-Safeguard, BlindGuard, T-Guard, M-Guard) to the AgentCF multi-agent collaborative filtering framework, and evaluates them under two connectivity axes: candidate count k (user decision bandwidth, k=1,2,3) and catalog concentration rho (user/item ratio, rho=0.5,1,2). Outcomes are tracked per turn and per role partition (user/item) using an LLM judge to produce ASR curves. The central empirical findings are F1 (growth-saturation dynamics), F2 (role asymmetry), and F3 (faster transient growth does not always imply higher steady-state ASR). The paper also proposes an epidemic-inspired static predictor (Eq. 2) for ranking configurations by expected attack outcome. It concludes that single-scalar ASR is insufficient and that robustness reporting should separate temporal regimes and role partitions. Implementation is publicly released.","tokens_in":22910,"tokens_out":2354,"duration_ms":27979,"significance":"If the findings hold, the paper makes a useful contribution to agentic RecSys safety: it demonstrates that connectivity, measured through k and rho, produces non-trivial and role-dependent effects on attack/defense outcomes, and it provides a structured set of eight reproducibility elements (§5) that could inform future evaluation practice. The open codebase, cross-LLM ablations (§6.3), and attacker-ratio ablations (§6.1) are concrete strengths. The exploratory epidemic predictor (§7) is a promising idea for cost-efficient robustness screening. However, the empirical foundation is currently weakened by single-run curves, an uncalibrated LLM judge, and acknowledged confounds in the connectivity manipulations. These issues are load-bearing because the paper's main claims—notably F3—rely on small, sometimes non-monotonic ordering differences between configurations.","major_comments":[{"comment":"Every plotted ASR curve and every Table 1 symbol appears to come from a single run per configuration; the paper itself lists 'statistical significance across repeated seeds' as future work. The claims that carry the paper—MASLeak steady-state k=2 > k=3 despite k=3's faster transient; NetSafe item-side k=1/k=2 reversal; TOMA edge-FPR reversal; CORBA item-side rho non-monotonicity—are exactly small-gap orderings that can flip between stochastic LLM runs. Without repeated-seed confidence intervals and pairwise significance tests, F3 and the 'single scalar ASR is not sufficiently descriptive' policy claim lack empirical support. This is not a presentation issue; it is central to the paper's core characterization.","section":"§4, Table 1, §8 Limitations (4)"},{"comment":"The LLM judge is the primary instrument for ASR, the main outcome variable. The paper states that deterministic checks are only used to 'spot-check' the judge, and that human-annotated calibration (FPR/FNR, prompt sensitivity) is a limitation. If the judge has systematic biases or high prompt sensitivity, all reported ASR levels and their cross-configuration orderings shift. Given that ASR is the basis for every qualitative finding (F1-F3), the paper should provide at least a human-annotated validation set with judge error rates per partition and attack type, and ideally report results with deterministic labels where available (e.g., exact canary matching) as a robustness check.","section":"§3.6, §8 Limitations (4)"},{"comment":"The paper acknowledges that 'adjusting k co-varies with prompt and ranking task structures, while resampling for rho shifts item degree distributions.' This is a serious confound for the central claim that connectivity modulates vulnerability. For example, k=1 is binary like/dislike feedback, k=2 is ranking two items, and k=3 is ranking three items; observed differences could reflect task format rather than interaction density per se. Similarly, rho variation resamples the catalog, changing item degree distributions as well as overlap. To support the connectivity interpretation, the paper should either add controls (e.g., hold task format fixed for different k, or use degree-matched resampling), or reframe the claims as joint effects of connectivity and task/format changes. As written, the causal attribution to connectivity is not supported.","section":"§3.3, §8 Limitations (2)"},{"comment":"The Recovery-Aware First-Order Connectivity predictor is presented as enabling 'cost-efficient robustness assessment,' but the correlation results are obtained by fitting (gamma, r) per attack and partition to maximise in-sample Spearman correlation with the observed outcomes. This is an in-sample, per-configuration fit with two free parameters per partition, and the 'one-hop freezing' steady-state assumption is an ad-hoc simplification. Without held-out evaluation across configurations not used for fitting, or a proper complexity penalty, the reported predictive skill could be overfitting. The paper should report out-of-sample correlation (e.g., leave-one-configuration-out) and error bars on the fitted correlation, and state clearly that the predictor is a descriptive summary rather than a validated forecasting tool.","section":"§7, Eq. (2)"}],"minor_comments":[{"comment":"The heading 'TO MA' appears to be a typo for 'TOMA'. Also, Fig. 3 captions could list which subplot corresponds to which variant more explicitly; the marker/color legends are dense.","section":"§4.3.1"},{"comment":"Mixed British/American spelling ('defence' vs 'defense') is present; this should be harmonized. Some references (e.g., [2], [13], [34], [80]) are dated 2026 and may not be universally available; please verify they are publicly accessible.","section":"Throughout"},{"comment":"Table 1 uses '↕' for 'curve behaviors differ but lack consistent ordering.' This conflates genuinely non-monotonic but reproducible effects with run-to-run noise. Once repeated-seed data are added, it would be helpful to mark which '↕' entries are statistically distinguishable from flat ordering.","section":"§5, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is relevant to RecSys's safety and agentic systems line, and the adaptation of MAS attacks to AgentCF plus the per-partition temporal analysis are valuable. However, the empirical spine is currently single-run and partially confounded; the authors themselves acknowledge the missing repeated-seed and judge-calibration analyses. With those added, the paper could meet the bar for a solid empirical contribution. I recommend major revision rather than rejection because the claimed phenomena are plausible and the limitations are explicit; the missing experiments are within scope of the current manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper gives the first connectivity map of agentic CF vulnerability, splitting connectivity into candidate count k and catalog concentration rho, and tracking user/item partitions separately. It reproduces six attacks and four defenses in AgentCF, ships code, and is unusually open about what it cannot claim. That is real credit: for a system class that is being deployed, this is a useful first map.\n\nWhat is genuinely new is F3: faster early slope does not always mean higher steady-state ASR (e.g., MASLeak k=2 > k=3 at plateau despite k=3's faster transient). The paper uses F3 to argue that single-scalar ASR is insufficient and that robustness should be reported per temporal regime. That is exactly where the evidence is thinnest. Every curve and table entry appears to come from a single run; section 8(4) explicitly lists statistical significance across repeated seeds as future work. LLM agents are stochastic, and the orderings that carry F3 — NetSafe item-side k=2 vs k=1 reversal, TOMA's FPR-driven reversal, MASLeak's k=2/k=3 swap — are small-gap effects that can flip between runs. If those orderings are within noise, F3 collapses and the temporal-regime recommendation loses its empirical basis. The stress-test concern lands.\n\nTwo more soft spots, both acknowledged in the manuscript. First, the primary instrument, the LLM judge, is uncalibrated (sections 3.6 and 8(4)); deterministic spot-checks help but don't validate false-positive/negative rates or prompt sensitivity. Second, the k and rho manipulations co-vary with task format and item degree distributions (section 8(2)), so the causal label 'connectivity' is not isolated. The section 7 epidemic predictor also has a fit-then-correlate problem: (gamma, r) are fitted per attack and partition on the same data they are then correlated with, which inflates the reported rank correlations. The paper calls this extension exploratory, but the circularity should be stated.\n\nOn balance: as a first characterization, it is defensible and honestly scoped. As support for F3 and the 'report per temporal regime' advice, it needs repeated-seed confidence intervals, judge calibration, and a cleaner decomposition of the axes. I would send it to peer review — the map is worth drawing even if the contour lines need redrawing — but the revision bar should be explicit about error bars and confound analysis. I wouldn't cite the specific numbers in my own work yet, but I'd cite the framing. Worth a reading group slot as a case study in evaluating agentic RecSys security.","headline":"First connectivity map of agentic CF vulnerability, honestly scoped, but single-run curves and acknowledged confounds mean the specific F3 ordering claims need replication before they carry weight.","tokens_in":23484,"tokens_out":3010,"would_cite":false,"duration_ms":30655,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that connectivity—candidate count and catalog overlap—shapes attack and defense outcomes in multi-agent collaborative filtering, with user and item agents responding differently and fast early attacks not predicting final","keywords":["responsible AI","multi-agent systems","collaborative filtering","connectivity","attack success rate","LLM agents","recommender system security","adversarial robustness"],"falsifier":"Run the same attack suite with $k$ varied while the response format is held constant (always binary like/dislike or always a ranking), and with $\\rho$ varied while item degree distributions are held fixed by constrained resampling; if the reported orderings across $k$ and $\\rho$ vanish or invert, the connectivity claim is confounded. Separately, have human annotators label a sample of agent memories and compare with the LLM judge's contamination labels; a materially high false-positive or false-negative rate would invalidate the ASR measurements.","tokens_in":22530,"feed_emoji":"🛡️","tokens_out":8838,"duration_ms":91228,"temperature":0.7,"pith_summary":"The paper tries to establish that in LLM-powered multi-agent collaborative filtering systems, connectivity is a primary driver of how attacks spread and how defenses hold. It adapts six attacks and four defenses from general multi-agent-systems research into the AgentCF recommender, then sweeps two connectivity axes: how many item candidates each user sees per turn ($k$) and how concentrated the item catalog is across users ($\\rho$). Its central empirical finding is that attack success is role-asymmetric and temporally non-monotonic: user agents and item agents respond differently, and a fast early contamination spike does not predict the eventual plateau. The authors argue this makes a single scalar attack-success-rate metric insufficient, and they propose per-partition, per-regime reporting plus an epidemic-style static predictor as a cheap way to rank configurations. A sympathetic reader would care because it tells recommender-system builders which wiring choices amplify or dampen security risks before deployment.","feed_headline":"Fast attacks don't always win in multi-agent recommenders","feed_subtitle":"Candidate count and catalog overlap shape attack success differently for user agents and item agents.","key_machinery":"The central instrument is a two-axis connectivity sweep on the bipartite AgentCF architecture: candidate count $k\\in\\{1,2,3\\}$ and catalog concentration $\\rho=n_U/n_I\\in\\{0.5,1,2\\}$, realized by resampling MovieLens-100K. Outcomes are per-turn, per-partition Attack Success Rates (ASR) from an LLM judge, split into transient slope and steady-state mean. A secondary mechanism is the recovery-aware first-order connectivity predictor, a closed-form SIS mean-field formula $R_U, R_I$ that ranks configurations from $k$, $\\rho$, attacker fractions, and fitted recovery parameters without running full simulations.","core_discovery":"The authors claim that the connectivity of a multi-agent CF system—candidate count $k$ and catalog concentration $\\rho$—systematically shapes vulnerability, with three reproducible patterns. F1: contamination and leakage grow fast and saturate, like epidemic spread. F2: user and item partitions respond differently to both axes, an asymmetry with no analogue in role-homogeneous MAS. F3: the transient growth slope and steady-state mean are decoupled; raising connectivity can accelerate early attacks without raising, or even while lowering, the eventual plateau, and vice versa. They further claim that extraction attacks (privacy leaks, reverse engineering) are modulated mainly by $k$, while dis","pith_inferences":["If the decoupling result generalizes, security benchmarks for agentic recommenders should report transient and steady-state ASR separately; a short red-team run may rank configurations opposite to their long-run risk.","The user/item asymmetry suggests a concrete defensive allocation: because user agents resist multi-hop spread while item agents do not, memory-update rules for item agents and bridge items are likely the higher-value defense surface.","The first-order predictor could be extended to second-hop transmission and heterogeneous degree distributions; if it holds up, it offers a configuration-screening tool that avoids costly full simulations.","A testable next step is to de-confound $k$ from task format (binary vs ranking) and $\\rho$ from degree distribution; the paper flags this as a limitation, and an experiment that controls these would either sharpen or overturn the connectivity characterization."],"forward_implications":["Raising $k$ generally accelerates early-phase attack growth and often raises steady-state contamination, but the plateau is capped: $k=2$ and $k=3$ frequently converge, and for some attacks $k=2$ ends ahead of $k=3$.","Catalog concentration $\\rho$ has little effect on pure extraction attacks, but denser catalogs increase item-side contamination and reverse-engineering success for dissemination and bidirectional attacks.","User and item partitions are asymmetric under both axes and across defenses; most defenses protect users better than items, with item-side gains concentrated in dense catalogs.","A single scalar ASR is insufficient; per-role, per-regime metrics are needed to rank configurations, and the paper's epidemic-inspired static predictor offers a partial but uneven shortcut."],"supporting_citations":[{"why":"AgentCF: the bipartite mesh CF system under attack; defines the canonical $k=2$ configuration and memory update mechanism.","marker":"[72]"},{"why":"NetSafe: source of the lower-connectivity-is-safer topology hypothesis and the misinformation attack adapted here.","marker":"[67]"},{"why":"CORBA: the contagious recursive blocking attack adapted for dissemination and resource-exhaustion experiments.","marker":"[81]"},{"why":"MAMA: topology-aware memory leakage attack adapted to seed and measure PII extraction.","marker":"[26]"},{"why":"MASLeak: worm-based IP extraction attack adapted for reverse-engineering metrics.","marker":"[46]"},{"why":"TOMA: topology-guided adversarial propagation attack and T-Guard defense adapted for bidirectional evaluation.","marker":"[24]"},{"why":"MASTER: role/topology attack framework and M-Guard defense adapted for bidirectional evaluation.","marker":"[83]"},{"why":"G-Safeguard: supervised GNN defense re-implemented on bipartite user-item graphs.","marker":"[48]"},{"why":"Discrete-time Markov chain contact-based spreading model that the recovery-aware first-order connectivity predictor is built on.","marker":"[7]"},{"why":"MovieLens-100K: dataset resampled to realize the catalog concentration $\\rho$ levels.","marker":"[8]"}],"fun_headline_variants":["Connectivity splits attack outcomes in multi-agent CF","User and item agents: separate attack responses in CF","Early attack speed and final damage diverge in recommenders","Catalog overlap and candidate count shape attack goals"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The central claim collapses if the connectivity sweeps do not isolate connectivity—since raising $k$ changes the per-turn ranking task format and raising $\\rho$ shifts item degree distributions—or if the LLM judge's binary contamination labels misclassify at a material rate.","fun_headline_variants_meta":{"raw":{"variants":["Connectivity splits attack outcomes in multi-agent CF","User and item agents: separate attack responses in CF","Early attack speed and final damage diverge in recommenders","Catalog overlap and candidate count shape attack goals"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000216,"raw_usage":{"total_tokens":1284,"prompt_tokens":773,"completion_tokens":511,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":450}},"tokens_in":517,"tokens_out":511,"duration_ms":5952,"temperature":1.0,"reasoning_tokens":450,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T22:03:14.617222+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same attack suite with $k$ varied while the response format is held constant (always binary like/dislike or always a ranking), and with $\\rho$ varied while item degree distributions are held fixed by constrained resampling; if the reported orderings across $k$ and $\\rho$ vanish or invert, the connectivity claim is confounded. Separately, have human annotators label a sample of agent memories and compare with the LLM judge's contamination labels; a materially high false-positive or false-negative rate would invalidate the ASR measurements.","supporting_citations":[],"review_version":1}