{"id":"21534e63-2d15-4a28-8d10-294d8d33ce5a","arxiv_id":"2506.17338","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":12,"one_line_summary":"The Co-Forgetting Protocol combines DistilBERT-based semantic voting, multi-scale temporal decay, and PBFT-style consensus to synchronize memory pruning in multi-agent systems, with a four-agent simulation reporting 52% memory reduction and 88% forgetting accuracy.","lead":"This paper proposes a protocol for groups of software agents to collectively prune their shared memory using semantic relevance scores, time-based decay, and a Byzantine fault-tolerant voting mechanism. The reported simulation shows reduced memory footprint and high voting agreement, but the evaluation is small-scale and the implementation omits key details.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1 is not PBFT: it lacks pre-prepare, view change, and authentication, and the reported 92% success under f=1, N=4 contradicts PBFT's liveness guarantee for the three non-faulty replicas, so the Byzantine-fault-tolerance claim is unsupported.","rationale":"The reader's weakest assumption is exactly the load-bearing concern: Algorithm 1 is not PBFT, and the reported 92% success rate under f=1, N=4 contradicts the liveness guarantee for the three non-faulty replicas. I agree with this identification. This is the most load-bearing issue because the abstract's central novelty is fault-tolerant consensus for memory pruning; if that fails, the protocol reduces to a two-round quorum vote with no formal Byzantine guarantee, regardless of the plausibility of the semantic voting and decay components. The paper does provide a code link, a concrete implementation stack, and repeated runs over five random seeds, which are genuine pieces of support, but they do not repair the consensus mismatch. The self-referential draft notes in Section V are concerning as editorial artifacts, but I do not rest the verdict on them; the PBFT problem is sufficient on its own. Therefore the rejection verdict should stand.","tokens_in":15642,"tokens_out":6126,"duration_ms":63941,"concrete_test":"Run a deterministic controlled experiment with the released code: N=4, f=1, coordinator fixed as one of the three honest agents, the faulty agent silent (sends no PREPARE/COMMIT messages), reliable delivery among the three honest agents, and 100 memory items proposed for forgetting; record whether all three honest agents reach the same keep/forget outcome for every item. Standard PBFT guarantees a decision for each item; if the released protocol leaves any item undecided or produces divergent honest decisions, Algorithm 1 lacks PBFT's liveness and agreement properties and the abstract's Byzantine-tolerance claim is unsupported. If all 100 items are decided, re-run with the faulty agent as coordinator to test the missing view-change path.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim (Abstract and Section V) is that the Co-Forgetting Protocol provides PBFT-backed Byzantine fault tolerance. Algorithm 1 is not PBFT: it contains only Prepare and Commit phases, with no pre-prepare, no sequence numbers, no view change, and no message authentication; its prepare threshold is 2f (line 8) even though Section III.B states that consensus requires 2f+1 matching messages in each phase. The decisive evidence is in Section V.C: under f=1 and N=4, the reported PBFT success rate is 92%, and failures are attributed to 'the faulty agent was critical for forming the 2f+1 quorum in a specific phase.' This attribution contradicts PBFT's liveness argument: with N=4 and f=1 there are 3 non-faulty replicas, exactly 2f+1, and they can certify decisions without the faulty replica once their messages are delivered. An 8% epoch-level failure rate is exactly what one would expect from a two-phase quorum protocol without view change when the faulty agent is the coordinator or delays messages. Therefore the 92% figure is evidence against the claimed guarantee, not for it, and the abstract's promise of agreement 'even in the presence of up to f Byzantine agents' is unsupported by the presented mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the Co-Forgetting Protocol, a framework for synchronized memory pruning in multi-agent systems. It combines three components: DistilBERT-based semantic relevance voting, multi-scale exponential temporal decay functions, and a consensus mechanism described as Practical Byzantine Fault Tolerance (PBFT) implemented over gRPC. The protocol is evaluated in a simulated four-agent system, with reported results of a 52% memory footprint reduction over 500 epochs, 88% voting accuracy against human-annotated labels on 200 items, a 92% PBFT consensus success rate under Byzantine faults with f=1, and an 82% LRU cache hit rate. The paper also describes implementation details using Pinecone, SQLite, and cachetools, and the authors make code available at a GitHub repository.","tokens_in":15789,"tokens_out":2701,"duration_ms":28916,"significance":"If the central claims held, the protocol would be a useful blueprint for fault-tolerant, semantics-aware collective memory management in distributed AI systems. The paper has some genuine strengths: the component architecture is clearly described, the authors provide a public code repository, and the limitations section is unusually candid about the absence of ablation studies, the subjectivity of the ground-truth annotations, and the need for larger-scale validation. However, the central load-bearing claim—that the protocol provides PBFT-grade Byzantine fault tolerance—is not supported by the presented algorithm or the reported experiments. In addition, the semantic relevance score R that drives the voting accuracy results is never defined, and the evaluation is heavily dependent on parameters described as chosen by preliminary tuning. These issues together mean the headline quantitative results cannot be interpreted as evidence for the paper's main contributions.","major_comments":[{"comment":"Algorithm 1 is not PBFT, so the abstract's claim of agreement 'even in the presence of up to f Byzantine agents' is unsupported. The algorithm contains only a prepare phase and a commit phase, with no pre-prepare phase, no sequence numbers, no view change, and no message authentication, all of which are essential to PBFT's safety and liveness. The prepare threshold in line 8 is 'received ≥ 2f identical votes', which is inconsistent with Section III.B's statement that consensus requires 2f+1 matching messages in each phase. More importantly, the reported 92% success rate under f=1, N=4 is evidence against the claimed PBFT guarantee, not for it: with N=4 and f=1, the three non-faulty replicas constitute exactly 2f+1 and can certify decisions without the faulty replica once their messages are delivered, so an 8% epoch-level failure rate attributed to 'the faulty agent being critical for forming the 2f+1 quorum' contradicts PBFT's liveness argument. The failure mode described is precisely what one would expect from a two-phase quorum protocol without view change, not from PBFT.","section":"Algorithm 1; Section III.B; Section V.C"},{"comment":"The relevance score R is never defined. Algorithm 3 simply states 'R ← LLM Relevance(text(m))' without specifying how DistilBERT's output is converted to a scalar, what range R takes, or how it is normalized with respect to D(t). Since the voting decision is C = ωD·D(t) + ωR·R compared against threshold θ, the 88% voting accuracy is not interpretable without a precise definition of R and the combination function. The paper also does not report any ablation separating the semantic component from the decay component, even though Section VI acknowledges that no extensive ablation study was performed.","section":"Algorithm 3; Section V.A; Section V.C"},{"comment":"The headline quantitative results are not sufficiently supported as independent validation. Section V.B states that parameters including δ, θ, ωD, ωR, and α were set as described in the methodology, but Section VI's internal-validity paragraph concedes that these settings 'were chosen based on preliminary tuning and literature review.' The voting accuracy of 88% is therefore, to an unknown degree, a measurement of how well the tuned thresholds and weights fit the 200-item human-annotated benchmark, rather than an independent test of the protocol. In addition, the evaluation is presented as a textual summary with no tables or figures, and Section V.C itself says that 'in a full paper, these would be presented with accompanying tables and figures.' The five-run averaging with no reported variance or significance testing further limits the strength of the claims.","section":"Section V.B; Section VI (Internal Validity)"},{"comment":"Sections III.D and III.E are duplicated verbatim, including the full text of the epoch synchronization description and the repeated 'Epoch Synchronization and Collective Management' heading. This duplication suggests the manuscript is not in a publishable form and also creates ambiguity about which version of Algorithm 4 is authoritative, since both sections contain the same algorithm but the second occurrence is placed after Section III.E rather than as a continuation of Section III.D.","section":"Section III.D and Section III.E"}],"minor_comments":[{"comment":"The text states that the pbft.proto file defines RPCs for 'propose, pre-prepare, prepare, commit' phases, but Algorithm 1 has no pre-prepare phase; either the implementation or the pseudocode is inconsistent, and this needs to be reconciled.","section":"Section IV.A"},{"comment":"The repeated references to a '16% improvement' in voting accuracy are unresolved: the text says this could be an improvement over a baseline, but no baseline is defined or measured, so the phrase should be removed or converted into a concrete comparison.","section":"Section V.C"},{"comment":"The step labels in Algorithm 7 are misaligned: the comment '// Step 2: Gather proposals from agents' appears after a loop that already collects proposals, and '// Step 3: PBFT Voting' appears after PBFT voting has already been performed. The comments should be renumbered or moved to match the actual control flow.","section":"Algorithm 7"},{"comment":"The definition of Memory Deletion Rate is written as a fraction without explicitly stating that it is multiplied by 100 for a percentage; this is a minor notational inconsistency with the other metrics.","section":"Section V.A"}],"recommendation":"reject","confidential_remarks":"The manuscript reads as an early draft rather than a completed submission: the evaluation section explicitly defers tables and figures to 'a full paper,' two sections are duplicated verbatim, and the reported PBFT failure mode contradicts the standard liveness argument for PBFT with N=4, f=1. The central Byzantine-fault-tolerance claim cannot be fixed by local edits; it would require reimplementing a genuine PBFT protocol (including pre-prepare, view change, and authentication) or substantially reframing the contribution as a weaker quorum-based voting protocol. The undefined relevance score R and the explicitly tuned parameters further undermine the quantitative claims. I recommend rejection, though the authors' willingness to release code and to state limitations openly is commendable and could serve as a foundation for a revised, more carefully scoped submission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The idea — agents using a small language model to score memory relevance, then voting to prune shared memory, with anti-Byzantine consensus — is a reasonable research direction, and the paper describes a concrete implementation with code on GitHub. But the central fault-tolerance claim is wrong as stated. Algorithm 1 is a two-phase quorum vote: prepare with a 2f threshold, commit with 2f+1. That is not PBFT — no pre-prepare, no sequence numbers, no view change, no authentication. The evaluation compounds the problem: under f=1, N=4, the paper reports a 92% consensus success rate and blames failures on the faulty agent being needed for the 2f+1 quorum. But with three non-faulty replicas you already have 2f+1=3; PBFT's liveness argument says they can decide without the faulty one. So the reported failures are exactly what you'd expect from this simplified protocol, not from PBFT, and the abstract's promise about tolerating up to f Byzantine agents isn't supported by the mechanism shown.\n\nWhat the paper does well: it integrates three components (decay, semantic voting, consensus) into a working prototype, it is transparent about the stack (gRPC, Pinecone, SQLite, DistilBERT), and the discussion section honestly lists threats to validity. That level of candor is rare.\n\nThe soft spots, in decreasing severity. The relevance score R from DistilBERT is never defined — no equation, no pooling description, just 'LLM Relevance(text(m))'. The voting-accuracy figure of 88% is largely a fit: weights and thresholds were hand-tuned on the same synthetic distribution as the 200-item human-annotated test set, with no baseline, no ablation, no error bars. The PBFT evaluation uses one faulty agent in one configuration, and the paper's own account of the 8% failures contradicts the advertised guarantee. There are also signs of haste: a duplicated section on epoch synchronization, placeholder text about where tables would go in 'a full paper', and a self-referential note about the 'user's original draft.'\n\nWho is this for? Someone working on MAS memory management might mine the system design and the code. The research direction deserves exploration, but the current draft doesn't have a single measured claim that I'd trust without reimplementation. If the authors fix the consensus protocol (or rename it), define the relevance score, and run proper baselines and ablations, it could become a decent workshop or short-conference paper. As is, I'd desk-reject with an invitation to resubmit after those changes.","headline":"A promising but overclaimed prototype: the consensus mechanism isn't PBFT, and the headline fault-tolerance number undercuts the guarantee it's meant to show.","tokens_in":16492,"tokens_out":3150,"would_cite":false,"duration_ms":31043,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The Co-Forgetting Protocol claims to make multi-agent memory pruning a synchronized, semantics-aware, Byzantine-fault-tolerant process, reporting 52% footprint reduction and 88% agreement with human judgments.","keywords":["multi-agent systems","memory pruning","collective forgetting","Byzantine fault tolerance","PBFT","semantic voting","temporal decay","DistilBERT"],"falsifier":"Instrument the protocol so every PBFT message is logged, then run $N=4$ with one faulty agent that sends conflicting votes but stays reachable. PBFT's guarantee is that the three non-faulty agents still decide; if any epoch ends without a decision, or if the 92% success rate is reproduced, the implementation is not realizing PBFT. The direct check is whether each decision is backed by at least $2f+1=3$ identical prepare messages and $2f+1=3$ identical commit messages from distinct agents.","tokens_in":15233,"feed_emoji":"🧠","tokens_out":8985,"duration_ms":80309,"temperature":0.7,"pith_summary":"Multi-agent systems accumulate shared memories that grow stale, and this paper tries to make forgetting a collective, fault-tolerant decision rather than a private one. The proposed Co-Forgetting Protocol has each agent score every memory by combining semantic relevance (computed by DistilBERT) with multi-scale temporal decay, then uses PBFT-style consensus over gRPC to decide which items to delete. In a four-agent simulation the paper reports a 52% reduction in memory footprint over 500 epochs, 88% agreement with human keep/forget annotations, 92% consensus success with one faulty agent, and an 82% cache hit rate. The paper presents these results as evidence that the protocol is a working blueprint for synchronized, semantics-aware memory pruning in distributed AI systems.","feed_headline":"Collective forgetting cuts multi-agent memory 52%","feed_subtitle":"DistilBERT semantic votes plus PBFT consensus decide what to forget, matching humans 88% of the time.","key_machinery":"The load-bearing object is the Co-Forgetting Protocol itself, and inside it the decisive identity is the comparison between the weighted forgetting score $S_m=\\sum_{i:\\mathrm{vote}_i(m)=\\mathrm{forget}} w_i c_i$ and the dynamic quorum $Q=\\alpha \\sum_i w_i \\mathrm{IsActive}(a_i)$: an item is deleted only if both the PBFT consensus and this quorum test pass. Staleness is summarized by the multi-scale decay $D(t)=\\sum_{i=1}^n \\gamma_i \\exp(-(t-t_{\\mathrm{last}}(m))/S_i)$, and each agent's individual vote is $C=\\omega_D D(t)+\\omega_R R$, with $R$ the DistilBERT relevance score. PBFT—a Byzantine fault-tolerant consensus protocol that lets $N\\ge 3f+1$ agents agree through pre-prepare, prepare, and commit phases despite up to $f$ faulty agents—is the component that is supposed to give the collective decision its fault-tolerance guarantee.","core_discovery":"The central claim is that synchronized memory pruning can be engineered as a three-part loop: agents independently propose forgetting based on a combined score $C = \\omega_D D(t) + \\omega_R R$, where $D(t)$ is a multi-scale exponential decay of last-access time and $R$ is DistilBERT's relevance judgment; a quorum-weighted forgetting score $S_m = \\sum w_i c_i$ for 'forget' votes is then checked against a dynamic threshold $Q = \\alpha \\sum w_i \\mathrm{IsActive}(a_i)$; and PBFT consensus, requiring $2f+1$ matching prepare and commit messages, is used so that the decision survives up to $f$ Byzantine agents in an $N \\ge 3f+1$ system. The paper argues this combination aligns forgetting with human judgment, prevents unilateral deletion of shared knowledge, and keeps memory growth in check. Its evidence is a four-agent simulation reporting 52% memory-footprint reduction, 88% voting accuracy, 92% PBFT success under one faulty agent, and 82% cache hit rate.","pith_inferences":["If the missing PBFT machinery (pre-prepare, view change, signed messages) were added, the '92% success' figure should approach 100% for decisions among live non-faulty agents; the current number likely describes the simplified two-phase implementation, not the PBFT protocol's limit.","Replacing the fixed thresholds $\\delta, \\theta, \\omega_D, \\omega_R$ with a meta-learned adaptation rule would let the same voting machinery track changing task contexts, a direction the paper lists for future work.","An ablation that deactivates the DistilBERT term $R$ while keeping decay, and vice versa, would isolate which component produces the 88% agreement with human annotations; the paper notes such ablation studies are still pending.","Because the consensus layer is separable from the vector store, the voting logic could be benchmarked against open-source storage backends, turning the reported cache and latency figures into a reproducible comparison rather than a single-cloud measurement."],"forward_implications":["At the boundary case $N=4, f=1$, the protocol is meant to keep deciding correctly even when one agent is malicious or unresponsive; the reported 92% success rate implies roughly 8% of fault-injected epochs defer some memory decisions to the next epoch.","Because deletion requires both PBFT consensus and the quorum threshold, no single agent can unilaterally remove a shared memory; this is the property that separates the protocol from local, per-agent pruning.","An 82% cache hit rate and a roughly 45% reduction in Pinecone write calls keep per-epoch latency near 1250 ms, making a 100-interaction epoch loop practical in the simulation.","The protocol's claim of Byzantine tolerance is stated for any $N\\ge 3f+1$, but the paper's own evidence is limited to four agents, so larger systems remain an extrapolation rather than a demonstrated result."],"supporting_citations":[{"why":"Defines PBFT and its $N \\ge 3f+1$, $2f+1$-quorum guarantee, the fault-tolerance property the protocol claims.","marker":"[1]"},{"why":"Establishes why multi-agent memory bloat and inconsistency degrade system performance, motivating synchronized pruning.","marker":"[2]"},{"why":"Raft is the crash-only consensus alternative that the paper contrasts with PBFT to justify the stronger Byzantine model.","marker":"[6]"},{"why":"Supplies the continual-learning notion of protecting important weights, which the paper adapts into memory retention logic.","marker":"[10]"},{"why":"Shows that selective knowledge removal in language models can reduce footprint without major degradation, a premise for LLM-based forgetting votes.","marker":"[14]"},{"why":"Provides the machine-unlearning concept of deleting specific data points that the multi-agent vote generalizes to shared memory.","marker":"[15]"}],"fun_headline_variants":["Semantic voting and PBFT trim multi-agent memory 52%","PBFT consensus helps multi-agents forget 52% of shared memory","AI agents vote to forget, cutting memory 52% via PBFT","Co-Forgetting Protocol: PBFT-based semantic pruning cuts memory 52%","Byzantine fault tolerance meets semantic voting to prune agent memories"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the simplified two-phase voting procedure over gRPC provides PBFT's Byzantine fault-tolerance guarantees, even though it omits PBFT's pre-prepare phase, view change, and message authentication; the paper's own 92% success under $f=1, N=4$ is below the unanimous non-faulty decision that PBFT promises.","fun_headline_variants_meta":{"raw":{"variants":["Semantic voting and PBFT trim multi-agent memory 52%","PBFT consensus helps multi-agents forget 52% of shared memory","AI agents vote to forget, cutting memory 52% via PBFT","Co-Forgetting Protocol: PBFT-based semantic pruning cuts memory 52%","Byzantine fault tolerance meets semantic voting to prune agent memories"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000798,"raw_usage":{"total_tokens":3569,"prompt_tokens":1062,"completion_tokens":2507,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":678,"completion_tokens_details":{"reasoning_tokens":2426}},"tokens_in":678,"tokens_out":2507,"duration_ms":14992,"temperature":1.0,"reasoning_tokens":2426,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:29:09.905955+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Instrument the protocol so every PBFT message is logged, then run $N=4$ with one faulty agent that sends conflicting votes but stays reachable. PBFT's guarantee is that the three non-faulty agents still decide; if any epoch ends without a decision, or if the 92% success rate is reproduced, the implementation is not realizing PBFT. The direct check is whether each decision is backed by at least $2f+1=3$ identical prepare messages and $2f+1=3$ identical commit messages from distinct agents.","supporting_citations":[{"cited_title":"Practical Byzantine Fault Tolerance,","cited_arxiv_id":null,"evidence_quote":"Defines PBFT and its $N \\ge 3f+1$, $2f+1$-quorum guarantee, the fault-tolerance property the protocol claims."},{"cited_title":"Wooldridge, *An Introduction to MultiAgent Systems*, 2nd ed","cited_arxiv_id":null,"evidence_quote":"Establishes why multi-agent memory bloat and inconsistency degrade system performance, motivating synchronized pruning."},{"cited_title":"In search of an understandable consensus algorithm,","cited_arxiv_id":null,"evidence_quote":"Raft is the crash-only consensus alternative that the paper contrasts with PBFT to justify the stronger Byzantine model."},{"cited_title":"Overcoming catastrophic forgetting in neural networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the continual-learning notion of protecting important weights, which the paper adapts into memory retention logic."}],"review_version":1}