Pith. sign in

REVIEW 4 major objections 5 minor 46 references

A reusable online memory of who to trust lets multi-agent LLMs coordinate without retraining.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 22:12 UTC pith:IFAOU7TZ

load-bearing objection Useful reliability-memory primitive with real multi-readout experiments; the OOD story is mostly online label-driven adaptation, not zero-shot domain transfer. the 4 major comments →

arxiv 2607.27958 v1 pith:IFAOU7TZ submitted 2026-07-30 cs.MA cs.AI

Sigma-Mem: An Online Reliability Memory for LLM-based Multi-Agent Systems

Peilin Feng , Suorong Yang , Soujanya Poria This is my paper
classification cs.MA cs.AI
keywords multi-agent systemsreliability memoryLLM agentspeer selectionresidual steeringonline adaptationWeyl inequalitytrust and reputation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Long-horizon multi-agent systems need more than a record of what was said; they need a running account of which peers are trustworthy on which kinds of tasks. This paper introduces Σ-Mem, an online reliability memory that stores two kinds of evidence: how competent each peer has been on similar tasks, and how peers’ correctness patterns relate to one another. Both are kept as real symmetric matrices and updated from post-decision correctness feedback, with each update spectrally bounded so the memory adapts stably without retraining the models. The same frozen memory can steer a central model’s residual stream, route work to a peer before any response is generated, or weight votes by reliability. Across five Qwen-family central models it adapts when reliability shifts, generalizes to unseen peers and domains, and its direct readouts beat majority voting and the best fixed peer on a full out-of-distribution set, with accuracy rising as more feedback arrives.

Core claim

Reliability memory—task-conditioned historical competence per peer plus peer-relationship structure, maintained as real symmetric states and written from external correctness feedback—is a reusable coordination state for LLM multi-agent systems. By Weyl’s inequality each event-level write changes the spectrum only by a bounded amount, so persistent aligned evidence accumulates while noise stays controlled, enabling online adaptation without retraining and supporting residual steering, response-free routing, and reliability-weighted voting from the same memory.

What carries the argument

Σ-Mem: peer-specific symmetric competence matrices M_p and a symmetric relationship matrix G, updated as decayed state plus a rank-one correctness-weighted outer product along a task competence direction φ(x). Readout is either residual steering of a central model or direct scores s_p = φ(x)^T M_p φ(x) for routing and weighted voting; Weyl’s inequality bounds the spectral shift of each write.

Load-bearing premise

After each decision the system gets external ground-truth labels saying which peers were right or wrong; without those labels the memory cannot write new evidence and only decays.

What would settle it

On a mixed counterfactual and OOD peer-selection stream, freeze the memory and withhold post-decision correctness labels (or replace them with pure noise): if accuracy no longer rises with feedback availability and direct M-route/M-vote stop beating majority voting and the best fixed peer on the full OOD set, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • The same frozen reliability state can drive residual steering, response-free routing, or weighted voting without extra training.
  • New peers can be added at test time by allocating new memory states; learned parameters stay fixed.
  • When reliability patterns are stable, memory-only routing can match or beat a weak central judge; when history is ambiguous, current response judgment becomes essential.
  • Accuracy on OOD tasks improves as more correctness feedback is revealed, so the memory accumulates actionable trust signal over the stream.
  • Content memory and reliability memory play complementary roles: one stores what happened, the other who to trust when verification is hard.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Deployment without cheap post-hoc labels may need surrogate feedback (self-consistency, delayed human review, or environment reward) or the memory will stall in the low-feedback decay regime the paper already measures.
  • Adaptive readouts that down-weight historical guidance when the stream is ambiguous (as in CF@50) are a natural next control knob the conclusion already flags.
  • Task-conditioned spectral memory could sit beside conventional global reputation scores whenever peer strength varies sharply by domain.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Σ-Mem, an online reliability memory for LLM multi-agent systems that stores per-peer historical competence as real symmetric matrices {M_p} and peer-relationship structure as a symmetric matrix G, both updated from external post-decision correctness labels via decayed rank-one writes. Spectral stability is motivated by Weyl’s inequality and a concentration argument (Theorem 1) showing that persistent competence signal can dominate bounded noise in the leading eigenvalue. The same frozen memory state is read in three ways: residual steering of a central model for utility-based peer selection, response-free M-route, and reliability-weighted M-vote. Experiments on five Qwen-family centers cover counterfactual reliability shifts (CF@0/50/70/90), enlarged peer pools with unseen models, six OOD benchmarks, direct-readout baselines versus majority and best-fixed-peer, and feedback-availability ablations.

Significance. If the claims hold under realistic feedback, the work cleanly separates content memory from reliability memory and offers a reusable write–read interface that is not tied to one aggregation rule—an important coordination primitive for long-horizon MAS. Strengths include: (i) explicit separation of selection-time utilities from external write labels (Sec. 3.3); (ii) a standard but well-applied spectral stability argument with a full appendix proof of Weyl and Theorem 1; (iii) multi-axis evaluation (Tables 1–4, Fig. 3) including an honest CF@50 failure mode and feedback-ratio curves; and (iv) demonstration that direct memory readouts can beat majority voting and the best fixed peer without a central judge. These are concrete contributions beyond ad-hoc reputation scalars.

major comments (4)
  1. [Abstract; Sec. 4.4–4.5; Fig. 3; Table 4] Abstract, Sec. 4.4–4.5, and the contribution bullet on “generalizes to unseen… task domains” overstate what Table 3–4 show. During OOD evaluation, M_p and G are written from post-prediction correctness labels on the OOD stream itself (Sec. 3.1 Eqs. 6–9; Sec. 4.5). Fig. 3 is decisive: at 5–10% feedback, M-route and Σ-Mem sit at or below majority (59.12%); gains appear only as OOD labels increase. Table 4’s ~1.5–2 pt edge over majority is therefore largely online reputation on the test distribution, not evidence that competence directions learned on math/RAG/code transfer a useful reliability prior without new labels. Please add a frozen-memory / zero-update OOD control (train on the 2,963-event mix, then evaluate OOD with decay only or no writes) and rephrase “domain generalization” as “online adaptation on new domains given feedback,” unless that control shows transfer.
  2. [Sec. 4.3; Table 2] The same issue affects the unseen-peer claim in Sec. 4.3 / Table 2. New peers receive fresh M_p = 0 and must accumulate CF-stream feedback; trainable components are shared, but reliability evidence for unseen peers is not transferred from training. The paper should distinguish parameter generalization (shared P, g, φ) from reliability-state generalization, and report how many labeled events a new peer needs before M-route or Σ-Mem beats the base model / majority.
  3. [Sec. 3.1 Eqs. 6–9; Sec. 3.3; Fig. 3; Conclusion] Deployment assumption: every write requires external c_{p,t} ∈ {+1,−1} from the benchmark or environment (Sec. 3.1, 3.3). Fig. 3 already shows the low-feedback regime is hard, but the main claims and abstract still present Σ-Mem as a foundation for adaptive MAS coordination without a clear limitation statement on label cost. Real coordination often lacks cheap ground truth after each step. Please add a dedicated limitations subsection on label availability, and either (a) a weak-label / self-consistency / delayed-feedback experiment, or (b) an explicit scope restriction that results assume verifiable environments (code tests, closed QA).
  4. [Sec. 3.2–3.3; Sec. 4.1; Appendix B] Training and free parameters are under-specified relative to the experimental claims. Sec. 4.1 says Σ-Mem is “trained” on 2,963 events, and Sec. 3.2 introduces learned P and g, yet γ, η, γ_G, η_G, w_u, w_g, rank r, and the competence encoder φ are not reported with values, selection protocol, or sensitivity. Without this, Tables 1–4 are not reproducible and it is unclear what is frozen versus fit on the mixed set. Please report the full hyperparameter table, what objective trains P/g (and whether the center LLM is frozen), and a short sensitivity check on γ and η—especially given Corollary 1’s memory-length/adaptivity trade-off and the CF@50 pathology.
minor comments (5)
  1. [Figure 2] Fig. 2 panel labels repeat “c) Σ-mem Update” twice; panel (b) vs (c) layout is hard to parse in grayscale.
  2. [Sec. 2–3] Notation switches between Σ-Mem, Σ-mem, and 𝚺-Mem; pick one and use it consistently, including in theorem statements.
  3. [Sec. 5; Appendix C] Related Work cites classical MAS trust (TRAVOS, FIRE, REGRET) but does not clearly state what is new versus a task-conditioned matrix form of Beta reputation; Appendix C’s B1 baseline helps—point to it earlier in the main text.
  4. [Abstract; Sec. 4.2 Obs.2] Table 1 CF@50 accuracy drops are explained as faithfulness, which is fair; a one-sentence pointer in the abstract or intro that ambiguous streams yield ambiguous memory would set expectations.
  5. [Table 1; Figure 2; Sec. 4.3] Typos / wording: “Counter Factual” → “Counterfactual”; “Metrix” in Fig. 2 → “Matrix”; “BitCPM-CANN-3B” may be MiniCPM—verify model id.

Circularity Check

0 steps flagged

No significant circularity: online memory writes from external labels and spectral bounds are independent of the decision readouts they later support.

full rationale

Σ-Mem’s load-bearing chain is (i) external post-decision correctness labels c_{p,t} write symmetric states M_p and G via decayed rank-one updates (Eqs. 6, 9), (ii) Weyl’s inequality and Theorem 1 bound spectral change and show persistent signal can dominate noise under stated assumptions, and (iii) the same frozen states are read for residual steering, M-route, or M-vote and scored against held-out CF and OOD streams. Memory writing is explicitly separated from decision-time utilities U_{p,t} (Sec. 3.3), so selection does not define the labels that update memory. Weyl/Theorem 1 are standard perturbation and concentration arguments proved in the appendix, not tautologies of the empirical claim. Learned projection P and gain g are ordinary trainable interface parameters fit on a fixed 2,963-event mix and then evaluated on different streams; that is supervised adaptation, not a fitted quantity renamed as a prediction of itself. Related-work self-citations (e.g., epistemic-context papers) are not used as uniqueness theorems that force the mechanism. Overclaim concerns about calling label-driven test-time adaptation “OOD generalization” are framing/correctness issues, not derivation circularity. No step reduces Eq. X to Eq. Y by construction in the sense of the circularity taxonomy.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 2 invented entities

The central empirical claim rests on standard spectral math, the availability of post-hoc correctness labels, a learned task encoder into competence directions, and several hand-chosen decay/update/aggregation weights. No new physical entities; the invented objects are the memory state design itself.

free parameters (5)
  • decay γ and update η for M_p
    Control how fast competence evidence is forgotten vs written (Eq. 6). Chosen for stable online adaptation; not derived from data-free principles.
  • topology decay γ_G and update η_G for G
    Analogous free rates for the peer-relationship graph (Eq. 9).
  • utility/relationship weights w_u, w_g
    Scale competence utilities vs pairwise G terms in the latent correctness posterior (Eq. 15).
  • projection P and scalar gain g = learned
    Learned maps from memory readout r_{p,t} into residual-stream steers δ_{p,t} (Eq. 11); fit so steering helps the center model.
  • memory rank/dimension r and competence encoder φ
    Define the feature space in which competence evidence accumulates; architecture choice that shapes all readouts.
axioms (5)
  • standard math Weyl’s inequality: |λ_i(M+E)−λ_i(M)| ≤ ||E||_2 for real symmetric M,E
    Invoked in Sec. 2.1 and Appendix A.1 to bound per-event spectral change of memory states.
  • domain assumption After each decision, external binary correctness labels c_{p,t}∈{+1,−1} are observed for every peer
    Required for Eqs. 6 and 9; experiments always write from benchmark labels (Sec. 3.3, Fig. 3).
  • domain assumption Task inputs map to unit competence directions φ(x) that align similar tasks so historical evidence transfers
    Underpins readout r_p=M_p φ(x) and OOD transfer claims (Sec. 3.2, 4.4).
  • ad hoc to paper Task-conditional competence model c_{p,t}=μ_p(x_t)+ξ_t with independent bounded zero-mean noise (Assumption 1)
    Used to prove Theorem 1’s signal-vs-noise bound in Appendix A.2; not independently validated on LLM peer errors.
  • ad hoc to paper Peer relationship structure is well captured by centered outer products of correctness vectors with fixed diagonal 1
    Defines G updates (Eqs. 8–9); alternative trust topologies are not compared.
invented entities (2)
  • Per-peer competence memory matrices M_p and peer-relationship matrix G (Σ-Mem state) no independent evidence
    purpose: Persistent online store of historical competence and correlated correctness for MAS coordination.
    Core postulated representation; justified by experiments and spectral stability arguments rather than prior measurement of such matrices in LLM MAS.
  • Competence direction φ(x) as the task-conditioned write/read key no independent evidence
    purpose: Aligns events so reliability evidence generalizes across similar tasks and to OOD domains.
    Encoder-defined feature; transfer results support usefulness but φ is not an externally measured latent.

pith-pipeline@v1.2.0-daily-grok45 · 33538 in / 3857 out tokens · 77439 ms · 2026-07-31T22:12:04.487091+00:00 · methodology

0 comments
read the original abstract

Memory is central to long-horizon LLM agents, yet existing memory systems primarily preserve interaction content rather than modeling which agents can be trusted and under what conditions. This limitation is particularly important in multi-agent systems, where a central model may be unable to directly verify plausible or correlated peer responses. We introduce $\Sigma$-Mem, an online reliability memory that records historical competence evidence for individual peers and peer relationship evidence across the peer set. Both forms of evidence are maintained as real symmetric states and updated from post-decision correctness feedback. By Weyl's inequality, the spectral change caused by each event-level update is bounded, enabling stable online adaptation without retraining the underlying models. $\Sigma$-Mem provides a general write-and-read interface: the same memory can be used for residual steering of a central model, response-free peer routing, or reliability-weighted voting. Across five Qwen-family models, $\Sigma$-Mem adapts to counterfactual reliability shifts and generalizes to unseen peers and task domains. Direct memory readouts also outperform majority voting and the best fixed peer over the full OOD evaluation set. Moreover, performance improves consistently as more correctness feedback becomes available, indicating that $\Sigma$-Mem progressively accumulates actionable reliability information. These results establish reliability memory as a reusable foundation for adaptive coordination in LLM-based multi-agent systems.

Figures

Figures reproduced from arXiv: 2607.27958 by Peilin Feng, Soujanya Poria, Suorong Yang.

Figure 1
Figure 1. Figure 1: Motivation of Σ-Mem. We hide the supporting context from the RAG task so that the central model cannot directly verify the correctness of the candidate answers. It is insufficient to aggregate peers for the central model when multiple peers provide confident or correlated answers. Historical performance and peer relationship can provide complementary reliability evidence to steer the central model toward b… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of Σ-Mem. Given a task and multiple peer answers: For each peer 𝑝, Σ-Mem reads the task-conditioned historical competence state M𝑝 and converts it into a peer-specific residual steering signal, which influences the center model’s attention and produces a utility score. The utilities are then combined with the peer relationship G for final aggregation. After external correctness labels become avail… view at source ↗
Figure 3
Figure 3. Figure 3: OOD accuracy under different feedback availability ratios, with Qwen3.5-4B and Qwen3.5-9B as the central models. We vary the fraction of events whose correctness feedback is revealed after prediction, from 5% to 100%. Unlabeled events still apply temporal decay, which makes low-feedback settings strictly harder. The blue curve denotes M-based routing, the red curve denotes Σ-Mem w/ G, and the gray dashed l… view at source ↗
Figure 4
Figure 4. Figure 4: SuperGLUE dataset breakdown across six subtasks. Each horizontal bar decomposes performance into the base model accuracy and the change introduced by Σ-Mem: positive gains are shown in red, while degradations are shown in blue. The orange dashed line denotes the oracle accuracy obtained when at least one peer provides a correct answer, which is the upper bound of accuracy. As shown in [PITH_FULL_IMAGE:fig… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 25 linked inside Pith

  1. [1]

    Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras.arXiv preprint arXiv:2503.01743,

    Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras.arXiv preprint arXiv:2503.01743,

    Pith/arXiv arXiv

  2. [3]

    Nicolò Cesa-Bianchi and Gábor Lugosi.Prediction, Learning, and Games

    arXiv:2503.13657. Nicolò Cesa-Bianchi and Gábor Lugosi.Prediction, Learning, and Games. Cambridge University Press,

    Pith/arXiv arXiv

  3. [8]

    This allows historical reliability evidence from related training domains to be reused for new tasks

    B.3 Generalization Beyond Training Domains Although Σ-Mem is trained only on mathematics, RAG, and coding events, it can map an unseen task into the learned competence-direction space through𝝓(x). This allows historical reliability evidence from related training domains to be reused for new tasks. To evaluate this capability, we test the three fixed peers...

    2020

  4. [9]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Pith/arXiv arXiv

  5. [10]

    Efficiently modeling long sequences with structured state spaces.arXiv preprint arXiv:2111.00396,

    Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces.arXiv preprint arXiv:2111.00396,

    Pith/arXiv arXiv

  6. [11]

    Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300,

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300,

    Pith/arXiv arXiv 2009

  7. [13]

    Memory in the age of ai agents.arXiv preprint arXiv:2512.13564,

    Yuyang Hu, Shichun Liu, Yanwei Yue, Guibin Zhang, Boyang Liu, Fangyi Zhu, Jiahang Lin, Honglin Guo, Shihan Dou, Zhiheng Xi, et al. Memory in the age of ai agents.arXiv preprint arXiv:2512.13564,

    Pith/arXiv arXiv

  8. [14]

    Jennings, and Nigel Shadbolt

    Declare Lab · NTU · Under review 13 Trung Dong Huynh, Nicholas R. Jennings, and Nigel Shadbolt. Fire: An integrated trust and reputation model for open multi-agent systems. InEuropeanConferenceonArtificialIntelligence,2004. URL https://api.semanticscholar.org/CorpusID:10678849. Naman Jain, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armand...

    2004

  9. [15]

    Debating with more persuasive llms leads to more truthful answers.arXiv preprint arXiv:2402.06782,

    Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R Bowman, Tim Rocktäschel, and Ethan Perez. Debating with more persuasive llms leads to more truthful answers.arXiv preprint arXiv:2402.06782,

    Pith/arXiv arXiv

  10. [16]

    No free labels: Limitations of llm-as-a-judge without human grounding.arXiv preprint arXiv:2503.05061,

    Michael Krumdick, Charles Lovering, Varshini Reddy, Seth Ebner, and Chris Tanner. No free labels: Limitations of llm-as-a-judge without human grounding.arXiv preprint arXiv:2503.05061,

    arXiv

  11. [17]

    J. Lei, D. Zhang, J. Li, W. Wang, K. Fan, X. Liu, Q. Liu, X. Ma, B. Chen, and S. Poria.𝛿-mem: Efficient online memory for large language models.arXiv preprint arXiv:2605.12357,

    Pith/arXiv arXiv

  12. [18]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InInternational Conference on Learning Representations, volume 2024, pages 39578–39601,

    2024

  13. [20]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. InProceedings of the 2018 conference on empirical methods in natural language processing, pages 2381–2391,

    2018

  14. [22]

    CONSENSAGENT: Towards efficient and effective consensus in multi-agent LLM interactions through sycophancy mitigation

    Priya Pitre, Naren Ramakrishnan, and Xuan Wang. CONSENSAGENT: Towards efficient and effective consensus in multi-agent LLM interactions through sycophancy mitigation. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors,Findings of the Association for Computational Linguistics: ACL 2025, pages 22112–22133, Vienna, Austria, July

    2025

  15. [23]

    ISBN 979-8-89176-256-5

    Association for Computational Linguistics. ISBN 979-8-89176-256-5. . URLhttps://aclanthology.org/2025.find ings-acl.1141/. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. InProceedings of the 2016 conference on empirical methods in natural language processing, pages 2383–2392,

    2025

  16. [25]

    Trust me, i’m wrong: Llms hallucinate with certainty despite knowing the answer.arXiv preprint arXiv:2502.12964,

    Declare Lab · NTU · Under review 14 Adi Simhi, Itay Itzhak, Fazl Barez, Gabriel Stanovsky, and Yonatan Belinkov. Trust me, i’m wrong: Llms hallucinate with certainty despite knowing the answer.arXiv preprint arXiv:2502.12964,

    Pith/arXiv arXiv

  17. [26]

    Vighnesh Subramaniam, Yilun Du, Joshua B Tenenbaum, Antonio Torralba, Shuang Li, and Igor Mordatch

    URLhttps://arxiv.org/abs/2508.18321. Vighnesh Subramaniam, Yilun Du, Joshua B Tenenbaum, Antonio Torralba, Shuang Li, and Igor Mordatch. Multiagent finetuning: Self improvement with diverse reasoning chains. InInternational Conference on Learning Representations, volume 2025, pages 10840–10862,

    arXiv 2025

  18. [27]

    Challenging big-bench tasks and whether chain-of-thought can solve them

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. InFindings of the Association for Computational Linguistics: ACL 2023, pages 13003–13051,

    2023

  19. [28]

    Mathscale: Scaling instruction tuning for mathematical reasoning

    Zhengyang Tang, Xingxing Zhang, Benyou Wang, and Furu Wei. Mathscale: Scaling instruction tuning for mathematical reasoning. arXiv preprint arXiv:2403.02884,

    Pith/arXiv arXiv

  20. [30]

    Minicpm4: Ultra-efficient llms on end devices.arXiv preprint arXiv:2506.07900, 2025b

    MiniCPM Team, Chaojun Xiao, Yuxuan Li, Xu Han, Yuzhuo Bai, Jie Cai, Haotian Chen, Wentong Chen, Xin Cong, Ganqu Cui, et al. Minicpm4: Ultra-efficient llms on end devices.arXiv preprint arXiv:2506.07900, 2025b. Qwen Team. Qwen3. 5-omni technical report.arXiv preprint arXiv:2604.15804,

    Pith/arXiv arXiv

  21. [32]

    Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171,

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171,

    Pith/arXiv arXiv

  22. [34]

    Remember when it matters: Proactive memory agent for long-horizon agents.arXiv preprint arXiv:2607.08716, 2026b

    Yifan Wu, Lizhu Zhang, Yuhang Zhou, Mingyi Wang, Bo Peng, Serena Li, Xiangjun Fan, and Zhuokai Zhao. Remember when it matters: Proactive memory agent for long-horizon agents.arXiv preprint arXiv:2607.08716, 2026b. Andrea Wynn, Harsh Satija, and Gillian Hadfield. Talk isn’t always cheap: Understanding failure modes in multi-agent debate.arXiv preprint arXi...

    Pith/arXiv arXiv

  23. [35]

    Qwen2.5-mathtechnicalreport: Towardmathematicalexpertmodelviaself-improvement.arXivpreprintarXiv:2409.12122,

    AnYang,BeichenZhang,BinyuanHui,BofeiGao,BowenYu,ChengpengLi,DayihengLiu,JianhongTu,JingrenZhou,JunyangLin, etal. Qwen2.5-mathtechnicalreport: Towardmathematicalexpertmodelviaself-improvement.arXivpreprintarXiv:2409.12122,

    Pith/arXiv arXiv

  24. [36]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  25. [37]

    Hotpotqa: A dataset for diverse, explainable multi-hop question answering

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. InProceedings of the 2018 conference on empirical methods in natural language processing, pages 2369–2380,

    2018

  26. [38]

    Swarmagentic: Towards fully automated agentic system generation via swarm intelligence

    Yao Zhang, Chenyang Lin, Shijie Tang, Haokun Chen, Shijie Zhou, Yunpu Ma, and Volker Tresp. Swarmagentic: Towards fully automated agentic system generation via swarm intelligence. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 1778–1818, 2025a. Zeyu Zhang, Quanyu Dai, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Ji...

    2025

  27. [39]

    Zhou et al

    R. Zhou et al. Epistemic context learning: Building trust in LLM-based multi-agent systems.arXiv preprint arXiv:2601.21742, 2026a. Ruiwen Zhou, Maojia Song, Xiaobao Wu, Sitao Cheng, Xunjian Yin, Yuxi Xie, Zhuoqun Hao, Wenyue Hua, Liangming Pan, Soujanya Poria, et al. Epistemic context learning: Building trust the right way in llm-based multi-agent systems...

    arXiv

  28. [40]

    LetM,E∈R 𝑟×𝑟 be real symmetric matrices, and order the eigenvalues of any real symmetric matrixAas 𝜆1(A)≥𝜆 2(A)≥···≥𝜆 𝑟(A). According to the Courant–Fischer variational principle (Bhatia, 1997), the𝑖-th eigenvalue ofAsatisfies 𝜆𝑖(A)=min S⊆R 𝑟 dim(S)=𝑟−𝑖+1 max v∈S ∥v∥ 2=1 v⊤Av.(21) For every unit vectorv, the definition of the spectral norm gives v⊤Ev ≤∥v∥...

    1997

  29. [41]

    training split. For each task, we independently generate one Declare Lab · NTU · Under review 20 response from each of three fixed peer models: Gemma-3-4B-it (Team et al., 2025a), Phi-4-mini-instruct (Abouelenin et al., 2025), and Qwen2.5-Coder-7B-Instruct (Yang et al., 2025). Each peer response is then evaluated to obtain a binary correctness label used ...

    2025

  30. [42]

    96325.86%8.10% 17.65% Overall 2,963 40.06%51.77%46.88% B.2 Counterfactual Attack Dataset We construct a counterfactual attack dataset from nine benchmarks covering mathematics, retrieval-augmented question answering (RAG), and coding. The mathematics subset is drawn from MATH-500 (Hendrycks et al., 2021; Lightman et al., 2024), AMC23 (Yang et al., 2024), ...

    2021

  31. [43]

    Each counterfactual split contains the same 2,685 tasks, including 1,216 mathematics tasks, 1,075 RAG tasks, and 394 coding tasks

    and TriviaQA (Joshi et al., 2017); and the coding subset is drawn from HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), and LiveCodeBench (Jain et al., 2025). Each counterfactual split contains the same 2,685 tasks, including 1,216 mathematics tasks, 1,075 RAG tasks, and 394 coding tasks. For each constituent dataset, we first measure the clean ...

    2017

  32. [45]

    All splits contain the same 2,685 task events

    Declare Lab · NTU · Under review 21 Table 7:Peer accuracy under different counterfactual attack ratios in the four-peer setting. All splits contain the same 2,685 task events. The best-performing peer in each split is highlighted inbold. CF Ratio N Gemma-3-4B-it Phi-4-mini Qwen2.5-Coder Llama-3.2t CF@0 2,68555.68%46.93% 48.42% 38.70% CF@50 2,685 40.45% 37...

    2017

  33. [46]

    The results reflect two factors: the judgment ability of the central model and the validity of the historical signal in the stream. For the Qwen3 center models, whose response-level judgment is weaker, M-route matches or exceedsΣ w/G in Declare Lab · NTU · Under review 22 Table 10:Counter Factual (CF) Attack results under different CF@ratios.Acc denotes t...

    1993

  34. [1912]

    Scaling teams or scaling time? memory enabled lifelong learning in llm multi-agent systems.arXiv preprint arXiv:2604.03295, 2026a

    Shanglin Wu, Yuyang Luo, Yueqing Liang, Kaiwen Shi, Yanfang Ye, Ali Payani, and Kai Shu. Scaling teams or scaling time? memory enabled lifelong learning in llm multi-agent systems.arXiv preprint arXiv:2604.03295, 2026a. Yaxiong Wu, Sheng Liang, Chen Zhang, Yichao Wang, Yongyue Zhang, Huifeng Guo, Ruiming Tang, and Yong Liu. From human memory to ai memory:...

    Pith/arXiv arXiv

  35. [1979]

    Risk analysis techniques for governed LLM-based multi-agent systems.arXiv preprint arXiv:2508.05687,

    Gradient Institute. Risk analysis techniques for governed LLM-based multi-agent systems.arXiv preprint arXiv:2508.05687,

    Pith/arXiv arXiv

  36. [2001]

    Towards understanding sycophancy in language models

    Mrinank Sharma, Meg Tong, Tomek Korbak, David Duvenaud, Amanda Askell, Sam Bowman, Esin Durmus, Zac Hatfield-Dodds, Scott Johnston, Shauna Kravec, et al. Towards understanding sycophancy in language models. InInternational Conference on Learning Representations, volume 2024, pages 110–144,

    2024

  37. [2006]

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Bey...

    Pith/arXiv arXiv

  38. [2018]

    Llamar: Long-horizon planning for multi-agent robots in partially observable environments.arXiv preprint arXiv:2407.10031,

    Siddharth Nayak, Adelmo Morrison Orozco, Marina Ten Have, Vittal Thirumalai, Jackson Zhang, Darren Chen, Aditya Kapoor, Eric Robinson, Karthik Gopalakrishnan, James Harrison, et al. Llamar: Long-horizon planning for multi-agent robots in partially observable environments.arXiv preprint arXiv:2407.10031,

    Pith/arXiv arXiv

  39. [2019]

    Mixture-of-agents enhances large language model capabilities

    Declare Lab · NTU · Under review 15 Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Y Zou. Mixture-of-agents enhances large language model capabilities. InInternational Conference on Learning Representations, volume 2025, pages 33944–33963,

    2025

  40. [2020]

    Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    DanHendrycks, CollinBurns, SauravKadavath, AkulArora, StevenBasart, EricTang, DawnSong, andJacobSteinhardt. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    Pith/arXiv arXiv

  41. [2021]

    Mem0: Building production-ready ai agents with scalable long-term memory.arXiv preprint arXiv:2504.19413,

    Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. Mem0: Building production-ready ai agents with scalable long-term memory.arXiv preprint arXiv:2504.19413,

    Pith/arXiv arXiv

  42. [2022]

    Routing to the expert: Efficient reward-guided ensemble of large language models

    Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. Routing to the expert: Efficient reward-guided ensemble of large language models. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1964–1974,

    2024

  43. [2023]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374,

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374,

    Pith/arXiv arXiv

  44. [2024]

    Frugalgpt: How to use large language models while reducing cost and improving performance.arXiv preprint arXiv:2305.05176,

    Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance.arXiv preprint arXiv:2305.05176,

    Pith/arXiv arXiv

  45. [2025]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732,

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models.arXiv preprint arXiv:2108.07732,

    Pith/arXiv arXiv

  46. [2026]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Pith/arXiv arXiv