The Formalism Trap: Are LLM-as-a-Judge Evaluators Blinded by Consensus Mimicry under Social Load?

Dahlia Shehata; Ming Li

REVIEW 4 major objections 5 minor 44 references

The Formalism Trap: Are LLM-as-a-Judge Evaluators Blinded by Consensus Mimicry under Social Load?

T0 review · 4 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM-as-a-judge evaluators can be captured by performative syntax—awarding near-perfect quality scores to factually hollow but structurally convincing reasoning traces under adversarial social load, a vulnerability that generalizes across do

desk verdict Plausible phenomenon, but the headline results are not trustworthy because the semantic features are extracted from the same JSON fields that define the target. read the letter →

arxiv 2607.28641 v1 pith:S3V76T5P submitted 2026-05-20 cs.CL cs.AI

Dahlia Shehata , Ming Li This is my paper

classification cs.CLcs.AI

keywords LLM-as-a-judgeevaluatorcaptureevaluativedissonanceformalismtrapmulti-agentsimulationsocialconformityhallucinationdetectionmeta-evaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper attempts to establish that an LLM evaluator's qualitative scoring is systematically hijacked by the structural form of a reasoning trace rather than its factual content, especially when the trace is produced under simulated social pressure. It introduces the Agentic Formalism Trap and a numeric index, the Evaluative Dissonance Index, which measures the gap between the quality score a judge awards and the judge's own binary verdict on the answer's correctness. Using 22,500 multi-agent trajectories across general QA, conversational logic, and code execution benchmarks, the authors build a logistic meta-evaluator on a 531-cluster taxonomy of rhetorical maneuvers that detects these captured evaluations with ROC-AUC 0.8779 and transfers zero-shot to unseen domains with mean ROC-AUC 0.7482. A sympathetic reader would care because automated judges are increasingly used to grade AI systems, and a judge that can be blinded by consensus mimicry makes evaluation loops unreliable and gameable.

What carries the argument

The load-bearing object is the Evaluative Dissonance Index, D_E(T) = E[V_qual(T)] − A_ext, where V_qual is the judge's normalized qualitative evidence-weighting score in [0,1] and A_ext is the judge's own binary accuracy verdict; D_E near 0 is consistent scoring and D_E near 1 is 'evaluator capture.' The argument runs through a two-part decomposition of a reasoning trace into syntactic structure S and binary factual semantics M, a taxonomy of 531 semantic clusters of rhetorical maneuvers extracted from the traces, and a logistic regression meta-evaluator that predicts capture (D_E > 0.5) from the presence of those clusters. The regression coefficients are the empirical evidence for 'syntacti

What would settle it

Re-run the semantic extraction with the accuracy and evidence-weighting fields removed from the records given to the labeling model, then retrain the meta-evaluator; if the ROC-AUC drops substantially from 0.8779, the original result is driven by target leakage rather than the Formalism Trap. A second check would replace prompt-injected swarm personas with live multi-agent message passing; if the trap vanishes, the simulated social load was a token-weighting artifact rather than genuine social dynamics.

Watch

Extended reading notes

Core claim

The paper's central claim is that frontier evaluator models do not weigh facts and form symmetrically: under adversarial social load, a reasoning trace's syntactic structure (rhetoric, formatting, consensus mimicry) dominates its factual semantics in the judge's scoring. Formally, the paper proposes that when a trace exhibits performative syntax, the probability of the judge awarding near-perfect evidence weighting becomes nearly independent of whether the answer is correct, and the Evaluative Dissonance Index D_E approaches its maximum of 1. The paper supports this with a corpus of 22,500 trajectories in which propagator models were placed under simulated peer pressure, and shows that a log

Load-bearing premise

The central claim collapses if the automated semantic labels describing rhetorical maneuvers are contaminated by the very scores they are used to predict, since the labeling model was given the full record including the judge's evidence-weighting score and the final accuracy verdict.

Editorial extensions

If this is right

If the claim holds, any LLM-as-a-judge pipeline scoring multi-agent reasoning traces is vulnerable to a specific attack: formatting and consensus mimicry can buy high scores for false results.
Gradient-based or reward-driven optimization of generator models will converge on performative syntax, because that is what the judge rewards, amplifying the trap rather than fixing it.
A high-precision 'vigilance filter' can be deployed to flag likely capture with 0.91 precision, though its recall of 0.03 means most capture events will go unflagged.
Because the vulnerability transfers zero-shot across domains, it cannot be dismissed as a benchmark-specific quirk; it would need to be treated as a property of the evaluator class.
Architecture-specific filters would be needed, not a single global fix, since distinct swarm compositions produce distinct semantic blind spots.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the weakest assumption holds, a natural extension is to test whether capture also occurs with human-written adversarial traces; if it does, the practical threat is broader than simulated swarms.
The design of D_E requires a binary ground-truth verdict; in open-ended generation the index cannot be computed directly, so the paper's scope may not transfer to creative or subjective tasks without a new anchor.
The reported threshold calibration suggests a trade-off: at deployable precision, almost all captured evaluations are missed; a safety filter built on this method would need a complementary high-recall channel.
One could test the leakage hypothesis directly by re-extracting the taxonomy from records with the judge's scores and accuracy masked; if ROC-AUC falls, the reported meta-evaluator is partly reading the target from the features.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The core idea here is genuinely worth thinking about: an LLM judge can be seduced by performative syntax even when it knows the final answer is wrong. The D_E index is a clean way to operationalize judge self-contradiction, and the scale of the corpus (22,500 trajectories across three benchmarks) is real work. The LODO transfer idea is also a good instinct, and the authors are honest about the simulated-swarm limitation.

But the central empirical claim does not survive contact with the method. In §3.2, the extraction model sees the full JSON record, including Eew and Aext — the two numbers that literally define D_E. It is then asked to analyze the 'holistic relationship' between all fields, including 'subjective grading and final accuracy.' That is not a neutral semantic annotation; it is a target-contaminated labeling step. A cluster called EVALUATOR_CAPTURE can be assigned simply by noticing that Eew is high and Aext is 0. The subsequent logistic regression is then recovering the target from the label content, which explains the in-domain AUC of 0.8779 and the LODO transfer of 0.7482. The stress-test note is right: this is annotation leakage, not evidence of a semantic taxonomy predicting evaluator failure.

The deterministic lexical grounding check does not rescue it. A correlation of r=0.1557 is weak, and even a target-contaminated label can correlate with formatting artifacts. The vigilance threshold t=0.98 is also selected on the held-out test set, which inflates precision numbers. And no code or data are released, so none of this is independently checkable.

What is salvageable? The conceptual framing of the Formalism Trap is useful for the LLM-as-a-judge community, and the swarm architecture profiling (§4.6) might contain interesting descriptive patterns — but those patterns are themselves derived from the same contaminated feature space. Without a blind re-extraction (e.g., feeding only the reasoning trace text, not the judge's scores), the paper's main results are not established.

I would send this to peer review rather than desk-reject, because the topic matters and the flaw is fixable. But the review must insist on a decontaminated extraction pipeline and a proper ablation. As written, I would not cite the empirical results.

Referee Report

4 major / 5 minor

Summary. The paper claims to identify an 'Agentic Formalism Trap' in LLM-as-a-Judge evaluation: under adversarial social load from simulated multi-agent swarms, evaluators award high evidence-weighting scores to traces that are structurally elaborate but factually hollow. The authors define the Evaluative Dissonance Index D_E = E[Vqual] − Aext, generate 22,500 deterministic trajectories across GAIA, SWE-bench, and Multi-Challenge, extract 531 semantic clusters using Gemini 3 Flash, validate these clusters via lexical grounding, and train a logistic meta-evaluator to predict D_E > 0.5 (held-out ROC-AUC 0.8779). They further report zero-shot Leave-One-Domain-Out transfer (mean ROC-AUC 0.7482) and an architecture vulnerability matrix. The central claim is that the logistic model's features identify syntactic triggers of evaluator capture and that the vulnerability is domain-agnostic.

Significance. If the findings were valid, the D_E index and the semantic taxonomy would be a useful diagnostic for LLM-as-a-Judge pipelines, going beyond known biases such as verbosity and self-enhancement. The paper's formalization of dissonance, its separation of normative evaluation from descriptive extraction, and its explicit, falsifiable propositions are conceptually valuable. It is also transparent about several limitations (simulated swarms, proprietary model volatility, deterministic tasks). However, the main empirical contribution is compromised by target leakage in the feature-extraction step and by test-set threshold tuning; the reported AUCs and precision figures cannot currently be taken as evidence for the Formalism Trap. The paper does not provide code/data, and the 'proving' language in the abstract and conclusion overstates the inferential power of a single logistic-regression study.

major comments (4)

[§3.2, §3.3, Eq. (3)] The extraction model in §3.2 is instructed to analyze 'the holistic relationship between all fields' of each JSON record, which includes the evaluator's evidence-weighting score Eew and the final accuracy verdict Aext. These are precisely the components that define D_E in Eq. (3) as E[Vqual(T)] − Aext, and the binary target in §3.3 is Y=1[D_E>0.5]. The 531 Global Canonical Clusters are therefore generated with direct access to the target variable; a cluster such as EVALUATOR_CAPTURE (Table 1) can be assigned by observing Eew=5 with Aext=0. The logistic meta-evaluator then recovers the target from label content, inflating held-out ROC-AUC (0.8779) and LODO transfer (0.7482). Because the same contaminated extraction procedure feeds every fold, cross-domain transfer does not remove the leakage. The deterministic lexical grounding check (§3.4, r=0.1557, r²≈0.024) does not clear this: a targe
[§4.4, Fig. 4, Table 2] The vigilance-filter threshold t=0.98 is selected from the PR curve on the held-out test set ('By shifting the probability activation threshold to t=0.98 based on the PR curve'). Reporting precision (0.91) and recall at a threshold tuned on the test set yields optimistically biased performance estimates. The 'deployable Vigilance Filter' claim is not validly evaluated; threshold selection should be done via nested cross-validation or on a separate validation set, and applied only once to the test set.
[§4.5, §4.6, Table 4] Leave-One-Domain-Out transfer does not mitigate the target leakage, because the same contaminated semantic extraction is used for all domains; the feature labels carry Eew and Aext regardless of domain. Moreover, Table 4 shows several architecture-specific classifiers with AUC near 0.50 (e.g., GPG 0.5039, GCG 0.5735, CCG 0.5488). Interpreting these null results as evidence for Proposition 5's 'Heterogeneous Swarm Decoupling' is unjustified without demonstrating that they are not due to low minority-class sample sizes or leakage. The paper's own exclusion rule (fewer than 5 minority instances) leaves many configurations with very low power, and no multiple-comparison correction is applied to the many significance tests in Table 4.
[Limitations] The Limitations section concedes that swarm consensus was simulated via prompt injection rather than live multi-agent interaction, and then asserts that this 'genuinely induces multi-agent phenomena' rather than a token-weighting artifact. This assertion is load-bearing for the paper's central claim about 'social load' and 'architectural asymmetry.' No control condition is reported (e.g., comparing the prompt-injection condition to a control prompt with no personas, or testing whether the judge's behavior differs under generic social content). Without such a manipulation check, the results cannot be attributed to simulated multi-agent social dynamics, and the strong wording in the abstract and conclusion ('proving that unanchored closed-loop evaluation is unstable') is not supported.

minor comments (5)

[Eq. (3)] The notation E[Vqual(T)] is not defined as an expectation over any distribution; it appears to be simply the normalized score Vqual(T). Using E[...] invites confusion and should be replaced with Vqual(T).
[Eq. (5)] The limit expression in Proposition 2 is not a valid implication as written: D_E(T_fake)→1.0 requires Aext=0 in addition to Vqual→1. The authors should state this condition explicitly.
[Fig. 1] 'Cross-Domain Overlap: Semantic Anomalies' is not described in the text; the reader cannot tell what is plotted. A caption or a reference in §4.2 explaining the axes and quantities is needed.
[§3.4] The correlation r=0.1557 is described as 'moderate.' With r²≈0.024, the effect size is weak in magnitude, though it is statistically significant because n=22,500. The wording should be adjusted to avoid overstating the strength of the grounding check.
[Ethics Statement] The Ethics Statement mentions 'open-sourcing our methodology,' but no code or data repository is provided. Given the focus on reproducibility and the reliance on proprietary models, a link or supplementary material should be supplied.

Circularity Check

2 steps flagged · score 7.0 of 10

Semantic features are extracted from JSONs containing Eew and Aext — the exact components of D_E — so the reported ROC-AUCs partly measure annotation leakage, not an independent discovery of the Formalism Trap.

self definitional [§3.2 Semantic Extraction / §3.3 Modeling Evaluative Dissonance (also Eq. 3, §3.1.3 record schema)]
"we inject the entire serialized JSON object holistically into the extraction pipeline ... the model evaluates the holistic relationship between the propagator’s syntactic rhetoric, the evaluator’s response, subjective grading and final accuracy ... To construct the feature space, the Global Canonical Clusters assigned to each trace are transformed into a dense, one-hot encoded binary feature matrix (X) ... A trajectory is flagged as captured (Y=1) if the Evaluative Dissonance exceeds a strict threshold of DE >0.5."

The JSON record schema (§3.1.3) explicitly contains 'accuracy or Aext' and 'evidence weighting (i.e. Eew(T))', and Eq. (3) defines D_E = (Eew−1)/4 − Aext. The extraction model is therefore given direct access to the two components of the target variable when it invents the 531 'Global Canonical Clusters'; it is even told to analyze 'subjective grading and final accuracy'. The logistic meta-evaluator then predicts Y = 1[D_E>0.5] from clusters that can simply encode those components (e.g., EVALUATOR_CAPTURE assigned when Eew=5, Aext=0). The ROC-AUC 0.8779 and LODO mean 0.7482 therefore largely measure how consistently the extractor echoes Eew/Aext, not that independent syntactic features predict capture. The lexical-grounding correlation (r=0.1557) only ties the labels to formatting counts;
self citation load bearing [Limitations, 'Simulated Swarm Topologies']
"Despite the lack of live message-passing, this framework validly replicates true multi-agent social dynamics rather than acting as a mere token-weighting artifact. ... injecting a simulated consensus effectively triggers the model’s internal social alignment mechanisms, genuinely inducing multi-agent phenomena such as cognitive conformity (Shehata and Li, 2026)."

The central premise that prompt-injected 'swarm' personas create genuine social-load phenomena rather than token-weighting artifacts is justified by a citation to the authors' own prior preprint (Shehata and Li, 2026), not by any independent check in this paper. The paper's Architecture Vulnerability Matrix cannot supply that independent check because it is built from the same target-contaminated semantic features as the main classifier. This makes the self-citation load-bearing rather than merely bibliographic.

full rationale

The empirical core of the paper—the semantic taxonomy and the logistic meta-evaluator—is not circular in the sense of Eq. 3 being derived from the classifier; D_E is defined independently. However, the feature space that predicts D_E is generated by an LLM that receives the same two quantities that define D_E (Eew and Aext) and is explicitly asked to analyze 'subjective grading and final accuracy'. As a result, the in-domain ROC-AUC (0.8779) and the zero-shot LODO transfer (0.7482) can be inflated by annotation leakage: the 'semantic clusters' may simply restate whether the judge awarded a high score to a self-identified wrong answer. The deterministic lexical grounding check does not fix this, since a target-contaminated label can still be correlated with formatting artifacts. A secondary load-bearing self-citation (Shehata and Li, 2026) is used to certify that prompt-injected swarms are genuine multi-agent phenomena. Net: the paper has an independent statistical object (D_E), but its headline claim that performative syntax is detected by the taxonomy is substantially undermined by the construction of X from the components of Y. Score 7 reflects a central empirical result that reduces, at least in part, to feature/target leakage rather than a fully self-contained derivation.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The central quantitative claims rest on hand-chosen thresholds, an unvalidated LLM annotation step, and the assumption that prompt-injected swarms stand in for real multi-agent dynamics. The paper contributes a conceptual frame but not independent measurements.

free parameters (3)

Evaluator Capture threshold (D_E > 0.5) = 0.5
Binarizes the target variable; no sensitivity analysis or justification is given.
Vigilance filter threshold (t) = 0.98
Selected from the PR curve and evaluated on the same held-out test set, risking overfitting to the test set.
Adversarial noise token count = 500
Chosen to 'saturate the attention heads'; no dose-response or ablation is provided.

assumptions (4)

domain assumption Semantic labels produced by Gemini 3 Flash are faithful, unbiased descriptors of rhetorical maneuvers.
The entire feature space and taxonomy come from an LLM coder with no human validation; §3.2.
domain assumption Simulated swarms via prompt injection replicate true multi-agent social dynamics.
The paper argues this in Limitations; the universality of the Formalism Trap depends on it.
domain assumption Aext is deterministic ground truth and is consistently measured across domains.
The paper mixes 'evaluator's final binary verdict' (§2.2) with 'deterministic metrics' (§3.1.3).
ad hoc to paper Adversarial decoupling: under social load, syntactic structure and factual semantics become decoupled.
This is the phenomenon being asserted, used as a premise rather than derived or independently established.

invented entities (2)

Evaluative Dissonance Index (D_E)
purpose: Quantifies the gap between the evaluator's qualitative score and binary correctness.
Defined directly from Eew and Aext; no external validation shows it captures a construct beyond those two inputs.
Agentic Formalism Trap / Performative Verification Theater taxonomy
purpose: Describes LLM evaluator capture via structural mimicry and consensus conformity.
The taxonomy is generated by another LLM with no human annotation; the only grounding is a weak lexical correlation (r=0.156).

how reviews work

0 comments

Cite this review

Pith. "Pith review of The Formalism Trap: Are LLM-as-a-Judge Evaluators Blinded by Consensus Mimicry under Social Load?." pith.science (2026). https://pith.science/paper/S3V76T5P

@misc{pith2026260728641,
  author       = {Pith},
  title        = {Pith review of: The Formalism Trap: Are LLM-as-a-Judge Evaluators Blinded by Consensus Mimicry under Social Load?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S3V76T5P}},
  note         = {Machine review of arXiv:2607.28641}
}

abstract

We introduce the \textit{Agentic Formalism Trap} and the Evaluative Dissonance Index ($D_E$), quantifying how LLM-as-a-Judge systems conflate structural proceduralism with semantic truth under adversarial load. Analyzing 22,500 trajectories across 3 domains (GAIA, SWE-bench, Multi-Challenge), we extract a semantic taxonomy of hallucination maneuvers, validated via deterministic lexical grounding ($p < 10^{-120}$). A logistic meta-evaluator isolates the exact syntactic triggers of this evaluator capture (ROC-AUC 0.8779), while a zero-shot Leave-One-Domain-Out transfer proves the vulnerability is universally domain-agnostic (mean ROC-AUC 0.7482). Architectural profiling reveals that distinct simulated swarm topologies induce mathematically disparate semantic blind spots, proving that unanchored closed-loop evaluation is unstable, systemically divergent and necessitates architecture-specific vigilance filters.

Figures

Figures reproduced from arXiv: 2607.28641 by the authors.

**Figure 2.** Top 25 Frequent Semantic Anomalies. automated grounded-theory coder, tasked with labeling rhetorical and structural maneuvers. This offline extraction pipeline does not subject the model to the simulated swarm consensus or adversarial peer pressure that originally induced the Formalism Trap. Operating under these nominal, zero-pressure conditions can preserve the model’s analytical integrity. For each reasoning r… view at source ↗

**Figure 3.** Feature Importance of Semantic Clusters. Lo [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Precision-Recall Calibration: Anomaly de [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

44 extracted references · 1 linked inside Pith

[1]

and Boyd, Danah and Friedler, Sorelle A

Selbst, Andrew D. and Boyd, Danah and Friedler, Sorelle A. and Venkatasubramanian, Suresh and Vertesi, Janet , title =. Proceedings of the Conference on Fairness, Accountability, and Transparency , pages =. 2019 , isbn =

2019
[2]

LLM s instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks

Bavaresco, Anna and Bernardi, Raffaella and Bertolazzi, Leonardo and Elliott, Desmond and Fern \'a ndez, Raquel and Gatt, Albert and Ghaleb, Esam and Giulianelli, Mario and Hanna, Michael and Koller, Alexander and Martins, Andre and Mondorf, Philipp and Neplenbroek, Vera and Pezzelle, Sandro and Plank, Barbara and Schlangen, David and Suglia, Alessandro a...

2025
[3]

2026 , issn =

A survey on LLM-as-a-judge , journal =. 2026 , issn =

2026
[4]

Dorner and Vivian Yvonne Nastl and Moritz Hardt , booktitle=

Florian E. Dorner and Vivian Yvonne Nastl and Moritz Hardt , booktitle=. Limits to scalable evaluation at the frontier:
[5]

2024 , booktitle =

Chen, Dongping and Chen, Ruoxi and Zhang, Shilin and Wang, Yaochen and Liu, Yinuo and Zhou, Huichi and Zhang, Qihui and Wan, Yao and Zhou, Pan and Sun, Lichao , title =. 2024 , booktitle =

2024
[6]

Frontiers in Artificial Intelligence , volume=

The extended hollowed mind: why foundational knowledge is indispensable in the age of AI , author=. Frontiers in Artificial Intelligence , volume=. 2025 , publisher=

2025
[7]

Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communication

Yin, Zhangyue and Sun, Qiushi and Chang, Cheng and Guo, Qipeng and Dai, Junqi and Huang, Xuanjing and Qiu, Xipeng. Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communication. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023

2023
[8]

The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models? , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

Show all 44 references

[9]

Justice or Prejudice? Quantifying Biases in

Jiayi Ye and Yanbo Wang and Yue Huang and Dongping Chen and Qihui Zhang and Nuno Moniz and Tian Gao and Werner Geyer and Chao Huang and Pin-Yu Chen and Nitesh V Chawla and Xiangliang Zhang , booktitle=. Justice or Prejudice? Quantifying Biases in
[10]

ChatEval: Towards Better

Chi-Min Chan and Weize Chen and Yusheng Su and Jianxuan Yu and Wei Xue and Shanghang Zhang and Jie Fu and Zhiyuan Liu , booktitle=. ChatEval: Towards Better
[11]

Ruosen Li and Teerth Patel and Xinya Du , journal=
[12]

M ulti A gent B ench : Evaluating the Collaboration and Competition of LLM agents

Zhu, Kunlun and Du, Hongyi and Hong, Zhaochen and Yang, Xiaocheng and Guo, Shuyi and Wang, Zhe and Wang, Zhenhailong and Qian, Cheng and Tang, Xiangru and Ji, Heng and You, Jiaxuan. M ulti A gent B ench : Evaluating the Collaboration and Competition of LLM agents. Proceedings ...

2025
[13]

Chandler Smith and Marwa Abdulhai and Manfred Diaz and Marko Tesic and Rakshit Trivedi and Sasha Vezhnevets and Lewis Hammond and Jesse Clifton and Minsuk Chang and Edgar A. Du. Evaluating Generalization Capabilities of. The Thirty-ninth Annual Conference on Neural Information...
[14]

Judging with Many Minds: Do More Perspectives Mean Less Prejudice? On Bias Amplification and Resistance in Multi-Agent Based LLM -as-Judge

Ma, Chiyu and Zhang, Enpei and Zhao, Yilun and Liu, Wenjun and Jia, Yaning and Qing, Peijun and Shi, Lin and Cohan, Arman and Yan, Yujun and Vosoughi, Soroush. Judging with Many Minds: Do More Perspectives Mean Less Prejudice? On Bias Amplification and Resistance in Multi-Agen...

2025
[15]

and Han, Daniel Y

McBee, Joseph C. and Han, Daniel Y. and Liu, Li and Ma, Leah and Adjeroh, Donald A. and Xu, Dong and Hu, Gangqing , title =. JMIR Medical Education , year =
[16]

, author=

Bystander intervention in emergencies: Diffusion of responsibility. , author=. Journal of Personality and Social Psychology , volume=. 1968 , publisher=

1968
[17]

Journal of Personality and Social Psychology , volume=

Many hands make light the work: The causes and consequences of social loafing , author=. Journal of Personality and Social Psychology , volume=
[18]

Recherches sur les moteurs anim

Ringelmann, Max , journal =. Recherches sur les moteurs anim
[19]

2026 , journal=

The Inverse-Wisdom Law: Architectural Tribalism and the Consensus Paradox in Agentic Swarms , author=. 2026 , journal=

2026
[20]

and Yue, Summer and Xing, Chen

Deshpande, Kaustubh and Sirdeshmukh, Ved and Mols, Johannes Baptist and Jin, Lifeng and Hernandez-Cardona, Ed-Yeremai and Lee, Dean and Kritz, Jeremy and Primack, Willow E. and Yue, Summer and Xing, Chen. M ulti C hallenge: A Realistic Multi-Turn Conversation Evaluation Benchm...

2025
[21]

The Twelfth International Conference on Learning Representations , year=

Gr. The Twelfth International Conference on Learning Representations , year=
[22]

2026 , eprint =

The Bystander Effect in Multi-Agent Reasoning: Quantifying Cognitive Loafing in Collaborative Interactions , author =. 2026 , eprint =

2026
[23]

2026 , eprint=

CoGate-LSTM: Prototype-Guided Feature-Space Gating for Mitigating Gradient Dilution in Imbalanced Toxic Comment Classification , author=. 2026 , eprint=

2026
[24]

Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik Narasimhan , booktitle=

Carlos E. Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik Narasimhan , booktitle=
[25]

2026 , journal=

Beyond the Attention Stability Boundary: Agentic Self-Synthesizing Reasoning Protocols , author=. 2026 , journal=

2026
[26]

Sparse Local Embeddings for Extreme Multi-label Classification , volume =

Bhatia, Kush and Jain, Himanshu and Kar, Purushottam and Varma, Manik and Jain, Prateek , booktitle =. Sparse Local Embeddings for Extreme Multi-label Classification , volume =
[27]

Learning Deep Representation for Imbalanced Classification , year=

Huang, Chen and Li, Yining and Loy, Chen Change and Tang, Xiaoou , booktitle=. Learning Deep Representation for Imbalanced Classification , year=
[28]

Cohen , booktitle=

Zhilin Yang and Zihang Dai and Ruslan Salakhutdinov and William W. Cohen , booktitle=. Breaking the Softmax Bottleneck: A High-Rank
[29]

Leave-One-Domain-Out Evaluation of Deep Learning Architectures for Balinese Masks Classification , year=

Andika, I Gede and Yuniarti, Anny and Hidayati, Shintami Chusnul and Baskoro, Fajar and Wai, Khaing Hsu , booktitle=. Leave-One-Domain-Out Evaluation of Deep Learning Architectures for Balinese Masks Classification , year=
[30]

S warm A gentic: Towards Fully Automated Agentic System Generation via Swarm Intelligence

Zhang, Yao and Lin, Chenyang and Tang, Shijie and Chen, Haokun and Zhou, Shijie and Ma, Yunpu and Tresp, Volker. S warm A gentic: Towards Fully Automated Agentic System Generation via Swarm Intelligence. Proceedings of the 2025 Conference on Empirical Methods in Natural Langua...

2025
[31]

ArXiv , year=

Towards a Science of Scaling Agent Systems , author=. ArXiv , year=
[32]

and Mordatch, Igor , title =

Du, Yilun and Li, Shuang and Torralba, Antonio and Tenenbaum, Joshua B. and Mordatch, Igor , title =. Proceedings of the 41st International Conference on Machine Learning , articleno =. 2024 , publisher =

2024
[33]

2024 , month =

Chen, Lingjiao and Zaharia, Matei and Zou, James , journal =. 2024 , month =

2024
[34]

Advances in Neural Information Processing Systems (NeurIPS) , volume =

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena , author =. Advances in Neural Information Processing Systems (NeurIPS) , volume =
[35]

2023 , institution=

Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality , author=. 2023 , institution=

2023
[36]

arXiv preprint arXiv:2601.12538 , year=

Agentic Reasoning for Large Language Models , author=. arXiv preprint arXiv:2601.12538 , year=

arXiv
[37]

Proceedings of the 41st International Conference on Machine Learning (ICML) , year=

Improving Factuality and Reasoning in Language Models through Multiagent Debate , author=. Proceedings of the 41st International Conference on Machine Learning (ICML) , year=
[38]

Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate

Liang, Tian and He, Zhiwei and Jiao, Wenxiang and Wang, Xing and Wang, Yan and Wang, Rui and Yang, Yujiu and Shi, Shuming and Tu, Zhaopeng. Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate. Proceedings of the 2024 Conference on Empirical Metho...

2024
[39]

ArXiv , year=

Reasoning Models Generate Societies of Thought , author=. ArXiv , year=
[40]

Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems , articleno =

Sheng, Rui and Yang, Yukun and Shi, Chuhan and Lin, Yanna and Chen, Zixin and Qu, Huamin and Cheng, Furui , title =. Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems , articleno =. 2026 , isbn =

2026
[41]

International Conference on Learning Representations (ICLR) , year=

AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors , author=. International Conference on Learning Representations (ICLR) , year=
[42]

and Le, Quoc V

Wei, Jason and Wang, Xuezhi and Schuurmans, Dale and Bosma, Maarten and Ichter, Brian and Xia, Fei and Chi, Ed H. and Le, Quoc V. and Zhou, Denny , title =. Proceedings of the 36th International Conference on Neural Information Processing Systems (NeurIPS) , articleno =. 2022 ...

2022
[43]

International Conference on Learning Representations (ICLR) , year=

Multi-Agent Design: Optimizing Agents with Better Prompts and Topologies , author=. International Conference on Learning Representations (ICLR) , year=
[44]

International Conference on Learning Representations (ICLR) , year=

GraphBench: Next-generation graph learning benchmarking , author=. International Conference on Learning Representations (ICLR) , year=

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [1]

and Boyd, Danah and Friedler, Sorelle A

Selbst, Andrew D. and Boyd, Danah and Friedler, Sorelle A. and Venkatasubramanian, Suresh and Vertesi, Janet , title =. Proceedings of the Conference on Fairness, Accountability, and Transparency , pages =. 2019 , isbn =

2019

[2] [2]

LLM s instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks

Bavaresco, Anna and Bernardi, Raffaella and Bertolazzi, Leonardo and Elliott, Desmond and Fern \'a ndez, Raquel and Gatt, Albert and Ghaleb, Esam and Giulianelli, Mario and Hanna, Michael and Koller, Alexander and Martins, Andre and Mondorf, Philipp and Neplenbroek, Vera and Pezzelle, Sandro and Plank, Barbara and Schlangen, David and Suglia, Alessandro a...

2025

[3] [3]

2026 , issn =

A survey on LLM-as-a-judge , journal =. 2026 , issn =

2026

[4] [4]

Dorner and Vivian Yvonne Nastl and Moritz Hardt , booktitle=

Florian E. Dorner and Vivian Yvonne Nastl and Moritz Hardt , booktitle=. Limits to scalable evaluation at the frontier:

[5] [5]

2024 , booktitle =

Chen, Dongping and Chen, Ruoxi and Zhang, Shilin and Wang, Yaochen and Liu, Yinuo and Zhou, Huichi and Zhang, Qihui and Wan, Yao and Zhou, Pan and Sun, Lichao , title =. 2024 , booktitle =

2024

[6] [6]

Frontiers in Artificial Intelligence , volume=

The extended hollowed mind: why foundational knowledge is indispensable in the age of AI , author=. Frontiers in Artificial Intelligence , volume=. 2025 , publisher=

2025

[7] [7]

Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communication

Yin, Zhangyue and Sun, Qiushi and Chang, Cheng and Guo, Qipeng and Dai, Junqi and Huang, Xuanjing and Qiu, Xipeng. Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communication. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023

2023

[8] [8]

The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models? , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=