Pith. sign in

REVIEW 4 major objections 5 minor 300 references

Chain-of-Models: Cross-Model Auditing for Bias-Robust LLM Judges

T0 review · 4 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A different-family second model that reads a judge's reasoning trace corrects LLM bias better than any single fixed auditor.

desk verdict Real insight about cross-family auditors and the limits of standalone resistance, but a protocol contradiction in the appendix and an overstrong claim need fixing. read the letter →

arxiv 2607.28636 v1 pith:SLG2VHPZ submitted 2026-05-19 cs.CL cs.CY

Qian Wang , Zhanzhi Lou , Zhenheng Tang , Nuo Chen , Bingsheng He This is my paper · ORCID
classification cs.CLcs.CY
keywords LLM-as-judgecognitivebiascross-modelauditingreasoningtraceauditorselectionfunctionaldiversitysycophancymitigation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the best way to correct a biased LLM judge is not to prompt it harder or swap in the most bias-resistant model, but to add a second model from a different training family that reads the judge's full reasoning trace and issues the final judgment. Across nine models from six families, four cognitive biases, and four factual datasets, it argues that this auditor must be chosen per bias type: the model most resistant to a bias when answering alone is often the worst at correcting another model's biased reasoning, and no single auditor wins on every bias. The paper operationalizes this as Chain-of-Models (CoM), a routing rule that scores candidate auditors by functional diversity, standalone bias resistance, and a calibrated audit-effectiveness estimate, and shows under a calibration/test split that the rule reaches 0.884 accuracy across four biased slices, beating the strongest fixed auditor (0.824) and the no-audit baseline (0.805). A sympathetic reader would care because this offers a deployable, weight-free way to improve LLM-as-judge pipelines in high-stakes domains where human evaluation does not scale.

What carries the argument

The load-bearing mechanism is the two-model Chain-of-Models pipeline: M1 produces a reasoning trace and answer, then M2 receives the same prompt plus M1's full trace and re-judges, with no bias-specific instructions. Auditor selection is driven by a weighted score for each candidate: functional diversity (cosine distance between the generator's and candidate's behavioral fingerprint vectors), per-bias standalone resistance (candidate's own biased accuracy), and calibrated audit effectiveness e(M1, M2, b), the chain accuracy estimated on a held-out calibration split. The e term is what rules out the highly standalone-resistant Kimi-K2.5; conditioning on bias type b is what routes sycophancy t

What would settle it

Withhold the bias label and infer it with a learned classifier, or bootstrap the 25 calibration examples per cell: if the chosen auditor frequently flips or the selector's margin over always-on GPT-4o disappears, the central claim fails.

Watch

Extended reading notes

Core claim

Auditor identity, not standalone quality, decides whether a second model corrects a biased judge. With generator fixed at Qwen2.5-72B-Instruct, standalone bias resistance does not predict audit effectiveness: Kimi-K2.5, most resistant standalone on three biases, is the weakest auditor on bandwagon (0.400) and authority (0.500). The best auditor is bias-specific: GPT-4o wins bandwagon, authority, and distraction; GLM-5 wins sycophancy, where GPT-4o auditing is worse than no audit. A per-bias rule combining functional diversity, standalone resistance, and calibrated audit effectiveness reaches 0.884 test accuracy versus 0.824 for the best fixed auditor and 0.805 for no audit.

Load-bearing premise

The selection advantage depends on knowing each query's bias type at inference time and on the 25-example calibration estimate of audit effectiveness being accurate enough to rank candidate auditors reliably.

Editorial extensions

If this is right

  • A deployed LLM judge can be made more accurate on biased inputs without retraining or prompt engineering, by routing flagged queries to the auditor calibrated for that bias.
  • Standalone bias-resistance leaderboards should not be used to choose auditors; a model's own robustness says little about how well it corrects another model's biased reasoning.
  • Same-model or same-family self-audit is insufficient; the auditor's training lineage should differ from the judge's to catch correlated blind spots.
  • Per-bias routing beats always-on auditing at the same per-query cost, with the gain concentrated on the bias where the fixed auditor is weakest.
  • The selection rule transfers to subjective preference judging, but audit-effectiveness estimates must be re-estimated on the target domain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The resistance/effectiveness inversion suggests audit quality is about complementary failure modes between generator and auditor, not individual capability; a cheap trace-level disagreement score might predict it without costly calibration runs.
  • The rule assumes the bias type is known per query; a learned bias detector tested end-to-end would determine whether the method works on unlabeled in-the-wild prompts.
  • With only 25 calibration examples per bias–dataset cell, the effectiveness estimate is noisy; bootstrapping those cells would show how often the auditor ranking flips, and larger calibration sets could justify learning the scoring weights instead of fixing them.
  • The best auditor differing between factual and subjective sycophancy implies auditor rankings can be task-family-labile, so transferring a routing policy across domains without recalibration is risky.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Chain-of-Models (CoM), a sequential auditing pipeline in which a second LLM receives the first model's reasoning trace and produces a final judgment. Fixing the generator as Qwen2.5-72B-Instruct, the authors evaluate five cross-family auditors and one same-family ladder across four cognitive biases on MMLU-Pro and a subjective DPO sycophancy track. They report two main empirical findings: standalone bias resistance does not predict audit effectiveness (e.g., Kimi-K2.5 is most standalone-resistant but the weakest auditor on bandwagon and authority), and the best auditor is bias-specific (GPT-4o for bandwagon/authority/distraction; GLM-5 for factual sycophancy). They then propose a per-bias selector scoring auditors by functional DNA distance, standalone resistance, and calibration-split audit effectiveness, and claim 0.884 held-out accuracy across biased slices versus 0.824 for always-on GPT-4o and 0.805 for no audit.

Significance. If substantiated, the paper would make a useful contribution: it treats auditor identity as a design variable, provides a concrete selection rule, and releases code and configurations. The calibration/test split is a sound way to avoid selection-on-test, and the two headline findings are practically relevant. However, the protocol inconsistency in Appendix K.3 and the lack of uncertainty quantification currently prevent acceptance.

major comments (4)
  1. [§2.1 / §G.3 vs. Appendix K.3] The stated protocol and Appendix K.3 contradict each other. Section 2.1 says the auditor 'is not told that bias may be present' and G.3 says 'the original prompt—including any injected bias cue—is passed verbatim to Mi.' Appendix K.3, described as 'the key mechanism,' instead shows M2 receiving the clean prompt without the bandwagon cue and with the trace labeled 'Advisory: Another analyst's reasoning (may contain biases).' If K.3 reflects the actual experimental condition, the reported chain accuracies are confounded by cue removal and explicit bias flagging, and the central claim that a cross-family auditor resists the same cue M1 fell for is not supported. The authors must reconcile this: either K.3 is outside the evaluation protocol and must be relabeled/replaced with a verbatim G.3 trace, or G.3 is wrong and the experiments must be redone.
  2. [§3.3, Eq. (4), Table 12] The paper claims the selector combines functional diversity, standalone resistance, and empirical audit effectiveness, but the evidence shows the empirical-effectiveness term alone reproduces the result. Table 12's 'effectiveness-only' weighting (0,0,1) selects GPT-4o with the same 0.830 authority accuracy as the default (0.2,0.3,0.5); the diversity-only and resistance-only weightings collapse. No ablation tests whether removing d or r from the full score changes the selection. The three-axis claim is therefore not supported by the reported data. Please either demonstrate that d and r contribute beyond e (e.g., ablations that drop each term, or cases where e is unavailable) or reframe the selector as primarily e-based.
  3. [§3.3, Table 7; Appendix F] The calibration split uses only n=25 examples per bias–dataset cell to estimate e(M1, Mi, b), and all reported accuracies are point estimates without confidence intervals. With 25 calibration examples, the auditor ranking can easily flip under sampling noise; the overall 0.884 vs. 0.824 improvement could be inflated by a lucky calibration draw, especially since the entire gain is concentrated in the sycophancy slice. The paper acknowledges this in the limitations but does not quantify the risk. Please provide bootstrap confidence intervals, a calibration-size sensitivity analysis, or a larger calibration set.
  4. [§3.2] The statement that audit effectiveness 'cannot be predicted' from standalone bias resistance is stronger than the evidence supports. The paper demonstrates two counterexamples (Kimi-K2.5 and DeepSeek-V3) and shows that the best standalone model is not the best auditor, but this does not establish a general impossibility of any prediction from standalone resistance. A correlational analysis or a more tempered wording—e.g., 'standalone resistance is not a reliable predictor'—would match the data. This matters because the finding is one of the two headline claims motivating the selector.
minor comments (5)
  1. [Appendix C] Appendix C references 'the D-6 collapse on bandwagon (§3.2),' but Section 3.2 does not mention D-6 chains. Either add the D-6 results to the main text or fix the cross-reference.
  2. [Appendix F vs. Appendix I] Appendix F says 'An earlier trace-only detector pilot reported in §I had much lower recall on bandwagon and distraction,' but Appendix I does not describe such a pilot. The cross-reference appears to point to nonexistent content.
  3. [References] Several references are incomplete or placeholder-style, e.g., 'Jane Li and Others. 2024' and 'Nurit Cohen Inger and 1 others. 2026 ... Co-author list pending verification before camera-ready.' These need full author lists and verified details for a journal submission.
  4. [Appendix K.3] The K.3 trace uses GPT-4o-mini as M1, whereas the main experiments fix M1 as Qwen2.5-72B-Instruct. If this trace is intended as an illustrative example rather than a main-protocol run, it should be explicitly labeled as such.
  5. [§2.2] The text says 'over 500 experiments (∼100,000 API calls),' but the stated design (9 models × 4 biases × 4 datasets × 50 plus chains) corresponds to a different count. Please clarify the arithmetic or describe how the count was derived.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the selector is calibrated on a disjoint split and the headline result is an empirical evaluation, not a derivation from its own inputs.

full rationale

The paper's only predictive claim is the per-bias selector (Eq. 4), which combines d, r, and e. The empirical-effectiveness term e is estimated on a held-out calibration split (25 examples per bias-dataset cell) and the reported 0.884 accuracy is measured on disjoint test examples, so the headline result is not fitted to the target. The finding that standalone resistance does not predict audit effectiveness is an empirical comparison of measured r and e, not a definitional equivalence. The functional-diversity term comes from Wu et al. (2025), which shares two co-authors with this paper, but that citation is not load-bearing: the Appendix H ablation shows that dropping e (while retaining d and r) selects Kimi-K2.5 and collapses to 0.480, while the default selection is driven mainly by e. I also flag a serious internal inconsistency that is a validity concern rather than a circularity: Section 2.1 and Appendix G.3 state that the auditor receives the original biased prompt verbatim and is not told bias may be present, whereas Appendix K.3 describes the auditor as seeing the clean prompt without the bandwagon cue plus an 'Advisory: Another analyst's reasoning (may contain biases)' label and calls this 'the key mechanism.' If K.3 reflects the actual experimental protocol, the chain-accuracy measurements that populate e are confounded by cue removal and bias flagging, but that is a correctness/confound issue, not a derivation-equivalence or self-citation circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central result relies on a hand-set routing rule with three terms, a known-bias assumption, and a calibration split of 25 samples. No new physical entities are introduced. The diversity metric comes from the authors' own prior work, adding a mild self-referential component.

free parameters (3)
  • Routing weights (alpha, beta, gamma) = (0.2, 0.3, 0.5)
    Hand-set to prioritize empirical effectiveness; swept in Appendix H but not learned from data. Selection rule collapses to 0.480 when e is omitted.
  • Calibration split size = 25 examples per (bias, dataset) cell
    Chosen split; e estimates are noisy at this n and no confidence intervals are reported.
  • Bias cue injection templates = e.g., '87%', '90%'
    Cue strengths are fixed and not varied; detector achieves 100% recall by construction on these templates.
assumptions (4)
  • domain assumption Functional diversity d(Mi, Mj) from Wu et al. (2025) captures shared blind spots
    Eq. 4 includes d; the paper inherits the LLM-DNA representation without revalidating it on these models besides Figure 2 distances.
  • domain assumption Bias type b is known at inference time
    Section 3.3 states b comes from data-source labels or deployment context; if not, the selector cannot route.
  • domain assumption Auditors are not informed that bias may be present
    G.3 prompt template omits bias warnings; however Appendix K.3 shows a bias advisory, creating inconsistency.
  • standard math Cosine distance is a valid functional diversity metric
    Def. 1 uses cosine distance; standard, but the probes are only 5 prompts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Chain-of-Models: Cross-Model Auditing for Bias-Robust LLM Judges." pith.science (2026). https://pith.science/paper/SLG2VHPZ

@misc{pith2026260728636,
  author       = {Pith},
  title        = {Pith review of: Chain-of-Models: Cross-Model Auditing for Bias-Robust LLM Judges},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SLG2VHPZ}},
  note         = {Machine review of arXiv:2607.28636}
}
abstract

LLMs increasingly serve as automated judges, but their judgments remain vulnerable to cognitive biases. Existing mitigations mostly rely on prompt-driven debiasing, which is brittle across bias types, or human evaluation, which does not scale. We study \emph{Chain-of-Models} (CoM), an automated audit pipeline in which a second model inspects the first model's reasoning trace before producing the final judgment. The key design question is whether the auditor should be the same model, a same-family model, or a different-family model. Across 9 models from 6 families, 4 cognitive biases, and 4 factual datasets, we find that auditor identity matters in two ways. First, standalone bias resistance does not predict audit effectiveness: Kimi-K2.5 is the strongest standalone model on several biases, yet is a weak auditor for Qwen2.5-72B's biased traces. Second, the best auditor is bias-specific: GPT-4o is strongest on bandwagon, authority, and distraction, while GLM-5 is strongest on sycophancy. We operationalize these findings with a per-bias auditor selection rule that, given the bias type, scores candidates along functional diversity, per-bias standalone resistance, and calibrated audit effectiveness. Under a calibration/test split, the selector reaches the highest accuracy across the four biased slices ($0.884$ vs.\ $0.824$ for the strongest single fixed auditor and $0.805$ for the no-audit baseline). We release data, configurations, and an LLM-agent skill at https://anonymous.4open.science/r/chain-of-models-B585 .

Figures

Figures reproduced from arXiv: 2607.28636 by the authors.

Figure 1
Figure 1. Auditor identity matters. (A) Bias blind spot intuition: bias is easier to detect in another reasoner than in oneself, so the auditor should not be the same model as the judge. (B) M1 answers under a cognitive-bias cue: valid evidence in its trace supports answer B, but the cue pushes toward A and M1 pivots to the wrong answer. (C) A different-family auditor M2 receives M1’s trace and final answer, challenges the cu… view at source ↗
Figure 2
Figure 2. Pairwise functional DNA distances (co￾sine, computed over 5 probe prompts; cf. Wu et al., 2025). Bold cells: within-family pairs (Qwen→Qwen and GPT-mini→GPT-4o). Kimi-K2.5 is the largest func￾tional outlier (distances 0.103–0.145 to all other mod￾els); Qwen↔GPT distances (0.045–0.065) are at within￾family levels, indicating functional convergence despite distinct training organizations. family gain we report would n… view at source ↗
Figure 3
Figure 3. Two bias-induced reasoning patterns visible in the trace. Left: acknowledge-but-defer (factual)— M1 identifies correct evidence then abandons it un￾der authority pressure. Right: fabricated justification (subjective)—M1 invents a rationale aligned with the bandwagon cue. Both signatures are exposed in the trace and detectable by a heterogeneous M2. 50 samples—enough to detect the headline effects but not small ones;… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

300 extracted references · 1 canonical work pages

  1. [1]

    Personality and Social Psychology Bulletin , volume=

    The bias blind spot: Perceptions of bias in self versus others , author=. Personality and Social Psychology Bulletin , volume=. 2002 , publisher=

    2002

  2. [2]

    1957 , publisher=

    A Theory of Cognitive Dissonance , author=. 1957 , publisher=

    1957

  3. [3]

    Psychological Bulletin , volume=

    The case for motivated reasoning , author=. Psychological Bulletin , volume=. 1990 , publisher=

    1990

  4. [4]

    Essai sur l'application de l'analyse

    de Condorcet, Marquis , year=. Essai sur l'application de l'analyse

  5. [5]

    Wu, Zhaomin and Zhao, Haodong and Wang, Ziyang and Guo, Jizhou and Wang, Qian and He, Bingsheng , journal=

  6. [6]

    International Workshop on Multiple Classifier Systems , pages=

    Ensemble methods in machine learning , author=. International Workshop on Multiple Classifier Systems , pages=. 2000 , publisher=

    2000

  7. [7]

    International Conference on Learning Representations , year=

    Self-Consistency Improves Chain of Thought Reasoning in Language Models , author=. International Conference on Learning Representations , year=

  8. [8]

    International Conference on Machine Learning , year=

    Improving Factuality and Reasoning in Language Models through Multiagent Debate , author=. International Conference on Machine Learning , year=

Show all 300 references
  1. [9]

    Min, Sewon and Krishna, Kalpesh and Lyu, Xinxi and Lewis, Mike and Yih, Wen-tau and Koh, Pang Wei and Iyyer, Mohit and Zettlemoyer, Luke and Hajishirzi, Hannaneh , booktitle=

  2. [10]

    International Conference on Learning Representations , year=

    Teaching Large Language Models to Self-Debug , author=. International Conference on Learning Representations , year=

  3. [11]

    arXiv preprint arXiv:2305.20050 , year=

    Let's Verify Step by Step , author=. arXiv preprint arXiv:2305.20050 , year=

    arXiv

  4. [12]

    International Conference on Learning Representations , year=

    Mixture-of-Agents Enhances Large Language Model Capabilities , author=. International Conference on Learning Representations , year=

  5. [13]

    International Conference on Learning Representations , year=

    Towards Understanding Sycophancy in Language Models , author=. International Conference on Learning Representations , year=

  6. [14]

    arXiv preprint arXiv:2503.10814 , year=

    Thinking machines: A survey of llm based reasoning strategies , author=. arXiv preprint arXiv:2503.10814 , year=

    arXiv

  7. [15]

    arXiv preprint arXiv:2402.14016 , year=

    Is llm-as-a-judge robust? investigating universal adversarial attacks on zero-shot llm assessment , author=. arXiv preprint arXiv:2402.14016 , year=

    arXiv

  8. [16]

    Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    Deceptive Semantic Shortcuts on Reasoning Chains: How Far Can Models Go without Hallucination? , author=. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    2024

  9. [17]

    arXiv preprint arXiv:2410.05229 , year=

    Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models , author=. arXiv preprint arXiv:2410.05229 , year=

    arXiv

  10. [18]

    arXiv preprint arXiv:2502.16169 , year=

    Patterns over principles: The fragility of inductive reasoning in llms under noisy observations , author=. arXiv preprint arXiv:2502.16169 , year=

    arXiv

  11. [19]

    arXiv preprint arXiv:2506.04210 , year=

    Does Thinking More always Help? Understanding Test-Time Scaling in Reasoning Models , author=. arXiv preprint arXiv:2506.04210 , year=

  12. [20]

    Yuanbing Zhu and Zhenheng Tang and Xiang Liu and Ang Li and Bo Li and Xiaowen Chu and Bo Han , booktitle=. Oracle. 2025 , url=

    2025

  13. [21]

    2025 , eprint=

    One Token to Fool LLM-as-a-Judge , author=. 2025 , eprint=

    2025

  14. [22]

    2025 , eprint=

    Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety , author=. 2025 , eprint=

    2025

  15. [23]

    JailbreakLo

    Fanjunduo Wei and Zhenheng Tang and Rongfei Zeng and Tongliang Liu and Chengqi Zhang and Xiaowen Chu and Bo Han , booktitle=. JailbreakLo. 2025 , url=

    2025

  16. [24]

    ICML 2025 Workshop on Data in Generative Models - The Bad, the Ugly, and the Greats , year=

    Ghost in the Cloud: Your Geo-Distributed Large Language Models Training is Easily Manipulated , author=. ICML 2025 Workshop on Data in Generative Models - The Bad, the Ugly, and the Greats , year=

    2025

  17. [25]

    ICLR 2025 Workshop on Foundation Models in the Wild , year=

    AgentTaxo: Dissecting and Benchmarking Token Distribution of LLM Multi-Agent Systems , author=. ICLR 2025 Workshop on Foundation Models in the Wild , year=

    2025

  18. [26]

    The 63rd Annual Meeting of the Association for Computational Linguistics , year=

    MegaAgent: A Large-Scale Autonomous LLM-based Multi-Agent System Without Predefined SOPs , author=. The 63rd Annual Meeting of the Association for Computational Linguistics , year=

  19. [27]

    2025 , journal=

    Mediator: Memory-efficient LLM Merging with Less Parameter Conflicts and Uncertainty Based Routing , author=. 2025 , journal=

    2025

  20. [28]

    2025 , journal=

    Can LLMs Maintain Fundamental Abilities under KV Cache Compression? , author=. 2025 , journal=

    2025

  21. [29]

    2025 , journal=

    ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference , author=. 2025 , journal=

    2025

  22. [30]

    Proceedings of the 42th International Conference on Machine Learning , series =

    Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression , author=. Proceedings of the 42th International Conference on Machine Learning , series =

  23. [31]

    Qian Wang and Zhenheng Tang and Bingsheng He , booktitle=. Can

  24. [32]

    The Lottery

    Zhenheng Tang and Xiang Liu and Qian Wang and Peijie Dong and Bingsheng He and Xiaowen Chu and Bo Li , booktitle=. The Lottery

  25. [33]

    2024 , journal=

    FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression , author=. 2024 , journal=

    2024

  26. [34]

    arXiv preprint arXiv:2308.13387 , year=

    Do-not-answer: A dataset for evaluating safeguards in llms , author=. arXiv preprint arXiv:2308.13387 , year=

    arXiv

  27. [35]

    Journal of Experimental Social Psychology , volume=

    The effort heuristic , author=. Journal of Experimental Social Psychology , volume=. 2004 , publisher=

    2004

  28. [36]

    2025 , eprint=

    Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge , author=. 2025 , eprint=

    2025

  29. [37]

    2024 , eprint=

    A Survey on Data Selection for LLM Instruction Tuning , author=. 2024 , eprint=

    2024

  30. [38]

    arXiv preprint arXiv:2309.07045 , year=

    Safetybench: Evaluating the safety of large language models with multiple choice questions , author=. arXiv preprint arXiv:2309.07045 , year=

    arXiv

  31. [39]

    arXiv preprint arXiv:2407.17436 , year=

    Air-bench 2024: A safety benchmark based on risk categories from regulations and policies , author=. arXiv preprint arXiv:2407.17436 , year=

    2024 arXiv

  32. [40]

    arXiv preprint arXiv:2406.14598 , year=

    Sorry-bench: Systematically evaluating large language model safety refusal behaviors , author=. arXiv preprint arXiv:2406.14598 , year=

    arXiv

  33. [41]

    arXiv preprint arXiv:2501.09686 , year=

    Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models , author=. arXiv preprint arXiv:2501.09686 , year=

    arXiv

  34. [42]

    arXiv preprint arXiv:2504.18333 , year=

    Adversarial Attacks on LLM-as-a-Judge Systems: Insights from Prompt Injections , author=. arXiv preprint arXiv:2504.18333 , year=

    arXiv

  35. [43]

    arXiv preprint arXiv:2408.01605 , year=

    Cyberseceval 3: Advancing the evaluation of cybersecurity risks and capabilities in large language models , author=. arXiv preprint arXiv:2408.01605 , year=

    arXiv

  36. [44]

    arXiv preprint arXiv:2402.05044 , year=

    Salad-bench: A hierarchical and comprehensive safety benchmark for large language models , author=. arXiv preprint arXiv:2402.05044 , year=

    arXiv

  37. [45]

    arXiv preprint arXiv:2404.13161 , year=

    Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models , author=. arXiv preprint arXiv:2404.13161 , year=

    arXiv

  38. [46]

    First Conference on Language Modeling , year=

    AutoDAN: interpretable gradient-based adversarial attacks on large language models , author=. First Conference on Language Modeling , year=

  39. [47]

    Advances in Neural Information Processing Systems , volume=

    Jailbroken: How does llm safety training fail? , author=. Advances in Neural Information Processing Systems , volume=

  40. [48]

    arXiv preprint arXiv:2406.18510 , year=

    WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models , author=. arXiv preprint arXiv:2406.18510 , year=

    arXiv

  41. [49]

    arXiv preprint arXiv:2410.09024 , year=

    Agentharm: A benchmark for measuring harmfulness of llm agents , author=. arXiv preprint arXiv:2410.09024 , year=

    arXiv

  42. [50]

    arXiv preprint arXiv:2410.02644 , year=

    Agent security bench (asb): Formalizing and benchmarking attacks and defenses in llm-based agents , author=. arXiv preprint arXiv:2410.02644 , year=

    arXiv

  43. [51]

    arXiv preprint arXiv:2312.14197 , year=

    Benchmarking and defending against indirect prompt injection attacks on large language models , author=. arXiv preprint arXiv:2312.14197 , year=

    arXiv

  44. [52]

    arXiv preprint arXiv:2403.02691 , year=

    Injecagent: Benchmarking indirect prompt injections in tool-integrated large language model agents , author=. arXiv preprint arXiv:2403.02691 , year=

    arXiv

  45. [53]

    arXiv preprint arXiv:2308.01263 , year=

    Xstest: A test suite for identifying exaggerated safety behaviours in large language models , author=. arXiv preprint arXiv:2308.01263 , year=

    arXiv

  46. [54]

    arXiv preprint arXiv:2307.15043 , year=

    Universal and transferable adversarial attacks on aligned language models , author=. arXiv preprint arXiv:2307.15043 , year=

    arXiv

  47. [55]

    arXiv preprint arXiv:2408.15221 , year=

    Llm defenses are not robust to multi-turn human jailbreaks yet , author=. arXiv preprint arXiv:2408.15221 , year=

    arXiv

  48. [56]

    arXiv preprint arXiv:2410.05295 , year=

    Autodan-turbo: A lifelong agent for strategy self-exploration to jailbreak llms , author=. arXiv preprint arXiv:2410.05295 , year=

    arXiv

  49. [57]

    arXiv preprint arXiv:2404.07921 , year=

    Amplegcg: Learning a universal and transferable generative model of adversarial suffixes for jailbreaking both open and closed llms , author=. arXiv preprint arXiv:2404.07921 , year=

    arXiv

  50. [58]

    arXiv preprint arXiv:2501.12948 , year=

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year=

    arXiv

  51. [59]

    arXiv preprint arXiv:2412.19437 , year=

    Deepseek-v3 technical report , author=. arXiv preprint arXiv:2412.19437 , year=

    arXiv

  52. [60]

    2025 , url =

    OpenAI , title =. 2025 , url =

    2025

  53. [61]

    2024 , eprint=

    CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Models , author=. 2024 , eprint=

    2024

  54. [62]

    2024 , eprint=

    WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs , author=. 2024 , eprint=

    2024

  55. [63]

    arXiv preprint arXiv:2407.21783 , year=

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    arXiv

  56. [64]

    arXiv preprint arXiv:2406.12845 , year=

    Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts , author=. arXiv preprint arXiv:2406.12845 , year=

    arXiv

  57. [65]

    arXiv preprint arXiv:2409.10164 , year=

    Quantile regression for distributional reward models in rlhf , author=. arXiv preprint arXiv:2409.10164 , year=

    arXiv

  58. [66]

    arXiv preprint arXiv:2403.13787 , year=

    Rewardbench: Evaluating reward models for language modeling , author=. arXiv preprint arXiv:2403.13787 , year=

    arXiv

  59. [67]

    arXiv preprint arXiv:2410.21276 , year=

    Gpt-4o system card , author=. arXiv preprint arXiv:2410.21276 , year=

    arXiv

  60. [68]

    arXiv preprint arXiv:2310.03693 , year=

    Fine-tuning aligned language models compromises safety, even when users do not intend to! , author=. arXiv preprint arXiv:2310.03693 , year=

    arXiv

  61. [69]

    2017 , publisher =

    Expert Political Judgment: How Good Is It? How Can We Know? -- New Edition , author =. 2017 , publisher =

    2017

  62. [70]

    1993 , publisher=

    Ideals and illusions: On reconstruction and deconstruction in contemporary critical theory , author=. 1993 , publisher=

    1993

  63. [71]

    arXiv preprint arXiv:2501.18841 , year=

    Trading inference-time compute for adversarial robustness , author=. arXiv preprint arXiv:2501.18841 , year=

    arXiv

  64. [72]

    arXiv preprint arXiv:2411.01111 , year=

    Rule based rewards for language model safety , author=. arXiv preprint arXiv:2411.01111 , year=

    arXiv

  65. [73]

    2024 , author=

    Deliberative Alignment: Reasoning Enables Safer Language Models. 2024 , author=. URL https://arxiv. org/abs/2412.16339 , year=

    2024 arXiv

  66. [74]

    arXiv preprint arXiv:2410.12784 , year=

    Judgebench: A benchmark for evaluating llm-based judges , author=. arXiv preprint arXiv:2410.12784 , year=

    arXiv

  67. [75]

    2020 , eprint=

    BERTScore: Evaluating Text Generation with BERT , author=. 2020 , eprint=

    2020

  68. [76]

    arXiv preprint arXiv:2404.06395 , year=

    Minicpm: Unveiling the potential of small language models with scalable training strategies , author=. arXiv preprint arXiv:2404.06395 , year=

    arXiv

  69. [77]

    arXiv preprint arXiv:2308.12261 , year=

    Prompt2model: Generating deployable models from natural language instructions , author=. arXiv preprint arXiv:2308.12261 , year=

    arXiv

  70. [78]

    Py-DPO Dataset , howpublished =

  71. [79]

    NSFW (not safe for work) content DPO dataset , howpublished =

  72. [80]

    2023 , author =

    LMSys Chat Platform , howpublished =. 2023 , author =

    2023

  73. [81]

    2024 , eprint=

    MolX: Enhancing Large Language Models for Molecular Learning with A Multi-Modal Extension , author=. 2024 , eprint=

    2024

  74. [82]

    2024 , eprint=

    ScholarChemQA: Unveiling the Power of Language Models in Chemical Research Question Answering , author=. 2024 , eprint=

    2024

  75. [83]

    arXiv preprint arXiv:2406.10819 , year=

    GUI-WORLD: A Dataset for GUI-oriented Multimodal LLM-based Agents , author=. arXiv preprint arXiv:2406.10819 , year=

    arXiv

  76. [84]

    arXiv preprint arXiv:2406.18966 , year=

    UniGen: A Unified Framework for Textual Dataset Generation Using Large Language Models , author=. arXiv preprint arXiv:2406.18966 , year=

  77. [85]

    arXiv preprint arXiv:2407.16686 , year=

    Can Large Language Models Automatically Jailbreak GPT-4V? , author=. arXiv preprint arXiv:2407.16686 , year=

    arXiv

  78. [86]

    arXiv preprint arXiv:2310.19019 , year=

    Teacherlm: Teaching to fish rather than giving the fish, language modeling likewise , author=. arXiv preprint arXiv:2310.19019 , year=

    arXiv

  79. [87]

    arXiv preprint arXiv:2407.07061 , year=

    Internet of Agents: Weaving a Web of Heterogeneous Agents for Collaborative Intelligence , author=. arXiv preprint arXiv:2407.07061 , year=

    arXiv

  80. [88]

    arXiv preprint arXiv:2401.17882 , year=

    I Think, Therefore I am: Awareness in Large Language Models , author=. arXiv preprint arXiv:2401.17882 , year=

    arXiv

  81. [89]

    arXiv preprint arXiv:2407.12874 , year=

    SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning , author=. arXiv preprint arXiv:2407.12874 , year=

    arXiv

  82. [90]

    arXiv preprint arXiv:2403.17710 , year=

    Optimization-based Prompt Injection Attack to LLM-as-a-Judge , author=. arXiv preprint arXiv:2403.17710 , year=

    arXiv

  83. [91]

    arXiv preprint arXiv:2406.13662 , year=

    ObscurePrompt: Jailbreaking Large Language Models via Obscure Input , author=. arXiv preprint arXiv:2406.13662 , year=

    arXiv

  84. [92]

    arXiv preprint arXiv:2406.17675 , year=

    Quantifying AI Psychology: A Psychometrics Benchmark for Large Language Models , author=. arXiv preprint arXiv:2406.17675 , year=

  85. [93]

    2024 , eprint=

    I Think, Therefore I am: Benchmarking Awareness of Large Language Models Using AwareBench , author=. 2024 , eprint=

    2024

  86. [94]

    2024 , eprint=

    Judging the Judges: A Systematic Investigation of Position Bias in Pairwise Comparative Assessments by LLMs , author=. 2024 , eprint=

    2024

  87. [95]

    2024 , eprint=

    LLM Evaluators Recognize and Favor Their Own Generations , author=. 2024 , eprint=

    2024

  88. [96]

    2024 , eprint=

    Large Language Models are Inconsistent and Biased Evaluators , author=. 2024 , eprint=

    2024

  89. [97]

    What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks , url =

    Guo, Taicheng and Guo, kehan and Nan, Bozhao and Liang, Zhenwen and Guo, Zhichun and Chawla, Nitesh and Wiest, Olaf and Zhang, Xiangliang , booktitle =. What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks , url =

  90. [98]

    2024 , eprint=

    SceMQA: A Scientific College Entrance Level Multimodal Question Answering Benchmark , author=. 2024 , eprint=

    2024

  91. [99]

    2024 , eprint=

    Adapting Large Language Models for Education: Foundational Capabilities, Potentials, and Challenges , author=. 2024 , eprint=

    2024

  92. [100]

    arXiv preprint arXiv:2303.11032 , year=

    Deid-gpt: Zero-shot medical text de-identification by gpt-4 , author=. arXiv preprint arXiv:2303.11032 , year=

  93. [101]

    arXiv preprint arXiv:2307.07924 , url =

    ChatDev: Communicative Agents for Software Development , author =. arXiv preprint arXiv:2307.07924 , url =

    arXiv

  94. [102]

    arXiv preprint arXiv:2406.00380 , year=

    The Best of Both Worlds: Toward an Honest and Helpful Large Language Model , author=. arXiv preprint arXiv:2406.00380 , year=

    arXiv

  95. [103]

    Advances in neural information processing systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in neural information processing systems , volume=

  96. [104]

    2024 , eprint=

    Large Language Model based Multi-Agents: A Survey of Progress and Challenges , author=. 2024 , eprint=

    2024

  97. [105]

    arXiv preprint arXiv:2212.08073 , year=

    Constitutional ai: Harmlessness from ai feedback , author=. arXiv preprint arXiv:2212.08073 , year=

    arXiv

  98. [106]

    arXiv preprint arXiv:2405.14734 , year=

    Simpo: Simple preference optimization with a reference-free reward , author=. arXiv preprint arXiv:2405.14734 , year=

    arXiv

  99. [107]

    2019 , eprint=

    MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance , author=. 2019 , eprint=

    2019

  100. [108]

    2023 , eprint=

    DiscoScore: Evaluating Text Generation with BERT and Discourse Coherence , author=. 2023 , eprint=

    2023

  101. [109]

    Advances in Neural Information Processing Systems , volume=

    Judging llm-as-a-judge with mt-bench and chatbot arena , author=. Advances in Neural Information Processing Systems , volume=

  102. [110]

    arXiv preprint arXiv:2307.02762 , year=

    Prd: Peer rank and discussion improve large language model based evaluations , author=. arXiv preprint arXiv:2307.02762 , year=

    arXiv

  103. [111]

    2023 , eprint=

    Benchmarking Cognitive Biases in Large Language Models as Evaluators , author=. 2023 , eprint=

    2023

  104. [112]

    2024 , eprint=

    Humans or LLMs as the Judge? A Study on Judgement Biases , author=. 2024 , eprint=

    2024

  105. [113]

    Answers Unite! Unsupervised Metrics for Reinforced Summarization Models

    Scialom, Thomas and Lamprier, Sylvain and Piwowarski, Benjamin and Staiano, Jacopo. Answers Unite! Unsupervised Metrics for Reinforced Summarization Models. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint C...

    2019 doi

  106. [114]

    Rei, Ricardo and Farinha, Ana C and Zerva, Chrysoula and van Stigt, Daan and Stewart, Craig and Ramos, Pedro and Glushkova, Taisiya and Martins, Andr \'e F. T. and Lavie, Alon. Are References Really Needed? Unbabel- IST 2021 Submission for the Metrics Shared Task. Proceedings ...

    2021

  107. [115]

    On the Limitations of Reference-Free Evaluations of Generated Text

    Deutsch, Daniel and Dror, Rotem and Roth, Dan. On the Limitations of Reference-Free Evaluations of Generated Text. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022. doi:10.18653/v1/2022.emnlp-main.753

    2022 doi

  108. [116]

    2023 , eprint=

    Is ChatGPT a Good NLG Evaluator? A Preliminary Study , author=. 2023 , eprint=

    2023

  109. [117]

    2024 , eprint=

    Beyond Traditional Benchmarks: Analyzing Behaviors of Open LLMs on Data-to-Text Generation , author=. 2024 , eprint=

    2024

  110. [118]

    2024 , eprint=

    On the Limitations of Fine-tuned Judge Models for LLM Evaluation , author=. 2024 , eprint=

    2024

  111. [119]

    2023 , eprint=

    Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision , author=. 2023 , eprint=

    2023

  112. [120]

    2023 , eprint=

    Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models , author=. 2023 , eprint=

    2023

  113. [121]

    2021 , eprint=

    A General Language Assistant as a Laboratory for Alignment , author=. 2021 , eprint=

    2021

  114. [122]

    2022 , eprint=

    Constitutional AI: Harmlessness from AI Feedback , author=. 2022 , eprint=

    2022

  115. [123]

    arXiv preprint arXiv:2306.11507 , year=

    Trustgpt: A benchmark for trustworthy and responsible large language models , author=. arXiv preprint arXiv:2306.11507 , year=

    arXiv

  116. [124]

    arXiv preprint arXiv:2310.05046 , year=

    Harnessing the power of chatgpt in fake news: An in-depth exploration in generation, detection and explanation , author=. arXiv preprint arXiv:2310.05046 , year=

    arXiv

  117. [125]

    2024 , eprint=

    TrustLLM: Trustworthiness in Large Language Models , author=. 2024 , eprint=

    2024

  118. [126]

    ArXiv , year=

    Exploring AI Ethics of ChatGPT: A Diagnostic Analysis , author=. ArXiv , year=

  119. [127]

    Should ChatGPT be biased? Challenges and risks of bias in large language models , ISSN=

    Ferrara, Emilio , year=. Should ChatGPT be biased? Challenges and risks of bias in large language models , ISSN=. doi:10.5210/fm.v28i11.13346 , journal=

    doi

  120. [128]

    Kelly is a Warm Person, Joseph is a Role Model

    "Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letters , author=. 2023 , eprint=

    2023

  121. [129]

    Age Discrimination: An Historical and Contemporary Analysis , isbn =

    Macnicol, John , year =. Age Discrimination: An Historical and Contemporary Analysis , isbn =

  122. [130]

    2023 , eprint=

    Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine , author=. 2023 , eprint=

    2023

  123. [131]

    2023 , eprint=

    A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity , author=. 2023 , eprint=

    2023

  124. [132]

    2024 , eprint=

    Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment , author=. 2024 , eprint=

    2024

  125. [133]

    HuggingFace repository , howpublished =

    OpenOrca: An Open Dataset of GPT Augmented FLAN Reasoning Traces , author =. HuggingFace repository , howpublished =. 2023 , publisher =

    2023

  126. [134]

    2023 , eprint=

    Orca: Progressive Learning from Complex Explanation Traces of GPT-4 , author=. 2023 , eprint=

    2023

  127. [135]

    2023 , eprint=

    The Flan Collection: Designing Data and Methods for Effective Instruction Tuning , author=. 2023 , eprint=

    2023

  128. [136]

    2023 , eprint=

    Llama 2: Open Foundation and Fine-Tuned Chat Models , author=. 2023 , eprint=

    2023

  129. [137]

    2021 , eprint=

    Measuring Mathematical Problem Solving With the MATH Dataset , author=. 2021 , eprint=

    2021

  130. [138]

    2021 , eprint=

    Training Verifiers to Solve Math Word Problems , author=. 2021 , eprint=

    2021

  131. [139]

    2022 , eprint=

    Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering , author=. 2022 , eprint=

    2022

  132. [140]

    arXiv preprint arXiv:2311.18743 , year=

    Alignbench: Benchmarking chinese alignment of large language models , author=. arXiv preprint arXiv:2311.18743 , year=

    arXiv

  133. [141]

    2024 , eprint=

    GPT-4 Technical Report , author=. 2024 , eprint=

    2024

  134. [142]

    2024 , url =

    OpenAI , title =. 2024 , url =

    2024

  135. [143]

    2024 , url =

    Anthropic , title =. 2024 , url =

    2024

  136. [144]

    2024 , eprint=

    ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools , author=. 2024 , eprint=

    2024

  137. [145]

    arXiv preprint arXiv:2309.16609 , year=

    Qwen Technical Report , author=. arXiv preprint arXiv:2309.16609 , year=

    arXiv

  138. [146]

    2023 , eprint=

    Large Language Models are not Fair Evaluators , author=. 2023 , eprint=

    2023

  139. [147]

    2024 , eprint=

    Evaluating Large Language Models at Evaluating Instruction Following , author=. 2024 , eprint=

    2024

  140. [148]

    2023 , eprint=

    Style Over Substance: Evaluation Biases for Large Language Models , author=. 2023 , eprint=

    2023

  141. [149]

    2019 , eprint=

    CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge , author=. 2019 , eprint=

    2019

  142. [150]

    2022 , eprint=

    TruthfulQA: Measuring How Models Mimic Human Falsehoods , author=. 2022 , eprint=

    2022

  143. [151]

    2024 , url =

    Llama 3 Model Card , author=. 2024 , url =

    2024

  144. [152]

    AI@Mistral , title =

  145. [153]

    ROUGE : A Package for Automatic Evaluation of Summaries

    Lin, Chin-Yew. ROUGE : A Package for Automatic Evaluation of Summaries. Text Summarization Branches Out. 2004

    2004

  146. [154]

    Proceedings of the 40th annual meeting of the Association for Computational Linguistics , pages=

    Bleu: a method for automatic evaluation of machine translation , author=. Proceedings of the 40th annual meeting of the Association for Computational Linguistics , pages=

  147. [155]

    2024 , eprint=

    LLM-as-a-Coauthor: Can Mixed Human-Written and Machine-Generated Text Be Detected? , author=. 2024 , eprint=

    2024

  148. [156]

    arXiv preprint arXiv:2310.03128 , year=

    Metatool benchmark for large language models: Deciding whether to use tools and which to use , author=. arXiv preprint arXiv:2310.03128 , year=

    arXiv

  149. [157]

    2024 , eprint=

    1+1>2: Can Large Language Models Serve as Cross-Lingual Knowledge Aggregators? , author=. 2024 , eprint=

    2024

  150. [158]

    2023 , eprint=

    Wider and Deeper LLM Networks are Fairer LLM Evaluators , author=. 2023 , eprint=

    2023

  151. [159]

    2023 , eprint=

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models , author=. 2023 , eprint=

    2023

  152. [160]

    2024 , eprint=

    MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark , author=. 2024 , eprint=

    2024

  153. [161]

    2024 , eprint=

    Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges , author=. 2024 , eprint=

    2024

  154. [162]

    2024 , eprint=

    Bias and Fairness in Large Language Models: A Survey , author=. 2024 , eprint=

    2024

  155. [163]

    , editor=

    Slovic, Paul and Finucane, Melissa and Peters, Ellen and MacGregor, Donald G. , editor=. The Affect Heuristic , booktitle=. 2002 , pages=

    2002

  156. [164]

    2023 , eprint=

    Split and Merge: Aligning Position Biases in Large Language Model based Evaluators , author=. 2023 , eprint=

    2023

  157. [165]

    2024 , eprint=

    Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment , author=. 2024 , eprint=

    2024

  158. [166]

    2024 , eprint=

    Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement , author=. 2024 , eprint=

    2024

  159. [167]

    The Symposium on Future Directions in Information Access (FDIA) co-located with the 2023 European Summer School on Information Retrieval (ESSIR) , year=

    Examining query sentiment bias effects on search results in large language models , author=. The Symposium on Future Directions in Information Access (FDIA) co-located with the 2023 European Summer School on Information Retrieval (ESSIR) , year=

    2023

  160. [168]

    Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinement

    Xu, Wenda and Zhu, Guanglei and Zhao, Xuandong and Pan, Liangming and Li, Lei and Wang, William. Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinement. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024

    2024

  161. [169]

    The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track , year=

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark , author=. The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track , year=

  162. [170]

    arXiv preprint arXiv:2503.00555 , year=

    Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonable , author=. arXiv preprint arXiv:2503.00555 , year=

    arXiv

  163. [171]

    2023 , howpublished =

    Truthy-DPO-v0.1 , author =. 2023 , howpublished =

    2023

  164. [172]

    2024 , howpublished =

    Emerton-DPO-Pairs-Judge , author =. 2024 , howpublished =

    2024

  165. [173]

    2023 , howpublished =

    Orca-DPO-Pairs , author =. 2023 , howpublished =

    2023

  166. [174]

    2024 , howpublished =

    Py-DPO-v0.1 , author =. 2024 , howpublished =

    2024

  167. [175]

    2024 , howpublished =

    xDAN-SFT-DPO-Roleplay-NSFW-with-lf , author =. 2024 , howpublished =

    2024

  168. [176]

    Advances in Neural Information Processing Systems , volume=

    Direct preference optimization: Your language model is secretly a reward model , author=. Advances in Neural Information Processing Systems , volume=

  169. [177]

    arXiv preprint arXiv:2407.01085 , year=

    Rethinking LLM-based Preference Evaluation , author=. arXiv preprint arXiv:2407.01085 , year=

    arXiv

  170. [178]

    ArXiv , year =

    Gu, John and Others , title =. ArXiv , year =

  171. [179]

    EMNLP , year =

    Li, Jane and Others , title =. EMNLP , year =

  172. [180]

    Humans or LLM s as the Judge? A Study on Judgement Bias

    Chen, Guiming Hardy and Chen, Shunian and Liu, Ziche and Jiang, Feng and Wang, Benyou. Humans or LLM s as the Judge? A Study on Judgement Bias. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. doi:10.18653/v1/2024.emnlp-main.474

    2024 doi

  173. [181]

    OpenCompass: A Universal Evaluation Platform for Foundation Models , author=

  174. [182]

    Proceedings of the International Conference on Learning Representations (ICLR) , year=

    Measuring Massive Multitask Language Understanding , author=. Proceedings of the International Conference on Learning Representations (ICLR) , year=

  175. [183]

    Proceedings of the International Conference on Learning Representations (ICLR) , year=

    Aligning AI With Shared Human Values , author=. Proceedings of the International Conference on Learning Representations (ICLR) , year=

  176. [184]

    2023 , eprint=

    A Survey on Evaluation of Large Language Models , author=. 2023 , eprint=

    2023

  177. [185]

    2018 , eprint=

    HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering , author=. 2018 , eprint=

    2018

  178. [186]

    arXiv preprint arXiv:2108.07732 , year=

    Program Synthesis with Large Language Models , author=. arXiv preprint arXiv:2108.07732 , year=

    arXiv

  179. [187]

    The Benefits of a Concise Chain of Thought on Problem-Solving in Large Language Models , url=

    Renze, Matthew and Guven, Erhan , year=. The Benefits of a Concise Chain of Thought on Problem-Solving in Large Language Models , url=. doi:10.1109/fllm63129.2024.10852493 , booktitle=

    2024

  180. [188]

    arXiv preprint arXiv:2502.12659 , year=

    The hidden risks of large reasoning models: A safety assessment of r1 , author=. arXiv preprint arXiv:2502.12659 , year=

  181. [189]

    arXiv preprint arXiv:2306.02707 , year=

    Orca: Progressive learning from complex explanation traces of gpt-4 , author=. arXiv preprint arXiv:2306.02707 , year=

    arXiv

  182. [190]

    arXiv preprint arXiv:2504.00050 , year=

    Judgelrm: Large reasoning models as a judge , author=. arXiv preprint arXiv:2504.00050 , year=

  183. [191]

    Advances in Neural Information Processing Systems , volume=

    Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models , author=. Advances in Neural Information Processing Systems , volume=

  184. [192]

    33rd USENIX Security Symposium (USENIX Security 24) , pages=

    Formalizing and benchmarking prompt injection attacks and defenses , author=. 33rd USENIX Security Symposium (USENIX Security 24) , pages=

  185. [193]

    arXiv preprint arXiv:2504.07887 , year=

    Benchmarking Adversarial Robustness to Bias Elicitation in Large Language Models: Scalable Automated Assessment with LLM-as-a-Judge , author=. arXiv preprint arXiv:2504.07887 , year=

  186. [194]

    arXiv preprint arXiv:2409.08087 , year=

    Securing Large Language Models: Addressing Bias, Misinformation, and Prompt Attacks , author=. arXiv preprint arXiv:2409.08087 , year=

  187. [195]

    arXiv preprint arXiv:2310.10844 , year=

    Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks , author=. arXiv preprint arXiv:2310.10844 , year=

    arXiv

  188. [196]

    arXiv preprint arXiv:2410.02736 , year=

    Justice or prejudice? quantifying biases in llm-as-a-judge , author=. arXiv preprint arXiv:2410.02736 , year=

    arXiv

  189. [197]

    2025 , url =

    Claude , title =. 2025 , url =

    2025

  190. [198]

    arXiv preprint arXiv:2410.21819 , year=

    Self-preference bias in llm-as-a-judge , author=. arXiv preprint arXiv:2410.21819 , year=

    arXiv

  191. [199]

    arXiv preprint arXiv:2506.17088 , year =

    Chain-of-Thought Prompting Obscures Hallucination Cues in Large Language Models: An Empirical Evaluation , author =. arXiv preprint arXiv:2506.17088 , year =

  192. [200]

    2025 , note =

    Introducing GPT-5 , howpublished =. 2025 , note =

    2025

  193. [201]

    arXiv preprint arXiv:2506.12217 , year =

    From Emergence to Control: Probing and Modulating Self-Reflection in Language Models , author =. arXiv preprint arXiv:2506.12217 , year =

    arXiv

  194. [202]

    arXiv preprint arXiv:2505.16475 , year =

    ReflectEvo: Improving Meta Introspection of Small LLMs by Learning Self-Reflection , author =. arXiv preprint arXiv:2505.16475 , year =

    arXiv

  195. [203]

    arXiv preprint arXiv:2501.04519 , year=

    rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking , author=. arXiv preprint arXiv:2501.04519 , year=

    arXiv

  196. [204]

    University of Chicago Coase-Sandor Institute for Law & Economics Research Paper , number=

    Judge AI: Assessing Large Language Models in Judicial Decision-Making , author=. University of Chicago Coase-Sandor Institute for Law & Economics Research Paper , number=

  197. [205]

    arXiv preprint arXiv:2504.12522 , year=

    Evaluating the Diversity and Quality of LLM Generated Content , author=. arXiv preprint arXiv:2504.12522 , year=

  198. [206]

    arXiv preprint arXiv:2502.12893 , year=

    H-cot: Hijacking the chain-of-thought safety reasoning mechanism to jailbreak large reasoning models, including openai o1/o3, deepseek-r1, and gemini 2.0 flash thinking , author=. arXiv preprint arXiv:2502.12893 , year=

    arXiv

  199. [207]

    arXiv preprint arXiv:2502.17419 , year=

    From system 1 to system 2: A survey of reasoning large language models , author=. arXiv preprint arXiv:2502.17419 , year=

    arXiv

  200. [208]

    arXiv preprint arXiv:2412.16720 , year=

    Openai o1 system card , author=. arXiv preprint arXiv:2412.16720 , year=

    arXiv

  201. [209]

    2024 , eprint=

    The Llama 3 Herd of Models , author=. 2024 , eprint=

    2024

  202. [210]

    2024 , eprint=

    Large Language Models: A Survey , author=. 2024 , eprint=

    2024

  203. [211]

    2025 , eprint=

    Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models , author=. 2025 , eprint=

    2025

  204. [212]

    2022 , eprint=

    Towards Understanding Mixture of Experts in Deep Learning , author=. 2022 , eprint=

    2022

  205. [213]

    2022 , eprint=

    Training language models to follow instructions with human feedback , author=. 2022 , eprint=

    2022

  206. [214]

    2020 , eprint=

    LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning , author=. 2020 , eprint=

    2020

  207. [215]

    2020 , eprint=

    CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning , author=. 2020 , eprint=

    2020

  208. [216]

    2019 , eprint=

    GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding , author=. 2019 , eprint=

    2019

  209. [217]

    2017 , eprint=

    Long Text Generation via Adversarial Training with Leaked Information , author=. 2017 , eprint=

    2017

  210. [218]

    2021 , eprint=

    Program Synthesis with Large Language Models , author=. 2021 , eprint=

    2021

  211. [219]

    2024 , eprint=

    LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models , author=. 2024 , eprint=

    2024

  212. [220]

    2023 , eprint=

    Tree of Thoughts: Deliberate Problem Solving with Large Language Models , author=. 2023 , eprint=

    2023

  213. [221]

    2023 , eprint=

    Large Language Models are Zero-Shot Reasoners , author=. 2023 , eprint=

    2023

  214. [222]

    2025 , eprint=

    Reasoning-Enhanced Self-Training for Long-Form Personalized Text Generation , author=. 2025 , eprint=

    2025

  215. [223]

    2025 , eprint=

    Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges , author=. 2025 , eprint=

    2025

  216. [224]

    2025 , howpublished =

    Meta LLaMA 3: The Next Generation of Open Foundation Models , author =. 2025 , howpublished =

    2025

  217. [225]

    ChatGPT outperforms crowd workers for text-annotation tasks , volume=

    Gilardi, Fabrizio and Alizadeh, Meysam and Kubli, Maël , year=. ChatGPT outperforms crowd workers for text-annotation tasks , volume=. Proceedings of the National Academy of Sciences , publisher=. doi:10.1073/pnas.2305016120 , number=

    doi

  218. [226]

    2025 , eprint=

    RocketEval: Efficient Automated LLM Evaluation via Grading Checklist , author=. 2025 , eprint=

    2025

  219. [227]

    2024 , eprint=

    Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge , author=. 2024 , eprint=

    2024

  220. [228]

    2024 , eprint=

    Reasoning with Large Language Models, a Survey , author=. 2024 , eprint=

    2024

  221. [229]

    Can Large Language Models Be an Alternative to Human Evaluations?

    Chiang, Cheng-Han and Lee, Hung-yi. Can Large Language Models Be an Alternative to Human Evaluations?. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023. doi:10.18653/v1/2023.acl-long.870

    2023 doi

  222. [230]

    2024 , eprint=

    Explaining Length Bias in LLM-Based Preference Evaluations , author=. 2024 , eprint=

    2024

  223. [231]

    2023 , eprint=

    G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment , author=. 2023 , eprint=

    2023

  224. [232]

    2024 , eprint=

    LLMs are Biased Evaluators But Not Biased for Retrieval Augmented Generation , author=. 2024 , eprint=

    2024

  225. [233]

    2023 , eprint=

    PaLM 2 Technical Report , author=. 2023 , eprint=

    2023

  226. [234]

    2024 , eprint=

    Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate , author=. 2024 , eprint=

    2024

  227. [235]

    2023 , eprint=

    Self-Refine: Iterative Refinement with Self-Feedback , author=. 2023 , eprint=

    2023

  228. [236]

    The Need for a Leaderboard: A Survey of

    Anonymous , booktitle=. The Need for a Leaderboard: A Survey of. 2024 , url=

    2024

  229. [237]

    Transactions on Machine Learning Research , issn=

    Emergent Abilities of Large Language Models , author=. Transactions on Machine Learning Research , issn=. 2022 , url=

    2022

  230. [238]

    2020 , eprint=

    Language Models are Few-Shot Learners , author=. 2020 , eprint=

    2020

  231. [239]

    Comparing Automatic and Human Evaluation of NLG Systems

    Belz, Anja and Reiter, Ehud. Comparing Automatic and Human Evaluation of NLG Systems. 11th Conference of the E uropean Chapter of the Association for Computational Linguistics. 2006

    2006

  232. [240]

    Why We Need New Evaluation Metrics for NLG

    Novikova, Jekaterina and Du s ek, Ond r ej and Cercas Curry, Amanda and Rieser, Verena. Why We Need New Evaluation Metrics for NLG. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2017. doi:10.18653/v1/D17-1238

    2017 doi

  233. [241]

    2023 , eprint=

    Sparks of Artificial General Intelligence: Early experiments with GPT-4 , author=. 2023 , eprint=

    2023

  234. [242]

    Scaling Learning Algorithms Towards

    Bengio, Yoshua and LeCun, Yann , booktitle =. Scaling Learning Algorithms Towards

  235. [243]

    and Osindero, Simon and Teh, Yee Whye , journal =

    Hinton, Geoffrey E. and Osindero, Simon and Teh, Yee Whye , journal =. A Fast Learning Algorithm for Deep Belief Nets , volume =

  236. [244]

    2016 , publisher=

    Deep learning , author=. 2016 , publisher=

    2016

  237. [245]

    arXiv preprint arXiv:2302.13971 , year=

    Llama: Open and efficient foundation language models , author=. arXiv preprint arXiv:2302.13971 , year=

    arXiv

  238. [246]

    arXiv preprint arXiv:2304.10592 , year=

    Minigpt-4: Enhancing vision-language understanding with advanced large language models , author=. arXiv preprint arXiv:2304.10592 , year=

    arXiv

  239. [247]

    arXiv preprint arXiv:2404.18231 , year=

    From persona to personalization: A survey on role-playing language agents , author=. arXiv preprint arXiv:2404.18231 , year=

    arXiv

  240. [248]

    arXiv preprint arXiv:2310.05746 , year=

    Put your money where your mouth is: Evaluating strategic planning and execution of llm agents in an auction arena , author=. arXiv preprint arXiv:2310.05746 , year=

    arXiv

  241. [249]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    Llm-planner: Few-shot grounded planning for embodied agents with large language models , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  242. [250]

    2023 , publisher=

    Error analysis prompting enables human-like translation evaluation in large language models: A case study on chatgpt , author=. 2023 , publisher=

    2023

  243. [251]

    Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

    Towards mitigating LLM hallucination via self reflection , author=. Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

    2023

  244. [252]

    arXiv preprint arXiv:2310.15556 , year=

    Tcra-llm: Token compression retrieval augmented large language model for inference cost reduction , author=. arXiv preprint arXiv:2310.15556 , year=

    arXiv

  245. [253]

    arXiv preprint arXiv:2406.06461 , year=

    Reasoning in Token Economies: Budget-Aware Evaluation of LLM Reasoning Strategies , author=. arXiv preprint arXiv:2406.06461 , year=

    arXiv

  246. [254]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Vtimellm: Empower llm to grasp video moments , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  247. [255]

    arXiv preprint arXiv:2210.08726 , year=

    Rarr: Researching and revising what language models say, using language models , author=. arXiv preprint arXiv:2210.08726 , year=

    arXiv

  248. [256]

    Perspectives in clinical research , volume=

    Quality assurance: Importance of systems and standard operating procedures , author=. Perspectives in clinical research , volume=. 2011 , publisher=

    2011

  249. [257]

    arXiv preprint arXiv:2311.07491 , year=

    A Step Closer to Comprehensive Answers: Constrained Multi-Stage Question Decomposition with Large Language Models , author=. arXiv preprint arXiv:2311.07491 , year=

    arXiv

  250. [258]

    arXiv preprint arXiv:2302.12813 , year=

    Check your facts and try again: Improving large language models with external knowledge and automated feedback , author=. arXiv preprint arXiv:2302.12813 , year=

    arXiv

  251. [259]

    arXiv preprint arXiv:2303.08774 , year=

    Gpt-4 technical report , author=. arXiv preprint arXiv:2303.08774 , year=

    arXiv

  252. [260]

    arXiv preprint arXiv:2401.02038 , year=

    Understanding llms: A comprehensive overview from training to inference , author=. arXiv preprint arXiv:2401.02038 , year=

    arXiv

  253. [261]

    arXiv preprint arXiv:2408.00989 , year=

    On the Resilience of Multi-Agent Systems with Malicious Agents , author=. arXiv preprint arXiv:2408.00989 , year=

    arXiv

  254. [262]

    arXiv preprint arXiv:2402.03578 , year=

    LLM multi-agent systems: Challenges and open problems , author=. arXiv preprint arXiv:2402.03578 , year=

    arXiv

  255. [263]

    arXiv preprint arXiv:2408.09955 , year=

    Megaagent: A practical framework for autonomous cooperation in large-scale llm agent systems , author=. arXiv preprint arXiv:2408.09955 , year=

    arXiv

  256. [264]

    arXiv preprint arXiv:2402.18272 , year=

    Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key? , author=. arXiv preprint arXiv:2402.18272 , year=

    arXiv

  257. [265]

    Advances in Neural Information Processing Systems , volume=

    To repeat or not to repeat: Insights from scaling llm under token-crisis , author=. Advances in Neural Information Processing Systems , volume=

  258. [266]

    arXiv preprint arXiv:2410.02506 , year=

    Cut the crap: An economical communication pipeline for llm-based multi-agent systems , author=. arXiv preprint arXiv:2410.02506 , year=

    arXiv

  259. [267]

    arXiv preprint arXiv:2309.07870 , year=

    Agents: An open-source framework for autonomous language agents , author=. arXiv preprint arXiv:2309.07870 , year=

    arXiv

  260. [268]

    ACM Transactions on Information Systems , year=

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions , author=. ACM Transactions on Information Systems , year=

  261. [269]

    Scientific reports , volume=

    Quantifying the uncertainty of LLM hallucination spreading in complex adaptive social networks , author=. Scientific reports , volume=. 2024 , publisher=

    2024

  262. [270]

    arXiv preprint arXiv:2402.13184 , year=

    What if LLMs Have Different World Views: Simulating Alien Civilizations with LLM-based Agents , author=. arXiv preprint arXiv:2402.13184 , year=

    arXiv

  263. [271]

    arXiv preprint arXiv:2308.03688 , year=

    Agentbench: Evaluating llms as agents , author=. arXiv preprint arXiv:2308.03688 , year=

    arXiv

  264. [272]

    arXiv preprint arXiv:2403.02839 , year=

    An empirical study of llm-as-a-judge for llm evaluation: Fine-tuned judge models are task-specific classifiers , author=. arXiv preprint arXiv:2403.02839 , year=

    arXiv

  265. [273]

    arXiv preprint arXiv:2410.22932 , year=

    Multi-Agent Large Language Models for Conversational Task-Solving , author=. arXiv preprint arXiv:2410.22932 , year=

    arXiv

  266. [274]

    arXiv preprint arXiv:2401.01313 , year=

    A comprehensive survey of hallucination mitigation techniques in large language models , author=. arXiv preprint arXiv:2401.01313 , year=

    arXiv

  267. [275]

    arXiv preprint arXiv:2310.06271 , year=

    Towards mitigating hallucination in large language models via self-reflection , author=. arXiv preprint arXiv:2310.06271 , year=

    arXiv

  268. [276]

    arXiv preprint arXiv:2407.07791 , year=

    Flooding spread of manipulated knowledge in llm-based multi-agent communities , author=. arXiv preprint arXiv:2407.07791 , year=

    arXiv

  269. [277]

    arXiv preprint arXiv:2410.07283 , year=

    Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems , author=. arXiv preprint arXiv:2410.07283 , year=

    arXiv

  270. [278]

    arXiv preprint arXiv:2409.11239 , year=

    Llm-as-a-judge & reward model: What they can and cannot do , author=. arXiv preprint arXiv:2409.11239 , year=

    arXiv

  271. [279]

    Autonomous agents and multi-agent systems , volume=

    A roadmap of agent research and development , author=. Autonomous agents and multi-agent systems , volume=. 1998 , publisher=

    1998

  272. [280]

    arXiv preprint arXiv:2311.17227 , year=

    War and peace (waragent): Large language model-based multi-agent simulation of world wars , author=. arXiv preprint arXiv:2311.17227 , year=

    arXiv

  273. [281]

    Artificial intelligence, learning and computation in economics and finance , pages=

    Agent-based computational economics: Overview and brief history , author=. Artificial intelligence, learning and computation in economics and finance , pages=. 2023 , publisher=

    2023

  274. [282]

    arXiv preprint arXiv:2406.20094 , year=

    Scaling Synthetic Data Creation with 1,000,000,000 Personas , author=. arXiv preprint arXiv:2406.20094 , year=

    arXiv

  275. [283]

    , author=

    Symphony: Towards Natural Language Query Answering over Multi-modal Data Lakes. , author=. CIDR , year=

  276. [284]

    arXiv preprint arXiv:2312.13010 , year=

    Agentcoder: Multi-agent-based code generation with iterative testing and optimisation , author=. arXiv preprint arXiv:2312.13010 , year=

    arXiv

  277. [285]

    arXiv preprint arXiv:2402.16823 , year=

    Language agents as optimizable graphs , author=. arXiv preprint arXiv:2402.16823 , year=

    arXiv

  278. [286]

    arXiv preprint arXiv:2412.04093 , year=

    Practical Considerations for Agentic LLM Systems , author=. arXiv preprint arXiv:2412.04093 , year=

    arXiv

  279. [287]

    arXiv preprint arXiv:2409.05746 , year=

    Llms will always hallucinate, and we need to live with this , author=. arXiv preprint arXiv:2409.05746 , year=

    arXiv

  280. [288]

    arXiv preprint arXiv:2402.04788 , year=

    Mllm-as-a-judge: Assessing multimodal llm-as-a-judge with vision-language benchmark , author=. arXiv preprint arXiv:2402.04788 , year=

    arXiv

  281. [289]

    arXiv preprint arXiv:2104.05938 , year=

    QMSum: A new benchmark for query-based multi-domain meeting summarization , author=. arXiv preprint arXiv:2104.05938 , year=

    arXiv

  282. [290]

    arXiv preprint arXiv:2412.03563 , year=

    From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agents , author=. arXiv preprint arXiv:2412.03563 , year=

    arXiv

  283. [291]

    The Thirty-eighth Annual Conference on Neural Information Processing Systems , year=

    Cooperate or collapse: Emergence of sustainable cooperation in a society of llm agents , author=. The Thirty-eighth Annual Conference on Neural Information Processing Systems , year=

  284. [292]

    arXiv preprint arXiv:2410.06108 , year=

    ConceptAgent: LLM-Driven Precondition Grounding and Tree Search for Robust Task Planning and Execution , author=. arXiv preprint arXiv:2410.06108 , year=

    arXiv

  285. [293]

    arXiv preprint arXiv:2310.02170 , year=

    Dynamic llm-agent network: An llm-agent collaboration framework with agent team optimization , author=. arXiv preprint arXiv:2310.02170 , year=

    arXiv

  286. [294]

    arXiv preprint arXiv:2403.16971 , year=

    Llm agent operating system , author=. arXiv preprint arXiv:2403.16971 , year=

    arXiv

  287. [295]

    arXiv preprint arXiv:2404.04834 , year=

    LLM-Based Multi-Agent Systems for Software Engineering: Vision and the Road Ahead , author=. arXiv preprint arXiv:2404.04834 , year=

    arXiv

  288. [296]

    Proceedings of the AAAI Symposium Series , volume=

    Memory Matters: The Need to Improve Long-Term Memory in LLM-Agents , author=. Proceedings of the AAAI Symposium Series , volume=

  289. [297]

    arXiv preprint arXiv:2309.11696 , year=

    Memory-augmented llm personalization with short-and long-term memory coordination , author=. arXiv preprint arXiv:2309.11696 , year=

    arXiv

  290. [298]

    arXiv preprint arXiv:2402.05733 , year=

    Timearena: Shaping efficient multitasking language agents in a time-aware simulation , author=. arXiv preprint arXiv:2402.05733 , year=

    arXiv

  291. [299]

    arXiv preprint arXiv:2305.05252 , year=

    Distilling script knowledge from large language models for constrained language planning , author=. arXiv preprint arXiv:2305.05252 , year=

    arXiv

  292. [300]

    arXiv preprint arXiv:2312.11805 , year=

    Gemini: a family of highly capable multimodal models , author=. arXiv preprint arXiv:2312.11805 , year=

    arXiv

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.