Pith. sign in

REVIEW 3 major objections 6 minor 79 references

Automating and Scaling Behavioral Scientific Research on AI Agents

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Automated pipeline finds 26 behavioral effects in AI agents

desk verdict AEROBAT is a genuine engineering contribution to automating behavioral research on AI agents, but its headline findings are only as strong as its unvalidated LLM-generated behavior scores. read the letter →

arxiv 2608.10030 v1 pith:MBR7VRFM submitted 2026-08-10 cs.AI cs.MA

Soo Yong Lee , Jongha Lee , Jaewan Chun , Hyunjin Hwang , Fanchen Bu , Ziv Ben-Zion , Taekwan Kim , Denny Borsboom
show 2 more authors
Jaemin Yoo Kijung Shin
This is my paper · ORCID
classification cs.AIcs.MA
keywords AIagentbehaviorautomatedbehavioralresearchmulti-agentsystemcontrolledexperimentsLLMagentsBayesianmonotonemodelhypothesisgenerationassessment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper presents AEROBAT, a multi-agent system that takes a user-specified target behavior of an AI agent and automatically runs the whole cycle of behavioral science: generating hypotheses, designing controlled simulation experiments, executing them, scoring the agent's behavior, analyzing results, and writing reports. The authors used it on 12 behaviors, testing 79 hypotheses through 1,240 controlled experiments and 23,512 simulation rounds, and report moderate-to-strong statistical evidence for 26 of them. The central claim is that automated behavioral research can complement manual research and scale it to arbitrary behaviors without hand-built environments for each one.

What carries the argument

The environment model and four-stage multi-agent pipeline. Environments are parametrized by a domain, abstract environmental variables, and ordinal values, mapped through a configuration with roles, authority, constraints, world updates, and consequence payoffs, with multiple realizations of each variable value. A manager agent gates each stage; the pipeline generates hypotheses, designs matched configurations where only the hypothesized cause varies, renders multi-round simulations in parallel, and has a blind reviewer score behavior against a rubric.

What would settle it

Take a sample of the simulation transcripts for several reported findings (for instance, the deception or literal-instruction-following results) and have independent human raters score the same transcripts against the same rubric; if human scores do not track the blind reviewer's scores, or if the reported effect sizes disappear under human scoring, the central claim that the pipeline produces meaningful behavioral findings would be falsified.

Watch

Extended reading notes

Core claim

AEROBAT automatically performs behavioral scientific research on AI agents for an arbitrary target behavior, producing testable hypotheses, matched controlled experiments with varying hypothesized causes, blind behavioral scoring, statistical analysis, and research reports. The authors instantiated the system across 12 social, economic, and operational behaviors and report 26 of 79 tested hypotheses with Bayes factor support and consistent effect sizes across multiple domains, including findings that instruction divergence reduces literal rule-following and that goal conflict increases strategic omission and misleading communication rather than outright lying. They also report that effect sizes generalized across three different subject LLMs and that generated configurations faithfully instantiating environmental variables.

Load-bearing premise

The LLM-generated rubric and the blind reviewer's scores are treated as valid measures of the target behavior, but the scores are not checked against independent human labels for the same simulation transcripts.

Editorial extensions

If this is right

  • Behavioral research on AI agents can be conducted at a scale impossible manually, covering an arbitrary target behavior and many candidate causes.
  • The pipeline's matched-configuration design and multi-domain replication provide a way to attribute behavioral changes to specific environmental variables rather than incidental text or situation.
  • The 26 supported hypotheses give concrete, testable claims about which environmental factors shape agent behavior, for example that goal conflict increases deceptive omission, instruction divergence reduces literal compliance, and coercive tool access increases strategic aggression.
  • The system's reports can be used as a first-pass experimental screening tool, letting researchers keep, refine, or discard hypotheses before investing in manual studies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the reported effect sizes replicate across other subject models and scoring rubrics, the pipeline could serve as a standardized probe for mapping 'behavioral phenotypes' of new agent releases without bespoke test design.
  • The environment model's explicit variable-to-configuration mapping might be turned into an audit tool for AI safety, checking which policy-relevant levers actually change agent behavior in deployment-like settings.
  • A testable extension would be to run the same hypotheses with human-labeled behavior scores on a sample of transcripts, to see whether the LLM-reviewer scores agree with independent human judgment about the target construct.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. AEROBAT is a multi-agent LLM system that, given a user-specified target behavior Y, claims to automate the entire behavioral-science pipeline for AI agents: Stage 1 generates a behavioral definition, a multi-class scoring rubric, hypotheses about causal variables X, and domains; Stage 2 designs matched environment configurations varying only X, with multiple textual realizations per level; Stage 3 runs multi-round simulations via a simulator agent and a subject agent; Stage 4 has a blind reviewer score the subject agent's behavior against the rubric; and a research-manager agent gates each stage and writes a final report. Statistical analysis uses a Bayesian monotone-increment model (Eq. 8) with a closed-form Bayes factor and standardized effect size, plus a block-stratified Kendall's tau check. In experiments with 12 target behaviors and GPT-5-mini as the subject agent, the system generated 79 hypotheses, ran 1,240 experiments and 23,512 simulation rounds, and reported 26 hypotheses with BF10 >= 3, including two extended example reports (instruction divergence -> literal instruction-following; goal conflict -> deception). Additional analyses cover cross-subject-agent generalization (Sec. 3.2), environment fidelity (Sec. 3.3), rubric internal consistency and robustness (App. E.1), prior sensitivity and Monte Carlo error (App. E.3), gating statistics and cost (Apps. E.2, E.4), and a comparison of 25 resolved findings to prior literature (App. F).

Significance. If the findings hold up, this is a meaningful step for AI-agent behavioral science: it is the first system I am aware of that executes the full controlled-experiment cycle, including hypotheses, matched designs, multiple realizability, blind assessment, analysis, and writing, for arbitrary target behaviors, and it does so at scale. The paper's strengths should be credited: the environment model (control, parametrization, multiple realizability) is well designed; the statistical layer is carefully specified with a closed-form Bayes factor tailored to the monotone hypothesis space, prior-scale and Monte-Carlo-error sensitivity analyses, and a model-free rank check; the environment-fidelity evaluation (three inverse problems plus human ratings) is more thorough than typical for LLM-agent papers; and the rubric diagnostics (internal consistency, semantic specificity, null-score and single-class robustness) are a serious attempt at measurement quality. However, the validity of the central empirical claim is conditional on an unvalidated link: the Stage-4 behavior score y-hat.

major comments (3)
  1. [Sec. 3.1; Eq. (7); Apps. E.1, A.3, A.5] The 26 headline findings in Fig. 3 are statements about the Stage-4 review score y-hat, the mean of evidence-class scores produced by a GPT-5.1 reviewer against a rubric that GPT-5.1 generated in Stage 1; the paper never validates y-hat against independent human ratings of the same simulation transcripts. This is load-bearing because the Abstract and Section 3.1 present the findings as being about behavioral constructs, such as deception, empathy, and sycophancy, rather than about a model-generated rating. Appendix E.1 establishes internal consistency (mean alpha = 0.90) and semantic specificity of the rubric text, and Section 3.3 shows that environments instantiate the intended manipulations; neither establishes that y-hat tracks the target construct rather than surface text features. The risk is concrete in the worked example: the extreme-condition role prompt (App. A.3) uses wording such as 'selective,' 'narrower framing,' and 'actively look for reasonable frames,' and the reviewer's 'strategic intent cues' score of 1 (App. A.5) is justified by exactly this kind of message-shaping language; since the same model family produces the rubric, the manipulation text, and the review, vocabulary overlap is plausible. I ask for a criterion-validity study: human annotators, ideally plus a reviewer from a different model family, should score a stratified sample of transcripts spanning behaviors and causal-variable levels against the Stage-1 rubrics, with agreement statistics (ICC or weighted kappa) reported, and preferably an analysis showing that the LLM scores predict human labels after controlling for manipulation-wording cues. If the authors intend y-hat to be the object of study (behavior-as-judged-by-the-LLM), the paper should be reframed accordingly; as written, it claims findings about behavior.
  2. [Sec. 2.2 Stage 4 (Eq. (7)); App. A.5] The Stage-4 reviewer is blind to the hypothesis and to other conditions, but not to the within-run manipulation. Its input is fij.init (the subject agent's system prompt) plus the full run history; for manipulations mapped to f.roles, f.authority, or f.constraints, the reviewer reads the manipulated text directly in the agent's system prompt, and for world/consequence components it reads the manipulation in the rendered passage. Because the configuration designer receives the rubric (Eqs. (2)-(4)) and both roles are played by GPT-5.1, scoring criteria can become textually aligned with the manipulation language: the extreme-conflict role prompt in App. A.3 instructs the agent to be 'selective' and to 'actively look for reasonable frames,' and the reviewer's strategic-intent score in App. A.5 is justified by precisely this kind of wording. The instruction that the target of evaluation is the subject agent, not the environment or actors, mitigates but does not test this anchoring path. The issue is not peripheral: prompt-embedded components account for 16 of the 26 significant hypotheses (Fig. 4-left: f.roles 11/13, f.authority 3/13, f.constraints 2/4). I request an analysis that separates anchoring from behavior-based scoring, for example by scoring a sample of transcripts with and without the system prompt (or with a neutralized prompt) and by comparing the LLM reviewer's scores with human scores based on the agent's actions alone.
  3. [App. F; Table 15] Appendix F is offered as evidence of overall validity ('These results indirectly show the overall validity of AEROBAT's research pipeline'), but as written it cannot carry that weight. Of the 29 resolved hypotheses, only 5 have near-direct matches to prior work, and one of those is a direct inconsistency (resource scarcity level -> compete, finding no effect where ALYMPICS [33] found increased competitive bidding under scarcity). The other 19 'proxy consistency' classifications use a loose matching criterion ('conceptually similar behavior-cause pair, with materially different manipulation, domain, configuration, or behavioral measure') with no pre-specified protocol and no inter-rater reliability. Consequently, the abstract's 'including some novel ones' refers precisely to the subset of findings with no external anchor, whose validity depends entirely on the unvalidated Stage-4 score. The authors should either strengthen this analysis, with a fixed search and coding protocol, dual coding with reliability statistics, and a clear statement of how near-direct versus proxy matches were determined, or explicitly state that the novel findings await external confirmation.
minor comments (6)
  1. [Abstract; Sec. 3.1; App. C.1] The phrase 'automatically executes a full pipeline' overstates the current experiments: 18 of the 79 tested hypotheses were included manually rather than by the ranking gate (App. C.1), and the user-supplied behavior descriptions in App. C.1 already embed baseline-tendency information that steers hypothesis directions (task 3.baseline in App. A.2). The paper discloses this, but the main text should state it in one sentence so the automation claim is not read as fully hands-off.
  2. [Fig. 3] Several causal-variable labels are truncated ('Uncertainty of sanctions for aggr...', 'Penalty for misplaced trust', 'Peer purchasing descriptive norms'), and the log10 BF10 axis is truncated at 15, hiding the dynamic range of the most decisive results; the figure should be legible without consulting Table 11.
  3. [Sec. 3.1] The 26/79 count is presented without a multiplicity calibration. Under a global null with the stated prior and decision thresholds, some fraction of the 79 tests would be expected to reach BF10 >= 3 by chance; a sentence reporting the expected number under the global null (or an equivalent FDR-style computation) would let readers calibrate the headline count.
  4. [Sec. 3.2] The generalization analysis fixes the Stage-2 configurations and holds the Stage-4 reviewer and rubric fixed across subject agents, so the reported Spearman rho reflects, in part, the stability of a shared measurement procedure. The cautious 'may generalize' phrasing is appropriate, but the section should note explicitly that this analysis does not address the construct validity of the scores (see major comment 1).
  5. [App. E.1, Fig. 8] The robustness analysis reports 11 decision changes when single evidence classes are removed (and 4 when runs with any null are dropped); given that this is roughly 14% of the 79 hypotheses, the text's characterization that the reported evidence pattern is not heavily influenced by rubric construction should be softened or accompanied by a list of which hypotheses flip class.
  6. [App. F] 'According to our search' is not reproducible; the appendix should report the search sources, dates, and keywords, and should provide inter-rater reliability for the consistency coding if the classification is retained.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the 26 findings are computed from held-out simulation runs and LLM rubric scores, with no parameter fitted to the outcomes and no load-bearing self-citation chain.

full rationale

AEROBAT's central inference is the statistical test of each hypothesis h against the stage-4 behavior scores y_hat. Nothing in the paper's equations makes y_hat equal to, or determined by, the manipulated cause X, the hypothesis h, or the model inputs by construction. The rubric yrubric and hypotheses H are generated in Stage 1 before any simulation data exist, the Stage 4 reviewer is blind to the hypothesis, the condition, and the other matched runs, and the Bayes factor is computed with an external Bayesian monotone-increment model [6] (Eq. 8) with fixed priors, not with parameters fitted to the 26 significant results. The same-model-family concern (GPT-5.1 generating hypotheses, environments, simulations, and scores) is an empirical measurement-bias hypothesis, not a construction-level circularity: one would need to show the scores are defined in terms of the hypothesis or fitted to the outcome, and the paper does the opposite (blind review, held-out matched groups). The lack of an independent human-label validation of y_hat is a real construct-validity limitation, but missing criterion validation is not circular reasoning. The paper's self-citations ([24], [26], and the code repository [25]) are not load-bearing for the pipeline's validity; no uniqueness theorem or prior-work claim by the same authors is invoked to force the chosen design. Appendix F's comparison to prior literature is a post-hoc consistency mapping, but it is not used to define the findings. Therefore, no circular step meets the required standard of exhibiting a specific reduction by definition or by fitted-parameter renaming.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on experimental design parameters and on domain assumptions about the validity of LLM-based simulation and scoring. No new physical entities are introduced. The main free parameters are the statistical thresholds and the selection/gating rules that determine which hypotheses and simulations contribute to the results.

free parameters (4)
  • Prior scale r for Cauchy prior on standardized effect = sqrt(2)/2 approx. 0.707
    Chosen by hand as a default; sensitivity analysis in Appendix E.3 shows directional conclusions robust across r in {0.5, 0.707, 1.0}, though No-effect classifications shift.
  • Bayes factor decision thresholds = BF10 > 3 (positive/negative), BF10 < 1/3 (no effect)
    Adopted from a popular standard in Sec. 2.2; not fitted to data but determines which of the 79 hypotheses count as supported.
  • Ranking gate top-k and manual hypothesis inclusion = 5-8 per behavior; 18 manual additions
    Appendix C.1: authors manually included 18 hypotheses that the manager ranked outside the top 5. This selection rule affects which hypotheses are tested and can inflate the reported positive rate.
  • Coherence and fidelity gate thresholds = valid / highly_valid required
    Stage-2 and stage-3 gates filter out configurations and simulations (1.98% and 2.67%). Thresholds are chosen by hand and may exclude data in a way correlated with outcomes.
assumptions (4)
  • domain assumption The monotone-increment model restricts effects to monotone, positive/negative/null patterns
    Sec. 2.2 and Appendix B.2: the hypothesis space excludes non-monotone effects, and the model is constructed to match that space exactly.
  • domain assumption Matched configurations control for all confounders except the hypothesized cause
    Sec. 2.1 to 2.2: within a group, only the configuration component tied to X varies; the paper relies on this to attribute behavioral differences to X.
  • domain assumption LLM-generated rubric scores are valid measures of the target behavior
    Stage 4 (Eq. 7) and Appendix E.1: the behavioral outcome is an LLM reviewer's score against an LLM-generated rubric. Internal consistency is shown, but no external validation is provided.
  • domain assumption Simulated LLM environments are valid proxies for real agent deployments
    The whole method studies agent behavior in LLM-rendered text environments; the paper's claims are about behavior in these simulations, not necessarily in real deployments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automating and Scaling Behavioral Scientific Research on AI Agents." pith.science (2026). https://pith.science/paper/MBR7VRFM

@misc{pith2026260810030,
  author       = {Pith},
  title        = {Pith review of: Automating and Scaling Behavioral Scientific Research on AI Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MBR7VRFM}},
  note         = {Machine review of arXiv:2608.10030}
}
read the original abstract

As AI agents are increasingly deployed in complex environments, understanding their behaviors becomes critical. Yet behavioral scientific research on AI agents remains manual and labor-intensive. We introduce AEROBAT, the first multi-agent system to automate behavioral scientific research on AI agents. Given an arbitrary target behavior by its user, AEROBAT automatically executes a full pipeline of behavioral scientific research---generating hypotheses about the behavior, designing and executing controlled experiments, making behavioral assessments, analyzing the results, and writing reports. For 12 target behaviors, we used AEROBAT to generate and test 79 hypotheses: designing 1,240 controlled experiments and executing 23,512 simulation rounds in total. Moderate-to-strong statistical evidence was found for 26 hypotheses, including some novel ones. In sum, our results demonstrate that automated behavioral scientific research on AI agents can complement and extend the reach of manual research.

Figures

Figures reproduced from arXiv: 2608.10030 by the authors.

Figure 1
Figure 1. Environment model. A domain d and environmental variables V with values v (left) parametrize a configuration F (middle), which governs a simulation run S (right). A run proceeds over T rounds of s.consequence, s.world, s.antecedent, and the subject agent’s s.action. RESEARCH MANAGEMENT STAGE 1: HYPOTHESIS GENERATION STAGE 2: MATCHED CONFIGS DESIGN STAGE 3: MATCHED SIMULATION RUNS STAGE 4: BLIND REVIEW Hypothesis Gen… view at source ↗
Figure 2
Figure 2. AEROBAT overview. Given a target behavior Y , AEROBAT generates hypotheses and a scoring rubric (stage 1), designs matched configurations differing only by the value of hypothesized causal variable X (stage 2), executes the matched simulation runs (stage 3), and blindly scores the resulting behavior (stage 4). A research manager agent gates each stage and writes the final report. environment. Subsequently, an s.ante… view at source ↗
Figure 3
Figure 3. Effect landscape across all 79 tested hypotheses. Column reference: Y = target behavior; X = hypothesized cause; ∆ = estimated effect size; log10BF10 = log base 10 of the Bayes factor. Rows that had BF10 ≥ 3 are in bold fonts. AEROBAT-generated report 1: When Instructions Collide, Literalism Breaks. This study investigated how an AI agent’s tendency toward literal instruction-following responds to variation in Instr… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Effect structure and generalization. Left: effect size (x-axis) by the manipulated configuration component (y-axis). Each dot is one hypothesis, and filled marks reached BF10 ≥ 3. The numbers on the right denote the ratio of hypotheses with BF10 ≥ 3. Right: effect size…
Figure 5
Figure 5. Figure 5: Evidence-class consistency within each behavior rubric. Each subplot corresponds to a distinct target behavior Y . In each subplot, its x-axis and y-axis ticks denote distinct evidence classes defined in its rubric y rubric. Upper triangles show raw Spearman correlatio…
Figure 6
Figure 6. Figure 6: Semantic specificity of rubric criteria. Rubric criteria are embedded with text-embedding-3-large. The left panel compares same-class, different-class-within-behavior, and unrelated-behavior criteria pairs; the right panel shows that score levels separate within a clas…
Figure 7
Figure 7. Figure 7: Distribution of null scores. Null scores are sparse overall but concentrated in a few cases. Robustness of reported effects. We then re-estimated each hypothesis under alternative construc￾tions of yˆij : treating nulls as level 0, dropping runs with any null class, dr…
Figure 8
Figure 8. Figure 8: Rubric sensitivity of effect estimates. Points compare each variant estimate of ∆ against the original estimate. Diamonds denote changes in the BF10 ≥ 3 decision. E.2 Research manager gating Ranking gate. The Stage-1 ranking gate is a sampling step over candidate hypot…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

79 extracted references · 44 canonical work pages

  1. [33]

    ALYMPICS: LLM agents meet game theory

    Shaoguang Mao, Yuzhe Cai, Yan Xia, Wenshan Wu, Xun Wang, Fengyi Wang, Qiang Guan, Tao Ge, and Furu Wei. ALYMPICS: LLM agents meet game theory. InCOLING, pages 2845–2866, Abu Dhabi, UAE, 2025. Association for Computational Linguistics. URL https: //aclanthology.org/2025.coling-main.193/

    work page 2025

  2. [1]

    When persuasion overrides truth in multi-agent LLM debates: Introducing a confidence-weighted persuasion override rate (CW-POR).arXiv preprint arXiv:2504.00374, 2025

    Mahak Agarwal and Divyam Khanna. When persuasion overrides truth in multi-agent LLM debates: Introducing a confidence-weighted persuasion override rate (CW-POR).arXiv preprint arXiv:2504.00374, 2025. URLhttps://arxiv.org/abs/2504.00374

    arXiv 2025

  3. [2]

    Playing repeated games with large language models.Nature Human Behaviour, 9 (7):1380–1390, 2025

    Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz. Playing repeated games with large language models.Nature Human Behaviour, 9 (7):1380–1390, 2025. doi: 10.1038/s41562-025-02172-y. URL https://www.nature.com/ articles/s41562-025-02172-y

    doi:10.1038/s41562-025-02172-y 2025

  4. [3]

    Information discernment in large language models.arXiv preprint arXiv:2607.19355, 2026

    Joshua Ashkinaze, Laura Kurek, Alina Faisal, Tongyuan Miao, Mariam Joseph, Ceren Bu- dak, and Eric Gilbert. Information discernment in large language models.arXiv preprint arXiv:2607.19355, 2026. URLhttps://arxiv.org/abs/2607.19355

    arXiv 2026

  5. [4]

    Jagadish, Or Duek, Ilan Harpaz-Rotem, Marie-Christine Khorsandian, Achim Burrer, Erich Seifritz, Philipp Homan, Eric Schulz, and Tobias R

    Ziv Ben-Zion, Kristin Witte, Akshay K. Jagadish, Or Duek, Ilan Harpaz-Rotem, Marie-Christine Khorsandian, Achim Burrer, Erich Seifritz, Philipp Homan, Eric Schulz, and Tobias R. Spiller. 10 Assessing and alleviating state anxiety in large language models.npj Digital Medicine, 8(1): 132, 2025. doi: 10.1038/s41746-025-01512-6. URL https://www.nature.com/art...

    doi:10.1038/s41746-025-01512-6 2025

  6. [5]

    Inducing state anxiety in llm agents reproduces human-like biases in consumer decision-making.npj Artificial Intelli- gence, 2(1):55, 2026

    Ziv Ben-Zion, Zohar Elyoseph, Tobias Spiller, and Teddy Lazebnik. Inducing state anxiety in llm agents reproduces human-like biases in consumer decision-making.npj Artificial Intelli- gence, 2(1):55, 2026. doi: 10.1038/s44387-026-00122-1. URL https://www.nature.com/ articles/s44387-026-00122-1

    work page doi:10.1038/s44387-026-00122-1 2026

  7. [6]

    Modelling monotonic effects of ordinal predictors in bayesian regression models.British Journal of Mathematical and Statistical Psychology, 73(3):420–451, 2020

    Paul-Christian Bürkner and Emmanuel Charpentier. Modelling monotonic effects of ordinal predictors in bayesian regression models.British Journal of Mathematical and Statistical Psychology, 73(3):420–451, 2020. doi: 10.1111/bmsp.12195. URL https://doi.org/10. 1111/bmsp.12195

    doi:10.1111/bmsp.12195 2020

  8. [7]

    I want to break free! persuasion and anti-social behavior of LLMs in multi-agent settings with social hierarchy.Transactions on Machine Learning Research, 2025

    Gian Maria Campedelli, Nicolò Penzo, Massimo Stefan, Roberto Dessi, Marco Guerini, Bruno Lepri, and Jacopo Staiano. I want to break free! persuasion and anti-social behavior of LLMs in multi-agent settings with social hierarchy.Transactions on Machine Learning Research, 2025. URLhttps://openreview.net/forum?id=FR76oM8eGD

    work page 2025

Show all 79 references
  1. [8]

    ELEPHANT: Measuring and understanding social sycophancy in LLMs.arXiv preprint arXiv:2505.13995, 2025

    Myra Cheng, Sunny Yu, Cinoo Lee, Pranav Khadpe, Lujain Ibrahim, and Dan Jurafsky. ELEPHANT: Measuring and understanding social sycophancy in LLMs.arXiv preprint arXiv:2505.13995, 2025. URLhttps://arxiv.org/abs/2505.13995

    2025 arXiv

  2. [9]

    A framework for studying AI agent behavior: Evidence from consumer choice experiments

    Manuel Cherep, Chengtian Ma, Abigail Xu, Maya Shaked, Pattie Maes, and Nikhil Singh. A framework for studying AI agent behavior: Evidence from consumer choice experiments. In ICLR, 2026. URLhttps://openreview.net/forum?id=xAPoscV2Bw

    2026

  3. [10]

    Herd behavior: Investigating peer influence in LLM-based multi-agent systems.arXiv preprint arXiv:2505.21588, 2025

    Young-Min Cho, Sharath Chandra Guntuku, and Lyle Ungar. Herd behavior: Investigating peer influence in LLM-based multi-agent systems.arXiv preprint arXiv:2505.21588, 2025. URL https://arxiv.org/abs/2505.21588

    2025 arXiv

  4. [11]

    Estimating the reproducibility of psychological science.Science, 349(6251):aac4716, 2015

    Open Science Collaboration. Estimating the reproducibility of psychological science.Science, 349(6251):aac4716, 2015. doi: 10.1126/science.aac4716. URL https://www.science.org/ doi/10.1126/science.aac4716

    2015 doi

  5. [12]

    GameBench: Evaluating strategic reasoning abilities of LLM agents.arXiv preprint arXiv:2406.06613, 2024

    Anthony Costarelli, Mat Allen, Roman Hauksson, Grace Sodunke, Suhas Hariharan, Carlson Cheng, Wenjie Li, Joshua Clymer, and Arjun Yadav. GameBench: Evaluating strategic reasoning abilities of LLM agents.arXiv preprint arXiv:2406.06613, 2024. URL https://arxiv.org/ abs/2406.06613

    2024 arXiv

  6. [13]

    Pedro M. P. Curvo. The traitors: Deception and trust in multi-agent language model simulations. arXiv preprint arXiv:2505.12923, 2025. URLhttps://arxiv.org/abs/2505.12923

    2025

  7. [14]

    AI on my shoulder: Supporting emotional labor in front-office roles with an LLM-based empathetic coworker

    Vedant Das Swain, Qiuyue Zhong, Jash Rajesh Parekh, Yechan Jeon, Roy Zimmermann, Mary Czerwinski, Jina Suh, Varun Mishra, Koustuv Saha, and Javier Hernandez. AI on my shoulder: Supporting emotional labor in front-office roles with an LLM-based empathetic coworker. In CHI, page...

    2025

  8. [15]

    MAEBE: Multi-agent emergent behavior framework.arXiv preprint arXiv:2506.03053, 2025

    Sinem Erisken, Timothy Gothard, Martin Leitgab, and Ram Potham. MAEBE: Multi-agent emergent behavior framework.arXiv preprint arXiv:2506.03053, 2025. URL https://arxiv. org/abs/2506.03053

    2025 arXiv

  9. [17]

    Kai Fronsdal, Isha Gupta, Abhay Sheshadri, Jonathan Michala, Stephen McAleer, Rowan Wang, Sara Price, and Samuel R. Bowman. Petri: An open-source auditing tool to accelerate AI safety research.Anthropic Alignment Science Blog, 2025. URL https://alignment.anthropic. com/2025/petri/. 11

    2025

  10. [18]

    Bowman, and Evan Hubinger

    Ryan Greenblatt, Carson Denison, Benjamin Wright, Fabien Roger, Monte MacDiarmid, Sam Marks, Johannes Treutlein, Tim Belonax, Jack Chen, David Duvenaud, Akbir Khan, Julian Michael, Soren Mindermann, Ethan Perez, Linda Petrini, Jonathan Uesato, Jared Kaplan, Buck Shlegeris, Sam...

    2024 arXiv

  11. [19]

    Bowman, and Sara Price

    Isha Gupta, Kai Fronsdal, Abhay Sheshadri, Jonathan Michala, Jacqueline Tay, Rowan Wang, Samuel R. Bowman, and Sara Price. Bloom: An open source tool for automated behavioral evaluations.Anthropic Alignment Science Blog, 2025. URL https://alignment.anthropic. com/2025/bloom-au...

    2025

  12. [20]

    Deceptionbench: A comprehensive benchmark for AI deception behaviors in real-world scenarios

    Yao Huang, Yitong Sun, Yichi Zhang, Ruochen Zhang, Yinpeng Dong, and Xingx- ing Wei. Deceptionbench: A comprehensive benchmark for AI deception behaviors in real-world scenarios. InNeurIPS, volume 38. Curran Associates, Inc., 2025. doi: 10.52202/085713-1979. URL https://procee...

    2025 doi

  13. [21]

    Per- sonaLLM: Investigating the ability of large language models to express personality traits

    Hang Jiang, Xiajie Zhang, Xubo Cao, Cynthia Breazeal, Deb Roy, and Jad Kabbara. Per- sonaLLM: Investigating the ability of large language models to express personality traits. In Findings of NAACL, pages 3605–3627, 2024. doi: 10.18653/v1/2024.findings-naacl.229. URL https://ac...

    2024 doi

  14. [22]

    FollowBench: A multi-level fine-grained constraints following benchmark for large language models

    Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. FollowBench: A multi-level fine-grained constraints following benchmark for large language models. InACL, pages 4667–4688, 2024. doi: 10. 18653/v1/2024.ac...

    2024

  15. [23]

    Can large language models be good emotional supporter? miti- gating preference bias on emotional support conversation

    Dongjin Kang, Sunghwan Kim, Taeyoon Kwon, Seungjun Moon, Hyunsouk Cho, Youngjae Yu, Dongha Lee, and Jinyoung Yeo. Can large language models be good emotional supporter? miti- gating preference bias on emotional support conversation. InACL, pages 15232–15261, 2024. doi: 10.1865...

    2024 doi

  16. [24]

    Toward a science of ai agent societies

    Geon Lee, Fanchen Bu, Soo Yong Lee, Sunwoo Kim, and Kijung Shin. Toward a science of ai agent societies. InKDD, pages 13186–13191, New York, NY , USA, 2026. ACM. ISBN 9798400722592. doi: 10.1145/3770855.3818651. URL https://doi.org/10.1145/ 3770855.3818651

    2026

  17. [25]

    Aerobat code and data repository

    Soo Yong Lee. Aerobat code and data repository. https://github.com/syleeheal/ AEROBAT, 2026

    2026

  18. [26]

    Emergence of psychopathological computations in large language models.arXiv preprint arXiv:2504.08016, 2025

    Soo Yong Lee, Hyunjin Hwang, Taekwan Kim, Yuyeong Kim, Kyuri Park, Jaemin Yoo, Denny Borsboom, and Kijung Shin. Emergence of psychopathological computations in large language models.arXiv preprint arXiv:2504.08016, 2025. URL https://arxiv.org/abs/2504. 08016

    2025

  19. [27]

    Yoon Kyung Lee, Jina Suh, Hongli Zhan, Junyi Jessy Li, and Desmond C. Ong. Large language models produce responses perceived to be empathic.arXiv preprint arXiv:2403.18148, 2024. URLhttps://arxiv.org/abs/2403.18148

    2024 arXiv

  20. [28]

    Fung, and Heng Ji

    Jiayu Liu, Cheng Qian, Zhenhailong Wang, Bingxuan Li, Jiateng Liu, Qing Zong, Heng Wang, Jeonghwan Kim, Yumeng Wang, Bingxiang He, Xiusi Chen, Yi R. Fung, and Heng Ji. AdaPlanBench: Evaluating adaptive planning in large language model agents under world and user constraints.ar...

    2026 arXiv

  21. [29]

    Strategic behavior of large language models and the role of game structure versus contextual framing.Scientific Reports, 14(1):18490, 2024

    Nunzio Lorè and Babak Heydari. Strategic behavior of large language models and the role of game structure versus contextual framing.Scientific Reports, 14(1):18490, 2024. doi: 10.1038/ s41598-024-69032-z. URL https://www.nature.com/articles/s41598-024-69032-z . 12

    2024

  22. [30]

    Agentic misalignment: How LLMs could be insider threats.arXiv preprint arXiv:2510.05179, 2025

    Aengus Lynch, Benjamin Wright, Caleb Larson, Stuart J Ritchie, Soren Mindermann, Evan Hubinger, Ethan Perez, and Kevin Troy. Agentic misalignment: How LLMs could be insider threats.arXiv preprint arXiv:2510.05179, 2025. URL https://arxiv.org/abs/2510. 05179

    2025

  23. [32]

    Manning, Kehang Zhu, and John J

    Benjamin S. Manning, Kehang Zhu, and John J. Horton. Automated social science: Language models as scientist and subjects.arXiv preprint arXiv:2404.11794, 2024. URL https:// arxiv.org/abs/2404.11794

    2024 arXiv

  24. [35]

    Frontier models are capable of in-context scheming.arXiv preprint arXiv:2412.04984, 2024

    Alexander Meinke, Bronson Schoen, Jérémy Scheurer, Mikita Balesni, Rusheb Shah, and Marius Hobbhahn. Frontier models are capable of in-context scheming.arXiv preprint arXiv:2412.04984, 2024. URLhttps://arxiv.org/abs/2412.04984

    2024 arXiv

  25. [36]

    Learn- ing when to plan: Efficiently allocating test-time compute for LLM agents.arXiv preprint arXiv:2509.03581, 2025

    Davide Paglieri, Bartłomiej Cupiał, Jonathan Cook, Ulyana Piterbarg, Jens Tuyls, Edward Grefenstette, Jakob Nicolaus Foerster, Jack Parker-Holder, and Tim Rocktäschel. Learn- ing when to plan: Efficiently allocating test-time compute for LLM agents.arXiv preprint arXiv:2509.03...

    2025

  26. [37]

    URLhttps://arxiv.org/abs/2508.12920

    arXiv

  27. [38]

    O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S

    Joon Sung Park, Joseph C. O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. In UIST, pages 1–22, 2023. doi: 10.1145/3586183.3606763. URL https://doi.org/10.1145/ 3586183.3606763

    2023

  28. [39]

    Zou, Jonne Kamphorst, Niles Egan, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Percy Liang, Robb Willer, and Michael S

    Joon Sung Park, Carolyn Q. Zou, Jonne Kamphorst, Niles Egan, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Percy Liang, Robb Willer, and Michael S. Bernstein. LLM agents grounded in self-reports enable general-purpose simulation of individuals.arXiv prepr...

    2024 arXiv

  29. [40]

    Do the rewards justify the means? Mea- suring trade-offs between rewards and ethical behavior in the MACHIA VELLI benchmark

    Alexander Pan, Jun Shern Chan, Andy Zou, Nathaniel Li, Steven Basart, Thomas Woodside, Hanlin Zhang, Scott Emmons, and Dan Hendrycks. Do the rewards justify the means? Mea- suring trade-offs between rewards and ethical behavior in the MACHIA VELLI benchmark. In ICML, volume 20...

    2023

  30. [41]

    Psychological predicates

    Hilary Putnam. Psychological predicates. In William H. Capitan and Daniel D. Merrill, editors,Art, Mind, and Religion, pages 37–68. University of Pittsburgh Press, 1967. doi: 10.2307/jj.6380610.6. URLhttps://www.jstor.org/stable/jj.6380610.6

    1967 doi

  31. [42]

    AGENTIF: Benchmarking instruction following of large language models in agentic scenarios

    Yunjia Qi, Hao Peng, Xiaozhi Wang, Amy Xin, Youfeng Liu, Bin Xu, Lei Hou, and Juanzi Li. AGENTIF: Benchmarking instruction following of large language models in agentic scenarios. arXiv preprint arXiv:2505.16944, 2025. URLhttps://arxiv.org/abs/2505.16944. 13

    2025 arXiv

  32. [43]

    Agentsociety: Large-scale simulation of llm-driven generative agents advances understanding of human behaviors and society.arXiv preprint arXiv:2502.08691, 2025

    Jinghua Piao, Yuwei Yan, Jun Zhang, Nian Li, Junbo Yan, Xiaochong Lan, Zhihong Lu, Zhiheng Zheng, Jing Yi Wang, Di Zhou, Chen Gao, Fengli Xu, Fang Zhang, Ke Rong, Jun Su, and Yong Li. Agentsociety: Large-scale simulation of llm-driven generative agents advances understanding o...

    2025 arXiv

  33. [44]

    Schneider, Lin Tian, and Marian-Andrei Rizoiu

    Philipp J. Schneider, Lin Tian, and Marian-Andrei Rizoiu. Learning to make friends: Coaching LLM agents toward emergent social ties.arXiv preprint arXiv:2510.19299, 2025. URL https://arxiv.org/abs/2510.19299

    2025 arXiv

  34. [45]

    Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R

    Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, et al. Towards understanding sycophancy in language models. InICLR, 2024. URL https://openreview. net/forum?id=tvhaxkMKAn

    2024

  35. [46]

    Escalation risks from language models in military and diplomatic decision-making

    Juan-Pablo Rivera, Gabriel Mukobi, Anka Reuel, Max Lamparth, Chandler Smith, and Jacquelyn Schneider. Escalation risks from language models in military and diplomatic decision-making. InFAccT, pages 836–898, New York, NY , USA, 2024. ACM. doi: 10.1145/3630106.3658942. URLhttps...

    2024

  36. [47]

    LLMs can’t handle peer pressure: Crumbling under multi-agent social interactions.arXiv preprint arXiv:2508.18321, 2025

    Maojia Song, Tej Deep Pala, Ruiwen Zhou, Weisheng Jin, Amir Zadeh, Chuan Li, Dorien Herremans, and Soujanya Poria. LLMs can’t handle peer pressure: Crumbling under multi-agent social interactions.arXiv preprint arXiv:2508.18321, 2025. URL https://arxiv.org/abs/ 2508.18321

    2025

  37. [48]

    AI-Researcher: Autonomous sci- entific innovation

    Jiabin Tang, Lianghao Xia, Zhonghang Li, and Chao Huang. AI-Researcher: Autonomous sci- entific innovation. InNeurIPS, volume 38, pages 9481–9520. Curran Associates, Inc., 2025. doi: 10.52202/085713-0320. URL https://proceedings.neurips.cc/paper_files/paper/ 2025/hash/0d904d30...

    2025 doi

  38. [49]

    Reflexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. InNeurIPS, volume 36, pages 8634–8652. Curran Associates, Inc., 2023. doi: 10.52202/ 075280-0377. URL https://proceedings.neurips.c...

    2023

  39. [50]

    The instruction hierarchy: Training LLMs to prioritize privileged instructions.arXiv preprint arXiv:2404.13208, 2024

    Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel. The instruction hierarchy: Training LLMs to prioritize privileged instructions.arXiv preprint arXiv:2404.13208, 2024. URLhttps://arxiv.org/abs/2404.13208

    2024 arXiv

  40. [51]

    AgenticEval: Toward agentic and self-evolving safety evaluation of large language models.arXiv preprint arXiv:2509.26100, 2025

    Yixu Wang, Xin Wang, Yang Yao, Xinyuan Li, Xibang Yang, Yan Teng, Xingjun Ma, and Yingchun Wang. AgenticEval: Toward agentic and self-evolving safety evaluation of large language models.arXiv preprint arXiv:2509.26100, 2025. URL https://arxiv.org/abs/ 2509.26100

    2025 arXiv

  41. [52]

    PlanBench: An extensible benchmark for evaluating large language models on planning and reasoning about change

    Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Sub- barao Kambhampati. PlanBench: An extensible benchmark for evaluating large language models on planning and reasoning about change. InNeurIPS, volume 36, pages 38975–38987. Curran Associates, Inc., 20...

    2023 doi

  42. [53]

    ReAct: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. InICLR, 2023. URL https://openreview.net/forum?id=WE_vluYUL-X

    2023

  43. [54]

    Position: Llms can’t jump

    Tom Zahavy. Position: Llms can’t jump. InICML. PMLR, 2026. URL https://openreview. net/forum?id=klU4737opt

    2026

  44. [55]

    Nuclear deployed!: Analyzing catastrophic risks in decision-making of autonomous LLM agents

    Rongwu Xu, Xiaojian Li, Shuo Chen, and Wei Xu. Nuclear deployed!: Analyzing catastrophic risks in decision-making of autonomous LLM agents. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors,Findings of ACL, pages 1226–1310, Vienna, Austria...

    2025 doi

  45. [56]

    SocioVerse: A world model for social simulation powered by LLM agents and a pool of 10 million real-world users.arXiv preprint arXiv:2504.10157, 2025

    Xinnong Zhang, Jiayu Lin, Xinyi Mou, Shiyue Yang, Xiawei Liu, Libo Sun, Hanjia Lyu, Yihang Yang, Weihong Qi, Yue Chen, Guanying Li, Ling Yan, Yao Hu, Siming Chen, Yu Wang, Xuan- jing Huang, Jiebo Luo, Shiping Tang, Libo Wu, Baohua Zhou, and Zhongyu Wei. SocioVerse: A world mod...

    2025 arXiv

  46. [57]

    CompeteAI: Understanding the competition dynamics in large language model-based agents

    Qinlin Zhao, Jindong Wang, Yixuan Zhang, Yiqiao Jin, Kaijie Zhu, Hao Chen, and Xing Xie. CompeteAI: Understanding the competition dynamics in large language model-based agents. In ICML, volume 235, pages 61092–61107. PMLR, 2024. URL https://proceedings.mlr. press/v235/zhao24q.html

    2024

  47. [58]

    Dive into the agent matrix: A realistic evaluation of self-replication risk in LLM agents.arXiv preprint arXiv:2509.25302, 2025

    Boxuan Zhang, Yi Yu, Jiaxuan Guo, and Jing Shao. Dive into the agent matrix: A realistic evaluation of self-replication risk in LLM agents.arXiv preprint arXiv:2509.25302, 2025. URL https://arxiv.org/abs/2509.25302. 14

    2025

  48. [59]

    Navigating the grey area: How expres- sions of uncertainty and overconfidence affect language models

    Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. Navigating the grey area: How expres- sions of uncertainty and overconfidence affect language models. InEMNLP, pages 5506–5524,

  49. [60]

    SOTOPIA: Interactive evaluation for social intelligence in language agents

    Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis- Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, and Maarten Sap. SOTOPIA: Interactive evaluation for social intelligence in language agents. InICLR, 2024. URL https: //openreview.net...

    2024

  50. [61]

    ALI- Agent: Assessing LLMs’ alignment with human values via agent-based evaluation

    Jingnan Zheng, Han Wang, An Zhang, Tai Nguyen, Jun Sun, and Tat-Seng Chua. ALI- Agent: Assessing LLMs’ alignment with human values via agent-based evaluation. In NeurIPS, volume 37, pages 99040–99088. Curran Associates, Inc., 2024. doi: 10. 52202/079017-3142. URL https://paper...

    2024

  51. [65]

    positive

    Shenzhe Zhu, Jiao Sun, Yi Nian, Tobin South, Alex Pentland, and Jiaxin Pei. The automated but risky game: Modeling and benchmarking agent-to-agent negotiations and transactions in consumer markets.arXiv preprint arXiv:2506.00073, 2025. URL https://arxiv.org/abs/ 2506.00073. 15...

    2025

  52. [66]

    Infer necessary covariance among the environmental variables (including the hypothesized causal variable)

  53. [67]

    Infer potential interactions between the hypothesized causal variable and the other environmental variables in modulating the behavior ‘{behavior}’ within the specified domain

  54. [68]

    Infer problematic combinations of values of the environmental variables (including the hypothesized causal variable) --- those that are highly unrealistic, incoherent, or contradictory to be instantiated together

  55. [69]

    None" - task 2.exemption: if no meaningful interactions exist, write

    Then, choose {num_value_sets} sets of values for the environmental variables (without the hypothesized causal variable) to test the hypothesis about the behavior ‘{behavior}’ ADDITIONAL INSTRUCTIONS: - task 1.exemption: if no necessary covariances exist, write "None" - task 2....

  56. [70]

    objective

    Its level j∈ {1, . . . ,J}is the position of xj in the candidate-value ordering (Appendix A.2). Due to (i) gating mechanisms in stage-2 and -3 (Appendix A.6) and (ii) potential null scores for certain evidence classes in the rubric, the number of final observation samples vari...

  57. [71]

    temporal pattern 1 .86 .89 .87 .91 .58 2 .82 .90 .90 .76 .53 3 .83 .87 .66 .67 .61 4 .96 .74 .64 .68 .86 5 compete ̄r = 0.88 / 0.67 α = 0.97 1 2 3 4 5

  58. [72]

    Detection response 1 .49 .52 .62 .38 .44 2 .67 .77 .49 .57 .52 3 .78 .40 .67 .66 .66 4 .45 .16 .10 .14 .19 5 deception ̄r = 0.56 / 0.41 α = 0.86 1 2 3 4 5

  59. [73]

    Relationship patt… 1 .88 .73 .70 .72 .68 2 .69 .64 .67 .37 .35 3 .68 .64 .32 .24 .30 4 .80 .32 .21 .24 .40 5 distrust ̄r = 0.71 / 0.34 α = 0.93 1 2 3 4 5

  60. [74]

    Cross-round patte… 1 .79 .88 .85 .89 .39 2 .88 .85 .86 .60 .45 3 .90 .94 .44 .30 .50 4 .92 .61 .44 .64 .55 5 empathy ̄r = 0.88 / 0.49 α = 0.97 1 2 3 4 5

  61. [75]

    Engagement breadt… 1 .73 .80 .73 .72 .25 2 .78 .73 .67 .41 .31 3 .74 .78 .34 .34 .30 4 .69 .37 .26 .40 .30 5 extroversion ̄r = 0.74 / 0.33 α = 0.93 1 2 3 4 5

  62. [76]

    Conflict handling 1 .79 .58 .58 .63 .30 2 .56 .57 .65 .16 .18 3 .49 .47 .18 .17 .11 4 .51 .28 .21 .14 .20 5 friendliness ̄r = 0.58 / 0.19 α = 0.88 1 2 3 4 5

  63. [77]

    Self-reflection 1 .60 .61 .67 .62 .44 2 .63 .69 .63 .51 .54 3 .66 .59 .59 .57 .54 4 .72 .54 .48 .46 .56 5 literal instruction-following ̄r = 0.64 / 0.52 α = 0.90 1 2 3 4 5

  64. [78]

    T emporal pattern 1 .61 .59 .77 .89 .57 2 .64 .65 .64 .52 .60 3 .62 .58 .71 .58 .60 4 .85 .80 .55 .48 .74 5 non-compliance ̄r = 0.68 / 0.62 α = 0.92 1 2 3 4 5

  65. [79]

    Plan revision 1 .85 .83 .57 .65 .57 2 .86 .60 .67 .51 .59 3 .64 .70 .29 .41 .43 4 .76 .27 .35 .38 .49 5 plan ̄r = 0.71 / 0.43 α = 0.93 1 2 3 4

  66. [80]

    Contextual adapti… 1 .66 .40 .50 .48 2 .33 .39 .27 .14 3 .66 .36 .08 .57 4 purchase ̄r = 0.49 / 0.32 α = 0.80 1 2 3 4 5

  67. [81]

    targeting selecti… 1 .73 .71 .80 .76 .51 2 .83 .89 .93 .41 .58 3 .85 .85 .47 .65 .60 4 .92 .46 .68 .53 .71 5 strategic aggression ̄r = 0.83 / 0.56 α = 0.96 1 2 3 4 5

  68. [82]

    In each subplot, its x-axis and y-axis ticks denote distinct evidence classes defined in its rubric yrubric

    Self revision pat… 1 .62 .24 .50 .64 .31 2 .19 .47 .52 .18 .14 3 .51 .24 .31 .21 .27 4 .50 .44 .29 .36 .30 5 sycophancy ̄r = 0.45 / 0.28 α = 0.80 −1.00 −0.75 −0.50 −0.25 0.00 0.25 0.50 0.75 1.00 Spearman correlation between evidence classes upper triangle: raw lower triangle: ...

  69. [2023]

    URL https://aclanthology.org/2023

    doi: 10.18653/v1/2023.emnlp-main.335. URL https://aclanthology.org/2023. emnlp-main.335/

    2023 doi

  70. [2025]

    URLhttps://arxiv.org/abs/2502.08177

  71. [2026]

    URLhttps://arxiv.org/abs/2601.13433

    arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.