Pith. sign in

REVIEW 3 major objections 6 minor 75 references

Personality and emotion profiles change multi-agent LLM software teams’ results and cost: shared profiles swing code pass rates by 7–11 points, mixed roles often win, and fear or high conscientiousness drives extra revision without reliable

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 04:12 UTC pith:DSPZ4KA7

load-bearing objection Solid multi-agent SE experiment: profile prompts move pass@1 by ~7–11 pp and raise revision/token cost, but the psychology labels are mostly unvalidated prompt packaging. the 3 major comments →

arxiv 2607.05659 v1 pith:DSPZ4KA7 submitted 2026-07-06 cs.SE

Agents with Feelings? Personality and Emotion in Multi-Agent Software Teams

Yunyan Ding , Thomas Zimmermann , Iftekhar Ahmed This is my paper
classification cs.SE
keywords large language modelsmulti-agent systemscode generationcode reviewpersonalityemotionsoftware engineeringpersona prompting
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Multi-agent LLM systems for software work usually distinguish agents by roles and workflows, not by how they behave. This paper argues that psychology-style profiles—Big Five traits, basic emotions, and related work styles—are a real design lever: they change both task success and how teams collaborate. The authors build a fixed persona framework, assign 78 shared and mixed team profiles, and evaluate code generation and code review on four models and 659 task instances. They find large best-to-worst gaps under shared profiles, frequent gains when different roles get different profiles, and a cost pattern in which fear and high conscientiousness increase revision, over-revision, and token use without consistent quality payoffs. A sympathetic reader cares because this reframes agent “feelings” and dispositions as engineering choices that affect outcomes and efficiency, not as decorative prompt color.

Core claim

Profile choice substantially affects both performance and team behavior in multi-agent LLM software teams. For code generation, the best and worst shared-profile configurations differ by 7.1–11.3 percentage points in pass@1 across models; the best mixed-profile configuration beats the best shared-profile configuration in six of eight model–task settings; and fear and high-conscientiousness profiles raise revision activity, over-revision, and token usage without consistent performance gains. Agent profiles are therefore an important design dimension alongside roles and workflows.

What carries the argument

A psychology-informed persona framework that turns Big Five levels (conscientiousness, openness, extraversion), basic emotions, SE-relevant O*NET work styles, and task roles into fixed natural-language persona descriptions assigned to agents in shared- or mixed-profile teams.

Load-bearing premise

The study assumes fixed Claude-written 120–180-word persona paragraphs cleanly and comparably encode the intended personality, emotion, and work-style settings across roles and models, so measured gaps come from those constructs rather than uncontrolled wording or style differences.

What would settle it

Rewrite the same profile conditions so persona texts match on length, structure, and surface style while only flipping the intended trait and emotion labels (or replace them with matched human-written personas) and re-run the shared-profile code-generation suite: if the 7–11 point best–worst pass@1 gaps collapse or the fear/high-conscientiousness cost pattern disappears, the claim that these psychological profiles drive the effects fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Profile assignment should be treated as an experimental factor and validated per model and task, not copied as a universal default.
  • Role-specific mixed profiles can outperform uniform teams, so systems should allow different behavioral settings for planner, implementer, reviewer, and similar roles.
  • Fear and high-conscientiousness profiles raise collaboration cost; practitioners should optimize for both quality and token/revision overhead.
  • Multi-agent SE studies should report profile configurations because they change both outcomes and collaboration efficiency.
  • Tool builders can add automated or adaptive profile search that balances performance against revision and token cost.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same profile levers may matter in non-SE multi-agent settings where agents plan, critique, or negotiate, not only in code generation and review.
  • Because best profiles are model- and task-specific, profile “libraries” will likely need continuous re-tuning as models change rather than one-time design.
  • Over-revision under scrutiny-oriented profiles points to a control problem: multi-agent loops may need stop rules keyed to tests or confidence, not only agent personality.
  • If persona wording is the real mechanism, lighter behavioral keywords might capture much of the effect without full psychology-informed generation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper studies whether psychology-informed agent profiles—Big Five traits (C/O/E), basic emotions, O*NET work styles, and SE task roles—affect multi-agent LLM team performance and collaboration on code generation and code review. The authors generate fixed role-specific persona texts, evaluate 54 shared-profile and 24 mixed-profile configurations on four instruction-tuned LLMs (1.5B–32B) over 282 LiveCodeBench problems and 377 review instances, and analyze pass@1, BLEU-4, revision/over-revision rates, token cost, and sentiment with mixed-effects logistic models. They report large best–worst shared-profile gaps (7.1–11.3 pp pass@1; up to ~19% relative BLEU), frequent gains from role-heterogeneous mixed profiles (6/8 model–task settings), and a cost–performance tradeoff in which fear and high conscientiousness increase revision activity and tokens without consistent quality gains. The central claim is that agent profiles are a first-class design dimension for multi-agent SE systems, not merely decorative prompt text.

Significance. If the results hold under stronger construct controls, the paper makes a clear and timely contribution to multi-agent SE: it elevates behavioral profile assignment from ad-hoc persona prompting to an experimentally measurable design factor with both effectiveness and cost implications. Strengths include scale (78 configurations × 4 models × two tasks), deterministic greedy decoding for isolation of profile effects, execution-based pass@1, an explicit over-revision metric, mixed-effects models with FDR-corrected odds ratios, and honest reporting that best profiles are model- and task-specific rather than universal. The mixed-profile and cost analyses are practically useful for tool builders. Even under a weaker reading—that different fixed persona prompts matter a lot—the empirical magnitude is still valuable for the multi-agent SE literature.

major comments (3)
  1. [§III-A; Findings 1–3; §IV-D4] §III-A (persona generation) and Findings 1–3: The causal reading that performance and revision gaps are effects of Big Five levels, basic emotions, and O*NET work styles rests on unvalidated Claude-generated 120–180-word persona texts that are fixed and reused across all evaluated LLMs. The manuscript reports no construct checks (independent trait/emotion ratings, lexical/embedding diagnostics, length/style matching, or ablations that hold wording density fixed while varying only the intended factor). Without such evidence, the 7.1–11.3 pp gaps and the fear/high-C revision effects could be driven by uncontrolled differences in imperative style, instruction density, or length rather than the psychological labels used as factors in the mixed-effects models (§IV-D4). This is load-bearing for the psychology-informed framing; either add validation/ablation or substantially soften claims from
  2. [§IV-C; RQ2; Finding 2] §IV-C and RQ2/Finding 2: Mixed-profile teams are constructed only by recombining the top-three shared-profile configurations per model–task pair (24 assignments after excluding uniforms). This is a reasonable tractability choice, but it conditions the mixed-vs-shared comparison on already-strong homogeneous profiles and does not test whether heterogeneity helps when weaker or complementary profiles are included. The claim that “role-specific profile specialization can provide additional gains” is therefore supported only inside a narrow, performance-selected subspace. Please either expand the candidate set (e.g., include mid/low shared profiles or role-targeted hypotheses) or restate Finding 2 as limited to recombinations of strong shared profiles, with corresponding caution in the abstract and discussion.
  3. [§IV-D2; Fig. 5(b); Table I–II] §IV-D2 and code-review results: Task performance for review is measured solely by smoothed BLEU-4 against human comments. The paper correctly notes this is a relative alignment metric, not review quality, yet abstract/RQ1/RQ2 still present BLEU gaps as “performance” parallel to pass@1. BLEU is known to be weak for short, multi-aspect review text; best/worst rankings and mixed-profile wins on review may not survive a semantic or human preference metric. At minimum, add one complementary metric (e.g., embedding similarity, aspect coverage, or a small human preference study on a subset) or clearly demote review results relative to execution-based generation findings when stating overall conclusions.
minor comments (6)
  1. [§IV-A] §IV-A: Temperature-0 greedy decoding is well justified for isolation and pass@1, but a short sensitivity note (or a small non-zero-temperature spot check on a subset of profiles) would help readers who expect persona effects to appear partly as sampling diversity.
  2. [§III-A; Fig. 2] §III-A / Fig. 2: Report basic statistics of the generated personas (token length distribution, work-style selection frequencies per trait/emotion) so readers can assess prompt-length confounds even before full construct validation.
  3. [§V RQ3; Table III] §V / Table III: Over-revision is defined only for code generation; consider an analogous “unnecessary revision” proxy for code review if Supervisor requests changes when Writers already match reference aspects, or explicitly state why that is infeasible.
  4. [§IV-C; Table I] Self-report baseline (§IV-C): Clarify the exact prompt used for self-report and whether NEUTRAL answers were frequent; for Qwen 32B the baseline is already near the best shared profile, which affects interpretation of “improvement opportunity.”
  5. [Figs. 5–6; [73]] Presentation: Expand figure captions for Figs. 5–6 to state n (54 vs 24 configs) and that diamonds/triangles are single reference points, not distributions. Ensure companion artifact link and full per-configuration tables are available at camera-ready.
  6. [§II-B] Related work: A brief contrast with single-agent personality-guided code generation [17] on whether multi-agent interaction amplifies or dampens persona effects would sharpen the novelty claim.

Circularity Check

0 steps flagged

Empirical multi-agent prompt study with external benchmarks; no derivation reduces claimed effects to fitted inputs or self-definition.

full rationale

The paper is a controlled comparative experiment: 78 profile configurations are assigned as fixed persona prompts, then teams are scored on external criteria (LiveCodeBench public+private pass@1; BLEU-4 vs human review comments) under fixed workflows, greedy decoding, and shared instance sets. Best–worst gaps, mixed-vs-shared comparisons, and mixed-effects odds ratios for revision/over-revision are measurements of those outcomes, not quantities recovered from parameters fitted to the same outcomes. The self-report baseline is an extra reference condition obtained by asking each model for a default profile; it does not define success and is not used to construct the performance metrics. Mixed-profile teams recombine top shared profiles as a search heuristic, then re-evaluate on the same external metrics. No uniqueness theorem, self-citation chain, or ansatz from the authors’ prior work is load-bearing for the central claims. Construct-validity concerns about whether Claude-generated personas faithfully encode Big Five/emotion labels are real but are validity threats, not circular reductions of prediction to input. Score 0 is therefore appropriate.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 2 invented entities

This is an empirical multi-agent prompting study. Load-bearing content is mostly domain assumptions that human psychology constructs and O*NET work styles can be usefully induced in LLMs via short natural-language personas, plus several hand-chosen experimental constants (work-style count, revision budget, persona length, mixed-profile search restriction). No new physical entities are postulated; the main invented construct is the combined profile/persona framework used as the experimental factor.

free parameters (5)
  • number of work styles per persona = 3
    Fixed at three by author choice for concreteness vs. conciseness (§III-A); not derived from data or theory uniqueness.
  • max revision rounds = 3
    Capped at three for both tasks, following prior multi-agent studies rather than optimized (§III-B, §III-C).
  • persona description length = 120-180 words
    Constrained to 120–180 words by design choice (§III-A).
  • mixed-profile candidate set = top-3 shared per model-task
    Only recombinations of the top-3 shared profiles per model–task are evaluated, excluding the full 54³ space (§IV-C).
  • task sample sizes = 282 / 377
    282 generation and 377 review instances chosen via finite-population sample-size formula at 95% CI / 5% MoE with fixed seed (§IV-B).
axioms (6)
  • domain assumption Conscientiousness, Openness, and Extraversion (but not Agreeableness/Neuroticism) are the Big Five dimensions most relevant to programming aptitude and thus to SE agent profiles.
    Invoked in §III-A citing Gnambs (2015); drives the 2³+neutral personality design.
  • domain assumption Anger, fear, disgust, sadness, happiness, and neutral are an adequate emotion set for SE agent behavior.
    Selected from basic emotion theory and SE emotion reviews (§III-A); other emotions excluded to keep the design tractable.
  • domain assumption O*NET work-style poles can be paired and used as concrete behavioral descriptors that make personality/emotion profiles actionable for SE roles.
    §III-A constructs 14 work-style poles from O*NET software-developer impacts after excluding dependability/integrity/perseverance.
  • domain assumption Natural-language persona descriptions can induce stable, profile-consistent behavioral differences in instruction-tuned LLMs under multi-agent workflows.
    Core experimental premise throughout; supported by cited persona/emotion prompting literature but not independently validated here with psychometric checks.
  • domain assumption pass@1 on LiveCodeBench-style public/private tests and smoothed BLEU-4 against human review comments are adequate primary performance metrics for comparing profiles.
    §IV-D; BLEU is explicitly only a relative reference-alignment measure.
  • ad hoc to paper Temperature-0 greedy decoding isolates profile effects without needing multi-sample variance estimates.
    §IV-A chooses deterministic decoding for pass@1 comparability; trades off behavioral diversity.
invented entities (2)
  • psychology-informed multi-factor agent profile (Big Five C/O/E × basic emotion × O*NET work styles × SE role) no independent evidence
    purpose: Defines the experimental factor used to generate role-specific persona descriptions and compare team configurations.
    Composite construct assembled for this study; components are prior, combination and SE multi-agent use are paper-specific.
  • over-revision rate (code generation) no independent evidence
    purpose: Captures Reviewer-requested revisions after the current implementation already passes the full evaluation suite.
    Operational metric defined in §IV-D to separate productive from unnecessary revision; useful but study-specific.

pith-pipeline@v1.1.0-grok45 · 25500 in / 3560 out tokens · 38325 ms · 2026-07-11T04:12:01.814899+00:00 · methodology

0 comments
read the original abstract

Multi-agent LLM systems for Software Engineering (SE) typically differentiate agents through roles and workflows, but little is known about how agents' behavioral profiles affect team performance. We investigate the impact of personality and emotion profiles on LLM agent teams using a psychology-informed framework that combines Big Five personality traits, basic emotions, SE-relevant work styles, and task roles. We evaluate 78 team-profile configurations across code generation and code review using four LLMs and 659 task instances. Results show that profile choice substantially affects both performance and team behavior. For code generation, the gap between the best and worst shared-profile configurations reaches 7.1-11.3 percentage points in pass@1 across models, while the best mixed-profile configuration outperforms the best shared-profile configuration in six of eight model-task settings. Profiles also influence collaboration dynamics and cost: fear and high-conscientiousness profiles increase revision activity, over-revision, and token usage without consistent performance gains. These findings identify agent profiles as an important design dimension in multi-agent SE systems, affecting not only task outcomes but also the efficiency of collaboration.

Figures

Figures reproduced from arXiv: 2607.05659 by Iftekhar Ahmed, Thomas Zimmermann, Yunyan Ding.

Figure 1
Figure 1. Figure 1: Overview of our study design, including profile specification, persona [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Example prompt and generated persona for the Planner role under the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Code-generation workflow. The Planner produces one plan, and the [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Code-review workflow. Two Writers cover complementary review di [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Performance distributions across shared-profile configurations. Dots [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Performance distributions of mixed-profile assignments. Boxes show [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

75 extracted references · 44 canonical work pages · 1 internal anchor

  1. [1]

    Large language models for software engineering: Sur- vey and open problems,

    A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, “Large language models for software engineering: Sur- vey and open problems,” in2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE). IEEE, 2023, pp. 31–53

    2023

  2. [2]

    Large language models for software engi- neering: A systematic literature review,

    X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engi- neering: A systematic literature review,”ACM Transactions on Software Engineering and Methodology, vol. 33, no. 8, pp. 1–79, 2024

    2024

  3. [3]

    Chatdev: Communicative agents for software development,

    C. Qian, W. Liu, H. Liu, N. Chen, Y . Dang, J. Li, C. Yang, W. Chen, Y . Su, X. Conget al., “Chatdev: Communicative agents for software development,” inProceedings of the 62nd annual meeting of the associ- ation for computational linguistics (volume 1: Long papers), 2024, pp. 15 174–15 186

    work page 2024

  4. [4]

    Metagpt: Meta programming for a multi-agent collaborative framework,

    S. Hong, M. Zhuge, J. Chen, X. Zheng, Y . Cheng, J. Wang, C. Zhang, S. Yau, Z. Lin, L. Zhouet al., “Metagpt: Meta programming for a multi-agent collaborative framework,” inInternational Conference on Learning Representations, vol. 2024, 2024, pp. 23 247–23 275

    2024

  5. [5]

    The big five personality dimensions and job performance: a meta-analysis,

    M. R. Barrick and M. K. Mount, “The big five personality dimensions and job performance: a meta-analysis,”Personnel psychology, vol. 44, no. 1, pp. 1–26, 1991

    work page 1991

  6. [6]

    Big five personality traits and performance: A quantitative synthesis of 50+ meta-analyses,

    E. Zell and T. L. Lesick, “Big five personality traits and performance: A quantitative synthesis of 50+ meta-analyses,”Journal of personality, vol. 90, no. 4, pp. 559–573, 2022

    work page 2022

  7. [7]

    Personality and team performance: a meta-analysis,

    M. A. Peeters, H. F. Van Tuijl, C. G. Rutte, and I. M. Reymen, “Personality and team performance: a meta-analysis,”European journal of personality, vol. 20, no. 5, pp. 377–396, 2006

    work page 2006

  8. [8]

    Deep-level composition variables as predictors of team performance: a meta-analysis

    S. T. Bell, “Deep-level composition variables as predictors of team performance: a meta-analysis.”Journal of applied psychology, vol. 92, no. 3, p. 595, 2007

    work page 2007

  9. [9]

    Software engineering team diversity and performance,

    V . Pieterse, D. G. Kourie, and I. P. Sonnekus, “Software engineering team diversity and performance,” inProceedings of the 2006 annual research conference of the South African institute of computer scientists and information technologists on IT research in developing countries, 2006, pp. 180–186

    work page 2006

  10. [10]

    A follow up study of the effect of personality on the performance of software engineering teams,

    J. Karn and T. Cowling, “A follow up study of the effect of personality on the performance of software engineering teams,” inProceedings of the 2006 ACM/IEEE international symposium on Empirical software engineering, 2006, pp. 232–241

    work page 2006

  11. [11]

    Are team personality and climate related to satisfaction and software quality? aggregating results from a twice replicated experiment,

    S. T. Acu ˜na, M. N. G ´omez, J. E. Hannay, N. Juristo, and D. Pfahl, “Are team personality and climate related to satisfaction and software quality? aggregating results from a twice replicated experiment,”Information and Software Technology, vol. 57, pp. 141–156, 2015

    work page 2015

  12. [12]

    Affective events theory,

    H. M. Weiss and R. Cropanzano, “Affective events theory,”Research in organizational behavior, vol. 18, no. 1, pp. 1–74, 1996

    work page 1996

  13. [13]

    Evaluating and inducing personality in pre-trained language models,

    G. Jiang, M. Xu, S.-C. Zhu, W. Han, C. Zhang, and Y . Zhu, “Evaluating and inducing personality in pre-trained language models,”Advances in Neural Information Processing Systems, vol. 36, pp. 10 622–10 643, 2023

    work page 2023

  14. [14]

    Personality traits in large language models,

    G. Serapio-Garc ´ıa, M. Safdari, C. Crepy, L. Sun, S. Fitz, P. Romero, M. Abdulhai, A. Faust, and M. Matari ´c, “Personality traits in large language models,”arXiv preprint arXiv:2307.00184, 2023

    Pith/arXiv arXiv 2023

  15. [15]

    Out of one, many: Using language models to simulate human samples,

    L. P. Argyle, E. C. Busby, N. Fulda, J. R. Gubler, C. Rytting, and D. Wingate, “Out of one, many: Using language models to simulate human samples,”Political Analysis, vol. 31, no. 3, pp. 337–351, 2023

    2023

  16. [16]

    Large language models understand and can be enhanced by emotional stimuli,

    C. Li, J. Wang, Y . Zhang, K. Zhu, W. Hou, J. Lian, F. Luo, Q. Yang, and X. Xie, “Large language models understand and can be enhanced by emotional stimuli,”arXiv preprint arXiv:2307.11760, 2023

    Pith/arXiv arXiv 2023

  17. [17]

    Personality-guided code generation using large language models,

    Y . Guo, Z. Chen, J. M. Zhang, Y . Liu, and Y . Ma, “Personality-guided code generation using large language models,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025, pp. 1068–1080

    work page 2025

  18. [18]

    The power of personality: A human simulation perspective to investigate large language model agents,

    Y . Duan, Y . Tang, X. Bai, K. Chen, J. Li, and M. Zhang, “The power of personality: A human simulation perspective to investigate large language model agents,”arXiv preprint arXiv:2502.20859, 2025

    Pith/arXiv arXiv 2025

  19. [19]

    The next big five inventory (bfi-2): Developing and assessing a hierarchical model with 15 facets to enhance bandwidth, fidelity, and predictive power

    C. J. Soto and O. P. John, “The next big five inventory (bfi-2): Developing and assessing a hierarchical model with 15 facets to enhance bandwidth, fidelity, and predictive power.”Journal of personality and social psychology, vol. 113, no. 1, p. 117, 2017

    work page 2017

  20. [20]

    An introduction to the five-factor model and its applications,

    R. R. McCrae and O. P. John, “An introduction to the five-factor model and its applications,”Journal of personality, vol. 60, no. 2, pp. 175–215, 1992

    work page 1992

  21. [21]

    O*NET content model,

    National Center for O*NET Development, “O*NET content model,” https://www.onetcenter.org/content.html, 2026, accessed: 2026-06-14

    work page 2026

  22. [22]

    The O* NET content model: strengths and limitations,

    M. J. Handel, “The O* NET content model: strengths and limitations,” Journal for Labour Market Research, vol. 49, no. 2, pp. 157–176, 2016

    work page 2016

  23. [23]

    An argument for basic emotions,

    P. Ekman, “An argument for basic emotions,”Cognition & emotion, vol. 6, no. 3-4, pp. 169–200, 1992

    1992

  24. [24]

    Taking the emotional pulse of software engineering—a systematic literature review of empiri- cal studies,

    M. S ´anchez-Gord´on and R. Colomo-Palacios, “Taking the emotional pulse of software engineering—a systematic literature review of empiri- cal studies,”Information and Software Technology, vol. 115, pp. 23–43, 2019

    work page 2019

  25. [25]

    A survey on large language models for code generation,

    J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim, “A survey on large language models for code generation,”ACM Transactions on Software Engineering and Methodology, vol. 35, no. 2, pp. 1–72, 2026

    2026

  26. [26]

    Automated code review using large language models at ericsson: An experience report,

    S. Ramesh, J. Bose, H. Singh, A. Raghavan, S. R. Chowdhury, G. Sridhara, N. Saini, and R. Britto, “Automated code review using large language models at ericsson: An experience report,” in2025 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2025, pp. 602–607

    work page 2025

  27. [27]

    Hydra- reviewer: A holistic multi-agent system for automatic code review comment generation,

    X. Ren, C. Dai, Q. Huang, Y . Wang, C. Liu, and B. Jiang, “Hydra- reviewer: A holistic multi-agent system for automatic code review comment generation,”IEEE Transactions on Software Engineering, 2025

    work page 2025

  28. [28]

    Autogen: Enabling next-gen llm applications via multi-agent conversations,

    Q. Wu, G. Bansal, J. Zhang, Y . Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liuet al., “Autogen: Enabling next-gen llm applications via multi-agent conversations,” inFirst conference on language modeling, 2024

    2024

  29. [29]

    Camel: Communicative agents for “Mind

    G. Li, H. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem, “Camel: Communicative agents for “Mind” exploration of large language model society,”Advances in neural information processing systems, vol. 36, pp. 51 991–52 008, 2023

    work page 2023

  30. [30]

    Agentverse: Facilitating multi- agent collaboration and exploring emergent behaviors,

    W. Chen, Y . Su, J. Zuo, C. Yang, C. Yuan, C.-M. Chan, H. Yu, Y . Lu, Y .-H. Hung, C. Qianet al., “Agentverse: Facilitating multi- agent collaboration and exploring emergent behaviors,” inInternational Conference on Learning Representations, vol. 2024, 2024, pp. 20 094– 20 136

    work page 2024

  31. [31]

    Sentiment analysis for software engineering: How far can we go?

    B. Lin, F. Zampetti, G. Bavota, M. Di Penta, M. Lanza, and R. Oliveto, “Sentiment analysis for software engineering: How far can we go?” in Proceedings of the 40th international conference on software engineer- ing, 2018, pp. 94–104

    work page 2018

  32. [32]

    Towards Automated Crowdsourced Testing via Personified-LLM

    S. Yu, Y . Ling, C. Fang, Z. Chen, and C. Chen, “Towards au- tomated crowdsourced testing via personified-llm,”arXiv preprint arXiv:2603.24160, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  33. [33]

    Mimic: Integrating diverse personality traits for better game testing using large language model,

    Y . Chen, S. Habchi, and L. Wei, “Mimic: Integrating diverse personality traits for better game testing using large language model,”arXiv preprint arXiv:2510.01635, 2025

    arXiv 2025

  34. [34]

    Generative agents: Interactive simulacra of human behavior,

    J. S. Park, J. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein, “Generative agents: Interactive simulacra of human behavior,” inProceedings of the 36th annual acm symposium on user interface software and technology, 2023, pp. 1–22

    2023

  35. [35]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockmanet al., “Evaluating large language models trained on code,”arXiv preprint arXiv:2107.03374, 2021

    Pith/arXiv arXiv 2021

  36. [36]

    Program synthesis with large language models,

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Leet al., “Program synthesis with large language models,”arXiv preprint arXiv:2108.07732, 2021

    Pith/arXiv arXiv 2021

  37. [37]

    Livecodebench: Holistic and contamination free evaluation of large language models for code,

    N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica, “Livecodebench: Holistic and contamination free evaluation of large language models for code,” in The Thirteenth International Conference on Learning Representations,

  38. [38]

    Available: https://openreview.net/forum?id=chfJJYC3iL

    [Online]. Available: https://openreview.net/forum?id=chfJJYC3iL

  39. [39]

    Codegen: An open large language model for code with multi-turn program synthesis,

    E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y . Zhou, S. Savarese, and C. Xiong, “Codegen: An open large language model for code with multi-turn program synthesis,”arXiv preprint arXiv:2203.13474, 2022

    Pith/arXiv arXiv 2022

  40. [40]

    Starcoder: may the source be with you!

    R. Li, L. B. Allal, Y . Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chimet al., “Starcoder: may the source be with you!”arXiv preprint arXiv:2305.06161, 2023

    Pith/arXiv arXiv 2023

  41. [41]

    Code llama: Open foundation models for code,

    B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, R. Sauvestre, T. Remezet al., “Code llama: Open foundation models for code,”arXiv preprint arXiv:2308.12950, 2023

    Pith/arXiv arXiv 2023

  42. [42]

    Re- flexion: Language agents with verbal reinforcement learning,

    N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Re- flexion: Language agents with verbal reinforcement learning,”Advances in neural information processing systems, vol. 36, pp. 8634–8652, 2023

    2023

  43. [43]

    Self-refine: Iter- ative refinement with self-feedback,

    A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y . Yanget al., “Self-refine: Iter- ative refinement with self-feedback,”Advances in neural information processing systems, vol. 36, pp. 46 534–46 594, 2023

    2023

  44. [44]

    Agentcoder: Multi-agent-based code generation with iterative testing and optimisation,

    D. Huang, J. M. Zhang, M. Luck, Q. Bu, Y . Qing, and H. Cui, “Agentcoder: Multi-agent-based code generation with iterative testing and optimisation,”arXiv preprint arXiv:2312.13010, 2023

    Pith/arXiv arXiv 2023

  45. [45]

    Mapcoder: Multi-agent code generation for competitive problem solving,

    M. A. Islam, M. E. Ali, and M. R. Parvez, “Mapcoder: Multi-agent code generation for competitive problem solving,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 4912–4944

    work page 2024

  46. [46]

    Expectations, outcomes, and challenges of modern code review,

    A. Bacchelli and C. Bird, “Expectations, outcomes, and challenges of modern code review,” in2013 35th international conference on software engineering (ICSE). IEEE, 2013, pp. 712–721

    2013

  47. [47]

    Automating code review activities by large-scale pre-training,

    Z. Li, S. Lu, D. Guo, N. Duan, S. Jannu, G. Jenks, D. Majumder, J. Green, A. Svyatkovskiy, S. Fuet al., “Automating code review activities by large-scale pre-training,” inProceedings of the 30th ACM joint European software engineering conference and symposium on the foundations of software engineering, 2022, pp. 1035–1047

    work page 2022

  48. [48]

    What are emotions? and how can they be measured?

    K. R. Scherer, “What are emotions? and how can they be measured?” Social science information, vol. 44, no. 4, pp. 695–729, 2005

    work page 2005

  49. [49]

    National Research Council,A Database for a Changing Economy: Re- view of the Occupational Information Network (O*NET), N. T. Tippins and M. L. Hilton, Eds. Washington, DC: The National Academies Press,

    work page

  50. [50]

    Available: https://nap.nationalacademies.org/catalog/ 12814/a-database-for-a-changing-economy-review-of-the-occupational

    [Online]. Available: https://nap.nationalacademies.org/catalog/ 12814/a-database-for-a-changing-economy-review-of-the-occupational

    work page

  51. [51]

    What makes a computer wiz? linking personality traits and programming aptitude,

    T. Gnambs, “What makes a computer wiz? linking personality traits and programming aptitude,”Journal of Research in Personality, vol. 58, pp. 31–34, 2015

    work page 2015

  52. [52]

    15-1252.00 – software developers,

    National Center for O*NET Development, “15-1252.00 – software developers,” O*NET OnLine, 2026, accessed: 2026-06-14. [Online]. Available: https://www.onetonline.org/link/summary/15-1252.00

    work page 2026

  53. [53]

    Introducing claude sonnet 4.6,

    Anthropic, “Introducing claude sonnet 4.6,” https://www.anthropic.com/ news/claude-sonnet-4-6, Feb. 2026, accessed: 2026-06-14

    work page 2026

  54. [54]

    Claude sonnet 4.6 system card,

    ——, “Claude sonnet 4.6 system card,” Anthropic, Tech. Rep., Feb. 2026, accessed: 2026-06-14. [Online]. Available: https://www-cdn. anthropic.com/78073f739564e986ff3e28522761a7a0b4484f84.pdf

    work page 2026

  55. [55]

    Scaling large language model-based multi- agent collaboration,

    C. Qian, Z. Xie, Y . Wang, W. Liu, K. Zhu, H. Xia, Y . Dang, Z. Du, W. Chen, C. Yanget al., “Scaling large language model-based multi- agent collaboration,” inInternational Conference on Learning Repre- sentations, vol. 2025, 2025, pp. 41 488–41 505

    work page 2025

  56. [56]

    MAgICoRe: Multi-agent, iterative, coarse-to-fine refinement for reasoning,

    J. Chen, A. Prasad, S. Saha, E. Stengel-Eskin, and M. Bansal, “MAgICoRe: Multi-agent, iterative, coarse-to-fine refinement for reasoning,” inProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V . Peng, Eds. Suzhou, China: Association for Computational Linguistics, Nov...

    work page 2025

  57. [57]

    Qwen2.5 technical report,

    Qwen, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Ren, Y . Fan, Y . Su, Y . Zhang, Y . Wan, Y . Liu, Z. Cui, Z. Zhang, and...

    Pith/arXiv arXiv 2025

  58. [58]

    The llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughanet al., “The llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  59. [59]

    Mistral-Small-24B-Instruct-2501,

    Mistral AI, “Mistral-Small-24B-Instruct-2501,” https://huggingface.co/ mistralai/Mistral-Small-24B-Instruct-2501, 2025, Hugging Face model card

    work page 2025

  60. [60]

    Codeif: Benchmarking the instruction-following capabilities of large language models for code generation,

    K. Yan, H. Guo, X. Shi, S. Cao, D. Di, and Z. Li, “Codeif: Benchmarking the instruction-following capabilities of large language models for code generation,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), 2025, pp. 1272–1286

    work page 2025

  61. [61]

    Humaneval pro and mbpp pro: Evaluating large language models on self-invoking code generation task,

    Z. Yu, Y . Zhao, A. Cohan, and X.-P. Zhang, “Humaneval pro and mbpp pro: Evaluating large language models on self-invoking code generation task,” inFindings of the Association for Computational Linguistics: ACL 2025, 2025, pp. 13 253–13 279

    work page 2025

  62. [62]

    Can llms generate high-quality test cases for algorithm problems? testcase-eval: A systematic evaluation of fault coverage and exposure,

    Z. Yang, Z. Kuang, X. Xia, and Y . Zhao, “Can llms generate high-quality test cases for algorithm problems? testcase-eval: A systematic evaluation of fault coverage and exposure,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2025, pp. 1050–1063

    work page 2025

  63. [63]

    How many tries does it take? iterative self-repair in llm code generation across model scales and benchmarks,

    J. J. Arimbur, “How many tries does it take? iterative self-repair in llm code generation across model scales and benchmarks,”arXiv preprint arXiv:2604.10508, 2026

    Pith/arXiv arXiv 2026

  64. [64]

    Awq: Activation-aware weight quanti- zation for on-device llm compression and acceleration,

    J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han, “Awq: Activation-aware weight quanti- zation for on-device llm compression and acceleration,”Proceedings of machine learning and systems, vol. 6, pp. 87–100, 2024

    2024

  65. [65]

    Personallm: Investigating the ability of large language models to ex- press personality traits,

    H. Jiang, X. Zhang, X. Cao, C. Breazeal, D. Roy, and J. Kabbara, “Personallm: Investigating the ability of large language models to ex- press personality traits,” inFindings of the association for computational linguistics: NAACL 2024, 2024, pp. 3605–3627

    work page 2024

  66. [66]

    Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions,

    T. Y . Zhuo, M. C. Vu, J. Chim, H. Hu, W. Yu, R. Widyasari, I. N. B. Yusuf, H. Zhan, J. He, I. Paulet al., “Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions,” inInternational Conference on Learning Representations, vol. 2025, 2025, pp. 66 602–66 656

    work page 2025

  67. [67]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,

    J. Liu, C. S. Xia, Y . Wang, and L. Zhang, “Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,”Advances in neural information processing systems, vol. 36, pp. 21 558–21 572, 2023

    work page 2023

  68. [68]

    W. G. Cochran,Sampling techniques, 3rd ed. John Wiley & Sons, 1977

    work page 1977

  69. [69]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” inProceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, pp. 311–318

    2002

  70. [70]

    Senticr: A customized sentiment analysis tool for code review interactions,

    T. Ahmed, A. Bosu, A. Iqbal, and S. Rahimi, “Senticr: A customized sentiment analysis tool for code review interactions,” in2017 32nd IEEE/ACM International Conference on Automated Software Engineer- ing (ASE). IEEE, 2017, pp. 106–111

    work page 2017

  71. [71]

    Generalized linear mixed models: a practical guide for ecology and evolution,

    B. M. Bolker, M. E. Brooks, C. J. Clark, S. W. Geange, J. R. Poulsen, M. H. H. Stevens, and J.-S. S. White, “Generalized linear mixed models: a practical guide for ecology and evolution,”Trends in ecology & evolution, vol. 24, no. 3, pp. 127–135, 2009

    work page 2009

  72. [72]

    Fitting linear mixed- effects models using lme4,

    D. Bates, M. M ¨achler, B. Bolker, and S. Walker, “Fitting linear mixed- effects models using lme4,”Journal of statistical software, vol. 67, pp. 1–48, 2015

    2015

  73. [73]

    Controlling the false discovery rate: a practical and powerful approach to multiple testing,

    Y . Benjamini and Y . Hochberg, “Controlling the false discovery rate: a practical and powerful approach to multiple testing,”Journal of the Royal statistical society: series B (Methodological), vol. 57, no. 1, pp. 289–300, 1995

    1995

  74. [74]

    A circumplex model of affect

    J. A. Russell, “A circumplex model of affect.”Journal of personality and social psychology, vol. 39, no. 6, p. 1161, 1980

    1980

  75. [75]

    Personality and emotion in Multi-Agent Software Teams,

    Y . Ding, “Personality and emotion in Multi-Agent Software Teams,” https://github.com/personas-matter-llms/companion-artifact, 2026, com- panion repository

    work page 2026