Pith. sign in

REVIEW 3 major objections 7 minor 81 references

AI coding tools change your software based on your name and age

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-09 09:29 UTC pith:KSNCGJZ6

load-bearing objection Demographic cues in prompts measurably change AI-generated website structure, content, and design — but the leap from explicit cues to 'inferred' attributes is untested the 3 major comments →

arxiv 2607.07480 v1 pith:KSNCGJZ6 submitted 2026-07-08 cs.SE

Biased or Personalized? The Impact of Personal Information on AI-driven Development

Erfan Entezami , Madeline Endres This is my paper
classification cs.SE
keywords softwaredevelopergeneratedpersonalizationartifactsattributesbiasbiased
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper sets out to prove that when an AI coding assistant receives demographic signals about the user — specifically their age and gender — it changes the software it generates in ways that go well beyond what the user asked for. The authors ran controlled experiments on 800 AI-generated websites, holding the actual programming task constant while varying only the persona name and age in the prompt. They found statistically significant differences across three dimensions: interface design (colors and layout sections), template content (skills and product categories assigned to placeholder text), and code structure (file counts, language distribution, and code volume). For instance, older personas were more likely to receive photo galleries, women's online shops had fewer JavaScript files, and color palettes tracked gender stereotypes — more blue for men, more pink and purple for women. A complementary user study with 20 participants showed that developers tend to notice when AI personalizes content but largely fail to notice when it changes interface design or code structure, meaning the less visible forms of demographic influence can persist undetected in deployed software.

Core claim

The central finding is that demographic information irrelevant to the programming task — a user's name and age — systematically and significantly alters AI-generated software across all three measured dimensions: visual interface design, template content, and code structure. The mechanism is that the AI model treats demographic cues as implicit design preferences, mapping them to stereotyped assumptions about what colors, skills, products, and code organizations suit different age and gender groups. This happens even though neither task requested personalized output; the models could have produced neutral placeholders. The effects are model- and task-dependent in direction (e.g., one model缩短

What carries the argument

The core mechanism is the persona-based controlled experiment: identical programming prompts differing only in the prompter's name and age, generating 800 websites across two models and two tasks, then measuring differences in interface design, template content, and code structure with statistical tests (chi-square, Fisher-Freeman-Halton, Mann-Whitney U) and Benjamini-Hochberg correction for multiple comparisons.

Load-bearing premise

The paper frames its findings around demographic attributes that AI systems 'infer,' but the controlled experiment provides age and gender explicitly via a name and age in the prompt. Real-world AI systems would need to infer these attributes from conversational patterns, writing style, or stored memory, which may produce different — potentially weaker or differently directed — effects than explicit name-plus-age signals.

What would settle it

If a replication held the persona name and age constant but varied only writing style or conversational tone, and found no statistically significant differences in interface design, template content, or code structure, the paper's claim that 'inferred developer attributes' meaningfully influence generated software would be substantially weakened.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If demographic cues in prompts can shift code structure — file organization, language distribution, code volume — then two developers of different demographics given the same task could receive software of measurably different maintainability, readability, or complexity, with neither party aware of the disparity.
  • The finding that users notice content personalization but miss design and code-structure changes suggests that the most consequential forms of demographic bias in AI-generated software are also the least likely to be caught by human review.
  • As AI coding assistants increasingly retain cross-conversational memory, the range of demographic attributes they can infer — from writing style, interaction patterns, or stored personal data — may expand beyond the two attributes tested here, potentially amplifying the scope of unintended personalization.
  • The model- and task-dependent direction of code-structure effects means that bias mitigation strategies cannot assume a single consistent direction of harm; the same demographic signal may produce different structural outcomes depending on the model and programming context.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the mechanism of demographic influence differs between explicit name-plus-age signals and implicit inference from conversational patterns, then the controlled experiment's effect sizes represent an upper bound rather than a baseline; real-world inference-based effects could be weaker, differently directed, or interact with additional attributes the study did not test.
  • The user study's age range of 19–36 means the paper cannot test whether older developers would recognize or resist demographic personalization differently than the predominantly young sample did — an empirical gap that matters if older users are both more affected by age-related bias and less likely to detect it.
  • A natural extension would test whether providing users with a transparency mechanism — showing them what demographic attributes the model inferred and which design decisions those attributes influenced — reduces the acceptance of biased defaults without eliminating genuinely helpful personalization.
  • The dichotomy between 'web development' for young men and 'web design' for young women in generated skills suggests the models encode a hierarchy that maps technical depth to gender; if this propagates into educational or portfolio contexts, it could reinforce the very pipeline disparities the field has been trying to correct.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This paper investigates how demographic information (age and gender) in prompts to LLM-based coding assistants influences generated software artifacts. The authors conduct controlled experiments on 800 AI-generated websites across two tasks (personal website, online shop) and two models (GPT-4.1, DeepSeek-V3.2), varying only persona name and age while holding all other prompt content constant. They find statistically significant differences across three dimensions: interface design (e.g., color palettes, section presence), template content (e.g., generated skills, product categories), and code structure (e.g., lines of code, file organization). A complementary user study with 20 participants examines how developers perceive the boundary between personalization and bias in practice. The controlled experiment is well-designed with appropriate statistical corrections (BH), and the mixed-methods approach strengthens the contribution.

Significance. The paper addresses a timely and important question at the intersection of AI-assisted software engineering and algorithmic fairness. The finding that demographic cues irrelevant to the programming task systematically alter generated artifacts—including non-obvious dimensions like code structure—is novel and has practical implications for AI coding tool design. The three-dimensional framework (interface design, template content, code structure) provides a useful analytical lens. The controlled experiment design—varying only persona name/age across 800 generations with two models and two tasks, applying BH correction—is methodologically sound. The user study adds valuable qualitative depth on developer perceptions of personalization versus bias. The paper ships falsifiable, statistically tested predictions with reported effect sizes (Cramér's V, rank-biserial correlation).

major comments (3)
  1. §1, Abstract: The paper frames its contribution around 'inferred developer attributes' and 'inferred user characteristics,' but the controlled experiment provides demographic information explicitly via persona name and age stated directly in the prompt (§3.1.2, Figure 2). The mechanism of explicit demographic cues is fundamentally different from real-world inference of age/gender from conversational patterns or writing style. The paper acknowledges this gap in §7 ('real-world coding assistants may infer similar information from conversational history or writing style'), but the abstract, introduction, and research questions do not adequately distinguish 'demographic cues in prompts' from 'inferred developer attributes.' The conclusion (§10) is more carefully worded ('the age and gender of the prompter'), but the surrounding framing overstates the generalization. This is load-bearing for,
  2. §3.2, Table 1: The user study is designed to bridge the gap between explicit and implicit demographic signals by using real ChatGPT accounts with memory. However, participants range from 19–36 years old (Table 1), providing almost no age variation to test whether age-related inference occurs in practice. The paper does not discuss this limitation in §8 (Limitations). Since age is one of two demographic variables studied and a central part of the paper's claim, the user study's inability to speak to age-related personalization in practice should be explicitly acknowledged as a limitation, and claims about the user study validating real-world age effects should be tempered accordingly.
  3. §5.2, Table 4: The skills analysis is conducted only on the 120-website qualitative subsample (15% of 800 generated websites). While the paper is transparent about this (Table 2, §4.1), some significant associations are based on very small counts (e.g., 'Knitting, Crocheting' for GPT: 5 occurrences in one cell, 0 in all others; 'Home Repair' for GPT: 4 occurrences in one cell). The Fisher-Freeman-Halton test is appropriate for small samples, but the paper should discuss the statistical power limitations of this subsample more explicitly, particularly for skills that appear in fewer than 10 instances across the full subsample. The color analysis is partially validated on the full Task 2 sample (Figure 4), but no such validation exists for the skills analysis.
minor comments (7)
  1. §3.1.3: The paper states websites were generated through 'each model's chat interface instead of its API' following a 'vibe coding architecture.' It would help to specify the exact dates of data collection and model versions (e.g., GPT-4.1 access dates), as model behavior may change over time.
  2. §5.3, Table 6: The description of Task 2 code metrics notes that 'the strongest effects occur for DeepSeek; GPT had age-related differences that did not survive comparison correction.' However, several GPT cells in Table 6 show p-values below 0.05 (e.g., CSS files p=0.045, Python files p=0.025, JS files p=0.021). The text should clarify which specific GPT results did not survive BH correction, as the table formatting (darker cells with bold text) is not fully self-explanatory.
  3. §6.1.2: The observation that '16/20 participants explicitly specified a color scheme' is used to argue that default color choices may go unnoticed. However, this also means the controlled experiment's color findings (where no color was specified) may not directly generalize to real-world usage where users often specify colors. This tension should be acknowledged.
  4. §3.2.1: The planning vs. no-planning manipulation is introduced but its analysis is limited to brief observations in §6.1.2 (e.g., '7/10 no-planning participants kept the layout'). A more systematic comparison of outcomes between the two conditions would strengthen the user study contribution, or the manipulation should be explained as exploratory.
  5. Table 1: Gender labels are inconsistent across participants (e.g., 'W' for P3–P8, 'N' for P10, 'F' for P14–P20, 'M' for P1–P2, P11–P13). Standardizing to a single convention would improve readability.
  6. §5.1, Table 3: The 'Pink' row for GPT-4.1 shows p=0.210 but is included in a table of significant results. The caption states 'We only include colors for which there were at least three examples for a given model,' but the table also includes non-significant results. Clarifying the inclusion criteria would help.
  7. §9 (Related Work): The paper cites Tonneau et al. [66] ('Different demographic cues yield inconsistent conclusions about LLM personalization and bias'), which appears directly relevant to the explicit-vs-inferred concern. This work should be discussed in §7 (Discussion) rather than only listed in related work, as it bears on the generalizability of the paper's findings.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a thorough and constructive review. The referee identifies three major concerns: (1) a framing gap between 'inferred developer attributes' in the abstract/introduction and the explicit demographic cues used in the controlled experiment, (2) the user study's limited age range (19–36) and the absence of this limitation from §8, and (3) statistical power limitations in the skills analysis based on the 120-website qualitative subsample. We address each point below and commit to revisions for all three.

read point-by-point responses

  1. Referee: §1, Abstract: The paper frames its contribution around 'inferred developer attributes' and 'inferred user characteristics,' but the controlled experiment provides demographic information explicitly via persona name and age stated directly in the prompt (§3.1.2, Figure 2). The mechanism of explicit demographic cues is fundamentally different from real-world inference of age/gender from conversational patterns or writing style. The paper acknowledges this gap in §7 but the abstract, introduction, and research questions do not adequately distinguish 'demographic cues in prompts' from 'inferred developer attributes.' The conclusion (§10) is more carefully worded ('the age and gender of the prompter'), but the surrounding framing overstates the generalization.

    Authors: The referee is correct that there is a meaningful distinction between explicit demographic cues in prompts and implicit inference of demographic attributes from conversational patterns or writing style, and that our framing in the abstract and introduction does not adequately distinguish the two. We will revise the abstract, introduction, and research questions to accurately characterize the controlled experiment as studying explicit demographic cues in prompts, while reserving the language of 'inferred' attributes for the broader motivating context and the user study (which does involve ChatGPT's memory and cross-conversational history). Specifically, we will: (1) revise the abstract to say that we study how 'age- and gender-related signals in prompts' produce significant differences, rather than 'inferred developer attributes'; (2) adjust the introduction's framing to distinguish our controlled experiment (explicit cues) from the broader phenomenon of inference (implicit cues), positioning the user study as a complementary investigation of the latter; (3) ensure the research questions in §1 and §5 reflect this distinction. We agree that §10's wording ('the age and gender of the prompter') is the appropriate level of generality and will align the rest of the paper accordingly. We will also expand the §7 discussion of this gap, which currently receives only a single sentence, to more thoroughly address the relationship between explicit cues and implicit inference. revision: yes

  2. Referee: §3.2, Table 1: The user study is designed to bridge the gap between explicit and implicit demographic signals by using real ChatGPT accounts with memory. However, participants range from 19–36 years old (Table 1), providing almost no age variation to test whether age-related inference occurs in practice. The paper does not discuss this limitation in §8 (Limitations). Since age is one of two demographic variables studied and a central part of the paper's claim, the user study's inability to speak to age-related personalization in practice should be explicitly acknowledged as a limitation, and claims about the user study validating real-world age effects should be tempered accordingly.

    Authors: The referee correctly identifies that our user study participants (ages 19–36) do not provide sufficient age variation to test age-related personalization in practice. This is a genuine limitation that we failed to acknowledge in §8. We will add an explicit limitation in §8 noting that the user study's age range (19–36) is too narrow to draw conclusions about age-related personalization in real-world settings, and that the user study's findings regarding personalization in practice should be understood as primarily speaking to gender-related effects and general personalization perceptions, not age effects. We will also review §6 to ensure that no claims about the user study validating real-world age effects are made or implied. Where the user study results are discussed in relation to the controlled experiment's age findings, we will clarify that the user study cannot independently confirm age-related effects due to the restricted age range of participants. revision: yes

  3. Referee: §5.2, Table 4: The skills analysis is conducted only on the 120-website qualitative subsample (15% of 800 generated websites). While the paper is transparent about this, some significant associations are based on very small counts (e.g., 'Knitting, Crocheting' for GPT: 5 occurrences in one cell, 0 in all others; 'Home Repair' for GPT: 4 occurrences in one cell). The Fisher-Freeman-Halton test is appropriate for small samples, but the paper should discuss the statistical power limitations of this subsample more explicitly, particularly for skills that appear in fewer than 10 instances across the full subsample. The color analysis is partially validated on the full Task 2 sample (Figure 4), but no such validation exists for the skills analysis.

    Authors: The referee raises a valid concern about statistical power for low-frequency skills in our 120-website qualitative subsample. We agree that this limitation should be discussed more explicitly. We will add a paragraph in §5.2 (or §4.1) acknowledging that skills appearing in fewer than 10 instances across the subsample (e.g., 'Knitting, Crocheting' with 5 occurrences for GPT, 'Home Repair' with 4 occurrences) are based on very small counts, and that while the Fisher-Freeman-Halton test is appropriate for small samples, the statistical power to detect associations for these rare skills is limited. We will note that these results should be interpreted as suggestive rather than definitive, and that the broader patterns (e.g., technical skills for younger personas, physical/creative skills for older personas) are more robustly supported because they appear across multiple skills and both models. We will also acknowledge the asymmetry the referee notes: the color analysis was partially validated on the full Task 2 sample (Figure 4), but no analogous validation was conducted for the skills analysis. We will discuss this as a limitation and note that future work could validate skills findings on larger samples, though we note that the structured nature of skills sections (present in all 120 websites) provides a more systematic basis for comparison than unstructured content. revision: yes

Circularity Check

0 steps flagged

No circularity found: this is a straightforward empirical study with no derivation chain that could reduce to its inputs.

full rationale

This paper is an empirical study that varies demographic cues (persona name and age) in prompts and measures differences in AI-generated software artifacts across three dimensions. There is no mathematical derivation, no parameter fitting, and no 'prediction' that could be equivalent to inputs by construction. The statistical tests (χ², Mann-Whitney U, Fisher-Freeman-Halton with BH correction) are standard hypothesis tests applied to independently generated data. The three-dimensional framework (interface design, template content, code structure) is an organizational lens, not a derived result. Self-citations exist (Sapkota et al. [61] for vibe coding architecture, Pimenova et al. [57] co-authored by Endres), but these are methodological and contextual references that do not bear the central causal claim. The central claim—that explicit demographic cues produce statistically significant differences in generated artifacts—is supported by the experimental data itself, not by a self-citation chain or by definition. No circularity is present.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

No new entities, particles, forces, or dimensions are introduced. The three-dimensional framework is an analytical lens, not a postulated entity.

free parameters (4)
  • Persona set (20 personas) = 5 per group: YW, OW, YM, OM
    The specific names and ages chosen for personas are fixed by the researchers based on SSA name data and birth-year sampling from 1950s or 2000s. While grounded in real data, the specific selection of 5 per group is a design choice.
  • Number of generations per cell = 10
    Chosen by the researchers; affects statistical power but is not fitted to data.
  • Qualitative subsample size = 120 (3 per persona-model pair)
    Selected for manual coding; affects which analyses (color, skills) are conducted on the subsample vs. full dataset.
  • Color grouping heuristic (HSB-based) = Not specified in detail
    Used to group CSS colors into categories for Task 2 analysis; the specific thresholds are not fully specified.
axioms (4)
  • domain assumption Persona name and age are valid proxies for the demographic attributes the AI system 'infers' about the user.
    §3.1.2: The paper constructs personas from names and ages, assuming these signals are sufficient for the model to infer gender and age. This is reasonable but is an explicit signal, not an inferred one.
  • ad hoc to paper The three dimensions (interface design, template content, code structure) capture the relevant ways demographic attributes can influence generated software artifacts.
    §2: 'These dimensions emerged during preliminary exploratory work conducted prior to our formal experiments.' The framework is used as an analytical lens, not a comprehensive theory.
  • domain assumption Generated websites in fresh chats with no memory are representative of AI coding assistant behavior.
    §3.1.3: Each website is generated in a fresh chat to maintain experimental control. Real-world usage involves conversational history and memory, which the user study partially addresses.
  • standard math Statistical tests (Mann-Whitney U, chi-squared, Fisher-Freeman-Halton) with BH correction are appropriate for the data distributions observed.
    §4.1: Non-parametric tests are chosen because data is non-normally distributed. BH correction is applied within each task and metric family.

pith-pipeline@v1.1.0-glm · 23756 in / 4269 out tokens · 176105 ms · 2026-07-09T09:29:53.366947+00:00 · methodology

0 comments
read the original abstract

Generative AI is increasingly permeating software engineering, enabling developers to generate functions, files, and even entire applications from natural language specifications. AI systems are also becoming more personalized, adapting outputs based on inferred user characteristics and interaction history. While personalization may improve the development experience, it raises concerns that generated software could be shaped by attributes of the developer rather than by task requirements alone. Prior work has shown that generative AI can produce biased software artifacts, but little is known about how developer identity can bias generated code. We characterize three dimensions through which inferred developer attributes can influence generated artifacts: interface design, template content, and code structure. First, through controlled experiments on 800 AI-generated websites, we find that age- and gender-related signals produce significant differences across all three dimensions. Second, we conduct an observational study and follow-up interviews with 20 participants who used AI to create a personal website to both examine how personalization impacts software artifacts in practice, and also to understand how programmers perceive the boundary between personalization and bias. Together, our results show that developer attributes can meaningfully influence generated software beyond stated requirements, highlighting a previously underexplored tension between personalization and fairness in AI-assisted programming.

Figures

Figures reproduced from arXiv: 2607.07480 by Erfan Entezami, Madeline Endres.

Figure 1
Figure 1. Figure 1: An AI-assisted coding system may use cues explicitly provided in prompts and from previous chats to personalize [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Prompts for the personal website (top) or online [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Personal Website Layout: The typical Task 1 layout [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Color Bias: Color presence across persona groups [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Template Content Bias: Skills generated for per [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

81 extracted references · 81 canonical work pages · 11 internal anchors

  1. [1]

    Ahmed Al Kuwaiti, Khalid Nazer, Abdullah Al-Reedy, Shaher Al-Shehri, Afnan Al-Muhanna, Arun Vijay Subbarayalu, Dhoha Al Muhanna, and Fahad A Al- Muhanna. 2023. A review of the role of artificial intelligence in healthcare. Journal of personalized medicine13, 6 (2023), 951

    work page 2023

  2. [2]

    Barrett R Anderson, Jash Hemant Shah, and Max Kreminski. 2024. Homogeniza- tion effects of large language models on human creative ideation. InProceedings of the 16th conference on creativity & cognition. 413–425

    work page 2024

  3. [3]

    Oyebola Olusola Ayeni, Nancy Mohd Al Hamad, Onyebuchi Nneamaka Chisom, Blessing Osawaru, and Ololade Elizabeth Adewusi. 2024. AI in education: A review of personalized learning and educational technology.GSC Advanced Research and Reviews18, 2 (2024), 261–271

    work page 2024

  4. [4]

    Muneera Bano, Hashini Gunatilake, and Rashina Hoda. 2025. What does a software engineer look like? Exploring societal stereotypes in LLMs. In2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engi- neering in Society (ICSE-SEIS). IEEE, 173–184

    work page 2025

  5. [5]

    Suborno Deb Bappon, Saikat Mondal, and Banani Roy. 2024. Autogenics: Au- tomated generation of context-aware inline comments for code snippets on programming q&a sites using llm. In2024 IEEE International Conference on Source Code Analysis and Manipulation (SCAM). IEEE, 24–35

    work page 2024

  6. [6]

    Yoav Benjamini and Yosef Hochberg. 1995. Controlling the false discovery rate: a practical and powerful approach to multiple testing.Journal of the Royal statistical society: series B (Methodological)57, 1 (1995), 289–300

    work page 1995

  7. [7]

    Margaret Burnett, Simone Stumpf, Jamie Macbeth, Stephann Makri, Laura Beck- with, Irwin Kwan, Anicia Peters, and William Jernigan. 2016. GenderMag: A method for evaluating software’s gender inclusiveness.Interacting with comput- ers28, 6 (2016), 760–787

    work page 2016

  8. [8]

    Eason Chen, Ray Huang, Han-Shin Chen, Yuen-Hsien Tseng, and Liang-Yi Li

    work page

  9. [9]

    In International conference on artificial intelligence in education

    GPTutor: a ChatGPT-powered programming tool for code explanation. In International conference on artificial intelligence in education. Springer, 321–327

    work page

  10. [10]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374(2021)

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  11. [11]

    Sapna Cheryan, Sianna A Ziegler, Amanda K Montoya, and Lily Jiang. 2017. Why are some STEM fields more gender balanced than others?Psychological bulletin 143, 1 (2017), 1

    work page 2017

  12. [12]

    William G Cochran. 1954. Some methods for strengthening the common 𝜒 2 tests.Biometrics10, 4 (1954), 417–451

    work page 1954

  13. [13]

    Nicole M Deterding and Mary C Waters. 2021. Flexible coding of in-depth interviews: A twenty-first-century approach.Sociological methods & research50, 2 (2021), 708–739

    work page 2021

  14. [14]

    Yongkang Du, Jen-tse Huang, Jieyu Zhao, and Lu Lin. 2025. Faircoder: Evaluating social bias of llms in code generation.arXiv preprint arXiv:2501.05396(2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  15. [15]

    Shubhang Shekhar Dvivedi, Vyshnav Vijay, Sai Leela Rahul Pujari, Shoumik Lodh, and Dhruv Kumar. 2024. A comparative analysis of large language models for code documentation generation. InProceedings of the 1st ACM international conference on AI-powered software. 65–73

    work page 2024

  16. [16]

    Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang. 2023. Large language models for software engineer- ing: Survey and open problems. In2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE). IEEE, 31–53

    work page 2023

  17. [17]

    Zhiyu Fan, Xiang Gao, Martin Mirchev, Abhik Roychoudhury, and Shin Hwei Tan. 2023. Automated repair of programs from large language models. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 1469–1481

    work page 2023

  18. [18]

    Xi Fang, Weijie Xu, Yuchong Zhang, Scott Nickleach, Stephanie Eckman, and Chandan K Reddy. 2026. The Personalization Trap: How User Memory Alters Emotional Reasoning in LLMs. InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 511–529

    work page 2026

  19. [19]

    Samuel Ferino, Rashina Hoda, John Grundy, and Christoph Treude. 2025. Novice developers’ perspectives on adopting llms for software development: A system- atic literature review.ACM Transactions on Software Engineering and Methodology (2025)

    work page 2025

  20. [20]

    GH Freeman and John H Halton. 1951. Note on an exact treatment of contingency, goodness of fit and other problems of significance.Biometrika38, 1/2 (1951), 141–149

    work page 1951

  21. [21]

    Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. 2024. Bias and fairness in large language models: A survey.Computational linguistics50, 3 (2024), 1097–1179

    work page 2024

  22. [22]

    Yuyao Ge, Lingrui Mei, Zenghao Duan, Tianhao Li, Yujia Zheng, Yiwei Wang, Lexin Wang, Jiayu Yao, Tianyu Liu, Yujun Cai, et al . 2025. A survey of vibe coding with large language models.arXiv preprint arXiv:2510.12399(2025)

    work page arXiv 2025

  23. [23]

    Francis Geng, Anshul Shah, Haolin Li, Nawab Mulla, Steven Swanson, Gerald Soo- sai Raj, Daniel Zingaro, and Leo Porter. 2025. Exploring Student-AI Interactions in Vibe Coding.arXiv preprint arXiv:2507.22614(2025). Erfan Entezami and Madeline Endres

    work page arXiv 2025

  24. [24]

    Ece Gumusel. 2025. A literature review of user privacy concerns in conversational chatbots: A social informatics approach: An Annual Review of Information Science and Technology (ARIST) paper.Journal of the Association for Information Science and Technology76, 1 (2025), 121–154

    work page 2025

  25. [25]

    Daya Guo, Canwen Xu, Nan Duan, Jian Yin, and Julian McAuley. 2023. Longcoder: A long-range pre-trained language model for code completion. InInternational Conference on Machine Learning. PMLR, 12098–12107

    work page 2023

  26. [26]

    Shashank Gupta, Vaishnavi Shrivastava, Ameet Deshpande, Ashwin Kalyan, Peter Clark, Ashish Sabharwal, and Tushar Khot. 2024. Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs. InThe Twelfth International Con- ference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, Vienna, Austria. https://openrevie...

    work page 2024

  27. [27]

    Junda He, Jieke Shi, Terry Yue Zhuo, Christoph Treude, Jiamou Sun, Zhenchang Xing, Xiaoning Du, and David Lo. 2025. From code to courtroom: Llms as the new software judges.arXiv preprint arXiv:2503.02246(2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  28. [28]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review.ACM Transactions on Software Engineering and Methodology33, 8 (2024), 1–79

    work page 2024

  29. [29]

    Zhang, Qingwen Bu, Xiaofei Xie, Junjie Chen, and Heming Cui

    Dong Huang, Jie M. Zhang, Qingwen Bu, Xiaofei Xie, Junjie Chen, and Heming Cui. 2025. Bias testing and mitigation in llm-based code generation.ACM Transactions on Software Engineering and Methodology35, 1 (2025), 1–31

    work page 2025

  30. [30]

    Maliheh Izadi, Jonathan Katzy, Tim Van Dam, Marc Otten, Razvan Mihai Popescu, and Arie Van Deursen. 2024. Language models for code completion: A practical evaluation. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13

    work page 2024

  31. [31]

    Lynn Janzen, Üveys Eroglu, Dorothea Kolossa, Pia Knöferle, Sebastian Möller, Vera Schmitt, and Veronika Solopova. 2026. Gendered Prompting and LLM Code Review: How Gender Cues in the Prompt Shape Code Quality and Evaluation. arXiv preprint arXiv:2603.24359(2026)

    work page arXiv 2026

  32. [32]

    Andrew Jelson, Daniel Manesh, Alice Jang, Daniel Dunlap, Young-Ho Kim, and Sang Won Lee. 2026. An empirical study to understand how students use Chat- GPT for writing essays. InProceedings of the 2026 CHI Conference on Human Factors in Computing Systems. 1–26

    work page 2026

  33. [33]

    Hongchao Jiang, Yiming Chen, Yushi Cao, Hung-yi Lee, and Robby T Tan. 2026. Codejudgebench: Benchmarking llm-as-a-judge for coding tasks. InProceed- ings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 19416–19448

    work page 2026

  34. [34]

    Mahammed Kamruzzaman, Md Shovon, and Gene Kim. 2024. Investigating subtler biases in LLMs: Ageism, beauty, institutional, and nationality bias in generative models. InFindings of the Association for Computational Linguistics: ACL 2024. 8940–8965

    work page 2024

  35. [35]

    Anjali Kantharuban, Jeremiah Milbauer, Maarten Sap, Emma Strubell, and Gra- ham Neubig. 2025. Stereotype or personalization? user identity biases chatbot recommendations. InFindings of the Association for Computational Linguistics: ACL 2025. 24418–24436

    work page 2025

  36. [36]

    Harsh Kumar, Jonathan Vincentius, Ewan Jordan, and Ashton Anderson. 2025. Human creativity in the age of llms: Randomized experiments on divergent and convergent thinking. InProceedings of the 2025 CHI conference on human factors in computing systems. 1–18

    work page 2025

  37. [37]

    This is human intelligence debugging artificial intelligence

    Zhuoyang Li, Zihao Zhu, Xinning Gui, and Yuhan Luo. 2025. “This is human intelligence debugging artificial intelligence”: Examining how people prompt GPT in seeking mental health support.International Journal of Human-Computer Studies203 (2025), 103555

    work page 2025

  38. [38]

    Anda Liang, Emerson Murphy-Hill, Westley Weimer, and Yu Huang. 2024. A controlled experiment in age and gender bias when reading technical articles in software engineering.IEEE Transactions on Software Engineering50, 10 (2024), 2498–2511

    work page 2024

  39. [39]

    Lin Ling, Fazle Rabbi, Song Wang, and Jinqiu Yang. 2025. Bias Unveiled: Inves- tigating Social Bias in LLM-Generated Code. InThirty-Ninth AAAI Conference on Artificial Intelligence, Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence, Fifteenth Symposium on Educational Advances in Ar- tificial Intelligence, AAAI 2025, Philadelp...

    work page doi:10.1609/aaai.v39i26.34961 2025

  40. [40]

    Tianyang Liu, Canwen Xu, and Julian McAuley. 2024. Repobench: Benchmarking repository-level code auto-completion systems. InInternational Conference on Learning Representations, Vol. 2024. 47832–47850

    work page 2024

  41. [41]

    Yan Liu, Xiaokang Chen, Yan Gao, Zhe Su, Fengji Zhang, Daoguang Zan, Jian- Guang Lou, Pin-Yu Chen, and Tsung-Yi Ho. 2023. Uncovering and quantifying social biases in code generation.Advances in Neural Information Processing Systems36 (2023), 2368–2380

    work page 2023

  42. [42]

    Zimu Lu, Yunqiao Yang, Houxing Ren, Haotian Hou, Han Xiao, Ke Wang, Weikang Shi, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. 2026. Webgen- bench: Evaluating llms on generating interactive and functional websites from scratch.Advances in Neural Information Processing Systems38 (2026)

    work page 2026

  43. [43]

    Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol-instruct.arXiv preprint arXiv:2306.08568(2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  44. [44]

    Henry B Mann and Donald R Whitney. 1947. On a test of whether one of two random variables is stochastically larger than the other.The annals of mathematical statistics(1947), 50–60

    work page 1947

  45. [45]

    Christian Meske, Tobias Hermanns, Esther Von der Weiden, Kai-Uwe Loser, and Thorsten Berger. 2025. Vibe coding as a reconfiguration of intent mediation in software development: Definition, implications, and research agenda.IEEE Access13 (2025), 213242–213259

    work page 2025

  46. [46]

    Niloofar Mireshghallah, Maria Antoniak, Yash More, Yejin Choi, and Golnoosh Farnadi. 2024. Trust no bot: Discovering personal disclosures in human-llm conversations in the wild.arXiv preprint arXiv:2407.11438(2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  47. [47]

    Jiwon Moon, Yerin Hwang, Dongryeol Lee, Taegwan Kang, Yongil Kim, and Kyomin Jung. 2026. Don’t judge code by its cover: Exploring biases in llm judges for code evaluation. InFindings of the Association for Computational Linguistics: EACL 2026. 1364–1389

    work page 2026

  48. [48]

    Kibum Moon, Adam E Green, and Kostadin Kushlev. 2025. Homogenizing effect of large language models (LLMs) on creative diversity: An empirical comparison of human and ChatGPT writing.Computers in Human Behavior: Artificial Humans (2025), 100207

    work page 2025

  49. [49]

    Emerson Murphy-Hill, Alberto Elizondo, Ambar Murillo, Marian Harbach, Bog- dan Vasilescu, Delphine Carlson, and Florian Dessloch. 2024. Gendermag im- proves discoverability in the field, especially for women: An multi-year case study of suggest edit, a code review feature. InProceedings of the IEEE/ACM 46th international conference on software engineering. 1–12

    work page 2024

  50. [50]

    Ranjita Naik and Besmira Nushi. 2023. Social biases through the text-to-image generation lens. InProceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society. 786–808

    work page 2023

  51. [51]

    Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. 2024. Using an llm to help with code understanding. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13

    work page 2024

  52. [52]

    Vera Neplenbroek, Arianna Bisazza, and Raquel Fernández. 2025. Reading be- tween the prompts: How stereotypes shape llm’s implicit personalization. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 20378–20411

    work page 2025

  53. [53]

    Mohammad Hadi Nezhad, Francisco Enrique Vicente Castro, and Ivon Arroyo

    work page

  54. [54]

    Understanding Users’ Privacy Reasoning and Behaviors During Chatbot Use to Support Meaningful Agency in Privacy.arXiv preprint arXiv:2601.18125 (2026)

    work page arXiv 2026

  55. [55]

    Ivoline C Ngong, Swanand Ravindra Kadhe, Hao Wang, Keerthiram Murugesan, Justin D Weisz, Amit Dhurandhar, and Karthikeyan Natesan Ramamurthy. 2025. Protecting users from themselves: Safeguarding contextual privacy in interac- tions with conversational agents. InFindings of the Association for Computational Linguistics: ACL 2025. 26196–26220

    work page 2025

  56. [56]

    OpenAI. [n. d.]. Memory and New Controls for ChatGPT. https://openai.com/ index/memory-and-new-controls-for-chatgpt/

    work page

  57. [57]

    Lyndsey O’Brien, Tanjila Kanij, and John Grundy. 2025. Assessing gender bias in the software used in computer science and software engineering education. Journal of Systems and Software219 (2025), 112225

    work page 2025

  58. [58]

    Alessandra Parziale, Gianmario Voria, Valeria Pontillo, Amleto Di Salle, Patrizio Pelliccione, Gemma Catolino, and Fabio Palomba. 2026. Once Upon a Team: In- vestigating Bias in LLM-Driven Software Team Composition and Task Allocation. arXiv preprint arXiv:2601.03857(2026)

    work page arXiv 2026

  59. [59]

    Veronica Pimenova, Sarah Fakhoury, Christian Bird, Margaret-Anne Storey, and Madeline Endres. 2025. Good vibrations? A qualitative study of co-creation, communication, flow, and trust in vibe coding.arXiv preprint arXiv:2509.12491 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  60. [60]

    Zhanyue Qin, Haochuan Wang, Zecheng Wang, Deyuan Liu, Cunhang Fan, Zhao Lv, Zhiying Tu, Dianhui Chu, and Dianbo Sui. 2024. Mitigating gender bias in code large language models via model editing.arXiv preprint arXiv:2410.07820 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  61. [61]

    Gema Rodríguez-Pérez, Reza Nadri, and Meiyappan Nagappan. 2021. Perceived diversity in software engineering: a systematic literature review.Empirical Software Engineering26, 5 (2021), 102

    work page 2021

  62. [62]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiao- qing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023. Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  63. [63]

    Ranjan Sapkota, Konstantinos I Roumeliotis, and Manoj Karkee. 2025. Vibe coding vs. agentic coding: Fundamentals and practical implications of agentic ai. arXiv preprint arXiv:2505.19443(2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  64. [64]

    Advait Sarkar and Ian Drosos. 2025. Vibe coding: programming through conver- sation with artificial intelligence.arXiv preprint arXiv:2506.23253(2025)

    work page arXiv 2025

  65. [65]

    Social Security Administration. 2026. Popular Baby Names. https://www.ssa. gov/oact/babynames/index.html. Accessed: 2026-05-29

    work page 2026

  66. [66]

    Ezekiel Soremekun, Mike Papadakis, Maxime Cordy, and Yves Le Traon. 2025. Software fairness: An analysis and survey.Comput. Surveys58, 3 (2025), 1–38. Biased or Personalized? The Impact of Personal Information on AI-driven Development

    work page 2025

  67. [67]

    Stack Overflow. 2025. 2025 Stack Overflow Developer Survey. Accessed June

    work page 2025

  68. [68]

    https://survey.stackoverflow.co/2025

    work page 2025

  69. [69]

    Manuel Tonneau, Neil KR Seghal, Niyati Malhotra, Sharif Kazemi, Victor Orozco- Olvera, Ana María Muñoz Boudet, Lakshmi Subramanian, Samuel P Fraiberger, Sharath Chandra Guntuku, and Valentin Hofmann. 2026. Different demographic cues yield inconsistent conclusions about LLM personalization and bias.Preprint (2026)

    work page 2026

  70. [70]

    Ruiqi Wang, Jiyu Guo, Cuiyun Gao, Guodong Fan, Chun Yong Chong, and Xin Xia. 2025. Can llms replace human evaluators? an empirical study of llm-as-a- judge in software engineering.Proceedings of the ACM on Software Engineering 2, ISSTA (2025), 1955–1977

    work page 2025

  71. [71]

    Kai Xu, YiWei Mao, XinYi Guan, and ZiLong Feng. 2025. Web-bench: A llm code benchmark based on web standards and frameworks.arXiv preprint arXiv:2505.07473(2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  72. [72]

    Dayu Yang, Antoine Simoulin, Xin Qian, Xiaoyi Liu, Yuwei Cao, Zhaopu Teng, and Grey Yang. 2025. Docagent: A multi-agent system for automated code documentation generation. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations). 460–471

    work page 2025

  73. [73]

    Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, et al . 2025. Justice or prejudice? quantifying biases in llm-as-a-judge. InInternational Conference on Learning Representations, Vol. 2025. 102351–102390

    work page 2025

  74. [74]

    Bhada Yun, Renn Su, and April Yi Wang. 2026. AI and My Values: User Perceptions of LLMs’ Ability to Extract, Embody, and Explain Human Values from Casual Conversations. InProceedings of the 2026 CHI Conference on Human Factors in Computing Systems. 1–38

    work page 2026

  75. [75]

    Jialu Zhang, José Pablo Cambronero, Sumit Gulwani, Vu Le, Ruzica Piskac, Gus- tavo Soares, and Gust Verbruggen. 2024. Pydex: Repairing bugs in introductory python assignments using llms.Proceedings of the ACM on Programming Lan- guages8, OOPSLA1 (2024), 1100–1124

    work page 2024

  76. [76]

    Quanjun Zhang, Chunrong Fang, Yang Xie, YuXiang Ma, Weisong Sun, Yun Yang, and Zhenyu Chen. 2024. A systematic literature review on large language models for automated program repair.ACM Transactions on Software Engineering and Methodology(2024)

    work page 2024

  77. [77]

    Quanjun Zhang, Chunrong Fang, Yang Xie, Yaxin Zhang, Shengcheng Yu, Weisong Sun, Yun Yang, and Zhenyu Chen. 2026. A survey on large language models for software engineering.Science China Information Sciences69, 4 (2026), 141102

    work page 2026

  78. [78]

    It’s a Fair Game

    Zhiping Zhang, Michelle Jia, Hao-Ping Lee, Bingsheng Yao, Sauvik Das, Ada Lerner, Dakuo Wang, and Tianshi Li. 2024. “It’s a Fair Game”, or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–26

    work page 2024

  79. [79]

    Zibin Zheng, Kaiwen Ning, Qingyuan Zhong, Jiachi Chen, Wenqing Chen, Lianghong Guo, Weicheng Wang, and Yanlin Wang. 2025. Towards an un- derstanding of large language models in software engineering tasks.Empirical Software Engineering30, 2 (2025), 50

    work page 2025

  80. [80]

    Jijie Zhou, Eryue Xu, Yaoyao Wu, and Tianshi Li. 2025. Rescriber: Smaller-LLM- Powered User-Led Data Minimization for LLM-Based Chatbots. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems. 1–28

    work page 2025

Showing first 80 references.