Pith. sign in

REVIEW 3 major objections 5 minor 51 references

HiLDe: Intentional Code Generation via Human-in-the-Loop Decoding

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper reports that exposing the token-level decisions of an LLM code assistant cut security vulnerabilities by 31% in a user study.

desk verdict A genuinely new interaction technique with a promising security result, but the main quantitative claim is confounded by a security-tuned prompt, so the effect of the interface itself is not yet identified. read the letter →

arxiv 2505.22906 v2 pith:IQMV5ALB submitted 2025-05-28 cs.HC cs.AIcs.PL

Emmanuel Anaya González , Raven Rothkopf , Sorin Lerner , Nadia Polikarpova This is my paper
classification cs.HCcs.AIcs.PL
keywords Human-in-the-LoopDecodingcodecompletionassistantLLMtoken-leveldecisionssoftwaresecurityhuman-AIcollaborationintentionalgenerationwithin-subjectsuserstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes Human-in-the-Loop Decoding, a way of letting programmers inspect and influence the token-by-token decisions an LLM makes while generating code, instead of accepting a single "best" completion. The authors implement this idea in HiLDe, a code completion assistant that highlights critical decision points in red and offers local alternative tokens with short explanations of how each would change the code. In a within-subjects study of 18 programmers working on security-related tasks, the paper reports that HiLDe produced 31% fewer security vulnerabilities on average than a baseline assistant using the same underlying model ($M = 2.67$ vs. $3.89$, $p = 0.01$), and that participants intentionally repaired more vulnerabilities with the new interface. The authors interpret this as evidence that giving users visibility into the model's choices and control over them can reduce overreliance and align generated code with user intent, without requiring security expertise. The finding matters because it suggests interface design, not just better models or prompts, can improve the safety of AI-generated code.

What carries the argument

The load-bearing mechanism is the combination of critical decision highlighting with local alternatives. HiLDe first asks an analysis LLM to classify each alternative token at every generation step as Significant, Minor, or Incorrect, with an importance score from 0 to 1; it then computes a corrected entropy that upweights the token distribution by these importance scores, so only steps with behavior-changing alternatives are highlighted. When the user opens a highlighted token, the assistant shows short explanations of each alternative and, on selection, regenerates the completion suffix, choosing the most similar suffix to the original so the edit feels local. These two affordances—pointing at the decisions that matter and letting the user act on them with explanations—carry the argument that intentionality can be designed into LLM code generation.

What would settle it

Run the same security tasks with identical underlying prompts and completion model in both conditions, toggling only the highlighting and local-alternative UI; if the vulnerability count stays roughly equal, the reported 31% reduction is not attributable to the interface. Conversely, if the tuned prompt alone, without the UI, reproduces the reduction, the affordances are not the active ingredient.

Watch

Extended reading notes

Core claim

The paper's central discovery is that involving the user in the decoding process—the sequence of token choices the model makes—improves the security of the code they end up with. At each critical step HiLDe shows alternatives the model was weighing, ordered by likelihood and annotated with explanations, and lets the user swap a token in; the rest of the completion is regenerated to fit. The measured effect is a 31% average reduction in vulnerabilities ($M = 2.67$ vs. $3.89$, $p = 0.01$, $r = 0.53$), with 71% of the 42 intentional security repairs in the study occurring under HiLDe and a significantly higher proportion of those repairs being made via the UI rather than by re-prompting. The authors also report that participants spent longer reviewing HiLDe suggestions before accepting them, and that the slower pace was associated with more deliberate decision-making rather than higher cognitive load.

Load-bearing premise

The study's conclusion that the interface caused the security improvement rests on the assumption that HiLDe and the baseline differ only in the highlighting and local-alternative affordances; the paper itself notes the HiLDe prompt was "slightly tuned to surface security-related decision points," so the two conditions differ in prompt content as well.

Editorial extensions

If this is right

  • If the effect holds, code assistants can reduce security vulnerabilities by changing the interface rather than by retraining the model or adding a safety filter.
  • The 31% reduction came with a cost: participants took about twice as long to accept a suggestion, so the technique trades speed for intentionality—a trade-off the paper argues is worth making for critical domains.
  • The same interaction framework can be aimed at other non-functional goals such as efficiency, maintainability, or personal style by re-tuning the prompt that asks the analysis model what counts as significant; the study only tuned it for security.
  • Because participants with low security knowledge benefited most from local alternatives, the approach may specifically mitigate overreliance in users who cannot easily spot insecure suggestions.
  • The finding that HiLDe users often discovered their own intent while looking at alternatives suggests the technique changes not just output quality but the user's understanding of what they want.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not isolate the UI effect from the prompt-tuning confound; a cleaner test would run both conditions with identical prompts and toggle only the highlighting and alternatives, which would tell us whether the 31% reduction is an interface effect or partly a prompt effect.
  • The corrected-entropy mechanism could be validated in isolation: if importance-weighted highlighting reliably marks the tokens whose alternatives actually change security, then the highlight-and-select loop is the active ingredient; if not, the explanations or the act of choosing may be doing the work.
  • The downstream-regeneration fragility the paper reports—where swapping a token can overwrite prior user edits—could be addressed by anchoring regeneration to the user's edited context rather than the original completion; solving it would likely determine whether the technique scales beyond short completions to real refactorings.
  • A direct comparison with a no-highlighting, only-alternatives condition would clarify how much of the benefit comes from drawing attention to decisions versus from offering choices; the paper's design bundles both affordances together.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper introduces Human-in-the-Loop Decoding (HiLDe), an interaction technique for LLM-based code completion that highlights token-level "critical decision points" and lets users view and select local alternative tokens with explanatory annotations. The authors implement HiLDe as a VS Code extension and evaluate it in a within-subjects user study (N=18) with four security-related programming tasks, comparing it against a baseline assistant that uses the same underlying model but offers only global alternative completions. The main quantitative claim is that participants using HiLDe produced code with 31% fewer security vulnerabilities on average (M_HiLDe = 2.67, M_BASELINE = 3.89, p = 0.01, r = 0.53), and that 71% of intentional security repairs occurred with HiLDe. The paper also presents qualitative evidence that HiLDe encouraged users to discover, interpret, and act on implementation alternatives.

Significance. If the effect is real, HiLDe addresses an important problem: users of AI code assistants often over-rely on suggestions and fail to consider security-relevant alternatives. The paper's strengths include a concrete system implementation with a clear interaction design, a controlled within-subjects study with manual vulnerability counting by two authors, and rich qualitative case studies that illustrate the proposed discover-interpret-act loop. The central quantitative result is directionally compelling and the effect size is large. However, the comparison is compromised by a confound between the interaction affordances and the underlying prompt content, which is explicitly acknowledged in Section VI-B but not controlled for or listed as a limitation. Because the study's causal claim about the interaction technique depends on this confound being absent, the current evidence is not yet sufficient to support the title's claim about Human-in-the-Loop Decoding itself.

major comments (3)
  1. [Section VI-B / IV-C] The central comparison between HiLDe and BASELINE is confounded by prompt content. Section VI-B states that "the underlying prompt for HILDE is slightly tuned to surface security-related decision points," whereas Section IV-C describes BASELINE as using the same underlying model without highlighting or local alternatives and does not state that BASELINE uses the same tuned prompt. The observed reduction in vulnerabilities (M_HiLDe = 2.67 vs. M_BASELINE = 3.89, p = 0.01) and the availability of scrypt/pbkdf2_hmac alternatives could therefore be caused by the prompt tuning rather than by the interactive affordances. Section VI-D lists limitations but does not mention this confound. Please add a condition that holds the prompt content fixed across interfaces, or otherwise demonstrate that the prompt tuning does not affect which tokens and alternatives are surfaced.
  2. [Section III-B.2] The corrected-entropy highlighting method is not specified reproducibly. The paper states that after pilot experiments "we arrived at a configuration" for the effect of Importance Score on corrected entropy, but it does not provide the exact formula or the parameter values. Since highlighting is the mechanism claimed to drive the effect, the empirical results cannot be replicated or compared with related uncertainty-highlighting work (e.g., Vasconcelos et al.) without this information. Please report the full correction formula and the final parameter values.
  3. [Section V-A.2] The intentional-repair analysis conflates opportunity with agency. Participants using HiLDe had more opportunities to perform UI-driven repairs because HiLDe surfaced local alternatives; the comparison of repair counts (71% vs. 29%) does not account for the number of insecure suggestions or exposed decision points in each condition. The Fisher's exact test on strategy proportions (p = 0.03, r = 0.17) appears to be computed over repair instances rather than over participants, so the effective sample size and independence assumptions are unclear. Please report per-participant repair rates or use a mixed-effects model with participant as a random effect.
minor comments (5)
  1. [Section V-B.3] The text says participants reported significantly higher trust in AI-generated code with HiLDe, but the reported means (M_HiLDe = 3.39, M_BASELINE = 3.72) indicate the opposite direction; please correct either the wording or the means.
  2. [Abstract / Section V-C] The abstract's claim that HiLDe helps users "better align code generation with their goals" is supported mainly by qualitative case studies; the self-reported measures for control, helpfulness of alternatives, and understanding were not significantly different between conditions. Please soften the abstract or report which specific goals were better aligned.
  3. [Figure 3 caption] There is a typo: "yeild" should be "yield."
  4. [Section V-B.1] The reported acceptance times (76.20 and 35.55) and task completion times (514.98 and 396.39) should include units (seconds) to avoid ambiguity.
  5. [Section IV-E] The paper reports many significance tests across security counts, survey items, NASA-TLX subscales, and repair strategies without correcting for multiple comparisons; a brief note on this or a pre-registered primary outcome would strengthen the analysis.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the security outcome is an external behavioral measurement, but the system's security-tuned prompt (Sec. VI-B) confounds the causal attribution to the highlighting interface.

full rationale

The paper's central claim is an empirical A/B result: vulnerability counts were measured externally by two authors using criteria from prior work (Sec. IV-E: 'two authors used this list to independently count the number of vulnerabilities in each participant's final solutions'), not derived from HiLDe's own highlighting or alternatives, so the reported reduction (M_HiLDE=2.67 vs M_BASELINE=3.89, p=0.01) does not equal any system input by construction. The self-referential elements are minor and disclosed. First, the corrected-entropy weighting was tuned in pilots only to limit highlight volume: 'After experimenting with parameter choices for the effect of Importance Score on the corrected entropy, we arrived at a configuration that consistently highlights only a handful of critical decision points' (Sec. III-B.2); this calibrates UI salience, not the measured security outcome, and the tuning was not fit to the evaluation data. Second, the paper admits a genuine between-condition confound: 'The underlying prompt for HILDE is slightly tuned to surface security-related decision points' (Sec. VI-B), while the baseline 'uses the same underlying model as HILDE, but without the uncertainty highlighting or local alternatives' (Sec. IV-C); this threatens attribution of the security gain to the interaction technique rather than to prompt content and is not listed among limitations in Sec. VI-D. That is a validity/identification limitation, not a circular reduction: participants could and sometimes did ignore surfaced alternatives (P7 found highlighting overwhelming; baseline participants saw secure options yet accepted insecure code), so the effect is unforced human behavior. Third, self-citations [3] and [13] serve only as background and as an interpretive lens ('exploration' and 'acceleration' modes, Sec. VI-C); neither is load-bearing for the quantitative security claim. No parameter is fitted to the evaluation data and renamed a prediction, no uniqueness theorem is imported from the authors' prior work, and the highlighting design is not justified by a self-cited ansatz. The central result therefore has independent empirical content; the honest finding is no significant circularity, with the prompt-tuning confound flagged as a correctness risk.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper does not introduce new physical or formal entities. The main load-bearing inputs are the participant sample, the vulnerability rubric, and the analysis model's judgments, all of which are assumed rather than independently validated. The corrected-entropy weighting is a tuned design parameter rather than a derived constant.

free parameters (2)
  • Importance Score weighting in corrected entropy = not reported (tuned in pilot experiments)
    Sec. III-B2 states that the authors 'experimented with parameter choices for the effect of Importance Score on the corrected entropy' and 'arrived at a configuration' that highlights only a few critical points. This tuning affects which tokens are highlighted and therefore shapes the intervention itself.
  • Number of regenerated suffixes (10) for preserving downstream code = 10
    Sec. III-B3 chooses 10 generated suffixes and keeps the one most similar to the base completion. This is a design choice that affects how often user edits are preserved, which in turn affects user experience and outcomes.
assumptions (3)
  • domain assumption The sample of 18 participants with mixed experience levels generalizes to the intended population of AI-assistant-using programmers.
    The study recruits a small convenience sample, and the causal claims depend on this sample being representative enough to support conclusions about programmers generally.
  • domain assumption The manually constructed vulnerability rubrics, adapted from prior security work, correctly identify security-relevant deficiencies in the four tasks.
    Section IV-E says two authors used a compiled list of vulnerabilities to count issues. If the rubric is incomplete or biased, the measured difference could be distorted.
  • domain assumption The analysis LLM's importance scores and explanations accurately reflect the semantic significance of token alternatives.
    Sec. III-B2 uses the Importance Score from gpt4.1-nano to compute corrected entropy and decide which tokens to highlight. If these scores are noisy or biased, the highlighting may not correspond to genuinely critical decisions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HiLDe: Intentional Code Generation via Human-in-the-Loop Decoding." pith.science (2026). https://pith.science/paper/IQMV5ALB

@misc{pith2026250522906,
  author       = {Pith},
  title        = {Pith review of: HiLDe: Intentional Code Generation via Human-in-the-Loop Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IQMV5ALB}},
  note         = {Machine review of arXiv:2505.22906}
}
read the original abstract

While AI programming tools hold the promise of increasing programmers' capabilities and productivity to a remarkable degree, they often exclude users from essential decision-making processes, causing many to effectively "turn off their brains" and over-rely on solutions provided by these systems. These behaviors can have severe consequences in critical domains, like software security. We propose Human-in-the-loop Decoding, a novel interaction technique that allows users to observe and directly influence LLM decisions during code generation, in order to align the model's output with their personal requirements. We implement this technique in HiLDe, a code completion assistant that highlights critical decisions made by the LLM and provides local alternatives for the user to explore. In a within-subjects study (N=18) on security-related tasks, we found that HiLDe led participants to generate significantly fewer vulnerabilities and better align code generation with their goals compared to a traditional code completion assistant.

Figures

Figures reproduced from arXiv: 2505.22906 by the authors.

Figure 1
Figure 1. HILDE: an assistant that highlights critical decision points in an LLM code completion, displays local alternatives the model considered at a particular decision point, explains differences between these alternatives, and lets users select a preferred alternative over the original completion, aligning code generation with their personal goals. Abstract—While AI programming tools hold the promise of increasing progra… view at source ↗
Figure 2
Figure 2. HILDE is a VSCODE extension that visualizes an LLM’s token-level uncertainty and encourages interactive exploration of code completions: Users prompt the model via comments and/or code context. As code is generated, HILDE highlights tokens where the model is uncertain in a red gradient (1)—-darker indicates higher uncertainty. Press ENTER to preview local alternative completions that the model was considering at tha… view at source ↗
Figure 3
Figure 3. The HILDE architecture. When a user requests a code completion , HILDE sends the user’s prompt to the completion LLM, receiving a base completion and the top-k tokens (with probabilities) at each generation step . For each alternative token at every step, HILDE asks the same completion LLM to generate a code preview (up to the next line break) showing what the completion would look like with that token . For each pr… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Summary of all security vulnerabilities identified in H [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Intentional security repairs using HILDE (blue) vs. BASELINE (yellow) for each repair strategy for each task [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Distribution for average time to accept a completion [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

51 extracted references · 29 canonical work pages

  1. [1]

    Beyond code generation: An observational study of chatgpt usage in software engineering practice,

    R. Khojah, M. Mohamad, P. Leitner, and F. G. de Oliveira Neto, “Beyond code generation: An observational study of chatgpt usage in software engineering practice,” Proceedings of the ACM on Software Engineering, vol. 1, no. FSE, pp. 1819–1840, 2024

    work page 2024

  2. [2]

    A large-scale survey on the usability of ai programming assistants: Successes and challenges,

    J. T. Liang, C. Yang, and B. A. Myers, “A large-scale survey on the usability of ai programming assistants: Successes and challenges,” in Proceedings of the 46th IEEE/ACM international conference on software engineering, 2024, pp. 1–13

    2024

  3. [3]

    Grounded copilot: How programmers interact with code-generating models,

    S. Barke, M. B. James, and N. Polikarpova, “Grounded copilot: How programmers interact with code-generating models,” Proceedings of the ACM on Programming Languages , vol. 7, no. OOPSLA1, pp. 85–111, 2023

    2023

  4. [4]

    Supporting sensemaking of large language model outputs at scale,

    K. I. Gero, C. Swoopes, Z. Gu, J. K. Kummerfeld, and E. L. Glassman, “Supporting sensemaking of large language model outputs at scale,” in Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, 2024, pp. 1–21

    work page 2024

  5. [5]

    Do users write more insecure code with ai assistants?

    N. Perry, M. Srivastava, D. Kumar, and D. Boneh, “Do users write more insecure code with ai assistants?” in Proceedings of the 2023 ACM SIGSAC conference on computer and communications security , 2023, pp. 2785–2799

    2023

  6. [6]

    Asleep at the keyboard? assessing the security of github copilot’s code con- tributions,

    H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, “Asleep at the keyboard? assessing the security of github copilot’s code con- tributions,” Communications of the ACM , vol. 68, no. 2, pp. 96–105, 2025

    work page 2025

  7. [7]

    Poisoned chatgpt finds work for idle hands: Exploring developers’ coding practices with inse- cure suggestions from poisoned ai models,

    S. Oh, K. Lee, S. Park, D. Kim, and H. Kim, “Poisoned chatgpt finds work for idle hands: Exploring developers’ coding practices with inse- cure suggestions from poisoned ai models,” in 2024 IEEE Symposium on Security and Privacy (SP) . IEEE, 2024, pp. 1141–1159

    work page 2024

  8. [8]

    Vulnerabilities in ai code generators: Exploring targeted data poisoning attacks,

    D. Cotroneo, C. Improta, P. Liguori, and R. Natella, “Vulnerabilities in ai code generators: Exploring targeted data poisoning attacks,” in Pro- ceedings of the 32nd IEEE/ACM International Conference on Program Comprehension, 2024, pp. 280–292

    work page 2024

Show all 51 references
  1. [9]

    Large language models for code: Security hardening and adversarial testing,

    J. He and M. Vechev, “Large language models for code: Security hardening and adversarial testing,” in Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security , 2023, pp. 1865–1879

    2023

  2. [10]

    Can we trust large language models generated code? a framework for in-context learning, security patterns, and code evaluations across diverse llms,

    A. Mohsin, H. Janicke, A. Wood, I. H. Sarker, L. Maglaras, and N. Jan- jua, “Can we trust large language models generated code? a framework for in-context learning, security patterns, and code evaluations across diverse llms,” arXiv preprint arXiv:2406.12513 , 2024

    2024 arXiv

  3. [11]

    Ai-resilient interfaces,

    E. L. Glassman, Z. Gu, and J. K. Kummerfeld, “Ai-resilient interfaces,” arXiv preprint arXiv:2405.08447 , 2024

    2024 arXiv

  4. [12]

    An ai- resilient text rendering technique for reading and skimming documents,

    Z. Gu, I. Arawjo, K. Li, J. K. Kummerfeld, and E. L. Glassman, “An ai- resilient text rendering technique for reading and skimming documents,” in Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, 2024, pp. 1–22

    2024

  5. [13]

    Validating ai-generated code with live programming,

    K. Ferdowsi, R. Huang, M. B. James, N. Polikarpova, and S. Lerner, “Validating ai-generated code with live programming,” in Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems , 2024, pp. 1–8

    2024

  6. [14]

    Neural machine translation of rare words with subword units,

    R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” arXiv preprint arXiv:1508.07909, 2015

    2015 arXiv

  7. [15]

    Sentencepiece: A simple and language inde- pendent subword tokenizer and detokenizer for neural text processing,

    T. Kudo and J. Richardson, “Sentencepiece: A simple and language inde- pendent subword tokenizer and detokenizer for neural text processing,” arXiv preprint arXiv:1808.06226 , 2018

    2018 arXiv

  8. [16]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021

    2021 arXiv

  9. [17]

    Generation probabilities are not enough: Uncertainty highlighting in ai code completions,

    H. Vasconcelos, G. Bansal, A. Fourney, Q. V . Liao, and J. W. Vaughan, “Generation probabilities are not enough: Uncertainty highlighting in ai code completions,” ACM Transactions on Computer-Human Interaction, 2024

    2024

  10. [18]

    Effect of confidence and explanation on accuracy and trust calibration in ai-assisted decision mak- ing,

    Y . Zhang, Q. V . Liao, and R. K. Bellamy, “Effect of confidence and explanation on accuracy and trust calibration in ai-assisted decision mak- ing,” in Proceedings of the 2020 conference on fairness, accountability, and transparency, 2020, pp. 295–305

    2020

  11. [19]

    On the design of ai-powered code assistants for notebooks,

    A. M. McNutt, C. Wang, R. A. Deline, and S. M. Drucker, “On the design of ai-powered code assistants for notebooks,” in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems , 2023, pp. 1–16

    2023

  12. [20]

    Do llms consider security? an empirical study on responses to programming questions,

    A. Sajadi, B. Le, A. Nguyen, K. Damevski, and P. Chatterjee, “Do llms consider security? an empirical study on responses to programming questions,” Empirical Software Engineering, vol. 30, no. 3, p. 101, 2025

    2025

  13. [21]

    Secuguard: Leveraging pattern-exploiting training in language models for advanced software vulnerability detec- tion,

    M. Basharat and M. Omar, “Secuguard: Leveraging pattern-exploiting training in language models for advanced software vulnerability detec- tion,” International Journal of Mathematics and Computer in Engineer- ing, vol. 3, no. 1, pp. 47–56, 2024

    2024

  14. [22]

    Finetuning large language models for vulnerability detection,

    A. Shestov, R. Levichev, R. Mussabayev, E. Maslov, P. Zadorozhny, A. Cheshkov, R. Mussabayev, A. Toleu, G. Tolegen, and A. Krasso- vitskiy, “Finetuning large language models for vulnerability detection,” IEEE Access, 2025

    2025

  15. [23]

    Vuldetectbench: Evaluating the deep capability of vulnerability de- tection with large language models,

    Y . Liu, L. Gao, M. Yang, Y . Xie, P. Chen, X. Zhang, and W. Chen, “Vuldetectbench: Evaluating the deep capability of vulnerability de- tection with large language models,” arXiv preprint arXiv:2406.07595 , 2024

    2024 arXiv

  16. [24]

    Examining zero-shot vulnerability repair with large language models,

    H. Pearce, B. Tan, B. Ahmad, R. Karri, and B. Dolan-Gavitt, “Examining zero-shot vulnerability repair with large language models,” in 2023 IEEE Symposium on Security and Privacy (SP) . IEEE, 2023, pp. 2339–2356

    2023

  17. [25]

    Evaluating large language models for real-world vulnerability repair in c/c++ code,

    L. Zhang, Q. Zou, A. Singhal, X. Sun, and P. Liu, “Evaluating large language models for real-world vulnerability repair in c/c++ code,” in Proceedings of the 10th ACM International Workshop on Security and Privacy Analytics, 2024, pp. 49–58

    2024

  18. [26]

    A study of vulnerability repair in javascript programs with large language models,

    T. K. Le, S. Alimadadi, and S. Y . Ko, “A study of vulnerability repair in javascript programs with large language models,” in Companion Proceedings of the ACM Web Conference 2024 , 2024, pp. 666–669

    2024

  19. [27]

    Security implications of large language model code assistants: A user study,

    G. Sandoval, H. Pearce, T. Nys, R. Karri, B. Dolan-Gavitt, and S. Garg, “Security implications of large language model code assistants: A user study,” arXiv preprint arXiv:2208.09727 , 2022

    2022 arXiv

  20. [28]

    Coladder: Manipulating code generation via multi-level blocks,

    R. Yen, J. S. Zhu, S. Suh, H. Xia, and J. Zhao, “Coladder: Manipulating code generation via multi-level blocks,” in Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology , 2024, pp. 1–20

    2024

  21. [29]

    Waitgpt: Monitoring and steering conversational llm agent in data analysis with on-the-fly code visualization,

    L. Xie, C. Zheng, H. Xia, H. Qu, and C. Zhu-Tian, “Waitgpt: Monitoring and steering conversational llm agent in data analysis with on-the-fly code visualization,” in Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology , 2024, pp. 1–14

    2024

  22. [30]

    Intention is all you need: Refining your code from your intention,

    Q. Guo, X. Xie, S. Liu, M. Hu, X. Li, and L. Bu, “Intention is all you need: Refining your code from your intention,” arXiv preprint arXiv:2502.08172, 2025

    2025 arXiv

  23. [31]

    Dynavis: Dynamically synthesized ui widgets for visualization editing,

    P. Vaithilingam, E. L. Glassman, J. P. Inala, and C. Wang, “Dynavis: Dynamically synthesized ui widgets for visualization editing,” in Pro- ceedings of the 2024 CHI Conference on Human Factors in Computing Systems, 2024, pp. 1–17

    2024

  24. [32]

    An ex- ploratory study of ml sketches and visual code assistants,

    L. F. Gomes, V . J. Hellendoorn, J. Aldrich, and R. Abreu, “An ex- ploratory study of ml sketches and visual code assistants,” arXiv preprint arXiv:2412.13386, 2024

    2024 arXiv

  25. [33]

    Improving steering and verification in ai- assisted data analysis with interactive task decomposition,

    M. Kazemitabaar, J. Williams, I. Drosos, T. Grossman, A. Z. Henley, C. Negreanu, and A. Sarkar, “Improving steering and verification in ai- assisted data analysis with interactive task decomposition,” in Proceed- ings of the 37th Annual ACM Symposium on User Interface Software...

    2024

  26. [34]

    Beyond code generation: Llm-supported exploration of the program design space,

    J. Zamfirescu-Pereira, E. Jun, M. Terry, Q. Yang, and B. Hartmann, “Beyond code generation: Llm-supported exploration of the program design space,” in Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems , 2025, pp. 1–17

    2025

  27. [35]

    Luminate: Structured generation and exploration of design space with large language models for human-ai co-creation,

    S. Suh, M. Chen, B. Min, T. J.-J. Li, and H. Xia, “Luminate: Structured generation and exploration of design space with large language models for human-ai co-creation,” in Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems , ser. CHI ’24. New York, NY...

    2024

  28. [36]

    Ivie: Lightweight anchored explanations of just-generated code,

    L. Yan, A. Hwang, Z. Wu, and A. Head, “Ivie: Lightweight anchored explanations of just-generated code,” in Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems , 2024, pp. 1–15

    2024

  29. [37]

    Biscuit: Scaffolding llm-generated code with ephemeral uis in computational notebooks,

    R. Cheng, T. Barik, A. Leung, F. Hohman, and J. Nichols, “Biscuit: Scaffolding llm-generated code with ephemeral uis in computational notebooks,” in 2024 IEEE Symposium on Visual Languages and Human- Centric Computing (VL/HCC) . IEEE, 2024, pp. 13–23

    2024

  30. [38]

    Investigating explainability of generative ai for code through scenario-based design,

    J. Sun, Q. V . Liao, M. Muller, M. Agarwal, S. Houde, K. Talamadupula, and J. D. Weisz, “Investigating explainability of generative ai for code through scenario-based design,” in Proceedings of the 27th International Conference on Intelligent User Interfaces , 2022, pp. 212–228

    2022

  31. [39]

    Expectation vs. experi- ence: Evaluating the usability of code generation tools powered by large language models,

    P. Vaithilingam, T. Zhang, and E. L. Glassman, “Expectation vs. experi- ence: Evaluating the usability of code generation tools powered by large language models,” in Chi conference on human factors in computing systems extended abstracts , 2022, pp. 1–7

    2022

  32. [40]

    ”i’m not sure, but

    S. S. Y . Kim, Q. V . Liao, M. V orvoreanu, S. Ballard, and J. W. Vaughan, “”i’m not sure, but...”: Examining the impact of large language models’ uncertainty expression on user reliance and trust,” in Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Tra...

    2024

  33. [41]

    Binary Codes Capable of Correcting Deletions, Insertions and Reversals,

    V . I. Levenshtein, “Binary Codes Capable of Correcting Deletions, Insertions and Reversals,” Soviet Physics Doklady , vol. 10, p. 707, Feb. 1966

    1966

  34. [42]

    Qwen2. 5-coder technical report,

    B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Dang et al., “Qwen2. 5-coder technical report,” arXiv preprint arXiv:2409.12186, 2024

    2024 arXiv

  35. [43]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , 2023

    2023

  36. [44]

    Github codespaces,

    GitHub, “Github codespaces,” 2024. [Online]. Available: https: //github.com/features/codespaces

    2024

  37. [45]

    Development of nasa-tlx (task load index): Results of empirical and theoretical research,

    S. G. Hart and L. E. Staveland, “Development of nasa-tlx (task load index): Results of empirical and theoretical research,” in Advances in psychology. Elsevier, 1988, vol. 52, pp. 139–183

    1988

  38. [46]

    Using thematic analysis in psychology,

    V . Braun and V . Clarke, “Using thematic analysis in psychology,” Qualitative research in psychology , vol. 3, no. 2, pp. 77–101, 2006

    2006

  39. [47]

    Content analysis and thematic analysis: Implications for conducting a qualitative descriptive study,

    M. Vaismoradi, H. Turunen, and T. Bondas, “Content analysis and thematic analysis: Implications for conducting a qualitative descriptive study,” Nursing & health sciences , vol. 15, no. 3, pp. 398–405, 2013

    2013

  40. [48]

    The impact of ai on developer productivity: Evidence from github copilot,

    S. Peng, E. Kalliamvakou, P. Cihon, and M. Demirer, “The impact of ai on developer productivity: Evidence from github copilot,” arXiv preprint arXiv:2302.06590, 2023

    2023 arXiv

  41. [49]

    Trade-offs for substi- tuting a human with an agent in a pair programming context: the good, the bad, and the ugly,

    S. K. Kuttal, B. Ong, K. Kwasny, and P. Robe, “Trade-offs for substi- tuting a human with an agent in a pair programming context: the good, the bad, and the ugly,” in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems , 2021, pp. 1–20

    2021

  42. [50]

    Measuring github copilot’s impact on productivity,

    A. Ziegler, E. Kalliamvakou, X. A. Li, A. Rice, D. Rifkin, S. Simister, G. Sittampalam, and E. Aftandilian, “Measuring github copilot’s impact on productivity,”Communications of the ACM, vol. 67, no. 3, pp. 54–63, 2024

    2024

  43. [51]

    Assistance or disruption? exploring and evaluating the design and trade-offs of proactive ai programming support,

    K. Pu, D. Lazaro, I. Arawjo, H. Xia, Z. Xiao, T. Grossman, and Y . Chen, “Assistance or disruption? exploring and evaluating the design and trade-offs of proactive ai programming support,” arXiv preprint arXiv:2502.18658, 2025

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.