Pith. sign in

REVIEW 3 major objections 6 minor 55 references

When comments and code disagree, language models often follow the misleading cue—and the conflict is recoverable from a small set of internal residual-stream states.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 05:22 UTC pith:KRJX5ZU4

load-bearing objection Clean first causal study of cue-vs-code conflicts in LLMs; the behavioral drop and residual staging are real on their stimuli, with external validity—not circular math—as the main limit. the 3 major comments →

arxiv 2607.05587 v1 pith:KRJX5ZU4 submitted 2026-07-06 cs.SE

A Mechanistic Lens on Semantic Conflicts: Using Activation Patching to Understand LLM Behavior

Youssef Abdelsalam , Norman Peitek , Anna-Maria Maurer , Marvin Wyrich , Sven Apel This is my paper
classification cs.SE
keywords large language modelsmechanistic interpretabilitysemantic conflictsprogram comprehensionactivation patchingunit-test generationcode commentssoftware engineering
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks what large language models do when executable code and non-executable semantic cues (comments, names) point to different behavior. Using 45 carefully paired Python snippet triplets and four open-weight models, it shows that such conflicts sharply cut execution-grounded correctness on both final-output prediction and unit-test generation, with many errors matching the misleading cue rather than the running code. The authors then intervene inside the models with residual-stream activation patching: swapping internal states between aligned and conflicting prompts reveals a staged pattern in which the edited cue or code region, a sparse set of intermediate carrier tokens, and the late readout site carry most of the recoverable causal signal. For generated tests, that pattern continues into the assertion prefix before the expected value is written. The practical stake is clear for AI-assisted development: stale docs and misleading names are not just noise—they can steer models toward wrong but plausible behavior, and the paper offers a causal way to localize where that decision takes shape.

Core claim

Semantic conflicts between cues and implementation significantly reduce execution-grounded correctness on output prediction and unit-test generation across four open-weight LLMs, and models frequently produce cue-consistent errors. Residual-stream activation patching recovers a consistent multi-stage causal pattern: early recovery at the changed cue or code region, middle-layer recovery at sparse intermediate carrier tokens (including generated assertion-prefix sites for tests), and late aggregation at the readout where the final answer is produced.

What carries the argument

Residual-stream activation patching on token-aligned aligned/conflicting Python snippet pairs: residual activations at matched token-layer sites are swapped between prompts, and a recovery score measures how much the destination model’s preference shifts toward the source behavior, localizing which states causally carry the conflict.

Load-bearing premise

The claim rests on 45 hand-built, minimal, token-aligned Python micro-examples that were selected so at least one model already shows a cue-consistent error—so the staged internal pattern may not hold for larger, messier, real-world conflicts.

What would settle it

Run the same residual-stream patching protocol on a larger set of naturally occurring code-comment mismatches (not constructed micro-triplets) and check whether the early-changed-region / sparse-carrier / late-readout recovery staging still appears with comparable strength and sparsity.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies how open-weight LLMs behave when executable code and non-executable semantic cues (comments/identifiers) disagree. The authors construct 45 token-aligned Python triplets (aligned, cue-varied, implementation-varied), evaluate four ~7–8B models on final-output prediction and unit-test generation, and use residual-stream activation patching to localize token–layer states that causally shift outputs between paired conditions. Behaviorally, conflicts reduce execution-grounded correctness and often yield cue-consistent errors; mechanistically, recoverable conflict signal concentrates at the edited region (early layers), sparse intermediate “carrier” tokens (middle layers), and the readout (late layers), with an analogous pattern extending into generated assertion prefixes for tests.

Significance. If the results hold under clearer stimulus accounting and broader settings, this is a useful methodological contribution at the SE–mechanistic-interpretability boundary: it shows how controlled, token-aligned code contrasts plus residual-stream patching can turn ambiguous cue/code reliance into a causal localization problem rather than only a behavioral failure mode. Strengths include a paired experimental design, McNemar/Wilcoxon tests with Holm correction, bidirectional patching, recovery-threshold and margin-gap sensitivity checks, dual tasks (comprehension and test generation), and a replication package with the 45 triplets. The staged localization pattern and the observation that conflict information is recoverable at generated assertion sites are concrete, falsifiable findings that can guide follow-up circuit work and conflict-aware tooling.

major comments (3)
  1. Section IV-A and VII-B state that each pair must produce a meaningful conflict for at least one studied LLM, and VII-B further requires aligned-correct plus cue-consistent error on the conflicting variant for that model. This selection rule is load-bearing for RQ1.1/RQ2.1: large correctness drops and high cue-consistent error rates are partly expected by construction for selector models, so the claim that conflicts “significantly reduce” correctness and are “frequently directed toward the misleading cue” is not fully independent of filtering. Please report the pre-filter candidate pool, discard rates, which model(s) licensed each pair, and either (i) behavioral results restricted to non-selector model–pair cells or (ii) a clearly scoped claim that findings characterize this contrastive set rather than arbitrary real-world conflicts.
  2. Relatedly, the abstract and RQ1.1/RQ2.1 framing present the behavioral effect as a general discovery about LLM program comprehension under semantic conflict, while the mechanistic staging (RQ1.2/RQ2.2) is measured on pairs deliberately chosen so the conflict is behaviorally decisive. Activation patching then localizes information that was selected to flip outputs. The staging result can still be informative, but the manuscript should separate (a) existence of a recoverable staged pathway on decisive micro-conflicts from (b) prevalence/severity of cue-following in unfiltered or multi-site conflicts, and temper generalization language in the abstract, §V, and §VIII accordingly (cf. VII-C).
  3. Table I / Figure 2 and Table II / Figure 5 report strong effects on 45 hand-curated minimal snippets. For the central SE claim—that such conflicts matter for downstream artifacts—the unit-test results are important, yet assertion inputs are model-chosen and many tests are both/non-discriminating (Fig. 5). The paper notes this, but does not quantify how much of the pass-rate drop is cue-following versus input avoidance versus brittle formatting. A breakdown of pass-rate change attributable to cue-consistent vs non-discriminating vs neither assertions (per model/conflict) is needed so RQ2.1 is not over-read as pure semantic-cue dominance.
minor comments (6)
  1. Define the recovery score formula in the main text (currently deferred to the replication package). Even a short equation for recovery and Δ = m_S − m_D in §IV-E would make the 0.3 threshold and τ filters interpretable without external files.
  2. Figure 3 is a single representative heatmap; a small multi-example appendix figure (one cue-varied, one implementation-varied, two models) would better support the claim of a “consistent pattern.”
  3. Clarify tokenization of multi-token edits: §IV-E says consecutive changed tokens are one patch unit—state whether partial-token or subword splits ever forced redesign of a stimulus and how often.
  4. “Carrier tokens” are a useful operational category; add an explicit decision rule (max recovery ≥ 0.3 over layers, excluding changed/readout sites) in §IV-E so the term is not only introduced in Results.
  5. Minor prose: abstract says “first controlled, mechanistic study” while intro says “among the first”—align the priority claim. Also fix occasional grammar (“a models’s algorithm,” “incite a cue–implementation conflict”).
  6. Report decoding details for unit-test generation beyond temp=0 (max tokens, stop sequences, whether few-shot examples were used), since assertion form affects RQ2.2 prompt construction.

Circularity Check

1 steps flagged

Empirical intervention study: recovery and staging are measured against independently defined execution/cue candidates; stimulus filtering ensures a patchable contrast but does not force the staged localization by construction.

specific steps
  1. other [Section IV-A; VII-B Internal Validity]
    "To ensure meaningful contrasts, each pair must incite a cue–implementation conflict for, at least, one studied LLM for output generation. [...] During dataset construction, each snippet pair had to show the intended behavioral contrast for at least one studied LLM. The same LLM had to answer the aligned variant correctly and produce a cue-consistent error on the conflicting variant, ensuring a meaningful patching contrast."

    Stimulus filtering requires that the intended behavioral flip already occurs for ≥1 model before a pair enters the set. This can inflate the reported McNemar/Wilcoxon effect magnitudes relative to an unfiltered sample of conflicts, and it guarantees a non-trivial destination–source margin for patching. It is selection for contrast, not a fitted parameter renamed as prediction, and it does not define recovery staging in terms of the labels—so it is only a mild circularity-adjacent design choice, not a load-bearing self-definitional loop.

full rationale

This paper is a controlled behavioral-plus-causal intervention study, not a first-principles derivation. Execution- and cue-consistent labels are defined from program execution and the paired cue/implementation variants, not from the patching metric. Residual-stream recovery scores compare patched vs unpatched logits on those fixed candidates; no free parameters are fitted to produce the early-changed / middle-carrier / late-readout staging, and bidirectional patching plus multi-model reporting further separate measurement from construction. The only mild circularity-adjacent concern is dataset construction: pairs were required to produce a cue–implementation conflict for at least one studied LLM so that a meaningful patching contrast exists. That is standard contrast selection for causal mediation, not a self-definitional or fitted-input prediction, and it does not make the localization pattern equivalent to the inputs by construction. Self-citations are methodological background (activation patching, TransformerLens, related SE work), not load-bearing uniqueness theorems. Score 1 reflects that minor selection filter only; the central claims remain independently measured.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 1 invented entities

The central claims rest on standard causal-intervention methodology, public model weights, and a hand-constructed stimulus set. No new physical entities or fitted physical constants; free parameters are analysis thresholds chosen for reporting, not fitted to invent the staging pattern. Domain assumptions about residual-stream sufficiency and token alignment are explicit.

free parameters (2)
  • recovery_score_threshold = 0.3 (primary)
    Primary cutoff 0.3 (with sensitivity at 0.2 and 0.5) used to define carrier tokens and reportable sites; chosen by authors, not derived.
  • margin_gap_filter_tau = 0.10 (primary)
    Minimum |Δ| = 0.10 logits (robustness at 0.05/0.25) to avoid inflated normalized recovery; analysis hyperparameter.
axioms (4)
  • domain assumption Residual-stream activations at token-layer sites are a sufficient causal channel for localizing conflict-relevant information via activation patching.
    Standard in mechanistic interpretability (Elhage et al., Meng et al.); paper acknowledges it abstracts over heads/MLPs/paths (Section VII-A).
  • domain assumption Token-aligned, localized 1–2 token edits preserve syntactic validity and isolate cue-vs-implementation conflict without introducing uncontrolled confounds.
    Core design premise of the 45 triplets (Section IV-A).
  • domain assumption Execution behavior is the ground-truth reference for correctness; cue-suggested behavior is an independent contrast label.
    Stated evaluation policy for both tasks (Section IV-C).
  • standard math Paired non-parametric tests (McNemar, Wilcoxon) with Holm correction adequately control family-wise error for the reported comparisons.
    Standard statistical practice invoked in Section IV-D.
invented entities (1)
  • carrier tokens (prompt carriers / response carriers) no independent evidence
    purpose: Label sparse intermediate residual sites that recover substantial conflict signal between the changed region and the readout.
    Operational definition based on recovery threshold; useful descriptive construct, not a new physical entity. Independent evidence is the patching recovery itself within this study.

pith-pipeline@v1.1.0-grok45 · 22820 in / 2853 out tokens · 25210 ms · 2026-07-11T05:22:09.423273+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) are increasingly used in software-engineering tasks processing executable code and non-executable semantic cues such as comments or identifiers. These two sources can conflict when semantic cues suggest different program behavior than the code itself. It remains unclear how such semantic conflicts affect LLM behavior and which source dominates their outputs. We present the first controlled, mechanistic study of LLM behavior under semantic conflicts. To this end, we construct 45 Python snippet triplets that isolate conflicts by varying either semantic cues or implementation while keeping token-aligned pairs for causal intervention. We evaluate four open-weight LLMs on two tasks (output prediction and unit-test generation) using behavioral performance measures and residual-stream activation patching to identify token-layer states that causally contribute to behavioral differences between aligned and conflicting inputs. Our results show that semantic conflicts significantly reduce execution-grounded correctness in both tasks and that all tested LLMs often follow misleading semantic cues. Residual-stream activation patching reveals a consistent pattern for final-output prediction: The changed cue/code region and a small set of intermediate tokens carry most of the recoverable causal signal before aggregation near the output readout. For unit-test generation, this pattern extends beyond the prompt, showing that conflict-related information is recoverable at generated sites before producing expected values. Overall, our findings show that semantic conflicts affect program comprehension and downstream tasks, with relevant information concentrated in a small number of causally active residual-stream states, and demonstrate a framework for mechanistically analyzing how LLMs integrate code-related information under controlled semantic variations.

Figures

Figures reproduced from arXiv: 2607.05587 by Anna-Maria Maurer, Marvin Wyrich, Norman Peitek, Sven Apel, Youssef Abdelsalam.

Figure 1
Figure 1. Figure 1: Overview of our experimental framework. We construct 45 Python snippet triplets (aligned, implementation-varied, and cue-varied) and provide them [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: RQ1.1 Distribution of response labels on conflicting prompts. Cue￾consistent responses are incorrect under the execution-grounded evaluation but indicate sensitivity to the misleading cue. [Context] [...] odd -> even result . def compute (x ): <3 spaces> return x% <space> 2 == <space> 0 <2 newlines> print (com pute ( 4 )) `` ` Output : 0 5 10 15 20 25 Layer -1 -0.5 0 0.5 1 signed recovery [PITH_FULL_IMAGE… view at source ↗
Figure 3
Figure 3. Figure 3: RQ1.2 Representative residual-recovery heatmap for Qwen2.5 7B, cue-varied pair 001, patching the conflicting-cue source into the aligned-cue destination. The changed cue token recovers in early layers, token 0 exhibits a later recovery band, and the readout token recovers in the final layers. exhibit a directed bias under conflict, with many failures aligning with the incorrect but semantically salient cue… view at source ↗
Figure 4
Figure 4. Figure 4: RQ1.2 Site-group comparison at the primary carrier threshold. Changed region peaks early, carrier tokens in middle layers, and readout site late; recovery is strongest at changed and readout sites and substantial for carriers. number of intermediate tokens before being aggregated at the readout site. Analyzing the recovery distribution across layers shows the same staged pattern. Median best recovery occur… view at source ↗
Figure 5
Figure 5. Figure 5: RQ2.1 Assertion labels for conflicting generated unit tests, in which both/non-discriminating assertions are runtime-correct assertions and share the same expected value under both execution semantics and misleading cue. as shown in Table II. Aligned assertions result in a pass rate between 77% to almost 90%. In comparison, the cue￾varied pass rate is reduced by 18.5 to 31.9 percentage points; this reducti… view at source ↗
Figure 6
Figure 6. Figure 6: RQ2.2 Assertion-specific staging and recovery strength of conflict information during unit-test generation at the primary carrier threshold. while adding a distinct response-generation stage before the readout site. Median best recovery occurs around layer 2 at changed-prompt sites, around layers 8–10 at prompt carriers, around layers 11–13 at response carriers, and at the final layer (27 or 31) at the exp… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

55 extracted references · 11 linked inside Pith

  1. [1]

    Large Language Models for Software Engineering: A Systematic Literature Review,

    X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Liet al., “Large Language Models for Software Engineering: A Systematic Literature Review,” Transactions on Software Engineering and Methodology (TOSEM), vol. 33, no. 8, pp. 1–79, 2024

    2024

  2. [2]

    A Survey on Large Language Models for Software Engineering,

    Q. Zhang, C. Fang, Y . Xie, Y . Zhang, S. Yu, W. Sunet al., “A Survey on Large Language Models for Software Engineering,”Science China Information Sciences, vol. 69, no. 4, p. 141102, 2026

    2026

  3. [3]

    Software Testing with Large Language Models: Survey, Landscape, and Vision,

    J. Wang, Y . Huang, C. Chen, Z. Liu, S. Wang, and Q. Wang, “Software Testing with Large Language Models: Survey, Landscape, and Vision,” Transactions on Software Engineering (TSE), vol. 50, no. 4, pp. 911– 936, 2024

    2024

  4. [4]

    A Systematic Approach for Assessing Large Language Models’ Test Case Generation Capability,

    H.-F. Chang and M. Shokrolah Shirazi, “A Systematic Approach for Assessing Large Language Models’ Test Case Generation Capability,” Software, vol. 4, no. 1, p. 5, 2025

    2025

  5. [5]

    Automated Repair of Programs from Large Language Models,

    Z. Fan, X. Gao, M. Mirchev, A. Roychoudhury, and S. H. Tan, “Automated Repair of Programs from Large Language Models,” inProc. International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 1469–1481

    2023

  6. [6]

    ThinkRepair: Self- Directed Automated Program Repair,

    X. Yin, C. Ni, S. Wang, Z. Li, L. Zeng, and X. Yang, “ThinkRepair: Self- Directed Automated Program Repair,” inProc. International Symposium on Software Testing and Analysis (ISSTA), 2024, pp. 1274–1286

    2024

  7. [7]

    LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision, and the Road Ahead,

    J. He, C. Treude, and D. Lo, “LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision, and the Road Ahead,” Transactions on Software Engineering and Methodology (TOSEM), vol. 34, no. 5, May 2025

    2025

  8. [8]

    Large Language Model-Based Agents for Software Engineering: A Sur- vey,

    J. Liu, K. Wang, Y . Chen, X. Peng, Z. Chen, L. Zhang, and Y . Lou, “Large Language Model-Based Agents for Software Engineering: A Sur- vey,”Transactions on Software Engineering and Methodology (TOSEM), 2026

    2026

  9. [9]

    Precise Documentation: The Key to Better Software,

    D. L. Parnas, “Precise Documentation: The Key to Better Software,” in The Future of Software Engineering. Springer, 2010, pp. 125–148

    2010

  10. [10]

    Are Your Comments Out- dated? Toward Automatically Detecting Code-Comment Consistency,

    Y . Huang, Y . Chen, X. Chen, and X. Zhou, “Are Your Comments Out- dated? Toward Automatically Detecting Code-Comment Consistency,” Journal of Software: Evolution and Process, vol. 37, no. 1, p. e2718, 2025

    2025

  11. [11]

    Software Documentation: The Practitioners’ Perspec- tive,

    E. Aghajani, C. Nagy, M. Linares-V ´asquez, L. Moreno, G. Bavota, M. Lanzaet al., “Software Documentation: The Practitioners’ Perspec- tive,” inProc. International Conference on Software Engineering (ICSE), 2020, pp. 590–601

    2020

  12. [12]

    Towards Detecting Inconsistent Comments in Java Source Code Automatically,

    N. Stulova, A. Blasi, A. Gorla, and O. Nierstrasz, “Towards Detecting Inconsistent Comments in Java Source Code Automatically,” inInterna- tional Working Conference on Source Code Analysis and Manipulation (SCAM). IEEE, 2020, pp. 65–69

    2020

  13. [13]

    Two Sides of the Same Coin: Exploiting the Impact of Identifiers in Neural Code Comprehension,

    S. Gao, C. Gao, C. Wang, J. Sun, D. Lo, and Y . Yu, “Two Sides of the Same Coin: Exploiting the Impact of Identifiers in Neural Code Comprehension,” inInternational Conference on Software Engineering (ICSE). IEEE, 2023, pp. 1933–1945

    2023

  14. [14]

    When Names Disappear: Revealing What LLMs Actually Understand About Code,

    C. C. Le, M. V . Pham, C. D. Van, H. N. Phan, H. N. Phan, and T. N. Nguyen, “When Names Disappear: Revealing What LLMs Actually Understand About Code,”arXiv preprint arXiv:2510.03178, 2025

    arXiv 2025

  15. [15]

    Bugs in Large Language Models Generated Code: An Empirical Study,

    F. Tambon, A. Moradi-Dakhel, A. Nikanjam, F. Khomh, M. Desmarais, and G. Antoniol, “Bugs in Large Language Models Generated Code: An Empirical Study,”Empirical Software Engineering, vol. 30, no. 3, p. 65, 2025

    2025

  16. [16]

    What Is Wrong with Your Code Generated by Large Language Models? An Extensive Study,

    S. Dou, H. Jia, S. Wu, H. Zheng, M. Wu, Y . Tao, M. Zhanget al., “What Is Wrong with Your Code Generated by Large Language Models? An Extensive Study,”Science China Information Sciences, vol. 69, no. 1, p. 112107, 2026

    2026

  17. [17]

    Towards Best Practices of Activation Patching in Language Models: Metrics and Methods,

    F. Zhang and N. Nanda, “Towards Best Practices of Activation Patching in Language Models: Metrics and Methods,” inInternational Conference on Learning Representations (ICLR), 2024, pp. 1651–1678

    2024

  18. [18]

    How to Use and Interpret Activation Patching,

    S. Heimersheim and N. Nanda, “How to Use and Interpret Activation Patching,”arXiv preprint arXiv:2404.15255, 2024

    Pith/arXiv arXiv 2024

  19. [19]

    Localizing Model Behavior with Path Patching,

    N. Goldowsky-Dill, C. MacLeod, L. Sato, and A. Arora, “Localizing Model Behavior with Path Patching,”arXiv preprint arXiv:2304.05969, 2023

    Pith/arXiv arXiv 2023

  20. [20]

    Towards Automated Circuit Discovery for Mechanistic In- terpretability,

    A. Conmy, A. Mavor-Parker, A. Lynch, S. Heimersheim, and A. Garriga- Alonso, “Towards Automated Circuit Discovery for Mechanistic In- terpretability,”Advances in Neural Information Processing Systems, vol. 36, pp. 16 318–16 352, 2023

    2023

  21. [21]

    Replication Package - Mechanistic Lens and Semantic Conflicts,

    Y . Abdelsalam, N. Peitek, A.-M. Maurer, M. Wyrich, and S. Apel, “Replication Package - Mechanistic Lens and Semantic Conflicts,” Jun. 2026. [Online]. Available: https://github.com/brains-on-code/ mechanistic-interpretability-semantic-conflicts

    2026

  22. [22]

    Evaluating Large Language Models Trained on Code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplanet al., “Evaluating Large Language Models Trained on Code,”arXiv preprint arXiv:2107.03374, 2021

    Pith/arXiv arXiv 2021

  23. [23]

    Program Synthesis with Large Language Models,

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan et al., “Program Synthesis with Large Language Models,”arXiv preprint arXiv:2108.07732, 2021

    Pith/arXiv arXiv 2021

  24. [24]

    Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation,

    J. Liu, C. S. Xia, Y . Wang, and L. Zhang, “Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 36, 2023, pp. 21 558–21 572

    2023

  25. [25]

    TestE- val: Benchmarking Large Language Models for Test Case Generation,

    W. Wang, C. Yang, Z. Wang, Y . Huang, Z. Chu, D. Songet al., “TestE- val: Benchmarking Large Language Models for Test Case Generation,” inFindings of the Association for Computational Linguistics: NAACL 2025, L. Chiruzzo, A. Ritter, and L. Wang, Eds. Association for Computational Linguistics, Apr. 2025, pp. 3547–3562

    2025

  26. [26]

    Benchmarking LLMs for Unit Test Generation from Real-World Functions,

    D. Huang, J. Zhang, M. Harman, Q. Zhang, M. Du, and S.-K. Ng, “Benchmarking LLMs for Unit Test Generation from Real-World Functions,”Transactions on Software Engineering and Methodology (TOSEM), Mar. 2026

    2026

  27. [27]

    TestForge: Feedback-Driven, Agentic Test Suite Generation,

    K. Jain and C. Le Goues, “TestForge: Feedback-Driven, Agentic Test Suite Generation,”arXiv preprint arXiv:2503.14713, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    Towards More Effective Fault Detection in LLM-Based Unit Test Generation,

    G. Wang, Q. Xu, L. Briand, and K. Liu, “Towards More Effective Fault Detection in LLM-Based Unit Test Generation,”arXiv preprint arXiv:2506.02954, 2025

    Pith/arXiv arXiv 2025

  29. [29]

    Correctness Assessment of Code Generated by Large Language Models Using Internal Representations,

    T.-D. Bui, T. T. Vu, T.-T. Nguyen, S. Nguyen, and H. D. V o, “Correctness Assessment of Code Generated by Large Language Models Using Internal Representations,”Journal of Systems and Software, p. 112570, 2025

    2025

  30. [30]

    Model- Agnostic Quality Assessment for LLM-Generated Code via Dynamic Internal Representation Selection,

    T. T. Vu, T.-D. Bui, T.-T. Nguyen, S. Nguyen, and H. D. V o, “Model- Agnostic Quality Assessment for LLM-Generated Code via Dynamic Internal Representation Selection,”Journal of Systems and Software, p. 112852, 2026

    2026

  31. [31]

    On LLMs’ Internal Rep- resentation of Code Correctness,

    F. Ribeiro, C. Spiess, P. Devanbu, and S. Nadi, “On LLMs’ Internal Rep- resentation of Code Correctness,” inProc. of International Conference on Software Engineering (ICSE), 2026

    2026

  32. [32]

    Mechanistic Interpretability of Code Correctness in LLMs via Sparse Autoencoders,

    K. Tahimic and C. Cheng, “Mechanistic Interpretability of Code Correctness in LLMs via Sparse Autoencoders,”arXiv preprint arXiv:2510.02917, 2025

    arXiv 2025

  33. [33]

    CodeCircuit: Toward Inferring LLM-Generated Code Correctness via Attribution Graphs,

    Y . He, Z. Zhao, Z. Kaiyu, B. Dai, J. Fu, and Y . Yang, “CodeCircuit: Toward Inferring LLM-Generated Code Correctness via Attribution Graphs,”arXiv preprint arXiv:2602.07080, 2026

    arXiv 2026

  34. [34]

    Investigating Gender Bias in Language Models Using Causal Mediation Analysis,

    J. Vig, S. Gehrmann, Y . Belinkov, S. Qian, D. Nevo, S. Sakenis et al., “Investigating Gender Bias in Language Models Using Causal Mediation Analysis,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 33. Curran Associates, Inc., 2020, pp. 9583– 9595

    2020

  35. [35]

    Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability,

    A. Geiger, D. Ibeling, A. Zur, M. Chaudhary, S. Chauhan, J. Huang et al., “Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability,”Journal of Machine Learning Research, vol. 26, no. 83, pp. 1–64, 2025

    2025

  36. [36]

    Locating and Editing Factual Associations in GPT,

    K. Meng, D. Bau, A. Andonian, and Y . Belinkov, “Locating and Editing Factual Associations in GPT,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 35. Curran Associates, Inc., 2022, pp. 17 359–17 372

    2022

  37. [37]

    In- terpretability in the Wild: A Circuit for Indirect Object Identification in GPT-2 Small,

    K. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt, “In- terpretability in the Wild: A Circuit for Indirect Object Identification in GPT-2 Small,” inInternational Conference on Learning Representations (ICLR), 2023

    2023

  38. [38]

    A Mathematical Framework for Transformer Circuits,

    N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mannet al., “A Mathematical Framework for Transformer Circuits,” Transformer Circuits Thread, 2021. [Online]. Available: https://transformer-circuits.pub/2021/framework/index.html

    2021

  39. [39]

    Scaling and Evaluating Sparse Autoencoders,

    L. Gao, T. D. la Tour, H. Tillman, G. Goh, R. Troll, A. Radford et al., “Scaling and Evaluating Sparse Autoencoders,” inInternational Conference on Learning Representations (ICLR), 2025, pp. 26 721– 26 754

    2025

  40. [40]

    An Empir- ical Study on Code Comment Completion,

    A. Mastropaolo, E. Aghajani, L. Pascarella, and G. Bavota, “An Empir- ical Study on Code Comment Completion,” inInternational Conference on Software Maintenance and Evolution (ICSME). IEEE, 2021, pp. 159–170

    2021

  41. [41]

    Code Comment Inconsistency Detection with BERT and Longformer,

    T. Steiner and R. Zhang, “Code Comment Inconsistency Detection with BERT and Longformer,”arXiv preprint arXiv:2207.14444, 2022

    Pith/arXiv arXiv 2022

  42. [42]

    The Effect of Comments on Program Comprehension: An Eye-tracking Study,

    Y . Abdelsalam, N. Peitek, A. Bergum, and S. Apel, “The Effect of Comments on Program Comprehension: An Eye-tracking Study,” Empirical Software Engineering, vol. 31, no. 4, p. 94, Mar 2026

    2026

  43. [43]

    9.6 Million Links in Source Code Comments: Purpose, Evolution, and Decay,

    H. Hata, C. Treude, R. G. Kula, and T. Ishio, “9.6 Million Links in Source Code Comments: Purpose, Evolution, and Decay,” inProc. International Conference on Software Engineering (ICSE). IEEE, 2019, pp. 1211–1221

    2019

  44. [44]

    Code Llama: Open Foundation Models for Code,

    B. Rozi `ere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan et al., “Code Llama: Open Foundation Models for Code,”arXiv preprint arXiv:2308.12950, 2024

    Pith/arXiv arXiv 2024

  45. [45]

    Qwen2 Technical Report,

    A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhouet al., “Qwen2 Technical Report,”arXiv preprint arXiv:2407.10671, 2024

    Pith/arXiv arXiv 2024

  46. [46]

    Mistral 7b,

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casaset al., “Mistral 7b,”arXiv preprint arXiv:2310.06825, 2023

    Pith/arXiv arXiv 2023

  47. [47]

    The Llama 3 Herd of Models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al- Dahle, A. Letmanet al., “The Llama 3 Herd of Models,”arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  48. [48]

    TransformerLens,

    N. Nanda and J. Bloom, “TransformerLens,” https://github.com/ TransformerLensOrg/TransformerLens, 2022

    2022

  49. [49]

    Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages,

    Q. McNemar, “Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages,”Psychometrika, vol. 12, no. 2, pp. 153–157, 1947

    1947

  50. [50]

    Cohen,Statistical Power Analysis for the Behavioral Sciences

    J. Cohen,Statistical Power Analysis for the Behavioral Sciences. Lawrence Erlbaum Associates, 1977

    1977

  51. [51]

    Individual Comparisons by Ranking Methods,

    F. Wilcoxon, “Individual Comparisons by Ranking Methods,”Biometrics Bulletin, vol. 1, no. 6, pp. 80–83, 1945

    1945

  52. [52]

    A Simple Sequentially Rejective Multiple Test Procedure,

    S. Holm, “A Simple Sequentially Rejective Multiple Test Procedure,” Scandinavian Journal of Statistics, pp. 65–70, 1979

    1979

  53. [53]

    An Empirical Study of the Non-Determinism of ChatGPT in Code Generation,

    S. Ouyang, J. Zhang, M. Harman, and M. Wang, “An Empirical Study of the Non-Determinism of ChatGPT in Code Generation,”Transactions on Software Engineering and Methodology (TOSEM), vol. 34, no. 2, pp. 1–28, 2025

    2025

  54. [54]

    40 Years of Designing Code Comprehension Experiments: A Systematic Mapping Study,

    M. Wyrich, J. Bogner, and S. Wagner, “40 Years of Designing Code Comprehension Experiments: A Systematic Mapping Study,”ACM Comput. Surv., vol. 56, no. 4, Nov. 2023

    2023

  55. [55]

    Open Science in Software Engineering,

    D. Mendez, D. Graziotin, S. Wagner, and H. Seibold, “Open Science in Software Engineering,” inContemporary Empirical Methods in Software Engineering. Cham: Springer International Publishing, 2020, pp. 477– 501

    2020