Pith. sign in

REVIEW 3 major objections 7 minor 55 references

Removing LLM refusal changes vulnerability analysis quality, not just answer rate

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-08 22:39 UTC pith:DMRTOJK6

load-bearing objection Same-lineage study of aligned vs. abliterated LLMs for vulnerability analysis; good design, no inferential statistics on small samples the 3 major comments →

arxiv 2607.05842 v1 pith:DMRTOJK6 submitted 2026-07-07 cs.SE cs.AIcs.CR

Beyond Refusal: A Same-Lineage Study of Aligned and Abliterated LLMs for Vulnerability Analysis

Mingchen Li , Meikang Qiu , Zifan Peng , Heng Fan , Song Fu , Junhua Ding , Yunhe Feng This is my paper
classification cs.SE cs.AIcs.CR
keywords abliteratedalignedsafetywhetherlocalizationmodelmodelsterminology
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper investigates whether a large language model's safety alignment — specifically, whether its refusal behavior is intact (Aligned) or surgically suppressed (Abliterated) — changes the quality of its output on legitimate software-security tasks, beyond simply changing whether it answers at all. The authors compare same-lineage model pairs (Gemma and Qwen) where the only intended difference is the suppression of a learned refusal direction in activation space. They evaluate both states across a ladder of tasks: vulnerability detection, CWE attribution, vulnerable-line localization, root-cause localization, and executable patch generation with compilation and test validation. They also vary prompt wording from neutral code-review language to security-terminology-dense framings with explicit authorization context. The central finding is that safety-state effects are context-dependent rather than monotonic. The aligned state retains an advantage on shallow diagnostic tasks (detection, CWE classification) under neutral wording. The abliterated state becomes stronger as tasks get more code-grounded and actionable: it improves vulnerable-line localization in both model families and produces substantially more patches that parse, apply, and compile in the Java repair pipeline (67.8% usable vs 29.9% for aligned; 32.8% compiled vs 9.0%). The advantage for the abliterated state is driven primarily by security-explicit prompt framings — when prompts use professional cybersecurity terminology or authorization language, the aligned model's throughput drops sharply. The paper also shows that prompt wording changes not just scores but the concrete answers themselves: the aligned model exhibits higher decision-flip rates and label drift when terminology shifts from neutral to security-dense, suggesting safety mechanisms respond to surface cues associated with cyber misuse, not just to the legitimacy of the task.

Core claim

The paper's central discovery is that suppressing the refusal direction in an LLM changes the model's utility profile across a vulnerability-analysis workflow in ways that refusal rate alone cannot capture. The effect is not uniform: it depends on task depth and prompt framing. For shallow classification tasks under neutral wording, alignment helps. For code-grounded localization and early-stage executable repair — especially when prompts use security terminology or authorization context — abliteration helps, sometimes dramatically. The aligned model also shows greater output instability (decision flips, label drift) when prompt wording shifts toward cybersecurity language, indicating that a

What carries the argument

The central object is the refusal direction — a vector in the model's activation space identified by contrasting hidden representations of refusal-eliciting prompts against assistance-permitting prompts. Suppressing this direction (via activation projection or weight-space surgery) produces the Abliterated state. The study's measurement framework decomposes model utility into coverage (does the model return a usable answer), answer quality (is the non-refused answer correct), and end-to-end utility (does the output survive executable validation). Two experimental axes cross this decomposition: a task-depth ladder from binary detection through CWE attribution, line localization, root-cause,

Load-bearing premise

The study assumes that the publicly released abliterated model artifacts differ from their aligned counterparts only in the suppression of the refusal direction. The paper itself acknowledges this is not a perfectly isolated causal intervention: the abliteration process and subsequent quantization may have altered other model capabilities, meaning the observed differences cannot be cleanly attributed to safety state alone.

What would settle it

If the abliterated model artifacts have collateral weight changes beyond refusal-direction suppression (e.g., from quantization differences or unintended representational shifts), then the performance differences attributed to safety state could instead reflect unrelated capability changes. A clean falsification would require reproducing the abliteration intervention from the aligned base model with controlled quantization matched to the aligned artifact, then re-running the full evaluation suite. If the same task-depth and prompt-framing patterns disappeared under matched conditions, the

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Cyber-safety evaluations for LLMs should measure answer correctness, localization precision, and executable actionability — not just refusal rates. A model that rarely refuses but produces incorrect line predictions or non-compiling patches may be no more useful to a defender than one that refuses frequently.
  • Safety mechanisms that respond to cybersecurity terminology as a surface cue risk degrading legitimate defensive work. Security professionals naturally use exploit-path, attacker-controlled-input, and vulnerability-pattern language; if that language triggers safety-related degradation, the model becomes less useful precisely when the task is most security-critical.
  • The task-depth interaction suggests that safety alignment may disproportionately affect deeper, more code-grounded reasoning rather than shallow classification. If confirmed, this means alignment costs are not evenly distributed across the capability stack — they may concentrate in the tasks that matter most for actual repair work.
  • Prompt-framing sensitivity in the aligned state implies that evaluation benchmarks using neutral code-review language may systematically overestimate aligned-model utility for real security workflows, where professional terminology is standard.
  • The cross-language variation in neutral-prompt repair (aligned advantage in C/C++, abliterated advantage in Java under security framing) suggests that safety-state effects may interact with programming-language ecosystem characteristics,

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If safety mechanisms respond to surface terminology cues rather than task legitimacy, one could test this directly by constructing prompt pairs that are semantically identical but lexically varied (neutral vs. security-dense) on the same code sample and measuring whether the aligned model's internal activations show safety-related engagement on the security-dense variant. The paper's drift results
  • The finding that abliteration improves localization but not necessarily final repair quality (in neutral Java, aligned gets slightly more full validations) suggests the refusal direction may interact with the model's tendency to commit to specific code regions early. A testable hypothesis: abliterated models may cast a wider net in line prediction (higher recall, lower precision), which would expl
  • The sharp drop in aligned throughput under security-explicit framing could be partly due to the aligned model internally detecting potential harm and shifting into a more conservative generation mode — not fully refusing but narrowing its output distribution. This could be tested by comparing token-level output distributions (entropy, diversity) between aligned and abliterated states across prompt
  • If the refusal direction overlaps with representations for security-relevant reasoning (not just refusal per se), then abliteration could inadvertently suppress some security-analysis capability even as it removes refusal. The paper's mixed results — abliterated wins on localization but loses on some shallow tasks — are consistent with this kind of collateral representational overlap, but the stud

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This paper studies how LLM safety state—aligned versus refusal-ablated (abliterated)—affects defender-side utility across vulnerability-analysis workflows. Using matched same-lineage model pairs (Gemma and Qwen), the authors evaluate detection, CWE attribution, vulnerable-line localization, root-cause localization, and executable patch validation under controlled prompt framings. The key finding is that safety-state effects are context-dependent: the aligned state tends to be stronger on shallow diagnostic tasks under neutral wording, while the abliterated state becomes more competitive on code-grounded localization and early-stage executable repair, especially under security-explicit prompt framing. The experimental design is careful: same-lineage models, deterministic decoding (temperature 0, seed 42), taxonomy-driven prompt construction fixed before result inspection, and a staged repair-validation funnel with executable gates.

Significance. The paper addresses a genuine gap in cyber-safety evaluation: most prior work compares unrelated model families or focuses on refusal rate alone. The same-lineage design and the decomposition of utility into coverage, answer quality, and end-to-end actionability are well-motivated. The staged patch-validation funnel (usable → applied → compiled → PoV pass → full validation) is a concrete and reproducible evaluation contribution. The prompt-intensity study, crossing authorization context with lexical intensity, is a thoughtful experimental design. However, the practical significance of several results is limited by very low absolute scores (line-F1 of 2–4%, Top-1 of 4–10%) and the absence of any statistical testing, which is a load-bearing gap for a paper whose central claims rest on percentage-point differences.

major comments (3)
  1. The paper reports no statistical tests anywhere. Several headline differences for the primary Gemma pair are not distinguishable from noise at conventional thresholds. For the 'clearest cross-pair result' (Section V-A, Table II): Gemma Top-1 is 9.00% vs 10.10% (189 vs 212 hits out of n=2100 per state), giving a two-proportion z-test p≈0.13. Gemma mean line-F1 differs by 0.16 percentage points (3.84% vs 4.00%). For executable repair (Section VI-A, Figure 3): 6 vs 2 full-validation passes out of 59 gives Fisher's exact p≈0.27; the usable-answer gate (40 vs 32 out of 59) gives p≈0.16. The Qwen localization result (Top-1 4.10% vs 6.95%, n=2100) does reach significance (p≈0.001), but it stands alone; the Gemma pair—the paper's primary model—does not. The central claim that safety state affects localization quality, prompt sensitivity, and repair-validation outcomes rests entirely on raw point
  2. Section III-A: The paper acknowledges that the abliterated artifacts differ from their aligned counterparts through 'CRACK-style per-layer refusal-vector surgery followed by quantization and GGUF conversion' and states this is 'not a perfectly isolated causal alignment intervention.' However, the paper's framing throughout (e.g., 'varies only safety state' in the contributions, Section I) implies that safety state is the sole varying factor. The Gemma aligned model uses Q4_K_M GGUF while the Qwen pair uses MXFP4, and the abliterated artifacts involve both refusal-vector surgery AND quantization changes. The paper should explicitly state which confounds are uncontrolled (quantization format, potential collateral weight changes from surgery) and temper causal claims accordingly. The phrase 'varies only safety state' in the contributions list is not justified.
  3. Section VI-A, Figure 3: The Vul4J repair results are presented as a major finding (abstract highlights 67.8% vs 29.9% usable, 32.8% vs 9.0% compiled), but the sample size is 59 vulnerabilities × 3 frames = 177 tasks per state. The late-gate results (6 vs 2 full validation passes) are based on extremely sparse counts. The paper should report confidence intervals or at minimum acknowledge that these differences are not statistically significant, and should frame the early-gate throughput gap as suggestive rather than established, especially given that the neutral-frame final-gate result actually favors Aligned (2 vs 1).
minor comments (7)
  1. The abstract states 'Abliterated achieves higher early-stage validation rates, with 67.8%, 65.0%, and 32.8%...' These are the all-frame rates from Figure 3. The abstract should clarify that these are all-frame aggregates and that the neutral-frame late-gate results actually favor Aligned, as shown in Figure 3.
  2. Section III-C: The model names (gemma-4-31B-it, Qwen3.6-27B-MXFP4) appear to be fictional or future-dated versions. If these are placeholder names, they should be corrected to the actual model versions used.
  3. Table I: The 'Usable Answers' row for Qwen shows notable coverage differences (96.56% vs 91.67% for detection, 97.67% vs 92.67% for CWE), suggesting the abliterated Qwen model produces more unparseable responses. This is mentioned but not analyzed. A brief discussion of why coverage drops would strengthen the paper.
  4. Figure 2: The heatmap values in the right panel are described as 'ALIGNED-minus-ABLITERATED differences' but the caption says 'blue cells favor ALIGNED and red cells favor ABLITERATED.' The color convention should be verified for consistency, as some values appear to have signs that may be confusing without a colorbar.
  5. Section VII-B: The 'linear fit' to ABLITERATED-minus-ALIGNED differences (+0.25 pp per lexical level for localization, +0.81 for CWE) is described as descriptive, but with only 4 lexical levels and no goodness-of-fit reported, the linear trend claim is weak. This should be explicitly labeled as descriptive only.
  6. The paper would benefit from a brief discussion of whether the absolute localization scores (2-4% line-F1, 4-10% Top-1) are practically useful. Even the better state achieves very low accuracy—is this a limitation of the models, the task formulation, or the evaluation metric? This is especially relevant given the concerns in Major Comment 1.
  7. Reference [20] (Defensive Refusal Bias) is cited as the closest prior work and primary motivation. The paper should clarify what is novel beyond this work's findings—[20] apparently already showed over-refusal in cyber-defense contexts. The contribution should be framed as extending measurement from refusal to answer quality and actionability, which is already done in the contributions list but could be stated more clearly in the introduction to strengthen positioning.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for a careful and constructive review. The referee raises three major points: (1) absence of statistical testing, (2) overstated causal claims given uncontrolled confounds (quantization format, collateral weight changes from refusal-vector surgery), and (3) over-interpretation of sparse late-gate repair results. We agree with all three points in substance and will revise the manuscript accordingly. Specifically, we will add statistical tests (two-proportion z-tests and Fisher's exact tests) with explicit significance thresholds, report confidence intervals where sample sizes permit, temper all causal language (including removing 'varies only safety state' from the contributions), and reframe early-gate repair gaps as suggestive rather than established. We respectfully note that the Qwen localization result (p≈0.001) and the early-gate Gemma repair throughput gap (usable answers: 120/177 vs 53/177, p<0.001) do reach conventional significance, so the core empirical phenomenon survives the statistical correction—even as we agree that several Gemma overview differences do not.

read point-by-point responses

  1. Referee: The paper reports no statistical tests anywhere. Several headline differences for the primary Gemma pair are not distinguishable from noise at conventional thresholds. For the 'clearest cross-pair result' (Section V-A, Table II): Gemma Top-1 is 9.00% vs 10.10% (189 vs 212 hits out of n=2100 per state), giving a two-proportion z-test p≈0.13. Gemma mean line-F1 differs by 0.16 percentage points (3.84% vs 4.00%). For executable repair (Section VI-A, Figure 3): 6 vs 2 full-validation passes out of 59 gives Fisher's exact p≈0.27; the usable-answer gate (40 vs 32 out of 59) gives p≈0.16. The Qwen localization result (Top-1 4.10% vs 6.95%, n=2100) does reach significance (p≈0.001), but it stands alone; the Gemma pair—the paper's primary model—does not. The central claim that safety state affects localization quality, prompt sensitivity, and repair-validation outcomes rests entirely on raw point

    Authors: The referee is correct on every numerical point, and we will add the statistical tests the referee specifies. We have independently verified the referee's calculations: Gemma Top-1 localization (189/2100 vs 212/2100) gives z-test p≈0.13; Gemma line-F1 differs by 0.16 pp; Vul4J full-validation (6 vs 2/59) gives Fisher p≈0.27; Vul4J usable-answer gate (40 vs 32/59) gives Fisher p≈0.16. These differences are not statistically significant at conventional thresholds, and the current manuscript does not claim otherwise explicitly enough. We will add a statistical-testing subsection reporting two-proportion z-tests for all localization comparisons and Fisher's exact tests for all repair-funnel gates, with explicit p-values and significance thresholds. We will also revise all prose to distinguish significant from non-significant differences. Two important qualifications: (1) The Qwen localization result is statistically significant (Top-1: 86/2100 vs 146/2100, p≈0.001; line-F1: 2.08% vs 3.91%), so the cross-pair localization finding is not based solely on the non-significant Gemma result. (2) The Gemma Vul4J usable-answer gate is more favorable than the referee's per-frame calculation suggests: the all-frame comparison is 120/177 vs 53/177 (67.8% vs 29.9%), which gives Fisher p<0.001. The per-frame neutral-only comparison (40 vs 32/59, p≈0.16) is indeed non-significant, and we will report it as such. The compiled gate (58/177 vs 16/177, p<0.001) is also significant in the all-frame aggregate. We agree that the late gates (PoV: 12 vs 3/177; full validation: 6 vs 2/177) are sparse and should be framed as suggestive, not established. We will restructure the claims accordingly: significant results will be stated as findings; non-significant results will be described as directional或 revision: no

Circularity Check

0 steps flagged

No circularity found — empirical study with externally defined benchmarks and no derivation chain

full rationale

This paper is an empirical evaluation study, not a derivation chain. It compares two publicly released model artifacts (aligned vs. abliterated) on external benchmarks (Vul4J, PrimeVul, LineVul, PatchEval, Vul4C) using pre-defined prompt taxonomies fixed before result inspection (Section III-B3: 'The prompt choices are taxonomy-driven and fixed before result inspection'). The utility decomposition U(y) = C(y)·Q(y) (Section III-B1) is a measurement framework definition, not a claimed prediction that could be circular. The refusal-direction mathematics in Section III-A is background description of the abliteration procedure (citing [50], [51] — external authors Arditi et al. and Wang et al.), not a result the paper claims to derive. No parameter is fitted to a subset of data and then 'predicted' on related data. No uniqueness theorem is invoked. Self-citation is minimal: the paper cites prior work [20] (Defensive Refusal Bias, by non-overlapping authors Campbell et al.) as motivation, but this is not load-bearing for any claimed derivation. The central claims are empirical observations grounded in external benchmarks with independent ground truth. There is no step where an output reduces to an input by construction.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The paper introduces no new entities, particles, or forces. It evaluates existing model artifacts against existing benchmarks. The free parameters are decoding settings, not fitted values. The primary axiom is the assumption that abliteration is a clean intervention, which the paper itself flags as imperfect.

free parameters (1)
  • None fitted by authors = N/A
    The study is empirical. Decoding parameters (temperature=0, top-p=1, seed=42) are fixed, not fitted. No model weights are trained or adjusted by the authors; they evaluate existing released artifacts.
axioms (4)
  • domain assumption The publicly released abliterated model artifacts differ from their aligned counterparts primarily in the suppression of the refusal direction, preserving other capabilities.
    Section III-A states the treatment is 'a matched same-lineage safety-state comparison rather than a perfectly isolated causal alignment intervention.' The entire causal attribution depends on this assumption.
  • standard math The refusal direction can be represented as a normalized difference between refusal and allowed-response centroids in activation space.
    Section III-A cites Arditi et al. [50] and Wang et al. [51] for this formulation. It is a standard result from mechanistic interpretability literature.
  • domain assumption Vul4J, PrimeVul, and LineVul provide realistic and representative vulnerability-analysis tasks.
    Section III-B2 uses these datasets as the task substrate. The generalizability of findings depends on their representativeness.
  • domain assumption Deterministic decoding (temperature 0) yields stable and representative model responses for evaluation.
    Section III-C uses temperature 0 with seed 42. This assumes no significant degeneration or sampling artifacts affect the results.

pith-pipeline@v1.1.0-glm · 21618 in / 2221 out tokens · 254111 ms · 2026-07-08T22:39:19.228611+00:00 · methodology

0 comments
read the original abstract

Large language model (LLM)-assisted software security operates at a difficult boundary: the vulnerability-analysis terminology needed for legitimate code review, triage, and repair can closely resemble terminology associated with misuse. Existing safety and cybersecurity evaluations are difficult to interpret in this setting because they often compare unrelated model families, thereby conflating safety behavior with differences in architecture, scale, training data, and deployment. To isolate this factor, we study safety state: whether refusal behavior remains intact (Aligned) or has been refusal-ablated (Abliterated) within same-lineage models. We ask how this safety state affects defensive utility across software-security workflows. We compare aligned instruction-tuned models with publicly released refusal-ablated descendants from two model families, Gemma and Qwen. We evaluate Aligned and Abliterated states on vulnerability detection, CWE attribution, vulnerable-line localization, root-cause localization, and executable patch validation. We further treat prompt wording as a controlled framing dimension: prompts begin with neutral code-review language, add authorization context, and vary the density of cybersecurity terminology. In a Gemma-based Java/Vul4J repair-validation study, Abliterated achieves higher early-stage validation rates, with 67.8%, 65.0%, and 32.8% of patches judged usable, successfully applied, and successfully compiled, respectively, compared with 29.9%, 24.9%, and 9.0% for Aligned. In the Qwen pair, Abliterated improves localization performance, increasing line-level F1 from 2.08% to 3.91% and Top-1 accuracy from 4.10% to 6.95%. These findings suggest that evaluations of LLM-based security assistants should jointly measure whether models respond, whether their usable responses are correct, and whether their outputs remain actionable across the engineering workflow.

Figures

Figures reproduced from arXiv: 2607.05842 by Heng Fan, Junhua Ding, Meikang Qiu, Mingchen Li, Song Fu, Yunhe Feng, Zifan Peng.

Figure 1
Figure 1. Figure 1: Study design for measuring defender-side utility across task depth and prompt framing. Left: the overview design crosses three prompt-framings [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Root-cause localization on Vul4J with the Gemma pair. The bar panels report Top1/Top3/Top5 hit rates and mean line-F1 by prompt frame; the [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Vul4J patch-validation funnel for the Gemma pair across prompt frames and model states. The left panel reports all-frame pass rates across the five [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Prompt-intensity utility under Baseline Context and Authorized Context. The figure reports localization and CWE-attribution utility across Neutral, [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Output drift induced by prompt wording relative to Neutral wording. The figure reports localization decision flips, localization line-set drift, and [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

55 extracted references · 55 canonical work pages · 22 internal anchors

  1. [1]

    Evaluating Large Language Models Trained on Code

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockmanet al., “Evaluating large language models trained on code,”arXiv preprint arXiv:2107.03374, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  2. [2]

    Competition- level code generation with alphacode,

    Y . Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lagoet al., “Competition- level code generation with alphacode,”Science, vol. 378, no. 6624, pp. 1092–1097, 2022

    work page 2022

  3. [3]

    Code Llama: Open Foundation Models for Code

    B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, R. Sauvestre, T. Remezet al., “Code llama: Open foundation models for code,”arXiv preprint arXiv:2308.12950, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  4. [4]

    CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation

    S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tanget al., “Codexglue: A machine learning benchmark dataset for code understanding and generation,” arXiv preprint arXiv:2102.04664, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  5. [5]

    Codebert: A pre-trained model for programming and natural languages,

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jianget al., “Codebert: A pre-trained model for programming and natural languages,” inFindings of the association for computational linguistics: EMNLP 2020, 2020, pp. 1536–1547

    work page 2020

  6. [6]

    Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,

    Y . Wang, W. Wang, S. Joty, and S. C. Hoi, “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” inProceedings of the 2021 conference on empirical methods in natural language processing, 2021, pp. 8696–8708

    work page 2021

  7. [7]

    SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?”arXiv preprint arXiv:2310.06770, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  8. [8]

    Swe-agent: Agent-computer interfaces enable automated soft- ware engineering,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable automated soft- ware engineering,”Advances in Neural Information Processing Systems, vol. 37, pp. 50 528–50 652, 2024

    work page 2024

  9. [9]

    CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Models

    M. Bhatt, S. Chennabasappa, Y . Li, C. Nikolaidis, D. Song, S. Wan, F. Ahmad, C. Aschermann, Y . Chen, D. Kapilet al., “Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models,” arXiv preprint arXiv:2404.13161, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  10. [10]

    CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Models

    S. Wan, C. Nikolaidis, D. Song, D. Molnar, J. Crnkovich, J. Grace, M. Bhatt, S. Chennabasappa, S. Whitman, S. Dinget al., “Cyberseceval 3: Advancing the evaluation of cybersecurity risks and capabilities in large language models,”arXiv preprint arXiv:2408.01605, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  11. [11]

    Ctibench: A benchmark for evaluating llms in cyber threat intelligence,

    M. T. Alam, D. Bhusal, L. Nguyen, and N. Rastogi, “Ctibench: A benchmark for evaluating llms in cyber threat intelligence,”Advances in Neural Information Processing Systems, vol. 37, pp. 50 805–50 825, 2024

    work page 2024

  12. [12]

    {PentestGPT}: Evaluating and harnessing large language models for automated penetration testing,

    G. Deng, Y . Liu, V . Mayoral-Vilches, P. Liu, Y . Li, Y . Xu, T. Zhang, Y . Liu, M. Pinzger, and S. Rass, “{PentestGPT}: Evaluating and harnessing large language models for automated penetration testing,” in33rd USENIX Security Symposium (USENIX Security 24), 2024, pp. 847–864

    work page 2024

  13. [13]

    Vulnerability Detection with Code Language Models: How Far Are We?

    Y . Ding, Y . Fu, O. Ibrahim, C. Sitawarin, X. Chen, B. Alomair, D. Wagner, B. Ray, and Y . Chen, “Vulnerability detection with code language models: How far are we?”arXiv preprint arXiv:2403.18624, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  14. [14]

    When llms meet cybersecurity: A systematic literature review,

    J. Zhang, H. Bu, H. Wen, Y . Liu, H. Fei, R. Xi, L. Li, Y . Yang, H. Zhu, and D. Meng, “When llms meet cybersecurity: A systematic literature review,”Cybersecurity, vol. 8, no. 1, p. 55, 2025

    work page 2025

  15. [15]

    Patcheval: A new benchmark for evaluating llms on patching real-world vulnerabilities,

    Z. Wei, J. Zeng, M. Wen, Z. Yu, K. Cheng, Y . Zhu, J. Guo, S. Zhou, L. Yin, X. Suet al., “Patcheval: A new benchmark for evaluating llms on patching real-world vulnerabilities,”arXiv preprint arXiv:2511.11019, 2025

    work page arXiv 2025

  16. [16]

    Claude fable 5 and claude mythos 5,

    Anthropic, “Claude fable 5 and claude mythos 5,” https://www.anthropic. com/news/claude-fable-5-mythos-5, Jun. 2026, accessed: 2026-06-27

    work page 2026

  17. [17]

    Claude platform api release notes,

    ——, “Claude platform api release notes,” https://platform.claude.com/ docs/en/release-notes/overview, Jun. 2026, accessed: 2026-06-27

    work page 2026

  18. [18]

    Anthropic’s claude fable 5 is a version of mythos the public can access today,

    R. Bellan, “Anthropic’s claude fable 5 is a version of mythos the public can access today,” https://techcrunch.com/2026/06/09/ anthropics-claude-fable-5-is-a-version-of-mythos-the-public-can-access-today/, Jun. 2026, accessed: 2026-06-27

    work page 2026

  19. [19]

    Cybersecurity researchers aren’t happy about the guardrails on anthropic’s fable,

    L. Franceschi-Bicchierai, “Cybersecurity researchers aren’t happy about the guardrails on anthropic’s fable,” https://techcrunch.com/2026/06/10/ cybersecurity-researchers-arent-happy-about-the-guardrails-on-anthropics-fable/, Jun. 2026, accessed: 2026-06-27

    work page 2026

  20. [20]

    Defensive refusal bias: How safety alignment fails cyber defenders,

    D. Campbell, N. Kale, U. M. Sehwag, B. Herring, N. Price, D. Borges, A. Levinson, and C. Q. Knight, “Defensive refusal bias: How safety alignment fails cyber defenders,”arXiv preprint arXiv:2603.01246, 2026

    work page arXiv 2026

  21. [21]

    Gemma: Open Models Based on Gemini Research and Technology

    G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivi `ere, M. S. Kale, J. Loveet al., “Gemma: Open models based on gemini research and technology,”arXiv preprint arXiv:2403.08295, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  22. [22]

    Qwen3.6-27B: Flagship-level coding in a 27B dense model,

    Qwen Team, “Qwen3.6-27B: Flagship-level coding in a 27B dense model,” April 2026. [Online]. Available: https://qwen.ai/blog?id=qwen3. 6-27b

    work page 2026

  23. [23]

    GraphCodeBERT: Pre-training Code Representations with Data Flow

    D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fuet al., “Graphcodebert: Pre-training code repre- sentations with data flow,”arXiv preprint arXiv:2009.08366, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2009

  24. [24]

    Unified pre- training for program understanding and generation,

    W. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, “Unified pre- training for program understanding and generation,” inProceedings of the 2021 conference of the North American chapter of the association for computational linguistics: human language technologies, 2021, pp. 2655–2668

    work page 2021

  25. [25]

    Unixcoder: Unified cross-modal pre-training for code representation,

    D. Guo, S. Lu, N. Duan, Y . Wang, M. Zhou, and J. Yin, “Unixcoder: Unified cross-modal pre-training for code representation,” inProceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), 2022, pp. 7212–7225

    work page 2022

  26. [26]

    StarCoder: may the source be with you!

    R. Li, L. B. Allal, Y . Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chimet al., “Starcoder: may the source be with you!”arXiv preprint arXiv:2305.06161, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  27. [27]

    WizardCoder: Empowering Code Large Language Models with Evol-Instruct

    Z. Luo, C. Xu, P. Zhao, Q. Sun, X. Geng, W. Hu, C. Tao, J. Ma, Q. Lin, and D. Jiang, “Wizardcoder: Empowering code large language models with evol-instruct,”arXiv preprint arXiv:2306.08568, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  28. [28]

    DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence

    D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y . Wu, Y . Liet al., “Deepseek-coder: when the large language model meets programming–the rise of code intelligence,”arXiv preprint arXiv:2401.14196, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  29. [29]

    Autocoderover: Autonomous program improvement,

    Y . Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, “Autocoderover: Autonomous program improvement,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024, pp. 1592–1604

    work page 2024

  30. [30]

    Agentless: Demystifying LLM-based Software Engineering Agents

    C. S. Xia, Y . Deng, S. Dunn, and L. Zhang, “Agentless: De- mystifying llm-based software engineering agents,”arXiv preprint arXiv:2407.01489, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  31. [31]

    Cvefixes: automated collec- tion of vulnerabilities and their fixes from open-source software,

    G. Bhandari, A. Naseer, and L. Moonen, “Cvefixes: automated collec- tion of vulnerabilities and their fixes from open-source software,” in Proceedings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering, 2021, pp. 30–39

    work page 2021

  32. [32]

    Diversevul: A new vulnerable source code dataset for deep learning based vulnerability detection,

    Y . Chen, Z. Ding, L. Alowain, X. Chen, and D. Wagner, “Diversevul: A new vulnerable source code dataset for deep learning based vulnerability detection,” inProceedings of the 26th international symposium on research in attacks, intrusions and defenses, 2023, pp. 654–668

    work page 2023

  33. [33]

    Linevul: A transformer-based line- level vulnerability prediction,

    M. Fu and C. Tantithamthavorn, “Linevul: A transformer-based line- level vulnerability prediction,” inProceedings of the 19th international conference on mining software repositories, 2022, pp. 608–620

    work page 2022

  34. [34]

    Megavul: Ac/c++ vulner- ability dataset with comprehensive code representations,

    C. Ni, L. Shen, X. Yang, Y . Zhu, and S. Wang, “Megavul: Ac/c++ vulner- ability dataset with comprehensive code representations,” inProceedings of the 21st International Conference on Mining Software Repositories, 2024, pp. 738–742

    work page 2024

  35. [35]

    Vul4j: A dataset of reproducible java vulnerabilities geared towards the study of program repair techniques,

    Q.-C. Bui, R. Scandariato, and N. E. D. Ferreyra, “Vul4j: A dataset of reproducible java vulnerabilities geared towards the study of program repair techniques,” inProceedings of the 19th International Conference on Mining Software Repositories, 2022, pp. 464–468

    work page 2022

  36. [36]

    Sok: automated vulnerability repair: methods, tools, and assessments,

    Y . Hu, Z. Li, K. Shu, S. Guan, D. Zou, S. Xu, B. Yuan, and H. Jin, “Sok: automated vulnerability repair: methods, tools, and assessments,” in Proceedings of the 34th USENIX Security Symposium (USENIX Security 25), 2025

    work page 2025

  37. [37]

    Grace: Empowering llm-based software vulnerability detection with graph structure and in- context learning,

    G. Lu, X. Ju, X. Chen, W. Pei, and Z. Cai, “Grace: Empowering llm-based software vulnerability detection with graph structure and in- context learning,”Journal of Systems and Software, vol. 212, p. 112031, 2024

    work page 2024

  38. [38]

    A case study of llm for automated vulnerability repair: Assessing impact of reasoning and patch validation feedback,

    U. Kulsum, H. Zhu, B. Xu, and M. d’Amorim, “A case study of llm for automated vulnerability repair: Assessing impact of reasoning and patch validation feedback,” inProceedings of the 1st ACM International Conference on AI-Powered Software, 2024, pp. 103–111

    work page 2024

  39. [39]

    AgentBench: Evaluating LLMs as Agents

    X. Liu, H. Yu, H. Zhang, Y . Xu, X. Lei, H. Lai, Y . Gu, H. Ding, K. Men, K. Yanget al., “Agentbench: Evaluating llms as agents,”arXiv preprint arXiv:2308.03688, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  40. [40]

    Agentdojo: A dynamic environment to evaluate prompt injection attacks and defenses for llm agents,

    E. Debenedetti, J. Zhang, M. Balunovic, L. Beurer-Kellner, M. Fischer, and F. Tram`er, “Agentdojo: A dynamic environment to evaluate prompt injection attacks and defenses for llm agents,”Advances in Neural Information Processing Systems, vol. 37, pp. 82 895–82 920, 2024

    work page 2024

  41. [41]

    Identifying the Risks of LM Agents with an LM-Emulated Sandbox

    Y . Ruan, H. Dong, A. Wang, S. Pitis, Y . Zhou, J. Ba, Y . Dubois, C. J. Maddison, and T. Hashimoto, “Identifying the risks of lm agents with an lm-emulated sandbox,”arXiv preprint arXiv:2309.15817, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  42. [42]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Rayet al., “Training language models to follow instructions with human feedback,”Advances in neural information processing systems, vol. 35, pp. 27 730–27 744, 2022

    work page 2022

  43. [43]

    Constitutional AI: Harmlessness from AI Feedback

    Y . Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnonet al., “Constitutional ai: Harmlessness from ai feedback,”arXiv preprint arXiv:2212.08073, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  44. [44]

    Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations

    H. Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y . Mao, M. Tontchev, Q. Hu, B. Fuller, D. Testuggineet al., “Llama guard: Llm-based input-output safeguard for human-ai conversations,”arXiv preprint arXiv:2312.06674, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  45. [45]

    Jailbroken: How does llm safety training fail?

    A. Wei, N. Haghtalab, and J. Steinhardt, “Jailbroken: How does llm safety training fail?”Advances in neural information processing systems, vol. 36, pp. 80 079–80 110, 2023

    work page 2023

  46. [46]

    Universal and Transferable Adversarial Attacks on Aligned Language Models

    A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,”arXiv preprint arXiv:2307.15043, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  47. [47]

    HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal

    M. Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang, N. Mu, E. Sakhaee, N. Li, S. Basart, B. Liet al., “Harmbench: A standardized evaluation framework for automated red teaming and robust refusal,”arXiv preprint arXiv:2402.04249, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  48. [48]

    OR-Bench: An Over-Refusal Benchmark for Large Language Models

    J. Cui, W.-L. Chiang, I. Stoica, and C.-J. Hsieh, “Or-bench: An over-refusal benchmark for large language models,”arXiv preprint arXiv:2405.20947, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  49. [49]

    FORTRESS: Frontier Risk Evaluation for National Security and Public Safety

    C. Q. Knight, K. Deshpande, V . Sirdeshmukh, M. Mankikar, S. R. Team, S. Team, and J. Michael, “Fortress: Frontier risk evaluation for national security and public safety,”arXiv preprint arXiv:2506.14922, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  50. [50]

    Refusal in language models is mediated by a single direction,

    A. Arditi, O. Obeso, A. Syed, D. Paleka, N. Panickssery, W. Gurnee, and N. Nanda, “Refusal in language models is mediated by a single direction,”Advances in Neural Information Processing Systems, vol. 37, pp. 136 037–136 083, 2024

    work page 2024

  51. [51]

    Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablation

    X. Wang, C. Hu, P. R ¨ottger, and B. Plank, “Surgical, cheap, and flexible: Mitigating false refusal in language models via single vector ablation,” arXiv preprint arXiv:2410.03415, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  52. [52]

    gemma-4-31b-it,

    Google, “gemma-4-31b-it,” https://huggingface.co/google/ gemma-4-31B-it, Jun. 2026, hugging Face model card. Accessed: 2026-07-01

    work page 2026

  53. [53]

    Gemma-4-31b-jang 4m-crack-gguf,

    douyamv, “Gemma-4-31b-jang 4m-crack-gguf,” https://huggingface.co/ douyamv/Gemma-4-31B-JANG 4M-CRACK-GGUF, Apr. 2026, hug- ging Face model card. Accessed: 2026-07-01

    work page 2026

  54. [54]

    Qwen3.6-27b-mxfp4,

    OsaurusAI, “Qwen3.6-27b-mxfp4,” https://huggingface.co/OsaurusAI/ Qwen3.6-27B-MXFP4, Apr. 2026, hugging Face model card. Accessed: 2026-07-01

    work page 2026

  55. [55]

    Qwen3.6-27b-mxfp4-crack,

    dealignai, “Qwen3.6-27b-mxfp4-crack,” https://huggingface.co/ dealignai/Qwen3.6-27B-MXFP4-CRACK, Apr. 2026, hugging Face model card. Accessed: 2026-07-01

    work page 2026