{"id":"38458aa4-edca-401c-8f34-b0deb133d4b2","arxiv_id":"2502.01806","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper outlines the Neurosymbolic Program Comprehension (NsPC) framework and provides preliminary evidence that SHAP-value patterns over AST-tagged token positions can be interpreted as symbolic rules for insecure code detection.","lead":"A team at William & Mary proposes combining large code models with symbolic rules, extracted from SHAP explanations, to make vulnerability detection more interpretable. They present a preliminary case study on CodeBERT, finding that certain token types in certain positions correlate with insecure code, but they do not yet show the full system working.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pattern evidence lacks a null model, so Table I's rule candidates are not separated from chance, and RQ1's symbolic rules are never actually defined.","rationale":"The reader's weakest assumption identifies the same core fragility: the 60% threshold and x-intercept heuristic are used to declare symbolic patterns without a statistical baseline. My stress test sharpens this in two ways. First, the design lacks any null model, so the qualified cells in Table I could reflect the LCM's general accuracy rather than a genuine SHAP-position pattern; a label-permutation test directly settles whether the reported selections exceed chance. Second, RQ1's affirmative answer says the patterns 'facilitated the definition of symbolic rules,' but no concrete rule is ever written down or evaluated, so the chain from pattern to rule to detector is incomplete. This second point is not fatal by itself because the paper is explicitly a 'toward' contribution with a stated limitation, but it reinforces that the empirical foundation is not yet established. A conditional verdict remains appropriate: the direction is plausible and the authors are appropriately cautious, but the central evidence needs a null-model check and an actual rule instantiation before the claim can be accepted. No independent support such as released artifacts or machine-checked proofs is present, so the weight rests entirely on the Table I analysis, making the permutation test the decisive next step.","tokens_in":8266,"tokens_out":8396,"duration_ms":85820,"concrete_test":"Run a label-permutation null model over the Section III-A pipeline: shuffle the secure/insecure labels across the 600 snippets, recompute the logistic regressions for all 8 AST types and 6 position ranges using the original SHAP tensors, and count how many cells satisfy the same accuracy>60% and x-intercept criterion under 1,000 permutations. If the observed number of qualified cells falls inside the null distribution, the reported patterns are indistinguishable from chance and the central claim loses its empirical support; if it is a clear outlier, the concern is settled in the paper's favor.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is Section III-A: a Table I cell is treated as evidence of a symbolic rule when a logistic regression exceeds 60% accuracy and has an x-intercept. The central claim requires that these qualified cells reflect stable, non-chance structure rather than post-hoc selection over 48 cells, and that the identified patterns can be turned into rules that detect secure and insecure snippets. Neither is currently demonstrated. The pipeline has no null model, no control task, no confidence intervals, and no correction for multiple comparisons. Because SHAP values are computed from a model that already predicts the labels, a logistic regression on them can separate classes simply by inheriting the model's predictive signal, without showing that the AST-type and position grouping adds any explanatory value. Separately, the actual rules promised by RQ1 are never stated: Section III-B gives only a template, and the case study stops at Table I. The x-intercept criterion is also formally undefined, so the reader cannot verify which cells qualify. The paper's own limitation statement is honest, but it does not repair the missing baseline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NsPC, a neurosymbolic framework for program comprehension: compute SHAP values from a fine-tuned CodeBERT model on Java snippets, align tokens to AST types, group SHAP values by AST type and position bin, train logistic regression models on each bin, and use cells with accuracy above 60% and an x-axis intersection as evidence of symbolic rules for vulnerability detection. A case study on 300 insecure and 300 secure snippets reports Table I and answers RQ1 affirmatively, while acknowledging that the evidence is model-specific and not yet generalizable.","tokens_in":8435,"tokens_out":4269,"duration_ms":41205,"significance":"If the proposed pipeline were validated, it would be an interesting step toward interpretable, rule-augmented code models, responding to legitimate concerns about the opacity and cost of large code models. The paper is honest about limitations and builds on relevant prior work in probing and neurosymbolic SE. However, the current evidence does not establish the central claim: there is no null model, no concrete rule set, no evaluation of rules, and the design creates a potential internal loop between SHAP values and the model being explained. The contribution at present is a promising idea plus a preliminary, under-controlled case study.","major_comments":[{"comment":"The criterion of \"accuracy exceeding 60%\" plus an \"intersection with the x-axis\" is not validated against any null model. With 8 AST types and 6 position ranges, some cells will pass the 0.60 threshold by chance, yet the paper reports no confidence intervals, significance tests, or multiple-comparison correction, and the x-intercept criterion is never formally defined for a logistic regression. This is load-bearing because RQ1 is answered solely from the qualified cells in Table I.","section":"Section III-A, Table I"},{"comment":"The symbolic rules promised by RQ1 are never actually stated. Section III-B gives only a template (\"if a feature linked to an AST node consistently shows high SHAP values...\"), and the case study stops at Table I plus narrative interpretations in Section IV-A. To support the claim that patterns \"facilitated the definition of symbolic rules,\" the manuscript must present a concrete rule syntax, the rules derived for the studied model, and a held-out evaluation of those rules against the base model.","section":"Section III-B and Section IV-A"},{"comment":"Because the SHAP values are computed from the very CodeBERT model whose predictions define the target labels, a logistic regression on those SHAP values can separate classes simply by inheriting the model's predictive signal. The paper does not test whether the AST-type/position grouping adds explanatory value beyond raw SHAP values; a control using shuffled SHAP values, ungrouped SHAP values, or a label permutation test is needed to distinguish real structure from model-reproduced signal.","section":"Section III-A, Figure 1"},{"comment":"With only 300 snippets per class and most qualifying accuracies clustered around 0.60, the reported numbers are too fragile to support the conclusion. The paper gives no variance estimates, confidence intervals, or tests against chance, and the threshold is close to the observed values. A bootstrap or cross-validated estimate, together with reporting which cells fail the threshold under resampling, would be needed to establish the stability of the patterns.","section":"Section IV, Evaluation Dataset and Results"}],"minor_comments":[{"comment":"The text says the qualified AST types include \"primitive,\" but Table I shows no primitive cell with both accuracy above 0.60 and an x-intercept; the list of qualifying types and the gray shading need to be reconciled.","section":"Section IV-A"},{"comment":"The paper states a 500-token limit in dataset construction but then says \"considering a maximum token length of 300 per snippet\"; clarify whether snippets were truncated to 300 tokens for the SHAP analysis.","section":"Section IV, Evaluation Dataset"},{"comment":"The phrases \"positions [0 − 43]\" and \"positions [251 − 280]\" conflate the fixed position bins ([0-50], [251-300]) with the reported x-intercepts; specify whether the proposed rule applies to the whole bin or to a subrange defined by the intercept.","section":"Section IV-A, Figure 2"},{"comment":"The prose says \"curves with an accuracy exceeding 60%\" and later that the \"logistic regression model surpassed\" the threshold; accuracy belongs to the classifier, not to the fitted curve, and the text should distinguish the decision curve from the classifier's accuracy.","section":"Section III-A"},{"comment":"The paper cites [15] on problems with SHAP-based feature importance but does not return to those caveats when interpreting Table I; a short discussion of SHAP's known limitations in this setting would strengthen the validity discussion.","section":"Section II, Reference [15]"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads as a vision/position paper with a pilot case study. For a full research paper, the empirical core is currently too thin: the central evidence reduces to a small table interpreted without a baseline. If the venue is willing to publish early-stage frameworks, the revision should explicitly frame the contribution as a proposal with illustrative pilot data; otherwise the case study needs substantial strengthening along the lines of the major comments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this paper. First, the actual result is narrow: for a fine-tuned CodeBERT on Devign, SHAP values grouped by AST type and position range produce a few logistic regression cells above 60% accuracy. Second, the paper frames this as the seed of a neurosymbolic framework, but the symbolic rules are never actually written down anywhere. RQ1's answer promises rules in Section III-B, but the case study stops at Table I and a generic template.\n\nCredit where due: the combination of SHAP with AST tagging and position binning for vulnerability detection is new in the SE literature, as far as I know. The related work is well-chosen and includes the SHAP-as-feature-importance critique (Kumar et al.) and probing control tasks (Hewitt et al.), which shows they know the methodological bar. The limitation statement in Section IV-A is honest: they say the evidence doesn't generalize beyond this model. As a position paper, the direction is coherent and timely.\n\nSoft spots, in order of severity. No null model: a logistic regression on SHAP values from a model that already predicts the labels can separate classes just by inheriting that signal; nothing shows the AST-type/position grouping adds explanatory value. They select patterns post hoc across 48 cells with no multiple-comparison correction, no confidence intervals, and no significance tests. The 60% threshold is arbitrary, and the x-intercept criterion is never formally defined, so the reader can't verify which cells qualify. Most importantly, the advertised symbolic rules are never defined or tested—Section III-B gives only a template. These are fixable problems: a permuted-SHAP baseline, a control task, confidence intervals, and a stated rule set would make the case study meaningful. Without those, Table I is anecdotal.\n\nWho is this for: people working on interpretability of code models, and program chairs for NIER/vision tracks. A serious referee would need major revisions to the empirical section, but the idea deserves referee time. I'd accept it for peer review with that expectation.","headline":"A well-scoped vision paper whose empirical seed—SHAP-by-AST-type patterns—lacks a null model and whose promised rules are never actually stated.","tokens_in":8971,"tokens_out":1934,"would_cite":false,"duration_ms":18349,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SHAP explanations of a code model, grouped by AST type and token position, can be converted into symbolic rules for identifying secure and insecure code snippets.","keywords":["neuro-symbolic AI","program comprehension","vulnerability detection","SHAP","interpretability","large code models","symbolic rules","probing classifiers"],"falsifier":"Permute the SHAP values within each abstract-syntax-tree type and position bin, retrain the same logistic-regression probes, and count how often the 60%-accuracy plus x-intercept criterion is met; if shuffled attributions produce patterns at a comparable rate, the proposed rules are artifacts rather than signal.","tokens_in":8047,"feed_emoji":"🛡️","tokens_out":8187,"duration_ms":75813,"temperature":0.7,"pith_summary":"This paper proposes NsPC, a neurosymbolic program comprehension framework that extracts symbolic rules from the SHAP (Shapley Additive exPlanations) values of a large code model. The authors claim that SHAP values, aligned with abstract-syntax-tree types and bucketed by token position, reveal patterns that distinguish insecure from secure code, and they demonstrate the idea on a CodeBERT model fine-tuned for defect detection. Their case study reports that logistic-regression probes on these SHAP tensors exceed a 60% accuracy threshold for several AST types, which they read as evidence of symbolic rules. The motivation is to offer an interpretable, deterministic complement to ever-larger language models in software engineering.","feed_headline":"SHAP values can define symbolic rules for insecure code","feed_subtitle":"A neurosymbolic framework turns SHAP patterns by syntax type and position into deterministic detection rules.","key_machinery":"The central object is the SHAP tensor, a per-token record $(i, w_i, \\varphi_i, \\mu_i)$ where $i$ is the token's position, $w_i$ is the token, $\\varphi_i$ is its SHAP value, and $\\mu_i$ is its abstract-syntax-tree type. An alignment function maps each token to its AST type, and the framework trains supervised models (logistic regression in the case study) on SHAP values grouped by AST type across position ranges; a curve with accuracy above 60% and an x-axis intersection is treated as evidence of a symbolic rule.","core_discovery":"The paper's central claim is that position-sensitive SHAP patterns carry enough signal to define symbolic rules for vulnerability classification. In the case study, literals appearing early in a snippet are associated with insecure predictions, while operators appearing late are associated with secure predictions; the authors take these as instances of the general claim that AST-type and position configurations with high SHAP influence can be turned into rules. They answer their research question affirmatively, stating that the NsPC framework identified meaningful insecure-prone patterns within specific position ranges that facilitated rule definition.","pith_inferences":["I infer that the six fixed position bins and the 60% threshold need calibration: without a null baseline or multiple-comparison control, some cells in Table I may reflect chance, and a permutation test would settle this.","I infer that the same pipeline could be run with other attribution methods, such as integrated gradients, to check whether the derived rules reflect the model's behavior or SHAP's particular value allocation.","A test the paper leaves open is whether applying the extracted rules to low-confidence predictions improves overall detection accuracy, which would close the loop from pattern to rule to model adjustment.","Tokenization interacts with position ranges, so the rules may encode subword-tokenizer artifacts as much as syntax; controlling for token length would make the position signal cleaner."],"forward_implications":["Symbolic rules derived this way can be attached in the post-training stage, letting low-confidence predictions be guided by deterministic conditions.","Practitioners could inspect which syntax types in which token ranges contribute to insecure predictions, turning part of the model's judgment into checkable statements.","The evidence is model-specific, so any new code model used for vulnerability detection would need its own SHAP-tensor analysis before its rules are trusted.","Because the tensor format is class-agnostic, the same extraction pipeline can target other labels beyond secure and insecure, such as bug-prone or style-violating code."],"supporting_citations":[{"why":"Supplies the SHAP attribution method whose per-token values form the tensors analyzed for patterns.","marker":"[13]"},{"why":"Provides the probing-classifier methodology that the framework adapts to train supervised models on SHAP values.","marker":"[20]"},{"why":"Supplies prior probing of pretrained code models that motivates pattern identification from model representations.","marker":"[22]"},{"why":"Defines probing classifiers for hidden representations, the conceptual basis for treating SHAP distributions as probes.","marker":"[23]"},{"why":"Supplies CodeBERT, the large code model whose SHAP values are analyzed in the case study.","marker":"[27]"},{"why":"Supplies the Devign dataset on which the model was fine-tuned for insecure-code detection.","marker":"[32]"},{"why":"Supplies the CodeXGLUE validation split from which the secure and insecure evaluation datasets were built.","marker":"[33]"}],"fun_headline_variants":["From SHAP to symbolic rules for insecure code","Neurosymbolic rules from SHAP patterns","SHAP patterns become deterministic security rules","Rules from SHAP: a neurosymbolic path","Symbolic rules from position-sensitive SHAP"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach assumes that a logistic-regression probe crossing a 60% accuracy threshold on SHAP values grouped into six fixed position ranges indicates a genuine, stable symbolic pattern rather than noise or dataset artifact.","fun_headline_variants_meta":{"raw":{"variants":["From SHAP to symbolic rules for insecure code","Neurosymbolic rules from SHAP patterns","SHAP patterns become deterministic security rules","Rules from SHAP: a neurosymbolic path","Symbolic rules from position-sensitive SHAP"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000159,"raw_usage":{"total_tokens":1197,"prompt_tokens":882,"completion_tokens":315,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":246}},"tokens_in":498,"tokens_out":315,"duration_ms":2990,"temperature":1.0,"reasoning_tokens":246,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T14:22:05.727357+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Permute the SHAP values within each abstract-syntax-tree type and position bin, retrain the same logistic-regression probes, and count how often the 60%-accuracy plus x-intercept criterion is met; if shuffled attributions produce patterns at a comparable rate, the proposed rules are artifacts rather than signal.","supporting_citations":[{"cited_title":"A unified approach to interpreting model predictions,","cited_arxiv_id":null,"evidence_quote":"Supplies the SHAP attribution method whose per-token values form the tensors analyzed for patterns."},{"cited_title":"Designing and Interpreting Probes with Control Tasks,","cited_arxiv_id":null,"evidence_quote":"Provides the probing-classifier methodology that the framework adapts to train supervised models on SHAP values."},{"cited_title":"Probing pretrained models of source codes,","cited_arxiv_id":null,"evidence_quote":"Supplies prior probing of pretrained code models that motivates pattern identification from model representations."},{"cited_title":"Probing classifiers: Promises, shortcomings, and ad- vances,","cited_arxiv_id":null,"evidence_quote":"Defines probing classifiers for hidden representations, the conceptual basis for treating SHAP distributions as probes."}],"review_version":1}