{"id":"9a1f9a46-9d40-4060-aee5-7bc0edd3f6f9","arxiv_id":"2509.10287","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"URL2Graph++ fuses BERT semantics, character CNN features, and dual word/character co-occurrence graphs to report state-of-the-art malicious URL detection on three public datasets.","lead":"URL2Graph++ is a malicious URL detector that combines BERT semantics, character-level CNN features, and word/character co-occurrence graphs, then fuses them with a learned gate. The authors report higher accuracy and lower false-positive rates than existing detectors, including large language models, on three public URL datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Graph construction split discipline is the load-bearing risk: if NPMI graphs are built on the full dataset rather than each training split, test-set co-occurrences leak into structural features and the claimed SOTA margins may be inflated.","rationale":"The reader's weakest assumption—that NPMI graphs must be built only from each experiment's training partition and that F1 must use a stated, consistent formula—identifies the same load-bearing risk I see. I agree with the CONDITIONAL verdict and do not think the concern, if confirmed, would necessarily be fraud; it is a standard split-discipline pitfall that can be settled by code inspection and a controlled recomputation. The graph leakage issue is the most load-bearing because the headline SOTA claim is an empirical comparison, and the 1–2% margins over strong baselines are exactly the scale at which transductive feature leakage would matter. The paper's own ambiguity about whether 'training corpus' means the full dataset or each training split makes this not merely a stylistic gap. The F1 inconsistency in Table III strengthens the concern that the evaluation pipeline is under-specified, but it is secondary to the leakage risk. Since the reader already conditioned the verdict on this assumption, my stress-test does not move the verdict; it sharpens the specific experiment needed to resolve it.","tokens_in":20927,"tokens_out":5547,"duration_ms":61082,"concrete_test":"Run the released code's GramBeddings 5% experiment twice, holding the random split fixed: (A) construct NPMI graphs from the 32,000 training URLs only; (B) construct them from the full 800,000 URLs before splitting. Compare Table III accuracy/F1/AUC and Table V TPR@FPR. If (B) beats (A) by more than ~1–2% accuracy or ~1% AUC, the reported margin over LLaMA-2/URLBERT is inflated by test-set leakage; if the code already builds graphs per-split, verify that by code inspection and assert it in the manuscript.","verdict_should_be":"UNCHANGED","load_bearing_attack":"URL2Graph++'s central claim—exceeding SOTA including URLBERT and LLaMA-2—is empirical, and the largest unverified prerequisite is that the global NPMI graphs (Section III.A, Eq. 3) are built from the training partition only and frozen before the split. The paper says 'training corpus' but never states that the graph is rebuilt for each random training subset. In the scaling experiments (Table III), training sizes are 1–5% of the dataset. If the graph is instead built from the full 800k/1.56M dataset, then each test URL's induced subgraph (Eq. 10) contains edges whose existence was computed using co-occurrences from the test set itself. This is transductive leakage: the model sees corpus-level structure derived from the exact URLs it is later scored on. Since the reported gains over URLBERT and LLaMA-2 are only 1–2% in accuracy/F1 and about 1% in AUC, leakage of even modest size can invert the conclusion. No split protocol is given in Section IV, and the GitHub link is not executed in the paper, so the manuscript alone does not rule this out. Secondary but related: several Mendeley F1 values in Table III are inconsistent with the reported precision/recall (e.g., 1%: P=0.9818, R=0.6712 → binary F1≈0.797 vs table 0.7578; 5%: P=0.9851, R=0.7302 → ≈0.838 vs table 0.8022), so the metric formula must also be pinned down before any SOTA comparison is meaningful.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes URL2Graph++, a malicious URL detection architecture that fuses three signal types: BERT subword semantics, character-level CNN features, and dual-granularity (subword and character) NPMI co-occurrence graphs processed by GCNs, combined through a learned gated fusion. The method is evaluated on GramBeddings, Mendeley, and Kaggle datasets under balanced, imbalanced, cross-dataset, adversarial, and short-URL settings. The central claim is that URL2Graph++ exceeds state-of-the-art performance, including URLBERT and LLaMA-2, with reported gains of roughly 1–2% in accuracy/F1 and around 1% in AUC at the main training scales, and larger gains at small training sizes and on short URLs. The architecture is clearly described, but the experimental protocol has several load-bearing gaps: graph construction split discipline is unspecified, F1 values in Table III are not consistent with the reported precision/recall, baseline adaptations (especially LLaMA-2) are not detailed, and some textual claims about TPR@FPR are contradicted by the tables.","tokens_in":21327,"tokens_out":6086,"duration_ms":69453,"significance":"If the empirical claims hold, URL2Graph++ is a sensible and moderately novel contribution: it combines a semantic encoder, character-level morphology, and corpus-derived co-occurrence graphs in a unified, end-to-end trainable model, and the evaluation covers several practically important regimes. The ablation study and the cross-dataset and low-FPR experiments are useful. The paper is not circular in the supervised-learning sense: the NPMI graphs are unsupervised, and the learned parameters are fitted by standard supervised training. However, the headline SOTA claim rests on experimental reporting that is currently not reproducible or internally consistent. The largest risk, identified by the stress-test, does land: if the global NPMI graphs are built on the full corpus rather than recomputed from each training split, test URLs contribute to the structural features used to score them, and the reported 1–2% margins over strong baselines can easily be explained by leakage. Because the central claim is empirical and the required fixes are localized, the paper is potentially salvageable but needs major revision.","major_comments":[{"comment":"The global NPMI graphs are described as constructed over 'the entire training corpus' (Eqs. 1–3), but the manuscript never states whether the graphs are rebuilt for each training subset used in the scaling experiments. Table III uses 1%, 2%, ..., 5% of GramBeddings/Mendeley as training sizes. If the graphs are built once on the full 800k/1.56M corpus before splitting, then every test URL contributes co-occurrence statistics to the edges in its induced subgraph (Eq. 10); this is transductive leakage. The reported gains over URLBERT and LLaMA-2 are only 1–2% in accuracy/F1 and about 1% in AUC, so leakage of even modest size can invert the conclusion. Please state explicitly and verifiably: for every experiment, the NPMI graphs are constructed from the training partition only and frozen before any evaluation, and describe how graph construction interacts with the 1–5% random subsets. If thi","section":"Section III.A and Table III"},{"comment":"The F1 values in Table III are not consistent with the reported precision and recall under any standard binary F1 definition. For example, the 1% Mendeley row for 'Our' reports P=0.9818, R=0.6712, whose harmonic mean is approximately 0.797, not 0.7578; the 5% row reports P=0.9851, R=0.7302, whose harmonic mean is approximately 0.838, not 0.8022. More egregiously, the 2% CGAN row reports P=0.7096, R=0.6695, and F1=0.8140, which is mathematically impossible for a binary F1 (harmonic mean ≈0.689). The F1 formula (macro, micro, weighted, or binary) is never defined. Since the paper's headline comparisons rely on these numbers, all metrics must be recomputed and the metric definitions stated before any SOTA comparison is meaningful.","section":"Table III (Mendeley rows)"},{"comment":"The baseline adaptation is underspecified. All baselines are said to 'follow the settings reported in their respective publications,' but LLaMA-2 is a generative decoder-only language model, and the paper does not explain how it is adapted to binary URL classification: is it fine-tuned with a classification head, prompted, or evaluated zero-shot? What tokenizer/truncation is used? What hyperparameters, epochs, and learning rates? Similarly, URLBERT is a multi-task pretrained model; the manuscript should clarify whether the pretraining corpus overlaps with the evaluation datasets and whether the reported URLBERT numbers come from the original implementation or from a re-implementation by the authors. Without these details, the baseline comparison cannot be independently reproduced.","section":"Section V, Baselines (esp. LLaMA-2)"},{"comment":"Several textual claims about TPR@FPR are directly contradicted by the tables. The text says the proposed method achieves 'consistently superior TPR across all FPR levels and training sizes,' but Table V shows URLBERT has a higher TPR at FPR=0.0001 (0.1671 vs. 0.1466 for 'Our'). In the adversarial evaluation, Table VII shows LLaMA-2 has a higher TPR at FPR=0.01 (0.7235 vs. 0.6822), and Table VIII shows LLaMA-2 (0.5920) and CGAN (0.6617) both exceed 'Our' (0.4823) at FPR=0.001. These contradictions affect the robustness and low-FPR claims, which are a major part of the paper's practical motivation. The claims or the tables must be corrected, and the corrected conclusions stated accurately.","section":"Section V.A.c and V.D, Tables IV–VIII"},{"comment":"All results are reported as single point estimates with no seeds, error bars, or significance tests. Since the headline advantage over LLaMA-2 and URLBERT is often small (e.g., Table IX: F1 0.8615 vs. 0.8513; AUC 0.9536 vs. 0.9402; and Table III margins of ~1%), a single run is insufficient to establish 'exceeds SOTA.' The paper should report the number of independent runs, standard deviations or confidence intervals, and, where appropriate, a paired significance test. This is not merely a presentation issue: it is load-bearing for the central claim.","section":"All experiments (Tables III–IX, Figures 2–3)"}],"minor_comments":[{"comment":"Notation is inconsistent: Eq. (15) uses \\tilde{h}^{word}_u and \\tilde{h}^{char}_u, but Eq. (17) uses \\tilde{h}^{sub}_u. Please unify the superscripts.","section":"Section III.C, Eqs. (14)–(17)"},{"comment":"The text says 'we use this new adversarial test dataset to evaluate our best performing baseline model' — should be 'baseline models' (plural), and the sentence structure is awkward. Also, the adversarial generation process is described only briefly; more detail is needed for reproducibility.","section":"Section V.D"},{"comment":"The x-axis labels '0-9 100-109 200-209 300+' are confusing: they appear to denote decade bins but omit intermediate labels. Please clarify the binning and the axis label.","section":"Figure 4"},{"comment":"The claim to be 'the first method to model URLs as graphs' is too strong given that the related work section acknowledges existing graph-based URL approaches. Please temper this to 'the first to model intra-URL dual-granularity co-occurrence graphs' or similar.","section":"Section I Contributions"},{"comment":"The BERT encoder is said to be 'optionally fine-tuned depending on training configuration,' but the experiments never state which configuration is used. Specify the BERT variant, whether it is frozen or fine-tuned, and the training hyperparameters (batch size, learning rate, epochs) for both the proposed method and the baselines.","section":"Section III.B and V"},{"comment":"The 'Graph Only' variant uses randomly initialized node features and consequently performs at chance. The text interprets this as showing structural information alone is inadequate, but the experiment confounds graph structure with feature initialization. A conclusion about the value of structure would require informative node features or a clearer caveat.","section":"Table VI"}],"recommendation":"major_revision","confidential_remarks":"The URLBERT baseline [38] shares authors with the current paper. This is not by itself a problem, but it raises a checklist item: the authors should state whether the URLBERT results are from the original implementation or re-implemented, and whether URLBERT's pretraining data overlaps with the GramBeddings/Mendeley/Kaggle evaluation sets. The shared-authorship overlap should not be allowed to substitute for a clearly specified and independently checkable baseline protocol. I also recommend that the editor ask the authors to provide the exact configuration, seeds, and graph-construction code upon revision, since the manuscript alone currently cannot rule out transductive leakage in the NPMI graphs."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a careful read, and I think the reader's conditional verdict is right. The combo of dual NPMI graphs, CharCNN node initialization, BERT, and gated fusion is genuinely new as a package, and the ablations make a decent case that each piece earns its place. I'd give credit for the clear architecture description and for an honest limitations section.\n\nThe soft spots are real and load-bearing. The split discipline question isn't a nitpick: in the scaling experiments they train on 1–5% of the data, and if the global subword/character graphs are built from the full 800k/1.56M dataset, then the test URLs' induced subgraphs (Eq. 10) contain edges computed with test-set co-occurrences. That's transductive leakage, and since the reported gains over URLBERT and LLaMA-2 are only 1–2% in accuracy/F1, leakage could easily create margins that don't exist. The paper says 'training corpus' but never states that the graph is rebuilt for each training subset, and no split protocol is given in Section IV. This must be fixed before the SOTA claim is meaningful.\n\nSecond, the metric definitions are sloppy. On Mendeley, the table reports P=0.9818, R=0.6712 at 1%, which gives binary F1≈0.797, but the table says 0.7578. At 5%, P=0.9851, R=0.7302 gives ≈0.838 vs the table's 0.8022. So either they're using macro-F1 or something else, and they don't say. That makes cross-model F1 comparisons uninterpretable.\n\nThird, the 'first' claims are overstated. The related work already mentions graph-based URL approaches, and the paper itself concedes that prior graph work exists. The novelty is the specific multi-granularity combination, not being first. Also missing: error bars/seeds, baseline adaptation details (especially how LLaMA-2 was fine-tuned or prompted), and no executed code to confirm the GitHub repo matches the experiments.\n\nThe architecture itself is credible and the ablations are a real strength. I'm not saying the results are fabricated—likely not—but the manuscript as written doesn't rule out the leakage path. This paper is useful for researchers working on URL detection, graph-based security, and anyone who cares about clean evaluation in empirical security. I'd like to see this go to peer review, because the idea is worth testing properly, but the authors need to supply the split protocol, metric definitions, seeds, and baseline details. If those check out, this is a solid subfield-level contribution.","headline":"A clever new architecture for URL detection, but the reported SOTA margins can't be trusted until the authors clarify whether their global NPMI graphs are built from each training split alone.","tokens_in":21842,"tokens_out":4102,"would_cite":true,"duration_ms":39369,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"URL2Graph++ claims that fusing BERT semantics with dual co-occurrence graphs over subwords and characters makes a malicious-URL detector that beats strong baselines, including LLMs, especially on short or obfuscated links.","keywords":["malicious URL detection","phishing","graph neural networks","BERT","character CNN","gated fusion","NPMI","co-occurrence graphs"],"falsifier":"Take GramBeddings at 6,400 training samples, freeze the NPMI graphs using only the training partition before any test split, and recompute Table III with an explicitly stated macro-F1; if the gap over LLaMA-2 narrows to noise or reverses, the central SOTA claim is false.","tokens_in":20817,"feed_emoji":"🛡️","tokens_out":2963,"duration_ms":28570,"temperature":0.7,"pith_summary":"The paper tries to establish that treating a URL not as a linear string but as a graph at two granularities—subword tokens and characters—lets a detector capture non-local dependencies and morphological obfuscations that sequential models miss. It proposes URL2Graph++, which builds corpus-wide NPMI co-occurrence graphs at both levels, initializes graph nodes with character-level CNN features, enriches the whole thing with BERT embeddings, and merges the three views with a learned gating weight. The authors report consistent gains over CNN, GAN, BERT, and LLaMA-2 baselines across balanced, imbalanced, cross-dataset, adversarial, and short-URL settings. If these results hold, a hybrid graph-plus-semantics architecture is the more robust detector for diverse and evasive malicious URLs.","feed_headline":"Graph-based URL detector beats LLMs on malicious links","feed_subtitle":"Fusing BERT semantics with character- and subword-level graphs lifts phishing detection across scales and obfuscation.","key_machinery":"The load-bearing object is the pair of global NPMI co-occurrence graphs: a subword-level graph and a character-level graph built from the training corpus, with edges where normalized pointwise mutual information exceeds a threshold. These graphs encode long-range token associations and character-level anomalies; per-URL subgraphs are extracted, node features are initialized by a shared CharCNN, refined by two jointly trained GCNs, and then merged with BERT's pooled representation through a gated convex combination (Eq. 16-17). The gating mechanism is what lets the model weight semantic vs structural vs character evidence per instance.","core_discovery":"The paper claims that a URL's meaning is carried by three complementary carriers—contextual subword semantics (BERT), global token co-occurrence (subword graph), and character-level morphology (character graph)—and that jointly training these with a gated fusion produces representations that generalize under distribution shift, label imbalance, obfuscation, and lexical sparsity. Concretely, on the balanced GramBeddings benchmark at 32,000 training samples it reports accuracy 0.9490 and AUC 0.9943, above URLBERT and LLaMA-2; on imbalanced Mendeley at 78,100 samples it reports F1 0.8022 and AUC 0.8903; on short malicious URLs (<40 chars) it reports accuracy 0.8920 and AUC 0.9536; and under adv","pith_inferences":["The dual-graph construction is effectively a corpus-level prior: if the training distribution shifts after deployment, the global graphs and BERT tokenizer need to be refreshed; the paper does not test continuous drift.","The 'short URL' result hints that the character-level graph carries much of the signal; a testable extension would train a variant with only CharCNN + character graph to isolate that contribution.","A natural stress test is adaptive adversarial attacks: the paper's adversarial evaluation uses a fixed insertion heuristic (hyphens), so robustness against an attacker who sees the model's gradients is open.","The method's compute cost (BERT + two GCNs + CharCNN) means the practical win may only materialize offline; latency-sensitive deployments might distill to a lighter student."],"forward_implications":["If correct, URL2Graph++ provides a template for detecting phishing and malware URLs that exploits non-local token relationships (e.g., 'account' far from 'verify') rather than only adjacent patterns.","The reported robustness at low FPR means security pipelines could set tight false-alarm budgets while still catching a meaningful share of malicious links.","Cross-dataset results suggest the representation transfers across URL distributions without fine-tuning, which matters for deployment on new data sources.","The gated fusion architecture implies that the optimal mix of semantic, structural, and character signals is instance-dependent; a single static concatenation is not enough.","Even under a 1:43 positive:negative ratio on Mendeley, the method keeps usable precision-recall trade-offs, indicating the graphs help resist label imbalance."],"fun_headline_variants":["Graph+BERT fusion beats LLMs on malicious URL detection","Hybrid URL model outguns zero-shot LLMs with graph semantics","Multi-scale graph learning boosts URL detection past LLM baselines","Combining BERT and graphs defeats LLMs on obfuscated URLs","Semantic-structural-character model outscores LLMs on malicious links"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The headline numbers presume that the subword and character co-occurrence graphs are built from training URLs only, and that the reported F1/precision/recall use one consistent formula; if test URLs leak into graph construction or metrics are computed inconsistently, the SOTA margins are not comparable.","fun_headline_variants_meta":{"raw":{"variants":["Graph+BERT fusion beats LLMs on malicious URL detection","Hybrid URL model outguns zero-shot LLMs with graph semantics","Multi-scale graph learning boosts URL detection past LLM baselines","Combining BERT and graphs defeats LLMs on obfuscated URLs","Semantic-structural-character model outscores LLMs on malicious links"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000844,"raw_usage":{"total_tokens":3562,"prompt_tokens":842,"completion_tokens":2720,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":2630}},"tokens_in":586,"tokens_out":2720,"duration_ms":20445,"temperature":1.0,"reasoning_tokens":2630,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T17:59:37.638922+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take GramBeddings at 6,400 training samples, freeze the NPMI graphs using only the training partition before any test split, and recompute Table III with an explicitly stated macro-F1; if the gap over LLaMA-2 narrows to noise or reverses, the central SOTA claim is false.","supporting_citations":[],"review_version":1}