{"id":"06ca93dc-46de-4a21-a192-3e49f8faea95","arxiv_id":"2411.09393","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Rolling-buffer Additive Gaussian Processes deliver competitive, interpretable, uncertainty-aware URL phishing classification in an online setting.","lead":"This paper applies interpretable additive models, especially Additive Gaussian Processes, to online URL phishing detection, using a rolling buffer to limit retraining cost. It shows these models classify phishing URLs nearly as well as a neural network while providing feature-level explanations and uncertainty estimates.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never measures the computational cost of its proposed rolling buffer, so the central 'eases the scaling issue' claim is asserted rather than demonstrated, and the performance comparisons lack error bars and statistical tests, leaving 'no significant loss' unquantified.","rationale":"The paper's advertised contribution is a pipeline that makes AGPs scalable in online settings (§3.3, Abstract). The experiments, however, only evaluate predictive performance: Fig. 6 and 7 report F1, §4.2 reports AUC, and no runtime or memory measurements appear anywhere. A rolling buffer of 20% of 11k samples still requires inverting a ~2200×2200 covariance matrix at every update for GP/AGP (cost O(n^3) per §2.1); whether that is 'easing the scaling issue' depends on the actual per-update latency and on comparison with full-data retraining or with NAM/NN baselines, none of which is provided. The performance side is similarly thin: single-run F1 curves without error bars cannot support the phrase 'no largely statistically significant difference' (§4.2). These are fixable omissions, so CONDITIONAL remains appropriate. The temporal-correlation issue raised by the reader is real but secondary: the paper explicitly assumes sample independence (§2) and its random-batch protocol is internally consistent with that assumption; the missing piece is evidence on real ordered streams, whereas the scaling claim is unsupported even on the paper's own terms.","tokens_in":12595,"tokens_out":5887,"duration_ms":54676,"concrete_test":"Re-run the released code on the URL dataset, recording per-update wall-clock time and peak memory for GP and AGP at window proportions 5%, 10%, 20%, 50%, and 100%, alongside NN and NAM, and repeat the 20%-window condition with 10 independent seeds to report mean±std F1/AUC. If AGP update time does not decrease with window size according to the O(m^3) rate, or if the AGP-to-NN F1 gap is larger than the seed-to-seed variance, then the §3.3 scalability and 'no significant loss' claims are not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Abstract, §3.3) is that a rolling buffer of recent plus randomly drawn historical data eases AGP/GP scaling 'without incurring a significant loss in terms of model performance.' The experiments measure only accuracy: Fig. 6 plots F1 against window proportion, Fig. 7 plots F1 against labelled batch proportion, and §4.2 reports AUC. No experiment records update time, memory, or FLOPs, and there is no comparison to retraining on the full dataset. The paper itself states GP/AGP cost is O(n^3) (§2.1), so reducing n from ~11,000 to a 20% window (~2,200) still leaves a cubic solve per update; whether that is manageable for online security workloads is never quantified. Additionally, the F1/AUC numbers appear to be single-run: no error bars, repeated seeds, or statistical tests are reported, so the claim of 'no largely statistically significant difference' (§4.2) is unsupported. The weaker claim that AGPs are usable could survive this, but the advertised contribution—alleviating AGPs' main drawback—requires evidence on the compute/accuracy trade-off that is absent.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes using Additive Gaussian Processes (AGPs) and Neural Additive Models (NAMs) for online supervised learning in cybersecurity, specifically phishing URL classification. The central claim is that a rolling buffer of recent plus randomly drawn historical data eases AGPs' cubic scaling without incurring significant performance loss (Abstract, §3.3), while retaining inherent interpretability and uncertainty quantification. The authors compare AGP, GP, NAM, and NN on a Kaggle phishing dataset (11,000+ URLs, 30 features) using ROC/AUC and F1 curves across varying window proportions and active-learning label budgets (§4). They also show feature-level contribution and variance plots (§4.4) to illustrate interpretability and uncertainty awareness.","tokens_in":12854,"tokens_out":2352,"duration_ms":23933,"significance":"If the central claim were fully supported, the paper would be a useful practical contribution: it would show that an inherently interpretable, uncertainty-aware model can be applied online to a cybersecurity task at competitive accuracy with a concrete mechanism (rolling buffer) to mitigate the main known drawback of full GPs. The paper explicitly provides code (Abstract), and the model family (AGPs) is standard and correctly cited to external work, so the methodological core is sound. However, the significance is currently limited by the evidence provided: the scalability claim is asserted without timing or complexity measurements, the performance claims lack error bars and statistical tests, and the active-learning experiment lacks a random baseline. The interpretability and uncertainty visualizations are interesting but qualitative. The paper contributes a plausible pipeline, yet the advertised advantages over standard GPs and over non-additive baselines are not rigorously demonstrated.","major_comments":[{"comment":"The central scalability claim that the rolling buffer 'eases the scaling issue' of AGPs is never directly tested. The paper reports F1 versus window proportion (Fig. 6) and versus labelled batch proportion (Fig. 7), but no runtime, memory, or FLOPs measurements are presented, and there is no comparison against retraining on the full dataset. Since §2.1 states the O(n^3) cost of GPs, the reader cannot assess whether a 20% window (about 2,200 points) is actually manageable for online cyber-security workloads. This is a load-bearing omission: the abstract and conclusion both rest on the claimed computational improvement.","section":"§3.3, §4.3"},{"comment":"The statement that 'no largely statistically significant difference' exists between the models is unsupported: the AUC values (NN 0.978, GP 0.955, NAM 0.969, AGP 0.952) are single-run numbers with no error bars, confidence intervals, or statistical tests. The same issue affects Figs. 6 and 7, where F1 curves are shown without repeated seeds or variance estimates, so the claim of 'no significant losses' at small window proportions is not quantified.","section":"§4.2, Fig. 5"},{"comment":"The active-learning experiment does not include a random-labelling baseline. Without comparing uncertainty-sampling to randomly selecting labels from the new batch, the conclusion that 'uncertainty-based active sampling is able to select highly representative samples' is not established. The observed upward trend in F1 as the labelled proportion increases could equally occur with random selection.","section":"§4.3, Fig. 7"},{"comment":"The online-learning setup assumes consecutive samples are independent: '(xt, yt) is independent of (xt+1, yt+1)' (§2). However, the experiments explicitly draw 'a random batch of new URLs' (§4.1), which removes any temporal ordering present in the original dataset. Real security event streams are temporally correlated (e.g., campaign-style phishing attacks), so the reported F1/AUC and the rolling-buffer design may not transfer to actual online deployment. The manuscript provides no experiment with temporally ordered data, which weakens the claim of applicability to 'the dynamic learning environment' of cyber-security.","section":"§2, §4.1"}],"minor_comments":[{"comment":"Equation (1) contains a typographical error: 'g(E[Y ]) = β0 + Pp j=1 fj(xj) = (1) = β0 + f1(x1) + ...' has a duplicated equals sign and an unnecessary '(1)=' fragment that should be removed.","section":"Eq. (1)"},{"comment":"The statement that the input-specific functions fj(xj) can be interpreted 'as Shapley values' is imprecise. GAM components are additive feature contributions, but they are not the Shapley values of the underlying game in general; the connection requires additional assumptions. Either clarify the sense in which this holds or cite a source that establishes it.","section":"§3"},{"comment":"The experimental setup does not state how many runs are averaged for the F1 and AUC results, nor whether the same train/test splits are used across models. Reporting the number of independent repetitions and the variance of the metrics would substantially improve the paper.","section":"§4.1"},{"comment":"The abstract says 'The source code is available here' but the full text only says 'available here' without a visible URL. Please include an explicit link or repository identifier.","section":"Abstract"},{"comment":"The ROC curves in Fig. 5 are not labelled in the legend in a way that allows the reader to match each curve to the AUC values listed in the text; consider adding the AUC values directly to the legend.","section":"Fig. 5"}],"recommendation":"major_revision","confidential_remarks":"The paper's central contribution is a scalability claim that is currently supported only by accuracy plots, not by any compute-time evidence. The independence assumption in §2 also conflicts with the random-batch update procedure in §4.1. I believe the manuscript can be repaired by adding (a) wall-clock or complexity measurements for the rolling buffer, (b) error bars or significance tests for all main F1/AUC comparisons, (c) a random baseline in the active-learning experiment, and (d) at least one experiment on temporally ordered data or an explicit discussion of why the independence assumption is acceptable for the chosen problem. If these are added, the paper could be a solid empirical contribution; without them, the advertised advantages over standard GPs are not demonstrated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a straightforward but competently executed application of Additive Gaussian Processes to online phishing URL classification, and if you work on interpretable ML for security it is worth knowing about. What is actually new: nobody has evaluated AGPs in this setting, and the rolling-buffer plus uncertainty-sampling active learning pipeline is a sensible combination that security practitioners could adopt. The exposition of AGPs and NAMs is clear, and the feature-contribution plots (Figs. 8, 9) give a genuine sense of what interpretability buys you: both interpretable models land on HTTPS and anchor URL features as important, and AGP correctly flags feature 13 as high-variance.\n\nWhere the paper goes soft is exactly where the reader's report says. The central claim—that the rolling buffer eases AGP scaling without significant loss—is never directly measured. No runtime, memory, or FLOPs numbers appear anywhere; Figure 6 only shows F1 against window proportion, and there is no comparison against retraining on the full dataset. Given that the paper itself quotes O(n^3) GP cost, a 20% window on 11k points still leaves a non-trivial cubic solve per update. The phrase \"no largely statistically significant difference\" also effectively asserts a statistical claim without any tests, standard errors, or repeated seeds. The active-learning experiment lacks a random-labelling baseline, so you cannot tell whether uncertainty sampling is actually doing anything beyond the benefit of more labels. The independence assumption ((x_t,y_t) independent across t) is stated but never tested, and real security event streams are rarely that clean.\n\nNone of these are fatal for the basic feasibility message. The models do achieve competitive AUC on a public dataset, and the interpretability/uncertainty outputs look plausible. But the advertised contribution—fixing AGP's main drawback—rests on a measurement that is missing. The authors seem aware of this in the conclusion, where they admit computational complexity remains a significant challenge; that honesty is to their credit but it undercuts the abstract.\n\nA serious referee should engage with this. The paper deserves review, but only with the expectation that the authors add compute measurements, repeated runs with error bars, and a random baseline for active learning. For me, this is a maybe for the reading group and not something I would cite in my own work yet.\n\nRecommendation: send to peer review, but treat the scalability claim as unverified and require the missing experiments before acceptance.","headline":"A competent application of AGPs to online phishing classification whose feasibility claim survives but whose advertised scalability fix is never actually measured.","tokens_in":13360,"tokens_out":2005,"would_cite":false,"duration_ms":19543,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that Additive Gaussian Processes, retrained on a rolling buffer of recent and randomly drawn historical data, can scale to online cyber-security classification while keeping interpretability and uncertainty…","keywords":["additive Gaussian processes","interpretable machine learning","uncertainty quantification","online learning","cyber-security","phishing detection","active learning","generalized additive models"],"falsifier":"Run the same pipeline on a timestamped stream of real URLs or network connections in chronological order, compare its F1 and AUC against an offline model retrained on all data, and measure the autocorrelation of labels and features. A material drop in performance, or a buffer that cannot track drift, would refute the claim.","tokens_in":12403,"feed_emoji":"🛡️","tokens_out":8635,"duration_ms":73930,"temperature":0.7,"pith_summary":"The paper tries to establish that Additive Gaussian Processes (AGPs)—Gaussian-process versions of generalized additive models, where each feature gets its own kernel and its own function—can be made practical for online cyber-security classification. Its proposed fix is a rolling buffer: at each update, retrain on the most recent batch plus randomly sampled historical batches, instead of the whole dataset. On a URL phishing classification task with over 11,000 samples and 30 features, the AGP reaches AUC 0.952 and F1 scores in the same range as the other models, close to a black-box neural network (AUC 0.978), while providing per-feature contributions and posterior variances the network cannot. If the claim holds, security analysts get a model that is transparent by construction and uncertainty-aware without paying a large accuracy penalty, and the scalability objection that has blocked AGP adoption is eased.","feed_headline":"Additive GPs scale online with little accuracy loss","feed_subtitle":"A 20% rolling window keeps F1 and AUC near black-box nets while preserving per-feature uncertainty.","key_machinery":"The load-bearing object is the additive kernel $k(x,x') = k_0 + \\sum_{j=1}^p k_j(x_j,x_j')$, which makes a Gaussian Process take the generalized additive form $g(E[Y]) = \\beta_0 + \\sum_{j=1}^p f_j(x_j)$. Because each feature has its own independent kernel, the per-feature function $f_j(x_j)$ can be recovered and interpreted as a Shapley value, and it carries its own posterior variance, giving both interpretability and uncertainty quantification. The scalability mechanism is the rolling buffer: retraining on the most recent data plus randomly drawn historical data, with the recent-to-historical ratio a tunable hyperparameter, instead of retraining on the full dataset or learning sparse inducing points at every step.","core_discovery":"The central claim is that the rolling-buffer pipeline eases the scaling issue of AGPs \"without incurring a significant loss in terms of model performance,\" in the paper's words. The demonstration is an online URL phishing classification problem: models are warmed up on an initial window, then updated with random batches of 50–100 new URLs, and optionally with uncertainty-based active learning that labels only the most uncertain samples. Across window proportions from roughly 10% to 100%, the AGP's F1 score stays close to those of a full GP, a Neural Additive Model, and a neural network, and at the reported 20% window its AUC is 0.952 versus 0.978 for the neural network. The AGP also concentrates its posterior variance on a small set of meaningful features—absence of HTTPS and absence of an anchor URL—while the neural additive model disperses its variance across many features, which the paper reads as evidence that NAM suffers from the over-confidence typical of deep networks. The paper positions AGP as the first inherently interpretable and uncertainty-aware model class applied to cyber-security online learning.","pith_inferences":["The rolling-buffer recipe is model-agnostic: any GP variant whose bottleneck is the $O(n^3)$ covariance inversion could be dropped into the same online loop, so the paper's scalability fix may carry over to sparse, deep, or multi-output GPs without new theory.","The independence assumption is likely the first thing to break in deployment: real security event streams are autocorrelated, and the reported metrics come from randomly drawn batches, so a chronological re-run on real traffic is the natural stress test.","The feature-specific variance signal suggests a cheap novelty detector: flag samples whose highest-variance feature shifts away from the features the model relies on, and route those to human analysts; this is a direct, testable use of the AGP output that the paper does not implement.","If the result transfers, the main practical competitor is not the neural network but the Neural Additive Model, since NAM matches the interpretability at lower computational cost; the decisive advantage claimed for AGP is exactly the uncertainty quantification that NAM lacks, so deployment decisions hinge on how much that uncertainty is worth in practice."],"forward_implications":["At a window proportion of 20–30%, models keep most of their full-data F1, so the buffer defines a practical operating point for online retraining where the computational saving is large and the accuracy cost is small.","AGP outputs can be read as per-feature Shapley contributions, so an analyst can see directly that missing HTTPS and missing anchor URLs drive phishing decisions, without running a post-hoc explainer.","The posterior variance gives a principled acquisition function for active learning: labelling only 10–20% of each new batch maintains performance, lowering labelling cost.","Because the GP posterior separates epistemic from aleatoric uncertainty, the model can flag out-of-sample inputs, something the neural baselines cannot do reliably.","The additive-kernel construction extends naturally to regression and to pairwise interaction terms, so the pipeline is not limited to binary phishing detection."],"supporting_citations":[{"why":"Defines Additive Gaussian Processes and the additive kernel that gives per-feature functions; supplies the model class the paper scales.","marker":"[16]"},{"why":"Introduces Generalized Additive Models, the functional form that gives AGP and NAM their inherent interpretability.","marker":"[24]"},{"why":"Provides the Gaussian Process regression and classification machinery, including the posterior variance that underlies AGP uncertainty estimates.","marker":"[52]"},{"why":"Introduces Neural Additive Models, the interpretable neural baseline that AGP is compared against.","marker":"[3]"},{"why":"Presents sparse inducing-point GPs, the scalability alternative the paper argues is less suited to online retraining than a rolling buffer.","marker":"[38]"},{"why":"Proposes a random Fourier feature approximation of GP-NAMs that trades full GP uncertainty for scalability, a trade-off the paper avoids.","marker":"[57]"},{"why":"Defines Shapley values, used to interpret the per-feature additive functions as contributions to the final classification.","marker":"[48]"}],"fun_headline_variants":["Interpretable AGPs go online with small accuracy trade-off","Rolling-window AGPs keep F1, AUC near neural nets","Uncertainty-aware AGPs scale for cyber-security online learning","Additive GPs scale online with little accuracy loss"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the assumption that consecutive samples in the online stream are independent; real security traffic arrives in correlated bursts, so the reported numbers may not survive a chronological deployment.","fun_headline_variants_meta":{"raw":{"variants":["Interpretable AGPs go online with small accuracy trade-off","Rolling-window AGPs keep F1, AUC near neural nets","Uncertainty-aware AGPs scale for cyber-security online learning","Additive GPs scale online with little accuracy loss"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000771,"raw_usage":{"total_tokens":3682,"prompt_tokens":979,"completion_tokens":2703,"prompt_tokens_details":{"cached_tokens":896},"prompt_cache_hit_tokens":896,"prompt_cache_miss_tokens":83,"completion_tokens_details":{"reasoning_tokens":2633}},"tokens_in":83,"tokens_out":2703,"duration_ms":21588,"temperature":1.0,"reasoning_tokens":2633,"cache_read_input_tokens":896,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:40:28.378077+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pipeline on a timestamped stream of real URLs or network connections in chronological order, compare its F1 and AUC against an offline model retrained on all data, and measure the autocorrelation of labels and features. A material drop in performance, or a buffer that cannot track drift, would refute the claim.","supporting_citations":[{"cited_title":"Generalized additive models","cited_arxiv_id":null,"evidence_quote":"Introduces Generalized Additive Models, the functional form that gives AGP and NAM their inherent interpretability."},{"cited_title":"A unifying view of sparse approximate gaussian process regression","cited_arxiv_id":null,"evidence_quote":"Presents sparse inducing-point GPs, the scalability alternative the paper argues is less suited to online retraining than a rolling buffer."},{"cited_title":"Gaussian process neural additive models","cited_arxiv_id":null,"evidence_quote":"Proposes a random Fourier feature approximation of GP-NAMs that trades full GP uncertainty for scalability, a trade-off the paper avoids."}],"review_version":1}