{"id":"60dd29a0-7776-44b5-a43d-0d58f3e912df","arxiv_id":"2509.07532","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":10,"one_line_summary":"UGSR uses hierarchical uncertainty sampling plus codebook retrieval to improve continual Android malware detection under class imbalance and concept drift.","lead":"An Android malware detector that chooses which apps to label using a two-level uncertainty score, then uses a growing codebook of known app embeddings to spot new malware variants. It reports higher accuracy under a 50-label monthly budget than the strongest baseline, but the evaluation protocol and release of code need clarification.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1 separates Testing and Continual Learning into sequential loops; if evaluation on month n uses the codebook updated with month n's labels, the reported TPR/mACC are inflated.","rationale":"After reading the full manuscript, I agree with the reader's weakest assumption. The paper's central claim is an empirical one: UGSR obtains 92.95% TPR and 94.26% mACC on APIGraph with only 50 labels per month. For that claim to be meaningful, the evaluation on each month's data must occur before the model is updated with labels from that same month; otherwise the retrieval codebook could contain vectors extracted from the test month's labeled samples, giving the method direct access to test labels. Algorithm 1 is the only place where the order is specified, and it is ambiguous. The Testing loop (lines 8–18) and Continual Learning loop (lines 20–32) are both inside the outer while loop but are written as two separate sequential loops over T_n. This structure cannot represent the intended per-month 'test-then-update' flow: if all testing happens before any update, the continual learning steps would have no effect on the reported numbers, contradicting the claimed improvements over baselines; if the loops are meant to be interleaved, the pseudocode is missing an explicit 'for each month' outer loop that interleaves them. Moreover, the retrieval-based detection (Section III.D) uses the codebook V, and line 29 updates V with new features during the Continual Learning phase. If the update for month n occurs before the testing for month n (e.g., because the loops are executed in the written order), the codebook would contain features from D'_n, which are a subset of D_n, and the Top-k matching in line 15 would leak labels. The paper does not state in prose that evaluation on D_n precedes any update with D_n's labels, and Section III.C describes sampling from D_n as part of the continual learning stage, which could be read as happening before evaluation. This is not a matter of 'outside consensus' or 'internally inconsistent' in a narrow mathematical sense; it is a direct threat to the validity of the central empirical result. If the temporal order is indeed 'test first, then update', the paper should say so explicitly and ideally provide code; if it is not, the headline numbers are not trustworthy. Other issues (e.g., the misnamed 'ETF' operations in Eq. 12–14, missing error bars, unreported hyperparameters) are secondary because they do not by themselves invalidate the empirical comparison; the temporal ambiguity does. Therefore, I recommend keeping the reader's CONDITIONAL verdict: the paper should be revised to specify and verify the evaluation order, provide the code or a precise algorithm, and then the claims can be re-evaluated.","tokens_in":14576,"tokens_out":8756,"duration_ms":95695,"concrete_test":"Implement or inspect the exact evaluation loop structure. For each month n: (1) evaluate F_D and codebook V on all of D_n; (2) select D'_n and obtain labels; (3) fine-tune F_S/F_D and update V; (4) move to n+1. Re-run the APIGraph and AndroZoo experiments with this explicit ordering, and compare TPR/mACC to Tables I/II. If the reported numbers are produced after the model or codebook is updated with D_n's labels—e.g., if V used in line 15 contains feature vectors from D_n—the results should be recomputed; if the reported numbers drop, the central claim is invalid. Also request the authors' code to check whether the Testing loop and Continual Learning loop are interleaved or sequential.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim (TPR 92.95%, mACC 94.26% under a 50-label budget) rests on a strict temporal protocol: the model and codebook must be evaluated on each month's data D_n before any labels from D_n are used to update them. Algorithm 1 makes this order ambiguous. Lines 8–18 (Testing) and 20–32 (Continual Learning) are two separate loops both iterating over T_n; the testing loop runs completely before the continual learning loop. This implies either (a) all months are tested with the initial static model, then all months are used for updates—which would make the reported continual-learning improvements inexplicable—or (b) the loops are intended to be interleaved per month, but the pseudocode does not say so. If in interpretation (b) the codebook V is updated with features extracted from D'_n after fine-tuning on D'_n but before testing the remainder of D_n, then the retrieval module (line 15) has direct access to vectors derived from the test month's labeled samples, inflating TPR and mACC. The text in Section III.C ('For each incoming monthly test batch D_n, F_S calculates uncertainty scores...') further suggests that sampling from D_n is part of the continual learning phase, not explicitly post-evaluation. The paper never states in words that evaluation on D_n precedes any update using D_n's labels. This is the load-bearing assumption: if it fails, the headline numbers are not a valid measure of generalization to unseen monthly data.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"UGSR is a continual-learning framework for Android malware detection under class imbalance and concept drift. It couples a hierarchical uncertainty sampler (family-level multi-class and binary modules) with a detector trained via contrastive and weighted BCE losses, and a vector codebook used for retrieval-based detection. Static training initializes sampler, detector, and codebook; each month, a fixed budget of samples is actively selected, labeled, stored in a memory bank, and used to fine-tune the sampler and detector and to update the codebook; test predictions fuse the classifier output with top-3 codebook matches. Experiments on APIGraph and AndroZoo report TPR 92.95% and mACC 94.26% under a 50-label monthly budget, with comparisons to MLP/SVM/GBDT, CADE, and USENIX23 baselines, plus ablations and few-shot budgets down to 2 samples/month.","tokens_in":14945,"tokens_out":6726,"duration_ms":66680,"significance":"If the results hold, UGSR addresses a practical problem: sustaining Android malware detection with a very small monthly analyst labeling budget. The hierarchical sampler is a sensible response to the known majority-class bias of uncertainty sampling, and the retrieval module is a plausible non-parametric complement to the classifier. The ablations (Tables V and VI) suggest that both the retrieval module and the multi-class selector contribute to performance. However, the manuscript's current rigor does not support the headline claims: the temporal evaluation/update protocol is ambiguous, all results are single runs with no variance or significance testing, and the claimed ETF component is asserted rather than implemented. The framework is potentially valuable, but the evidence as presented is not yet convincing. No code or data are released, which further limits verification.","major_comments":[{"comment":"The temporal ordering of evaluation and model update is load-bearing and ambiguous. Algorithm 1 places Testing (lines 8–18) and Continual Learning (lines 20–32) as two separate sequential loops over T_n. Under a literal reading, all months are tested with the initial model and codebook before any updates occur, which cannot produce the reported continual-learning gains. Under the intended interleaved reading, the pseudocode does not state that evaluation on D_n precedes sampling/updating with D'_n drawn from D_n. Section III.C's description, 'For each incoming monthly test batch D_n, F_S calculates uncertainty scores...,' reinforces this ambiguity. If V is updated with features from D'_n before the remainder of D_n is tested, the retrieval module has direct access to vectors derived from the current month's labels, inflating TPR and mACC. The paper must unambiguously specify the protocol","section":"Algorithm 1, §III.C"},{"comment":"All performance tables report a single run per configuration with no standard deviations, confidence intervals, or significance tests. The abstract's claim that the framework 'significantly outperforms state-of-the-art methods' is therefore unsupported statistically. For a continual-learning evaluation over multiple months, the natural unit of comparison is monthly performance; the authors should report mean ± std over seeds (or over monthly periods) and a paired significance test (e.g., Wilcoxon signed-rank) against each baseline. This is especially important because several few-shot improvements are small (e.g., AndroZoo Count=2: TPR 62.67 vs. 59.54, mACC 79.92 vs. 79.33 in Table IV) and could be within noise.","section":"§IV.E, Tables I–IV and VII"},{"comment":"The ETF (Equiangular Tight Frame) claim is not implemented. Equations (12)–(13) pull stored vectors toward their class centroids, and Eq. (14) subtracts a benign-centroid component from malware vectors. These operations do not construct or enforce an equiangular tight frame; no ETF loss, simplex constraint, or angle equalization is defined anywhere. Algorithm 1's line 31, 'Apply ETF principle to V,' is therefore vacuous, and the contribution list's 'ETF-based feature regularization' is not demonstrated. The authors should either remove the ETF terminology or provide the actual ETF construction/regularization and an ablation quantifying its effect.","section":"§III.B, Eqs. (12)–(14); Algorithm 1 line 31"},{"comment":"The sampling procedure is internally inconsistent. Section III.C selects μB samples by multi-class uncertainty and (1−μ)B by binary uncertainty with μ=0.5, then enforces 'a minimum of 10% benign quota' by replacing low-scoring malware samples. Section IV.B states that continual learning 'maintaining a 40% benign sample ratio in incremental batches.' These two rules yield different class compositions for the 50-label budget, and the paper does not say which was used for Tables I and II. Since the headline result depends on the composition of the budget, the exact sampling rule and its hyperparameters must be stated without ambiguity.","section":"§III.C Eq. (15) vs. §IV.B"}],"minor_comments":[{"comment":"The mACC for 'w/o F_mul' is reported as 87.45, but (73.43+89.06)/2 = 81.25. Please correct this arithmetic inconsistency.","section":"Table VI"},{"comment":"The summation index in the first term is garbled ('Σ_{k=c}^{C}'), and S_c is not defined. Please rewrite the evidential loss expression carefully.","section":"Eq. (4)"},{"comment":"The matching threshold is variously denoted θ, τ, and k. In Eq. (17), θ is set equal to k, while §IV.B says 'matching thresholds set to τ=3.' Unify the notation.","section":"§III.D and Algorithm 1"},{"comment":"The paper says the codebook contains 'a total of 50 benign samples' and 'up to 3 samples per malware family,' but the total codebook size and family coverage are not stated. Please specify how many malware families are represented in the codebook.","section":"§III.B"},{"comment":"The figures show monthly TPR trajectories, but the captions do not define the x-axis time range or the number of months. Please add a textual summary of the variability, since this is central to the robustness claims.","section":"Figures 4–6"},{"comment":"The phrase 'Continue Learning' should be 'Continual Learning.'","section":"Index Terms"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope and addresses an important problem, but the current version overclaims. The temporal ambiguity in Algorithm 1 is the most serious issue because it directly affects the validity of every reported number; the authors must specify the protocol precisely and, ideally, release code. The single-run results and the unimplemented ETF claim also need substantive work. I see these as fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper has a legitimate new combination and the ablations are instructive, but the central empirical claim is not yet trustworthy because Algorithm 1 does not specify when testing happens relative to updating. If the codebook is refreshed with month n's labels before testing month n, the retrieval module has seen the test month's ground truth and the TPR/mACC numbers are inflated. The stress-test note is on target. Off the record, my guess is the authors meant to interleave testing and learning per month and just wrote the pseudocode poorly, but 'meant to' is not something a referee can verify from the text.\n\nWhat's actually new: the hierarchical dual-uncertainty sampler—using multi-class family-level uncertainty alongside binary boundary uncertainty to pick replay and incremental samples—plus a small cosine-similarity codebook that is updated over time with centroid-pulling operations. That combination is not in the cited prior work. The paper does well to compare against USENIX23 at budgets of 10 and 2, and the ablations show the retrieval module and the F_mul module each make a large difference. If those effects are real, it is a practically interesting result for security teams with tiny labeling budgets.\n\nSoft spots, in proportion:\n1. The temporal ordering is the load-bearing one. Algorithm 1's Testing loop runs completely before the Continual Learning loop. If that is literal, then all months are tested with the initial static model and the continual learning loop is only used to produce a final model that is never evaluated. If it is not literal, the text needs to say explicitly that evaluation on D_n strictly precedes any update using labels from D_n. This is not a minor typo.\n2. No standard deviations, no repeated runs, no significance tests. The abstract's 'significantly outperforms' is not supported.\n3. The 'ETF' label is wrong. Eqs. (13)–(14) pull vectors toward a centroid and subtract a scaled benign centroid. That is not an equiangular tight frame. Either implement actual ETF or call it centroid shrinkage.\n4. Several hyperparameters (λ_1, λ_2, λ_3, τ, θ_1/θ_2/θ_3) are not reported. With this many knobs, that matters.\n\nVerdict: this deserves a serious referee, not a desk reject. The reviewer needs to pin the authors down on the temporal protocol and ask for variance reporting. If the ordering is clean, the paper is a solid subfield contribution. If it is not clean, the headline TPR/mACC are not valid.","headline":"Plausible idea with a serious evaluation-ordering ambiguity; the reported numbers could be inflated if month n labels are used before testing month n.","tokens_in":15445,"tokens_out":3557,"would_cite":false,"duration_ms":36925,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One method sustains Android malware detection at 94% balanced accuracy with only 50 monthly analyst labels.","keywords":["Android malware detection","continual learning","uncertainty sampling","class imbalance","vector retrieval","concept drift","few-shot learning","codebook"],"falsifier":"Re-run the APIGraph experiment with the codebook update strictly moved after evaluation on each month's test set (i.e., update with D_n labels only after scoring D_n). If the reported TPR and mACC drop to the level of the baseline or lower, the retrieval module is leaking test labels. A simpler check: run the retrieval-only decision (without the neural classifier) on the test month using only codebooks from prior months; if its malware TPR exceeds the classifier's standalone TPR, the retrieval is exploiting temporally future information.","tokens_in":14448,"feed_emoji":"🛡️","tokens_out":2042,"duration_ms":25832,"temperature":0.7,"pith_summary":"This paper argues that continual Android malware detection fails not because of concept drift alone, but because uncertainty-based sample selection is biased toward the abundant benign class. The authors claim that decoupling sample selection from the detector, using a hierarchical uncertainty scorer over both malware families and binary labels, repairs this bias. They add a similarity-retrieval module whose codebook is updated with representative embeddings that are pushed toward class centroids and orthogonalized away from the benign direction. Under a strict budget of 50 labeled samples per monthly phase, they report a malware true positive rate of 92.95 percent and balanced accuracy of 94.26 percent on the APIGraph dataset, and 80.56 percent TPR with 88.29 percent balanced accuracy on AndroZoo, beating the strongest prior continual-learning baseline by roughly 3 to 11 percent balanced accuracy. If correct, this would make sustained 'label-scarce' malware detection practical: a small daily analyst effort keeps detection accurate over years of drift.","feed_headline":"50 monthly labels sustain Android malware detection at 94% accuracy","feed_subtitle":"A hierarchical uncertainty sampler plus embedding codebook beats prior continual-learning detectors on two large app datasets.","key_machinery":"The load-bearing mechanism is the hierarchical uncertainty sampler paired with an ETF-influenced codebook. The sampler has a multi-class module trained with an evidential Dirichlet loss (to estimate family-wise uncertainty) and a binary module trained with supervised contrastive plus binary cross-entropy; their union with a minimum benign quota selects balanced, high-information samples for analyst labeling. The codebook stores fixed-size class-balanced feature vectors, refines them by centroid compaction and by subtracting the benign centroid component from malware vectors, then classifies test samples by top-k cosine matching with fallback to the neural classifier. Together the two parts p","core_discovery":"The central claim is that the cause of poor continual malware detection under low-label budgets is the structural imbalance in which samples get selected for labeling, not merely model forgetting. The paper's UGSR framework separates the sampler from the detector: a two-stage sampler estimates uncertainty at the malware-family level and at the benign-versus-malicious boundary, selecting a confidence-balanced mix that enforces a minimum benign quota; a separate detector plus a vector codebook then detects by combining neural classification with retrieval. The retrieval codebook stores only 50 benign vectors and 3 per malware family, pulling stored vectors toward class centroids and making mal","pith_inferences":["The paper leaves implicit a concrete deployment procedure: the 50 monthly labels should be drawn from the most uncertain family- and binary-level samples, but the paper does not state who labels them or how label turnaround time interacts with the monthly cadence; a testable extension is to measure performance when labels arrive with a one-month delay.","The retrieval codebook's ETF-style orthogonalization is a geometric post-processing on stored embeddings; an untested but natural extension is to apply the same orthogonalization to the classifier's decision function itself, which if effective would reduce dependence on the hand-tuned threshold parameters.","A further implicit consequence is that the method as presented assumes the analyst labels samples from the same month's distribution before that month's test is run; any real deployment must quarantine a month's test set before updating on its labels, which the algorithm description does not yet make explicit.","Connecting to drift-detection literature, the retrieval module could double as a drift alarm: a spike in retrieval fallback (when neither benign nor malware matches reach threshold) identifies months in which the encoder has met genuinely new behavior, a signal the paper does not exploit."],"forward_implications":["If the reported gains hold, real-world security teams could maintain effective Android malware detection with about 50 labels per month, roughly 1-2 analyst-hours per phase.","The hierarchical family-level uncertainty idea could transfer to other highly imbalanced continual-learning tasks (fraud, intrusion, medical screening) where the minority class is the one that matters.","Retrieval-plus-classifier fusion appears to stabilize performance across time, smoothing month-to-month spikes that current baselines suffer, implying a safer operational profile for deployed detectors.","Ablations indicate that dropping the family-level sampler causes catastrophic overfitting to the benign class, i.e., the 'accuracy' of past methods may partly reflect predicting everything benign.","With only 2 labels per phase, the method still improves TPR over the best baseline, suggesting that extremely scarce-label regimes remain usable in practice."],"supporting_citations":[{"why":"The main baseline (USENIX23) that the paper must beat and whose methodology it adapts; supplies the continual-learning setup and the comparison numbers for label budgets.","marker":"[1]"},{"why":"API-Graph dataset: the primary benchmark of about 320,000 apps over 2012-2018 used for the headline 92.95% TPR / 94.26% mACC results.","marker":"[42]"},{"why":"AndroZoo dataset: the second benchmark used to test the method's transfer to 2019-2021 apps.","marker":"[43]"},{"why":"Provides the evidential Dirichlet formulation that underpins the multi-class uncertainty module F_mul within the hierarchical sampler.","marker":"[38]"},{"why":"CADE, an out-of-distribution drift baseline used as a comparison method in the experimental tables, showing the paper's edge over drift-specialized approaches.","marker":"[50]"},{"why":"BODMAS baseline with a GBDT learner, used as a strong traditional ML comparison under the same continual label budget.","marker":"[49]"},{"why":"TOPIC, the few-shot class-incremental learning baseline that motivates the low-label continual setting the paper claims to improve.","marker":"[28]"}],"fun_headline_variants":["50 labels per phase? This continual malware detector reaches 94% accuracy","Uncertainty-driven hierarchical sampling defeats class imbalance in malware CL","Retrieval codebook and balanced sampler improve continual malware detection","Low-label continual malware detection fixed by uncertainty-guided sampling","Class-balancing sampler plus embedding retrieval hits 94% in malware CL"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The paper's results depend on the evaluation order in Algorithm 1: the model must be tested on month n's data before it is updated with any labels from that same month. If instead the codebook and classifier are updated with month n's labels before testing month n, the retrieval module would have direct access to the test labels, inflating the reported TPR and mACC. The manuscript does not make this ordering unambiguous.","fun_headline_variants_meta":{"raw":{"variants":["50 labels per phase? This continual malware detector reaches 94% accuracy","Uncertainty-driven hierarchical sampling defeats class imbalance in malware CL","Retrieval codebook and balanced sampler improve continual malware detection","Low-label continual malware detection fixed by uncertainty-guided sampling","Class-balancing sampler plus embedding retrieval hits 94% in malware CL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"total_tokens":1218,"prompt_tokens":738,"completion_tokens":480,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":392}},"tokens_in":482,"tokens_out":480,"duration_ms":6516,"temperature":1.0,"reasoning_tokens":392,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T22:01:32.029004+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the APIGraph experiment with the codebook update strictly moved after evaluation on each month's test set (i.e., update with D_n labels only after scoring D_n). If the reported TPR and mACC drop to the level of the baseline or lower, the retrieval module is leaking test labels. A simpler check: run the retrieval-only decision (without the neural classifier) on the test month using only codebooks from prior months; if its malware TPR exceeds the classifier's standalone TPR, the retrieval is exploiting temporally future information.","supporting_citations":[{"cited_title":"Continuous learning for Android malware detection","cited_arxiv_id":null,"evidence_quote":"The main baseline (USENIX23) that the paper must beat and whose methodology it adapts; supplies the continual-learning setup and the comparison numbers for label budgets."},{"cited_title":"Enhancing state-of-the-art classifiers with API semantics to detect evolved Android malware","cited_arxiv_id":null,"evidence_quote":"API-Graph dataset: the primary benchmark of about 320,000 apps over 2012-2018 used for the headline 92.95% TPR / 94.26% mACC results."},{"cited_title":"https://androzoo.uni.lu/","cited_arxiv_id":null,"evidence_quote":"AndroZoo dataset: the second benchmark used to test the method's transfer to 2019-2021 apps."},{"cited_title":"Evidential deep learning to quantify classification uncertainty.Advances in Neural Information Processing Systems, 2018, 31","cited_arxiv_id":null,"evidence_quote":"Provides the evidential Dirichlet formulation that underpins the multi-class uncertainty module F_mul within the hierarchical sampler."},{"cited_title":"CADE: Detecting and explaining concept drift samples for security applications","cited_arxiv_id":null,"evidence_quote":"CADE, an out-of-distribution drift baseline used as a comparison method in the experimental tables, showing the paper's edge over drift-specialized approaches."},{"cited_title":"BODMAS: An open dataset for learning based temporal analysis of PE malware","cited_arxiv_id":null,"evidence_quote":"BODMAS baseline with a GBDT learner, used as a strong traditional ML comparison under the same continual label budget."},{"cited_title":"Few-shot class-incremental learning","cited_arxiv_id":null,"evidence_quote":"TOPIC, the few-shot class-incremental learning baseline that motivates the low-label continual setting the paper claims to improve."}],"review_version":1}