{"id":"3136ee21-f024-43e5-b796-4a9ac149afe9","arxiv_id":"2506.23314","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"MH-AutoML is a domain-specific AutoML framework for Android malware detection that combines automated modeling with built-in interpretability, and its evaluation shows competitive recall and higher transparency scores than seven general-purpose tools.","lead":"This paper introduces MH-AutoML, an automated machine learning framework tailored for Android malware detection that adds transparency, interpretability, and experiment tracking to the usual AutoML pipeline. It compares MH-AutoML against seven general-purpose AutoML tools on nine Android malware datasets, reporting competitive recall and better explainability scores.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'more transparency and control' claim rests on a self-designed rubric (Section V-C) applied by the authors to their own tool; the scoring formula is malformed and no external validation is reported, so the central differentiator is unfalsifiable.","rationale":"Stress-testing the central claim requires checking both prongs: better recall and more transparency. The recall prong is fragile: Section V-D describes a single holdout with no repeated runs or error bars, and Section IV says MH-AutoML tunes with Recall prioritized while competitors run with defaults, so the observed recall advantages may reflect objective mismatch rather than capability. However, the paper's distinctive contribution—and its title—is the interpretability/transparency dimension. The evidence for that dimension is entirely internal: the authors constructed a questionnaire, assigned scores to their own tool, and reported superiority. The two concrete defects—the malformed normalization formula (Section V-C) and the inconsistency between Table VI (all 100s) and the reported interpretability scores (e.g., 58.33)—mean the scores are not reproducible from the information given. Even a perfect performance comparison would not rescue the transparency claim if the rubric is a tailored self-assessment. The proposed external-rater check directly tests whether the rubric is impartial and whether MH-AutoML's advantage is real; unless that test succeeds, the central claim 'more transparency and control' is unfalsifiable. This does not change the reader's CONDITIONAL verdict: the conditions are exactly to supply this validation (plus the other methodological fixes). Therefore the verdict stays UNCHANGED.","tokens_in":19312,"tokens_out":6002,"duration_ms":66463,"concrete_test":"Recruit at least three independent raters with ML/security background who are blinded to the paper's goal. Give each rater the eight tools (or their public documentation and a default installation) and ask them to score every question in Table VI, following the stated 0/1/2 rules, with the corrected formula S = (sum P_i / (N*W))*100. Compute per-tool mean scores and inter-rater agreement (e.g., Cohen's kappa or Krippendorff's alpha). The transparency claim is settled only if MH-AutoML's mean score exceeds the best competitor by more than the inter-rater SD, and if agreement is at least moderate (alpha >= 0.6). Publish the raw per-question scores and the filled-out justification for each score. If the advantage does not replicate or is within noise, the 'more transparency' claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central differentiator—'more transparency and control'—is supported only by the authors' own five-dimension questionnaire (Section V-C, Table VI). The authors score their own framework, which was designed to implement the exact features the rubric rewards (MLflow tracking, Optuna logs, SHAP/LIME; Section IV-A). No external raters, blinded assessment, inter-rater reliability, or independent audit is reported. More seriously, the quantitative formula in Section V-C is not a formula: it defines S as a piecewise function of N alone (0 if N=0, 1 if N=1, 2 if N>=2), with no use of the question scores P_i or the maximum score W, so the reported 0-100% category scores cannot be recomputed from the stated procedure. The score table (Table VI) confusingly lists every category as 100, which conflicts with the text reporting non-perfect interpretability scores (e.g., MH-AutoML's 58.33 in Section VI-A), making the scoring procedure internally inconsistent. Because the headline transparency advantage is the paper's main contribution, a rubric that is biased, unvalidated, or unreproducible collapses the 'more transparency' claim. The recall comparison is also under-powered (single holdout, no error bars, and MH-AutoML tunes for recall while competitors use defaults), but the transparency measurement is the more fundamental weakness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MH-AutoML, a domain-specific AutoML framework for Android malware detection that automates data preprocessing, feature engineering, model selection, and hyperparameter tuning, and integrates MLflow experiment tracking, Optuna hyperparameter optimization, and SHAP/LIME interpretability tools. It reports a comparative evaluation against seven general-purpose AutoML frameworks (Auto-Sklearn, AutoGluon, TPOT, HyperGBM, Auto-PyTorch, LightAutoML, and MLJAR) on nine Android malware datasets, using recall, MCC, and execution time, together with a custom five-dimension transparency/interpretability questionnaire. The central claims are that MH-AutoML achieves better recall rates than the competitors on most datasets while providing more transparency and control.","tokens_in":19594,"tokens_out":5952,"duration_ms":59551,"significance":"If the results are reliable, the paper would offer a useful domain-specific AutoML baseline for security analysts, combining automated pipeline construction with interpretability, experiment tracking, and debugging support. The comparison across nine Android-specific datasets with both original and balanced versions is a valuable contribution, and the authors make the code and datasets publicly available. The choice of recall, MCC, and execution time as evaluation axes is appropriate for malware detection. However, the headline differentiator—'more transparency and control'—rests on a self-designed and self-applied questionnaire whose scoring formula is internally inconsistent, and the recall advantage is potentially confounded by the optimization objective used exclusively for MH-AutoML. These issues currently prevent the evidence from supporting the abstract's claims as stated.","major_comments":[{"comment":"The normalization formula is not a function of the question scores: it defines S as 0, 1, or 2 depending only on the question count N, while the variables P_i and W described in the surrounding text are never used. Consequently, the 0-100% category scores reported in Section VI-A and Figure 4 cannot be recomputed from the stated procedure. In addition, Table VI assigns 100 to every category, including Interpretability and Internal Analysis, while Section VI-A reports MH-AutoML scores of 58.33 and 50 for those same categories. This internal inconsistency makes the quantitative transparency assessment unreproducible and unfalsifiable; the formula must be corrected, and the scoring should be performed by independent raters or at least in a blinded manner.","section":"Section V-C, Eq. (2), Table VI"},{"comment":"The paper states in Section V-B that experiments use 'eight carefully selected datasets (Table VII)' but Section V-E and Table VII list nine datasets. Table II's 'This work' row reports 'Holdout + 5-fold CV' as the validation method, while Section V-B explicitly states that 'we employ the Holdout method.' No repeated runs, error bars, or statistical tests are reported, and Section V-D does not specify numerical time limits despite saying that time limits were adjusted. The balancing procedure used to create the 'balanced datasets with unique samples' is never described. These inconsistencies affect every performance claim in the paper and should be resolved with a single clear protocol.","section":"Section V-B, Table II, Section V-E"},{"comment":"MH-AutoML's tuning stage is described as prioritizing Recall through Optuna, whereas Section V-D states that all competitors were run with 'default configuration.' The observation that MH-AutoML leads in recall on six of nine datasets could therefore reflect the choice of optimization objective rather than a structural advantage of the framework. The paper should either tune all frameworks for recall, use a common objective, or explicitly discuss this confound and its effect on the reported recall comparisons.","section":"Section IV-A and Section V-D"},{"comment":"The balanced-versus-original analysis contains a direct numerical contradiction: 'the KronoDroid Emulator dataset shows improved recall after balancing, with HyperGBM increasing from 94.03% to 91.69%.' A decrease from 94.03% to 91.69% is described as an increase. Furthermore, the term 'balanced datasets with unique samples' is not defined, so the reader cannot determine whether balancing was performed by undersampling, oversampling, or another method. The entire balanced-versus-original discussion needs to be reworked with correct numbers and a precise description of the balancing protocol.","section":"Section VI-D"},{"comment":"The transparency questionnaire is designed by the authors, and the scores are assigned by the authors to their own framework, which was explicitly built to include the features the questionnaire rewards. No external validation, inter-rater reliability, or blinded assessment is reported. Even if the scoring formula is repaired, the central claim of 'more transparency and control' is not supported unless the assessment is independently audited or the paper explicitly frames the result as a self-assessment and discusses the associated bias risk.","section":"Section V-C and Section VI-A"}],"minor_comments":[{"comment":"Tool names contain typos: 'LinghtAutoML' should be 'LightAutoML', 'AutoGloun' should be 'AutoGluon', and 'MlJar' should be 'MLJar'.","section":"Figure 4"},{"comment":"The text contains an odd spacing in 'ANOV A', which should read 'ANOVA'.","section":"Section IV-A"},{"comment":"The text refers to a dataset named 'MH-100-4k', but Table VII lists 'MH-100K Real'; please align the naming.","section":"Section VI-B"},{"comment":"The sentence 'Auto-Sklearn achieves 91.41% recall compared to 99.19% in the original data' reports an original-data value that should be tied to a specific figure or table for verification.","section":"Section VI-D"},{"comment":"Reference [63] is listed as 'Anonymized for review' and is a placeholder that must be replaced before publication.","section":"References"},{"comment":"The phrase 'eight carefully selected datasets' conflicts with the nine datasets listed in Table VII and Section V-E; the count should be corrected.","section":"Section V-B"}],"recommendation":"major_revision","confidential_remarks":"The paper's headline distinction—transparency and control—is supported only by a self-authored rubric with a malformed formula and no external validation. The recall comparison is also under-powered and potentially confounded by the tuning objective. These problems are fixable, but they touch the central claims, so I recommend major revision rather than rejection. Please ensure the authors address the dataset-count, validation-method, and balancing-protocol inconsistencies in their revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper's headline claim—more transparency and control—is not supported by its own methods. The scoring formula in Section V-C defines S as a piecewise function of N alone (0 if N=0, 1 if N=1, 2 if N>=2) and never uses the per-question scores Pi or the maximum W. Table VI shows every category at 100, yet the text reports scores like 58.33 for MH-AutoML's Interpretability. The reported percentages cannot be recomputed from the stated procedure. The stress-test note is correct: the differentiator is effectively unfalsifiable as written.\n\nWhat is actually new: the framework and the benchmark. MH-AutoML is a real artifact—MLflow tracking, Optuna tuning, SHAP/LIME, LASSO-style feature selection, a VotingClassifier ensemble—and the code is on GitHub. The comparison covers nine Android malware datasets and seven general AutoML tools, which is broader than the authors' previous SBSeg'24 work. The recall results are competitive: MH-AutoML leads on six of nine original datasets, and the balanced-dataset analysis adds practical information.\n\nSoft spots beyond the scoring formula: Section V-B says eight datasets while Table VII lists nine; Table II claims 'Holdout + 5-fold CV' but only Holdout is described; there are no repeated runs or error bars; time limits are unstated; the balancing procedure is not described; and reference [63] is still 'Anonymized for review.' Also, MH-AutoML optimizes for recall while competitors run with defaults, so the recall advantage is partly baked in.\n\nThe transparency evaluation is the bigger problem. The five-dimension rubric is self-designed and self-administered, with no external raters or inter-rater reliability. Even with a corrected formula, the claim would need independent validation, or at minimum the complete questionnaire and per-tool answers. As it stands, the main differentiator collapses.\n\nWho should read this? Practitioners working on Android malware detection who want a hands-on tool and a map of how seven tools behave on these datasets. AutoML transparency researchers might find it a useful negative example. It deserves a serious referee: the artifact and comparison are genuine, and most flaws are fixable in revision. I would send it to peer review with a major-revision expectation, and I'd insist that the transparency score be reworked or removed, and the protocol fully specified.","headline":"Useful benchmark, broken transparency metric: the scoring formula in Section V-C cannot produce the reported percentages, so the paper's main differentiator is unsupported as written.","tokens_in":20136,"tokens_out":4726,"would_cite":false,"duration_ms":46976,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A domain-specific AutoML framework for Android malware out-recalls seven general-purpose tools while keeping the pipeline exposed.","keywords":["automated machine learning","Android malware detection","interpretability","transparency","feature selection","recall","MLOps","domain-specific AutoML"],"falsifier":"Ask independent analysts to apply a pre-registered transparency questionnaire to all eight tools under blinded conditions, and run all tools on a fresh Android malware dataset with default settings; if MH-AutoML no longer scores highest on transparency or no longer leads in recall, the paper's central claims fail.","tokens_in":19134,"feed_emoji":"🛡️","tokens_out":7985,"duration_ms":78890,"temperature":0.7,"pith_summary":"This paper sets out to show that a purpose-built AutoML framework for Android malware detection can be more transparent than general-purpose AutoML tools without sacrificing detection performance. It presents MH-AutoML, which automates the full pipeline and builds explanation, logging, and debugging into every stage. Comparing against seven established AutoML frameworks on nine Android malware datasets, the paper reports that MH-AutoML leads in recall on six of the nine original datasets and scores at or near the top of its own five-dimension transparency assessment. If these results hold, security teams could get both high detection rates and the ability to inspect and trust the models.","feed_headline":"AutoML for Android malware out-recalls general-purpose rivals","feed_subtitle":"A purpose-built pipeline stays inspectable while taking the recall lead on six of nine datasets.","key_machinery":"The load-bearing mechanism is the MH-AutoML pipeline itself: an automated sequence that exposes its decisions. It applies PCA, ANOVA, and LASSO for feature selection, combines six classifiers through a voting ensemble, tunes hyperparameters with recall as the prioritized objective, and attaches local explanation and experiment-tracking artifacts at each stage. The supporting evaluative device is a five-dimension questionnaire, normalized to a 0-100 scale, used to compare transparency and interpretability across tools. The pipeline's transparency is what the paper claims as its differentiator, and the questionnaire is what converts that claim into a reported score.","core_discovery":"The central claim is that MH-AutoML achieves better recall than seven general-purpose AutoML frameworks on Android malware detection while providing more transparency and control, with computational efficiency comparable to the other tools. The paper supports this by running all eight tools on nine datasets in both original and balanced forms, measuring recall, the Matthews correlation coefficient (MCC), and execution time, and by scoring each tool with a custom questionnaire on functional description, statistical analysis, algorithmic transparency, interpretability, and internal analysis. In the authors' evaluation, MH-AutoML leads the other tools in recall on six of the nine original datasets and receives the highest overall transparency and interpretability score. The intended upshot is that a domain-specific AutoML tool can close the explainability gap that black-box general-purpose tools leave open, without sacrificing detection quality in a security context.","pith_inferences":["The five-dimension scoring questionnaire is a reusable checklist, but its real weight depends on independent application; a natural next step is to have third parties score all tools blind.","The framework's recall-first tuning likely shifts the precision-recall trade-off; a deployment-oriented follow-up would report cost-sensitive thresholds or precision at matched recall levels for each dataset.","The same evaluation protocol (multiple datasets, balanced and imbalanced versions, plus a transparency rubric) could be transferred to other security domains such as network intrusion detection or phishing URL classification, where explainability also matters.","If the recall advantage persists across new Android malware families, it would suggest that feature-selection choices grounded in domain data, rather than generic optimization alone, are what close the gap with larger general-purpose systems."],"forward_implications":["Non-specialist security teams could deploy a malware detector that reaches high recall without hand-tuning a pipeline stage by stage.","Detection models built this way would come with the information needed to audit which features and models drove a prediction, supporting incident review and organizational accountability.","Because tuning optimizes recall, the framework is positioned for settings where missing a malicious sample is costlier than investigating a false alarm.","The comparison suggests that domain-specific AutoML does not necessarily trade performance for transparency: a tool built for one problem class can hold its own against general-purpose frameworks."],"supporting_citations":[{"why":"Baseline framework compared on recall, MCC, and execution time.","marker":"[57]"},{"why":"Baseline framework whose weighted-ensemble design is contrasted with MH-AutoML's approach.","marker":"[33]"},{"why":"Baseline framework contributing the genetic-programming comparison point.","marker":"[58]"},{"why":"Baseline framework supplying the fast-execution comparison point.","marker":"[59]"},{"why":"Baseline framework representing the deep-learning AutoML comparison point.","marker":"[60]"},{"why":"Baseline framework whose blending and early-stopping design is a key speed-accuracy rival.","marker":"[61]"},{"why":"Baseline framework serving as another general-purpose comparison point in the benchmark.","marker":"[62]"},{"why":"Grounds the transparency and interpretability questionnaire categories used to score all tools.","marker":"[66]"},{"why":"Supplies a model-agnostic explanation method assessed in the interpretability category.","marker":"[19]"},{"why":"Supplies a feature-attribution explanation method assessed in the interpretability category.","marker":"[20]"}],"fun_headline_variants":["MH-AutoML out-recalls general AutoML on Android malware","Domain-specific AutoML wins on recall, stays transparent for Android malware","Android malware AutoML: better recall, full transparency","MH-AutoML: transparent AutoML that out-recalls rivals for Android","Purpose-built AutoML out-recalls general tools for Android malware"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The transparency advantage rests on a five-dimension scoring rubric the authors designed and applied to their own tool; if that rubric is not a valid, impartial measure of interpretability, the main differentiator is not established.","fun_headline_variants_meta":{"raw":{"variants":["MH-AutoML out-recalls general AutoML on Android malware","Domain-specific AutoML wins on recall, stays transparent for Android malware","Android malware AutoML: better recall, full transparency","MH-AutoML: transparent AutoML that out-recalls rivals for Android","Purpose-built AutoML out-recalls general tools for Android malware"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002012,"raw_usage":{"total_tokens":7828,"prompt_tokens":911,"completion_tokens":6917,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":6828}},"tokens_in":527,"tokens_out":6917,"duration_ms":52654,"temperature":1.0,"reasoning_tokens":6828,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:45:51.073688+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask independent analysts to apply a pre-registered transparency questionnaire to all eight tools under blinded conditions, and run all tools on a fresh Android malware dataset with default settings; if MH-AutoML no longer scores highest on transparency or no longer leads in recall, the paper's central claims fail.","supporting_citations":[{"cited_title":"Efficient and robust automated machine learning,","cited_arxiv_id":null,"evidence_quote":"Baseline framework compared on recall, MCC, and execution time."},{"cited_title":"Scaling tree-based automated machine learning to biomedical big data with a feature set selector,","cited_arxiv_id":null,"evidence_quote":"Baseline framework contributing the genetic-programming comparison point."},{"cited_title":"HyperGBM: A Full Pipeline AutoML Tool Integrated With Various GBM Models,","cited_arxiv_id":null,"evidence_quote":"Baseline framework supplying the fast-execution comparison point."},{"cited_title":"Auto-pytorch: multi-fidelity metalearning for efficient and robust autodl,","cited_arxiv_id":null,"evidence_quote":"Baseline framework representing the deep-learning AutoML comparison point."},{"cited_title":"Mljar: State-of-the-art automated machine learning framework for tabular data,","cited_arxiv_id":null,"evidence_quote":"Baseline framework serving as another general-purpose comparison point in the benchmark."},{"cited_title":"Ex- plainable artificial intelligence (xai): Precepts, models, and opportunities for research in construction,","cited_arxiv_id":null,"evidence_quote":"Grounds the transparency and interpretability questionnaire categories used to score all tools."},{"cited_title":"” why should i trust you?","cited_arxiv_id":null,"evidence_quote":"Supplies a model-agnostic explanation method assessed in the interpretability category."},{"cited_title":"A unified approach to interpreting model predictions,","cited_arxiv_id":null,"evidence_quote":"Supplies a feature-attribution explanation method assessed in the interpretability category."}],"review_version":1}