{"id":"cf710a7a-7298-43c6-a194-f3aacd4718d2","arxiv_id":"2507.02110","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"Internal code metrics predict app popularity only as a binary popular/unpopular classification, reaching macro F1 0.72 for ratings and 0.69 for downloads per year, while regression predictions are no better than the mean.","lead":"This study tests whether software metrics measured from an app's earliest available source code can predict its later popularity on Google Play. It finds that regression models fail, but a neural network that classifies apps as popular or unpopular reaches F1 scores of 0.72 for ratings and 0.69 for downloads per year.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Voting feature selection is run on the full dataset before LOOCV, so the headline F1 0.72/0.69 may be inflated by label leakage; a nested-selection re-run is needed.","rationale":"Both the abstract and the RQ summaries hinge on the Voting-set classification metrics; the regression results are explicitly weak. The necessary condition for those metrics to support 'meaningful correlations' is that the held-out app's target is not used to construct the features. Section 3.4 applies selection to the full dataset, and Section 3.5 then runs LOOCV, so the paper's evaluation violates that condition for the only feature set that produces the headline numbers. This is an internal flaw, not a disagreement with field consensus: the same leakage logic is what the paper itself uses to criticize Catolino's version-based 10-fold setup. I agree with the reader's weakest assumption. The secondary threshold concern (2000 downloads/year chosen from the data CDF) affects construct validity, but the ratings result already carries the feature-selection problem, so the verdict does not depend on it. Credit is due for the transparent regression analysis, the F-Droid/Play Store mapping effort, the public data, and the explicit threats section; those are real strengths. However, they do not establish the headline positive claim if the nested-selection check fails. I would keep the reader's REJECT; no adjustment.","tokens_in":30705,"tokens_out":6505,"duration_ms":73222,"concrete_test":"Using the public repository, rerun the RQ1 and RQ2 classification pipelines for the Voting feature set, but move feature selection inside each LOOCV fold: for each held-out app, run Pearson, chi-square, SVC-wrapper, logistic-regression, random-forest, and LightGBM selection on the 445 training apps only, take the majority-vote features (at least 3 of 6), train the MLP (with SMOTE applied inside the fold for RQ1), and predict the held-out app. Compare nested macro F1, MCC, and AUC against Tables 7 and 10, and against a majority-class macro-F1 baseline. If nested-selection F1 drops by more than about 0.05 or approaches the baseline, the headline signal is largely a feature-selection artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on the classification results for the Voting feature set: macro F1 0.72 for ratings (MLP+SMOTE, Table 7) and 0.69 for downloads (MLP, Table 10). Those results are obtained after the Voting set is selected once on the full 446-app dataset using target-aware algorithms (Pearson correlation, chi-square, SVC wrapper, logistic regression, random forest, LightGBM; Section 3.4), with LOOCV applied only afterward. In each LOOCV fold, the held-out app's label has therefore already influenced which features survive voting. With 1,167 candidate features and 446 samples, this can inflate apparent generalization; the paper reports no nested feature selection and no ablation showing that the F1 survives selection inside the loop. The download threshold of 2000/year is also read from the data's CDF (Section 4.2.2), making that binary target partly self-defined; the ratings threshold is externally motivated. As reported, the positive 'meaningful correlations' conclusion is not supported by a leakage-free evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether internal software metrics extracted from the first available source-code version of an Android app can predict its later popularity, measured by average user rating and downloads per year. The authors construct a filtered dataset of 446 Java-based apps available on both F-Droid and Google Play, extract 1,167 features (system-, class-, and method-level metrics, code smells, metadata), and evaluate regression and binary classification models under three feature sets: Size-only, Handpicked, and a Voting set produced by six feature-selection algorithms. Regression results are poor, with R² values near zero or negative for both targets. When the task is reframed as binary classification (Popular vs. Unpopular), the best models achieve macro F1 of 0.72 for ratings (MLP with SMOTE on the Voting set, Section 4.1.2, Table 7) and 0.69 for downloads per year (MLP on the Voting set, Section 4.2.2, Table 10). The paper concludes that internal metrics alone are insufficient for exact prediction but carry meaningful correlation with popularity in a classification setting.","tokens_in":30903,"tokens_out":3816,"duration_ms":46673,"significance":"If the classification results are valid, the paper makes a useful empirical contribution: it provides a carefully constructed dataset, honestly reports negative regression results, and challenges overly dismissive conclusions about internal code metrics by showing a moderate but real predictive signal at launch time. The authors share their data and repository, use the oldest available F-Droid version to approximate inception, manually validate the F-Droid-to-Play mapping, and correctly perform resampling inside cross-validation folds. The significance is, however, conditional on the evaluation protocol being leakage-free; the headline F1 values currently rest on feature selection performed on the full dataset before cross-validation, which is a load-bearing threat to the central claim.","major_comments":[{"comment":"The Voting feature set is selected once on the full 446-app dataset using target-aware algorithms (Pearson correlation, chi-squared, SVC wrapper, logistic regression, random forest, LightGBM) before leave-one-out cross-validation is run. In each LOOCV fold, the held-out app's popularity label has therefore already influenced which features survive voting; with 1,167 candidate features and 446 samples, this selection leakage can materially inflate the reported macro F1 of 0.72 (Table 7) and 0.69 (Table 10). The paper reports no nested feature selection and no ablation showing that the F1 is preserved when feature selection is performed inside each training fold. Because the abstract's 'meaningful correlations' conclusion rests on these classification results, this is a load-bearing issue that must be addressed by re-running the evaluation with per-fold feature selection or an equivalent leakage-free protocol.","section":"§3.4, §4.1.2, §4.2.2"},{"comment":"The threshold of 2000 downloads per year used to define Popular versus Unpopular is read from the cumulative distribution function of the same data and splits the sample almost exactly in half (191 Popular vs. 190 Unpopular). This makes the binary target partly self-defined rather than externally anchored. The authors do not report sensitivity analyses over alternative thresholds or justify the threshold a priori. Since RQ2's classification result is one of the two pillars of the paper's positive conclusion, the target definition should be shown to be robust to reasonable threshold choices or grounded in an independent criterion.","section":"§4.2.2"}],"minor_comments":[{"comment":"The introductory sentence says 'three research questions,' but the paper presents only RQ1 and RQ2; please correct the count.","section":"§4 (intro)"},{"comment":"The MLP row for DownloadsPerYear regression is formatted as '15549.398320.82-0.31' with missing spaces, which is difficult to read; please fix the table formatting.","section":"Table 9"},{"comment":"The AUC-ROC plot labels the positive class as '1' while the confusion matrix uses 'Popular' and 'Unpopular'; please clarify which class is treated as positive.","section":"Figure 7"},{"comment":"For the rating-based classification, apps with no user reviews are assigned to the Unpopular class. This is a reasonable operational choice, but it would be helpful to state explicitly how many of the 119 Unpopular apps have zero reviews, since the absence of reviews is a very different signal from a low average rating.","section":"§4.1.2"},{"comment":"The voting threshold is described as 'at least 50% of the algorithms,' which with six algorithms means at least three; this is clear, but the number n=25 for each algorithm's ranked list is stated without justification, so a brief rationale or sensitivity check would strengthen the presentation.","section":"§3.4"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the global feature selection before LOOCV. This is not merely a framing concern: the headline F1 values are produced by a protocol that allows test labels to influence feature choice. A nested re-analysis is feasible within the manuscript's scope and would determine whether the 'meaningful correlations' conclusion survives. I would not recommend acceptance without that re-analysis, but I also do not see the current state as unrecoverable, hence major_revision rather than reject."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honest read: this paper has a genuinely useful new dataset and a cleanly reported set of negative regression results, but the headline classification numbers should not be taken at face value. The Voting feature set is selected once on the full 446-app dataset using target-aware methods (Pearson, chi-square, SVC wrapper, etc.) before LOOCV is run. That means in every fold the held-out app's label has already influenced which features survive. With 1,167 candidate features and 446 samples, this is real leakage, not a rounding error. The stress-test note is right, and reading the paper confirms no nested feature selection or ablation. So the F1 0.72 (ratings) and 0.69 (downloads) cannot be read as leakage-free evidence that internal metrics carry a launch-time popularity signal.\n\nWhat the paper does well: the dataset construction is careful. They map F-Droid source to Play Store metadata by package name, manually verify 30 mappings, filter to Java-only apps with at least 50% Java, drop apps younger than one year, and use the oldest available F-Droid version to approximate first release. That is a real step up from decompiled-APK datasets like Catolino's, and they avoid version leakage by evaluating at the app level. The regression analysis is honestly reported: R2 near zero, mean-prediction behavior, skewed rating distribution, and they do not oversell it. The threats section is transparent.\n\nSoft spots, in proportion. The leakage is load-bearing because the positive claim rests on the Voting set. It also matters that the Voting set is not purely internal code metrics: the selected features include genreId and containsAds, so part of the signal is metadata, not code. The title and abstract emphasize internal software metrics; that framing needs adjustment. The download threshold of 2000/year is read from the data's CDF to split the sample almost in half, making that binary target partly self-defined. The ratings threshold of 3.5 is externally motivated, which is better. Finally, the 'first release' claim is approximate because the oldest archived version is not guaranteed to be first, but they acknowledge this in the threats section.\n\nBottom line: this deserves a serious referee. The dataset and the negative regression results are worth publishing, and the classification claim can be fixed with nested feature selection, an external or predefined download threshold, and ablations that separate metadata from code metrics. I would send it to review with major revision, and I would bring it to a reading group as a clean case study of selection leakage.","headline":"Useful dataset and honest negative regressions, but the headline F1 numbers are inflated by target-aware feature selection on the full dataset; the central claim needs a re-analysis.","tokens_in":31486,"tokens_out":3184,"would_cite":false,"duration_ms":37050,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Internal software metrics carry a real but limited signal for whether an app will be popular at launch, with binary classifiers reaching macro F1 0.72 for ratings and 0.69 for downloads per year.","keywords":["app popularity prediction","internal software metrics","code metrics","code smells","Android apps","F-Droid and Google Play","feature selection","binary classification"],"falsifier":"Re-run the classification pipeline with the Voting feature selection performed inside each leave-one-out fold (or on the training fold only) and compare the macro F1; if the rating-classification F1 drops noticeably from 0.72, part of the reported signal is an artifact of target-aware feature selection.","tokens_in":30466,"feed_emoji":"📱","tokens_out":4544,"duration_ms":48005,"temperature":0.7,"pith_summary":"This paper asks whether software metrics that can be measured from source code before an app is released—complexity, coupling, cohesion, code smells, architecture quality, and basic metadata—can predict how popular the app will become. It reports that precise prediction is out of reach: regression models for user ratings and downloads per year explain almost none of the variance, with $R^2$ near zero or negative. But when the question is narrowed to a binary call, Popular versus Unpopular, the same internal metrics carry a moderate signal: a multilayer perceptron with SMOTE oversampling and an ensemble-selected feature set reaches macro F1 0.72 for ratings and 0.69 for downloads per year. The paper concludes that internal metrics are not sufficient for accurate popularity prediction but are far from useless, and that context—especially framing the task as classification rather than regression—determines whether they help.","feed_headline":"Code metrics give a moderate read on app popularity","feed_subtitle":"Classifying apps as popular or unpopular hits 0.72 F1 using only pre-launch code metrics.","key_machinery":"The load-bearing mechanism is the combination of ensemble feature selection and binary reframing. The Voting feature set is built by running six feature selection algorithms—two filters, one wrapper, and three embedded methods—and keeping features recommended by at least half of them; this shrinks 1,167 features to a handful (five for rating classification, five for download classification) and consistently beats both the size-only baseline and a 28-feature handpicked set. The second half of the mechanism is the Popular/Unpopular binarization: rating $\\geq 3.5$ for ratings (with review-less apps treated as unpopular) and 2000 downloads per year for downloads, where the latter threshold splits the dataset almost in half. Classification metrics on these binary targets, evaluated with leave-one-out cross-validation, are what the paper's conclusion rests on.","core_discovery":"Using 446 open-source Java Android apps available on both F-Droid and Google Play, the authors extract 1,167 features from the earliest available version of each app: 17 system-level architecture metrics from the DV8 tool, class-level and method-level metrics aggregated to percentiles, 34 code smells, and metadata such as genre, permissions, ad support, and activity count. Across three feature sets—size-only, handpicked, and an ensemble Voting set—they find that regression on ratings and downloads per year performs poorly, with best $R^2$ values of 0.05 and 0.08, because the targets are skewed or scattered. Reframed as binary classification, the Voting feature set, an MLP classifier, and SMOTE yield macro F1 0.72 (MCC 0.45) for rating-based popularity and macro F1 0.69 (MCC 0.37) for download-based popularity. The central conclusion is that internal code metrics carry real, but moderate, predictive signal for coarse popularity classes, challenging studies that dismiss them entirely while agreeing that they do not tell the whole story.","pith_inferences":["Editorial extension: because the Voting feature set is chosen on the full 446-app dataset before leave-one-out cross-validation, the reported F1 scores should be treated as optimistic; a nested feature-selection evaluation would likely land lower.","Editorial extension: the same metric pipeline could be tested on other binary outcomes that matter at launch, such as crash-proneness or abandonment, where a moderate signal may be more actionable than popularity.","Editorial extension: the authors' own genre findings suggest genre-stratified models, but only 446 apps and hundreds of features make such models fragile; gathering more app-store pairs is the binding constraint.","Editorial extension: the moderate ceiling likely reflects the irreducible role of external factors, so the practical path is hybrid models that combine internal metrics with market signals, not more code metrics alone."],"forward_implications":["Before launch, developers can treat internal metrics as an early-warning check: an app that looks structurally unhealthy is more likely to land in the Unpopular class, though many other factors intervene.","Size alone is a weak predictor; the ensemble-selected features, including code smells, architecture metrics, and metadata, outperform the size-only baseline by a wide margin on both tasks.","The failure of regression here is not evidence that metrics are useless—the same features show signal once the task is posed as coarse classification.","App store operators could plausibly incorporate such metric-based signals into ranking or quality-assurance workflows, which the paper explicitly suggests.","The F1 0.72 and 0.69 results are the benchmark any richer model—one adding marketing, UI, or developer reputation features—would need to beat."],"supporting_citations":[{"why":"The closest prior study whose dataset and cross-validation flaws this work explicitly avoids.","marker":"(Catolino, 2018)"},{"why":"Supplies the F-Droid/Play Store dataset-building approach and the decompiled-APK methodology the paper rejects in favor of original source code.","marker":"(Grano et al., 2017)"},{"why":"Defines the C&K class-level metrics that form a core feature family.","marker":"(Chidamber and Kemerer, 1994)"},{"why":"Introduces Decoupling Level, the system-level architecture metric used via DV8.","marker":"(Mo et al., 2016a)"},{"why":"Provides the readability score used as a method-level feature.","marker":"(Buse and Weimer, 2010)"},{"why":"SMOTE, the oversampling technique that produces the best rating classifier.","marker":"(Chawla et al., 2002)"},{"why":"Motivates the ensemble feature-selection strategy behind the Voting set.","marker":"(Li et al., 2017)"},{"why":"Documents external factors such as size and marketing that motivate why internal metrics alone cannot fully predict popularity.","marker":"(Tian et al., 2015)"}],"fun_headline_variants":["Code metrics: moderate popularity signal at app launch","Internal code metrics classify app popularity moderately","Pre-launch code metrics predict popularity coarsely","App popularity: code metrics are modest predictors","Moderate might: code metrics predict popularity class"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results depend on the assumption that selecting the Voting features once on the full 446-app dataset does not leak popularity information into the cross-validation folds, so the held-out app's label never influences its own training set.","fun_headline_variants_meta":{"raw":{"variants":["Code metrics: moderate popularity signal at app launch","Internal code metrics classify app popularity moderately","Pre-launch code metrics predict popularity coarsely","App popularity: code metrics are modest predictors","Moderate might: code metrics predict popularity class"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000276,"raw_usage":{"total_tokens":1724,"prompt_tokens":1099,"completion_tokens":625,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":715,"completion_tokens_details":{"reasoning_tokens":557}},"tokens_in":715,"tokens_out":625,"duration_ms":7149,"temperature":1.0,"reasoning_tokens":557,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:37:43.252912+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the classification pipeline with the Voting feature selection performed inside each leave-one-out fold (or on the training fold only) and compare the macro F1; if the rating-classification F1 drops noticeably from 0.72, part of the reported signal is an artifact of target-aware feature selection.","supporting_citations":[],"review_version":1}