Pith. sign in

REVIEW 2 major objections 5 minor 3 references

Moderately Mighty: To What Extent Can Internal Software Metrics Predict App Popularity at Launch?

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Internal software metrics carry a real but limited signal for whether an app will be popular at launch, with binary classifiers reaching macro F1 0.72 for ratings and 0.69 for downloads per year.

desk verdict Useful dataset and honest negative regressions, but the headline F1 numbers are inflated by target-aware feature selection on the full dataset; the central claim needs a re-analysis. read the letter →

arxiv 2507.02110 v3 pith:V4AM7JQZ submitted 2025-07-02 cs.SE

Md Nahidul Islam Opu , Fatima Islam Mouri , Rick Kazman , Yuanfang Cai , Shaiful Chowdhury This is my paper · ORCID
classification cs.SE
keywords apppopularitypredictioninternalsoftwaremetricscodesmellsAndroidappsF-DroidandGooglePlayfeatureselectionbinaryclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether software metrics that can be measured from source code before an app is released—complexity, coupling, cohesion, code smells, architecture quality, and basic metadata—can predict how popular the app will become. It reports that precise prediction is out of reach: regression models for user ratings and downloads per year explain almost none of the variance, with $R^2$ near zero or negative. But when the question is narrowed to a binary call, Popular versus Unpopular, the same internal metrics carry a moderate signal: a multilayer perceptron with SMOTE oversampling and an ensemble-selected feature set reaches macro F1 0.72 for ratings and 0.69 for downloads per year. The paper concludes that internal metrics are not sufficient for accurate popularity prediction but are far from useless, and that context—especially framing the task as classification rather than regression—determines whether they help.

What carries the argument

The load-bearing mechanism is the combination of ensemble feature selection and binary reframing. The Voting feature set is built by running six feature selection algorithms—two filters, one wrapper, and three embedded methods—and keeping features recommended by at least half of them; this shrinks 1,167 features to a handful (five for rating classification, five for download classification) and consistently beats both the size-only baseline and a 28-feature handpicked set. The second half of the mechanism is the Popular/Unpopular binarization: rating $\geq 3.5$ for ratings (with review-less apps treated as unpopular) and 2000 downloads per year for downloads, where the latter threshold splits the dataset almost in half. Classification metrics on these binary targets, evaluated with leave-one-out cross-validation, are what the paper's conclusion rests on.

What would settle it

Re-run the classification pipeline with the Voting feature selection performed inside each leave-one-out fold (or on the training fold only) and compare the macro F1; if the rating-classification F1 drops noticeably from 0.72, part of the reported signal is an artifact of target-aware feature selection.

Watch

Extended reading notes

Core claim

Using 446 open-source Java Android apps available on both F-Droid and Google Play, the authors extract 1,167 features from the earliest available version of each app: 17 system-level architecture metrics from the DV8 tool, class-level and method-level metrics aggregated to percentiles, 34 code smells, and metadata such as genre, permissions, ad support, and activity count. Across three feature sets—size-only, handpicked, and an ensemble Voting set—they find that regression on ratings and downloads per year performs poorly, with best $R^2$ values of 0.05 and 0.08, because the targets are skewed or scattered. Reframed as binary classification, the Voting feature set, an MLP classifier, and SMOTE yield macro F1 0.72 (MCC 0.45) for rating-based popularity and macro F1 0.69 (MCC 0.37) for download-based popularity. The central conclusion is that internal code metrics carry real, but moderate, predictive signal for coarse popularity classes, challenging studies that dismiss them entirely while agreeing that they do not tell the whole story.

Load-bearing premise

The results depend on the assumption that selecting the Voting features once on the full 446-app dataset does not leak popularity information into the cross-validation folds, so the held-out app's label never influences its own training set.

Editorial extensions

If this is right

  • Before launch, developers can treat internal metrics as an early-warning check: an app that looks structurally unhealthy is more likely to land in the Unpopular class, though many other factors intervene.
  • Size alone is a weak predictor; the ensemble-selected features, including code smells, architecture metrics, and metadata, outperform the size-only baseline by a wide margin on both tasks.
  • The failure of regression here is not evidence that metrics are useless—the same features show signal once the task is posed as coarse classification.
  • App store operators could plausibly incorporate such metric-based signals into ranking or quality-assurance workflows, which the paper explicitly suggests.
  • The F1 0.72 and 0.69 results are the benchmark any richer model—one adding marketing, UI, or developer reputation features—would need to beat.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: because the Voting feature set is chosen on the full 446-app dataset before leave-one-out cross-validation, the reported F1 scores should be treated as optimistic; a nested feature-selection evaluation would likely land lower.
  • Editorial extension: the same metric pipeline could be tested on other binary outcomes that matter at launch, such as crash-proneness or abandonment, where a moderate signal may be more actionable than popularity.
  • Editorial extension: the authors' own genre findings suggest genre-stratified models, but only 446 apps and hundreds of features make such models fragile; gathering more app-store pairs is the binding constraint.
  • Editorial extension: the moderate ceiling likely reflects the irreducible role of external factors, so the practical path is hybrid models that combine internal metrics with market signals, not more code metrics alone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper investigates whether internal software metrics extracted from the first available source-code version of an Android app can predict its later popularity, measured by average user rating and downloads per year. The authors construct a filtered dataset of 446 Java-based apps available on both F-Droid and Google Play, extract 1,167 features (system-, class-, and method-level metrics, code smells, metadata), and evaluate regression and binary classification models under three feature sets: Size-only, Handpicked, and a Voting set produced by six feature-selection algorithms. Regression results are poor, with R² values near zero or negative for both targets. When the task is reframed as binary classification (Popular vs. Unpopular), the best models achieve macro F1 of 0.72 for ratings (MLP with SMOTE on the Voting set, Section 4.1.2, Table 7) and 0.69 for downloads per year (MLP on the Voting set, Section 4.2.2, Table 10). The paper concludes that internal metrics alone are insufficient for exact prediction but carry meaningful correlation with popularity in a classification setting.

Significance. If the classification results are valid, the paper makes a useful empirical contribution: it provides a carefully constructed dataset, honestly reports negative regression results, and challenges overly dismissive conclusions about internal code metrics by showing a moderate but real predictive signal at launch time. The authors share their data and repository, use the oldest available F-Droid version to approximate inception, manually validate the F-Droid-to-Play mapping, and correctly perform resampling inside cross-validation folds. The significance is, however, conditional on the evaluation protocol being leakage-free; the headline F1 values currently rest on feature selection performed on the full dataset before cross-validation, which is a load-bearing threat to the central claim.

major comments (2)
  1. [§3.4, §4.1.2, §4.2.2] The Voting feature set is selected once on the full 446-app dataset using target-aware algorithms (Pearson correlation, chi-squared, SVC wrapper, logistic regression, random forest, LightGBM) before leave-one-out cross-validation is run. In each LOOCV fold, the held-out app's popularity label has therefore already influenced which features survive voting; with 1,167 candidate features and 446 samples, this selection leakage can materially inflate the reported macro F1 of 0.72 (Table 7) and 0.69 (Table 10). The paper reports no nested feature selection and no ablation showing that the F1 is preserved when feature selection is performed inside each training fold. Because the abstract's 'meaningful correlations' conclusion rests on these classification results, this is a load-bearing issue that must be addressed by re-running the evaluation with per-fold feature selection or an equivalent leakage-free protocol.
  2. [§4.2.2] The threshold of 2000 downloads per year used to define Popular versus Unpopular is read from the cumulative distribution function of the same data and splits the sample almost exactly in half (191 Popular vs. 190 Unpopular). This makes the binary target partly self-defined rather than externally anchored. The authors do not report sensitivity analyses over alternative thresholds or justify the threshold a priori. Since RQ2's classification result is one of the two pillars of the paper's positive conclusion, the target definition should be shown to be robust to reasonable threshold choices or grounded in an independent criterion.
minor comments (5)
  1. [§4 (intro)] The introductory sentence says 'three research questions,' but the paper presents only RQ1 and RQ2; please correct the count.
  2. [Table 9] The MLP row for DownloadsPerYear regression is formatted as '15549.398320.82-0.31' with missing spaces, which is difficult to read; please fix the table formatting.
  3. [Figure 7] The AUC-ROC plot labels the positive class as '1' while the confusion matrix uses 'Popular' and 'Unpopular'; please clarify which class is treated as positive.
  4. [§4.1.2] For the rating-based classification, apps with no user reviews are assigned to the Unpopular class. This is a reasonable operational choice, but it would be helpful to state explicitly how many of the 119 Unpopular apps have zero reviews, since the absence of reviews is a very different signal from a low average rating.
  5. [§3.4] The voting threshold is described as 'at least 50% of the algorithms,' which with six algorithms means at least three; this is clear, but the number n=25 for each algorithm's ranked list is stated without justification, so a brief rationale or sensitivity check would strengthen the presentation.

Circularity Check

1 steps flagged · score 6.0 of 10

Voting feature selection is run on the full dataset before LOOCV, so the headline F1 0.72/0.69 is partly fitted to the target labels; the strongest 'prediction' claim is not leakage-free.

  1. fitted input called prediction [Section 3.4 (Voting feature selection) applied before LOOCV in Sections 4.1.2 and 4.2.2]
    "For classification, the algorithms were categorized into three types: filter methods, which included Pearson Correlation and Chi2 Correlation; wrapper methods, represented by the Support Vector Classifier; and embedded methods, consisting of Logistic Regression, Random Forest Classifier, and Light Gradient Boosting Classifier. ... The Voting method highlighted five key features: containsAds, innerClassesQty 50 class, genreId TRAVEL AND LOCAL, parametersQty 20, and parametersQty 99."

    Pearson correlation, chi-squared, and the SVC wrapper are target-aware: they rank the 1,167 features by association with the Popular/Unpopular label. Section 3.4 runs them on the full 446-app dataset and keeps features voted by at least 3 of 6 algorithms. Section 4.1.2 then applies LOOCV only after those features are fixed. In every LOOCV fold, the held-out app's label has already contributed to selecting the Voting features, so the reported macro F1 0.72 (and similarly 0.69 for downloads in 4.2.2) is not an independent out-of-sample prediction from internal metrics; the feature set itself is fitted to the target. No nested feature selection or within-fold selection is described.

full rationale

The paper is an empirical study with no formal derivation, so most circularity categories do not apply. The one concrete circular step is the Voting feature-selection protocol: target-aware selectors run on the full dataset before LOOCV, so the headline classification F1 is partly fitted to the labels it claims to predict. The Handpicked and Size-only results are not affected by this leakage, which is why the circularity is partial (score 6) rather than total. The RQ2 download threshold (2000/year) is chosen from the same data's CDF and is a construct-validity concern, but it is not a circular derivation because the threshold does not depend on the internal-metric features. Self-citations (Chowdhury et al. 2022/2024/2025, Cai and Kazman 2023) are used for motivation and tool background, not as the load-bearing evidence for the paper's conclusion, so they do not add circularity.

Assumptions & free parameters 9 free parameters · 5 assumptions · 0 invented entities

The central claim rests on dataset-construction choices (Java-only, F-Droid/Play availability, oldest-version proxy), target-label definitions (rating threshold 3.5, download threshold 2000 chosen from the data), and an ML pipeline that selects the Voting features on the full dataset before LOOCV. No new physical or theoretical entities are introduced. The free parameters are hand-set thresholds and aggregation choices rather than fitted model coefficients.

free parameters (9)
  • Rating popularity threshold = 3.5 stars
    Apps with mean rating >= 3.5 labeled Popular; taken from prior studies (Khalid et al., 2015; Catolino, 2018), not fitted to data, but it determines the target variable.
  • Downloads-per-year threshold = 2000 downloads/year
    Chosen from the CDF in Figure 9 to split the dataset almost in half (191 vs 190 apps), making the classification target data-defined and balanced.
  • Feature-selection vote threshold = at least 50% of 6 algorithms (3 of 6)
    Features must be recommended by at least half of the six selection algorithms; this hand-set threshold controls the size and composition of the Voting set.
  • Per-algorithm feature count n = 25
    Each feature selection algorithm returns its top 25 features; chosen by hand.
  • Percentile aggregation set = 10th, 20th, ..., 90th, 95th, 99th
    Eleven percentiles chosen to summarize class/method metric distributions; hand-set.
  • Java code proportion filter = >= 50%
    Apps with less than 50% Java excluded to ensure metric validity.
  • Minimum normal classes = 5
    Apps with fewer than five normal classes excluded for statistical reliability.
  • App age cutoff = 1 year
    Apps younger than one year discarded to enable age normalization.
  • IQR outlier removal for downloads = IQR method, 65 apps removed
    Outliers removed before download regression, potentially affecting the target distribution and later classification threshold.
assumptions (5)
  • domain assumption package_name is a unique identifier across F-Droid and Google Play, and manual validation of 30 apps suffices to confirm app identity
    Used to map apps between repositories (Section 3.1); if mapping is wrong, the entire feature-label pairing fails.
  • domain assumption The oldest archived F-Droid version approximates the app's first release
    Explicitly acknowledged in Section 5.1 as a threat; the whole inception framing depends on this proxy.
  • ad hoc to paper Global feature selection before LOOCV does not cause target leakage
    The Voting feature set is selected on the full dataset using target-aware algorithms before cross-validation (Sections 3.4 and 4.1.2); if leakage occurs, reported F1 scores are inflated.
  • domain assumption Aggregating class/method metrics via percentiles preserves predictive signal
    The paper compresses variable-size metric sets into fixed-length percentile summaries (Section 3.3); loss of distributional detail is accepted without sensitivity analysis.
  • domain assumption Code metrics from source code are stable and tool outputs are reliable
    The study trusts DV8, CK, DesigniteJava, cloc, and readability scores as accurate measurements of the intended constructs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Moderately Mighty: To What Extent Can Internal Software Metrics Predict App Popularity at Launch?." pith.science (2026). https://pith.science/paper/V4AM7JQZ

@misc{pith2026250702110,
  author       = {Pith},
  title        = {Pith review of: Moderately Mighty: To What Extent Can Internal Software Metrics Predict App Popularity at Launch?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V4AM7JQZ}},
  note         = {Machine review of arXiv:2507.02110}
}
read the original abstract

Predicting a mobile app's popularity before its first release can provide developers with a strategic advantage in a competitive marketplace, yet it remains a challenging problem. This study explores the extent to which internal software metrics, measurable from source code before deployment, can predict an app's popularity (i.e., ratings and downloads per year) at inception. For our analysis, we constructed a rigorously filtered dataset of 446 open-source Java-based Android apps that are available on both F-Droid and Google Play Store. Using app source code from F-Droid, we extracted a wide array of internal metrics, including system-, class-, and method-level code metrics, code smells, and app metadata. Popularity-related information, including reviews and download counts, was collected from the Play Store. We evaluate regression and classification models across three feature sets: a minimal Size-only baseline, a domain-informed Handpicked set, and a Voting set derived via feature selection algorithms. Our results show that, for both app ratings and number of downloads, regression models perform poorly due to skewed rating distributions and a highly scattered range of download counts in our dataset. However, when reframed as a binary classification (Popular vs. Unpopular), performance improves significantly-the best model, a Multilayer Perceptron, achieves an F1-score of 0.72. We conclude that, although internal code metrics alone are insufficient for accurately predicting an app's future popularity, they do exhibit meaningful correlations with it. Thus, our findings challenge prior studies that have entirely dismissed internal metrics as valid indicators of software quality. Instead, our results align with research suggesting that internal code metrics can be valuable when evaluated within the appropriate context-specifically, we found them useful for classification tasks.

Figures

Figures reproduced from arXiv: 2507.02110 by the authors.

Figure 1
Figure 1. Overview of methodology that involves app collection (with source code and [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. Cumulative distribution functions for (a) the size (LOC) of the applications in [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Cumulative Distribution Functions (CDFs) of actual and predicted ratings. [PITH_FULL_IMAGE:figures/full_fig_p022_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Approximately 90% of the absolute prediction errors fall below 1.0, aligning [PITH_FULL_IMAGE:figures/full_fig_p023_4.png]
Figure 5
Figure 5. Figure 5: Scatter plot of true ratings and predicted ratings for each app. [PITH_FULL_IMAGE:figures/full_fig_p024_5.png]
Figure 6
Figure 6. Figure 6: Scatter Plot of True Ratings vs Predicted Ratings. [PITH_FULL_IMAGE:figures/full_fig_p024_6.png]
Figure 7
Figure 7. Figure 7: Confusion matrix and AUC-ROC curve for rating-based popularity prediction [PITH_FULL_IMAGE:figures/full_fig_p027_7.png]
Figure 8
Figure 8. Figure 8: CDF of DownloadsPerYear. We apply the Interquartile Range (IQR) method for outlier detection to remove the outliers from the dataset. This approach filters out 65 apps, leaving us with 381 apps for the analysis, with DownloadsPerYear. The CDF of DownloadsPerYear after …
Figure 9
Figure 9. Figure 9: CDF of DownloadsPerYear after removing outliers [PITH_FULL_IMAGE:figures/full_fig_p028_9.png]
Figure 10
Figure 10. Figure 10: CDF of absolute error for the Ridge regression model in predicting [PITH_FULL_IMAGE:figures/full_fig_p029_10.png]
Figure 11
Figure 11. Figure 11: Scatter plot of true DownloadsPerYear vs predicted DownloadsPerYear. In the previous research question, internal software metrics proved ineffective for regression due to the skewed distribution of app ratings. However, in this research question for DownloadsPerYear, …
Figure 12
Figure 12. Figure 12: Confusion matrix and AUC-ROC curve for DownloadsPerYear-based pop￾ularity prediction using MLP classifier. 5 Discussion Our empirical investigation into the predictive power of internal software metrics for application popularity reveals both promising insights and no…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

3 extracted references · 2 canonical work pages

  1. [1]

    In: 2011 15th European Conference on Software Maintenance and Reengineering, pp 181–190, DOI 10.1109/CSMR.2011.24, URLhttps://ieeexplore.ieee

    Abbes M, Khomh F, Gu ´eh´eneuc YG, Antoniol G (2011) An Empirical Study of the Impact of Two Antipatterns, Blob and Spaghetti Code, on Program Comprehension. In: 2011 15th European Conference on Software Maintenance and Reengineering, pp 181–190, DOI 10.1109/CSMR.2011.24, URLhttps://ieeexplore.ieee. org/document/5741260, iSSN: 1534-5351 Moderately Mighty:...

    work page arXiv 2011

  2. [10]

    Journal of Machine Learning Research 12:2825–2830 Rachow P, Riebisch M (2022) An architecture smell knowledge base for managing architecture technical debt

    1016/j.jss.2019.110493, URLhttps://www.sciencedirect.com/science/ article/pii/S0164121219302675 Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay E (2011) Scikit-learn: Machine learning in Python. Journal of Machine Learni...

    work page arXiv 2011

  3. [2023]

    URLhttps:// sonarsource.navattic.com/kdq70t3x Spadini D, Palomba F, Zaidman A, Bruntink M, Bacchelli A (2018) On the Relation of Test Smells to Software Code Quality. In: 2018 IEEE Interna- tional Conference on Software Maintenance and Evolution (ICSME), pp 1– 12, DOI 10.1109/ICSME.2018.00010, URLhttps://ieeexplore.ieee.org/ document/8529832, iSSN: 2576-3...

    arXiv 2018

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.