{"id":"5dfd36cb-93ee-4f35-b9bf-e09210b5bcd6","arxiv_id":"1908.05376","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper extends mRMR feature selection with non-linear redundancy and model-based relevance measures, evaluates them on Uber marketing data, and selects FCQ for production deployment.","lead":"This paper evaluates eight feature selection methods, including three new variants of the mRMR framework, on marketing classification data at Uber. It finds that the FCQ variant, using an F-test for relevance and correlation for redundancy, performs well and fast, and describes its production deployment in an automated machine learning platform.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Real-data evidence for FCQ's 'robust' claim lacks error bars; Table V differences as small as 0.001–0.005 AUC could be within 4-fold CV noise.","rationale":"The reader flagged representativeness and the online experiment. I agree with the direction but would sharpen it: the core problem is not only external validity—it is that the real-data ranking itself has no quantified uncertainty. The synthetic section includes a 100-trial significance analysis, so the authors know how to report variability; the omission for Table V is conspicuous. Given that the production decision was made on these real-data rankings, the missing error bars are load-bearing. I also considered the possibility that Equations (6)-(7) use raw Pearson correlation, which would make FCQ's denominator unstable near zero and would fail to penalize negatively correlated redundancy; this is worth a code check, but the more direct threat to the stated claim is the absence of uncertainty in the decisive empirical comparison. With the proposed repeated-CV confidence intervals, the concern can be settled without new data.","tokens_in":12855,"tokens_out":9776,"duration_ms":104894,"concrete_test":"Re-run the Section IV.C real-data evaluation with repeated 4-fold cross-validation (e.g., 20 repeats). For each classifier and feature count (10, 20, 30), compute FCQ minus each competitor (RFCQ, RFRQ, RF, etc.) with a paired bootstrap 95% CI or a paired permutation test across folds/repeats. If the intervals for FCQ versus the best competitor exclude a meaningful AUC drop (say > 0.005) and are stable across datasets, the robustness claim stands; otherwise the conclusion should be softened to 'competitive on these datasets pending replication'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the conclusion that FCQ is robust across classification models and efficient. The efficiency part is supported by Table III. The robustness part rests on synthetic results and on Table V, where each real-data entry is a single averaged AUC over a 4-fold cross-validation, with no standard deviation, confidence interval, or paired comparison. Several comparisons that distinguish FCQ from alternatives are extremely small: in Dataset 3, for Logistic Regression at 30 features, FCQ scores 0.812 and at least four other methods also score 0.812; in Dataset 1, for Random Forest at 30 features, FCQ is 0.715 while RFCQ and RF are 0.725 and 0.727. With 4 folds, the standard error of an AUC average is typically several times 10^-3, so the observed ordering of methods within a column—and the claim that FCQ is consistently near-optimal—can be fold-dependent. Section IV.A says metrics are 'averaged over the iterations' but never reports dispersion; the online experiment in Section V.D validates the deployed model against a baseline, not FCQ against an alternative feature selector. Thus the load-bearing condition 'FCQ is robust' is not empirically pinned down by the real-data evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies feature selection for classification problems in Uber's marketing machine learning platform. It proposes two extensions to the minimum Redundancy Maximum Relevance (mRMR) framework: using the randomized dependence coefficient (RDC) as a nonlinear redundancy measure, giving the FRQ method (Eq. 8), and using random-forest feature importance as the relevance measure, giving the RFCQ and RFRQ methods (Eqs. 9-10). The paper then evaluates seven mRMR variants plus RF importance and an all-features baseline on one synthetic dataset and three real Uber marketing datasets, using Naive Bayes, logistic regression, and random forests as downstream classifiers and reporting AUC, F1 (for synthetic data), and computation time. Based on the evaluation, FCQ is selected for production implementation in a Scala/Spark pipeline, and an online experiment for a cross-sell model is described.","tokens_in":13042,"tokens_out":6884,"duration_ms":71996,"significance":"The paper's applied contribution is valuable: the proposed variants are defined explicitly in Eqs. (8)-(10), the synthetic study includes a 100-trial comparison relative to RF-based selection, and the production implementation in Section V gives unusual detail about how feature selection is embedded in an AutoML pipeline. If the performance claims are supported, FCQ would be a practical default choice for marketing classification models. The main weakness is empirical: the real-data evidence for the central conclusion of FCQ's robustness lacks uncertainty quantification. The paper does not ship proofs or reproducible code, but its explicit method definitions and the breadth of the evaluation make it a useful case study for practitioners, provided the claims are tightened.","major_comments":[{"comment":"The real-data results are reported only as AUC values averaged over the four cross-validation folds, with no standard deviations, confidence intervals, or paired comparisons. Several discriminating comparisons are extremely small; for example, in Dataset 3 with Logistic Regression at 30 features, FCQ reaches 0.812 but at least four other methods also reach 0.812, and in Dataset 1 with Random Forest at 30 features, FCQ is 0.715 while RFCQ and RF are 0.725 and 0.727, respectively. With only four folds, the standard error of an averaged AUC is typically on the order of 10^-3 or larger, so the observed ordering of methods within a column can be fold-dependent. The conclusion in Section VI that FCQ shows \"robust performance for different classification models\" is therefore not empirically pinned down by the real-data evidence; please add measures of uncertainty and paired tests, or substantially weaken the claim.","section":"Section IV.A (Step 5) and Table V"},{"comment":"The online experiment demonstrates that a model trained with FCQ-selected features identifies high-propensity users and generates incremental adoption relative to an untargeted baseline, but it does not compare FCQ with any alternative feature selection method. As presented, this experiment is evidence of business impact for the deployed model, not evidence that FCQ is superior to other selectors; the text should say so explicitly.","section":"Section V.D"},{"comment":"The only repeated-trial statistical comparison, the 100-trial synthetic study, uses the Random Forest classifier exclusively. Since the central robustness claim concerns multiple downstream model families, the repeated-trial evidence should either include Naive Bayes and Logistic Regression, or the conclusion should be restricted to tree-based models.","section":"Figure 4 and Section IV.A"}],"minor_comments":[{"comment":"The displayed formula appears to have an unmatched parenthesis; the expression for the logistic probability is missing a closing parenthesis.","section":"Eq. (11)"},{"comment":"The \"Number of Users\" column appears to render \"105\" and \"106\" where superscripts 10^5 and 10^6 are intended; please use proper superscript formatting.","section":"Table IV"},{"comment":"The text states that F1-score patterns for the real data are consistent with AUC patterns, but no real-data F1 results are shown; including a summary or plot would make this statement checkable.","section":"Section IV.C"},{"comment":"The RDC implementation is not fully specified: the number of random projections, the choice of nonlinear features, and any regularization parameters would be needed to reproduce the FRQ and RFRQ results.","section":"Section II.A / Eq. (8)"},{"comment":"The computation-time results are reported as single numbers without stating whether they are one-off measurements or averages, and without specifying the compute environment; a brief methodology sentence would help.","section":"Table III"}],"recommendation":"major_revision","confidential_remarks":"This is an industry paper with modest methodological novelty, but the detailed empirical comparison and the production case study give it value for a practitioner-oriented venue. The main barrier is the lack of uncertainty quantification in the real-data comparisons, which directly affects the central claim of FCQ's robustness. I would not reject the paper; the requested changes are feasible and should not require new experiments beyond reporting variance estimates and paired comparisons, or tempering the conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this one. First, it is a genuine engineering paper: it defines three new mRMR variants (FRQ, RFCQ, RFRQ) as explicit recombination of existing measures, evaluates them on a synthetic dataset and three real Uber marketing datasets, and describes a production deployment in an Auto ML platform. Second, the central empirical claim—that FCQ is robust across classification models—is weaker than the prose suggests because the real-data tables report single averaged AUCs from 4-fold CV with no variance or significance tests. The stress-test note gets this right.\n\nWhat is actually new: the three variants are simple, clearly specified extensions of the well-known mRMR framework. FRQ swaps Pearson correlation for RDC in the redundancy term; RFCQ and RFRQ replace the F-statistic relevance with random forest importance, keeping correlation or RDC for redundancy. The paper does not pretend these are deep theoretical contributions, and the evaluation is honest about their modest novelty. The synthetic study is a real strength: 100 trials, boxplots of AUC differences against RF baseline, and a design with known informative, redundant, and irrelevant features. The computation time table is useful, and the production implementation section (Scala/Spark, optimization choices) is the kind of practical detail most papers skip.\n\nWhere the paper is soft: the real-data evaluation. Table V shows differences of 0.001–0.005 AUC between methods, and with 4-fold CV those differences are within the noise. The conclusion that FCQ is 'robust' is therefore not fully supported by the real data alone; it is carried partly by the synthetic results, which are more reliable. Also, the online experiment validates the deployed model against a baseline, not FCQ against an alternative feature selector, so it does not directly test the paper's main claim. The datasets are proprietary, which limits reproducibility, though the methods are simple enough to reimplement. None of this is fatal—the paper is clear about what it does and does not claim, and the production experience has value.\n\nFor a reader: this is for practitioners building Auto ML pipelines or considering mRMR variants in applied work. It deserves a serious referee, not because it advances theory, but because it is a careful, reproducible-in-spirit empirical study with a real deployment. I would send it to peer review with a request for error bars or paired significance tests on the real-data tables, and a toned-down conclusion about robustness. A solid case study that needs tightening, not a flawed one.\n\nWould I cite it? Not in my own methods work, but possibly as an industrial application example. Bring it to a reading group? Maybe, if the group cares about the gap between offline evaluation and production deployment.","headline":"Useful industrial case study of mRMR feature selection; the FCQ robustness claim is plausible but under-supported by the real-data results without error bars.","tokens_in":13592,"tokens_out":1121,"would_cite":false,"duration_ms":12490,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that a simple mRMR variant, FCQ, is the most practical feature-selection method for marketing classification, and that model-based variants win only when the downstream model is Random Forest.","keywords":["feature selection","mRMR","minimum redundancy maximum relevance","filter methods","classification","marketing machine learning","random forest feature importance","automated machine learning"],"falsifier":"A concrete test: take a marketing dataset in which the useful features are connected to the label mainly through nonlinear transformations while redundancy among features is also nonlinear. If a method like FRQ or RFRQ, which uses the RDC nonlinear dependence measure, consistently beats FCQ by a large AUC margin across several such datasets, the paper's claim that FCQ is the robust production default would fail to generalize. A second check is to rerun the reported online experiment with two additional arms—models built from RFCQ-selected features and from all features—and compare incremental adoption; the current experiment lacks those arms.","tokens_in":12630,"feed_emoji":"🎯","tokens_out":7995,"duration_ms":74491,"temperature":0.7,"pith_summary":"The paper sets out to find a feature-selection method that can serve as a reliable default in an automated marketing machine-learning platform, where hundreds to thousands of user-level features are available for targeting and personalization models. Starting from the mRMR (minimum redundancy, maximum relevance) framework, it adds two ingredients—a nonlinear dependence measure (RDC) for redundancy and Random-Forest importance for relevance—and evaluates seven mRMR variants plus plain Random-Forest ranking on one synthetic and three real-world marketing datasets. Its central finding is that the simplest quotient variant, FCQ, which divides an F-statistic relevance score by the average Pearson correlation with already-selected features, is nearly as accurate as the fancier extensions while being far faster. The paper concludes that FCQ's robustness across Naive Bayes, Logistic Regression, and Random Forest makes it the right production default, and that the model-aware variants RFCQ and RFRQ are worth the extra compute when the downstream classifier is known to be Random Forest. It also reports that FCQ was deployed in production and that an online campaign using the resulting model produced a statistically significant 12% lift in adoption among targeted high-propensity users.","feed_headline":"One formula beats fancier feature selectors for marketing models","feed_subtitle":"The simple F-test divided by average correlation beat fancier selectors on accuracy and speed and now runs in production.","key_machinery":"The machinery is the mRMR scoring step: at each greedy selection round, a candidate feature $X_i$ is scored by relevance to the label $Y$ minus or divided by its average redundancy with the already-selected set $S$. The production-recommended form is FCQ, $$f_{FCQ}(X_i)=\\frac{F(Y,X_i)}{\\frac{1}{|S|}\\sum_{X_s\\in S}\\rho(X_s,X_i)},$$ where $F$ is the one-way ANOVA F-statistic and $\\rho$ is Pearson correlation. Two extensions change the inputs: FRQ replaces $\\rho$ with the randomized dependence coefficient (RDC), a copula-based nonlinear association measure; RFCQ and RFRQ replace $F$ with Random-Forest Gini importance, either keeping Pearson correlation (RFCQ) or RDC (RFRQ). The quotient form is load-bearing because relevance and redundancy have different units and scales, so a difference form either lets one term dominate or requires careful normalization. The entire method is a greedy filter: independent of the downstream classifier, so it can be computed once and reused, and its per-step statistics are cheap enough to implement at production scale.","core_discovery":"The paper's discovery is that, for marketing classification problems with many redundant features, the balance between relevance and redundancy matters more than the sophistication of either measure. In head-to-head comparison, mutual-information-based mRMR variants (MID, MIQ) perform poorly because of the difficulty of estimating feature distributions, and the difference-scheme FCD suffers from mismatched scales between its F-statistic and correlation terms. The quotient scheme in FCQ—relevance divided by average redundancy—avoids the scale problem, and its reliance on univariate F-statistics and Pearson correlations makes it fast enough for production. The model-based extensions are the right tool when the final classifier is a Random Forest: RFCQ and RFRQ reach good AUC with fewer selected features than plain Random-Forest importance ranking, because they remove redundant features the tree would otherwise double-count. The paper claims these conclusions hold across its synthetic data and the three real marketing datasets, and that the production implementation with FCQ delivers the expected accuracy, scaling, and interpretation benefits.","pith_inferences":["Editorial inference: because FCQ depends only on univariate F-statistics and pairwise Pearson correlations, it can be computed in a single distributed pass over the data; this makes it an attractive baseline in any large-scale automated machine-learning pipeline, not just marketing.","Editorial inference: the RFCQ result suggests a self-reinforcing loop—train a quick Random Forest to obtain importances, select features, retrain—which should be compared against end-to-end embedded selection such as L1-regularized logistic regression; the paper does not include that baseline.","Editorial inference: the extension recipe is modular, so one could plug in gradient-boosting importances or other nonlinear dependence measures without changing the greedy selection; the authors point toward this but do not test it."],"forward_implications":["Automated marketing platforms can safely default to a univariate F-statistic plus average-correlation score for feature selection, avoiding expensive mutual-information estimation.","When the downstream model is a tree ensemble, using tree importance in the mRMR quotient (RFCQ or RFRQ) yields the best accuracy and the smallest feature sets, at the cost of extra computation.","Feature selection improves model quality mainly through redundancy removal and overfitting prevention, visible when Naive Bayes or Logistic Regression with all features underperforms the same model on 10 to 30 selected features.","The selected feature sets are less correlated with each other, which improves model interpretability and diagnosis compared with plain importance ranking."],"supporting_citations":[{"why":"Supplies the original mRMR relevance-minus-redundancy criterion that all evaluated variants build on.","marker":"[6]"},{"why":"Defines the F-statistic/correlation difference and quotient variants FCD and FCQ.","marker":"[11]"},{"why":"Establishes that the best individual features are not necessarily the best feature subset, motivating the redundancy control.","marker":"[5]"},{"why":"Defines Random Forests, used both as a downstream classification model and as the source of the model-based importance measure.","marker":"[13]"},{"why":"Provides the Random-Forest feature importance score used in the RFCQ and RFRQ extensions.","marker":"[18]"},{"why":"Introduces the randomized dependence coefficient (RDC) used to measure nonlinear redundancy in the FRQ and RFRQ variants.","marker":"[15]"}],"fun_headline_variants":["For marketing features, simple relevance-redundancy quotient wins","FCQ feature selection: fast, accurate, and production-ready","Marketing ML: divide relevance by redundancy for best features","Quotient of F-stat and correlation beats complex selectors","Simple feature selection wins for marketing models at scale"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the three real marketing datasets used for offline comparison—cross-sell, up-sell, and app cross-sell—represent the full range of classification problems the automated platform serves, so the observed ranking of feature selectors will carry over to future campaigns; the single online experiment does not compare FCQ against alternative feature-selection methods.","fun_headline_variants_meta":{"raw":{"variants":["For marketing features, simple relevance-redundancy quotient wins","FCQ feature selection: fast, accurate, and production-ready","Marketing ML: divide relevance by redundancy for best features","Quotient of F-stat and correlation beats complex selectors","Simple feature selection wins for marketing models at scale"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000309,"raw_usage":{"total_tokens":1791,"prompt_tokens":999,"completion_tokens":792,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":711}},"tokens_in":615,"tokens_out":792,"duration_ms":8440,"temperature":1.0,"reasoning_tokens":711,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:15:03.394463+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: take a marketing dataset in which the useful features are connected to the label mainly through nonlinear transformations while redundancy among features is also nonlinear. If a method like FRQ or RFRQ, which uses the RDC nonlinear dependence measure, consistently beats FCQ by a large AUC margin across several such datasets, the paper's claim that FCQ is the robust production default would fail to generalize. A second check is to rerun the reported online experiment with two additional arms—models built from RFCQ-selected features and from all features—and compare incremental adoption; the current experiment lacks those arms.","supporting_citations":[{"cited_title":"Feature selection based on mu- tual information: criteria of max-dependency, max-relevance, and min- redundancy,","cited_arxiv_id":null,"evidence_quote":"Supplies the original mRMR relevance-minus-redundancy criterion that all evaluated variants build on."},{"cited_title":"The best two independent measurements are not the two best,","cited_arxiv_id":null,"evidence_quote":"Establishes that the best individual features are not necessarily the best feature subset, motivating the redundancy control."},{"cited_title":"A comparison of random forest and its gini importance with standard chemometric methods for the fea- ture selection and classification of spectral data,","cited_arxiv_id":null,"evidence_quote":"Provides the Random-Forest feature importance score used in the RFCQ and RFRQ extensions."},{"cited_title":"The randomized de- pendence coefficient,","cited_arxiv_id":null,"evidence_quote":"Introduces the randomized dependence coefficient (RDC) used to measure nonlinear redundancy in the FRQ and RFRQ variants."}],"review_version":1}