{"id":"0b4ab418-5308-47b7-959f-d4e3f04f6a5e","arxiv_id":"2411.11436","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A multi-label feature selection method, mFSIR, uses Hadamard-product parameterization and latent label embedding to reduce bias and promote benign overfitting.","lead":"This paper proposes a new feature selection method for multi-label data that replaces the usual penalty term with a clever parameterization and a label-embedding step. If it works, the method could reduce bias and overfitting compared with standard sparse feature selection approaches.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lemma 3.3 is the load-bearing support for the implicit-regularization claim, but it is unproved and Algorithm 1 does not satisfy its Assumption 2 initialization.","rationale":"The strongest claim cannot stand on the experiments alone because the experiments do not isolate the implicit-regularization effect: Figure 3 compares objective values of two different objectives, Table 2 is mixed (mFSIR loses on ranking loss and macro-AUC for many datasets), and Figure 7 compares overfitting on original versus feature-reduced data, which is feature-selection capacity reduction rather than benign overfitting. What remains is the theoretical assertion that the Hadamard-product parameterization plus nonnegative initialization yields sparse-feature-selection behavior comparable to explicit l2,1 regularization. That assertion is Lemma 3.3. It is not proven, and the geometric conditions borrowed from [46] are nontrivial for Eq. (8). The mismatch between Assumption 2 (nonnegative initialization) and Algorithm 1 (Uniform(-ϖ,ϖ)) is concrete: the reported experiments may not instantiate the lemma's hypothesis. A multi-start experiment on a small synthetic problem can settle whether final objectives and feature rankings are initialization-dependent. If they are, the estimator is not a well-defined implicit-regularization estimator and the paper's headline needs revision; if they are not, Lemma 3.3 should be replaced by a statement of the empirically observed invariance. Either way, the current conditional verdict remains appropriate until this is resolved.","tokens_in":17121,"tokens_out":10388,"duration_ms":116267,"concrete_test":"On a synthetic dataset with n=50, m=200, q=10, l=3, where only 10 features are truly relevant and Y is generated from X and a nonnegative low-rank label matrix, run Algorithm 1 to completion under the stopping rule in Eq. (15) for 100 random seeds under two initialization schemes: (a) Uniform(-1e-5, 1e-5) as written in Algorithm 1, and (b) Uniform(0, 1e-5) as required by Assumption 2. Record the final value of Eq. (8), the row norms of W = G ⊙ H, and the top-30% feature set. If the final objective values differ by more than 1e-6 across seeds or the selected feature sets differ materially between (a) and (b), Lemma 3.3 is not satisfied by the implemented algorithm and the claimed feature-selection optimality is initialization-dependent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Eq. (8) replaces the explicit l2,1 penalty with implicit regularization and suffers less bias rests on Lemma 3.3, which asserts convergence of the alternating updates to a global minimum. No proof is supplied: the lemma states that under Assumption 3 the objective has no local maxima, all local minima are global, and all saddle points are strict, citing [46], but these geometric conditions are not verified for Eq. (8), which couples a Hadamard-parametrized regression term, a nonnegative NMF term, and a graph-Laplacian regularizer. Worse, Lemma 3.3 is attributed to Assumption 2, which requires initial values of G or H to be nonnegative, while Algorithm 1 initializes all blocks from Uniform(-ϖ,ϖ) and does not project G or H during updates. Figure 4 shows that positive versus random initialization produces different sparsity patterns. Consequently, the final W = G ⊙ H may be an initialization-dependent stationary point; the row-norm ranking then lacks the claimed optimality, and the reported experiments may not implement the mechanism the theory is meant to justify.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes mFSIR, a multi-label feature selection method based on implicit regularization via Hadamard product parameterization (W = G ⊙ H) and a latent semantic label embedding (V, B). The objective function in Eq. (8) replaces the explicit l2,1 penalty in Eq. (2) with an implicit regularizer, and the authors present an alternating gradient descent algorithm (Algorithm 1) with projected updates for V and B. The paper claims global convergence to a minimum (Lemma 3.3) and reports experiments on ten benchmark datasets comparing mFSIR against MIFS, MICO, GRRO, and LassoNet. It also includes a study (Section 4.3.7) suggesting that mFSIR may lead to benign overfitting on the bibtex dataset.","tokens_in":17468,"tokens_out":5496,"duration_ms":79465,"significance":"The combination of Hadamard-product implicit regularization with a latent label embedding for multi-label feature selection is a novel and potentially useful idea, and the paper addresses a real limitation of explicit penalties—extra bias. The experimental evaluation is broad, covering ten datasets and three evaluation metrics with statistical testing, which is a strength. However, the central convergence claim is not rigorously supported, and the benign-overfitting experiment does not directly demonstrate the proposed estimator's behavior. If the theoretical gap is closed and the experiments are sharpened, the approach could be a meaningful contribution to multi-label feature selection.","major_comments":[{"comment":"Lemma 3.3 asserts that Eq. (8) converges to a global minimum, but no proof is supplied. The text only states that under Assumption 3 the objective has no local maxima, all local minima are global, and all saddle points are strict, citing [46]. These geometric conditions are not verified for the specific objective in Eq. (8), which couples a Hadamard-parameterized regression term, a nonnegative NMF term α||Y - VB||_F^2, and the graph-Laplacian regularizer β tr(V^T L V). Without a proof or a verification of these conditions, the final W = G ⊙ H may be an arbitrary stationary point, and the feature ranking lacks the claimed optimality. Please provide a rigorous proof or a weaker, provable convergence statement that is sufficient for the method's practical use.","section":"Section 3.3, Lemma 3.3"},{"comment":"There is a mismatch between the theory and the algorithm. Lemma 3.3 is stated under Assumption 2, which requires initial values of G or H to be nonnegative. However, Algorithm 1 initializes all four blocks from Uniform(-ϖ, ϖ) in line 1, which produces negative entries, and the updates for G and H in lines 3–4 are unconstrained gradient steps with no projection onto the nonnegative orthant. Thus the algorithm may not satisfy the assumption under which the global convergence lemma is stated. Figure 4 empirically shows that random versus nonnegative initialization yields different sparsity patterns, indicating that the final solution is initialization-dependent. Please align the algorithm with the assumption, for example by using nonnegative initialization or by extending the theoretical analysis to the actual initialization scheme.","section":"Algorithm 1 / Assumption 2"},{"comment":"The benign-overfitting experiment does not support the abstract's claim that \"the proposed estimator ... may lead to benign overfitting.\" The experiment trains a baseline classifier on the original bibtex data and on bibtex reduced by mFSIR, and observes that validation performance tracks training performance better on the reduced data. This demonstrates that feature-selection preprocessing can reduce overfitting—a well-known effect of dimensionality reduction—but it does not isolate the effect of the implicit regularizer or of the proposed estimator itself. To substantiate the claim, the authors should compare mFSIR against an explicit-regularization feature selector (e.g., MIFS) under the same preprocessing protocol, or directly study the training and test error of the fitted X(G ⊙ H) versus the latent semantics V. As written, the experiment is consistent with any feature-selection method and is not load-bearing for the benign-overfitting claim.","section":"Section 4.3.7"},{"comment":"The \"extra bias\" comparison is not well posed. Figure 3 plots the objective function value of mFSIR (Eq. (8)) against that of MIFS, which minimizes a different objective with an explicit l2,1 penalty. A lower value of a different objective does not establish that mFSIR has less bias. The paper should either define a quantitative bias measure (for instance, estimation error against known ground-truth coefficients in a simulated sparse setting) or compare both methods under a common task-oriented metric on the selected features. The current experiment is suggestive but does not directly support the claimed advantage in bias.","section":"Section 4.3.2"}],"minor_comments":[{"comment":"The definition of the Frobenius norm is malformed; the standard definition is ||X||_F = (∑_{i,j} X_{ij}^2)^{1/2}. Please correct the formula.","section":"Eq. (1)"},{"comment":"The text says \"number of selected instances\" but should read \"number of selected features.\"","section":"Section 4.3.1"},{"comment":"The input list includes \"Validation data: (X,Y)\" but the validation set is never used in the procedure. Please remove it or explain its role in early stopping or hyperparameter selection.","section":"Algorithm 1"},{"comment":"The caption defining the markers • and ◦ is ambiguous. Please clarify that • indicates mFSIR is statistically superior to the column method and ◦ indicates inferiority, and ensure the win/tie/loss counts are fully consistent with the markers.","section":"Table 2"},{"comment":"The stated complexity O(nml + nlq + n^2l) is per iteration; Algorithm 1 runs for Tmax iterations, so the total cost is Tmax times this value. Please state this explicitly.","section":"Section 3.4, Lemma 3.4"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an interesting problem and the general idea is worth publishing if the theoretical and experimental gaps are addressed. My main concerns are the unproved global convergence lemma, the inconsistency between Assumption 2 and Algorithm 1's random initialization, and the weak support for the benign-overfitting and extra-bias claims. These are fixable in a revision, so I recommend major revision rather than rejection. The authors should also consider tempering the abstract's claims if the convergence result is weakened to a stationary-point guarantee."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"One thing to know: the mFSIR objective in Eq (8) is a genuinely new combination — Hadamard product parameterization for implicit sparsity plus nonnegative label embedding with graph Laplacian — but the paper's central theoretical claim, Lemma 3.3 (global convergence), is not proved. And the algorithm as stated doesn't satisfy the lemma's own initialization assumption.\n\nWhat's good: the experiments are substantial and follow the standard multi-label protocol. Ten datasets, three metrics, Friedman/Nemenyi, sensitivity and stability analyses. The convergence curves in Fig 3 and sparsity analysis in Fig 4 are informative. The motivation against explicit l2,1 bias is clearly laid out. The idea of using a latent semantic space V to guide feature selection is sensible and connects well to the implicit regularization.\n\nWhere it's soft. First, Lemma 3.3: the text asserts that under Assumption 2 the objective converges to a global minimum, citing [46]. But the conditions from [46] — no local maxima, all local minima global, strict saddle points — are not verified for Eq (8), which couples Hadamard regression with NMF and a graph Laplacian. Lemma 3.3 is attributed to Assumption 2, which requires G or H to start nonnegative. Algorithm 1 initializes all blocks from Uniform(-ϖ, ϖ) and never projects G or H, so the assumption is not met. Figure 4 even shows sparsity depends on initialization. That means the final W = G⊙H might be an arbitrary stationary point, and the feature ranking lacks the claimed optimality. This is the load-bearing support for the 'less bias' claim, so it needs a real proof or a weaker statement.\n\nSecond, the benign overfitting section is indirect. They train a baseline classifier on the original vs. mFSIR-reduced bibtex and show the validation curve tracks training on the reduced version. That says something about the reduced data, not about mFSIR's own overfitting behavior. The abstract's 'may lead to benign overfitting' is a stretch from this evidence.\n\nThird, Table 2 has many zero standard deviations (e.g., .011±.000); that looks like rounding or a single run. Per-dataset hyperparameters and standard errors are missing, and no code is released. Also LassoNet's hamming loss around .98 on most datasets suggests it was not well adapted to multi-label, so the comparison against it is not very informative.\n\nVerdict: this is a real idea with a substantial experimental load, but the theory is not yet there. A serious referee should ask for a proof or a reframing of Lemma 3.3, proper reporting of variance and hyperparameters, and code. I would accept it for review — it deserves referee time — but I'd expect major revisions.\n\nWho is this for: people working on multi-label feature selection, especially those interested in implicit regularization alternatives to explicit penalties. Reading group: maybe. It's worth a mention, but not a canonical discussion paper. I wouldn't cite it in the next year.","headline":"Novel combination of Hadamard implicit regularization with label embedding for multi-label feature selection, but the central convergence claim is asserted, not proved, and the algorithm contradicts its own assumption.","tokens_in":17901,"tokens_out":3213,"would_cite":false,"duration_ms":27807,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that replacing the explicit l2,1 penalty with a Hadamard-product parameterization plus a latent label embedding yields a multi-label feature selector that suffers less extra bias and may lead to benign overfitting.","keywords":["multi-label feature selection","implicit regularization","Hadamard product parameterization","latent semantic analysis","label embedding","benign overfitting","l2,1 norm","nonnegative matrix factorization"],"falsifier":"Run the alternating gradient updates from many random and nonnegative initializations on one fixed dataset and compare the resulting feature rankings and objective values; if the final $W = G \\odot H$ or the zero-pattern of $G$ and $H$ changes materially with initialization, the claimed global-minimum guarantee and sparsity property fail, and a second check would be to test whether the benign-overfitting gap closes on several datasets besides bibtex.","tokens_in":16942,"feed_emoji":"🎯","tokens_out":7326,"duration_ms":69139,"temperature":0.7,"pith_summary":"Multi-label feature selection usually adds an explicit sparsity penalty such as the $\\ell^2$,1 norm to a regression loss, and the paper argues that this extra term injects bias that limits accuracy. It proposes instead to write the coefficient matrix as the element-wise product $W = G \\odot H$ and to guide selection by a low-dimensional latent embedding of the label matrix, so sparsity and stability emerge without a penalty term. The estimator is smooth, simple to optimize by alternating gradient descent, and fast on ten benchmark datasets. The paper's headline experimental claim is that this implicit-regularization estimator suffers much less extra bias than explicit-regularization baselines and can exhibit benign overfitting on a denoised version of the bibtex dataset.","feed_headline":"No penalty term: implicit regularization for feature selection","feed_subtitle":"Hadamard-product estimator embeds labels in a latent space, trims bias, and may overfit benignly.","key_machinery":"The central object is the Hadamard product parameterization $W = G \\odot H$, which replaces the explicit $l_{2,1}$ penalty: instead of penalizing a single coefficient matrix, the method learns two matrices multiplied element-wise, so sparsity emerges when either factor has zero entries. The second mechanism is latent semantic label embedding, in which the label matrix is decomposed as $Y \\approx VB$ with nonnegative $V$ and $B$, and the graph Laplacian term $\\mathrm{tr}(V^T L V)$ enforces that instances close in input space have close latent semantics. The optimization alternates gradient updates for $G, H, V, B$ with a projection that keeps $V$ and $B$ nonnegative, and the final row norms of $G \\odot H$ provide the feature ranking.","core_discovery":"The paper proposes that explicit sparsity-inducing penalties are unnecessary for multi-label feature selection: parameterizing the coefficient matrix as $W = G \\odot H$ and adding a latent-label embedding term induces sparsity and stability implicitly. The estimator solves $\\min_{G,H,V,B \\ge 0} \\|X(G \\odot H) - V\\|_F^2 + \\alpha\\|Y - VB\\|_F^2 + \\beta\\,\\mathrm{tr}(V^T L V)$, where $V$ is a low-dimensional nonnegative latent semantics matrix, $B$ its coefficient matrix, and $L$ the graph Laplacian built from input similarities. The Hadamard form smooths the original non-smooth problem, and the authors argue that under their assumptions the alternating gradient scheme converges to a global minimum, so the row norms of $\\hat{W} = \\hat{G} \\odot \\hat{H}$ rank the features. Experiments on ten multi-label benchmark datasets indicate the estimator is competitive with or better than explicit-regularization baselines on Hamming loss and macro-averaged AUC, often much faster, and can exhibit benign overfitting on a cleaned version of the bibtex dataset.","pith_inferences":["An implication the authors leave implicit is that the convergence argument leans on unverified saddle-point conditions; a direct check of the strict-saddle property for the full objective with the graph Laplacian and nonnegative factorization terms would decide whether the global-minimum claim generalizes beyond the cited linear-regression setting.","Because the paper ties sparsity to nonnegative initialization of $G$ or $H$, the algorithm's practical behavior may change if those matrices are initialized with negative entries, a sensitivity the paper only partially explores in its sparsity experiments.","The benign-overfitting demonstration uses a single dataset after reducing features to 30 percent; a systematic sweep across datasets, noise levels, and label cardinalities would show whether the phenomenon is intrinsic to the implicit regularizer or specific to that reduction.","The estimator suggests a general recipe of implicit regularization plus latent output embedding that could transfer to semi-supervised or online multi-label learning, directions the authors name as future work but do not pursue."],"forward_implications":["The Hadamard parameterization converts the non-smooth $l_{2,1}$-regularized problem into a smooth unconstrained objective, so multi-label feature selection can run with plain gradient updates and no proximal operators.","If the bias claim holds, practitioners can drop the penalty-strength hyperparameter and rely on $\\alpha$ and $\\beta$ that tune the label embedding and manifold terms instead.","The reported convergence in roughly three to ten iterations with per-iteration complexity $O(nml + nlq + n^2l)$ makes the method practical on high-dimensional multi-label data.","The benign-overfitting observation suggests that implicit regularization outside neural-network training can also close the train-validation gap, at least when the label space is first denoised by latent semantic embedding."],"supporting_citations":[{"why":"Supplies the explicit $l_{2,1}$-regularized estimator in Eq. (2) that mFSIR replaces.","marker":"[20]"},{"why":"Provides the Hadamard product parameterization as a change-of-variable that smooths non-smooth sparse estimation problems.","marker":"[31]"},{"why":"Supports the latent semantic analysis assumption that the multi-labeled output space can be decomposed into low-dimensional nonnegative factors.","marker":"[37]"},{"why":"Cited for the conditions under which implicit regularization via Hadamard product over-parameterization leads to global minima with no local maxima and strict saddle points.","marker":"[46]"},{"why":"Underlies the claim that implicit regularization in high-dimensional linear regression adapts to problem difficulty and motivates replacing the explicit penalty.","marker":"[57]"},{"why":"Connects implicit regularization to benign overfitting in sparse linear regression, providing the theoretical framing for the paper's benign-overfitting experiments.","marker":"[33]"},{"why":"Establishes implicit bias as a mechanism of benign overfitting, used to interpret the observed train-validation behavior.","marker":"[34]"},{"why":"Defines the MIFS baseline that exploits label correlations for multi-label feature selection and is compared against in the experiments.","marker":"[38]"}],"fun_headline_variants":["Implicit regularization drops explicit penalties in multi-label FS","Hadamard product parameterization makes sparsity norms obsolete","Label embedding plus Hadamard: less bias, benign overfitting","No $l_{2,1}$ norm needed: Hadamard-based feature selection","Sparse penalties replaced by implicit regularization in multi-label FS"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on the unproven assertion that the non-convex objective has no misleading local optima, so the alternating gradient scheme's final $W = G \\odot H$ is a global solution rather than an arbitrary stationary point.","fun_headline_variants_meta":{"raw":{"variants":["Implicit regularization drops explicit penalties in multi-label FS","Hadamard product parameterization makes sparsity norms obsolete","Label embedding plus Hadamard: less bias, benign overfitting","No $l_{2,1}$ norm needed: Hadamard-based feature selection","Sparse penalties replaced by implicit regularization in multi-label FS"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000249,"raw_usage":{"total_tokens":1521,"prompt_tokens":884,"completion_tokens":637,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":549}},"tokens_in":500,"tokens_out":637,"duration_ms":6363,"temperature":1.0,"reasoning_tokens":549,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:31:44.915052+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the alternating gradient updates from many random and nonnegative initializations on one fixed dataset and compare the resulting feature rankings and objective values; if the final $W = G \\odot H$ or the zero-pattern of $G$ and $H$ changes materially with initialization, the claimed global-minimum guarantee and sparsity property fail, and a second check would be to test whether the benign-overfitting gap closes on several datasets besides bibtex.","supporting_citations":[{"cited_title":"Efficient and robust feature selec- tion via joint l2, 1-norms minimization,","cited_arxiv_id":null,"evidence_quote":"Supplies the explicit $l_{2,1}$-regularized estimator in Eq. (2) that mFSIR replaces."},{"cited_title":"Lasso, fractional norm and structured sparse estimation using a hadamard product parametrization,","cited_arxiv_id":null,"evidence_quote":"Provides the Hadamard product parameterization as a change-of-variable that smooths non-smooth sparse estimation problems."},{"cited_title":"Latent semantic analysis,","cited_arxiv_id":null,"evidence_quote":"Supports the latent semantic analysis assumption that the multi-labeled output space can be decomposed into low-dimensional nonnegative factors."},{"cited_title":"High-Dimensional Linear Regression via Implicit Regularization","cited_arxiv_id":"1903.09367","evidence_quote":"Cited for the conditions under which implicit regularization via Hadamard product over-parameterization leads to global minima with no local maxima and strict saddle points."},{"cited_title":"High-Dimensional Linear Regression via Implicit Regularization,","cited_arxiv_id":null,"evidence_quote":"Underlies the claim that implicit regularization in high-dimensional linear regression adapts to problem difficulty and motivates replacing the explicit penalty."},{"cited_title":"Implicit Regularization Leads to Benign Overfitting for Sparse Linear Regression","cited_arxiv_id":"2302.00257","evidence_quote":"Connects implicit regularization to benign overfitting in sparse linear regression, providing the theoretical framing for the paper's benign-overfitting experiments."},{"cited_title":"The implicit bias of benign overfitting,","cited_arxiv_id":null,"evidence_quote":"Establishes implicit bias as a mechanism of benign overfitting, used to interpret the observed train-validation behavior."},{"cited_title":"Exploiting multilabel information for noise- resilient feature selection,","cited_arxiv_id":null,"evidence_quote":"Defines the MIFS baseline that exploits label correlations for multi-label feature selection and is compared against in the experiments."}],"review_version":1}