{"id":"70db372a-b878-4f8a-a2d4-4c5bf0294ced","arxiv_id":"2501.04718","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"RIGPS combines ensemble pre-filtering with multi-agent reinforcement learning to select compact gene panels, reporting improved clustering and annotation across 24 scRNA-seq datasets, though the final selection step may leak evaluation labels.","lead":"This paper introduces RIGPS, an automated pipeline that selects compact gene panels from label-free single-cell RNA sequencing data by combining ensemble gene-ranking knowledge with multi-agent reinforcement learning. The authors report that RIGPS outperforms eight baseline methods on 24 datasets for clustering and cell-type annotation, but the evaluation contains a critical label-selection step that undermines the label-free claim.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1's final selection metric E is never specified; if it uses NMI/ARI with true labels, the reported label-free advantage is an artifact of evaluation-guided selection.","rationale":"The central claim requiring support is that RIGPS is label-free and outperforms all baselines on clustering and annotation benchmarks. The most load-bearing condition is that no true cell-type label influences any selection decision, especially the final choice of G*. Algorithm 1 lines 15-21 choose G* by maximizing E(C(X[G'])), and Section 4.2 defines the evaluation metrics used in this paper as NMI, ARI, and SI. Since NMI and ARI require ground-truth labels, the paper must state which E is used and whether labels are passed. The pseudocode input omits labels, but that is not conclusive: Section 2's Eq. (1) and Section 4.2 describe E as the same evaluation metric used for benchmarking, and the paper never explicitly forbids label use in selection. If E is NMI or ARI, the method is not label-free and its comparison against baselines is unfair; if E is label-free, the best-over-history selection still needs to be justified relative to baselines. The reader's weakest assumption identifies exactly this ambiguity, and I agree with the reader's verdict: the central claim is not established. The proposed check (code inspection or a label-free rerun) would resolve the issue. No change to the reader's verdict is needed.","tokens_in":40369,"tokens_out":10557,"duration_ms":105705,"concrete_test":"Inspect the public code linked in Section 4.1 (currently a Dropbox link without commit hash) and locate the function implementing Algorithm 1 lines 15-21. Determine whether E is computed from ground-truth labels, e.g., a call such as `nmi(pred_labels, y_true)`, or from a label-free metric such as silhouette. If the code shows label-based final selection, the label-free claim is not supported. If the code is unavailable, re-run the 24-dataset benchmark with E fixed to silhouette and no history selection; if RIGPS no longer beats all baselines in NMI/ARI, the reported advantage is an artifact of label-informed or history-optimized selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 1 (lines 15-21) returns the history subset G' with the highest clustering metric E(C(X[G'])). The paper never states what E is. In the evaluation protocol (Section 4.2), the metrics are NMI, ARI, and SI; NMI and ARI are computed against true cell-type labels, whereas SI is label-free. If Algorithm 1 uses NMI or ARI, the final gene panel is chosen by oracle access to the ground-truth labels that define the reported performance. That would contradict the label-free claim and make the comparison against baselines unfair, because baselines do not receive an equivalent label-informed final selection. If, instead, E is SI or pseudo-label-based NMI, no true labels are used, but the final step is an optimistic best-over-history selection whose effect on the reported margins is never quantified. The pseudocode input list in Algorithm 1 omits ground-truth labels, yet Section 4.2 and Eq. (1) treat E as the same clustering metric used for evaluation, so both readings are plausible. This ambiguity is load-bearing because the benchmark numbers in Section 5.1 and Table 7 are the primary evidence for the claim that RIGPS outperforms all baselines.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RIGPS, a gene panel selection method for label-free single-cell RNA-seq data that combines an ensemble pre-filtering step, injection of prior knowledge from existing gene selection algorithms, and multi-agent reinforcement learning. Each gene is controlled by an agent that selects or discards it; the reward combines a pseudo-label-based NMI term (Eq. 2) and a parsimony term (Eq. 3). The final panel is chosen in Algorithm 1 by evaluating a clustering metric E over all candidate subsets encountered during training and keeping the best. The authors report that RIGPS outperforms eight baseline methods on 24 scRNA-seq datasets for clustering (NMI, ARI, SI) and for supervised cell-type annotation, alongside ablations, hyperparameter studies, robustness analysis, and biological case studies.","tokens_in":40551,"tokens_out":3985,"duration_ms":41947,"significance":"If the label-free claim and the reported performance hold, RIGPS would be a practically useful contribution: it targets a real problem, is evaluated on 24 datasets across species and tissues, includes ablations of each component, and provides qualitative downstream analyses. The code and selected gene sets are promised to be publicly available, which is a strength for reproducibility. However, the central quantitative claim is currently undermined by an unspecified and potentially label-aware final selection step, and by the alignment between the training reward and the evaluation metric. These issues are load-bearing because the benchmark numbers, rather than the architecture alone, are the main evidence for the method.","major_comments":[{"comment":"The final selection step evaluates E(C(X[G'])) over every candidate subset in the history H and returns the subset with the highest E. The paper never states what E is in this step. Section 4.2 defines the evaluation metrics as NMI, ARI, and SI, with NMI and ARI computed against true cell-type labels, while Eq. (1) uses the same E and C notation as Algorithm 1. If E is NMI or ARI, the final gene panel is selected using oracle access to the ground-truth labels that also define the reported performance, which contradicts the label-free claim (Section 3.2, Algorithm 1 input) and makes the comparison against baselines unfair, because the baselines do not receive an equivalent label-informed final selection. If E is instead SI or a pseudo-label-based metric, this must be stated explicitly and the effect of the best-over-history selection on the reported margins must be quantified. As written, both readings are plausible, and this ambiguity is load-bearing for the Section 5.1 claim that RIGPS outperforms all baselines on 19 of 24 datasets in NMI.","section":"Algorithm 1, lines 15-21; Section 4.2; Eq. (1)"},{"comment":"The reward function is NMI between the selected gene expression matrix and pseudo-labels obtained from Louvain clustering of that same expression matrix, and the benchmark metrics are NMI and ARI computed from Louvain clustering of the selected genes. This means the agent is trained to maximize essentially the same quantity on which it is evaluated. Combined with the best-over-history selection in Algorithm 1, the reported gains may be partly a selection artifact rather than evidence of independent predictive quality. Please report the performance of the final trained policy without best-over-history selection, or using a held-out evaluation protocol, so that the reader can separate the contribution of the learned policy from the contribution of the selection rule.","section":"Section 3.2, Eq. (2); Section 4.2; Section 5.1"},{"comment":"All clustering and annotation results are reported as single point estimates. No standard deviations, confidence intervals, or numbers of independent RL runs are given. Since the policies are stochastic and the final panel is selected by scanning the training history, the claim that the average NMI of RIGPS outperforms all baselines needs variance quantification to establish that the difference is not within run-to-run noise. This is particularly important because the Nemenyi test in Figure 3 is presumably computed from a single set of per-dataset values.","section":"Section 5.1 and Table 7"}],"minor_comments":[{"comment":"The text says Figure 3 compares RIGPS with ten gene panel selection methods, but Section 4.3 lists eight baselines and Table 7 additionally reports Original Dataset and Random, so the count is inconsistent.","section":"Section 5.1 and Figure 3 caption"},{"comment":"The denominator in Eq. (5) is written as 'P rj ∈P rj', which is confusing; it should be the sum of r_j over all j in P, written as Σ_j r_j.","section":"Section 3.3, Eq. (5)"},{"comment":"The sentence 'We adopt the Louvain community detection algorithm to generate pseudo-labels for reward estimation and those supervised selection methods' should clarify whether the Random Forest, SVM, RFE, and KBest pre-filtering methods are trained on Louvain pseudo-labels from the full dataset or from a candidate subset. This is important for assessing possible information leakage in the pre-filtering step.","section":"Section 4.4"},{"comment":"The 2-sigma threshold in Eq. (7) is described as forming a coarse boundary with high recall, but no justification or empirical support is given for the 2-sigma choice; the hyperparameter study in Section 5.8 varies the set of basic methods but not the threshold.","section":"Section 3.3 and Eq. (7)"},{"comment":"Reference [46] is listed with a CrossRef 'Deleted DOI' placeholder; a proper citation for normalized mutual information should be provided.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline case between major revision and rejection. The core ambiguity in Algorithm 1 is serious, but it is resolvable in principle: the authors can explicitly define E, or remove the label-informed selection and re-run the benchmarks, or provide a held-out evaluation. Because the contribution is potentially useful and the experimental infrastructure is extensive, I prefer major revision over rejection. The editor may wish to ensure that the authors also provide standard errors or repeated-run statistics, since the stochastic RL pipeline and best-over-history selection make point estimates difficult to interpret."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The stress-test note holds up on reading: Algorithm 1's final selection step is the load-bearing problem. Lines 16-21 choose the panel that maximizes E(C(X[G'])) over the history, and E is never defined. Section 4.2 lists NMI, ARI, and SI as evaluation metrics; NMI and ARI need ground-truth labels, SI does not. If E uses true labels, the label-free claim is false and the baseline comparison is unfair, since baselines get no equivalent oracle selection. If E uses SI or pseudo-label NMI, the final step is an optimistic best-over-history pick whose effect on the reported margins is never quantified. Either way, the headline result cannot be taken at face value.\n\nThat said, the core idea is worth taking seriously. Applying multi-agent RL to gene panel selection, with an ensemble meta-vote pre-filter and a parsimony-aware reward on pseudo-label clustering, is a genuine new combination. The benchmark is broad: 24 datasets across species and tissues, with clustering and annotation comparisons plus ablation studies. The ablations (pre-filtering, knowledge injection, reinforcement) are informative and mostly support the design choices.\n\nBeyond Algorithm 1, reproducibility is thin. There are no error bars or seeds, and the code is a Dropbox link without a commit hash. Table 7 contains at least one impossible value: gpsFISH on Chu2 shows ARI 102.11, which exceeds the stated 0-1 range even as a percentage. That kind of error makes me less confident in the table's accuracy. Also, the reward and the evaluation both use NMI-based clustering purity, so the optimization is partially aligned with the metric; not fatal, but it should be stated.\n\nWho is this for? Method developers in single-cell genomics and ML-for-bio who want a label-free gene panel selector. The paper deserves a serious referee: the idea is novel enough and the experiments are extensive, but the evaluation protocol needs a major correction. I would send it to review with a request to specify E, rerun without any label-informed selection (or with a validation-set split), add variance estimates, and fix the table errors.\n\nRecommendation: engage, but require the clarification and re-analysis before trusting the numbers.","headline":"A genuinely new RL-based gene panel selector, but Algorithm 1's unspecified final-selection metric makes the reported gains unverifiable.","tokens_in":41122,"tokens_out":3807,"would_cite":false,"duration_ms":36996,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes RIGPS, a label-free gene panel selection method that combines ensembled prior knowledge from existing selectors with multi-agent reinforcement learning, and claims it outperforms all baselines on 24 single-cell RNA-seq…","keywords":["gene panel selection","single-cell RNA-seq","reinforcement learning","multi-agent reinforcement learning","label-free biomarker discovery","clustering","knowledge ensemble","scRNA-seq feature selection"],"falsifier":"Rerun RIGPS with the final selection step restricted to rewards computed only from pseudo-labels (no true labels anywhere), and compare the resulting NMI on the same 24 datasets; if the margin over the baselines collapses, the reported advantage depends on label-informed selection rather than on the reinforcement-learning pipeline.","tokens_in":40132,"feed_emoji":"🧬","tokens_out":4941,"duration_ms":44207,"temperature":0.7,"pith_summary":"RIGPS is an iterative, label-free gene panel selection method for single-cell RNA-seq. It first combines several existing gene-selection algorithms into a coarse candidate gene set, then uses multiple reinforcement-learning agents to refine that set through repeated clustering and reward feedback. The paper claims that on 24 public datasets the resulting panels outperform all tested baselines on clustering quality, rank first in NMI on 19 of 24 datasets, and also improve supervised cell-type annotation, while producing smaller gene panels. If correct, this would give biologists compact marker sets for cheaper targeted assays and more robust downstream analysis without needing cell-type labels.","feed_headline":"RL-guided gene panels beat all baselines on 24 single-cell datasets","feed_subtitle":"Label-free method RIGPS tops clustering and annotation benchmarks while choosing fewer genes.","key_machinery":"The load-bearing mechanism is a multi-agent reinforcement-learning loop in which one agent per candidate gene decides to select or discard its gene, sharing a state representation built from descriptive statistics of the current gene subset and compressed by an autoencoder. The reward combines a biological-distinctiveness term, the normalized mutual information between the selected genes' expression and Louvain pseudo-labels, and a parsimony term that penalizes large panels; the loop is bootstrapped by injecting gene subsets produced by existing selectors as initial experiences. That combination lets the search start from ensemble knowledge while still moving beyond it.","core_discovery":"The central claim is that RIGPS finds a better gene panel than existing methods precisely because it treats panel selection as an expert-like sequential decision process: ensemble knowledge from simpler selectors provides a high-recall starting boundary, multi-agent reinforcement learning then explores and optimizes within that boundary, and a reward function that combines an unsupervised separability term with a parsimony term guides the agents toward genes that distinguish cell populations. The paper reports that this pipeline achieves the highest average NMI, ARI, and silhouette scores across 24 scRNA-seq datasets from multiple species and tissues, with the best NMI rank on 19 of 24 datasets, and that it surpasses all baselines on balanced accuracy and F1 metrics for cell-type annotation.","pith_inferences":["Editorial inference: if the final selection step in Algorithm 1 uses true cell-type labels to pick among candidate panels, the headline 'label-free' claim would be undercut, and the reported advantage might partly reflect that label access rather than the RL refinement.","Editorial inference: because the reward is a cluster-quality score computed from pseudo-labels, the method may favor genes that produce the easiest clusters rather than the most biologically meaningful ones, so independent validation against known marker genes or perturbation data would strengthen the biological claim.","Editorial inference: the same architecture could be transferred to other high-dimensional molecular measurements, such as spatial transcriptomics or ATAC-seq, by replacing the state statistics while keeping the reward and agent loop unchanged."],"forward_implications":["The selected panels are compact enough to lower the cost of targeted assays, since the paper shows RIGPS chooses fewer genes than the second-best baseline while keeping higher clustering performance.","Because the method needs no cell-type labels for training, it can be applied to new tissues and species without manual annotation, a scenario the 24-dataset evaluation is meant to support.","Ablation results show that both the pre-filtering step and the knowledge injection contribute to the final gain, implying the ensemble of basic selectors is doing real work rather than being a formality.","Reported linear scaling of training time with cell number suggests the approach remains usable on atlas-scale datasets, at least within the sizes tested."],"supporting_citations":[{"why":"Supplies CellBRF, a supervised feature-selection baseline and one of the knowledge sources injected into the RL agents.","marker":"[26]"},{"why":"Supplies geneBasis, an iterative unsupervised panel-selection baseline and a knowledge source for initialization.","marker":"[27]"},{"why":"Supplies HRG, a graph-based gene selection baseline and a knowledge source.","marker":"[28]"},{"why":"Supplies Seurat v3's variance-stabilizing gene selection, one of the baselines.","marker":"[22]"},{"why":"Supplies the Louvain community-detection clustering used both for pseudo-labels in reward estimation and for downstream evaluation.","marker":"[45]"},{"why":"Supplies prioritized experience replay, the mechanism by which agents store and sample transitions during policy optimization.","marker":"[36]"},{"why":"Supplies the actor-critic algorithm used to update each gene agent's policy.","marker":"[37]"},{"why":"Supplies Pearson residuals, a normalization-based baseline for comparison.","marker":"[50]"}],"fun_headline_variants":["Reinforcement learning picks better gene panels for scRNA-seq","Expert-shaped RL improves single-cell gene panel discovery","RIGPS: RL-based gene panel selection beats all baselines","RL-guided gene panels beat baselines on 24 scRNA-seq datasets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Algorithm 1's final step picks the candidate gene panel with the best clustering metric, but the paper never says whether that metric uses the true cell-type labels; if it does, the method is not fully label-free and the benchmark gains could come from that label access.","fun_headline_variants_meta":{"raw":{"variants":["Reinforcement learning picks better gene panels for scRNA-seq","Expert-shaped RL improves single-cell gene panel discovery","RIGPS: RL-based gene panel selection beats all baselines","RL-guided gene panels beat baselines on 24 scRNA-seq datasets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000759,"raw_usage":{"total_tokens":3320,"prompt_tokens":842,"completion_tokens":2478,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":458,"completion_tokens_details":{"reasoning_tokens":2407}},"tokens_in":458,"tokens_out":2478,"duration_ms":16606,"temperature":1.0,"reasoning_tokens":2407,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:34:32.236414+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun RIGPS with the final selection step restricted to rewards computed only from pseudo-labels (no true labels anywhere), and compare the resulting NMI on the same 24 datasets; if the margin over the baselines collapses, the reported advantage depends on label-informed selection rather than on the reinforcement-learning pipeline.","supporting_citations":[{"cited_title":"Cellbrf: a feature selection method for single-cell clustering using cell balance and random forest,","cited_arxiv_id":null,"evidence_quote":"Supplies CellBRF, a supervised feature-selection baseline and one of the knowledge sources injected into the RL agents."},{"cited_title":"genebasis: an iterative approach for unsupervised selection of targeted gene panels from scrna-seq,","cited_arxiv_id":null,"evidence_quote":"Supplies geneBasis, an iterative unsupervised panel-selection baseline and a knowledge source for initialization."},{"cited_title":"Highly regional genes: graph-based gene selection for single-cell rna-seq data,","cited_arxiv_id":null,"evidence_quote":"Supplies HRG, a graph-based gene selection baseline and a knowledge source."},{"cited_title":"Comprehensive integration of single-cell data,","cited_arxiv_id":null,"evidence_quote":"Supplies Seurat v3's variance-stabilizing gene selection, one of the baselines."},{"cited_title":"Spatial reconstruction of single-cell gene expression data,","cited_arxiv_id":null,"evidence_quote":"Supplies the Louvain community-detection clustering used both for pseudo-labels in reward estimation and for downstream evaluation."},{"cited_title":"Analytic pearson residuals for normalization of single-cell rna-seq umi data,","cited_arxiv_id":null,"evidence_quote":"Supplies Pearson residuals, a normalization-based baseline for comparison."}],"review_version":1}