{"id":"d061c884-e3b2-4d62-8325-8f3f6ab780ef","arxiv_id":"2412.06805","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"HiCat fuses CatBoost predictions with DBSCAN clusters in a multi-resolution embedding to annotate known cell types and distinguish multiple novel cell types in scRNA-seq data.","lead":"HiCat is a semi-supervised pipeline that combines supervised cell type classification with unsupervised clustering to label known cell types and identify new ones in single-cell RNA sequencing data. It builds a 53-dimensional feature space from reference and query data and uses confidence scores to decide when to trust the classifier versus the clusters.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"HiCat's headline capability — separating multiple novel cell types — is supported only by a post hoc aligned confusion matrix from two datasets; no fixed cluster-to-label rule, metric, or baseline is given, so the 'surpasses' claim is not yet established.","rationale":"The reader's concern about Harmony erasing query-only populations is a real mechanistic risk, but the manuscript actually reports successful recovery of such populations, so the more urgent issue is whether that success is measured fairly. Experiment 3 is the unique contribution, and it lacks a fixed mapping, a quantitative metric, and any comparator. This is a falsifiable evaluation gap rather than an internal inconsistency: the method could be correct, but the evidence as presented cannot support the 'surpasses' claim for the multi-novel-type setting. The paper deserves credit for releasing code and for being transparent about deferred broader comparisons, but the headline capability needs stronger support. I keep the reader's conditional verdict because the gap is fixable by additional analysis with the public code.","tokens_in":16095,"tokens_out":5071,"duration_ms":52696,"concrete_test":"Rerun Experiment 3 on all 33 pairs that contain unseen types, with a pre-registered mapping: for each query-only true type, assign the DBSCAN cluster whose majority of low-confidence CatBoost query cells is that type; then report per-type sensitivity/specificity, macro-F1, and adjusted Rand index between HiCat's novel cluster IDs and true unseen labels. Run the same protocol with (a) DBSCAN on UMAP alone using the same Steps 1–3 embedding, and (b) scNym with pseudo-labels. If HiCat's macro-F1/ARI under the fixed mapping is not significantly above DBSCAN-alone, the fusion's claimed advantage in differentiating multiple novel types is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To establish the abstract's central claim — that HiCat 'surpasses other methods, particularly in differentiating and identifying multiple new cell types' — the paper must show that the fused pipeline recovers distinct novel populations under a fixed, non-arbitrary labeling rule and beats reasonable baselines. Experiment 3 does neither. It uses only two pancreas query pairs; the confusion-matrix columns are numeric DBSCAN cluster IDs, and the caption states 'Cluster labels are arranged to align as closely as possible with the likely true labels' (Fig. 4). Because the mapping from clusters to true unseen types is chosen after the fact to maximize diagonal concentration, the matrices can only demonstrate that DBSCAN can form clusters — not that HiCat's supervised/unsupervised fusion outperforms alternatives at distinguishing multiple novel types. No metric (ARI, NMI, macro-F1, cluster purity) is reported for this task, and no baseline is run, even though DBSCAN on UMAP alone would also produce multiple clusters. The paper's own Limitations defer a broader method comparison to a journal paper, but the specific 'differentiating multiple new cell types' advantage is the paper's headline novelty and its only support is this post hoc illustration.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"HiCat is a semi-supervised pipeline for cell type annotation from single-cell RNA sequencing data. The method harmonizes reference and query data with Harmony into a 50-dimensional principal component embedding, further reduces to two dimensions with UMAP, and obtains one-dimensional cluster labels with DBSCAN; these are concatenated into a 53-dimensional feature space on which a CatBoost classifier is trained using reference cell type labels. For query cells, the CatBoost prediction is used when its confidence is high, and otherwise the DBSCAN cluster label is assigned, which is intended to label cell types absent from the reference. The authors benchmark HiCat on 33 reference-query pairs derived from 10 public datasets, reporting improved accuracy over nine existing methods for known cell types and improved F1 for detecting unseen cell types, and they illustrate the ability to separate multiple novel cell types on two pancreas query datasets.","tokens_in":16325,"tokens_out":4388,"duration_ms":40360,"significance":"The proposed architecture is simple and practical: multi-resolution feature construction from both reference and query data is a reasonable idea, and the supervised/unsupervised fusion is clearly described. The authors also provide code on GitHub, which supports reproducibility. If the evaluation were made rigorous, the multi-resolution feature engineering and the confidence-based fusion of CatBoost and DBSCAN would be a useful contribution to cell annotation. As it stands, however, the headline claim of distinguishing multiple novel cell types rests on post hoc aligned confusion matrices, and the comparisons to existing methods lack statistical error bars and any semi-supervised baseline. The central claims are therefore plausible but not yet established.","major_comments":[{"comment":"The central claim that HiCat differentiates multiple novel cell types is supported only by confusion matrices whose cluster IDs are arranged post hoc to align with true labels, as the caption states. Such an alignment can demonstrate that DBSCAN forms separable clusters, but it does not show that HiCat's fusion outperforms any alternative at assigning novel cell types. The paper should define a fixed, deterministic rule for mapping DBSCAN cluster IDs to cell type labels (or at least one that does not use query ground truth), report standard metrics such as ARI, NMI, macro-F1, or cluster purity for the novel-cell task, and compare against at least one baseline such as DBSCAN on the UMAP embedding alone or a semi-supervised method.","section":"Section 3, 'Distinguishing Multiple Unseen Cell Types' and Figure 4"},{"comment":"The confidence threshold rule is under-specified: 'The threshold for determining high confidence is based on the largest drop in predicted probabilities across both the reference and query datasets.' It is unclear whether the 'drop' refers to the difference between the top two predicted probabilities per cell, the gap in the probability distribution, or some other quantity, and how a single global threshold is extracted from a drop. Because this threshold determines which cells are assigned to DBSCAN/novel labels, it is load-bearing for the method's main claimed advantage. The authors should give the exact algorithm and report sensitivity of the results to the threshold.","section":"Section 2, Step 6"},{"comment":"The evaluation aggregates results over 33 dataset pairs but provides no error bars, confidence intervals, or significance tests. The unseen-type scenarios are created by 'randomly remov[ing] cell types,' yet the number of random repetitions and seeds is not reported. Without repeated subsampling or paired statistical tests, the claims that HiCat 'consistently ranks highest' and 'clearly excels' cannot be distinguished from chance variation. The authors should add repeated random subsampling with standard errors and, for the unseen-type experiments, paired tests across the same dataset pairs.","section":"Section 3, Figures 2 and 3"},{"comment":"HiCat is motivated as a semi-supervised method, and the Introduction cites CALLR, scNym, scSemiGAN, and scBERT as related semi-supervised approaches, but none of these appears in the benchmark. The Limitations section defers a broader comparison to a journal paper, yet the abstract claims HiCat 'surpasses other methods' without qualification. At minimum, the known-type and unseen-type experiments should include one or two of the most directly related semi-supervised baselines to support the comparative novelty.","section":"Section 1 versus Section 3"}],"minor_comments":[{"comment":"Reference entries 1–3 contain 'ezproxy.library.uvic.ca' proxy URLs that will not resolve for most readers; these should be replaced by DOIs or canonical publisher URLs.","section":"References"},{"comment":"There are several typos: 'pipline' in Section 1, 'spares logistic regression' in the description of CALLR, and 'Embedings' in the keywords.","section":"Introduction and keywords"},{"comment":"The Figure 4 caption states that HiCat identified unseen types 'even when they contained fewer than 10 cells,' while the text reports that endothelial cells contained 'about 20 cells' and epsilon cells had 4 or 8 cells; please reconcile these inconsistent numbers.","section":"Figure 4 caption and Section 3 text"},{"comment":"The sentence 'Except for this sentence, this document is exactly the same as the submitted version for RECOMB 2025 on October 28, 03:06 GMT' is a submission artifact and should be removed from a journal manuscript.","section":"Abstract"},{"comment":"The description of Harmony as using 'its default configuration' with 'top 50 principal components' should clarify which software's defaults are meant and how the PCA is computed (for example, whether variable genes are used); this affects reproducibility.","section":"Section 2, Step 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely a conference submission, and the evaluation gaps are substantial but addressable within the scope of a revision. The post hoc alignment of cluster IDs in Experiment 3 is the most serious issue because it directly undermines the unique selling point. The lack of semi-supervised baselines is also notable given the Introduction's framing. I would not reject the paper, but the revision must add a fixed labeling rule, quantitative novel-cell metrics, baseline comparisons, and statistical error bars."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. The pipeline is a genuine new combination: it merges Harmony-corrected PCs, a 2D UMAP embedding, and DBSCAN cluster labels into a 53-dimensional feature space, trains CatBoost on that space, and then uses a confidence-based rule to decide whether to trust CatBoost or defer to the DBSCAN label. That is a sensible design for the problem it targets, and the multi-resolution idea is not just noise: Figure 5 shows features at all three resolutions being selected. The second thing is that the headline claim, that HiCat surpasses other methods 'particularly in differentiating and identifying multiple new cell types,' is not yet backed by the evidence. Experiment 3, the only support for that capability, uses just two pancreas query pairs. The confusion matrices are aligned after the fact – the caption says cluster labels are arranged to match likely true labels – and no metric (ARI, NMI, purity) or baseline is reported. This is a demonstration, not a benchmark.\n\nWhat the paper does well is the evaluation for known and unseen types. The 33 data pairs across 10 datasets are a reasonable test bed, and HiCat comes out top on accuracy in most pairs in Experiment 1 and on F1 for the 'unseen' category in Experiment 2. That is a meaningful result. The code is on GitHub. And the Limitations section is unusually honest: it flags the need for more baselines, subgroup analyses by platform and species, and the risk of Harmony erasing query-specific biology.\n\nThe soft spots are proportionate to a conference paper. The confidence threshold is described only as 'the largest drop in predicted probabilities,' which is under-specified and needs to be pinned down. There are no significance tests or repeated subsampling, so I would not rank methods on these numbers alone. The stress-test note about post hoc cluster alignment is fair, but it is not a reason to reject the idea; it is a reason to demand a fixed labeling rule and quantitative cluster-quality metrics before the 'surpasses' claim is published.\n\nWho should read this: anyone building or benchmarking cell-annotation tools will want this pipeline as a comparison point, and practitioners with a reference atlas and a messy query set may find it useful. It deserves a serious referee. The right outcome is major revision: keep the method, rebuild Experiment 3 with a non-arbitrary cluster-to-label rule, add ARI/NMI and a DBSCAN-on-UMAP baseline, and specify the threshold rule. Then the paper would deliver on its promise.","headline":"Plausible new semi-supervised pipeline with a genuine multi-resolution feature space and a strong known-type benchmark, but the headline claim of separating multiple novel cell types is supported only by post hoc aligned confusion matrices from two datasets.","tokens_in":16864,"tokens_out":3143,"would_cite":true,"duration_ms":29261,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HiCat claims that fusing CatBoost confidence with DBSCAN cluster labels in a 53-dimensional embedding lets researchers annotate known cell types and separate novel ones.","keywords":["single-cell RNA sequencing","cell type annotation","semi-supervised learning","novel cell type discovery","DBSCAN clustering","CatBoost","multi-resolution feature space","Harmony batch correction"],"falsifier":"Construct a reference-query pair where the query contains a known shared cell type and a spiked-in rare population with about 30 cells, run HiCat, and check whether the spiked cells form their own DBSCAN cluster; if the spiked population is absorbed into a shared type after Harmony, the claimed rare-novel-cell detection collapses.","tokens_in":15882,"feed_emoji":"🧬","tokens_out":6708,"duration_ms":56560,"temperature":0.7,"pith_summary":"The paper proposes HiCat, a semi-supervised pipeline for single-cell RNA sequencing that combines a supervised classifier (CatBoost) with an unsupervised clustering step (DBSCAN). The central claim is that by embedding reference and query cells together in a 53-dimensional feature space built from Harmony-corrected principal components, UMAP coordinates, and DBSCAN cluster labels, the method can accurately label known cell types and, more importantly, identify and separate multiple cell types that never appeared in the reference data. The authors benchmark on 33 reference-query pairs from 10 published datasets and report that HiCat outperforms existing methods on known cell types and is especially accurate at flagging unseen cells, including rare novel populations with as few as 20 cells. The reason to care is that standard supervised annotators degrade when query data contains new cell types, while purely unsupervised clustering suffers from cluster impurity; HiCat couples the two signals so that low-confidence supervised predictions fall back to clustering labels.","feed_headline":"New pipeline labels known and novel cell types in one pass","feed_subtitle":"HiCat fuses CatBoost confidence with DBSCAN clustering to flag unseen cell types, even ones with ~20 cells.","key_machinery":"The central object is the 53-dimensional multi-resolution feature space, formed by concatenating Harmony's 50 principal components, UMAP's two coordinates, and DBSCAN's one cluster-membership vector for every cell in both reference and query sets. CatBoost is trained on the reference portion of this space, and for query cells it produces both a predicted cell type and a probability. The fusion rule is what carries the argument: when the predicted probability is near chance, HiCat treats the classifier as out of its depth and assigns the DBSCAN cluster label, which is how novel cell types receive labels; the confidence cutoff is chosen automatically from the largest gap in predicted probabilities across both datasets.","core_discovery":"HiCat's core claim is that cell-type annotation is best done by merging supervised and unsupervised evidence across multiple resolutions of the same data. The pipeline first applies Harmony to the combined reference and query expression matrices, producing a 50-dimensional principal-component embedding in which shared cell types are aligned across batches. UMAP then compresses this to two dimensions, and DBSCAN assigns each cell a cluster label or marks it as noise. These three outputs are concatenated into a 53-dimensional feature space, on which CatBoost is trained using the reference labels. When CatBoost's predicted probability for a query cell is low, HiCat defers to the DBSCAN cluster label, which is what lets genuinely novel cell types surface as separate clusters. On 33 benchmark pairs the authors report that this procedure maintains accuracy when unseen cell types are present and, unlike the compared methods, can tell multiple unknown cell types apart.","pith_inferences":["If the fusion idea transfers, the same 53-dimensional construction could be applied with other supervised classifiers and other density-based clusterers; the key hypothesis is that low-confidence supervised predictions indicate out-of-distribution cells that clustering can resolve.","A practical implication the authors do not spell out: researchers can use HiCat's per-cluster outputs as a discovery screen, then validate candidate novel populations with curated marker genes rather than accepting the numeric labels as biological identities.","The reported instability on 4-cell epsilon populations suggests a natural stress test: benchmark HiCat on query sets with novel populations of 5, 10, 20, and 50 cells to quantify the minimum reliable population size.","Because the paper pools cross-platform and cross-species pairs, a natural next analysis is to compare HiCat's novel-type recovery separately for same-platform and cross-platform pairs, since Harmony's alignment quality is likely the deciding factor."],"forward_implications":["For query sets that contain no unseen cell types, HiCat is reported to achieve the highest accuracy in most of the 33 benchmark pairs.","When one to three unseen cell types are present, HiCat keeps its median accuracy roughly stable, while the compared methods decline; it also has higher F1 for flagging cells as unseen.","HiCat can assign distinct labels to multiple unseen cell types; in the two pancreas examples it separated acinar, ductal, stellate, and endothelial cells with over 90% accuracy, including an endothelial population of about 20 cells.","Rare novel populations are not always recovered: the paper reports unstable results for 4 epsilon cells in one query set versus 8 in another, so very small novel clusters may be missed or split.","The decision boundary between 'trust CatBoost' and 'trust DBSCAN' is set by the largest gap in predicted probabilities, so the method has no tunable confidence parameter in its default form."],"supporting_citations":[{"why":"Supplies the Harmony batch-effect correction that aligns reference and query cells into the 50-dimensional embedding.","marker":"[32]"},{"why":"Supplies the UMAP step that reduces the harmonized embedding to two dimensions for clustering.","marker":"[38]"},{"why":"Supplies DBSCAN, the density-based clustering that produces the unsupervised labels used for novel-cell detection.","marker":"[20]"},{"why":"Supplies CatBoost, the gradient-boosting classifier whose confidence scores gate the final labels.","marker":"[18]"},{"why":"A batch-effect benchmark the authors cite for choosing Harmony in Step 1.","marker":"[50]"},{"why":"Defines the 33 reference-query benchmark pairs that all three evaluation experiments rely on.","marker":"[6]"},{"why":"One of the two query pancreas datasets used to show HiCat separates multiple unseen cell types.","marker":"[39]"},{"why":"The other query pancreas dataset used in the multiple-unseen-types experiment.","marker":"[44]"},{"why":"The Xin pancreas dataset used as the reference with only four shared cell types in Experiment 3.","marker":"[55]"}],"fun_headline_variants":["HiCat: one pass to label known and novel cell types","Semi-supervised pipeline finds unseen cell types with DBSCAN","CatBoost plus DBSCAN: hybrid cell annotation","HiCat annotates known and novel cell types together","Merging supervision and clustering reveals new cell types"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that Harmony can strip away technical differences between the reference and query datasets while keeping intact the gene-expression signatures of cell types that exist only in the query set; if Harmony treats those query-only populations as batch variation and merges them into shared cell types, HiCat's downstream unsupervised step has nothing left to detect.","fun_headline_variants_meta":{"raw":{"variants":["HiCat: one pass to label known and novel cell types","Semi-supervised pipeline finds unseen cell types with DBSCAN","CatBoost plus DBSCAN: hybrid cell annotation","HiCat annotates known and novel cell types together","Merging supervision and clustering reveals new cell types"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000158,"raw_usage":{"total_tokens":1236,"prompt_tokens":968,"completion_tokens":268,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":187}},"tokens_in":584,"tokens_out":268,"duration_ms":2929,"temperature":1.0,"reasoning_tokens":187,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:34:16.594214+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a reference-query pair where the query contains a known shared cell type and a spiked-in rare population with about 30 cells, run HiCat, and check whether the spiked cells form their own DBSCAN cluster; if the spiked population is absorbed into a shared type after Harmony, the claimed rare-novel-cell detection collapses.","supporting_citations":[{"cited_title":"In: Proceedings of the Second International Conference on Knowledge Discovery and Data Mining","cited_arxiv_id":null,"evidence_quote":"Supplies DBSCAN, the density-based clustering that produces the unsupervised labels used for novel-cell detection."},{"cited_title":"https://doi.org/10.21203/rs.3.rs-3219242/v1, https://www","cited_arxiv_id":null,"evidence_quote":"Defines the 33 reference-query benchmark pairs that all three evaluation experiments rely on."},{"cited_title":"Cell Metabolism24(4), 608–615 (Oct 2016)","cited_arxiv_id":null,"evidence_quote":"The Xin pancreas dataset used as the reference with only four shared cell types in Experiment 3."}],"review_version":1}