Pith. sign in

REVIEW 4 major objections 4 minor 24 references

CohortHijack: Robustness of Single Cell Annotation to Companion Cell Removal

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Removing selected non-target companion cells from a query cohort can change a single cell's refined annotation even when its expression profile, base prediction, and trained model are unchanged.

desk verdict The core finding is real and well supported, but the headline flip rates are measured on a deliberately vulnerable audit set, so the paper overstates the typical risk and under-reports variability. read the letter →

arxiv 2608.05900 v1 pith:JFURIGA7 submitted 2026-08-06 cs.LG

Arash Vashagh , Yasmin Vashagh This is my paper · ORCID
classification cs.LG
keywords single-cellRNAsequencingcell-typeannotationadversarialrobustnesscohortdependencelabelrefinementneighborhoodvotingcompanion-cellremovalstabilitytesting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that cohort-dependent label refinement is a target-preserving attack surface in single-cell annotation. Removing a small set of non-target companion cells from the query cohort can change a target cell's refined annotation, even though the target cell's expression profile, the base classifier, and all trained parameters are left untouched. The authors test this with random, structured, and search-based removal strategies on two annotated single-cell datasets, finding that structured removal flips substantially more targets than random removal and that a multi-start search changes about 24% of targeted linear-SVM cells and about 20% of logistic-regression cells while mean collateral flip rates stay below 0.4%. Ablations show that the effect vanishes when the neighborhood-refinement stage is disabled, which isolates that stage as the mechanism. If the paper is right, clean annotation accuracy alone does not capture how reliable a refined label is, and cells with uncertain identities can have labels that depend on which other cells happen to be in the query cohort.

What carries the argument

The load-bearing mechanism is the neighborhood-refinement stage, in which the refined probability $r_{ic}$ for cell $i$ and class $c$ combines the frozen base probability $p_{ic}$ with a neighborhood support term $h_{ic}$ that is a confidence-weighted average of neighboring cells' base probabilities (equivalently, a uniform average under majority voting). When a removal set $S$ is deleted from the query cohort, the neighbor sets $\mathcal{N}_k(i)$ are recomputed and $h_{ic}$ changes, which can shift the argmax of $r_{ic}$. The paper's removal strategies differ in how they choose $S$: random baseline, removing same-class cells, removing nearest cells, and search procedures (greedy, multi-start greedy, and beam search) that lexicographically optimize the target's margin while penalizing collateral flips.

What would settle it

Apply the same removal budget and search procedure to the full test cohort, including high-confidence and misclassified cells, and compare target flip rates to the preselected target set. If the full-cohort flip rate is indistinguishable from random subsampling noise, the claimed attack surface is an artifact of target selection rather than a general property of the refinement stage.

Watch

Extended reading notes

Core claim

The central claim is that the final label produced by a cohort-dependent annotation pipeline is not determined by the target cell alone. In the paper's formulation, the refined probability for a target cell is a weighted combination of its base-classifier probability and a neighborhood-support term computed from the surrounding cells; removing any non-target cell changes the neighbor set and therefore the support term. The paper demonstrates, for two datasets and two frozen linear classifiers, that small structured removals flip the refined labels of a meaningful fraction of low-confidence but correctly annotated cells, and that the same pattern appears in an established majority-voting annotation tool where independent predictions never change but cohort-voted labels do. The paper concludes by identifying query cohort composition as a target-preserving attack surface and by recommending that pipelines report independent and refined predictions separately.

Load-bearing premise

The load-bearing assumption is that the low-confidence correctly annotated cells selected as targets represent the practically relevant vulnerable population; if such cells are rare in a real query cohort, the measured flip rates overstate how often companion-cell removal will change an arbitrary cell's label.

Editorial extensions

If this is right

  • If the claim holds, any pipeline that refines labels by neighborhood voting or cluster majority is potentially sensitive to which companion cells survive quality control and downsampling.
  • The mechanism ablation implies that the fix should target the refinement stage: when the context weight is zero, no removal changes any target label.
  • In the majority-voting validation, all target flips occurred with unchanged independent predictions, so cohort-level voting is the point of failure, not the base classifier.
  • Reporting independent and refined labels separately, and flagging cells where they disagree, would give users a practical warning that a label may be cohort-dependent.
  • Repeated controlled subsampling could serve as a stability check, with unstable cells assigned broader lineage labels or routed to manual review.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The target set is deliberately biased toward lower-confidence correctly annotated cells, so the headline flip rates are likely upper bounds for a typical query cohort; the paper does not measure the base rate of such vulnerable cells.
  • If vulnerable cells are concentrated along developmental transitions or overlapping populations, then cohort-removal flips could also distort downstream trajectory or composition analyses, not just the labels themselves.
  • A natural testable extension is to compute a stability score for every cell from an ensemble of controlled subsamples and check whether that score predicts disagreement between independent and refined labels or manual-review outcomes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper proposes CohortHijack, a robustness audit for single-cell annotation pipelines that include a cohort-dependent refinement stage. The method removes a small set of non-target 'companion' cells from the query cohort while keeping the target cell's expression profile, base classifier, and trained parameters unchanged, and it measures whether the refined label changes. The authors evaluate random, same-class, and nearest-cell removal, plus greedy, multi-start greedy, and beam search, on PBMC3K and Paul15 with logistic regression and calibrated linear SVM. They supplement the controlled pipeline with a CellTypist majority-voting validation. They report that structured removal is stronger than random removal on Paul15, that multi-start search changes 24.33% of linear-SVM targets and 19.67% of logistic-regression targets with low collateral damage, that the effect disappears when the context weight is zero, and that CellTypist refined labels are similarly unstable while its independent labels remain unchanged.

Significance. If the result is taken at face value, the paper identifies a genuinely new target-preserving attack surface for cohort-dependent single-cell annotation and offers a reusable audit procedure. The controlled experimental design is sound, the lambda = 0 ablation is a clean mechanistic check, and the CellTypist validation provides independent evidence that the phenomenon is not an artifact of the authors' own pipeline. The main weakness is that the headline flip rates are measured on a deliberately constructed audit set of lower-confidence, correctly annotated targets, and the paper does not quantify the base rate of such cells; the practical frequency of flips in an ordinary query cohort is therefore not established. As stated in the paper, the rates are vulnerability rates within the targeted audit set, not estimates over all cells. The contribution is best characterized as demonstrating that cohort-removal flips exist and can be induced, with the practical significance depending on how common context-sensitive cells are in routine cohorts.

major comments (4)
  1. [Abstract and Section III.A] The headline claim in the Abstract—'Multi-start search changed 24.33% of linear-SVM targets and 19.67% of logistic-regression targets'—omits the qualifier stated in Section III.A: these rates were measured on 100 correctly annotated, lower-confidence targets per dataset, classifier, and seed, and 'should be interpreted as vulnerability rates within this targeted audit set, rather than as estimates over all cells in the dataset.' Because the target-selection policy deliberately enriches for context-sensitive cells, the abstract's percentages cannot be read as population-level flip rates and should carry the audit-set qualifier. The paper should also discuss how the base rate of lower-confidence, context-sensitive cells affects the practical frequency of flips in a typical query cohort.
  2. [Section III.C and Table I] The search-based success rates are reported as single point estimates aggregated over seeds 13, 37, and 73, with no per-seed breakdown, confidence interval, or p-value. Because the 100 targets per dataset, classifier, and seed are sampled without replacement, the 24.33% and 19.67% figures could vary across target samples and seeds; the paper should report per-seed results or otherwise demonstrate that the aggregated rates are stable.
  3. [Section III.A and Section III.E] There is a numerical inconsistency in the CellTypist validation. Section III.A describes 10 context-sensitive and 10 initially stable targets per seed over 3 seeds, i.e., 60 targets total; with 3 removal methods and 2 budgets, this gives 360 evaluations for each outcome, not the '840 evaluations' claimed in Section III.E. In addition, the 'Context-sensitive 2%' row in Table II reports 27.33%, which would correspond to 8.2 out of 30 targets and is not an integer count. These inconsistencies should be corrected because they directly affect the credibility of the validation statistics.
  4. [Section II.F, Section II.G, and Section III.C] The search methods optimize the refined-probability margin m_t(S) defined in Section II.F, and the attack-success indicator in Section II.G is exactly a negative margin. This is a legitimate audit procedure, but it means the multi-start success rates are an upper-bound-style worst-case measure under a matched objective, not an estimate of naturally occurring flips. The paper should state this interpretation explicitly and should not present the search-based rates on the same footing as the structured-removal rates, which do not optimize the success criterion.
minor comments (4)
  1. [Section III.B] The statement that 'paired tests confirmed' the Paul15 differences is not verifiable without naming the test (e.g., paired t-test, Wilcoxon signed-rank) and reporting the test statistics or p-values; please add these details.
  2. [Section II.D] The phrase 'The same target manifests are reused' appears to be a typo for 'target sets' or 'target lists'; the intended meaning is clear but the wording should be corrected.
  3. [Section III.A] Please state explicitly whether each structured-removal method was run once per seed or repeated; the ten repeats are specified for random removal only, and this affects how the reported flip rates should be interpreted.
  4. [Table II] The percentages in Table II are based on only 30 targets per group across three seeds, so each 3.33% increment corresponds to one target; this should be stated in the caption to prevent over-interpretation of small differences between conditions.

Circularity Check

1 steps flagged · score 2.0 of 10

Search-based flip rates partly reflect the optimizer's own objective, but the core vulnerability claim is independently supported by random/structured removal and CellTypist; Section III.A's audit-set qualifier is explicit.

  1. other [Section II.F 'Search-Based Removal', Section II.G Eq. (3), Table I]
    "For the target's clean refined label, let c_t denote its corresponding class index. We measure its margin after removing S as m_t(S) = r_tc_t - max_{c != c_t} r_tc ... A target flip is preferred over a non-flip. ... The target flip rate is the mean of A_t(S) across evaluated targets."

    The search methods directly optimize the refined-probability margin m_t(S), and the measured 'target flip' is exactly the event m_t(S) < 0. The headline multi-start flip rates (24.33% and 19.67%) are therefore the success rates of an optimizer that explicitly ranks a flip as preferred and minimizes the same margin used to define success. This is partly by construction: the reported rates describe how often a search engineered to flip a label succeeds, not how often random cohort perturbation flips a cell.

full rationale

The paper's derivation chain is largely self-contained. The annotation pipeline, refinement equation (Eq. 2), removal strategies, and flip/collateral definitions are stated explicitly, and the claimed mechanism is directly ablated: when lambda = 0, refinement cannot change the target prediction by construction, and the paper reports no flips. The strongest independent evidence is the random versus structured removal comparison on Paul15 and the CellTypist majority-voting experiment, where no margin was optimized and the target's independent prediction stayed fixed while refined labels changed. These results are not fitted to the outcome and do not reduce to an input definition. The main qualification, already present in the paper, is that the headline percentages apply to 100 lower-confidence correct targets per dataset/classifier/seed and should not be read as population-level estimates; this is a generalization caveat rather than a circularity. The self-citations in the introduction ([17]-[19]) are background context and are not load-bearing. The only notable by-construction element is that the search-based attack success rate equals the rate at which the search achieves its own stated objective, which the paper acknowledges by framing the results as an audit of a targeted set. Overall this is a minor, partial circularity, warranting a low score.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim is an empirical existence result, so the ledger is dominated by modeling and evaluation choices rather than fitted physical constants. The quantitative flip rates depend on the hand-chosen neighborhood size, context weight, and the selection of lower-confidence targets; no new biological entities are introduced.

free parameters (5)
  • Neighborhood size k = 25 (main; ablations 10, 25, 50)
    Chosen by hand for the controlled pipeline; the ablation shows flip rates depend on k, so the main quantitative rates are specific to this setting.
  • Context weight lambda = 1 (with alpha = 1)
    Chosen by hand; vulnerability increases with lambda and disappears at lambda = 0, so the main flip rates reflect this specific weighting.
  • PCA components = 40
    Dimensionality reduction setting chosen for the controlled pipeline; affects the Euclidean neighbor graph.
  • Class cap and minimum class size = 500 cells per class cap, 30-cell minimum
    Preprocessing choices that determine cohort composition and which classes are retained.
  • Target audit size = 100 lower-confidence targets per dataset-classifier-seed
    Flip rates are defined over this selected subset, not over the full test cohort.
assumptions (4)
  • domain assumption Euclidean nearest-neighbor graph in PCA space is a valid model of the cohort context used by annotation refinement.
    The controlled pipeline defines refinement via Eqs. (1)-(2) on this graph; if real pipelines use different context, such as cluster-level voting, the controlled findings may not transfer.
  • domain assumption Clean refined labels, used to select correctly annotated targets, are trustworthy ground truth.
    Target selection in Section II.D relies on the refined label being correct; errors in reference labels would alter the meaning of a flip.
  • domain assumption The frozen classifier's probabilities remain valid after cells are removed from the query cohort.
    The pipeline reuses base probabilities without retraining or recalibration; in practice, pipelines may recompute normalization or recalibrate.
  • domain assumption Reference labels in PBMC3K and Paul15 are correct enough for evaluating annotation flips.
    The paper uses reference labels as ground truth for correctly annotated targets.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CohortHijack: Robustness of Single Cell Annotation to Companion Cell Removal." pith.science (2026). https://pith.science/paper/JFURIGA7

@misc{pith2026260805900,
  author       = {Pith},
  title        = {Pith review of: CohortHijack: Robustness of Single Cell Annotation to Companion Cell Removal},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JFURIGA7}},
  note         = {Machine review of arXiv:2608.05900}
}
read the original abstract

Many single-cell annotation tools refine an initial cell label using nearby cells or cluster-level voting. We study whether this refinement can be manipulated without changing the target cell. We introduce CohortHijack, a robustness audit that removes selected non-target cells from the query cohort while preserving the target expression profile, base prediction, and trained model. We evaluate random and structured removal methods, together with greedy, multi-start, and beam search, on PBMC3K and Paul15 using logistic regression and calibrated linear SVM classifiers. Structured removal was consistently stronger than random removal on Paul15. Multi-start search changed 24.33% of linear-SVM targets and 19.67% of logistic-regression targets while removing a small fraction of the cohort and keeping mean collateral changes below 0.4%. Ablations confirmed that the effect disappeared when neighborhood refinement was disabled. We also evaluated CellTypist majority voting, where independent predictions remained unchanged across all evaluations, but refined labels changed after small companion-cell removals. These findings identify query cohort composition as a target-preserving attack surface in single-cell annotation.

Figures

Figures reproduced from arXiv: 2608.05900 by the authors.

Figure 1
Figure 1. Overview of the CohortHijack threat model. Under ordinary annotation, the target cell is classified using its own [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Effect of context weight on target flip rates for same [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 22 canonical work pages

  1. [1]

    SCANPY: Large-scale single- cell gene expression data analysis,

    F. A. Wolf, P. Angerer, and F. J. Theis, “SCANPY: Large-scale single- cell gene expression data analysis,”Genome Biology, vol. 19, no. 1, p. 15, 2018

    work page 2018

  2. [2]

    Cross-tissue immune cell analysis reveals tissue-specific features in humans,

    C. D. Conde, C. Xu, L. B. Jarvis, D. B. Rainbow, S. B. Wells, T. Gomes, S. K. Howlett, O. Suchanek, K. Polanski, H. W. King, L. Mamanova, N. Huang, P. A. Szabo, L. Richardson, L. Bolt, E. S. Fasouli, K. T. Mahbubani, M. Prete, L. Tuck, N. Richoz, Z. K. Tuong, L. Campos, H. S. Mousa, E. J. Needham, S. Pritchard, T. Li, R. Elmentaite, J. Park, E. Rahmani, D...

    work page 2022

  3. [3]

    A comparison of automatic cell identifi- cation methods for single-cell rna sequencing data,

    T. Abdelaal, L. Michielsen, D. Cats, D. Hoogduin, H. Mei, M. J. T. Reinders, and A. Mahfouz, “A comparison of automatic cell identifi- cation methods for single-cell rna sequencing data,”Genome Biology, vol. 20, no. 1, p. 194, 2019

    work page 2019

  4. [4]

    scmap: Projection of single- cell RNA-seq data across data sets,

    V . Y . Kiselev, A. Yiu, and M. Hemberg, “scmap: Projection of single- cell RNA-seq data across data sets,”Nature Methods, vol. 15, no. 5, pp. 359–362, 2018

    work page 2018

  5. [5]

    Probabilistic cell-type assignment of single-cell rna-seq for tumor microenvironment profiling,

    A. W. Zhang, C. O’Flanagan, E. A. Chavez, J. L. P. Lim, N. Ceglia, A. McPherson, M. Wiens, P. Walters, T. Chan, B. Hewitson, D. Lai, A. Mottok, C. Sarkozy, L. Chong, T. Aoki, X. Wang, A. P. Weng, J. N. McAlpine, S. Aparicio, C. Steidl, K. R. Campbell, and S. P. Shah, “Probabilistic cell-type assignment of single-cell rna-seq for tumor microenvironment pro...

    work page 2019

  6. [6]

    Probabilistic harmonization and annotation of single-cell transcrip- tomics data with deep generative models,

    C. Xu, R. Lopez, E. Mehlman, J. Regier, M. I. Jordan, and N. Yosef, “Probabilistic harmonization and annotation of single-cell transcrip- tomics data with deep generative models,”Molecular Systems Biology, vol. 17, no. 1, p. e9620, 2021

    work page 2021

  7. [7]

    Ucell: Robust and scalable single-cell gene signature scoring,

    M. Andreatta and S. J. Carmona, “Ucell: Robust and scalable single-cell gene signature scoring,”Computational and Structural Biotechnology Journal, vol. 19, pp. 3796–3798, 2021

    work page 2021

  8. [8]

    Consensus prediction of cell type labels in single-cell data with popv,

    C. Ergen, G. Xing, C. Xu, M. Kim, M. Jayasuriya, E. McGeever, A. Oliveira Pisco, A. Streets, and N. Yosef, “Consensus prediction of cell type labels in single-cell data with popv,”Nature Genetics, vol. 56, no. 12, pp. 2731–2738, 2024

    work page 2024

Show all 24 references
  1. [9]

    Efficient and precise single-cell reference atlas mapping with symphony,

    J. B. Kang, A. Nathan, K. Weinand, F. Zhang, N. Millard, L. Rumker, D. B. Moody, I. Korsunsky, and S. Raychaudhuri, “Efficient and precise single-cell reference atlas mapping with symphony,”Nature Communi- cations, vol. 12, no. 1, p. 5890, 2021

    2021

  2. [10]

    scplan: a hierarchical computational framework for single transcriptomics data annotation, integration and cell-type label refinement,

    Q. Guo, M. Yuan, L. Zhang, and M. Deng, “scplan: a hierarchical computational framework for single transcriptomics data annotation, integration and cell-type label refinement,”Briefings in Bioinformatics, vol. 25, no. 4, p. bbae305, 2024

    2024

  3. [11]

    Jind: joint integration and discrimination for automated single-cell annotation,

    M. Goyal, G. Serrano, J. Argemi, I. Shomorony, M. Hernaez, and I. Ochoa, “Jind: joint integration and discrimination for automated single-cell annotation,”Bioinformatics, vol. 38, no. 9, pp. 2488–2495, 2022

    2022

  4. [12]

    Nico identifies extrinsic drivers of cell state modulation by niche covariation analysis,

    A. Agrawal, S. Thomann, S. Basu, and D. Gr ¨un, “Nico identifies extrinsic drivers of cell state modulation by niche covariation analysis,” Nature Communications, vol. 15, no. 1, p. 10628, 2024

    2024

  5. [13]

    scsampler: fast diversity- preserving subsampling of large-scale single-cell transcriptomic data,

    D. Song, N. M. Xi, J. J. Li, and L. Wang, “scsampler: fast diversity- preserving subsampling of large-scale single-cell transcriptomic data,” Bioinformatics, vol. 38, no. 11, pp. 3126–3127, 2022

    2022

  6. [14]

    scvalue: value-based subsampling of large-scale single-cell transcriptomic data for machine and deep learning tasks,

    L. Huang, W. Gong, and D. Chen, “scvalue: value-based subsampling of large-scale single-cell transcriptomic data for machine and deep learning tasks,”Briefings in Bioinformatics, vol. 26, no. 3, p. bbaf279, 2025

    2025

  7. [15]

    Intriguing properties of neural networks,

    C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,”arXiv preprint arXiv:1312.6199, 2013

    2013 arXiv

  8. [16]

    Explaining and harnessing adversarial examples,

    I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,”arXiv preprint arXiv:1412.6572, 2014

    2014 arXiv

  9. [17]

    Recent ad- vances in adversarial attacks on model utility, privacy, and explainability: A comprehensive survey,

    A. Vashagh, R. Razavi-Far, M. Meymani, and B. Biggio, “Recent ad- vances in adversarial attacks on model utility, privacy, and explainability: A comprehensive survey,”TechRxiv, 2026

    2026

  10. [18]

    Defense against adversarial attacks: Foundations, strategies, and future direc- tions,

    M. Meymani, R. Razavi-Far, A. Vashagh, and B. Biggio, “Defense against adversarial attacks: Foundations, strategies, and future direc- tions,”Preprints, 2026

    2026

  11. [19]

    Conformalshift: Targeted event reordering against adaptive ecg monitoring,

    A. Vashagh and Y . Vashagh, “Conformalshift: Targeted event reordering against adaptive ecg monitoring,”arXiv preprint arXiv:2608.03628, 2026

    2026 arXiv

  12. [20]

    adverscarial: assessing the vulnerability of single-cell rna-sequencing classifiers to adversarial attacks,

    G. Fievet, J. Bros ´eus, D. Meyre, and S. Hergalant, “adverscarial: assessing the vulnerability of single-cell rna-sequencing classifiers to adversarial attacks,”Bioinformatics, vol. 41, no. 4, p. btaf168, 2025

    2025

  13. [21]

    Adversarial attack on graph structured data,

    H. Dai, H. Li, T. Tian, X. Huang, L. Wang, J. Zhu, and L. Song, “Adversarial attack on graph structured data,” inProceedings of the 35th International Conference on Machine Learning, vol. 80, 2018, pp. 1115–1124

    2018

  14. [22]

    Adversarial attacks on graph neural networks: Perturbations and their patterns,

    D. Z ¨ugner, O. Borchert, A. Akbarnejad, and S. G¨unnemann, “Adversarial attacks on graph neural networks: Perturbations and their patterns,”ACM Transactions on Knowledge Discovery from Data (TKDD), vol. 14, no. 5, pp. 1–31, 2020

    2020

  15. [23]

    Uncertainty- aware single-cell annotation with a hierarchical reject option,

    L. Theunissen, T. Mortier, Y . Saeys, and W. Waegeman, “Uncertainty- aware single-cell annotation with a hierarchical reject option,”Bioinfor- matics, vol. 40, no. 3, p. btae128, 2024

    2024

  16. [24]

    Interpreting single-cell and spatial omics data using deep neural network training dynamics,

    J. Karin, R. Mintz, B. Raveh, and M. Nitzan, “Interpreting single-cell and spatial omics data using deep neural network training dynamics,” Nature Computational Science, vol. 4, no. 12, pp. 941–954, 2024

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.