Pith. sign in

REVIEW 5 major objections 7 minor 1 cited by

PathSelect turns whole-slide token pruning into a learned sequential choice, keeping ~45 patches on average and matching full-context diagnostic accuracy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A learnable sequential token router with noise-gated Soft Top-K training and Hard Top-K inference cuts WSI visual context ~36.6× while holding 74% SlideBench accuracy on frozen SlideChat.

T0 review reviewed 2026-07-30 challenge →

load-bearing objection Practical frozen-backbone plugin that gets near-SlideChat accuracy on ~45 tokens via sequential pointer + VP noise/diagonal denoise; solid applied result with the usual train–infer surrogate gap. the 5 major comments →

arxiv 2607.23631 v1 pith:KSEREAEY submitted 2026-07-26 cs.CV

PathSelect: Sequential Token Selection for Whole Slide Pathology

Jingzhi Chen , Landi He , Zehong Chen , Peihang Wu , Lijian Xu This is my paper
classification cs.CV
keywords Gigapixel Whole-Slide ImagesSequential Token SelectionToken PruningVision-Language Modelscomputational pathologySoft Top-Kvariance-preserving noise gate
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Gigapixel pathology slides produce so many visual tokens that vision-language models cannot afford to read them all. Static sampling and training-free pruning often throw away sparse but decisive tumor cues. This paper reframes pruning as an autoregressive routing problem: a lightweight scorer walks the slide, picks complementary diagnostic regions one by one, and stops when enough evidence is in hand. During training a variance-preserving noise gate and a diagonal-attention denoiser make the discrete choices differentiable while freezing the pretrained slide encoder and language model; at inference only the scorer and a hard top-k remain. The result is an empirical average of 44.86 tokens under a 128-token cap, roughly 36.6 times fewer than the dense baseline, with 74 percent overall accuracy on SlideBench (TCGA) and competitive zero-shot transfer.

Core claim

The authors show that a decoupled sequential selection plugin, trained end-to-end with continuous noise gating and then discarded at test time, can reduce whole-slide visual context to an average of 44.86 tokens while recovering essentially the full diagnostic accuracy of the uncompressed SlideChat baseline and beating compute-matched sampling and biomedical VLMs.

What carries the argument

PathSelect: an alive-weighted Soft Top-K score drives a variance-preserving noise gate that mixes each patch with isotropic Gaussian noise; a diagonal-attention Denoiser restores the representation without letting unselected tokens leak semantics; at inference the gate and denoiser are removed and a deterministic Hard Top-K plus learned stop token execute adaptive trajectory termination.

Load-bearing premise

The continuous noisy training path must teach a scorer whose hard discrete selections at inference still carry the same diagnostic information the language-model loss optimized.

What would settle it

Measure the train–inference selection mismatch directly: if the set of patches retained by Hard Top-K at test time diverges sharply from the soft alive-weighted scores that received gradient, or if accuracy collapses when the diagonal denoiser mask is replaced by full attention, the claimed transfer fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Frozen encoder–LLM stacks can be adapted to gigapixel inputs with a single-GPU, single-stage plugin rather than full fine-tuning.
  • Adaptive, sample-wise token budgets replace fixed retention rates, automatically spending fewer tokens on simple slides and more on complex ones.
  • Clinical reasoning can improve over the dense baseline once redundant stroma is stripped away.
  • Quadratic LLM prefill cost scales with intrinsic specimen complexity rather than raw patch count N.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same noise-gate-plus-stop-token pattern could transfer to other long-context multimodal settings (long video, multi-page documents) where discrete selection must stay end-to-end trainable.
  • Because the scorer is conditioned on the text instruction, the method implicitly performs question-aware region retrieval; this could be evaluated as a stand-alone localization task.
  • If the stop token is reliable, one could expose the selected trajectory to a pathologist as an interactive “evidence path” without extra supervision.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper reformulates token pruning for gigapixel whole-slide images as a text-conditioned autoregressive selection process. A pointer-decoder Scorer sequentially selects patch tokens with a learned stop action, enabling per-sample adaptive budgets. Because hard top-K selection is non-differentiable, training proceeds through a continuous surrogate: alive-weighted, geometrically decayed pointer scores (Eq. 10) are polarized by a Soft Top-K operator (Eq. 11), then used in a variance-preserving Gaussian noise gate (Eq. 12) whose output is repaired by a diagonal-attention Denoiser (Eq. 13) before the frozen SlideChat LLM. At inference, the gate and Denoiser are detached and a deterministic Hard Top-K policy on the Scorer alone selects tokens. On SlideBench (TCGA) the method reports 74.00% overall accuracy with an empirical average of ~45 tokens under a K=128 cap, versus 74.81% for the uncompressed SlideChat upper bound at ~1642 average tokens (a claimed 36.6× reduction), plus zero-shot results on SlideBench (BCNB) and WSI-VQA*. Ablations cover the Denoiser mask, score aggregation, length penalty λ, and budget K.

Significance. If the numbers hold, this is a useful contribution to a real bottleneck: WSI token counts make dense VLM ingestion expensive, and a frozen-backbone plugin that recovers ~99% of the uncompressed model's accuracy at ~3% of the tokens, trained single-stage on one GPU, is a practically attractive recipe. Specific strengths worth crediting: (i) the framework makes per-sample token counts data-dependent via a learned stop action rather than a fixed budget, which is the right formulation for slides of heterogeneous complexity; (ii) the ablation matrix (Table 2a/b, Table 3) is broader than typical, including mean±std over 2–3 runs for the λ sweep; (iii) zero-shot transfer to BCNB and WSI-VQA* exceeds the uncompressed SlideChat baseline (56.39 vs 54.14; 60.76 vs 60.18), which is a non-trivial falsifiable signal that selection is not merely overfitting TCGA. The train/inference decoupling via a VP noise gate is a reasonable surrogate-gradient design, and the diagonal-attention ablation gives it at least indirect empirical support.

major comments (5)
  1. [§4.2 Table 1 / §4.4 Table 3] Table 1 vs Table 3 inconsistency in the headline numbers. Table 1 reports 74.00% overall with 'Pruning Rate=97.27%, LP=2', and the abstract and §4.3 cite an empirical average of 44.86 tokens. Table 3's LP=2.0, K=128 row — nominally the identical configuration — reports 73.80±0.29% average and 45.84 average tokens, and Table 2 again quotes 74.00% for the full model. Either Table 1 is a different run/cherry-picked seed than the mean reported in Table 3, or the configurations differ in an unstated way. Since 74.00% vs 74.81% (SlideChat upper bound) is the paper's central 'near-lossless' claim, the authors must reconcile which number is the mean over runs and report Table 1 with the same mean±std protocol as Table 3.
  2. [§4.1 Baselines / Table 1] The claim of 'consistently outperforming sampling-based counterparts' (abstract, §4.2) is not supported by the baseline matrix. Table 1 contains no token-reduction baseline applied to the same SlideChat backbone — no uniform/random sampling of SlideChat to K=128, and no training-free pruner (VisionZip [45], ToMe [11], attention-magnitude ranking, all cited) plugged into the identical frozen base at a matched token budget. As written, the only same-backbone comparison is the uncompressed upper bound, and the remaining baselines are different VLMs wholesale (LLaVA-Med, Quilt-LLaVA, MedDr, GPT-4o), so the gains cannot be attributed to the selection mechanism rather than the SlideChat base. At minimum: SlideChat + random-K and SlideChat + a training-free scorer at matched average token counts are needed to isolate the contribution of the learned sequential policy. This is the load-bearing ab
  3. [§4.3 Efficiency Analysis / Table 1] The efficiency claims are unquantified where it matters. §4.3 and the abstract assert 'exceptionally low sequential token selection latency' and 'significantly accelerating downstream generative processing', and Table 1 reports FLOPs (1.76T vs 133.3T). But the method's selection loop is itself autoregressive — up to K=128 serial pointer-decode steps over N+1 memory slots — so wall-clock latency is not obviously favorable, and no latency, throughput, or memory measurements are reported anywhere. Figure 3 shows only token counts. At least one timing column (selection latency, end-to-end prefill+decode time per slide, vs SlideChat) is required to substantiate the practical-efficiency contribution; FLOPs alone do not capture the serial selection overhead.
  4. [§3.3–3.4 / Table 2] The train–inference mismatch is argued but not directly measured, and one ablation is conspicuously absent. Table 2 ablates the Denoiser mask and the score aggregation, but there is no ablation removing the VP noise gate / Denoiser pathway entirely (e.g., STE-only training feeding the soft-weighted features directly, or training with hard top-K + REINFORCE-style estimation). Since the entire methodological contribution rests on the VP-gate-plus-diagonal-Denoiser surrogate being faithful to the hard-drop deployment path (Eqs. 10–13), the paper should show that this machinery outperforms the simpler differentiable alternatives it replaces — otherwise the added complexity is unjustified. Relatedly, the Soft Top-K target k = running batch average Kb (Eq. 11) makes the training polarization target batch-composition-dependent; its stability should at least be commented on.
  5. [§4.2 bullet on Clinical Reasoning] The Clinical-subset improvement over the upper bound (+3.06%, 74.49% vs 71.43%) is presented as evidence that selection 'enhances' reasoning (§4.2), but Table 3 shows the Clinical column has std up to ±1.44 across runs, and Table 1 appears to be single-run. A +3.06 point gain over the uncompressed model is within noise at this reported variance and should be hedged or supported with significance testing; as stated it invites over-interpretation of a single seed.
minor comments (7)
  1. [§4.3 / Figure 4] Figure 4's claim of accuracy stability 'even in extreme high-sparsity scenarios where only 4 to 5 tokens are preserved' is not verifiable as presented: no per-bin sample counts or numeric values are given. Please annotate bin counts; stability in sparsely populated tail bins may be an artifact.
  2. [§4.1] Hyperparameters are under-specified for reproduction: values of β, τ, T_max, the g_txt annealing schedule, Scorer/Denoiser depth and width, and training epochs/optimizer settings do not appear in §4.1. No code release is mentioned. Please add an implementation-details table and state whether code will be released.
  3. [§3.2 / §4.3] Terminology: the text repeatedly calls the selection 'sequential token selection' with 'exceptionally low sequential token selection latency', but never defines the per-step cost relative to the N-sized memory; complexity of the pointer loop (O(T·(N+1)·D)) should be stated explicitly alongside the LLM prefill savings.
  4. [§3.4, Eq. (15)] Eq. (15): K is described as 'determined dynamically by the initial activation of the stop action' and the loss is said to backpropagate through the alive trajectory; please make explicit how the discrete count K enters the differentiable length penalty (presumably via Σ_t A_t), since as written the penalty term's gradient path is ambiguous.
  5. [§2, §4.3, Table 1] Several typos and formatting issues: double period after Eq. (10) reference in §4.3; 'LLaV A-Med' spacing in Table 1; 'a inflated budget' in §2.1 should be 'an inflated budget'; 'Pain Point 1/2/3' labels are introduced but only 1 and 3 are numbered in §2.1, with 2 appearing in §2.2 — consider consolidating.
  6. [§4.3 / Figure 5] The UMAP visualization (Figure 5) is suggestive but the interpretive claim — that marginal-occupancy of selected tokens reflects 'complementary pathological outliers' — is post hoc. Either soften the language or provide a quantitative counterpart (e.g., pathologist-annotated ROI overlap of selected patches, which would substantially strengthen the clinical-relevance argument).
  7. [§4.1 / Table 1] SlideBench (BCNB) and WSI-VQA* are described only as zero-shot; please state dataset sizes and whether the 'Random Baseline' row in Table 1 is chance-level accuracy for each benchmark, to help readers calibrate the ~24–25% values.

Circularity Check

1 steps flagged

Standard empirical ML paper; accuracy and token counts are measured on external benchmarks, not forced by definition or self-citation.

specific steps
  1. self citation load bearing [Sec. 2.2 Related Work; Refs. [12]–[15], [8]]
    "To decouple from the independent, one-shot top-K selection paradigm [47, 48, 49], our framework reformulates diagnostic token selection as an autoregressive sequential decision process. ... [12] Landi He et al. Beyond surrogate gradients... [13] ... noise gating. [14] Stepwise token selection... [15] Learnable token sparsification... [8] TC-SSA: Token compression via semantic slot aggregation..."

    A cluster of concurrent/related token-pruning and noise-gating preprints shares authors with the present paper and is used to position the sequential-selection framing. This is ordinary lineage citation, not a load-bearing uniqueness result: the accuracy and token-count claims are independently measured on SlideBench/WSI-VQA* and do not reduce to those citations. Flagged only as minor self-citation, not as forcing the main result.

full rationale

PathSelect’s central claims (74.00% SlideBench-TCGA overall accuracy under an empirical average of 44.86 tokens, ~36.6× reduction vs. frozen SlideChat’s ~1641-token average, competitive zero-shot on BCNB/WSI-VQA*) are ordinary held-out measurements against public-style baselines with the slide encoder and LLM frozen. The training path (alive-weighted soft scores → VP noise gate Eq. 12 → diagonal Denoiser Eq. 13 → LM loss + length penalty Eq. 15) and the inference path (Hard Top-K / stop token from the trained Scorer) are a standard continuous-relaxation design choice; hyperparameters λ, K, and temperature are swept and reported as ablations, not rebranded as predictions. Related-work citations include several arXiv preprints by overlapping authors on token pruning/noise gating, but none supply a uniqueness theorem or load-bearing premise that forces the reported numbers. No equation equates a fitted quantity to a claimed prediction by construction. Residual train–test selection mismatch is a correctness/transfer risk, not circularity. Score 1 only for the mild non-load-bearing self-citation cluster.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 2 invented entities

Load-bearing content is mostly standard deep-learning practice plus pathology sparsity assumptions and several hand-chosen training knobs. The central accuracy/compression claim rests on the SlideChat frozen prior, the Soft-to-Hard train/test surrogate, and λ/K/β/τ choices rather than on new physical entities.

free parameters (6)
  • length penalty λ (LP) = 2.0 (main config)
    Trades LM loss against K/N in Eq. 15; swept 0–6 and used to drive average tokens down to ~45.
  • max token budget K = 128
    Hard cap on selection length; main results use 128, with 64/256 ablations.
  • geometric depth decay β = in (0,1), exact value unspecified
    Discounts later pointer steps in soft score s_i (Eq. 10); ablated on/off but value not fully specified.
  • softmax / Soft Top-K temperature τ
    Controls sharpness of pointer and SoftTopK polarization; shared temperature, not reported numerically.
  • SoftTopK target k = running batch-average Kb = batch-average Kb
    Adaptive soft cardinality target tied to hard selection count during training.
  • text gate g_txt schedule = annealed in [0,1]
    Annealed gate on projected text tokens in joint encoder to stabilize early training.
axioms (5)
  • domain assumption Decisive diagnostic evidence in WSIs is extremely sparse and spatially diffuse (often <<1% of patches), so static independent top-K or random sampling loses critical cues.
    Motivates sequential complementary selection throughout Intro and Related Work; not re-proven here.
  • domain assumption A fully pre-trained SlideChat slide encoder + LLM already encodes sufficient multimodal alignment; only a router and projection bridge need training.
    Sec. 3–4 freeze both backbones; claim quality inherits SlideChat’s priors.
  • ad hoc to paper Diagonal-attention denoising prevents unselected tokens from reconstructing dropped semantics (no semantic leakage), aligning train noise path with inference hard drop.
    Sec. 3.3 and ablation Table 2(a); central to train–test consistency.
  • standard math Straight-through estimator through one-hot minus stop-grad(p)+p yields usable gradients for discrete pointer feedback (Eq. 9).
    Standard STE practice; assumed adequate for this scorer.
  • ad hoc to paper Variance-preserving isotropic Gaussian mixing (Eq. 12) is a valid continuous surrogate for hard token drop under SoftTopK α_i.
    Core PathSelect training mechanism; justified by continuity and ablation, not a theorem.
invented entities (2)
  • PathSelect plugin (VP noise gate + Soft Top-K + diagonal Denoiser + stop-token pointer Scorer) no independent evidence
    purpose: Provide end-to-end gradients for sequential WSI token routing while detaching to Hard Top-K at inference.
    Composite training/inference module introduced by the paper; evaluated only via downstream SlideBench metrics inside this work.
  • Alive-weighted geometrically decayed selection score s_i no independent evidence
    purpose: Aggregate sequential pointer probabilities into a soft importance vector for noise gating.
    Defined in Eq. 10; ablation shows full form beats last-step/uniform variants, but no external validation.

reviewed 2026-07-30 · how reviews work

0 comments
Cite this review

Pith. "Pith review of PathSelect: Sequential Token Selection for Whole Slide Pathology." pith.science (2026). https://pith.science/paper/KSEREAEY

@misc{pith2026260723631,
  author       = {Pith},
  title        = {Pith review of: PathSelect: Sequential Token Selection for Whole Slide Pathology},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KSEREAEY}},
  note         = {Machine review of arXiv:2607.23631}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Gigapixel Whole-Slide Images (WSIs) present a fundamental computational bottleneck for vision-language models (VLMs) due to extreme sequence lengths. Existing approaches predominantly rely on spatial sampling or training-free pruning, which risk diluting weak but informative signals, leading to the loss of critical diagnostic evidence due to the spatially diffuse nature of pathological cues. We reformulate WSI token pruning as a sequential selection process, enabling the model to autonomously learn an optimal routing strategy rather than relying on static heuristics. We herein propose a decoupled routing framework integrated as an active plugin into the fully pre-trained SlideChat base model, leaving both the slide encoder and large language model frozen. To provide continuous gradients for the non-differentiable pruning operation during training, we introduce PathSelect. PathSelect employs a variance-preserving noise gate to modulate each patch's information flow via a differentiable Soft Top-K operator, paired with a diagonal-attention Denoiser that recovers the perturbed representations without semantic leakage. At inference, the PathSelect module is entirely detached. Relying solely on the trained Scorer, a deterministic Hard Top-K operator executes adaptive, data-dependent trajectory termination, significantly accelerating downstream generative processing with exceptionally low sequential token selection latency. Driven by an empirical average of only 44.86 tokens under a maximum constraint of K = 128, our framework achieves 74.00% overall accuracy on SlideBench (TCGA), representing an approximate 36.6x spatial token reduction relative to the uncompressed baseline average while consistently outperforming sampling-based counterparts.

Figures

Figures reproduced from arXiv: 2607.23631 by Jingzhi Chen, Landi He, Lijian Xu, Peihang Wu, Zehong Chen.

Figure 1
Figure 1. Figure 1: Illustration of text-conditioned sequential token selection in PathSelect. The top panel presents a histology [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of our framework. Frozen encoders extract visual and textual features. Autoregressive Selection Loop: The joint cross-modal contextual representations are organized into a candidate token pool augmented with a virtual stop token sstop. Driven by a trainable conditional Scorer network, the pointer decoding trajectory sequentially selects prominent diagnostic regions step-by-step while calculating a… view at source ↗
Figure 3
Figure 3. Figure 3: Performance and token efficiency comparison. Our framework achieves competitive diagnostic accuracy using an empirical average of only 44.86 tokens under the LP = 2.0 configuration. This represents an approximate 36.6× spatial token reduction relative to the SlideChat [19] baseline average of 1, 641.87 tokens, while operating far below the maximum context ceiling of 4, 096 tokens. suppresses immediate spat… view at source ↗
Figure 4
Figure 4. Figure 4: Accuracy stability under LP = 2.0 training configuration. The downstream classification accuracy (orange line) exhibits remarkable stability across all intervals of dynamically selected tokens, demonstrating consistent diagnostic robustness independent of the adaptive sample-wise compression rates. dynamics of the pointer. (iii) Removing the geometric depth decay while retaining the termination probability… view at source ↗
Figure 5
Figure 5. Figure 5: UMAP visualizations of the latent feature space across three diverse WSIs. Unselected tokens cluster into dense centralized manifolds. In contrast, our autoregressive pointer uses the history of prior selections to dy￾namically update candidate scores, isolating complementary pathological feature outliers at the extreme topological margins [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. DiffPrune: differentiable information throttling for token pruning in vision-language models

    cs.CV 2026-08 conditional novelty 4.0

    DiffPrune replaces Gumbel-Softmax surrogate gradients with a differentiable noise-throttling path for visual token scoring, achieving high accuracy retention under aggressive pruning on three VLM families.

Reference graph

Works this paper leans on

54 extracted references · 10 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Scalar: Spatial- concept alignment for robust vision in harsh open world.Pattern Recognition, page 113203, 2026

    Xiaoyu Yang, Lijian Xu, Xingyu Zeng, Xiaosong Wang, Hongsheng Li, and Shaoting Zhang. Scalar: Spatial- concept alignment for robust vision in harsh open world.Pattern Recognition, page 113203, 2026

    2026

  2. [2]

    A unified multi-task framework enables interpretable chest radiograph analysis.Med, 2026

    Lijian Xu, Ziyu Ni, Xinglong Liu, Xiaosong Wang, Hongsheng Li, and Shaoting Zhang. A unified multi-task framework enables interpretable chest radiograph analysis.Med, 2026

    2026

  3. [3]

    A foundation model for generalizable disease diagnosis in chest x-ray images.arXiv preprint arXiv:2410.08861, 2024

    Lijian Xu, Ziyu Ni, Hao Sun, Hongsheng Li, and Shaoting Zhang. A foundation model for generalizable disease diagnosis in chest x-ray images.arXiv preprint arXiv:2410.08861, 2024

    Pith/arXiv arXiv 2024

  4. [4]

    Xrayclaw: Cooperative-competitive multi-agent alignment for trustworthy chest x-ray diagnosis.arXiv preprint arXiv:2604.02695, 2026

    Shawn Young and Lijian Xu. Xrayclaw: Cooperative-competitive multi-agent alignment for trustworthy chest x-ray diagnosis.arXiv preprint arXiv:2604.02695, 2026

    Pith/arXiv arXiv 2026

  5. [5]

    Clinical-grade computational pathology using weakly supervised deep learning on whole slide images.Nature medicine, 25(8):1301–1309, 2019

    Gabriele Campanella, Matthew G Hanna, Luke Geneslaw, et al. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images.Nature medicine, 25(8):1301–1309, 2019

    2019

  6. [6]

    A graph-transformer for whole slide image classification

    Yi Zheng, Rushin H Gindra, Emily J Green, et al. A graph-transformer for whole slide image classification. IEEE transactions on medical imaging, 41(11):3003–3015, 2022

    2022

  7. [7]

    Streaming convolutional neural networks for end-to- end learning with multi-megapixel images.IEEE transactions on pattern analysis and machine intelligence, 44(3):1581–1590, 2020

    Hans Pinckaers, Bram Van Ginneken, and Geert Litjens. Streaming convolutional neural networks for end-to- end learning with multi-megapixel images.IEEE transactions on pattern analysis and machine intelligence, 44(3):1581–1590, 2020

    2020

  8. [8]

    Tc-ssa: Token compression via semantic slot aggregation for gigapixel pathology reasoning.MICCAI, 2026

    Zhuo Chen, Shawn Young, and Lijian Xu. Tc-ssa: Token compression via semantic slot aggregation for gigapixel pathology reasoning.MICCAI, 2026

    2026

  9. [9]

    Scaling vision transformers to gigapixel images via hierar- chical self-supervised learning

    Richard J Chen, Cheng Chen, Yicong Li, et al. Scaling vision transformers to gigapixel images via hierar- chical self-supervised learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16144–16155, 2022

    2022

  10. [10]

    Dynamicvit: Efficient vision transformers with dynamic token sparsification

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. InAdvances in Neural Information Processing Systems, volume 34, pages 13937–13949, 2021

    2021

  11. [11]

    Token merging: Your ViT but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaodong Dai, Peize Zhang, and Judy Hoffman. Token merging: Your ViT but faster. InInternational Conference on Learning Representations, 2023

    2023

  12. [12]

    Beyond surrogate gradients: Fully differentiable token pruning for vision-language models.arXiv preprint arXiv:2605.28051, 2026

    Landi He, Mingde Yao, Shawn Young, and Lijian Xu. Beyond surrogate gradients: Fully differentiable token pruning for vision-language models.arXiv preprint arXiv:2605.28051, 2026

    Pith/arXiv arXiv 2026

  13. [13]

    The model knows which tokens matter:automatic token selection via noise gating.arXiv preprint arXiv:2603.07135, 2026

    Landi He, Xiaoyu Yang, and Lijian Xu. The model knows which tokens matter:automatic token selection via noise gating.arXiv preprint arXiv:2603.07135, 2026

    arXiv 2026

  14. [14]

    Stepwise token selection for efficient multimodal large language models

    Landi He, Shawn Young, and Lijian Xu. Stepwise token selection for efficient multimodal large language models. WAICA, 2026

    2026

  15. [15]

    Learnable token sparsification for efficient gigapixel whole slide image reasoning.arXiv preprint arXiv:2606.08641, 2026

    Jingzhi Chen, Landi He, Zhuo Chen, Shawn Young, and Lijian Xu. Learnable token sparsification for efficient gigapixel whole slide image reasoning.arXiv preprint arXiv:2606.08641, 2026

    Pith/arXiv arXiv 2026

  16. [16]

    Stochastic beams and where to find them: The gumbel-top- k trick for sampling sequences without replacement

    Wouter Kool, Herke Van Hoof, and Max Welling. Stochastic beams and where to find them: The gumbel-top- k trick for sampling sequences without replacement. InInternational conference on machine learning, pages 3499–3508. PMLR, 2019. 11 APREPRINT- JULY28, 2026

    2019

  17. [17]

    Estimating or propagating gradients through stochastic neurons for conditional computation.arXiv preprint arXiv:1308.3432, 2013

    Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation.arXiv preprint arXiv:1308.3432, 2013

    Pith/arXiv arXiv 2013

  18. [18]

    Learnpruner: Rethinking attention-based token pruning in vision language models.arXiv preprint arXiv:2604.23950, 2026

    Rinyoichi Takezoe, Yaqian Li, Zihao Bo, Anzhou Hou, Mo Guang, and Kaiwen Long. Learnpruner: Rethinking attention-based token pruning in vision language models.arXiv preprint arXiv:2604.23950, 2026

    Pith/arXiv arXiv 2026

  19. [19]

    Slidechat: A large vision-language assistant for whole-slide pathol- ogy image understanding

    Ying Chen, Guoan Wang, Yuanfeng Ji, et al. Slidechat: A large vision-language assistant for whole-slide pathol- ogy image understanding. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 5134–5143, 2025

    2025

  20. [20]

    One leaf reveals the season: Occlusion-based contrastive learning with semantic-aware views for efficient visual representation

    Xiaoyu Yang, Lijian Xu, Hongsheng Li, and Shaoting Zhang. One leaf reveals the season: Occlusion-based contrastive learning with semantic-aware views for efficient visual representation. InInternational Conference on Machine Learning, pages 71425–71440, 2025

    2025

  21. [21]

    Fewer tokens, greater scaling: Self-adaptive visual bases for efficient and expansive representation learning.arXiv preprint arXiv:2511.19515, 2026

    Shawn Young, Xingyu Zeng, and Lijian Xu. Fewer tokens, greater scaling: Self-adaptive visual bases for efficient and expansive representation learning.arXiv preprint arXiv:2511.19515, 2026

    arXiv 2026

  22. [22]

    Attention-based deep multiple instance learning

    Maximilian Ilse, Jakub Tomczak, and Max Welling. Attention-based deep multiple instance learning. InInter- national conference on machine learning, pages 2127–2136. PMLR, 2018

    2018

  23. [23]

    Data-efficient and weakly supervised computational pathology on whole-slide images.Nature biomedical engineering, 5(6):555–570, 2021

    Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, et al. Data-efficient and weakly supervised computational pathology on whole-slide images.Nature biomedical engineering, 5(6):555–570, 2021

    2021

  24. [24]

    Transmil: Transformer based correlated multiple instance learning for whole slide image classification.Advances in neural information processing systems, 34:2136–2147, 2021

    Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, et al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification.Advances in neural information processing systems, 34:2136–2147, 2021

    2021

  25. [25]

    Noriaki Hashimoto, Hiroyuki Hanada, Hiroaki Miyoshi, et al. Multimodal gated mixture of experts using whole slide image and flow cytometry for multiple instance learning classification of lymphoma.Journal of Pathology Informatics, 15:100359, 2024

    2024

  26. [26]

    Learning heterogeneous tissues with mixture of experts for gigapixel whole slide images

    Junxian Wu, Minheng Chen, Xinyi Ke, Tianwang Xun, Xiaoming Jiang, Hongyu Zhou, Lizhi Shao, and Youyong Kong. Learning heterogeneous tissues with mixture of experts for gigapixel whole slide images. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 5144–5153, 2025

    2025

  27. [27]

    Multi-modal gated mixture of local-to-global experts for dynamic image fusion

    Bing Cao, Yiming Sun, Pengfei Zhu, and Qinghua Hu. Multi-modal gated mixture of local-to-global experts for dynamic image fusion. InProceedings of the IEEE/CVF international conference on computer vision, pages 23555–23564, 2023

    2023

  28. [28]

    Feature re-embedding: Towards foundation model-level performance in computational pathology

    Wenhao Tang, Fengtao Zhou, Sheng Huang, et al. Feature re-embedding: Towards foundation model-level performance in computational pathology. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11343–11352, 2024

    2024

  29. [29]

    Revisiting end-to-end learning with slide-level supervision in computational pathology.Advances in Neural Information Processing Systems, 38:160279–160312, 2026

    Wenhao Tang, Rong Qin, Heng Fang, Fengtao Zhou, Hao Chen, Xiang Li, and Ming-Ming Cheng. Revisiting end-to-end learning with slide-level supervision in computational pathology.Advances in Neural Information Processing Systems, 38:160279–160312, 2026

    2026

  30. [30]

    Towards a general-purpose foundation model for computational pathology.Nature medicine, 30(3):850–862, 2024

    Richard J Chen, Tong Ding, Ming Y Lu, et al. Towards a general-purpose foundation model for computational pathology.Nature medicine, 30(3):850–862, 2024

    2024

  31. [31]

    A whole-slide foundation model for digital pathology from real-world data.Nature, 630(8015):181–188, 2024

    Hanwen Xu, Naoto Usuyama, Jaspreet Bagga, et al. A whole-slide foundation model for digital pathology from real-world data.Nature, 630(8015):181–188, 2024

    2024

  32. [32]

    Multimodal model for computational pathology: Representation learning and image compression.arXiv preprint arXiv:2603.18660, 2026

    Peihang Wu, Zehong Chen, and Lijian Xu. Multimodal model for computational pathology: Representation learning and image compression.arXiv preprint arXiv:2603.18660, 2026

    arXiv 2026

  33. [33]

    Medvilam: A multimodal large language model with advanced generalizability and explainability for medical data understanding and generation.arXiv preprint arXiv:2409.19684, 2024

    Lijian Xu, Hao Sun, Ziyu Ni, Hongsheng Li, and Shaoting Zhang. Medvilam: A multimodal large language model with advanced generalizability and explainability for medical data understanding and generation.arXiv preprint arXiv:2409.19684, 2024

    Pith/arXiv arXiv 2024

  34. [34]

    A visual–language foun- dation model for pathology image analysis using medical twitter.Nature medicine, 29(9):2307–2316, 2023

    Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas J Montine, and James Zou. A visual–language foun- dation model for pathology image analysis using medical twitter.Nature medicine, 29(9):2307–2316, 2023

    2023

  35. [35]

    A visual-language foundation model for computational pathology.Nature medicine, 30(3):863–874, 2024

    Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, et al. A visual-language foundation model for computational pathology.Nature medicine, 30(3):863–874, 2024

    2024

  36. [36]

    Cpath-omni: A unified multimodal foundation model for patch and whole slide image analysis in computational pathology

    Yuxuan Sun, Yixuan Si, Chenglu Zhu, et al. Cpath-omni: A unified multimodal foundation model for patch and whole slide image analysis in computational pathology. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10360–10371, 2025

    2025

  37. [37]

    Segmentation and vascular vectorization for coronary artery by geometry-based cascaded neural network.IEEE Transactions on Medical Imaging, 44(1):259–269, 2024

    Xiaoyu Yang, Lijian Xu, Simon Yu, Qing Xia, Hongsheng Li, and Shaoting Zhang. Segmentation and vascular vectorization for coronary artery by geometry-based cascaded neural network.IEEE Transactions on Medical Imaging, 44(1):259–269, 2024. 12 APREPRINT- JULY28, 2026

    2024

  38. [38]

    Geometry-based end- to-end segmentation of coronary artery in computed tomography angiography

    Xiaoyu Yang, Lijian Xu, Simon Yu, Qing Xia, Hongsheng Li, and Shaoting Zhang. Geometry-based end- to-end segmentation of coronary artery in computed tomography angiography. InInternational Workshop on Trustworthy Machine Learning for Healthcare, pages 190–196. Springer, 2023

    2023

  39. [39]

    Efficient chest x-ray representation learning via semantic- partitioned contrastive learning.arXiv preprint arXiv:2603.07113, 2026

    Wangyu Feng, Shawn Young, and Lijian Xu. Efficient chest x-ray representation learning via semantic- partitioned contrastive learning.arXiv preprint arXiv:2603.07113, 2026

    arXiv 2026

  40. [40]

    When tokens talk too much: A survey of multimodal long-context token compression across images, videos, and audios.arXiv preprint arXiv:2507.20198, 2025

    Kele Shao, Keda Tao, Kejia Zhang, et al. When tokens talk too much: A survey of multimodal long-context token compression across images, videos, and audios.arXiv preprint arXiv:2507.20198, 2025

    arXiv 2025

  41. [41]

    Mmtok: Multimodal coverage maximization for efficient inference of vlms.International Conference on Learning Representations, 2026

    Sixun Dong, Juhua Hu, Mian Zhang, et al. Mmtok: Multimodal coverage maximization for efficient inference of vlms.International Conference on Learning Representations, 2026

    2026

  42. [42]

    Beyond attention or similarity: Maximizing conditional diversity for token pruning in mllms.Advances in Neural Information Processing Systems, 38:25438–25468, 2026

    Qizhe Zhang, Mengzhen Liu, Lichen Li, et al. Beyond attention or similarity: Maximizing conditional diversity for token pruning in mllms.Advances in Neural Information Processing Systems, 38:25438–25468, 2026

    2026

  43. [43]

    Dynamic-llava: Efficient multimodal large language models via dynamic vision-language context sparsifi- cation

    Wenxuan Huang, Zijie Zhai, Yunhang Shen, Shaosheng Cao, Fei Zhao, Xiangfeng Xu, Zheyu Ye, and Shaohui Lin. Dynamic-llava: Efficient multimodal large language models via dynamic vision-language context sparsifi- cation. InInternational Conference on Learning Representations, volume 2025, pages 69927–69955, 2025

    2025

  44. [44]

    Hidrop: Hierarchical vision token reduction in mllms via late injection, concave pyramid pruning, and early exit.International Conference on Learning Representations, 2026

    Hao Wu, Yingqi Fan, Jinyang Dai, Junlong Tong, Yunpu Ma, and Xiaoyu Shen. Hidrop: Hierarchical vision token reduction in mllms via late injection, concave pyramid pruning, and early exit.International Conference on Learning Representations, 2026

    2026

  45. [45]

    Visionzip: Longer is better but not necessary in vision language models

    Senqiao Yang, Yukang Chen, Zhuotao Tian, et al. Visionzip: Longer is better but not necessary in vision language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19792– 19802, 2025

    2025

  46. [46]

    Categorical reparameterization with gumbel-softmax.arXiv preprint arXiv:1611.01144, 2016

    Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax.arXiv preprint arXiv:1611.01144, 2016

    Pith/arXiv arXiv 2016

  47. [47]

    Loc-path: Learning to compress for pathology multimodal large language models.arXiv preprint arXiv:2512.05391, 2025

    Qingqiao Hu, Weimin Lyu, Meilong Xu, et al. Loc-path: Learning to compress for pathology multimodal large language models.arXiv preprint arXiv:2512.05391, 2025

    arXiv 2025

  48. [48]

    Wsisum: Wsi summarization via dual-level semantic reconstruc- tion.Medical Image Analysis, page 103970, 2026

    Baizhi Wang, Kun Zhang, Yuhao Wang, et al. Wsisum: Wsi summarization via dual-level semantic reconstruc- tion.Medical Image Analysis, page 103970, 2026

    2026

  49. [49]

    Focus: Knowledge-enhanced adaptive visual compression for few-shot whole slide image classification

    Zhengrui Guo, Conghao Xiong, Jiabo Ma, et al. Focus: Knowledge-enhanced adaptive visual compression for few-shot whole slide image classification. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 15590–15600, 2025

    2025

  50. [50]

    Wsi-vqa: Interpreting whole slide images by generative visual question answering

    Pingyi Chen, Chenglu Zhu, Sunyi Zheng, Honglin Li, and Lin Yang. Wsi-vqa: Interpreting whole slide images by generative visual question answering. InEuropean Conference on Computer Vision, pages 401–417. Springer, 2024

    2024

  51. [51]

    Llava-med: Training a large language-and-vision assistant for biomedicine in one day.Advances in Neural Information Processing Systems, 36:28541–28564, 2023

    Chunyuan Li, Cliff Wong, Sheng Zhang, et al. Llava-med: Training a large language-and-vision assistant for biomedicine in one day.Advances in Neural Information Processing Systems, 36:28541–28564, 2023

    2023

  52. [52]

    Quilt-llava: Visual instruction tuning by extracting localized narratives from open-source histopathology videos

    Mehmet Saygin Seyfioglu, Wisdom O Ikezogwo, Fatemeh Ghezloo, et al. Quilt-llava: Visual instruction tuning by extracting localized narratives from open-source histopathology videos. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13183–13192, 2024

    2024

  53. [53]

    Meddr: Diagnosis-guided bootstrapping for large-scale medical vision-language learning.arXiv preprint arXiv:2404.15127, 1(3):6, 2024

    Sunan He, Yuxiang Nie, Zhixuan Chen, Zhiyuan Cai, Hongmei Wang, Shu Yang, and Hao Chen. Meddr: Diagnosis-guided bootstrapping for large-scale medical vision-language learning.arXiv preprint arXiv:2404.15127, 1(3):6, 2024

    Pith/arXiv arXiv 2024

  54. [54]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024. 13

    Pith/arXiv arXiv 2024

This paper was first reviewed by grok-4.5 on July 30, 2026.