{"id":"e4c3729b-98cc-438d-800d-a9fc9bfad77d","arxiv_id":"2608.10989","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A learned task register per task, instead of one global pruning rule, coordinates token selection, depth-wise budget allocation, and dense recovery, yielding 47.0 mIoU and 53.7 box AP at 1.3 times encoder speedup on ADE20K and COCO.","lead":"This paper introduces task registers, one learned token per task, that control token pruning in a shared Vision Transformer by deciding which patches to keep, how to spread a fixed removal budget across layers, and how to reconstruct pruned positions for dense tasks. It reports that this task-adaptive pruning keeps accuracy nearly intact at about 1.3 times encoder speedup on ADE20K segmentation and COCO detection, suggesting a path toward efficient multi-task vision serving.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Probe-based cross-task conclusions are confounded by resolution, head, and eligible-layer differences; the caveat appears in the main Controlled-study text, but the abstract and Figure 1 still overstate task semantics.","rationale":"The reader's weakest assumption correctly identifies a real confound in the controlled-probe comparisons, and this is the most load-bearing concern about the paper's stated motivation. I flag one factual correction: the caveat that probes compare complete pipelines rather than task semantics alone appears in the main Controlled-study paragraph, not only in Supplement J. The concern still lands because the abstract and Figure 1 are presented without that caveat, and the ablations in Table 4 do not isolate task semantics from resolution/head/layer-eligibility effects. The central quantitative claims for TAP-J at rho=0.5 are direct empirical results and are supported by careful controls: the exact-budget property is proved in Supplement A.3, paired recovery is properly seeded, and ablation tables show the register-driven choices matter. The probe confound therefore does not justify rejection, but it does support a conditional verdict with the probe interpretation softened or a matched-pipeline check added. Since the reader already reached CONDITIONAL, my recommendation is UNCHANGED.","tokens_in":44,"tokens_out":7888,"duration_ms":205024,"concrete_test":"Re-run the controlled-probe protocol for detection with the two largest confounds removed: use ADE20K's 512 input resolution and make pruning eligible at blocks 3, 6, and 9 by replacing ViTDet's early window attention with global attention (or equivalently, run the segmentation probe at 1024 with global-only pruning). Keep the native heads and metrics. Recompute Table S9's Spearman correlation between segmentation and detection criterion rankings and Table S10's recovery-endpoint preferences. If the negative correlation and opposite recovery endpoints persist under the matched resolution and eligible-layer schedule, the task-semantics interpretation survives; if they weaken, flip, or vanish, the probe differences are pipeline artifacts and the abstract's 'three differences' claim should be softened to 'differences across the evaluated pipelines.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central motivation is the controlled-probe finding that segmentation and detection rank pruning criteria differently and prefer opposite recovery endpoints. The probes compare complete pipelines: ImageNet at 224 with a linear head and pruning at blocks 3/6/9; ADE20K at 512 with a SegFormer MLP head at blocks 3/6/9; COCO at 1024 with Cascade Mask R-CNN and pruning only at global-attention blocks 3/6/9 (plus 12 in the probe). The Spearman rank correlation of -0.62 and the recovery-endpoint reversal in Table S10 could therefore be driven by the detector's higher resolution, its global-only pruning schedule, or its head geometry rather than by task semantics. The main text does contain the caveat 'the probes compare complete pipelines rather than task semantics alone' in the Controlled-study paragraph; the reader's claim that this appears only in Supplement J is inaccurate. However, the abstract and Figure 1 present the findings without that caveat, and the probe-based 'three differences' claim remains load-bearing for the paper's stated motivation. Table 4 ablations show that task-conditioned selection, allocation, and recovery help under TAP, but they do not attribute that help specifically to task semantics as opposed to pipeline geometry. Thus the confound weakens the abstract's interpretive claim, even though the main quantitative results for TAP-J are not directly undermined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies whether token-pruning policies transfer across image classification, semantic segmentation, and object detection with a ViT backbone. It first runs controlled probes on frozen no-pruning checkpoints, applying parameter-free reduction criteria one layer at a time, and reports three task-level differences: dense tasks rank criteria differently, classification is especially sensitive to attention-based pruning in early layers, and the dense pipelines prefer opposite recovery endpoints. These findings motivate Task-Adaptive Pruning (TAP), in which a learned per-task register token propagates through the backbone and conditions token scoring, layerwise removal allocation, and dense-feature recovery. Two regimes are evaluated: TAP-J (per-task full fine-tuning) and TAP-F (frozen shared backbone with LoRA and task registers). At a final keep rate of 0.5, TAP-J reaches 47.0 mIoU at 1.30x encoder throughput on ADE20K and 53.7 box AP at 1.32x encoder throughput on COCO, with 83.2 top-1 accuracy at 1.26x on ImageNet-1K; TAP-F reports larger drops. The supplement contains an exact-budget proof, unit tests, verification tables, and extensive ablations.","tokens_in":31719,"tokens_out":12593,"duration_ms":119565,"significance":"If the results hold, the paper provides a practical and unusually lightweight token-pruning controller. The exact-budget guarantee is proved in Proposition 1 and supported by unit tests, the main operating points are verified in Table S11, and the empirical package is strong: multiple seeds with standard deviations, calibration-split baselines, provenance tables for baselines, and a claim-to-evidence map in Table S23. The dense-task results are competitive with fully fine-tuned methods at about 1.3x encoder speedups. However, the central interpretive claim that pruning behavior differs by task semantics is not established by the evidence as presented, because the probes compare complete pipelines with different resolutions, heads, and eligible-layer schedules, and the headline TAP-J result does not demonstrate cross-task sharing because it trains a separate backbone per task. The paper's value lies more in the controlled comparison and the TAP controller than in the causal task-semantics narrative.","major_comments":[{"comment":"The probe evidence in Figure 1 and Tables S9 and S10 is presented as revealing differences between tasks ('segmentation and detection rank the criteria differently', 'the dense tasks prefer opposite recovery endpoints'), but the probes simultaneously vary input resolution (224 vs 512 vs 1024), task head (linear, SegFormer MLP, Cascade Mask R-CNN), and eligible pruning layers (blocks 3/6/9 for classification and segmentation; only global-attention blocks 3/6/9/12 for detection). The Spearman correlation of -0.62 and the recovery-endpoint reversal in Table S10 therefore measure complete-pipeline differences, not task semantics. The main controlled-study paragraph does contain the caveat 'compare complete pipelines rather than task semantics alone,' but the abstract, the contribution list, and Figure 1 do not carry that caveat and instead use causal task language. Because these probe findings are the paper's stated motivation and first contribution, the authors should either add a matched-pipeline control (for example, fixing resolution and pruning-layer schedule across tasks while varying the head/objective) or consistently reframe all probe claims as pipeline-level transfer results rather than task-level ones.","section":"Abstract; Figure 1; Controlled study; Supplement J"},{"comment":"The sentence 'The largest gains occur on the dense tasks, linking the conflicts identified in Figure 1(b-d) to TAP's task-conditioned decisions' is not supported by the comparisons in Tables 1-3. TAP-J is fully fine-tuned per task, whereas the static rule is parameter-free and is not fine-tuned, so the comparison conflates task-conditioning with the benefit of fine-tuning. The ablations in Table 4 are conducted in the frozen-base TAP-F regime and show relative gains, but TAP-F itself loses 1.0 top-1, 1.4 mIoU, and 1.5 box AP against no pruning and is worse than the static rule on classification and detection. To attribute the observed gains to the task-conditioned decisions, the paper needs fully fine-tuned controls, such as a fine-tuned attention top-k policy, a fine-tuned static layer schedule, and a TAP-J variant whose register readouts are replaced by fixed or task-mean policies.","section":"Experiments, Main comparison; Tables 1-3; Figure 4"},{"comment":"The paper's headline model TAP-J is not a shared-backbone model, since it trains a separate backbone, register, and readouts for each task. The only shared-backbone model is TAP-F, whose published operating points are substantially weaker (Table S12: TAP-F loses 1.0 top-1, 1.4 mIoU, and 1.5 box AP at rho=0.5). This weakens the framing in the abstract and introduction that pretrained ViTs are 'reused across tasks' and that TAP is a mechanism for unified vision systems. The authors should either present TAP-J explicitly as a per-task upper bound and TAP-F as the cross-task result, or add a cross-task training variant that yields competitive numbers while using a genuinely shared backbone.","section":"Introduction; Method; Tables 1-3 and S12"}],"minor_comments":[{"comment":"The labels 'seg needs coverage' and 'det needs attention' in Figure 1(b) are causal and task-level; add a qualifier such as 'under these task pipelines' so the caption does not contradict the caveat later in the Controlled-study paragraph.","section":"Figure 1 caption and panel (b)"},{"comment":"Please clarify the batch size used for the per-image component latencies in Table S20; as written, the segmentation end-to-end value of 5.58 ms per image appears inconsistent with the 280 im/s encoder throughput reported in Table 1, and the table should state whether it uses the throughput protocol's batch size or a single-image latency protocol.","section":"Supplement, Table S20"},{"comment":"The Spearman correlation of -0.62 is computed over only 11 criteria whose scores share the same underlying data; report a permutation-based confidence interval or p-value to indicate the stability of the rank reversal.","section":"Supplement, Table S9"},{"comment":"The supplement correctly notes that adding an early-layer offset to a later endpoint 'is an approximation, since the intervening blocks do not provide an explicit transport map'; the main text should at least point to this limitation where Eq. (3) is introduced, since it is a core assumption of the recovery mechanism.","section":"Method, Eq. (3) and Supplement A.5"},{"comment":"The name TAP-J ('jointly adapted') is confusing because the model trains a separate backbone per task; state in the abstract or method that 'joint' refers to jointly fine-tuning the backbone, register, and readouts within each task, not to joint training across tasks.","section":"Abstract and Method"}],"recommendation":"major_revision","confidential_remarks":"The quantitative core of the paper appears sound and the reproducibility package is a clear strength. The main risk is overclaiming causal task-semantics conclusions from confounded probes and presenting per-task fine-tuned TAP-J as the central evidence for a task-adaptive controller. I would not reject, because the exact-budget guarantee and the dense-task operating points are valuable, but the framing and at least one control experiment need to be corrected before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a well-executed token-pruning paper that does something genuinely new — a per-task learned register that controls selection, depth-wise budget, and dense recovery — and backs it with a proved exact-budget allocation and a thorough experimental protocol. The main quantitative claims hold up. But the probe study that motivates the design is less controlled than the abstract implies, and despite describing an anonymous artifact, they ship no code or commit hash.\n\nWhat's new: the task register is a real mechanism, not a rehash. One learned vector per task, only the active one enters the backbone, and its evolving state decides which tokens to drop, how to split the global budget across layers, and how much of a stored feature offset to restore at dense readout. The exact-budget property is proved (Proposition 1) and unit-tested. The headline results are strong: TAP-J hits 47.0 mIoU on ADE20K (baseline 47.2) and 53.7 box AP on COCO (baseline 54.0) at roughly 1.3x encoder speedup, with only 2.3K pruning parameters. That is a genuinely useful operating point for dense ViT pipelines. The probe methodology — freezing a no-pruning checkpoint and varying one pruning choice at one layer at a time — is a nice empirical contribution in itself. They also report seeds, standard deviations, calibration-split baselines, and a claim-to-evidence map in the supplement.\n\nSoft spots: the cross-task probe comparison compares complete pipelines that differ in input resolution (224 vs 512 vs 1024), head geometry (linear, SegFormer MLP, Cascade Mask R-CNN), and eligible pruning layers (including global-only for detection). The main Controlled-Study paragraph does say the probes \"compare complete pipelines rather than task semantics alone,\" so the reader's complaint that the caveat appears only in Supplement J is inaccurate. But the abstract and Figure 1 present the \"three differences\" as if they reflect task semantics, and that is an overstatement. The rank reversal and recovery-endpoint flip could be driven by resolution or head or global-only pruning. This weakens the story, not the numbers: the TAP-J results stand independently of the probe interpretation. The bigger practical issue is that no public code or commit hash is provided. For a methods paper with this much pipeline complexity, that is a real reproducibility gap. TAP-F, the frozen-shared-backbone variant, loses 1–1.5 points on each task — fine for a shared-base offering, but clearly secondary.\n\nWho should read it: anyone working on efficient ViTs, especially for dense prediction, and researchers interested in task-conditioned computation. It deserves a serious referee; I would not desk-reject it. My recommendation: send it to review and ask the authors to soften the cross-task semantic claims, actually release the code, and ideally add a probe that keeps resolution and head fixed across tasks. After those revisions it would be a solid accept.","headline":"A genuinely new task-register pruning mechanism with strong dense-task results, but the cross-task probe story is exaggerated and the code is not actually released.","tokens_in":32291,"tokens_out":3357,"would_cite":true,"duration_ms":33476,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Token-pruning choices differ across vision tasks, and a single evolving task register can adapt them, keeping dense predictions close to the full model at 1.3x encoder speed.","keywords":["token pruning","vision transformer","task register","semantic segmentation","object detection","image classification","adaptive computation","efficient inference"],"falsifier":"Run the same controlled probe on segmentation and detection at a matched input resolution with the same eligible pruning layers; if the rank correlation between criterion drops is no longer negative, the paper's motivating task-conflict claim collapses.","tokens_in":31167,"feed_emoji":"⚡","tokens_out":10944,"duration_ms":83490,"temperature":0.7,"pith_summary":"Token pruning in Vision Transformers is usually designed as a single policy, but pretrained backbones are reused across tasks with very different spatial demands. This paper asks which pruning decisions transfer across image classification, semantic segmentation, and object detection, and answers with controlled probes: the same parameter-free criteria rank differently per task, classification is especially hurt by attention-based pruning in the earliest layers, and the two dense tasks prefer opposite ways to reconstruct removed positions. On the strength of those findings it proposes Task-Adaptive Pruning (TAP), which gives each task its own learned register token whose evolving state selects tokens, distributes an exact removal budget over depth, and sets a recovery scale for dense features. At a keep rate of half the tokens, the jointly fine-tuned TAP-J drops 0.2 mIoU on ADE20K and 0.3 box AP on COCO while accelerating the encoder about 1.3x, with ImageNet-1K accuracy competitive.","feed_headline":"Prune half the tokens, lose 0.3 points, run 1.3x faster","feed_subtitle":"A per-task register picks surviving tokens and rebuilds pruned ones, keeping dense-task drops within 0.3 points.","key_machinery":"The central object is the task register: one learned $d$-dimensional vector per task that is the only task-specific token allowed into the shared backbone. It evolves through ordinary transformer blocks, and at each pruning layer its normalized state is turned into a selection score $f_i^{(l)}$ by dotting the register query against each patch key using the block's existing projections, a per-layer removal count through a sigmoid readout $w^\\top r_t^{(l)} + b$ that provably spends the exact global budget, and a recovery scale $\\alpha_t^{(l)} = \\sigma(u^\\top r_t^{(l)} + v)$ used only at dense readouts. Removed tokens are matched to the retained token with the highest key-space cosine similarity, and their stored feature offset $\\delta_i$ is added back scaled by $\\alpha_t^{(l)}$, keeping reconstructed features out of later encoder blocks. The register is what lets one sparse backbone carry task-dependent computation without per-task selectors.","core_discovery":"The paper's central claim is that token reduction cannot be treated as a single transferable rule across vision tasks. Controlled probes freeze each no-pruning pipeline and change one pruning choice at one eligible layer at a time; they show that segmentation and detection rank eleven criteria almost oppositely (Spearman rank correlation -0.62), that attention-based selection causes larger drops than spatial coverage in the earliest classification layers, and that segmentation benefits from restoring the full stored feature offset at removed positions while detection prefers the surviving stand-in alone. TAP turns these observations into a mechanism: one learned task register per task is activated per forward pass, and its state drives three decisions: register-to-patch scoring for token selection (reusing the block's query and key projections, so no extra projection parameters), an exact integer allocation of the global removal budget across pruning layers, and a per-layer recovery scale $\\alpha_t^{(l)}$ that interpolates between the stand-in endpoint and the full offset. The reported consequence is that at $\\rho=0.5$ TAP-J reaches 47.0 mIoU at $1.30\\times$ encoder throughput on ADE20K and 53.7 box AP at $1.32\\times$ on COCO while remaining competitive on ImageNet-1K.","pith_inferences":["The probe comparison bundles task semantics with pipeline differences such as input resolution, head architecture, and eligible layers, so a matched-pipeline study would be needed to attribute the ranking conflict purely to the task.","If the opposite recovery endpoints generalize, other dense tasks such as depth estimation, video segmentation, or panoptic segmentation should learn their own reconstruction rule rather than reusing one endpoint.","Because the task register is a tiny interface and scoring reuses existing projections, the same recipe could be probed for adding a new task to a frozen backbone without retraining the pruning controller."],"forward_implications":["A single fixed pruning policy leaves measurable accuracy on the table on dense tasks; the same backbone can support task-specific pruning paths with only a few thousand extra parameters.","At half the tokens, dense predictions stay within 0.2 to 0.3 points of the unpruned model while the encoder runs about 1.3 times faster.","One mechanism can coordinate which tokens survive, where the budget is spent, and how removed positions are rebuilt, and it can be trained exactly to a target token count without auxiliary rate losses.","A frozen shared backbone with per-task registers and low-rank updates is a workable multi-task operating point, at the cost of a larger accuracy drop than per-task fine-tuning."],"supporting_citations":[{"why":"Supplies the register-token idea that TAP changes from passive artifact storage into an active pruning controller.","marker":"(Darcet et al. 2024)"},{"why":"Provides the attention-based pruning criterion used as a representative probe and the EViT classification baseline.","marker":"(Liang et al. 2022)"},{"why":"Provides DynamicViT, the learned-selector baseline that motivates the comparison against parameter-free criteria.","marker":"(Rao et al. 2021)"},{"why":"Supplies the MAE-pretrained ViT-B checkpoints that all three task pipelines start from.","marker":"(He et al. 2022)"},{"why":"Defines the SegFormer MLP head used for semantic segmentation in the ADE20K pipeline.","marker":"(Xie et al. 2021)"},{"why":"Defines ViTDet, whose global and windowed attention structure determines where detection can prune.","marker":"(Li et al. 2022)"},{"why":"Supplies the Cascade Mask R-CNN head used in the COCO detection pipeline.","marker":"(Cai and Vasconcelos 2018)"},{"why":"Provides the low-rank updates that let TAP-F adapt a frozen shared backbone per task.","marker":"(Hu et al. 2022)"},{"why":"Supplies the Gumbel perturbations used in the cardinality-constrained training of token selection.","marker":"(Jang, Gu, and Poole 2017)"},{"why":"Provides the straight-through gradient estimator that allows hard top-k selection to be trained.","marker":"(Bengio, Léonard, and Courville 2013)"}],"fun_headline_variants":["Task registers tailor token pruning for each vision task","One register per task: smarter token pruning, 1.3x faster","Adaptive pruning keeps dense tasks within 0.3 points at half tokens","Token pruning that adapts: up to 1.32x faster with minimal loss"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the differences in criterion ranking and recovery preference come from the tasks themselves, not from the fact that the segmentation and detection pipelines differ in input resolution, head, and eligible pruning layers.","fun_headline_variants_meta":{"raw":{"variants":["Task registers tailor token pruning for each vision task","One register per task: smarter token pruning, 1.3x faster","Adaptive pruning keeps dense tasks within 0.3 points at half tokens","Token pruning that adapts: up to 1.32x faster with minimal loss"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000251,"raw_usage":{"total_tokens":1609,"prompt_tokens":1047,"completion_tokens":562,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":663,"completion_tokens_details":{"reasoning_tokens":483}},"tokens_in":663,"tokens_out":562,"duration_ms":4767,"temperature":1.0,"reasoning_tokens":483,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:41:44.858762+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same controlled probe on segmentation and detection at a matched input resolution with the same eligible pruning layers; if the rank correlation between criterion drops is no longer negative, the paper's motivating task-conflict claim collapses.","supporting_citations":[],"review_version":1}