REVIEW 3 major objections 5 minor 42 references
Memory Efficient Matting with Adaptive Token Routing
T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read This paper claims that a content-aware router can send only the informative image tokens through global attention in a matting transformer, cutting peak memory by about 88% and latency by about 50% while keeping matte quality close to the…
desk verdict Memory and latency savings check out and the new dataset is a real contribution, but the router's 'global' feature is constant across tokens and cannot influence routing, so the local-global story does not hold. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the combination of the router, BATR, and the Lightweight Token Refinement Module (LTRM) acting inside each transformer block. The router computes a per-token log-probability from a local feature (the first half of a projected token) and a global feature (the average of the second half over all tokens), so each scoring decision has both fine-grained and context information; BATR makes the final 0/1 decision, reselecting every token at every block so no information is permanently discarded, with only a batch-level compression loss steering the average attention ratio; and the LTRM updates the non-routed tokens with depthwise convolution and efficient channel attention so they are refined rather than skipped. An inference-time cap on the number of attended tokens is what ultimately lets ultra high-resolution images fit on a single consumer GPU.
What would settle it
Retrain MEMatte with the learned router replaced by random routing that sends the same average fraction of tokens to global attention; if the sum-of-absolute-differences matte error (SAD) on Composition-1K stays near 21.90, the router's scoring is not what preserves quality, whereas a large SAD jump would confirm the routing decision is load-bearing.
Extended reading notes
Core claim
The central claim is that adaptive token routing can replace full global self-attention in image matting without a meaningful quality cost. Each router predicts a two-class routing probability from a local token feature concatenated with a globally pooled feature; the Batch-constrained Adaptive Token Routing mechanism (BATR) turns those probabilities into per-block binary decisions with Gumbel-Softmax during training and argmax at inference, and it constrains only the batch-level average attention ratio, so the attended token set adapts to image content and network stage. The non-attended tokens are processed by the Lightweight Token Refinement Module, a depthwise-convolution-plus-channel-attention block taught by distillation to imitate global attention. On the Composition-1K benchmark, MEMatte-S is reported to drop peak memory from 6.20 GB to 0.71 GB and latency from 186 ms to 84.99 ms while raising SAD from 21.46 to 21.90, and on the new UHR-395 and PPT-344 high-resolution tests the full-resolution outputs beat downsampling and crop-and-stitch variants of earlier methods.
Load-bearing premise
The efficiency gain rests on the router's local-global scoring reliably identifying which tokens actually need global attention; if it misroutes informative tokens, matte quality degrades and the memory savings come at the cost of accuracy.
Editorial extensions
If this is right
- If the reported numbers hold, full-resolution matting of 4K and 8K images becomes feasible on consumer GPUs, eliminating the artifacts of downsampling and crop-and-stitch pipelines.
- Because the routing ratio is content- and stage-adaptive rather than fixed, the method distributes compute where it is needed instead of paying a flat pruning cost on every image.
- The ablation indicates that the quality retention depends on distillation and LTRM working together; using either alone does not close the gap to the teacher.
- The UHR-395 dataset provides a high-resolution benchmark where many prior matting models run out of memory, so future efficiency claims can be tested at the resolution that motivated the method.
Reading between the lines
- Beyond the paper, the same local-global router design could transfer to other dense prediction tasks with full-resolution inputs, such as segmentation, depth estimation, or document analysis, wherever a few informative tokens need global context and the rest can be refined locally.
- Beyond the paper, the router's scores could serve as a built-in saliency or uncertainty map, since the visualizations indicate that attention is concentrated on object boundaries and fine textures; that could be tested by correlating the scores with boundary annotations.
- Beyond the paper, the inference-time maximum token count k is a single memory-quality knob, and one could extend MEMatte by predicting k per image instead of setting it globally, lowering memory further on easy images.
- Beyond the paper, combining adaptive routing with a hierarchical or sparse-attention backbone could compound the savings, since the two mechanisms reduce cost at different scales.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes MEMatte, a ViT-based matting framework that inserts a router before each global self-attention block. The router predicts, for every token, whether it should be processed by global attention or by a lightweight token refinement module (LTRM); a batch-constrained mechanism adapts the routed fraction rather than using a fixed ratio, and a distillation loss transfers knowledge from a frozen ViTMatte teacher. The paper also introduces UHR-395, a large ultra-high-resolution matting dataset. On Composition-1K, MEMatte-S/B report roughly 88% memory reduction and about 50% latency reduction relative to ViTMatte, with small SAD increases, and it reports improved results on high-resolution and real-world benchmarks.
Significance. If the reported results hold, the efficiency contribution is practically valuable: full-resolution matting at 4K/8K on a single consumer GPU is a real enabler, and the dataset is a plausible resource for the community. The paper also contains useful ablations (routing compression, maximum token cap, distillation/LTRM) and release code. However, the paper's core architectural novelty—local-global routing—is undermined by a mathematical issue in the router definition, and the high-resolution comparisons mix the efficiency contribution with an input-resolution confound. The measured savings are still consistent with the architecture, but the paper needs revision before it can be accepted.
major comments (3)
- [Adaptive Token Routing, Eqs. (5)-(7)] The global feature zg_i defined in Eq. (5) is the same vector for every token i. In Eq. (7), f'_theta is a linear layer followed by LogSoftmax, so the contribution W_g zg_i is an additive constant across all tokens and cancels in the softmax/argmax. Consequently the routing probability p_i and decision delta_i depend only on the local feature zl_i; the 'local-global strategy' is formally present but decision-inert. This removes the stated mechanism by which the router identifies globally informative tokens and weakens the interpretation of Figure 5, which is equally consistent with a purely local edge/texture heuristic. Please either modify the router so that the global feature can affect per-token decisions (for example by using a non-constant interaction with local features) or rewrite the methodology and claims to describe a local-only router.
- [Tables 3 and 4] The high-resolution comparisons let ViTMatte run only on downsampled input (D) or patches (P), while MEMatte runs on full-resolution input. This confounds the efficiency of the routing architecture with the resolution of the input. The reported gains on UHR-395 and PPT-344 could be driven substantially by the full-resolution processing rather than by adaptive token routing. Please add a full-resolution ViTMatte baseline on PPT-344 (which is small enough to be feasible) or otherwise quantify the contribution of input resolution, and adjust the 'outperforms existing methods' wording accordingly.
- [Adaptation to Ultra High-Resolution Images and Figure 6] The maximum token cap k is introduced and ablated, but the default value used for the reported results is never given in the main text or in Table 2 or Table 7. Please report the default k and the exact inference recipe (per-stage or global cap). In addition, all quantitative tables report a single run without error bars or seeds, which is insufficient to support the fine performance differences claimed, such as the 0.02-0.11 MSE differences on AIM-500 in Table 5.
minor comments (5)
- [Eq. (5)] The notation in Eq. (5) uses the subscript i on the left-hand side while the right-hand side averages over all tokens; please use a pooled symbol such as z^g without the token index to avoid implying a per-token global feature.
- [Table 2] The formatted values '0.7188.5%↓' and '84.9954.3%↓' are difficult to read; please insert spaces or use separate columns for memory/latency and the percentage reduction.
- [Abstract and Experiments] The abstract states the average resolution of UHR-395 is 4872×6017, while the experiments section states the test set average is 5318×7051; please clarify which number applies to the whole dataset and which to the test subset.
- [Figure 2] The figure caption mentions the RTX 1060 in the text but the experiments are run on the RTX 3090; please ensure all GPUs used for the memory curves are identified in the figure or caption.
- [Table 3] MEMatte-S* is fine-tuned on UHR-395, but the ViTMatte baselines in the same table are not fine-tuned on UHR-395; please add a fine-tuned ViTMatte baseline or state why such a comparison is not possible.
Circularity Check
No significant circularity: benchmark metrics are measured against a frozen external ViTMatte teacher, and the router-global-feature concern is an architectural nuance, not a circular derivation.
full rationale
I walked the claimed derivation chain. The router (Eqs. 3-7) outputs routing probabilities, BATR (Eq. 8) makes discrete decisions via Gumbel-Softmax or argmax, LTRM (Eq. 10) is a lightweight refinement branch, and the losses (Eqs. 11-13) use a frozen external ViTMatte teacher plus a compression regularizer. None of these equations defines a headline benchmark number in terms of the value being predicted: memory, latency, SAD, MSE, Grad, and Conn in Tables 2-5 are measured after training, not algebraically derived from rho, k, or the router scores. The distillation setup compares MEMatte with the same ViTMatte model used as teacher, but the teacher is a fixed external checkpoint and the student's metrics are evaluated on held-out sets, so this is standard practice rather than a logical circle. The only self-referential flavor is that several related-work citations share authors with the present paper, but none of these is load-bearing for the memory-efficiency or routing claims. I also considered the Eq. 5 concern that the global feature is an average over all tokens and hence token-invariant; even taken at face value, this is an architectural limitation about whether global context can re-rank tokens, not a circular derivation in which a result is equivalent to its own inputs. Therefore the circularity score is 0.
Assumptions & free parameters
free parameters (2)
- target compression degree rho =
0.25 (default)
- maximum token cap k =
not stated in main text
assumptions (6)
- standard math Global self-attention has O(N^2) memory complexity as defined in Eq. 2.
- domain assumption In early ViT stages, global attention behaves approximately like convolution, so few tokens need full attention there.
- domain assumption A lightweight branch made of depthwise convolution and ECA can refine uninformative tokens as well as global attention when trained with distillation.
- ad hoc to paper The local-global router, whose global feature is a single average over all tokens (Eq. 5), carries enough context to identify informative tokens.
- domain assumption Ultra-high-resolution synthetic composites in UHR-395 are a valid benchmark for high-resolution matting quality.
- domain assumption Models trained on DIM transfer to UHR-395, PPT-344, and AIM-500 without additional adaptation (except the fine-tuned starred variant).
Cite this review
Pith. "Pith review of Memory Efficient Matting with Adaptive Token Routing." pith.science (2026). https://pith.science/paper/5ZJKQG33
@misc{pith2026241210702,
author = {Pith},
title = {Pith review of: Memory Efficient Matting with Adaptive Token Routing},
year = {2026},
howpublished = {\url{https://pith.science/paper/5ZJKQG33}},
note = {Machine review of arXiv:2412.10702}
}
abstract
Transformer-based models have recently achieved outstanding performance in image matting. However, their application to high-resolution images remains challenging due to the quadratic complexity of global self-attention. To address this issue, we propose MEMatte, a \textbf{m}emory-\textbf{e}fficient \textbf{m}atting framework for processing high-resolution images. MEMatte incorporates a router before each global attention block, directing informative tokens to the global attention while routing other tokens to a Lightweight Token Refinement Module (LTRM). Specifically, the router employs a local-global strategy to predict the routing probability of each token, and the LTRM utilizes efficient modules to simulate global attention. Additionally, we introduce a Batch-constrained Adaptive Token Routing (BATR) mechanism, which allows each router to dynamically route tokens based on image content and the stages of attention block in the network. Furthermore, we construct an ultra high-resolution image matting dataset, UHR-395, comprising 35,500 training images and 1,000 test images, with an average resolution of $4872\times6017$. This dataset is created by compositing 395 different alpha mattes across 11 categories onto various backgrounds, all with high-quality manual annotation. Extensive experiments demonstrate that MEMatte outperforms existing methods on both high-resolution and real-world datasets, significantly reducing memory usage by approximately 88% and latency by 50% on the Composition-1K benchmark. Our code is available at https://github.com/linyiheng123/MEMatte.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Bolya, D.; Fu, C.-Y.; Dai, X.; Zhang, P.; Feichtenhofer, C.; and Hoffman, J. 2023. Token Merging: Your ViT but Faster. In International Conference on Learning Representations
work page 2023
-
[4]
Cai, H.; Xue, F.; Xu, L.; and Guo, L. 2022. Transmatting: Enhancing transparent objects matting with transformers. In European conference on computer vision, 253--269. Springer
work page 2022
-
[5]
Chen, Q.; Li, D.; and Tang, C.-K. 2013. KNN matting. IEEE transactions on pattern analysis and machine intelligence, 35(9): 2175--2188
work page 2013
-
[6]
Dai, Y.; Price, B.; Zhang, H.; and Shen, C. 2022. Boosting robustness of image matting with context assembling and strong data augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 11707--11716
work page 2022
-
[7]
Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929
arXiv 2020
-
[8]
Engelcke, M.; Rao, D.; Wang, D. Z.; Tong, C. H.; and Posner, I. 2017. Vote3deep: Fast object detection in 3d point clouds using efficient convolutional neural networks. In 2017 IEEE International Conference on Robotics and Automation (ICRA), 1355--1361. IEEE
work page 2017
Show all 42 references
-
[9]
K.; Winn, J.; and Zisserman, A
Everingham, M.; Van Gool, L.; Williams, C. K.; Winn, J.; and Zisserman, A. 2010. The pascal visual object classes (voc) challenge. International journal of computer vision, 88: 303--338
2010
-
[10]
Forte, M.; and Piti \'e , F. 2020. F , B , Alpha Matting. arXiv preprint arXiv:2003.07711
2020 arXiv
-
[11]
S.; and Oliveira, M
Gastal, E. S.; and Oliveira, M. M. 2010. Shared sampling for real-time alpha matting. In Computer Graphics Forum, volume 29, 575--584. Wiley Online Library
2010
-
[12]
G.; and Goldstein, T
Ghiasi, A.; Kazemi, H.; Borgnia, E.; Reich, S.; Shu, M.; Goldblum, M.; Wilson, A. G.; and Goldstein, T. 2022. What do vision transformers learn? a visual exploration. arXiv preprint arXiv:2212.06727
2022 arXiv
-
[13]
He, K.; Rhemann, C.; Rother, C.; Tang, X.; and Sun, J. 2011. A global sampling method for alpha matting. In CVPR 2011, 2049--2056. Ieee
2011
-
[14]
He, K.; Sun, J.; and Tang, X. 2010. Fast matting using large kernel matting laplacian matrices. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2165--2172. IEEE
2010
-
[15]
Hu, Y.; Lin, Y.; Wang, W.; Zhao, Y.; Wei, Y.; and Shi, H. 2025. Diffusion for natural image matting. In European Conference on Computer Vision, 181--199. Springer
2025
-
[16]
Ke, Z.; Sun, J.; Li, K.; Yan, Q.; and Lau, R. W. 2022. Modnet: Real-time trimap-free portrait matting via objective decomposition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 1140--1147
2022
-
[17]
Levin, A.; Lischinski, D.; and Weiss, Y. 2007. A closed-form solution to natural image matting. IEEE transactions on pattern analysis and machine intelligence, 30(2): 228--242
2007
-
[18]
Li, J.; Goel, V.; Ohanyan, M.; Navasardyan, S.; Wei, Y.; and Shi, H. 2024 a . Vmformer: End-to-end video matting with transformer. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 6678--6687
2024
-
[19]
Li, J.; Zhang, J.; and Tao, D. 2021. Deep automatic natural image matting. arXiv preprint arXiv:2107.07235
2021 arXiv
-
[20]
Li, Y.; Huang, Z.; Yu, G.; Chen, L.; Wei, Y.; and Jiao, J. 2024 b . Disentangled Pre-training for Image Matting. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 169--178
2024
-
[21]
Li, Y.; and Lu, H. 2020. Natural image matting via guided contextual attention. In Proceedings of the AAAI conference on artificial intelligence, volume 34, 11450--11457
2020
-
[22]
Li, Y.; Mao, H.; Girshick, R.; and He, K. 2022. Exploring plain vision transformer backbones for object detection. In European conference on computer vision, 280--296. Springer
2022
-
[23]
Liang, Y.; Ge, C.; Tong, Z.; Song, Y.; Wang, J.; and Xie, P. 2022. Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations. In International Conference on Learning Representations
2022
-
[24]
L.; Seitz, S
Lin, S.; Ryabtsev, A.; Sengupta, S.; Curless, B. L.; Seitz, S. M.; and Kemelmacher-Shlizerman, I. 2021. Real-time high-resolution background matting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8762--8771
2021
-
[25]
Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Doll \'a r, P.; and Zitnick, C. L. 2014. Microsoft coco: Common objects in context. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 1...
2014
-
[26]
Liu, H.; Tan, Z.; Tan, C.; Wei, Y.; Wang, J.; and Zhao, Y. 2024. Forgery-aware adaptive transformer for generalizable synthetic image detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10770--10780
2024
-
[27]
Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; and Guo, B. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, 10012--10022
2021
-
[28]
Lu, H.; Dai, Y.; Shen, C.; and Xu, S. 2019. Indices Matter: Learning to Index for Deep Image Matting. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV)
2019
-
[29]
Park, G.; Son, S.; Yoo, J.; Kim, S.; and Kwak, N. 2022. Matteformer: Transformer-based image matting via prior-tokens. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 11696--11706
2022
-
[30]
Park, N.; and Kim, S. 2022. HOW DO VISION TRANSFORMERS WORK? In 10th International Conference on Learning Representations, ICLR 2022
2022
-
[31]
Qiao, Y.; Liu, Y.; Yang, X.; Zhou, D.; Xu, M.; Zhang, Q.; and Wei, X. 2020. Attention-guided hierarchical structure aggregation for image matting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13676--13685
2020
-
[32]
Rao, Y.; Zhao, W.; Liu, B.; Lu, J.; Zhou, J.; and Hsieh, C.-J. 2021. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34: 13937--13949
2021
-
[33]
Sun, Y.; Tang, C.-K.; and Tai, Y.-W. 2021. Semantic image matting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 11120--11129
2021
-
[34]
Sun, Y.; Tang, C.-K.; and Tai, Y.-W. 2023. Ultrahigh resolution image/video matting with spatio-temporal sparsity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14112--14121
2023
-
[35]
N.; Kaiser, .; and Polosukhin, I
Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, .; and Polosukhin, I. 2017. Attention is all you need. Advances in neural information processing systems, 30
2017
-
[36]
M.; Porikli, F.; and Habibian, A
Venkataramanan, S.; Ghodrati, A.; Asano, Y. M.; Porikli, F.; and Habibian, A. 2023. Skip-attention: Improving vision transformers by paying less attention. arXiv preprint arXiv:2301.02240
2023 arXiv
-
[37]
Wang, Q.; Wu, B.; Zhu, P.; Li, P.; Zuo, W.; and Hu, Q. 2020. ECA-Net: Efficient channel attention for deep convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 11534--11542
2020
-
[38]
Xu, N.; Price, B.; Cohen, S.; and Huang, T. 2017. Deep image matting. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2970--2979
2017
-
[39]
Yao, J.; Wang, X.; Yang, S.; and Wang, B. 2024. Vitmatte: Boosting image matting with pre-trained plain vision transformers. Information Fusion, 103: 102091
2024
-
[40]
Yu, H.; Xu, N.; Huang, Z.; Zhou, Y.; and Shi, H. 2021 a . High-resolution deep image matting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 3217--3224
2021
-
[41]
Yu, Q.; Zhang, J.; Zhang, H.; Wang, Y.; Lin, Z.; Xu, N.; Bai, Y.; and Yuille, A. 2021 b . Mask guided matting via progressive refinement network. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 1154--1163
2021
-
[42]
Zhang, Y.; Wei, Y.; Jiang, D.; Zhang, X.; Zuo, W.; and Tian, Q. 2023. ControlVideo: Training-free Controllable Text-to-Video Generation. arXiv preprint arXiv:2305.13077
2023 arXiv
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.