REVIEW 4 major objections 7 minor 25 references
TACO: Rethinking Semantic Communications with Task Adaptation and Context Embedding
T0 review · 4 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read The paper claims that splitting an image into a compact context latent and a small set of task-salient tokens lets a single lightweight VQ-VAE beat diffusion-based semantic communication on reconstruction quality, bandwidth, and latency.
desk verdict A genuinely new lightweight VQ-based semantic communication architecture, but its headline LSF results are invalid because the transmitter uses ground-truth labels to select per-image token percentages. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the shared VQ-VAE latent space with a compression factor of 16 and a fixed codebook of 8192 entries, used for both the context and the task tokens. Two mechanisms select what enters that space: a four-times downsampling of the image produces the context latent, and GradCAM, a gradient-based saliency map from the downstream model, produces a mask whose coordinates pick the task-specific latent embeddings. The fusion step is the latent-level mixup $z_r = (1-M)\odot z_u + M\odot z_i$, where $M$ marks the selected coordinates, $z_u$ is the latent of the upsampled context, and $z_i$ carries the transmitted task tokens. This mixup is what lets a single decoder rebuild the image, and the local semantic feedback (LSF) loop is what chooses how many coordinates to send.
What would settle it
Run TACO without LSF at a fixed 10% token share on STL-10 and measure classification accuracy against the original image's 94.30%; if it falls below that, the headline gain depends entirely on the oracle label-based search rather than on the latent context/task fusion itself.
Extended reading notes
Core claim
The central claim is that context and task information can be separated in the latent space of one VQ-VAE and recombined at the receiver without retraining the network. The transmitter downsamples the image by a factor of four and encodes the small image to obtain a context latent; it also runs GradCAM on the original image to find the pixels the downstream task model attends to and maps those pixels to the corresponding latent coordinates. The receiver decodes the context latent, upsamples it back to full resolution, re-encodes it to recover a full latent, and replaces the selected coordinates with the transmitted task tokens. A local semantic feedback loop at the transmitter tries different shares of task tokens, picks the smallest share whose reconstruction wins on the target metric, and in doing so reports a 96.49% classification accuracy at 0.65 KB, above the original image's 94.30% at 59.93 KB. The paper also reports that the same procedure lets a receiver switch from classification to object detection after one short feedback round.
Load-bearing premise
The whole LSF result rests on the transmitter being able to evaluate downstream task performance against ground-truth labels when choosing the token percentage.
Editorial extensions
If this is right
- Reconstruction drops from thousands of diffusion steps to a single VQ-VAE decode, about 0.0015 seconds per image on an A100 in the paper's measurement.
- Bandwidth can be pushed below the raw image size by two orders of magnitude while downstream accuracy rises, because only task-relevant tokens are transmitted.
- Adding more task tokens raises classification accuracy in the reported tables, so the transmitted rate can be tuned to a receiver's quality-of-service requirement.
- A receiver can switch goals with one round of coordinate feedback rather than retraining, as shown in the classification-to-detection scenario.
- The rate bound tightens from $R_i + R_c$ to $R_{fb} \le R_i$, meaning the feedback mechanism removes the redundancy of sending context tokens that overlap with task tokens.
Reading between the lines
- The LSF gains depend on evaluating reconstructions against ground-truth labels at the transmitter; a label-free proxy such as prediction entropy or reconstruction uncertainty would make the method applicable where labels are unavailable, and testing that proxy is a direct next step.
- Because the context/task split lives in a spatially aligned VQ latent, the same design should transfer to video, audio, or point clouds whenever such a quantized latent exists; the paper only demonstrates images.
- The finding that a 0.65 KB reconstruction can beat the original image suggests the method is doing semantic denoising, removing distracting background; a stress test under channel noise would show whether that benefit survives in a real link, which the paper leaves to future work.
- One could replace the grid search over {10, 20, 30, 50, 70, 90, 100} with a learned selector that predicts the right token share per image, reducing the feedback cost further.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript introduces TACO, a semantic image communication framework built on a pretrained VQ-VAE. The transmitter sends a downsampled context latent and a set of task-salient latent tokens selected by GradCAM, and the receiver fuses the two. The paper also proposes 'local semantic feedback' (LSF), where the transmitter tries several task-token percentages and chooses the one that yields the best downstream accuracy. Evaluation on Cityscapes, STL-10, and Flickr reports large improvements over diffusion-based semantic communication and JPEG/JPEG2000 baselines, including 96.49% classification accuracy at 0.65 KB, higher than the 94.30% accuracy of the original image at 59.93 KB, and reconstruction time of 0.0015 s.
Significance. If the experimental claims were valid, TACO would be a useful contribution: the context-plus-task-token decomposition is simple, the reconstruction speed gain is large, and the paper demonstrates adaptation to a new task via receiver feedback. No code or repeated-seed statistics are provided, and the core performance claim depends on an oracle-like use of ground-truth labels at the transmitter during LSF. The paper is therefore better read as a promising architecture with an unsupported headline evaluation.
major comments (4)
- [Sec. III.A.4 and Tables III-IV] The LSF protocol selects the transmitted token percentage per image by evaluating downstream accuracy against ground-truth labels. The text says the transmitter 'evaluate[s] the downstream task performance' and keeps the percentage whose performance 'is compatible with the ground truth.' In a real system the transmitter (e.g., a camera) does not have the class or detection ground truth, so the reported TACO-LSF accuracy of 96.49% at 0.65 KB is an oracle-selected upper bound, not the result of a fixed transmission policy. The bandwidth is also per-image selected, so it is not an achievable fixed rate. The no-LSF rows in Table III (e.g., ζ+10% gives 85.51% at 1.51 KB) fall below the original image's 94.30%, so the central claim that TACO beats the original image depends entirely on this label-leaking selection. The authors should either design a selection rule that uses only transmitter-side information (e.g., downstream confidence on the reconstructed context) and report that policy's performance, or clearly present LSF as an oracle upper bound and re-derive the system-level claims.
- [Tables III-V] Tables III-V report single-run accuracy, mAP, and mIoU values without error bars or repeated seeds. Because LSF performs a per-image search over the test set, a single split can overfit to the ground-truth labels used in selection. At minimum the tables need means and standard deviations over at least three independent runs or bootstrap intervals, and the conclusions should be drawn from the resulting intervals.
- [Sec. V and Tables II-IV] The paper contains no channel model or channel coding: token indices are assumed to arrive error-free, and the 'Bandwidth' columns do not account for coding overhead, the cost of transmitting the mask M in Eq. (7), or bit errors. Section V explicitly lists channel noise as future work. As written, the contribution is a noiseless source-coding system, and claims about 'semantic communications' and 'ultra-high bandwidth efficiency' should be narrowed accordingly, or the experiments should include a noisy-token channel with error correction.
- [Tables III and IV and Table I] Tables III and IV compare TACO only against JPEG and JPEG2000 for classification and object detection; no learned image codec or existing deep semantic-communication baseline appears in these tables, so the abstract's claim of outperforming 'existing work' is not supported on these tasks. The comparisons in Table I are also uneven: TACO is a single feed-forward pass, yet its 'Steps' entry is listed as 'NA,' while the text claims it outperforms diffusion models in number of steps; the number of inference steps should be stated explicitly.
minor comments (7)
- [Sec. III.A.3, Sec. III.A.4, Fig. 2] The candidate percentage sets are inconsistent: Sec. III.A.3 lists {10,20,30,50,70}, Sec. III.A.4 lists {10,20,30,50,70,90,100}, and the Figure 2 caption says [10,20,50,100]; please unify them.
- [Table III] 'DATAEST' is a typo for 'DATASET' in the table title.
- [Abstract and Sec. II.B] The phrase 'ˆx is unnecessarily to be in the format of original data' should read 'ˆx does not need to be in the format of the original data.'
- [Table II] The row 'Diff-GO+(n=1024,L=1024) ... +GO-EV AE(n=1024,L=16)' appears to be a formatting error; please correct the baseline name and parameters.
- [Table VI] The 'Steps' entry for TACO is listed as '-'; since the paper's latency argument depends on TACO being single-step, report the exact number of forward passes.
- [Throughout] The paper does not provide a code or data release statement; adding one would improve reproducibility.
- [Sec. IV.A.3] The object-detection experiments use a dataset referred to only as 'Flickr,' but the cited reference [19] is a sentence-to-image retrieval dataset; please specify the exact dataset, annotations, and detection training protocol.
Circularity Check
TACO-LSF's headline gains are obtained by per-image grid search over transmitted token percentage using ground-truth labels, so the reported accuracy/bandwidth is the selected optimum, not a fixed-configuration prediction.
-
fitted input called prediction
[Sec. III.A.4 (Local Semantic Feedback), Eqs. (6)-(8); Table III]
"We first set a task-specific information percentage search set as{10, 20, 30, 50, 70, 90, 100}. Then, for each percentage in the set, we perform Eq. (6) to Eq. (8) at the transmitter and evaluate the downstream task performance. If the performance is compatible with the ground truth we share the top p% task-dependent latent embeddings with the context latent embeddings."
The LSF protocol runs the full receive-side reconstruction (Eqs. (6)-(8)) for every p in the search set, evaluates the downstream task accuracy against ground-truth labels, and selects the p that is 'compatible with the ground truth.' The reported TACO-LSF row (96.49% at 0.65 KB) is therefore the value of this per-image oracle selection, i.e., the maximum/fitted result over the percentage grid, not a prediction at a fixed p. The bandwidth is likewise chosen per image by the same search, so it is not an achievable fixed-rate system. Without ground-truth labels at the transmitter, LSF cannot be executed as described, and the claimed advantage over the original image (94.30%) is not established by the paper.
full rationale
Apart from the LSF procedure, the TACO encoder/decoder and context/task latent fusion are a self-contained constructive design: a pretrained VQ-VAE, GradCAM saliency selection, and the latent mixup rule in Eq. (7) define a deterministic transmitter/receiver pair whose no-LSF results are fixed-configuration measurements. Those results are not circular. The circularity is concentrated in the LSF evaluation: the method selects the transmitted token percentage per image by evaluating downstream task performance against ground-truth labels, and then reports the resulting accuracy and bandwidth as the method's performance. That is a fitted input (the task-percentage p) selected on the test labels, presented as a predicted outcome; it reduces by construction to the selection objective. The comparison baseline numbers in Table I partly come from the authors' prior work [10], which is a self-citation concern for reproducibility but is not load-bearing for TACO's own derivation and is not the main circular step.
Assumptions & free parameters
free parameters (2)
- Task-specific information percentage p =
Unknown; searched over {10,20,30,50,70,90,100} per image
- Context downsampling factor f =
4
assumptions (4)
- domain assumption The pretrained VQ-VAE from [17] provides a valid latent space for Cityscapes, STL-10, and Flickr images without any fine-tuning.
- domain assumption GradCAM heatmaps identify the latent embeddings that are most important for the downstream task.
- domain assumption The channel is noiseless, so transmitted integer tokens and masks arrive intact.
- domain assumption The downstream task models (ResNet-18 and Fast R-CNN) are fixed and evaluation on reconstructed images with these models is a valid proxy for semantic fidelity.
Cite this review
Pith. "Pith review of TACO: Rethinking Semantic Communications with Task Adaptation and Context Embedding." pith.science (2026). https://pith.science/paper/I3SYSEUQ
@misc{pith2026250510834,
author = {Pith},
title = {Pith review of: TACO: Rethinking Semantic Communications with Task Adaptation and Context Embedding},
year = {2026},
howpublished = {\url{https://pith.science/paper/I3SYSEUQ}},
note = {Machine review of arXiv:2505.10834}
}
read the original abstract
Recent advancements in generative artificial intelligence have introduced groundbreaking approaches to innovating next-generation semantic communication, which prioritizes conveying the meaning of a message rather than merely transmitting raw data. A fundamental challenge in semantic communication lies in accurately identifying and extracting the most critical semantic information while adapting to downstream tasks without degrading performance, particularly when the objective at the receiver may evolve over time. To enable flexible adaptation to multiple tasks at the receiver, this work introduces a novel semantic communication framework, which is capable of jointly capturing task-specific information to enhance downstream task performance and contextual information. Through rigorous experiments on popular image datasets and computer vision tasks, our framework shows promising improvement compared to existing work, including superior performance in downstream tasks, better generalizability, ultra-high bandwidth efficiency, and low reconstruction latency.
Figures
Reference graph
Works this paper leans on
-
[1]
Intelliot: intelligent iot environments,
A. Br ¨oring, V . Kulkarni, A. Zirkler, P. Buschmann, K. Fysarakis, S. Mayer, B. Soret, L. D. Nguyen, P. Popovski, S. Samarakoon et al., “Intelliot: intelligent iot environments,” in Global IoT Summit , Dublin, Ireland, 2022, pp. 55–68
work page 2022
-
[2]
Diff- go: Diffusion goal-oriented communications with ultra-high spectrum efficiency,
A. Wijesinghe, S. Zhang, S. Wanninayaka, W. Wang, and Z. Ding, “Diff- go: Diffusion goal-oriented communications with ultra-high spectrum efficiency,” in 2024 IEEE International Conference on Communications Workshops (ICC Workshops), Denver, CO, USA, 2024, pp. 1079–1084
work page 2024
-
[3]
Deep learning enabled semantic communication systems,
H. Xie, Z. Qin, G. Y . Li, and B.-H. Juang, “Deep learning enabled semantic communication systems,” IEEE Transactions on Signal Pro- cessing, vol. 69, pp. 2663–2675, 2021
2021
-
[4]
Deep learning based semantic communications: An initial inves- tigation,
——, “Deep learning based semantic communications: An initial inves- tigation,” in GLOBECOM 2020, Taipei, Taiwan, 2020, pp. 1–6
work page 2020
-
[5]
Deep learning enabled semantic communications with speech recognition and synthesis,
Z. Weng, Z. Qin, X. Tao, C. Pan, G. Liu, and G. Y . Li, “Deep learning enabled semantic communications with speech recognition and synthesis,” IEEE Transactions on Wireless Communications , vol. 22, no. 9, pp. 6227–6240, 2023
work page 2023
-
[6]
Generative semantic communication: Diffusion models beyond bit recovery,
E. Grassucci, S. Barbarossa, and D. Comminiello, “Generative semantic communication: Diffusion models beyond bit recovery,” arXiv preprint arXiv:2306.04321, 2023
arXiv 2023
-
[7]
Diff-go n: Enhancing diffusion models for goal-oriented communications,
S. Wanninayaka, A. Wijesinghe, W. Wang, Y .-C. Chao, S. Zhang, and Z. Ding, “Diff-go n: Enhancing diffusion models for goal-oriented communications,” arXiv preprint arXiv:2412.06980 , 2024
arXiv 2024
-
[8]
Diff-go+: An efficient diffusion goal-oriented communication system with local feedback,
A. Wijesinghe, S. Zhang, S. Wanninayaka, W. Wang, and Z. Ding, “Diff-go+: An efficient diffusion goal-oriented communication system with local feedback,” IEEE Transactions on Wireless Communications , 2025
work page 2025
Show all 25 references
-
[9]
Neural discrete representation learning,
A. Van Den Oord, O. Vinyals et al. , “Neural discrete representation learning,” in Advances in Neural Information Processing Systems , vol. 30, Long Beach, CA, USA, 2017
2017
-
[10]
Lami-go: Latent mixture integration for goal-oriented communications achieving high spectrum efficiency,
A. Wijesinghe, S. Wanninayaka, W. Wang, Y .-C. Chao, S. Zhang, and Z. Ding, “Lami-go: Latent mixture integration for goal-oriented communications achieving high spectrum efficiency,” arXiv preprint arXiv:2412.17839, 2024
2024 arXiv
-
[11]
Token communications: A unified framework for cross-modal context-aware semantic communications,
L. Qiao, M. B. Mashhadi, Z. Gao, R. Tafazolli, M. Bennis, and D. Niyato, “Token communications: A unified framework for cross-modal context-aware semantic communications,” arXiv preprint arXiv:2502.12096, 2025
2025 arXiv
-
[12]
Token-domain mul- tiple access: Exploiting semantic orthogonality for collision mitigation,
L. Qiao, M. B. Mashhadi, Z. Gao, and D. G ¨und¨uz, “Token-domain mul- tiple access: Exploiting semantic orthogonality for collision mitigation,” arXiv preprint arXiv:2502.06118 , 2025
2025 arXiv
-
[13]
A multi-task semantic commu- nication system for natural language processing,
Y . Sheng, F. Li, L. Liang, and S. Jin, “A multi-task semantic commu- nication system for natural language processing,” in 2022 IEEE 96th Vehicular Technology Conference, London/Beijing, 2022, pp. 1–5
2022
-
[14]
Grad-cam: Visual explanations from deep networks via gradient-based localization,
R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in Proceedings of the IEEE International Conference on Computer Vision , Venice, Italy, 2017, pp. 618–626
2017
-
[15]
The unreasonable effectiveness of deep features as a perceptual metric,
R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Utah, USA, 2018, pp. 586–595
2018
-
[16]
Gans trained by a two time-scale update rule converge to a local nash equilibrium,
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” in Advances in Neural Information Processing Systems , vol. 30, Long Beach, CA, USA, 2017
2017
-
[17]
A novel sampling scheme for text-and image-conditional image synthesis in quantized latent spaces,
D. Rampas, P. Pernias, and M. Aubreville, “A novel sampling scheme for text-and image-conditional image synthesis in quantized latent spaces,” arXiv preprint arXiv:2211.07292 , 2022
2022 arXiv
-
[18]
The cityscapes dataset for semantic urban scene understanding,
M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Be- nenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in Proceedings of the IEEE CVPR, Las Vegas, NV , USA, 2016, pp. 3213–3223
2016
-
[19]
From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions,
P. Young, A. Lai, M. Hodosh, and J. Hockenmaier, “From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions,” Transactions of the Association for Computational Linguistics, vol. 2, pp. 67–78, 2014
2014
-
[20]
An analysis of single-layer networks in unsupervised feature learning,
A. Coates, A. Ng, and H. Lee, “An analysis of single-layer networks in unsupervised feature learning,” in Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , Fort Lauderdale, FL, USA, 2011, pp. 215–223
2011
-
[21]
Semantic image synthesis with spatially-adaptive normalization,
T. Park, M.-Y . Liu, T.-C. Wang, and J.-Y . Zhu, “Semantic image synthesis with spatially-adaptive normalization,” in 2019 IEEE/CVF CVPR, Long Beach, CA, USA, 2019, pp. 2332–2341
2019
-
[22]
Learning to predict layout- to-image conditional convolutions for semantic image synthesis,
X. Liu, G. Yin, J. Shao, X. Wang et al. , “Learning to predict layout- to-image conditional convolutions for semantic image synthesis,” in NeurIPS, vol. 32, Vancouver, Canada, 2019
2019
-
[23]
Semantically multi-modal image synthesis,
Z. Zhu, Z. Xu, A. You, and X. Bai, “Semantically multi-modal image synthesis,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 5467–5476
2020
-
[24]
You only need adversarial supervision for semantic image synthesis,
V . Sushko, E. Sch ¨onfeld, D. Zhang, J. Gall, B. Schiele, and A. Khoreva, “You only need adversarial supervision for semantic image synthesis,” arXiv preprint arXiv:2012.04781 , 2020
2012 arXiv
-
[25]
Semantic image synthesis via diffusion models,
W. Wang, J. Bao, W. Zhou, D. Chen, D. Chen, L. Yuan, and H. Li, “Semantic image synthesis via diffusion models,” arXiv preprint arXiv:2207.00050, 2022
2022
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.