REVIEW 3 major objections 5 minor 15 references
Generalized User-Oriented Image Semantic Coding Empowered by Large Vision-Language Model
T0 review · 3 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read User-intent image coding generalizes to unseen objects, lifting answer match by 34%
desk verdict A plausible query-aware image semantic coding system whose evaluation is circular (LLaVA defines both the loss and the metric); the architecture is worth a look, but the numbers don't yet support the claims. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a segment-wise vector quantization pipeline applied to CLIP features that are text-conditioned through Feature-wise Linear Modulation (FiLM). The text query and image are projected into a shared embedding space; the text features generate scale and shift matrices that modulate the image features before quantization into discrete codewords. A user-intent relevance loss, computed as the cosine distance between LLaVA hidden states of the original and reconstructed images under the same query, is added to ℓ1, quantization, and adversarial losses in a two-phase training procedure. This machinery lets the transmitter send a compact sequence of integer indices that preserve
What would settle it
Run a human study in which raters are shown original and reconstructed images with the same query and asked whether the reconstruction preserves the queried content, then compare human agreement rates with the LLaVA-based answer match rates; if human gains are substantially smaller than the 34% and 5% improvements, the central claim of user-intent preservation is not supported.
Extended reading notes
Core claim
The paper discovers that integrating a frozen, pretrained VLM (CLIP) as the image/text feature extractor makes user-intent-driven semantic image coding generalize to unseen object categories, while a separately defined user-intent relevance loss (computed by LLaVA) aligns the reconstruction with the user's query. In zero-shot evaluations on animal images absent from the VQA training set, the proposed UO-ISC framework achieves the lowest user-intent relevance loss and the highest answer match rate among four baselines, including a state-of-the-art LLM-based query-aware codec. The improvement is attributed to the pretrained backbone's transferable semantic representations and to the FiLM-based
Load-bearing premise
The load-bearing assumption is that LLaVA's hidden-state cosine similarity and its exact-match answers are faithful, self-consistent proxies for what a human user actually intends; if LLaVA's judgments diverge from human intent, the reported gains overestimate true intent preservation.
Editorial extensions
If this is right
- If the claim holds, wireless image transmission can become intent-adaptive: the same source image can be encoded differently for different user queries without retraining the backbone.
- The zero-shot results suggest that semantic coding systems can be deployed on out-of-distribution imagery by leveraging pretrained VLMs, reducing the need for task-specific training data.
- The framework transmits 82% fewer symbols per image than JPEG2000 at comparable segment settings, indicating potential bandwidth savings for semantic-aware communication.
- The two-phase training recipe (semantic reconstruction first, adversarial refinement second) may be reusable for other discrete-feature generative transmission tasks.
Reading between the lines
- Since the same LLaVA model is used for both the training loss and the evaluation metric, the reported answer-match gains may partly reflect the model learning to satisfy LLaVA's specific hidden-state geometry rather than genuine semantic preservation; a human-evaluation study or a different VLM as judge would clarify this.
- The approach could be extended to multi-query or dynamically changing user intent, where the transmitter would need to prioritize content relevant to multiple simultaneous queries or update the transmitted features as intent shifts.
- The segment-wise codebook design creates an explicit rate-quality trade-off (shorter segments improve relevance but increase symbol count), suggesting a rate-intent-relevance optimization problem that the paper leaves open.
- If the user-intent relevance loss is robust across different VLM backbones, the methodology could also apply to video or 3D scene transmission, though the paper only demonstrates still images.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes UO-ISC, a user-oriented image semantic coding framework. The transmitter extracts CLIP image features and aligns them with a user-provided text query via FiLM-conditioned projections, then quantizes the aligned features segment-wise and transmits discrete indices over a fading channel. The receiver reconstructs the image through a decoder with skip connections. Training uses an L1 loss, a quantization loss, a user-intent relevance loss (Eq. 11) defined as the cosine distance between LLaVA hidden states of the original and reconstructed images given the same query, and, in a second phase, an adversarial loss. The framework is evaluated on a zero-shot animal subset of VQA, reporting answer match rate and user-intent relevance loss versus SNR, plus ablations and a symbol-efficiency comparison with JPEG2000.
Significance. If the results hold, the framework would be a practical way to inject user intent into image semantic coding while leveraging pretrained vision-language models for zero-shot generalization. The architecture is plausible and the use of frozen CLIP with learnable projections and FiLM conditioning is well motivated. However, the evaluation has a load-bearing self-reference issue: the training objective (Eq. 11) and the main metric (LLaVA answer match rate) both rely on the same nanoLLaVA model. Without an independent human or cross-model validation of user-intent preservation, the reported gains may reflect optimization toward LLaVA's internal geometry rather than genuine semantic alignment. The single split and absence of error bars further weaken the empirical support. The contribution is potentially useful, but the evidence presented is not yet sufficient to support the central claim.
major comments (3)
- [Section IV and Eq. (11)] The evaluation metric is self-referential. The user-intent loss Luser in Eq. (11) minimizes the cosine distance between nanoLLaVA hidden states of the original and reconstructed images, while the answer match rate is computed by asking the same nanoLLaVA (or its LLaVA variant) to generate answers from both images. Since the generated answers depend directly on those hidden states, optimizing Eq. (11) can inflate the answer match rate without actually preserving human-judged intent. The paper does not validate nanoLLaVA against human annotations or a different VLM, so the central claim of 'user-intent preservation' is not established. Please add an independent evaluation, e.g., human raters or a different pretrained VLM, and report agreement between nanoLLaVA and human judgments.
- [Section IV, ablation and statistics] The experimental evidence rests on a single zero-shot split (train on non-animal VQA samples, test on animal samples) with no error bars, no multiple seeds, and no statistical significance tests. The curves in Fig. 4 show differences of a few percent, but without variance information we cannot assess whether the reported 5%, 34%, and 4.8% improvements are meaningful. Moreover, the text 'improves the answer match rate by 5% through text alignment and by 34% by using a large VLM backbone' is ambiguous. If the 34% comes from comparing 'UO-ISC without text alignment' with 'UO-ISC without text alignment and pretrained VLM', the isolation is valid but should be stated explicitly; if it instead compares the full UO-ISC with the random-initialized baseline, the contribution of the pretrained VLM is conflated with text alignment and Luser. Please clarify and provide multiple runs with confidence
- [Section IV, baseline selection] The paper claims to outperform the 'state-of-the-art query-aware image semantic coding' but compares against a single method, reference [7], which is an arXiv preprint not established as state of the art. The JPEG2000 baseline is not query-aware. To support the SOTA claim, please either expand the comparison to include other query-aware semantic coding methods or temper the claim to 'the compared baseline'.
minor comments (5)
- [Eq. (6)] The segment indexing uses '(l−1)L' in the subscript, but the segment length is N_L, not L. It should be '(l−1)N_L + 1' etc.
- [Eq. (11) and Eq. (13)] The argument order of Luser is inconsistent: Eq. (11) defines Luser(Y, X_I; x_T) while Eq. (13) calls Luser(X_I, Y; x_T). Please unify.
- [Section IV] The footnote says 'nanoLLaV A' with an extra space due to a line break. Please use the correct model name consistently.
- [Section IV, Fig. 6] The number of transmitted symbols per image is not fully defined. Please specify whether it includes LDPC coding overhead, and how the symbol count for JPEG2000 is computed.
- [Section II, channel model] The paper assumes LDPC coding and modulation but does not discuss residual channel errors after decoding. In a fading channel with noise, the received symbol vector may contain errors that affect the decoded indices. Please clarify whether the evaluation accounts for residual errors or assumes error-free decoding after LDPC.
Circularity Check
Evaluation is partially circular: Luser is both a training loss and a reported metric, and the answer match rate is scored by the same LLaVA model used in Luser.
-
fitted input called prediction
[Section III Eq. (13)-(14); Section IV Fig. 4(b)]
"Results in Figs. 4 (a) and (b) show the answer match rate and the user-intent relevance loss Luser versus the SNR."
Eq. (13) includes Luser as a term in the training loss, so the model is explicitly optimized to minimize it. Fig. 4(b) then reports Luser as a performance metric. Thus the 'lowest user-intent relevance loss' claimed for UO-ISC is by construction the value of the training objective, not an independent measure of user intent.
-
other
[Section III Eq. (11); Section IV answer match rate definition]
"The answer match rate measures the proportion of samples where the answer generated by LLaVA from the reconstructed image exactly matches the one from the original image under the same user query."
The training loss Luser (Eq. 11) minimizes the cosine distance between LLaVA hidden states of the original and reconstructed images, and the evaluation metric asks the same LLaVA model to generate answers from those same hidden states. The answer is a deterministic function of these hidden states, so the metric is not independent of the training signal. Improvements in answer match rate may reflect alignment of LLaVA's internal geometry rather than preservation of human-perceived user intent. This is a partial reduction, not an exact equality, but the same model is load-bearing on both sides.
full rationale
The paper's architectural derivation and quantization steps are self-contained and do not rely on load-bearing self-citations. However, the evaluation of the central user-intent claim is substantially circular. First, the user-intent relevance loss Luser is a training objective in Eqs. (13)-(14) and is then reported as an evaluation metric in Fig. 4(b), making that result true by construction. Second, the primary metric, answer match rate, is computed using the same LLaVA model whose hidden states define Luser, creating a self-referential loop: the model is optimized against LLaVA's internal representations and then judged by LLaVA-generated answers. The comparison with the query-aware baseline retains some independent content, and the answer match rate is not identical to Luser, so the paper is not fully circular, but the reported gains (+5% text alignment, +34% VLM backbone) are likely inflated by this same-model evaluation. The 34% ablation is also confounded because 'without text alignment and pretrained VLM' differs in two factors at once, though that is an experimental-design issue rather than circularity. Overall, the central claim is not independently established with respect to human user intent.
Assumptions & free parameters
free parameters (3)
- Regularization weights lambda_user, lambda_quant, beta, lambda_gen =
Not reported
- Segment length N_L, number of segments L, codebook size N_cw =
N_L=32, L=16, N_cw=64
- Selected CLIP layers M_S =
{3,6,9,11}
assumptions (4)
- domain assumption LLaVA's hidden-state similarity and its generated answers are valid, consistent proxies for human user intent.
- domain assumption Held-out animal samples from VQA constitute a meaningful zero-shot 'unseen object' generalization test.
- domain assumption Frozen CLIP features retain enough spatial and semantic information for high-quality image reconstruction after quantization.
- domain assumption The simulated flat fading channel with h~CN(0,1), LDPC coding, and 8-ary modulation adequately represents wireless transmission.
Cite this review
Pith. "Pith review of Generalized User-Oriented Image Semantic Coding Empowered by Large Vision-Language Model." pith.science (2026). https://pith.science/paper/JIIA3CSP
@misc{pith2026250908913,
author = {Pith},
title = {Pith review of: Generalized User-Oriented Image Semantic Coding Empowered by Large Vision-Language Model},
year = {2026},
howpublished = {\url{https://pith.science/paper/JIIA3CSP}},
note = {Machine review of arXiv:2509.08913}
}
read the original abstract
Semantic communication has shown outstanding performance in preserving the overall source information in wireless transmission. For semantically rich content such as images, human users are often interested in specific regions depending on their intent. Moreover, recent semantic coding models are mostly trained on specific datasets. However, real-world applications may involve images out of the distribution of training dataset, which makes generalization a crucial but largely unexplored problem. To incorporate user's intent into semantic coding, in this paper, we propose a generalized user-oriented image semantic coding (UO-ISC) framework, where the user provides a text query indicating its intent. The transmitter extracts features from the source image which are relevant to the user's query. The receiver reconstructs an image based on those features. To enhance the generalization ability, we integrate contrastive language image pre-training (CLIP) model, which is a pretrained large vision-language model (VLM), into our proposed UO-ISC framework. To evaluate the relevance between the reconstructed image and the user's query, we introduce the user-intent relevance loss, which is computed by using a pretrained large VLM, large language-and-vision assistant (LLaVA) model. When performing zero-shot inference on unseen objects, simulation results show that the proposed UO-ISC framework outperforms the state-of-the-art query-aware image semantic coding in terms of the answer match rate.
Figures
Reference graph
Works this paper leans on
-
[7]
Semantic communication based on large language model for underwa- ter image transmission,
W. Chen, W. Xu, H. Chen, X. Zhang, Z. Qin, Y . Zhang, and Z. Han, “Semantic communication based on large language model for underwa- ter image transmission,”arXiv preprint arXiv:2408.12616, 2024
arXiv 2024
-
[1]
Semantic communication: A survey on research landscape, challenges, and future directions,
T. M. Getu, G. Kaddoum, and M. Bennis, “Semantic communication: A survey on research landscape, challenges, and future directions,”Proc. of the IEEE, vol. 112, no. 11, pp. 1649–1685, Nov. 2024
2024
-
[2]
Beyond transmitting bits: Context, semantics, and task-oriented communications,
D. G ¨und¨uz, Z. Qin, I. E. Aguerri, H. S. Dhillon, Z. Yang, A. Yener, K. K. Wong, and C.-B. Chae, “Beyond transmitting bits: Context, semantics, and task-oriented communications,”IEEE J. Sel. Areas Commun., vol. 41, no. 1, pp. 5–41, Jan. 2023
2023
-
[3]
SCAN: Semantic communication with adaptive channel feedback,
G. Zhang, Q. Hu, Y . Cai, and G. Yu, “SCAN: Semantic communication with adaptive channel feedback,”IEEE Trans. Cogn. Commun. Netw., vol. 10, no. 5, pp. 1759–1773, Oct. 2024
2024
-
[4]
Toward semantic communications: Deep learning-based image semantic coding,
D. Huang, F. Gao, X. Tao, Q. Du, and J. Lu, “Toward semantic communications: Deep learning-based image semantic coding,”IEEE J. Sel. Areas Commun., vol. 41, no. 1, pp. 55–71, Jan. 2023
2023
-
[5]
Visual language model based cross-modal semantic communication systems,
F. Jiang, C. Tang, L. Dong, K. Wang, K. Yang, and C. Pan, “Visual language model based cross-modal semantic communication systems,” IEEE Trans. Wireless Commun., vol. 24, no. 5, pp. 3937–3948, May 2025
2025
-
[6]
LaMoSC: Large language model-driven semantic communication system for visual trans- mission,
Y . Zhao, Y . Yue, S. Hou, B. Cheng, and Y . Huang, “LaMoSC: Large language model-driven semantic communication system for visual trans- mission,”IEEE Trans. Cogn. Commun. Netw., vol. 10, no. 6, pp. 2005– 2018, Dec. 2024
2005
-
[8]
Leveraging MoE-based large language model for zero-shot multi-task semantic communication,
S.-Y . Huang, R. Liao, and V . W. S. Wong, “Leveraging MoE-based large language model for zero-shot multi-task semantic communication,” in Proc. of IEEE Int. Conf. Commun. (ICC), Montreal, Canada, Jun. 2025
2025
Show all 15 references
-
[9]
Learning transferable visual models from natural language supervi- sion,
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervi- sion,” inProc. of Int’l Conf. on Machine Learning (ICML), Jul. 2021
2021
-
[10]
Generative adversarial nets,
I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y . Bengio, “Generative adversarial nets,” in Proc. of Adv. Neural Inf. Process. Syst., Montreal, Canada, Dec. 2014
2014
-
[11]
VQA: Visual Question Answering,
S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh, “VQA: Visual Question Answering,” inProf. of Int’l Conf. Comput. Vis. (ICCV), Santiago, Chile, Dec. 2015
2015
-
[12]
U-Net: Convolutional net- works for biomedical image segmentation,
O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional net- works for biomedical image segmentation,” inProc. of Int’l Conf. Med. Ima. Comput. Comput.-Assisted Intervention (MICCAI), Munich, Germany, Oct. 2015
2015
-
[13]
Feature-wise transformations,
V . Dumoulin, E. Perez, N. Schucher, F. Strub, H. de Vries, A. Courville, and Y . Bengio, “Feature-wise transformations,”Distill, Jul. 2018
2018
-
[14]
Visual instruction tuning,
H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” inProc. of Advances in Neural Inf. Process. Syst. (NeurIPS), New Orleans, LA, Dec. 2023
2023
-
[15]
Information Technology - JPEG 2000 Image Coding System Part 1: Core Coding System,
ISO/IEC 15444-1, “Information Technology - JPEG 2000 Image Coding System Part 1: Core Coding System,” 5th edition, Nov. 2024
2000
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.