{"id":"63d70f0a-d0a6-4cad-a6ea-eb0296353582","arxiv_id":"2411.13033","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A single LMM generates and compresses image captions for ultra-low-bitrate learned image compression, improving LPIPS BD-rate by 41.58% over MISC.","lead":"This paper presents an image compression system that uses a large vision-language model to write a short caption for each image, compresses that caption with the model's own token probabilities, and uses it to guide a diffusion model that rebuilds the image. The authors report 41.58% bitrate savings on a perceptual quality metric compared with a recent method, and release code and weights.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 41.58% LPIPS BD-rate gain is not attributable to the proposed semantic-perceptual loss: it includes the separate text-compression module, and MISC is evaluated without its three-stage diffusion, so the controlled comparison is missing.","rationale":"The reader's CONDITIONAL verdict is appropriate. I read the paper as a serious engineering contribution: the LMM rank+gzip text compression is a plausible, independently checkable module, and the semantic-perceptual fine-tuning is a reasonable extension of existing LIC training. The weakest point is not the text compression itself but the causal attribution and baseline control around the headline BD-rate. The paper includes Table II, which is useful: it shows the semantic loss's marginal effect on LPIPS is small or slightly negative, and the jump to 41.58% comes from the separate text path. Therefore the abstract's wording overstates the fine-tuning contribution. The MISC diffusion omission is a missing control; the authors assert no quality contribution, but without an ablation the reader cannot verify. These concerns are empirical and repairable, not signs of an invalid architecture. Since the reader already marked CONDITIONAL and the proposed check can settle the issue, I leave the verdict unchanged.","tokens_in":6501,"tokens_out":11088,"duration_ms":118790,"concrete_test":"Re-run the evaluation with MISC's full diffusion pipeline (or with the identical one-stage DiffBIR for both methods) and with identical caption prompts and text-bit accounting for both codecs. Then recompute LPIPS BD-rate for: (1) MISC, (2) the proposed model with semantic loss but no LMM text compression, and (3) the proposed full model. If the difference between (2) and (3) accounts for most of the 41.58%, or if matching MISC's diffusion reduces the gap to under roughly 10%, the headline claim is not supported as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim conflates two independent contributions. Table II shows that the semantic-perceptual fine-tuning alone achieves -38.07% LPIPS BD-rate vs MISC, while adding the LMM rank+gzip text path yields the headline -41.58%; perceptual-only fine-tuning achieves -39.76%. The abstract credits the fine-tuning method with the 41.58% result, but the ablation says the fine-tuning component is not the source of the headline margin. More importantly, the MISC comparison is not controlled: Section III-A states that the original three-stage diffusion process was omitted 'as it does not contribute to image quality', but no ablation or point-wise RD data is given. If MISC with its full diffusion has lower LPIPS at the same bpp, or if its raw caption text cost differs from the proposed caption cost, the BD-rate gap can shrink or change sign. The text-compression path itself appears sound and independently testable, but the headline performance claim requires either re-baselining MISC under identical diffusion and rate accounting or decomposing the gain into its two sources.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-part system for ultra-low-bitrate learned image compression: (i) an LMM-driven text compression path that entropy-codes caption tokens by ranking them with LLaVA's next-token probabilities and then gzipping the rank sequence, and (ii) a semantic-perceptual fine-tuning loss for the image coding network that combines MSE, LPIPS, CLIP-IQA, and CLIP image-to-text similarity. The reported claims are a 65.14% text bit saving, a 41.58% LPIPS BD-rate saving and a 60.99% CLIP image-to-image BD-rate saving against MISC, with code and pretrained weights released.","tokens_in":6761,"tokens_out":3575,"duration_ms":35242,"significance":"The central empirical claim is the 41.58%/60.99% BD-rate improvement over MISC; if this result is cleanly supported, the paper would demonstrate that a single LMM can serve as both caption generator and entropy model, and that adding semantic and perceptual losses improves ultra-low-bitrate reconstruction quality. The text-compression component is concrete, independently testable, and does not suffer from the metric-circularity issue that affects the image-loss evaluation. The public release of implementation and pretrained weights is a clear strength and should allow the community to verify the comparisons. However, the current manuscript does not isolate the source of the headline BD-rate gains, because the reported number includes the separate text-compression path and because the MISC baseline is evaluated in a modified configuration.","major_comments":[{"comment":"The abstract and Section IV-C attribute the 41.58% LPIPS BD-rate improvement to the proposed semantic-perceptual fine-tuning, but Table II shows that the fine-tuning component alone yields -38.07% on LPIPS without text compression, while perceptual-only loss yields -39.76%. The headline -41.58% therefore includes the separate text-compression path, and on LPIPS the semantic loss is slightly worse than the perceptual-only loss. The same issue appears on CLIP I2I, where the text-compression path contributes roughly 3.5 percentage points. Please decompose the reported gain into the contributions from (a) perceptual loss, (b) semantic loss, and (c) text compression, and rephrase the abstract, Section IV-C, and conclusion so that the 41.58% figure is not attributed to the fine-tuning method alone.","section":"Section IV-C and Table II"},{"comment":"The MISC anchor is evaluated with its original three-stage diffusion process omitted, with the statement that it 'does not contribute to the image quality,' but no ablation or point-wise rate-distortion data is provided to justify that omission. Since every reported BD-rate saving is computed against MISC, a configuration-mismatched baseline can change the magnitude or even the sign of the gains. Please either re-run MISC with its full diffusion pipeline under identical rate accounting, prompts, and decoding settings, or provide a per-image RD comparison with and without the omitted diffusion stages for both methods.","section":"Section III-A"},{"comment":"The image-coding network is trained by minimizing a loss that directly includes LPIPS, CLIP-IQA, and CLIP image-to-text similarity, and then the method is evaluated on LPIPS and CLIP I2I similarity. The reported BD-rate gains are therefore partially circular for the image-compression component, and the hand-set weights kappa_0..kappa_3 in Equation (2) are not analyzed for sensitivity. The text-compression result is not affected by this concern, but the image-coding claim should be supported by at least one metric not present in the training loss, such as FID, DISTS, or a subjective evaluation, and by a sensitivity study over the kappa values.","section":"Equation (2) and Section V"},{"comment":"The text-compression result is reported as total bits and a compression ratio, but the manuscript does not state the number of captions evaluated, the average caption token length, the size of the token vocabulary used for ranking, whether the end-of-sequence token is included, or how the rank probabilities are converted into the gzip input. These details matter because LLMZip-style rank coding is sensitive to tokenization and to the underlying probability model; without them the 65.14% figure cannot be independently reproduced or compared with other text-compression approaches. Please report the average caption length, bit allocation between image and text streams, and the exact entropy-coding pipeline.","section":"Table I and Section III-B"}],"minor_comments":[{"comment":"The text says the rank sequence is encoded with 'glib (adaptive Huffman coding)' but the rest of the paper and Table I refer to gzip; please unify the terminology and describe the actual implementation.","section":"Section III-B"},{"comment":"There is a typo: 'LIPIS-VGG' should be 'LPIPS-VGG'.","section":"Section II-B"},{"comment":"The caption-generation prompt and question template are not described in the paper and are only referenced to the GitHub repository; please include the exact prompt wording used for the 50-word caption generation.","section":"Section IV-A"},{"comment":"The RD curves are shown without point markers or error bars; providing per-image scatter or the number of test images would make the BD-rate comparison more transparent, especially because generative baselines can produce high-variance LPIPS values at low bitrates.","section":"Figures 4 and 5"},{"comment":"The phrase 'Clip Image-to-Image Similarity' is capitalized inconsistently with 'CLIP' used elsewhere in the paper; please standardize the notation.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The code release and the independent text-compression result are commendable. The main risk is not novelty but the lack of a controlled baseline and the conflation of two separate contributions in the headline number. If the authors re-run MISC in full configuration and decompose the BD-rate gains, the paper could become acceptable; otherwise the central empirical claim remains unsupported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the text compression idea is genuinely new and the engineering is competent. The headline 41.58% LPIPS BD-rate number, though, is not supported as a claim about the semantic-perceptual loss. Table II shows that loss alone gives -38.07% vs MISC, while perceptual-only gives -39.76%, and the extra margin comes from the separate text compression module. The abstract credits the fine-tuning with the headline gain, which the paper's own ablation contradicts.\n\nThe strongest part is the caption compression path. Feeding the caption back through the LMM to get token probabilities, converting tokens to ranks, and then gzipping the rank sequence is a clean application of LLM-as-entropy-model. The 65.14% compression ratio over raw text, with a 43.28% gzip-only baseline, is plausible and independently testable. If the code ships, this part will be reproducible.\n\nThe soft spots are real but not fatal. First, the MISC baseline is not controlled: the paper omits MISC's three-stage diffusion process because it 'does not contribute to image quality,' but no ablation or point-wise RD data supports that claim. If MISC's full pipeline has lower LPIPS at the same bpp, the BD-rate gap shrinks. Second, the semantic-perceptual loss has hand-set kappas without sensitivity analysis, and the ablation shows it slightly hurts LPIPS while marginally helping CLIPI2I; the claimed subjective improvement is qualitative and unmeasured. Third, training on LPIPS and CLIPI2I and evaluating on the same metric families introduces some circularity, though the text compression ratio is immune to that.\n\nThis is a solid conference paper, not a top paper. The text compression contribution is worth citing. The image quality claim needs a clearer decomposition and a properly controlled baseline before I'd trust the headline. I'd send it to peer review—a serious referee will ask for those controls—but the abstract's causal story should be rewritten. Worth a reading group discussion of the text compression idea.","headline":"The LMM-driven caption compression path is a real contribution, but the headline BD-rate gain is not attributable to the proposed semantic-perceptual loss and needs a controlled re-baselining.","tokens_in":7262,"tokens_out":1872,"would_cite":true,"duration_ms":18289,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single large multimodal model can generate and compress image captions together, saving 65% of text bits while semantic-aware training cuts image bitrate by 41.58%.","keywords":["learned image compression","large multimodal model","caption compression","semantic-perceptual loss","latent diffusion","ultra low bitrate","CLIP","LPIPS"],"falsifier":"Re-running the released code and weights against the prior method with the prior method's full diffusion pipeline, identical prompts, and identical rate accounting would settle the central claim: if the 41.58% LPIPS and 60.99% CLIP savings collapse to near zero, the improvement is an artifact of the configuration mismatch.","tokens_in":6312,"feed_emoji":"🖼️","tokens_out":4531,"duration_ms":39390,"temperature":0.7,"pith_summary":"This paper tries to establish that ultra-low-bitrate image compression can use one large multimodal model (LMM) for both caption generation and caption compression, replacing the separate text codec used by earlier text-conditioned methods. The authors also claim that fine-tuning the image codec with a weighted mixture of pixel, perceptual, and semantic losses—rather than pixel error alone—shifts the rate-distortion curve in the right direction. If true, text-conditioned generative compression becomes simpler and cheaper, and the caption model's own next-token probabilities become a usable entropy code for the caption. The headline quantitative claims are a 65.14% text bit saving from rank encoding plus gzip, and 41.58% LPIPS and 60.99% CLIP image-to-image bitrate savings against the prior method.","feed_headline":"One model generates and compresses captions, saving 65% of text bits","feed_subtitle":"Semantic-aware training then cuts image bitrate by 41.58% at equal perceptual quality.","key_machinery":"The load-bearing mechanism is the reuse of next-token prediction: the LMM both writes the caption and acts as an entropy model for it, since its output probabilities define the rank code. The second mechanism is a semantic-perceptual loss mixture that balances pixel fidelity against perceptual and semantic objectives during fine-tuning of the image codec, with a frozen diffusion model reconstructing the final image from the compressed image plus caption.","core_discovery":"The central discovery is that the LMM used to generate an image caption also supplies, for free, a conditional probability distribution over the next token at every step; replacing each token by its rank in that sorted distribution and gzipping the rank sequence yields a lossless text code with 65.14% fewer bits than the raw caption. On the image side, training the learned image codec with a normalized sum of MSE, LPIPS, a CLIP-based image-quality score, and a CLIP image-to-text score—instead of MSE alone—produces reconstructions that score better on LPIPS and CLIP image-to-image similarity at the same bitrate, with the largest reported gap coming when both semantic losses and text compression are active.","pith_inferences":["The same rank-encoding trick should apply to any structured output an LMM emits—sketches, layouts, audio captions, or code—wherever the receiver can run the same model.","If semantic losses prevent overfitting to a single network-based metric, then similar loss mixtures could stabilize other perceptually trained codecs and generative compressors.","A natural extension is to let the decoder's diffusion model choose how much detail the caption must carry, trading caption bits against image bits at fixed total rate."],"forward_implications":["Text-conditioned ultra-low-bitrate compression no longer needs a separate text codec; caption generation and entropy coding live in the same frozen LMM.","Because the LMM and diffusion model are frozen, the semantic-perceptual fine-tuning can be grafted onto any learned image codec, not just the one used here.","The rank-encoding scheme converts every generated token into a small integer, so text overhead stays low even when captions are long.","At equal bitrate, the method reports better LPIPS and CLIP image-to-image similarity than the prior text-conditioned baseline, implying better perceptual and semantic fidelity in the ultra-low-bitrate regime."],"supporting_citations":[{"why":"Supplies the baseline text-conditioned method and the anchor for all BD-rate comparisons.","marker":"[3]"},{"why":"Provides the LMM whose next-token probabilities are reused for caption compression.","marker":"[20]"},{"why":"Supplies the rank-encoding idea that turns next-token probabilities into a lossless code.","marker":"[6]"},{"why":"Defines CLIP embeddings used for both semantic losses and the image-to-image evaluation metric.","marker":"[22]"},{"why":"Defines LPIPS, the perceptual metric used in the loss and in evaluation.","marker":"[11]"},{"why":"Supplies the frozen diffusion reconstruction network conditioned on the caption.","marker":"[19]"},{"why":"Provides the base learned image codec that is fine-tuned with the new loss.","marker":"[18]"}],"fun_headline_variants":["Single LMM compresses captions 65% and cuts LPIPS BD-rate 41.58%","One LMM generates and compresses captions, saving 65% text bits","Semantic fine-tuning cuts LPIPS BD-rate by 41.58%","LMM caption compression saves 65% bits; semantic tuning cuts LPIPS BD-rate 41.58%","LMM-driven codec: caption bits down 65%, LPIPS BD-rate down 41.58%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains assume the prior method was evaluated with the same bitrate accounting, prompts, and diffusion settings, and that omitting its three-stage diffusion refinement—which the paper states but does not ablate—does not change comparative quality.","fun_headline_variants_meta":{"raw":{"variants":["Single LMM compresses captions 65% and cuts LPIPS BD-rate 41.58%","One LMM generates and compresses captions, saving 65% text bits","Semantic fine-tuning cuts LPIPS BD-rate by 41.58%","LMM caption compression saves 65% bits; semantic tuning cuts LPIPS BD-rate 41.58%","LMM-driven codec: caption bits down 65%, LPIPS BD-rate down 41.58%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001211,"raw_usage":{"total_tokens":4916,"prompt_tokens":803,"completion_tokens":4113,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":419,"completion_tokens_details":{"reasoning_tokens":3991}},"tokens_in":419,"tokens_out":4113,"duration_ms":27990,"temperature":1.0,"reasoning_tokens":3991,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:54:24.424824+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-running the released code and weights against the prior method with the prior method's full diffusion pipeline, identical prompts, and identical rate accounting would settle the central claim: if the 41.58% LPIPS and 60.99% CLIP savings collapse to near zero, the improvement is an artifact of the configuration mismatch.","supporting_citations":[],"review_version":1}