{"id":"94b13b62-27c5-4a73-9eb7-15649aa32ee6","arxiv_id":"2412.11379","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A pretrained neural image codec can be augmented with a decoder-side latent diffusion module and a τ slider that interpolates between low-distortion and high-perception reconstructions without changing the bitstream.","lead":"This paper adds a train-once decoder-side module that uses latent diffusion to turn any pretrained neural image codec into a system with a user-controlled balance between pixel fidelity and perceived realism. The module leaves the codec and its bitstream untouched, so the same compressed file can be reconstructed with different distortion-perception tradeoffs at inference time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (2) adds noise to the condition ˆy, but Eq. (3)–(4) use D as a DDIM denoiser of the target latent, so inference evaluates D on inputs (noisy target latents) not seen in training; the latent-diffusion mechanism is not justified by the stated objective.","rationale":"The reader's conditional verdict already flags the auxiliary-encoder reachability assumption and the over-claimed headline number. I go one step further and identify a concrete internal mismatch in the diffusion formulation. The stated training objective (Eq. 2) does not train D to reverse a diffusion process on the target ¯y; it trains D to map a noised condition into a clean target. The inference recursions (Eqs. 3–4) presuppose the former. Because the model is never trained on noisy versions of the target, the sampling trajectory is uncontrolled, and the paper's claim that this is a latent-diffusion method is not established. This is directly load-bearing for the central claim: if the trajectory drifts outside the decoder's trained manifold, the plug-and-play promise fails. The concern is checkable by a distributional analysis of the sampling trajectory and by a target-noised retraining control. Separately, the paper's headline numbers are internally inconsistent: the abstract promises 'without sacrificing more than 1 dB in PSNR' while the results section reports 1.08 dB average PSNR loss for the 158.75% LPIPS-BDRate improvement. That is a smaller but unambiguous error that should be corrected. I see no evidence of bad faith; the issues are technical and addressable. The verdict remains CONDITIONAL, so no change to the reader's recommendation is needed.","tokens_in":16527,"tokens_out":14515,"duration_ms":134532,"concrete_test":"Using the trained D and a fixed base codec (e.g., Cheng2020), run the 10-step DDIM inference on Kodak and record the actual inputs ˜y_t to D at every step. Separately, draw the training inputs √α_t ˆy + √(1−α_t)ϵ for the same t values and images. Compare the two empirical distributions per timestep, e.g., with a logistic-regression domain classifier or MMD. If the inference inputs are distinguishable from training inputs for mid/low t, D is evaluated out-of-distribution and the DDIM derivation in Eq. (3) is not valid. A complementary retraining test: change the forward process to noise the auxiliary target instead of the condition (¯y_t = √α_t ¯y + √(1−α_t)ϵ) and re-train D under the same budget; if the τ=0 gains disappear or change materially, the reported results are tied to the mismatched training scheme and the diffusion claim should be restated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In the training objective (Eq. 2), the noisy input is formed by noising the decoded condition, ˆy_t = √α_t ˆy + √(1−α_t)ϵ, and D is trained to output the auxiliary perception target ¯y. In inference, D is inserted into the DDIM update (Eq. 3, 4) as if it were the posterior-mean predictor of the target: starting from ˜y_T ∼ N(0,I), each step computes ε from D(˜y_t, ˆy, t/T) and reconstructs the next sample. The training set only contains pairs (noised condition, clean target), never (noisy target, clean target). In training D can use the clean condition channel to identify the added noise; at inference the noisy input ˜y_t is not a corrupted version of that same condition, so that decomposition is unavailable. The recurrence is therefore not the reverse of the forward process used in training. At small t the training inputs are concentrated near the clean condition ˆy, while the inference inputs are supposed to be concentrated near the target ¯y; these are different regions of latent space. Hence the claimed latent-diffusion interpretation and the stability of the controllable sampling are not supported by the stated loss. The empirical PSNR/LPIPS curves may still be reproducible, but they would demonstrate a heuristic conditional mapping, not a latent diffusion process. This is also the concrete mechanism that could break the reader's weakest assumption: if the sampled latents drift off the frozen decoder's manifold, the plug-and-play promise fails.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a plug-and-play decoder-side module for pretrained neural image codecs that uses a latent diffusion process to transform decoded latent features, enabling a user-controlled distortion-perception trade-off without changing the bitstream. An auxiliary encoder is trained to produce a perception-oriented target latent, and an adaptive latent fusion module is trained with an LPIPS-based loss to map from the base codec's decoded latent to that target. At inference, the module is applied inside a DDIM-style sampling recurrence with a trade-off parameter τ that interpolates between the transformed and original latents. The method is evaluated on Kodak and CLIC2020 with Cheng2020, HiFiC, and MSHyper as base codecs, reporting improvements in LPIPS-BDRate, FID, CLIP-IQA, NIQE, and a user study.","tokens_in":16893,"tokens_out":8498,"duration_ms":67243,"significance":"If the technical foundation is sound, this is a practically valuable contribution: a single trained module can give a pretrained codec a wide, adjustable distortion-perception range at inference time, with no modification to the codec or the bitstream. The paper is strong on the empirical side: it reports eight image-quality metrics on two benchmark datasets, includes a user study, measures inference latency, and uses publicly available pretrained models for reproducibility. The authors also provide an ablation of the diffusion module and a stability analysis. However, the central claim that the method is a latent diffusion process is undermined by a mismatch between the training objective and the inference procedure, and the headline PSNR number in the abstract is inconsistent with the reported result. The paper's core idea remains interesting, but the technical story needs to be repaired before the results can be fully trusted.","major_comments":[{"comment":"The training objective trains the module D to predict the auxiliary perceptual latent y_bar from a noised condition y_hat_t = sqrt(alpha_t) y_hat + sqrt(1-alpha_t) epsilon, i.e., from a noisy version of the base codec's decoded feature. At inference, however, D is used inside the DDIM recurrence as the posterior-mean predictor of y_bar from y_tilde_t, which is a noisy version of the target y_bar itself and is initialized as y_tilde_T ~ N(0,I). These are different input distributions; for small t the training inputs concentrate around y_hat, while the inference inputs concentrate around y_bar. The statement that 'we directly learn transformed features y_tilde instead of the noise' does not reconcile this mismatch. As a result, the claimed latent-diffusion interpretation is not the reverse of the training forward process, and the stability of the sampling procedure is not guaranteed by Eq. (2). Please either reformulate the training to noise the actual target y_bar (as in standard DDPM/DDIM) or provide explicit evidence (e.g., an ablation training D on noised y_bar or a distribution-shift analysis) that the current training yields a valid posterior-mean predictor for the inference-time inputs.","section":"Adaptive Latent Fusion, Eq. (2) and Eqs. (3)-(4)"},{"comment":"The abstract's headline claim 'without sacrificing more than 1 dB in PSNR' is contradicted by the reported Cheng2020 result in the Main Results section, which states an average PSNR degradation of 1.08 dB. Since 1.08 dB exceeds 1 dB, the abstract's quantitative claim is inaccurate as written. Please correct the abstract to 'about 1.1 dB' or adjust the reported number so the headline assertion is consistent with the experimental results.","section":"Abstract and Main Results"},{"comment":"The training pipeline (auxiliary encoder g'_a and Eq. (2)) is formulated under the explicit assumption that the base codec is distortion-oriented. The paper nonetheless reports results for Ours(HiFiC), a perception-oriented base codec, and claims the method can transform features in both directions. The paper never specifies how g'_a is trained, what the target y_bar represents, or how Eq. (2) is adapted for a perception-oriented base codec. Without this information, the HiFiC results are not reproducible and the claimed generality across codec types is unsupported. Please add the training equations and implementation details for perception-oriented base codecs.","section":"Proposed Method and Main Results"}],"minor_comments":[{"comment":"The phrase 'On the low-distortion side ( τ = 0 )' appears to be a typo; τ = 0 corresponds to the perception-oriented end of the trade-off, not the low-distortion side.","section":"Main Results"},{"comment":"The text 'Their method shows little or no change from β = 0.64 to β = 2.56' does not match the β values shown in Fig. 8 (0, 0.08, 2.56); please correct the reference.","section":"Rate-Distortion-Perception Trade-off"},{"comment":"The sentence 'We disregard noise level t and directly input the result of D at each step into the decoder g_s.' is ambiguous; please clarify that the decoder is used during training to compute the image-domain loss term in Eq. (2).","section":"Adaptive Latent Fusion, Training"},{"comment":"The submitted PDF contains garbled character sequences in and around the figure captions (e.g., immediately after Figure 6 and within the caption of Figure 7); these should be cleaned in the final version.","section":"Figures"},{"comment":"Because Eq. (2) directly optimizes LPIPS and the headline result is LPIPS-BDRate, the paper should explicitly acknowledge that the LPIPS improvement is partially an outcome of optimizing the evaluation metric; the non-LPIPS metrics (FID, CLIP-IQA, NIQE, user study) provide independent evidence and should be emphasized accordingly.","section":"Evaluation"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the stress-test concern about the mismatch between Eq. (2) and Eqs. (3)-(4) is valid and should be the primary focus of the revision. If the authors can provide a corrected training formulation or strong empirical evidence that the current training yields a valid inference-time predictor, the paper could become acceptable. I also note that the paper is otherwise well executed on the experimental side."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a practically motivated decoder-side plug-in that lets you slide a pretrained codec's output between distortion and perception without re-encoding or changing the bitstream. The claimed headline result (more than 150% LPIPS-BDRate improvement for less than 1 dB PSNR loss) is internally inconsistent: they report an average PSNR degradation of 1.08 dB for the Cheng2020 case. That is a small but concrete accuracy problem in the framing.\n\nThe stress-test point is more substantive. The training objective in Eq. (2) only ever sees noisy versions of the decoded condition y_hat, with the auxiliary perceptual latent y_bar as target. At inference, D is inserted into the DDIM recurrence as if it predicted noise from a noisy target latent. Those are different input distributions. So the \"latent diffusion\" label is not justified by the stated loss; the module is better described as a learned conditional mapping with an ad-hoc sampling schedule. That does not mean the empirical curves are fake. The metrics are broad, the ablations are informative, the user study and latency numbers point in the right direction, and the comparison to CDC, MRIC, and HiFiC looks fair. But the mechanism section needs rework, either a proper derivation of the sampling procedure or an explicit statement that it is a heuristic.\n\nWhat is genuinely new: a train-once decoder-side module that works on multiple pretrained codecs, with a tau^2 interpolation rule and no bitstream change. That is useful for delivery pipelines and is not present in the cited DIRAC/HFD/MRIC work. The HiFiC extension, however, is never specified in the training equations; the paper assumes a distortion-oriented base codec and hand-waves the rest. No code or checkpoints are released, so reproducibility currently rests on the authors' figures.\n\nSoft spots in order: (1) the training/inference mismatch, which is central; (2) the 1 dB claim versus 1.08 dB; (3) missing code; (4) the appendix user study is thin (12 participants), though its direction is consistent.\n\nBottom line: this deserves a serious referee. The core idea is good and the empirical evidence is substantial. The diffusion mechanism, as written, does not hold together, and the authors should be asked to fix the derivation or reframe the method as a learned conditional refiner. If the revision addresses the mechanism and releases code, I would take it seriously.","headline":"Useful decoder-side plug-in for distortion-perception control on pretrained codecs, but the diffusion mechanism as written has a real training/inference mismatch that needs fixing before the claims hold.","tokens_in":17407,"tokens_out":1794,"would_cite":false,"duration_ms":18097,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that one decoder-side latent diffusion module can make a fixed pretrained neural codec slide continuously between high-PSNR and high-perceptual-quality reconstructions at the same bitrate, using a user parameter τ to…","keywords":["neural image compression","latent diffusion","distortion-perception tradeoff","plug-and-play decoder module","LPIPS-BDRate","adaptive latent fusion","DDIM","rate-distortion-perception"],"falsifier":"One concrete test is to attach the module to a pretrained codec with a narrow or highly quantized latent manifold, train on a fixed dataset, and check whether the PSNR-versus-$\\tau$ curve is monotone and whether $\\tau=0$ yields clean images; if the diffusion output drifts off the decoder's trained manifold, artifacts appear and the claimed 158.75% LPIPS-BDRate gain at about 1 dB PSNR loss will not reproduce.","tokens_in":16298,"feed_emoji":"🎛️","tokens_out":9396,"duration_ms":74056,"temperature":0.7,"pith_summary":"The paper tries to establish that a fixed, already-trained neural image codec can be upgraded on the receiver side alone: one plug-and-play module runs a latent diffusion process on the decoded features and lets a user choose, at inference time, where to sit on the distortion–perception curve without sending a single extra bit. The central claim is that the same compressed latent can be decoded either as a high-PSNR reconstruction or as a perceptually richer one, with a single parameter τ controlling the blend. If true, codec users no longer have to fix their fidelity-versus-realism trade-off at training time; they can dial it after the bitstream is already in hand. The headline numbers are a 158.75% LPIPS-BDRate improvement over the Cheng2020 base codec at an average PSNR cost of 1.08 dB, with similar flexibility shown on the HiFiC base codec.","feed_headline":"Decoder add-on wins 158% LPIPS bitrate gain for 1 dB PSNR","feed_subtitle":"Same compressed bitstream, new receiver-side dial: choose high fidelity or high realism without retraining the codec.","key_machinery":"The load-bearing object is the adaptive latent fusion module $D$, a latent diffusion network that operates directly on the compressed feature $\\hat{y}$ rather than on pixels. It is conditioned by concatenating the decoded feature with the diffusion input, and it is trained to predict the perception-oriented feature $\\bar{y}$ (produced by an auxiliary encoder $g'_a$ optimized with LPIPS against the frozen decoder) rather than predicting noise. Inference uses DDIM-style sampling with the trade-off parameter $\\tau$; the update rule $\\tilde{y}_{t-1} = \\sqrt{\\alpha_{t-1}}\\left[(1-\\tau^2)D(\\tilde{y}_t,\\hat{y},t/T)+\\tau^2\\hat{y}\\right] + (1-\\tau^2)\\sqrt{1-\\alpha_{t-1}}\\,\\epsilon(\\tilde{y}_t,\\hat{y},t/T)$ blends the diffusion output with the original latent, with $\\tau^2$ used so the control feels linear. The fixed decoder $g_s$ then maps the fused latent to an image, so the entire base codec stays frozen and the bitstream remains unchanged.","core_discovery":"On the paper's own terms, the discovery is that the distribution gap between distortion-optimized and perception-optimized latent features can be bridged in latent space by a diffusion module trained once against a frozen decoder, so the decoder output becomes a controllable interpolation. The method trains an auxiliary encoder $g'_a$ against the fixed decoder with an LPIPS loss to produce a perception-oriented target $\\bar{y}$; the adaptive latent fusion module $D$ then learns to denoise the base codec's decoded feature $\\hat{y}$ toward $\\bar{y}$. At inference, DDIM sampling with a user parameter $\\tau$ interpolates between the diffusion prediction and the original $\\hat{y}$: $\\tau=1$ reproduces the base codec exactly, and $\\tau=0$ gives the fully perception-oriented reconstruction. On Kodak, the paper reports a $158.75\\%$ LPIPS-BDRate improvement with an average $1.08$ dB PSNR drop on Cheng2020, matches or exceeds HiFiC on DISTS at $\\tau=0$ while keeping a $1.48$ dB PSNR advantage, and saves $22.59\\%$ bitrate for equal PSNR when applied to HiFiC.","pith_inferences":["Our inference: because the auxiliary encoder is trained without any rate constraint, the perception-oriented target can be richer than the bitstream would normally pay for; if that holds, the method effectively shifts some decoder-side computation into perceptual enhancement that costs no extra bits.","Our inference: the same latent-space interpolation idea could extend to other conditional targets, such as semantic maps, text prompts, or restoration cues, letting a receiver choose content-specific enhancements from one transmitted bitstream.","Our inference: a stress point is whether the frozen decoder remains on its trained manifold for fused latents at intermediate $\\tau$; the paper's stability runs show small variance, but applying the module to codecs with very narrow or strongly quantized latent manifolds could break the monotone $\\tau$ control."],"forward_implications":["A single trained module can be attached to an existing distortion-oriented codec (Cheng2020) and, at $\\tau=0$, improve LPIPS-BDRate by 158.75% while costing only 1.08 dB PSNR on average.","The same module attached to a perception-oriented codec (HiFiC) moves the operating point the other way, saving 22.59% bitrate for equal PSNR at $\\tau=0$.","Because the base codec is frozen and the bitstream is unchanged, users can decode the same file several times with different $\\tau$ values, effectively choosing sharpness versus realism after transmission.","Combined with a variable-bitrate scheme (MSHyper), the method produces a three-dimensional rate–distortion–perception operating range with a smooth, near-linear $\\tau$ traversal, unlike MRIC's less controllable $\\beta$ sweep.","The module works with as few as 10 DDIM steps, keeping inference latency at 331 ms per Kodak image on a 4090 GPU, well below pixel-domain diffusion codecs."],"supporting_citations":[{"why":"Distortion-oriented base codec whose frozen decoder and latent $\\hat{y}$ the module transforms; the 158.75% LPIPS-BDRate result is measured against it.","marker":"Cheng et al. 2020"},{"why":"HiFiC, the perception-oriented base codec that demonstrates the module also shifts a GAN-trained codec toward lower distortion.","marker":"Mentzer et al. 2020"},{"why":"Latent diffusion models; the module runs diffusion in the compressed-feature domain instead of pixels.","marker":"Rombach et al. 2022"},{"why":"DDIM; the inference recurrence equation that the method adapts for deterministic sampling.","marker":"Song, Meng, and Ermon 2021"},{"why":"DDPM; the noise schedule and conditional denoising formalism that the training builds on.","marker":"Ho, Jain, and Abbeel 2020"},{"why":"LPIPS; provides the perceptual loss for the auxiliary encoder and the metric behind LPIPS-BDRate.","marker":"Zhang et al. 2018"},{"why":"Rate-distortion-perception tradeoff theory; motivates why one bitstream cannot simultaneously maximize both goals, which the method addresses by receiver-side interpolation.","marker":"Blau and Michaeli 2019"},{"why":"MRIC; the main controllable-tradeoff baseline that trains the whole model from scratch, contrasted with the plug-and-play design.","marker":"Agustsson et al. 2023"},{"why":"CDC; a diffusion-based compression baseline compared on speed and quality, particularly the 17-step pixel-domain DDIM.","marker":"Yang and Mandt 2023"}],"fun_headline_variants":["Decoder plug-in swaps distortion for realism, 158% LPIPS gain","Latent diffusion dial tunes distortion vs perception per image","No retrain: 158% LPIPS bitrate win, 1 dB PSNR cost","Same codec, new decoder trick: adjustable realism up to 158%","Diffusion decoder add-on: 158% LPIPS gain for 1 dB PSNR"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the original decoder remains able to turn the diffusion-transformed feature codes into good images, and that a separately trained perception-oriented encoder produces a target those transformed codes can genuinely reach; if either fails, the whole trade-off dial stops working.","fun_headline_variants_meta":{"raw":{"variants":["Decoder plug-in swaps distortion for realism, 158% LPIPS gain","Latent diffusion dial tunes distortion vs perception per image","No retrain: 158% LPIPS bitrate win, 1 dB PSNR cost","Same codec, new decoder trick: adjustable realism up to 158%","Diffusion decoder add-on: 158% LPIPS gain for 1 dB PSNR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000151,"raw_usage":{"total_tokens":1208,"prompt_tokens":962,"completion_tokens":246,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":142}},"tokens_in":578,"tokens_out":246,"duration_ms":2838,"temperature":1.0,"reasoning_tokens":142,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:00:19.080629+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One concrete test is to attach the module to a pretrained codec with a narrow or highly quantized latent manifold, train on a fixed dataset, and check whether the PSNR-versus-$\\tau$ curve is monotone and whether $\\tau=0$ yields clean images; if the diffusion output drifts off the decoder's trained manifold, artifacts appear and the claimed 158.75% LPIPS-BDRate gain at about 1 dB PSNR loss will not reproduce.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Distortion-oriented base codec whose frozen decoder and latent $\\hat{y}$ the module transforms; the 158.75% LPIPS-BDRate result is measured against it."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"HiFiC, the perception-oriented base codec that demonstrates the module also shifts a GAN-trained codec toward lower distortion."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Latent diffusion models; the module runs diffusion in the compressed-feature domain instead of pixels."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DDIM; the inference recurrence equation that the method adapts for deterministic sampling."},{"cited_title":"A.; Shechtman, E.; and Wang, O","cited_arxiv_id":null,"evidence_quote":"LPIPS; provides the perceptual loss for the auxiliary encoder and the metric behind LPIPS-BDRate."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Rate-distortion-perception tradeoff theory; motivates why one bitstream cannot simultaneously maximize both goals, which the method addresses by receiver-side interpolation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MRIC; the main controllable-tradeoff baseline that trains the whole model from scratch, contrasted with the plug-and-play design."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CDC; a diffusion-based compression baseline compared on speed and quality, particularly the 17-step pixel-domain DDIM."}],"review_version":1}