{"id":"44d84fc4-f07e-4d00-b5d8-00aee8a46295","arxiv_id":"2504.21445","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A survey of end-to-end learned video compression that categorizes P-frame and B-frame neural codecs, reviews optimization and system implementation, and benchmarks several learned codecs against standard codecs.","lead":"This paper is a survey of learned video compression, a field that replaces the hand-designed blocks of traditional video codecs with neural networks optimized end to end. It organizes recent models, optimization tricks, standardization efforts, and hardware deployments, and reports benchmark comparisons of learned codecs against HEVC and VVC.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 5's 'significantly outperform VTM in MS-SSIM' is not supported by the paper's own tables: PSNR shows VTM ahead on average, the MS-SSIM advantage is dataset-dependent, and it rests on a single author-affiliated model with a nonstandard RGB-space MS-SSIM computation.","rationale":"The reader's conditional verdict already flags the benchmark fairness issue, and my read agrees that the experimental section is the weak link. I go further in pointing out that the paper's own Table 3 contradicts the abstract's broad 'superior compression performance' claim, and that the only positive evidence—the MS-SSIM result—is dataset-dependent, restricted to one author-affiliated model, and computed with a nonstandard RGB-space MS-SSIM protocol. This is an internal-evidence problem, not merely a missing-artifact problem. However, the survey's taxonomy, model summaries, and system-level discussion are still useful as a literature map, so a full rejection would be too harsh. Keeping the verdict at CONDITIONAL, with the condition that the benchmark be redone under standard, auditable conditions or the claims be softened, is appropriate. The concrete re-run described above would settle whether the perceptual-superiority claim survives a fair comparison.","tokens_in":12714,"tokens_out":7019,"duration_ms":74604,"concrete_test":"Re-run the Section 5 benchmark under a single controlled protocol: use the original YUV420 test sources converted to RGB with BT.709 for all codecs, set HM-16.20, VTM-23.0, and EEV-0.5 all to randomaccess with intra-period=32 and GOP=32, encode the full sequences (not just the first 96 frames), and compute PSNR and MS-SSIM on the luma plane of the reconstructed frames after a single defined conversion back to YUV. Then recompute average BD-rate for EEV-0.5 versus VTM-23.0. If the MS-SSIM advantage drops below a meaningful threshold or reverses, the paper's headline 'superior perceptual quality' claim fails. Additionally, release the per-sequence rate-distortion points so the comparison can be audited.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that learned video compression is competitive with, and perceptually superior to, the latest conventional codecs—is undercut by the paper's own experimental section. In Table 3, which reports PSNR BD-rate in RGB, VTM-23.0 achieves -37.72% average versus -33.63% for EEV-0.5 and -26.19% for DCVC-FM; VTM is better on every dataset class except Class E. Table 4 shows EEV-0.5 ahead of VTM in MS-SSIM on average (-41.53 vs -37.00), but the text's 'significantly outperform VTM in MS-SSIM' is an overgeneralization: VTM wins on UVG (-38.77 vs -33.20) and MCL-JCV (-42.07 vs -28.17). The only bi-directional model evaluated is EEV-0.5, a model from the authors' own MPAI-EEV standardization work, and no code, rate-distortion points, error bars, or significance tests are provided. Compounding this, MS-SSIM is reported 'in RGB colorspace' although the standard metric is defined on luminance; computing it per RGB channel is nonstandard and can favor codecs with different chroma error distributions. Combined with the unconventional pipeline (RGB-to-YUV444 for HM/VTM, RGB-native for LVC) and the use of only the first 96 frames, the benchmark cannot support the abstract's blanket 'superior compression performance' claim. Section 5 itself concedes the PSNR gap and the dependence on GOP size, so the conclusion rests on one fragile, unvalidated perceptual comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript is a survey of end-to-end learned video compression (LVC), organizing the literature into uni-directional and bi-directional prediction architectures, reviewing optimization techniques (online encoder adaptation, decoder-side adaptation, variable bitrate, rate control, perceptual quality, quantization), discussing system and hardware deployment (FPGA, mobile, semantic communications), reporting standardization efforts (MPAI-EEV, AVS-EEM), and presenting an experimental comparison of selected LVC models against HM-16.20 and VTM-23.0 using PSNR and MS-SSIM BD-rates. The paper's central claim, stated in the abstract and conclusion, is that LVC models demonstrate superior compression performance over conventional codecs and are poised to be the next major advancement in video compression.","tokens_in":13016,"tokens_out":2177,"duration_ms":22747,"significance":"If the empirical claims were fully supported, the survey would be a useful systematization of a fast-moving field, and its sections on model taxonomies, optimization methods, and hardware/system deployment are genuinely informative. The paper also provides a valuable service by documenting MPAI-EEV standardization progress and connecting learned coding to system-level considerations. The experimental section, however, does not substantiate the headline claim of 'superior compression performance': on PSNR, VTM-23.0 clearly outperforms every evaluated LVC model on average, and the MS-SSIM advantage attributed to bi-directional LVC is dataset-dependent and rests on a single author-affiliated model. The paper ships no code, no error bars, and no significance testing, so the quantitative evidence is not reproducible at the level the abstract's claim requires. The survey content alone could still be a useful contribution, but the experimental overclaim must be corrected.","major_comments":[{"comment":"The abstract's claim of 'superior compression performance of LVC models' and Section 5's statement that bi-directional NVC models 'significantly outperform VTM in MS-SSIM' are not supported by the paper's own tables. In Table 3 (PSNR BD-rate, RGB), VTM-23.0 achieves -37.72% average versus -33.63% for EEV-0.5, -26.19% for DCVC-FM, and -26.45% for DCVC-DC; VTM is better on every dataset class except Class E. In Table 4 (MS-SSIM), EEV-0.5 is ahead of VTM on average (-41.53 versus -37.00), but VTM wins on UVG (-38.77 versus -33.20) and MCL-JCV (-42.07 versus -28.17). The word 'significantly' is also unsupported because no error bars, confidence intervals, or significance tests are reported. The conclusion should be rephrased to state precisely which codec wins on which metric and dataset, or additional evidence must be provided.","section":"Abstract; Section 5, Tables 3 and 4"},{"comment":"The benchmark configuration is not demonstrably fair for the comparative claim. Traditional codecs are applied to RGB-to-YUV444-converted input with HM randomaccess main rext and VTM randomaccess vtm configurations, while LVC models operate in native RGB with intra-period and GOP set to 32, and only the first 96 frames of each sequence are evaluated. The color-space conversion, the GOP-size mismatch (HM GOP=16, VTM GOP=32, LVC GOP=32), and the truncated sequence length can each systematically affect BD-rate. The paper states that the YUV444 pipeline 'achieve[s] a better compression ratio for the traditional codecs,' but no ablation or sensitivity analysis is given to establish that the reported differences are not artifacts of these choices. At minimum, the authors should report results under matched configurations (e.g., same GOP size, same color space for all codecs) or state clearly that the comparison is configuration-specific.","section":"Section 5, Test Conditions"},{"comment":"The MS-SSIM evaluations are reported 'in RGB colorspace,' whereas the standard MS-SSIM index is computed on luminance. Computing MS-SSIM per RGB channel is a nonstandard variant, and its sensitivity to chroma error distribution can favor codecs with different color-space handling. This methodological choice is not justified, and it materially affects the central perceptual-quality conclusion. The authors should either compute MS-SSIM on the luminance channel in a common color space (e.g., YUV444 or YUV420) or provide a validation that the RGB-channel MS-SSIM agrees with luminance MS-SSIM on these codecs.","section":"Section 5, Experimental Results; Table 4"},{"comment":"The only bi-directional LVC model evaluated is EEV-0.5 (Ye et al., 2024), whose first-listed author is also an author of this survey and which is the product of the MPAI-EEV standardization effort in which several co-authors participate. The paper's claim that 'bi-directional predictive NVC models... significantly outperform VTM in MS-SSIM' therefore rests on a single model with direct author affiliation, and no independent bi-directional baseline (e.g., DCVC-B, B-CANF, or TLZMC) is included. To support the general statement about bi-directional LVC, the authors should evaluate at least one additional bi-directional model from a different group, or explicitly restrict the claim to EEV-0.5.","section":"Section 5, Tables 3-4; Section 2.2"}],"minor_comments":[{"comment":"The phrase 'visual intelligence' is used repeatedly but never defined; consider replacing it with a precise term such as 'visual signal processing' or defining it at first use.","section":"Abstract and Section 1"},{"comment":"The table cells use '!' and '%' symbols without a legend explaining their meaning (presumably checkmark/not-applicable). Please add a caption legend for these symbols.","section":"Tables 1 and 2"},{"comment":"The paper mentions that MPAI-EEV released 'a public benchmark for UAV video coding' but this benchmark is not used in the experiments; either clarify why it is omitted or cite it explicitly as future work.","section":"Section 5, Test Conditions"},{"comment":"The phrase 'the author borrowed the knowledge' should be 'the authors borrowed'; there are several similar grammatical issues throughout (e.g., 'emphasizing their technical innovations, advantages' in the abstract) that need copyediting.","section":"Section 3.2"},{"comment":"Several cited works appear in the reference list but are not discussed in the text (e.g., Feng et al., 2021; Hu et al., 2020; Li et al., 2022; Liu et al., 2021; Lu et al., 2021; Ladune et al., 2021; Shi et al., 2022; Yılmaz and Tekalp, 2021). Please either cite them in the relevant sections or remove them.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The survey's taxonomy and system-level coverage are useful, but the experimental section is the load-bearing part for the paper's central claim and it currently overstates the evidence. The issues are fixable by re-scoping the claims, adding matched-configuration experiments, and including independent bidirectional baselines, but they require substantive additional work rather than copyediting. I would also encourage the editor to verify that the authors' direct involvement in the MPAI-EEV standardization is disclosed wherever EEV-0.5 results are used as evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid survey with a weak empirical tail. If someone wants a structured map of learned video compression models, optimization tricks, and hardware/system work, this is actually a handy read. The taxonomy in Tables 1 and 2 is careful, the six optimization areas are well chosen, and the system section (FPGA, mobile, semantic communications) is something many surveys skip. That part deserves genuine credit.\n\nThe soft spots are all in Section 5. The abstract promises 'superior compression performance,' but the body is more hedged: on PSNR, all learned codecs lose to VTM on average, and on MS-SSIM, the average gain for EEV-0.5 over VTM is driven by the JCT-VC classes; VTM actually wins on UVG and MCL-JCV. So 'significantly outperform VTM in MS-SSIM' is an overgeneralization, especially with no error bars or significance tests anywhere. The only bi-directional model evaluated is EEV-0.5, which comes from the authors' MPAI-EEV line, so the claim about 'bi-directional predictive NVC models' rests on one self-affiliated point. Also, computing MS-SSIM per RGB channel is nonstandard; MS-SSIM is normally defined on luminance, and the paper doesn't justify the choice or show it doesn't bias the comparison. The RGB-to-YUV444 conversion for HM/VTM while LVC runs natively in RGB is another potential confound, though at least they cite the prior work for that procedure. Using only the first 96 frames is a common shortcut but limits generalizability. None of this sinks the survey, but the benchmark can't be used as evidence for the field-level claims until the authors either release code/points, add error bars, or soften the language.\n\nThe survey half is for newcomers, standardization folks, and people who need a quick literature map. I'd send it to peer review with a clear request to fix Section 5's claims. The taxonomy and system coverage are worth refereeing properly.","headline":"Competent survey with a useful map of the field; the experimental section overclaims on the strength of one author-affiliated model and a nonstandard MS-SSIM comparison.","tokens_in":13600,"tokens_out":2481,"would_cite":false,"duration_ms":26108,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This survey argues that learned video compression now matches the latest conventional codecs and is poised to be the next major advance in video coding.","keywords":["learned video compression","end-to-end neural codec","rate-distortion optimization","bi-directional prediction","conditional coding","video coding standardization","hardware acceleration","perceptual quality"],"falsifier":"Re-run the same rate-distortion comparison on all frames of the JCT-VC, UVG, and MCL-JCV sequences, with the conventional codecs fed RGB directly (or both sides in the same color space), matching GoP sizes (e.g., 32 for both) and using both PSNR and MS-SSIM; if bi-directional learned models no longer beat VTM-23.0 on MS-SSIM under this configuration, the survey's central comparative claim would be undercut.","tokens_in":12459,"feed_emoji":"🎥","tokens_out":7270,"duration_ms":68215,"temperature":0.7,"pith_summary":"This survey reviews end-to-end learned video compression and argues that the field has reached competitiveness with the latest conventional codecs. Its own experiments back the claim: on standard test sequences, bi-directional prediction based learned codecs score slightly below the VVC reference encoder (VTM-23.0) in PSNR but clearly above it in MS-SSIM, a perceptually motivated quality metric. The survey also documents optimization techniques, standardization efforts, and hardware implementations, and concludes that learned video compression is poised to be the next major advancement in video coding.","feed_headline":"Learned codecs now beat VVC on perceptual quality","feed_subtitle":"A benchmark across standard test sequences shows bi-directional neural codecs ahead of the VVC reference in MS-SSIM.","key_machinery":"The carrying mechanism is the end-to-end jointly optimized neural video codec, in which learned motion estimation, motion compensation, and entropy-coded latent transforms are trained together against a rate-distortion objective, in contrast to the locally optimized modules of conventional hybrid codecs. Within this design space, the survey identifies conditional coding in feature space as a key architectural shift: instead of transmitting pixel or feature residuals, the decoder conditions reconstruction on learned contextual features, which the survey reports as more efficient than residual coding. Combined with bi-directional prediction and hierarchical GoP structures, this is the architecture behind the models that outperform VTM on MS-SSIM.","core_discovery":"The paper's central claim is that end-to-end learned video compression has matured into a practical alternative to block-based hybrid codecs. Concretely, its rate-distortion benchmark on JCT-VC, UVG, and MCL-JCV sequences shows that bi-directional predictive NVC models are slightly behind VTM-23.0 in PSNR but significantly ahead in MS-SSIM, meaning their reconstructions are perceptually preferable at equal bitrates. Uni-directional models trail VTM in PSNR but are competitive in low-latency settings and benefit from larger GoP sizes. From this the paper concludes that LVC is the next major advancement in video compression, a conclusion reinforced by ongoing standardization activity and demonstrations of real-time decoding on FPGA and mobile devices.","pith_inferences":["If the reported MS-SSIM advantage survives a matched comparison (same color space, same GoP length, full-length sequences), the practical role of PSNR as the default codec benchmark would weaken, and learned codecs could be adopted first in applications where perceptual quality matters more than pixel fidelity.","The benchmark's use of the first 96 frames leaves drift behavior unmeasured; longer sequences and varied content would test whether the perceptual edge persists under error propagation.","The taxonomy suggests a convergence hypothesis: future LVC models will combine bi-directional hierarchical prediction with conditional coding and quantization-aware hardware design, making robustness and adaptation speed the main differentiators.","A direct RGB-to-RGB comparison with VTM, rather than conversion to YUV444, would clarify whether the PSNR gap is a color-space artifact or a genuine coding-efficiency difference."],"forward_implications":["If LVC is truly competitive, next-generation video coding standards will absorb neural tools, and future standardization will focus on interoperability and complexity rather than raw rate-distortion gains.","Perceptual metrics like MS-SSIM will gain weight in codec evaluation, potentially changing how bitrate savings are reported and optimized.","The low-latency strength of uni-directional models makes LVC a plausible candidate for interactive and real-time video services once hardware acceleration matures.","Deployment requirements such as fixed-point quantization and cross-platform bitstream compatibility will shape the next wave of LVC model design."],"supporting_citations":[{"why":"Provides the bi-directional hierarchical motion model behind EEV-0.5, the learned codec with the strongest MS-SSIM results in the benchmark.","marker":"[Ye et al., 2024]"},{"why":"Supplies the DCVC-DC uni-directional codec whose PSNR and MS-SSIM numbers appear in the benchmark tables.","marker":"[Li et al., 2023]"},{"why":"Supplies the DCVC-FM feature-modulation codec evaluated against VTM and HM.","marker":"[Li et al., 2024]"},{"why":"Documents the VVC standard and its reference software, establishing VTM-23.0 as the conventional baseline to beat.","marker":"[Bross et al., 2021]"},{"why":"Introduced an early end-to-end deep video compression framework that the surveyed uni-directional lineage extends.","marker":"[Lu et al., 2019]"},{"why":"Reports the standardization project for end-to-end AI video coding, supporting the survey's claim that LVC is moving toward adoption.","marker":"[Jia et al., 2023]"}],"fun_headline_variants":["Learned codecs edge out VVC in MS-SSIM tests","Survey shows neural codecs beat VVC perceptually","Neural video codecs ahead on perceptual metric","LVC benchmark: bi-directional models lead in MS-SSIM","Perceptual quality: Learned codecs surpass VVC"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark comparisons assume the chosen test configuration is fair and representative: conventional codecs are fed YUV444 converted from RGB and evaluated on the first 96 frames with a GoP of 16 or 32, while learned models operate directly in RGB with GoP 32; if this mismatch systematically favors the learned models, the reported perceptual advantage would not generalize.","fun_headline_variants_meta":{"raw":{"variants":["Learned codecs edge out VVC in MS-SSIM tests","Survey shows neural codecs beat VVC perceptually","Neural video codecs ahead on perceptual metric","LVC benchmark: bi-directional models lead in MS-SSIM","Perceptual quality: Learned codecs surpass VVC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000608,"raw_usage":{"total_tokens":2787,"prompt_tokens":858,"completion_tokens":1929,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":474,"completion_tokens_details":{"reasoning_tokens":1848}},"tokens_in":474,"tokens_out":1929,"duration_ms":14179,"temperature":1.0,"reasoning_tokens":1848,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:02:54.151885+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same rate-distortion comparison on all frames of the JCT-VC, UVG, and MCL-JCV sequences, with the conventional codecs fed RGB directly (or both sides in the same color space), matching GoP sizes (e.g., 32 for both) and using both PSNR and MS-SSIM; if bi-directional learned models no longer beat VTM-23.0 on MS-SSIM under this configuration, the survey's central comparative claim would be undercut.","supporting_citations":[{"cited_title":"Overview of the versatile video coding (vvc) stan- dard and its applications","cited_arxiv_id":null,"evidence_quote":"Documents the VVC standard and its reference software, establishing VTM-23.0 as the conventional baseline to beat."},{"cited_title":"Mpai-eev: Standardiza- tion efforts of artificial intelligence based end-to-end video coding","cited_arxiv_id":null,"evidence_quote":"Reports the standardization project for end-to-end AI video coding, supporting the survey's claim that LVC is moving toward adoption."}],"review_version":1}