REVIEW 3 major objections 6 minor 33 references
Application Space and the Rate-Distortion-Complexity Analysis of Neural Video CODECs
T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read The paper reduces codec choice to a point on a (λ, γ) plane and finds that only five neural codecs can ever be the best choice.
desk verdict The application-space framework is genuinely useful; the empirical winner list is an artifact of four-point sampling. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the cost plane D + λR + γC = 0 in rate-distortion-complexity space. Orthogonal projection onto this plane turns each codec's operating point into a distance proportional to the Lagrangian cost J; the codec's cost is then the minimum or mean of its points' costs, or the segment-weighted average distance along its RDC curve (Eq. 12). The (λ, γ) plane itself is the 'application space': an application's λ and γ come from mapping R, D and C into a common unit, and scanning the plane produces the winner map B(λ, γ) = argmin_i J_i(λ, γ). That map is the mechanism that compresses the 17-codec comparison down to a five-codec answer.
What would settle it
Use the same 17 codecs and 55 sequences, but set C to the measured end-to-end decoding energy per frame instead of decoder kMAC/pixel, and recompute B(λ, γ); if the winner set differs, the five-codec result is an artifact of the chosen complexity axis.
Extended reading notes
Core claim
A codec, the paper argues, should be judged not by RD curves alone but by its set of operating points in a three-dimensional RDC space, using a single linear cost J = D + λR + γC. Each application fixes λ and γ through a linear mapping of rate, distortion and complexity into a common cost (for example, dollars); the codec whose minimum-cost point is closest to the plane D + λR + γC = 0 is the best for that application. Sweeping all (λ, γ) yields a best-codec map B(λ, γ) = argmin_i J_i(λ, γ) over the whole application space. On 17 recent neural video codecs, measured over 55 test sequences, the map is dominated by only five codecs — DCVC-DC, DCVC-FM, MaskCRT, CR16 and HyTIP — and CR16 wins th
Load-bearing premise
The claim that only five codecs can ever win rests on trusting that a codec's complexity is fully captured by its decoder's multiply-accumulate count per pixel and that four sampled quality levels per codec are enough to decide the winner.
Editorial extensions
If this is right
- A standards body or streaming service that accepts the J = D + λR + γC cost can rank any set of neural codecs without ad hoc runtime comparisons: compute each codec's RDC points, pick (λ, γ) from the application's costs, and read the winner off B(λ, γ).
- The five surviving codecs define the effective frontier in RDC space; work on codecs outside this set would need a different complexity metric or a different cost model to justify itself.
- The map gives a quantitative explanation for why one codec wins: in the low-γ region winners alternate between DCVC-DC and HyTIP as λ varies, while CR16 wins wherever complexity is penalized heavily.
- Because the method accepts any cost vector u, the same analysis can be rerun with different complexity axes or with additional dimensions, and the winner map would change accordingly.
- The streaming case study yields a concrete, testable prediction: for the assumed prices, CR16 should be chosen over the other 16 codecs.
Reading between the lines
- Running the same map with 8–10 rate points per codec, rather than four, would test whether the winners' territory is stable; variable-rate codecs like DCVC-DC and DCVC-FM could gain or lose regions.
- An application is rarely a single (λ, γ) point; if an application's requirements are a distribution over the plane, the robust choice would be the codec that wins the largest weighted area, a notion the paper does not develop.
- The linear cost assumption means that adding a fourth dimension (memory bandwidth or encoder complexity) would force a redefinition of C; a multi-objective version of B(λ, γ) would be needed instead of a single scalar map.
- The procedure's practical value is that any organization can substitute its own prices for bandwidth, energy and hardware and get a decision map; the paper's five-codec result is tied to its illustrative streaming prices and its kMAC/pixel choice.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a rate-distortion-complexity (RDC) analysis framework for comparing neural video codecs. It defines a linear Lagrangian cost J = D + λR + γC, argues that an application can be mapped to a (λ,γ) point in an 'application space,' and constructs maps B(λ,γ) that indicate the lowest-cost codec across the application plane. It also discusses extensions of Bjontegaard-delta metrics to 3D. Using four operating points per codec for 17 neural codecs, the paper reports that only a small set of codecs (DCVC-DC, DCVC-FM, MaskCRT, CR16, HyTIP) is ever selected as best, with CR16 chosen for a toy streaming application. The authors are explicit that the results are conditioned on their chosen complexity metric and cost assumptions.
Significance. The methodological contribution—a transparent, linear-cost RDC comparison that yields a best-codec map—is potentially useful for codec selection and standardization discussions. The geometric interpretation of the cost plane and the explicit treatment of complexity as an axis are strengths, and the paper is commendably honest about its assumptions. The framework is generalizable to other complexity measures. However, the empirical winner set is not robustly established: it depends on sparse sampling of variable-rate codecs and on a single complexity scalar, and the abstract/conclusion discrepancy in the number of winning codecs undermines the headline. With additional robustness analysis, the approach could become a practical tool; in its current form, the strong empirical claims outrun the evidence.
major comments (3)
- [§VIII, Eq. (19)] The best-codec map B(λ,γ) is computed by applying min_j to four RDC points per codec. For the variable-rate codecs (DCVC-HEM, DCVC-DC, DCVC-FM), these four points are arbitrary samples of a continuous achievable RDC curve, not the curve itself. A codec whose true lower envelope dips below the sampled hull for some (λ,γ) is therefore incorrectly excluded. This concern is not hypothetical: the authors themselves state in §V that Eq. (12) is the preferred cost for smoothly parameterized codecs and note that variable-rate neural codecs fall in this class, yet the experiments use only Eq. (19)/(20). Denser sampling of the variable-rate codecs, or use of Eq. (12), could change the winner set and the headline 'only five codecs' claim. Please report the sensitivity of B(λ,γ) to the number/placement of operating points.
- [§III and Table I] The entire empirical ranking rests on a single complexity scalar: decoder kMAC/pixel. The paper acknowledges other complexity axes (memory bandwidth, power, encoder cost), but the B(λ,γ) maps are direct functions of this C. Since the central conclusion is that only DCVC-DC, DCVC-FM, MaskCRT, CR16, and HyTIP can be optimal, this conclusion is conditional on that specific choice of C. A different defensible complexity measure, e.g., total MACs including the encoder or measured runtime, could change the set of winners. The authors should either (i) state more prominently that the winner set is a property of the chosen complexity metric, not of the codecs themselves, or (ii) provide a sensitivity analysis over alternative C definitions. Without this, the practical recommendation is underdetermined.
- [Abstract vs. §IX] The abstract states that 'only four neural video codecs came out as the best suited for any application,' while §IX concludes that 'only DCVC-DC, DCVC-FM, MaskCRT, CR16 and HyTIP have a chance of being the best codecs'—five codecs. The same five are visible in the Fig. 8 map described in §VIII. This inconsistency is central to the advertised contribution and must be corrected. Please also verify whether the count refers to the min-based map (Fig. 8) or the mean-based map (Fig. 9), since the two maps can differ.
minor comments (6)
- [§VIII] The text first says 'four RDC points of 17 neural video codecs,' but later says '16 neural video codecs' and '16 costs... for the respective 16 codecs.' Table I lists indices 0–16, i.e., 17 codecs. The count should be made consistent.
- [§V–§VI] The term 'wild guess' for the distortion-to-cost mapping is informal; more importantly, the resulting toy application point (λ≈7.02, γ≈1.14) is illustrative only. Recommend adding an explicit statement that this calibration is a placeholder and that the resulting numerical winner (CR16) is not a validated monetary recommendation.
- [§IV–§V] The 'distance' z_i in Eq. (5) and the 'cost' J in Eq. (13) differ by a constant factor 1/√(1+λ^2+γ^2). Since the factor does not affect minimization, the text should note this equivalence to avoid confusion.
- [Figs. 6–11] The axes are labeled 'dB' for 10log10(λ), 10log10(γ), and 10log10(J), but the reference baseline for the logarithms is not specified. Add the conversion formulas, e.g., that a value of x is displayed as 10log10(x) with x≥0.
- [Eq. (20)] The average cost in Eq. (20) is a codec-level summary but does not correspond to any achievable (R,D,C) point. When the goal is selecting an operating point for an application, the min in Eq. (19) is the appropriate criterion. The paper uses both without explaining the practical meaning of the average-cost map.
- [References] Reference [32] appears to be 'MCL-JVC' (not 'MCL-JCV') based on the dataset name. Please verify and correct.
Circularity Check
No significant circularity: the best-codec map is computed directly from stated R,D,C measurements and the Lagrangian cost; no result is assumed as an input.
full rationale
The paper's central claim — that only a small set of codecs can be optimal across the (lambda,gamma) plane — is obtained by evaluating Eqs. (18) and (19) over the measured R,D values and the tabulated C values. This is a direct computation from stated inputs, not a derivation that assumes its own conclusion. The lower-convex-hull observation in Sec. VI is a standard consequence of minimizing a linear cost over a finite set, not an imported uniqueness theorem. Several compared codecs come from the authors' own prior work ([21], [27], [28], [29]), and the complexity axis is chosen as decoder kMAC/pixel; however, the paper explicitly conditions its conclusions on these choices ('These observations are conditioned on our RDC measures' and 'within our RDC computation constraints'), so the self-citation supplies evaluation objects rather than a load-bearing logical premise. The use of only four operating points per codec, and the decision not to apply Eq. (12) to variable-rate codecs, is a potential representativeness or methodological limitation, but it is not circular: no fitted parameter is renamed as a prediction, and no equation is defined in terms of the result it is used to produce. Therefore no circular step can be exhibited with a specific reduction, and the appropriate score is 0.
Assumptions & free parameters
free parameters (6)
- alpha_1, cost per unit MSE =
5,127 US$
- alpha_2, cost per Mb/s of rate =
36,000 US$
- alpha_3, cost per kMAC/pixel of complexity =
5,843 US$
- lambda and gamma for the toy streaming application =
lambda ~ 7.02, gamma ~ 1.14
- Reference HEVC operating point =
3 Mb/s, 7 kMAC/pixel
- Operating points per codec =
4
assumptions (6)
- domain assumption Cost is a linear combination J = D + lambda R + gamma C
- domain assumption Complexity is a scalar equal to decoder kMAC/pixel
- domain assumption Rate and distortion averages over 55 sequences are representative
- domain assumption Normalization of 12-bit MSE by dividing by 256 makes it comparable to 8-bit MSE
- ad hoc to paper An application is fully described by one (lambda,gamma) point
- domain assumption Codec cost uses min (Eq. 19) or mean (Eq. 20) over available operating points
invented entities (2)
-
Application space (the (lambda,gamma) plane)
-
Cost plane D + lambda R + gamma C = 0
Cite this review
Pith. "Pith review of Application Space and the Rate-Distortion-Complexity Analysis of Neural Video CODECs." pith.science (2026). https://pith.science/paper/G6NE6G4W
@misc{pith2026250905929,
author = {Pith},
title = {Pith review of: Application Space and the Rate-Distortion-Complexity Analysis of Neural Video CODECs},
year = {2026},
howpublished = {\url{https://pith.science/paper/G6NE6G4W}},
note = {Machine review of arXiv:2509.05929}
}
abstract
We study the decision-making process for choosing video compression systems through a rate-distortion-complexity (RDC) analysis. We discuss the 2D Bjontegaard delta (BD) metric and formulate generalizations in an attempt to extend its notions to the 3D RDC volume. We follow that discussion with another one on the computation of metrics in the RDC volume, and on how to define and measure the cost of a coder-decoder (codec) pair, where the codec is characterized by a cloud of points in the RDC space. We use a Lagrangian cost $D+\lambda R + \gamma C$, such that choosing the best video codec among a number of candidates for an application demands selecting appropriate $(\lambda, \gamma)$ values. Thus, we argue that an application may be associated with a $(\lambda, \gamma)$ point in the application space. An example streaming application was given as a case study to set a particular point in the $(\lambda, \gamma)$ plane. The result is that we can compare Lagrangian costs in an RDC volume for different codecs for a given application. Furthermore, we can span the plane and compare codecs for the entire application space filled with different $(\lambda, \gamma)$ choices. We then compared several state-of-the-art neural video codecs using the proposed metrics. Results are informative and surprising. We found that, within our RDC computation constraints, only four neural video codecs came out as the best suited for any application, depending on where its desirable $(\lambda, \gamma)$ lies.
Figures
Figures from the paper (7 more)
Reference graph
Works this paper leans on
-
[1]
Overview of the versatile video coding (vvc) standard and its applications,
B. Bross, Y .-K. Wang, Y . Ye, S. Liu, J. Chen, G. J. Sullivan, and J.- R. Ohm, “Overview of the versatile video coding (vvc) standard and its applications,”IEEE Transactions on Circuits and Systems for Video Technology, vol. 31, no. 10, pp. 3736–3764, 2021
2021
-
[2]
J.-R. Ohm, G. J. Sullivan, H. Schwarz, T. K. Tan, and T. Wiegand, “Com- parison of the coding efficiency of video coding standards—including high efficiency video coding (hevc),”IEEE Transactions on Circuits and Systems for Video Technology, vol. 22, no. 12, pp. 1669–1684, 2012
work page 2012
-
[3]
Toward a practical perceptual video quality metric,
Z. Li, A. Aaron, I. Katsavounidis, A. Moorthy, and M. Manohara, “Toward a practical perceptual video quality metric,” Netflix Tech Blog, 2016, [Online]. Available: https://netflixtechblog.com/ toward-a-practical-perceptual-video-quality-metric-653f208b9652
work page 2016
-
[4]
The Bjøntegaard Bible: Why your way of comparing video codecs may be wrong,
C. Herglotz, H. Och, A. Meyer, G. Ramasubbu, L. Eicherm ¨uller, M. Kr ¨anzler, F. Brand, K. Fischer, D. T. Nguyen, A. Regenskyet al., “The Bjøntegaard Bible: Why your way of comparing video codecs may be wrong,”IEEE Transactions on Image Processing, 2024
work page 2024
-
[5]
Energy-rate-quality tradeoffs of state-of-the-art video codecs,
A. Katsenou, J. Mao, and I. Mavromatis, “Energy-rate-quality tradeoffs of state-of-the-art video codecs,” in2022 Picture Coding Symposium (PCS). IEEE, 2022, pp. 265–269
work page 2022
-
[6]
A Comprehensive Review of Software and Hardware Energy Efficiency of Video Decoders
M. Kr ¨anzler, C. Herglotz, and A. Kaup, “A comprehensive review of software and hardware energy efficiency of video decoders,”arXiv preprint arXiv:2402.09001, 2024
work page Pith review arXiv 2024
-
[7]
Rate-complexity-distortion evaluation for hybrid video coding,
X. Li, M. Wien, and J.-R. Ohm, “Rate-complexity-distortion evaluation for hybrid video coding,” in2010 IEEE International Conference on Multimedia and Expo. IEEE, 2010, pp. 685–690
work page 2010
-
[8]
Advanced video coding for generic audiovisual services,
ITU-T and ISO/IEC, “Advanced video coding for generic audiovisual services,” ITU-T Recommendation H.264 and ISO/IEC 14496-10, 2003, version updated as of 2019
work page 2003
Show all 33 references
-
[9]
High efficiency video coding,
——, “High efficiency video coding,” ITU-T Recommendation H.265 and ISO/IEC 23008-2, 2013, version updated as of 2020
2013
-
[10]
Versatile video coding,
——, “Versatile video coding,” ITU-T Recommendation H.266 and ISO/IEC 23090-3, 2020
2020
-
[11]
Av1: Aomedia video codec,
A. for Open Media, “Av1: Aomedia video codec,” Available online: https://aomedia.org/av1/, 2018
2018
-
[12]
Fast decision of block size, prediction mode, and intra block for h.264 intra prediction,
Y .-H. Huang, T.-S. Ou, and H. H. Chen, “Fast decision of block size, prediction mode, and intra block for h.264 intra prediction,”IEEE Transactions on Circuits and Systems for Video Technology, vol. 20, no. 8, pp. 1122–1132, 2010
2010
-
[13]
Fast h.264/avc to hevc transcoding based on machine learning,
E. Peixoto, B. Macchiavello, R. L. de Queiroz, and E. M. Hung, “Fast h.264/avc to hevc transcoding based on machine learning,” in2014 International Telecommunications Symposium (ITS), 2014, pp. 1–4
2014
-
[14]
Fast cu splitting and pruning for suboptimal cu partitioning in hevc intra coding,
S. Cho and M. Kim, “Fast cu splitting and pruning for suboptimal cu partitioning in hevc intra coding,”IEEE Transactions on Circuits and Systems for Video Technology, vol. 23, no. 9, pp. 1555–1564, 2013
2013
-
[15]
Reducing complexity of hevc: A deep learning approach,
M. Xu, T. Li, Z. Wang, X. Deng, R. Yang, and Z. Guan, “Reducing complexity of hevc: A deep learning approach,”IEEE Transactions on Image Processing, vol. 27, no. 10, pp. 5044–5059, 2018
2018
-
[16]
Fast hevc encoding decisions using data mining,
G. Correa, P. A. Assuncao, L. V . Agostini, and L. A. da Silva Cruz, “Fast hevc encoding decisions using data mining,”IEEE Transactions on Circuits and Systems for Video Technology, vol. 25, no. 4, pp. 660– 673, 2015
2015
-
[17]
Low- complexity ctu partition structure decision and fast intra mode decision for versatile video coding,
H. Yang, L. Shen, X. Dong, Q. Ding, P. An, and G. Jiang, “Low- complexity ctu partition structure decision and fast intra mode decision for versatile video coding,”IEEE Transactions on Circuits and Systems for Video Technology, vol. 30, no. 6, pp. 1668–1682, 2020
2020
-
[18]
Deepqtmt: A deep learning approach for fast qtmt-based cu partition of intra-mode vvc,
T. Li, M. Xu, R. Tang, Y . Chen, and Q. Xing, “Deepqtmt: A deep learning approach for fast qtmt-based cu partition of intra-mode vvc,” IEEE Transactions on Image Processing, vol. 30, pp. 5377–5390, 2021
2021
-
[19]
Cnn oriented complexity reduction of vvc intra encoder,
A. Tissier, W. Hamidouche, J. Vanne, F. Galpin, and D. Menard, “Cnn oriented complexity reduction of vvc intra encoder,” in2020 IEEE International Conference on Image Processing (ICIP), 2020, pp. 3139– 3143
2020
-
[20]
Calculation of average PSNR differences between RD curves,
B. Gisle, “Calculation of average PSNR differences between RD curves,” inITU-T SG16/Q6, 13th VCEG Meeting, Austin, Texas, USA, April 2001, 2001
2001
-
[21]
CANF- VC: Conditional augmented normalizing flows for video compression,
Y .-H. Ho, C.-P. Chang, P.-Y . Chen, A. Gnutti, and W.-H. Peng, “CANF- VC: Conditional augmented normalizing flows for video compression,” inComputer Vision – ECCV 2022, S. Avidan, G. Brostow, M. Ciss ´e, G. M. Farinella, and T. Hassner, Eds. Cham: Springer Nature Switzerland,...
2022
-
[22]
Deep contextual video compression,
J. Li, B. Li, and Y . Lu, “Deep contextual video compression,” in Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y . Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34. Curran Associates, Inc., 2021, pp. 18 114–18 125
2021
-
[23]
Temporal context min- ing for learned video compression,
X. Sheng, J. Li, B. Li, L. Li, D. Liu, and Y . Lu, “Temporal context min- ing for learned video compression,”IEEE Transactions on Multimedia, 2022
2022
-
[24]
Hybrid spatial-temporal entropy modelling for neural video compression,
J. Li, B. Li, and Y . Lu, “Hybrid spatial-temporal entropy modelling for neural video compression,” inProceedings of the 30th ACM Interna- tional Conference on Multimedia, 2022
2022
-
[25]
Neural video compression with diverse contexts,
——, “Neural video compression with diverse contexts,” inIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, V ancouver , Canada, June 18-22, 2023, 2023
2023
-
[26]
Neural video compression with feature modulation,
——, “Neural video compression with feature modulation,” in 12 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 17-21, 2024, 2024
2024
-
[27]
MaskCRT: Masked conditional residual transformer for learned video compression,
Y .-H. Chen, H.-S. Xie, C.-W. Chen, Z.-L. Gao, M. Benjak, W.-H. Peng, and J. Ostermann, “MaskCRT: Masked conditional residual transformer for learned video compression,”IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 11, pp. 11 980–11 992, 2024
2024
-
[28]
On the rate-distortion-complexity trade-offs of neural video coding,
Y .-H. Chen, K.-W. Ho, M. Benjak, J. Ostermann, and W.-H. Peng, “On the rate-distortion-complexity trade-offs of neural video coding,” in2024 IEEE 26th International Workshop on Multimedia Signal Processing (MMSP), 2024, pp. 1–6
2024
-
[29]
Hytip: Hybrid temporal information propagation for masked conditional residual video coding,
Y .-H. Chen, Y .-C. Yao, K.-W. Ho, C.-H. Wu, H.-T. Phung, M. Benjak, J. Ostermann, and W.-H. Peng, “Hytip: Hybrid temporal information propagation for masked conditional residual video coding,” 2025. [Online]. Available: https://arxiv.org/abs/2508.02072
2025 arXiv
-
[30]
UVG dataset: 50/120fps 4K sequences for video codec analysis and development,
A. Mercat, M. Viitanen, and J. Vanne, “UVG dataset: 50/120fps 4K sequences for video codec analysis and development,” inProc. ACM Multimedia Syst. Conf., Istanbul, Turkey, June 2020
2020
-
[31]
Common test conditions and software reference configurations,
F. Bossenet al., “Common test conditions and software reference configurations,”JCTVC-L1100, vol. 12, no. 7, 2013
2013
-
[32]
MCL-JCV: A JND-based H.264/A VC video quality assessment dataset,
H. Wanget al., “MCL-JCV: A JND-based H.264/A VC video quality assessment dataset,” in2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA, 2016, pp. 1509–1513. Ricardo L. de Queiroz(F’17) received the Engineer degree from Universidade de Brasilia , B...
2016
-
[2025]
Peng is a Fellow of the Higher Education Academy (FHEA), and a Fellow of the IEEE
Dr. Peng is a Fellow of the Higher Education Academy (FHEA), and a Fellow of the IEEE. Luciano Volcan Agostini(Senior Member, IEEE) received his M.S. and Ph.D. degrees in Computer Science from the Federal University of Rio Grande do Sul, Brazil, in 2002 and 2007, respectively....
2002
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.