REVIEW 4 major objections 5 minor 36 references
Learned Rate Control for Frame-Level Adaptive Neural Video Compression via Dynamic Neural Network
T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read A single learned video codec, with four coding routes and a bitrate-estimating agent, meets target bitrates to within 1.66% on average while cutting bitrate by 14.8% over a strong baseline.
desk verdict A plausible engineering contribution to frame-level rate control for learned video codecs, but the headline 1.66% bitrate-error figure is a windowed average with I-frames exempt and the rate-control agent's training is unspecified; worth a serious referee but needs revision. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the pair of a Dynamic-Route Autoencoder (DRA) and a Rate Control Agent (RCA) trained by Joint-Routes Optimization (JRO). DRA is a slimmable autoencoder: slimmable convolutions let it run four overlapping sub-networks—routes—that share weights but decode only the first 24, 48, 72, or 96 channels, each route giving a distinct RD trade-off at a distinct computational cost. Channel-wise auto-regression with a Feature Modulation network estimates entropy parameters for each newly added channel slice without re-encoding previous routes. RCA's three modules do the actual control: a Rate Estimation network (conv+FC heads) predicts the bitrate of each route from the curr
What would settle it
Run the agent with the sliding window shrunk to 1 (no averaging) on a high-motion sequence and measure per-frame bitrate error; also compare route choices made by the learned estimator versus an oracle that knows true coded bitrates—if per-frame errors or the choice gap are large, the load-bearing assumption fails.
Extended reading notes
Core claim
In the paper's own terms, the central claim is that rate control for neural video compression can be cast as route selection over a dynamic network. The Dynamic-Route Autoencoder is a slimmable encoder-decoder whose four routes use the first 24, 48, 72, or 96 channels of the latent representation; each route lands on a different point of the rate-distortion curve. The Rate Control Agent estimates the bitrate of every route from the current frame and its motion vector, allocates a target for the current frame with a 30-frame sliding window, and selects the route whose estimated bitrate is closest to that target—above if the clip is under budget, below if it is over. The routes are trained joi
Load-bearing premise
The whole 1.66% rate-control claim depends on the agent's bitrate estimates, learned on Vimeo-90k content, ranking the four routes correctly on unseen HEVC/UVG frames; if those estimates are biased on high-motion or unusual content, the sliding-window allocation smooths over per-frame misses and the reported accuracy would not reflect true frame-level control.
Editorial extensions
If this is right
- A single NVC model can span a range of target bitrates by per-frame route switching, removing the need to train or store separate models per rate point.
- Average bitrate error near 1.66% makes learned codecs viable for bandwidth-constrained delivery, where exceeding the target bitrate fails the contract.
- Computational cost tracks bitrate: the lowest route uses 2.1T MACs and ~0.216 s/frame encoding versus 2.8T and ~0.280 s/frame for the highest, so saving bits also saves compute.
- Joint-Routes Optimization lets shared parameters converge near the global-optimal RD curve; the paper attributes the 14.8% BD-Rate gain over DCVC-HEM to this training strategy.
- The RCA's overhead is small (~0.05 s/frame over the autoencoder), so the rate-control precision does not come at a large latency penalty.
Reading between the lines
- Outside the paper's own tests, the same RCA design could be attached to other variable-rate NVC models—feature modulation or quantizer-step codecs—by training its route-bitrate heads on their operating points; the likely payoff is similar bitrate-error gains at negligible compute overhead.
- The 1.66% average is computed over a 30-frame window with I-frames exempt; per-frame error on hard cuts or fast motion is probably larger, and reporting a percentile or per-frame curve would show how much of the accuracy comes from the window averaging.
- A natural extension is to feed the actual bits consumed by the previous frame back into the agent as an online correction term, turning the open-loop estimate into a closed-loop controller that could shrink the error further.
- The diverging-point training recipe may generalize beyond video: any shared-parameter supernet with multiple capacities could tune each capacity's Lagrange multiplier to where its RD slope separates from the richest subnet, rather than optimizing one global loss.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes DynamicDVC, a learned video compression framework that combines a Dynamic-Route Autoencoder (DRA) with a Rate Control Agent (RCA). The DRA provides four nested coding routes with different channel widths, each targeting a distinct rate-distortion-complexity trade-off. The RCA estimates per-route bitrates from the current frame and reference motion vector, then selects one route per frame to hit a target bitrate using a 30-frame sliding window allocation. The Joint-Routes Optimization (JRO) strategy trains routes collaboratively by decaying Lagrange multipliers toward 'diverging points'. Experiments on HEVC and UVG claim an average BD-Rate reduction of 14.8% and BD-PSNR gain of 0.47 dB relative to DCVC-HEM, an average bitrate error of 1.66% (vs 6.05% for the R-D-λ baseline), and reduced MACs and coding time at lower-bitrate routes. Code is provided.
Significance. If the claims hold, the paper is a useful step toward practical neural video codecs with per-frame rate control and complexity scalability. It combines slimmable routing with a learned bitrate predictor in an NVC pipeline, and the RD performance is competitive with strong learned baselines. The manuscript includes comparisons to multiple baselines, an ablation of the Feature Modulation and JRO components, and publicly available code. However, the headline rate-control claim is not yet demonstrated at frame level, and several implementation details are missing, so the contribution is promising but not fully substantiated.
major comments (4)
- [3.3, Eq. (5), Table 2] The 1.66% bitrate error reported in Table 2 is an average over the sequence/window, computed with a 30-frame sliding window (Eq. 5) and with GoP I-frames exempt from allocation. This does not establish frame-level rate control: large per-frame under/overshoots can cancel within the window. Since the abstract claims 'frame-level adaptive' rate control, the paper should report per-frame bitrate error (e.g., distribution, max, 95th percentile) and the fraction of frames for which the actual bits are within a tolerance of T_tar. Without this, the rate-control accuracy claim is not fully supported.
- [3.3] The Rate Estimation module is central to the method, but its training loss and procedure are not specified. The selection rule in Eq. (6) relies entirely on R_est being an accurate, content-adaptive predictor of actual route bitrates. If R_est is biased (especially on high-motion content), the route selection will be unreliable and the feedback loop in Eq. (5) may absorb the error. Please specify the loss function, training data, and provide a calibration analysis (R_est vs. actual bits) on held-out sequences.
- [Abstract, Section 1, Table 1] The abstract and introduction make inconsistent claims: the abstract states 'average BD-Rate reduction of 14.8% and BD-PSNR gain of 0.47dB over state-of-the-art methods,' while the introduction states 13.1%, and Table 1 shows that the 14.8%/0.47 dB values are specifically versus DCVC-HEM, not all state-of-the-art methods. The abstract should be corrected to name the baseline, and the two numbers should be reconciled.
- [3.4, Algorithm 2] The Joint-Routes Optimization strategy is central to the claimed RD performance, but the 'diverging point' and stopping criterion ('until ξ < ξpre and R < Rpre') are not formalized. Algorithm 2 could terminate before any update if the initial validation condition fails, and there is no evidence that the procedure converges to a 'global-optimal' solution. Please provide a precise definition of the diverging point and per-route RD curves or quantitative ablation results showing the effect of JRO, rather than a single illustrative figure.
minor comments (5)
- [Section 4.1, reference [34]] The HEVC standard test sequences are cited to [34], which is the H.264/AVC overview. The correct reference is likely [29] (the HEVC standard) or the JCT-VC common test conditions document.
- [Figure 8 caption] The caption under Figure 8 appears to be a leftover from another figure ('Non-Linear Transform ... Inverse Transform Quantization ...'), and does not match the ablation plots shown.
- [Algorithm 1, line 5] The loss is written as \sum_{i=0}^{K-1} R + \lambda_i D, but the loop is over route k and R,D are from route k. This should likely be R + \lambda_k D, otherwise the objective is unclear.
- [Section 4.2, Table 3] Table 3 reports coding times only for Route 0 and Route 3, while Table 4 lists all four routes. For completeness, include times for Routes 1 and 2.
- [Section 4.2, Table 2] The R-D-λ baseline comparison reports only per-class average bitrate errors; per-sequence results and test conditions (target rates, GOP length, I-frame interval) are not given, which limits the reproducibility of the rate-control comparison.
Circularity Check
No load-bearing circularity; central claims are external benchmark results and measured rate-control behavior, with only non-load-bearing self-citation to prior Adanic work.
full rationale
The paper's central assertions are empirical comparisons: the 14.8% BD-Rate reduction and 0.47 dB BD-PSNR gain are measured against external baselines (DCVC-HEM, DCVC, etc., Table 1), and the 1.66% bitrate error is a measured system outcome (Table 2) versus the external R-D-λ model. The rate-control loop of Eq. 5 uses actual consumed bits Rcoded in the sliding-window target, so the final average bitrate is not forced by definition; it depends on the learned R_est and the actual bits produced by the chosen route. If R_est were arbitrary, Eq. 6 would not automatically yield 1.66%. The DRA borrows slimmable-route design from the authors' prior Adanic [30], but no claim in this paper is justified solely by that citation, and no uniqueness theorem is imported to rule out alternatives. The RCA training objective is not specified in Sec. 3.3, which is a reproducibility limitation, not a circular step. Thus no derivation reduces to its own inputs or to a self-citation chain.
Assumptions & free parameters
free parameters (6)
- Per-route Lagrange multipliers lambda_0..lambda_3 =
not reported numerically
- Sliding window length SW =
30
- Number of routes K =
4
- Route output channels =
(24, 48, 72, 96)
- JRO decay coefficient kappa and stopping thresholds =
not reported
- I-frame bitrate allocation target =
not specified
assumptions (5)
- standard math Lagrangian relaxation of the rate-control objective (Eq. 1 and Eq. 7) with lambda_i per route
- ad hoc to paper Lower-bitrate routes can share parameters with higher-bitrate routes, and the 'diverged RD model' yields global-optimal joint training
- domain assumption y<=k-1 (partial latent of current route) plus Feature Modulation approximates the previous route's output
- domain assumption Rate Control Agent's bitrate estimates generalize from Vimeo-90k training to HEVC/UVG test sequences
- domain assumption Sliding-window budget redistribution (Eq. 5) is valid within SW=30 frames, and I-frames are exempt from the constraint
invented entities (1)
-
Diverged RD curve model with 'diverging points'
Cite this review
Pith. "Pith review of Learned Rate Control for Frame-Level Adaptive Neural Video Compression via Dynamic Neural Network." pith.science (2026). https://pith.science/paper/LRUW7AFL
@misc{pith2026250820709,
author = {Pith},
title = {Pith review of: Learned Rate Control for Frame-Level Adaptive Neural Video Compression via Dynamic Neural Network},
year = {2026},
howpublished = {\url{https://pith.science/paper/LRUW7AFL}},
note = {Machine review of arXiv:2508.20709}
}
read the original abstract
Neural Video Compression (NVC) has achieved remarkable performance in recent years. However, precise rate control remains a challenge due to the inherent limitations of learning-based codecs. To solve this issue, we propose a dynamic video compression framework designed for variable bitrate scenarios. First, to achieve variable bitrate implementation, we propose the Dynamic-Route Autoencoder with variable coding routes, each occupying partial computational complexity of the whole network and navigating to a distinct RD trade-off. Second, to approach the target bitrate, the Rate Control Agent estimates the bitrate of each route and adjusts the coding route of DRA at run time. To encompass a broad spectrum of variable bitrates while preserving overall RD performance, we employ the Joint-Routes Optimization strategy, achieving collaborative training of various routes. Extensive experiments on the HEVC and UVG datasets show that the proposed method achieves an average BD-Rate reduction of 14.8% and BD-PSNR gain of 0.47dB over state-of-the-art methods while maintaining an average bitrate error of 1.66%, achieving Rate-Distortion-Complexity Optimization (RDCO) for various bitrate and bitrate-constrained applications. Our code is available at https://git.openi.org.cn/OpenAICoding/DynamicDVC.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Mediamakersmeet.com (2022), https://mediamakersmeet.com
Over 82% of internet traffic will be online videos by 2022. Mediamakersmeet.com (2022), https://mediamakersmeet.com
work page 2022
-
[3]
Bjøntegaard, G.: Calculation of average psnr differences between rd-curves (2001), https://api.semanticscholar.org/CorpusID:61598325
work page 2001
-
[4]
Bross, B., Wang, Y.K., Ye, Y., Liu, S., Chen, J., Sullivan, G.J., Ohm, J.R.: Overview of the versatile video coding (vvc) standard and its applications. IEEE Trans. Circuit Syst. Video Technol.31(10), 3736–3764 (2021).https://doi.org/ 10.1109/TCSVT.2021.3101953
-
[5]
Chen, P.Y., Peng, W.H.: CANF-VC++: Enhancing Conditional Augmented Nor- malizing Flows for Video Compression with Advanced Techniques (2023).https: //doi.org/10.48550/arXiv.2309.05382
work page Pith review arXiv doi:10.48550/arxiv.2309.05382 2023
-
[6]
Cui, Z., Wang, J., Gao, S., Guo, T., Feng, Y., Bai, B.: Asymmetric gained deep image compression with continuous rate adaptation. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 10527–10536 (2021).https://doi.org/10.1109/CVPR46437. 2021.01039
-
[7]
He, D., Yang, Z., Peng, W., Ma, R., Qin, H., Wang, Y.: ELIC: efficient learned image compression with unevenly grouped space-channel contextual adaptive cod- ing. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 5708–5717 (2022).https: //doi.org/10.1109/CVPR52688.2022.00563
arXiv 2022
-
[8]
In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T
Ho, Y.H., Chang, C.P., Chen, P.Y., Gnutti, A., Peng, W.H.: CANF-VC: Con- ditional Augmented Normalizing Flows for Video Compression. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Eur. Conf. Comput. Vis. pp. 207–223 (2022).https://doi.org/10.1007/978-3-031-19787-1_12
-
[9]
Hu, Z., Lu, G., Guo, J., Liu, S., Jiang, W., Xu, D.: Coarse-to-fine deep video coding with hyperprior-guided mode prediction. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 5921–5930 (2022)
work page 2022
Show all 36 references
-
[11]
In: IEEE International Work- shop on Multimedia Signal Processing
Ladune, T., Philippe, P., Hamidouche, W., Zhang, L., Déforges, O.: Optical flow and mode selection for learning-based video coding. In: IEEE International Work- shop on Multimedia Signal Processing. pp. 1–6 (2020)
2020
-
[12]
IEEE Trans
Li, B., Li, H., Li, L., Zhang, J.:λ domain rate control algorithm for high efficiency video coding. IEEE Trans. Image Process.23(9), 3841–3854 (2014).https://doi. org/10.1109/TIP.2014.2336550
2014
-
[13]
Li, J., Li, B., Lu, Y.: Deep Contextual Video Compression. In: Adv. Neural Inform. Process. Syst. vol. 34, pp. 18114–18125 (2021)
2021
-
[14]
https://doi
Li, J., Li, B., Lu, Y.: Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression.In:ACMInt.Conf.Multimedia.pp.1503–1511(2022). https://doi. org/10.1145/3503161.3547845
2022
-
[16]
In: ICASSP
Li, Y., Chen, X., Li, J., Wen, J., Han, Y., Liu, S., Xu, X.: Rate Control for Learned Video Compression. In: ICASSP. pp. 2829–2833 (2022). https://doi.org/10. 1109/ICASSP43922.2022.9746080
2022
-
[17]
In: IEEE Conf
Lin, J., Liu, D., Li, H., Wu, F.: M-LVC: Multiple Frames Prediction for Learned Video Compression. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 3543–3551 (2020). https://doi.org/10.1109/CVPR42600.2020.00360
2020
-
[18]
In: IEEE Int
Lin, J., Liu, D., Liang, J., Li, H., Wu, F.: A deeply modulated scheme for variable- rate video compression. In: IEEE Int. Conf. Image Process. pp. 3722–3726 (2021). https://doi.org/10.1109/ICIP42928.2021.9506269
2021
-
[19]
In: IEEE Int
Liu, M., Guo, Y., Li, H., Chen, C.W.: Low-complexity rate control based onρ- domain model for scalable video coding. In: IEEE Int. Conf. Image Process. pp. 1277–1280 (2010). https://doi.org/10.1109/ICIP.2010.5653340
2010
-
[20]
In: IEEE Conf
Lu, G., Ouyang, W., Xu, D., Zhang, X., Cai, C., Gao, Z.: DVC: An End-To-End Deep Video Compression Framework. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 10998–11007 (2019).https://doi.org/10.1109/CVPR.2019.01126
2019
-
[21]
IEEE Trans
Ma, S., Gao, W., Lu, Y.: Rate-distortion analysis for h.264/avc video coding and its application to rate control. IEEE Trans. Circuit Syst. Video Technol.15(12), 1533–1544 (2005). https://doi.org/10.1109/TCSVT.2005.857300
2005
-
[22]
In: Toni, L., Begen, A.C., Alay, Ö., Timmerer, C
Mercat,A.,Viitanen,M.,Vanne,J.:UVGdataset:50/120fps4ksequencesforvideo codec analysis and development. In: Toni, L., Begen, A.C., Alay, Ö., Timmerer, C. (eds.) ACM Multimedia Systems Conference. pp. 297–302 (2020).https://doi. org/10.1145/3339825.3394937
2020
-
[23]
In: IEEE Int
Minnen, D., Singh, S.: Channel-wise autoregressive entropy models for learned image compression. In: IEEE Int. Conf. Image Process. pp. 3339–3343 (2020). https://doi.org/10.1109/ICIP40778.2020.9190935
2020
-
[26]
IEEE Trans
Sheng, X., Li, J., Li, B., Li, L., Liu, D., Lu, Y.: Temporal Context Mining for Learned Video Compression. IEEE Trans. Multimedia25, 7311–7322 (2022). https://doi.org/10.1109/TMM.2022.3220421
2022
-
[27]
Shi, Y., Ge, Y., Wang, J., Mao, J.: AlphaVC: High-performance and efficient learned video compression. In: Eur. Conf. Comput. Vis. pp. 616–631 (2022)
2022
-
[28]
Song, M., Choi, J., Han, B.: Variable-rate deep image compression through spatially-adaptive feature transform. In: Int. Conf. Comput. Vis. pp. 2360–2369 (2021). https://doi.org/10.1109/ICCV48922.2021.00238
2021
-
[29]
IEEE Trans
Sullivan, G.J., Ohm, J.R., Han, W.J., Wiegand, T.: Overview of the high efficiency video coding (hevc) standard. IEEE Trans. Circuit Syst. Video Technol.22(12), 1649–1668 (2012). https://doi.org/10.1109/TCSVT.2012.2221191
2012
-
[30]
Tao, L., Gao, W., Li, G., Zhang, C.: Adanic: Towards practical neural image com- pression via dynamic transform routing. In: Int. Conf. Comput. Vis. pp. 16833– 16842 (2023). https://doi.org/10.1109/ICCV51070.2023.01548
2023
-
[31]
Veit, A., Belongie, S.J.: Convolutional networks with adaptive inference graphs. In: Eur. Conf. Comput. Vis. Lecture Notes in Computer Science, vol. 11205, pp. 3–18 (2018). https://doi.org/10.1007/978-3-030-01246-5_1 Learned Rate Control for Adaptive NVC 17
2018 doi
-
[32]
In: IEEE Conf
Wang, W., Dai, J., Chen, Z., Huang, Z., Li, Z., Zhu, X., Hu, X., Lu, T., Lu, L., Li, H., Wang, X., Qiao, Y.: Internimage: Exploring large-scale vision foundation models with deformable convolutions. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 14408–14419 (2023).https://doi....
2023
-
[34]
IEEE Trans
Wiegand, T., Sullivan, G., Bjontegaard, G., Luthra, A.: Overview of the h.264/avc video coding standard. IEEE Trans. Circuit Syst. Video Technol.13(7), 560–576 (2003). https://doi.org/10.1109/TCSVT.2003.815165
2003
-
[35]
In: IEEE Conf
Wu, Y., Qi, Z., Zheng, H., Tao, L., Gao, W.: Deep image compression with latent optimization and piece-wise quantization approximation. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 1926–1930 (2021)
1926
-
[36]
In: Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., Scarlett, J
Xu, T., Gao, H., Gao, C., Wang, Y., He, D., Pi, J., Luo, J., Zhu, Z., Ye, M., Qin, H., Wang, Y., Liu, J., Zhang, Y.Q.: Bit allocation using optimization. In: Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., Scarlett, J. (eds.) Int. Conf. Mach. Learn. vol. 202, p...
2023
-
[37]
Xue, T., Chen, B., Wu, J., Wei, D., Freeman, W.T.: Video enhancement with task- oriented flow. Int. J. Comput. Vis.127(8), 1106–1125 (2019).https://doi.org/ 10.1007/S11263-018-01144-2
2019 doi
-
[38]
Yang, C., Wang, X., Yao, L., Long, G., Xu, G.: Dyformer: A dynamic transformer- based architecture for multivariate time series classification. Inf. Sci.656, 119881 (2024). https://doi.org/10.1016/J.INS.2023.119881
2024
-
[39]
In: IEEE Conf
Yang, F., Herranz, L., Cheng, Y., Mozerov, M.G.: Slimmable compressive autoen- coders for practical neural image compression. In: IEEE Conf. Comput. Vis. Pat- tern Recog. (2021)
2021
-
[40]
IEEE Trans
Yang, Z., Gao, W., Li, G., Yan, Y.: Sur-driven video coding rate control for jointly optimizing perceptual quality and buffer control. IEEE Trans. Image Process. (2023)
2023
-
[41]
In: AAAI
Zheng, H., Gao, W.: End-to-end rgb-d image compression via exploiting channel- modality redundancy. In: AAAI. vol. 38, pp. 7562–7570 (2024)
2024
-
[42]
In: IEEE Int
Çetin, E., Yılmaz, M.A., Tekalp, A.M.: Flexible-rate learned hierarchical bi- directional video compression with motion refinement and frame-level bit allo- cation. In: IEEE Int. Conf. Image Process. pp. 1206–1210 (2022).https://doi. org/10.1109/ICIP46576.2022.9897455
2022
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.