REVIEW 3 major objections 3 minor 31 references
Compressed Video Super-Resolution based on Hierarchical Encoding
T0 review · 3 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A per-frame hierarchical transformer trained on HEVC-compressed video with perceptual and GAN losses reports higher PSNR, SSIM, MS-SSIM, and VMAF than bicubic, Lanczos, EDSR, CVEGAN, and SwinIR at 4x upscaling.
desk verdict A modest, honest challenge paper whose headline claim overreaches its evidence: the in-distribution CVEGAN comparison is fair and shows small gains, but the EDSR/SwinIR baselines are out-of-distribution and no per-sequence numbers back up 'consistent' improvement. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The core mechanism is the HiET block, a hierarchical encoding transformer layer previously proposed for image super-resolution; here it is repurposed for compressed video upscaling with a specific configuration (window sizes [64, 32, 8, 32, 64], B=6 blocks, 126 channels). The two-stage training objective couples a perceptual loss ($L_p = 0.3L_{\mathrm{L1}} + 0.2L_{\mathrm{SSIM}} + 0.1L_{\mathrm{L2}} + 0.4L_{\mathrm{MS-SSIM}}$) with an adversarial term ($L_{\mathrm{total}} = L_p + 0.05L_{\mathrm{GAN}}$), which together push the network to suppress compression artifacts and synthesize perceptually realistic texture.
What would settle it
Fine-tune EDSR and SwinIR on the same HEVC-compressed training pairs (low-resolution degraded inputs with high-resolution ground truth) and re-run Table I; if the PSNR and VMAF gaps vanish or reverse, the claimed advantage is due to training data rather than the hierarchical encoding architecture.
Extended reading notes
Core claim
The central claim is that an architecture built from hierarchical encoding transformer (HiET) blocks, configured with window sizes [64, 32, 8, 32, 64], six stacked blocks, and a hidden dimension of 126, can serve as a general-purpose upscaler for heavily compressed H.265/HEVC video. Trained in two stages—first with a weighted perceptual loss combining L1, L2, SSIM, and MS-SSIM, then with an additional GAN loss—the model is reported to outperform both classical interpolation and recent image super-resolution networks (EDSR and SwinIR) on both challenge tracks across all four reported metrics. The paper attributes this to the hierarchical attention capturing both local structure and long-range dependencies, and to training on diverse QP values from HEVC HM 18.0.
Load-bearing premise
The reported advantage over EDSR and SwinIR assumes a fair comparison, but those baselines are off-the-shelf models trained on clean image super-resolution while VSR-HE is trained specifically on HEVC-compressed video.
Editorial extensions
If this is right
- If the reported gains hold, per-frame transformer SR can be dropped into existing HEVC decoding pipelines without temporal modeling, simplifying deployment in low-delay conferencing.
- Training on multiple QP values (17–37) suggests robustness across bitrates, so a single model could replace separate artifact-removal stages for different compression levels.
- VMAF improvements of roughly 21 points over bicubic on Track 1 and 19 points over Lanczos on Track 2 indicate meaningful perceptual gains, not just pixel-metric changes.
- The model's 5.43M parameters and 140 ms/frame runtime on an RTX4090 make it a candidate for near-real-time enhancement on modern GPUs.
- Since the method outperforms image SR networks trained on clean data, it implies that training on compressed-domain data is at least as important as network capacity for this task.
Reading between the lines
- An open test is whether the gap over EDSR and SwinIR persists when those baselines are fine-tuned on the same compressed training pairs, which would separate the contribution of training data from the architecture.
- The evaluation covers only H.265/HEVC and five challenge sequences; a broader test across codecs (AV1, VVC) and a larger video corpus would tell whether the advantage is codec-specific.
- Because frames are processed independently, temporal coherence across upscaled frames is not enforced; a temporal extension might further improve perceived quality while preserving the artifact-removal benefits.
- Supplementing with BVI-AOM while using the challenge datasets suggests training-data diversity matters, but a controlled ablation of dataset composition would quantify each source's contribution.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes VSR-HE, a 4x video super-resolution method targeted at HEVC-compressed content. The method uses hierarchical encoding transformer (HiET) blocks with perceptual and GAN-based training, and it is evaluated on the ICME 2025 Grand Challenge validation sequences for both Track 1 (general-purpose content) and Track 2 (talking head videos). The authors report average PSNR-Y, SSIM, MS-SSIM, and VMAF improvements over bicubic/Lanczos upsampling, EDSR, CVEGAN, and SwinIR, along with model complexity and runtime. The central claim is that VSR-HE consistently outperforms both conventional and recent learning-based super-resolution baselines on compressed video.
Significance. If the reported results hold, the paper provides a useful demonstration that a per-frame transformer trained with a perceptual-plus-GAN objective can improve the quality of 4x upscaled, HEVC-compressed video, with a relatively compact model (5.43M parameters). The inclusion of BVI-AOM training content and multiple QP levels is a practical recipe for challenge-style compressed video SR. However, the evaluation is not statistically rigorous: there are no error bars, per-sequence results, or significance tests, and the comparison with EDSR and SwinIR is confounded by the fact that these baselines are used off-the-shelf on out-of-distribution compressed inputs while VSR-HE is trained in-domain. The significance is therefore conditional on the results surviving a fairer and more thorough comparison.
major comments (3)
- [Section III, Table I] The central claim of "consistently outperforms EDSR and SwinIR" is not supported by the presented comparison because EDSR and SwinIR are off-the-shelf models trained on clean, uncompressed image super-resolution, evaluated directly on HEVC-compressed video, whereas VSR-HE is trained specifically on compressed inputs with paired compression QPs (Section II-C). This conflates architecture quality with training-domain match. To support the claim, the authors should either fine-tune the baselines on the same compressed training pairs, or explicitly discuss the domain mismatch as a limitation and temper the claim accordingly.
- [Section III, Table I] No error bars, per-sequence results, or statistical significance tests are reported. With only five test sequences and small margins over the in-domain baseline CVEGAN (e.g., +0.32 dB PSNR and +0.86 VMAF for Track 1), the reported averages may be driven by a few favorable clips. The paper should include per-sequence and per-QP scores, and ideally a measure of variance or significance, to substantiate the word "consistently."
- [Section II-C] The text says "five different quantization parameter (QP) values: 17, 22, 27, 32, 34, and 37," but lists six values; Section III correctly states "six different QPs." Please correct the count and ensure the QP list is unambiguous.
minor comments (3)
- [Throughout] Table references are inconsistent: "TABLE. I" and "TABLE. II" appear in some places and "Table I" in others; unify the formatting.
- [References] Several references contain placeholder text such as "Enter Date Accessed" (e.g., references [2], [3], [4], [5]); complete these before final submission.
- [Section II-A] The phrase "wind ow sizes" contains a formatting error; it should read "window sizes."
Circularity Check
No constructional circularity: the paper's central results are externally measured benchmark scores, and the heavy self-citation is not load-bearing for the reported gains.
full rationale
This is an empirical systems paper rather than a derivation; VSR-HE is an application of the authors' previous HiET block and a CVEGAN-style training loss to the ICME 2025 compressed-video SR task. The reported improvements are measured on external challenge sequences (Table I), not quantities implied by the loss definition: the training losses in Eqs. 1-2 (L1, L2, SSIM, MS-SSIM, GAN) do not include the evaluation metrics PSNR-Y, SSIM, MS-SSIM, or VMAF as fitted targets, so outperforming bicubic, EDSR, SwinIR, and CVEGAN on those metrics is not forced by construction. The comparison against EDSR and SwinIR may be unfair because those baselines are evaluated off-the-shelf without compressed-domain fine-tuning, but that is an experimental-fairness concern, not circularity. Self-citations are numerous, but they provide architecture, loss, and dataset building blocks that are independently checkable, and none is invoked to forbid alternatives or to justify the empirical outcome. The paper makes no uniqueness claim and offers no theoretical reduction, so the central claim retains independent empirical content; the score reflects only the presence of repeated self-citation without load-bearing circularity.
Assumptions & free parameters
free parameters (4)
- loss weights in Eq. (1) =
0.3, 0.2, 0.1, 0.4
- GAN loss weight =
0.05
- HiET hyperparameters =
window sizes [64,32,8,32,64], B=6, channel dim 126
- learning rate schedule =
1e-4 halved at 50k, 100k, 200k, and 300k iterations
assumptions (4)
- domain assumption Bicubic (Track 1) and Lanczos (Track 2) 4x downsampling followed by HM 18.0 HEVC compression at QPs 17 to 37 approximates real-world challenge degradation.
- domain assumption Per-frame independent processing is sufficient for video super-resolution.
- ad hoc to paper HiET blocks from [12] transfer to compressed-domain restoration with only hyperparameter changes.
- domain assumption Average metrics across five test sequences adequately represent model quality.
Cite this review
Pith. "Pith review of Compressed Video Super-Resolution based on Hierarchical Encoding." pith.science (2026). https://pith.science/paper/FC65FGHS
@misc{pith2026250614381,
author = {Pith},
title = {Pith review of: Compressed Video Super-Resolution based on Hierarchical Encoding},
year = {2026},
howpublished = {\url{https://pith.science/paper/FC65FGHS}},
note = {Machine review of arXiv:2506.14381}
}
read the original abstract
This paper presents a general-purpose video super-resolution (VSR) method, dubbed VSR-HE, specifically designed to enhance the perceptual quality of compressed content. Targeting scenarios characterized by heavy compression, the method upscales low-resolution videos by a ratio of four, from 180p to 720p or from 270p to 1080p. VSR-HE adopts hierarchical encoding transformer blocks and has been sophisticatedly optimized to eliminate a wide range of compression artifacts commonly introduced by H.265/HEVC encoding across various quantization parameter (QP) levels. To ensure robustness and generalization, the model is trained and evaluated under diverse compression settings, allowing it to effectively restore fine-grained details and preserve visual fidelity. The proposed VSR-HE has been officially submitted to the ICME 2025 Grand Challenge on VSR for Video Conferencing (Team BVI-VSR), under both the Track 1 (General-Purpose Real-World Video Content) and Track 2 (Talking Head Videos).
Figures
Reference graph
Works this paper leans on
-
[1]
CISCO visual networking index: forecast and method- ology, 2017–2022,
CISCO, “CISCO visual networking index: forecast and method- ology, 2017–2022,” November 2018
work page 2017
-
[2]
Overview of the High Efficiency Video Coding (HEVC) Stan- dard,
G. J. Sullivan, J.-R. Ohm, W.-J. Han, and T. Wiegand, “Overview of the High Efficiency Video Coding (HEVC) Stan- dard,”IEEE Transactions on Circuits and Systems for Video Technology, vol. 22, no. 12, pp. 1649–1668, 2012
work page 2012
-
[3]
“VVCSoftware VTM.” https://vcgit.hhi.fraunhofer.de/jvet/ VVCSoftware VTM. Accessed: Enter Date Accessed
- [4]
-
[5]
“VP9.” https://www.webmproject.org/vp9/. Accessed: Enter Date Accessed
-
[6]
Enhanced Compression Model (ECM) 12.0 Library
Joint Video Experts Team (JVET), “Enhanced Compression Model (ECM) 12.0 Library .” https://vcgit.hhi.fraunhofer.de/ ecm/ECM, 2024. Accessed: 2024-04-10
work page 2024
-
[7]
AOM Video Model (A VM) Codec 2.0.0 Library
Alliance for Open Media, “AOM Video Model (A VM) Codec 2.0.0 Library.” https://gitlab.com/AOMediaCodec/avm, 2024. Accessed: 2024-04-10
work page 2024
-
[8]
Benchmarking conventional and learned video codecs with a low-delay configuration,
S. Teng, Y . Jiang, G. Gao, F. Zhang, T. Davis, Z. Liu, and D. Bull, “Benchmarking conventional and learned video codecs with a low-delay configuration,” in2024 IEEE International Conference on Visual Communications and Image Processing (VCIP), pp. 1–5, IEEE, 2024
work page 2024
Show all 31 references
-
[9]
MTKD: Multi- teacher knowledge distillation for image super-resolution,
Y . Jiang, C. Feng, F. Zhang, and D. Bull, “MTKD: Multi- teacher knowledge distillation for image super-resolution,” arXiv preprint arXiv:2404.09571, 2024
2024 arXiv
-
[10]
SwinIR: Image restoration using swin transformer,
J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, and R. Tim- ofte, “SwinIR: Image restoration using swin transformer,” in Proceedings of the IEEE/CVF international conference on com- puter vision, pp. 1833–1844, 2021
2021
-
[11]
Hiif: Hierarchical encoding based implicit image function for continuous super-resolution,
Y . Jiang, H. M. Kwan, T. Peng, G. Gao, F. Zhang, X. Zhu, J. Sole, and D. Bull, “Hiif: Hierarchical encoding based implicit image function for continuous super-resolution,”arXiv preprint arXiv:2412.03748, 2024
2024
-
[12]
C2d-isr: Optimizing attention-based image super- resolution from continuous to discrete scales,
Y . Jiang, C. Zeng, S. Teng, F. Zhang, X. Zhu, J. Sole, and D. Bull, “C2d-isr: Optimizing attention-based image super- resolution from continuous to discrete scales,”arXiv preprint arXiv:2503.13740, 2025
2025 arXiv
-
[13]
Blind video super-resolution based on implicit kernels,
Q. Zhu, Y . Jiang, S. Zhu, F. Zhang, D. Bull, and B. Zeng, “Blind video super-resolution based on implicit kernels,”arXiv preprint arXiv:2503.07856, 2025
2025
-
[14]
Enhanced deep residual networks for single image super-resolution,
B. Lim, S. Son, H. Kim, S. Nah, and K. Mu Lee, “Enhanced deep residual networks for single image super-resolution,” in Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 136–144, 2017
2017
-
[15]
Instance data condensation for image super-resolution,
T. Peng, H. M. Kwan, Y . Jiang, G. Gao, F. Zhang, X. Xu, S. Liu, and D. Bull, “Instance data condensation for image super-resolution,”arXiv preprint arXiv:2505.21099, 2025
2025
-
[16]
Convolutional neural network-based fractional-pixel motion compensation,
N. Yan, D. Liu, H. Li, B. Li, L. Li, and F. Wu, “Convolutional neural network-based fractional-pixel motion compensation,” IEEE Transactions on Circuits and Systems for Video Technol- ogy, vol. 29, no. 3, pp. 840–853, 2018
2018
-
[17]
Enhancing vvc through cnn- based post-processing,
F. Zhang, C. Feng, and D. R. Bull, “Enhancing vvc through cnn- based post-processing,” in2020 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6, IEEE, 2020
2020
-
[18]
CVEGAN: a perceptually- inspired gan for compressed video enhancement,
D. Ma, F. Zhang, and D. R. Bull, “CVEGAN: a perceptually- inspired gan for compressed video enhancement,”arXiv preprint arXiv:2011.09190, 2020
2011 arXiv
-
[19]
MFRNet: a new CNN architecture for post-processing and in-loop filtering,
D. Ma, F. Zhang, and D. R. Bull, “MFRNet: a new CNN architecture for post-processing and in-loop filtering,”IEEE Journal of Selected Topics in Signal Processing, vol. 15, no. 2, pp. 378–387, 2020
2020
-
[20]
Rtsr: A real-time super-resolution model for av1 compressed content,
Y . Jiang, J. Nawała, C. Feng, F. Zhang, X. Zhu, J. Sole, and D. Bull, “Rtsr: A real-time super-resolution model for av1 compressed content,”arXiv preprint arXiv:2411.13362, 2024
2024 arXiv
-
[21]
Compressing deep image super-resolution models,
Y . Jiang, J. Nawała, F. Zhang, and D. Bull, “Compressing deep image super-resolution models,” in2024 Picture Coding Symposium (PCS), pp. 1–5, IEEE, 2024
2024
-
[22]
Vistra3: Video coding with deep parameter adaptation and post process- ing,
C. Feng, D. Danier, C. Tan, F. Zhang, and D. Bull, “Vistra3: Video coding with deep parameter adaptation and post process- ing,” in2022 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 824–828, IEEE, 2022
2022
-
[23]
Aim 2024 challenge on efficient video super-resolution for av1 compressed content,
M. V . Conde, Z. Lei, W. Li, C. Bampis, I. Katsavouni- dis, and R. Timofte, “Aim 2024 challenge on efficient video super-resolution for av1 compressed content,”arXiv preprint arXiv:2409.17256, 2024
2024 arXiv
-
[24]
Cpga: Coding priors-guided aggregation network for compressed video quality enhancement,
Q. Zhu, J. Hao, Y . Ding, Y . Liu, Q. Mo, M. Sun, C. Zhou, and S. Zhu, “Cpga: Coding priors-guided aggregation network for compressed video quality enhancement,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2964–2974, 2024
2024
-
[25]
Fcvsr: A frequency-aware method for compressed video super- resolution,
Q. Zhu, F. Zhang, F. Chen, S. Zhu, D. Bull, and B. Zeng, “Fcvsr: A frequency-aware method for compressed video super- resolution,”arXiv preprint arXiv:2502.06431, 2025
2025 arXiv
-
[26]
BVI-AOM: A new training dataset for deep video compression optimization,
J. Nawała, Y . Jiang, F. Zhang, X. Zhu, J. Sole, and D. Bull, “BVI-AOM: A new training dataset for deep video compression optimization,”arXiv preprint arXiv:2408.03265, 2024
2024 arXiv
-
[27]
Esrgan: Enhanced super-resolution generative adversarial networks,
X. Wang, K. Yu, S. Wu, J. Gu, Y . Liu, C. Dong, Y . Qiao, and C. Change Loy, “Esrgan: Enhanced super-resolution generative adversarial networks,” inProceedings of the European confer- ence on computer vision (ECCV) workshops, pp. 0–0, 2018
2018
-
[28]
Pytorch: An imperative style, high-performance deep learning library,
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga,et al., “Pytorch: An imperative style, high-performance deep learning library,” Advances in neural information processing systems, vol. 32, 2019
2019
-
[29]
Adam: A method for stochastic optimization,
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”arXiv preprint arXiv:1412.6980, 2014
2014 arXiv
-
[30]
Ntire 2019 challenge on video deblurring and super-resolution: Dataset and study,
S. Nah, S. Baik, S. Hong, G. Moon, S. Son, R. Timofte, and K. Mu Lee, “Ntire 2019 challenge on video deblurring and super-resolution: Dataset and study,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 0–0, 2019
2019
-
[31]
Vcd: A video confer- encing dataset for video compression,
B. Naderi, R. Cutler, N. S. Khongbantabam, Y . Hosseinkashi, H. Turbell, A. Sadovnikov, and Q. Zou, “Vcd: A video confer- encing dataset for video compression,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3970–3974,...
2024
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.