REVIEW 4 major objections 5 minor 36 references
Enhancing Multi-Exposure High Dynamic Range Imaging with Overlapped Codebook for Improved Representation Learning
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper claims that modeling exposure bracketing inside a vector-quantized codebook—via overlapping partial codebooks for short, mid, and long exposures—improves implicit HDR representation learning, and that the resulting dual-decoder…
desk verdict A genuinely new HDR codebook idea with solid ablations, but the central 'overlapped' mechanism is contradicted by Eq. 4 as printed; fix that and release code before trusting the headline claims. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key machinery is the Overlapped Codebook (OLC): a master codebook $\mathcal{Z}=\{z_k\}_{k=1}^K$ partitioned into overlapping segments so that the quantization of an input image of type $\eta$ (short, mid, long, or HDR) uses a conditioned partial codebook $\mathcal{Z}'$: $\mathcal{Z}_\eta$ for LDR frames and the full $\mathcal{Z}$ for HDR. During training, an input type $\eta \sim \mathrm{Unif}[1,4]$ is sampled, the encoder produces features $\bar{z}=E(X)$, and quantization $\hat{z}_j = \arg\min_{z_k \in \mathcal{Z}'}\|\bar{z}_j - z_k\|_2$ maps each feature to the nearest code vector, with straight-through gradient copying and a commitment loss. The pre-trained VQ decoder is then used in the HDR network, where a Parallel Alignment module aligns non-reference frames, a Frame-Selective Merging module aggregates valid frame contexts, and a Residual Fusing module combines VQ decoder features with context via an affine transformation and residual connection.
What would settle it
Inspect the indexing in the released training or inference code (or, if no code is released, directly compute the intersection of the code vector sets used for the short and mid exposure frames in a trained OLC). If the intersection is empty for all codebook updates, the OLC's shared-representation mechanism is absent; the method would then reduce to a per-exposure codebook conditioning scheme, and the ablation against the vanilla codebook would need re-interpretation.
Extended reading notes
Core claim
The central discovery is that the representation space of a VQGAN can be organized to reflect the exposure-bracketing process, so that HDR content is represented by the union of code vectors used for the LDR frames rather than by an unrelated set. Concretely, the OLC uses three partial codebooks $\mathcal{Z}_1,\mathcal{Z}_2,\mathcal{Z}_3$, each meant to contain $K/2$ of the total $K$ code vectors and to overlap with its neighbors by $K/4$ vectors, while the HDR image is quantized against the full codebook. A quantization-conditioning scheme selects the partial or full codebook based on whether the input is an LDR frame or an HDR image. The authors show that this scheme trains a VQ decoder whose latent features, when fused into a dual-decoder HDR network, recover saturated regions and fine details better than the same network with a vanilla codebook, and better than prior CNN- and Transformer-based HDR methods on two public benchmarks; for example, PSNR-$\mu$ reaches 44.89 versus 44.64 on Kalantari and 48.73 versus 48.46 on Hu.
Load-bearing premise
The claimed benefit depends on the partial codebooks used for adjacent LDR frames actually sharing code vectors during training and inference; if the implementation uses disjoint segments, the overlap mechanism described as the core of OLC does not exist, and the reported gains must come from a different component.
Editorial extensions
If this is right
- The OLC improves HDR reconstruction without adding inference-time computation, since the extra cost is paid only during VQGAN training.
- The frozen VQ decoder provides reusable HDR priors: the same pre-trained OLC and decoder are plugged into the HDR network without fine-tuning.
- On Kalantari's dataset the method reports PSNR-$\mu$ 44.89 versus 44.64 for the previous best, and on Hu's dataset 48.73 versus 48.46, with similar gains in HDR-VDP-2.
- Larger codebooks benefit more from the overlap structure (K=1024 gives a larger improvement over vanilla than K=512), so OLC is a scalable way to exploit big codebooks.
- Qualitative comparisons on the Tursun dataset, which has no ground truth, show the method recovering saturated regions that prior attention- and transformer-based methods wash out.
Reading between the lines
- The OLC idea could be applied to other multi-frame fusion tasks—e.g., low-light burst photography or multi-exposure super-resolution—where the input frames share content but differ by a known physical parameter such as exposure, gain, or focus.
- Because the printed equation (4) defines disjoint partial codebooks, the reported behavior depends on the implementation actually using overlapping indexing; verifying the code would determine whether the gains come from the overlap or from the conditioning scheme alone.
- The shared code vectors between neighboring exposures could be inspected to see if they encode exposure-invariant content such as edges and textures, while the exclusive vectors encode exposure-specific radiance; if so, the codebook becomes an interpretable exposure dictionary.
- The mapping loss $\mathcal{L}_{\mathrm{map}}$ that ties the HDR network's latent features to the pre-trained codebook suggests a general recipe: any restoration network can be regularized by a fixed VQ representation, which may help in domains beyond HDR.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a two-step method for multi-exposure HDR imaging. In the first step, a VQGAN is trained with a proposed Overlapped Codebook (OLC), in which LDR frames are quantized with exposure-dependent subsets of a shared codebook while HDR images use the full codebook, with the aim of learning implicit HDR representations aligned with the exposure bracketing process. In the second step, a dual-decoder HDR network combines a pretrained VQ decoder with a fidelity decoder, using a parallel alignment module (PA), a frame-selective merging module (FSM), and a residual fusing module (RF). The method is evaluated on the Kalantari, Hu, and Tursun datasets, reporting improved PSNR, SSIM, HDR-VDP-2, and PU21 scores over prior methods, with ablations attributing gains to the OLC and the proposed modules.
Significance. If the proposed OLC mechanism works as described, the idea of structuring a VQ codebook to match the exposure fusion process is a novel and potentially valuable contribution to learning-based HDR imaging. The reported quantitative gains are consistent and replicated across several metrics and two public benchmarks, and the ablation study gives a plausible attribution of the gains to the OLC and the proposed modules. The paper also includes extensive qualitative comparisons and a code-index distribution analysis. However, the absolute improvements over the strongest prior methods are small (e.g., 44.89 vs. 44.64 PSNR-μ on Kalantari; 48.73 vs. 48.46 on Hu), no statistical significance or error bars are reported, and no code or pretrained models are released, which limits the strength of the central claim.
major comments (4)
- [Sec. 3.1, Eq. (4) and Fig. 1(b)] The printed definition of the partial codebook contradicts the described overlap and the visualization. With α=K/4, Eq. (4) defines Z_i = {z_{iα+1}, ..., z_{(i+1)α}} for i∈{1,2,3}, giving three disjoint blocks of size K/4: Z_1={z_{K/4+1},...,z_{K/2}}, Z_2={z_{K/2+1},...,z_{3K/4}}, Z_3={z_{3K/4+1},...,z_K}. These sets have empty pairwise intersections, so the claimed K/4 shared code vectors between adjacent LDR partial codebooks cannot exist under this definition. The text also calls Z_i a (K/2)×n_z codebook, whereas Eq. (4) yields only K/4 elements. The example in the same paragraph (Z_1 and Z_2 sharing {z_{α+1},...,z_{2α}}) matches the corrected indexing Z_i={z_{(i−1)α+1},...,z_{(i+1)α}}, not the printed one. Since every quantization step in Eq. (5) conditions on Z', this indexing error is load-bearing: if the implementation follows Eq. (4) literally, the OLC is not overlapped at all, and the Table 2 gains cannot be attributed to overlapping codebooks. The authors must correct the formula, state the actual indexing used in the implementation, and ideally release code or checkpoints to resolve the ambiguity.
- [Sec. 3.1, Eq. (10); Sec. 3.2, Eq. (16)] The loss weights in the two final training objectives are not reported. Eq. (10) defines L_OLC = λ_rec L_rec + λ_per L_per + λ_vq L_vq + λ_adv L_adv, and Eq. (16) defines L_HDR = L_rec + λ_per L_per + λ_map L_map, but no values or schedules are given for λ_rec, λ_per, λ_vq, λ_adv, or λ_map. Without these weights, the reported results are not reproducible, and the relative contribution of the VQ/OLC losses to the learned representation cannot be assessed. Please provide the exact values and any annealing schedules, as is standard in VQGAN-based training.
- [Sec. 4.3, Table 1] The quantitative comparison reports only point estimates on test sets of 15 samples (Kalantari) and a similarly small synthetic set (Hu). The reported margins over the best prior methods are small (0.25 dB and 0.27 dB PSNR-μ, respectively), and no error bars, multiple-seed variances, or paired significance tests are given. Given the small test sets and the sensitivity of HDR metrics to a few saturated regions, the authors should report standard deviations over multiple training runs or at least a paired test across the test samples to support the claim of consistent improvement.
- [Sec. 3.2, Eq. (15)] The notation in the mapping loss is confusing and potentially inconsistent. The text says L_map calculates the distance between the extracted feature z_gt in the HDR network and the ground-truth VQ representation z_gt = Q(E(H), Z), but both quantities are named z_gt and the equation as printed is L_map = ||z_vq − z_gt||_2^2. Please rename the extracted feature (e.g., z_vq) and the ground-truth quantized feature (e.g., z_gt), clarify the dimensions, and specify whether H is the ground-truth HDR image provided during training only. This is important because the mapping loss uses ground-truth latents as supervision, which is legitimate but must be precisely defined.
minor comments (5)
- [Sec. 4.3, Table 1 caption] The caption states 'H.V-2 is HDR-VDP-2 metric,' which contains a typo; it should read 'HDR-VDP-2.'
- [Sec. 3.1, Eq. (10)] The adversarial loss definition L_adv = −E_Xhat[D(Xhat))] contains a mismatched parenthesis; the equation should read L_adv = −E_Xhat[D(Xhat)].
- [Sec. 4.2] The phrase 'horizon/vertical flipping' should be 'horizontal/vertical flipping.'
- [Sec. 4.3 and Fig. 6(b)] The comparison on the Tursun dataset is qualitative only because no ground-truth HDR images exist; this limitation should be stated explicitly in the main text, not only in the figure caption, and the claim of 'more realistic HDR images' should be framed as a perceptual visual assessment rather than a quantitative superiority.
- [References] Several references use abbreviated author lists such as 'Grosch, T. & Others' and 'Van Den Oord, A., Vinyals, O. & Others.' Please use the complete author lists or the standard citation style of the venue to improve reproducibility of the bibliography.
Circularity Check
No circularity: the method is trained and evaluated on external benchmarks with no fitted-input predictions or load-bearing self-citations.
full rationale
The paper's derivation chain is not circular. The OLC scheme is an architectural modification of VQGAN codebook indexing (Eqs. 4-6), pre-trained on LDR/HDR pairs from Kalantari and Hu and then evaluated on held-out test splits. The claimed HDR-network improvement is supported by an ablation (Table 2) that trains vanilla VQ and OLC under identical conditions, so the gain is an empirical result rather than a quantity forced by construction. The mapping loss in Eq. 15 uses z_gt = Q(E(H), Z) as a distillation target derived from the ground-truth HDR, not from the network's own prediction, so it does not make the prediction self-defined. There are no load-bearing self-citations or imported uniqueness theorems; prior work is cited only as standard components (VQGAN, HDR datasets, metrics). One non-circular technical flaw should be noted: Eq. 4 with alpha=K/4 defines Z_i={z_{i*alpha+1},...,z_{(i+1)*alpha}} as three disjoint K/4-element blocks, contradicting the text's claim that partial codebooks share K/4 code vectors and the statement that Z_i is in R^{(K/2)*n_z}. This is an internal inconsistency/reproducibility issue, not a circularity, because the OLC benefit is claimed empirically and the equations do not derive the result from themselves. A corrected offset or released code is needed to confirm the mechanism, but the derivation is not self-referential.
Assumptions & free parameters
free parameters (6)
- Loss weights in OLC objective =
not reported
- Loss weights in HDR network =
not reported
- Codebook size K =
1024
- Overlap offset α =
K/4
- Commitment weight β =
0.25
- Base channel C =
32
assumptions (4)
- domain assumption The VQGAN framework with straight-through gradient estimation yields discrete codebook representations that are useful priors for image restoration.
- domain assumption The triangle-weight exposure fusion model of Kalantari et al. (Eqs. 1-2) is a faithful target model for multi-exposure HDR, and mirroring it in a discrete codebook layout improves representation learning.
- domain assumption VGG-16 perceptual similarity and µ-law tone mapping are valid training objectives for HDR reconstruction.
- domain assumption Models trained on Kalantari and Hu training splits generalize to the held-out test samples of the same datasets and to Tursun's no-reference scenes.
Cite this review
Pith. "Pith review of Enhancing Multi-Exposure High Dynamic Range Imaging with Overlapped Codebook for Improved Representation Learning." pith.science (2026). https://pith.science/paper/IUBHW45G
@misc{pith2026250701588,
author = {Pith},
title = {Pith review of: Enhancing Multi-Exposure High Dynamic Range Imaging with Overlapped Codebook for Improved Representation Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/IUBHW45G}},
note = {Machine review of arXiv:2507.01588}
}
read the original abstract
High dynamic range (HDR) imaging technique aims to create realistic HDR images from low dynamic range (LDR) inputs. Specifically, Multi-exposure HDR imaging uses multiple LDR frames taken from the same scene to improve reconstruction performance. However, there are often discrepancies in motion among the frames, and different exposure settings for each capture can lead to saturated regions. In this work, we first propose an Overlapped codebook (OLC) scheme, which can improve the capability of the VQGAN framework for learning implicit HDR representations by modeling the common exposure bracket process in the shared codebook structure. Further, we develop a new HDR network that utilizes HDR representations obtained from a pre-trained VQ network and OLC. This allows us to compensate for saturated regions and enhance overall visual quality. We have tested our approach extensively on various datasets and have demonstrated that it outperforms previous methods both qualitatively and quantitatively
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Grosch, T. & Others Fast and robust high dynamic range image generation with camera and object movement.Vision, Modeling And Visualization, RWTH Aachen. 277284, 2 (2006)
work page 2006
- [2]
-
[3]
Kang, S., Uyttendaele, M., Winder, S. & Szeliski, R. High dynamic range video. ACM Transactions On Graphics (TOG).22, 319-325 (2003)
work page 2003
-
[4]
& Others Neural discrete representation learning
Van Den Oord, A., Vinyals, O. & Others Neural discrete representation learning. Advances In Neural Information Processing Systems.30(2017)
work page 2017
-
[5]
Esser, P., Rombach, R. & Ommer, B. Taming transformers for high-resolution image synthesis.Proceedings Of The IEEE/CVF Conference On Computer Vision And Pattern Recognition. pp. 12873-12883 (2021)
work page 2021
-
[6]
Sen, P., Kalantari, N., Yaesoubi, M., Darabi, S., Goldman, D. & Shechtman, E. Robust patch-based hdr reconstruction of dynamic scenes..ACM Trans. Graph.. 31, 203-1 (2012)
work page 2012
-
[7]
& Others Deep high dynamic range imaging of dynamic scenes..ACM Trans
Kalantari, N., Ramamoorthi, R. & Others Deep high dynamic range imaging of dynamic scenes..ACM Trans. Graph..36, 144-1 (2017)
work page 2017
- [8]
Show all 36 references
-
[9]
& Zhang, Y
Yan, Q., Gong, D., Shi, Q., Hengel, A., Shen, C., Reid, I. & Zhang, Y. Attention- guided network for ghost-free high dynamic range imaging.Proceedings Of The IEEE/CVF Conference On Computer Vision And Pattern Recognition. pp. 1751- 1760 (2019)
2019
-
[10]
& Bengio, Y
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. & Bengio, Y. Generative adversarial networks.Communications Of The ACM.63, 139-144 (2020)
2020
-
[11]
& Lau, R
Niu, Y., Wu, J., Liu, W., Guo, W. & Lau, R. Hdr-gan: Hdr image reconstruction from multi-exposed ldr images with large motions.IEEE Transactions On Image Processing.30pp. 3885-3896 (2021)
2021
-
[12]
& Houlsby, N
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J. & Houlsby, N. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. International Conference On Learn...
2021
-
[13]
& Guo, B
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S. & Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows.Proceedings Of The IEEE/CVF International Conference On Computer Vision. pp. 10012-10022 (2021)
2021
-
[14]
& Yang, M
Zamir, S., Arora, A., Khan, S., Hayat, M., Khan, F. & Yang, M. Restormer: Efficient transformer for high-resolution image restoration.Proceedings Of The IEEE/CVF Conference On Computer Vision And Pattern Recognition. pp. 5728- 5739 (2022)
2022
-
[15]
Kingma, D. & Ba, J. Adam: A Method for Stochastic Optimization.International Conference On Learning Representations (ICLR). (2015)
2015
-
[16]
& Liu, S
Liu, Z., Wang, Y., Zeng, B. & Liu, S. Ghost-free high dynamic range imaging with context-aware transformer.European Conference On Computer Vision. pp. 344-360 (2022)
2022
-
[17]
& Zhang, Y
Yan, Q., Zhang, L., Liu, Y., Zhu, Y., Sun, J., Shi, Q. & Zhang, Y. Deep HDR imaging via a non-local network.IEEE Transactions On Image Processing.29pp. 4308-4322 (2020)
2020
-
[18]
& Heidrich, W
Mantiuk, R., Kim, K., Rempel, A. & Heidrich, W. HDR-VDP-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions.ACM Transactions On Graphics (TOG).30, 1-14 (2011)
2011
-
[19]
& Others PU21: A novel perceptually uniform encoding for adapting existing quality metrics for HDR.2021 Picture Coding Symposium (PCS)
Azimi, M. & Others PU21: A novel perceptually uniform encoding for adapting existing quality metrics for HDR.2021 Picture Coding Symposium (PCS). pp. 1-5 (2021)
2021
-
[20]
& Zisserman, A
Simonyan, K. & Zisserman, A. Very Deep Convolutional Networks for Large-Scale Image Recognition.International Conference On Learning Representations (ICLR). (2015)
2015
-
[21]
& Sun, X
Hu, J., Gallo, O., Pulli, K. & Sun, X. HDR deghosting: How to deal with satu- ration?.Proceedings Of The IEEE Conference On Computer Vision And Pattern Recognition. pp. 1163-1170 (2013)
2013
-
[22]
& Weickert, J
Zimmer, H., Bruhn, A. & Weickert, J. Freehand HDR imaging of moving scenes with simultaneous resolution enhancement.Computer Graphics Forum.30, 405-414 (2011)
2011
-
[23]
& Erdem, E
Tursun, O., Akyüz, A., Erdem, A. & Erdem, E. An objective deghosting quality metric for HDR images.Computer Graphics Forum.35, 139-152 (2016)
2016
-
[24]
& Polley, M
Hu, J., Choe, G., Nadir, Z., Nabil, O., Lee, S., Sheikh, H., Yoo, Y. & Polley, M. Sensor-realistic synthetic data engine for multi-frame high dynamic range photogra- phy.Proceedings Of The IEEE/CVF Conference On Computer Vision And Pattern Recognition Workshops. pp. 516-517 (2...
2020
-
[25]
& Kang, S
Song, J., Park, Y., Kong, K., Kwak, J. & Kang, S. Selective transhdr: Transformer- based selective hdr imaging using ghost region mask.European Conference On Com- puter Vision. pp. 288-304 (2022)
2022
-
[26]
& Guo, S
Chen, C., Shi, X., Qin, Y., Li, X., Han, X., Yang, T. & Guo, S. Real-world blind super-resolution via feature matching with implicit high-resolution priors.Proceed- ings Of The 30th ACM International Conference On Multimedia. pp. 1329-1338 (2022)
2022
-
[27]
& Cheng, M
Gu, Y., Wang, X., Xie, L., Dong, C., Li, G., Shan, Y. & Cheng, M. Vqfr: Blind face restoration with vector-quantized dictionary and parallel decoder.European Conference On Computer Vision. pp. 126-143 (2022)
2022
-
[28]
Wang, X., Girshick, R., Gupta, A. & He, K. Non-local neural networks.Proceedings Of The IEEE Conference On Computer Vision And Pattern Recognition. pp. 7794- 7803 (2018)
2018
-
[29]
& Wang, Y
Guo, B., Zhang, X., Wu, H., Wang, Y., Zhang, Y. & Wang, Y. LAR-SR: A Local Autoregressive Model for Image Super-Resolution.Proceedings Of The IEEE/CVF Conference On Computer Vision And Pattern Recognition. pp. 1909-1918 (2022)
2022
-
[30]
& Malik, J
Debevec, P. & Malik, J. Recovering high dynamic range radiance maps from pho- tographs.Seminal Graphics Papers: Pushing The Boundaries, Volume 2.pp.643-652 (2023)
2023
-
[31]
& Yang, J
Li, X., Wang, W., Hu, X. & Yang, J. Selective kernel networks.Proceedings Of The IEEE/CVF Conference On Computer Vision And Pattern Recognition. pp. 510-519 (2019)
2019
-
[32]
Yan,Q.,Chen,W.,Zhang,S.,Zhu,Y.,Sun,J.&Zhang,Y.AUnifiedHDRImaging Method with Pixel and Patch Level.Proceedings Of The IEEE/CVF Conference On Computer Vision And Pattern Recognition. pp. 22211-22220 (2023)
2023
-
[33]
& Liu, S
Liu, Z., Lin, W., Li, X., Rao, Q., Jiang, T., Han, M., Fan, H., Sun, J. & Liu, S. ADNet: Attention-guided deformable convolutional network for high dynamic range imaging.Proceedings Of The IEEE/CVF Conference On Computer Vision And Pattern Recognition. pp. 463-470 (2021)
2021
-
[34]
& Change Loy, C
Wang, X., Chan, K., Yu, K., Dong, C. & Change Loy, C. Edvr: Video restoration with enhanced deformable convolutional networks.Proceedings Of The IEEE/CVF Conference On Computer Vision And Pattern Recognition Workshops. pp. 0-0 (2019)
2019
-
[35]
& Yuan, S
Chen, R., Zheng, B., Zhang, H., Chen, Q., Yan, C., Slabaugh, G. & Yuan, S. Improving dynamic hdr imaging with fusion transformer.Proceedings Of The AAAI Conference On Artificial Intelligence.37, 340-349 (2023)
2023
-
[36]
& Gin- hac, D
Tel, S., Wu, Z., Zhang, Y., Heyrman, B., Demonceaux, C., Timofte, R. & Gin- hac, D. Alignment-free HDR Deghosting with Semantics Consistent Transformer. Proceedings Of The IEEE/CVF International Conference On Computer Vision. pp. 12836-12845 (2023)
2023
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.