HAAT: Hybrid Attention Aggregation Transformer for Image Super-Resolution

Ka-Chun Fung; Kai-wen Xue; Kin-man Lam; Song-Jiang Lai; Tsun-Hin Cheung

REVIEW 5 major objections 5 minor 22 references

HAAT: Hybrid Attention Aggregation Transformer for Image Super-Resolution

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read HAAT combines channel, sparse, and window attention to push image super-resolution accuracy ahead of prior state-of-the-art models on standard benchmarks.

desk verdict HAAT's headline SOTA claim is unsupported because Table 1 mixes RGB-channel PSNR for HAAT against published Y-channel baseline numbers, and the reported margins are tiny. read the letter →

arxiv 2411.18003 v3 pith:EXPIX73C submitted 2024-11-27 eess.IV cs.AIcs.CV

Song-Jiang Lai , Tsun-Hin Cheung , Ka-Chun Fung , Kai-wen Xue , Kin-Man Lam This is my paper · ORCID

classification eess.IVcs.AIcs.CV

keywords imagesuper-resolutiontransformerattentionmechanismchannelsparsewindowSwindenseresidualconnection

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces HAAT, a transformer for single-image super-resolution that combines Swin-Dense-Residual-Connected Blocks (SDRCB) with Hybrid Grid Attention Blocks (HGAB). The authors argue that existing window-based transformer models ignore information across channels, and that a hybrid attention that adds channel attention and sparse (grid) attention to shifted-window attention can better fuse nonlocal features. On the Set5 and Set14 benchmarks, HAAT reports the highest PSNR among compared methods at scales x2, x3, and x4, surpassing DRCT by 0.02–0.03 dB. The significance is that it shows a modest but consistent accuracy gain from attending to channel and grid structure rather than only local windows.

What carries the argument

The load-bearing mechanism is the attention split inside HGAB. Input features are copied into a channel-attention branch and split into two halves: one half goes to window-based multi-head self-attention (W-MSA), the other to shifted-window attention (SW-MSA), while a grid multi-head self-attention (Grid-MSA) branch processes a divided feature map with sparse attention. The outputs are concatenated and added to the channel-attention output, then layer-normalized and residually added to the input. SDRCB wraps Swin Transformer Layers in dense residual connections, concatenating features from multiple transition layers before a 1×1 convolution fuses them, with a residual scale of 0.2.

What would settle it

Train DRCT and HAAT from scratch under the same protocol (patch size 256×256, same DF2K training data, same border handling) and compare PSNR on Set5 and Set14; if HAAT no longer leads at all three scales, the claimed advantage is an artifact of training setup.

Watch

Extended reading notes

Core claim

The central discovery claimed is that a hybrid attention aggregation design improves super-resolution above current transformer baselines. HAAT builds on DRCT's deep residual groups but replaces the standard attention block with HGAB, which splits features into parallel branches: channel attention over the whole input, window and shifted-window attention on one half, and grid (sparse) attention on the other half, then merges them with a residual connection. With SDRCB's dense-residual connections expanding the receptive field, the network reaches 38.74 dB PSNR on Set5 at x2, 35.17 dB on Set5 at x3, and 33.12 dB on Set5 at x4, compared with DRCT's 38.72, 35.15, and 33.09 dB, respectively.

Load-bearing premise

The claimed state-of-the-art depends on comparing numbers measured by the authors with numbers from other papers, which used different training patches and possibly different evaluation details; under identical conditions the small margins may not hold.

Editorial extensions

If this is right

The consistent 0.02–0.03 dB advantage over DRCT suggests the hybrid attention branches capture useful information the dense-residual base misses.
Because SDRCB and HGAB are modular, they can be dropped into other transformer-based super-resolution or restoration models and would be expected to transfer their gains.
The reported results establish that channel and grid attention can be combined with shifted windows at a training patch size of 256×256 without instability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The reported margins over DRCT are small enough that a fair re-test with matched training data, patch size, and evaluation border handling could shift which model leads.
Applying the same hybrid attention split to larger windows or to multi-scale grid attention could further improve performance beyond what the paper reports.
Because the evaluation covers only Set5 and Set14, whether the gains generalize to larger or more diverse benchmarks such as Urban100 or Manga109 remains untested.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Xiang, quick take on HAAT: the SOTA claim is not interpretable as presented. Table 1 mixes HAAT's RGB-channel PSNR (all channels, border of 2×scale pixels excluded) with published Y-channel numbers from prior papers, which is the standard in SISR. RGB and Y PSNR are different metrics; the 0.01–0.03 dB margins over DRCT could easily be a protocol artifact. That alone sinks the headline claim.

What is genuinely new: nothing beyond a direct combination. The paper takes SDRCB from DRCT and HGAB from HMANet, bolts them together, and reports numbers. To its credit, it is explicit about the provenance—the introduction says it "enhances the DRCT architecture," and the equations match the cited blocks. No new mechanism, no analysis, no ablation.

The other soft spots are proportionate. Only Set5 and Set14 are reported, the two easiest benchmarks; B100, Urban100, and Manga109 are absent, where the margin might reverse. Training patch size is 256×256 versus 192×192 for most baselines; training data mixes DIV2K-only and DF2K. No code, no error bars, no controlled re-evaluation of baselines under HAAT's protocol. The phrase "significantly better" is an overstatement for 0.02 dB.

The math and equations are consistent; there is no internal contradiction. The citation pattern is honest. But the empirical support is not.

Bottom line: desk reject. The central claim is unsupported by the evidence in the table, and the novelty is too low to warrant referee time. I would not cite it, and I would not bring it to a journal club except as an example of protocol mismatch. If the authors re-evaluate baselines under identical conditions and add the harder benchmarks, the story might be recoverable, but as it stands the paper does not substantiate its title claim.

Referee Report

5 major / 5 minor

Summary. The paper proposes HAAT, a hybrid attention transformer for single-image super-resolution that combines Swin-Dense-Residual-Connected Blocks (SDRCB) with Hybrid Grid Attention Blocks (HGAB). SDRCB integrates Swin Transformer layers with dense-residual connections, while HGAB fuses channel attention, sparse (grid) attention, and window attention. The model is trained on DF2K and evaluated on Set5 and Set14, reporting PSNR/SSIM margins of 0.01–0.03 dB over DRCT and claiming state-of-the-art performance. The paper contains no code, no ablations, no error bars, and evaluates only two small benchmarks.

Significance. If validated, HAAT would be an incremental improvement over existing Transformer-based SR models such as HAT and DRCT. The architecture is a plausible combination of known components, and the dense-residual design is a reasonable direction. However, the only empirical support is a cross-paper comparison on two datasets under mismatched evaluation protocols, so the central significance claim is not currently established. The paper should be credited for clearly describing the architecture and the training/evaluation settings, but it does not provide reproducibility artifacts such as code or detailed per-component analysis.

major comments (5)

[3, Table 1] The comparison in Table 1 is not a like-for-like comparison. The text in Section 3 states that HAAT is evaluated on all RGB channels with the outermost (2 × scale) border pixels excluded, while the baseline entries (EDSR, RCAN, SwinIR, HAT, DRCT, etc.) are taken from their original papers, which report PSNR/SSIM on the Y channel of YCbCr. RGB-channel and Y-channel PSNR are different metrics with image-dependent gaps, so HAAT's 38.74 dB on Set5 ×2 is not numerically comparable to DRCT's 38.72 dB. Please re-evaluate all baselines under HAAT's protocol or report HAAT under the standard Y-channel protocol, and then re-examine the 0.01–0.03 dB margins.
[3, Table 1] Baseline training settings are not controlled. The table mixes models trained on DIV2K only (EDSR, RCAN, SAN, IGNN, HAN, NLSN) with models trained on DF2K (SwinIR, CAT-A, HAT, DAT, DRCT, HAAT), and HAAT uses 256×256 HR patches while HAT and DRCT typically use 192×192. Any of these differences can plausibly explain the reported 0.01–0.03 dB gains. Please provide a controlled comparison, for example by re-running baselines under identical training data, patch size, and evaluation protocol, or by using a shared public implementation with the same settings.
[3, Table 1] The claim that HAAT surpasses state-of-the-art methods is supported only on Set5 and Set14, two small benchmarks. The standard SISR evaluation also includes B100, Urban100, and Manga109, where the relative ordering of HAT/DRCT-type models can differ. Without results on these datasets, the abstract's 'benchmark datasets' claim is an overgeneralization. Please add these datasets or qualify the claim.
[2.2, Eqs. (8)–(9)] The definition of HGAB appears to contain a technical error: Equation (8) computes FM = LN(MAL(Fin)) + Fin, and Equation (9) then computes FM = LN(MAL(FM)) + FM. Since the text says HGAB consists of an MAL followed by an MLP layer, the second equation should use an MLP rather than another MAL. As written, the MLP does not appear in the block, making the architecture description inconsistent and the results hard to reproduce.
[3, Experiments] No ablation study isolates the contributions of SDRCB, HGAB, channel attention, sparse attention, or window attention. Because these are existing components combined in a new arrangement, the claim that the combination yields complementary gains is not evidenced. Please include ablations over the main components and attention branches.

minor comments (5)

[3, Table 1] The training dataset column in Table 1 spells 'DIFK' in several rows; this should be 'DF2K'.
[3, Table 1] The list of compared methods contains 'CDRT' and 'IGN'; these should be 'DRCT' and 'IGNN', respectively.
[2.2, Eq. (7)] The notation 'XM AL' in Equation (7) appears to be a formatting error; it should be 'X_MAL' or a similar subscript.
[2.2] The block is called 'Hybrid Grid Attention Block' in the abstract and Section 2.2 title, but 'Hybrid Global Attention Block' in the body text; please unify the terminology.
[References] Reference [8] for DRCT is cited as an arXiv preprint; if a peer-reviewed version exists, it should be cited instead.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: HAAT is an empirical architecture paper; the SOTA claim rests on benchmark comparisons, not on a derivation that feeds its inputs back into its outputs.

full rationale

The paper's load-bearing claim is an empirical performance comparison, not a derived prediction. HAAT's architecture is defined constructively by Eqs. (1)-(9): SDRCB is a residual-dense composition of Swin Transformer layers and 1x1 convolutions, and HGAB is a channel-split combination of W-MSA, SW-MSA, Grid-MSA, and channel attention. No equation defines an output quantity in terms of the benchmark metrics or in terms of the baseline numbers; the performance numbers in Table 1 come from training on DF2K and evaluating on held-out Set5 and Set14. The baseline entries are quoted from external published papers, and no parameter of HAAT is fitted to the test sets. The paper contains no self-citations by the present authors; its building-block citations (SwinIR, HAT, DRCT, HMANet) are to external work and are used as architectural inspiration, not as an unverified premise that forces the conclusion. The comparability concerns raised by the RGB-channel evaluation and border exclusion are real threats to the validity of the SOTA claim, but they are empirical-protocol issues, not circularity: the claim does not reduce to its inputs by construction. Accordingly, no circular step is identified.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. Its load-bearing assumptions are about comparability of benchmark results and the effectiveness of combining two existing blocks, neither of which is validated by controlled experiments or ablations.

free parameters (6)

residual scaling factor alpha = 0.2
Set to 0.2 for training stability; chosen by hand, not derived.
LeakyReLU negative slope = 0.2
Set to 0.2 in the feature transition convolution; chosen by hand.
channel squeeze factor = 16
Used in the channel attention branch of HGAB; chosen by hand.
window size = 16
Window size for W-MSA and SW-MSA; chosen by hand.
attention head counts = 6, 3, 2
Number of heads for W-MSA, Grid-MSA, and SW-MSA respectively; chosen by hand.
HR patch size = 256x256
Patch size used for training; larger than typical 192x192, affects comparability with baselines.

assumptions (4)

domain assumption Bicubic downsampling is the degradation model used to generate low-resolution training and test images.
Section 3 states LR versions are generated by bicubic downsampling with scales 2, 3, and 4; this is the standard SISR benchmark assumption.
domain assumption PSNR and SSIM on Set5 and Set14 are accepted measures of image super-resolution quality.
Section 3 uses these metrics and datasets for evaluation; the paper assumes they are sufficient to establish SOTA.
ad hoc to paper The combination of SDRCB and HGAB yields complementary gains.
No ablation study is provided to isolate the contribution of each block; the claimed benefit is asserted without direct evidence.
ad hoc to paper Published baseline numbers in Table 1 are directly comparable to HAAT numbers.
The paper does not retrain baselines under identical conditions; it assumes cross-paper comparability despite differences in training patch size and evaluation protocol.

how reviews work

0 comments

Cite this review

Pith. "Pith review of HAAT: Hybrid Attention Aggregation Transformer for Image Super-Resolution." pith.science (2026). https://pith.science/paper/EXPIX73C

@misc{pith2026241118003,
  author       = {Pith},
  title        = {Pith review of: HAAT: Hybrid Attention Aggregation Transformer for Image Super-Resolution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EXPIX73C}},
  note         = {Machine review of arXiv:2411.18003}
}

read the original abstract

In the research area of image super-resolution, Swin-transformer-based models are favored for their global spatial modeling and shifting window attention mechanism. However, existing methods often limit self-attention to non overlapping windows to cut costs and ignore the useful information that exists across channels. To address this issue, this paper introduces a novel model, the Hybrid Attention Aggregation Transformer (HAAT), designed to better leverage feature information. HAAT is constructed by integrating Swin-Dense-Residual-Connected Blocks (SDRCB) with Hybrid Grid Attention Blocks (HGAB). SDRCB expands the receptive field while maintaining a streamlined architecture, resulting in enhanced performance. HGAB incorporates channel attention, sparse attention, and window attention to improve nonlocal feature fusion and achieve more visually compelling results. Experimental evaluations demonstrate that HAAT surpasses state-of-the-art methods on benchmark datasets. Keywords: Image super-resolution, Computer vision, Attention mechanism, Transformer

Figures

Figures reproduced from arXiv: 2411.18003 by the authors.

**Figure 2.** The structure of HGAB. The structure of Hybrid Grid Attention Block (HGAB) is illustrated in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 21 canonical work pages

[1]

Swinir: Image restoration using swin transformer,

Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., and Timofte, R., “Swinir: Image restoration using swin transformer,” in Proceedings of the IEEE/CVF international conference on computer vision, 1833–1844 (2021)

work page 2021
[2]

Swin transformer: Hierarchical vision transformer using shifted windows,

Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B., “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision, 10012–10022 (2021)

work page 2021
[3]

Deep learning for single image super-resolution: A brief review,

Yang, W., Zhang, X., Tian, Y., Wang, W., Xue, J.-H., and Liao, Q., “Deep learning for single image super-resolution: A brief review,” IEEE Transactions on Multimedia21(12), 3106–3121 (2019)

work page 2019
[4]

Accurate image super-resolution using very deep convolutional networks,

Kim, J., Lee, J. K., and Lee, K. M., “Accurate image super-resolution using very deep convolutional networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 1646–1654 (2016)

work page 2016
[5]

Photo-realistic single image super-resolution using a generative adversarial network,

Ledig, C., Theis, L., Husz´ ar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al., “Photo-realistic single image super-resolution using a generative adversarial network,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 4681–4690 (2017)

work page 2017
[6]

Introduction to Bayesian image analysis,

Hanson, K. M., “Introduction to Bayesian image analysis,” in Medical Imaging: Image Processing, Loew, M. H., ed., Proc. SPIE1898, 716–731 (1993)

work page 1993
[7]

Activating more pixels in image super-resolution transformer,

Chen, X., Wang, X., Zhou, J., Qiao, Y., and Dong, C., “Activating more pixels in image super-resolution transformer,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 22367–22377 (2023)

work page 2023
[8]

Drct: Saving image super-resolution away from information bottleneck,

Hsu, C.-C., Lee, C.-M., and Chou, Y.-S., “Drct: Saving image super-resolution away from information bottleneck,” arXiv preprint arXiv:2404.00722(2024)

arXiv 2024

Show all 22 references

[9]

Hmanet: Hybrid multi-axis aggregation network for image super-resolution,

Chu, S.-C., Dou, Z.-C., Pan, J.-S., Weng, S., and Li, J., “Hmanet: Hybrid multi-axis aggregation network for image super-resolution,” arXiv preprint arXiv:2405.05001(2024)

2024 arXiv
[10]

Memnet: A persistent memory network for image restoration,

Tai, Y., Yang, J., Liu, X., and Xu, C., “Memnet: A persistent memory network for image restoration,” in Proceedings of the IEEE international conference on computer vision, 4539–4547 (2017)

2017
[11]

Ntire 2017 challenge on single image super-resolution: Dataset and study,

Agustsson, E. and Timofte, R., “Ntire 2017 challenge on single image super-resolution: Dataset and study,” in Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 126–135 (2017)

2017
[12]

Ntire 2017 challenge on single image super-resolution: Methods and results,

Timofte, R., Agustsson, E., Van Gool, L., Yang, M.-H., and Zhang, L., “Ntire 2017 challenge on single image super-resolution: Methods and results,” in Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 114–125 (2017)

2017
[13]

Low-complexity single-image super- resolution based on nonnegative neighbor embedding,

Bevilacqua, M., Roumy, A., Guillemot, C., and Alberi-Morel, M. L., “Low-complexity single-image super- resolution based on nonnegative neighbor embedding,” (2012)

2012
[14]

On single image scale-up using sparse-representations,

Zeyde, R., Elad, M., and Protter, M., “On single image scale-up using sparse-representations,” in Curves and Surfaces: 7th International Conference, Avignon, France, June 24-30, 2010, Revised Selected Papers 7, 711–730, Springer (2012)

2012
[15]

Enhanced deep residual networks for single image super- resolution,

Lim, B., Son, S., Kim, H., Nah, S., and Mu Lee, K., “Enhanced deep residual networks for single image super- resolution,” in Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 136–144 (2017)

2017
[16]

Image super-resolution using very deep residual channel attention networks,

Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., and Fu, Y., “Image super-resolution using very deep residual channel attention networks,” in Proceedings of the European conference on computer vision (ECCV), 286– 301 (2018)

2018
[17]

Second-order attention network for single image super-resolution,

Dai, T., Cai, J., Zhang, Y., Xia, S.-T., and Zhang, L., “Second-order attention network for single image super-resolution,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 11065–11074 (2019)

2019
[18]

Cross-scale internal graph neural network for image super- resolution,

Zhou, S., Zhang, J., Zuo, W., and Loy, C. C., “Cross-scale internal graph neural network for image super- resolution,” Advances in neural information processing systems33, 3499–3509 (2020)

2020
[19]

Single image super-resolution via a holistic attention network,

Niu, B., Wen, W., Ren, W., Zhang, X., Yang, L., Wang, S., Zhang, K., Cao, X., and Shen, H., “Single image super-resolution via a holistic attention network,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16, 191–...

2020
[20]

Image super-resolution with non-local sparse attention,

Mei, Y., Fan, Y., and Zhou, Y., “Image super-resolution with non-local sparse attention,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 3517–3526 (2021)

2021
[21]

Cross aggregation transformer for image restora- tion,

Chen, Z., Zhang, Y., Gu, J., Kong, L., Yuan, X., et al., “Cross aggregation transformer for image restora- tion,” Advances in Neural Information Processing Systems35, 25478–25490 (2022)

2022
[22]

Dual aggregation transformer for image super-resolution,

Chen, Z., Zhang, Y., Gu, J., Kong, L., Yang, X., and Yu, F., “Dual aggregation transformer for image super-resolution,” in Proceedings of the IEEE/CVF international conference on computer vision, 12312– 12321 (2023)

2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Swinir: Image restoration using swin transformer,

Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., and Timofte, R., “Swinir: Image restoration using swin transformer,” in Proceedings of the IEEE/CVF international conference on computer vision, 1833–1844 (2021)

work page 2021

[2] [2]

Swin transformer: Hierarchical vision transformer using shifted windows,

Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B., “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision, 10012–10022 (2021)

work page 2021

[3] [3]

Deep learning for single image super-resolution: A brief review,

Yang, W., Zhang, X., Tian, Y., Wang, W., Xue, J.-H., and Liao, Q., “Deep learning for single image super-resolution: A brief review,” IEEE Transactions on Multimedia21(12), 3106–3121 (2019)

work page 2019

[4] [4]

Accurate image super-resolution using very deep convolutional networks,

Kim, J., Lee, J. K., and Lee, K. M., “Accurate image super-resolution using very deep convolutional networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 1646–1654 (2016)

work page 2016

[5] [5]

Photo-realistic single image super-resolution using a generative adversarial network,

Ledig, C., Theis, L., Husz´ ar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al., “Photo-realistic single image super-resolution using a generative adversarial network,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 4681–4690 (2017)

work page 2017

[6] [6]

Introduction to Bayesian image analysis,

Hanson, K. M., “Introduction to Bayesian image analysis,” in Medical Imaging: Image Processing, Loew, M. H., ed., Proc. SPIE1898, 716–731 (1993)

work page 1993

[7] [7]

Activating more pixels in image super-resolution transformer,

Chen, X., Wang, X., Zhou, J., Qiao, Y., and Dong, C., “Activating more pixels in image super-resolution transformer,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 22367–22377 (2023)

work page 2023

[8] [8]

Drct: Saving image super-resolution away from information bottleneck,

Hsu, C.-C., Lee, C.-M., and Chou, Y.-S., “Drct: Saving image super-resolution away from information bottleneck,” arXiv preprint arXiv:2404.00722(2024)

arXiv 2024