Pith. sign in

REVIEW 3 major objections 6 minor 28 references

Waterfall Transformer for Multi-person Pose Estimation

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read WTPose, a single-pass transformer for multi-person pose estimation, reaches 77.1 AP on COCO validation by feeding multi-scale backbone features through a waterfall of alternating dilated and local window attention blocks, outperforming…

desk verdict WTPose is a modest but real improvement over Swin for pose estimation; the missing stem-only ablation and thin evaluation make the headline claim too broad. read the letter →

arxiv 2411.18944 v1 pith:SFM7OOW7 submitted 2024-11-28 cs.CV

Navin Ranjan , Bruno Artacho , Andreas Savakis This is my paper · ORCID
classification cs.CV
keywords humanposeestimationmulti-personvisiontransformerwaterfallmoduledilatedneighborhoodattentionmulti-scalefeaturesCOCOkeypointdetectionSwin
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

WTPose is a single-pass, end-to-end trainable model for top-down multi-person 2D pose estimation. It combines a Swin transformer backbone, with its patch-partition layer replaced by two 3×3 convolutions plus a ResNet-101 residual block, with a transformer-based waterfall module that fuses feature maps from all four backbone stages. The waterfall module runs those features through a cascade of alternating dilated and non-dilated window self-attention blocks, with dilation rates (2,1), (4,1), (4,1), (8,1), which enlarges the receptive field while preserving local detail. On COCO validation the model reaches 77.1 AP, beating the Swin-B baseline by 1.2 AP and the much larger Swin-L by 0.8 AP. If this holds up, it suggests that multi-scale feature fusion plus alternating wide and narrow attention is a cheap way to make transformer backbones better at keypoint localization.

What carries the argument

The Waterfall Transformer Module (WTM). It upsamples Swin stage-2, -3, and -4 features to the stage-1 resolution (1/4 scale), concatenates them with low-level ResNet bottleneck features, and reduces channels to 128 with 1×1 convolutions. The resulting tensor flows through four Waterfall Transformer Blocks (WTBs), each pairing a dilated multi-head neighborhood self-attention (D-MHSA) with an MLP and then a non-dilated multi-head neighborhood self-attention (N-MHSA) with an MLP; dilation rates are (2,1), (4,1), (4,1), (8,1) at a window size of 7 with 8 heads. Each WTB branch is concatenated with a depth-wise pooling stream of the input, then low-level features are added and refined with 1×1 and 3×3 convolutions. The mechanism's job is to expand the field of view from 7×7 up to 49×49 while retaining fine local attention, so the decoder gets both global context and local precision.

What would settle it

Train WTPose and its Swin-B baseline several times with different random seeds under identical training settings and compare AP on a held-out test set; if the average gap drops below roughly 0.5 AP or run-to-run intervals overlap, the consistent-improvement claim loses support. A second check is to replace the alternating dilation schedule with non-dilated attention only: if AP does not fall, the dilation pattern is not the active ingredient.

Watch

Extended reading notes

Core claim

The central claim is that waterfall-style multi-scale fusion, implemented with dilated neighborhood attention, gives a consistent performance gain for transformer-based pose estimation. Concretely, WTPose with a modified Swin-B reports AP 77.1, AP75 84.1, AR 82.0 at 89.3M parameters, versus Swin-B at 75.9 AP and Swin-L at 76.3 AP. The ablation in Table 2 separates the gain: adding the waterfall transformer module to Swin-B raises AP from 75.9 to 76.5, and adding the convolutional stem plus ResNet bottleneck raises it further to 77.1. The paper concludes that combining the two components improves feature representation and keypoint accuracy without post-processing.

Load-bearing premise

The reported 1.2-point average-precision gain over Swin-B rests on the specific alternating dilation schedule (2,1), (4,1), (4,1), (8,1) and the stem-plus-bottleneck front-end, both chosen after experiments on the COCO validation set; if that configuration is over-tuned to this validation split, or if the gain is within run-to-run variation, the improvement attributed to the waterfall transformer is not established.

Editorial extensions

If this is right

  • If the result is right, a Swin-B-sized model (89.3M parameters) can beat Swin-L (197M parameters) by 0.8 AP, making the waterfall module a parameter-efficient accuracy boost.
  • The alternating dilation schedule (2,1), (4,1), (4,1), (8,1) outperforms dilated-only rates (2,4,4,8), so interleaving wide and narrow attention is a better recipe inside a cascade.
  • The stem plus ResNet bottleneck front-end contributes roughly 0.6 AP on top of the waterfall module, meaning the input stem matters nearly as much as the multi-scale fusion.
  • WTM's multi-scale fusion also improves hard cases: AP on medium instances rises to 73.4 and on large instances to 83.9, both above the corresponding Swin baselines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same waterfall module should transfer to other hierarchical backbones and dense prediction tasks, since it only consumes multi-scale feature maps and attaches at the decoder side; a direct test would be plugging it into an HRNet-style or ConvNeXt-style backbone for semantic segmentation.
  • Part of the gain may come from the convolutional stem replacing patch partition, a change known to help hierarchical transformers generally; the paper's ablation does not fully separate the stem effect from the waterfall effect beyond the joint table.
  • A natural extension is to feed the 1/8- or 1/16-resolution features into additional WTM branches, since the current design already uses all four stages but branch count and dilation schedule could be tuned per task rather than fixed.
  • The approach could be tested on bottom-up pose estimation by running WTM on full-image features without person detection boxes, following the same waterfall idea used in prior atrous-convolution pose models.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The manuscript proposes WTPose, a top-down multi-person pose estimation framework built on a Swin transformer backbone with a modified stem (two 3x3 convolutions followed by the first ResNet-101 block) and a Waterfall Transformer Module (WTM). The WTM upsamples and concatenates feature maps from all four Swin stages, then applies a cascade of dilated and non-dilated neighborhood attention transformer blocks with dilation rates (2,1), (4,1), (4,1), (8,1). Experiments on COCO train/val 2017 report AP 77.1 at 89.3M parameters, compared with 75.9 for Swin-B, 76.3 for Swin-L, and 76.9 for ViTPose-B. Table 2 ablates the dilation schedule and the stem/ResNet bottleneck, showing that the alternating schedule improves over uniform dilation and that adding the stem/bottleneck further improves AP. The paper concludes that WTPose outperforms other transformer architectures for multi-person pose estimation.

Significance. If the reported result is reproducible, the contribution is a modest but useful empirical improvement: +1.2 AP over Swin-B and +0.8 AP over Swin-L at 54% fewer parameters than Swin-L, achieved with a relatively simple architectural module. The WTM design is clearly inspired by D-WASP and DiNAT, and the paper is transparent about that lineage. The main limitations are evidential rather than conceptual: the results are from a single validation split with no variance estimates, the ablation table does not isolate the stem effect, and the headline claim is broader than the set of compared baselines. These issues are addressable with additional experiments and careful rewording.

major comments (3)
  1. [Section 4.1, Table 2] The ablation table does not include a Swin-B + BN condition without the WTM (i.e., the Stem and ResNet bottleneck added to Swin-B only). As a result, the +0.6 AP gain from row 3 (76.5) to row 4 (77.1) cannot be definitively attributed to the stem; it may reflect an interaction between the stem and the WTM. Please add this missing condition so that the stem effect and the WTM effect can be separated. Note that comparing row 1 (75.9) and row 3 (76.5) already supports a WTM-only gain of 0.6 AP, so this gap is about precise attribution rather than the existence of any improvement.
  2. [Abstract and Section 5] The claim that WTPose "outperforms other transformer architectures for multi-person pose estimation" is broader than the evidence in Table 1, which compares only Swin-B, Swin-L, ViTPose-B, and HRNet (a CNN). Important transformer-based pose estimators such as HRFormer, TokenPose, and TransPose are not evaluated. Please either add these baselines or restrict the claim to the specific transformer baselines compared in the paper.
  3. [Section 4.1, Table 1] Only one COCO validation split is reported, with no repeated seeds, standard deviations, or test-dev numbers. The 0.2 AP advantage over ViTPose-B (77.1 vs 76.9) is within typical run-to-run variability for COCO pose estimation, so the ranking between WTPose and ViTPose-B is not established without additional runs or a test-dev evaluation. At minimum, report the mean and standard deviation over at least three training runs for the main configurations, and ideally provide a test-dev submission.
minor comments (6)
  1. [Section 3] In the sentence following the equations, "N-MSHA" appears to be a typo for "N-MHSA"; please correct it.
  2. [Section 4.1] The phrases "by 1.2%" and "by 0.9%" should be "by 1.2 AP" and "by 0.9 AR" (or "percentage points"), since AP and AR are already reported as percentages.
  3. [Section 3, Eq. (1)] The text says "summation denotes concatenation," but Eq. (1) uses the plus symbol and the text elsewhere says "summation operator" while Figure 3 labels the operation "Concatenate." Please use a consistent notation (e.g., \oplus) and define the concatenation axis explicitly.
  4. [Table 1] The Swin-B and Swin-L rows cite reference [7], which is the MMPose codebase; the Swin architecture should be cited as [14] instead.
  5. [Section 1] The sentence "The contribution of this paper are the following" has a subject-verb agreement error; it should be "The contributions of this paper are the following."
  6. [Section 4] The decoder used to produce the K heatmaps is not described beyond the figure. Please specify its architecture (e.g., the default MMPose decoder used for Swin-B) so that the method is fully reproducible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: WTPose is an empirical architecture comparison; self-citations are acknowledged inspiration and no prediction reduces to a fitted input.

full rationale

The paper makes an empirical architecture claim: WTPose with a modified Swin backbone and waterfall transformer module achieves 77.1 AP on COCO val, compared with 75.9 for Swin-B and 76.9 for ViTPose-B. There is no theoretical derivation chain whose output is equivalent to its input by construction. The WTM equations (Eq. 1 and Eq. 2) are architectural compositions of convolution, upsampling, concatenation, and attention blocks; they do not define the reported accuracy in terms of any fitted parameter. The dilation schedule (2,1), (4,1), (4,1), (8,1) was selected by experimenting on the same validation set, but this is ordinary hyperparameter model selection, not a fitted parameter being renamed as a prediction: the reported AP values are measurements, not algebraic consequences of the chosen rates. Self-citations to UniPose+, OmniPose, and BAPose are explicitly labeled as inspiration for the waterfall design and are not load-bearing evidence for the COCO numbers. The ablation in Table 2 omits the Swin-B + Stem/ResNet-bottleneck condition without WTM, which weakens attribution of the +0.6 AP to the stem versus the WTM, but that is a missing experimental control, not circular reasoning. No step in the paper reduces a claimed result to its own input or to an unverified self-citation chain, so the appropriate circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard training assumptions and on several architecture hyperparameters chosen after validation experiments. No new physical or mathematical entities are introduced. The most important free parameter is the dilation schedule, and the stem/bottleneck also contributes. These choices are common in empirical deep learning, but they mean the result is measured performance after model selection, not a prediction from first principles.

free parameters (4)
  • WTB dilation schedule = (2,1), (4,1), (4,1), (8,1)
    Chosen after experiments on COCO val in Section 4. The alternating schedule is the key mechanism claimed to enlarge receptive fields while preserving local detail.
  • Stem design = two 3x3 convolutions plus first ResNet bottleneck
    Hand-designed front-end. Table 2 shows it adds 0.6 AP over the WTM-only variant, so it contributes directly to the final reported number.
  • Attention heads per WTB = 8
    Selected in the experimental setup; no ablation is shown for this choice.
  • WTM channel reduction = 128
    The concatenated multi-scale features are reduced to 128 channels by a 1x1 convolution; no sensitivity analysis is provided.
assumptions (4)
  • domain assumption COCO annotations and the OKS metric are a valid measure of pose estimation quality
    The paper evaluates only on COCO val and treats AP/AR as the ground truth for its contribution.
  • domain assumption Pretrained Swin-B weights transfer effectively when fine-tuned for pose estimation
    WTPose is initialized with Swin-B pretrained weights from [14]; the stem modification assumes this transfer still works.
  • domain assumption The mmpose default training recipe with AdamW, 210 epochs, and the stated learning rate schedule is sufficient and produces stable results
    No repeated seeds or learning-rate sensitivity are reported.
  • domain assumption Dilated neighborhood attention as defined in DiNAT [9] is a valid, trainable attention mechanism
    The waterfall transformer blocks are built on DiNAT; the paper does not re-derive or verify the mechanism beyond citing it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Waterfall Transformer for Multi-person Pose Estimation." pith.science (2026). https://pith.science/paper/SFM7OOW7

@misc{pith2026241118944,
  author       = {Pith},
  title        = {Pith review of: Waterfall Transformer for Multi-person Pose Estimation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SFM7OOW7}},
  note         = {Machine review of arXiv:2411.18944}
}
read the original abstract

We propose the Waterfall Transformer architecture for Pose estimation (WTPose), a single-pass, end-to-end trainable framework designed for multi-person pose estimation. Our framework leverages a transformer-based waterfall module that generates multi-scale feature maps from various backbone stages. The module performs filtering in the cascade architecture to expand the receptive fields and to capture local and global context, therefore increasing the overall feature representation capability of the network. Our experiments on the COCO dataset demonstrate that the proposed WTPose architecture, with a modified Swin backbone and transformer-based waterfall module, outperforms other transformer architectures for multi-person pose estimation

Figures

Figures reproduced from arXiv: 2411.18944 by the authors.

Figure 1
Figure 1. WTPose examples from the COCO dataset. • We introduce the novel Waterfall Transformer archi￾tecture for pose estimation, a single-pass, end-to-end trainable, multi-scale approach for top-down multi￾person 2D pose estimation. • We propose a waterfall transformer module with multi￾scale attention, that employs a dilated attention mecha￾nism enabling a larger receptive field to capture global and local context. • Our e… view at source ↗
Figure 2
Figure 2. Waterfall transformer framework for multi-person pose estimation. The input color image is fed through the modified Swin [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The proposed waterfall transformer module. The inputs are multi-scale feature maps from all four stages of the Swin backbone [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 21 canonical work pages

  1. [1]

    OmniPose: A Multi-Scale Framework for Multi-Person Pose Estimation

    Bruno Artacho and Andreas Savakis. Omnipose: A multi-scale framework for multi-person pose estima- tion. arXiv preprint arXiv:2103.10180, 2021

    work page Pith review arXiv 2021

  2. [2]

    Unipose+: A unified framework for 2d and 3d human pose es- timation in images and videos

    Bruno Artacho and Andreas Savakis. Unipose+: A unified framework for 2d and 3d human pose es- timation in images and videos. IEEE Trans. Pat- tern Analysis and Machine Intelligence, 44(12):9641– 9653, 2021

    work page 2021

  3. [3]

    BAPose: Bottom-up pose estimation with disentangled water- fall representations

    Bruno Artacho and Andreas Savakis. BAPose: Bottom-up pose estimation with disentangled water- fall representations. In Winter Conference on Applica- tions of Computer Vision Workshops (WACVW), pages 528–537, 2023

    work page 2023

  4. [4]

    Full-BAPose: Bottom up framework for full body pose estimation

    Bruno Artacho and Andreas Savakis. Full-BAPose: Bottom up framework for full body pose estimation. Sensors, 23(7), 2023

    work page 2023

  5. [5]

    Realtime multi-person 2d pose estimation us- ing part affinity fields

    Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation us- ing part affinity fields. InComputer Vision and Pattern Recognition, pages 7291–7299, 2017

    work page 2017

  6. [6]

    Yuille, and Xiaogang Wang

    Xiao Chu, Wei Yang, Wanli Ouyang, Cheng Ma, Alan L. Yuille, and Xiaogang Wang. Multi-context attention for human pose estimation. In Computer Vi- sion and Pattern Recognition (CVPR), 2017

    work page 2017

  7. [7]

    Openmmlab pose estimation toolbox and benchmark

    MMPose Contributors. Openmmlab pose estimation toolbox and benchmark. https://github.com/ open-mmlab/mmpose, 2020

    work page 2020

  8. [8]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    arXiv 2010

Show all 28 references
  1. [9]

    Dilated neigh- borhood attention transformer

    Ali Hassani and Humphrey Shi. Dilated neigh- borhood attention transformer. arXiv preprint arXiv:2209.15001, 2022

    2022 arXiv

  2. [10]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016

    2016

  3. [11]

    Rethinking on multi- stage networks for human pose estimation

    Wenbo Li, Zhicheng Wang, Binyi Yin, Qixiang Peng, Yuming Du, Tianzi Xiao, Gang Yu, Hongtao Lu, Yichen Wei, and Jian Sun. Rethinking on multi- stage networks for human pose estimation. ArXiv, abs/1901.00148, 2019

    1901 arXiv

  4. [12]

    Token- Pose: Learning keypoint tokens for human pose es- timation

    Yanjie Li, Shoukui Zhang, Zhicheng Wang, Sen Yang, Wankou Yang, Shu-Tao Xia, and Erjin Zhou. Token- Pose: Learning keypoint tokens for human pose es- timation. In International Conference on Computer Vision (ICCV), pages 11313–11322, October 2021

    2021

  5. [13]

    Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C

    Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision (ECCV), 2014

    2014

  6. [14]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In International Conference on Computer Vision (ICCV), pages 10012–10022, 2021

    2021

  7. [15]

    Stacked hourglass networks for human pose estimation

    Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In European Conference on Computer Vision (ECCV) , pages 483–499. Springer, 2016

    2016

  8. [16]

    On the convergence of adam and beyond

    Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. arXiv preprint arXiv:1904.09237, 2019

    1904 arXiv

  9. [17]

    15 keypoints is all you need

    Michael Snower, Asim Kadav, Farley Lai, and Hans Peter Graf. 15 keypoints is all you need. InCom- puter Vision and Pattern Recognition (CVPR) , pages 6737–6747, 2020

    2020

  10. [18]

    Deep high-resolution representation learning for hu- man pose estimation

    Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for hu- man pose estimation. In Computer Vision and Pattern Recognition (CVPR), pages 5693–5703, 2019

    2019

  11. [19]

    Deeply learned com- positional models for human pose estimation

    Wei Tang, Pei Yu, and Ying Wu. Deeply learned com- positional models for human pose estimation. In Eu- ropean conference on computer vision (ECCV), pages 190–206, 2018

    2018

  12. [20]

    DeepPose: Human pose estimation via deep neural networks

    Alexander Toshev and Christian Szegedy. DeepPose: Human pose estimation via deep neural networks. In Computer Vision and Pattern Recognition (CVPR) , pages 1653–1660, 2014

    2014

  13. [21]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems , 30, 2017

    2017

  14. [22]

    Deep high- resolution representation learning for visual recogni- tion

    Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high- resolution representation learning for visual recogni- tion. IEEE Trans. Pattern Analysis and Machine In- telligence, 43(10):3349–3364, 2020

    2020

  15. [23]

    Convolutional pose machines

    Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh. Convolutional pose machines. In Com- puter Vision and Pattern Recognition (CVPR) , pages 4724–4732, 2016

    2016

  16. [24]

    Vitpose: Simple vision transformer base- lines for human pose estimation

    Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. Vitpose: Simple vision transformer base- lines for human pose estimation. arXiv preprint arXiv:2204.12484, 2022

    2022 arXiv

  17. [25]

    TransPose: Keypoint localization via transformer

    Sen Yang, Zhibin Quan, Mu Nie, and Wankou Yang. TransPose: Keypoint localization via transformer. In International Conference on Computer Vision (ICCV), pages 11802–11812, October 2021

    2021

  18. [26]

    HRFormer: High-resolution vision transformer for dense predict

    Yuhui Yuan, Rao Fu, Lang Huang, Weihong Lin, Chao Zhang, Xilin Chen, and Jingdong Wang. HRFormer: High-resolution vision transformer for dense predict. In Advances in Neural Information Processing Sys- tems, 2021

    2021

  19. [27]

    Human pose estimation with spatial contextual information

    Hong Zhang, Hao Ouyang, Shu Liu, Xiaojuan Qi, Xi- aoyong Shen, Ruigang Yang, and Jiaya Jia. Human pose estimation with spatial contextual information. arXiv preprint arXiv:1901.01760, 2019

    1901 arXiv

  20. [28]

    3d human pose estimation with spatial and temporal transform- ers

    Ce Zheng, Sijie Zhu, Matias Mendieta, Taojiannan Yang, Chen Chen, and Zhengming Ding. 3d human pose estimation with spatial and temporal transform- ers. In International Conference on Computer Vision (ICCV), pages 11656–11665, 2021

    2021

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.