REVIEW 3 major objections 5 minor 38 references
Deep Generalized Max Pooling
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper introduces Deep Generalized Max Pooling, a trainable global pooling layer that equalizes frequent and rare activation vectors and shows it outperforms global average and max pooling on medieval manuscript classification and…
desk verdict Honest adaptation of GMP into a trainable layer with real empirical gains, but the paper misstates its own dimensions and the burstiness-explanation is unverified in the D>N regime. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the closed-form ridge-regression solve over the Gram matrix of local activation vectors. Generalized max pooling was originally a bag-of-words aggregation that equalizes each descriptor's contribution to the image-to-image similarity. The paper turns it into a network layer by identifying each spatial position's depth vector as a local descriptor. With activation vectors $\varphi_i$ and Gram matrix $K$, the weights are $\alpha=(K+\lambda I)^{-1}\mathbf{1}_N$, making frequent activation patterns count no more than rare ones; $\lambda$ interpolates between sum pooling as $\lambda\to\infty$ and the raw least-squares solution as $\lambda\to0$. The layer is differentiable, so $\lambda$ is trained rather than cross-validated, and the authors increase spatial resolution by reducing the last ResNet block's stride to keep the linear solve stable.
What would settle it
Run the same ResNet-50 and training protocol on a standard object-category benchmark where backgrounds are not bursty; if DGMP does not beat global average pooling there, the claimed superiority is specific to repetitive document imagery rather than a general pooling improvement.
Extended reading notes
Core claim
The central claim is that a global pooling layer derived from generalized max pooling produces better image embeddings than the two standard choices. For an activation volume of height $h$, width $w$, and depth $d$, the layer reads each spatial location as a $d$-dimensional vector, forms their Gram matrix $K$, and computes weights $\alpha=(K+\lambda I)^{-1}\mathbf{1}_N$; the pooled vector is the weighted sum $\xi=\Phi\alpha$, normalized to unit norm. Because the closed-form weights are differentiable, $\lambda$ can be trained by backpropagation, and the layer can sit wherever global average pooling usually sits. On writer retrieval with a triplet loss the layer raises mAP from roughly 50% for average or max pooling to 54.8%, and on the CLAMM 16 and CLAMM 17 script-classification benchmarks it gives consistently higher accuracy than both baselines, including a single-model improvement over prior systems on CLAMM 16.
Load-bearing premise
The load-bearing premise is that the vector of CNN activations at one spatial location behaves like a local descriptor in a bag-of-words model, so equalizing frequent and rare such vectors improves deep image embeddings; if deep activation vectors are too highly correlated for the Gram-matrix solve to produce stable weights, the pooling advantage could vanish.
Editorial extensions
If this is right
- Any CNN that ends in global average pooling can swap in this layer at the cost of one extra parameter; the paper demonstrates this on a standard ResNet-50.
- The layer makes image embeddings robust to repetitive visual structure, so performance on script classification and writer identification rises without extra training data or multi-crop testing.
- Since the weights depend only on the current sample's Gram matrix, the pooling adapts per image, not per class or per dataset, while $\lambda$ remains the only learned scalar.
- DGMP can be combined with the same training losses used for embeddings—the paper uses hard triplet mining for writer identification and softmax for script classification—suggesting it is a drop-in pooling layer for metric-learning pipelines.
Reading between the lines
- The same burstiness argument should transfer to other domains with repetitive textures, such as medical imaging, histopathology, or scene text, where background patches dominate the representation; this is a direct extension the paper does not test.
- Because the Gram-matrix solve couples all spatial positions, the layer's benefit should depend on spatial resolution: reducing the stride, as the paper does for script classification, increases the number of equations and stabilizes the weights, so other architectures may need a similar adjustment.
- If the balancing weights are interpreted as a kind of attention, one could combine DGMP with content-based attention; the paper notes this possibility informally, but does not pursue it.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Deep Generalized Max Pooling (DGMP), a global pooling layer that replaces standard global average or max pooling in CNNs. The layer interprets each spatial location's depth vector of activations as a local descriptor and re-weights these locations by solving a ridge-regularized least-squares problem derived from Generalized Max Pooling (GMP), originally developed for bag-of-words models. The closed-form solution for the weights is given in Eq. (9), and the resulting weighted sum is ℓ2-normalized. The authors evaluate DGMP on three historical document analysis tasks: writer identification/retrieval on ICDAR 17-WI and script type classification on CLAMM 16 and CLAMM 17, using a ResNet-50 backbone with different pooling layers. They report that DGMP outperforms average and max pooling in most of their experiments, and they also compare with mixed pooling, log-sum-exp pooling, and state-of-the-art methods.
Significance. If the central claim holds, DGMP would be a simple, differentiable drop-in replacement for global pooling layers that improves embeddings for structured, repetitive images, with only one extra learnable parameter. The paper has several strengths: the closed-form solution in Eqs. (7)-(9) is mathematically correct and the layer is fully differentiable, allowing end-to-end training; the code is publicly available; CLAMM experiments report means and standard deviations over five runs; and the evaluation includes a range of pooling baselines. However, the significance is moderated by the fact that GMP itself is an existing technique and the novelty is primarily in phrasing it as a neural network layer, and by the empirical results being modest and not universally in favor of DGMP. The theoretical justification connecting DGMP to burstiness equalization is also weakened by the dimensional mismatch between the original GMP setting and the actual CNN activation volumes used, which is a central concern addressed in the major comments.
major comments (3)
- [Section III.B] The dimensional claim in Section III.B is incorrect and load-bearing: the text states 'D < N for the normal ResNet50 version with 400×400 input image sizes and N≈D for the modified ResNet50 version,' but in both experimental settings D=2048 while N=169 (13×13 for 400×400 input) or N=441 (21×21 for the stride-reduced version). Thus D>N in both configurations. The original GMP formulation in [7] is motivated for N≫D, where ridge regression balances many descriptors; in the underdetermined regime the linear system in Eq. (7) can be satisfied exactly (or nearly so for small λ) for each sample, so the layer may be overfitting the spatial positions rather than equalizing frequent versus rare activations. The authors should correct the dimensional statements, provide diagnostics such as the distribution and norm of the learned weights α, the condition number of K+λI, or the residual ‖Φ^T ξ − 1_N‖, and discuss whether the reported gains are attributable to burstiness equalization or to sample-specific interpolation.
- [Abstract and Table I] The abstract and contribution list claim that DGMP is 'superior to both average and max pooling' without qualification, but Table I shows a counterexample: for margin 0.5 and λ=10^5, DGMP achieves mAP 45.24 and top-1 64.47, while average pooling achieves 48.60 and 67.19, respectively. The text in Section IV.C.1 later acknowledges that 'the performance of DGMP relative to the average pooling baseline depends on the proper choice of λ,' but this nuance is absent from the abstract and conclusion. The authors should either qualify the superiority claim (e.g., 'for appropriate λ initialization' or 'for small margins') or restrict it to the configurations where it holds. Additionally, no standard deviations or significance tests are reported for the writer-ID experiments in Table I, so it is unclear whether the observed differences are statistically meaningful.
- [Section III.A and IV.C.2] The transfer of GMP from hand-crafted local descriptors to deep CNN activations is not sufficiently justified. In the original GMP framework, each φ(x) is a local descriptor in a large set (N≫D), and the goal is to reduce the influence of bursty descriptors in the global similarity. In Section III.B, the authors instead define φ_i as the depth vector at one spatial location; with D=2048 and N=169 or 441, the number of equations is far smaller than the number of unknowns. The statement in Section IV.C.2 that 'more activation vectors are pooled and the solution for the optimization problem becomes more reliable' addresses only N, not the ratio N/D. The authors should provide empirical evidence that the weight vector α behaves as in the intended GMP regime, for example by comparing weight statistics on real activations before and after training, or by testing with a lower-dimensional feature map where N≫D can be achieved.
minor comments (5)
- [Throughout] There are several typos and inconsistencies: 'fix-sized' in the abstract, inconsistent use of 'CLAMM' vs 'CLaMM', and 'Equ.' vs 'Eq.' in equations. These should be harmonized.
- [Figure 2] The caption says 'ICDAR 17-WI indicative samples' but the sub-caption references IDs such as '11-3-IMG MAX 1005484'; it would be clearer to explicitly state that these are example patches from the dataset.
- [Section IV.C.1] The experimental setup for writer identification omits the batch size and the number of epochs used for validation selection; providing these details would improve reproducibility.
- [Equation (8)] In Eq. (8), the notation '1n' should be '1_N' for consistency with Eq. (6); also, the subscripts in 'αgmp,λ' are not defined before use.
- [Table II] Table II does not report standard deviations or the number of runs for the writer-ID results, unlike the CLAMM tables; adding this information would strengthen the comparison.
Circularity Check
No circularity: DGMP is defined by an unsupervised per-sample linear solve and evaluated on external public benchmarks.
full rationale
The pooling construction is self-contained: Eq. (5) defines the pooled representation as a weighted sum, Eq. (6) imposes constant similarity to every local descriptor, Eq. (7) turns this into a ridge-regression problem, and Eq. (9) is the closed-form solution of the dual formulation. No benchmark label, class frequency, or target metric enters these equations. The claim of superiority is empirical, supported by controlled comparisons in Tables I–III, where the same ResNet-50 backbone, loss, and training schedule are used and only the pooling layer is varied. The learnable ridge parameter lambda is optimized by backpropagation and its initialization is selected with validation data; this is standard hyperparameter tuning, not a fitted prediction of the reported test metrics. The self-citations [3] and [27] motivate generalized max pooling for writer recognition, but the layer's equations are attributed to the external GMP work [7] and the evaluation uses independent public datasets (ICDAR 17-WI, CLAMM 16/17). The skeptical concern about the D>N regime is a numerical-conditioning and generalization risk, not a circularity, because the pooling weights are computed per sample from activations rather than fitted to any external target. No step in the derivation reduces a predicted result to its own input by construction.
Assumptions & free parameters
free parameters (4)
- lambda (DGMP regularization weight) =
writer ID best with init 1e3; CLAMM17 uses 5000; lambda then learned during training
- triplet loss margin m =
0.1 (best); 0.2 and 0.5 also evaluated
- Canny edge sum threshold for patch filtering =
2000
- learning rate multiplier for lambda =
1000 for writer ID; 100 for CLAMM17; none for CLAMM16
assumptions (4)
- standard math Closed-form ridge regression solution and differentiability of the matrix inverse with respect to lambda.
- domain assumption CNN activation depth vectors behave like local descriptors for which GMP's burstiness equalization is beneficial.
- domain assumption The Gram matrix K is well-conditioned enough after adding lambda I, and N is large enough for a reliable solve.
- domain assumption ImageNet-pretrained ResNet-50 features transfer to historical document images.
Cite this review
Pith. "Pith review of Deep Generalized Max Pooling." pith.science (2026). https://pith.science/paper/6GQHCGYA
@misc{pith2026190805040,
author = {Pith},
title = {Pith review of: Deep Generalized Max Pooling},
year = {2026},
howpublished = {\url{https://pith.science/paper/6GQHCGYA}},
note = {Machine review of arXiv:1908.05040}
}
read the original abstract
Global pooling layers are an essential part of Convolutional Neural Networks (CNN). They are used to aggregate activations of spatial locations to produce a fixed-size vector in several state-of-the-art CNNs. Global average pooling or global max pooling are commonly used for converting convolutional features of variable size images to a fix-sized embedding. However, both pooling layer types are computed spatially independent: each individual activation map is pooled and thus activations of different locations are pooled together. In contrast, we propose Deep Generalized Max Pooling that balances the contribution of all activations of a spatially coherent region by re-weighting all descriptors so that the impact of frequent and rare ones is equalized. We show that this layer is superior to both average and max pooling on the classification of Latin medieval manuscripts (CLAMM'16, CLAMM'17), as well as writer identification (Historical-WI'17).
Figures
Reference graph
Works this paper leans on
-
[7]
Interferences in Match Kernels,
N. Murray, H. Jegou, F. Perronnin, and A. Zisserman, “Interferences in Match Kernels,” PAMI, vol. 39, no. 9, pp. 1797–1810, 2016. 1, 2, 3, 4
work page 2016
-
[1]
Convolutional Neural Networks for Font Classification,
C. Tensmeyer, D. Saunders, and T. Martinez, “Convolutional Neural Networks for Font Classification,” in ICDAR, 2017. 1, 2, 6
work page 2017
-
[2]
PHOCNet: A Deep Convolutional Neural Network for Word Spotting in Handwritten Documents,
S. Sudholt and G. A. Fink, “PHOCNet: A Deep Convolutional Neural Network for Word Spotting in Handwritten Documents,” in ICFHR, 2016, pp. 277–282. 1, 2, 7
work page 2016
-
[3]
Unsupervised Feature Learning for Writer Identification and Writer Retrieval,
V . Christlein, M. Gropp, S. Fiel, and A. Maier, “Unsupervised Feature Learning for Writer Identification and Writer Retrieval,” in ICDAR, 2017. 1, 3, 5, 7
work page 2017
-
[4]
Deep Residual Learning for Image Recognition,
K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in CVPR, 2016, pp. 770–778. 1, 2, 4
work page 2016
-
[5]
Densely connected convolutional networks,
G. Huang, Z. Liu, L. v. d. Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in CVPR, July 2017, pp. 2261–2269. 1
work page 2017
-
[6]
Is Object Localization for Free? - Weakly-Supervised Learning with Convolutional Neural Networks,
M. Oquab, L. Bottou, I. Laptev, and J. Sivic, “Is Object Localization for Free? - Weakly-Supervised Learning with Convolutional Neural Networks,” in CVPR, 2015, pp. 685–694. 1
work page 2015
-
[8]
On the Burstiness of Visual Elements,
H. J ´egou, M. Douze, and C. Schmid, “On the Burstiness of Visual Elements,” in CVPR Workshops, 2009, pp. 1169–1176. 2
work page 2009
Show all 38 references
-
[9]
Improving the Fisher Kernel for Large-Scale Image Classification,
F. Perronnin, J. S ´anchez, and T. Mensink, “Improving the Fisher Kernel for Large-Scale Image Classification,” in ECCV, ser. Lecture Notes in Computer Science, K. Daniilidis, P. Maragos, and N. Paragios, Eds. Springer, Berlin, Heidelberg, 2010, pp. 143–156. 2
2010
-
[10]
Generalized Max Pooling,
N. Murray and F. Perronnin, “Generalized Max Pooling,” in CVPR, 2014, pp. 2473–2480. 2
2014
-
[11]
Triangulation Embedding and Democratic Aggregation for Image Search,
H. J ´egou and A. Zisserman, “Triangulation Embedding and Democratic Aggregation for Image Search,” in CVPR, 2014, pp. 3310–3317. 2
2014
-
[12]
Gradient-based Learning Applied to Document Recognition,
Y . LeCun, L. Bottou, Y . Bengio, and P. Haffner, “Gradient-based Learning Applied to Document Recognition,” Proceedings of the IEEE , vol. 86, no. 11, pp. 2278–2324, 1998. 2
1998
- [13]
-
[14]
Learning and Transferring Mid-Level Image Representations using Convolutional Neural Networks,
M. Oquab, L. Bottou, I. Laptev, and J. Sivic, “Learning and Transferring Mid-Level Image Representations using Convolutional Neural Networks,” in CVPR, 2014, pp. 1717–1724. 2
2014
-
[15]
Generalizing pooling functions in convolutional neural networks: Mixed, gated, and tree,
C.-Y . Lee, P. W. Gallagher, and Z. Tu, “Generalizing pooling functions in convolutional neural networks: Mixed, gated, and tree,” in 19th International Conference on Artificial Intelligence and Statistics , ser. Machine Learning Research, A. Gretton and C. C. Robert, Eds., vol...
2016
-
[16]
From image-level to pixel-level labeling with convolutional networks,
P. O. Pinheiro and R. Collobert, “From image-level to pixel-level labeling with convolutional networks,” in CVPR, June 2015, pp. 1713–1721. 2
2015
-
[17]
Clustering of medieval scripts through computer image analysis: towards an evaluation protocol,
D. Stutzmann, “Clustering of medieval scripts through computer image analysis: towards an evaluation protocol,” Digital Medievalist, vol. 10,
-
[18]
Artificial Paleography: Computational Approaches to Identifying Script Types in Medieval Manuscripts,
M. Kestemont, V . Christlein, and D. Stutzmann, “Artificial Paleography: Computational Approaches to Identifying Script Types in Medieval Manuscripts,” Speculum, vol. 92, no. S1, pp. 86–109, 2017. 2, 6
2017
-
[19]
ICFHR2016 Competition on the Classification of Medieval Handwritings in Latin Script,
F. Cloppet, V . Eglin, V . C. Kieu, D. Stutzmann, N. Vincent, V . ´Eglin, V . C. Kieu, D. Stutzmann, and N. Vincent, “ICFHR2016 Competition on the Classification of Medieval Handwritings in Latin Script,” in ICFHR, 2016, pp. 590–595. 2, 4
2016
-
[20]
ICDAR2017 Competition on the Classification of Medieval Handwritings in Latin Script,
F. Cloppet, V . Eglin, M. Helias-Baron, C. Kieu, N. Vincent, and D. Stutzmann, “ICDAR2017 Competition on the Classification of Medieval Handwritings in Latin Script,” in ICDAR, 2017, pp. 1371–
2017
-
[21]
Handwriting Analysis with Focus on Writer Identification and Writer Retrieval,
V . Christlein, “Handwriting Analysis with Focus on Writer Identification and Writer Retrieval,” Ph.D. dissertation, FAU Erlangen-N ¨urnberg, 2018. 2, 6
2018
-
[22]
Writer Identification and Retrieval Using a Convolutional Neural Network,
S. Fiel and R. Sablatnig, “Writer Identification and Retrieval Using a Convolutional Neural Network,” in CAIP, G. Azzopardi and N. Petkov, Eds. Springer International Publishing, 2015, pp. 26–37. 3
2015
-
[23]
Offline Writer Identification Using Convolutional Neural Network Activation Features,
V . Christlein, D. Bernecker, A. Maier, and E. Angelopoulou, “Offline Writer Identification Using Convolutional Neural Network Activation Features,” in GCPR, J. Gall, P. Gehler, and B. Leibe, Eds., 2015, pp. 540–552. 3
2015
-
[24]
Text-Independent Writer Identification via CNN Features and Joint Bayesian,
Y . Tang and X. Wu, “Text-Independent Writer Identification via CNN Features and Joint Bayesian,” in ICFHR, 2016, pp. 566–571. 3
2016
-
[25]
Semi-supervised feature learning for improving writer identification,
S. Chen, Y . Wang, C.-T. Lin, W. Ding, and Z. Cao, “Semi-supervised feature learning for improving writer identification,” Information Sciences, vol. 482, pp. 156 – 170, 2019. 3
2019
-
[26]
Aggregating Local Image Descriptors into Compact Codes,
H. J ´egou, F. Perronnin, M. Douze, J. S ´anchez, P. P ´erez, and C. Schmid, “Aggregating Local Image Descriptors into Compact Codes,” PAMI, vol. 34, no. 9, pp. 1704–1716, 2012. 3
2012
-
[27]
Encoding CNN Activations for Writer Recognition,
V . Christlein and A. Maier, “Encoding CNN Activations for Writer Recognition,” in 13th IAPR International Workshop on Document Analysis Systems, 2018, pp. 169—-174. 3
2018
-
[28]
Deep adaptive learning for writer identification based on single handwritten word images,
S. He and L. Schomaker, “Deep adaptive learning for writer identification based on single handwritten word images,” Pattern Recognition, vol. 88, pp. 64 – 74, 2019. 3
2019
-
[29]
Learning Features for Writer Retrieval and Identification using Triplet CNNs,
M. Keglevic, S. Fiel, and R. Sablatnig, “Learning Features for Writer Retrieval and Identification using Triplet CNNs,” in ICFHR, 2018, pp. 211–216. 3
2018
-
[30]
Precision Learning: Towards Use of Known Operators in Neural Networks,
A. Maier, F. Schebesch, C. Syben, T. W ¨urfl, S. Steidl, J.-H. Choi, and R. Fahrig, “Precision Learning: Towards Use of Known Operators in Neural Networks,” in ICPR, J. K. T. Tan, Ed., 2018, pp. 183–188. 3
2018
-
[31]
ICDAR2017 Competition on Historical Document Writer Identification (Historical-WI),
S. Fiel, F. Kleber, M. Diem, V . Christlein, G. Louloudis, N. Stamatopou- los, and B. Gatos, “ICDAR2017 Competition on Historical Document Writer Identification (Historical-WI),” in ICDAR, vol. 01, 2017, pp. 1377–
2017
-
[32]
A Computational Approach to Edge Detection,
J. Canny, “A Computational Approach to Edge Detection,” PAMI, vol. 8, no. 6, pp. 679–698, 1986. 5
1986
-
[33]
FaceNet: A Unified Embedding for Face Recognition and Clustering,
F. Schroff, D. Kalenichenko, and J. Philbin, “FaceNet: A Unified Embedding for Face Recognition and Clustering,” in CVPR, 2015, pp. 815–823. 5
2015
-
[34]
In Defense of the Triplet Loss for Person Re-Identification,
A. Hermans, L. Beyer, and B. Leibe, “In Defense of the Triplet Loss for Person Re-Identification,” 2017, arXiv:1703.07737. 5
2017 arXiv
-
[35]
On the Convergence of Adam and Beyond,
S. J. Reddi, S. Kale, and S. Kumar, “On the Convergence of Adam and Beyond,” in ICLR, 2018. 5
2018
-
[36]
Adam: A Method for Stochastic Optimization,
D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” in ICLR, 2015. 5
2015
-
[37]
Efficient Sparse Coding Algorithms,
H. Lee, A. Battle, R. Raina, and A. Y . Ng, “Efficient Sparse Coding Algorithms,” in NIPS, P. B. Sch ¨olkopf, J. C. Platt, and T. Hoffman, Eds. MIT Press, 2007, pp. 801–808. 5
2007
-
[38]
Person ReID Triplet-Loss Baseline,
H. Huang, “Person ReID Triplet-Loss Baseline,” 2019. [Online]. Avail- able: https://github.com/huanghoujing/person-reid-triplet-loss-baseline 6
2019
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.