REVIEW 3 major objections 4 minor 42 references
HiCoRe: Visual Hierarchical Context-Reasoning
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A graph over an image's bounding boxes can predict the scene's global class without any object labels.
desk verdict The core idea is novel, but the MIT67 superclass results are likely a transductive label-leakage artifact, and the baselines are too weak to support the advertised gains. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The context-reasoning core: for each image, a complete graph is built over its bounding-box nodes, with an adjacency matrix encoding their pairwise relations; a graph convolutional network then propagates node features through the layer rule $H^{(l+1)} = \sigma(\hat{A} H^{(l)} S^{(l)} W^{(l)})$, where $S^{(l)}$ carries complementary (e.g., spatial) features and $\hat{A}$ is the renormalized adjacency matrix with self-loops. At the superclass level, images themselves become nodes in a supergraph that nests the per-image subgraphs of boxes. This construction is what lets the model combine intrinsic object features with relational structure while staying independent of object labels.
What would settle it
Fine-tune ResNet50 end-to-end on the same datasets and compare: if a properly trained CNN reaches HiCoRe's reported Unrel accuracy of 63.86% (vs 35.25% reported for the baseline), the graph's contribution is not the source of the gain. A second test: replace the GCN with a graph-free MLP over the same node features; if it matches HiCoRe, the complete-graph edges add nothing.
Extended reading notes
Core claim
The paper's central claim is that semantic context—the global class of an image—emerges from the interactions between the objects in it, and that these interactions can be captured by a graph-convolutional network operating on a complete graph of bounding boxes. The framework, HiCoRe, treats each bounding box as a node carrying a CNN-extracted feature vector, connects every box to every other box, and propagates information through a GCN layer; at coarser granularities the same machinery is applied to images within a superclass. Because the graph is complete, the method needs no pre-existing knowledge graph and no object class labels—only the boxes and the image-level class. The authors report consistent gains over standard CNN baselines across three datasets and three granularities, and show that a semi-supervised variant using one-fifth of the labels matches the supervised one.
Load-bearing premise
The reported margins over CNNs assume the comparison models were trained and tuned at least as well as the graph pipeline, so the gains come from the graph reasoning and not from a weaker baseline; the method also assumes bounding-box proposals are given at test time.
Editorial extensions
If this is right
- Image classification can be reformulated as reasoning over object proposals, making object-label annotation unnecessary for scene-level tasks.
- Hierarchical context is exploitable: the same graph machinery works at superclass, subclass, and combined hierarchical granularities, with superclass accuracy near 99 percent on MIT67.
- Semi-supervised training with one-fifth of the labeled data matches the fully supervised pipeline on the VRD dataset, suggesting strong label-efficiency for context reasoning.
- The approach is dataset-agnostic given boxes and global labels, and copes with ambiguous scenes where the same objects appear under different contexts.
- Dimensionality reduction (PCA) and spatial-feature fusion give small but measurable improvements, indicating the core signal is relational rather than purely feature-based.
Reading between the lines
- Because the paper does not report a no-graph baseline that uses the same node features, the specific contribution of the edges could be tested by comparing HiCoRe against a multilayer perceptron on the same pooled box features; that comparison would isolate what the graph structure adds.
- The pipeline only requires boxes and image labels, so a natural test is feeding it region proposals from an off-the-shelf detector rather than ground-truth boxes, to see whether the gains survive noisy object locations.
- The near-perfect superclass result suggests the relational signal is strongest when class counts are small; stressing the method with more superclasses (or finer subclass splits) would reveal where the graph advantage saturates.
- The semi-supervised equality hints that unlabeled same-context images supply most of the relational information; exploring labeled fractions below 20 percent would map how quickly the advantage degrades.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes HiCoRe, a graph-convolutional framework for image-level global-context classification. It extracts CNN features from bounding-box proposals and/or whole images, builds complete graphs at different granularities, and applies GCN/GAT layers to propagate information; object labels are not required. Experiments on Unrel, MIT67, and VRD compare HiCoRe with 'traditional' CNN baselines. The central claim is that HiCoRe substantially outperforms standard CNNs, e.g., 63.86% vs 35.25% on Unrel and large gains on MIT67.
Significance. If the reported numbers are valid, the paper offers a novel way to use context and object proposals without object-label supervision, with potential relevance to scene and relation understanding. The contribution is empirical; the GCN formulation is standard, and I agree with the stress-test note that there is no circularity-by-construction issue. The manuscript is also transparent about filtering 14 MIT67 classes and about using center-bias heuristics on VRD. However, the evaluation currently does not establish the headline claims, because the graph-construction protocol for superclass/hierarchical experiments is under-specified and the baselines appear substantially weaker than standard end-to-end CNNs.
major comments (3)
- [Algorithm 1, lines 11–15 and 17–24; Section 4.1] The paper never states that the 80/20 train/test split is enforced when building the superclass and hierarchical graphs. As written, Algorithm 1 constructs Vj from 'images from a same context j' over the whole dataset, so a test image can be connected by a complete graph to all training images of its ground-truth superclass. A two-layer GCN (Eqs. 1–3) then averages features across exactly that class, which would make the 99.00% superclass accuracy in Table 5 an artifact of label leakage rather than visual reasoning. The hierarchical result (58.96%) inherits the same construction through the supergraph. Please specify the graph-construction protocol at test time: how a test image is assigned to a context group without using its label, or state that graphs are built only from training images and describe the inference procedure.
- [Section 4.1 and Tables 1, 5] The baselines are described only as grid-searched classifiers on ImageNet-pretrained features; Section 4.1 does not say that the CNN backbones are fine-tuned end-to-end. If the 'traditional' numbers come from a shallow classifier on frozen features, they are not representative of standard trained CNNs, and the reported gains (81.2% on Unrel, 4.79x on MIT67 subclass) may reflect the weakness of the baseline rather than the graph architecture. Please report the exact baseline pipeline (architecture, whether the backbone is frozen, classifier specification) and, ideally, include fine-tuned end-to-end CNNs.
- [Section 4.3, Table 5] Table 5 reports only HiCoRe accuracies; the claimed 'gains of 4.79 and 1.2 times greater than ResNet50' for subclass and hierarchical granularities cannot be checked because the corresponding baseline values are omitted. The text gives 52.00% for the superclass only. Please provide the baseline numbers for all granularities, along with the number of random splits and variance/error bars.
minor comments (4)
- [Section 4.3] Filtering 14 of 67 MIT67 classes changes the benchmark; the paper should state that results are not directly comparable to standard MIT67 accuracy and ideally report numbers on the full dataset.
- [Section 4.2, Table 2] The text says that fusing visual and spatial features led to a 'slightly improvement', but for VGG19 visual+spatial (61.86%) is below visual-only (62.31%); please qualify this claim.
- [Section 4.1] The hyperparameter description is incomplete: 'epochs ( 2000)' lacks a range, and the text should clarify whether the same training procedure and epoch count are used for the baselines and for HiCoRe.
- [Section 4.4, Table 7] The semi-supervised ResNet50 row is marked with '–', so the claimed 55% gain compares semi-supervised HiCoRe to the supervised ResNet50; this comparison should be stated explicitly.
Circularity Check
MIT67 superclass/hierarchical graphs are built from the very superclass labels the method reports predicting, so the 99% accuracy is forced by construction.
-
self definitional
[Algorithm 1 (superclass branch, lines 11-15; hierarchical branch, lines 17-24); Section 4.1 train/test split; Table 5]
"X← visual features of the images using CNNs; for each image i from dataset do Vj is composed of images from a same context j; Build a complete graph Gj = (Vj,E j); end ... For all image datasets we randomly split them, in a stratified way, generating their respective training (80% of images) and test sets (20%)."
In superclass mode the graph's node partition is defined by 'a same context j', where context j is the superclass label that HiCoRe is claimed to predict (Table 5 reports 99.00% visual-superclass accuracy). Algorithm 1 iterates over 'each image i from dataset' and never states that test images are excluded from the grouping or how a test image would be assigned to a context without knowing its label. With a complete graph per context, the GCN propagation (Eq. 1) averages features over all images sharing the target label, so a test node's representation is dominated by training images already known to be in its class.
full rationale
Most of the paper is a standard empirical comparison and does not contain a derivation chain: the GCN update (Eq. 1-2) and GAT attention are standard tools cited to Kipf & Welling and Velickovic et al.; the visual features are ImageNet-pretrained CNN outputs; no parameter is fitted to a subset and then renamed as a prediction. The Unrel and VRD experiments build complete graphs among bounding boxes within each image, so their predictions are not defined in terms of the image-class label. However, the MIT67 superclass branch in Algorithm 1 defines graph nodes as 'images from a same context j', and context j is exactly the superclass label reported in Table 5. The paper specifies a random 80/20 train/test split but does not describe removing test images from this label-defined graph or a test-time procedure that inserts an unknown image without using its context label. Under the transductive GCN update, same-context neighbor averaging makes the 99.00% superclass accuracy substantially an artifact of the graph being built from the target labels; the hierarchical result (58.96%) uses the same supergraph construction. This is a partial, construction-level circularity affecting the hierarchical claims, while the rest of the paper's comparisons are not circular in the same way.
Assumptions & free parameters
free parameters (4)
- GMM components for spatial feature discretization =
400
- PCA components =
200
- GCN depth =
2 layers
- Classifier and GCN hyperparameters =
grid-searched per architecture
assumptions (4)
- domain assumption Bounding-box proposals are available for all images at both training and test time.
- domain assumption Pre-trained ImageNet features transfer to the target datasets without fine-tuning.
- ad hoc to paper A complete graph among all nodes is an appropriate model of context.
- standard math The GCN propagation rule with normalized adjacency is a valid aggregation for this task.
Cite this review
Pith. "Pith review of HiCoRe: Visual Hierarchical Context-Reasoning." pith.science (2026). https://pith.science/paper/U4KKP6RN
@misc{pith2026190900848,
author = {Pith},
title = {Pith review of: HiCoRe: Visual Hierarchical Context-Reasoning},
year = {2026},
howpublished = {\url{https://pith.science/paper/U4KKP6RN}},
note = {Machine review of arXiv:1909.00848}
}
read the original abstract
Reasoning about images/objects and their hierarchical interactions is a key concept for the next generation of computer vision approaches. Here we present a new framework to deal with it through a visual hierarchical context-based reasoning. Current reasoning methods use the fine-grained labels from images' objects and their interactions to predict labels to new objects. Our framework modifies this current information flow. It goes beyond and is independent of the fine-grained labels from the objects to define the image context. It takes into account the hierarchical interactions between different abstraction levels (i.e. taxonomy) of information in the images and their bounding-boxes. Besides these connections, it considers their intrinsic characteristics. To do so, we build and apply graphs to graph convolution networks with convolutional neural networks. We show a strong effectiveness over widely used convolutional neural networks, reaching a gain 3 times greater on well-known image datasets. We evaluate the capability and the behavior of our framework under different scenarios, considering distinct (superclass, subclass and hierarchical) granularity levels. We also explore attention mechanisms through graph attention networks and pre-processing methods considering dimensionality expansion and/or reduction of the features' representations. Further analyses are performed comparing supervised and semi-supervised approaches.
Figures
Reference graph
Works this paper leans on
-
[1]
P. W. Battaglia, J. B. Hamrick, V . Bapst, A. Sanchez- Gonzalez, V . F. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, C. Gulcehre, F. Song, A. J. Ballard, J. Gilmer, G. E. Dahl, A. Vaswani, K. Allen, C. Nash, V . Langston, C. Dyer, N. Heess, D. Wierstra, P. Kohli, M. Botvinick, O. Vinyals, Y . Li, and R. Pascanu. Relational in...
arXiv 2018
- [2]
-
[3]
L. Bottou. Stochastic gradient descent tricks. In Neural Net- works: Tricks of the Trade - Second Edition, pages 421–436. Springer-Verlag, 2012. 5
work page 2012
-
[4]
X. Bu, Y . Wu, Z. Gao, and Y . Jia. Deep convolutional net- work with locality and sparsity constraints for texture classi- fication. Pattern Recognition, 91:34–46, 2019. 1
work page 2019
-
[5]
H. Chen, B. Perozzi, Y . Hu, and S. Skiena. Harp: Hierar- chical representation learning for networks. In AAAI Con- ference on Artificial Intelligence , pages 2127–2134. AAAI Press, 2018. 3
work page 2018
-
[6]
W. Chen, D. Xie, Y . Zhang, and S. Pu. All you need is a few shifts: Designing efficient convolutional neural networks for image classification. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1–10. IEEE Computer Society, 2019. 1
-
[7]
X. Chen and A. Gupta. Spatial memory for context reason- ing in object detection. In IEEE International Conference on Computer Vision, pages 4106–4116. IEEE Computer Soci- ety, 2017. 3
work page 2017
-
[8]
X. Chen, L. Li, L. Fei-Fei, and A. Gupta. Iterative vi- sual reasoning beyond convolutions. In IEEE Conference on Computer Vision and Pattern Recognition , pages 7239–
Show all 42 references
-
[9]
X. Chen, A. Shrivastava, and A. Gupta. Neil: Extracting vi- sual knowledge from web data. In IEEE International Con- ference on Computer Vision, pages 1409–1416. IEEE Com- puter Society, 2013. 3
2013
-
[10]
Defferrard, X
M. Defferrard, X. Bresson, and P. Vandergheynst. Convolu- tional neural networks on graphs with fast localized spectral filtering. In International Conference on Neural Information Processing Systems, pages 3844–3852, USA, 2016. Curran Associates Inc. 1
2016
-
[11]
K. He, G. Gkioxari, P. Doll ´ar, and R. B. Girshick. Mask R-CNN. In IEEE International Conference on Computer Vi- sion, pages 2980–2988. IEEE Computer Society, 2017. 1
2017
-
[12]
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learn- ing for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778. IEEE Com- puter Society, 2016. 1, 6
2016
-
[13]
T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In International Con- ference on Learning Representations, pages 1–14, 2017. 3, 6
2017
-
[14]
J. B. Lee, R. A. Rossi, S. Kim, N. K. Ahmed, and E. Koh. At- tention models in graphs: A survey. CoRR, abs/1807.07984,
-
[15]
R. Li, M. Li, and J. Li. Connection sensitive attention U-NET for accurate retinal vessel segmentation. CoRR, abs/1903.05558:1–10, 2019. 1
1903 arXiv
-
[16]
Y . Li, W. Ouyang, Z. Bolei, S. Jianping, Z. Chao, and X. Wang. Factorizable net: An efficient subgraph-based framework for scene graph generation. In European Con- ference on Computer Vision, pages 346–363, 2018. 2
2018
-
[17]
Y . Li, W. Ouyang, B. Zhou, K. Wang, and X. Wang. Vip- cnn: Visual phrase guided convolutional neural network. In IEEE Conference on Computer Vision and Pattern Recogni- tion, pages 7244–7253, 2017. 2
2017
-
[18]
D. Lin, J. Dai, J. Jia, K. He, and J. Sun. Scribble- Sup: Scribble-supervised convolutional networks for seman- tic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3159–3167. IEEE Computer Society, 2016. 1
2016
-
[19]
W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. E. Reed, C. Fu, and A. C. Berg. SSD: single shot multibox detector. In European Conference on Computer Vision, pages 21–37. Elsevier, 2016. 1
2016
-
[20]
C. Lu, R. Krishna, M. S. Bernstein, and F. Li. Visual relation- ship detection with language priors. InEuropean Conference on Computer Vision, pages 852–869, 2016. 8
2016
-
[21]
Marino, R
K. Marino, R. Salakhutdinov, and A. Gupta. The more you know: Using knowledge graphs for image classification. In IEEE Conference on Computer Vision and Pattern Recogni- tion, pages 20–28. IEEE Computer Society, 2017. 3
2017
-
[22]
Peyre, I
J. Peyre, I. Laptev, C. Schmid, and J. Sivic. Weakly- supervised learning of visual relations. In IEEE Interna- tional Conference on Computer Vision , pages 5189–5198. IEEE Computer Society, 2017. 5, 6
2017
-
[23]
Quattoni and A
A. Quattoni and A. Torralba. Recognizing indoor scenes. In IEEE Conference on Computer Vision and Pattern Recogni- tion, pages 413–420. IEEE Computer Society, 2009. 2, 6, 7
2009
-
[24]
A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carls- son. CNN features off-the-shelf: An astounding baseline for recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 512–519. IEEE Computer Soci- ety, 2014. 6
2014
-
[25]
S. Ren, K. He, R. B. Girshick, and J. Sun. Faster R-CNN: towards real-time object detection with region proposal net- works. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6):1137–1149, 2017. 1
2017
-
[26]
Ronneberger, P
O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolu- tional networks for biomedical image segmentation. In In- ternational Conference on Medical Image Computing and Computer-Assisted Intervention, pages 234–241, 2015. 1
2015
-
[27]
Russakovsky, J
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. Imagenet large scale visual recog- nition challenge. International Journal of Computer Vision, 115(3):211–252, 2015. 6
2015
-
[28]
Sandler, A
M. Sandler, A. G. Howard, M. Zhu, A. Zhmoginov, and L. Chen. Mobilenetv2: Inverted residuals and linear bot- tlenecks. In IEEE Conference on Computer Vision and Pat- tern Recognition, pages 4510–4520. IEEE Computer Soci- ety, 2018. 1
2018
-
[29]
Simonyan and A
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, pages 1–14, 2015. 6
2015
-
[30]
Vaswani, N
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Annual Conference on Neural Information Pro- cessing Systems, pages 6000–6010, 2017. 7
2017
-
[31]
Veli ˇckovi´c, G
P. Veli ˇckovi´c, G. Cucurull, A. Casanova, A. Romero, P. Li`o, and Y . Bengio. Graph Attention Networks. In International Conference on Learning Representations, pages 1–12, 2018. 5, 7
2018
-
[32]
Z. Wang, T. Chen, J. Ren, W. Yu, H. Cheng, and L. Lin. Deep reasoning with knowledge graph for social relationship un- derstanding. In International Joint Conferences on Artificial Intelligence, pages 1021–1028, 2018. 2
2018
-
[33]
D. Xu, Y . Zhu, C. B. Choy, and L. Fei-Fei. Scene graph generation by iterative message passing. InIEEE Conference on Computer Vision and Pattern Recognition , pages 3097– 3106, 2017. 2, 6, 8
2017
-
[34]
J. Yang, J. Lu, S. Lee, D. Batra, and D. Parikh. Graph r- cnn for scene graph generation. In European Conference on Computer Vision, pages 690–706, 2018. 2
2018
-
[35]
W. Yang, X. Wang, A. Farhadi, A. Gupta, and R. Mottaghi. Visual semantic navigation using scene priors. In Interna- tional Conference on Learning Representations, pages 1–13,
-
[36]
W. Yao, Z. Zeng, C. Lian, and H. Tang. Pixel-wise regression using u-net and its application on pansharpening.Neurocom- puting, 312:364–371, 2018. 1
2018
-
[37]
X. Zeng, W. Ouyang, J. Yan, H. Li, T. Xiao, K. Wang, Y . Liu, Y . Zhou, B. Yang, Z. Wang, H. Zhou, and X. Wang. Crafting gbd-net for object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(9):2109–2123, 2018. 1
2018
-
[38]
Zhang, P
Z. Zhang, P. Cui, and W. Zhu. Deep learning on graphs: A survey. CoRR, abs/1812.04202, 2018. 1
2018 arXiv
-
[39]
Zheng, Y
Q. Zheng, Y . Wang, and P. A. Heng. Online robust im- age alignment via subspace learning from gradient orienta- tions. In IEEE International Conference on Computer Vi- sion, pages 1771–1780. IEEE Computer Society, 2017. 5
2017
-
[40]
J. Zhou, G. Cui, Z. Zhang, C. Yang, Z. Liu, and M. Sun. Graph neural networks: A review of methods and applica- tions. CoRR, abs/1812.08434, 2018. 1
2018 arXiv
-
[41]
Y . Zhou, A. Rangarajan, and P. D. Gader. A gaussian mix- ture model representation of endmember variability in hyper- spectral unmixing. IEEE Transactions on Image Processing, 27(5):2242–2256, 2018. 5
2018
-
[7248]
IEEE Computer Society, 2018. 3
2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.