REVIEW 4 major objections 6 minor 27 references
HCNQA: Enhancing 3D VQA with Hierarchical Concentration Narrowing Supervision
T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Supervising three checkpoints of spatial narrowing makes 3D VQA models reason before answering and cuts shortcut behavior roughly in half compared with an answer-centric baseline.
desk verdict HCNQA has a genuinely new three-phase supervision idea, but the evidence for its shortcut-suppression claim is undermined by inconsistent baselines and an unvalidated pseudo-label pipeline. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Hierarchical Supervision Module (HSM), a sequence of three lightweight MLP submodules that predict object-level masks: Blocks of Interest (BoI, a cell in a 5x5 spatial grid that contains target or anchor objects), Objects of Interest (OoI, the target plus anchor objects), and Object of Target (OoT, the object directly answering the question). Each submodule's feature is concatenated with the base scene feature and passed to the next, so the model must commit to earlier masks before computing later ones. The masks are trained with weighted binary cross-entropy to counter class imbalance, and the final OoT mask reweights object tokens before a Modular Co-Attention Network (MCAN) VQA head generates the answer.
What would settle it
Take a random sample of ScanQA questions and have independent human annotators produce BoI, OoI, and OoT masks from each question and scene; if HCNQA's errors concentrate on the questions where the heuristic masks disagree with human masks, or if retraining on human masks does not improve accuracy over the heuristic masks, then the claim that the heuristic pathway itself is the mechanism of improvement fails.
Extended reading notes
Core claim
The paper's central claim is that supervising a general, question-independent three-phase reasoning pathway—coarse grounding (which room block matters), fine grounding (which objects matter), and inference (which object is the target)—is sufficient to keep a 3D VQA model from taking shortcuts, and that the resulting model answers more accurately. The evidence is that HCNQA reaches 27.01 EM@1 on ScanQA's Test w/ obj split, versus 25.94 for the answer-centric 3D-VisTA baseline, and that on a synonym-substitution perturbation test the model's accuracy drop is 0.24 versus 0.47, roughly half the baseline's degradation. The paper also shows in ablations that each supervised phase contributes, and that fully supervised models outperform partially supervised ones.
Load-bearing premise
The load-bearing premise is that the heuristic rules that turn ScanQA's object_ids into Blocks of Interest, Objects of Interest, and Object of Target labels describe, for every question, the same correct reasoning pathway a human would follow; if those labels are wrong for a question, the model is trained to commit to an erroneous intermediate conclusion and the error can propagate to the final answer.
Editorial extensions
If this is right
- 3D VQA models trained with checkpoint supervision on intermediate masks outperform answer-centric-only models on ScanQA, with EM@1 improving from 25.94 to 27.01 and gains across BLEU, ROUGE-L, Meteor, and CIDEr.
- Supervised reasoning checkpoints suppress superficial shortcuts: under synonym perturbation, HCNQA's EM@1 drop is 0.24 versus 0.47 for the baseline, which is about half the degradation.
- The hierarchical supervision adds only about 7.30% FLOPs, so the reasoning-pathway regularization is computationally cheap relative to its accuracy benefit.
- Ablation results show that removing any of the three supervision phases lowers performance, implying that each phase's checkpoint contributes to the final result.
- The method underperforms on questions with short or implicit reasoning chains, where the multi-stage mechanism adds complexity and can propagate early mispredictions, a limitation the paper explicitly discusses.
Reading between the lines
- (Editorial inference) The same three-phase narrowing idea could transfer to other 3D grounding tasks or embodied question answering, where object-level checkpoints can be generated from existing annotations with little extra cost.
- (Editorial inference) Because the paper's masks come from heuristic rules on ScanQA annotations, one testable extension is to make the masks learnable or human-annotated for a subset of questions; if human masks improve accuracy further, the heuristic pathway, not the supervision mechanism, is the current bottleneck.
- (Editorial inference) The shortcut-suppression result suggests intermediate checkpoint supervision could act as a general debiasing regularizer for vision-language models, for example on biased VQA splits where answer distributions correlate with question wording.
- (Editorial inference) The paper's framing against underthinking in slow-thinking LLMs suggests that checkpoint supervision could be applied to longer chain-of-thought generation as a commitment mechanism, though this goes beyond what the paper tests.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes HCNQA, a 3D VQA model that augments answer-centric supervision with three hierarchical object-mask supervision objectives (BoI, OoI, OoT), intended to force the model to perform coarse-to-fine spatial grounding before answering. Masks are generated by unpublished heuristic rules applied to ScanQA's object_ids annotations. The model is built on 3D-VisTA-style encoders plus a sequential Hierarchical Supervision Module (HSM) and a MCAN VQA head. Experiments on ScanQA report EM@1 of 27.01 vs 25.94 for 3D-VisTA on Test w/ obj, and a smaller degradation under synonym perturbation (0.24 vs 0.47). Ablations and annotation-quality tables are also presented.
Significance. If the mechanism is validated, the work would make a useful contribution by showing that checkpoint supervision of intermediate spatial grounding can suppress shortcuts in 3D VQA, with only 7.30% FLOPs overhead. The paper's strengths are the clear problem motivation, the inclusion of a shortcut-perturbation evaluation, and the release of code. However, the central evidence is currently weakened by the unspecified mask-generation rules, an inconsistent baseline number between Tables 3 and 4, and an ablation in which full supervision is not the best configuration. These issues prevent the paper from establishing the claimed causal mechanism.
major comments (4)
- [§3.1] The exact heuristic rules used to convert ScanQA's object_ids annotations into BoI, OoI, and OoT masks are not given (no pseudocode, no thresholds, no examples), and the text states that the annotation 'itself does not have a clear meaning.' Because the HSM is trained to commit to these masks at three sequential checkpoints, any systematic error in the masks will propagate into the final answer; the paper itself concedes in Section 4.7 that early-stage mispredictions can mislead on short reasoning chains. The authors should fully specify the rules, release the generated masks, and validate them against human judgments or an oracle; otherwise the EM@1 gain over 3D-VisTA cannot be attributed to the claimed reasoning pathway rather than to generic auxiliary supervision.
- [§4.4, §4.5 (Tables 3 and 4)] The 3D-VisTA baseline is 25.94 EM@1 in Table 3 but 23.25 in Table 4. If these are different splits or evaluation settings, the shortcut-degradation comparison is not controlled; if they are the same setting, at least one number is wrong. This matters because the shortcut-suppression claim is a central piece of evidence. The authors must reconcile the numbers and report the perturbation experiment on the same split and model checkpoint used in Table 3.
- [§4.3 (Table 2)] The full-supervision row (CG+FG+IF+VQA, EM@1 22.95) is not the best configuration; the three-head combination without the VQA head reaches 22.98, and the difference to CG-only (22.33) is small. This undermines the claim that all three phases and the full pipeline contribute, and no significance tests are reported. Please provide multiple runs with standard deviations or confidence intervals and a paired test, and discuss whether the differences are within noise.
- [§4.2 (Table 1)] The annotation-quality verification shows that training with BoI, OoI, or OoT improves EM@1 relative to no localization supervision, but trainability is not evidence that the masks encode a human-like reasoning pathway. No agreement metric between the heuristic masks and human annotations is reported, and the γ ratio only compares improvements over object_ids. Please add direct validation, e.g., inter-annotator agreement, oracle-mask upper bounds, or per-question inspection of mask quality.
minor comments (6)
- [§4.6, Fig. 4] The section heading and figure caption say 'Quantitative results' but the content is qualitative; rename to 'Qualitative results'.
- [§3.3, Eq. (1)] Equation (1) is missing a normalization convention: the prefactor (c0+c1)/N and the per-term division by c0 and c1 are introduced without explaining the effective class balance; please clarify the derivation.
- [§4.1] The loss weights λ_cg=0.2, λ_fg=0.3, λ_if=0.5 and grid size S=5 are chosen without sensitivity analysis; report at least a small sweep over these hyperparameters.
- [References] References [24] and [25] are the same paper (Ye and Kovashka) and should be merged.
- [Abstract/Code] The code link in the abstract has no commit hash or license; for reproducibility, please pin a version.
- [§4.3, Table 2] Table 2's checkmark rows are ambiguous because three rows have identical '✓ ✓ ✓' symbols; label each configuration explicitly (e.g., 'CG+FG' vs 'CG+IF' vs 'FG+IF').
Circularity Check
No significant circularity: the intermediate BoI/OoI/OoT masks are auxiliary supervisions derived from ScanQA annotations, not re-expressions of the final answer, and the reported gains are evaluated against an independent baseline and a shortcut-perturbation test.
full rationale
The paper's core claim is that supervising intermediate masks (BoI, OoI, OoT) enforces a three-phase spatial reasoning pathway before the answer is produced. These masks are generated by heuristic rules over ScanQA's object_ids and question anchors, as stated in Section 3.1: 'we generate pseudo-labels for ground truths by applying heuristic rules on ScanQA's annotation (which itself does not have a clear meaning).' Although OoT is conceptually identical to ScanQA's object_ids (the target object), BoI and OoI are coarser object sets derived from the target and anchors, so they are not merely the final answer relabeled. The answer supervision comes from the same ScanQA benchmark, but that is standard supervised training, not circularity. The empirical evaluation compares against 3D-VisTA, which already uses object_ids supervision, and includes a synonym-perturbation shortcut test that is independent of the pseudo-label construction. No equation-level reduction, no fitted parameter renamed as a prediction, and no load-bearing self-citation chain is present; the paper cites no prior work by its own authors. The main weakness is that the heuristic pseudo-labels are not fully specified or validated, which is a reproducibility and assumption-risk concern rather than a circular-derivation concern. Therefore the derivation chain is self-contained with respect to circularity, and the appropriate score is 0.
Assumptions & free parameters
free parameters (4)
- loss weights λ_cg, λ_fg, λ_if =
0.2, 0.3, 0.5
- BoI grid size S =
5
- VQA head attention constant =
1
- HSM MLP depth =
4 layers
assumptions (5)
- ad hoc to paper The three-phase concentration narrowing pathway (BoI to OoI to OoT) is a general and rational reasoning pathway for 3D VQA.
- domain assumption Supervising intermediate checkpoints suppresses shortcuts and improves answer accuracy.
- domain assumption ScanQA object_ids annotations reliably identify the target and anchors for every question.
- domain assumption Object segmentation (ground truth or Mask3D predictions) is accurate and consistent across experiments.
- domain assumption Pretraining on Masked Language Modeling, Masked Object Modeling, and Scene-Text Matching following 3D-VisTA transfers to the HSM training.
invented entities (3)
-
Block of Interest (BoI) mask
independent evidence
-
Objects of Interest (OoI) mask
independent evidence
-
Object of Target (OoT) mask
independent evidence
Cite this review
Pith. "Pith review of HCNQA: Enhancing 3D VQA with Hierarchical Concentration Narrowing Supervision." pith.science (2026). https://pith.science/paper/XALJDTPY
@misc{pith2026250701800,
author = {Pith},
title = {Pith review of: HCNQA: Enhancing 3D VQA with Hierarchical Concentration Narrowing Supervision},
year = {2026},
howpublished = {\url{https://pith.science/paper/XALJDTPY}},
note = {Machine review of arXiv:2507.01800}
}
read the original abstract
3D Visual Question-Answering (3D VQA) is pivotal for models to perceive the physical world and perform spatial reasoning. Answer-centric supervision is a commonly used training method for 3D VQA models. Many models that utilize this strategy have achieved promising results in 3D VQA tasks. However, the answer-centric approach only supervises the final output of models and allows models to develop reasoning pathways freely. The absence of supervision on the reasoning pathway enables the potential for developing superficial shortcuts through common patterns in question-answer pairs. Moreover, although slow-thinking methods advance large language models, they suffer from underthinking. To address these issues, we propose \textbf{HCNQA}, a 3D VQA model leveraging a hierarchical concentration narrowing supervision method. By mimicking the human process of gradually focusing from a broad area to specific objects while searching for answers, our method guides the model to perform three phases of concentration narrowing through hierarchical supervision. By supervising key checkpoints on a general reasoning pathway, our method can ensure the development of a rational and effective reasoning pathway. Extensive experimental results demonstrate that our method can effectively ensure that the model develops a rational reasoning pathway and performs better. The code is available at https://github.com/JianuoZhu/HCNQA.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Azuma, D., Miyanishi, T., Kurita, S., Kawanabe, M.: Scanqa: 3d question answer- ing for spatial scene understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 19129–19139 (June 2022)
work page 2022
-
[2]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Cai, D., Zhao, L., Zhang, J., Sheng, L., Xu, D.: 3djcg: A unified framework for joint dense captioning and visual grounding on 3d point clouds. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 16464–16473 (June 2022)
work page 2022
-
[3]
Chen, S., Tapaswi, M., Guhur, P.L., Schmid, C., Laptev, I.: Language conditioned spatial relation reasoning for 3d object grounding. In: NeurIPS (2022)
work page 2022
-
[4]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Chen, S., Chen, X., Zhang, C., Li, M., Yu, G., Fei, H., Zhu, H., Fan, J., Chen, T.: Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 26428–26438 (June 2024)
work page 2024
-
[5]
In: Proceed- ings of the IEEE/CVF International Conference on Computer Vision (ICCV)
Dancette, C., Cadène, R., Teney, D., Cord, M.: Beyond question-based biases: Assessing multimodal shortcut learning in visual question answering. In: Proceed- ings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 1574–1583 (October 2021)
work page 2021
-
[6]
DeepSeek-AI: Deepseek-r1: Incentivizing reasoning capability in llms via reinforce- ment learning (2025),https://arxiv.org/abs/2501.12948
arXiv 2025
-
[7]
ArXivabs/2306.02329 (2023), https://api
Delitzas, A., Parelli, M., Hars, N., Vlassis, G., Anagnostidis, S., Bachmann, G., Hofmann, T.: Multi-clip: Contrastive vision-language pre-training for ques- tion answering tasks in 3d scenes. ArXivabs/2306.02329 (2023), https://api. semanticscholar.org/CorpusID:259076122
arXiv 2023
-
[8]
In: North American Chapter of the Association for Computational Linguistics (2019)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidi- rectional transformers for language understanding. In: North American Chapter of the Association for Computational Linguistics (2019)
work page 2019
Show all 27 references
-
[9]
NeurIPS (2023)
Hong,Y.,Zhen,H.,Chen,P.,Zheng,S.,Du,Y.,Chen,Z.,Gan,C.:3d-llm:Injecting the 3d world into large language models. NeurIPS (2023)
2023
-
[10]
Proceedings of Machine Learning Research, vol
Huang, J., Yong, S., Ma, X., Linghu, X., Li, P., Wang, Y., Li, Q., Zhu, S.C., Jia, B., Huang,S.:Anembodiedgeneralistagentin3dworld.In:Salakhutdinov,R.,Kolter, Z.,Heller,K.,Weller,A.,Oliver,N.,Scarlett,J.,Berkenkamp,F.(eds.)Proceedings of the 41st International Conference on Ma...
2024
-
[11]
Huang,S.,Chen,Y.,Jia,J.,Wang,L.:Multi-viewtransformerfor3dvisualground- ing.In:ProceedingsoftheIEEE/CVFConferenceonComputerVisionandPattern Recognition (CVPR). pp. 15524–15533 (June 2022)
2022
-
[12]
In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR)
Jin, Z., Hayat, M., Yang, Y., Guo, Y., Lei, Y.: Context-aware alignment and mu- tual masking for 3d-language pre-training. In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR). pp. 10984–10994 (June 2023) 12 S. Zhou et al
2023
-
[13]
Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H.W., Tay, Y., Zhou, D., Le, Q.V., Zoph, B., Wei, J., Roberts, A.: The flan collection: Designing data and meth- ods for effective instruction tuning (2023),https://arxiv.org/abs/2301.13688
2023 arXiv
-
[14]
In: International Conference on Learning Representations (2023), https://openreview.net/forum?id=IDJx97BC38
Ma, X., Yong, S., Zheng, Z., Li, Q., Liang, Y., Zhu, S.C., Huang, S.: Sqa3d: Sit- uated question answering in 3d scenes. In: International Conference on Learning Representations (2023), https://openreview.net/forum?id=IDJx97BC38
2023
-
[15]
Proceedings of the AAAI Conference on Artificial In- telligence 38(5), 4261–4268 (Mar 2024).https://doi.org/10.1609/aaai.v38i5
Mo, W., Liu, Y.: Bridging the gap between 2d and 3d visual question answering: A fusion approach for 3d vqa. Proceedings of the AAAI Conference on Artificial In- telligence 38(5), 4261–4268 (Mar 2024).https://doi.org/10.1609/aaai.v38i5. 28222, https://ojs.aaai.org/index.php/AA...
2024 doi
-
[16]
https://openai.com/index/ learning-to-reason-with-llms/ (2024), accessed: 2024-07-26
OpenAI: Learning to reason with llms. https://openai.com/index/ learning-to-reason-with-llms/ (2024), accessed: 2024-07-26
2024
-
[17]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops
Parelli, M., Delitzas, A., Hars, N., Vlassis, G., Anagnostidis, S., Bachmann, G., Hofmann, T.: Clip-guided vision-language pre-training for question answering in 3d scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. ...
2023
-
[18]
In: Proceedings of the 31st International Con- ference on Neural Information Processing Systems
Qi, C.R., Yi, L., Su, H., Guibas, L.J.: Pointnet++: deep hierarchical feature learn- ing on point sets in a metric space. In: Proceedings of the 31st International Con- ference on Neural Information Processing Systems. p. 5105–5114. NIPS’17, Curran Associates Inc., Red Hook, N...
2017
-
[19]
In: Meila, M., Zhang, T
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. In: Meila, M., Zhang, T. (eds.) Proceedings of the 38th Int...
2021
-
[20]
In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2016)
Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2016)
2016
-
[21]
International Confer- ence on Robotics and Automation (ICRA) (2023)
Schult, J., Engelmann, F., Hermans, A., Litany, O., Tang, S., Leibe, B.: Mask3D: Mask Transformer for 3D Semantic Instance Segmentation. International Confer- ence on Robotics and Automation (ICRA) (2023)
2023
-
[22]
Sourulahti, S., Janssen, C.P., Jokinen, J.P.: Modeling rational adaptation of visual search to hierarchical structures (2024),https://arxiv.org/abs/2409.08967
2024 arXiv
-
[23]
Wang, Y., Liu, Q., Xu, J., Liang, T., Chen, X., He, Z., Song, L., Yu, D., Li, J., Zhang, Z., Wang, R., Tu, Z., Mi, H., Yu, D.: Thoughts are all over the place: On the underthinking of o1-like llms (2025),https://arxiv.org/abs/2501.18585
2025 arXiv
-
[24]
Ye, K., Kovashka, A.: A case study of the shortcut effects in visual commonsense reasoning. Proceedings of the AAAI Conference on Artificial Intelligence35(4), 3181–3189 (May 2021).https://doi.org/10.1609/aaai.v35i4.16428, https:// ojs.aaai.org/index.php/AAAI/article/view/16428
2021 doi
-
[25]
In: Proceedings of the Thirty-Fifth AAAI Conference on Artificial In- telligence (AAAI) (February 2021)
Ye, K., Kovashka, A.: A case study of the shortcut effects in visual commonsense reasoning. In: Proceedings of the Thirty-Fifth AAAI Conference on Artificial In- telligence (AAAI) (February 2021)
2021
-
[26]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2019)
Yu, Z., Yu, J., Cui, Y., Tao, D., Tian, Q.: Deep modular co-attention networks for visual question answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2019)
2019
-
[27]
In: Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision (ICCV)
Zhu, Z., Ma, X., Chen, Y., Deng, Z., Huang, S., Li, Q.: 3d-vista: Pre-trained trans- former for 3d vision and text alignment. In: Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision (ICCV). pp. 2911–2921 (October 2023)
2023
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.