REVIEW 4 major objections 5 minor 33 references
Learn 3D VQA Better with Active Selection and Reannotation
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Coupling variance-based active selection with oracle reannotation trains 3D VQA models at roughly half the cost.
desk verdict Active selection part is solid; the reannotation half is undercut by the paper's own ScanQA ablation, but the work deserves serious refereeing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the weighted semantic variance of the model's answer distribution, $\operatorname{var}_p(\phi) = \sum_{c \in C} p_c \|\phi_c - \bar{\phi}\|_2^2$, where $\phi_c$ is a pretrained BERT embedding of answer term $c$ and $\bar{\phi}$ is the probability-weighted mean embedding. The paper shows that its uncertainty metric, a KL divergence between the prediction and a Gaussian in semantic space minus the KL divergence to the uniform distribution, reduces up to an additive constant to exactly this variance, so ranking instances by the log determinant of the weighted covariance matrix $\log|\mathrm{cov}_p(\phi)|$ implements the same criterion while avoiding the curse of dimensionality. This same quantity, combined with the term classification loss, defines the three-way partition of labeled instances (confident-correct, uncertain, and confident-wrong); only the confident-wrong group, where the model is certain of an answer different from the ground truth, is sent for reannotation.
What would settle it
Have independent annotators re-examine all instances that the filtration rule flags (low weighted variance, high loss) on ScanQA; if the majority turn out to be correctly labeled hard examples rather than mislabeled ones, the active-reannotation premise fails. A cheaper check is already visible in the paper's own ablation: on ScanQA the diligent oracle's exact-match accuracy (19.42) is lower than the lazy oracle's (19.83), so any setting where that gap widens would falsify the claim that reannotation helps.
Extended reading notes
Core claim
The central discovery the paper argues for is that coupling active selection with active reannotation turns a noisy, scarce 3D VQA training set into a stronger curriculum. Active selection ranks unlabeled questions by the log determinant of the weighted covariance of predicted answer embeddings, $\log|\mathrm{cov}_p(\phi)|$, which the paper derives from an uncertainty metric $\Delta = D_{\mathrm{KL}}(P \,\|\, \mathcal{N}(\bar{\phi}, k^{-1} I)) - D_{\mathrm{KL}}(P \,\|\, U)$ that reduces exactly to the weighted variance $\operatorname{var}_p(\phi) = \sum_{c \in C} p_c \|\phi_c - \bar{\phi}\|^2$; this respects semantic similarity between answer terms instead of treating all pairs as equally distinct. Active reannotation then flags labeled instances whose predictions are confident (low weighted variance) yet wrong (high loss), and requests the oracle to check and correct them; a hierarchical oracle first maps non-canonical answers to canonical terms automatically and only resorts to manual reannotation for the leftovers. Experiments on ScanQA and SQA3D with the ScanQA and 3D-VisTA models show that the weighted-variance selection beats entropy and information-gain baselines on all reported metrics, and that the full multi-turn loop reaches fixed accuracy thresholds in roughly half the training cost of random selection.
Load-bearing premise
The reannotation filter assumes that a model that is confident but wrong about an answer is looking at a mislabeled example rather than a hard-but-correct one, so if the model's confidence is systematically biased, correct labels will be sent for unnecessary rewriting.
Editorial extensions
If this is right
- Training with the multi-turn loop reaches fixed exact-match thresholds (e.g., 20–23% on ScanQA) in roughly half the steps of random selection, and the advantage persists on SQA3D.
- Weighted-variance selection outperforms entropy and information-gain selection on every reported metric, so semantic similarity should be part of uncertainty estimation for free-form answer generation.
- Reannotation for low-variance/high-loss instances improves most quality metrics on both models, with the largest gains on 3D-VisTA (BLEU-1 +6.56% relative, CIDEr +4.44%).
- The hierarchical reannotation strategy, which resolves most non-canonical answers by term mapping before manual review, lowers the cost of fine-grained data cleaning.
- Because the framework is agnostic to the semantic embedding, it is portable to other answer encoders without changing the selection rule.
Reading between the lines
- Editorial extension: the same confident-wrong filter could be reused as a dataset-cleaning audit for other free-form generation tasks where mislabeled or non-canonical answers are common, such as 2D VQA or image captioning.
- Editorial extension: the method's advantage should depend on how noisy the labels are; on cleaner datasets the reannotation step may hurt, and the ScanQA exact-match drop with the diligent oracle already hints that the filter needs a noise-detection calibration.
- Editorial extension: combining weighted variance with a diversity term could prevent the selector from repeatedly choosing near-duplicate questions in large unlabeled pools.
- Editorial extension: a robust variant could demand disagreement across model checkpoints before flagging an instance for reannotation, reducing false positives from systematic scene-level biases.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes AQuA, a multi-turn interactive active learning strategy for 3D visual question answering. The method combines active selection based on the log-determinant of a weighted covariance matrix of BERT semantic embeddings with an active reannotation filter that sends low-variance/high-loss labeled instances to an oracle for correction. The oracle uses a hierarchical strategy that first maps non-canonical answers to a refined corpus and resorts to manual reannotation only for remaining cases. Experiments on ScanQA and SQA3D with ScanQA and 3D-VisTA models report improved exact-match/BLEU/CIDEr/ROUGE scores and reduced training cost to reach fixed accuracy thresholds, including a claimed halving of training cost.
Significance. If the claims hold, the paper offers a practical way to train 3D VQA models with fewer labels and to clean noisy free-form answer annotations with modest human effort. The paper has several strengths: it ships code, evaluates on two datasets and two base models, provides qualitative examples of real annotation errors, and the variance-based selection metric is a reasonable alternative to entropy-based uncertainty for free-form answers. The active selection results are plausible and the cost-reduction numbers are interesting. However, the active reannotation component is not consistently supported by the primary benchmark, and the theoretical derivation of the uncertainty metric contains a mathematical gap, so the current evidence does not fully support the paper's central claims.
major comments (4)
- [Section 3.2, Eqs. (2)-(4), Appendix A] The quantity D_KL(P || N(\bar{\phi}, k^{-1} I)) is not a well-defined KL divergence as written: P is a discrete distribution over corpus terms while N is a continuous Gaussian density, so the log-ratio p_c / N(\phi_c | ...) mixes probability masses with density values and the derivation silently cancels the \sum_c p_c \log p_c terms. The weighted-variance score can be motivated directly as the expected squared distance of predicted terms from their semantic mean, but the stated KL derivation is mathematically invalid. Please rewrite the derivation using a discrete reference distribution over the corpus, or explicitly present the weighted-variance score as an independent objective rather than as a KL-based quantity.
- [Section 4.2.2, Table 3] On ScanQA, the Diligent Oracle (active selection plus active reannotation) achieves 19.42 exact match, which is lower than the Lazy Oracle (active selection only) at 19.83; on the exact-match metric used for the cost-reduction headline, active reannotation therefore reduces performance by 0.41 points. The statement that "results under most metrics show improvements" does not address the primary metric, and Table 3 reports no error bars, so the reannotation-specific contribution is not supported on the main benchmark. This needs either corrected results with multiple seeds or a clearly scoped claim that reannotation helps only on non-EM metrics and on specific model/dataset combinations.
- [Section 4.2.2, Tables 1-3] All experimental tables appear to report single runs without standard deviations, confidence intervals, or significance tests. Since many reported gains are below 0.5 EM points (e.g., 19.42 vs 19.83 in Table 3) and the training-cost reductions are computed from cumulative best scores, it is impossible to assess whether the reported differences are reproducible. Please provide multiple seeds or a statistical comparison, or at a minimum report variance across runs.
- [Section 3.3, Figure 3, Appendix C] The filtration rule assumes that low weighted variance plus high loss indicates a mislabeled instance, but Appendix C Table 6 shows that the filter flags many correct or acceptable answers (CA/MA) at several epochs, and Figure 3 reports a nontrivial false-filtration ratio. The manuscript should quantify the precision of the filter on a held-out set of labeled examples and discuss how the false-filtration ratio affects the training pipeline; as written, the empirical support for the load-bearing "confident-and-wrong means mislabeled" assumption is weak.
minor comments (5)
- [Section 4.1.1] The thresholds Z_cov=-1 and Z_loss=3 are tuned on the validation set; please report sensitivity to these thresholds or show that the conclusions are stable across reasonable choices.
- [Section 4.2.1, Table 1] The text says the variance-based strategy outperforms other strategies "across all metrics," but Table 1 reports only cost-reduction percentages and AUC; please report the underlying metric curves or clarify the claim.
- [Algorithm 1] The predicates require_selection(epoch_idx) and require_reannotation(epoch_idx) are not defined in a single place; please specify the epoch schedule for each dataset/model in a table or in the main text.
- [Appendix A, Eq. (16)] The proof labels the quantity in Eq. (16) as proportional to weighted variance, but the additive constant includes -m/2 log(k/2pi) - log|C|; this is fine when m, k, and |C| are fixed, but the text should state that these are fixed per problem instance.
- [Figure 3] The y-axis label "ratio of instances filtered out due to false filtration and correct filtration" is ambiguous; please specify the denominator explicitly, e.g., the number of filtered instances.
Circularity Check
No significant circularity found: the uncertainty metric is derived from an explicit KL objective, the comparisons use external benchmarks, and reannotation thresholds are validation-tuned hyperparameters rather than fitted restatements of the reported metrics.
full rationale
The derivation chain is self-contained. Section 3.2 defines Δ = D_KL(P||N(φbar,k^-1 I)) − D_KL(P||U), and Appendix A proves Δ ∝ Σ_c p_c ||φ_c − φbar||^2 = var_p(φ); this is an algebraic identity with no reported accuracy appearing on either side. The semantic vectors come from a fixed pretrained BERT model, not from the downstream ScanQA/SQA3D metrics, and active selection is compared against random, entropy, and information-gain baselines on external benchmarks, so the selection claim is not circular. Active reannotation (Section 3.3) uses a three-case heuristic (low variance and high loss indicates a possible mislabel), and the filtration thresholds Z_cov = −1 and Z_L = 3 in Section 4.1.1 are tuned on validation-set scatter plots. That is conventional hyperparameter selection, not a fitted parameter renamed as a prediction: the reported EM/BLEU/CIDEr numbers are not used to construct the thresholds, and the ablation includes a negative result (ScanQA diligent-oracle EM 19.42 below lazy-oracle EM 19.83). That negative result shows the method can fail, which is inconsistent with the claim being forced by construction. Figure 3 and Appendix C explicitly acknowledge false filtration as training proceeds, again indicating the outcome is not predetermined. The only coauthor citation ([22]) appears in a survey list of discriminative 3D VQA approaches and is not load-bearing. There is no uniqueness theorem, no ansatz smuggled in via self-citation, and no equation in the paper reduces to the experimental outcome. Therefore, no significant circularity is present.
Assumptions & free parameters
free parameters (4)
- Z_cov threshold =
-1
- Z_loss threshold =
3
- Selection budgets k_AS =
ScanQA: 2000 initial, 1000 per epoch; 3D-VisTA: 3000 initial, 1500 per epoch, with a piecewise schedule in Eq (6)
- Evaluation score thresholds =
20/21/22/23% for ScanQA, 45/48/49% for SQA3D
assumptions (4)
- domain assumption Semantically related answer terms are close in the BERT embedding space, so weighted variance in that space reflects answer uncertainty.
- ad hoc to paper An ideal predictive distribution over terms is approximately Gaussian in semantic space centered at the weighted mean (N(phi_bar, k^-1 I)).
- ad hoc to paper Instances with low weighted variance and high loss are likely mislabeled rather than hard but correct.
- domain assumption The model trained on selected data is a reliable judge of label correctness after a few epochs.
Cite this review
Pith. "Pith review of Learn 3D VQA Better with Active Selection and Reannotation." pith.science (2026). https://pith.science/paper/274I6AO5
@misc{pith2026250704630,
author = {Pith},
title = {Pith review of: Learn 3D VQA Better with Active Selection and Reannotation},
year = {2026},
howpublished = {\url{https://pith.science/paper/274I6AO5}},
note = {Machine review of arXiv:2507.04630}
}
read the original abstract
3D Visual Question Answering (3D VQA) is crucial for enabling models to perceive the physical world and perform spatial reasoning. In 3D VQA, the free-form nature of answers often leads to improper annotations that can confuse or mislead models when training on the entire dataset. While other text generation tasks can mitigate this issue by learning on large-scale datasets, the scarcity of 3D scene data enlarges the negative effect of misleading annotations. Although active learning strategies can select valuable instances for training, they fail to identify and resolve misleading labels, which the oracle inevitably provides in practice. To address this issue, we propose a multi-turn interactive active learning strategy. This strategy selects data based on models' semantic uncertainty to form a solid knowledge foundation more effectively and actively requests reannotation from an oracle to resolve potentially misleading labels. For uncertainty assessment, we utilize a variance-based metric that takes semantic relationships between terms into consideration, thus avoiding the uniform inter-class similarity assumption of previous assessment metrics. Extensive experiments exhibit better model performance and a substantial reduction in training costs, with a halving of training costs for achieving relatively high accuracy. The code is available at https://github.com/fz-zsl/AQuA.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. 2022. ScanQA: 3D Question Answering for Spatial Scene Understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . 19129–19139
work page 2022
-
[2]
Beluch, Tim Genewein, Andreas Nurnberger, and Jan M
William H. Beluch, Tim Genewein, Andreas Nurnberger, and Jan M. Kohler. 2018. The Power of Ensembles for Active Learning in Image Classification. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . 9368–9377. https://doi.org/10.1109/CVPR.2018.00976
arXiv 2018
-
[3]
Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. 2024. LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding Reasoning and Planning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . 26428–26438
work page 2024
-
[4]
ActiveAnno3D -- An Active Learning Framework for Multi-Modal 3D Object Detection
Ahmed Ghita, Bjørk Antoniussen, Walter Zimmer, Ross Greer, Christian Creß, Andreas Møgelmose, Mohan M. Trivedi, and Alois C. Knoll. 2024. Ac- tiveAnno3D - An Active Learning Framework for Multi-Modal 3D Object Detec- tion. CoRR abs/2402.03235 (2024). https://doi.org/10.48550/ARXIV.2402.03235 arXiv:2402.03235
work page Pith review arXiv doi:10.48550/arxiv.2402.03235 2024
-
[5]
Bin Gu, Zhou Zhai, Cheng Deng, and Heng Huang. 2021. Efficient Active Learning by Querying Discriminative and Representative Samples and Fully Exploiting Unlabeled Data. IEEE Transactions on Neural Networks and Learning Systems 32, 9 (2021), 4111–4122. https://doi.org/10.1109/TNNLS.2020.3016928
-
[6]
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 2023. 3D-LLM: Injecting the 3D World into Large Language Models. NeurIPS (2023)
work page 2023
-
[7]
Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. 2024. An Embodied Generalist Agent in 3D World. In Proceedings of the 41st Interna- tional Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235), Ruslan Salakhutdinov, Zico Kolter, Katherine ...
work page 2024
-
[8]
Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. 2023. VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models. arXiv preprint arXiv:2307.05973 (2023)
arXiv 2023
Show all 33 references
-
[9]
Anita Krishnakumar. 2007. Active Learning Literature Survey
2007
-
[10]
Lewis and Jason Catlett
David D. Lewis and Jason Catlett. 1994. Heterogeneous Uncertainty Sampling for Supervised Learning. In Machine Learning Proceedings 1994 , William W. Cohen and Haym Hirsh (Eds.). Morgan Kaufmann, San Francisco (CA), 148–156. https://doi.org/10.1016/B978-1-55860-335-6.50026-X
1994 doi
-
[11]
Jingyuan Li and Eli Shlizerman. 2020. Sparse Semi-Supervised Action Recognition with Active Learning. arXiv:2012.01740 [cs.CV] https://arxiv.org/abs/2012.01740
2020 arXiv
-
[12]
Rui Li, Baopeng Zhang, Jun Liu, Wei Liu, Jian Zhao, and Zhu Teng. 2023. Het- erogeneous Diversity Driven Active Learning for Multi-Object Tracking. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV) . 9898–9907. https://doi.org/10.1109/ICCV51070.2023.00911
2023
-
[13]
Xiongquan Li, Yao Lu, Hongpeng Fu, et al. 2023. Unlabeled Data Selection for Active Learning in Image Classification. PREPRINT (Version 1) available at Research Square. https://doi.org/10.21203/rs.3.rs-2988304/v1
2023 doi
-
[14]
Jinpeng Lin, Zhihao Liang, Shengheng Deng, Lile Cai, Tao Jiang, Tianrui Li, Kui Jia, and Xun Xu. 2024. Exploring Diversity-Based Active Learning for 3D Object Detection in Autonomous Driving. IEEE Transactions on Intelligent Transportation Systems 25, 11 (2024), 15454–15466. h...
2024
-
[15]
Zhuoming Liu, Hao Ding, Huaping Zhong, Weijia Li, Jifeng Dai, and Conghui He. 2021. Influence Selection for Active Learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) . 9274–9283
2021
-
[16]
Yadan Luo, Zhuoxiao Chen, Zijian Wang, Xin Yu, Zi Huang, and Mahsa Bak- tashmotlagh. 2023. Exploring Active 3D Object Detection from a Generalization Perspective. In The Eleventh International Conference on Learning Representations . https://openreview.net/forum?id=2RwXVje1rAh
2023
-
[17]
McDuff, and Yale Song
Shuang Ma, Zhaoyang Zeng, Daniel J. McDuff, and Yale Song. 2021. Active Con- trastive Learning of Audio-Visual Video Representations. In International Con- ference on Learning Representations . https://api.semanticscholar.org/CorpusID: 235613436
2021
-
[18]
Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. 2023. SQA3D: Situated Question Answering in 3D Scenes. In International Conference on Learning Representations . https://openreview.net/ forum?id=IDJx97BC38
2023
-
[19]
Radek Mackowiak, Philip Lenz, Omair Ghori, Ferran Diego, Oliver Lange, and Carsten Rother. 2018. CEREALS - Cost-Effective REgion-based Active Learning for Semantic Segmentation. ArXiv abs/1810.09726 (2018). https: //api.semanticscholar.org/CorpusID:52288581
2018 arXiv
-
[20]
Raghav Mehta, Changjian Shui, Brennan Nichyporuk, and Tal Arbel. 2022. Infor- mation Gain Sampling for Active Learning in Medical Image Classification. In Uncertainty for Safe Utilization of Machine Learning in Medical Imaging , Carole H. Sudre, Christian F. Baumgartner, Adria...
2022
-
[21]
Schäfer, and Thomas Brox
Sudhanshu Mittal, Joshua Niemeijer, Jörg P. Schäfer, and Thomas Brox. 2023. Best Practices in Active Learning for Semantic Segmentation. arXiv:2302.04075 [cs.CV] https://arxiv.org/abs/2302.04075
2023 arXiv
-
[22]
Wentao Mo and Yang Liu. 2024. Bridging the Gap between 2D and 3D Visual Question Answering: A Fusion Approach for 3D VQA. Proceedings of the AAAI Conference on Artificial Intelligence 38, 5 (Mar. 2024), 4261–4268. https://doi.org/ 10.1609/aaai.v38i5.28222
2024 doi
-
[23]
Nguyen and Arnold Smeulders
Hieu T. Nguyen and Arnold Smeulders. 2004. Active learning using pre-clustering. In Proceedings of the Twenty-First International Conference on Machine Learning (Banff, Alberta, Canada) (ICML ’04). Association for Computing Machinery, New York, NY, USA, 79. https://doi.org/10....
2004
-
[24]
Aneesh Rangnekar, Christopher Kanan, and Matthew Hoffman. 2023. Seman- tic Segmentation With Active Semi-Supervised Learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (W ACV). 5966– 5977
2023
-
[25]
Nicholas Roy and Andrew McCallum. 2001. Toward Optimal Active Learning through Sampling Estimation of Error Reduction. In International Conference on Machine Learning. https://api.semanticscholar.org/CorpusID:14949756
2001
-
[26]
Meng Shen, Yizheng Huang, Jianxiong Yin, Heqing Zou, Deepu Rajan, and Simon See. 2023. Towards Balanced Active Learning for Multimodal Classification. In Proceedings of the 31st ACM International Conference on Multimedia, MM 2023, Ottawa, ON, Canada, 29 October 2023- 3 Novembe...
2023
- [27]
-
[28]
Jingwen Wang, Yuguang Yan, Yubing Zhang, Guiping Cao, Ming Yang, and Michael K. Ng. 2020. Deep Reinforcement Active Learning for Medical Image Classification. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2020, Anne L. Martel, Purang Abolmaesumi, Danai...
2020
-
[29]
Di Yuan, Xiaojun Chang, Qiao Liu, Yi Yang, Dehua Wang, Minglei Shu, Zhenyu He, and Guangming Shi. 2024. Active Learning for Deep Visual Tracking. IEEE Transactions on Neural Networks and Learning Systems 35, 10 (2024), 13284–13296. https://doi.org/10.1109/TNNLS.2023.3266837
2024
-
[30]
Jiakang Yuan, Bo Zhang, Xiangchao Yan, Tao Chen, Botian Shi, Yikang Li, and Yu Qiao. 2023. Bi3D: Bi-Domain Active Learning for Cross-Domain 3D Object Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 15599–15608
2023
-
[31]
Guang Zhao, Edward Dougherty, Byung-Jun Yoon, Francis Alexander, and Xi- aoning Qian. 2021. Uncertainty-aware Active Learning for Optimal Bayesian Classifier. In International Conference on Learning Representations . https: //openreview.net/forum?id=Mu2ZxFctAI
2021
-
[32]
Zhu Ziyu, Ma Xiaojian, Chen Yixin, Deng Zhidong, Huang Siyuan, and Li Qing
-
[2023]
yes, ” regardless of the actual question. In these scenes, the model finds it optimal to memorize the scene pat- tern and answer “yes
3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment. In ICCV. MM ’25, October 27–31, 2025, Dublin, Ireland Shengli Zhou, Yang Liu, and Feng Zheng A Correctness Proof of the Uncertainty Metric To formulate the uncertainty of the model with respect to input, we de...
2025
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.