{"id":"02aa9a1c-ce60-4f4f-9f8b-643f55236995","arxiv_id":"2507.04630","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An active learning loop with semantic-variance uncertainty selection and oracle reannotation improves 3D VQA training efficiency, but gains are small and validation tuning is a concern.","lead":"The paper proposes a training strategy for 3D visual question answering in which the model selects the most informative questions and asks an oracle to reannotate likely-mislabeled answers. It reports faster training and small accuracy gains, though the evaluation has gaps around validation-set tuning and missing error bars.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reannotation component is undercut by the paper's own ScanQA ablation: diligent-oracle exact-match (19.42) is below lazy-oracle (19.83), so low-variance/high-loss instances are not a reliable mislabel signal.","rationale":"I focused on the reannotation filter because it is the methodological novelty beyond ordinary active selection and because the paper's own Table 3 provides a direct internal check. On ScanQA, adding active reannotation to active selection reduces exact-match from 19.83 to 19.42, so the claim that reannotation reliably improves performance is not supported on the primary metric of the primary dataset. The paper's Appendix C further documents that the filter flags many correctly labeled instances once the model overfits, confirming that the underlying assumption is fragile. The reader's weakest-assumption analysis already identified this as the key risk; my proposed test settles it by moving the comparison off the threshold-tuned validation split to a held-out test split with seed variance. The remaining issues (no error bars, validation-set tuning, omitted selection/annotation cost) are real but secondary; they would not change the conditional verdict if the reannotation check passes. Therefore the verdict remains conditional pending this test.","tokens_in":16673,"tokens_out":8372,"duration_ms":97020,"concrete_test":"Re-run the ScanQA ablation (Random, Ablation, Lazy Oracle, Diligent Oracle) on the official test split with five seeds and fixed validation-tuned thresholds, and report exact-match with standard deviations. If the diligent-oracle EM is not above the lazy-oracle EM, or if the 95% confidence interval covers zero, the active reannotation claim fails its decisive test.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3's three-case categorization assumes that low weighted variance plus high loss identifies an improperly annotated instance, and Algorithm 1 sends exactly those instances to the oracle. Table 3 contradicts this on ScanQA: Random (no AS/AR) is 19.19 EM, Ablation (random selection + AR) is 19.59, Lazy Oracle (AS, labels unchanged) is 19.83, and Diligent Oracle (AS+AR) is 19.42. Active reannotation therefore lowers exact-match by 0.41 relative to active selection alone, on the same exact-match metric used for the cost-reduction headline. The paper's own Appendix C (Table 6) shows that at later epochs the filter flags many correct answers (CA/MA), and Figure 3 acknowledges the false-filtration ratio rises with overfitting. The load-bearing premise that 'confident-and-wrong means mislabeled' is thus not reliable in the deployed setting, and the reannotation-specific contribution is not supported on the primary benchmark.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AQuA, a multi-turn interactive active learning strategy for 3D visual question answering. The method combines active selection based on the log-determinant of a weighted covariance matrix of BERT semantic embeddings with an active reannotation filter that sends low-variance/high-loss labeled instances to an oracle for correction. The oracle uses a hierarchical strategy that first maps non-canonical answers to a refined corpus and resorts to manual reannotation only for remaining cases. Experiments on ScanQA and SQA3D with ScanQA and 3D-VisTA models report improved exact-match/BLEU/CIDEr/ROUGE scores and reduced training cost to reach fixed accuracy thresholds, including a claimed halving of training cost.","tokens_in":16903,"tokens_out":5190,"duration_ms":58460,"significance":"If the claims hold, the paper offers a practical way to train 3D VQA models with fewer labels and to clean noisy free-form answer annotations with modest human effort. The paper has several strengths: it ships code, evaluates on two datasets and two base models, provides qualitative examples of real annotation errors, and the variance-based selection metric is a reasonable alternative to entropy-based uncertainty for free-form answers. The active selection results are plausible and the cost-reduction numbers are interesting. However, the active reannotation component is not consistently supported by the primary benchmark, and the theoretical derivation of the uncertainty metric contains a mathematical gap, so the current evidence does not fully support the paper's central claims.","major_comments":[{"comment":"The quantity D_KL(P || N(\\bar{\\phi}, k^{-1} I)) is not a well-defined KL divergence as written: P is a discrete distribution over corpus terms while N is a continuous Gaussian density, so the log-ratio p_c / N(\\phi_c | ...) mixes probability masses with density values and the derivation silently cancels the \\sum_c p_c \\log p_c terms. The weighted-variance score can be motivated directly as the expected squared distance of predicted terms from their semantic mean, but the stated KL derivation is mathematically invalid. Please rewrite the derivation using a discrete reference distribution over the corpus, or explicitly present the weighted-variance score as an independent objective rather than as a KL-based quantity.","section":"Section 3.2, Eqs. (2)-(4), Appendix A"},{"comment":"On ScanQA, the Diligent Oracle (active selection plus active reannotation) achieves 19.42 exact match, which is lower than the Lazy Oracle (active selection only) at 19.83; on the exact-match metric used for the cost-reduction headline, active reannotation therefore reduces performance by 0.41 points. The statement that \"results under most metrics show improvements\" does not address the primary metric, and Table 3 reports no error bars, so the reannotation-specific contribution is not supported on the main benchmark. This needs either corrected results with multiple seeds or a clearly scoped claim that reannotation helps only on non-EM metrics and on specific model/dataset combinations.","section":"Section 4.2.2, Table 3"},{"comment":"All experimental tables appear to report single runs without standard deviations, confidence intervals, or significance tests. Since many reported gains are below 0.5 EM points (e.g., 19.42 vs 19.83 in Table 3) and the training-cost reductions are computed from cumulative best scores, it is impossible to assess whether the reported differences are reproducible. Please provide multiple seeds or a statistical comparison, or at a minimum report variance across runs.","section":"Section 4.2.2, Tables 1-3"},{"comment":"The filtration rule assumes that low weighted variance plus high loss indicates a mislabeled instance, but Appendix C Table 6 shows that the filter flags many correct or acceptable answers (CA/MA) at several epochs, and Figure 3 reports a nontrivial false-filtration ratio. The manuscript should quantify the precision of the filter on a held-out set of labeled examples and discuss how the false-filtration ratio affects the training pipeline; as written, the empirical support for the load-bearing \"confident-and-wrong means mislabeled\" assumption is weak.","section":"Section 3.3, Figure 3, Appendix C"}],"minor_comments":[{"comment":"The thresholds Z_cov=-1 and Z_loss=3 are tuned on the validation set; please report sensitivity to these thresholds or show that the conclusions are stable across reasonable choices.","section":"Section 4.1.1"},{"comment":"The text says the variance-based strategy outperforms other strategies \"across all metrics,\" but Table 1 reports only cost-reduction percentages and AUC; please report the underlying metric curves or clarify the claim.","section":"Section 4.2.1, Table 1"},{"comment":"The predicates require_selection(epoch_idx) and require_reannotation(epoch_idx) are not defined in a single place; please specify the epoch schedule for each dataset/model in a table or in the main text.","section":"Algorithm 1"},{"comment":"The proof labels the quantity in Eq. (16) as proportional to weighted variance, but the additive constant includes -m/2 log(k/2pi) - log|C|; this is fine when m, k, and |C| are fixed, but the text should state that these are fixed per problem instance.","section":"Appendix A, Eq. (16)"},{"comment":"The y-axis label \"ratio of instances filtered out due to false filtration and correct filtration\" is ambiguous; please specify the denominator explicitly, e.g., the number of filtered instances.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for ACM MM and the active-selection component is interesting, but the reannotation ablation on ScanQA undermines the paper's headline claim. The authors should be encouraged to address the ScanQA Table 3 contradiction directly rather than relying on \"most metrics,\" and to provide multi-seed results before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The active selection side of this paper is the real contribution. The semantic-variance metric is a sensible adaptation of uncertainty sampling to free-form 3D VQA answers, and the lazy-oracle results show a consistent training-cost reduction over random, entropy, and information-gain baselines on ScanQA, with a smaller but positive effect on 3D-VisTA. The derivation in Section 3.2 is algebraically fine, though the KL between a discrete distribution and a continuous Gaussian is notationally abusive; in practice the cancellation makes it a weighted variance, so the method stands. The paper is also transparent: it includes the ablation that exposes the reannotation weakness and an appendix showing false filtrations at later epochs. That honesty earns credit.\n\nThe reannotation contribution does not hold up on the primary benchmark. The stress-test note is correct: in Table 3, on ScanQA, the diligent oracle (AS+AR) gets 19.42 exact match, below the lazy oracle's 19.83. The paper claims improvement \"under most metrics,\" but exact match is the metric used for the cost-reduction headline, and there active reannotation is a net negative. The load-bearing assumption that low variance plus high loss reliably identifies mislabeled instances is contradicted by the paper's own filtration examples: many flagged items are correct answers or acceptable alternatives. On 3D-VisTA the diligent oracle does beat the lazy oracle, so the idea is not dead, but the evidence is mixed and the mechanism is not as clean as claimed.\n\nBeyond that, the evaluation is methodologically thin: no error bars or multiple seeds, thresholds Z_cov=-1 and Z_loss=3 tuned on the validation set that is also used for reporting, and a cost comparison that ignores the computational expense of running selection and the human effort of reannotation. These are not fatal, but they need addressing before the central claims can be trusted.\n\nWho is this for? Researchers working on active learning for 3D VQA or on label-noise correction for small multimodal datasets. The active selection method is worth citing; the reannotation loop needs a careful re-run with proper controls. I would send this to peer review, but require independent seeds, a held-out test evaluation, and a cost model that includes selection and oracle overhead before acceptance.","headline":"Active selection part is solid; the reannotation half is undercut by the paper's own ScanQA ablation, but the work deserves serious refereeing.","tokens_in":17399,"tokens_out":1675,"would_cite":true,"duration_ms":20381,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Coupling variance-based active selection with oracle reannotation trains 3D VQA models at roughly half the cost.","keywords":["active learning","3D visual question answering","reannotation","weighted variance","semantic uncertainty","ScanQA","SQA3D"],"falsifier":"Have independent annotators re-examine all instances that the filtration rule flags (low weighted variance, high loss) on ScanQA; if the majority turn out to be correctly labeled hard examples rather than mislabeled ones, the active-reannotation premise fails. A cheaper check is already visible in the paper's own ablation: on ScanQA the diligent oracle's exact-match accuracy (19.42) is lower than the lazy oracle's (19.83), so any setting where that gap widens would falsify the claim that reannotation helps.","tokens_in":16443,"feed_emoji":"🎯","tokens_out":7735,"duration_ms":73644,"temperature":0.7,"pith_summary":"The paper's claim is that a 3D VQA model can be trained better and at roughly half the cost if, instead of using the whole dataset, the training loop repeatedly lets the model pick the questions it is most semantically uncertain about and also asks an oracle to recheck answers the model is confidently wrong about. The motivation is that free-form 3D VQA answers are frequently mislabeled (wrong answers, or correct information in non-canonical forms such as '2 chairs' instead of '2'), and the small size of 3D scene datasets makes these bad labels more damaging than in large-scale text tasks. To do this, the paper replaces the usual uniform-similarity assumption of active learning with a variance-based uncertainty score computed in answer-embedding space, and couples selection with a reannotation filter targeting low-variance, high-loss predictions. If the claim holds, the same accuracy can be reached with about half the training steps, and mislabeled training data can be caught and corrected with modest human effort.","feed_headline":"Active learning + reannotation halves 3D VQA training cost","feed_subtitle":"Picking semantically uncertain questions and rechecking suspicious answers lifts accuracy on fewer labels.","key_machinery":"The load-bearing object is the weighted semantic variance of the model's answer distribution, $\\operatorname{var}_p(\\phi) = \\sum_{c \\in C} p_c \\|\\phi_c - \\bar{\\phi}\\|_2^2$, where $\\phi_c$ is a pretrained BERT embedding of answer term $c$ and $\\bar{\\phi}$ is the probability-weighted mean embedding. The paper shows that its uncertainty metric, a KL divergence between the prediction and a Gaussian in semantic space minus the KL divergence to the uniform distribution, reduces up to an additive constant to exactly this variance, so ranking instances by the log determinant of the weighted covariance matrix $\\log|\\mathrm{cov}_p(\\phi)|$ implements the same criterion while avoiding the curse of dimensionality. This same quantity, combined with the term classification loss, defines the three-way partition of labeled instances (confident-correct, uncertain, and confident-wrong); only the confident-wrong group, where the model is certain of an answer different from the ground truth, is sent for reannotation.","core_discovery":"The central discovery the paper argues for is that coupling active selection with active reannotation turns a noisy, scarce 3D VQA training set into a stronger curriculum. Active selection ranks unlabeled questions by the log determinant of the weighted covariance of predicted answer embeddings, $\\log|\\mathrm{cov}_p(\\phi)|$, which the paper derives from an uncertainty metric $\\Delta = D_{\\mathrm{KL}}(P \\,\\|\\, \\mathcal{N}(\\bar{\\phi}, k^{-1} I)) - D_{\\mathrm{KL}}(P \\,\\|\\, U)$ that reduces exactly to the weighted variance $\\operatorname{var}_p(\\phi) = \\sum_{c \\in C} p_c \\|\\phi_c - \\bar{\\phi}\\|^2$; this respects semantic similarity between answer terms instead of treating all pairs as equally distinct. Active reannotation then flags labeled instances whose predictions are confident (low weighted variance) yet wrong (high loss), and requests the oracle to check and correct them; a hierarchical oracle first maps non-canonical answers to canonical terms automatically and only resorts to manual reannotation for the leftovers. Experiments on ScanQA and SQA3D with the ScanQA and 3D-VisTA models show that the weighted-variance selection beats entropy and information-gain baselines on all reported metrics, and that the full multi-turn loop reaches fixed accuracy thresholds in roughly half the training cost of random selection.","pith_inferences":["Editorial extension: the same confident-wrong filter could be reused as a dataset-cleaning audit for other free-form generation tasks where mislabeled or non-canonical answers are common, such as 2D VQA or image captioning.","Editorial extension: the method's advantage should depend on how noisy the labels are; on cleaner datasets the reannotation step may hurt, and the ScanQA exact-match drop with the diligent oracle already hints that the filter needs a noise-detection calibration.","Editorial extension: combining weighted variance with a diversity term could prevent the selector from repeatedly choosing near-duplicate questions in large unlabeled pools.","Editorial extension: a robust variant could demand disagreement across model checkpoints before flagging an instance for reannotation, reducing false positives from systematic scene-level biases."],"forward_implications":["Training with the multi-turn loop reaches fixed exact-match thresholds (e.g., 20–23% on ScanQA) in roughly half the steps of random selection, and the advantage persists on SQA3D.","Weighted-variance selection outperforms entropy and information-gain selection on every reported metric, so semantic similarity should be part of uncertainty estimation for free-form answer generation.","Reannotation for low-variance/high-loss instances improves most quality metrics on both models, with the largest gains on 3D-VisTA (BLEU-1 +6.56% relative, CIDEr +4.44%).","The hierarchical reannotation strategy, which resolves most non-canonical answers by term mapping before manual review, lowers the cost of fine-grained data cleaning.","Because the framework is agnostic to the semantic embedding, it is portable to other answer encoders without changing the selection rule."],"supporting_citations":[{"why":"Supplies the main 3D VQA dataset (ScanQA) whose noisy free-form annotations motivate the method and on which most experiments run.","marker":"[1]"},{"why":"Supplies the second benchmark dataset (SQA3D) used to show the selection strategy transfers beyond ScanQA.","marker":"[18]"},{"why":"Supplies the second 3D VQA model (3D-VisTA) whose training loop the active learning strategy wraps and on which all metrics improve.","marker":"[32]"}],"fun_headline_variants":["Ask oracle to fix wrong answers, halve 3D VQA training cost","3D VQA training cost halved by active reannotation","Weighted variance picks clearer 3D VQA training data","Semantic uncertainty beats entropy for 3D VQA data selection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reannotation filter assumes that a model that is confident but wrong about an answer is looking at a mislabeled example rather than a hard-but-correct one, so if the model's confidence is systematically biased, correct labels will be sent for unnecessary rewriting.","fun_headline_variants_meta":{"raw":{"variants":["Ask oracle to fix wrong answers, halve 3D VQA training cost","3D VQA training cost halved by active reannotation","Weighted variance picks clearer 3D VQA training data","Semantic uncertainty beats entropy for 3D VQA data selection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0006,"raw_usage":{"total_tokens":2852,"prompt_tokens":1041,"completion_tokens":1811,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":657,"completion_tokens_details":{"reasoning_tokens":1735}},"tokens_in":657,"tokens_out":1811,"duration_ms":13681,"temperature":1.0,"reasoning_tokens":1735,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:43:22.666145+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent annotators re-examine all instances that the filtration rule flags (low weighted variance, high loss) on ScanQA; if the majority turn out to be correctly labeled hard examples rather than mislabeled ones, the active-reannotation premise fails. A cheaper check is already visible in the paper's own ablation: on ScanQA the diligent oracle's exact-match accuracy (19.42) is lower than the lazy oracle's (19.83), so any setting where that gap widens would falsify the claim that reannotation helps.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the main 3D VQA dataset (ScanQA) whose noisy free-form annotations motivate the method and on which most experiments run."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the second benchmark dataset (SQA3D) used to show the selection strategy transfers beyond ScanQA."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the second 3D VQA model (3D-VisTA) whose training loop the active learning strategy wraps and on which all metrics improve."}],"review_version":1}