{"id":"fc36d59f-7ae2-4e40-bd34-41d230c3d1b1","arxiv_id":"2502.04469","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A questions-only replay plus attention distillation method outperforms image-replay baselines for continual visual question answering while storing no past images.","lead":"This paper presents QUAD, a continual learning method for visual question answering that stores only past questions, not images, and uses them to replay and distill attention patterns. It reports higher average performance and lower forgetting than image-storing baselines on VQAv2 and NExT-QA, which matters because it suggests privacy-preserving continual VQA is possible.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Question-only replay's sufficiency claim depends on the VQACL benchmark's recurring object groups; without them, Eq. (4)'s cross-pairs are semantically mismatched, and the paper's own §8 admits visually detailed tasks still need images.","rationale":"The reader's weakest assumption correctly targets the pairing of stored past questions with current-task images, and I agree that this is the main load-bearing premise. My stress test sharpens the same concern: the method works only if (a) the object categories in current images make the stored questions answerable, and (b) the previous model provides reliable pseudo-labels on these never-seen pairs. The VQACL benchmark's fixed object groups maximize (a), while the paper never measures (b). The ablation results are internally consistent with this story: LQR alone reaches 30.72 AP on VQAv2, and ACD alone is far weaker (13.34 AP), so the method's success depends on the specific synergy between question replay and attention distillation rather than on either component alone. I also note the absence of error bars: the headline 1.79-point margin over VQACL on VQAv2 is a single run, and the out-of-answer-set evidence in Sec. 14 uses only 10 questions per task. These issues do not justify rejection; the code is released, the benchmark comparisons are standard, and the paper honestly states its limitations in Sec. 8. They do, however, support the reader's CONDITIONAL verdict: the sufficiency claim should be scoped to settings where object recurrence and teacher reliability hold, and the main quantitative comparisons should be accompanied by seeded variance. My recommendation is therefore to keep the verdict unchanged.","tokens_in":29261,"tokens_out":8783,"duration_ms":96073,"concrete_test":"Run a pseudo-label reliability diagnostic on VQAv2: after each task transition, compare θ_{t-1}'s accuracy on original past triplets (x_p, q_p) with its accuracy on the replay pairs actually used by Eq. (4), i.e., current-task images x_c matched to stored questions q_p by object category. If the latter is markedly lower, the teacher signal is noisy on exactly the pairs LQR optimizes, and the retention gains in Table 1 cannot be attributed to recovering prior visual-linguistic associations. If the two accuracies are close and θ_t retains the original triplets, the sufficiency claim survives this check.","verdict_should_be":"UNCHANGED","load_bearing_attack":"QUAD's central assertion—that storing only questions is sufficient to mitigate forgetting (Abstract; §4.2)—is carried by Eq. (4), where the replay loss supervises the current model on pairs (x_t, q_m) formed from current-task images and stored questions. For these pairs to act as a replay of past knowledge, the previous model θ_{t-1} must produce trustworthy soft labels on image-question combinations it has never seen, and the current image must contain the object categories that make the stored question answerable. The object-matched selection in §3.3 is meaningful only because the VQACL benchmark reuses the same 80 COCO classes split into five groups across every macro-task; when macro-tasks have non-overlapping visual domains, the selected pairs are semantically incoherent and the pseudo-labels are uninformative. The paper's §8 concedes exactly this boundary: tasks like 'type' that depend on fine visual semantics degrade without stored images, and representative images 'may be necessary' for high-fidelity visual reasoning. This does not falsify the benchmark result, but it means the 'sufficiency' claim is conditional on object recurrence and on the teacher's reliability for cross-pairs, neither of which is measured.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses continual visual question answering under a new question-only rehearsal setting, VQACL-QR, in which past images are not stored. The proposed method, QUAD, combines question-only replay (Eq. 4), which pairs stored past questions with current-task images using pseudo-labels from the previous model, with attention consistency distillation (Eq. 5), which aligns self-attention distributions across tasks. Experiments on VQAv2 and NExT-QA, plus experiments with BLIP-2 and LLaVA, report state-of-the-art average performance and lower forgetting compared with image-storing baselines such as VQACL. Ablations isolate the contributions of the two loss terms, and a memory-size sensitivity study is provided.","tokens_in":29476,"tokens_out":5244,"duration_ms":53120,"significance":"If the claims hold, the central result is significant: it would show that storing only questions can mitigate forgetting in continual VQA even when image-rehearsal methods store thousands of image-question pairs, with clear benefits for memory and privacy. The paper's strengths include a clearly specified problem setting, public code, component ablations in Table 3, comparison of attention-distillation variants in Table 4, and extensions to large pretrained vision-language models. However, the paper currently overstates the sufficiency claim relative to its own evidence: the novel-composition results on NExT-QA do not actually beat the best image-storing baseline, and the main comparisons lack error bars or statistical tests.","major_comments":[{"comment":"The text in §4.2 states that QUAD achieves \"top AP scores of 40.00% on VQAv2 and 33.85% on NExT-QA\" for novel composition testing, but Table 1 reports QUAD at 33.21% for NExT-QA novel composition while VQACL reports 33.85%. Because the paper's headline claim is that question-only storage surpasses image-storing methods, this contradiction is load-bearing: on NExT-QA novel compositions, QUAD does not outperform VQACL. The text must be corrected and the claim tempered accordingly.","section":"§4.2, Table 1"},{"comment":"All main results are reported as single runs without standard deviations or statistical tests. For example, QUAD versus VQACL on the standard test is 39.25% vs 37.46% on VQAv2 and 31.70% vs 30.86% on NExT-QA; the latter margin is 0.84 percentage points, which could easily be within run-to-run variation. The paper should report mean±std over at least three random seeds (or confidence intervals) for the main tables before claiming state-of-the-art superiority.","section":"Tables 1–2, §4.2"},{"comment":"The claim that \"storing only questions is sufficient to mitigate forgetting\" is broader than the evidence. Eq. (4) only provides meaningful replay when current-task images share object categories with stored questions, and the paper's own limitation section (§8) concedes that tasks requiring detailed visual or spatial reasoning, such as 'type' in Fig. 4, still degrade and that storing representative images \"may be necessary\" for high-fidelity visual reasoning. The paper should either restrict the sufficiency claim to object-recurrent, conceptually driven tasks or provide a quantitative task-level analysis showing how much forgetting is actually prevented.","section":"Abstract, §4.2, §8, Eq. (4)"},{"comment":"The replay loss relies on soft pseudo-labels generated by the previous model θ_{t-1} for image-question pairs (x_t, q_m) that the teacher has never seen. The paper does not measure the reliability of these pseudo-labels, nor does it compare against an oracle-label variant. Since the entire question-only replay mechanism depends on these cross-pairs being semantically coherent and the pseudo-labels being trustworthy, the authors should report pseudo-label accuracy on the replayed pairs or ablate with ground-truth answers to establish that the replay signal is not systematically misleading.","section":"Eq. (4), §3.3"},{"comment":"The controlled out-of-answer-set experiment uses only 10 questions per task and no error bars, so the confusion matrices in Fig. 2 cannot quantitatively support the claimed severity of the phenomenon. The paper should report the full evaluation set with confidence intervals, or explicitly present Fig. 2 as an illustrative example rather than as an empirical evaluation.","section":"§14, Fig. 2"}],"minor_comments":[{"comment":"The caption mentions \"Without replay: Blue\" but does not explain what answer this refers to or which image-question pair produced it; please clarify.","section":"Fig. 1"},{"comment":"The example pairs counting cars with the stored question \"What's the color of the car?\", which is not a counting question; this obscures the intended object-matched selection. Clarify that selection matches object categories, not question types or reasoning skills.","section":"§3.3"},{"comment":"There are several typos, e.g., \"distilaltion\" in Supp. §7 and \"lean new skills\" in the Introduction; please proofread the text.","section":"Supp. §7 and Introduction"},{"comment":"The legend for the memory-type column is garbled (\"? and /ctre\"); the symbols used for question-only versus image-plus-question memory should be defined cleanly and consistently.","section":"Table 1"},{"comment":"Cross-entropy is applied to two-dimensional self-attention maps; please specify how the maps are flattened or normalized so that LCE is well-defined as a distributional loss.","section":"Eq. (5)"},{"comment":"The sensitivity analysis for λ is reported only on VQAv2; since λ is a central hyperparameter, the same analysis should be shown on NExT-QA or justified as unnecessary.","section":"Supp. §12"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the core idea is interesting, but I would not be comfortable with acceptance before the NExT-QA novel-composition discrepancy is fixed and the main tables include variance estimates. The limitations in §8 are honest but currently conflict with the abstract's sufficiency claim; the authors need to align the claims with the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a genuinely useful paper and the main result holds up on its own terms. It introduces VQACL-QR, a setting where only past questions are stored, and shows that question-only replay plus attention distillation beats image-storing baselines on VQAv2 and NExT-QA. The ablations are clean: LQR alone gives 30.72 AP, ACD alone is weak at 13.34, and together they reach 39.25. That is real evidence the components synergize.\n\nWhat is new: the problem setting itself, the object-matched question selection, and the cross-entropy attention distillation on normalized maps. The building blocks are established techniques, but the combination is new and, as far as I can tell, fairly evaluated against VQACL and other baselines under the same backbone. Code is released, which helps reproducibility. Citation pattern looks fine: the relevant continual learning and VQA literature is covered.\n\nSoft spots. First, no error bars or multi-seed runs anywhere in Tables 1 and 2. For a paper claiming state-of-the-art by 1.8 points on one benchmark and 0.84 on another, single-run numbers are a real weakness, though not disqualifying. Second, the sufficiency claim is overreached. The abstract and Sec. 4.2 say storing only questions is sufficient to mitigate forgetting, but the paper's own Sec. 8 admits tasks like 'type' degrade without stored images, and the NExT-QA discussion in Sec. 4.2 concedes that storing images may be necessary for temporal and causal reasoning. That tension should be resolved by reframing the claim as \"sufficient on benchmark settings with recurring object groups.\"\n\nThe stress-test concern is legitimate and worth saying out loud: the object-matched selection in Sec. 3.3 only works because the VQACL benchmark reuses the same 80 COCO classes split into five groups across macro-tasks. When macro-tasks have non-overlapping visual domains, the pairs (x_t, q_m) in Eq. (4) become semantically incoherent and the pseudo-labels from the previous model carry little signal. The paper does not measure this boundary. It is not a fatal flaw, but it should be stated as a scope condition. Also minor: the out-of-answer-set experiment in Sec. 14 uses only 10 questions per task, which makes it illustrative rather than evidential.\n\nWho this is for: anyone working in continual VQA or multimodal continual learning. It deserves a serious referee. I would ask for seeded runs, a corrected scope claim, and ideally an experiment that breaks object-group overlap to test the boundary condition.\n\nRecommendation: send to peer review; expect heavy but feasible revision.","headline":"A solid, well-ablated paper on question-only replay for continual VQA that deserves review, but its central 'sufficiency' claim should be softened and it needs error bars.","tokens_in":30056,"tokens_out":1767,"would_cite":true,"duration_ms":18284,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In continual visual question answering, storing only past question text—no images—and distilling attention from the previous model beats every tested method that stores full image-question pairs, reaching 39.25% average performance on…","keywords":["continual learning","visual question answering","catastrophic forgetting","question-only replay","attention distillation","out-of-answer-set problem","VQACL","privacy-preserving machine learning"],"falsifier":"The decisive experiment is to break the object-category match between stored questions and current images: permute the five object groups per subtask so the images a question is paired with never contain the objects the question asks about, keeping the model, losses, and memory buffer otherwise identical. If QUAD's average performance on VQAv2 falls back toward the memory-free baseline rather than staying near 39%, the shared-category pairing is the load-bearing assumption; if it stays high, question replay generalizes beyond object overlap.","tokens_in":29039,"feed_emoji":"💬","tokens_out":9030,"duration_ms":74562,"temperature":0.7,"pith_summary":"This paper aims to establish a counterintuitive result in continual visual question answering: to keep a model from forgetting how to answer past questions, you do not need to keep any of the past images. The proposed method, QUAD, stores only the text of past questions, replays them against current-task images, and uses two regularizers—soft pseudo-labels from the previous model and cross-entropy alignment of attention maps—to hold the model's knowledge in place. On the VQACL benchmark the method reaches 39.25% average performance on VQAv2 and 31.70% on NExT-QA, both above the strongest image-storing baseline (VQACL at 37.46% and 30.86%), with lower forgetting. If this holds, the practical payoff is that privacy-conscious continual VQA is not a compromise: storage falls from image-plus-question to question-text alone, and performance does not drop. The paper is candid that visually heavy skills such as fine-grained object type identification still suffer without stored images, so the claim is about sufficiency for the benchmark's skill mix rather than for all VQA skills.","feed_headline":"Questions-only replay beats image memory in continual VQA","feed_subtitle":"Keeping only the text of past questions beats storing whole images on VQAv2 and NExT-QA, with less forgetting.","key_machinery":"The load-bearing machinery is the stability loss $L_{\\mathrm{Stability}} = L_{\\mathrm{QR}} + L_{\\mathrm{ACD}}$, evaluated on pairs $(x_t, q_m)$ of a current-task image and a stored past question. $L_{\\mathrm{QR}}$ is a cross-entropy between the current model's output and the previous model's soft output on that pair---soft pseudo-labels, deliberately not argmaxed, so the full output distribution is preserved. $L_{\\mathrm{ACD}}$ is a cross-entropy between the previous and current models' softmax-normalized self-attention maps, summed over all layers and heads, which weights corrections toward highly attended regions and leaves low-attended regions flexible. A third, selection-side mechanism carries the argument: questions are drawn from memory only if their object category matches the current subtask's object group, which keeps replayed pairs semantically coherent (a counting question is not paired with an image of cows when the current task counts cars). The named problem the mechanism targets is the out-of-answer-set problem, the multimodal analogue of class recency bias, where fine-tuning overfits the answer vocabulary of the newest task.","core_discovery":"On its own terms, the paper's discovery is that question-only replay plus attention consistency distillation makes past images dispensable in the VQACL setting. The question-only replay term $L_{\\mathrm{QR}}$ builds new image-question pairs from current images and stored past questions—matched so the question's object category and the image's content overlap—and trains the current model to imitate the previous model's soft output distribution on these pairs, without the argmax. The attention-consistency term $L_{\\mathrm{ACD}}$ distills the previous model's normalized self-attention maps into the current model, preserving both intra-modal (text-text, image-image) and inter-modal (text-image) focus patterns. Together the two terms counteract the out-of-answer-set problem, the paper's name for the tendency of sequential fine-tuning to collapse all responses into the latest task's answer vocabulary. The empirical claim is that this recipe outperforms every tested method that does store images, on standard and on novel-composition tests, which the paper reads as evidence that questions alone can carry the visual-linguistic associations needed to mitigate forgetting.","pith_inferences":["The same trick may transfer to other multimodal continual settings—video question answering, image captioning, visual dialogue—where the text side is the cheaper and less sensitive modality to retain; the paper does not test these settings.","The visible weakness on 'type' tasks suggests a hybrid that the paper only gestures at: keep a small buffer of images exclusively for visually intensive question types and rely on question replay everywhere else; per-task AP curves could be measured against how many images per type are retained.","Because question text can still correlate with sensitive visual content (a question about a person's identity is not anonymous), QUAD reduces storage-related privacy risk but does not eliminate information leakage; an adversarial re-identification test on stored questions would quantify what remains.","The 'judge' task jump (62.6% with the full method versus 35.0% with replay alone, per Fig. 4) is attributed to attention distillation; a targeted ablation that freezes $L_{\\mathrm{ACD}}$ only for spatially demanding tasks would test whether attention alignment is the carrier or whether soft pseudo-labels alone suffice."],"forward_implications":["On the VQACL benchmark, a model that never stores images forgets less than every tested image-storing method: 4.91% average forgetting on VQAv2 versus 5.99% for the best prior method, and 2.91% versus 4.12% on NExT-QA.","Storage cost drops from image-plus-question to question-text alone, reducing the memory footprint from $O(N \\cdot (I + L_q + L_a))$ to $O(N \\cdot L_q)$ with less than 5% GPU-memory impact, since the number of processed pairs is unchanged.","Performance on novel compositions—unseen object-skill pairings—lands within 0.75 percentage points of standard performance on VQAv2, so the question-only signal transfers to combinations the model never trained on.","Larger memory buffers help monotonically, and object-matched question selection beats random pairing at every buffer size, so the selection criterion, not the buffer alone, is doing the work."],"supporting_citations":[{"why":"Defines the VQACL setting and evaluation protocol, and provides the 5000-sample image-question buffer baseline that QUAD must beat on standard and novel-composition tests.","marker":"[95]"},{"why":"The simple experience-replay baseline that stores image-question pairs; QUAD's question-only replay directly replaces its buffer contents.","marker":"[9]"},{"why":"Dark experience replay, the rehearsal baseline with stored-logit distillation, a key comparison in the main tables.","marker":"[8]"},{"why":"Virtual-sample rehearsal baseline, providing the second rehearsal competitor and its feature-consistency losses.","marker":"[77]"},{"why":"Supplies the asymmetric attention-regularization baseline (Asym-Attn) that QUAD's cross-entropy attention distillation is compared against in the ablation and analysis.","marker":"[63]"},{"why":"Contributes the attention-distillation mechanism and the regularization framing that the question-only replay and attention-consistency losses build upon.","marker":"[14]"},{"why":"Justifies using soft pseudo-labels rather than argmax outputs in the question-only replay loss, allowing full-distribution alignment.","marker":"[30]"},{"why":"Provides the learning-without-forgetting template of distilling the previous model's outputs on replayed inputs, which the question-only replay loss instantiates with question-only memory.","marker":"[47]"}],"fun_headline_variants":["Forget images, remember questions: QUAD boosts continual VQA","Question-only replay outshines image memory in continual VQA","No images stored, yet less forgetting: QUAD sets new VQA-CL SOTA","QUAD asks and remembers: text-only replay for continual VQA","Drop the images, keep the questions: better continual VQA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method works only if pairing a stored past question with a current-task image that shares the same object category recreates enough of the original image-question association for the previous model's answers to be a trustworthy teaching signal.","fun_headline_variants_meta":{"raw":{"variants":["Forget images, remember questions: QUAD boosts continual VQA","Question-only replay outshines image memory in continual VQA","No images stored, yet less forgetting: QUAD sets new VQA-CL SOTA","QUAD asks and remembers: text-only replay for continual VQA","Drop the images, keep the questions: better continual VQA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000178,"raw_usage":{"total_tokens":1332,"prompt_tokens":1016,"completion_tokens":316,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":632,"completion_tokens_details":{"reasoning_tokens":222}},"tokens_in":632,"tokens_out":316,"duration_ms":3527,"temperature":1.0,"reasoning_tokens":222,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T22:35:43.562052+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"The decisive experiment is to break the object-category match between stored questions and current images: permute the five object groups per subtask so the images a question is paired with never contain the objects the question asks about, keeping the model, losses, and memory buffer otherwise identical. If QUAD's average performance on VQAv2 falls back toward the memory-free baseline rather than staying near 39%, the shared-category pairing is the load-bearing assumption; if it stays high, question replay generalizes beyond object overlap.","supporting_citations":[{"cited_title":"Vqacl: A novel visual question answering continual learning setting","cited_arxiv_id":null,"evidence_quote":"Defines the VQACL setting and evaluation protocol, and provides the 5000-sample image-question buffer baseline that QUAD must beat on standard and novel-composition tests."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Virtual-sample rehearsal baseline, providing the second rehearsal competitor and its feature-consistency losses."},{"cited_title":"Towards exemplar-free continual learning in vision transformers: an account of at- tention, functional and weight regularization","cited_arxiv_id":null,"evidence_quote":"Supplies the asymmetric attention-regularization baseline (Asym-Attn) that QUAD's cross-entropy attention distillation is compared against in the ablation and analysis."},{"cited_title":"Learning without forgetting","cited_arxiv_id":null,"evidence_quote":"Provides the learning-without-forgetting template of distilling the previous model's outputs on replayed inputs, which the question-only replay loss instantiates with question-only memory."}],"review_version":1}