{"id":"3432b841-1244-4dbe-808e-a9706c0e7760","arxiv_id":"2501.02964","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A self-questioning training and inference framework for lightweight multimodal LLMs is claimed to reduce hallucinations and improve zero-shot visual reasoning.","lead":"This paper trains an AI system that looks at a picture, asks itself questions about the details, answers them, then writes a short description. The authors say this reduces made-up details and improves scores on visual question-answering tests.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CapQA's headline metric is internally inconsistent: Eq. 2 defines HalS = pred/gt while Table 6's prompt says lower raw hallucination scores are better, so higher HalS would mean more hallucination; the 31.2% improvement may be sign-reversed.","rationale":"The paper has some independent support: POPE F1 improves from 87.34 to 88.78 (Table 3), and MMHal AvgS from 2.04 to 2.16 with HalR 0.61 to 0.57 (Table 4). These external benchmarks are not generated by the same GPT-4V loop and provide partial evidence for the method. However, the paper's headline quantification of hallucination reduction relies on CapQA HalS, which is the largest effect (31.2%) and is the only metric that is both self-referential and internally inconsistent. The reader's weakest assumption (GPT-4 evaluator loop) is valid, but I identify a more elementary defect: the metric direction as defined in Eq. 2 and Table 6 is contradictory. This is an internal inconsistency, not a disagreement with consensus, so it directly undermines the headline number. The external benchmark deltas are small (at most about 1.5 points) and no error bars are reported, so once the CapQA metric is set aside, the central claim of substantial hallucination reduction is only weakly supported. Thus the reader's CONDITIONAL verdict remains appropriate; it should be conditioned on fixing the metric definition and validating it independently, ideally with human evaluation.","tokens_in":18493,"tokens_out":10107,"duration_ms":82075,"concrete_test":"Compute HalS = s_pred/s_gt for synthetic raw scores s_pred = 1..10 with a fixed s_gt (e.g., 5) under the Table 6 definition 'lower raw score = fewer hallucinations'; if HalS is monotonically increasing in s_pred, then Table 2's 'higher HalS = less hallucination' is contradicted, and the 31.2% 'improvement' in HalS actually corresponds to more hallucinations; the authors must then recompute Table 2 with the correct direction or with a corrected raw-score definition.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that SQ reduces hallucinations rests primarily on the CapQA HalS metric in Table 2 (69.3 -> 90.9, reported as '+31.2%'). That metric is not well-defined as written. Section 5.1 defines HalS = HalSpred/HalSgt and states 'higher scores indicating less hallucination.' However, the evaluation prompt in Table 6 instructs GPT-4 to rate hallucinations on a 1-10 scale 'where a lower score indicates fewer hallucinations and better performance.' Under that literal definition, a model with fewer hallucinations than the ground-truth label gets a lower raw score than the label, so HalS = pred/gt is below 100, and HalS increases as the predicted output hallucinates more relative to the label. Thus higher HalS corresponds to more hallucination, the opposite of the paper's claim. The reported numbers are therefore not interpretable: if the prompt is taken literally, LLaVA-1.5's HalS of 69.3 means it hallucinates less than GPT-4V's ground-truth labels, and SQ's 90.9 is worse, reversing the claimed 31.2% improvement. If instead the raw score is intended as a quality score (higher = less hallucination), then the prompt in Table 6 is mis-specified and the ratio's direction still needs explicit justification. Because both numerator and denominator come from the same GPT-4 family that generated the CapQA training labels, evaluator style-matching is a plausible confound; the paper's own Table 12 shows SQ HalS of 69.0 vs LLaVA's 52.0 under a GPT-4o-mini evaluator on 30 samples, far from the Table 2 ratio, suggesting high evaluator sensitivity.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Socratic Questioning (SQ), a multi-round self-questioning framework for lightweight multimodal LLMs. Given an image and a question, the model generates a list of questions, answers them, writes a detailed description, and produces a summarized caption. The authors automatically create a ~1k-image dataset (CapQA) by prompting GPT-4V, fine-tune LLaVA-1.5-7B on CapQA plus the LLaVA-1.5 mix, and evaluate on CapQA, POPE, MMHal, MME, LLaVA-QA90, LLaVA-Bench, ScienceQA-IMG, TextVQA, and VCR/SNLI-VE. The headline result is a 31.2% improvement in a GPT-4-scored hallucination metric (HalS) on CapQA.","tokens_in":18875,"tokens_out":9122,"duration_ms":82799,"significance":"The core idea — training a small MLLM to ask itself fine-grained visual questions before answering — is simple, inexpensive, and potentially useful, and the detailed hyperparameters plus the planned release of CapQA are practical contributions. If the reported hallucination reduction were robust, this would be a valuable low-cost intervention. The external benchmarks (POPE, MMHal, MME) give weak but directionally consistent support. However, the manuscript's own CapQA metric is internally inconsistent and confounded by generator-evaluator overlap, so the central quantitative claim is not currently credible. The paper contains no machine-checked proofs or released code, and the main empirical claims lack error bars.","major_comments":[{"comment":"The HalS score direction is internally inconsistent. Eq. (2) defines HalS = HalSpred/HalSgt and the text states 'higher scores indicating less hallucination,' but the GPT-4 evaluation prompt in Table 6 says 'a lower score indicates fewer hallucinations and better performance.' Under the prompt's literal meaning, an output with fewer hallucinations than the ground-truth label receives a lower raw score, so HalS < 1, and a higher HalS means more hallucination relative to the label. Read literally, the claimed improvement from 69.3 to 90.9 in Table 2 is a regression, not a 31.2% improvement. If the authors intended the raw scores as quality scores (higher = better), the prompt is mis-specified. Table 12's raw columns (LLaVA-1.5 pred 41.0 vs SQ pred 53.0 on the same 30 samples) illustrate the ambiguity. The metric must be redefined and re-run before the headline claim can be evaluated.","section":"Section 5.1, Eq. (2), Table 6"},{"comment":"The CapQA evaluation is a same-generator loop. GPT-4V produced the CapQA training annotations (Section 4.2) and the reference labels used in Eq. (2); the GPT-4 judge scores model outputs against those GPT-4V labels, and the QQS prompt even supplies the expected GPT-4V questions to the judge. A model fine-tuned to imitate GPT-4V's style is therefore rewarded by the same model family that created the data, independent of visual grounding. No human evaluation or correlation with human judgments is reported. This confound must be addressed (e.g., human ratings on a subset or a metric not based on GPT-4V-generated references) before CapQA can support the 31.2% claim.","section":"Section 5.1, Section 4.2, Table 6"},{"comment":"The independent external evidence is too thin to carry the hallucination claim on its own. POPE F1 gains over LLaVA-1.5 are 1.44, 0.77, and 0.69 points in the random, popular, and adversarial settings; MMHal Average Score moves from 2.04 to 2.16 and Hallucination Rate from 0.61 to 0.57. These are single-run differences, reported without confidence intervals or significance tests. Once the CapQA metric is corrected, the paper needs either repeated runs with variance estimates or a human evaluation to establish that SQ reduces hallucinations.","section":"Section 5.3, Tables 3 and 4"},{"comment":"The inference mode used for each reported result is not specified. The paper says 1-turn inference suits straightforward problems and 3-turn inference suits complicated ones, and Table 2 shows a non-negligible difference between the two modes (90.9 vs 93.0 HalS). Without stating which mode produced each benchmark number, the comparisons are not reproducible.","section":"Section 4.5, Tables 3, 4, 9-11"}],"minor_comments":[{"comment":"'Our proposed SQ method leads to a 31.2% improvement in the hallucination score' should be reworded to state the metric direction, the baseline, and the sign.","section":"Abstract and Section 5.1"},{"comment":"'archtecture' should be 'architecture'; 'LLaV A' spacing is inconsistent throughout; 'NeurlPS' in references [26] and [42] should be 'NeurIPS'; references [31] and [32] are duplicated.","section":"Section 3.1, References"},{"comment":"The column header is difficult to parse; a table with separate 'run', 'pred/gt', 'gt', and 'pred' columns would clarify the reported numbers.","section":"Table 12"},{"comment":"The CapQA30 HalS of 130.0 for SQ-caponly exceeds 100; the paper should explain whether normalized scores can exceed 100 and what that implies.","section":"Table 10"},{"comment":"The prompt asks for 5-8 questions and later allows up to 20, while the final format in Table 8 shows 13 questions; the refinement and deduplication procedure should be described.","section":"Section 4.2, Table 1"},{"comment":"The GitHub link should point to a released repository, and the dataset download location should be given.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper's main technical issue is not a disagreement with community consensus but an internal inconsistency in the evaluation metric, compounded by a same-generator evaluation loop. The external benchmarks are directionally consistent but small and unreplicated. I would encourage the editor to request a corrected and re-run CapQA metric, explicit inference-mode reporting, and variance estimates or human validation before further consideration. The editorial presentation (duplicate references, inconsistent notation, garbled table header) also needs cleanup."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this one. First, the idea is reasonable: fine-tune a lightweight MLLM (Vicuna-7B via LoRA) to act as question generator, answerer, and summarizer on a 1k-image GPT-4V-generated dataset, and then optionally run a 3-turn inference. That is a genuinely new training setup, and the CapQA dataset is new. Second, the headline result is not interpretable as written.\n\nThe CapQA metric has a sign problem. Eq. 2 defines HalS = HalSpred / HalSgt, and the text says higher HalS means less hallucination. But the scoring prompt in Table 6 tells GPT-4 that a lower raw score means fewer hallucinations. If you take that literally, a model that hallucinates less than the ground-truth label gets a ratio below 100, and a higher HalS means the model is actually closer to (or worse than) the label. So the reported +31.2% improvement from 69.3 to 90.9 may be backwards. The paper never resolves this, and Table 12 makes it worse: swapping the evaluator to GPT-4o-mini on 30 samples gives SQ 69.0 vs LLaVA-1.5 52.0, a completely different picture from Table 2's 90.9 vs 69.3. The metric is fragile.\n\nThe other soft spot is the evaluation loop. GPT-4V generated the CapQA training labels, the model is fine-tuned to imitate them, and GPT-4 scores the model against those same labels. That is a same-generator loop, and the QQS prompt even hands GPT-4 the expected questions, which rewards mimicry rather than visual grounding. The paper claims no human evaluation, no error bars, and code/data are promised but not released. It also uses IdealGPT as a baseline in Table 9 without citing it in related work.\n\nTo be fair, the independent benchmarks are not all empty. POPE F1, MMHal, MME, ScienceQA, and TextVQA all show small positive deltas over LLaVA-1.5. That is real evidence the method is not worthless, just much weaker than the headline claims. The missing control is fine-tuning on an extra 1k random captions to rule out that the gains come from extra data rather than the SQ structure.\n\nBottom line: this deserves a serious referee because the idea is plausible and the external signal is nonzero. But the authors need to fix the metric, release artifacts, add human evaluation and error bars, and run an extra-data control. Send it to review, expect heavy revision, and do not let the 31.2% figure survive contact with the equation.","headline":"The training recipe is plausible and cheap, but the headline hallucination metric is sign-confused as written and the evaluation loop is same-generator, so the main claim does not survive close reading.","tokens_in":19447,"tokens_out":3889,"would_cite":false,"duration_ms":69628,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a lightweight multimodal model on a small set of self-question-answer-description-caption conversations is claimed to reduce hallucinations by 31.2% and improve zero-shot visual reasoning.","keywords":["Socratic Questioning","multimodal large language models","hallucination mitigation","visual reasoning","chain-of-thought","visual instruction tuning","CapQA","zero-shot generalization"],"falsifier":"Ask human annotators to score the same CapQA test outputs for visual groundedness and question usefulness against the original images, then compare the SQ model with the base model; if humans do not find roughly the same 31% hallucination reduction, the central claim fails.","tokens_in":18290,"feed_emoji":"🧠","tokens_out":7756,"duration_ms":72997,"temperature":0.7,"pith_summary":"The paper proposes Socratic Questioning (SQ), a training-and-inference routine in which a lightweight multimodal language model asks itself a list of questions about an image, answers those questions one by one, and then consolidates the question-answer pairs first into a detailed description and finally into a short caption. The authors claim that adding just under a thousand such multi-turn exchanges—the CapQA dataset—to a standard visual-instruction mix teaches the model to ground its descriptions in fine-grained visual detail and to reason about everyday human activities it was not trained on. On their CapQA benchmark, SQ raises the question-quality metric from 31.5 to 92.3 and improves the hallucination metric by 31.2% relative to the base model, while on the POPE object-existence benchmark it reports the best F1 across random, popular, and adversarial sampling. If the claim holds, hallucination in small models can be reduced by changing the training format alone, without adding parameters, raising image resolution, or introducing extra decoding machinery.","feed_headline":"Self-questioning training cuts vision-language hallucinations by 31%","feed_subtitle":"A small training set of self-asked questions and answers teaches lightweight models to stay visually grounded.","key_machinery":"The load-bearing object is the Socratic loop—self-ask, self-answer, consolidate and organize, summarize and condense—encoded as a structured multi-turn instruction-tuning conversation. The first turn trains the model to generate a question list, the middle turns train it to answer those questions one at a time, and the last two turns train it to produce a detailed description and then a condensed caption. At inference, the same weights run the loop in 3-turn mode or skip to a caption in 1-turn mode. This loop is what the paper credits with forcing the model to retrieve fine-grained visual information before producing its final output, and therefore with reducing hallucinations and improving zero-shot reasoning.","core_discovery":"SQ is a four-step reasoning loop made into a multi-turn conversation: the model first self-asks a set of 5–8 questions targeted at the activity in the image, then self-answers each question with visually grounded VQA responses, then consolidates and organizes the Q&A into a detailed description, and finally summarizes and condenses that description into a caption retaining the core content. All four roles are played by the same lightweight language model, trained jointly; at inference the model can run the whole loop (3-turn mode) or jump straight to the final caption (1-turn mode). The authors argue this loop counters a central cause of hallucination they identify: multimodal models tend to ignore visual features during autoregressive generation, whereas the self-questioning step forces attention back to visual clues before any final claim is made. They report that this formulation, fine-tuned with LoRA on 666K image-text pairs including 882 CapQA samples, transfers zero-shot to six standard benchmarks and reduces object-existence hallucinations on POPE.","pith_inferences":["Because the CapQA training labels and the GPT-4 scoring prompt come from the same model family, the reported 31.2% improvement may partly reflect stylistic imitation rather than visual grounding; a human-scored evaluation would be needed to separate the two, and the paper does not provide one.","If SQ works by forcing visual attention, its benefit should be sensitive to the order and content of the self-asked questions; a controlled variant that substitutes generic or random questions would isolate the mechanism, but the paper does not run that ablation.","The framework suggests a cheap route to domain adaptation beyond human activities: generating CapQA-style self-questioning conversations for other fine-grained domains, such as medical images, satellite imagery, or industrial scenes, could transfer SQ benefits, a direction the paper leaves implicit.","The 1-turn versus 3-turn trade-off points toward an adaptive controller that decides whether to invoke self-questioning based on expected problem complexity, something not explored here."],"forward_implications":["If the claims hold, a 7B-parameter multimodal model can gain zero-shot visual reasoning ability from roughly a thousand extra multi-turn conversations, making the annotation budget for instruction tuning much smaller than current practice.","Hallucination mitigation would no longer require larger vision encoders, higher input resolution, or decoding-time intervention; a training-format change would suffice.","The large jump in question-quality metric (31.5 to 92.3) implies that models can learn to ask informative, answerable questions from very little data, which could be harnessed for other self-guided tasks.","The 3-turn inference mode's extra 2.3% reduction in hallucination indicates that running the full question-answer-caption loop at inference is worthwhile for complex questions, while 1-turn mode remains cheaper for simple ones."],"supporting_citations":[{"why":"Supplies the GPT-4V annotations: all CapQA questions, answers, detailed descriptions, and summarized captions used for fine-tuning.","marker":"[1]"},{"why":"Provides the source video clips and 512 fine-grained activity labels from which the 1,000 CapQA images are extracted.","marker":"[4]"},{"why":"Defines the base model and the 665k mixed instruction-tuning data that SQ is fine-tuned on and compared against.","marker":"[25]"},{"why":"Supplies the multi-round conversation format, the 595k pretraining data, and the LLaVA benchmarks used for evaluation.","marker":"[26]"},{"why":"Provides GPT-4 as the judge that computes the hallucination and question-quality metrics in the CapQA evaluation.","marker":"[29]"},{"why":"Supplies the POPE object-hallucination benchmark where SQ reports the best F1 scores across three sampling settings.","marker":"[24]"}],"fun_headline_variants":["Socratic self-questions slash vision-language hallucinations 31%","Self-questioning loop grounds MLLMs, cuts hallucinations 31%","Socratic training: teach models to question images, cut false claims","Teach MLLMs to self-ask visual questions, cut hallucinations by a third","Socratic self-questioning grounds vision models, cuts hallucinations 31%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that GPT-4's ratings are a valid and unbiased measure of hallucination and question quality, because the same model family generated the training annotations and scores the outputs, and if the judge merely rewards outputs that imitate its own generation style, the reported improvements would be an artifact rather than visual grounding.","fun_headline_variants_meta":{"raw":{"variants":["Socratic self-questions slash vision-language hallucinations 31%","Self-questioning loop grounds MLLMs, cuts hallucinations 31%","Socratic training: teach models to question images, cut false claims","Teach MLLMs to self-ask visual questions, cut hallucinations by a third","Socratic self-questioning grounds vision models, cuts hallucinations 31%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000688,"raw_usage":{"total_tokens":3131,"prompt_tokens":971,"completion_tokens":2160,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":2063}},"tokens_in":587,"tokens_out":2160,"duration_ms":14725,"temperature":1.0,"reasoning_tokens":2063,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:59:50.040403+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask human annotators to score the same CapQA test outputs for visual groundedness and question usefulness against the original images, then compare the SQ model with the base model; if humans do not find roughly the same 31% hallucination reduction, the central claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the GPT-4V annotations: all CapQA questions, answers, detailed descriptions, and summarized captions used for fine-tuning."},{"cited_title":"Fine-grained activities of people worldwide, 2022","cited_arxiv_id":null,"evidence_quote":"Provides the source video clips and 512 fine-grained activity labels from which the 1,000 CapQA images are extracted."},{"cited_title":"Visual instruction tuning.NeurlPS,","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-round conversation format, the 595k pretraining data, and the LLaVA benchmarks used for evaluation."}],"review_version":1}