{"id":"bb2b179d-02f7-460b-9100-1a0d6bad7227","arxiv_id":"2507.22369","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Four open-source VQA models reach moderate accuracy on a new classroom video dataset, with yes/no questions easiest and counting/reasoning hardest.","lead":"The paper introduces a small video question answering dataset for classroom behavior and tests four open-source VQA models on it. The models handle simple yes/no questions well but stumble on counting and reasoning tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Load-bearing concern: Qwen3 and NVILA results come from public web demos, not local inference, so the 'all four models promising' conclusion is unsupported.","rationale":"The reader's REJECT verdict at moderate confidence is appropriate, and the weakest assumption is correctly identified. The feasibility claim is plausible in principle—open-source VQA models may well handle simple classroom questions—but the evidence in the paper does not establish it. The most load-bearing flaw is the uncontrolled evaluation of Qwen3 and NVILA via public web interfaces, because it breaks the link between the reported numbers and the named models. This is not merely a reproducibility nicety: if the web backend differs from the stated checkpoint, then the paper's own tables are measuring different systems. The paper's future-work sentence about adding test samples (§5.2) further suggests the current dataset is not yet a benchmark with a defined test split; combined with the absence of N values, one cannot tell whether 80% accuracy reflects 4/5 questions or 40/50. A local rerun with released data and item counts would settle the concern. I do not see an internal inconsistency in the core pipeline; the problem is evidential support, which the paper does not currently provide.","tokens_in":9339,"tokens_out":5917,"duration_ms":63354,"concrete_test":"Run the exact evaluation set on the released/open-weight checkpoints locally—e.g., NVILA-8B and the Qwen3/VL checkpoint named in §3.3.1—with the same prompts, a fixed random seed or temperature setting, and the paper's own accuracy/ROUGE-L/BERTScore scripts; report per-model item counts and 95% CIs. If either web-demo result differs by more than ~5 accuracy points or ~0.05 metric points from the local run, or if the evaluation set and item counts are not released, the 'all four models' claim is not supported by the paper as written.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that all four open-source VQA models achieve promising performance (Abstract)—requires Tables 1–3 to be faithful measurements of those models. Section 3.3.1 states that VideoLLaMA2/VideoLLaMA3 were run from Hugging Face, but Qwen3 and NVILA were 'tested via the official websites' (chat.qwen.ai and vila.hanlab.ai). Public web demos are not controlled inference: they may serve different checkpoints than the ones named (e.g., Qwen3-235B-A22B vs. the current chat backend), apply hidden prompt formatting or sampling, impose rate limits, and change across time, so the reported accuracies and metric scores cannot be attributed to the named open-source models. Since Qwen3 and NVILA are the strongest open-ended scorers in Tables 2–3, the 'all four models' conclusion depends critically on these two measurements. The weakness is compounded by the absence of dataset release, item-level counts, and a clear test split—§5.2 says test samples 'will be added in future versions'—so the claimed performance levels cannot be independently checked.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BAV-Classroom-VQA, a new dataset of classroom video clips annotated with close-ended and open-ended visual questions, and benchmarks four open-source VQA models (VideoLLaMA2, VideoLLaMA3, Qwen3, and NVILA) on this dataset. The authors report accuracy for Yes/No and Single-Choice questions and BERTScore/ROUGE-L for open-ended questions, and they discuss per-model strengths and weaknesses. The central claim, stated in the Abstract and Section 5.1, is that all four models achieve promising performance on behavior-related visual questions, suggesting feasibility of VQA-based classroom analytics. The paper also describes a data collection and annotation methodology for classroom videos from the Banking Academy of Vietnam.","tokens_in":9565,"tokens_out":4292,"duration_ms":48572,"significance":"If the central claim were fully supported, this paper would provide useful initial evidence that zero-shot VQA models can assist in classroom behavior monitoring, and the BAV-Classroom-VQA dataset could be a valuable resource for a domain that currently lacks a dedicated VQA benchmark. The authors cover a real-world application, include multiple question categories (factual, spatial, counting, reasoning, etc.), and report both a lexical metric (ROUGE-L) and a semantic metric (BERTScore). These are commendable choices for an exploratory study. However, the significance is substantially weakened by the evaluation protocol: two of the four models were tested through public web chat interfaces rather than controlled local inference, the dataset is not released and test samples are promised only in future versions, and no sample sizes or statistical tests are provided. The paper does not ship code or a reproducible evaluation harness, so the quantitative results are not independently verifiable in their current form.","major_comments":[{"comment":"Qwen3 and NVILA were evaluated through public web interfaces (chat.qwen.ai and vila.hanlab.ai), not through controlled local inference. Public demos can serve different checkpoints than the named ones, apply hidden prompt formatting or sampling parameters, and change over time, so the reported accuracies and scores for these two models cannot be reliably attributed to the open-source Qwen3-235B-A22B and NVILA-8B models. Because Qwen3 and NVILA are the strongest open-ended scorers in Table 2 and contribute heavily to the Abstract's claim that 'all four models achieve promising performance,' this is a load-bearing issue. The authors should rerun both models locally with fixed checkpoints and report the exact inference configuration, including prompt template, sampling settings, and model version.","section":"Section 3.3.1; Tables 1–3"},{"comment":"The BAV-Classroom-VQA dataset is neither released nor fully specified. Section 5.2 states that test samples 'will be added in future versions of the dataset,' implying that the current version has no test split or that all samples were used for evaluation. The paper also does not report the number of video clips, the number of questions per type, or the number of questions per model. Without these counts and a public release, the percentages in Tables 1–3 cannot be interpreted or independently checked. The authors should provide detailed dataset statistics, state whether any training or validation split is used, and describe a release plan.","section":"Section 3.2; Section 5.2"},{"comment":"No sample sizes, confidence intervals, error bars, or significance tests are reported. Differences such as 80% versus 75% Yes/No accuracy in Table 1, or a 0.03 difference in BERTScore in Table 2, are treated as meaningful, but with unknown N they may be well within chance variation. The authors should report the number of questions per category and per model and apply appropriate statistical tests or at least bootstrap confidence intervals to support the qualitative ranking of models.","section":"Section 4; Tables 1–3"}],"minor_comments":[{"comment":"The model names are inconsistent: the Abstract says 'LLaMA2, LLaMA3, QWEN3, and NVILA,' while Section 3.3.1 lists 'VideoLLaMA2, VideoLLaMA3, NVILA-8B, and Qwen3-235B-A22B'; the Introduction also refers to 'LLaMA2, LLaMA3.' Please use the full, consistent model names throughout.","section":"Abstract; Section 3.3.1"},{"comment":"There is an apparent contradiction: the text first lists 'Single-Choice, Multiple-Choice, Factual, Reasoning, Spatial' as included question types, then says the authors excluded 'Attribute Classification, Comparative, and Multiple-Choice' questions. Please clarify whether multiple-choice questions are included or excluded, and distinguish multiple-choice from single-choice.","section":"Section 3.2"},{"comment":"The sentence 'the Factual and Object Detection questions yielded the highest scores, ranging from 0.7 to 0.9' is inconsistent with Table 3, where Factual BERTScore values are 0.93–0.95; please correct the range or the description.","section":"Section 4.2.1"},{"comment":"The claim that VideoLLaMA3 'showed lower performance in Counting (0.7265, 0.2000) compared to other models' is imprecise because NVILA and QWEN3 have the same ROUGE-L value of 0.2000 for Counting; please rephrase to state which models it ties with or outperforms.","section":"Section 4.2.2"},{"comment":"Reference [19] appears to be a garbled citation: 'Dgm: A deep learning algorithm for solving partial differential equations' is not a VQA dataset and seems unrelated to the surrounding sentence about VQA datasets. Please verify and replace this reference.","section":"References"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe novel part here is the BAV-Classroom-VQA dataset—classroom video clips with behavior-focused QA pairs—and the idea of testing open-source VQA models on it. That's a legitimate niche for educational analytics, and the authors did the obvious right things: privacy masking, a question taxonomy borrowed from the survey literature, and both BERTScore and ROUGE-L for open-ended answers. The per-category results in Table 3 are a reasonable first cut.\n\nBut the central claim—'all four models achieve promising performance'—isn't supported by the evidence. The biggest problem is that Qwen3 and NVILA were tested through public web chat interfaces, not local inference. Those endpoints can serve different checkpoints, apply hidden prompt formatting or sampling, and change over time, so the numbers for the two strongest open-ended scorers can't be attributed to the named model versions. On top of that, the dataset is not released, and Section 5.2 admits that test samples 'will be added in future versions,' so nothing can be independently rechecked. There are also no sample sizes, error bars, or significance tests. And a smaller internal inconsistency: the text says multiple-choice questions were excluded, yet the earlier list of question types includes them.\n\nThe flaws aren't evenly spread. The web-demo issue is load-bearing; without local inference, the accuracy and BERTScore numbers for Qwen3 and NVILA are effectively uninterpretable. The other shortcomings just deepen the problem. The paper is an honest exploratory workshop effort, not hostile to scrutiny—the 'initial experimental results' phrasing and the future-work section at least concede the dataset is incomplete. But the conclusion overstates what the data can show, especially when NVILA's 50% on Yes/No is already at chance.\n\nWho gets value from this? Someone scoping whether VQA for classroom monitoring is worth pursuing, and who wants a checklist of what a proper evaluation would need. The quantitative benchmark table itself should not be cited.\n\nMy recommendation: don't reject out of hand—send it to peer review, because the dataset idea and application question are real, and the flaws are fixable. A serious referee should ask for controlled local inference, a released dataset with a proper split, and conclusions that match the evidence.","headline":"A sensible new application and dataset idea, but the web-demo evaluation and unreleased data make the 'promising performance' claim unverifiable.","tokens_in":10072,"tokens_out":3067,"would_cite":false,"duration_ms":33457,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Off-the-shelf VQA models hit 80% on classroom yes/no questions","keywords":["Visual Question Answering","Classroom Activity Monitoring","VideoQA","Multimodal Learning","Large Language Models","Dataset Construction","Behavior Recognition","Benchmarking"],"falsifier":"Run the full BAV-Classroom-VQA question set through locally hosted QWEN3 and NVILA checkpoints with deterministic decoding and compare the scores to the web-interface results; a substantial drop in accuracy would show that the paper's 'promising performance' reflects interface behavior rather than model capability.","tokens_in":9173,"feed_emoji":"🎓","tokens_out":7305,"duration_ms":70158,"temperature":0.7,"pith_summary":"This paper tries to establish that off-the-shelf, open-source visual question answering (VQA) models can already support basic classroom behavior monitoring without any classroom-specific fine-tuning. To test this, the authors build BAV-Classroom-VQA, a new dataset of short clips from real university lectures annotated with closed-ended and open-ended questions across five categories. They benchmark four open-source models and report yes/no accuracies up to 80%, single-choice accuracies between roughly 48% and 62%, and open-ended BERTScore values around 0.69–0.75. The paper's central claim is that these numbers show VQA is a feasible first step toward semi-automated classroom analytics and intervention systems.","feed_headline":"Off-the-shelf VQA models hit 80% on classroom yes/no questions","feed_subtitle":"Four open-source VQA models show promise answering classroom activity questions on a new benchmark.","key_machinery":"The central object is BAV-Classroom-VQA, a new dataset of 20–30 second clips from a university classroom video repository, each clip paired with questions in five categories (single-choice, multiple-choice, factual, reasoning, spatial) and two answer formats (closed-ended and open-ended). The dataset carries the argument: the paper's feasibility conclusion is derived entirely from how four open-source VQA models score on it. The supporting mechanism is the two-metric evaluation protocol—accuracy for closed-ended questions, and BERTScore plus ROUGE-L for open-ended questions—which separates a model's ability to pick a correct option from its ability to phrase a semantically acceptable free-form answer.","core_discovery":"The paper claims that current open-source VQA models, used as-is, perform at a 'promising' level on behavior-related visual questions about real classroom video. The evidence is the new BAV-Classroom-VQA dataset and an evaluation protocol that treats closed-ended and open-ended questions separately. For yes/no questions the best models reach 80% accuracy, single-choice questions score between 47.6% and 61.9%, and open-ended answers achieve BERTScore of 0.69–0.75 with ROUGE-L of 0.31–0.41. The authors read the strong factual and object-detection scores alongside the weak action-recognition and reasoning scores as showing that the technology can handle quick recognition checks but still struggles with higher-order interpretation of classroom events.","pith_inferences":["A controlled re-run of QWEN3 and NVILA on local checkpoints, rather than web chat interfaces, is needed before their reported accuracies can be taken as measures of model capability; the paper's own evaluation setup leaves that open.","If the 80% yes/no accuracy survives controlled inference, then a small amount of classroom-specific fine-tuning could plausibly push these models across a practical deployment threshold.","The paper does not report inter-annotator agreement on the gold answers, so the true ceiling of its accuracy numbers is unknown; measuring agreement would clarify how much of the remaining error is model error versus label noise.","The question taxonomy aligns with standard classroom observation coding schemes, so the same dataset could double as a testbed for comparing VQA models against human observer agreement."],"forward_implications":["Existing open-source VQA models could be deployed for quick classroom checks such as counting students or detecting object use, without custom training.","Yes/no accuracy of up to 80% suggests binary monitoring questions are close to being practical, so such questions could serve as a first-pass filter that flags clips for human review.","The low scores on action recognition and reasoning mean automated interpretation of complex interactions is not yet reliable.","Because the videos were privacy-masked and access-controlled, the approach shows that classroom analytics need not expose personally identifiable information.","The BAV-Classroom-VQA dataset provides a reusable benchmark for future classroom VQA research and for domain-specific fine-tuning experiments."],"supporting_citations":[{"why":"Supplies the VideoLLaMA2 model evaluated in the experiments and its prior benchmark results.","marker":"[14]"},{"why":"Supplies the VideoLLaMA3 model evaluated in the experiments.","marker":"[15]"},{"why":"Supplies the NVILA model and its reported accuracy on video QA benchmarks.","marker":"[16]"},{"why":"Supplies the Qwen3 model and its claimed competitiveness with leading models.","marker":"[17]"},{"why":"Provides the factoid/inference question taxonomy used to structure the dataset's questions.","marker":"[24]"},{"why":"Provides the closed-ended vs open-ended answer format classification and the accuracy metric methodology.","marker":"[25]"},{"why":"Defines the ROUGE-L metric used to evaluate open-ended answers.","marker":"[26]"},{"why":"Defines the BERTScore metric used to evaluate semantic similarity of open-ended answers.","marker":"[27]"}],"fun_headline_variants":["Open-source VQA models hit 80% on classroom yes/no, but struggle on actions","Classroom VQA benchmark: models ace yes/no, falter on reasoning","New class dataset shows VQA limits: strong on facts, weak on actions","Classroom VQA: yes/no 80%, open-ended barely passable","VQA in classrooms: simple questions answered, reasoning still lags"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation of QWEN3 and NVILA through public web chat interfaces assumes those interfaces behave like controlled local model inference, and if they do not, the reported accuracies for these two models are not reliable measures of their capability.","fun_headline_variants_meta":{"raw":{"variants":["Open-source VQA models hit 80% on classroom yes/no, but struggle on actions","Classroom VQA benchmark: models ace yes/no, falter on reasoning","New class dataset shows VQA limits: strong on facts, weak on actions","Classroom VQA: yes/no 80%, open-ended barely passable","VQA in classrooms: simple questions answered, reasoning still lags"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000804,"raw_usage":{"total_tokens":3487,"prompt_tokens":854,"completion_tokens":2633,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":470,"completion_tokens_details":{"reasoning_tokens":2531}},"tokens_in":470,"tokens_out":2633,"duration_ms":18987,"temperature":1.0,"reasoning_tokens":2531,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:44:58.546452+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full BAV-Classroom-VQA question set through locally hosted QWEN3 and NVILA checkpoints with deterministic decoding and compare the scores to the web-interface results; a substantial drop in accuracy would show that the paper's 'promising performance' reflects interface behavior rather than model capability.","supporting_citations":[{"cited_title":"Visual question answering: from early developments to recent advances – a survey,","cited_arxiv_id":null,"evidence_quote":"Provides the closed-ended vs open-ended answer format classification and the accuracy metric methodology."},{"cited_title":"ROUGE: A package for automatic evaluation of summaries,","cited_arxiv_id":null,"evidence_quote":"Defines the ROUGE-L metric used to evaluate open-ended answers."},{"cited_title":"A detailed comparative analysis of automatic neural metrics for machine translation: Bleurt & bertscore,","cited_arxiv_id":null,"evidence_quote":"Defines the BERTScore metric used to evaluate semantic similarity of open-ended answers."}],"review_version":1}