{"id":"bfd777e7-5220-4cc5-b758-42101a9bd27b","arxiv_id":"1909.02218","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A tree-structured memory network that uses parse trees and distinguishes visual from verbal words improves video question answering accuracy over flat sequence attention baselines.","lead":"This paper introduces a video question answering model that reads questions as syntax trees rather than as flat word sequences, and it lets only 'visual' words pay attention to the video frames. The authors report large accuracy gains over older attention baselines on two video QA datasets, especially for long and complex questions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline numbers rest on self-harvested versions of YouTube-QA and TGIF-QA; without evidence matching official splits/answer vocabularies, the SOTA and superiority claims over E-SA/E-SS are not established.","rationale":"The paper's central empirical assertion is comparative: HTreeMN outperforms Simple/E-SS/E-SA, and the conclusion generalizes this to 'state-of-the-art results on video question answering.' The ablations are internally consistent in aggregate (TreeMN > E-SA; HTreeMN-noh > TreeMN; HTreeMN > HTreeMN-noh), which supports a relative claim about the proposed components when trained and evaluated on the authors' pipeline. What is not established is the transfer of those numbers to the published benchmarks. Section IV-A explicitly says the datasets are re-harvested following [41] because the original is not fully available. Thus the baselines in Tables III/IV were re-run on a custom environment; differences in question generation, split composition, video selection, and answer vocabulary (K=500/1000) could easily shift accuracy by more than the reported gaps (e.g., roughly 2-4 points over E-SA). The conclusion's state-of-the-art wording is additionally under-supported because no post-2017 methods are compared. Given the reader already judged this conditional, the same concern does not change the verdict, but it is the load-bearing spot to test.","tokens_in":14638,"tokens_out":4628,"duration_ms":49178,"concrete_test":"Obtain the official YouTube-QA and TGIF-QA test splits and answer vocabularies (e.g., from [41] or the public TGIF-QA benchmark), run the released E-SA/E-SS and HTreeMN implementations on those exact splits with the same evaluation script, and compare accuracies/WUPS. If HTreeMN does not beat E-SA/E-SS on the official splits by the reported margins, or if the harvested datasets' question distributions and splits are shown to differ materially from official ones, the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is in Section IV-A: the authors state that the original dataset from [41] is 'not totally available till now,' so the two datasets are 'harvested following the method in [41]' rather than taken as published benchmarks. Every headline comparison in Section IV-F (HTreeMN vs. Simple, E-SS, E-SA, TreeMN, HTreeMN-noh) and the state-of-the-art conclusion in Section V is computed on these author-created datasets, with author-selected train/validation/test splits and answer vocabularies K=500/1000. If the harvested question distributions, video-text pairings, splits, or answer spaces differ from the official benchmarks, the reported gains may reflect dataset construction rather than the model's attention mechanism. The paper provides no quantitative check that the harvested data matches [41] or official TGIF-QA/YouTube-QA releases. A secondary reproducibility inconsistency (word labeling via NLTK in Section III-C vs. concreteness threshold in Section IV-B.2) also makes exact replication harder, but the dataset-equivalence problem is the primary threat to the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HTreeMN, a heterogeneous tree-structured memory network for video question answering. The model encodes questions using syntax parse trees, distinguishes 'visual' from 'verbal' words, applies word-level attention only to visual leaves, and adds a hierarchical attention mechanism over intermediate tree nodes. The approach is evaluated on two self-harvested datasets, YouTube-QA and TGIF-QA, and the authors report that the full HTreeMN outperforms the E-SA, E-SS, and several ablated baselines, and conclude that it achieves state-of-the-art results on video question answering.","tokens_in":14900,"tokens_out":4969,"duration_ms":48397,"significance":"If the empirical results were validated on standard benchmarks, the tree-structured attention mechanism would be a clearly described and potentially useful alternative to chain-based word-level attention for video QA. The paper's strengths include explicit model equations (1)-(8), a clear ablation sequence (TreeMN to HTreeMN-noh to HTreeMN), and qualitative analysis of attention distributions. However, the central quantitative claim is not currently established because the experiments are conducted on author-harvested datasets rather than the official YouTube-QA/TGIF-QA benchmarks, and because the reported per-type gains are not accompanied by statistical significance testing.","major_comments":[{"comment":"The datasets are self-harvested following the method of [41], not the official YouTube-QA or TGIF-QA benchmarks. Because the baselines E-SA and E-SS were originally evaluated on the official datasets, any comparison on different question-generation templates, splits, or answer vocabularies is not a valid test of model superiority. The paper provides no evidence, such as overlap statistics, template distribution comparisons, or split alignment, that the harvested data are equivalent to [41]. Consequently, the global claims in Section IV-F ('our full HTreeMN model performs the best on both the two datasets') and Section V ('we achieve the state-of-the-art results') are not supported. The authors should either evaluate on the official datasets or provide a rigorous equivalence analysis and restrict all claims to 'on our harvested datasets.'","section":"Section IV-A, Tables I-IV"},{"comment":"The per-question-type results are computed on very small test sets: for example, YouTube-QA Location has only 88 test questions and Time has 69; TGIF-QA Location has 520 and Time has 791. On these subsets, accuracy differences between HTreeMN and the next-best model are often within the binomial standard error. For instance, on YouTube-QA Location, HTreeMN accuracy is 0.2159 while HTreeMN-noh achieves 0.2386, a difference of about 2 percentage points, which is within the roughly 4 percentage-point standard error for a sample of 88. No error bars, confidence intervals, or multiple-seed experiments are reported anywhere. The claim that the tree models perform better on complex questions requires statistical support.","section":"Section IV-F.2, Tables V-X"},{"comment":"The word labeling procedure is described inconsistently. Section III-C states that the labeling is performed using the NLTK tool depending on word properties (nouns, verbs, adjectives, etc.), while Section IV-B.2 states that words are classified by the concreteness ratings of [3] with a threshold of 0.5. These are different methods and will generate different visual/verbal partitions, so the exact HTreeMN model is not reproducible from the paper. Please specify the actual algorithm used and report the fraction of visual versus verbal words, since this choice is load-bearing for the heterogeneous-node ablation.","section":"Section III-C vs Section IV-B.2"},{"comment":"The conclusion's claim of state-of-the-art results is not established by the experiments. The only baselines are the three models from [41] plus the authors' own ablations; no comparison to other published video QA methods is provided. For the claim to stand, the evaluation must be on the official benchmarks used by prior work, or the claim must be restricted to the compared baselines.","section":"Section V"}],"minor_comments":[{"comment":"Equation (4) uses 'n∈Rz' for the bias term, which should be 'b∈Rz'.","section":"Equation (4)"},{"comment":"The abstract gives the code URL as https://github.com/ZJULearning/TreeAttention while Section IV-A gives https://github.com/xuehy/TreeAttention; please unify the two URLs.","section":"Abstract and Section IV-A"},{"comment":"The phrase 'For an verbal node' should be 'For a verbal node.'","section":"Section III-C"},{"comment":"The sentence 'The size of the training set is almost equal to the size of both the validation and the testing set' is ambiguous; the training set is roughly equal to validation+test combined, not to each individually.","section":"Section IV-A"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central claim depends on datasets that are not the public benchmarks, which is a serious scope issue for a journal like TIP. If the official datasets, or the released data from [41], are available, I would require the authors to rerun the experiments on them; otherwise the paper should be repositioned as a study on the authors' collected datasets and the 'state-of-the-art' claim removed. The inconsistency between NLTK-based and concreteness-based word labeling is easily fixable but indicates a need for a careful reproducibility pass."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read Xue et al. on tree attention for video QA. Short version: the model is a reasonable, well-specified extension of TreeLSTM and E-SA, but the empirical claims overreach because everything is measured on self-harvested versions of YouTube-QA and TGIF-QA that are never checked against the official datasets.\n\nWhat's new: The heterogeneous leaf nodes (visual words attend, verbal words don't) and the recursive rule for deciding which intermediate nodes attend are not in prior work. The paper does a clean job specifying the model equations and running ablations (TreeMN, HTreeMN-noh, full HTreeMN) that are internally consistent. The qualitative attention comparison for verbal words is a nice touch, and the framing—tree structure helps long compositional questions—is plausible.\n\nSoft spots: The dataset problem is real and load-bearing. Section IV-A says the original dataset is 'not totally available till now' and that the two datasets are 'harvested following the method in [41]'. That means every comparison to E-SA and E-SS is on the authors' re-created splits, with their own answer vocabularies (K=500/1000) and their own question types. There is no check that these match the official YouTube-QA/TGIF-QA releases. If question distributions or splits differ, the reported gains over E-SA could be artifacts of dataset construction. The per-type test sets are tiny (Location test set in YouTube-QA has 88 questions; Time has 69), and there are no error bars or repeated runs, so the large per-type gains in Number and Time are not stable. The conclusion's 'state-of-the-art' line is also unsupported because no contemporary baselines are included. A minor reproducibility issue: the code link in the abstract and the one in Section IV-A point to different repos, and the visual/verbal labeling is described inconsistently (NLTK in one place, concreteness threshold in another).\n\nWho it's for: people working on structured attention for language in multimodal tasks. The model idea and ablations are worth engaging, but the empirical claims need discipline.\n\nRecommendation: worth sending to serious peer review, with major revision. Ask for dataset-equivalence evidence (at least matching official splits/answer vocabulary), error bars, and comparisons on a current benchmark before the SOTA claims stand.\n\nBest,\n[Your name]","headline":"The tree-attention model is clearly specified and the ablations are internally consistent, but the headline numbers rest on self-harvested datasets that are never checked against the official benchmarks, so the SOTA claim overreaches.","tokens_in":15374,"tokens_out":2429,"would_cite":false,"duration_ms":25631,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that video question answering improves when attention follows the syntax parse tree of the question, with the largest gains on long and complex questions.","keywords":["video question answering","tree-structured memory network","attention mechanism","syntax parse tree","temporal attention","visual and verbal words","hierarchical attention","long question reasoning"],"falsifier":"Run the three baselines and HTreeMN on the official published train/validation/test splits of YouTube-QA and TGIF-QA with identical visual features and answer vocabularies, and compare accuracy within each question-length bin; if the tree model does not beat the flat attention baselines on questions longer than fifteen words, the paper's central claim fails.","tokens_in":14453,"feed_emoji":"🎬","tokens_out":7832,"duration_ms":71414,"temperature":0.7,"pith_summary":"Video question answering needs to find the frames that answer a question, and most attention models encode the question as a flat sequence of words. This paper tries to establish that the attention computation should follow the syntax parse tree of the question instead: visual words such as nouns and most verbs attend to the video, verbal words such as linking verbs and conjunctions act only as sentence structure, and the two kinds of representation are merged recursively from the leaves to the root. The proposed model, the heterogeneous tree-structured memory network (HTreeMN), is reported to outperform flat attention baselines on two video-QA datasets, with the largest gains on long questions and on number and time questions. If the claim holds, it would show that syntactic structure is a practical asset for grounding language in video, not just for text-only NLP.","feed_headline":"Tree-structured attention wins on video question answering","feed_subtitle":"Routing attention to visual words and merging by syntax tree lifts accuracy on long questions across two datasets.","key_machinery":"The central object is the parse tree of the question sentence, with words at the leaves and phrase nodes (such as NP and VP) inside. A leaf is visual if its word has a high concreteness rating; visual leaves query the video frames with a temporal attention module and add the attended feature to the word embedding, while verbal leaves are transformed linearly without attention. All parent nodes sum their children's outputs through type-dependent linear layers, so information is composed from the leaves to the root, whose state is fed to a softmax classifier over candidate answers. A recursive rule decides which internal nodes attend: a node is verbal exactly when every child is verbal, which makes the attention hierarchy follow the sentence structure. This tree composition is what lets long questions keep their semantic structure instead of being collapsed into one linear embedding.","core_discovery":"The paper's central claim is that the syntactic parse tree of a question is the right skeleton for video attention. HTreeMN labels each word as visual or verbal using concreteness ratings, computes temporal attention over video frames only for the visual words, and then propagates joint word-video representations bottom-up through the tree, with separate linear transformations for the two word types. Intermediate phrase nodes are also classified recursively as verbal only when all of their children are verbal, and visual intermediate nodes receive their own attention, yielding a hierarchical attention mechanism. The full model is reported to outperform the simple attention baseline, the E-SS sequence-to-sequence model, and the E-SA word-level attention model on both YouTube-QA and TGIF-QA in accuracy, WUPS@0.0, and WUPS@0.9, with the clearest margin on long and complex questions. The conclusion states that these results are state-of-the-art for video question answering.","pith_inferences":["Because the parse tree comes from an off-the-shelf parser, the same visual/verbal split and recursive attention could transfer to other video-language tasks with long compositional queries, such as temporal grounding from natural language; the paper does not test this extension.","The reported gap between HTreeMN and the non-hierarchical HTreeMN-noh is small, suggesting that the hierarchical attention on intermediate nodes adds little once the leaf nodes are heterogeneous; a cheaper model might be enough, though the authors still report the full model.","The state-of-the-art claim is conditional on the harvested datasets matching the originals; re-running on official splits would settle whether the gains generalize outside the authors' own data.","The visual/verbal split could be learned end-to-end instead of fixed by a concreteness threshold, which might adapt better to specialized video domains."],"forward_implications":["Video question answering systems should encode questions as parse trees rather than flat LSTM chains, particularly when questions are long and compositional.","Skipping attention for verbal words removes a source of noise from the attended video features, since verbal words lack direct visual correspondence.","The largest practical gains should appear on number and time questions, where the reported accuracy jumps by large margins over the flat baselines.","The tree-structured models are claimed to make better use of small training data, because semantic structure compensates for fewer examples.","The flat baselines should be expected to degrade as question length grows, while the tree model stays stable."],"supporting_citations":[{"why":"defines the video-QA task, the E-SA and E-SS baselines, and the harvesting method used to build both datasets.","marker":"[41]"},{"why":"generates the syntax parse trees that the entire tree-structured model is built on.","marker":"[16]"},{"why":"introduces tree-structured LSTM composition, which the memory network adapts for recursive merging.","marker":"[26]"},{"why":"is the attention model from which the E-SA baseline is extended.","marker":"[36]"},{"why":"motivates the hierarchical attention mechanism through stacked attention reasoning.","marker":"[35]"},{"why":"supplies the pre-trained word vectors used to embed question tokens.","marker":"[22]"},{"why":"provides the convolutional network visual features extracted from video frames.","marker":"[24]"},{"why":"provides the concreteness ratings used to split words into visual and verbal categories.","marker":"[3]"}],"fun_headline_variants":["Syntax tree attention lifts video question answering accuracy","Tree-structured attention: a better path for video QA","HTreeMN: syntax-guided attention for video QA","Route attention via question syntax for sharper video QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the two datasets used in the experiments, re-harvested by the authors following the method of reference [41], are equivalent enough to the original published benchmarks that the gains over E-SA and E-SS reflect model quality rather than differences in question generation or data splits.","fun_headline_variants_meta":{"raw":{"variants":["Syntax tree attention lifts video question answering accuracy","Tree-structured attention: a better path for video QA","HTreeMN: syntax-guided attention for video QA","Route attention via question syntax for sharper video QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000818,"raw_usage":{"total_tokens":3604,"prompt_tokens":990,"completion_tokens":2614,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":2554}},"tokens_in":606,"tokens_out":2614,"duration_ms":19306,"temperature":1.0,"reasoning_tokens":2554,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:56:08.725437+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the three baselines and HTreeMN on the official published train/validation/test splits of YouTube-QA and TGIF-QA with identical visual features and answer vocabularies, and compare accuracy within each question-length bin; if the tree model does not beat the flat attention baselines on questions longer than fifteen words, the paper's central claim fails.","supporting_citations":[{"cited_title":"Pennington, R","cited_arxiv_id":null,"evidence_quote":"supplies the pre-trained word vectors used to embed question tokens."},{"cited_title":"Brysbaert, A","cited_arxiv_id":null,"evidence_quote":"provides the concreteness ratings used to split words into visual and verbal categories."},{"cited_title":"Zeng, T.-H","cited_arxiv_id":null,"evidence_quote":"defines the video-QA task, the E-SA and E-SS baselines, and the harvesting method used to build both datasets."},{"cited_title":"Klein and C","cited_arxiv_id":null,"evidence_quote":"generates the syntax parse trees that the entire tree-structured model is built on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"is the attention model from which the E-SA baseline is extended."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"motivates the hierarchical attention mechanism through stacked attention reasoning."}],"review_version":1}