{"id":"2c7f4874-49a9-44e5-a136-34588382b2c1","arxiv_id":"2412.05821","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A multimodal QA system that generates and iteratively corrects entailment trees, using a mixture-of-experts small model, reports first place on the WebQA leaderboard and strong results on MultimodalQA.","lead":"The authors built a question-answering system that turns images, tables, and text into a short list of facts, asks an AI to arrange those facts into a logical reasoning tree, and then trains a smaller model to answer questions from the tree while correcting the tree. On the WebQA leaderboard the system reports the highest answer accuracy, and it also does well on MultimodalQA, with the added benefit that each answer comes with a visible reasoning path.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"FRG supervision in Eq. 13 may come from the LLM-generated tree, not gold labels, so iterative feedback may lack an independent signal to correct leaf-selection errors, undermining the central correction claim.","rationale":"The reader's weakest assumption is that the QA decoder relies on the entailment tree as a sufficient information channel, and that tree errors could be unrecoverable. That is a valid concern about the architecture. My focus is on the mechanism designed to mitigate that risk: the FRG decoder and iterative feedback. The paper's Eq. 13 appears to train the FRG on LLM-generated tree indices, while later text claims gold-label supervision. This internal inconsistency is more load-bearing than the architectural bottleneck alone because it determines whether the correction loop actually has access to ground-truth evidence. If the FRG only imitates the LLM, the iterative feedback cannot fix leaf-selection errors, and the claimed advantage over prior work is not explained by the stated method. The empirical SOTA claim on WebQA is plausible and externally visible on the leaderboard, so I am not recommending rejection; however, the training-target ambiguity must be resolved before the central contribution can be accepted as described. The verdict remains CONDITIONAL (UNCHANGED), with the condition now including a precise description of FRG supervision and an ablation isolating the correction effect.","tokens_in":14961,"tokens_out":3263,"duration_ms":34116,"concrete_test":"Run an ablation comparing FRG training targets: (i) the LLM-generated tree's fact indices as stated in Eq. 13, and (ii) gold evidence labels from the dataset. Measure FRG retrieval recall and final QA accuracy on the WebQA dev set. If the two conditions yield identical results, the FRG provides no independent correction and the IFM's claimed benefit is vacuous. Additionally, inspect the actual training code or data pipeline to confirm which target is used, resolving the internal inconsistency between Eq. 13 and the case-study claim of supervision from dataset labels.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the method achieves SOTA by correcting LLM-generated entailment tree errors through joint learning and iterative feedback. The load-bearing mechanism is the FRG decoder's ability to retrieve facts that are better than the LLM's initial leaf selection. But §3.2.1, Eq. 13 defines the FRG training target as the 'fact index sequence extracted from the entailment tree generated by LLM.' This means the FRG is trained to imitate the LLM's leaf selection, not to correct it. The case study and quality analysis later claim that 'the small model can supervise the selection of evidence with detailed labels in the dataset' and thereby correct the LLM's leaf-selection errors. These two statements are mutually inconsistent. If Eq. 13 accurately describes the training, then FRG's retrieved facts are a copy of the LLM tree, and the iterative feedback mechanism cannot provide an independent correction signal: any initial leaf-selection error is passed forward as supervision. If instead the FRG uses gold evidence labels, Eq. 13 is mis-specified and the actual supervision source is undisclosed. Either way, the paper's explanation of how LLM errors are corrected—a key component of the claimed contribution—is not supported by the described training procedure. Since the QA decoder (Eq. 14) cross-attends only to the tree text and question, the final answer depends critically on the tree being faithful; if leaf errors are never independently corrected, the method's improvement over baselines may stem from other components (e.g., question decomposition, filtered image captions) rather than from iterative tree correction. This does not disprove the empirical leaderboard results, but it makes the stated mechanism unverifiable from the paper.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage method for multimodal multi-hop question answering (MMQA). In the first stage, GPT-3.5 decomposes the original question into sub-questions, builds a fact base from text, image, and table evidence via VQA and table linearization, and generates an entailment tree whose internal nodes are filled by the LLM. In the second stage, a T5-based multi-task mixture-of-experts model is trained jointly on fact retrieval generation (FRG) and question answering (QA), where the QA decoder is restricted to cross-attend only over the entailment tree description plus the question. An iterative feedback mechanism feeds the retrieved facts and predicted answer back to GPT-3.5 to regenerate the entailment tree for another round. The paper reports first place on the WebQA leaderboard, competitive results on MultimodalQA, an ablation study, and a human evaluation of reasoning-path quality.","tokens_in":15265,"tokens_out":2889,"duration_ms":28560,"significance":"If the central claims hold, the paper makes a useful contribution by introducing entailment tree generation into multimodal multi-hop QA and showing that interpretable reasoning structures can be coupled with a smaller trained model to improve both answer accuracy and explanation quality. The use of an external leaderboard (WebQA), a human evaluation of decomposed questions and reasoning paths (Section 4, Table 6), and ablations isolating the proposed modules (Table 5) are concrete strengths. However, the claimed mechanism for correcting LLM-generated entailment tree errors rests on an internal inconsistency in the description of the FRG supervision signal; until that is resolved, the central corrective-loop claim is not supported by the evidence as written.","major_comments":[{"comment":"Equation (13) defines the FRG training target as 'The fact index sequence extracted from the entailment tree generated by LLM.' This means the fact retrieval decoder is trained to imitate the LLM's leaf-node selection rather than to correct it. The case study in Section 4, however, states that 'the small model can supervise the selection of evidence with detailed labels in the dataset' and thereby correct the LLM's leaf-selection errors during iterative feedback. These two statements are mutually inconsistent. If Eq. (13) is accurate, the FRG decoder has no independent supervision signal, so any initial leaf-selection error in the LLM tree is passed forward through training, and the iterative feedback loop cannot provide the correction claimed in the contributions. If the FRG actually uses gold evidence labels, then Eq. (13) is mis-specified and the true supervision source is undisclosed. Either way, the load-bearing claim that errors from the LLM-generated entailment tree are corrected by the joint learning and feedback mechanism is not supported by the current description. The authors should clarify the exact supervision source for Eq. (13), and if the LLM tree is used, they should provide evidence that the FRG can still correct leaf errors (e.g., by comparing FRG outputs against gold evidence independently of the LLM tree).","section":"§3.2.1, Eq. (13) and §4 Case Study"},{"comment":"The QA decoder cross-attends only over the entailment tree description and the question, not over the fact base or retrieved evidence directly (Eq. 14). The design rationale is that the tree should be a sufficient and faithful channel for the answer. Given the inconsistency in the FRG supervision noted above, the paper does not currently demonstrate that the tree channel is faithful after iterative feedback. The case study in Figure 5 shows one qualitative example of correction, but no quantitative comparison is provided between (a) answers produced from tree-only cross-attention and (b) answers produced when the QA decoder also has access to the fact base or retrieved facts. Without such a comparison, the central architectural restriction is not empirically justified, and the failure mode of an unfixable leaf-selection error remains unaddressed.","section":"§3.2.1, Eq. (14) and §3.2.2"},{"comment":"The reported results are presented as single numbers from the WebQA leaderboard and the authors' own MultimodalQA runs, with no error bars, no multiple-seed statistics, and no released code. On MultimodalQA, the Single-Modal EM improvement over some baselines is small (69.8 vs. 69.7 for PERQA and Solar), while the claimed Multi-Modal improvement is large (64.7 vs. 55.5 for Solar). Since the core claim is state-of-the-art performance, the absence of statistical significance testing or repeated-run variance makes it hard to assess whether the gains are robust. The authors should at least report multiple runs with seeds or variance across validation folds, and ideally release code to enable independent verification.","section":"§4, Tables 2 and 4"}],"minor_comments":[{"comment":"Equation (2) includes 'E_k' inside the argmax over generated tokens, but E_k is a retrieved evidence item, not a decode variable; this is likely a typo that should be corrected to avoid confusion about what the LLM generates.","section":"§3.1.1, Eq. (2)"},{"comment":"The introductory example contains the typo 'churchill down truck' instead of 'Churchill Downs track', and Figure 2 has 'Gneration' in the box label 'Entailment Tree Gneration Module'. These typos should be fixed.","section":"Figure 1 and Figure 2"},{"comment":"The rows 'VitaminC', 'CMU ITL', 'HIT TMG', and 'SDU' are not cited or described in the text; the reader cannot tell which published systems or team entries these are. Please add references or a note on the leaderboard source.","section":"§4, Table 3"},{"comment":"The iterative feedback mechanism uses the validation set to decide whether to continue iterating. This should be described as a model-selection step, and the authors should clarify whether any validation-set accuracy was used in choosing the final test-set configuration, since repeated selection on the leaderboard can inflate reported numbers.","section":"§3.2.2"},{"comment":"The ablation in Table 5 shows that removing the Mixture-of-Experts module changes the overall EM by only 0.9 points (68.2 to 67.3) and leaves Single-Modal EM unchanged at 69.8. This weakens the claim that the MoE component is a key contributor; the authors should discuss this more carefully or provide additional analysis.","section":"§4, Ablation Study"}],"recommendation":"major_revision","confidential_remarks":"The core inconsistency between Eq. (13) and the case-study description is the main barrier to acceptance. If the authors can clarify the supervision source and provide quantitative evidence for the correction mechanism, the paper could become acceptable. The leaderboard results are promising but lack reproducibility details; since no code is released and no error bars are given, I would ask the editor to weigh whether the current evidence meets the standard for a top-tier venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this paper is worth reading but not as-is. It applies entailment tree generation to multimodal multi-hop QA for the first time, and reports first place on the WebQA leaderboard. That leaderboard claim is the main event. If it holds, it's a useful result for practitioners who want interpretable reasoning paths and less redundant evidence.\n\nWhat's genuinely new: the integration of entailment tree generation with a small multi-task MoE model and iterative LLM feedback. The question-decomposition-guided image captioning is also a sensible idea. The paper gives credit to the prior components and doesn't oversell the novelty as a new principle. The ablation study is helpful and shows each module contributes something.\n\nThe soft spot is real and load-bearing. Section 3.2.1's Eq. 13 says the FRG decoder is trained to retrieve the fact indices 'extracted from the entailment tree generated by LLM.' But the case study says the small model supervises evidence selection 'with detailed labels in the dataset.' Those are different supervision sources. If Eq. 13 is accurate, the small model is trained to imitate the LLM's leaf choices, so the iterative feedback loop cannot independently correct leaf-selection errors—it just repeats them. If the prose is accurate, Eq. 13 is mis-specified and the real training target is undisclosed. Either way the paper's central correction claim is not supported by the described procedure. That's not a minor typo; it's the mechanism the whole system hangs on.\n\nThere are smaller issues. No code or data released, no error bars, and the human evaluation is small (10 volunteers, 50 questions each). I also noticed the QA score in Tables 1 and 2 doesn't match the stated product of QA-FL and QA-Acc (e.g., 53.0 × 57.2 = 30.3, but they report 34.5). That's probably a harmless reporting slip, but it doesn't build trust.\n\nWho is this for? Anyone working on multimodal QA, interpretable reasoning, or retrieval-augmented generation. The idea is worth building on. But it needs a serious referee who will ask for code, clarification of the training target, and variance numbers. I'd send it to review, not desk reject, but the revision needs to be substantive.","headline":"A plausible SOTA claim on WebQA with a genuinely new application of entailment trees, but the paper's account of how errors are corrected is internally inconsistent and needs scrutiny.","tokens_in":15851,"tokens_out":3857,"would_cite":false,"duration_ms":35177,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes treating multimodal multi-hop QA as a joint entailment tree generation and question answering problem, with a multi-task mixture-of-experts model and iterative feedback, and reports state-of-the-art results on WebQA…","keywords":["multimodal multi-hop question answering","entailment tree","mixture-of-experts","iterative feedback","interpretable reasoning","question decomposition","WebQA","MultimodalQA"],"falsifier":"Take a set of questions where the initially generated entailment tree omits a fact that is present in the fact base and needed for the correct answer, run the full pipeline including iterative feedback, and compare accuracy on that subset against questions with complete initial trees. If accuracy on the incomplete-tree subset does not drop sharply, the QA decoder must be drawing information from somewhere besides the tree description, contradicting the paper's claimed information bottleneck.","tokens_in":14776,"feed_emoji":"🌳","tokens_out":12502,"duration_ms":99831,"temperature":0.7,"pith_summary":"This paper claims that multimodal multi-hop question answering is best framed as a joint entailment tree generation and question answering problem, rather than feeding all retrieved evidence to a language model as flat text. The authors build a two-stage pipeline: first a large language model decomposes the question, constructs a fact base from images, tables, and text, and generates an entailment tree whose leaves are facts and whose internal nodes are intermediate conclusions; then a smaller multi-task mixture-of-experts model jointly retrieves the correct leaf facts and answers the question while attending only to the tree's description. An iterative feedback loop feeds the small model's retrieved facts and answer back to the language model to regenerate a better tree. On the WebQA benchmark the method reports state-of-the-art results with first place on the official leaderboard, and on MultimodalQA it reports competitive results, with human evaluation rating the reasoning paths correct about 86% of the time. The authors argue this makes multimodal reasoning both more accurate and more inspectable.","feed_headline":"Entailment tree reasoning tops the WebQA leaderboard","feed_subtitle":"A compact reasoning tree filters redundant evidence and iteratively corrects itself, lifting answer accuracy.","key_machinery":"The carrying mechanism is the entailment tree, defined as a tuple $T = (h, L, E, S)$ with the question as hypothesis $h$, leaf nodes $L$ drawn from the fact base, internal nodes $E$ as intermediate conclusions supplied by the language model, and entailment steps $S$ linking children to parents. Around this object the paper builds a two-stage pipeline: in the initialization stage, a large language model decomposes the question into sub-questions, fills a fact base using a VQA model for images and linearized tables for tables, generates the tree structure with symbolic notation ('fact1 & fact2 -> int1'), and then refines intermediate nodes; in the optimization stage, a shared encoder encodes the tree description concatenated with the question, two gating networks with top-2 token-choice routing select among a shared expert and two task-specific experts, and two decoders perform fact retrieval generation and question answering. The load-bearing design choice is that the QA decoder attends only to the entailment tree description, so the tree must be sufficient for the answer; any failure is addressed by the iterative feedback mechanism, which replaces the fact indices with the retrieved facts, appends the predicted answer, and asks the language model to regenerate the tree.","core_discovery":"The paper's discovery is that the entailment tree, a structure borrowed from textual entailment reasoning, can serve as the information bottleneck that solves two problems at once in multimodal multi-hop QA: it filters the redundant evidence that comes from converting entire images and tables into text, and it makes the reasoning steps explicit so errors can be located and corrected. In the proposed formulation, the input to entailment tree generation is a corpus of premises (facts from a fact base) and a hypothesis (the original question); the output is a tree $T = (h, L, E, S)$ where leaf nodes $l_i \\in L$ are selected facts, internal nodes $e_i \\in E$ are intermediate conclusions generated by the language model, and the root is the hypothesis. Because the QA decoder is deliberately restricted to cross-attend only over the entailment tree's textual description plus the question, the tree is not just an explanation attached after the fact; it is the channel through which all answer-relevant information must pass. The paper argues that the small model's joint fact-retrieval and QA losses, combined with an iterative feedback loop that returns the predicted facts and answer to the language model, correct the leaf-selection and structural errors that the language model makes during initialization, and this is what yields the leaderboard results.","pith_inferences":["The paper's information-bottleneck design implies a sharper comparison than the one reported: holding the tree fixed, a QA decoder with access to the full fact base should outperform one restricted to the tree if raw facts carry information the tree discards, so such a comparison would directly test whether the tree is a necessary filter or merely a helpful one.","Because the paper notes that only about 1% of WebQA questions require three or more reasoning hops, the reported gains may be concentrated on simple two-hop cases; a stress test on a dataset with more genuinely multi-hop multimodal questions would reveal whether the entailment-tree machinery scales to the cases it is designed for.","The iterative feedback cycle between a small supervised model and an LLM has a natural transfer: the same correct-then-regenerate loop could be applied to chain-of-thought refinement, program synthesis, or tool-use planning, where an LLM's initial structural errors are corrected by a small model that sees ground-truth labels."],"forward_implications":["On the WebQA official test set, the method reports a QA score of 0.54 with QA-FL 0.68 and QA-Acc 0.73, surpassing the previous best result of 0.44, and it holds first place on the official leaderboard since April 10, 2024.","On MultimodalQA, the method reports EM 69.8 on single-modal questions and 64.7 on multi-modal questions, exceeding the previous best on both splits, with an overall EM of 68.2 and F1 of 70.9.","Ablation on MultimodalQA shows every component contributes: removing the fact-retrieval generation decoder drops overall EM from 68.2 to 65.2, removing entailment tree generation drops it to 65.3, and removing the iterative feedback mechanism drops it to 66.1.","Human evaluation on the WebQA dev set rates 88.6% of the decomposed questions correct and 85.8% of the entailment-tree reasoning paths correct, which the paper offers as evidence that the reasoning paths are usable explanations."],"supporting_citations":[{"why":"Defines the entailment tree formalism (hypothesis, leaves, intermediates, steps) and the dataset whose structure-generation accuracy this method extends and corrects.","marker":"[6]"},{"why":"Provides the WebQA dataset, its clue-retrieval and QA evaluation, and the leaderboard used to report first-place results.","marker":"[3]"},{"why":"Provides the MultimodalQA benchmark over text, tables, and images with EM and F1 metrics.","marker":"[22]"},{"why":"Supplies the unified text representation for converting tables and images and the evidence-retrieval approach that grounds the fact base construction.","marker":"[28]"},{"why":"Serves as the image-caption-then-LLM baseline that the proposed question-decomposition captioning is compared against.","marker":"[11]"},{"why":"Supplies the top-2 token-choice routing and training-stability practices used in the mixture-of-experts model.","marker":"[24]"},{"why":"Supplies the task-aware gating mechanism for sparse multi-task mixture-of-experts that shapes the two-gate design.","marker":"[9]"},{"why":"Supports the iterative retrieval-and-question-answering loop and the choice of two feedback rounds.","marker":"[13]"}],"fun_headline_variants":["Tree reasoning filters noise, iterates to WebQA top","Entailment trees prune evidence, revise answers, win WebQA","Iterative tree feedback lifts WebQA accuracy","Self-correcting entailment trees top WebQA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach assumes that the entailment tree, as initially generated by the language model and then patched by the feedback loop, is a faithful and complete channel for every fact needed to answer the question, because the QA model never sees the raw evidence and can only recover a missing fact if the fact-retrieval decoder selects it and the feedback loop repairs the tree.","fun_headline_variants_meta":{"raw":{"variants":["Tree reasoning filters noise, iterates to WebQA top","Entailment trees prune evidence, revise answers, win WebQA","Iterative tree feedback lifts WebQA accuracy","Self-correcting entailment trees top WebQA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000545,"raw_usage":{"total_tokens":2668,"prompt_tokens":1064,"completion_tokens":1604,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":680,"completion_tokens_details":{"reasoning_tokens":1539}},"tokens_in":680,"tokens_out":1604,"duration_ms":11887,"temperature":1.0,"reasoning_tokens":1539,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:18:45.414467+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of questions where the initially generated entailment tree omits a fact that is present in the fact base and needed for the correct answer, run the full pipeline including iterative feedback, and compare accuracy on that subset against questions with complete initial trees. If accuracy on the incomplete-tree subset does not drop sharply, the QA decoder must be drawing information from somewhere besides the tree description, contradicting the paper's claimed information bottleneck.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the entailment tree formalism (hypothesis, leaves, intermediates, steps) and the dataset whose structure-generation accuracy this method extends and corrects."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the WebQA dataset, its clue-retrieval and QA evaluation, and the leaderboard used to report first-place results."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the unified text representation for converting tables and images and the evidence-retrieval approach that grounds the fact base construction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the iterative retrieval-and-question-answering loop and the choice of two feedback rounds."}],"review_version":1}