{"id":"f0d889b9-5fae-4d1b-b660-884444160ceb","arxiv_id":"2412.08378","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new hybrid visual encoder with multi-layer ConvNeXt-ViT fusion improves high-resolution image understanding in vision-language models.","lead":"This paper introduces FILA, a vision-language model that combines a CLIP-ViT encoder with a ConvNeXt high-resolution branch through a deep fusion module to reduce errors from cropping high-resolution images. The authors report state-of-the-art results on nine of ten multimodal benchmarks, including large gains on text-heavy document tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central SOTA claim is not yet controlled for training data: Table 1 compares against published baselines with different data mixtures, and the abstract's 9.6% TextVQA gain is not reproduced by Table 1's 4.9-point gap.","rationale":"The reader's weakest assumption already identifies the training-data confound, and my analysis agrees: the central claim is an empirical superiority claim, yet the evidence for same-setting comparison is qualitative only. The internal ablations are honest and controlled, but they compare variants of FILA rather than a same-data external baseline, so they cannot separate architectural gains from data-mixture gains. The arithmetic mismatch on TextVQA (abstract 9.6% vs. Table 1 4.9 points) further weakens the headline claim and supports the need for controlled numbers. I found no stronger internal inconsistency than this; the architecture is well specified and the fusion mechanism is plausible. Keeping the conditional verdict is appropriate pending release of code/checkpoints or a controlled baseline comparison. If the controlled experiment shows the no-CVFM baseline matches FILA, the claim should be rejected or severely narrowed; if FILA retains the margin, the architecture claim is confirmed.","tokens_in":917,"tokens_out":1134,"duration_ms":61901,"concrete_test":"Run one controlled experiment: using FILA's exact data recipe, hyperparameters, and LLM, train a baseline that replaces the Hybrid Encoder with LLaVA-NeXT's CLIP-ViT encoder (or disables CVFM by keeping the tanh gate at zero and omitting ConvNeXt features), then evaluate on TextVQA, DocVQA, and the other eight benchmarks. If this no-CVFM baseline closes the reported 4.9-point TextVQA gap or matches the DocVQA margin, the claimed gains are due to training data rather than architecture. Also verify whether Table 1's 76.5 vs. 71.6 is ever reproducible as a 9.6% improvement; if not, the abstract requires correction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"For the central claim to hold, the 9/10 benchmark wins and the reported 9.6% TextVQA and 6.9% DocVQA margins must be caused by the Hybrid Encoder/CVFM rather than by differences in training data. Table 1 does not establish this: the baselines are published numbers from heterogeneous training setups, and the only explicitly same-data comparison is qualitative (Figure 4), which reports no score. The text claims MiniGemini-HD's training data are 'almost identical' to FILA's, but no data recipe is given for that baseline, and LLaVA-NeXT was not trained on the OCR-heavy mixture in Section 4.1 that includes 28K DocVQA/ChartQA/DVQA/AI2D pairs. Since MLLM benchmark scores are highly sensitive to instruction data, the reported advantages could be data-driven rather than architectural. Additionally, the 9.6% claim is arithmetically inconsistent with Table 1: HyViLM 76.5 vs. LLaVA-NeXT 71.6 is a 4.9-point gap (about 6.8% relative), not 9.6%; the 9.6% relative value matches 76.5 vs. 69.7 HiRes-LLaVA, but the text names LLaVA-NeXT as the comparison. The internal ablations in Tables 2-4 do support CVFM over other fusion variants, but they do not include a same-data external baseline, so the headline SOTA margins remain unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FILA/HyViLM, a high-resolution multimodal large language model that combines a CLIP-ViT encoder with a ConvNeXt auxiliary branch through a new interaction module (CVFM). The Hybrid Encoder interleaves multi-level ConvNeXt features with CLIP-ViT hidden states so that each dynamically cropped sub-image is encoded with access to global context, addressing the semantic fragmentation caused by resolution-adaptive cropping. The model is trained in two stages and evaluated on ten benchmarks, with the paper claiming state-of-the-art results on nine of them, including large reported gains on TextVQA and DocVQA. The main technical novelty is the staged deep fusion between the two visual encoders, and the internal ablations show consistent improvements over last-layer interaction and other fusion variants.","tokens_in":14722,"tokens_out":11022,"duration_ms":100532,"significance":"If the reported gains are caused by the architecture, this is a valuable contribution: dynamic cropping is widely used in high-resolution MLLMs, and the paper identifies a real failure mode (fragmentation at crop boundaries) with concrete qualitative examples. The design of CVFM is clean, the gating initialization at zero is a sensible stability choice, and the internal ablations in Tables 2-4 consistently favor the Hybrid Encoder over the examined alternatives. However, the central 'same setting' claim is not demonstrated: the main table compares against published baselines with heterogeneous training data, the architectural choices are tuned on benchmarks that later appear in the main results, and the headline percentages are not stated in consistent units. The paper is therefore a promising architecture study whose headline empirical claims need stronger controls.","major_comments":[{"comment":"The abstract and Section 4.3 claim that HyViLM outperforms state-of-the-art MLLMs 'under the same setting,' but the main evidence, Table 1, compares against published numbers from models trained on different data mixtures and with different LLMs and resolutions. The only explicitly same-data comparison is the qualitative Figure 4, which reports no scores. Section 4.1 shows that FILA's instruction mixture includes 28K DocVQA/ChartQA/DVQA/AI2D pairs, and MLLM benchmark numbers are known to be sensitive to such instruction data, so the Table 1 margins could be data-driven rather than architectural. The sentence claiming that MiniGemini-HD's training data are 'almost identical' to FILA's is not backed by a data recipe for MiniGemini-HD. To support the central claim that the Hybrid Encoder and CVFM cause the reported nine-of-ten wins, the paper needs a same-data external baseline (e.g., LLaVA-NeXT or MiniGemini-HD trained on FILA's exact mixture) or a detailed per-dataset comparison of training mixtures.","section":"§4.3, Table 1; abstract"},{"comment":"The design choices that define the method—number of ViT interaction layers (4), channel-concatenation fusion, and interpolation resizing—are selected by ablations run on TextVQA, InfoVQA, and MME, and the same three benchmarks appear in the main result table, Table 1. This means the headline architecture is fitted to the evaluation benchmarks, so the Table 1 results do not provide an unbiased estimate of the architecture's advantage. The paper should select hyperparameters on a held-out split, or report results for all ablated configurations on all ten benchmarks, and should include variance over at least several seeds; the reported advantages are often small (e.g., MME 1618/388 vs. 1604/368 and MMMU 41.8 vs. 41.7), and without error bars it is difficult to tell whether they are meaningful.","section":"§4.4, Tables 2–4"},{"comment":"The headline numbers are arithmetically inconsistent in units. Under the natural column alignment of Table 1, HyViLM and LLaVA-NeXT differ by 9.6 points on TextVQA (74.6 vs. 65.0) and by 6.9 points on DocVQA (85.1 vs. 78.2), so the abstract's '9.6% improvement' and '6.9% enhancement' are percentage-point differences, not percentage improvements. Section 4.3, in contrast, reports relative gains: for example, 74.6/71.6 − 1 ≈ 4.2% on TextVQA relative to MiniGemini-HD, and similar relative figures for the other document tasks. The paper should state the reference model and whether differences are absolute points or relative percentages; as written, the central numerical claims cannot be checked against Table 1.","section":"Abstract, §1, §4.3"}],"minor_comments":[{"comment":"The model name is inconsistent: the title and Section 1 introduce FILA, while the abstract, Table 1, Section 4, and Section 5 use HyViLM. Please unify the name throughout.","section":"Throughout"},{"comment":"The claim that HyViLM 'surpasses LLaVA-NeXT by 1.7% on MME' is not directly derivable from Table 1, which reports 1618/388 for HyViLM and 1604/368 for LLaVA-NeXT; please specify which MME component or combined metric is being compared.","section":"§4.3"},{"comment":"The module name is written as 'CVMF' in Section 5 and in Table 4's note; it should be 'CVFM' to match the rest of the paper.","section":"§5 and Table 4"},{"comment":"The alignment table says the example uses a 372 × 372 image, while Section 3.2 and the surrounding text say 336 × 336; please correct the inconsistency.","section":"Table 5"},{"comment":"The phrase 'our contributions are three folds' should read 'threefold' or 'three contributions.'","section":"§1"},{"comment":"The column header 'Doc GeneralTextVQA DocVQA AI2D InfoVQA OCRBenchMMBench MME MMMU MathVista HallBench' is malformed; please list the benchmark names clearly (TextVQA, DocVQA, AI2D, InfoVQA, OCRBench, MMBench, MME, MMMU, MathVista, HallBench).","section":"Table 1"},{"comment":"There is a typo in Table 2: 'Channle Concat' should be 'Channel Concat.'","section":"§4.4"}],"recommendation":"major_revision","confidential_remarks":"The reader's take's strongest arithmetic criticism appears to misread Table 1's column alignment: the numbers 76.5 and 71.6 are not the TextVQA entries. However, the more important concerns about the same-data control and test-set selection stand, and they are the basis of my recommendation. The internal ablations are encouraging, but the paper's headline claims currently exceed what the controlled evidence supports."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: FILA/HyViLM (the naming is inconsistent, which should be fixed) has a real idea—deep multi-layer fusion of a high-res ConvNeXt branch into CLIP-ViT via gated channel concatenation (CVFM)—and the ablations back it up. But the paper overclaims on the benchmark comparisons, and the abstract's numbers don't align with its own Table 1.\n\nWhat's genuinely new: while dual-encoder high-res MLLMs exist (Mini-Gemini, LLaVA-HR), they interact mostly at the last layer. FILA inserts four interaction layers, matching ConvNeXt stages to corresponding ViT stages, and gates the fusion with a zero-initialized tanh. The ablation in Tables 2–4 is systematic: it compares against channel-wise concat, local/global cross-attention, last-layer interaction, different interaction depths, and resizing methods. On TextVQA, InfoVQA, and MME the deep-fusion variant consistently wins. That is credible evidence for the mechanism.\n\nThe soft spots, in order:\n1. The central SOTA claim is not controlled for training data. Table 1 compares against published baselines trained on various mixtures, and FILA's instruction data include 28K OCR/QA pairs (DocVQA, ChartQA, DVQA, AI2D). The only same-data comparison is qualitative (Figure 4), which reports no scores. The text says MiniGemini-HD's data are 'almost identical,' but no recipe is given. So the 9/10 wins could be substantially data-driven.\n2. The abstract says a 9.6% improvement on TextVQA, and the intro says it surpasses LLaVA-NeXT by 9.6%. Table 1 shows HyViLM 76.5 vs LLaVA-NeXT 71.6—a 4.9-point gap, about 6.8% relative. The 9.6% matches the relative gap to HiRes-LLaVA (76.5 vs 69.7). So either the baselines are mislabeled or the percent is miscalculated. The 6.9% on DocVQA is similarly ambiguous.\n3. Ablation configurations (number of layers, fusion mode, resizing) are chosen using TextVQA, InfoVQA, and MME, and the same three benchmarks appear in the main results. So the architecture is partly fitted to those benchmarks. That doesn't invalidate the approach, but it tempers the cross-benchmark claim.\n4. No code, checkpoints, or error bars. For a paper claiming SOTA, artifacts matter.\n\nNone of this kills the core idea. The architecture is plausible, the ablations are internally consistent, and the fragmentation problem is real. But the paper would be stronger if the authors released code, re-ran a same-data baseline (at least LLaVA-NeXT) for the main table, and fixed the numeric discrepancies.\n\nWho is this for? People working on high-resolution MLLMs and visual encoders for dense documents. It deserves a serious referee, and I'd send it out with a request for revisions rather than desk-reject it. My verdict: conditional accept, contingent on controlled comparisons and artifact release.","headline":"Real architectural contribution, but the headline SOTA numbers are not yet controlled for training data; the abstract's 9.6% doesn't match Table 1's 4.9 points.","tokens_in":15279,"tokens_out":3676,"would_cite":true,"duration_ms":34507,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A vision-language model that keeps global context while encoding high-resolution crops outperforms prior models on nine of ten benchmarks.","keywords":["fine-grained vision-language models","high-resolution image understanding","dynamic cropping","image fragmentation","hybrid visual encoder","deep feature fusion","multimodal large language model","document visual question answering"],"falsifier":"A decisive test is to train LLaVA-NeXT and HyViLM on identical data and compare all ten benchmarks; if the 9.6% TextVQA gap shrinks or disappears, the advantage is data-driven rather than architectural.","tokens_in":14183,"feed_emoji":"🖼️","tokens_out":8020,"duration_ms":73398,"temperature":0.7,"pith_summary":"The paper sets out to fix a specific failure mode in high-resolution vision-language models: dynamic cropping, which cuts an image into sub-images to fit a low-resolution pre-trained encoder, often slices through objects or text and breaks the spatial context. To address this, it introduces HyViLM (also titled FILA), whose Hybrid Encoder pairs a CLIP-ViT processing the global view and crops with a ConvNeXt branch processing the whole image at high resolution; a deep fusion module, CVFM, injects the ConvNeXt features into the ViT's hidden states at four stages. The paper reports that this design outperforms existing MLLMs on nine of ten vision-language benchmarks, with a 9.6% improvement on TextVQA and 6.9% on DocVQA over LLaVA-NeXT. The paper argues that the architecture itself, not extra data or more tokens, is what restores fine-grained recognition.","feed_headline":"Hybrid encoder beats LLaVA-NeXT by 9.6% on TextVQA","feed_subtitle":"Fusing global context into every cropped patch fixes fragmentation and lifts document-QA scores.","key_machinery":"The load-bearing mechanism is the Hybrid Encoder with its ConvNeXt-ViT Deep Fusion Module (CVFM). The encoder sends the original image to two branches: CLIP-ViT receives the low-resolution global view plus the dynamically cropped sub-images, while ConvNeXt receives the same image resized to $\\frac{32}{14}$ times the crop resolution, preserving aspect ratio. The CVFM aligns each ConvNeXt stage's feature map to the spatial size of the corresponding ViT hidden state, concatenates them along the channel dimension, and runs a gated MLP, $F'_{vl} = F_{vl} + \\tanh(\\alpha_{\\text{dense}}) \\cdot \\operatorname{MLP}(F_{vl} \\oplus F'_{vh})$, so the fusion starts at zero and is learned gradually. Because ConvNeXt's receptive field covers the whole image, each ViT sub-image gains global context, which is the paper's answer to the fragmentation problem.","core_discovery":"The central claim is that image fragmentation from dynamic cropping can be resolved inside the visual encoder rather than patched after the fact. The paper's model keeps the standard two-branch setup of a low-resolution CLIP-ViT and a high-resolution ConvNeXt, but changes where and how they interact: instead of a single cross-attention at the last layer, CVFM concatenates the channel-aligned ConvNeXt features into the ViT's hidden states at four selected layers, with a tanh-gated MLP that lets the fusion be switched on gradually. The paper shows this beats the dynamic-slicing baseline and last-layer interaction baselines on TextVQA, InfoVQA, and MME, and argues that the global context carried by the high-resolution branch is what lets the model read text and objects that fall on crop boundaries.","pith_inferences":["The ablation table suggests that interaction depth, not the fusion operator, is the main driver: channel concatenation beat both local and global cross-attention. An unstated implication is that other dual-encoder MLLMs could gain more by deepening their fusion than by improving the attention scheme.","The paper's same-data comparison with LLaVA-NeXT appears only in the qualitative examples; a controlled head-to-head with identical training data across all ten benchmarks would isolate the architectural contribution from the data contribution. That experiment is not in the paper.","The described mechanism predicts that performance should degrade gracefully as the high-resolution branch's input resolution is lowered, and that the sweet spot for interaction layers depends on the ViT size. Both are testable without new data.","The idea of injecting global context at multiple depths generalizes beyond image cropping: the same CVFM pattern could apply to video frames or multi-page documents, where each frame or page is a 'crop' of a larger scene, though the paper does not test this."],"forward_implications":["If the gains are real, any MLLM built on a low-resolution CLIP-ViT can be upgraded to high-resolution document understanding by adding a ConvNeXt branch and the CVFM interaction, without retraining the ViT from scratch.","Because the interaction is at four internal layers rather than the last layer, features that would otherwise be lost by the time the final output is computed are preserved; the ablation table supports four layers as the sweet spot.","The token count fed to the LLM stays the same as LLaVA-NeXT, so the reported accuracy gains on TextVQA and DocVQA come without extra language-model compute.","The two-stage training recipe (low-resolution alignment first, then full high-resolution instruction tuning) is a reusable schedule for other hybrid encoders.","The paper's framing suggests OCR and fine-grained VQA are the most direct beneficiaries, since those tasks stress exact reading across crop boundaries; general benchmarks like MME and MathVista also improve, but less dramatically."],"supporting_citations":[{"why":"LLaVA-NeXT is the primary baseline the paper compares against; the reported TextVQA and DocVQA gains and the same-data qualitative comparison are relative to this model.","marker":"[Liu et al., 2024b]"},{"why":"Mini-Gemini is the closest dual-encoder approach that interacts low-resolution CLIP-ViT with a high-resolution branch at the last layer; the paper's deep-fusion design is positioned against it, and MiniGemini-HD is a near-matched baseline with almost identical data.","marker":"[Li et al., 2024a]"},{"why":"CLIP provides the pre-trained ViT used as the main low-resolution branch; its 336x336 training resolution is the limitation dynamic cropping targets.","marker":"[Radford et al., 2021]"},{"why":"ConvNeXt provides the high-resolution convolutional branch that supplies global features to the ViT.","marker":"[Liu et al., 2022]"},{"why":"Monkey is a representative dynamic-cropping high-resolution MLLM that the paper builds on and compares against.","marker":"[Li et al., 2024b]"},{"why":"TextVQA is one of the two benchmarks where the paper reports its largest gains, testing fine-grained reading across image boundaries.","marker":"[Singh et al., 2019]"},{"why":"DocVQA is the other benchmark with the largest reported gain, providing the document-understanding testbed.","marker":"[Mathew et al., 2021b]"}],"fun_headline_variants":["Fusion encoder fixes crop fragmentation, lifting TextVQA by 9.6%","HyViLM: hybrid encoder solves image cropping, boosts DocVQA by 6.9%","Global context fused into crops: 9.6% TextVQA gain","New encoder avoids truncation, wins 9 of 10 multimodal tasks","In-encoder fusion beats crop truncation: 9.6% better TextVQA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire performance claim rests on the assumption that the reported gains come from the hybrid encoder rather than from the training data, because the main table compares against published baselines with heterogeneous data and only the qualitative section shows a same-data head-to-head.","fun_headline_variants_meta":{"raw":{"variants":["Fusion encoder fixes crop fragmentation, lifting TextVQA by 9.6%","HyViLM: hybrid encoder solves image cropping, boosts DocVQA by 6.9%","Global context fused into crops: 9.6% TextVQA gain","New encoder avoids truncation, wins 9 of 10 multimodal tasks","In-encoder fusion beats crop truncation: 9.6% better TextVQA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000754,"raw_usage":{"total_tokens":3346,"prompt_tokens":928,"completion_tokens":2418,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":2308}},"tokens_in":544,"tokens_out":2418,"duration_ms":16501,"temperature":1.0,"reasoning_tokens":2308,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:51:39.118136+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive test is to train LLaVA-NeXT and HyViLM on identical data and compare all ten benchmarks; if the 9.6% TextVQA gap shrinks or disappears, the advantage is data-driven rather than architectural.","supporting_citations":[],"review_version":1}