{"id":"60b213cb-8e11-4264-8312-1794ec6a6d04","arxiv_id":"2412.05983","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Chimera fuses frozen domain-expert encoders into a generalist multimodal LLM via routing and a 30% masking of general tokens, lifting InternVL2-8B from 61.6 to 64.9 on MathVista and from 31.3 to 32.4 on MathVerse.","lead":"The paper introduces Chimera, a pipeline that attaches frozen specialist models for charts, tables, and math to a general vision-language model by feeding the specialist's visual features into the general model and masking part of the general image features during training. It reports state-of-the-art math reasoning scores for an 8B model and near-specialist results on document and table structure extraction.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training set includes MathVista source datasets (AI2D, ChartQA, Geometry3K, etc.) with no reported decontamination, so the 64.9 SOTA may reflect benchmark leakage rather than Chimera's expert-integration method.","rationale":"I read the paper in good faith: the controlled ablations in Supp. Table 11 (NF vs Chimera-0.0 vs Chimera-0.3) are the right kind of evidence, and the GSCM idea is plausible. The problem is that the headline evaluation is run on MathVista testmini, whose source datasets are also used for training, and no decontamination is reported. This is a standard correctness check for benchmark claims; without it, the strongest empirical support for the central claim is ambiguous. The reader's concern about mask-ratio selection on the testmini is valid and related, but it applies after decontamination is established; contamination would affect all model variants and the baselines differently. Hence I view the source-data overlap as the single most load-bearing concern. The recommendation remains conditional acceptance, with decontamination as a required condition, so the reader's verdict category does not change.","tokens_in":25447,"tokens_out":14347,"duration_ms":140816,"concrete_test":"Compute image-level and text-level overlap between all Stage 2 training examples and MathVista testmini: use perceptual hashes for images and exact/near-duplicate matching for question strings, then report the matched fraction per source dataset. If any non-trivial fraction matches, re-run Chimera-8B on the unmatched subset; if the re-computed accuracy drops materially, the 64.9 SOTA claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"MathVista's testmini is not a collection of unseen images; it is assembled from existing datasets such as AI2D, ChartQA, PlotQA, DVQA, FigureQA, Geometry3K, GeoQA+, InterGPS, TQA, CLEVR-Math, Super-CLEVR, ScienceQA, IconQA, and MapQA. Supplementary Table 8 lists nearly all of these as Chimera's Stage 2 visual instruction tuning data, with no split-level decontamination notes. The paper reports 64.9 on MathVista testmini (Table 1) and also selects the GSCM mask ratio on the same testmini (Supp. Table 11), but it never reports whether any training images or questions overlap with the evaluation set. If overlap exists, the headline gain over InternVL2-8B (61.6) could come from the model having seen the test distribution during training, not from frozen expert features, routing, or GSCM. This is more load-bearing than the mask-ratio issue alone: even a perfectly tuned GSCM cannot validate a method if the benchmark itself is contaminated, and the internal ablations do not control for it because all Chimera variants share the same training data. A decontamination analysis is therefore a prerequisite for the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Chimera, a training pipeline that augments a generalist large multi-modal model (InternVL2) with features from frozen domain-specific expert encoders (StructEqTable, ChartVLM, Math-CLIP, and GOT). A lightweight router selects which expert, if any, to invoke; trainable projectors map expert features into the LLM embedding space; and a proposed Generalist-Specialist Collaboration Masking (GSCM) mechanism randomly masks a fraction of general visual tokens during instruction tuning to encourage reliance on expert tokens. The authors report state-of-the-art results on MathVista (64.9 for Chimera-8B) and MathVerse (32.4), with a DPO post-training variant reaching 68.3 on MathVista, plus strong results on chart/table/document structural extraction benchmarks. The paper includes ablations on the mask ratio, router accuracy, and number of experts, and states that model weights and training/evaluation data will be released.","tokens_in":25707,"tokens_out":3036,"duration_ms":29852,"significance":"If the reported results are trustworthy, the paper would make a useful contribution: a low-cost, scalable recipe for injecting frozen specialist knowledge into generalist LMMs, with public data and weights promised and multiple standard benchmarks used. The GSCM mechanism is simple and the ablation of mask-ratio and expert-set size is informative. However, the headline claims rest on MathVista and MathVerse scores, and the manuscript does not currently establish that those scores are unaffected by overlap between training corpora and the evaluation sets. Until that is resolved, the central claim of state-of-the-art performance is not yet credible. The general idea of fusing frozen expert encoders with masking is plausible, but the empirical validation needs a decontamination analysis.","major_comments":[{"comment":"The Stage 2 visual instruction tuning data listed in Supplementary Table 8 includes AI2D, ChartQA, PlotQA, DVQA, FigureQA, Geometry3K, GeoQA+, InterGPS, TQA, CLEVR-Math, Super-CLEVR, ScienceQA, IconQA, and MapQA. These are the same source datasets that constitute MathVista's testmini. The paper reports the headline number 64.9 on MathVista testmini (Table 1) without any decontamination analysis, such as exact-duplicate or near-duplicate detection between training samples and evaluation samples. Since every Chimera variant in the ablations is trained on the same data, the internal comparisons cannot disentangle method gains from possible memorization of the test distribution. Please report overlap statistics (image-level and question-level) and, if any overlap exists, re-run the main comparisons on the disjoint subset or on a fully uncontaminated benchmark.","section":"Supp. Table 8 vs. Table 1"},{"comment":"The GSCM mask ratio is selected by ablating on the MathVista testmini subset (Supplementary Table 11), which is the same set used for the headline accuracy in Table 1. Tuning a hyperparameter on the evaluation set and then reporting that set's accuracy as the method's performance inflates the apparent gain and makes the specific choice of 0.3 and the resulting 64.9 score partially fitted. The mask-ratio ablation should be performed on a separate validation set, and the final mask ratio should be justified without reference to the testmini numbers.","section":"Supp. Table 11 / Sec. 4.4"},{"comment":"The DPO post-training constructs 60K preference pairs by sampling 10K problems from MathV360K and generating 16 responses per problem with Chimera. No decontamination is reported with respect to MathVista testmini or MathVerse, yet Chimera† is evaluated on exactly those benchmarks (Table 1 and Table 2). If MathV360K contains questions from MathVista or MathVerse, the DPO gain of 3.4 points could be due to preference optimization on the evaluation distribution rather than to the method itself. Please check for overlap and report results on a clean subset.","section":"Supp. Sec. 9 / Table 1"}],"minor_comments":[{"comment":"The sentence 'of which the visual content differing differs significantly from natural images' contains a grammatical error; please revise to 'the visual content of which differs significantly from natural images'.","section":"Sec. 2"},{"comment":"The table header contains the typo 'IntenrVL2'; it should read 'InternVL2'.","section":"Table 1"},{"comment":"The expression 'Sp_i (Se_i (Xv))' would be clearer with explicit function-application notation, e.g., 'Sp_i(Se_i(Xv))', and the index 'i' is used both as an argmax target and as a free index in the set definition; please disambiguate.","section":"Eq. (2)"},{"comment":"The caption in the supplementary text references 'Fig. ??' when discussing chart structural extraction; this undefined reference should be fixed.","section":"Supp. Fig. 13"},{"comment":"The word 'privides' in 'Eg privides the general visual features' is a typo for 'provides'.","section":"Sec. 3.1"},{"comment":"The router error analysis in Table 7 uses dataset-based labels and the paper itself notes that the 'general' category contains mixed-domain images (last sentence of Sec. 4.4); this is an acknowledged limitation, but it would be useful to quantify how much the 95.4% router accuracy depends on the dataset identity of the source rather than on visual content.","section":"Sec. 4.4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is well within the scope of the journal and the methodology is of interest. The decision hinges on the leakage question: the authors need to provide a rigorous decontamination analysis for MathVista testmini, MathVerse, and the DPO data before the state-of-the-art claim can be accepted. If the overlap is substantial, the headline results would need to be recomputed; if the overlap is negligible, a clear statement of the check would likely resolve the concern. The self-collected Table-SE and Doc-SE benchmarks are small (100 and 205 samples) and lack external validation, which is a secondary weakness but not a blocker if the main benchmark claims are cleaned up."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Chimera is a solid, well-engineered attempt at grafting frozen domain-specific encoders onto a generalist LMM. What is actually new is the GSCM masking trick and the two-stage progressive alignment with a lightweight token-level router. The paper cites MoVA and MoME honestly, and the attention analysis at least suggests GSCM moves the model toward using expert tokens. If the numbers hold, this is a practical recipe for adding specialist knowledge cheaply.\n\nBut the headline number does not hold up to scrutiny. The training set listed in Supplementary Table 8 includes nearly all of the source datasets that make up MathVista testmini (AI2D, ChartQA, PlotQA, DVQA, FigureQA, Geometry3K, GeoQA+, InterGPS, TQA, CLEVR-Math, Super-CLEVR, ScienceQA, IconQA, MapQA). The paper reports no decontamination analysis. If those exact images or questions appear in training, the 64.9 SOTA over InternVL2-8B could be pure benchmark leakage, not a property of the method. The fact that all variants in the ablation share the same training data means even the internal comparisons cannot wash this out. This is load-bearing, not a footnote.\n\nThe mask-ratio selection on the same testmini that generates the headline number is a milder version of the same problem, and the self-built Table-SE and Doc-SE benchmarks are not yet released, so those results are unverifiable. On the credit side, the paper does release code and model weights, trains at multiple scales, checks general performance on MME, and the GSCM idea is clever and worth testing in other settings.\n\nMy honest take: the method could be right, but the evidence as presented is not convincing enough to trust the SOTA claim. A referee should require a decontamination analysis (e.g., remove any training sample whose source and split match testmini, or evaluate on a fresh benchmark), a held-out validation split for hyperparameters, and release of the custom benchmarks. With those fixes, this could be a useful contribution. As submitted, I would not cite the MathVista number in my own work, and I'd be cautious about the rest.\n\nThe paper deserves a serious referee because the core idea is interesting and the flaws are addressable. Send it to review, but the review needs to be honest about the contamination issue.","headline":"Chimera's engineering is plausible and the GSCM mechanism is new, but the MathVista SOTA claim is compromised by an unreported train/test overlap with the testmini source datasets.","tokens_in":26280,"tokens_out":2535,"would_cite":false,"duration_ms":25216,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fusing frozen domain experts into a generalist vision-language model, via a router and token masking, lifts an 8B model to top scores on two math-reasoning benchmarks.","keywords":["large multimodal models","domain-specific experts","expert routing","visual token masking","multi-modal mathematical reasoning","visual structural extraction","Generalist-Specialist Collaboration Masking"],"falsifier":"Run the exact two-stage Chimera recipe on MathVista with the specialist token stream ablated at inference, forcing the router to choose no expert for every image, and compare accuracy with the full model: if the score does not fall well below 64.9, the expert features are not what carries the claimed gain.","tokens_in":25243,"feed_emoji":"🧠","tokens_out":9600,"duration_ms":82359,"temperature":0.7,"pith_summary":"The paper tries to show that a generalist vision-language model can acquire specialist skills—reading charts, tables, math diagrams, and document layouts—without retraining on private domain data or sacrificing its general abilities. The proposed pipeline, Chimera, keeps domain-expert encoders frozen, projects their features into the language model's input, and uses a small router to decide which expert, if any, should be invoked for a given image. The central empirical claim is that this reaches 64.9 on MathVista and 32.4 on MathVerse, state-of-the-art for models of comparable scale, while matching or approaching dedicated experts on chart, table, and document extraction. If true, this matters because it turns scarce specialist knowledge into a cheap modular upgrade that can be layered onto broadly capable models.","feed_headline":"Frozen specialists lift an 8B vision model to 64.9 on MathVista","feed_subtitle":"Randomly masking 30% of general image tokens forces the model to learn from chart, table, and math experts.","key_machinery":"The load-bearing mechanism is a trio: a lightweight linear router, frozen domain-expert encoders whose outputs are linearly projected into the language model's embedding space, and the Generalist-Specialist Collaboration Masking (GSCM) rule. The router takes the general encoder's classification token and outputs a distribution over 'no expert' plus one entry per specialist, deciding which expert token stream, if any, gets appended to the masked general visual tokens. GSCM is the sampling rule that, during instruction tuning, randomly selects 30% of the general visual tokens without replacement and blocks their attention, forcing the model to use expert features as a supplement. The paper's attention analysis shows that with masking the model attends substantially more to expert tokens, which is the mechanism that is claimed to balance the optimization between the well-aligned general encoder and the specialists.","core_discovery":"On its own terms, the paper's discovery is that the real obstacle to combining generalists with specialists is not the representational gap itself but an optimization imbalance: a well-aligned general visual encoder lets the model ignore the newly added expert tokens. Chimera counters this with Generalist-Specialist Collaboration Masking (GSCM), which randomly hides 30% of the general visual tokens during visual instruction tuning so the language model must draw on the specialist stream. With a progressive two-stage recipe that first aligns projections on structure-oriented tasks and then instruction-tunes with masking, Chimera turns an existing generalist LMM into a multi-domain model that sets new accuracy marks on MathVista (64.9) and MathVerse (32.4) and nearly matches specialist systems on chart, table, and document structural extraction. The paper further reports that a standard preference-optimization pass raises MathVista accuracy to 68.3.","pith_inferences":["Beyond the paper's claims, the same recipe should be testable on other frozen visual specialists—medical imaging, satellite imagery, code or diagram understanding—provided their encoders emit dense token features that a linear projector can align; the low training cost makes this an attractive experiment.","The paper's ablations suggest the optimal masking ratio and even the value of a given expert depend on the gap between that expert's pretraining task and the target task; a natural extension is to make the mask ratio or the router's decision task-aware rather than fixed at 0.3.","Because the router is trained with dataset-level labels and the paper notes that its 'general' category contains mixed-domain images, per-image expert annotation or confidence-based routing could reduce the reported 4.6% router error and further raise end-to-end accuracy."],"forward_implications":["Specialist capability becomes a plug-in: a new domain expert can be added to an existing generalist model by training only a router, projectors, and one instruction-tuning pass, with no need for private domain training data.","One model can hold several specialist skills at once: the router selects the right expert per image, so chart, table, math, and document abilities do not have to live in separate single-purpose systems.","The gains survive standard alignment training: a direct preference-optimization run on 60K pairs lifts MathVista accuracy from 64.9 to 68.3, so expert-augmented models remain compatible with preference tuning.","General capabilities do not collapse: on a general perception benchmark, Chimera stays comparable to its base model while gaining the specialist skills, suggesting the masking constraint does not destroy existing knowledge."],"supporting_citations":[{"why":"Supplies the table-expert encoder whose projected features Chimera appends to the input sequence.","marker":"[83]"},{"why":"Supplies the chart-expert encoder and its structural-extraction pretraining, tested on ChartQA-SE.","marker":"[84]"},{"why":"Supplies the math-expert encoder trained on geometry and function captions, which carries much of the reasoning gain.","marker":"[89]"},{"why":"Supplies the document-expert encoder used for visual structural extraction and sets the specialist comparison standard on Doc-SE.","marker":"[80]"},{"why":"Provides the base generalist LMM, its general visual encoder, dynamic high-resolution tokenization, and the main baseline for all Chimera variants.","marker":"[12]"},{"why":"Defines the MathVista benchmark whose testmini set is the primary multi-modal reasoning claim (64.9).","marker":"[48]"},{"why":"Defines the MathVerse benchmark whose accuracy of 32.4 is the second headline reasoning result.","marker":"[90]"}],"fun_headline_variants":["Chimera merges specialists with masking to hit 64.9 on MathVista","Masking 30% of image tokens lets experts boost generalist to 64.9","Random token masking forces model to use specialists, hitting 64.9","Frozen specialists, masked general tokens: recipe for 64.9 on MathVista"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that frozen specialist encoders still deliver useful domain information after a simple linear projection, and that randomly masking 30% of the general image tokens is enough to force the model to use that information rather than rely on its already-aligned general encoder.","fun_headline_variants_meta":{"raw":{"variants":["Chimera merges specialists with masking to hit 64.9 on MathVista","Masking 30% of image tokens lets experts boost generalist to 64.9","Random token masking forces model to use specialists, hitting 64.9","Frozen specialists, masked general tokens: recipe for 64.9 on MathVista"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000846,"raw_usage":{"total_tokens":3684,"prompt_tokens":951,"completion_tokens":2733,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":2642}},"tokens_in":567,"tokens_out":2733,"duration_ms":17653,"temperature":1.0,"reasoning_tokens":2642,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:07:02.397175+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the exact two-stage Chimera recipe on MathVista with the specialist token stream ablated at inference, forcing the router to choose no expert for every image, and compare accuracy with the full model: if the score does not fall well below 64.9, the expert features are not what carries the claimed gain.","supporting_citations":[{"cited_title":"Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186","cited_arxiv_id":null,"evidence_quote":"Defines the MathVerse benchmark whose accuracy of 32.4 is the second headline reasoning result."}],"review_version":1}