{"id":"8fb4f130-dab1-495f-8977-3a608a132266","arxiv_id":"2506.11515","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Manager aggregates multi-layer unimodal representations and improves both two-tower VLMs (ManagerTower) and MLLMs (LLaVA-OV-Manager) on 24 downstream tasks.","lead":"This paper introduces Manager, a lightweight module that combines features from multiple layers of a pretrained vision or language encoder and injects the mixture into a cross-modal model. The authors show it improves two-tower vision-language models and multimodal LLMs on 24 downstream datasets, including high-resolution document and OCR tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 20-dataset claim rests on an undisclosed post-hoc removal of datasets; without the full pre-removal leaderboard and variance estimates, the reported +1.06/+1.44 gains cannot be distinguished from selection effects.","rationale":"The reader's weakest assumption points to the same load-bearing concern: the 20-dataset evaluation is not shown to be a pre-specified, representative benchmark because Appendix C-E admits removing 'few some downstream datasets' without naming them. I agree that this is the main threat to the central claim. I also note that the reported aggregate tables contain an internal inconsistency in the General-category averages, which raises the cost of trusting the headline numbers as printed; however, this is best treated as a reporting/audit problem rather than a separate refutation of the method. In good faith, the paper has genuine support: the two-tower results are controlled against a re-implemented BridgeTower with matched settings, the MLLM comparisons hold all training data and objectives fixed between Baseline and Baseline+Manager, and all 20 reported per-dataset deltas are positive in the disclosed tables. But the strength of the claim ('significantly' across categories, images, and resolutions) exceeds what a single-run, post-hoc-selected 20-dataset evaluation can support. Since the reader already assigned CONDITIONAL, my read does not change the verdict; the condition should explicitly require disclosure of the full pre-removal dataset list, per-dataset results for all datasets, and error bars or multiple seeds before the strong wording is accepted.","tokens_in":48105,"tokens_out":12241,"duration_ms":111187,"concrete_test":"Re-run the four LLaVA-OV variants (Baseline, +Manager, +Grid, +Grid+Manager) on the complete lmms-eval suite used by the original LLaVA-OV, including every dataset considered before the 'few some' removal in Appendix C-E, and release the per-dataset table. Then recompute the reported averages, plus mean and standard deviation over 3 training seeds on the full 8M dataset. If the Manager-vs-Baseline and Grid+Manager-vs-Grid deltas on the full pre-removal suite are not positive at the category level, or if the corrected General-category averages differ materially from Table XI/XII, the 'significant' claim should be downgraded to conditional on the selected subset.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central MLLM claim—managers significantly improve LLaVA-OV on 20 downstream datasets—depends on those 20 datasets being a fair, fixed evaluation suite. In Appendix C-E the authors state: 'we remove few some downstream datasets for better efficiency and robustness' and do not name them. If any of those removals were decided after seeing results, the average gains in Table IV/Fig. 8 are partly selection artifacts: the headline set is the one that makes the plugin look best. The problem is compounded by the fact that the ablation in Section V-D uses only 9 datasets at 1/4 training data, and the final 20-dataset numbers are single runs with no standard errors, so small positive deltas (e.g., InfoVQA +0.01, OKVQA +0.10 in Table XII) could be noise. A related audit issue: the General-category averages in Table XI/Table XII do not match the per-dataset values in Table XII/Table XIII; for example, the six reported Baseline General scores average 50.44, not 48.71/48.72. This makes the aggregate tables hard to verify from their own data. The load-bearing issue is therefore not the architecture but the evaluative frame: before crediting the headline, the pre-removal benchmark and variance must be shown.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Manager, a lightweight module that aggregates representations from multiple layers of pretrained unimodal encoders and injects them into cross-modal or LLM layers. In the two-tower VLM setting, ManagerTower replaces BridgeTower's one-hot layer-wise bridges with learned static (SAUM) or adaptive (AAUM) managers, reporting gains on VQAv2, SNLI-VE, NLVR2, and Flickr30K under matched pretraining and finetuning. In the MLLM setting, LLaVA-OV-Manager injects visual managers into LLaVA-OneVision, with or without the multi-grid algorithm, and reports zero-shot gains on 20 datasets, together with attention-diversity analyses and case studies.","tokens_in":48409,"tokens_out":4143,"duration_ms":39097,"significance":"If the empirical claims are sound, the Manager plugin is a useful, architecture-agnostic contribution: it is lightweight, compatible with different encoders and cross-modal depths, and the paper provides a fairly extensive set of controlled comparisons, ablations, and computational-budget measurements. The ManagerTower results on four downstream tasks are internally consistent and appear to be a credible extension of BridgeTower. The MLLM extension is ambitious and the depth/width synergy viewpoint with the multi-grid algorithm is interesting. However, the headline 20-dataset claim is currently not auditable because the paper does not disclose which datasets were removed from the original LLaVA-OV evaluation suite, and several aggregate numbers in the MLLM section are internally inconsistent. These issues must be resolved before the central claims can be credited.","major_comments":[{"comment":"The sentence \"we remove few some downstream datasets for better efficiency and robustness\" does not identify which datasets were removed, how many, or the decision rule. Since the central claim is a significant improvement on the 20 reported datasets, the evaluation suite must be a fixed, pre-specified set; otherwise the reported averages in Table IV and Figure 8 could reflect post-hoc selection. Please report the full set of datasets evaluated, the removed datasets, and the results on all of them, or justify that the removals were independent of performance.","section":"Appendix C-E"},{"comment":"The aggregate numbers in the MLLM section are internally inconsistent. For the Baseline row in Table XII, the six listed General values (VQAv2 73.90, OKVQA 37.18, GQA 57.66, MMVet 21.40, SEED-Bench 61.50, RealWorldQA 50.98) average to 50.44, yet Table XII reports 48.71 and Table XI reports 48.72. The Table XI overall scores (50.54/51.50) also conflict with Table IV (50.61/51.67) for the same claimed 20-dataset averages. Please recompute and reconcile all aggregate tables, or explicitly state the averaging procedure and any excluded datasets.","section":"Tables XI-XIII"},{"comment":"The text states \"55.21−51.67 = 1.44\" and uses this to claim the synergy gain over the manager-only baseline. The arithmetic is incorrect: 55.21−51.67 = 3.54, and the manager-over-grid gain is 55.21−53.87 = 1.34, not 1.44. This error directly affects the claim that the manager and multi-grid algorithm provide an additional 1.44% synergy. Please correct the computation and the corresponding claim.","section":"Section V-C, Table IV"},{"comment":"The term \"significantly\" is used throughout, but no confidence intervals, standard deviations, or significance tests are reported. This is especially important because some per-dataset deltas are tiny (e.g., InfoVQA +0.01, OKVQA +0.10 in Table XII) and the headline gains of +1.06/+1.44 are averages over 20 datasets. Please provide multiple runs or at least a paired significance analysis for the main comparisons, and temper the word \"significantly\" accordingly.","section":"Section V-C and Tables IV, XII"}],"minor_comments":[{"comment":"The phrase \"The casual nature of the representation\" should be \"causal nature\", and \"casual attention\" should be \"causal attention\" in the same paragraph.","section":"Appendix A-A"},{"comment":"The phrase \"remove few some downstream datasets\" is ungrammatical; it should read \"remove a few downstream datasets\" or \"remove some downstream datasets\".","section":"Appendix C-E"},{"comment":"The x-axis label is inconsistent across figures: \"Unimodal Expert Index\", \"Uni-Modal Layer Index\", and \"Uni-Modal Expert Index\" are used interchangeably. Please unify the terminology.","section":"Figures 23-25 and Figure 7"},{"comment":"The caption states the ablation is on 9 datasets, but the figure reports only an overall average score. Please list the 9 datasets or cite the table containing them, for reproducibility.","section":"Section V-D2, Figure 10"},{"comment":"The table lists \"2407 and 2409\" for MM-LiveBench, but the text and Table XIII refer to \"July\" and \"Sep\". Please align these labels and verify the dates.","section":"Appendix C-F, Table X"}],"recommendation":"major_revision","confidential_remarks":"The footnote stating that the manuscript \"has been accepted by TCSVT\" with a DOI, combined with the arXiv submission under review, may require editorial checking on prior-publication and overlap issues. The core architecture appears sound, but the MLLM evaluation's undisclosed dataset removal and internal numeric inconsistencies are likely to be decisive for reviewers and readers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the MLLM half of this is the part worth reading. ManagerTower itself is the ACL 2023 result, re-presented with a few extra tables; the new material is the LLaVA-OV-Manager experiments where a 0.08M-parameter plugin gives ~1.1–1.4 points average over a controlled retrained baseline on 20 datasets, and the depth-vs-width story with the multi-grid algorithm is a genuinely useful way to think about the two interventions.\n\nThe empirical discipline is better than most: they retrain the baseline with the same data and settings, ablate manager design (SAM/SAUM/AAUM), swap backbones, and include the attention entropy/KL analysis. The appendix is candid about where AAUM fails in MLLMs and where the plugin hurts (non-semantic OCR). That honesty earns credit.\n\nNow the soft spots, in rough order of importance. First, the 20-dataset claim rests on an evaluation set that is not fully specified. In Appendix C-E they say they 'remove few some downstream datasets' for efficiency and robustness, and never name them. If any of those removals happened after looking at results, the headline +1.06/+1.44 is partly a selection artifact. That is the load-bearing weakness. Second, the aggregate tables don't agree with their own per-dataset numbers: the General-category averages in Table XI/Table XII are about 1.7 points lower than the mean of the six listed datasets (Baseline shows 48.71/48.72 vs an actual mean of ~50.44). The overall average in Table IV does match the per-dataset data, so the main effect is probably real, but the category numbers need a fix. Third, everything is single-run with no error bars or significance tests; the word 'significantly' is doing unearned work, especially on per-dataset deltas like InfoVQA +0.01.\n\nNone of these is fatal. The architecture is simple, the controlled baselines are the right comparison, and the mechanism is plausible. But before the headline claim can be trusted, the authors need to disclose the pre-removal dataset list, correct the aggregate arithmetic, and report variance across seeds. If they do that, this becomes a useful reference for anyone adding multi-layer visual features to an MLLM.\n\nBottom line: worth sending to a serious referee, not a desk reject, but it needs a revision round. I'd read the revised version with the dataset list and corrected tables.","headline":"Useful drop-in plugin paper with honest limitations, but the 20-dataset MLLM claim needs the removed datasets and variance estimates before it can be taken at face value.","tokens_in":48929,"tokens_out":5032,"would_cite":true,"duration_ms":44355,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A lightweight manager that aggregates multi-layer unimodal representations improves two-tower vision-language models and multimodal LLMs at near-zero added cost.","keywords":["vision-language model","multimodal large language model","two-tower architecture","manager module","multi-layer representation aggregation","multi-grid algorithm","zero-shot evaluation","visual representation learning"],"falsifier":"Run the four-baseline comparison on all datasets in the original LLaVA-OV zero-shot suite with no datasets removed and report per-dataset deltas; if the +Grid+Manager advantage over +Grid shrinks to zero or becomes negative once the excluded datasets are included, the 20-dataset averaging claim collapses.","tokens_in":47900,"feed_emoji":"🧩","tokens_out":7938,"duration_ms":71891,"temperature":0.7,"pith_summary":"The paper proposes a plugin-style module, the manager, that feeds aggregated representations from multiple layers of pre-trained unimodal encoders into a vision-language fusion module, instead of using only last-layer features or fixed layer-by-layer bridges. The claim is that this adaptive aggregation of insights from unimodal experts makes vision-language alignment more comprehensive and cheap: in a two-tower model (ManagerTower) it outperforms METER and BridgeTower under the same 4M-image pre-training and on four downstream tasks, and in the multimodal LLM LLaVA-OV it raises zero-shot averages across 20 datasets by about 1.06 points without multi-grid tiling and 1.44 points with it, at negligible parameter and compute overhead. A sympathetic reader would care because the module is architecture-agnostic, so if the claim holds it is a low-cost way to improve both older two-tower models and modern high-resolution multimodal LLMs.","feed_headline":"Manager module lifts vision-language scores on 24 datasets","feed_subtitle":"Multi-layer unimodal features, adaptively aggregated, improve two-tower VLMs and zero-shot MLLMs with tiny added cost.","key_machinery":"The central object is the manager, a lightweight aggregation layer inserted into each cross-modal layer (or, at intervals, into the LLM of an MLLM). Its main form, AAUM, computes per-token aggregation weights $W_A=\\mathrm{softmax}(\\mathrm{LN}(C^V_{\\ell-1})W_M+\\epsilon)$ and produces $\\tilde{C}^V_\\ell=\\sum_{i=1}^{6}W_{A,i}\\odot\\mathrm{LN}(V_{i+6})+W_C\\odot\\mathrm{LN}(C^V_{\\ell-1})$, so each token in each sample can draw differently on the top $N=6$ unimodal layer representations plus the previous cross-modal layer state. In the MLLM setting the paper uses an optimized SAUM variant with zero-initialized weights, no LayerNorm and no softmax, to avoid disturbing the pre-trained LLM early in training. This mechanism decouples the number of unimodal layers exploited from the number of cross-modal layers, and the authors use attention entropy and head-diversity measurements to attribute the gains to more diverse visual attention.","core_discovery":"On the paper's own terms, the discovery is that layers of a pre-trained unimodal encoder can be treated as experts, and a small trainable manager that learns to combine their outputs—token-by-token in ManagerTower, with simpler static weights in LLaVA-OV-Manager—supplies exactly the level of semantic detail each cross-modal layer needs. This repairs what the authors identify as BridgeTower's flaw: fixed one-to-one bridges force layer-by-layer reuse of unimodal representations and tether the number of bridges to the number of cross-modal layers. With managers, ManagerTower reaches 79.15% on VQAv2 test-std and 86.56% IR@1 on Flickr30K using the same 4M pre-training data as BridgeTower, and LLaVA-OV-Manager improves the zero-shot average on 20 datasets from 50.61% to 51.67% without grids and from 53.87% to 55.21% with grids, with 0.08M extra parameters. The paper further argues that the manager's multi-layer aggregation (depth) and the multi-grid algorithm (width) are orthogonal ways to add visual detail, and that their synergy counters the semantic ambiguity caused by cutting images into grids.","pith_inferences":["If the depth/width decomposition is the right explanation, managers should also stack with other width-style plugins—different tile sizes, high-resolution encoders, or multi-scale crops—rather than only with LLaVA-OV's specific multi-grid scheme; that is testable with the same four-baseline recipe.","The 20-dataset average should be re-computed on the full original LLaVA-OV evaluation suite; Appendix C-E says the authors removed 'few some downstream datasets' for efficiency and robustness, and the robustness of the central claim depends on whether those omitted datasets also benefit from the manager.","The attention-entropy mechanism suggests managers might help in other settings where vision tokens are ambiguous, such as small objects or heavily cropped inputs, even without grid tiling; a controlled experiment varying input resolution while holding architecture fixed could test this.","AAUM's failure to beat SAUM in the causal LLM setting hints that a bidirectional or causally compatible query might unlock more of the manager's potential in MLLMs, a direction the paper itself raises as future work."],"forward_implications":["ManagerTower with half the cross-modal layers ($L_C=3$) can beat BridgeTower with $L_C=6$, so the same data and backbones yield better results with less cross-modal depth.","Adding managers to LLaVA-OV is cheap: about 0.08M parameters, roughly 2% more FLOPs, and 3–4% more training or inference time, while adding about one absolute point without grids and 1.44 points with grids on the 20-dataset average.","The top half of the visual encoder carries most of the benefit; including all layers hurts, so practitioners can select a fixed layer range and reuse it across architectures.","The manager and the multi-grid algorithm are complementary: managers help most where grid splitting cuts objects and creates ambiguity, such as instance location and counting on SEED-Bench and artistic or handwriting text recognition on OCRBench.","Because the manager works with different visual and textual backbones and with both two-tower and MLLM architectures, the same recipe transfers across model families without retraining the unimodal encoders."],"supporting_citations":[{"why":"The conference version of ManagerTower whose manager design and experiments this paper extends.","marker":"[1]"},{"why":"METER supplies the two-tower baseline, the CLIP-ViT/RoBERTa backbone choices, and the pre-training and fine-tuning protocol reused for comparison.","marker":"[6]"},{"why":"BridgeTower is the immediate predecessor that introduces layer-by-layer bridges; ManagerTower is defined against it and compared with it throughout.","marker":"[7]"},{"why":"The linear combination of layers method is the starting point for the SAM and SAUM manager weights and the layer-fusion baseline.","marker":"[10]"},{"why":"LLaVA-OneVision is the MLLM baseline whose training data, multi-grid variant, and evaluation protocol the 20-dataset zero-shot study follows.","marker":"[11]"},{"why":"The multi-grid algorithm from LLaVA-NeXT is the width-side plugin whose implementation and semantic ambiguity are central to the MLLM experiments.","marker":"[12]"},{"why":"Switch Transformers supplies the Gaussian exploration noise and gating-style aggregation ideas used in AAUM and SAUM.","marker":"[22]"},{"why":"The zero-init attention adapter motivates zero-initializing manager weights to avoid disturbing a pre-trained LLM during early training.","marker":"[42]"}],"fun_headline_variants":["Manager aggregates unimodal experts to improve VLMs and MLLMs","Adaptive manager combines encoder layers, lifting vision-language performance","Small plugin adaptively fuses unimodal knowledge, boosting VLM and MLLM scores","Manager and multi-grid synergy adds visual detail from depth and width"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's claim of consistent gains on 20 datasets assumes that the datasets it kept after removing 'few some downstream datasets' are representative of the full evaluation suite, so if the removals systematically dropped cases where managers helped least, the reported averages overstate the general result.","fun_headline_variants_meta":{"raw":{"variants":["Manager aggregates unimodal experts to improve VLMs and MLLMs","Adaptive manager combines encoder layers, lifting vision-language performance","Small plugin adaptively fuses unimodal knowledge, boosting VLM and MLLM scores","Manager and multi-grid synergy adds visual detail from depth and width"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000277,"raw_usage":{"total_tokens":1739,"prompt_tokens":1126,"completion_tokens":613,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":742,"completion_tokens_details":{"reasoning_tokens":536}},"tokens_in":742,"tokens_out":613,"duration_ms":6988,"temperature":1.0,"reasoning_tokens":536,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:03:20.312195+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the four-baseline comparison on all datasets in the original LLaVA-OV zero-shot suite with no datasets removed and report per-dataset deltas; if the +Grid+Manager advantage over +Grid shrinks to zero or becomes negative once the excluded datasets are included, the 20-dataset averaging claim collapses.","supporting_citations":[],"review_version":1}