{"id":"87c869d6-8cff-4e64-85a4-cff61a2edbc5","arxiv_id":"2608.12724","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MAG selects in-context demonstrations by propagating relevance scores over text and image graphs, using a small set of pseudo-labeled unlabeled samples to boost few-shot multimodal accuracy.","lead":"A new method called MAG uses unlabeled image-text data to pick better examples for prompting multimodal AI models, improving few-shot accuracy on eight benchmarks. It propagates relevance scores through text and image graphs to choose which unlabeled samples to pseudo-label and which demonstrations to show.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never states that the 585-sample unlabeled pool is disjoint from the 300 test samples; without that guarantee, MAG's demonstrations may include test queries, and the reported gains could be leakage rather than semi-supervised selection.","rationale":"The reader's weakest_assumption concerns MLLM-generated description quality (Section 3.1). That is a real risk, but description noise would attenuate or corrupt MAG's gains; it does not by itself explain systematically positive results across eight benchmarks. The more load-bearing assumption is implicit in Section 4.2: the unlabeled pool must be disjoint from the test set. The paper never states this, and the evaluation protocol (Appendix D) uses substring containment, making any overlap directly rewarding. This is not an accusation of fraud; random sampling without an explicit hold-out is a common reporting gap, and code release would settle it. The reader did flag disjointness in the rationale (issue 3), so there is partial agreement, but the reader's weakest_assumption targeted a different component. Other weaknesses (no significance tests, hyperparameters tuned on evaluation benchmarks, no code release) are secondary: they affect confidence in the magnitude of gains, whereas split leakage would affect the existence of the effect. The recommended verdict is unchanged from the reader's CONDITIONAL: MAG is plausible and the ablations are internally consistent, but the split guarantee and data release are prerequisites for the central claim to be verified.","tokens_in":17979,"tokens_out":8810,"duration_ms":99155,"concrete_test":"Request the exact random split (seeds and sample indices) used for Table 1, or the released code/data. Compute the intersection between the 585 unlabeled samples and the 300 test samples, and also check for near-duplicate image/question pairs using the dataset's canonical split definitions. If any overlap exists, or if the split was drawn from the full dataset without holding out the test set, rerun the MAG and MAPLE rows of Table 1 with a strictly disjoint unlabeled pool (labeled and unlabeled drawn from the non-test portion, test never in DU) and compare the margins. If the MAG advantage over MAPLE and MMICES collapses or becomes non-significant, the central claim is an artifact of test leakage; if the margins persist, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that unlabeled data, selected via MAG's two-stage propagation, is what drives the gains. Section 4.2 says only 'we randomly sample 15 labeled examples, 585 unlabeled examples as the candidate pool, and 300 test samples.' No sentence states that the unlabeled pool and the test set are disjoint, and no code, data, or split indices are released. This is load-bearing because Stage 1 (Eq. 4) pseudo-labels the top-K unlabeled samples and Stage 2 (Eq. 8) selects top-k demonstrations from DE = DL ∪ DP. If any test query or a near-duplicate image appears in DU, that sample's pseudo-label can be the ground-truth answer, and it can be placed directly in the demonstration set. Under the substring-containment metric of Appendix D, even a partial answer in a demonstration leaks. MAPLE shares the unlabeled pool, so leakage would inflate both MAG and MAPLE relative to labeled-only baselines, but the paper's headline compares MAG against all baselines. Appendix Table 7 gives independent reason to worry: on Emotion6, MAG's pseudo-label accuracy is 40.0% versus 55.6% for random selection, so the pseudo-labels themselves are not uniformly high-quality; consistent gains across all eight tasks are more easily explained by split overlap than by uniformly better pseudo-labels.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MAG, a training-free two-stage framework for multi-modal in-context learning under label scarcity. Stage 1 builds a text-only k-NN graph over labeled and unlabeled samples, propagates relevance scores from labeled to unlabeled nodes, and pseudo-labels the top-K unlabeled samples using the MLLM. Stage 2 constructs separate visual and textual graphs over the expanded candidate pool, propagates query-conditioned relevance scores in each modality, fuses them late, and selects the top-k demonstrations. Experiments on eight vision-language benchmarks compare MAG against retrieval, pseudo-labeling, and zero-shot baselines, report better mean accuracy across all eight datasets, and include ablations on pseudo-labeling strategy, demonstration selection, modality usage, fusion strategy, and hyperparameter sensitivity.","tokens_in":18216,"tokens_out":4290,"duration_ms":42476,"significance":"If the empirical claims hold, this is a useful contribution: it provides a parameter-free, closed-form way to exploit unlabeled multi-modal data for ICL without fine-tuning, with a sparse-graph formulation that is computationally attractive. The paper ships a clearly specified algorithm, reasonable ablations, and a scalability analysis. However, the current validation has several load-bearing gaps: no explicit statement that the unlabeled pool and test set are disjoint, no significance testing on the headline gains, an internal inconsistency in the pseudo-label accuracy results on Emotion6, and hyperparameter choices informed by the same test sets used for the final numbers. These issues must be resolved before the central claim is credible.","major_comments":[{"comment":"The experimental setup never states that the 585-sample unlabeled pool is disjoint from the 300 test samples; because Stage 1 (Eq. 4) pseudo-labels the top-K unlabeled samples and Stage 2 (Eq. 8) selects demonstrations from DE, any overlap between DU and the test set would directly leak ground-truth labels into the prompt and could fully explain the reported gains. The authors must explicitly specify the data split construction (e.g., unlabeled samples drawn from the train split and test samples from the test split with no shared images) and, ideally, release the exact split indices.","section":"4.2 Implementation Details"},{"comment":"The abstract and Section 4.3 claim that MAG 'significantly' outperforms baselines, but with only 3 runs and no significance tests, many margins are within one standard deviation (e.g., EmoSet 76.3±1.4 vs. CVR-LLM 74.8±1.5; Emotion6 64.3±1.4 vs. Few-shot 63.0±1.9; GQA 59.0±3.9 vs. VICL 55.1±1.0; OKVQA 57.6±2.1 vs. VICL 56.3±1.5). Please report per-example paired tests or bootstrap confidence intervals across the three runs, and temper the 'significant' language accordingly.","section":"Table 1 (main results)"},{"comment":"On Emotion6 the manifold-guided pseudo-labeling gives lower pseudo-label accuracy (0.400) and slightly lower final accuracy (0.650) than random selection (0.556 and 0.657), which directly contradicts the paragraph's claim of a 'strong positive correlation between pseudo-label accuracy and final performance' and the main-text claim that relevance-guided pseudo-labeling 'significantly improves performance across all benchmarks.' The authors should explain why Emotion6 is an exception or qualify the universal claim.","section":"Appendix B.3, Table 7"},{"comment":"The hyperparameters (α, β, knn, pool size, label rate, demo count) are fixed after inspecting performance on EmoSet, Emotion6, and MMStar, which appear to be the same test sets used for the final evaluation in Table 1; this selection-on-test-set circularity can inflate the reported numbers on those three datasets. Please tune on held-out validation splits (or report sensitivity computed on validation folds) and state the split protocol explicitly.","section":"Section 4.7, Figure 3"},{"comment":"Stage 1 assumes that MLLM-generated descriptions faithfully represent the visual content needed for label propagation, but the paper provides no measure of description quality (e.g., human agreement or a reference-based metric) and no robustness analysis under corrupted or noisy descriptions. Because Eqs. (1) and (4) build the entire Stage-1 graph and pseudo-label selection on these descriptions, this assumption is load-bearing and should be validated.","section":"Section 3.1 (Textual Description Generation)"}],"minor_comments":[{"comment":"The heading 'Quantitave Evaluations' contains a typo; it should be 'Quantitative Evaluations.'","section":"Section 4.3"},{"comment":"Line 11 says to compute relevance scores 'via Eq. (4)', but Eq. (4) is the pseudo-labeling equation; the relevance scores are computed by the closed-form solution in Eq. (3), so the reference should be corrected.","section":"Appendix A, Algorithm 1"},{"comment":"The text encoder is named 'Contriver' but the reference is Contriever (Izacard et al., 2021); please correct the spelling throughout.","section":"Section 3.1"},{"comment":"Tables 3 and 4 report accuracy without standard deviations, unlike Tables 1 and 2, making it difficult to judge whether the modality and fusion differences are meaningful; please include variance information.","section":"Tables 3 and 4"},{"comment":"The caption should state which split is used for the sensitivity analysis; if it is the test split, this reinforces the concern in major comment 4.","section":"Figure 3 caption"}],"recommendation":"major_revision","confidential_remarks":"The most serious risk is the missing disjointness statement in Section 4.2: if the authors cannot provide split indices or a clear description of how DU and the test set are separated, the core empirical claim could be leakage. Please ask them to clarify this first. The Emotion6 row in Table 7 is also a red flag that should be resolved before acceptance, as it contradicts the paper's own correlation claim. The method itself is plausible and the writing is clear, so I see this as fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nMAG is a credible extension of graph-based pseudo-labeling to multi-modal in-context learning. The new part is the two-stage design: use text-only propagation to pick unlabeled samples worth pseudo-labeling, then late-fuse visual and textual relevance graphs to pick demonstrations. The authors show text-only propagation beats visual or combined in stage 1, and late fusion beats early/mid in stage 2. Those are concrete, non-obvious findings. The method is learning-free, cheap, and the ablations are honestly structured. The gains on TextOCR and CLEVR are large and probably real.\n\nThe main soft spot is the one that matters: the paper never states that the 585-sample unlabeled pool and the 300 test samples are disjoint. Section 4.2 says they \"randomly sample 15 labeled examples, 585 unlabeled examples as the candidate pool, and 300 test samples\" and nothing more. Under the substring-containment metric in Appendix D, if any test query or near-duplicate image is in the pool, its pseudo-label can be the ground-truth answer and that demo can leak the answer directly. This is load-bearing. The fact that MAPLE presumably shares the same pool doesn't help, because the headline comparison is MAG versus all baselines. Appendix Table 7 gives indirect reason for caution: on Emotion6 MAG's pseudo-label accuracy (40%) is worse than random (55.6%), so the pseudo-labels aren't uniformly better; the consistent gains across all eight tasks would be easier to explain by split overlap than by universally better pseudo-labels. I don't think this is necessarily fraud or even error — it could be an unstated standard convention — but the paper must state the split guarantee in the main text and, ideally, release the exact indices.\n\nWeaker issues: hyperparameters (alpha, k, beta, pool size, demo count) were tuned via sensitivity analysis on the same benchmarks used for evaluation (Figure 3), so the reported numbers include some selection effect. No significance tests, and a few margins (EmoSet 76.3 vs CVR-LLM 74.8, GQA 59.0±3.9) are within one standard deviation. No code or data release, and the pipeline depends on a proprietary model for descriptions and inference. Those are fixable in a revision.\n\nIf the split guarantee holds, this is a solid empirical contribution for label-scarce multi-modal ICL. It deserves a serious referee, but the referee should ask for a clean statement of data partitioning, validation-based hyperparameter selection, and release of code and split indices.","headline":"Plausible two-stage label-propagation method for few-shot multimodal ICL, but the undisclosed split between the unlabeled pool and test set could change the headline result.","tokens_in":18856,"tokens_out":2332,"would_cite":false,"duration_ms":22635,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Unlabeled image-question pairs, mined by graph propagation and pseudo-labeling, lift few-shot multi-modal in-context learning under label scarcity.","keywords":["in-context learning","multi-modal large language models","semi-supervised learning","label propagation","demonstration selection","pseudo-labeling","visual question answering","graph-based retrieval"],"falsifier":"Compute the overlap between Stage 1's top-ranked unlabeled samples and an oracle ranking built from ground-truth labels or human-written captions on the same eight benchmarks, then corrupt or paraphrase a random fraction of the MLLM-generated descriptions and re-run MAG. If overlap is low yet accuracy is unchanged, the paper's dependence on description quality is contradicted; if accuracy collapses under corruption, the dependence is confirmed.","tokens_in":17692,"feed_emoji":"🖼️","tokens_out":17516,"duration_ms":167483,"temperature":0.7,"pith_summary":"Multi-modal in-context learning adapts a model to a new task by showing it a few image-question-answer examples, with no weight updates, but performance depends heavily on which examples are shown. This paper claims that abundant unlabeled image-question pairs can be exploited without training: first, a relevance score is propagated on a graph built from descriptions produced by a multi-modal large language model (MLLM), isolating a compact set of unlabeled samples whose predicted answers are then used as provisional labels (pseudo-labels); second, the final demonstrations are chosen by propagating query-conditioned relevance on separate image and text graphs and combining the two scores. The reported result is consistent gains over labeled-only retrieval baselines across eight benchmarks under extreme label scarcity (15 labeled examples), with the largest gains on reasoning-heavy tasks. The paper concludes that pseudo-label quality matters more than quantity, that textual structure is the right carrier for the coarse filtering stage, and that late-fused visual and textual structure is needed for the final selection.","feed_headline":"Unlabeled data lifts few-shot multimodal AI up to 30 percent","feed_subtitle":"The method pseudo-labels only high-value unlabeled samples, then fuses image and text graphs to choose demonstrations.","key_machinery":"The carrying object is a pair of $k$-nearest-neighbour graphs (each node links to its $k$ most similar nodes) with closed-form relevance-score propagation, and the paper packages the two-stage use of that machinery under the name MAG (manifold-guided semi-supervised demonstration selection). In Stage 1, nodes are text embeddings $h_i^{(t)} = f_t([d_i, q_i])$, where $d_i$ is an MLLM-generated description and $q_i$ is the question; starting from a vector that is 1 on labeled nodes and 0 on unlabeled nodes, the relevance scores converge to $I^{(t)}_{(\\infty)} = (1-\\alpha)(I-\\alpha \\hat{W}^{(t)})^{-1} I^{(t)}_{(0)}$, and the top-scoring unlabeled nodes are pseudo-labeled. In Stage 2, the same propagation is run separately on a visual graph and a textual graph over the expanded candidate pool, each initialized from the query's nearest neighbours, and the two score vectors are combined by the late fusion $I = \\beta I^{(v)}_{(\\infty)} + (1-\\beta) I^{(t)}_{(\\infty)}$. The closed form is what makes the method training-free: no learning is involved, and computation reduces to solving a sparse linear system, which the paper reports scales near-linearly in the number of samples.","core_discovery":"The paper's central discovery is that relevance propagation on manifolds can turn an unlabeled pool into better in-context demonstrations. Specifically, the proposed MAG (manifold-guided semi-supervised demonstration selection) framework shows that closed-form label propagation on a graph of text embeddings—formed by concatenating MLLM-generated image descriptions with the task question—ranks unlabeled samples by relevance to the labeled data well enough that pseudo-labeling only the top 45 out of 585 yields larger ICL gains than random pseudo-labeling or no pseudo-labeling. It further shows that for selecting the demonstrations actually shown to the model, neither modality alone is enough: visual-only graphs mislead on language-heavy tasks such as emotion recognition and scene text reading, while text-only graphs collapse on visually grounded reasoning, and late fusion of two separately propagated relevance scores consistently beats early or mid fusion. In the paper's reported experiments, MAG outperforms the strongest baseline on all eight benchmarks, with the largest margins on CLEVR (+30.5% over the best retrieval baseline), TextOCR (+25.1%), and MMStar (+7.1%).","pith_inferences":["An unstated consequence is that the same triage pattern could generalize beyond ICL: use a cheap single-modality manifold to filter a large unlabeled pool, then reserve expensive cross-modal scoring for a shortlist; the paper's near-linear scaling measurements make this pattern practical as pools grow.","The paper does not test whether the MLLM that writes the descriptions and the MLLM that consumes the demonstrations should be the same; using one generator may select images that match that model's perceptual biases, which could be either a benefit or a confirmation-bias risk.","A testable extension left open is an adaptive pseudo-label budget: instead of a fixed 45 samples, Stage 1 relevance scores could be thresholded, and the reported accuracy peak at a 25% labeled rate suggests such a rule might transfer across tasks."],"forward_implications":["Under the paper's reported regime (15 labeled examples, 585 unlabeled, 45 pseudo-labeled), a deployment can improve multi-modal few-shot accuracy without finetuning, so the bottleneck shifts from labels to cheap MLLM calls for descriptions and pseudo-labels.","The largest reported gains on compositional and fine-grained reasoning benchmarks imply that graph-selected demonstrations matter most exactly where local similarity retrieval fails to cover the task's structure.","Random pseudo-labeling can hurt accuracy, so any adoption should keep relevance filtering; the pseudo-label budget is best spent on a compact, manifold-ranked subset rather than a large random sample.","Late fusion of separately propagated visual and textual relevance scores is the design that makes both modalities help; this recommends keeping modality-specific structure intact rather than concatenating features before graph construction."],"supporting_citations":[{"why":"Introduces the label-propagation idea that MAG reformulates as relevance propagation on unlabeled pools.","marker":"[Zhu et al., 2002]"},{"why":"Provides the closed-form propagation solution MAG uses for both stages.","marker":"[Zhou et al., 2004]"},{"why":"Supplies the pre-trained text encoder that turns descriptions plus questions into graph node embeddings.","marker":"[Izacard et al., 2021]"},{"why":"Supplies the pre-trained visual encoder used for the Stage 2 image graph.","marker":"[Radford et al., 2021]"},{"why":"Provides the multi-modal language model that generates descriptions, produces pseudo-labels, and answers the queries.","marker":"[Comanici et al., 2025]"},{"why":"Presents the graph-based pseudo-labeling baseline for text-only ICL that MAG extends to the multi-modal few-shot setting.","marker":"[Chen et al., 2025d]"},{"why":"Provides the strongest labeled-only retrieval baseline; the paper's headline gains are reported against it.","marker":"[Chen et al., 2025b]"},{"why":"Establishes the few-shot in-context learning setting and supplies the random-demonstration baseline.","marker":"[Brown et al., 2020]"}],"fun_headline_variants":["Unlabeled data lifts multimodal ICL by up to 30%","Manifold-guided pseudo-labeling sharpens few-shot multimodal learning","Graph-based selection improves multimodal in-context learning","45 unlabeled samples power a multimodal ICL improvement","Fusing text and image graphs selects better demonstrations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the generated image descriptions being faithful enough that textual similarity finds unlabeled samples worth labelling with provisional labels; if the descriptions are noisy or biased, the first stage can pick textually similar but visually irrelevant samples and the reported gain disappears.","fun_headline_variants_meta":{"raw":{"variants":["Unlabeled data lifts multimodal ICL by up to 30%","Manifold-guided pseudo-labeling sharpens few-shot multimodal learning","Graph-based selection improves multimodal in-context learning","45 unlabeled samples power a multimodal ICL improvement","Fusing text and image graphs selects better demonstrations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000748,"raw_usage":{"total_tokens":3348,"prompt_tokens":976,"completion_tokens":2372,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":2292}},"tokens_in":592,"tokens_out":2372,"duration_ms":20365,"temperature":1.0,"reasoning_tokens":2292,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:40:09.761428+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the overlap between Stage 1's top-ranked unlabeled samples and an oracle ranking built from ground-truth labels or human-written captions on the same eight benchmarks, then corrupt or paraphrase a random fraction of the MLLM-generated descriptions and re-run MAG. If overlap is low yet accuracy is unchanged, the paper's dependence on description quality is contradicted; if accuracy collapses under corruption, the dependence is confirmed.","supporting_citations":[],"review_version":1}