{"id":"3e982007-2f18-4895-a188-5146f7cf915c","arxiv_id":"2608.02830","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Many-shot in-context learning in vision-language models can collapse accuracy as demonstrations accumulate, and the failure is causally localized to the vision-language integration pathway, where a small adapter repairs it.","lead":"Adding more image-label demonstrations to a vision-language model can sharply reduce its accuracy, sometimes below the level of random guessing, a failure the authors call in-context collapse. The paper shows the collapse is an integration failure at the vision-language interface, localizes it to specific layers, and offers a cheap adapter-based vaccine that transfers to new tasks.","discovery_kind":"unification","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The remap 'contamination-free' assumption is violated by the paper's own Table 1 (zero-shot remap accuracy up to 0.50), so the Lrn axis and vaccine transfer may partly reflect token priors, not in-prompt binding.","rationale":"I agree with the reader that the contamination-free measurement of learning is the most load-bearing assumption. My read of the paper strengthens this concern by pointing to an internal inconsistency: the paper defines remap accuracy at K=0 as approximately chance 'by construction' (Sec. 3.1), yet Table 1 lists zero-shot remap values as high as 0.50 on the supposedly contamination-free shapes task. If no demonstrations are present and the labels are arbitrary, above-chance accuracy cannot arise from in-prompt binding; it must come from a token-class prior or from a measurement artifact, either of which would invalidate the clean separation between retrieval and learning that Definition 2 and the vaccine's transfer claims rely on. The central localization claim (Table 3) is more robust to this concern because it is comparative, but the interpretation of the connector/early rescue as 'restoring genuine learning' and the vaccine's reported transfer to unseen families both inherit the contaminated Lrn axis. The paper has real strengths: the behavioral collapse is documented across a panel with well-formed-output controls, the lesion-and-rescue uses a matched-capacity control and cross-architecture replication, the continual-learning dissociation is a useful independent result, and code/data are promised. But the token-prior control is missing, and the paper's own zero-shot remap numbers suggest it is needed. Since the reader already marked the paper CONDITIONAL, and my concern is the same one the reader identified, I do not move the verdict; I would make the token-prior control an explicit condition of acceptance.","tokens_in":33591,"tokens_out":6360,"duration_ms":61917,"concrete_test":"Re-run the remap protocol on Qwen2-VL-2B, Qwen2-VL-7B, and Qwen2.5-VL-3B with per-task random token strings (e.g., 'q7z', 'xw2', 'mr4', 'pl9') sampled independently for each task and each seed, using at least 200 class-balanced held-out queries per cell, and additionally measure K=0 remap accuracy with no demonstrations. If K=0 remap accuracy falls to 0.25 within sampling error, the fixed alpha/beta/gamma/delta tokens carry a usable prior and the current Lrn and vaccine-transfer numbers are contaminated; if K=0 remains at chance and Table 4 transfer gains persist, the contamination concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that remap accuracy measures only in-prompt rule learning rests on Sec. 3.1's assertion that the fixed tokens alpha/beta/gamma/delta are 'arbitrary' and carry no pretraining association, and on Definition 2's claim that 'any above-chance remap accuracy can come only from the demonstrations.' The paper's own data contradict this premise: Table 1 reports remap accuracy at K=0 (no demonstrations at all) as 0.50 for Qwen2-VL-7B on shapes, 0.48 for Qwen2.5-VL-7B on shapes, 0.44 for Qwen2.5-VL-3B on shapes, and 0.40 for LLaVA-OV-7B on CIFAR-4, all far above the 0.25 chance level. With no demonstrations and no in-prompt mapping, above-chance accuracy is impossible unless the model has a usable prior linking the fixed tokens to image classes, the exact contamination the verbalizer was designed to exclude. Because the same four tokens are reused across every task and every model, any such token-class prior would inflate the Lrn axis for all models, compress the separation between 'retrieval-only' and 'learner' regimes, and inflate the vaccine's transfer scores (Table 4), since the vaccinated model could partly succeed by exploiting the same fixed-token prior rather than by acquiring a general demonstration-binding skill. The lesion-and-rescue asymmetry in Table 3 is less exposed, because it compares relative remap accuracy across adapter loci on the same token set, but the mechanistic interpretation that the connector adapter 'restores genuine learning' depends on the same contamination-free premise. The paper provides no control with randomly sampled novel token strings per task, and the small query counts (25 per panel cell) make the observed K=0 departures from chance difficult to dismiss as noise.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a phenomenon it calls in-context collapse in vision-language models: for several VLMs, accuracy on classification and VQA tasks falls as in-context demonstrations are added, sometimes below chance, even though outputs remain well-formed. It introduces a robustness-learning dissociation, measured via 'remap' tasks using supposedly arbitrary tokens, and claims to localize the collapse to the vision-language integration pathway (connector and early/mid transformer layers) using a parameter-matched lesion-and-rescue. It further proposes CircA, whose main component is a 'vaccine': a one-time adapter trained on a single synthetic remap task that, the authors claim, transfers to unseen task families. The paper presents a large experimental corpus across 11 open VLMs and several frontier API models.","tokens_in":33805,"tokens_out":10617,"duration_ms":86427,"significance":"If the claims hold, the paper offers three substantial contributions: a novel, reproducible, graded failure mode for many-shot multimodal ICL; a clean dissociation of robustness from genuine in-context learning; and an equal-capacity causal intervention suggesting the collapse is editable at a specific interface rather than a diffuse scale artifact. The cross-architecture replication and the integration-vs-consolidation dissociation are valuable. The code release is commendable. However, the central measurement premise—that the fixed token set (alpha/beta/gamma/delta) is contamination-free—is contradicted by the paper's own K=0 remap data, which threatens the learning-axis definition and the vaccine's transfer interpretation. The lesion-and-rescue asymmetry itself is less exposed, but the mechanistic interpretation depends on the same premise.","major_comments":[{"comment":"The claim that A_rmp(0)≈1/C 'by construction' (Sec. 3.1) is contradicted by the paper's own Table 1, which reports K=0 remap accuracy of 0.50 (Qwen2-VL-7B, shapes), 0.48 (Qwen2.5-VL-7B, shapes), 0.44 (Qwen2.5-VL-3B, shapes), and 0.40 (LLaVA-OV-7B, CIFAR-4), all far above the 0.25 chance level. With no demonstrations, above-chance remap accuracy is impossible unless the fixed token set carries a pretraining association with the image classes—exactly the contamination the verbalizer was designed to exclude. Consequently, Definition 2's assertion that 'any above-chance remap accuracy can come only from the demonstrations' is empirically false, and the learning axis Lrn = A_rmp(K_max) − A_rmp(0) uses a contaminated baseline. This affects the three-regime classification and the labeling of models as 'genuine learners.' Please provide a token-prior control (e.g., novel random strings per task per model) and re-analyze the learning axis after removing the token-prior component, or explicitly quantify the contribution of token priors to the reported gains.","section":"Sec. 3.1, Definition 2; Table 1"},{"comment":"The vaccine transfer results are measured with the same fixed token set used in training. Given the K=0 contamination shown in Table 1, the improvements on held-out CIFAR-4 (0.34→0.71) and Fashion-4 (0.18→0.60) may partly reflect exploitation of token-class priors rather than acquisition of a general integration skill. This is load-bearing because the central claim of the vaccine is transfer to unseen task families. Please include a control in which the vaccine is trained and evaluated on different, task-specific random token sets, or otherwise demonstrate that the transfer gain cannot be explained by retrieval of fixed-token priors.","section":"Sec. 4.3.1, Table 4"}],"minor_comments":[{"comment":"The phrases 'anin-context collapse' and 'thecollapse-prone' contain missing spaces; please fix these typos along with any similar spacing errors elsewhere in the text.","section":"Abstract and Sec. 1"},{"comment":"The notation 'K= max' is unclear; please specify the shot-count maximum for each model and use a consistent arrow notation (e.g., '0→max') throughout the table.","section":"Table 1 header"},{"comment":"The caption refers to a 'shaded band' in the top row being magnified in the bottom row, but the bottom row is a separate zoomed panel and no shaded band appears in the figure; please clarify or add the band to match the description.","section":"Fig. 6 caption"},{"comment":"The y-axis label 'Forgetting (less negative = better)' is confusing because the plotted values are positive; please state explicitly that the figure plots −BWT (the negative of backward transfer) or adjust the label and sign convention accordingly.","section":"Sec. 4.4.3 and Fig. 18"},{"comment":"The continual-learning protocol says 'CIFAR-100 into five disjoint 4-class tasks,' which uses only 20 of the 100 classes; please state this explicitly to avoid the impression that the full CIFAR-100 benchmark is used.","section":"Sec. 4.1.9"},{"comment":"Reference [36] is incompletely formatted as 'Olsson C, others, Olah C'; please supply the full author list.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is ambitious, and the phenomenological findings are interesting. However, the internal contradiction between the contamination-free axiom and the K=0 remap data in Table 1 is a serious flaw that must be addressed before the learning-axis and vaccine claims can be accepted. The lesion-and-rescue asymmetry (Table 3) is more robust to this concern, so the core localization may survive, but the framing as a restoration of 'genuine learning' is not yet clean. I also note the high proportion of self-citations (refs [6,7,8,18,40] are all by the author); while not improper, the authors should ensure the related-work section clearly positions these prior contributions relative to the new claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this paper demonstrates a real and counterintuitive failure in several VLMs: accuracy falls as demonstrations accumulate, and on Qwen2-VL-2B it falls from 0.94 to 0.44 with zero malformed outputs, front-loaded at the first shot, and coupled to a recency-copy signature. That part is solid. Second, the causal localization — a rank-8 adapter on the connector/early layers restores remap accuracy from 0.39 to 0.91 at K=16 while an equal-capacity late adapter drives it to 0.08 — is the most interesting result, and the equal-capacity control makes the asymmetry persuasive.\n\nThe robustness-learning dissociation and the transferable vaccine are genuine contributions. The paper is honest about its limits, and the integration-not-consolidation dissociation (late layers are best for durable weight updates) is a useful design rule.\n\nThe soft spot is the 'contamination-free' remap measurement. Table 1 reports K=0 remap accuracy of 0.50 for Qwen2-VL-7B on shapes and 0.40 for LLaVA-OV-7B on CIFAR — far above chance. With no demonstrations, above-chance accuracy is only possible if the fixed tokens alpha/beta/gamma/delta carry a usable pretraining association. That violates the paper's own Definition 2 and Sec. 3.1. It does not kill the lesion-and-rescue result, which compares loci on the same token set, but it weakens the Lrn axis and the vaccine transfer numbers, which may partly reflect token priors rather than a general demonstration-binding skill. The fix is straightforward: run the remap condition with randomly sampled novel token strings per task and re-report. Without that control, the regime classification and vaccine headline are not fully clean.\n\nMinor issues: the abstract overstates sub-chance collapse for the clean case; the frontier standard curve is single-seed; Table 1 lacks error bars; and Sec. 4.1.6 says the adapter loss supervises the 'true class name' while Eq. (5) uses ρ(y_q).\n\nBottom line: this deserves a serious referee. The localization result is likely to survive, and the paper is a good candidate for the multimodal ICL and mechanistic interpretability communities, but the token-prior control is a required revision.","headline":"A genuinely interesting causal localization of a real VLM in-context failure, with a cheap transferable fix, but the paper's own zero-shot remap numbers undermine its 'contamination-free' learning measure.","tokens_in":34539,"tokens_out":6495,"would_cite":true,"duration_ms":65963,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that many-shot in-context learning in vision-language models can collapse below chance as demonstrations accumulate, and that the failure is a localized, repairable deficit of the vision–language integration pathway.","keywords":["in-context learning","vision-language models","many-shot prompting","mechanistic interpretability","low-rank adaptation","catastrophic forgetting","continual learning","integration failure"],"falsifier":"Run the remap probe with freshly generated random token strings per task instead of the reused 'alpha' through 'delta' set. If above-chance remap accuracy survives with novel tokens and the vaccine still transfers, the learning axis is clean; if accuracy drops to chance, the measured learning and transfer are inflated by token priors.","tokens_in":33182,"feed_emoji":"📉","tokens_out":7576,"duration_ms":60680,"temperature":0.7,"pith_summary":"This paper tries to establish that many-shot in-context learning in vision-language models is not the free lunch it is assumed to be: for a substantial subset of VLMs, accuracy falls as correctly labelled image–label demonstrations are added, sometimes below chance while the model still emits well-formed labels. It further claims that this 'in-context collapse' is a localized failure of the vision–language integration pathway, not of the readout, and that a small low-rank adapter placed at that locus restores genuine in-context learning and transfers to unseen tasks. If correct, the result would change where practitioners validate many-shot prompting and where they should spend adaptation capacity.","feed_headline":"More demos can wreck VLM accuracy; one adapter restores it","feed_subtitle":"Some VLMs get worse with more examples; a one-time adapter at the vision–language interface restores learning","key_machinery":"The central machinery is a depth partition of the model into an integration locus I (connector plus early/mid language-model blocks) and a readout locus R (late blocks), together with a remap verbalizer that replaces class names with arbitrary tokens so above-chance accuracy can only come from binding the in-prompt mapping. The causal probe is a low-rank adapter attached to one region at a time, trained only on remap episodes with all original weights frozen, then re-measured on the learning curve. What this does is turn a behavioral failure into an editable, location-specific computation: the capacity-matched asymmetry (integration rescues, readout does not) is what localizes the collapse.","core_discovery":"On the paper's own terms, the central discovery is the integration hypothesis (Definition 3): the in-context collapse is a deficit of the integration computation localized to the vision–language interface (connector plus early/mid language-model layers), not of the readout. As demonstrations accumulate, the integration pathway is overwhelmed and the model reverts to its prior or to copying the most recent label instead of binding the in-prompt mapping. The support is a parameter-matched lesion-and-rescue: a rank-8 adapter on the connector lifts contamination-free remap accuracy from 0.39 to 0.91 at 16 shots, while an equal-capacity late-readout adapter drives it to 0.08. A one-time vaccine trained on one synthetic remap task transfers this repaired capability to unseen CIFAR and Fashion task families (chance to 0.71 and 0.60).","pith_inferences":["Editorial extension: if the token-prior assumption holds, a natural next step is predicting collapse-proneness from weight statistics, such as the rank or conditioning of the connector-to-early-layer Jacobian, so models could be pre-vaccinated at release time; the paper sketches this direction but does not test it.","Editorial extension: the copy-rate diagnostic could be deployed online as a serving-time monitor that stops adding demonstrations when the model starts echoing the last label, even without ground-truth labels.","Editorial extension: because a training-free task-vector injection fails on collapse-prone models, the framework predicts that vaccine-then-inject ordering is necessary; this is testable on other architectures and connector families."],"forward_implications":["Many-shot multimodal ICL is not monotonically helpful: more demonstrations can destroy competence a model already had, so deployment pipelines should test the full K-curve per model rather than assume more is better.","Robustness to demonstrations and the ability to learn a novel rule in context are separable axes; the three regimes give practitioners a diagnostic instead of a single 'does ICL work' answer.","The collapse is causally removable at the integration locus: a small adapter on the connector and early/mid layers restores in-context learning, so the fix is cheap and offline.","A one-time vaccine trained on one synthetic remap task confers transferable collapse-resistance to unseen task families, implying integration capacity is a general skill rather than a task-specific lookup.","The integration locus is not the consolidation locus: late readout layers give more durable weight-based consolidation with fewer parameters, so fast prompting and slow weight updates belong in different places."],"supporting_citations":[{"why":"The prior work reporting that VLMs 'mimic rather than reason' and that accuracy degrades with more shots; it is the behavioral baseline this paper sharpens into the collapse and separates into three regimes.","marker":"[20]"},{"why":"The many-shot ICL study reporting log-linear improvements on closed models; it supplies the 'more is better' view that the collapse directly contradicts.","marker":"[22]"},{"why":"Supplies the low-rank adapter mechanism used for the lesion-and-rescue and the vaccine.","marker":"[17]"},{"why":"Established few-shot multimodal ICL via cross-attention; it defines the architectural context for the integration hypothesis.","marker":"[2]"},{"why":"Shows multimodal task vectors can relocate demonstration content into activation edits; it motivates the inject path and the integration-locus edit.","marker":"[19]"},{"why":"Finds cross-modal information flow is dominated by middle layers; it correlates with the integration locus identified by the intervention.","marker":"[23]"},{"why":"Argues visual information becomes decision-relevant only in deep layers; the paper refines this by separating readout from integration.","marker":"[35]"},{"why":"Documents a cross-modal gap in entity knowledge extraction; part of the readability picture the integration/readout dissociation qualifies.","marker":"[12]"}],"fun_headline_variants":["More demos, worse VLMs: one connector adapter saves learning","Paradox: extra examples break VLMs; a tiny patch restores","VLM collapse from context overload? Adapter at vision-language interface","One-time vaccine stops VLM in-context collapse","Less is more for VLMs? Training an adapter beats adding demos"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole learning axis rests on the assumption that the fixed arbitrary tokens 'alpha', 'beta', 'gamma', and 'delta' carry no usable pretraining association with the task classes or image features; if any token had even a weak prior correlation with a class, above-chance remap accuracy would partly reflect retrieval rather than in-prompt learning.","fun_headline_variants_meta":{"raw":{"variants":["More demos, worse VLMs: one connector adapter saves learning","Paradox: extra examples break VLMs; a tiny patch restores","VLM collapse from context overload? Adapter at vision-language interface","One-time vaccine stops VLM in-context collapse","Less is more for VLMs? Training an adapter beats adding demos"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00071,"raw_usage":{"total_tokens":3248,"prompt_tokens":1045,"completion_tokens":2203,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":661,"completion_tokens_details":{"reasoning_tokens":2112}},"tokens_in":661,"tokens_out":2203,"duration_ms":14683,"temperature":1.0,"reasoning_tokens":2112,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:59:16.122854+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the remap probe with freshly generated random token strings per task instead of the reused 'alpha' through 'delta' set. If above-chance remap accuracy survives with novel tokens and the vaccine still transfers, the learning axis is clean; if accuracy drops to chance, the measured learning and transfer are inflated by token priors.","supporting_citations":[],"review_version":1}