{"id":"b8346a8f-273a-4b7c-ae91-82a58951e174","arxiv_id":"2505.24143","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CrossICL reuses demonstrations from similar source tasks, rewriting them into the target task's format, and improves zero-shot ICL on Super-NI without target-task annotations.","lead":"This paper introduces CrossICL, a method that performs in-context learning with demonstrations borrowed from other tasks, then rewrites those demonstrations to fit the user's current task. On 875 Super-NI tasks across six language models it improves average performance without requiring users to write any examples.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported gains are not compute-matched: CrossICL spends 3n+1 LLM calls per query versus one for zero-shot baselines, so the central ICL claim is confounded.","rationale":"The reader's weakest assumption was the two-part query format, which is a real scope limitation but does not threaten the internal validity of the Super-NI experiments. The more load-bearing concern for the central causal claim is compute confounding: CrossICL deliberately spends many more LLM calls per query than the baselines, and no control matches that budget. The paper's ablations (Table 2) show that adaptation helps over no adaptation, and the exact-match check in Appendix G and multi-model consistency are good evidence against a ROUGE-L artifact, but none of these address the inference-budget imbalance. This is a concrete, addressable concern rather than a fatal flaw, so I do not change the reader's CONDITIONAL verdict. The decisive check is the compute-matched baseline described above.","tokens_in":25850,"tokens_out":4945,"duration_ms":63174,"concrete_test":"Re-run the main Llama3.1-8B experiment on the 119 Super-NI test tasks with two controls: (1) Zero-Shot with self-consistency over 16 samples and majority voting, matching CrossICL's 16 calls per query; (2) Self-ICL given the same budget as Progressive Task Adaptation, i.e., 3 generated demonstrations per query, each refined and labeled with the same number of LLM calls as CrossICL. If either control closes more than half of the reported 0.036 average ROUGE-L gap (0.532 to 0.568), the central claim is not supported; if the gap persists, the compute confound is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"CrossICL's pipeline (Sections 3.2 and 3.3) makes, for each target query, n separate Prompt-1 rewrites, n Prompt-2 refinements, n Prompt-3 label generations, and one final Prompt-4 ICL call (n=5, so 16 LLM calls), plus embedding retrieval. The Table 1 baselines are Zero-Shot and Zero-shot-CoT, which use one call, and Self-ICL, which does not spend this per-query adaptation budget. The central claim, that 'CrossICL consistently outperforms the baseline methods' (Section 4.4), is therefore compatible with a simpler explanation: extra test-time compute and the repeated target-task description in every demonstration improve ROUGE-L independently of cross-task transfer. Appendix D only compares annotation time against manual annotation, not inference cost against baselines. The final Prompt-4 repeats 'Task Instruction: [target task description]' n times, which can act as instruction amplification. Without a compute-matched or budget-controlled baseline, the causal attribution to cross-task demonstration transfer is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CrossICL, a zero-annotation paradigm for in-context learning in which demonstrations are taken from existing source-task datasets and transferred to a target task. The method has two stages: Minimum Gap Selection (§3.1) retrieves a source task whose task-description embedding is closest to the target query and then selects demonstrations whose input embeddings are closest, followed by Progressive Task Adaptation (§3.2) that rewrites each source query into the target-task format, refines the rewrite, and generates a label by referring to the original source demonstration. The aligned demonstrations are then used for standard ICL (§3.3). Experiments on 875 Super-NI tasks across six LLMs, including GPT-4o, report consistent ROUGE-L gains over Zero-Shot, Zero-shot-CoT, and Self-ICL, with additional analyses of selection criteria, ablations, an error taxonomy, source-task similarity, and combination with query-supervised ICL.","tokens_in":26032,"tokens_out":3839,"duration_ms":50573,"significance":"If the central claim is correct, CrossICL is a useful extension of ICL: it removes the need for target-task demonstrations and shows that existing labeled source tasks can reliably guide inference on new tasks. The paper has notable strengths: it evaluates across six model families, uses a large benchmark with 875 tasks, validates the ROUGE-L metric with exact-match checks in Appendix G, provides a detailed ablation of the adaptation stages in Table 2, and offers a concrete error taxonomy in §4.7. The selection-criteria analysis in §4.5 and the source-similarity analysis in §4.8 are also informative. These strengths make the empirical phenomenon worth investigating. However, the main comparison is not compute-matched, and the paper lacks error bars or significance tests, so the central claim of consistent improvement is not yet established at the standard expected for a strong empirical NLP contribution.","major_comments":[{"comment":"The main results are confounded by an unequal inference budget. For each query, CrossICL makes n Prompt-1 calls, n Prompt-2 calls, n Prompt-3 calls, and one Prompt-4 call (n=5 in the experiments, so 16 LLM calls per query), plus embedding retrieval, whereas the Zero-Shot and Zero-shot-CoT baselines use one call. Self-ICL also does not appear to spend this per-query adaptation budget. The paper's central claim in §4.4 that 'CrossICL consistently outperforms the baseline methods' is therefore compatible with a simpler explanation: additional test-time compute, or the repeated inclusion of the target task instruction in every demonstration, improves ROUGE-L independently of cross-task transfer. Appendix D only compares annotation time against manual annotation, not inference cost against baselines. To support the causal attribution to cross-task demonstration transfer, the authors should add a compute-matched or budget-controlled baseline—for example, zero-shot with self-consistency using the same number of LLM calls, or Self-ICL given the same number of generation/refinement calls—and report per-query cost and latency.","section":"§3.2–§3.3, Table 1"},{"comment":"No error bars, confidence intervals, or significance tests are reported for any of the main aggregate results. The paper states that 'the average score of three rounds of predictions' is reported (§4.1), but no variance is given across rounds or across tasks. Many of the claimed gains are small in absolute terms (for example, Llama3.1-8B average ROUGE-L rises from 0.532 to 0.568, and Deepseek-7B from 0.433 to 0.449 in Table 1), and the per-category differences are sometimes much smaller or negative (e.g., Extraction for Deepseek-7B drops from 0.451 to 0.428). Since the central claim is that CrossICL consistently outperforms all baselines, the authors should report standard errors or bootstrap confidence intervals and, where appropriate, paired significance tests over tasks or runs.","section":"Tables 1, 2, 4, 6; Figures 3–5"},{"comment":"The method assumes that every user query can be cleanly decomposed into a task description and a task input, and that the task description is available at inference time. The Limitations section acknowledges this constraint and suggests, without evaluation, that an LLM can transform arbitrary queries into this two-part form. Because the paper's motivating scenario is real-world users who do not provide demonstrations, the unvalidated query-transformation step is load-bearing for the claimed practical applicability. To make the scope of the contribution clear, the authors should either evaluate the automatic query decomposition step or explicitly restrict the claims to settings where the two-part format is available or readily provided.","section":"§2 and Limitations"}],"minor_comments":[{"comment":"There is a typo: 'al six types of LLMs' should be 'all six types of LLMs'.","section":"§1, first paragraph of findings"},{"comment":"The same paper appears twice as Gao et al. (2024a) and Gao et al. (2024b) with different venues but overlapping content; please unify the citation or clarify the distinction.","section":"References"},{"comment":"The ablation label 'w/o Src in All' is ambiguous: it means that no source-task information is referenced anywhere in the adaptation, but the target task description and target query are still used. Please state this explicitly in the table caption or in §4.6.","section":"Table 2"},{"comment":"The efficiency discussion reports only annotation time and performance for four example tasks. It would be more informative to also report the wall-clock inference time per query for CrossICL versus each baseline on the same tasks.","section":"Appendix D"},{"comment":"The exact-match results are reported only for four task categories; please state clearly that generation and rewriting are excluded because they are not suitable for exact match, as is already noted, but also indicate whether the same held-out test split and sample size are used as in Table 1.","section":"Appendix G and Table 6"}],"recommendation":"major_revision","confidential_remarks":"The work is a serious empirical study with a large-scale evaluation and a genuinely interesting idea. The main barrier to acceptance is the compute-matching issue: as written, the headline comparison to zero-shot baselines is not a controlled test of cross-task transfer. This is fixable, but it requires additional experiments rather than just rewriting. I would also urge the editor to require the authors to release code and exact prompts for reproducibility, since the paper relies on several bespoke prompt steps and the exact-match validation is only in an appendix."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: genuinely new ICL setup — cross-task demonstration transfer with no target-task labels — and the experiments are broad, mostly careful, and consistent across six models. The main results are probably real, but the paper doesn't yet rule out that a big chunk of the gain comes from spending far more LLM inference per query than the baselines, rather than from cross-task transfer. That needs fixing before I'd trust the headline claim.\n\nWhat's new: the two-stage alignment — selecting source demos by task-description and input similarity, then progressively adapting each demo into the target task's format — is clean and sensible. The evaluation on 875 Super-NI tasks with six LLMs is a serious effort. The selection-criteria study (12 criteria) and the interference taxonomy (7 error types) are useful and are the most citable parts. The ablations are well chosen, and the exact-match consistency check in Appendix G is a good sanity check.\n\nSoft spots, in order of importance. First, the compute confound is real. CrossICL makes n separate rewrite calls, n refinement calls, n label-generation calls, and one final ICL call — sixteen LLM calls per query at n=5. Zero-shot and Self-ICL make one. Appendix D only compares annotation time, not inference cost. The repeated 'Task Instruction' in the final prompt could be instruction amplification. A compute-matched control — e.g., Self-ICL with five generated demos, or zero-shot with budget-equivalent sampling — is missing. Without it, the central attribution is not established. Second, no variance or significance: three-round averages with no error bars. With six models the direction is consistent, but the magnitudes could be noise; easy to fix. Third, the two-part query format is a real constraint, acknowledged in Limitations, but the proposed LLM auto-extraction is not evaluated. Fourth, no code or data released for an empirical pipeline that needs replication.\n\nCitation pattern is fine; prior work is cited and differences drawn accurately. No circularity: target labels are never used for selection or adaptation.\n\nBottom line: real contribution to the zero-shot ICL subfield, thoughtful analysis, but the compute confound is load-bearing. I'd send it to review, not desk-reject, and would want a compute-matched baseline plus error bars before accepting. I'd bring it to reading group maybe, and would only cite it once the confound is addressed.","headline":"CrossICL is a genuinely new cross-task ICL setup with a solid empirical sweep, but the central claim is confounded by a large inference-budget gap between method and baselines.","tokens_in":26585,"tokens_out":2632,"would_cite":false,"duration_ms":28843,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Existing demonstrations from other tasks can serve as reliable in-context examples for a new task.","keywords":["in-context learning","cross-task transfer","zero-shot ICL","demonstration selection","task adaptation","large language models","Super-NI","cross-task interference"],"falsifier":"Run CrossICL on the Super-NI test tasks using prompts that omit the task-description part, with only raw input strings and no automatic splitting step; if performance drops to zero-shot levels, the core claim fails for the general user prompts the paper gestures at. A narrower check would remove the nearest source task by description embedding and measure whether the adaptation stage can still beat zero-shot.","tokens_in":1458,"feed_emoji":"🧩","tokens_out":1496,"duration_ms":68853,"temperature":0.7,"pith_summary":"This paper tries to establish that in-context learning does not need human-written examples for every new task: existing demonstrations from other tasks can be recycled as reliable guidance, provided the gap between source and target tasks is actively narrowed. The proposed method, CrossICL, selects the source-task demonstrations closest in meaning to the user's query and then has the language model rewrite, refine, and re-label them into the target task's format. On 875 Super-NI tasks across six LLMs, the paper reports consistent gains over zero-shot, chain-of-thought, and self-generated-demonstration baselines. If the claim holds, any repository of labelled examples becomes a reusable pool of demonstrations, removing a major practical barrier to in-context learning.","feed_headline":"Other tasks' demonstrations can replace manual ICL examples","feed_subtitle":"CrossICL selects and reshapes source-task examples, lifting six LLMs' scores on 875 Super-NI tasks.","key_machinery":"The load-bearing mechanism is a two-stage alignment strategy. Minimum Gap Selection uses an embedding model to choose, first, the source task whose task description is semantically closest to the target query's description, and then the demonstrations within that task whose inputs are closest to the query. Progressive Task Adaptation then reshapes each chosen demonstration: the LLM rewrites the source query into the target form, refines the rewrite to remove noise, and generates a label for the refined query by consulting the original source demonstration. These aligned demonstrations are prepended to the target query as standard in-context learning examples.","core_discovery":"The central discovery is that cross-task demonstration transfer works: feeding an LLM demonstrations adapted from a different task improves its answers to a target query, with no target-task annotation. Concretely, selecting the source task whose description embedding is closest to the target query and whose example inputs are closest to the query, then transforming those examples through query rewriting, refinement, and source-guided label generation, raises average Super-NI ROUGE-L from 0.532 to 0.568 on Llama3.1-8B and from 0.644 to 0.676 on GPT-4o. The paper also maps seven types of cross-task interference and shows that the method still helps even when the source task is not the most similar one.","pith_inferences":["If an LLM can reliably split arbitrary free-form prompts into task description plus task input, the method should carry over to the messy queries real users type; the paper suggests this but does not test it.","Because the selection stage only needs an embedding model and a pool of labelled examples, the recipe could transfer to other modalities, such as image or audio tasks, wherever embeddings capture input similarity.","The taxonomy of seven interference types gives a diagnostic checklist for prompt design: users could avoid cross-task confusion by explicitly separating task instruction from input in their queries.","A harder test would be to see whether CrossICL gains persist when the pool of source tasks is small or domain-distant, since the paper's gains come from a large 756-task source pool."],"forward_implications":["CrossICL outperforms zero-shot, zero-shot chain-of-thought, and Self-ICL on average across all six tested LLMs, including GPT-4o.","The method works even when the selected source task is not the most similar one available, because the adaptation stage absorbs much of the cross-task gap.","Five aligned demonstrations are near-optimal; both fewer and more demonstrations reduce the gain, mirroring standard in-context learning.","Combining CrossICL with query-supervised ICL improves that method further, because the cross-task-derived labels are more accurate than zero-shot labels.","Task-description-plus-task-input selection outperforms twelve alternative selection criteria on average, though individual task categories prefer different criteria such as output similarity or query length."],"supporting_citations":[{"why":"Establishes that in-context learning is sensitive to query-demonstration gaps and identifies input-label mapping and input distribution as what demonstrations teach, motivating the selection criteria.","marker":"Min et al. 2022"},{"why":"Disentangles task recognition from task learning and supports using task description and input similarity as the basis for minimum-gap selection.","marker":"Pan et al. 2023"},{"why":"Supplies the Self-ICL baseline and the two-part query representation the paper adopts, and is the prior method that generates demonstrations from task descriptions.","marker":"Chen et al. 2023"},{"why":"Shows that self-generated demonstrations can hurt in-context learning and proposes query-supervised ICL; used here as a baseline and as the method combined with CrossICL.","marker":"Su et al. 2024"},{"why":"Introduces the Super-NI benchmark used for all experiments, including the task pool and the ROUGE-L evaluation protocol.","marker":"Wang et al. 2022"},{"why":"Provides the 875-task split and the six-category grouping of test tasks used in the evaluation setup.","marker":"Wang et al. 2024c"},{"why":"Supplies BGE-EN-ICL, the embedding model used for source-task and demonstration selection.","marker":"Chen et al. 2024"},{"why":"Defines ROUGE-L, the primary evaluation metric used to score all tasks in the benchmark.","marker":"Lin 2004"}],"fun_headline_variants":["CrossICL borrows demos from other tasks to improve ICL","Replace manual ICL examples with cross-task demonstrations","No target labels needed: CrossICL transfers demos across tasks","CrossICL lifts average scores on 875 tasks using cross-task demos","Skip manual demonstrations by reshaping other tasks' examples"],"cache_read_input_tokens":28800,"weakest_assumption_plain":"The method assumes every user query can be split cleanly into a task description and a task input, and that this description is available or can be extracted; the paper's Limitations section admits this constraint and only suggests, without evaluating, that an LLM could do the extraction automatically.","fun_headline_variants_meta":{"raw":{"variants":["CrossICL borrows demos from other tasks to improve ICL","Replace manual ICL examples with cross-task demonstrations","No target labels needed: CrossICL transfers demos across tasks","CrossICL lifts average scores on 875 tasks using cross-task demos","Skip manual demonstrations by reshaping other tasks' examples"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000441,"raw_usage":{"total_tokens":2196,"prompt_tokens":868,"completion_tokens":1328,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":484,"completion_tokens_details":{"reasoning_tokens":1242}},"tokens_in":484,"tokens_out":1328,"duration_ms":13814,"temperature":1.0,"reasoning_tokens":1242,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:32:33.301632+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CrossICL on the Super-NI test tasks using prompts that omit the task-description part, with only raw input strings and no automatic splitting step; if performance drops to zero-shot levels, the core claim fails for the general user prompts the paper gestures at. A narrower check would remove the nearest source task by description embedding and measure whether the adaptation stage can still beat zero-shot.","supporting_citations":[],"review_version":1}