{"id":"4bf3c7d0-58b7-498d-bc13-b031408508b3","arxiv_id":"2605.13511","paper_version":3,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Many-shot CoT-ICL improves when demonstrations are ordered for smooth conceptual progression, with CDS delivering up to 5.42 percentage-point gains on math tasks using 64 examples.","lead":"This paper examines many-shot chain-of-thought in-context learning on reasoning tasks and proposes Curvilinear Demonstration Selection to order examples for better performance. A smart generalist might read it to see how long contexts in LLMs can function as structured learning rather than simple retrieval.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Gains from CDS may not isolate ordering effect from demonstration selection","rationale":"The reader's weakest assumption directly identifies the missing attribution control between ordering and selection; the abstract-only basis reinforces that this remains the load-bearing gap until full experimental details are examined.","tokens_in":1682,"tokens_out":282,"duration_ms":19089,"concrete_test":"On the same math task and 64-demonstration pool used for the reported 5.42 pp result, compare CDS ordering against a random permutation of exactly those demonstrations (same model, same prompt format); if the accuracy difference falls below statistical significance across 5+ seeds, the ordering principle does not drive the headline gain.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim credits the 5.42 pp gain on the math task to CDS ordering that enforces smooth conceptual progression (principle ii) after selecting easy-to-understand demonstrations (principle i). The abstract presents CDS as a simple ordering method, yet provides no description of controls that hold the demonstration set fixed while varying only order (e.g., random permutation of the identical 64 examples). Without such isolation, the observed improvement could arise from the particular choice of demonstrations rather than the curvilinear progression, undermining the reframing of many-shot CoT-ICL as structured in-context test-time learning.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper studies many-shot chain-of-thought in-context learning (CoT-ICL) on reasoning tasks, identifying distinctive scaling properties compared to non-reasoning tasks and models. It reframes many-shot CoT-ICL as in-context test-time learning rather than pattern matching, proposes two principles (easy-to-understand demonstrations and smooth conceptual progression), and introduces Curvilinear Demonstration Selection (CDS) as an ordering method that yields up to 5.42 percentage-point gains on a math task with 64 demonstrations.","tokens_in":1786,"tokens_out":358,"duration_ms":16139,"significance":"If the reported gains are shown to arise specifically from the ordering principles rather than demonstration selection, the work would provide a substantive reframing of long-context ICL as structured test-time learning, with implications for curriculum design in prompting. The empirical result on math tasks with many shots is a concrete contribution, though its attribution requires verification.","major_comments":[{"comment":"The abstract and method description attribute the 5.42 pp gain on the math task with 64 demonstrations to CDS ordering that enforces smooth conceptual progression (principle ii) after selecting easy demonstrations (principle i). No mention is made of controls that hold the demonstration set fixed while varying only the order (e.g., random permutation of the identical 64 examples versus CDS order). Without such isolation, the performance improvement cannot be unambiguously credited to the curvilinear progression rather than the choice of demonstrations, weakening the claim that many-shot CoT-ICL functions as structured in-context test-time learning.","section":"Abstract / CDS description and experimental results"}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for this constructive comment on isolating the contribution of ordering. We agree that the current presentation does not fully separate the effects of demonstration selection from ordering, and we will add the requested control experiments in the revision.","responses":[{"response":"We agree that an explicit ablation holding the demonstration set fixed while varying only the order is necessary to attribute gains specifically to the curvilinear progression. CDS currently combines selection of easy-to-understand examples (principle i) with curvilinear ordering (principle ii). In the revised manuscript we will add results comparing (a) the CDS-selected set in random order versus (b) the same set in CDS order, on the same math task with 64 demonstrations. These controls will be reported alongside the existing results, with updated discussion clarifying the separate contributions of selection and ordering.","revision_made":"yes","referee_comment":"[Abstract / CDS description and experimental results] The abstract and method description attribute the 5.42 pp gain on the math task with 64 demonstrations to CDS ordering that enforces smooth conceptual progression (principle ii) after selecting easy demonstrations (principle i). No mention is made of controls that hold the demonstration set fixed while varying only the order (e.g., random permutation of the identical 64 examples versus CDS order). Without such isolation, the performance improvement cannot be unambiguously credited to the curvilinear progression rather than the choice of demonstrations, weakening the claim that many-shot CoT-ICL functions as structured in-context test-time learning."}],"tokens_in":1342,"tokens_out":335,"duration_ms":18153,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper's core move is to treat many-shot CoT-ICL as structured test-time learning instead of pattern matching, then give two ordering principles and a simple CDS method that reportedly adds 5.42 points on a math task with 64 shots. That reframing and the CDS proposal are the genuinely new pieces.\n\nIt does a useful job laying out how many-shot behavior differs on reasoning versus non-reasoning tasks and across model types. The two principles (easy-to-understand demos first, then smooth conceptual progression) are stated plainly and the curvilinear selection is easy to implement.\n\nThe soft spot is exactly the one the stress-test flags. The abstract credits the gain to the ordering that enforces progression, yet it gives no sign that the authors held the 64 demonstrations fixed and only permuted their order. Without that control, the improvement could come from which examples were chosen rather than how they were sequenced. That gap weakens the claim that the work shows structured in-context learning rather than better selection. Soundness is therefore limited until those controls appear.\n\nThe paper is aimed at people who already work on ICL and prompting; anyone running many-shot experiments could try CDS in an afternoon. It is not yet strong enough for a serious referee because the central empirical claim rests on an unisolated comparison. I would not bring it to a reading group yet and would not cite it until the ordering effect is cleanly separated from selection.","headline":"CDS gives a modest gain on one math task but the abstract leaves the ordering effect unisolated from demo selection.","tokens_in":2238,"tokens_out":363,"would_cite":false,"duration_ms":12855,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Ordering demonstrations for smooth conceptual progression turns many-shot CoT-ICL into effective test-time learning on reasoning tasks.","keywords":["many-shot in-context learning","chain-of-thought","demonstration ordering","reasoning tasks","test-time learning","curvilinear selection"],"falsifier":"An experiment on the same math task with 64 demonstrations that uses random ordering or a non-curvilinear selection method and measures whether the 5.42-point gain disappears.","tokens_in":2597,"feed_emoji":"","tokens_out":645,"duration_ms":19217,"temperature":0.7,"pith_summary":"The paper examines many-shot chain-of-thought in-context learning on reasoning tasks and identifies properties distinct from non-reasoning cases. It interprets these as evidence that many-shot CoT-ICL functions as in-context test-time learning, leading to two principles: demonstrations should be easy for the model to understand and should be ordered to support smooth conceptual progression. Guided by these principles the authors introduce Curvilinear Demonstration Selection, an ordering method that produces gains of up to 5.42 percentage points on a math task with 64 demonstrations. The work reframes long context windows as structured curricula rather than retrieval buffers.","feed_headline":"Curvilinear ordering lifts many-shot CoT-ICL by 5.42 points on math","feed_subtitle":"Treating demonstrations as a smooth conceptual curriculum improves reasoning when examples progress from easy to complex for the model.","key_machinery":"Curvilinear Demonstration Selection (CDS), an ordering method that arranges demonstrations to support smooth conceptual progression from easier to more complex examples.","core_discovery":"Viewing many-shot CoT-ICL as in-context test-time learning rather than scaled pattern matching yields the principles that demonstrations should be easy to understand and ordered for smooth conceptual progression; Curvilinear Demonstration Selection based on these principles produces up to a 5.42 percentage-point gain on a math task with 64 demonstrations.","pith_inferences":["The curvilinear ordering idea might extend to other reasoning domains if an automatic way to measure conceptual progression can be defined.","Models that already receive curriculum-style training during pretraining could show larger or smaller gains from CDS at test time.","Whether CDS helps on non-reasoning tasks depends on whether a meaningful progression dimension exists for those tasks."],"forward_implications":["Many-shot CoT-ICL on reasoning tasks benefits when the context window is treated as a learning curriculum.","Performance on math reasoning improves when demonstrations follow a curvilinear ordering that matches the model's understanding.","Long context windows can serve as structured test-time curricula beyond simple pattern retrieval.","Distinctive scaling behaviors appear for both non-reasoning and reasoning-oriented models under many-shot CoT-ICL."],"fun_headline_variants":["Curvilinear demo selection yields 5.42 point math gain in many-shot CoT-ICL","Many-shot CoT-ICL as test-time learning via ordered easy demos","Demo ordering for smooth progression gives 5.42 point math CoT-ICL gain","Long context as structured curriculum improves many-shot reasoning ICL"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The performance gains come from the proposed ordering principles rather than from other aspects of how the demonstrations were chosen or from task-specific factors.","fun_headline_variants_meta":{"raw":{"variants":["Curvilinear demo selection yields 5.42 point math gain in many-shot CoT-ICL","Many-shot CoT-ICL as test-time learning via ordered easy demos","Demo ordering for smooth progression gives 5.42 point math CoT-ICL gain","Long context as structured curriculum improves many-shot reasoning ICL"]},"model":"grok-4.3","cost_usd":0.008131,"raw_usage":{"total_tokens":3682,"prompt_tokens":644,"num_sources_used":0,"completion_tokens":86,"cost_in_usd_ticks":81312000,"prompt_tokens_details":{"text_tokens":644,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2952,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":644,"tokens_out":86,"duration_ms":30025,"temperature":1.0,"reasoning_tokens":2952,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-30T21:31:49.136672+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"An experiment on the same math task with 64 demonstrations that uses random ordering or a non-curvilinear selection method and measures whether the 5.42-point gain disappears.","supporting_citations":[],"review_version":2}