{"id":"350bf557-e972-4fd1-90ba-f5ff1649e361","arxiv_id":"2606.23196","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Intrinsic self-correction produces gains on tasks whose structure supports explicit verification, reasoning revisits, or second opinions, but not as a uniform method.","lead":"This paper analyzes intrinsic self-correction in LLMs by testing it across tasks with different revision mechanisms such as constraint verification, reasoning revisits, and word-game strategy choice. A smart generalist might read it to decide when to apply self-correction prompts instead of treating the technique as generally reliable.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.3","headline":"Gains may arise from extra tokens/context rather than the claimed revision mechanisms","rationale":"The reader's weakest_assumption is exactly the load-bearing point. The abstract-only limitation noted by the reader does not change the nature of the concern; the same control is required even with full text.","tokens_in":1642,"tokens_out":254,"duration_ms":9029,"concrete_test":"Re-run the main tables with a matched-token control that appends an equivalent number of tokens via a non-revision instruction (e.g., 'Elaborate on your reasoning without altering the final answer') and test whether the SC delta remains statistically larger; if the control matches or exceeds SC, the mechanism attribution is unsupported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that performance deltas are caused by the three identified revision modes (constraint verification, reasoning revisit, second-opinion in games) rather than by the mechanical side-effects of any second-pass prompt. Self-correction instructions necessarily increase total tokens and context length; without an ablation that holds token budget fixed while removing the revision logic (e.g., neutral continuation prompts), the task-structure dependence cannot be isolated from simple length or compute effects.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript claims that intrinsic self-correction (SC) in LLMs is not generally reliable but produces consistent gains in tasks whose structure supports specific revision mechanisms: verifying explicit constraints, revisiting complex reasoning processes, or providing a second opinion over competing strategies in word-game tasks. It supports this through experiments across multiple benchmarks and models, concluding that SC should be viewed as a task-dependent inference-time strategy rather than a uniform improvement method.","tokens_in":1723,"tokens_out":325,"duration_ms":24115,"significance":"If the empirical results hold after controlling for confounds, the work supplies a useful task-sensitive lens on SC that could reconcile prior conflicting findings on its reliability. The multi-benchmark, multi-model scope is a strength, as is the focus on mechanistic explanations tied to task structure rather than blanket claims.","major_comments":[{"comment":"The experimental sections do not report an ablation that holds total token budget and context length fixed while removing the revision logic (e.g., neutral continuation prompts of matched length). Without this control, performance deltas cannot be attributed to the three claimed revision modes rather than the mechanical effects of any second-pass prompt. This directly undermines the central claim that task structure facilitates the identified modes of revision.","section":"Experimental sections"}],"minor_comments":[{"comment":"The abstract and introduction could more precisely list the specific benchmarks and models used so readers can immediately assess the scope of the task-sensitive findings.","section":null}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback. We respond to the major comment below.","responses":[{"response":"The task-dependent pattern of results already provides evidence against a purely mechanical second-pass effect. Gains appear selectively on tasks whose structure supports verification of explicit constraints, revisits to complex reasoning, or second opinions over competing strategies, while other tasks show no improvement or degradation under the same self-correction procedure. A generic effect from any matched-length continuation prompt would be expected to produce more uniform changes across benchmarks rather than the observed alignment with task structure. We therefore maintain that the differential outcomes support attribution to the proposed revision modes. We will add a brief discussion of this point to the revised manuscript.","revision_made":"partial","referee_comment":"[Experimental sections] The experimental sections do not report an ablation that holds total token budget and context length fixed while removing the revision logic (e.g., neutral continuation prompts of matched length). Without this control, performance deltas cannot be attributed to the three claimed revision modes rather than the mechanical effects of any second-pass prompt. This directly undermines the central claim that task structure facilitates the identified modes of revision."}],"tokens_in":1213,"tokens_out":257,"duration_ms":20882,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper's core finding is that intrinsic self-correction produces gains mainly on tasks whose structure supports constraint checking, reasoning revisits, or strategy comparison in games. It frames SC as useful only under those conditions rather than as a general fix.\n\nWhat is new is the explicit task-sensitive breakdown. Recent work showed SC often fails when applied broadly; this paper tests whether success depends on the revision role the second pass can play. The experiments run across several benchmarks and models and report consistent improvements precisely where the task structure matches one of the three modes. That reframing is direct and responds to the literature it cites.\n\nThe experiments give a practical signal for when to try SC at inference time. The citation pattern is standard and pulls in the right negative results without overclaiming novelty for the mechanism itself.\n\nThe main soft spot is the one the stress test flags. Any second-pass prompt adds tokens and context length. Without an ablation that holds total tokens fixed and uses neutral continuation prompts instead of the revision instructions, the performance deltas cannot be cleanly attributed to the identified mechanisms rather than simple length or compute effects. If the full paper contains that control, the claim strengthens; if not, the task-structure dependence remains partly confounded. The reader's low soundness score tracks with this gap.\n\nThis is for groups working on inference-time reliability in LLMs who need a decision rule rather than a blanket recommendation. A reader focused on deployment trade-offs would get usable guidance from the task categories. The work shows clear empirical engagement with the question and no internal contradictions, so it merits referee time even if the ablations need tightening.","headline":"SC helps on tasks with explicit revision structures, but gains may trace to extra tokens rather than the claimed mechanisms.","tokens_in":2200,"tokens_out":392,"would_cite":false,"duration_ms":13692,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Intrinsic self-correction yields gains in large language models only when task structure supports specific revision modes.","keywords":["intrinsic self-correction","large language models","task-sensitive analysis","inference-time strategies","reasoning benchmarks","word games","prompting mechanisms","performance evaluation"],"falsifier":"Re-run the same benchmarks while holding total token count and context length fixed, then remove the self-correction instruction and measure whether the performance difference disappears.","tokens_in":2549,"feed_emoji":"","tokens_out":641,"duration_ms":17458,"temperature":0.7,"pith_summary":"The paper asks when prompting a model to revisit its own answer improves results, rather than assuming self-correction works across the board. It shows gains appear reliably when tasks let the model verify explicit constraints, step back through a chain of reasoning, or weigh competing strategies in word games. A reader cares because this turns self-correction from a hoped-for general fix into a strategy that must be matched to the task's built-in revision opportunities. Experiments across several benchmarks and models confirm the pattern holds when those structural conditions are present.","feed_headline":"Self-correction lifts LLM scores only in tasks with matching revision structure","feed_subtitle":"Gains appear when prompts can verify constraints, revisit reasoning, or compare strategies, not as a blanket improvement.","key_machinery":"Task-sensitive revision modes that align the self-correction prompt with opportunities already present in the task structure.","core_discovery":"Intrinsic self-correction can yield consistent performance gains when the underlying task structure facilitates these modes of revision: verifying explicit constraints, revisiting a complex reasoning process, or providing a second opinion over competing strategies in word-game tasks. The results indicate that self-correction functions as a task-dependent inference-time strategy whose usefulness depends on the role the revision stage can play in a given task, rather than as a uniformly reliable method for improving initial model outputs.","pith_inferences":["Prompt engineering could include an upfront check for whether a task supplies one of the three revision opportunities before applying self-correction.","The same logic might extend to agent workflows where an internal critic is added only after the environment already supplies verifiable constraints.","Training data could be filtered or augmented to emphasize tasks whose structure already rewards explicit revision steps."],"forward_implications":["Models improve on constraint-checking tasks when self-correction prompts them to verify rules they already know.","Reasoning chains benefit when the model is asked to revisit earlier steps rather than accept the first pass.","Word-game performance rises when self-correction lets the model compare alternative solution paths.","Self-correction should be applied selectively rather than as a default add-on to every prompt.","Task structure, not model size alone, determines whether the second pass adds value."],"fun_headline_variants":["Self-correction aids LLMs only via matching task revision modes","Intrinsic self-correction depends on task revision capabilities","Gains appear when tasks allow constraint or reasoning revision","Self-correction not general but task-dependent on revision role","Task structure decides if self-correction improves LLM answers"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The observed gains come from the identified revision mechanisms and not from extra tokens, longer context, or other prompting details.","fun_headline_variants_meta":{"raw":{"variants":["Self-correction aids LLMs only via matching task revision modes","Intrinsic self-correction depends on task revision capabilities","Gains appear when tasks allow constraint or reasoning revision","Self-correction not general but task-dependent on revision role","Task structure decides if self-correction improves LLM answers"]},"model":"grok-4.3","cost_usd":0.006092,"raw_usage":{"total_tokens":2853,"prompt_tokens":617,"num_sources_used":0,"completion_tokens":76,"cost_in_usd_ticks":60924500,"prompt_tokens_details":{"text_tokens":617,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2160,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":617,"tokens_out":76,"duration_ms":11644,"temperature":1.0,"reasoning_tokens":2160,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T08:31:16.431873+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Re-run the same benchmarks while holding total token count and context length fixed, then remove the self-correction instruction and measure whether the performance difference disappears.","supporting_citations":[],"review_version":1}