{"id":"d3206dd8-2940-44aa-8ae4-dc811a7c745f","arxiv_id":"2412.15314","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"The paper introduces SCIE, a prompt optimization method that uses LLM-generated data and estimated proxy-feature effects to produce enhanced reasoning instructions; observed gains are marginal and unstable.","lead":"SCIE uses a large language model to generate its own prompt variants, estimates how textual features correlate with answer correctness, and then asks the model to write a new prompt that maximizes those effects. Gains over standard zero-shot prompts are small and inconsistent across models and datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ignorability is asserted, not established: unmeasured instruction properties (length, specificity, format) plausibly confound the Eq. (6) ATE estimates, so the causal optimization and OR reuse claims are unsupported.","rationale":"The paper's central claim is that (OR)-SCIE improves reasoning by eliciting causal abilities—specifically, by estimating average treatment effects of textual proxy features and optimizing instructions accordingly. The reader correctly identifies ignorability as the weakest assumption: the paper asserts it by requiring proxy features to be mutually independent, which is not the same as observing all confounders. Because the 'counterfactual instructions' are generated by a fallible LLM and the treatment is a high-dimensional text property, the ATE estimates are likely confounded by unmeasured attributes such as length, specificity, and format. This is load-bearing because if the ATEs are not causal, the 'actionable insights' and the OR reusability claim lose their basis; the empirical gains could be due to generic rewriting. The placebo test I propose settles whether the causal estimates are load-bearing for the empirical performance. If the placebo matches SCIE, the causal mechanism is not supported. The Appendix E ablation shows the counterfactual module matters, but it does not validate unconfoundedness. My read does not change the reader's rejection: the paper's own conceptual framing of ignorability is erroneous, and the causal claims are unsupported. I therefore agree with the reader and recommend no verdict change.","tokens_in":17884,"tokens_out":6253,"duration_ms":46350,"concrete_test":"Run a placebo SCIE: after computing ATEi for each task, shuffle the ATE vector across proxy features (or replace all ATEi with 0) and execute the 'Enhanced Instructions Generation' step with the same LLM, prompts, and budget to produce placebo-optimized instructions. Evaluate on the same test splits and student models. If placebo instructions achieve accuracy within noise of the SCIE instructions (e.g., within 1 point on GSM8K), the reported gains are not attributable to the ATE estimates and the causal mechanism claim fails. For a direct ignorability check, also compute standardized mean differences of instruction length and reading ease between treated and control groups for each proxy feature; SMD > 0.1 would indicate confounding.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing concern is that Eq. (6) does not identify a causal effect because ignorability is never established. In the 'High-quality Observational Data Generation' section, the paper equates ignorability with the proxy features being 'independent of each other,' but ignorability requires that all confounders of the treatment-outcome relationship be observed. The instructions are LLM-generated and vary along many dimensions not in the proxy set: length, lexical specificity, topic overlap with the training question, formatting, and imperative phrasing. The 'counterfactual instructions' are produced by prompting the LLM to vary one feature, which does not hold other features (observed or unobserved) fixed; changing 'Directness' inevitably changes word choice, length, and tone. Thus the conditional means in Eq. (6) are descriptive associations, not potential-outcome contrasts. The downstream instruction optimization and OR inheritance inherit this problem, so the claim that SCIE 'effectively generates instructions' through causal optimization is unsupported; the observed gains could equally arise from generic instruction rewriting. Additionally, the 'proof' in the Enhanced Instructions Generation section that a positive ATE implies a higher probability of a positive ITE is false (one large positive effect can offset many small negative ones), further undermining the theoretical grounding. These are internal flaws, not merely deviations from consensus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SCIE (Self-Causal Instruction Enhancement), a prompt-optimization method that treats LLM-generated instruction variants as observational data, extracts interpretable textual 'proxy features', estimates an Average Treatment Effect for each feature using a T-learner (implemented via an LLM with code interpreter), and then asks the LLM to compose an enhanced instruction that maximizes the estimated ATEs. An 'Object-Relational' (OR) module is added to reuse the discovered causal relationships across tasks. The authors claim improved accuracy on multiple reasoning benchmarks relative to Zero-Shot CoT, Plan-and-Solve, AgentInstruct, and OPRO, while offering interpretability and lower training cost. The evaluation covers ten datasets, three student models, and two base-instruction settings, with additional OR-reuse experiments.","tokens_in":18133,"tokens_out":6085,"duration_ms":37856,"significance":"If the causal interpretation were valid, the paper would offer a novel and interpretable alternative to gradient-based prompt optimization, with a plausible mechanism for transferring prompt improvements across tasks. The manuscript also has concrete strengths: it releases code, evaluates on a broad set of reasoning benchmarks, and includes an ablation study in Appendix E. However, the central causal claims are not established. The data-generation procedure does not satisfy the identification assumptions stated in Section 3.1, the theoretical justification for instruction optimization is flawed, and the empirical evaluation lacks error bars, confidence intervals, or significance tests. As a result, the paper currently reads as a heuristic instruction-rewriting method with a causal vocabulary, rather than a causally grounded optimization method. The significance of the claimed contribution is therefore not supported by the evidence as presented.","major_comments":[{"comment":"The paper asserts that ignorability is satisfied because the proxy features 'must be independent of each other (ignorability)'. This is incorrect: ignorability (Eq. 2) requires that all confounders of the treatment-outcome relationship are observed and controlled for, not that the treatment features are mutually independent. The 'counterfactual instructions' are LLM-generated rephrasings that vary one feature but inevitably alter many other dimensions (length, lexical specificity, topic overlap, formatting, tone), and these unmeasured dimensions plausibly affect the correctness outcome. Consequently, the conditional means in Eq. (6) are descriptive associations, not potential-outcome contrasts, and the downstream ATE-based instruction optimization and OR-reuse claims inherit this problem. This is load-bearing because the paper's stated contribution is causal prompt enhancement.","section":"3.2 (High-quality Observational Data Generation)"},{"comment":"The proof that ATEi > 0 justifies adjusting Ti from 0 to 1 for an individual is invalid. The law of large numbers only relates the average of individual treatment effects to the ATE; it does not imply that any given individual has a high probability of positive ITE, and the statement 'adjusting Ti from 0 to 1 will increase the value of ITE' is not a well-defined probabilistic consequence of a positive average. Even for binary outcomes, the argument conflates average with individual-level guarantees. Since this is the theoretical basis for generating instructions with optimized ATEoverall (Eq. 9), the causal optimization step is not supported.","section":"3.4 (Enhanced Instructions Generation)"},{"comment":"All reported accuracies are single numbers without error bars, confidence intervals, or significance tests. Many improvements are small (e.g., GSM8K 75.5 to 77.3, CommonsenseQA 71.7 to 72.0), and some results worsen under SCIE (Boolean Expressions on GPT-4o mini: 98.4 to 97.7; CommonsenseQA on Llama-3-70B: 88.6 to 85.4; Date Understanding decreases on multiple models). The comparison with OPRO in Table 2 rests on a 0.9-point difference (77.7 vs 78.6) with no indication of variance. Without repeated runs or statistical testing, the claim that SCIE 'effectively enhances instructions for most reasoning tasks' is not established.","section":"4 (Experiments and Results), Tables 1-2, Figure 5, Appendices B and E"},{"comment":"The OR-module evaluation is anecdotal. The aggregation experiment uses a single random 70% subset of GSM8K and reports one accuracy (77.9%), which the authors themselves describe as possibly due to 'the randomly extracted test data'. The generalization experiment reports one accuracy on MultiArith and a qualitative 'several more questions correct' on a 30-problem set. Separately, Appendix E shows that removing the counterfactual module makes the ATE estimates 'not stable and robust' and degrades accuracy, which underscores how sensitive the method is to data-generation choices. The paper provides no evidence of stability across random seeds, question subsets, or instruction-generation runs, so the claimed reusability and robustness are not demonstrated.","section":"4.3 (Evaluation of OR-SCIE) and Appendix E"}],"minor_comments":[{"comment":"In Eq. (2), the expression 'T/upmodelsY (t)' is a rendering artifact and should read 'T _||_ Y(t) | X'. Also, the equality in Eq. (5) between causal estimands and observed conditional expectations silently assumes ignorability and positivity; these assumptions should be stated explicitly at that point.","section":"3.1 (Preliminaries)"},{"comment":"The description of Figure 3 is unclear about how 'probability density distributions' are obtained for ordinal or categorical proxy features after binarizing only the treatment feature. The figure caption and surrounding text should specify the estimation method and the sample size.","section":"3.2 (High-quality Observational Data Generation)"},{"comment":"The connection between the numerical ATE values and the LLM's instruction-generation prompt is not made precise. The paper says the LLM will 'generate instructions that optimize ATEoverall', but it does not give the exact prompt template or state whether the LLM receives the full ATE vector or only the recommended directions. Providing this material would improve reproducibility.","section":"3.4 (Enhanced Instructions Generation)"},{"comment":"The Llama-3-70B table has a formatting error in the 'Other logical reasoning' row: the SCIE line contains six numbers instead of five, which obscures the comparison. The paper should also fix the typo 'Plan-and-Sove' in Appendix C.","section":"Appendix B"},{"comment":"The first contribution claims this is 'the first work that enhances the prompts of LLMs for reasoning tasks from a causal perspective'. Given prior work on causal effects of linguistic properties (e.g., Pryzant et al. 2021, Feder et al. 2022) and the paper's own use of those methods, the novelty claim should be more carefully qualified.","section":"Introduction"}],"recommendation":"reject","confidential_remarks":"The paper's causal framing is central to its contribution, yet the identification assumptions are not satisfied by the data-generation process, and the theoretical justification for the optimization step is flawed. The empirical evaluation is too thin to support the claimed gains: single runs without error bars, several negative results, and a 30-question OR-reuse test. These are load-bearing issues that cannot be fixed by local revisions; a convincing version would need a different data-generation design (e.g., truly randomized or controlled manipulation of individual features), a rigorous statistical evaluation with multiple seeds and confidence intervals, and a substantial rewriting of the causal claims. The current manuscript would likely mislead readers into treating correlations from LLM self-generation as causal effects."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper's central claim—that it causally optimizes instructions by estimating treatment effects of proxy textual features—does not survive close reading. The ignorability assumption is misstated (feature independence is not unconfoundedness), the \"counterfactual instructions\" are just alternative phrasings that change multiple dimensions at once, and the proof that a positive ATE raises the probability of a positive ITE is simply wrong: one large positive effect can outweigh many small negative ones. The reader's stress test lands on all of these, and I agree.\n\nThat said, the assembled pipeline is genuinely new: LLM-generated instruction variants, proxy features, T-Learner ATE estimates via a code interpreter, and LLM-generated enhanced instructions with an inheritance module. Each component is known, but the integration is not in the cited literature. The paper also does several things right. It releases code, evaluates on three LLMs and ten reasoning tasks, reports the ATE estimates against T-Learner and S-Learner, and includes an ablation (Appendix E) showing that removing the counterfactual step degrades both ATE stability and accuracy. That is good empirical practice and gives me some confidence the effect is real as a heuristic.\n\nThe soft spots are proportional to the paper's ambitions. The reported gains are mostly one to three points, several tasks decline, and there are no error bars or significance tests. The circularity—same LLM family generates the data, estimates the effects, and writes the optimized prompts—means the \"enhanced instructions\" may just be superficially similar to what the model already prefers. The OR module's claims are thin: a 70% random split \"slightly higher\" result, a single 30-question inherited-instruction test, and no baseline beyond the base prompt. This is not enough to establish reusability.\n\nFor a reader, the paper is useful as an example of how to assemble a prompt-rewriting pipeline and as a cautionary tale about importing causal language without the identifying assumptions. It is not a validated causal inference method. If I were an editor, I would send it to a serious referee: the novelty is real, the flaws are specific and fixable in framing, and the empirical work is substantial enough to justify the attention. But I would not cite it as evidence of causal optimization in my own work.","headline":"The causal inference story doesn't hold up, but the assembled prompt-rewriting pipeline is new, empirical, and worth a careful look as a heuristic.","tokens_in":18709,"tokens_out":2036,"would_cite":false,"duration_ms":16039,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that prompt wording can be treated as a causal treatment, and that LLM-generated observational data plus average treatment effect estimation yields rewritten instructions that improve reasoning accuracy at lower training…","keywords":["prompt optimization","causal inference","large language models","chain-of-thought","average treatment effect","meta-prompting","reasoning tasks","instruction enhancement"],"falsifier":"Generate many instruction pairs matched on all eight proxy features but differing only in surface wording; if accuracy varies substantially within matched pairs, the proxy features do not capture the full causal path and the ATE-based rewrite has no stable target.","tokens_in":17593,"feed_emoji":"🧠","tokens_out":7716,"duration_ms":57733,"temperature":0.7,"pith_summary":"This paper proposes treating the wording of a prompt instruction as a treatment in a causal study, and using the LLM itself to supply the observational data. Its Self-Causal Instruction Enhancement (SCIE) method has the LLM generate many instruction variants, score them on named textual features such as clarity and directness, estimate the average effect of each feature on answer correctness, and then write an enhanced instruction that amplifies features with positive effects. The paper reports that this rewritten instruction improves accuracy over the base 'Let's think step by step' prompt on nine of ten reasoning tasks, and beats the LLM-as-optimizer baseline OPRO on GSM8K (78.6 versus 77.7), without gradient training and with a plain-language explanation. A companion Object-Relational (OR) module treats the learned causal pattern as an inheritable class, so related tasks can reuse it at low cost.","feed_headline":"Causal prompt rewrites lift LLM accuracy on nine of ten tasks","feed_subtitle":"SCIE estimates which textual features of an instruction drive correct answers, then rewrites the prompt to amplify them.","key_machinery":"The central mechanism is the proxy feature: a named, interpretable textual property of an instruction (e.g., 'Directness', 'Clarity', 'Specification of Detail Level') that the LLM enumerates and scores. The pipeline treats each proxy feature in turn as a binary treatment; generates counterfactual instruction variants that flip that feature; Cartesian-products those variants with sampled questions; labels each instruction-question pair with answer correctness; and estimates the average treatment effect of each feature using a T-Learner, implemented by an LLM with a code interpreter via in-context learning. The LLM then generates an enhanced instruction that raises positive-ATE features and lowers negative-ATE features, and the Object-Relational (OR) module packages the discovered causal pattern as an inheritable class for other tasks.","core_discovery":"The central claim is that an LLM's own reasoning can be improved by first eliciting its causal estimates of how prompt features affect answer correctness, then having it rewrite the prompt to maximize the estimated overall average treatment effect. Under the potential-outcomes framework, the instruction is the treatment, answer correctness is the outcome, and proxy textual features are the treatment variables, each binarized in turn while the others serve as covariates. The paper asserts that the generated observational data satisfy ignorability, positivity, and consistency, and that LLM-estimated ATE values align closely with T-Learner estimates. On that basis, SCIE-enhanced instructions outperform the original prompts on most of the ten evaluated reasoning tasks and outperform OPRO on GSM8K, while the OR module transfers the learned causal relationships to new tasks.","pith_inferences":["Editorial inference: the same data-generation and ATE pipeline is outcome-agnostic, so it could optimize prompts for answer confidence, response length, or refusal rates rather than only correctness.","Editorial inference: because the 'counterfactual instructions' are LLM paraphrases rather than controlled edits of a single feature, the ATE numbers should be read as correlations over generated wordings; a randomized swap of one feature would test whether the causal interpretation holds.","Editorial inference: the OR module implies a cheap transfer recipe—learn which prompt features matter on one task and reuse that class on related tasks—but the paper's evidence for transfer covers only a few dataset pairs."],"forward_implications":["On GPT-3.5 Turbo, SCIE raises Zero-Shot CoT accuracy on nine of the ten reasoning tasks, with GSM8K moving from 75.5 to 77.3 and StrategyQA from 65.5 to 71.1.","SCIE also improves more elaborate base instructions: on GSM8K it beats Plan-and-Solve prompting and AgentInstruct while keeping the same student model.","Against OPRO, the LLM-as-optimizer baseline, SCIE reaches 78.6 versus 77.7 on GSM8K, and it produces an interpretable explanation of the rewrite.","The OR inheritance step transfers a causal class learned on GSM8K to MultiArith and to high-difficulty fresh-gaokao-math-2023 problems, improving accuracy without recomputing ATE from scratch.","Across tasks, the LLM's ATE estimates closely match T-Learner estimates, which the paper takes as evidence that the generated observational data supports causal effect estimation."],"supporting_citations":[{"why":"Supplies the base Zero-Shot CoT instruction that SCIE rewrites and the two-stage prompting protocol used in most experiments.","marker":"Kojima et al. 2022"},{"why":"Provides the OPRO LLM-as-optimizer baseline that SCIE compares against on GSM8K.","marker":"Yang et al. 2023"},{"why":"Defines the potential-outcomes framework and the ATE estimand that the whole instruction-effect analysis builds on.","marker":"Rubin 1974"},{"why":"Supplies the three identification assumptions (ignorability, positivity, consistency) that the data-generation design is meant to satisfy.","marker":"Feder et al. 2022"},{"why":"Defines the T-Learner and S-Learner metalearners used for ATE estimation and as the comparison baselines.","marker":"Kunzel et al. 2019"},{"why":"Provides the forward and reverse instruction-generation modes used to create the instruction variants in the data-generation step.","marker":"Zhou et al. 2022"},{"why":"Motivates the in-context learning strategy used to teach the LLM to estimate ATE from demonstrations.","marker":"Brown et al. 2020"},{"why":"Provides the GSM8K dataset that anchors the main accuracy comparisons and the OR reuse experiments.","marker":"Cobbe et al. 2021"}],"fun_headline_variants":["LLMs use causal estimates to rewrite prompts and lift accuracy","SCIE: causal self-training boosts LLM reasoning","Causal instruction enhancement improves LLM reasoning","LLMs estimate prompt causality to improve their own answers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline works only if the instruction variants generated by the LLM differ in the measured textual feature and in nothing else that also affects answer correctness, so the estimated effects are not secretly driven by length, topic leakage, or other hidden wording differences.","fun_headline_variants_meta":{"raw":{"variants":["LLMs use causal estimates to rewrite prompts and lift accuracy","SCIE: causal self-training boosts LLM reasoning","Causal instruction enhancement improves LLM reasoning","LLMs estimate prompt causality to improve their own answers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000274,"raw_usage":{"total_tokens":1608,"prompt_tokens":885,"completion_tokens":723,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":661}},"tokens_in":501,"tokens_out":723,"duration_ms":4704,"temperature":1.0,"reasoning_tokens":661,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:40:10.408169+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate many instruction pairs matched on all eight proxy features but differing only in surface wording; if accuracy varies substantially within matched pairs, the proxy features do not capture the full causal path and the ATE-based rewrite has no stable target.","supporting_citations":[{"cited_title":"A.; Manzoor, E.; Pryzant, R.; Sridhar, D.; Wood-Doughty, Z.; Eisenstein, J.; Grimmer, J.; Reichart, R.; Roberts, M","cited_arxiv_id":null,"evidence_quote":"Supplies the three identification assumptions (ignorability, positivity, consistency) that the data-generation design is meant to satisfy."},{"cited_title":"R.; Sekhon, J","cited_arxiv_id":null,"evidence_quote":"Defines the T-Learner and S-Learner metalearners used for ATE estimation and as the comparison baselines."},{"cited_title":"I.; Han, Z.; Paster, K.; Pitis, S.; Chan, H.; and Ba, J","cited_arxiv_id":null,"evidence_quote":"Provides the forward and reverse instruction-generation modes used to create the instruction variants in the data-generation step."}],"review_version":1}