{"id":"a4e44032-c9f4-4679-88ef-64abdd3f4f91","arxiv_id":"2508.09586","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"EvoCurr couples an LLM curriculum designer with an LLM code-generating solver, but its only reported success is 1 of 5 runs and no direct baseline is shown.","lead":"A two-agent LLM system generates a sequence of increasingly hard StarCraft II micro-management tasks and writes Python decision-tree code for each, adapting difficulty to the solver's win rate. One of five independent runs reached the final task; the paper claims gains over direct solving but does not report the baseline results.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Direct-baseline comparison promised in §4.1 is never reported; 1/5 curriculum success cannot support the abstract's 'significantly improves' claim.","rationale":"The reader's REJECT verdict is well supported. My pass confirms that the central comparative claim is unsupported because the direct baseline promised in §4.1 never appears. This is more decisive than the transfer assumption the reader highlighted: even if transfer works perfectly, the paper lacks the comparison needed to show the curriculum helps. The transfer assumption in §3.1 is a real secondary risk: with one successful path, the paper cannot separate curriculum composition from one-shot code generation luck. The proposed concrete test would settle both concerns by supplying the missing baseline and checking reproducibility. Agreement is partial because the reader's weakest_assumption (transfer) differs from my primary concern (missing baseline), though both reduce to insufficient evidence for the central claim.","tokens_in":15873,"tokens_out":2687,"duration_ms":29900,"concrete_test":"Run the promised direct-generation baseline: use the same solver LLM (and, if possible, the same planner-coder-critic wrapper) to generate decision-tree code for the final task in Table 1 without any curriculum, for at least 5 independent attempts matching the 5 EvoCurr paths, under the same opponent scripts and 67% win-rate threshold. Report per-attempt success/failure, win rates, and an efficiency metric (e.g., number of LLM calls or script length). If the baseline succeeds in at least 1 of 5 attempts, the claimed improvement is not demonstrated; if it succeeds in 0 of 5, a larger sample is still needed to establish significance. Independently re-run Path 1's full curriculum at least 5 times to test whether the final-task 100% win rate is reproducible or a lucky draw.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires a comparison, but the comparison is absent. The abstract states EvoCurr 'significantly improves task success rates and solution efficiency compared to direct-solving baselines.' Section 4.1 says 'We compare EvoCurr against a direct generation baseline,' yet Section 4.4 reports only EvoCurr outcomes: 20% success (1 of 5 paths). No baseline success count, win-rate table, or efficiency metric is provided anywhere in the paper. The reported 67% stage thresholds also lack the number of simulations and confidence intervals needed to interpret 'consistent' success. Without baseline numbers, the single successful path could reflect the solver LLM's one-shot capability or luck rather than the curriculum loop. The transfer/template assumption in §3.1 ('The evolved decision tree is preserved as a template for subsequent iterations') is a mechanism-level risk, but the missing direct baseline is the load-bearing gap for the paper's stated comparative claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EvoCurr, a two-LLM closed-loop framework for complex decision-making. A curriculum designer LLM generates progressively harder StarCraft II micro-management tasks, while a solver LLM produces executable python-sc2 decision-tree code; the solver's win rate feeds back into the curriculum, which adapts difficulty until the final task is solved. The main claims are that EvoCurr significantly improves task success rates and solution efficiency over direct-solving baselines, and that the curriculum loop is the driver of success. The experimental section reports five independent curriculum paths (Table 2), of which exactly one (Path 1) reaches the final task, yielding a 20% overall success rate (1/5 paths).","tokens_in":16144,"tokens_out":4669,"duration_ms":58598,"significance":"The idea of using one LLM to generate an adaptive curriculum for another LLM's inference-time code generation is timely and potentially useful for complex, interpretable decision-making. The pseudocode (Algorithms 1–3) and the detailed curriculum tables are valuable for reproducibility of the qualitative setup. If the comparative claim were actually demonstrated, this would be a meaningful contribution to LLM-agent curriculum learning. However, as presented the central empirical claim is unsupported: the promised direct-generation baseline is never reported, the single success out of five paths is far too weak to establish 'significant improvement,' and no confidence intervals or simulation counts accompany the win rates. The significance of the framework is therefore not currently established.","major_comments":[{"comment":"The setup states 'We compare EvoCurr against a direct generation baseline' (§4.1), but no baseline result appears in §4.4 or Table 2. The abstract's claim of 'significantly improves task success rates and solution efficiency compared to direct-solving baselines' is therefore unsupported. With only one successful path (1/5), the observed outcome could be due to the solver LLM's one-shot ability or chance. The authors must report the baseline's success count, win rates, and efficiency metrics, or remove the comparative claim.","section":"§4.1 vs §4.4"},{"comment":"Win-rate thresholds are reported as 67% or 100% without stating the number of simulation runs per curriculum stage or any confidence interval. The phrase 'across multiple simulation runs' is never quantified. This is load-bearing: a 67% success rate over, say, three runs is not the same as over 100 runs, and 1/5 path-level success cannot support 'significantly improves.' Please provide per-stage trial counts, intervals, or raw counts.","section":"§4.2/§4.4"},{"comment":"The claimed 'solution efficiency' is never defined or measured. The text reports only path outcomes and stage-level win rates; no metric for efficiency—such as number of LLM calls, wall-clock time, code size, or attempts to reach the final task—appears anywhere. Thus the second half of the abstract's headline claim is not testable.","section":"§4.4"},{"comment":"The paper attributes success to the self-evolving curriculum, but it never isolates the curriculum from the code-refinement loop. Algorithm 1 preserves the previous decision tree as a template and iteratively improves it (§3.1, lines 10–11), and Algorithm 3 already contains a planner–coder–critic refinement loop. Without a baseline that uses the same iterative template refinement but with a fixed, non-adaptive curriculum, the contribution of the adaptive curriculum is confounded. This should be tested or explicitly acknowledged as untested.","section":"§3.1 and Algorithm 1"}],"minor_comments":[{"comment":"The caption says 'The green points are the terminal nodes, red points represents failing settings and the green points are success curricula.' The repeated 'green' is presumably a typo; likely one of the two should be a different color.","section":"Figure 3 caption"},{"comment":"In §2, the text cites 'LLM-SMAC [10]', but reference [10] is titled 'SMAC-R1: The Emergence of Intelligence in Decision-Making Tasks.' Please align the citation label with the reference title.","section":"References"},{"comment":"The notation block lists L_cd and L_cr (critique LLM), but Algorithms 2 and 3 only use L_c and generic Plan/Code/Critique operations. Either remove unused notation or use it consistently in the pseudocode.","section":"§3.2 Algorithm Notation"},{"comment":"The appendix provides example code snippets, but there is no statement about releasing the full code, prompts, or evaluation harness. Given the empirical nature of the paper, a code/data availability statement would aid reproducibility.","section":"Reproducibility"}],"recommendation":"reject","confidential_remarks":"The reader's stress-test concern is fully validated: the central comparative claim is missing, and the one successful path out of five cannot support the abstract's 'significantly improves' wording. I do not see this as a simple presentation issue; it requires a new experimental study with a direct baseline, more repetitions, and proper uncertainty quantification. The framework idea may be salvageable, but the current manuscript does not meet the evidentiary bar for a journal publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The framework is the real thing: a two-agent inference-time loop where a curriculum LLM adjusts task difficulty from win-rate feedback and a solver LLM writes executable python-sc2 decision trees through a planner-coder-critic cycle. That combination is not in the cited prior work, and the paper is honestly engineered—algorithms are concrete, you can see the actual code, and they report five full curriculum paths including one successful solve. The integration of known components is legitimate, and the fact that they show failures as well as successes suggests they are not cherry-picking.\n\nThe problem is that the abstract's claim is 'significantly improves task success rates and solution efficiency compared to direct-solving baselines,' and the paper never shows the baseline. Section 4.1 says they compare against direct generation, but no baseline result appears anywhere in Section 4. We only learn that EvoCurr solved the final task once in five runs. With one success, you cannot tell whether the curriculum loop helped or whether the underlying code-generation LLM would have solved it directly, or just got lucky on that path. The stress-test note has this right. Also, the 67% stage thresholds are reported without the number of games or confidence intervals, so 'consistent' success is not quantifiable. The transfer assumption—preserving the decision tree as a template for later stages—is plausible but never isolated; several failed paths could stem from template transfer failure rather than curriculum design failure.\n\nThe paper itself flags another limitation in Section 5: a single-agent coder biases toward certain unit types under limited context. That is honest but does not fix the missing comparison. Minor points: novelty is moderate (recognizable components, new integration), and no code or data are released, which is a real gap for an empirical systems paper.\n\nWho gets value from this: anyone working on LLM agents and curriculum learning, or StarCraft II bot scripting. It is a useful existence proof that an LLM curriculum loop can, at least in one run, build up a complicated decision-tree policy. But as a published claim of 'significant improvement,' it falls short.\n\nI would not cite it yet. It does deserve a serious referee rather than a desk reject, because the framework is coherent, the description is clear, and the flaws are fixable. The right ask is: rerun with a real direct-generation baseline, report per-run win rates and sample sizes, add at least a second scenario, and release the code. With that, it could become a solid paper. As is, I would reject.","headline":"Useful integration of LLM-driven curriculum design with behavior-code generation, but the paper's central comparative claim is unsupported because the promised direct-solver baseline is missing and success rests on one of five paths.","tokens_in":16579,"tokens_out":1544,"would_cite":false,"duration_ms":20561,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EvoCurr claims that an inference-time, self-generated curriculum of StarCraft II micro-battles lets a code-writing LLM solve a final battle that direct one-shot code generation cannot solve reliably.","keywords":["curriculum learning","LLM agents","decision tree code","StarCraft II","self-evolving agents","inference-time adaptation","micro-management","code generation"],"falsifier":"Run EvoCurr on the same final task but present the intermediate curricula in scrambled or reversed order, with the same total number of code-generation attempts; if success persists, adaptive ordering is not doing the work. Alternatively, check whether branches from early stages actually execute in later stages—if the preserved template is largely overwritten, transfer is not the mechanism.","tokens_in":15835,"feed_emoji":"🎮","tokens_out":10303,"duration_ms":95851,"temperature":0.7,"pith_summary":"The paper proposes EvoCurr, a two-LLM loop for hard decision-making tasks. One LLM designs a sequence of easier StarCraft II micro-battles, and another LLM writes executable Python decision-tree code to play each battle, carrying the previous tree forward as a template. A 67% win-rate threshold decides whether the next task is harder or easier, so the curriculum is generated on the fly rather than hand-designed. The central claim is that this self-made curriculum lets the solver LLM assemble a policy for a complex final engagement, whereas asking it to write the same final policy directly is unreliable. In five autonomous runs, one run reached complete mastery of the full task, which the paper reads as evidence that the curriculum loop, not one-shot code generation, is what makes the task solvable.","feed_headline":"Self-written lessons solved a StarCraft II task direct code could not","feed_subtitle":"A designer LLM and a coder LLM cooperated to master the full final battle in one of five autonomous runs.","key_machinery":"The load-bearing mechanism is the closed-loop curriculum. A curriculum-designer LLM proposes task configurations (unit counts, abilities, map, objectives, difficulty level); a solver LLM turns each configuration into a decision tree in a planner-coder-critic loop; environment evaluation returns a win rate; and the designer uses threshold crossings to decide whether to escalate or ease. The decision tree itself is the memory: all learned behavior lives in its branches, and each new curriculum modifies the preserved tree rather than restarting from scratch.","core_discovery":"EvoCurr's central claim is that the hard part of complex decision-making is not the final code but the path to it. The framework pairs a curriculum-designer LLM with a code-writing solver LLM: the designer starts from a simplified version of the target task, the solver emits decision-tree code, and after simulated combat the designer receives the win rate and either escalates or eases the next task. The current decision tree is preserved and extended at each stage. On the paper's Terran-versus-Protoss micro-management benchmark, one of five autonomous curriculum paths completed the full final task with a 100% win rate in the final stage, after passing the 67% threshold at every earlier stage","pith_inferences":["A direct test of the transfer assumption would shuffle or reverse the intermediate curricula while holding the code-generation budget fixed; if success persists, the gradual ordering is not the cause. The paper does not run this control.","The 67% win-rate threshold is reported over multiple runs, but the number of rollouts is not quantified; in a stochastic simulator, threshold crossings could be partly noise, so the framework's reliability would sharpen with variance reporting.","The paper itself notes in its discussion that a single solver agent applies fine control to some unit types and coarse control to others, and proposes assigning each unit type its own specialist agent; that division of labor is a testable extension of the central claim.","The one-in-five success rate suggests curriculum generation is a stochastic search; treating curriculum paths as a population and selecting among them is a natural next step the paper leaves implicit."],"forward_implications":["If EvoCurr is right, inference-time curriculum design can replace manual curriculum engineering for LLM agents in any domain where tasks can be ordered by difficulty.","The decision-tree representation gives an audit trail: each stage's behavior is explicit code, so debugging, verification, and partial reuse are built into the loop.","The adaptive easing mechanism means a failed task does not reset progress; the designer steps back and reuses the latest successful tree, which is what lets one of five paths survive a mid-curriculum failure.","The same loop should transfer to other structured reasoning or program-synthesis problems, since it only requires a difficulty-ordered task space and a scalar performance signal."],"supporting_citations":[{"why":"Supplies the core premise that gradually increasing task difficulty improves learning, which EvoCurr operationalizes with an LLM in the loop.","marker":"[2]"},{"why":"Supplies the planner-coder-critic structure and the behavior-tree code format the solver uses to turn curricula into executable StarCraft II policies.","marker":"[10]"},{"why":"An exemplar of self-evolving agents that write their own code; EvoCurr extends this idea from open-ended skill acquisition to curriculum-governed task progression.","marker":"[36]"},{"why":"Shows LLMs can generate and adapt environments for agent training; EvoCurr adapts this idea to generate problem instances at inference time.","marker":"[41]"},{"why":"Provides the bootstrapping idea that self-generated incremental training can make hard reasoning tasks tractable, which underlies the staged self-curriculum.","marker":"[42]"}],"fun_headline_variants":["Self-taught AI masters StarCraft via adaptive curriculum","Two LLMs team up for perfect StarCraft micro win","Gradual AI lessons outperform direct code in StarCraft","AI designer and coder collaborate for StarCraft success"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing premise is that decision-tree code from early curriculum stages survives and remains useful when new units, abilities, and enemy compositions are added, so preserving the previous tree as a template composes into a final working policy.","fun_headline_variants_meta":{"raw":{"variants":["Self-taught AI masters StarCraft via adaptive curriculum","Two LLMs team up for perfect StarCraft micro win","Gradual AI lessons outperform direct code in StarCraft","AI designer and coder collaborate for StarCraft success"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000539,"raw_usage":{"total_tokens":2420,"prompt_tokens":739,"completion_tokens":1681,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":1625}},"tokens_in":483,"tokens_out":1681,"duration_ms":13622,"temperature":1.0,"reasoning_tokens":1625,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T20:57:25.198450+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run EvoCurr on the same final task but present the intermediate curricula in scrambled or reversed order, with the same total number of code-generation attempts; if success persists, adaptive ordering is not doing the work. Alternatively, check whether branches from early stages actually execute in later stages—if the preserved template is largely overwritten, transfer is not the mechanism.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the core premise that gradually increasing task difficulty improves learning, which EvoCurr operationalizes with an LLM in the loop."},{"cited_title":"SMAC-R1: The Emergence of Intelligence in Decision-Making Tasks","cited_arxiv_id":"2410.16024","evidence_quote":"Supplies the planner-coder-critic structure and the behavior-tree code format the solver uses to turn curricula into executable StarCraft II policies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the bootstrapping idea that self-generated incremental training can make hard reasoning tasks tractable, which underlies the staged self-curriculum."}],"review_version":1}