{"id":"8e669502-6fcb-4fe2-b8d2-bba044e60377","arxiv_id":"2506.18880","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"OMEGA is a templated math benchmark showing that LLMs, including RL-tuned models, generalize to harder instances but largely fail at compositional and transformative reasoning.","lead":"This paper introduces OMEGA, a math benchmark that tests whether AI models can generalize beyond what they were trained on, in three ways: harder versions of known problems, combinations of two known skills, and problems requiring a new strategy. The authors find frontier models and reinforcement-learned models improve on harder versions but mostly fail when they must combine skills or find novel approaches.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"OMEGA's compositional/transformative test items may not require the intended skills; matrix-rank and rectangle-count examples admit familiar-tactic shortcuts, so the central claim that RL cannot compose or transform is underdetermined.","rationale":"The paper's exploratory generalization results are supported by the reported experiments, and the programmatic generator plus open training setup are genuine assets. The conditional verdict is appropriate. The load-bearing gap is construct validity: the compositional/transformative conclusions are only as strong as the assumption that test items force the intended skills, and the two examples above show this assumption is not automatic. The appendix's 'decomposition' paragraphs are asserted after the fact, not validated by human protocols or a no-shortcut proof. Statistical limitations (single seed, no error bars, LLM-as-judge trace labeling) reinforce the concern but are secondary. I would keep the verdict CONDITIONAL and make the condition concrete: validate or prune the 14 OOD families before treating the qualitative conclusions as established.","tokens_in":25799,"tokens_out":8876,"duration_ms":96305,"concrete_test":"For each of the 14 compositional/transformative test families, implement a reference solver that uses only the training-distribution tactics plus standard textbook formulas (row-type rank bound for E_n; choose-two-diameters count for dodecagon rectangles; direct expansion for the integral product). Remove or redesign any test item that this solver solves without invoking the claimed recombined or novel skill, then re-run the before/after RL comparisons. If the 'no OOD gain' pattern disappears on the pruned set, the original qualitative claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that RL improves exploratory but not compositional/transformative reasoning—rests on OMEGA test items isolating the intended skill. The paper asserts this in Secs. 2.3–2.5 but gives no independent protocol check, and two of its own examples admit simpler routes. Transformative Setting 1 (Table 9) asks for rank(E_n) with e_ij=1 if i+j even. The paper's 'needed insight' is an outer-product decomposition, but standard rank reasoning suffices: rows alternate between [1,0,1,0,...] and [0,1,0,1,...], so rank(E_n)≤2 and equals 2 for n≥2. A model that fails this is failing on symbolic-n OOD, not necessarily on 'shifting to a novel paradigm.' Compositional Setting 2 (Table 7) asks to count rectangles in a regular 12-gon; the closed form C(6,2)=15 via choosing two diameters requires neither the trained polygon-rotation skill nor the pattern-matching skill. If several items are solvable by familiar tactics, the near-zero post-RL OOD accuracies in Figures 10–11 do not establish the paper's qualitative conclusions; they may only show failure on unfamiliar single tasks. The appendix's 'decomposition' notes are post-hoc interpretations, not evidence that the intended composition/transformation was forced.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces OMEGA, a programmatically generated math benchmark with matched training-test pairs along three axes of out-of-distribution generalization: exploratory, compositional, and transformative. The authors evaluate four frontier LLMs and fine-tune Qwen2.5-7B-Instruct and Qwen2.5-Math-7B with GRPO, reporting that accuracy degrades sharply with problem complexity, that RL improves explorative generalization but yields only modest compositional gains, and that transformative reasoning remains near zero. The paper additionally provides chain-of-thought error-pattern analyses, Pass@k scaling experiments, and an ablation of compositional skill pairings.","tokens_in":26076,"tokens_out":7186,"duration_ms":75393,"significance":"If its construct validity holds, OMEGA is a useful and unusually controlled testbed: the problems are templated with programmatic verification, the training distributions are explicit, and the RL results replicate across two Qwen variants and several domains. This makes the paper's negative result about RL and compositional/transformative generalization falsifiable and reproducible, and the release of code and dataset is a concrete asset. The main caveat is that the central claim depends on test items actually forcing the intended composition or transformation; this premise is currently asserted rather than verified, and at least two representative items admit simpler solution routes.","major_comments":[{"comment":"The load-bearing premise of the compositional and transformative experiments is that each test item can only be solved by the specified composed or reframed strategy, but the manuscript provides no independent check of this premise: the 'Decomposition' column in Table 7 and the 'Needed insight' column in Table 9 are post hoc descriptions of intended routes, not demonstrations that no simpler route exists. Two of the paper's own examples illustrate the risk. In Transformative Setting 1 (Table 9), rank(E_n) can be found by observing that the rows alternate between (1,0,1,0,...) and (0,1,0,1,...), so rank(E_n) ≤ 2 for n ≥ 2, without the named outer-product decomposition; a model failing this item may be failing symbolic-n out-of-distribution generalization rather than 'shifting to a novel paradigm.' In Compositional Setting 2 (Table 7), the number of rectangles in a regular 12-gon has the closed form C(6,2)=15 obtained by choosing two of the six diameters, and this route does not require either the trained polygon-rotation skill or the pattern-matching skill. Because the near-zero OOD accuracies in Figures 10–11 are the main evidence for the paper's central qualitative conclusions, the authors should add a validation step—for example, human or solver-generated solution protocols for every OOD item showing that the intended route is necessary, or an automated detector verifying that the trained skills are genuinely used.","section":"§2.4–2.5, Tables 7 and 9"},{"comment":"The headline quantitative claims rest on very small OOD test sets and single-seed RL runs. Section B.1 states that compositional and transformative OOD sets contain 20–50 samples per setting, and the GRPO runs use seed 3 only. With 20 items, the difference between 0% and 10% post-RL accuracy is within binomial sampling noise, and with no seed variance reported, the conclusion that RL yields 'little to no improvement' is not statistically supported as stated. Report per-setting test sizes, confidence intervals, or at least 3–5 seeds for the headline settings, and ideally the per-checkpoint trajectories rather than the average over the last five checkpoints.","section":"§B.1, Figures 10–11"},{"comment":"The matrix-rank result, reported as 'base model achieved decent OOD performance (70%)' with post-RL performance dropping 30 points, is an important exception to the general pattern and is currently supported by a single unseeded observation. Before drawing the conclusion that RL 'entrenches brittle heuristics,' the authors should verify that this drop is reproducible across seeds and is not an artifact of checkpoint selection or evaluation noise, and should report the per-instance trajectories that show the hypothesized entrenchment. This matters because the result is used to support the paper's characterization of how RL fails on transformative tasks.","section":"§3.2.3, Figure 11"}],"minor_comments":[{"comment":"Figure 7's caption refers to 'geometry rotation problems' while the accompanying text reports Pass@k on 'letter distribution' problems; the rotation results appear in Figure 13, so the caption and cross-reference should be corrected.","section":"Figure 7 and text"},{"comment":"The paper uses 'exploratory' and 'explorative' interchangeably (e.g., Section 2.3 versus Figure 8), and similarly 'transformative' and 'transformational'; a single term per axis would improve precision.","section":"Throughout"},{"comment":"Tables 11 and 12 are captioned as 'corresponding to Figure 11,' but the compositional-setting ablation tracks the settings reported in Figure 10; the cross-reference appears to be mislabeled.","section":"Tables 11–12"},{"comment":"Appendix D's token-cost estimate assumes 'each backtrack call translates to 20 tokens' without justification; since this estimate is used to argue that context length is not the bottleneck, the constant should be justified or replaced by an upper-bound accounting based on actual rollout lengths.","section":"Appendix D"},{"comment":"The entry 'OMEGA (Ours)' in Table 1 lists verification as 'N/A; Correct by Construction,' which is undercut by the appendix's use of OpenCV, grid search, and DP solvers; the verification column should match the actual pipeline.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the paper's central empirical pattern is plausible and the resource investment is substantial, but the construct-validity issue is the key risk. I do not see this as a case for rejection: the authors can address it by adding per-item solution protocol validation and by reporting small-sample uncertainty. If these additions are feasible, the paper would be a solid contribution. One editorial note: appendices A.2–A.3 contain the phrase 'By the submission deadline,' which is unusual for a formal submission and should be removed or rephrased."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: OMEGA is a genuinely useful new benchmark, and the exploratory-generalization results are solid, but the headline claim that RL cannot do compositional or transformative reasoning is underdetermined. Several key test items don't force the intended skills. The matrix-rank and rectangle-count examples admit simpler routes than the paper's stated 'needed insight,' so low OOD accuracy on those items may reflect failure on unfamiliar single tasks rather than a specific inability to compose or transform.\n\nWhat's new: 40 templated generators across six domains, with matched train/test pairs on three axes. That's a real step beyond GSM-Symbolic and GSM-Infinite in breadth and control. Programmatic verification, a plausible complexity analysis (e.g., token budget for distribution problems), and released code/data are all reproducible assets. The CoT analysis of overthinking, error spirals, and shrinking calculation budget is a nice empirical contribution that stands apart from the benchmark itself.\n\nWhere it's soft: The skill-isolation assumption in Secs 2.3–2.5 is asserted, not independently checked. Transformative Setting 1 (matrix rank) is supposed to require an outer-product decomposition, but standard rank reasoning—observing alternating rows—suffices. Compositional Setting 2 (rectangles in a 12-gon) is solved by the closed form C(6,2)=15, needing neither trained skill. The appendix 'decomposition' notes are post-hoc stories, not evidence that the intended strategy was forced. The test sets for compositional/transformative are 20–50 items, RL runs are single-seed, and there are no error bars. These are addressable, but they make the central claim about transformative failure much weaker than the abstract suggests. The exploratory results—strong performance degradation with complexity, RL gains that plateau—are much better supported, with 100 items per level and consistency across two model families.\n\nThe paper is worth engaging. For researchers building math reasoning benchmarks or studying RL generalization, this is a useful resource to build on, but it is not a definitive statement about LLM creativity. The compositional/transformative conclusions need either items with verified no-shortcut properties or a restatement of what the results show. I'd send it to peer review with the expectation of heavy revision and a request for protocol validation.","headline":"OMEGA is a useful new benchmark with solid exploratory results, but its compositional and transformative claims are undercut by test items that don't force the intended skills.","tokens_in":26592,"tokens_out":4245,"would_cite":true,"duration_ms":41041,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current reasoning LLMs, even after RL fine-tuning, master exploratory generalization but fail at compositional and transformative math reasoning.","keywords":["OMEGA","out-of-distribution generalization","mathematical reasoning","exploratory generalization","compositional generalization","transformative generalization","reinforcement learning","chain-of-thought"],"falsifier":"Take the transformative test set (for example, the 10×10 blocked-grid path-counting problem) and solve every item using the familiar tactic from the corresponding training problems, with unlimited compute and a simulator. If many items yield to the familiar tactic, the tests do not force a paradigm shift and the near-zero accuracy cannot be attributed to missing transformative reasoning; if nearly all items resist the familiar tactic but fall to the intended alternative strategy, the paper's isolation claim is supported.","tokens_in":25624,"feed_emoji":"🧮","tokens_out":6962,"duration_ms":72860,"temperature":0.7,"pith_summary":"The paper introduces OMEGA, a benchmark of programmatically generated math problems designed to isolate three kinds of out-of-distribution generalization: exploratory (applying a known skill to harder instances), compositional (combining skills learned separately), and transformative (abandoning a familiar tactic for a genuinely new one). Its central claim is that current reasoning models, including reinforcement-learning-tuned ones, perform well only close to the training distribution. As complexity rises, accuracy on exploratory tasks degrades sharply, and RL fine-tuning mostly reinforces in-distribution patterns: gains on exploratory tasks are real but plateau, compositional gains are modest, and transformative gains are near zero. A sympathetic reader should care because the benchmark turns vague talk of \"mathematical creativity\" into measurable, per-skill failure modes.","feed_headline":"RL boosts familiar math skills, not creative leaps","feed_subtitle":"OMEGA isolates three generalization axes and finds compositional and transformative gains near zero.","key_machinery":"The carrying object is OMEGA itself: a collection of 40 templated problem generators across arithmetic, algebra, combinatorics, number theory, geometry, and logic & puzzles, each with a scalar complexity measure $\\delta(\\theta)$ that ranks instances. Training-test pairs are constructed by fixing the template and varying the parameter region: low-complexity instances for training and higher-complexity instances for exploratory tests; two single-skill families in isolation for training and their integrated combination for compositional tests; familiar-tactic problems for training and problems whose familiar tactic is intractable for transformative tests. Solutions are generated and verified programmatically, so the benchmark can scale indefinitely. This design is what lets the authors attribute a performance drop to a specific missing reasoning skill rather than to unseen problem content.","core_discovery":"On its own terms, the paper establishes a tripartite diagnosis. First, frontier long-chain-of-thought models show near-ceiling performance at low complexity but decline to near zero as task complexity increases; trace analysis attributes this to overthinking, error spirals, and a growing preference for guesses over computation rather than arithmetic error alone. Second, RL fine-tuning on easy instances produces substantial gains on in-distribution and moderately harder exploratory tasks, but the gains plateau on high-complexity instances. Third, models trained separately on two skills fail to integrate them on compositional tests, and RL training that improves the isolated skills does not transfer to the composed task. Fourth, on transformative tests that require a different solution paradigm, RL shows negligible improvement and can even reverse prior ability. The paper concludes that current RL amplifies learned strategies but does not induce new reasoning patterns.","pith_inferences":["A natural extension is to use OMEGA-style train/test splits to test whether test-time search (e.g., sampling many candidates and selecting by verifier) rescues compositional tasks; the paper's Pass@k data suggest the benefit vanishes at high complexity, but the interaction with compositional and transformative tests is untested.","The failure of RL on transformative tasks implies that reward signals based on verifiable answers may be too sparse to discover a new solution paradigm; process rewards or demonstrations of alternate strategies would be a direct training change the paper's framework could evaluate.","If the isolation assumption holds, the same template-pair design could be moved to code generation or scientific reasoning, where \"exploratory-compositional-transformative\" maps onto refactoring, library composition, and novel algorithm design.","The near-zero transformative results also suggest a testable prediction: models pretrained on diverse solution strategies across many templates should show larger transformative transfer than RL-finetuned models, since transformation requires exposure to paradigm shifts."],"forward_implications":["If the OMEGA findings hold, current reasoning LLMs cannot be treated as reliable solvers on high-complexity mathematics even with long chain-of-thought and test-time search.","RL fine-tuning is a reliable way to broaden competence within a learned strategy, but it should not be expected to produce compositional integration by itself.","Training setups that isolate skills need to be paired with explicit composition training, because mastering the components does not yield the composite.","Benchmarks that measure only final accuracy on static problem sets will miss the plateau: performance on exploratory generalizations can improve while transformative ability stays at zero.","The three-axis typology provides a reusable diagnostic grid for evaluating future training methods."],"supporting_citations":[{"why":"Supplies the three-way creativity typology that defines the benchmark's exploratory, compositional, and transformative generalization axes.","marker":"[4]"},{"why":"Evidence that supervised fine-tuning leaves models relying on a narrow set of reasoning strategies; motivates the need for OMEGA-style skill isolation.","marker":"[32]"},{"why":"Reports distinct reasoning boundaries as problem complexity escalates on synthetic puzzles, aligning with the performance degradation OMEGA documents across more categories.","marker":"[31]"},{"why":"Prior result on limits of transformers on compositional tasks such as multiplication and dynamic programming; OMEGA extends this to a broader math benchmark.","marker":"[7]"},{"why":"Raises the question of whether RL incentivizes reasoning capacity beyond the base model and frames the paper's RL generalization experiments.","marker":"[38]"},{"why":"Claims RL is most effective when the base model initially struggles; cited when interpreting why RL gains plateau and vary across domains.","marker":"[24]"},{"why":"A controlled perturbation benchmark that OMEGA contrasts with, since numeric perturbation preserves solution paths while OMEGA varies strategy complexity.","marker":"[26]"},{"why":"Represents the long chain-of-thought model family whose performance OMEGA evaluates on high-complexity mathematics.","marker":"[12]"}],"fun_headline_variants":["RL sharpens math routines but not creative leaps","OMEGA shows LLMs fail at out-of-the-box math","AI math: familiar gains, zero creative transfer","LLMs can't generalize to novel math strategies","OMEGA: RL amplifies known skills, not new ones"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that each OMEGA problem family and complexity level isolates exactly the intended skill: that $\\delta(\\theta)$ measures genuine reasoning difficulty, that compositional tests cannot be solved by applying the two skills one after another, and that transformative tests cannot be solved by the familiar tactic; if any test admits an unintended shortcut, the observed failures would not measure the axis the paper claims.","fun_headline_variants_meta":{"raw":{"variants":["RL sharpens math routines but not creative leaps","OMEGA shows LLMs fail at out-of-the-box math","AI math: familiar gains, zero creative transfer","LLMs can't generalize to novel math strategies","OMEGA: RL amplifies known skills, not new ones"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000204,"raw_usage":{"total_tokens":1418,"prompt_tokens":999,"completion_tokens":419,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":342}},"tokens_in":615,"tokens_out":419,"duration_ms":4418,"temperature":1.0,"reasoning_tokens":342,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:41:17.219902+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the transformative test set (for example, the 10×10 blocked-grid path-counting problem) and solve every item using the familiar tactic from the corresponding training problems, with unlimited compute and a simulator. If many items yield to the familiar tactic, the tests do not force a paradigm shift and the near-zero accuracy cannot be attributed to missing transformative reasoning; if nearly all items resist the familiar tactic but fall to the intended alternative strategy, the paper's isolation claim is supported.","supporting_citations":[{"cited_title":"Creativity and artificial intelligence.Artificial intelligence, 103(1-2):347–356, 1998","cited_arxiv_id":null,"evidence_quote":"Supplies the three-way creativity typology that defines the benchmark's exploratory, compositional, and transformative generalization axes."}],"review_version":2}