{"id":"90c2a6bd-5b52-40d8-afa1-1b3bff86a335","arxiv_id":"2608.06933","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A language model trained only to write questions that split two weaker solvers improves at solving math problems, while even frontier models calibrate less than half the time.","lead":"Ask-E is a benchmark and training environment where a model earns credit for writing math questions that exactly one of two solver models can answer correctly. Even leading models can calibrate questions under 45% of the time, and training a 4B model as a question asker modestly improves its own scores on AIME, HMMT, and IMO benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Section 5 transfer claim rests on 1–3 pp gains with no paired significance test; the reported improvements are within plausible seed and sampling noise, so the central claim is not yet established.","rationale":"The paper is thoughtful, transparent about its limitations, and provides useful evidence for the benchmark itself, including a no-probing baseline and cross-check-solver robustness. The Reader's concern about stochastic single-sample calibration is legitimate and directly threatens the validity of the training reward. My stress-test focuses on a more immediately gating issue for the paper's headline transfer claim: the downstream improvements are small and are not shown to exceed sampling noise. If the transfer effect is not statistically distinguishable from zero, the central claim fails regardless of whether the calibration signal is meaningful. If a paired permutation test does show significance, then the Reader's chance-calibration concern becomes the decisive next question about mechanism. Both concerns point to the same remedy—quantitative null and noise controls—so I regard the Reader's CONDITIONAL verdict as appropriate and unchanged.","tokens_in":27436,"tokens_out":9682,"duration_ms":121291,"concrete_test":"Re-evaluate the baseline and all three trained seeds on the same 550-problem suite, recording per-problem pass@8 and avg@8 outcomes, then run a paired permutation test (e.g., 10,000 shuffles of baseline-versus-trained labels within each problem) on the pooled suite and per benchmark. Report the mean difference, 95% confidence interval, and p-value. If the pooled 95% CI includes 0 or p > 0.05, the Section 5 'improves' claim is unsupported; if the test is significant, the chance-calibration concern raised by the reader becomes the next required check.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the Section 5 statement that Ask-E training 'improves at question answering' on math benchmarks, implying that calibrated question-generation skill transfers to QA. The evidence in Table 1 and Figure 11 consists of small differences: pooled pass@8 rises by roughly 1.2–1.4 pp (about 6–7 problems out of 550), AIME 2022–24 pass@8 improves by 2.2 pp (2 of 90 problems), AIME 2025 pass@8 is a tie, and avg@8 on AIME 2022–24 actually decreases by 0.7 pp. No paired significance test is reported. A rough paired-noise calculation for a binary outcome with n=550 and p≈0.74 gives a standard error of roughly 2.6 pp for the difference, so a ~1.2 pp pooled effect is well within sampling variability. Appendix I argues that consistent per-seed direction 'indicates the transfer is real rather than seed noise,' but the three seeds share the same 550 problems and are not independent samples, so consistency across seeds does not bound the paired difference. The load-bearing empirical premise—that training changes downstream QA performance—is therefore not yet distinguished from chance variation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"Ask-E is an environment for calibrated question generation. A question-asker model receives a target difficulty range defined by two boundary-solver LMs, probes the pair over several dialogue rounds, and emits a final math question; a session is 'calibrated' when exactly one boundary solver matches the cross-check solver's answer. The paper benchmarks five asker models on 190 boundary pairs (10 sessions each), reporting calibration rates from 25.2% to 44.9% correlated with model ability; control analyses include a no-probing baseline (16.3%), regrading with a different cross-check solver (<=0.6 pp shifts), a difficulty probe, and a direction analysis. The paper also trains Qwen3.5-4B with RL in the environment, using only weak boundary solvers, a self-judge cross-check, and no correctness reward, and reports small downstream gains on AIME, HMMT, and IMO-AnswerBench, interpreted as transfer of question-generation skill to question answering.","tokens_in":27675,"tokens_out":18367,"duration_ms":177074,"significance":"Ask-E is a genuinely new evaluation paradigm: the asker is graded by the differential performance of solvers at or below its own level, inverting the usual supervision direction. The benchmark is carefully constructed (20 boundary solvers, 190 pairs, consensus cross-checking, released code and rollouts), and the control analyses — no-probing baseline, cross-check swap, difficulty probe, and the one-sidedness of calibrated wins — are exactly the checks a model-behavior benchmark needs. The transfer experiment is tested on external public benchmarks (AIME, HMMT, IMO-AnswerBench) rather than on Ask-E's own scoring, so the central result is not circular by construction. If the transfer effect held up under proper paired significance testing, it would be an important finding; as it stands, the environment and benchmark are a solid standalone contribution, while the transfer claim is not yet supported by the reported evidence.","major_comments":[{"comment":"The central claim that Ask-E training 'improves at question answering' on math benchmarks is not yet established statistically. On the pooled 550-problem suite pass@8 rises by 1.4 pp and avg@8 by 1.1 pp, but the per-benchmark changes are mixed: AIME 2022–24 pass@8 rises 2.2 pp (2 of 90 problems) while its avg@8 falls 0.7 pp; AIME 2025 pass@8 is a tie; HMMT pass@8 rises 2.2 pp on 30 problems; IMO-AnswerBench pass@8 rises 1.0 pp (4 of 400 problems). No paired significance test is reported. For a paired binary outcome with n = 550 and a baseline rate near 0.74, the standard error of the difference is approximately 2.5–2.6 pp, so the +1.4 pp pooled gain is within one standard error and would need a McNemar test (or a confidence interval excluding zero) to be distinguished from chance. The per-seed consistency argument in Appendix I does not answer this point: all three seeds are evaluated on the same 550 problems, so agreement across seeds only rules out training-run stochasticity, not benchmark sampling noise (and at the per-benchmark level even the direction is not uniform, e.g., AIME 2022–24 pass@8 seed 1 is 86.67% versus the 87.78% baseline). The abstract, Section 5.2, and Section 6 claim a transfer effect; this requires either paired significance tests that support it or a suitably weakened claim.","section":"§5.2, Table 1, Figure 11, Appendix I"},{"comment":"The benchmark's absolute calibration rates have no chance-level baseline. Each boundary solver is sampled once per session at temperature 0.7, so any question on which both solvers succeed with probability near 0.5 is labeled 'calibrated' with probability near 0.5 regardless of asker skill; the 'below 50% / substantial headroom' framing in the abstract and Section 4.2 uses 50% as an implicit reference without justifying it. The no-probing baseline (16.3%, Appendix G.3) and the direction analysis of Figure 10 (the empirically weaker solver wins only 8.8% of calibrated sessions) give partial evidence that calibrated outcomes track genuine capability gaps rather than coin-flip noise, and I do not believe the comparative ranking of askers is threatened by this issue. Nevertheless, a proper control — for example, the calibration rate of a set of existing math problems scored on the same 190 pairs, or multiple boundary-solver samples per session to estimate P(exactly one correct) per question — is needed before the absolute rates can be interpreted as measuring asker skill, and the same stochasticity directly enters the training reward of Section 5.1.","section":"§3, §4.1, Appendix G.3"},{"comment":"The training-time reward is less protected against the degenerate strategies that Appendix A itself acknowledges. Appendix A states that the benchmark's three-sample cross-check consensus filters out ambiguous questions, but during training the cross-check is a single sample at temperature 0.0 from the asker's own current weights, with no consensus and no equivalence check. Under this reward, a question for which each boundary solver is correct with probability 1/2 (for example, an ambiguous or unstable question) yields an expected reward of about +0.5 (P(calibrated) = 0.5, P(too easy) = 0.25, P(too hard) = 0.25 against +1.0/+0.2/−0.2) without the question lying in any genuine capability gap, and the 'bad cross-check' filter that would flag such questions is never applied to training rollouts. The manuscript reports that degenerate behavior is unobserved (under 1% bad-cross-check) at benchmark evaluation, but that is not evidence about the training distribution. Please measure the three-sample-consensus bad-cross-check rate on held-out training rollouts, or otherwise show that the calibration improvement during training survives the benchmark's stricter scoring.","section":"§5.1, Appendix A, Appendix E"}],"minor_comments":[{"comment":"Reference [7] still contains the placeholders 'Commit abcdefg • accessed DD Mon YYYY'; please fill in the actual commit hash and access date.","section":"References"},{"comment":"The decision to score responses that hit the token cap as incorrect is justified by a plausibility argument ('models that are uncertain tend to repeat flawed reasoning'); since this choice affects the dependent variable, a brief sensitivity check (for example, excluding truncated responses and re-ranking askers) would strengthen the measurement section.","section":"Appendix F"},{"comment":"The label 'Pooled (macro, n=550)' is ambiguous; please state explicitly whether the pooled number is a macro-average over the four benchmarks or a problem-level pool, and report the weighting used.","section":"Figure 11, Appendix I"},{"comment":"Boundary solvers are run with thinking effort lowered or disabled 'to encourage brevity'; because solver behavior defines the benchmark outcome, a sentence reporting whether results change under default thinking settings would increase confidence in the measurement.","section":"§4.1"},{"comment":"The conclusion restates the transfer claim as established; its wording should be aligned with whatever statistical support is reported after revision (see Major Comment 1).","section":"§6"}],"recommendation":"major_revision","confidential_remarks":"The benchmark portion of this paper is a solid, well-controlled contribution and I expect it will be publishable after revision. My main concern is that the headline transfer claim (abstract, contribution 3, conclusion) is currently supported only by effects that are within the benchmark's sampling noise; please require the authors to supply the paired analysis, and if the pooled effect is not significant, to reframe the claim accordingly. I would also ask the editor to verify the availability of the code and released rollouts, and to have the authors clean up the placeholder in reference [7] before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the benchmark is the real contribution: fixed boundary-pair targets, cross-check consensus scoring, and the weak-only RL setup are new relative to the self-play QG papers, and the paper does a decent job checking its own work (no-probing baseline, cross-check robustness, difficulty probe). Second, the headline transfer claim—that training on calibrated question generation improves downstream math QA—is not yet supported by the numbers. The gains are 1–3 pp, with no paired significance test, and the three training seeds share the same 550 problems, so consistency across seeds does not bound the paired difference. That section needs a McNemar-style test or fresh problems.\n\nThe benchmark results are plausible and interesting: frontier models land below 50% calibration, calibration correlates with capability, the no-probing drop to 16.3% shows probing matters, and Figure 6 shows askers actually adjust difficulty to the target. The weak-only boundary set and masked solver tokens make the \"no stronger teacher\" claim credible. Credit where due: this is a thoughtfully built environment with transparent limitations and released code and rollouts.\n\nTwo soft spots, proportionate. First, the transfer effect: pooled pass@8 +1.4 pp on n=550 is within noise; the paper's Appendix I claim that consistent per-seed direction indicates real transfer doesn't hold because seeds are not independent samples. Second, the scoring rule has no chance baseline: with one temperature-0.7 sample per boundary solver, a question near the decision boundary can look calibrated with probability near 0.5. The cross-check consensus and the one-sidedness analysis (weak solver wins only 8.8% of calibrated sessions) mitigate this, but a random-question control or multiple boundary samples would make the calibration rates interpretable.\n\nWho is this for? People building self-play or question-generation training environments, and anyone thinking about how to benchmark models beyond answering. The benchmark deserves a serious referee despite the weak transfer evidence; the transfer claim is a falsifiable hypothesis that can be tightened. Recommendation: engage, but ask for paired significance testing and a chance-level baseline before accepting the transfer story.","headline":"Ask-E's benchmark is a solid new evaluation setup, but the headline transfer claim is within sampling noise and needs a paired significance test before it can be believed.","tokens_in":28198,"tokens_out":2298,"would_cite":true,"duration_ms":26613,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A model trained only to write calibrated math questions improves at solving them, with no stronger teacher in the loop.","keywords":["calibrated question generation","difficulty calibration","language model benchmarking","reinforcement learning","mathematical reasoning","answer-distribution supervision","question-answer transfer"],"falsifier":"Re-score every benchmark session using multiple independent samples per boundary solver and count how often the session label flips between calibrated, too easy, and too hard; if label flips are common, the measured calibration rates are not a stable property of the generated questions.","tokens_in":27239,"feed_emoji":"🎯","tokens_out":7283,"duration_ms":71988,"temperature":0.7,"pith_summary":"This paper tries to establish that generating math questions at a precisely targeted difficulty is a measurable capability, harder than answering, and trainable on its own. The Ask-E environment sets a target as a pair of solver models: a question is \"calibrated\" when exactly one of the two solvers can answer it correctly. Even the strongest tested asker reaches only 44.9% calibration, far from the ceiling, and calibration rates correlate with general model strength. After reinforcement learning on calibration alone, with no new math data, no contact with stronger models, and no reward for the correctness of its own answers, a small asker model improves on several contest-math benchmarks. The payoff, if right, is a supervision loop that does not require a more capable teacher as models advance.","feed_headline":"Training on question-writing lifts math solving scores","feed_subtitle":"RL reward from calibrated questions alone improves AIME, HMMT, and IMO results.","key_machinery":"The load-bearing object is the calibrated-question criterion: a generated math question is scored as \"calibrated\" when exactly one of two boundary-solver models answers it correctly, placing its difficulty inside the capability gap between the pair. Around this criterion the environment adds a probing dialogue in which the asker tests both solvers before committing a final question, and a cross-check solver whose repeated independent answers filter out questions without a stable answer. The same criterion plays both roles in the paper: it is the benchmark's success label and the reward that drives RL training, so the asymmetry of supervision is built into the metric itself.","core_discovery":"The central claim is that calibrated question generation requires and reinforces the same reasoning needed to solve the questions, and that a model can be improved purely by being rewarded for asking questions that separate two weaker or equal solvers. The paper operationalizes this with a concrete criterion: a session succeeds when exactly one of two boundary solvers matches the cross-check answer, and this same binary outcome serves as both benchmark score and RL reward. The authors train a 4-billion-parameter open-weight asker in this environment using only boundary solvers at or below its own ability, masking all solver tokens and using its own current weights as the cross-check, so that no stronger model's signal can leak into the policy. They find that this training lifts pass@8 accuracy on AIME, HMMT, and IMO AnswerBench, and improves avg@8 on three of four benchmarks, which they present as evidence that asking well-calibrated questions transfers to answering them.","pith_inferences":["Because each boundary solver is sampled once per session at temperature 0.7, a question near both solvers' ability might produce the calibrated label by chance about half the time; subtracting this chance baseline from the reported rates would give a clearer measure of genuine calibration skill.","The transfer gains are modest (about +1.1 percentage points pooled avg@8), so the supporting case would be stronger with a sweep over asker sizes or more seeds; the same environment could be tested as a curriculum with progressively harder boundary pairs.","The same \"exactly one solver succeeds\" criterion could be ported to other verifiable domains, such as code, formal proofs, or knowledge-grounded QA, making Ask-E a template for frontier-targeting benchmarks that do not require a stronger teacher.","The reward signal is an answer-distribution property rather than a correctness check, so future versions should guard against stochastic-solver exploits, for example by repeated sampling or by measuring per-question variance in solver responses."],"forward_implications":["Benchmarking calibration rate ranks models in the same order as their general ability, and even the best model scores below 50%, leaving room to track progress.","RL training on calibrated question generation improves downstream math answering with no new math data, no interaction with stronger models, and no correctness reward for the model's own answers.","The environment scales with the field: as frontier models improve, the boundary set can be replaced by stronger solvers to raise the difficulty ceiling without changing the protocol.","Probing is a major component of success: removing probing rounds drops calibration from 44.9% to 16.3%.","A session is gradable only when three independent cross-check samples agree, so the scoring filters out questions without a single stable answer."],"supporting_citations":[{"why":"Supplies the multi-turn RL environment library used to run Ask-E sessions.","marker":"[7]"},{"why":"Supplies the training library for the RL policy used in Section 5.","marker":"[26]"},{"why":"Provides symbolic answer-equivalence checking used in scoring.","marker":"[30]"},{"why":"Provides the frontier cross-check solver used for benchmark grading.","marker":"[21]"},{"why":"Supplies the asker and boundary-solver checkpoints used in training and evaluation.","marker":"[45]"},{"why":"Supplies the AIME 2022-2024 dataset for downstream transfer measurement.","marker":"[2]"},{"why":"Supplies the AIME 2025 dataset for downstream transfer measurement.","marker":"[42]"},{"why":"Supplies the HMMT February 2025 dataset for downstream transfer measurement.","marker":"[14]"},{"why":"Supplies the IMO AnswerBench dataset and the LLM-judge grading protocol for downstream measurement.","marker":"[38]"}],"fun_headline_variants":["Question-writing training lifts math solving scores","Ask right, solve right: calibrated Q training boosts math","Training on calibrated question writing improves math answering","No math data, no answers: question gen improves solving","Better questions, better answers: AI math skills rise"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The calibration signal assumes that one sampled answer per solver at temperature 0.7 measures a stable capability gap, but a question that both solvers could solve with probability near 0.5 would be labeled \"calibrated\" about half the time by chance, and the reported rates have no chance baseline.","fun_headline_variants_meta":{"raw":{"variants":["Question-writing training lifts math solving scores","Ask right, solve right: calibrated Q training boosts math","Training on calibrated question writing improves math answering","No math data, no answers: question gen improves solving","Better questions, better answers: AI math skills rise"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000263,"raw_usage":{"total_tokens":1622,"prompt_tokens":986,"completion_tokens":636,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":563}},"tokens_in":602,"tokens_out":636,"duration_ms":7695,"temperature":1.0,"reasoning_tokens":563,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:04:20.039779+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-score every benchmark session using multiple independent samples per boundary solver and count how often the session label flips between calibrated, too easy, and too hard; if label flips are common, the measured calibration rates are not a stable property of the generated questions.","supporting_citations":[{"cited_title":"Verifiers: Environments for llm reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-turn RL environment library used to run Ask-E sessions."},{"cited_title":"Prime-rl, 2025","cited_arxiv_id":null,"evidence_quote":"Supplies the training library for the RL policy used in Section 5."},{"cited_title":"Math-verify: Math verification library, 2025","cited_arxiv_id":null,"evidence_quote":"Provides symbolic answer-equivalence checking used in scoring."},{"cited_title":"Gemini 3.1 pro model card","cited_arxiv_id":null,"evidence_quote":"Provides the frontier cross-check solver used for benchmark grading."},{"cited_title":"Aimo validation aime","cited_arxiv_id":null,"evidence_quote":"Supplies the AIME 2022-2024 dataset for downstream transfer measurement."},{"cited_title":"Aime 2025","cited_arxiv_id":null,"evidence_quote":"Supplies the AIME 2025 dataset for downstream transfer measurement."},{"cited_title":"Beyond benchmarks: Matharena as an evaluation platform for mathematics with llms","cited_arxiv_id":null,"evidence_quote":"Supplies the HMMT February 2025 dataset for downstream transfer measurement."},{"cited_title":"Towards robust mathematical rea- soning","cited_arxiv_id":null,"evidence_quote":"Supplies the IMO AnswerBench dataset and the LLM-judge grading protocol for downstream measurement."}],"review_version":1}