{"id":"a51e33f3-7dd4-4125-a550-3e19df470a73","arxiv_id":"2609.00196","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"WHALE jointly optimizes model weights and executable agent harness code through alternating rejection-sampling fine-tuning and harness search, outperforming single-component and prompt-only baselines.","lead":"WHALE alternates between training a language model's weights and searching for better code (harness) that manages tools, context, and control flow. The method improves accuracy by up to 24 percentage points across search, math, and chess tasks compared to optimizing either component alone.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Headline margins are maxima over cycles selected on the test set (Alg. 1 returns the best D_test pair), so the 4.15–24.38 pp claims may be test-set selection artifacts; a held-out validation selection check is needed.","rationale":"The reader's verdict is CONDITIONAL, and I keep it conditional, but for a different reason. The strongest claim rests on 'best mean@8' numbers, and Algorithm 1 makes explicit that this best is selected on D_test. That means the reported accuracy is not the accuracy of a model chosen by a reproducible selection rule; it is the maximum of several noisy estimates evaluated on the very data used for the claim. This is most dangerous in Math (60 test items) and Chess (256 items), and it can affect comparisons because the methods have different numbers of candidate checkpoints. I do not consider this a sign of bad faith; the protocol is stated openly, which makes the issue checkable. But it is a correctness risk in the central claim. The proposed check is cheap and decisive: use a validation split for selection and keep D_test untouched. If the margins survive, the concern is resolved; if not, the paper needs new experiments. The reader's weakest assumption about harness search space and proposer quality is legitimate but less load-bearing: a narrower search space would weaken generality, whereas test-set selection threatens the specific numbers in the abstract. The FST implementation caveat and lack of seeds remain worth fixing; a held-out selection run with 3 seeds addresses the seed issue and the selection issue simultaneously, and the FST label should be corrected to 'prompt-restricted control' in the abstract as the reader suggested.","tokens_in":26076,"tokens_out":8065,"duration_ms":87617,"concrete_test":"Hold out a selection split disjoint from the headline test set (e.g., 30% of D_test items per domain) and forbid its use in reported numbers. For each method and seed, choose the final (θ,h) by best mean@8 on the selection split (or by training reward for adaptive WHALE), then evaluate that single chosen pair once on the untouched 70% and average over 3 seeds. If WHALE's margin over the stronger single-component baseline remains ≥7.67 pp and over the prompt-restricted control ≥4.15 pp, the concern is resolved; if the margin shrinks materially or flips, the central claim should be revised to validation-selected accuracy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing issue is the evaluation protocol, not search expressivity. Algorithm 1 (line 17) returns '(θ_k,h_k) with the best accuracy on D_test,' and Fig. 2 and Table 3 report 'best test mean@8' over the run. No validation split is used for checkpoint or cycle selection, so the same D_test both selects and evaluates. Since mean@8 is a noisy estimate (8 rollouts per example; Math has only 60 test items), taking the maximum over 6–10 cycles biases the reported accuracy upward, and the bias is not necessarily equal across methods because WHALE and the baselines have different numbers of candidate checkpoints and different curve shapes. Thus the central quantitative claim—WHALE beats weight-only, harness-only, and prompt-restricted FST by 4.15–24.38 pp—is not yet established as a property of the alternating recipe; part of the reported gain may be a selection artifact. The reader's FST and seed concerns are real but secondary; even a faithful FST baseline would not fix test-set selection. The issue is transparently stated in the algorithm, so it is directly checkable.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"This paper proposes WHALE, an alternating optimization framework that jointly updates model weights (via online rejection-sampling fine-tuning, RSFT) and executable harness code (via Meta-Harness search, MH) for LLM agents. The central claim is that this joint alternation outperforms both single-component baselines (weight-only, harness-only) and a prompt-weight joint method (FST) by 4.15–24.38 percentage points in best mean@8 accuracy across three domains: search QA, mathematical reasoning, and chess puzzles. Additional findings include identification of harness-dominant and model-dominant bottleneck regimes, superiority of alternating over stagewise optimization, and an adaptive patience rule that replaces fixed phase budgets.","tokens_in":27888,"tokens_out":1648,"duration_ms":49442,"significance":"The problem of jointly optimizing model parameters and executable harness code is timely and practically important for agentic LLM systems. The observation that the bottleneck can shift between model and harness, and that small weight updates can 'unlock' otherwise ineffective harness search, is potentially valuable guidance for practitioners. If the central claims held, the paper would provide a clear, modular recipe for a significant improvement over existing methods. However, as detailed below, a methodological flaw in the evaluation protocol undermines confidence in the reported margins.","major_comments":[{"comment":"The evaluation protocol uses the same test set for both checkpoint selection and final reporting. Algorithm 1 (line 17) returns the pair with the best accuracy on D_test, and Fig. 2 and Table 3 report 'best test mean@8' over the run. Since mean@8 is a noisy estimate (e.g., only 60 test items for Math), taking the maximum over 6–10 cycles introduces optimistic selection bias that is not necessarily equal across methods, because methods have different numbers of candidate checkpoints and different trajectory shapes. The central quantitative claim—that WHALE beats baselines by 4.15–24.38 pp—is therefore not yet established as a property of the alternating recipe; part of the reported gain may be a selection artifact. The authors should either (a) use a held-out validation set for checkpoint selection and report accuracy on a separate test set, (b) report final-checkpoint accuracy instead of best-over-run, or (c) characterize the bias magnitude through, e.g., a control experiment on a fixed checkpoint schedule.","section":"Algorithm 1 and §5.2, Fig. 2, Table 3"},{"comment":"The FST comparison replaces the original RL weight update with RSFT and restricts Meta-Harness to prompt search. While the authors argue this isolates the effect of the harness search space, the resulting system is no longer the published Fast-Slow Training method. This is not a fatal flaw—it is a reasonable controlled comparison—but the paper should be more explicit in the main text (not just Appendix B.1) that this is a modified variant, not the original FST, and discuss whether the original FST's RL update could benefit from the joint prompt-weight setting in ways that RSFT cannot capture.","section":"§5.3 (FST comparison)"},{"comment":"The bottleneck analysis in §6.1 relies on the same test-set trajectories used for checkpoint selection. If the selected checkpoint systematically overfits to test-set idiosyncrasies (e.g., format compliance patterns in SearchQA), the behavioral metric analysis may reflect spurious correlations rather than genuine regime properties. The authors should confirm that the qualitative findings (harness-dominant vs. model-dominant, catalytic effect of small weight updates) hold when evaluated with a proper train/validation/test split.","section":"§6.1 (bottleneck analysis)"},{"comment":"The adaptive WHALE results in SearchQA (52.82%) and Math (26.46%) are compared against fixed-schedule baselines that themselves were selected based on test-set accuracy (best-over-run). Gien the test-set selection bias, it is unclear whether adaptive WHALE's superiority is genuine or an artifact of different selection dynamics. Moreover, the adaptive rule uses training signals (valid) but the final checkpoint is still chosen on the test set. The comparison should be repeated with a validation-based selection protocol.","section":"§6.2.3 and Fig. 6 (adaptive WHALE)"}],"minor_comments":[{"comment":"The return statement selects on D_test. This is stated transparently, but the paper should explicitly discuss the bias this introduces and why it is or is not acceptable (currently no such discussion exists).","section":"Algorithm 1, line 17"},{"comment":"The table reports 'Completed cycles' for WHALE: SearchQA 6, Math 9, Chess 6. The number of cycles differs across domains, but the authors do not explain why Math uses 9 cycles while the others use 6. If this was determined by a fixed budget or by a stopping rule, that should be stated.","section":"Appendix B.1 (Table 1)"},{"comment":"The 'best-so-far' curves in Fig. 2 are visually useful but would benefit from error bands (e.g., bootstrap confidence intervals) to convey the uncertainty in the mean@8 estimates, especially for Math where there are only 60 test examples.","section":"Fig. 2"},{"comment":"Abstract and intro claim gains of '4.15–24.38 percentage points' but the range spans a factor of 6. This wide range suggests the benefit is heterogeneous across domains; the authors should discuss what drives the extremes (Chess apparently gives the lower bound, SearchQA the higher).","section":"§1, Abstract"},{"comment":"Algorithm 2 introduces per-phase patience parameters (W, T_min, P_w, J_min, P_h) that are set by default to (0.2 epoch, 0.2 epoch, 0.2 epoch, 6, 2). These are hyperparameters, so the adaptive WHALE still requires tuning, contrary to the claim that it 'removes the schedule (E,I) from the hyperparameters.' The authors should clarify that it removes the per-cycle fixed schedule but introduces new patience hyperparameters.","section":"§4, Algorithm 2"},{"comment":"The Chess Puzzles domain uses Qwen3.5-4B (vs. 2B for the other two domains). The authors should discuss whether model capacity interacts with the benefits of joint optimization, since the observation that Chess is 'model-dominant' might be partially an artifact of using a larger model.","section":"§5.1 (Ches Puzzles)"}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"The core of WHALE is a natural idea: alternate RSFT weight updates with Meta-Harness code search, treating the harness as an optimizable artifact alongside the weights. The paper executes this idea cleanly — matched budgets, per-domain breakdowns, bottleneck analysis that actually separates harness-dominant from weight-dominant regimes, and released code. The schedule sensitivity analysis and the adaptive patience rule are thoughtful. If the central claim holds, this is a useful recipe for the agent-optimization toolkit.\n\nHere is the honest bottom line: I cannot trust the quantitative headline. Algorithm 1 selects the checkpoint that performed best on D_test, and Figure 2 and Table 3 report \"best test mean@8\" over the run. Mean@8 is a noisy estimate (8 rollouts per example; Math has only 60 test items), and taking the maximum over 6–10 cycles guarantees an upward bias. The bias is not constant across methods — different numbers of candidate checkpoints and different curve shapes mean different selection pressures. So the claim that WHALE beats weight-only, harness-only, and FST by 4.15–24.38 percentage points is not yet established as a property of the alternating recipe. To their credit, the paper is transparent about the protocol, but that does not make the bias go away.\n\nThe reader flags the modified FST comparison and the lack of error bars. Both are real but secondary. The FST modification is a restricted ablation, not a direct comparison to the published method, and the abstract's \"outperforms FST\" is misleading as written. But even a faithful FST baseline would not fix the test-set selection problem.\n\nThe paper otherwise shows solid engineering and careful analysis. The bottleneck regimes and the schedule study are informative even without the absolute accuracy numbers. This paper deserves a serious referee — the core idea is real — but the authors need to re-run their analysis with a held-out validation split or bootstrap-corrected selection before the quantitative claims can be taken at face value.","headline":"A clean alternating optimization recipe let down by a test-set selection protocol that makes the headline numbers uninterpretable as reported.","tokens_in":26814,"tokens_out":1747,"duration_ms":18024},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"Alternating weight fine-tuning with executable-harness search beats tuning either component alone, by 4.15 to 24.38 points in three agent domains.","keywords":["harness optimization","weight-harness co-optimization","alternating optimization","rejection-sampling fine-tuning","Meta-Harness","agentic language models","tool-use agents","joint adaptation"],"falsifier":"Run WHALE at a fixed rollout budget while swapping the Meta-Harness proposer for a distinctly weaker model: if the 4–24 point gap over weight-only training shrinks or disappears, the reported gains come substantially from proposer strength rather than from the act of alternating. A complementary check is to restart the same recipe from a random or adversarially chosen initial harness $h_0$ and observe whether alternating updates still converge to the reported accuracy; if they do not, the hand-designed starting harness is carrying part of the result.","tokens_in":27555,"feed_emoji":"🔄","tokens_out":10784,"duration_ms":95305,"temperature":0.7,"pith_summary":"An agentic language model's usefulness is set by two coupled components: the model weights, and the harness, the executable code that decides what enters context, how tools are called, how errors are handled, and when to stop. Optimizing one while freezing the other, the paper argues, leaves the system bottlenecked by its frozen half, and prior joint methods only adapted textual prompts, not the broader harness. WHALE instead alternates small rejection-sampling fine-tuning of the weights with Meta-Harness search over full harness programs, and reports gains of 4.15 to 24.38 percentage points in best mean@8 accuracy over weight-only, harness-only, and prompt-weight joint optimization across search QA, math reasoning, and chess puzzles. The paper's deeper claim is that a model and its harness should be treated as one jointly trained system, with the timing of switches between the two phases as the decisive design choice.","feed_headline":"Alternating weights and harness code lifts agent accuracy 4–24 points","feed_subtitle":"Tuning either half alone leaves the other as the bottleneck; alternating beats both and prompt-weight joint training.","key_machinery":"The load-bearing mechanism is alternation applied to the coupled objective $J(\\theta,h)$ over model parameters $\\theta$ and executable harness programs $h$, where the two are linked only through the trajectory distribution $\\pi_{\\theta,h}$ they induce. Each cycle holds one side fixed while updating the other, $\\theta_{k+1} = \\text{ModelUpdate}(\\theta_k; h_k)$ and then $h_{k+1} = \\text{HarnessSearch}(h_k; \\theta_{k+1})$, a coordinate-descent structure that keeps a changing counterpart from confounding credit assignment. The weight phase is online rejection-sampling fine-tuning, supervised learning on the model's own verifier-accepted rollouts; the harness phase is Meta-Harness, an iterative propose–evaluate–select search over harness code that keeps an archive of candidates so later proposals can refine or revert. The third ingredient is the switching schedule: too little evidence per phase adopts a noisy candidate and can destabilize training, while too much over-optimizes against a counterpart that will change, and a per-phase patience rule over training signals replaces the fixed budgets.","core_discovery":"WHALE's central claim is that the joint objective $J(\\theta,h)$ — expected verifier reward over trajectories produced by the pair $(\\theta,h)$ — is best optimized not by one long pass per component but by short alternating conditional updates, and that the harness side of the pair should be the full executable harness, not just the prompt text. With weights updated by online rejection-sampling fine-tuning and harnesses searched by Meta-Harness, one fixed schedule $(E,I)=(0.6,6)$ beats weight-only, harness-only, and a prompt-restricted Fast–Slow Training control in all three domains, improving on the stronger single-component baseline by 7.67–10.05 percentage points and on Fast–Slow Training by 4.15–13.00 points. The experiments identify two bottleneck regimes: SearchQA is harness-dominant, where harness search matches peak weight-only accuracy with 5.79% of the rollouts, while Math is model-dominant, where harness search alone scores almost nothing until a small weight update makes it effective. Stagewise weight-then-harness optimization is dominated by small alternating steps (5.32 and 9.16 percentage points behind in the two domains), and an adaptive patience rule that stops each phase when its training signal stops improving reaches the best overall result in SearchQA and trails only the best hand-tuned schedule in Math.","pith_inferences":["A robustness question the paper leaves open is how much of the gain belongs to alternation versus to the proposer used inside Meta-Harness; replacing the proposer with a much weaker model at matched budget would separate the two, since the paper does not characterize how performance degrades along that axis.","The per-domain initial harnesses are hand-designed, so the recipe presumes a reasonable starting point; starting WHALE from a deliberately bad or randomized harness would test whether alternation climbs on its own or mostly polishes a good head start.","The SearchQA trajectories show the searched harness writing turn-by-turn state lines, answer checklists, and stricter termination rules, which suggests harness search is partly automating prompt engineering inside the loop; restricting the search space to exactly those prompt-like fields would quantify how much of WHALE's edge survives.","The token-budget logic of the Math and Chess examples suggests the joint objective also implicitly rewards cost savings, not just accuracy; measuring tokens-per-solved-example under WHALE versus weight-only training would make that trade-off explicit."],"forward_implications":["In harness-dominant domains such as SearchQA, a short harness search matches or beats extensive weight training at a small fraction of the rollout cost, which makes harness search a cheap diagnostic to run before committing to model training.","In model-dominant domains such as Math, the weight phase is a prerequisite that unlocks harness search, so the order and interleaving of phases determines whether the harness side contributes anything at all.","One long weight pass followed by one long harness pass is dominated by small alternating updates, both in final test accuracy and in the rollouts needed to reach a target accuracy.","The same interleaving schedule, with small weight epochs and harness search long enough to beat noise, is near-optimal across domains at opposite bottleneck extremes, and the adaptive patience rule recovers that schedule without hand-tuned phase budgets.","Because WHALE exposes only black-box interfaces for the two phases, the recipe transfers to any weight-update and harness-search algorithm that share a fixed binary verifier."],"supporting_citations":[{"why":"Supplies Meta-Harness, the propose–evaluate–select search over executable harness programs that carries WHALE's harness phase.","marker":"(Lee et al., 2026)"},{"why":"Defines Fast–Slow Training, the joint prompt–weight baseline WHALE must beat, which the paper converts into a prompt-restricted control.","marker":"(Tiwari et al., 2026)"},{"why":"Provides the rejection-sampling fine-tuning scheme whose online variant carries WHALE's weight-update phase.","marker":"(Shao et al., 2024)"},{"why":"Supplies the alternating and coordinate-descent optimization framing that justifies conditional updates against a fixed counterpart.","marker":"(Wright, 2015)"},{"why":"Provides the Search-R1 setup, datasets, and retrieval environment on which the SearchQA domain and its harness are built.","marker":"(Jin et al., 2025)"},{"why":"Supplies the DAPO-Math-17K training data and the strict boxed-answer verification used for the Math domain.","marker":"(Yu et al., 2025)"},{"why":"Gives the code-execution tool environment for mathematical reasoning that the Math harness wraps.","marker":"(Feng et al., 2025)"}],"fun_headline_variants":["WHALE alternates weight and harness updates to lift agents up to 24 points","Joint weight-harness tuning beats single-component agent training by up to 24 points","Alternating weights and harness beats stagewise and prompt-only agent tuning","Search harness and weights in short alternating steps for bigger agent gains"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The result rests on the search space for harnesses and the proposer that generates harness candidates being strong enough to keep finding genuinely better programs than the hand-written starting harness, and the paper does not measure how WHALE degrades as either is weakened.","fun_headline_variants_meta":{"raw":{"variants":["WHALE alternates weight and harness updates to lift agents up to 24 points","Joint weight-harness tuning beats single-component agent training by up to 24 points","Alternating weights and harness beats stagewise and prompt-only agent tuning","Search harness and weights in short alternating steps for bigger agent gains"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000265,"raw_usage":{"cost":0.00026474,"is_byok":false,"cost_details":{"upstream_inference_cost":0.00026474,"upstream_inference_prompt_cost":0.0001799,"upstream_inference_completions_cost":8.484e-05},"total_tokens":1588,"prompt_tokens":1285,"completion_tokens":303,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":0,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":222}},"tokens_in":1285,"tokens_out":303,"cost_source":"openrouter_usage","duration_ms":4670,"temperature":1.0,"reasoning_tokens":222,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-09-09T02:38:45.466207+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-alltime-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Run WHALE at a fixed rollout budget while swapping the Meta-Harness proposer for a distinctly weaker model: if the 4–24 point gap over weight-only training shrinks or disappears, the reported gains come substantially from proposer strength rather than from the act of alternating. A complementary check is to restart the same recipe from a random or adversarially chosen initial harness $h_0$ and observe whether alternating updates still converge to the reported accuracy; if they do not, the hand-designed starting harness is carrying part of the result.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}