{"id":"4afa1a18-9c24-4c52-9bca-e756b0f02bc6","arxiv_id":"2608.09155","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Model substitution value in multi-call LLM workflows is deployment-conditioned: the same medium-to-strong swap helps, hurts, or reverses depending on role, scale, pool, and task.","lead":"This paper measures when swapping in a larger LLM inside a multi-call workflow actually helps, and shows the answer depends on the workflow role, scale, model pool, and task, not on the model's tier label. It offers a diagnostic protocol plus an allocator, and a warning that routing should be calibrated per deployment rather than inferred from model names.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The fixed-tier reversal at the heart of the claim is reported without paired significance tests; at 64 calls the Qwen-only medium-vs-strong gap is well within one standard deviation, so the 'reverse ordering' may be noise.","rationale":"The reader's weakest_assumption is that the verify node's expected_total makes adaptive routing depend on oracle labels. In the text, the verifier receives only aggregate totals and validity fields, not the full ground-truth frequency vector, and fixed-tier RMSE is computed from the exact target independently of the verifier; failure-aware policies do use verifier flags, so the reader's caveat is fair for EvoCap placement claims. But the more load-bearing uncertainty for the central claim is the lack of paired inference on the exact numbers quoted in the abstract. The Qwen-only 64-call comparison has overlapping standard deviations and no p-values anywhere, and because the same seeds and tasks are reused across scales, naive pooling over 12 cells treats dependent observations as independent. This is fixable by reporting paired analyses, so the right verdict remains CONDITIONAL rather than ACCEPT or REJECT. The semantic task and matched diagnostics provide independent support for deployment-conditioned value, so I do not regard the paper as unsound; I only ask that the headline reversal survive a proper test. S6 also states that a public archival release will come later, which further prevents independent verification of the current numbers.","tokens_in":15319,"tokens_out":12134,"duration_ms":120916,"concrete_test":"Using the three seed replicates per ladder-scale cell, compute the paired difference δ_scale = RMSE(medium) − RMSE(strong) for Qwen-only and Mixed at each N in {8, 16, 32, 64}, then bootstrap 95% CIs by resampling seeds (10,000 draws) and run a seed-paired permutation test for the cross-ladder interaction Δ = δ_Mixed − δ_Qwen-only pooled across scales. Also report the eight scale-level δ values. If any Qwen-only δ CI includes 0 or the interaction p ≥ 0.05, the 'average Qwen-only ordering reverses' sentence in §4.1 and the abstract should be downgraded to 'not significantly different,' which changes the strength of the central demonstration.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is Eq. (2)/Figure 2/Table 1: Mixed all-strong improves RMSE from 4.818 to 1.538, while Qwen-only reverses, with all-medium 4.843 versus all-strong 5.091. But these are means over 12 scale-seed cells; the only CIs given are at 64 calls (Table 7), where Qwen-only all-medium is 7.118 ± 1.047 and all-strong is 6.387 ± 1.184, a gap far smaller than the cell standard deviation. No seed-paired test is reported for the 8/16/32/64 contrasts or for the Mixed-versus-Qwen-only interaction, so the headline 'reverses' could reflect noise rather than a deployment-conditioned ordering. The semantic reversal (0.466 vs 0.156) and the matched Qwen-only merge split (65.6% improve vs 34.4% harm) are larger and likely robust, so the conceptual point may survive; but the specific numeric reversal that anchors the abstract is under-evidenced. The reader's verifier-oracle concern is real but secondary: it affects EvoCap-based placement/efficiency claims, not the fixed-tier sweeps or matched probes that establish the core conditional-value result.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies model substitution in multi-call LLM workflows. It proposes a predicate-action factorization that separates the decision of where to intervene (the risk predicate) from the model change to apply (the substitution action), and it measures node-conditioned substitution value through fixed-tier sweeps, input-matched probes, and allocation ablations. In a solve-merge-verify workflow spanning four model ladders, the paper reports that the same medium-to-strong substitution has different, sometimes opposite, value depending on model pool, workflow role, scale, and task. The authors introduce EvoCap, a trace-to-constraint allocator, and present a quality-price frontier together with ablations that separate coverage, selectivity, and evidence acquisition. The central conclusion is that tier labels do not define a portable capability order; substitution value must be calibrated per deployment.","tokens_in":15611,"tokens_out":9002,"duration_ms":83823,"significance":"If the empirical claims hold, the paper makes a useful conceptual contribution: it reframes model substitution as a deployment-conditioned action and provides a controlled experimental template for measuring role- and scale-dependent effects. The design has real strengths: deterministic task generation with disjoint held-out splits, three seeds per scale, exact-input matched probes that isolate role-level effects, and an auditable trace-to-constraint mechanism in EvoCap. The frozen-transfer and semantic-task experiments directly test portability, and the paper is careful to separate calibration, localization, and placement. However, the headline reversal of the Qwen-only ordering is not yet statistically supported, and the EvoCap results depend on a verifier signal whose informational status needs clarification. The core conceptual point may survive these concerns, but the specific numeric reversal that anchors the abstract is under-evidenced.","major_comments":[{"comment":"The central claim that the average Qwen-only ordering reverses is supported only by pooled means over 12 scale-seed cells, with no paired significance tests or confidence intervals. At the completed 64-call cell (Table 7), Qwen-only all-medium RMSE is 7.118 ± 1.047 and all-strong is 6.387 ± 1.184 (n = 3); the difference is 0.731, well within one standard deviation of either cell. The paper reports no seed-paired test across the four scales and no test of the ladder-by-tier interaction that would justify the \"reverses\" wording used in the abstract and §4.1. Please add paired cell-level tests (e.g., paired t-test or bootstrap over the 12 observations, and seed-paired per scale) with effect sizes and confidence intervals; if the reversal is not statistically robust, the abstract and §4.1 should be qualified accordingly.","section":"§4.1, Table 1, Table 7"},{"comment":"The EvoCap failure-aware updater consumes verifier ok/fail outcomes at inference to create escalation constraints (§S3), and the numeric VERIFY prompt supplies expected_total (§S2). Because expected_total is the true total of the target frequency vector, the verifier provides information that may not be available in a real deployment unless it is simply the known array length. The fixed-tier sweeps and input-matched probes do not use this channel, so the core conditional-value result is unaffected; however, the EvoCap-based placement and efficiency claims (§4.3, Figure S5, the \"failure-aware\" rows) depend on this signal. Please either state explicitly that expected_total is the known array length (a constant) and therefore not an oracle label, or provide an ablation of EvoCap without expected_total/verifier outcomes to show that the allocation results do not rely on privileged information.","section":"S2, S3, §4.3"}],"minor_comments":[{"comment":"The table formatting is ambiguous: the column header \"PolicynRMSE Strong calls/task\" lacks separators, and entries such as \"Qwen-only All-medium 37.118±1.0470.0\" appear to concatenate the repetition count, RMSE, and strong-call share. Please reformat with clear column separators and align the reported values with the text in §4.1.","section":"Table 7"},{"comment":"The text refers to \"Tables 2 and??\" but the second table number is missing; please replace the placeholder with the correct supplementary table reference.","section":"§4.2"},{"comment":"Section S2 states that the matched diagnostic re-executes \"the first 16 arrays from the deterministic 32-task held-out stream for each seed,\" which would yield 48 arrays across three seeds, while §3.2.1 says \"the same 48 held-out arrays.\" Please clarify whether the 48 arrays are 16 arrays per seed or a separate set of 48.","section":"S2 vs §3.2.1"},{"comment":"The \"Strong calls/task\" column for all-strong in Table 8 reports 32.0, which appears to be the mean over the four scales (10, 18, 34, and 66 calls), not a per-scale count. Please state explicitly that these are pooled averages over the 12 scale-seed observations, or report the per-scale values.","section":"Table 8"}],"recommendation":"major_revision","confidential_remarks":"The paper is a good fit for the journal's interest in multi-agent systems and resource-aware routing, and the controlled experimental design is a strength. The recommendation rests on the need to make the headline reversal statistically defensible and to clarify the verifier signal before the EvoCap placement claims can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's core idea is right and worth taking seriously: in a multi-call workflow, the value of swapping medium to strong is a property of the action in context, not of the tier label. The controlled solve-merge-verify setup, the four ladders, the 35k node-tier probes, and the matched intervention protocol are real contributions. The role-conditioned diagnostics (100% merge improvement in Mixed vs. 65.6% in Qwen-only, with the scale-dependent sign flip) are the strongest part, and the semantic task reversal (0.466 vs. 0.156 exact match) is large enough to survive the statistical criticism. The paper also does a good job of separating calibration, role-conditioned effect, and placement, and the ablations distinguish under-coverage, over-allocation, and blocked exploration cleanly.\n\nThe soft spots are real but secondary. The abstract's anchor claim—Mixed all-strong reduces RMSE from 4.818 to 1.538 while Qwen-only reverses—is based on means over twelve scale-seed cells with no paired significance tests. At 64 calls, the Qwen-only medium vs. strong gap is 7.118±1.047 vs. 6.387±1.184, well within one standard deviation, so that specific numeric reversal could be noise. The stress-test note lands here. However, the matched merge split and the semantic reversal are much larger and are the better evidence for the paper's central thesis, so I would not reject the paper over this. The bigger concern is that the verifier receives expected_total, the ground-truth sample total, as part of its input; the EvoCap adaptive allocation and efficiency claims then rely on oracle correctness labels. The fixed-tier sweeps and matched probes do not depend on this, so the main ordering results stand, but the practical routing sequence is weaker than presented. No code or data is publicly released yet, and EvoCap has several hand-set constants that are transparently labeled but still engineering choices.\n\nWho gets value: anyone building routers for multi-call LLM systems or studying workflow-internal model behavior. It deserves a serious referee, and a good referee report should ask for paired significance tests on the fixed-tier contrasts, a clarification of the verifier's oracle role, and an artifact release.\n\nRecommendation: send it to peer review. It is not a finished practical recipe, but it is a solid controlled demonstration with a reproducible-looking protocol and a claim that will shape how the field talks about routing.","headline":"A genuinely useful empirical protocol for measuring deployment-conditioned model substitution, but the headline reversal in the abstract is not statistically supported; the conceptual point survives on stronger evidence elsewhere.","tokens_in":766,"tokens_out":1578,"would_cite":true,"duration_ms":27033,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The value of swapping a medium model for a strong one inside a multi-call LLM workflow is a property of the deployment, not the tier label.","keywords":["LLM routing","multi-call workflows","model substitution","tier labels","predicate-action factorization","node-conditioned substitution value","allocation footprint","cost-quality tradeoff"],"falsifier":"Remove expected_total and observed_total from the VERIFY prompt and re-run the 64-call Mixed condition under failure-aware allocation; if the sparse 10% strong-call operating point with RMSE near 1.5 degrades toward all-medium RMSE, the placement and efficiency claims depend on oracle verification rather than on the deployment-conditioned action value.","tokens_in":15133,"feed_emoji":"🔄","tokens_out":9344,"duration_ms":84782,"temperature":0.7,"pith_summary":"Multi-call LLM workflows distribute a task across many dependent calls, and this paper argues that the benefit of replacing a medium model with a strong one is not a fixed property of the model tier. In solve-merge-verify pipelines of 8–64 calls, the same medium-to-strong substitution lowers frequency-count RMSE from 4.818 to 1.538 in a Mixed Qwen/GPT pool, while in the Qwen-only pool it raises RMSE from 4.843 to 5.091. The paper separates the decision of where to intervene (a risk predicate) from the decision of what model change to apply (a substitution action), and measures the action's value at a specific node. This makes model substitution a deployment-conditioned action: pool, role, scale, and task contract all change whether the action helps. The payoff is a concrete routing sequence—calibrate the action, localize its effect, then optimize placement—so that routers stop inheriting rankings from tier labels.","feed_headline":"Model swaps flip value across LLM workflow deployments","feed_subtitle":"Same swap cuts RMSE in one model pool, raises it in another; routers must calibrate per deployment.","key_machinery":"The predicate-action factorization is the central object: a risk predicate selects a node context that may warrant intervention (for example, a high-fan-in merge), and a substitution action names the concrete model change to apply there (for example, medium to strong in a particular pool). The associated quantity is node-conditioned substitution value $\\Delta_{\\mathrm{local}}(z)$, estimated by matched executions on identical inputs under a fixed deployment. EvoCap, a trace-to-constraint allocator, turns observed outcomes and resource use into auditable at-least and at-most tier rules, making every placement attributable. The factorization carries the argument because it makes calibration, role localization, and placement optimization separately testable.","core_discovery":"On the paper's own terms, the central discovery is that node-conditioned substitution value—the change in loss from applying a named model change at a given workflow node under a fixed deployment—is the right unit of analysis for workflow routing. Across four model ladders, fixed-tier sweeps show that the same 'medium to strong' action has opposite signs: all-strong wins in the Mixed pool (RMSE 1.538 vs 4.818), loses in Qwen-only (RMSE 5.091 vs 4.843), and loses again after a semantic task-and-contract shift (exact match 0.156 vs 0.466 for medium). Input-matched interventions show the effect is role-concentrated: in Mixed, 192 of 192 merge contexts improve while 5,732 of 5,760 solve pairs tie; in Qwen-only, the same merge action improves 126 contexts and harms 66. The conclusion is that tier labels describe an implementation choice, not a portable capability order, and a router's allocation footprint is interpretable only against this measured action-value map.","pith_inferences":["A natural extension the paper does not run is target-side action rebinding: keep the learned risk predicate but re-estimate the substitution action on the new pool; frozen transfer suggests the footprint would survive while the action would need new calibration.","The role-concentrated evidence implies a cheaper routing heuristic: spend strong-model budget almost entirely on merge and consolidation contexts, since solve-level swaps mostly tie locally; the paper reports the distribution but does not prescribe this as a deployment recipe.","Because the semantic reversal changes the task and contract together, the data do not yet say which ingredient—prompt, output schema, or outcome metric—drives the sign flip; a factorial decomposition would make that attribution.","If the oracle-verifier limitation is real, the fixed-tier calibration results still stand, but the adaptive allocation results should be read as upper bounds on what failure-aware routing can achieve without ground-truth verification."],"forward_implications":["A router must recalibrate the medium-to-strong action whenever the model pool or task contract changes; the same action can switch sign without any change in graph topology.","A small, constant number of strong calls can hold all-strong quality as workflows scale: in the homogeneous ladders, two strong calls per task preserved the quality regime while strong-call share fell from 20% to 3% as total calls grew from 10 to 66.","Allocation quality is three separable capabilities—coverage, selectivity, and evidence acquisition—and failing any one yields a distinct regime: under-coverage, over-allocation, or blocked exploration.","Sparse deployment can cut cost substantially: failure-aware allocation reached RMSE 1.512 at a $0.0662 test price versus all-strong RMSE 1.538 at $0.2533, a 73.9% price reduction in the same quality regime.","Structural predicates transfer across model pools even when the attached action does not; frozen transfer kept a stable footprint and a 5% strong-call share across training pools."],"supporting_citations":[{"why":"Defines the query-level routing paradigm whose core assumption—that substitution benefit is known—the paper sets out to challenge.","marker":"[23]"},{"why":"Establishes the cascade and deferral baseline that treats moving to a stronger model as a fixed utility gain.","marker":"[2]"},{"why":"Unifies routing and cascading, formalizing the decision the paper factors into predicate and action.","marker":"[5]"},{"why":"Represents workflow-state-dependent model selection in multi-agent systems, the line of work this paper extends by measuring action value.","marker":"[38]"},{"why":"Shows context-aware orchestration making selection dependent on workflow state, motivating the deployment-conditioned view.","marker":"[20]"},{"why":"Supplies the selective-computation analogy that motivates sparse placement, while the paper notes the analogy does not establish a tier ordering.","marker":"[27]"},{"why":"Provides evidence that adding agents alone does not guarantee better answers, motivating outcome-level measurement of substitutions.","marker":"[9]"}],"fun_headline_variants":["Same LLM swap wins in one pool, loses in another","Where a model lands in a workflow changes its value","Substitution value flips with role and deployment","Workflow routers need action-value maps, not tier order"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The adaptive and failure-aware results assume the verifier receives expected_total—the ground-truth frequency total—as part of its input, so the routing policies are trained on oracle correctness labels that a real deployment may not provide.","fun_headline_variants_meta":{"raw":{"variants":["Same LLM swap wins in one pool, loses in another","Where a model lands in a workflow changes its value","Substitution value flips with role and deployment","Workflow routers need action-value maps, not tier order"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000238,"raw_usage":{"total_tokens":1541,"prompt_tokens":1003,"completion_tokens":538,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":473}},"tokens_in":619,"tokens_out":538,"duration_ms":5001,"temperature":1.0,"reasoning_tokens":473,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:27:23.084801+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Remove expected_total and observed_total from the VERIFY prompt and re-run the 64-call Mixed condition under failure-aware allocation; if the sparse 10% strong-call operating point with RMSE near 1.5 degrades toward all-medium RMSE, the placement and efficiency claims depend on oracle verification rather than on the deployment-conditioned action value.","supporting_citations":[{"cited_title":"Routellm: Learning to route llms with preference data","cited_arxiv_id":null,"evidence_quote":"Defines the query-level routing paradigm whose core assumption—that substitution benefit is known—the paper sets out to challenge."},{"cited_title":"Frugalgpt: How to use large language models while reducing cost and improving performance.Transactions on Machine Learning Research, 2024","cited_arxiv_id":null,"evidence_quote":"Establishes the cascade and deferral baseline that treats moving to a stronger model as a fixed utility gain."},{"cited_title":"A unified approach to routing and cascading for llms","cited_arxiv_id":null,"evidence_quote":"Unifies routing and cascading, formalizing the decision the paper factors into predicate and action."},{"cited_title":"CASTER: Breaking the cost-performance barrier in multi-agent orchestration via context-aware strategy for task efficient routing.arXiv preprint arXiv:2601.19793, 2026","cited_arxiv_id":null,"evidence_quote":"Shows context-aware orchestration making selection dependent on workflow state, motivating the deployment-conditioned view."},{"cited_title":"Improving factuality and reasoning in language models through multiagent debate","cited_arxiv_id":null,"evidence_quote":"Provides evidence that adding agents alone does not guarantee better answers, motivating outcome-level measurement of substitutions."}],"review_version":1}