{"id":"4aa9b999-e2dd-4238-9bf6-0ec5d2bb5ef9","arxiv_id":"2607.10694","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An actor-critic policy for when to fine-tune a foundation model under a compute budget outperforms random budgeted fine-tuning by >4% accuracy and reaches 97% of full fine-tuning accuracy with 25% of the steps.","lead":"The paper learns a policy that decides when a resource-limited device should fine-tune a foundation model on arriving data batches under a fixed compute budget. It frames the choice as a constrained MDP and solves it with actor-critic RL, showing better accuracy than random budgeted fine-tuning on AG News.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.5","headline":"The reported gains rest on an untested Markov-state assumption and a single narrow experimental regime, so the 4% / 97%-of-full claim is not yet shown to be robust.","rationale":"The Reader correctly isolates the Markov-state sufficiency claim (Section III-B) as the weakest modelling assumption and correctly notes that the empirical support is narrow (one benchmark, three seeds, code not yet released). That assumption is precisely what must hold for the actor-critic policy to be optimal for the true process and therefore for the Abstract/Table I numbers to be reliable. My concrete test simply operationalizes the Reader's concern: if adding residual history changes the reported metrics, the claim is overstated; if not, the numbers stand. No stronger internal inconsistency or mathematical error is present; the mathematics is standard constrained MDP + actor-critic. Hence the verdict remains CONDITIONAL and agreement with the Reader is full.","tokens_in":11291,"tokens_out":626,"duration_ms":5970,"concrete_test":"Augment the state with a simple residual-history feature (e.g., a one-hot or embedding of the last k FT decisions, or the Frobenius norm of the most recent LoRA update) and re-run the exact Table I protocol (AG News, LoRA, budget 50/200, three seeds). If average reward or best accuracy drops by more than ~1-2 points relative to the original three-scalar policy, the Markov assumption is load-bearing and the 4%/97% claim weakens; if the numbers stay within noise, the concern is largely dispelled.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central empirical claim (Abstract + Table I) is that the actor-critic policy, under a hard budget of 50 FT steps out of 200, yields average reward 0.85 / best accuracy 89.2% with LoRA, beating random budgeted FT by >4% and reaching 97% of always-full-parameter accuracy. That claim is only as strong as the modelling premise that the three-scalar state s_t = (d_t, p_t, b_t) is Markov (Section III-B, Transition paragraph). The paper asserts that future data and post-FT accuracy depend on the past only through these summaries, yet never tests whether residual history (e.g., which LoRA adapters were last updated, non-class distributional features, or the precise sequence of previous FT decisions) still influences the next accuracy a(Theta_t; D_ev). If the residual is material, the learned policy is solving a different MDP from the true environment and the reported gains may not transfer. The experiments themselves are confined to one dataset (AG News), one model (RoBERTa), fixed batch size 200, and three seeds; no ablation of the state features or of the KL-class-distribution proxy is supplied. Thus the headline numbers rest on an unverified sufficiency claim plus a narrow regime.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper studies when to fine-tune a pre-trained foundation model under a finite compute budget when labeled batches arrive online. The decision is cast as a constrained MDP whose state is the triple (data-shift score d_t via class-frequency KL, moving-average performance p_t, remaining budget b_t). The objective is to maximize expected cumulative evaluation accuracy subject to an expected total cost constraint; the Lagrangian is optimized online by a simple actor-critic scheme (logistic actor, linear critic) with dual ascent on λ. A special case in which post-fine-tuning accuracy can be known a priori is reduced to a dynamic program. Experiments on AG News with RoBERTa (batch size 200, T=200, three seeds) report that, under a hard budget of 50 LoRA updates, the learned policy reaches average reward 0.85 / best accuracy 89.2 %, outperforming random budgeted fine-tuning by more than 4 % and attaining 97 % of always-full-parameter accuracy while using only 25 % of the steps.","tokens_in":11631,"tokens_out":956,"duration_ms":9306,"significance":"The timing-of-update question under an explicit cumulative budget is under-explored relative to PEFT capacity allocation or offline threshold rules. The CMDP/Lagrangian formulation is standard yet cleanly specialized to continual fine-tuning, and the DP special case supplies an exact baseline when accuracy is predictable. If the reported gains hold under broader conditions, the work would give practitioners a principled, online controller for resource-limited continual adaptation of foundation models. Strengths include an explicit dual formulation, a fully specified actor-critic algorithm, and a transparent comparison against always/never/random budgeted baselines under identical step budgets.","major_comments":[{"comment":"Section III-B (Transition paragraph) asserts that s_t = (d_t, p_t, b_t) is a sufficient Markov state, i.e., that future data arrivals and post-FT accuracy a(Θ_t; D_ev) depend on the past only through these three scalars. No diagnostic is provided (e.g., residual-history ablation, comparison against a richer state that includes last-adapted layers or non-class features). If residual history is material, the learned policy solves a different MDP from the true environment and the Abstract/Table I gains are not guaranteed to transfer.","section":null},{"comment":"The central empirical claim (Abstract and Table I: >4 % over random budgeted FT, 97 % of full-parameter accuracy at 25 % of the steps) rests on a single dataset (AG News), one base model (RoBERTa), fixed batch size B=200, horizon T=200, and three random seeds. No ablation of the class-frequency KL proxy, of the moving-average window w, or of alternative PEFT methods is reported. Without these checks the headline numbers remain regime-specific.","section":null}],"minor_comments":[{"comment":"Algorithm 1 and the surrounding text never state the concrete values of free parameters w, β_c, η_φ, η_θ, η_λ used in the reported runs; they should be listed for reproducibility.","section":null},{"comment":"Figure 2 caption and the by-label numbers in the text are not accompanied by error bars or standard deviations across the three seeds, making the visual comparison harder to assess.","section":null},{"comment":"The dual update in Algorithm 1 uses a smoothed per-step cost Ć; a short remark on how the choice of β_c affects constraint satisfaction would help readers implement the method.","section":null},{"comment":"Related-work discussion of Online-LoRA and CARA is accurate but could more explicitly contrast the online dual-ascent mechanism with their offline or loss-plateau triggers.","section":null}],"recommendation":"major_revision","confidential_remarks":"The modeling premise is standard CMDP material; the main risk is over-claiming generality from a narrow experimental regime. If the authors can supply even a modest state-sufficiency check and one additional dataset or model, the paper becomes a solid contribution for a specialized ML systems or continual-learning venue. Scope fit for a broad journal is borderline without that expansion."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful core is the framing: treat the timing of PEFT updates on a streaming single-task FM as a budget-constrained MDP whose state is (data-shift KL, moving-average performance, remaining budget), then solve it online with Lagrangian actor-critic. That is a legitimate new application of known tools. Prior cost-aware retraining (Zliobaite, CARA) and budget-adaptive PEFT (AdaLoRA, OA-Adapter) either optimize offline thresholds or decide how much capacity to allocate once a task arrives; they do not learn an online policy for when to spend a cumulative compute budget. The DP special case (when post-FT accuracy can be predicted) is cleanly derived and worth having.\n\nWhat they do well: the CMDP and dual are standard and correctly written; Algorithm 1 is transparent; the reward is actual held-out accuracy after the action, so there is no circularity. On AG News + RoBERTa with LoRA, under a hard 50/200 budget, they report average reward 0.85 / best 89.2% versus random budgeted 0.80 / 86.1%, and they sit close to always-full FT while using 25% of the steps. Three seeds, both all-labels and by-label streams. That is a real, checkable gain over the natural baseline.\n\nSoft spots, in proportion: the load-bearing claim that the three scalars form a sufficient Markov state is asserted, never ablated. If residual history (last adapters updated, non-class features, exact FT sequence) still moves next accuracy, the policy is solving a different MDP. Experiments stay on one dataset, one model, fixed B=200/T=200; free parameters (window w, β_c, learning rates) are not swept. Code is promised, not shipped. None of this sinks the paper; it just means the headline numbers are not yet shown to be robust.\n\nThis is for people who care about resource-aware continual adaptation of FMs on edge or base-station devices. A serious editor should send it to referees; the formulation is clean enough and the empirical claim simple enough to deserve a proper check. I would read the camera-ready and cite the CMDP framing if I am writing on budgeted continual PEFT.","headline":"Clean CMDP framing of when to PEFT under a hard budget, with a working actor-critic policy and a DP special case; the 4%/97% numbers are real on AG News but rest on an untested three-scalar Markov state and a narrow regime.","tokens_in":12245,"tokens_out":580,"would_cite":true,"duration_ms":6432,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"An actor-critic policy can decide when to fine-tune a foundation model under a hard compute budget and still reach near full-fine-tuning accuracy.","keywords":["foundation models","fine-tuning","continual learning","reinforcement learning","actor-critic","resource constraints","constrained MDP","parameter-efficient fine-tuning"],"falsifier":"Run the same RoBERTa/AG-News experiment while adding an extra state feature that records which layers were last adapted; if the learned policy’s accuracy gap over random budgeted fine-tuning shrinks or disappears, the original three-dimensional state is insufficient.","tokens_in":12162,"feed_emoji":"⚡","tokens_out":891,"duration_ms":11610,"temperature":0.7,"pith_summary":"A foundation model sitting on a resource-limited device receives successive batches of task data and must choose, at every step, whether to spend compute to fine-tune or to throw the batch away. The paper shows that this choice can be cast as a constrained Markov decision process whose state tracks recent model accuracy, remaining budget, and how much the new data has drifted from history. An actor-critic algorithm then learns a policy that triggers fine-tuning only when the expected accuracy gain justifies the cost. On a standard text-classification benchmark the learned policy, using only one-quarter of the possible fine-tuning steps, recovers 97 percent of the accuracy of unrestricted full-parameter fine-tuning and beats random budgeted fine-tuning by more than four points. The result matters because it turns the expensive question of “when to update” into a learnable control problem rather than a fixed schedule or a hand-tuned threshold.","feed_headline":"Fine-tune only 25% of the time, keep 97% of full accuracy","feed_subtitle":"Actor-critic decides when a foundation model on a tight device should update or skip","key_machinery":"The constrained MDP whose state is the triple (moving-average performance, remaining budget, KL data-shift) and whose Lagrangian is optimized by an actor-critic pair that jointly learns the fine-tune-or-skip policy and the dual multiplier for the budget.","core_discovery":"Under a fixed fine-tuning budget the optimal timing of updates can be learned online by treating model performance, residual budget and data-shift score as the state of a constrained MDP and solving it with an actor-critic policy; the resulting policy consistently outperforms equal-budget random or always-on baselines and approaches unrestricted full-parameter accuracy with far fewer steps.","pith_inferences":["The same state-and-budget formulation could be applied to multi-task or multi-device settings by simply expanding the action space to choose which model or which task to update next.","If energy harvests periodically replenish the budget, the dual variable itself becomes a time-varying signal that automatically schedules more aggressive fine-tuning after recharge events.","A lightweight surrogate that predicts accuracy gain from the current (p, d) pair would let the DP special case be used online, removing the need for post-decision evaluation."],"forward_implications":["A device can achieve nearly the same long-term accuracy as continuous full fine-tuning while spending only a quarter of the compute budget.","The same control loop can be reused for any PEFT method (LoRA, adapters, etc.) because the framework never depends on the internal mechanics of the fine-tuning step.","When post-fine-tuning accuracy can be predicted in advance, the identical problem reduces to a classical dynamic program whose optimal schedule can be computed offline.","Budget-aware fine-tuning becomes a standard constrained-RL problem rather than a collection of ad-hoc drift detectors or loss-plateau heuristics."],"fun_headline_variants":["Actor-critic learns when to fine-tune, keeps 97% accuracy at 25% steps","Budgeted fine-tuning via MDP policy beats equal-compute baselines by 4%","Skip or update: online policy hits near-full accuracy with 25% fine-tunes","Resource-limited devices: actor-critic decides optimal fine-tune timing","Constrained MDP fine-tuning retains 97% accuracy using only 25% budget"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The three summary numbers—recent accuracy, leftover budget and KL drift—are assumed to capture everything about the past that matters for future data and for the accuracy gain of the next fine-tuning step.","fun_headline_variants_meta":{"raw":{"variants":["Actor-critic learns when to fine-tune, keeps 97% accuracy at 25% steps","Budgeted fine-tuning via MDP policy beats equal-compute baselines by 4%","Skip or update: online policy hits near-full accuracy with 25% fine-tunes","Resource-limited devices: actor-critic decides optimal fine-tune timing","Constrained MDP fine-tuning retains 97% accuracy using only 25% budget"]},"model":"grok-4.5","effort":"low","cost_usd":0.006266,"raw_usage":{"total_tokens":1647,"prompt_tokens":856,"num_sources_used":0,"completion_tokens":113,"cost_in_usd_ticks":62660000,"prompt_tokens_details":{"text_tokens":856,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":678,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":856,"tokens_out":113,"duration_ms":6215,"temperature":1.0,"reasoning_tokens":678,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T09:55:47.553347+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run the same RoBERTa/AG-News experiment while adding an extra state feature that records which layers were last adapted; if the learned policy’s accuracy gap over random budgeted fine-tuning shrinks or disappears, the original three-dimensional state is insufficient.","supporting_citations":[],"review_version":1}