{"id":"45daa85d-dbc6-4202-9cb7-c862942940d5","arxiv_id":"2505.01449","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"COSMOS predicts the performance and cost of QLoRA fine-tuning and retrieval-augmented in-context learning across LLMs, achieving 1.09% mean absolute error and 92.72% average cost reduction on eight benchmarks.","lead":"This paper introduces COSMOS, a system that predicts how well fine-tuning or in-context learning will work for a given language model and budget, without running the full experiments. It reports about 1% average error in performance predictions while cutting the compute cost of choosing a strategy by roughly 93%.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Appendix C.1 selects the QLoRA predictor checkpoint on peak test-set accuracy, so the reported 1.09% MAE may reflect test-label leakage rather than a valid prediction protocol; this needs a validation-based reproduction.","rationale":"The paper proposes a practically appealing framework and the cost model is detailed, but the central accuracy claim rests on the QLoRA predictor being a genuine predictor. The manuscript explicitly says the projector is selected on peak test accuracy, while the calibration target is also test accuracy. That is a concrete methodological problem, not just an untested transfer assumption, and it directly threatens the 1.09% MAE headline. The reader's weakest_assumption focused on whether the linear calibration transfers across tasks; that is a valid secondary concern, but the test-set selection issue is more immediate and more specific. If the stated protocol is what was actually run, the reported alignment between predicted and actual fine-tuning performance is not an unbiased estimate, and the cost-saving claim is also vulnerable because the calibration runs needed to fit a and b are not itemized. The proposed reproduction is cheap and decisive: selecting checkpoints on a true validation split and recomputing Table 1 would show whether the result survives. Until that is done, the manuscript as written does not support acceptance, hence the recommendation to reject the current version while acknowledging that a corrected rerun could change the verdict.","tokens_in":22184,"tokens_out":10242,"duration_ms":113134,"concrete_test":"Re-run the QLoRA predictor for MMLU, Winogrande, ARC-Challenge, and HellaSwag with the projector checkpoint selected on the 10% validation split instead of the test set, keeping all other settings fixed, and recompute the Table 1 MAE and CRR. If the mean absolute error rises materially (e.g., from 1.09% to above roughly 2-3%) or the selected optimal strategies change, the central accuracy claim depends on test-set leakage. As a secondary check, report how many actual QLoRA calibration runs were used to fit a and b and include their cost in the reported Ours cost.","verdict_should_be":"REJECT","load_bearing_attack":"The most load-bearing concern is the fine-tuning predictor's evaluation protocol. Appendix C.1 states that for the four general-domain tasks, the contrastive linear projector is selected by 'peak test accuracy over 300 iterations,' while Appendix B says all adapted models are evaluated on the test set. Thus the same test labels are used both to choose the checkpoint of the projector performance π_φ'' and to compute the actual fine-tuning accuracy to which π_φ'' is calibrated via π_hat = a·π_φ'' + b (Section 4.2). This is selection on the test set: it can only make the predicted/actual alignment in Table 1 and Figures 6-8 look better than an honest held-out protocol would, and it gives no unbiased estimate of transfer to a new task. The 1.09% MAE and near-oracle selection results are therefore not clean evidence that COSMOS predicts QLoRA outcomes without expensive trials; they may be an artifact of test-label leakage. If 'test accuracy' was intended to mean the 10% validation split, the manuscript must state this and the pipeline must be rerun with checkpoints chosen on validation only. Relatedly, the calibration of a and b is underspecified: fitting two parameters requires paired actual fine-tuning outcomes, but the number and cost of those runs are not reported, so the 92.72% CRR may be understated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper formalizes the model--strategy selection problem for LLMs and proposes COSMOS, a unified prediction framework that estimates both performance and cost of adaptation strategies. For QLoRA fine-tuning, COSMOS trains a lightweight linear probe on frozen bidirectional embeddings and calibrates probe accuracy to actual fine-tuning accuracy through a linear map (Section 4.2). For retrieval-augmented ICL, it fits an exponential saturation curve to performance at a small number of shot counts (Section 4.3). Experiments across eight benchmarks with 55 QLoRA/ICL configurations report a mean absolute error of 1.09% and an average cost reduction of 92.72% (Table 1), with appendices extending the evaluation to five model families, limited data access, and a GPT-4o fine-tuning cost case study.","tokens_in":22574,"tokens_out":5091,"duration_ms":50024,"significance":"If the reported protocol is clean, the paper makes a practically valuable contribution: a cheap proxy-plus-calibration method for comparing training-time and test-time adaptation strategies, accompanied by a transparent cost model. The empirical scope is substantial, covering eight tasks, two strategy families, and up to 275 configurations, and the limited-data and expanded-model appendices are useful robustness checks. The two prediction assumptions---linear proxy-to-fine-tuning calibration and exponential ICL saturation---are simple and falsifiable. However, as written, the manuscript does not establish that the headline numbers come from an unbiased evaluation protocol: checkpoint selection on test accuracy and the inclusion of calibration points in the ICL evaluation undermine the central accuracy and cost-savings claims until corrected.","major_comments":[{"comment":"Appendix C.1 states that for the four general-domain tasks the contrastive linear projector is selected by 'peak test accuracy over 300 iterations,' while Appendix B states that all adapted models are evaluated on the test set. The projector accuracy pi_phi'' is the input to the calibration pi_hat = a*pi_phi'' + b in Section 4.2, and the actual fine-tuning accuracies to which it is calibrated are test-set accuracies. Choosing the projector checkpoint with the same test labels therefore constitutes selection on the test set: it can only improve the apparent predicted-actual agreement in Table 1 and Figures 6-8 relative to an honest validation-based protocol. The main claims of a 1.09% MAE and near-oracle strategy selection are not clean evidence of transfer to a new task. Please rerun the pipeline with checkpoints selected on the validation split only; if 'test accuracy' was intended to mean the 10% validation split, this must be stated explicitly and the implications for the reported protocol must be addressed.","section":"Appendix C.1 / Section 4.2"},{"comment":"The ICL predictor fixes pi0 as the lower of zero-shot and 1-shot performance and then fits alpha and beta to the measured 1-shot and 8-shot points (Appendix C.2). With two free parameters and two calibration points, the fitted curve passes exactly through the 1-shot and 8-shot measurements, so predictions at those shot counts are trivially accurate. Because the 55-configuration grid in Section 5 includes 1-, 2-, 4-, 8-, and 16-shot configurations, the reported aggregate MAE and selection results include configurations whose 'prediction' is a restatement of measured data. Please report held-out shot counts (e.g., 2, 4, 16) or use a leave-one-out protocol for the curve fit.","section":"Section 4.3 / Appendix C.2"},{"comment":"The calibration of a and b is underspecified. Learning two parameters requires paired actual fine-tuning outcomes, but the manuscript does not state how many QLoRA runs were performed for calibration, on which data portions and iteration counts they were run, or whether the same a and b are applied to all configurations. Since these calibration runs are genuine fine-tuning runs, their cost must be included in 'Ours Cost' in Tables 1 and 3; otherwise the reported 92.72% cost reduction is overstated. Please specify the calibration budget explicitly and account for it in the cost model, or justify why the calibration cost is negligible.","section":"Section 4.2 / Appendix C.1"},{"comment":"All headline numbers are averages over three seeds, but no error bars, standard deviations, or confidence intervals are reported for the MAE and CRR values in Tables 1 and 3. Figure 4 shows substantial seed variance in low-cost regimes, so it is not possible to assess whether the 1.09% MAE or the per-task CRR differences are stable. Please report variability for the main metrics.","section":"Section 5 / Tables 1 and 3"}],"minor_comments":[{"comment":"There is a typesetting error in the cost-efficiency condition: 'cpredict(Pj,k,Cj,k) much less than cadapt(T_j^omega, f_k)' is missing the subscript on the adaptation cost; the claim should read cpredict(Pj,k,Cj,k) << cadapt(T_j^omega, f_k).","section":"Section 3.2"},{"comment":"Table 4 uses fractional accuracy values (e.g., 0.944) while Tables 1 and 3 use percentages (e.g., 94.4); please unify the units across all tables.","section":"Table 4 / Tables 1 and 3"},{"comment":"The terms 'Range Avg Acc' and 'Range Avg Cost' are introduced without definition; please define them in the caption or text so the reader can interpret the baseline comparison.","section":"Appendix D.1"},{"comment":"Figure 5 appears to be a schematic; please state explicitly whether the old and new Pareto frontiers are computed from the experimental data or are illustrative.","section":"Figure 5"},{"comment":"The calibration set size is described as '200 examples or 10% of the training set, whichever is larger,' but the manuscript does not specify which of these was used for each task; please list the calibration set size per task in the appendix.","section":"Appendix C.1"},{"comment":"The statement that FPB achieves 'MAE of 0.007 for both performance and cost predictions' should specify the unit for the cost MAE (dollars) to avoid ambiguity.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The test-set checkpoint selection in Appendix C.1 is the most serious issue; if the authors reproduce the headline results with validation-only checkpoint selection and report error bars, the paper would be a solid empirical contribution. I also note that no code release or data artifacts are mentioned, which makes validation of the reported protocols harder."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me give you the short version. COSMOS is a sensible engineering package: linear probes on frozen embeddings for QLoRA, an exponential saturation fit for retrieval-augmented ICL, and a clean formalization of model-strategy selection under cost. The empirical study is broad—eight tasks, 55 strategy combinations, an expanded model pool—and the ICL results look genuinely solid. The 1.09% MAE and 92.72% cost reduction, however, are not trustworthy out of the box because the QLoRA predictor is selected on test labels, and the calibration cost is not auditable.\n\nWhat's actually new: the unified framework that lets you compare training-time and test-time adaptation under one cost model. The individual predictors are known components, but the packaging and the cost analysis are a real contribution. The ICL scaling-law predictor is honest and effective: fitting an exponential to 1-shot and 8-shot actual runs and predicting 2, 4, 16 is a legitimate extrapolation check, and the held-out errors are small.\n\nNow the soft spots, in order of severity. First, Appendix C.1 says the contrastive linear projector is selected on 'peak test accuracy over 300 iterations.' That is using the same test labels that later define actual fine-tuning accuracy, so the reported MAE and near-oracle selection are not unbiased. The authors need to rerun with checkpoints chosen on validation only. Second, the calibration of a and b is underspecified: how many actual QLoRA runs are required, on which data portions, and are those runs included in the reported 'Ours Cost'? Without that, the 92.72% CRR cannot be verified. Third, results are averaged over three seeds but no error bars are shown; that's a minor fix.\n\nThe paper's formalization and empirical scope make it worth a serious referee, but the QLoRA prediction claim needs a clean re-evaluation before I would trust it. This is a paper for practitioners who need a cheap way to choose between fine-tuning and ICL under a budget, and for researchers working on cheap model and strategy selection. I would not cite it in its current form, but I would bring it to reading group as a case study in evaluation pitfalls. My recommendation: send to peer review with a required fix for the checkpoint selection and a full disclosure of calibration costs and code.","headline":"Useful framework and broad experiments, but the headline QLoRA accuracy and cost claims rest on a test-set selection protocol and underspecified calibration that need fixing before I'd trust them.","tokens_in":23044,"tokens_out":4158,"would_cite":false,"duration_ms":41608,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"COSMOS claims that the performance and cost of QLoRA fine-tuning and retrieval-augmented in-context learning can be predicted cheaply enough to replace exhaustive strategy search, with an average prediction error of 1.09 percentage points…","keywords":["LLM adaptation","strategy selection","performance prediction","cost prediction","QLoRA fine-tuning","retrieval-augmented in-context learning","scaling laws","proxy models"],"falsifier":"Take a task outside the eight benchmarks, learn the linear rescaling coefficients on a 10% validation subset, predict QLoRA accuracy for all data portions from 0.1 to 1.0 and all training iterations from 4 to 8, then run those configurations; if the predictions miss by more than roughly two percentage points on average, or if coefficients learned on one task do not reduce error on another task, the central transfer assumption fails. Likewise, the exponential curve fit from one-shot and eight-shot measurements should predict four-shot and sixteen-shot performance; a persistent bias there would sink the in-context-learning predictor.","tokens_in":21990,"feed_emoji":"📉","tokens_out":11122,"duration_ms":98367,"temperature":0.7,"pith_summary":"Large language models can be adapted to a task in many ways, and choosing the best model–strategy combination normally means running expensive trials. This paper asks whether both the performance and the cost of adaptation can be predicted instead, and it answers yes with COSMOS, a framework that couples cheap, strategy-specific predictors. For QLoRA fine-tuning, a linear probe on frozen embeddings is calibrated to real fine-tuning accuracy; for retrieval-augmented in-context learning, an exponential saturation curve is fit from as few as two shot-count measurements. Across 55 strategy combinations on eight benchmarks, the predicted best strategy misses the actual best by 1.09 percentage points on average while cutting the cost of selection by 92.72% (up to 98.71%). If the prediction mechanism transfers beyond these benchmarks, model and strategy choice becomes a cheap data-driven decision rather than an exhaustive search.","feed_headline":"Predicting LLM adaptation cuts selection cost by 92.72%","feed_subtitle":"Lightweight probes and 2-point scaling laws pick near-optimal fine-tuning and RAG prompting without full runs.","key_machinery":"The load-bearing object is the predictor pair $(P_{j,k}, C_{j,k})$ inside a strategy navigator: a performance predictor and a cost predictor for each model–strategy pair. For fine-tuning, the performance predictor is an embedding-augmented linear probe trained on frozen bidirectional representations and calibrated by $\\hat{\\pi} = a\\,\\pi_{\\phi''}+b$, where the coefficients are learned on a small validation subset. For in-context learning, the performance predictor is the exponential saturation curve $\\hat{\\pi}(d)=\\alpha(1-e^{-\\beta d})+\\pi_0$, fit to sparse shot-count measurements. These predictors are what let the framework score all configurations without running them; the cost model makes the comparison concrete by converting GPU time and token use into dollars.","core_discovery":"COSMOS formalizes strategy selection as choosing a model, an adaptation strategy, and a configuration to maximize a score that trades off performance against cost, and it replaces exhaustive evaluation with a pair of predictors for each model–strategy combination. For QLoRA, the performance predictor turns a causal language model into a bidirectional encoder, trains a linear projector on frozen embeddings, and calibrates the projector's accuracy to true fine-tuning accuracy through an affine map $\\hat{\\pi} = a\\pi_{\\phi''}+b$ with coefficients learned on a small validation subset. For retrieval-augmented in-context learning, the predictor fits the exponential saturation law $\\hat{\\pi}(d)=\\alpha(1-e^{-\\beta d})+\\pi_0$ to one-shot and eight-shot measurements and then extrapolates to other shot counts. The cost predictor accounts for adaptation, evaluation, and prediction costs, so both strategies are compared on a common dollar axis. The experiments show that the predicted optimal strategy matches the actual optimal strategy with 1.09 percentage points mean absolute error while spending about 7% of the exhaustive-evaluation cost.","pith_inferences":["An untested extension: the same frozen-embedding calibration could be tried on full fine-tuning, unquantized LoRA, or adapter methods; if the linear transfer holds there, one predictor would cover most training-time strategies.","Another extension the paper leaves implicit: the exponential saturation law is fixed in form and fit to only two points, so a small learned prior over tasks could allow ICL gains to be predicted before running any few-shot measurements at all.","The headline cost reduction is computed under specific GPU-hour and token prices, so real-world savings scale with those prices and with the size of the configuration grid; the 92.72% figure is an illustration of the mechanism, not a universal constant."],"forward_implications":["A practitioner can select a near-optimal model–strategy configuration by running only lightweight proxy and validation experiments, not the full grid of fine-tuning and prompting trials.","Fine-tuning and in-context learning can be compared on one performance-cost axis, so the choice between them becomes a resource-level decision rather than a default preference.","Expanding model routing to include adaptation strategies moves the Pareto frontier, meaning a router can offer better accuracy at the same cost or lower cost at the same accuracy.","Cost savings grow with task scale and with the number of configurations searched, so the method becomes more valuable as the strategy pool grows.","The linear calibration and exponential saturation predictors together give an end-to-end estimate of total deployment cost, including prediction, adaptation, and evaluation, before any expensive run is started."],"supporting_citations":[{"why":"Supplies the bidirectional-embedding transformation that turns a causal language model into the encoder used for fine-tuning prediction.","marker":"[1]"},{"why":"Defines QLoRA, the training-time adaptation strategy whose accuracy the linear-probe predictor is calibrated to match.","marker":"[7]"},{"why":"Provides the Llama 3 8B model used as the strong base model in the main experiments.","marker":"[8]"},{"why":"Provides the Gemma 2B model used as the weak base model and routing reference point.","marker":"[10]"},{"why":"Supplies the BM25 retriever that selects demonstrations for retrieval-augmented in-context learning.","marker":"[25]"},{"why":"Shows that language-model performance is predictable from observable quantities, motivating the low-sample scaling-law predictor.","marker":"[26]"},{"why":"Documents test-time compute scaling behavior, providing context for predicting in-context learning gains from few measurements.","marker":"[32]"}],"fun_headline_variants":["COSMOS predicts LLM adaptation at 7% of the cost","92.72% cheaper LLM strategy selection with COSMOS","COSMOS: accurate adaptation forecasts, 92.72% cheaper","Predict fine-tuning and RAG performance without full runs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole prediction pipeline rests on the assumption that real fine-tuning accuracy tracks a cheap linear probe on frozen word embeddings through one fixed linear rescaling, and that the rescaling learned from a small validation set keeps working across every data size, training length, and task; the paper demonstrates this only on the eight tasks it evaluates.","fun_headline_variants_meta":{"raw":{"variants":["COSMOS predicts LLM adaptation at 7% of the cost","92.72% cheaper LLM strategy selection with COSMOS","COSMOS: accurate adaptation forecasts, 92.72% cheaper","Predict fine-tuning and RAG performance without full runs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000228,"raw_usage":{"total_tokens":1476,"prompt_tokens":944,"completion_tokens":532,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":460}},"tokens_in":560,"tokens_out":532,"duration_ms":5215,"temperature":1.0,"reasoning_tokens":460,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:09:17.617898+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a task outside the eight benchmarks, learn the linear rescaling coefficients on a 10% validation subset, predict QLoRA accuracy for all data portions from 0.1 to 1.0 and all training iterations from 4 to 8, then run those configurations; if the predictions miss by more than roughly two percentage points on average, or if coefficients learned on one task do not reduce error on another task, the central transfer assumption fails. Likewise, the exponential curve fit from one-shot and eight-shot measurements should predict four-shot and sixteen-shot performance; a persistent bias there would sink the in-context-learning predictor.","supporting_citations":[{"cited_title":"The probabilistic relevance framework: Bm25 and beyond","cited_arxiv_id":null,"evidence_quote":"Supplies the BM25 retriever that selects demonstrations for retrieval-augmented in-context learning."}],"review_version":1}