{"id":"a221d74a-524c-40c7-a9ff-d9bda84e40e7","arxiv_id":"2608.00632","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Frontier Learning — concatenating pretrained models' representations and predictions and fitting a regularized linear/logistic head on target labels — matches or improves on the best individual zero-shot, fine-tuned, or directly trained model in the reported experiments.","lead":"Frontier Learning concatenates internal representations from white-box models and prediction outputs from black-box models, then trains a small regularized model on target labels to combine them. It reports matching or beating the best single reuse strategy in simulations and two real-world distribution-shift benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The no-worse training-risk guarantee is only conditional on exact representability of each white-box candidate's head by the shallow frontier learner, a condition that fails for general nonlinear heads and is conceded in §3.4.","rationale":"The advertised theoretical contribution is the unconditional no-worse training-risk guarantee. Its weakest link is the containment F_baseline ⊆ F_FL, exactly as the reader identified. The paper's own §3.4 caveat narrows the claim to cases where the candidate head belongs to the frontier learner class, which is not generally true and is not stated in the abstract. The experiments may satisfy the condition because all used heads are linear in the supplied representations, which is consistent with the empirical results but does not rescue the general statement. A constructive counterexample with a nonlinear head would settle the issue. If the authors restate the guarantee as conditional or include final prediction outputs for white-box candidates, the theoretical claim becomes sound. This does not change the reader's conditional verdict.","tokens_in":16046,"tokens_out":11811,"duration_ms":152593,"concrete_test":"Use one white-box candidate with r(x)=x in R^2 and a nonlinear head c_m(x)=1[x1*x2 > 0], so the baseline has zero training loss on an XOR-labeled training sample. Run the Section 3.3 logistic frontier learner on the same target training set with only this candidate and λ=0. If its training loss is positive while the baseline's is zero, F_baseline ⊄ F_FL and the no-worse training-risk guarantee is violated. This isolates the containment step without regularization confounds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4's proof of the no-worse guarantee rests on F_baseline ⊆ F_FL. In the construction, white-box candidates contribute only r_m(x), not their prediction q_m(x); black-box candidates contribute q_m(x). A white-box baseline is f_m(x) = c_m(r_m(x)), so containment requires c_m to lie in the frontier learner class G used at the top level. The paper uses linear, logistic, and multinomial logistic heads (Section 3.3), while c_m is an arbitrary neural-network head in general. When c_m is nonlinear, no choice of g_theta ∈ G can reproduce f_m, so F_baseline ⊄ F_FL and the inequality inf_{f∈F_FL} R_hat_T(f) ≤ inf_{f∈F_baseline} R_hat_T(f) is not guaranteed. The paper itself concedes this in §3.4: exact recovery of f_m from r_m 'requires that the candidate head c_m(·) belongs to the class of frontier learners used.' Thus the abstract's unconditional statement that the hypothesis class 'contains predictors obtained by zero-shot reuse, fine-tuning, and direct training as special cases' is false for general nonlinear heads; it is only a conditional containment result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Frontier Learning, a framework that combines a library of pretrained models under mixed access regimes (white-box representations, black-box predictions) by concatenating their signals into a single vector and fitting a regularized shallow learner (linear, logistic, or multinomial logistic) on labeled target data. The central theoretical claim is that the resulting hypothesis class contains zero-shot, fine-tuned, and directly trained baselines as special cases, yielding a no-worse training-risk guarantee under empirical risk minimization. The method is evaluated in a controlled simulation with varying source-target feature overlap and in two real-world tasks: visual domain adaptation on DomainNet/VisDA and in-hospital mortality prediction on MIMIC-IV-Notes. The empirical results show that Frontier Learning improves over individual baselines on the primary metrics in all reported settings.","tokens_in":16322,"tokens_out":5919,"duration_ms":64383,"significance":"If the theoretical claim is stated precisely, the paper makes a useful and practical contribution: it formalizes a lightweight, mixed-access alternative to committing to a single reuse strategy under distribution shift. The framework is simple, computationally cheap, and backed by reproducible code and simulations with 100 repetitions. The empirical gains are largest when no single candidate dominates, which is the motivating regime. The theoretical no-worse property is a modest but real extension of stacking/Super Learner ideas to libraries that include internal representations, provided the candidate heads are within the class of final learners. The main weakness is that the guarantee is presented more strongly than the conditions justify.","major_comments":[{"comment":"The containment claim F_baseline ⊆ F_FL is asserted unconditionally, but for white-box candidates the frontier representation contains only r_m(x), not q_m(x)=c_m(r_m(x)). Exact recovery of f_m therefore requires c_m ∈ G, and the paper's own text in §3.4 concedes this ('requires that the candidate head c_m(·) belongs to the class of frontier learners used'). For general nonlinear heads, F_baseline ⊄ F_FL and the no-worse inequality does not follow. Please restate the guarantee as a conditional result with an explicit representability assumption (e.g., Assumption: for each white-box candidate, c_m ∈ G) and align the abstract and introduction with that conditional statement. The reported experiments appear to use linear/multinomial logistic heads, so the condition is plausibly satisfied there, but this should be stated explicitly.","section":"Abstract and §3.4"},{"comment":"The stated guarantee concerns inf_{f∈F_FL} \\hat R_T(f), i.e., the unregularized empirical risk minimizer. The actual procedure minimizes a penalized objective with λ selected on validation data. Unless λ=0 is guaranteed to be chosen, the deployed model is not an unregularized ERM and may have higher training risk than a baseline. As written, the no-worse property applies to an idealized ERM, not to the predictor produced by the algorithm. Please clarify this gap, and either prove that the chosen λ yields the ERM solution or explicitly scope the guarantee to the unregularized ERM over F_FL.","section":"§3.4 and §3.3"}],"minor_comments":[{"comment":"The abstract says Frontier Learning 'matches or outperforms the strongest individual reuse strategy' across all settings. In Table 3, however, Frontier Learning's AUROC (0.9452±0.0187) is lower than that of Fine-tuning (MICU) (0.9619±0.0065). Please qualify the claim to refer to the primary metric in each application (AUPRC for MIMIC, accuracy for DomainNet).","section":"Abstract and §5.2.2"},{"comment":"The notation f_zs, f_ft, f_dir is used ambiguously: sometimes it denotes the full candidate model, sometimes its prediction output, and sometimes its internal representation. Please define each symbol in terms of q_m and r_m.","section":"§3.4"},{"comment":"The numeric labels on the plotted MSE values are small and partially overlapping. Consider a table or larger font, or move the values to the caption.","section":"Figure 2"},{"comment":"The simulation 'shortcut shift' is created by training source and target models on different feature masks. This is a nonstandard use of the term 'shortcut shift'; please clarify the terminology in the setup, since the outcome mechanism is actually identical across domains.","section":"§4.2"},{"comment":"The dataset name is written inconsistently as 'MIMIC-IV-Note' and 'MIMIC-IV-Notes'. Use the official name consistently.","section":"§5.2.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical contribution with a clear practical message, but the theoretical guarantee is overstated in the abstract and Section 3.4. The fix is local rather than structural: state the containment condition explicitly and qualify the empirical 'matches or outperforms' claim. I would be comfortable with acceptance after these revisions. The choice of target domains (Painting, CVICU) is justified by transfer difficulty, but since only one target per application is used, the generality of the empirical gains is limited; this is acceptable for the paper's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe one-sentence read: this is stacking/Super Learner applied to a library of pretrained models under distribution shift, with the modest twist that white-box candidates contribute internal representations and black-box candidates contribute outputs. The paper is honest enough to cite Wolpert and van der Laan, so it does not oversell novelty on that front. What is useful is the practical framing: rather than committing to one reuse strategy, learn a shallow target-domain combination of whatever each candidate exposes.\n\nOn the positive side, the empirical evaluation is careful: held-out test data, validation-based hyperparameter selection, repeated runs with standard deviations. The simulation varies source-target compatibility and the method beats all individual baselines across the 0/10/20/30 shared-feature settings. The two real-world studies are honestly reported, and the gains are modest but consistent (1.38 accuracy points on Painting, 0.014 AUPRC on CVICU). The paper also ships code and states limitations. That is real evidence of careful work.\n\nNow the soft spots, in decreasing order of size.\n\nFirst, the no-worse training-risk guarantee is conditional, not unconditional. The formal containment F_baseline ⊆ F_FL requires that each baseline predictor be exactly representable by the final shallow learner. That holds for white-box heads that are linear in the representation and for black-box outputs if they enter as logits. It does not hold in general for nonlinear candidate heads, and may also fail for probability-valued black-box outputs when the frontier learner is logistic or multinomial. The paper does qualify this in §3.4, but the abstract and the formal claim state an unconditional containment, which is more than the argument supports. This should be restated as conditional, with the cases where it applies spelled out.\n\nSecond, the experiments omit the most natural baseline: ordinary stacking using the same candidate predictions. Without that comparison, the reader cannot tell how much of the gain comes from representation access versus stacking itself. It is an essential control.\n\nThird, the evaluation picks one target domain per dataset (Painting; CVICU) and selects candidate models using the same transfer matrices that motivated the choice. That is arguably a sensible stress test — Painting has heterogeneous incoming transfer, CVICU is the hardest target — but it is still one target per dataset, and the candidate selection is informed by the same source-target compatibility information. A second target would make the headline more robust.\n\nNone of these are load-bearing flaws: the empirical core is solid in the settings studied, and the theory is correct as a conditional statement. But the abstract and the formal section need a rewrite, and the stacking baseline should be added.\n\nFor peer review: yes, send it out. A good referee can fix the framing and ask for the missing baseline. This is a useful paper for people thinking about model reuse under shift, even if the headline is not as strong as the authors seem to want.","headline":"A clean stacking-style recipe for mixing black-box and white-box pretrained models under shift; the empirical story mostly holds, but the no-worse guarantee is conditional and the paper should say so up front.","tokens_in":16818,"tokens_out":4638,"would_cite":true,"duration_ms":58793,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single shallow learner can match or beat every individual reuse strategy under distribution shift.","keywords":["distribution shift","transfer learning","foundation models","model ensemble","representation learning","black-box models","zero-shot reuse","regularized empirical risk minimization"],"falsifier":"Take a white-box candidate with a nonlinear prediction head, include only its internal representation in the frontier, and fit the shallow frontier learner on target data while the candidate itself is fine-tuned on the same data; if the frontier learner's training risk exceeds the candidate's, the containment premise fails exactly as the paper predicts, separating representable from non-representable regimes.","tokens_in":15891,"feed_emoji":"🤖","tokens_out":4728,"duration_ms":66387,"temperature":0.7,"pith_summary":"Frontier Learning is a proposal for prediction under distribution shift that refuses to commit in advance to one pretrained model or one reuse strategy. Instead it takes everything the candidate models expose—hidden representations from white-box models, prediction outputs from black-box models—concatenates those signals into a single feature vector, and fits a regularized shallow predictor on labeled target data. Because zero-shot reuse, fine-tuning, and direct target training are all special cases of the resulting hypothesis class, the paper proves that empirical risk minimization over the frontier learner achieves training risk no worse than any of those baselines. In simulations, visual domain shift, and clinical ICU mortality prediction, the frontier learner matches or outperforms the strongest individual candidate, with the largest gains when no single baseline is reliable. The payoff, if the claim holds, is that model reuse becomes a learnable combination of heterogeneous candidates rather than a bet on one strategy.","feed_headline":"One shallow learner matches or beats every reuse strategy","feed_subtitle":"Concatenating hidden representations and black-box outputs yields a no-worse guarantee over zero-shot, fine-tuned, and direct models.","key_machinery":"The load-bearing object is the frontier representation Ψ(x) = concatenation over candidates of ψ_m(x), where ψ_m is the extracted latent representation for white-box models and the prediction output (logits preferred) for black-box models. A regularized linear, logistic, or multinomial-logistic learner is fit on Ψ(x) using labeled target data. This single construction does two jobs: it makes the candidate models complementary inputs rather than competing choices, and it supplies the containment F_baseline ⊆ F_FL that yields the no-worse training-risk guarantee.","core_discovery":"The central claim is that the effective Pareto frontier of predictive strategies under distribution shift—zero-shot, fine-tuning, direct training, and black-box API prediction—can be approximated by a single hypothesis class: linear or logistic functions of the concatenated signals from all candidate models. The paper calls this class Frontier Learning. Its statistical argument is containment: when a candidate's prediction output is part of the frontier representation, that candidate's predictor belongs to the frontier class; when only its internal representation is used, the candidate is exactly recoverable only if its prediction head belongs to the shallow learner class. Under that contain","pith_inferences":["I infer an immediate practical corollary the paper leaves implicit: Frontier Learning can wrap any library of open-weight and API-only models without retraining them, so it is directly applicable to current foundation-model ecosystems.","A testable extension would be to read the learned frontier weights as a transfer diagnostic, with blocks shrunk to zero identifying candidate signals that carry no target information under a given shift.","The containment argument suggests a cross-fitting variant of the same idea would convert the training-risk guarantee into an out-of-sample stacking guarantee, closing the gap the paper flags between train and test risk.","The paper's linear-case analysis implies that representation-level concatenation can outperform ordinary prediction stacking whenever the final heads discard task-relevant structure; testing this on a task where logits are uninformative but embeddings are predictive would isolate the mechanism."],"forward_implications":["An analyst no longer needs to pick zero-shot, fine-tuning, or direct training before seeing target labels; the frontier learner can recover whichever baseline is best and often does better by combining them.","Weak or poorly transferred candidates are not wasted: their outputs or representations can be included and downweighted by regularization, so the method degrades gracefully instead of collapsing to a bad choice.","The guarantee holds on the training sample for every baseline whose predictor is representable in the frontier class, which makes it a safe default for heterogeneous model libraries.","The largest empirical improvements occur when no single baseline is reliable across the shift range, so the method is best understood as a robust-combination tool rather than a way to polish an already-strong model.","Because candidate models remain frozen and only a shallow head is trained, the added computational cost is small relative to training or fine-tuning the candidates themselves."],"fun_headline_variants":["One shallow learner beats or ties all reuse strategies","Frontier Learning: one model that never underperforms","Concatenating outputs gives a no-worse guarantee","Linear model on combined features tops every baseline","Single frontier learner matches best of zero-shot, FT, direct"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The guarantee collapses if a baseline's predictor cannot be exactly represented by the frontier learner—most clearly when only an internal representation is used and the candidate's final prediction head is a nonlinear function outside the shallow class, a case the paper itself notes voids the containment.","fun_headline_variants_meta":{"raw":{"variants":["One shallow learner beats or ties all reuse strategies","Frontier Learning: one model that never underperforms","Concatenating outputs gives a no-worse guarantee","Linear model on combined features tops every baseline","Single frontier learner matches best of zero-shot, FT, direct"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000175,"raw_usage":{"total_tokens":1147,"prompt_tokens":794,"completion_tokens":353,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":277}},"tokens_in":538,"tokens_out":353,"duration_ms":6144,"temperature":1.0,"reasoning_tokens":277,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T01:38:06.126450+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a white-box candidate with a nonlinear prediction head, include only its internal representation in the frontier, and fit the shallow frontier learner on target data while the candidate itself is fine-tuned on the same data; if the frontier learner's training risk exceeds the candidate's, the containment premise fails exactly as the paper predicts, separating representable from non-representable regimes.","supporting_citations":[],"review_version":1}