{"id":"ac472285-99ce-4e4d-a208-bc27eaa422cb","arxiv_id":"2507.12604","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Encoders trained to reproduce landmarker distances capture that structure, but the resulting representations do not significantly improve warm-starting of Bayesian hyperparameter optimization.","lead":"Two machine learning algorithms were built to turn tabular datasets into short numeric fingerprints, aiming to match how well different xgboost settings perform on each dataset. The fingerprints matched their target in controlled tests, but did not significantly speed up or improve hyperparameter tuning, so the core result is a caution for meta-learning approaches.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'no HPO gain' conclusion is confounded by the k-NN warm-start selection: even the true-landmarker baseline is not an upper bound, so the central negative claim is not yet established.","rationale":"Reader identified the Requirement as the weakest assumption; my concern is the downstream realization of that Requirement. The paper trains encoders with distance-alignment losses and validates alignment via Table 1, then uses one specific translation: Euclidean k-NN in representation space with one best configuration per neighbor. The landmarker baseline is presented as an upper bound but is the same k-NN pipeline with true landmarker distances; it only upper-bounds the encoder pipeline, not warm-starting in general. Because the landmarker vector has 100 dimensions (portfolio size, §4.1/4.2), nearest-neighbor search is not obviously a faithful way to identify transferable configurations, and the paper gives no evidence that this selector is a strong warm-start method. Thus the core negative result—encoder alignment does not translate to HPO gains—could be an artifact of the selector rather than a property of the representations. This is not an accusation of misconduct; it is a missing control. The paper's own §5 admits the warm-start algorithm is a possible cause. A single oracle experiment (best true portfolio configuration as warm-start) would settle whether warm-starting can matter at all; if it cannot, the encoders' failure is moot but the paper's conclusion is robust; if it can, the encoder evaluation needs to be redone with a stronger transfer mechanism. I still credit the paper for honest reporting and reproducible intent, but the central claim is conditional on this control.","tokens_in":9096,"tokens_out":14527,"duration_ms":173534,"concrete_test":"Rerun the ADTM/CD evaluation of §4.3.2/§4.5.3 with an oracle warm-start: for each meta-validation dataset, use the true landmarker vector to pick the portfolio configuration(s) with the best actual performance on that dataset (top-1 or top-5) as the 5 warm-start points, and compare against no-warm-start and random-from-portfolio over the same 20 BO iterations. If this oracle is statistically superior at iteration 20, the k-NN landmarker baseline is not an upper bound and the reported absence of encoder gains is confounded by the selection mechanism; if the oracle also shows no gain, the negative claim is robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central negative claim (abstract; §5) is that encoders can align with landmarkers yet 'may not directly translate' to HPO warm-start gains. The load-bearing support is the landmarker baseline in §4.4, labeled an upper bound, but it is only a k-NN selector: §4.3.2 finds nearest datasets in landmarker/representation space and proposes the best configuration from each neighbor (k=5, matching the 5 warm-start points). This is not an upper bound for warm-starting: Euclidean nearest neighbors in a 100-dimensional landmarker space need not be the datasets whose best configurations transfer to the target, and choosing one configuration per neighbor is a weak way to use even perfect landmarker information. Consequently, the observation in §4.5.3/Fig. 4b that even the landmarker baseline is statistically indistinguishable from random after 20 BO iterations may reflect the selection algorithm, not the encoders or the Requirement. The paper itself lists 'the specific warm-start algorithm' as a possible cause (§5) but never tests an oracle selector (e.g., the portfolio configuration with the best true landmarker performance for the target). Until such an oracle is run, the abstract's implication that encoder alignment is insufficient for HPO gains is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two encoder-based methods for representing tabular datasets, specifically designed for warm-starting Bayesian hyperparameter optimization (HPO). The core idea is a \"Requirement\": distances between dataset representations should align with distances between landmarker vectors (performance vectors of a portfolio of configurations). The first method trains a Dataset2Vec-style encoder with a metric-learning loss (Eq. 1) that minimizes the MSE between representation distances and landmarker distances. The second method adds a reconstruction head that predicts the landmarker vector from the latent representation, using the predicted vector as the dataset representation (Eq. 2). The encoders are evaluated on a meta-dataset of ~700 OpenML binary classification tasks with xgboost, using (i) distance correlation (Eqs. 3–4, Table 1) and (ii) warm-start gains in Bayesian HPO measured by ADTM and CD plots (Figs. 3–4). The main findings are that the proposed encoders achieve higher correlation with landmarker distances than the Dataset2Vec baseline, but these representations do not yield statistically significant improvements over random or heuristic baselines after full HPO optimization; even the true-landmarker baseline is not statistically distinguishable from baselines after 20 iterations. The authors conclude that encoder alignment with landmarkers may not directly translate to HPO warm-starting gains.","tokens_in":9294,"tokens_out":4371,"duration_ms":45614,"significance":"If the results are taken at face value, the paper provides a valuable negative result for the meta-learning community: it demonstrates that optimizing a dataset-representation objective to match landmarker distances does not automatically improve downstream HPO warm-starting. This challenges the assumption that such distance alignment is a sufficient signal for transfer in Bayesian HPO. The paper also contributes two concrete encoder architectures and a reproducible evaluation pipeline, with code provided. The evaluation design is thoughtful in separating the alignment metric from the target meta-task. However, the central negative claim is currently confounded by the warm-start selection algorithm used for all methods, including the landmarker baseline, which prevents the paper from establishing that the representation objective itself is the limiting factor. If the oracle experiment recommended below confirms the results, the paper would be a solid, useful contribution.","major_comments":[{"comment":"The landmarker baseline is described as an upper bound (§4.4, item 4), but it is implemented as a k-NN selector that retrieves the k=5 nearest datasets in landmarker space and proposes the best configuration from each. This is not an upper bound for warm-starting: Euclidean nearest neighbors in a 100-dimensional landmarker space need not be the datasets whose best configurations transfer to the target, and choosing one configuration per neighbor is a weak way to use even perfect landmarker information. Consequently, the observation in §4.5.3 and Figure 4b that the landmarker baseline is statistically indistinguishable from random after 20 HPO iterations may reflect the selection algorithm rather than the informative content of landmarkers or the proposed encoders. To support the abstract's claim that encoder alignment with landmarkers 'may not directly translate' to HPO gains, the authors should add an oracle baseline: for each target dataset, select the portfolio configuration that has the best true landmarker performance (or the best actual performance after evaluation) as the warm-start point. If that oracle also shows no significant gains after full optimization, then the negative conclusion would be sound. Until such an experiment is run, the central claim is not fully established.","section":"§4.4 and §4.5.3 (Landmarkers baseline and Warm-start results)"},{"comment":"The numerical comparison for the landmark reconstructor is internally inconsistent: the text states 'landmarker reconstructor with reconstruction loss of 0.0307, which is lower than the initial value at the beginning of the training – 0.0256'. Since 0.0307 > 0.0256, the claim that the loss decreased is false under the stated numbers. Either the reported final loss should be lower than the initial loss (e.g., swapping the values or correcting the printed figures), or the sentence should be rewritten to say the final loss is higher, which would contradict the claim that the method learned. This is a load-bearing detail because it is part of the evidence that the reconstructor successfully minimized its objective. Please correct the numbers and the comparison.","section":"§4.5.1 (Encoders training)"},{"comment":"The distance-correlation metric defined in Eq. (3) measures exactly the alignment that the metric-learning encoder's loss (Eq. (1)) optimizes during training, namely the correspondence between Euclidean distances of representations and Euclidean distances of landmarkers. Therefore, the high correlation reported for the metric-based encoder (0.332) in Table 1 is, to a substantial degree, a check of whether the training objective generalizes to held-out dataset pairs, rather than an independent confirmation that the representation captures landmarker properties beyond the optimized objective. The paper should explicitly acknowledge this overlap and, ideally, add an evaluation that is not directly aligned with the training loss—for example, the correlation of the predicted landmarker distances (Eq. 4) for the metric method, or a downstream task that tests transfer of the representation without being part of the loss.","section":"Eq. (3) and Table 1"},{"comment":"Table 1 mixes two different correlation formulas without clear labeling: for Dataset2Vec basic and metric learning, it reports Eq. (3) (correlation between representation distances and landmarker distances), whereas for the reconstruction approach it reports Eq. (4) (correlation between distances from reconstructed landmarkers to true landmarkers and true landmarker distances). These metrics are not directly comparable, yet the table presents them side by side under a single 'Correlation' column. This makes the cross-method comparisons in the table misleading. The authors should either report a single common metric for all methods (e.g., Eq. (3) for all, or a separate table column for each metric) or clearly annotate which rows use which equation and avoid implicit comparisons across rows.","section":"Table 1 and §4.3.1"}],"minor_comments":[{"comment":"The sentence 'The is performed in three steps' contains a typo; it should read 'This is performed in three steps' or 'The selection is performed in three steps'.","section":"§4.2"},{"comment":"The reference to 'In Figure 3a' and 'In Figure 3b' is ambiguous: the text says 'In Figure 3a, we present an ADTM plot... In Figure 3b, we extend that comparison showing scaled values...' but it is not clear which panel corresponds to which description. Also, the sentence 'the landmarker reconstruction is better than metric learning' should cite the specific panel and metric supporting this claim.","section":"§4.5.3"},{"comment":"The notation for indices is unclear: in Eq. (1), the pair index i is used to sum over N pairs, but the terms inside the summation are written with superscripts (i,1) and (i,2). Please clarify the indexing of the datasets and the summation bounds. Similarly, in Eq. (2), the subscript n is used in the sum but the terms are written with index i; the notation should be made consistent.","section":"Eq. (1) and Eq. (2)"},{"comment":"The standard deviations are reported over S=20 samples, but the number of dataset pairs N=1000 is not stated in the table caption. Please include the sample size and the definition of 'correlation' (Spearman) directly in the caption, and add a statistical test (e.g., confidence intervals or p-values) for the claim that correlations are 'significantly higher than 0'.","section":"Table 1"},{"comment":"The term 'formulated by ourselves' in the abstract is informal; consider replacing with 'proposed'. Also, in §5, the phrase 'too weak information' is grammatically incorrect; it should be 'too weak' or 'insufficient information'.","section":"§1 and §5"},{"comment":"'meta-validation serves as the set of data on which we calculate metrics' should be 'set of datasets' (or 'data') for grammatical agreement. Additionally, the dashed/dotted/solid line explanation in Figure 1 is not referenced in the text; please add a brief description of what each line type indicates.","section":"§2"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an important and timely question—whether dataset encoders trained to align with landmarkers are actually useful for HPO warm-starting. The negative result, if properly established, would be a valuable cautionary finding. However, the current evidence is weakened by two issues: (1) the landmarker baseline is only a k-NN selector, not an oracle, so the absence of significant gains cannot be attributed solely to the representation; and (2) the reconstruction loss numbers are reported inconsistently. The authors should run the oracle experiment and correct the loss reporting; these are feasible within the manuscript's scope. The code and reproducibility efforts are commendable. I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely useful negative result—training encoders to align with landmarkers doesn't automatically buy you HPO warm-start gains—but the paper's central claim is not as solid as the abstract implies, because the landmarker baseline they treat as an upper bound is only a k-NN selector and may be a weak one.\n\nWhat's good: the paper does a clean job of setting up two encoder objectives (metric learning on landmarker distances, landmarker reconstruction), evaluating them against Dataset2Vec, and including sensible baselines (random from portfolio, rank heuristic). The honest discussion in Section 5 is a plus. The finding that representations can correlate with landmarkers (0.47 for reconstruction, 0.33 for metric) yet still not improve warm-starting after full optimization is worth reporting, and the CD plots make the null result clear. The protocol itself is reusable.\n\nWhere it's soft: the stress-test note is right. The 'Landmarkers' baseline in Section 4.4 is described as an upper bound, but it's just nearest neighbors in landmarker space with one configuration pulled from each neighbor. There's no reason that's the best possible warm-start, and there's no oracle test (e.g., take the portfolio configuration that actually performs best on the target dataset). Without that, saying 'even landmarkers don't help' is really saying 'this particular k-NN warm-start algorithm doesn't help.' The authors list the algorithm as a possible cause in Section 5, but they don't isolate it. That's the main gap.\n\nAlso: the reconstruction loss in Section 4.5.1 (0.0307) is reported as lower than the initial value (0.0256), which is backwards; that looks like a typo but undermines the 'all approaches learn' claim. The distance-correlation metric (Eq. 3, Table 1) is somewhat self-referential for the metric encoder, since Eq. 1 optimizes exactly that; it's a legitimate held-out generalization check, but it shouldn't be read as independent evidence of usefulness. Reproducibility is partial: anonymous URL without versioning, and the OpenML dataset IDs aren't listed.\n\nBottom line: this is for AutoML/meta-learning folks. It deserves a serious referee, but not as-is. The main fix is to add an oracle or stronger landmarker-based warm-start to test whether the negative result is about the encoders or the selection mechanism. Then it would be a solid empirical paper.","headline":"Useful but not conclusive: the paper's 'alignment doesn't transfer' claim needs an oracle warm-start baseline before it holds.","tokens_in":9891,"tokens_out":2974,"would_cite":false,"duration_ms":30665,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper asks whether neural encoders can learn tabular dataset representations that capture landmarker information and whether that helps warm-start Bayesian hyperparameter optimization (HPO).","keywords":["tabular dataset representation","landmarkers","warm-starting","Bayesian hyperparameter optimization","metric learning","Dataset2Vec","meta-learning"],"falsifier":"Run the same warm-start protocol but replace encoder representations with the exact landmarker vectors in the nearest-neighbor search; if this oracle representation fails to improve over random portfolio selection at the end of optimization, the paper's claim that HPO is largely robust to warm-start selection is confirmed, while if it succeeds, the encoders' weak alignment is the bottleneck.","tokens_in":8827,"feed_emoji":"⚙️","tokens_out":6218,"duration_ms":61861,"temperature":0.7,"pith_summary":"The paper asks whether neural encoders can learn tabular dataset representations that capture landmarker information and whether that helps warm-start Bayesian hyperparameter optimization (HPO). It proposes two encoders, one trained by metric learning to align representation distances with landmarker distances and one trained to reconstruct landmarker vectors. The encoders do learn landmarker-aligned representations, with distance correlations rising well above the Dataset2Vec baseline. However, in warm-starting Bayesian HPO, neither encoder yields a statistically significant improvement over random or heuristic portfolio baselines after the full optimization run. The paper concludes that distance alignment with landmarkers may be necessary but is not sufficient for this meta-task.","feed_headline":"Matching landmarker distances fails to boost hyperparameter tuning","feed_subtitle":"Two new encoders align dataset distances with landmarkers, yet warm-started Bayesian hyperparameter optimization shows no significant gain.","key_machinery":"The load-bearing mechanism is the Requirement formulated in the introduction: distances between representations of datasets should align with distances between the corresponding landmarkers, where a landmarker vector records the performances of a fixed portfolio of hyperparameter configurations on a dataset. The paper implements this requirement in two loss functions. The metric-learning loss (Eq. 1) minimizes the mean squared difference between the Euclidean distance $d(\\varphi(X^{(i,1)}, y^{(i,1)}), \\varphi(X^{(i,2)}, y^{(i,2)}))$ of two encoded datasets and the Euclidean distance $d(l^{(i,1)}, l^{(i,2)})$ of their landmarker vectors. The reconstruction loss (Eq. 2) minimizes the mean squared error between a predicted landmarker vector and the true one, using the predicted vector as the dataset representation. The evaluation metric (Eq. 3) measures the Spearman correlation between representation distances and landmarker distances, and it is this correlation that the paper uses to judge whether the Requirement is fulfilled.","core_discovery":"On the paper's own terms, the central discovery is a negative result with a positive component: the proposed encoders can satisfy the proposed Requirement, that Euclidean distances between dataset representations align with Euclidean distances between landmarker vectors, but this alignment does not translate into significant gains in warm-starting Bayesian HPO. Training Dataset2Vec with a metric-learning loss raises the Spearman correlation between representation distances and landmarker distances from 0.037 to 0.332, and the reconstruction approach reaches 0.470 when distances are measured to predicted landmarkers. Yet in the warm-start evaluation, after the full Bayesian HPO run, even the true-landmarker selection method is not statistically distinguishable from no warm-start or random portfolio selection, and the encoder-based methods are statistically indistinguishable from other portfolio-based baselines. The paper thus establishes that the link between representation-to-landmarker distance alignment and downstream HPO performance is weak.","pith_inferences":["A natural extension the authors leave implicit is to test the distance-alignment requirement on other meta-tasks, such as model selection or dataset recommendation, where neighbor quality may matter more than in a 20-iteration Bayesian HPO run.","The results suggest that portfolio construction itself, with 100 configurations chosen by tournament selection over clusters, may dominate warm-start quality; representation learning might show larger gains with a larger or more carefully stratified portfolio.","One could design a stricter test of the Requirement by training the encoder to preserve only the ordering of landmarker distances to the target dataset, which is the information the nearest-neighbor warm-start step actually uses."],"forward_implications":["Evaluating encoders by how well their distances correlate with landmarker distances can be misleading, because high correlation does not imply usefulness in warm-starting HPO.","Encoder-based warm-starting can partially substitute true landmarkers in the initial phase of Bayesian HPO, as the proposed methods are statistically indistinguishable from the landmarker upper bound just after warm-start.","After a full Bayesian HPO run, the choice of warm-start approach has no statistically significant effect, since even the true-landmarker method is indistinguishable from random portfolio selection.","General-purpose representations like plain Dataset2Vec carry almost no landmarker information, with a correlation of 0.037, so task-specific training is needed if landmarker information is the goal."],"supporting_citations":[{"why":"Supplies the Dataset2Vec encoder architecture and the original encoder-evaluation meta-task that all proposed methods build on.","marker":"[Jomaa et al., 2021]"},{"why":"Shows that general encoder representations are insufficient for warm-starting Bayesian HPO, motivating the task-specific requirement.","marker":"[Płudowski et al., 2024]"},{"why":"Provides the metric-learning-based warm-start approach that inspires the deep metric learning loss.","marker":"[Kim et al., 2018]"},{"why":"Defines the two-stage transfer surrogate model and the ADTM evaluation metric used to measure warm-start gains.","marker":"[Wistuba et al., 2016]"},{"why":"Introduces the idea of initializing Bayesian hyperparameter optimization via meta-learning, which the warm-start protocol tests.","marker":"[Feurer et al., 2015]"},{"why":"Frames the cold-start problem in Bayesian optimization that warm-start points are intended to address.","marker":"[Bai et al., 2023]"}],"fun_headline_variants":["Landmarker-aligned encoders fail to boost HPO warm-starting","Distance alignment with landmarkers yields no HPO gain","Encoders match landmarkers, but HPO warm-start stays flat","Learning landmarker distances doesn't speed up tuning","Warm-starting HPO: representation alignment insufficient"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that making dataset-representation distances match landmarker distances is enough signal to produce useful warm-start configurations for Bayesian HPO; if that assumption fails, the whole encoder-design strategy loses its justification.","fun_headline_variants_meta":{"raw":{"variants":["Landmarker-aligned encoders fail to boost HPO warm-starting","Distance alignment with landmarkers yields no HPO gain","Encoders match landmarkers, but HPO warm-start stays flat","Learning landmarker distances doesn't speed up tuning","Warm-starting HPO: representation alignment insufficient"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000544,"raw_usage":{"total_tokens":2566,"prompt_tokens":867,"completion_tokens":1699,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":1616}},"tokens_in":483,"tokens_out":1699,"duration_ms":12802,"temperature":1.0,"reasoning_tokens":1616,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:43:29.262933+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same warm-start protocol but replace encoder representations with the exact landmarker vectors in the nearest-neighbor search; if this oracle representation fails to improve over random portfolio selection at the end of optimization, the paper's claim that HPO is largely robust to warm-start selection is confirmed, while if it succeeds, the encoders' weak alignment is the bottleneck.","supporting_citations":[],"review_version":1}