{"id":"6983313c-da3c-43d5-a2ec-2653ec4205de","arxiv_id":"2506.07109","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A string-based, metadata-guided language model framework for universal offline black-box optimization, with two variants and two embedding regularizations.","lead":"This paper proposes UniSO, a framework that turns heterogeneous black-box optimization problems into strings and trains a language model to predict or regress scores across tasks. It adds metadata alignment and embedding smoothness regularization, and shows zero-shot and few-shot gains on unseen offline optimization tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Zero-shot universality may be carried by hand-crafted semantic metadata rather than by the learned string embedding space; the unseen-task experiments never remove or corrupt this metadata.","rationale":"The reader's weakest_assumption correctly identifies metadata as the most fragile part of the evidence for universality. I partially agree: the concern is real, but Table 14 shows that removing metadata does not collapse all zero-shot results, so the effect is not a uniform leakage artifact. Rather, the paper's own formulation makes metadata a legitimate input, yet the zero-shot evaluation does not separate 'learning a general string embedding representation' from 'exploiting pretrained semantic similarity between hand-crafted text descriptions and training tasks'. The concrete test proposed above would settle this: if generic or misleading metadata destroys the zero-shot advantage, then the central claim should be weakened to 'universal BBO conditioned on semantically informative task descriptions', not universal BBO from string embeddings alone. I found no internal inconsistency in the method description, and the code release is a real asset for running the test. Because the reader already issued a CONDITIONAL verdict and the concern refines rather than overturns that verdict, I leave the verdict unchanged.","tokens_in":50035,"tokens_out":6844,"duration_ms":82519,"concrete_test":"Using the released code, retrain improved UniSO-T on the same nine tasks, then evaluate zero-shot on RobotPush, Rover, and LunarLander under four metadata conditions: (1) the original metadata; (2) generic placeholders such as 'task A', 'task B', 'task C' with no semantic content; (3) misleading objective text, e.g., swapping 'maximize' for 'minimize'; and (4) no metadata (already reported in Table 14). If zero-shot performance on any task drops to or below D(best) under conditions (2) or (3), the generalization is metadata-conditioned rather than representation-based. Additionally, add one held-out task from a disjoint domain, such as a synthetic analytic function with a randomized task name, so that its metadata has no semantic overlap with training metadata; the central claim requires zero-shot performance above D(best) on that task as well.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The universal BBO claim rests most heavily on RQ3: a model trained on nine Design-Bench/SOO-Bench tasks is evaluated zero-shot on RobotPush, Rover, and LunarLander. At inference, the model receives metadata (Appendix D, Table 7) that gives each unseen task's name, domain, and explicit objective, e.g., 'maximize the mean terminal reward' or 'minimize the distance between a predefined target location and two objects'. This text is embedded by a pretrained T5-Small embedder and used to train the contrastive alignment in Sec. 3.3, so the model can implement text-conditioned regression: the metadata embedding retrieves the most similar training task and reuses its score model, without ever learning a general string-embedding representation for genuinely new tasks. The paper's own ablation (Table 14) supports this reading: removing all metadata drops zero-shot LunarLander to 6.251 ± 53.042, below D(best) = 7.038, while full metadata gives 31.186 ± 27.971; on RobotPush and Rover the no-metadata model remains above D(best), so the effect is not uniform, but the headline 'unseen-task generalization' is at least partly an artifact of semantic prior access. The authors explicitly define metadata as something that can 'hint the information of the unknown objective f' (Sec. 2.1), so this is not a hidden bug; it is an unresolved confound for the universality claim. The claim that unifying LM priors and learning string embeddings can 'overcome traditional barriers in universal BBO' is not tested under the condition that matters: a held-out task with no semantically informative metadata or with a genuinely novel domain.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes UniSO, a universal offline black-box optimization framework that represents designs as strings, conditions on hand-written task metadata (name, description, objective), and learns a regressor whose latent space is regularized by a contrastive metadata-alignment loss and a Lipschitz smoothness loss. Two instantiations are studied: UniSO-T, a token-targeted T5-style sequence-to-sequence regressor, and UniSO-N, a numeric-targeted regressor with an MLP head on top of an embedding model. Experiments cover nine Design-Bench and SOO-Bench tasks for multi-task training, plus zero-shot and few-shot evaluation on RobotPush, Rover, and LunarLander. The paper reports that improved UniSO-T outperforms the authors' batch-normalized single-task MLP experts in several tasks, and that zero-/few-shot results on unseen tasks exceed the best scores in the offline datasets. The central claim is that unifying language-model priors with a learned string-embedding space can overcome traditional barriers in universal offline BBO.","tokens_in":50402,"tokens_out":4619,"duration_ms":46808,"significance":"If the universality claim were fully established, the work would be a meaningful step toward a single model that solves heterogeneous offline BBO tasks and transfers to new problems. The paper deserves credit for releasing code, providing ablation studies for each proposed component, and honestly reporting a lower rank (9.8/22) against published single-task offline BBO methods in Appendix E.1. The attention-visualization analysis of pre-trained versus from-scratch embedders is also a useful empirical observation. However, the evidence for the headline claim is thin: only nine training tasks, only three held-out tasks from the same control/simulation family, and no significance testing in a regime where standard deviations are very large. The strongest result about 'unseen-task generalization' is confounded by the fact that the model receives each unseen task's explicit optimization objective as metadata, as shown by the paper's own ablation in Table 14. The contribution is promising but the claims currently outrun the evidence.","major_comments":[{"comment":"The zero-shot generalization result is confounded by the metadata. For each unseen task, the metadata explicitly states the optimization objective (e.g., 'maximize the mean terminal reward' for LunarLander and 'minimize the total velocity change' for GTOPX in Table 7). Because the contrastive loss in §3.3 aligns the input-embedding similarity to the metadata-embedding similarity, the model can implement text-conditioned retrieval or conditioning on the stated objective rather than learning a genuinely transferable string-embedding representation for new tasks. The paper's own ablation in Table 14 shows that removing all metadata drops zero-shot LunarLander from 31.186 ± 27.971 to 6.251 ± 53.042, below D(best) = 7.038, while full metadata gives the inflated value. This is a load-bearing issue for the universality claim: the authors should present zero-shot results with generic or ablated metadata and explicitly separate 'knowing the task objective from text' from 'generalizing to a new task.'","section":"§3.3, §4.2 (RQ3), Appendix D Table 7, Appendix E.5 Table 14"},{"comment":"The paper's central claim that UniSO 'achieves better results than single-task numeric-input experts' is based on the authors' own batch-normalized MLP baselines in Tables 1 and 2. Against 21 published single-task offline BBO methods using the field-standard z-score normalization protocol (Table 8), improved UniSO-T ranks 9.8 on average and does not outperform state-of-the-art methods. The abstract and conclusion should qualify the comparative claim to the specific weak baseline, and the inconsistency between the BN-based comparison (Tables 1 and 2) and the z-score-based comparison (Table 8) should be explicitly reconciled.","section":"§4.2 (RQ7), Appendix E.1 Table 8"},{"comment":"The performance comparisons lack statistical support. Many reported standard deviations are extremely large (e.g., Ant 241.350 ± 288.922 in Table 1; TF Bind 10 0.929 ± 0.802 in Table 2), and the overlapping confidence intervals across methods make it impossible to conclude that one method is better than another. The authors should report the number of seeds, paired differences, and significance tests (e.g., Wilcoxon signed-rank) for the claimed improvements, especially for the UniSO-T versus expert comparisons that are central to RQ1 and RQ2.","section":"§4.2 (RQ1, RQ2), Tables 1 and 2"},{"comment":"The evidence for universality is thin. Zero-shot generalization is tested on only three tasks (RobotPush, Rover, LunarLander), all drawn from the same simulation/control suite of Wang et al. (2018; 2024a), and the training tasks are only nine Design-Bench/SOO-Bench problems. This does not support the broad claim of 'overcoming traditional barriers in universal BBO' across heterogeneous design spaces and task types. Additional held-out tasks from different families (e.g., molecule, protein, or mixed categorical-continuous problems) and an analysis of what transfers (embedding geometry vs. per-task retrieval) are needed before the universality claim can be accepted.","section":"§4.2 (RQ3), Figure 4, Appendix C.3"}],"minor_comments":[{"comment":"The caption for Figure 15 says the plots are on the GTOPX 6 task, but the subfigures are labeled TF Bind 10; the caption should be corrected.","section":"Appendix F, Figure 15 caption"},{"comment":"The sentence 'where X could be be CONTINUOUS...' has a duplicated 'be' and is grammatically incomplete; it should read 'where X could be CONTINUOUS...'.","section":"§2.1"},{"comment":"There is a typo: 'BO is is a widely used sample-efficient method' should be 'BO is a widely used sample-efficient method'.","section":"Appendix A.1"},{"comment":"The word 'tabluar' should be 'tabular' in the sentence about Transformers performing well on tabular data.","section":"Appendix A.3"},{"comment":"The text contains 'the rank correlation between between predicted and ground-truth objective scores'; the duplicated 'between' should be removed.","section":"§4.2 (RQ4)"},{"comment":"The Journal of Machine Learning Research reference for Raffel et al. is spelled 'Maching' instead of 'Machine'.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper presents an interesting framework and a reproducible codebase, but the load-bearing claim of zero-shot universality is undermined by the metadata confound identified in Table 14. The authors' own Table 8 also shows a much more modest result against standard offline BBO baselines. I recommend major revision with a request for metadata-ablated zero-shot experiments, statistical significance testing, and a more balanced framing of the claims relative to the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper makes a serious attempt at universal offline BBO with string embeddings, and the specific combination of metadata-guided contrastive alignment plus Lipschitz smoothness is new. Both ingredients come from earlier work, but the adaptation to heterogeneous offline BBO is a genuine contribution. The paper ships code, detailed hyperparameters, and full ablations, which is more than most papers of this type. The RQ7 comparison is honest: the best variant lands at rank 9.8/22 against single-task experts, and the paper says there is room for improvement. The attention analysis on T5 versus Qwen versus DeepSeek is a nice exploratory piece suggesting LM priors can hurt numeric regression.\n\nThe biggest soft spot is the metadata confound. In the zero-shot evaluation, the model receives task name, description, and explicit objective text such as \"minimize the total velocity change.\" That text lets the model retrieve a similar training task and reuse its score model, which undermines the claim of learning a general string-embedding space for genuinely new tasks. The paper's own ablation (Table 14) shows removing all metadata drops zero-shot LunarLander from 31.19 to 6.25, below D(best). The stress-test note is right: the paper tests \"unseen tasks\" under a condition where the objective is effectively disclosed. A held-out task with generic or no metadata would be the honest test.\n\nA second concern is that the few-shot results look too good to be true. LunarLander jumps from D(best) 7.0 to 287 after fine-tuning on 100 low-scoring pairs. That strongly suggests the surrogate is being exploited outside the training distribution. The paper reports OOD rank correlation only for the UniSO-N pretraining comparison, not for the main few-shot results, so we cannot tell how reliable the surrogates are in the regions BO finds.\n\nMinor issues: only nine training tasks, no significance tests, and high variance (Ant std 289 in Table 1). These cap the strength of the universality claim but do not invalidate the proof of concept.\n\nWho gets value: anyone working on LM-based BBO or universal offline optimization. The paper is a solid, clearly written baseline for future work, but it does not yet deliver a practical universal optimizer. The core idea is worth building on, and the confound is fixable in revision. I would send it to peer review: the question is timely, the implementation is reproducible, and the flaws are identifiable rather than hidden. I would push the authors to (a) add a no-metadata or generic-metadata zero-shot condition, (b) report significance tests or more seeds on the main comparisons, and (c) report surrogate OOD error for the few-shot results.","headline":"A clear, reproducible proof of concept for LM-based universal offline BBO, but the zero-shot claim leans on task-revealing metadata and the few-shot numbers smell like surrogate overfitting.","tokens_in":50911,"tokens_out":3582,"would_cite":true,"duration_ms":38057,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that a single string-based model, UniSO, can solve offline black-box optimization across heterogeneous design spaces and generalize to unseen tasks once its embedding space is shaped by metadata-guided contrastive…","keywords":["offline black-box optimization","universal optimization","language model embeddings","string-based representation","multi-task regression","metadata conditioning","contrastive learning","zero-shot generalization"],"falsifier":"Train the improved UniSO-T on the same nine Design-Bench and SOO-Bench tasks, but replace every task's metadata with one fixed generic sentence such as \"optimize the given design\" while keeping all other hyperparameters and search settings identical. If the model then loses its edge over single-task numeric-input experts on the in-distribution tasks and its zero-shot advantage on RobotPush, Rover, and LunarLander, the claimed universality is attributable to metadata conditioning rather than to learning a general string-embedding space.","tokens_in":49896,"feed_emoji":"🎯","tokens_out":8957,"duration_ms":79720,"temperature":0.7,"pith_summary":"The paper sets out to establish that offline black-box optimization does not have to be rebuilt from scratch for every new problem. It proposes UniSO, a framework that writes each candidate design as a text string, prepends a short metadata note saying what task it is and what objective is being optimized, and trains one language-model-based regressor on all tasks together. The payoff would be a single optimizer that handles continuous, categorical, and high-dimensional spaces of different sizes, transfers knowledge between related tasks, and needs only a handful of examples to adapt to a task it has never seen. The paper reports that its improved token-targeted variant, UniSO-T, beats single-task numeric-input experts on several Design-Bench and SOO-Bench tasks, and that both variants generalize zero-shot and few-shot to unseen tasks. If the claim holds, it breaks the traditional fixed-dimensional, single-task barrier of offline BBO.","feed_headline":"String embeddings make black-box optimization universal across tasks","feed_subtitle":"A single model trained on stringified designs beats per-task experts and adapts to unseen tasks from a few examples.","key_machinery":"The machinery that carries the argument is a task-shared string embedding space plus two geometric regularizers. Designs are tokenized as JSON-like strings, prefixed with metadata that names the task, describes it, and states the objective; the same SentencePiece tokenizer maps these strings to inputs for a T5-style encoder. UniSO-T uses an encoder-decoder that autoregressively predicts the P10-encoded score tokens, while UniSO-N fits an MLP regressor on mean-pooled LM embeddings. The decisive components are the two losses added to the main regression objective: a contrastive loss that encourages agreement between the input-embedding similarity matrix and the metadata-embedding similarity matrix, which separates dissimilar tasks while keeping related tasks close, and a per-task Lipschitz loss that penalizes pairs whose embedding distance is small while their score difference is large, which enforces local smoothness. A loss-balancing rule scales the auxiliary gradients by the ratio of the main loss to each auxiliary loss, so the regularizers steer rather than dominate training.","core_discovery":"The central claim is that a unified string-based representation, combined with a well-shaped learned embedding space, is sufficient for universal offline BBO. Concretely, the paper shows that a single multi-task regressor can score designs from heterogeneous search spaces when every design is serialized as a JSON-like dictionary and prefixed by hand-written metadata consisting of task name, description, and objective. It instantiates this as two variants: UniSO-T predicts the numerical score token-by-token using the P10 digit encoding inside a T5-based encoder-decoder, while UniSO-N embeds the input strings with an LM encoder and regresses the score with an MLP. The paper's improvements are two losses applied to the embedding space: a contrastive loss that aligns input embeddings with metadata embeddings so that similar tasks cluster together, and a Lipschitz loss that enforces local smoothness so that nearby embeddings correspond to similar scores. In its main experiments, improved UniSO-T outperforms single-task numeric-input experts on several tasks, and both variants exceed the best scores in the offline datasets and beat a z-score-normalized expert after few-shot fine-tuning on unseen tasks. The paper also argues, from attention visualizations and from training-from-scratch comparisons, that pre-trained language-model priors can be harmful for numeric regression because they concentrate attention on structural tokens such as EOS rather than on the numeric tokens that matter for optimization.","pith_inferences":["Editorial inference: the strongest untested premise is whether the hand-written metadata is a legitimate task descriptor or a leakage channel; a fair stress test would give generic metadata to every task and check whether the cross-task advantage survives.","Editorial inference: the attention analysis suggests that future universal optimizers could improve by adapting language models to arithmetic-heavy and code-heavy text, but the paper stops short of proving that such adaptation transfers to downstream optimization performance.","Editorial inference: the framework is demonstrated on single-objective unconstrained tasks, leaving open whether the same string-embedding recipe supports constrained, multi-objective, or mixed-integer search spaces common in real engineering and scientific design."],"forward_implications":["If one universal regressor can replace per-task surrogates, practitioners no longer need to collect a large dataset for every new design problem; cross-task data can be pooled.","The zero-shot and few-shot results on RobotPush, Rover, and LunarLander suggest that a universal model can bootstrap a new task from only its worst 100 offline examples and already outperform a single-task expert trained on the same data.","Because the representation is string-based, the same model can in principle cover continuous, categorical, integer, and permutation variables without changing architecture.","The finding that pre-trained LM embeddings can hurt numeric regression implies that string-based BBO models should be trained from scratch or from checkpoints with strong mathematical content rather than defaulting to general-purpose LMs.","The model-inner search can use any off-the-shelf black-box optimizer over strings, including Bayesian optimization, evolutionary algorithms, and CMA-ES, so the universal surrogate plugs into existing search software."],"supporting_citations":[{"why":"It supplies the Design-Bench tasks and datasets and defines the offline-BBO setting the paper extends.","marker":"Trabucco et al., 2022"},{"why":"It contributes the SOO-Bench tasks and the dataset-generation procedure used for training and evaluation.","marker":"Qian et al., 2025"},{"why":"It introduces the string-based token-targeted universal regression formulation, OmniPred, that UniSO-T builds on.","marker":"Song et al., 2024a"},{"why":"It contributes the LM-embedding regressor paradigm and the score-normalization strategy used by UniSO-N.","marker":"Nguyen et al., 2024"},{"why":"It provides the T5 architecture and pre-trained checkpoints used as backbones for both UniSO variants.","marker":"Raffel et al., 2020"},{"why":"It supplies the Lipschitz loss formulation used to enforce local embedding smoothness.","marker":"Lee et al., 2023"},{"why":"It supplies the contrastive-learning objective that the metadata-guided alignment loss is based on.","marker":"Chen et al., 2020"},{"why":"It provides the P10 digit encoding used to tokenize objective scores in UniSO-T.","marker":"Charton, 2022"},{"why":"It motivates the need for task metadata through the no-free-lunch theorem, which the paper cites as the reason data alone cannot distinguish tasks.","marker":"Wolpert & Macready, 1997"}],"fun_headline_variants":["Universal BBO via learned string embeddings","String embeddings enable cross-task black-box optimization","One model to optimize all black-box tasks offline","Embedding spaces make black-box optimization universal","Stringify and embed: universal offline BBO"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the hand-written metadata supplied to the model, namely the task name, description, and objective, is a fair task description rather than a leak of the answer. The paper's own ablation shows that removing pieces of metadata degrades performance, so if generic metadata fails to preserve the gains, the universality claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["Universal BBO via learned string embeddings","String embeddings enable cross-task black-box optimization","One model to optimize all black-box tasks offline","Embedding spaces make black-box optimization universal","Stringify and embed: universal offline BBO"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000878,"raw_usage":{"total_tokens":3844,"prompt_tokens":1042,"completion_tokens":2802,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":658,"completion_tokens_details":{"reasoning_tokens":2735}},"tokens_in":658,"tokens_out":2802,"duration_ms":24941,"temperature":1.0,"reasoning_tokens":2735,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:41:37.824743+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the improved UniSO-T on the same nine Design-Bench and SOO-Bench tasks, but replace every task's metadata with one fixed generic sentence such as \"optimize the given design\" while keeping all other hyperparameters and search settings identical. If the model then loses its edge over single-task numeric-input experts on the in-distribution tasks and its zero-shot advantage on RobotPush, Rover, and LunarLander, the claimed universality is attributable to metadata conditioning rather than to learning a general string-embedding space.","supporting_citations":[{"cited_title":"D esign- B ench: B enchmarks for data-driven offline model-based optimization","cited_arxiv_id":null,"evidence_quote":"It supplies the Design-Bench tasks and datasets and defines the offline-BBO setting the paper extends."},{"cited_title":"SOO - B ench: B enchmarks for evaluating the stability of offline black-box optimization","cited_arxiv_id":null,"evidence_quote":"It contributes the SOO-Bench tasks and the dataset-generation procedure used for training and evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the T5 architecture and pre-trained checkpoints used as backbones for both UniSO variants."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the Lipschitz loss formulation used to enforce local embedding smoothness."},{"cited_title":"and Macready, W","cited_arxiv_id":null,"evidence_quote":"It motivates the need for task metadata through the no-free-lunch theorem, which the paper cites as the reason data alone cannot distinguish tasks."}],"review_version":1}