{"id":"42c850ee-aebf-4b4b-a16d-4bb4c986be5e","arxiv_id":"2504.21018","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A hypernetwork that maps external multilingual word vectors to a PLM's embedding space can initialize new target-language token embeddings, matching or slightly exceeding the OFA heuristic while beating random initialization.","lead":"This paper presents HYPEROFA, a method that trains a small neural network to predict embeddings for new language tokens from an external multilingual word-vector space, then uses those predicted embeddings to initialize a language model before continued training. It is tested against OFA and random initialization on RoBERTa and XLM-R across sentence retrieval, NER, and POS tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"BiLSTM hypernetwork is not permutation-invariant; without a canonical inference-time order, generated target embeddings and hence the headline comparisons may depend on input order.","rationale":"The paper's central claim is empirical, so I weighed whether the lack of error bars or the borrowed OFA numbers in Table 3 were more damaging. Those are real weaknesses, but they are standard and can be repaired without changing the method. The permutation-invariance issue is different: it strikes at the definition of the method. In §3.4 the authors themselves list permutation invariance as a design requirement, then adopt BiLSTM because it performed better than Setformer in a single comparison. Random shuffling is a training-time heuristic; it does not make the function invariant, and no canonical test-time order is defined for the matched-word set. Since the hypernetwork generates up to 179K target embeddings, any order-dependent variation propagates into every evaluation. The reader's weakest assumption captured exactly this point, and my proposed inference-time shuffling test would settle it. I am not moving the verdict because the concern is for conditional acceptance: the paper is publishable if the test passes or if the method is revised to an explicitly set-invariant aggregator.","tokens_in":24776,"tokens_out":8204,"duration_ms":81258,"concrete_test":"Using the released code, fix the trained hypernetwork for one model (e.g., HYPEROFA-multi-400) and a sample of wisely initialized target tokens, including tokens with more than 256 matched words where truncation interacts with order. Regenerate their coordinate embeddings for 10–20 random permutations of W{tj} at inference. Compute pairwise cosine similarity and max L1 distance across permutations for each token, then, if the embeddings differ, initialize two copies of the model with two different permuted predictions and measure SR-T before continual pretraining. If pairwise cosine similarity is above 0.99 and downstream metrics match within one point, the concern is resolved; if embeddings or SR-T vary materially with order, the method is not well-defined as stated and needs an invariant aggregation or a canonical ordering before the headline comparison can be trusted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4 states two requirements for HNθ: variable-length inputs and permutation invariance, then selects a BiLSTM 'despite it not inherently satisfying the permutation-invariance requirement,' relying on random shuffling during training. This choice is load-bearing because every non-copied, non-random target embedding produced in §3.5 comes from this network. If the network is not invariant, the generated embeddings—and therefore all before/after continual-pretraining numbers in Tables 3–4—are functions of whatever order the matched word vectors happen to have in the data pipeline. Random shuffling during training encourages invariance but does not guarantee it at inference, and the paper specifies no canonical ordering for W{tj}. The appendices report only a single SR-T comparison of BiLSTM versus Setformer (A.1), not a test of prediction stability under input permutations. A related secondary gap is that generalization from the 22K/103K source-language training pairs to target tokens in unseen languages is supported only by aggregated downstream scores, not by per-script or per-token diagnostics; if permutation sensitivity is material, those aggregates cannot be interpreted. The central claim that HYPEROFA is a viable alternative to OFA therefore rests on an architectural property the paper explicitly acknowledges it does not have.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes HYPEROFA, a hypernetwork-based method for initializing the embeddings of new target-language tokens when extending the vocabulary of a pretrained language model. The hypernetwork is a BiLSTM trained to map sets of external multilingual word vectors (from ColexNet+) to the factorized embedding space of a PLM (RoBERTa or XLM-R); at inference, it generates embeddings for new target tokens, which are then used as initialization before continual pre-training. The authors evaluate six configurations (three factorization dimensions for mono and multi setups) on sentence retrieval (SR-T, SR-B), NER, and POS before and after 4,000 steps of continual pre-training on 22 languages, comparing against random initialization and OFA. They report that HYPEROFA outperforms random initialization and performs comparably to OFA, with occasional gains.","tokens_in":24972,"tokens_out":8363,"duration_ms":67806,"significance":"If substantiated, HYPEROFA would provide a useful alternative to similarity-based convex combination for new token embedding initialization, with potentially higher expressiveness, and would be relevant to the growing area of vocabulary expansion for multilingual PLMs. The paper's strengths include making code public, following the established OFA evaluation protocol, covering many languages and tasks, and providing per-language breakdowns in the appendix. However, the current evidence is insufficient to support the stated claims: the comparison in Table 3 uses published OFA numbers rather than matched runs, no variance is reported, and the mono-100 results after continual pre-training contradict the 'matches or exceeds' claim. With additional experiments and a revised framing, the contribution could become a solid empirical study.","major_comments":[{"comment":"The claim that HYPEROFA 'matches or exceeds the performance of OFA' is contradicted by the paper's own results. In Table 4, after continual pre-training, HYPEROFA-mono-100 scores 11.3 on SR-T versus 15.2 for OFA, and 43.4 vs 45.7 on NER; Table 3 also shows HYPEROFA-mono-400 below OFA on all four tasks (e.g., 6.3 vs 7.2 on SR-T). The abstract and contribution bullet should be revised to a more accurate 'competitive' formulation, and the cases where HYPEROFA underperforms should be explained or at least explicitly acknowledged.","section":"Abstract and §5.2, Table 4"},{"comment":"The hypernetwork is a BiLSTM that does not satisfy the permutation-invariance requirement stated at the beginning of §3.4. The only mitigation is random shuffling during training, which does not guarantee that generated embeddings are independent of input order at inference. Because the paper specifies no canonical ordering of the set W{tj}, every embedding generated by the hypernetwork in §3.5, and hence every result in Tables 3 and 4 that depends on those embeddings, is potentially a function of an arbitrary implementation detail. Please add a permutation-sensitivity test (e.g., variance of generated embeddings across multiple random orderings) and either fix a canonical ordering or switch to a permutation-invariant architecture.","section":"§3.4, Appendix A.3"},{"comment":"The OFA baseline scores in Table 3 are taken from Liu et al. (2024a) rather than reproduced in the same pipeline. This makes the headline before-continual-pretraining comparison non-controlled: HYPEROFA and Random are run under the authors' implementation while OFA numbers come from the original paper. Differences as small as 0.1–0.5 points cannot be attributed to the initialization method with confidence. The authors should reproduce OFA under identical conditions, even for a subset of configurations, or restrict the 'matches or exceeds' claim to the fully controlled Table 4 setting.","section":"§5.1, Table 3"},{"comment":"The hypernetwork is trained only on source-vocabulary token pairs (22K for RoBERTa, 103K for XLM-R) and then applied to target tokens from the Glot500 tokenizer covering many languages and scripts. The paper does not provide any per-script or per-language diagnostic of prediction quality; aggregated downstream averages (Tables 3 and 4) can hide systematic failures for particular scripts or languages not seen in hypernetwork training. Please report prediction accuracy or cosine similarity for target tokens broken down by script or by whether the target language was present in the hypernetwork training set.","section":"§4.2, §3.3"},{"comment":"No error bars, confidence intervals, or multiple seeds are reported anywhere in the empirical evaluation. Given that many HYPEROFA-vs-OFA differences in Table 3 are below one point (e.g., NER mono-100: 24.9 vs 25.0; SR-T mono-100: 6.4 vs 6.2), single-run results cannot support the claim that HYPEROFA 'consistently outperforms' the baselines. At minimum, the six continually pre-trained models in Table 4 should be run with at least three seeds and the mean and standard deviation reported; ideally the Table 3 comparisons should be re-run with matched OFA baselines.","section":"Tables 3 and 4"}],"minor_comments":[{"comment":"The second occurrence of 'HYPEROFA-multi-100' should be 'HYPEROFA-multi-200'; the corresponding hypernetwork HN-X-200 is listed in Appendix Table 6.","section":"Table 1"},{"comment":"The definition of NEG in the contrastive loss is not typeset cleanly; the summation over k should be written explicitly with the correct argument to avoid ambiguity.","section":"§3.4"},{"comment":"The abstract uses 'matches or exceeds the performance of OFA' while the conclusion says 'performs competitively with OFA'; align the wording to reflect the actual results.","section":"Abstract and Conclusion"},{"comment":"In the text, 'HYPEROFA-mutli-400' is a typo for 'HYPEROFA-multi-400'.","section":"§4.2"},{"comment":"Appendix A.2 reports λ=0.1 and τ=0.5/0.25 but contains no sensitivity analysis for these hyperparameters; since they are free parameters chosen on validation cosine similarity, a brief sensitivity study would strengthen the method.","section":"Appendix A.2"},{"comment":"Figure 2 shows a single validation curve for one configuration; the claim that 'larger hypernetworks perform better' would benefit from quantitative results at convergence rather than a single curve.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The authors are from the same group as the OFA paper, which may explain the direct reuse of published OFA numbers; regardless, the comparison is not fully matched and should be re-run or appropriately caveated. The core idea is not flawed, but the evidence as presented is not strong enough for acceptance; I recommend major revision and would be willing to re-review."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"HYPEROFA is worth a look if you work on vocabulary expansion for multilingual PLMs, but go in with calibrated expectations. The new bit is concrete: instead of OFA's similarity-based convex combination, they train a BiLSTM hypernetwork to map external word vectors into the PLM's factorized embedding space, then use it to generate target-token embeddings. That is a real, if incremental, contribution. The evaluation follows OFA's setup closely, covers before/after continual pretraining, and the code is public. The authors also state their limitations plainly (encoder-only, low embedding dimensions, small CPT dataset), which I appreciate.\n\nThe central claim—HYPEROFA beats random initialization and roughly matches OFA—is broadly supported, but the evidence is thinner than the abstract implies. Most differences are under a point, there are no error bars or multiple seeds, and on mono-100 after continual pretraining, HYPEROFA is clearly worse than OFA on sentence retrieval (11.3 vs 15.2) and somewhat worse on NER. Also, Table 3 borrows OFA numbers from the prior paper rather than running OFA in-house, which weakens the comparison. These are not fatal, but they should be fixed with proper significance testing and same-running baselines.\n\nThe stress-test concern about permutation invariance holds up. Section 3.4 explicitly says the architecture should be permutation-invariant, then chooses BiLSTM because it works better empirically, relying on random shuffling during training. At inference, no canonical ordering is specified for the input word-vector sets, so generated target embeddings are, in principle, a function of input order. I don't think this invalidates the paper, but it is a load-bearing assumption about generalization to unseen target tokens, and the appendix only reports a single SR-T comparison of BiLSTM vs Setformer, not a stability test under input permutations. They should add a small experiment permuting the same target tokens and measuring embedding variance.\n\nOverall, this is a serious, honest paper with a useful negative result embedded in it: the learned nonlinear mapping does not decisively beat OFA's simple convex combination. That is a credible finding. The paper deserves a proper peer-review round, mainly to push the authors on variance, baseline provenance, and the permutation issue. I would not cite it as a breakthrough, but it is a legitimate alternative method in a space where people are still looking for better initialization strategies. Worth a reading-group slot if your group cares about multilingual adaptation.","headline":"HYPEROFA is a modest, honest incremental method that roughly matches OFA rather than clearly beating it; the evidence is thin in places, and the BiLSTM's lack of permutation invariance is a real but addressable gap.","tokens_in":25540,"tokens_out":1896,"would_cite":true,"duration_ms":18889,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HYPEROFA claims that a hypernetwork trained on source-language tokens can generate target-language token embeddings that beat random initialization and match OFA's similarity-based initialization for expanding a pretrained model's…","keywords":["hypernetwork","embedding initialization","vocabulary expansion","cross-lingual transfer","continual pre-training","multilingual language models","factorized embeddings","low-resource languages"],"falsifier":"Take a held-out target language whose script has no overlap with the source vocabulary, generate its token embeddings with HYPEROFA, and continually pre-train the model from both the HYPEROFA initialization and a random initialization with identical data and compute; if random initialization converges to equal or better validation loss and downstream scores, the central claim fails. A cheaper check is to measure whether HYPEROFA's predicted embeddings for such unseen-script tokens have higher cosine similarity to the embeddings the model eventually learns than random initializations do.","tokens_in":24533,"feed_emoji":"🌐","tokens_out":5520,"duration_ms":47790,"temperature":0.7,"pith_summary":"The paper proposes a way to give pretrained language models embeddings for new tokens in languages they were not trained on: instead of designing similarity weights by hand, train a small hypernetwork to map external multilingual word vectors into the model's embedding space, then use it to generate the new embeddings. Trained only on source-language tokens (22K for RoBERTa, 103K for XLM-R), the hypernetwork is meant to transfer to unseen target-language tokens. The paper reports that this consistently beats random initialization and lands in the same range as OFA, the current similarity-based convex-combination method, on sentence retrieval, named-entity recognition, and part-of-speech tagging, both before and after continual pre-training. The practical stakes are that vocabulary expansion for mid- and low-resource languages could replace a similarity heuristic with a learned, more flexible mapping.","feed_headline":"Hypernetwork beats random init for adding new-language LLM vocab","feed_subtitle":"A learned mapping from word vectors to embedding space matches OFA and beats random initialization on low-resource language adaptation.","key_machinery":"The central object is the hypernetwork, a BiLSTM that takes a set of matched external word vectors $W_{\\{s_i\\}}$ for a source token $i$ and predicts that token's coordinate row $F_i^s$ in the SVD-factorized embedding matrix $E_s \\approx F_s P$. Because the input is a set of variable length with no natural order, the BiLSTM is trained with random shuffling of the input word vectors as a form of order augmentation; the loss is $\\lambda$ times a contrastive cosine term plus $(1-\\lambda)$ times a normalized L1 term. After training, the same hypernetwork predicts coordinate rows for target tokens, with overlapping token rows copied directly from $F_s$ and unmatched tokens initialized from a Gaussian fitted to the source embedding statistics.","core_discovery":"HYPEROFA argues that a BiLSTM hypernetwork can learn the nonlinear map from an external multilingual word-vector space to a pretrained language model's factorized embedding space, and that the embeddings it produces for target-language tokens are a better or equal starting point for continual pre-training than random initialization and than OFA's convex combinations of source embeddings. On the paper's 22-language continual pre-training runs, HYPEROFA matches or slightly exceeds OFA on multilingual XLM-R and on most monolingual RoBERTa metrics, with the clearest gains over random initialization appearing in the monolingual case. The differences between HYPEROFA and OFA are generally small, and the paper treats both as viable strategies while expecting HYPEROFA to improve more as hypernetwork and continual-pre-training data grow.","pith_inferences":["A natural stress test the paper does not run is to train on source tokens in one script family and evaluate the hypernetwork on completely unseen scripts, isolating whether the transfer is genuinely cross-lexical or just language-similarity based.","Because the BiLSTM is not permutation-invariant, replacing it with a set encoder such as a deep set or set transformer could improve robustness; the paper's own appendix reports that a transformer encoder without positional encodings underperformed on the limited data, so this is a data-scarcity hypothesis worth testing.","The method's cost scales steeply with embedding dimension (210M hypernetwork parameters for a factorized dimension of 400), so applying it to 4096-dimensional LLM embeddings would likely require shrinking the coordinate dimension or changing the hypernetwork architecture.","The paper only tests encoder-only models; the stated open direction is decoder-only and encoder-decoder LLMs, where vocabulary expansion for new languages is equally pressing and where the same initialization comparison could be run."],"forward_implications":["For monolingual models such as RoBERTa, wise initialization matters: both HYPEROFA and OFA converge faster and score higher after continual pre-training than random initialization, so vocabulary expansion without such initialization leaves clear performance on the table.","For a multilingual model such as XLM-R, the choice of initialization has little impact after continual pre-training; the model's multilingual pretraining largely absorbs randomly initialized new embeddings.","HYPEROFA's nonlinear mapping reaches OFA-level performance before continual pre-training, suggesting that the convex-combination constraint is not the main bottleneck limiting OFA in this experimental regime.","Because the hypernetwork is trained on only 22K to 103K source-language pairs, the paper expects its advantage over OFA to grow with more hypernetwork training data and longer continual pre-training, though that expectation is stated rather than demonstrated."],"supporting_citations":[{"why":"OFA: the baseline method and the source of the factorized parameterization, target tokenizer, matching procedure, and experimental protocol that HYPEROFA extends and compares against.","marker":"Liu et al., 2024a"},{"why":"ColexNet+: the external multilingual word-vector space whose matched word vectors form the hypernetwork's input.","marker":"Liu et al., 2023b"},{"why":"MIMICK: the earlier demonstration that a network can predict word embeddings from subword information, motivating the hypernetwork design.","marker":"Pinter et al., 2017"},{"why":"Zero-shot tokenizer transfer: the hypernetwork-based tokenizer-transfer work that this method builds on.","marker":"Minixhofer et al., 2024"},{"why":"Glot500: provides the target tokenizer and the continual pre-training corpus used in the experiments.","marker":"Imani et al., 2023"},{"why":"RoBERTa: the monolingual pretrained language model adapted in the mono experiments.","marker":"Liu et al., 2019"},{"why":"XLM-R: the multilingual pretrained language model adapted in the multi experiments.","marker":"Conneau et al., 2020"},{"why":"Tatoeba: supplies the sentence retrieval benchmark used to evaluate the initialized embeddings.","marker":"Artetxe and Schwenk, 2019"},{"why":"WikiANN: the named-entity recognition benchmark used for zero-shot cross-lingual evaluation.","marker":"Pan et al., 2017"},{"why":"Universal Dependencies: the part-of-speech tagging benchmark used for zero-shot cross-lingual evaluation.","marker":"de Marneffe et al., 2021"}],"fun_headline_variants":["Hypernetwork maps word vectors to add new languages to LLMs","Learn to initialize embeddings for low-resource languages via hypernetwork","HYPEROFA: Better token embeddings for new languages via learned mapping","Hypernetwork learns to embed new-language tokens, boosts continual pretraining","HYPEROFA matches OFA and beats random init for new-language vocab"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The hypernetwork trained on source-language tokens generalizes to target-language tokens, including languages and scripts it never saw, even though its BiLSTM encoder is not permutation-invariant and relies on random shuffling to dampen order sensitivity.","fun_headline_variants_meta":{"raw":{"variants":["Hypernetwork maps word vectors to add new languages to LLMs","Learn to initialize embeddings for low-resource languages via hypernetwork","HYPEROFA: Better token embeddings for new languages via learned mapping","Hypernetwork learns to embed new-language tokens, boosts continual pretraining","HYPEROFA matches OFA and beats random init for new-language vocab"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000608,"raw_usage":{"total_tokens":2821,"prompt_tokens":925,"completion_tokens":1896,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":1804}},"tokens_in":541,"tokens_out":1896,"duration_ms":10975,"temperature":1.0,"reasoning_tokens":1804,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:26:04.009151+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out target language whose script has no overlap with the source vocabulary, generate its token embeddings with HYPEROFA, and continually pre-train the model from both the HYPEROFA initialization and a random initialization with identical data and compute; if random initialization converges to equal or better validation loss and downstream scores, the central claim fails. A cheaper check is to measure whether HYPEROFA's predicted embeddings for such unseen-script tokens have higher cosine similarity to the embeddings the model eventually learns than random initializations do.","supporting_citations":[],"review_version":1}