{"id":"f32271e3-117d-42f8-ac2f-7d181aca0b42","arxiv_id":"2505.11427","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Mergenetic packages 19 evolutionary algorithms, six merging strategies, and lightweight fitness estimators into one library for evolutionary model merging.","lead":"Mergenetic is a new open-source library that combines evolutionary optimization with LLM merging, letting users search for good ways to blend fine-tuned models without training. It targets consumer GPUs by adding cheap fitness estimates and small evaluation subsets, and it includes two case studies on multilingual and cross-lingual math models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central empirical claim is unsupported because fitness is computed on 20–100 examples; evolutionary search can overfit the subsample, and no full-dataset or baseline comparison is provided.","rationale":"The reader's weakest_assumption correctly identifies the tiny fitness subsets as the load-bearing risk: the paper's quantitative claims depend on subsampled fitness remaining correlated with true performance, and no evidence is provided that this correlation holds for 20 or 100 examples. My stress-test reading confirms this is the weakest point in the argument. The library itself is a plausible software contribution, with real integration of PyMoo, MergeKit, and LM-Eval-Harness, and the paper is honest about hardware and model limitations. The repository link, though not commit-pinned, is an independent artifact that could be inspected. The central problem is empirical rather than architectural: the demonstrations do not establish that evolutionary search on such tiny samples yields merges that generalize to the actual benchmarks. A specific, cheap test—rerunning with increasing subset sizes and comparing against baselines—would settle whether the concern lands. Since the reader already assigned CONDITIONAL for essentially this reason, no verdict change is needed.","tokens_in":10541,"tokens_out":2841,"duration_ms":31589,"concrete_test":"Rerun both case studies with fitness subsets of increasing size (20, 50, 100, and the full evaluation set) and with at least 5 random seeds per configuration, then evaluate every best merge on the full held-out test set. Report mean and standard deviation of test accuracy for Mergenetic's best merge, for simple weight averaging, for each source model, and for random search. If the best merge from the 20-item fitness condition is not significantly better than the untrained baselines on the full test set, or if increasing fitness-subset size changes which merge is selected, the 'competitive results' claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract claims Mergenetic 'produces competitive results across tasks and languages using modest hardware,' but the only evidence is two case studies whose fitness functions are computed on extremely small subsamples. In the multilingual case (Appendix A.2), fitness per language uses 20 translated ARC examples and NSGA-II runs with population 25 for 7 generations; a single response changes accuracy by 5 points, and the approximate 95% CI for a 0.7 accuracy is roughly ±0.2. Selection pressure can therefore exploit noise, favoring merges that look good on the 20-item sample without generalizing to the real ARC test set. The cross-lingual case (Appendix A.1) uses 100 MGSM-JA examples for fitness instead of the 1,069 search points used by Akiba et al. (2025), yet no sensitivity analysis or comparison with full-dataset fitness is reported. No repeated seeds, error bars, or untuned baselines (e.g., plain weight averaging or random search) are given. The text in §5.2 also says '10-20% accuracy improvement' while Figure 5 is not clearly consistent with that arithmetic, which undermines confidence in the reported quantitative claims. None of this shows the library is broken; but it does mean the central claim that Mergenetic produces competitive results is not yet empirically established. The Limitations section candidly describes hardware constraints, but it does not address the statistical adequacy of the tiny fitness subsets, which is the load-bearing gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Mergenetic, an open-source Python library for evolutionary model merging built on MergeKit and PyMoo, with a Python API, CLI, and GUI. It supports six merging strategies, nineteen evolutionary algorithms, custom fitness functions via LM-Eval-Harness, and lightweight estimators including IRT-based ones. The empirical section presents two case studies: a multilingual ARC-Challenge merge across Italian, English, German, and Dutch using NSGA-II, and a cross-lingual math transfer to Japanese MGSM using a genetic algorithm. The abstract claims the library 'produces competitive results across tasks and languages using modest hardware.' The central claim is twofold: (i) the library architecture enables flexible, low-cost evolutionary merging, and (ii) the resulting merges are competitive with, and generally better than, their constituent models.","tokens_in":10801,"tokens_out":5854,"duration_ms":57873,"significance":"If the empirical claims were supported, Mergenetic would fill a real gap: no existing framework currently provides flexible, open-source support for composing merging strategies with evolutionary algorithms for LLMs on consumer GPUs. The modular design, permissive license, integration with established ecosystems (MergeKit, PyMoo, LM-Eval-Harness), and the inclusion of both CLI and GUI are concrete strengths, as is the fact that the demonstrations are reproducible from publicly available models and datasets. However, the evidence presented is not sufficient to establish the headline claim of 'competitive results': fitness is computed on very small subsamples, no baselines beyond constituent models are provided, no error bars or repeated seeds are reported, and the quantitative claims are not consistently supported by the figures. The paper would be stronger as a systems/software contribution with calibrated claims, or as an empirical paper with proper evaluation methodology.","major_comments":[{"comment":"The multilingual case study computes fitness on 20 translated ARC examples per language (Appendix A.2). With n=20, a single response changes accuracy by 5 percentage points, and the approximate 95% CI for an observed accuracy of 0.70 is about ±0.20, which is comparable to or larger than some reported gains (e.g., English: 0.79 vs 0.75). The NSGA-II run (population 25, 7 iterations) is not repeated, no full-ARC or held-out evaluation is reported, and no analysis shows that 20-example fitness ranks merges in the same order as full-dataset fitness. The claim in §5.1 that the merged model 'consistently outperforms' each constituent is therefore not established by the data as presented.","section":"§5.1, Appendix A.2"},{"comment":"The cross-lingual case study uses 100 MGSM-JA examples for fitness, in place of the 1,069 search points used by Akiba et al. (2025), but no sensitivity analysis or comparison with full-dataset fitness is provided. There are also no baselines such as equal-weight averaging, random search, or default TIES/DARE, so the claimed '10-20% accuracy improvement' in §5.2 cannot be attributed to evolutionary search rather than to the merging recipe itself. The numerical claim is also not clearly consistent with Figure 5: the visible deltas range from 0.18 (Abel, 0.24 to 0.42) to 0.41, which is neither 10 percentage points nor a uniform 10-20% relative improvement. The figure and text should be reconciled, and the claims re-stated with explicit error bars or repeated runs.","section":"§5.2, Appendix A.1, Figure 5"},{"comment":"Neither case study reports repeated seeds, error bars, or significance tests, even though the search procedures are stochastic (initialization, SBX crossover, polynomial mutation). The final results are point estimates from a single evolutionary run, and it is unclear whether the reported model was selected based on dev-subset fitness or on final test-set performance. Given the small fitness subsets, the absence of variance estimates is a load-bearing gap for any quantitative claim about accuracy gains.","section":"§5.1, §5.2, Appendix A.1–A.2"},{"comment":"The abstract highlights 'lightweight fitness estimators to reduce evaluation costs,' but neither case study actually uses the IRT-based estimators (P-IRT, GP-IRT, MP-IRT, GMP-IRT) described in Appendix A.4; only plain random subsampling is used. No experiment demonstrates the accuracy or computational savings of these estimators within Mergenetic, nor compares them to full-dataset fitness. To substantiate the abstract claim, at least one case study should use an estimator and report correlation with full-dataset performance, or the claim should be narrowed to random subsampling.","section":"Abstract, §4.2.3, Appendix A.4"}],"minor_comments":[{"comment":"The SMS-EMOA row lists 'CTAEA' as its algorithm class; this appears to be a typo and should read 'SMS-EMOA'.","section":"Appendix A.3, Table 3"},{"comment":"Several references are duplicated or incomplete: Deb et al. 2002a and 2002b are the same entry, Ilharco et al. appears as both 2022 and 2023, and Matena & Raffel and Stoica et al. lack publication years and venues.","section":"References"},{"comment":"The caption and bars should explicitly identify which bar is the merged model and which are the constituents; the label 'EvoLLM-JP-7B' is ambiguous because that name is also associated with the merged result.","section":"Figure 5"},{"comment":"The term 'ConfigPE' is introduced without definition or citation; please spell it out or explain what it is.","section":"§4.2.3"},{"comment":"The CLI example uses an en dash in '–eval-method' and '–merge-type'; these should be double hyphens ('--eval-method', '--merge-type').","section":"§4.1, CLI example"},{"comment":"The caption states that Mergenetic makes it easy to produce 'new state-of-the-art LLMs,' but the paper does not provide evidence for state-of-the-art performance; rephrase to avoid overclaiming.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about being built on the authors' own prior work (Mencattini et al. 2025) and on MergeKit, and the library itself appears functional. The main concern is that the empirical validation is currently too weak to support the abstract's 'competitive results' claim, and the fixes (baselines, repeated seeds, sensitivity analysis, or a scaled-back claim) require real experiments rather than copy-editing. I would also note that Figure 5 and the '10-20%' text need careful reconciliation before the paper can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the library is real and worth having; the 'competitive results' claim is not established by the two case studies, which is fixable by recalibrating the abstract or adding a couple of baselines.\n\nThe actual contribution is the integration: 19 PyMoo EAs, six merging strategies, subsampling and IRT estimators, and API/CLI/GUI wrapped around MergeKit and LM-Eval-Harness. That is new as a package, and it fills a real gap. The design is sensible, the code is open, and the three-interface approach lowers the entry barrier. The paper is honest about hardware constraints and licensing issues with MergeKit, which is more than most tool papers do.\n\nThe soft spots are in the empirical sections. The abstract claims competitive results across tasks and languages, but the demonstrations use 20 ARC items per language and 100 MGSM-JA items for fitness. No error bars, no repeated seeds, no untuned baselines, no comparison with full-dataset fitness. With samples that small, evolutionary search can exploit noise; a single response on 20 items moves accuracy by 5 points, and the CI on a 0.7 estimate is roughly ±0.2. The text also says 10-20% accuracy improvement in §5.2 while Figure 5 shows a jump from 0.01 to 0.24 or 0.42 depending on which bar is the merged model; the arithmetic is not clearly consistent. None of this suggests the library is broken, but it does mean the empirical claim is currently unsupported. The repository is linked but not pinned to a commit, so the experiments can't be rerun as-is. That is a minor issue for a library paper but worth fixing.\n\nWho this is for: practitioners and researchers who want to run evolutionary model merging without building their own harness. For that audience, the paper is a useful pointer and the library will likely work. It deserves a serious referee: a good editor should send it out, and a good reviewer should push for either a more modest abstract or a proper sensitivity analysis, plus pinned code.\n\nRecommendation: revise and resubmit, not reject. The artifact is the contribution; the case studies should be framed as smoke tests.","headline":"A genuinely useful library for evolutionary model merging, with case studies that overstate confidence; send it out but push for a sharper abstract and proper sensitivity analysis.","tokens_in":11412,"tokens_out":2373,"would_cite":false,"duration_ms":22936,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Mergenetic is an open-source library that makes evolutionary model merging cheap enough for a single consumer GPU and flexible enough to compare many optimization strategies.","keywords":["model merging","evolutionary algorithms","multi-objective optimization","large language models","fitness estimation","cross-lingual transfer","consumer GPU","open-source library"],"falsifier":"Rerun both evolutionary searches with fitness computed on the full benchmarks (full ARC-Challenge per language and the full Japanese math benchmark) and compare the winning merge coefficients with those found on the small subsets; if the winning recipes diverge substantially, or if the subset-chosen merges lose most of their reported edge on the full test sets, the small-sample fitness assumption is falsified.","tokens_in":10313,"feed_emoji":"🧬","tokens_out":10027,"duration_ms":86463,"temperature":0.7,"pith_summary":"Mergenetic is an open-source library that frames the weights of a model merge as variables in an evolutionary search: an optimizer proposes merge coefficients, the library builds the merged checkpoint, and a cheap fitness estimate scores it, repeating the loop over generations. The paper argues this is the first general-purpose library for evolutionary model merging in language models, letting users mix merging strategies and optimization algorithms freely instead of being locked into one routine. The central benefit is cost: fitness is computed on small subsamples or with statistical estimators rather than full benchmark runs, so meaningful searches fit on a single consumer GPU. Two demonstrations support the claim: merging four language-specific fine-tuned models into one multilingual model that beats every single-language endpoint on ARC-Challenge by up to 19% accuracy, and merging an English math model with a Japanese-tuned model so the result improves Japanese math word-problem accuracy by 10-20% over either parent. If true, the contribution is a much lower barrier to a technique that previously had no flexible, accessible implementation.","feed_headline":"One library makes evolutionary model merging run on a single GPU","feed_subtitle":"Pairs 19 evolutionary optimizers with 6 merging strategies and cheap fitness checks to fuse LLMs without retraining.","key_machinery":"The load-bearing component is the evolutionary search loop embodied in the searcher and problem classes: a genotype is a vector of merge coefficients, the merger layer converts that genotype into a checkpoint, and the evaluator converts the checkpoint into fitness scores that can be computed on a subsample or estimated with IRT-based approximation. This loop is what lets arbitrary evolutionary algorithms be plugged in without rewriting the merging code, and it is also where the cost savings happen, because fitness, the most expensive step, is deliberately approximated.","core_discovery":"The paper introduces Mergenetic as the first library purpose-built for evolutionary model merging in language models, combining six merging strategies with nineteen evolutionary algorithms and supporting both single- and multi-objective search. The central design is a modular search loop: a merger maps a genotype (the coefficients of a merge recipe) to a checkpoint, an evaluator maps that checkpoint to one or more fitness scores, and a searcher orchestrates selection, crossover, and mutation over generations. To keep the loop affordable, fitness is computed on subsampled datasets or through item-response-theory-style estimators instead of full benchmark evaluation. The paper reports that this pipeline produces a multilingual ARC model surpassing each of its four language-specific parents by up to 19% accuracy, and a Japanese math model that beats its English and Japanese parents by 10-20% on the Japanese math benchmark, using modest GPUs and only small fitness subsets (100 examples in the math case, 20 per language in the multilingual case).","pith_inferences":["The small fitness subsets (20 examples per language and 100 math examples) are the main uncontrolled variable; a direct test would rerun the same searches with full-dataset fitness and compare the winning merge recipes, which would reveal whether the cost savings come with a quality cost.","The modular genotype-to-model-to-fitness loop should carry over to other domains with mergeable checkpoints and cheap evaluators, such as vision-language or multimodal models, but the paper demonstrates only text language models.","The built-in multi-objective support could be used to optimize per-language worst-case accuracy rather than average accuracy, turning the merge search into an explicitly fairness-aware procedure; the paper does not explore this direction."],"forward_implications":["Users can compare single-objective and multi-objective merging on the same pipeline by changing only the optimizer and the objective definitions, so questions like 'does balancing two languages beat optimizing one?' become configuration changes.","A practitioner with one consumer GPU and a set of already fine-tuned checkpoints can run an evolutionary search instead of hand-tuning merge weights or training a new model from scratch.","Merging becomes a way to transplant skills across languages, as the Japanese math experiment shows: an English math capability can be transferred into a Japanese-tuned model without retraining.","Because fitness is approximated, iterative workflows become practical: run a short search, inspect the logged population, adjust the objectives or the evaluation subset, and relaunch."],"supporting_citations":[{"why":"Supplies the underlying merging strategies that Mergenetic wraps and delegates to for checkpoint combination.","marker":"Goddard et al., 2024"},{"why":"Supplies the suite of single- and multi-objective evolutionary algorithms that drive the search.","marker":"Blank and Deb, 2020"},{"why":"Powers the fitness-evaluation backend with standardized tasks and metrics.","marker":"Gao et al., 2024"},{"why":"Establishes the evolutionary model-merging recipe approach that Mergenetic generalizes, including the cross-lingual evaluation protocol.","marker":"Akiba et al., 2025"},{"why":"Supplies the consumer-GPU evolutionary merging pipeline and the merging-specific IRT fitness estimators used for cheap evaluation.","marker":"Mencattini et al., 2025"},{"why":"Provides the item-response-theory basis for estimating full-dataset performance from few examples.","marker":"Polo et al., 2024"},{"why":"TIES is one of the merging strategies exposed by the library and is used in the multilingual case study.","marker":"Yadav et al., 2023"},{"why":"DARE is a supported merging strategy, combined with TIES or task arithmetic, and used in the multilingual case.","marker":"Yu et al., 2024"},{"why":"ARC-Challenge is the benchmark on which the multilingual merged model's gains are measured.","marker":"Clark et al., 2018"},{"why":"MGSM supplies the Japanese math benchmark used for the cross-lingual case study.","marker":"Shi et al., 2022"}],"fun_headline_variants":["First library for evolutionary model merging on one GPU","Mergenetic: 19 optimizers, 6 merging strategies, one library","Evolve your LLM merges with cheap fitness checks","Mergenetic fuses evolution and merging for better LLMs","Multilingual ARC up 19% with evolutionary merging"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that fitness computed on very small benchmark subsets (20 translated ARC examples per language and 100 Japanese math examples) ranks candidate merges the same way full-dataset evaluation would, and the paper provides no test of that correlation.","fun_headline_variants_meta":{"raw":{"variants":["First library for evolutionary model merging on one GPU","Mergenetic: 19 optimizers, 6 merging strategies, one library","Evolve your LLM merges with cheap fitness checks","Mergenetic fuses evolution and merging for better LLMs","Multilingual ARC up 19% with evolutionary merging"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000858,"raw_usage":{"total_tokens":3671,"prompt_tokens":839,"completion_tokens":2832,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":455,"completion_tokens_details":{"reasoning_tokens":2745}},"tokens_in":455,"tokens_out":2832,"duration_ms":20308,"temperature":1.0,"reasoning_tokens":2745,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:53:43.965118+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun both evolutionary searches with fitness computed on the full benchmarks (full ARC-Challenge per language and the full Japanese math benchmark) and compare the winning merge coefficients with those found on the small subsets; if the winning recipes diverge substantially, or if the subset-chosen merges lose most of their reported edge on the full test sets, the small-sample fitness assumption is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"TIES is one of the merging strategies exposed by the library and is used in the multilingual case study."}],"review_version":1}