GimmBO: Interactive Generative Image Model Merging via Bayesian Optimization

Alec Jacobson; Chenxi Liu; Selena Ling

REVIEW 3 major objections 4 minor 4 references

GimmBO replaces slider-based adapter merging with an interactive Bayesian optimizer that exploits sparsity and small sum-of-weights for image-style merges.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 07:55 UTC pith:2DCEBTQF

load-bearing objection Worth a serious look: the method is a real step for interactive BO at 20-30D, but the self-aligned evaluation means the reported gains are on the method's own prior. the 3 major comments →

arxiv 2601.18585 v2 pith:2DCEBTQF submitted 2026-01-26 cs.CV cs.GR

GimmBO: Interactive Generative Image Model Merging via Bayesian Optimization

Chenxi Liu , Selena Ling , Alec Jacobson This is my paper

classification cs.CV cs.GR

keywords model mergingpreferential Bayesian optimizationdiffusion modelsLoRA adaptershuman-in-the-loopcapped simplexsparsityinteractive image generation

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that the tedious manual workflow of blending 20–30 fine-tuned image adapters by hand-tuning sliders can be replaced by an interactive optimizer that queries the user only to rank a few generated images. The key move is to encode two empirically observed properties of real-world adapter merges, namely that only a few adapters are active and that the total sum of merge coefficients stays small, into the search geometry of a Bayesian optimizer. The resulting method, GimmBO, first explores a B-capped simplex (sum of coefficients ≤ 2) with a sparsity-favoring prior, then fixes the discovered active adapter set and refines the coefficients in the full [0,1]^z. If the claims hold, users can converge to a desired style in about 15–20 ranking rounds rather than fiddling with 20–30 separate sliders.

Core claim

GimmBO's central claim is that incorporating the geometry of real adapter usage—sparse active sets and coefficient sums of roughly 1 to 3—into a preferential Bayesian optimization backend yields faster convergence and higher-quality adapter merges than generic BO or slider-based line search in 20- to 40-dimensional design spaces. Concretely, in a 20D matching task GimmBO reports rapid Stage 1 gains via the capped-simplex constraint, further improvement at the onset of Stage 2 when the active set is fixed, average user-study similarity above 0.9, and the highest sparsity-recovery F1 among all baselines, with graceful degradation as dimensionality rises to 40D. The paper argues that previous i

What carries the argument

The B-capped simplex Δ_B = {α ∈ [0,1]^n : Σ α_i ≤ B}, with B=2 by default. Its volume in [0,1]^n shrinks super-exponentially with n (for example, Vol(30;2) < 10^-21%), which is precisely why naive sampling fails and why restricting early exploration to it is informative. Samples are drawn by a coordinate-ordered stick-breaking process, and the two-stage schedule first searches Δ_B for 10 iterations, then fixes the sparsity pattern of the current best and optimizes only the active coefficients in [0,1]^z for an additional 9 iterations. The surrogate is a Gaussian process with a sparse axis-aligned subspace (SAAS) prior, conditioned on pairwise user preferences via a probit likelihood, and acq

Load-bearing premise

The load-bearing premise is that good adapter merges are sparse and have a small total coefficient sum, so the B-capped simplex with B=2 contains the optimum; if real users' ideal merges frequently involve many adapters or summed weights above about 3–4, Stage 1 will exclude the very solution the pipeline is trying to find.

What would settle it

Run the same 20D matching task with ground-truth targets sampled uniformly from the full [0,1]^20 hypercube (or with sums greater than 4). GimmBO's Stage 1, restricted to sum ≤ 2, cannot represent such targets; if its final similarity and F1 drop to baseline levels while an unconstrained BO retains an advantage, the claimed benefit of the capped-simplex geometry is limited to the surveyed data distribution.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

Users can interactively steer merges of 20–30 adapters in roughly 20 ranking rounds, with content stabilized by lightweight SDEdit control so that style remains the variable of interest.
The approach extends beyond style to content merging (e.g., hair, glasses, expressions) and integrates with coarse text-based adapter retrieval methods to form an end-to-end retrieval-plus-exploration workflow.
A discovered adapter configuration is reusable: it can be applied to arbitrary new images for consistent downstream stylization.
GimmBO degrades gracefully in 30D and 40D stress tests, suggesting that the benefit comes from the geometry-informed search space rather than a fixed dimensionality limit.
In the user study, participants reached an average similarity of over 0.9 on the DreamSim metric with the ranking interface, while the slider baseline showed fluctuating exploration and lower final scores.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The strongest test of the paper's premise would be to draw target merges from a distribution that violates the surveyed geometry, such as uniformly from the full [0,1]^n hypercube or with coefficient sums above 4; if GimmBO's gains then collapse, the improvement is tied to an in-distribution assumption rather than a general optimization advantage.
The fixed 10/9 iteration split between stages suggests an adaptive transition rule based on surrogate uncertainty or plateau detection as a natural extension, which could improve robustness across different merge complexities.
In the user interface, candidates are pre-sorted by the surrogate estimate before the user ranks them; an ablation that randomizes or de-biases the displayed order could quantify whether this pre-sorting steers user feedback in a way that masks exploration.
The user study participants had technical backgrounds, so broader validation with general design users would test whether the ranking interaction itself, not just the backend, remains effective outside an expert population.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Worth a serious look: the method is a real step for interactive BO at 20-30D, but the self-aligned evaluation means the reported gains are on the method's own prior.

read the letter

Here's my read. The core idea — preference-based BO with a B-capped simplex search space and a two-stage sparse-then-refine procedure — is a genuine extension of interactive BO to the 20–30D regime. Prior systems top out around 12D, and the stick-breaking parameterization is a sensible way to sample from the capped simplex. The ablations are coherent and support the design choices: B=2 beats B=1 and unconstrained, top-5 beats top-1, and the two-stage split does what it claims. The Stylus integration is a nice practical touch.

My main reservation is the evaluation design. Appendix B says the test coefficient vectors are sampled so that their active-adapter count and magnitude distributions match the surveyed real-world statistics. Those same statistics motivate the B=2 cap and sparsity assumptions. So the benchmarks are partly a check of the authors' prior, not a test of whether the prior holds. The 30D/40D stress tests use the same sampling, so they don't lift that concern. I would have liked to see at least one evaluation with dense or high-sum targets, and a limitation note acknowledging this. The user study is also small — 12 CS/AI participants — and between-method differences are not statistically characterized; they only test within-method improvement. That's minor relative to the circularity, but worth noting.

The central method holds up. The paper is clearly written and the implementation details are unusually thorough — exact hyperparameters, prompt templates, negative prompts, SDEdit settings. I don't see internal incoherence. But the external validity is genuinely narrower than the abstract claims, because the evaluation never leaves the distribution that defines the method's key assumption.

Who should read it: researchers in interactive design optimization, model merging, and generative-model UX. I'd send it to peer review, with a strong request for off-prior evaluation and a more rigorous user-study analysis. If the authors add those, this is a solid TOG or UIST paper.

Referee Report

3 major / 4 minor

Summary. The paper proposes GimmBO, an interactive preferential-Bayesian-optimization backend for choosing adapter-merging coefficients in text-to-image diffusion models. Motivated by a Civitai survey suggesting that effective merges are sparse and have small total coefficient magnitude, the method restricts Stage 1 search to a B-capped simplex (Eq. 6, B=2), uses a stick-breaking initialization, and then switches in Stage 2 to an unconstrained search over the active adapter set extracted from the current best solution. The paper evaluates the method with simulated DreamSim-based users in 20D/30D/40D matching tasks, a 12-participant user study, ablations, and several extensions including content merging and integration with a retrieval system. The central claims are that GimmBO improves sampling efficiency and convergence in high-dimensional adapter-merging spaces and consistently outperforms BO and line-search baselines.

Significance. If the result holds, the paper addresses a real and growing problem: manually tuning 20–30 adapter sliders is impractical, and existing interactive BO methods are designed for lower-dimensional design spaces. The method is well engineered and clearly specified: it builds on SAAS prior, pairwise probit likelihood, batched UCB acquisition, and a two-stage sparsity-then-refine procedure. The paper provides ablations for B, top-k, and past-sample inclusion, a user study, and stress tests in higher dimensions. The main strength is the integration of domain statistics into the search geometry, which is plausible and well motivated. However, the evaluation is substantially weakened by the fact that the ground-truth test coefficients are sampled from the same Civitai-derived distribution used to select B=2 and the sparsity assumption, so the benchmark partly measures the method's ability to optimize within its own prior rather than its general applicability to adapter merging.

major comments (3)

[Sec. 4.5] The benchmark is circular. Section 3 infers the B-capped simplex (B=2) and sparsity from a Civitai survey, and App. B states that test coefficient vectors are sampled so that 'their distributions of active adapter count and coefficient magnitude match the surveyed real-world statistics.' Thus the simulated-user targets live inside the same small-summation, few-active-adapters prior that motivates the search-space restriction. Unconstrained baselines are then evaluated on a distribution that is, by construction, favorable to the constrained method. The 30D/40D stress tests use the same sampling procedure, and the user-study reference images are generated from the same Civitai-matched GT coefficients, so they inherit the issue. The paper's own histograms in Sec. 3 show nonzero mass at up to 9 active adapters and sums up to about 7.2, yet Table 1 contains no target with z>5 or sum>4. The ma
[Sec. 5.1.1] The two-stage design has a structural failure mode when the Stage-1 geometry is wrong. Stage 1 searches only Δ_B, so any optimum outside that set is never sampled. Stage 2 then fixes the sparsity pattern of the current top-ranked Stage-1 solution and optimizes only over those z active coefficients; it cannot reintroduce an adapter that Stage 1 dropped. If real user goals frequently involve more adapters or larger coefficient sums than B=2 allows, Stage 2 can only refine the wrong active set. The paper does not analyze this failure mode, and the ablation in Fig. 8 varies B only over {1,2,n}; it does not test sensitivity to B on target distributions that are not matched to the B=2 prior. Please add an experiment that starts from targets outside Δ_B and reports final similarity and active-set F1, and consider a safeguard that allows Stage 2 occasionally to revisit excluded dimensions.
[Sec. 5.1] The headline claim of consistent gains over baselines is reported as averaged running-best curves with no confidence intervals or between-method significance tests in the simulated experiments. The user study reports within-method improvement (p<0.05, p<10^-3) but not pairwise significance among interfaces. Given the visible spread of success rates and F1 across z=2,...,5, some indication of variability (e.g., shaded intervals or per-seed analysis) is needed to support the strength of the claim. This is secondary to the circularity concern but still relevant to the evaluation section.

minor comments (4)

[General] Typo: 'Civitiai' should be 'Civitai'.
[Eq. (10)] The notation E_φ max_j φ(α_j | f-hat) is informal. Please clarify that the expectation is over the GP posterior and that the optimization is over the batch of q points, and specify the reparameterization used in BoTorch.
[Sec. 5.1.2] The matched-time-budget argument ('20 baseline steps correspond to approximately 14 (slider) and 4 (gallery) steps of our method') is interesting but the derivation is not shown. Please make the per-step time accounting explicit, including how the 50.5 s vs 34.7 s vs 10.6 s values are computed and what exactly is included.
[Sec. 4.5] The stage split T1=10, T2=9 is fixed across all experiments. A brief statement of whether performance is sensitive to this split, or a one-line rationale, would improve reproducibility and help readers understand the design choice.

Circularity Check

0 steps flagged

No significant circularity: the only trace is a non-load-bearing self-citation; the benchmark-prior overlap is an external-validity concern, not a derivation cycle.

full rationale

GimmBO's derivation chain is not circular. The B-capped simplex search space (Eq. 6) is motivated by independent survey observations in Sec. 3: 'Surveying real-world practice, we observe that effective adapter merges are typically sparse... and that merging coefficients must lie within a limited magnitude range.' The benchmark targets are independently constructed ground-truth coefficient vectors (App. B) and corresponding images; the optimizer never observes the GT coefficients or the target image, only DreamSim-based preference comparisons (Sec. 5.1.1). Matching the test distribution to the surveyed real-world statistics is a deliberate realism choice, not a way of encoding GimmBO's own output into the evaluation. Indeed, the constructed GT vectors include weight sums up to (3,4] (Tables 1-2), outside the default B=2 cap, so success is not guaranteed by construction. The claim that 'GimmBO achieves rapid Stage 1 gains via the capped simplex constraint' is an empirical result, not a consequence of the definition of the target. The only self-citation is [Liu et al. 2024] in the related-work paragraph on community-model retrieval: 'A related line of work studies community-model retrieval [Liu et al. 2024; Luo et al. 2024; Sonmezer et al. 2025]'; it is not load-bearing for the method or the evaluation. The benchmark-prior overlap is an external-validity limitation (generalization to merges outside the sparse/small-sum prior is untested), but that is not a circular reduction in the derivation chain. Therefore no significant circularity.

Axiom & Free-Parameter Ledger

8 free parameters · 6 axioms · 0 invented entities

The central method rests on a small set of domain priors (sparsity, bounded sum, linear mergeability) plus standard BO modeling choices. The most consequential is the B-capped simplex: B=2 is estimated from the same Civitai data later used to construct test targets, so the benchmark is partly generated from the method's own prior. No invented entities are introduced.

free parameters (8)

B (simplex cap) = 2
Chosen from observed distribution of sum of merging coefficients on Civitai (Sec. 3); default in all experiments and ablations. This is the load-bearing search-space constraint.
lambda (UCB exploration) = 9
Default UCB acquisition parameter (App. A); no sensitivity analysis reported.
tau (sparsity threshold) = 0.1
Initialization threshold below which coefficients are zeroed (Sec. 4.3); hand-set.
T1/T2 stage split = 10/9
Number of iterations in initial and polishing stages; fixed for all experiments, no tuning analysis.
top-k k = 5
Number of ranked images per iteration; ablation shows top-5 beats top-1 (Fig. 8).
q batch size = 8
Inferences per iteration; reflects interaction budget.
SDEdit t0 = 0.8
Content-control strength; chosen empirically via style similarity over 100 combinations (App. A.4).
GT target coefficient distribution = active counts 2-5; sums (0,4]; bins Tables 1-2
Test targets sampled to match the same Civitai statistics used to motivate sparsity and B=2; introduces evaluation circularity.

axioms (6)

domain assumption Linear merging (W0 + sum alpha_i ΔW_i, Eq. 1) of adapters from a common base yields meaningful combinations.
Invoked in Sec. 3; justified by linear mode connectivity and practice, but assumed for LoRA style adapters.
domain assumption Effective adapter merges are sparse (typically 2-9 active adapters).
Sec. 3 survey of 20,780 Civitai images; drives two-stage sparsity.
domain assumption High-quality merges have small total coefficient sum (proxy B=2).
Sec. 3 histogram of sums; modeled as B-capped simplex Eq. 6. This is the key unproven geometric assumption.
domain assumption User preference is transitive and representable by a latent scalar utility with probit noise.
Sec. 4.1 and App. A.1; standard pairwise-preference model; authors flag transitivity as a limitation.
domain assumption DreamSim similarity to a target image is a valid proxy for human preference in the matching task.
Sec. 5.1; used as simulated user and performance metric; not independently validated for this task.
standard math A GP with SAAS prior is an adequate surrogate for the unknown preference function in 20-40D.
Sec. 4.1/App. A.1; standard BO machinery, but the SAAS prior sparsity assumption is itself domain-motivated.

pith-pipeline@v1.3.0-alltime-deepseek · 20740 in / 12384 out tokens · 121557 ms · 2026-08-03T07:55:15.566828+00:00 · methodology

0 comments

read the original abstract

Fine-tuning-based adaptation is widely used to customize diffusion-based image generation, leading to large collections of community-created adapters that capture diverse subjects and styles. Adapters derived from the same base model can be merged with weights, enabling the synthesis of new visual results within a vast and continuous design space. To explore this space, current workflows rely on manual slider-based tuning, an approach that scales poorly and makes weight selection difficult, even when the candidate set is limited to 20-30 adapters. We propose GimmBO to support interactive exploration of adapter merging for image generation through Preferential Bayesian Optimization (PBO). Motivated by observations from real-world usage, including sparsity and constrained weight ranges, we introduce a two-stage BO backend that improves sampling efficiency and convergence in high-dimensional spaces. We evaluate our approach with simulated users and a user study, demonstrating improved convergence, high success rates, and consistent gains over BO and line-search baselines, and further show the flexibility of the framework through several extensions.

Figures

Figures reproduced from arXiv: 2601.18585 by Alec Jacobson, Chenxi Liu, Selena Ling.

**Figure 1.** Figure 1: Given a prompt, default image generation may appear reasonable yet fail to match a user’s creative intent (left). GimmBO enables users to explore [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

**Figure 2.** Figure 2: The adapter design space is vast. GimmBO assists users to produce [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Figure 3: Previous BO-based interaction methods [Koyama et al. 2020], which do not account for the structure of the adapter-merging design space, may propose clamped or degraded samples in high-dimensional settings. reduce direct parameter manipulation, with Bayesian Optimization (BO) later adopted as a sample-efficient backend within these settings for visual design tasks [Koyama et al. 2020, 2017]. BO has also be… view at source ↗

**Figure 4.** Figure 4: Starting from initial samples, GimmBO iteratively proposes new [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

**Figure 5.** Figure 5: Batch samples at Steps 1 and 10 in a 20D matching task ( [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗

**Figure 6.** Figure 6: Running-best similarity curves averaged over 30 prompt-weight combinations and 5 random seeds, together with success rates and sparsity recovery [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗

**Figure 7.** Figure 7: Running-best similarities averaged over 12 input-user combinations. [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗

**Figure 8.** Figure 8: Left: running-best similarity curves under increasing dimensionality (30D and 40D), comparing GimmBO to Sequential Gallery [ [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗

**Figure 9.** Figure 9: Given a text prompt, 20-25 community-shared adapters are gathered via coarse text-based retrieval of Stylus [ [PITH_FULL_IMAGE:figures/full_fig_p008_9.png] view at source ↗

**Figure 10.** Figure 10: Applications of adapter merging discovered with GimmBO. Disclaimer: Image generators and style adapters may not preserve facial identities or [PITH_FULL_IMAGE:figures/full_fig_p009_10.png] view at source ↗

**Figure 11.** Figure 11: Given a text prompt, SDEdit [Meng et al. 2021] initializes the denoising process from a provided control image and injects noise corresponding to timestep 𝑡0 ≤ 1. Smaller 𝑡0 values preserve the control image more strongly (stronger control), while larger 𝑡0 values allow greater deviation (weaker control). discard entries with magnitude below 0.1. We then construct𝑚 = 30 merging coefficient vectors whose … view at source ↗

**Figure 12.** Figure 12: Example images displayed in our matching task description. These [PITH_FULL_IMAGE:figures/full_fig_p014_12.png] view at source ↗

**Figure 13.** Figure 13: Interfaces shown to participants in the user study. Each interface displays the target image for the matching task and provides utility buttons (bottom [PITH_FULL_IMAGE:figures/full_fig_p015_13.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

4 extracted references · 4 linked inside Pith

[3]

doi:10.1038/s41592-019-0686-2 Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman

SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python.Nature Methods17 (2020), 261–272. doi:10.1038/s41592-019-0686-2 Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman. 2023. p+: Extended textual conditioning in text-to-image generation.arXiv preprint arXiv:2303.09522 (2023). Weiran Wang and Canyi Lu. 2015. Projection onto the c...

Pith/arXiv arXiv 2020
[4]

a drawing of

TIES-Merging: Resolving Interference When Merging Models. InAd- vances in Neural Information Processing Systems 36: Annual Conference on Neu- ral Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023. http://papers.nips.cc/paper_files/paper/2023/hash/ 1644c9af28ab7916874f6fd6228a9bcf-Abstract-Conference.html Hu Ye...

Pith/arXiv arXiv 2023
[2020]

InProceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology

Design adjectives: a framework for interactive model-guided exploration of parameterized design spaces. InProceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. 261–278. Maria Shugrina, Ariel Shamir, and Wojciech Matusik. 2015. Fab forms: Customizable objects for fabrication with validity and geometry caching.ACM Transacti...

Pith/arXiv arXiv 2015
[2023]

InProceedings of the IEEE/CVF conference on computer vision and pattern recognition

Multi-concept customization of text-to-image diffusion. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 1931–1941. Dongxu Li, Junnan Li, and Steven Hoi. 2023. Blip-diffusion: Pre-trained subject repre- sentation for controllable text-to-image generation and editing.Advances in Neural Information Processing Systems36 (2...

Pith/arXiv arXiv 1931

[1] [3]

doi:10.1038/s41592-019-0686-2 Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman

SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python.Nature Methods17 (2020), 261–272. doi:10.1038/s41592-019-0686-2 Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman. 2023. p+: Extended textual conditioning in text-to-image generation.arXiv preprint arXiv:2303.09522 (2023). Weiran Wang and Canyi Lu. 2015. Projection onto the c...

Pith/arXiv arXiv 2020

[2] [4]

a drawing of

TIES-Merging: Resolving Interference When Merging Models. InAd- vances in Neural Information Processing Systems 36: Annual Conference on Neu- ral Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023. http://papers.nips.cc/paper_files/paper/2023/hash/ 1644c9af28ab7916874f6fd6228a9bcf-Abstract-Conference.html Hu Ye...

Pith/arXiv arXiv 2023

[3] [2020]

InProceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology

Design adjectives: a framework for interactive model-guided exploration of parameterized design spaces. InProceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. 261–278. Maria Shugrina, Ariel Shamir, and Wojciech Matusik. 2015. Fab forms: Customizable objects for fabrication with validity and geometry caching.ACM Transacti...

Pith/arXiv arXiv 2015

[4] [2023]

InProceedings of the IEEE/CVF conference on computer vision and pattern recognition

Multi-concept customization of text-to-image diffusion. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 1931–1941. Dongxu Li, Junnan Li, and Steven Hoi. 2023. Blip-diffusion: Pre-trained subject repre- sentation for controllable text-to-image generation and editing.Advances in Neural Information Processing Systems36 (2...

Pith/arXiv arXiv 1931