{"id":"9724f2b7-3cac-4fbd-82b7-b54500bccb1c","arxiv_id":"2501.08361","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Gradient-similarity-regularized weight averaging and WA+SAM fine-tuning are tested on OOD and few-shot domain adaptation benchmarks, with mixed results that do not support the claimed improvements.","lead":"This thesis combines weight averaging with a gradient-similarity regularizer to improve out-of-distribution generalization, and pairs weight averaging with sharpness-aware minimization for few-shot domain adaptation. The reported gains are mixed: the proposed method trails the DiWA baseline on two DomainBed datasets and loses to d-SNE on most digit adaptation tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own reported results contradict the abstract: WA+grad averages below DiWA-60 on both PACS (88.46 vs 89.0) and VLCS (77.86 vs 78.6), leaving the central OOD improvement claim unsupported.","rationale":"The reader's REJECT verdict is well supported by the paper's own numbers. My stress-test focused on the most load-bearing condition for the abstract's first claim: that gradient regularization improves OOD accuracy over weight averaging alone. The PACS/VLCS tables show the opposite—the averaged accuracy of WA+grad is lower than DiWA-60 on both benchmarks. Since DiWA-60 uses more models, the comparison is imperfect, but the burden is on the paper to show an improvement at matched cost; it does not. The proposed mechanism (gradient diversity → beneficial weight-space diversity) is never directly measured. I considered instead the few-shot DA claim, which also lacks a no-WA control and loses to d-SNE on VisDA-C by a large margin; however, the OOD claim is the more central contribution and the one most directly contradicted. The paper's Discussion honestly acknowledges the results were 'not entirely promising', which strengthens the case that the abstract overstates the findings. A matched-model-count replication would settle whether the negative comparison is an artifact of model count or a genuine failure of the method. No further concerns about internal consistency or novelty were identified; the limitations are explicitly acknowledged, but they are fatal to the central claim as stated.","tokens_in":22273,"tokens_out":5601,"duration_ms":54424,"concrete_test":"Rerun PACS and VLCS with DiWA and WA+grad at the same number of models (e.g., M=40), same hyperparameter search budget, and at least five seeds, reporting paired per-domain means and standard errors. If WA+grad does not exceed DiWA by a statistically significant margin, the central OOD improvement claim fails. During the same runs, log the pairwise cosine similarity between weight vectors; if the regularizer does not increase this angle relative to DiWA's unregularized models, the proposed mechanism is not operating.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central OOD claim is that explicitly encouraging gradient diversity between paired models improves the averaged model's OOD accuracy. For this to hold, WA+grad should at least match DiWA, the closest baseline, when compared fairly. The paper's Tables 4 and 5 report WA+grad averages of 88.46 on PACS (M=40) and 77.86 on VLCS (M=30), both below the DiWA-60 averages of 89.0 and 78.6. The comparison is not matched in model count, but the direction is unfavorable and no significance testing is provided. In addition, the mechanism is not verified: Algorithm 1 never reports the cosine similarity values or the resulting weight-angle changes, so there is no evidence that the regularizer actually increases weight-space diversity. Because the claimed improvement over prior WA is the paper's primary contribution, the absence of a positive, controlled comparison is load-bearing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript, originally an MSc thesis, studies weight averaging for out-of-distribution (OOD) generalization and few-shot domain adaptation. It proposes two extensions: (i) a gradient-similarity regularizer applied to pairs of models during fine-tuning, intended to increase model diversity before weight averaging, and (ii) combining weight averaging with sharpness-aware minimization (SAM) for few-shot domain adaptation, with adaptation performed either before or after averaging. Experiments are reported on CIFAR-100, PACS, VLCS, digit datasets (MNIST, SVHN, USPS, MNIST-M), and VisDA-C, with comparisons to ERM, DiWA, SWAD, MA, Coral, and several few-shot DA baselines. The abstract claims improved OOD generalization from gradient-diversity regularization and significantly improved few-shot domain adaptation accuracy from combining WA and SAM.","tokens_in":22498,"tokens_out":6008,"duration_ms":60817,"significance":"If the central claims were substantiated, the work would offer a simple training-time regularizer for improving weight averaging under distribution shift and a practical few-shot domain adaptation recipe. The manuscript is transparent about its limitations and contains algorithm listings, ablation studies, and public-benchmark evaluations. However, the reported evidence is mixed or negative on the decisive comparisons: the proposed WA+grad is below DiWA on the average over both DomainBed datasets, the VisDA-C result is far below two baselines, and the few-shot DA experiments lack a no-weight-averaging control under the same setup. The main empirical claims are therefore not established, and the manuscript would need substantial new experiments and a substantially more modest claim before it could be considered for publication.","major_comments":[{"comment":"The headline OOD claim is not supported by the paper's own tables. WA+grad obtains 88.46 on PACS with M=40 and 77.86 on VLCS with M=30, while DiWA with uniform selection and M=60 obtains 89.0 and 78.6, respectively. The comparison against DiWA restricted (M≤20) is also mixed: 88.46 vs 88.0 on PACS, but 77.86 vs 78.5 on VLCS. No standard deviations or significance tests are reported for the DiWA uniform or WA+grad numbers, and the number of averaged models differs. Matched comparisons with equal M, equal training budget, and multiple seeds are needed before any improvement over DiWA can be claimed.","section":"§5.1, Tables 4 and 5"},{"comment":"The mechanism underlying the proposed regularizer is not verified. The paper never reports the gradient cosine-similarity values, the resulting pairwise weight angles, or an ablation that isolates the gradient regularizer from the choice of optimizer and hyperparameters. Without such measurements, the claim that penalizing gradient similarity increases weight-space diversity and thereby improves the averaged model's OOD accuracy remains an unverified hypothesis. In addition, cossim is not formally defined in Algorithm 1, and no regularization coefficient is specified, so the procedure is not fully reproducible as written.","section":"§4.2, Algorithm 1"},{"comment":"The few-shot domain adaptation experiments lack a no-weight-averaging control under the same training setup. The comparisons to FADA, CCSA, and d-SNE use different architectures, backbones, and experimental protocols, so the reported gains cannot be attributed to weight averaging. The optimizer ablation in Table 8 compares Adam versus SAM but always within weight averaging; it does not compare WA to single-model fine-tuning. Adding a single-model baseline fine-tuned with the same optimizer, backbone, and adaptation protocol is necessary to isolate the effect of weight averaging.","section":"§5.2, Tables 6–8"},{"comment":"The VisDA-C result directly contradicts the abstract's claim that combining WA and SAM significantly increases few-shot domain adaptation accuracy. Table 9 reports WA at 49.41 versus 80.66 for d-SNE and 79.2 for LCCS, and the text describes this as a 'significant discrepancy' with additional analysis promised. The Discussion also states that the gradient-diversity experiments were 'not entirely promising.' These are explicit limitations in the manuscript itself, and they undermine both central claims as stated.","section":"§5.2, Table 9 and §6"},{"comment":"The proposed gradient-similarity regularizer is closely related to the ensemble-diversity method of Teney et al. [85], which is cited in Related Work but not included as a baseline or discussed in the experiments. The manuscript should explain the distinction between enforcing diversity in prediction ensembles and enforcing it in weight-averaged models, and it should compare against [85] empirically. As written, the claimed novelty of explicitly training models with a gradient-similarity regularizer for diversity is substantially weakened by this omission.","section":"§2.4 and §5.1"}],"minor_comments":[{"comment":"The condition 'where ρ≤0' should read ρ≥0 for the SAM neighborhood size; as printed, the constraint is impossible.","section":"§3.6, Eq. (17)"},{"comment":"The right-hand side of the SAM update approximation omits the learning rate λ that appears on the left-hand side; the notation should be made consistent.","section":"§3.6, Eq. (20)"},{"comment":"The dropout distribution entry '[0, 0.1l 0.5]' contains an apparent typo ('l' instead of a separator or bracket) and should be corrected.","section":"Table 2"},{"comment":"There are multiple typos, including 'nueral' in §3.2, 'covraiate' in §3.2, 'problmes' in §2.6, and 'tacked' in §6; a careful proofread is needed.","section":"Throughout"},{"comment":"The note that the best 7-shot results are underlined and best 10-shot results are bold is not fully consistent: for several columns the best value is not uniquely identifiable, and no standard deviations are reported for the WA rows, making the comparisons hard to assess.","section":"Table 6"}],"recommendation":"reject","confidential_remarks":"This submission reads as an unrevised MSc thesis. The core contribution is incremental relative to existing weight-averaging and diversity-regularization literature, and the paper's own experiments contradict the abstract on the PACS and VLCS averages and on VisDA-C. I would not encourage resubmission unless the authors can provide matched, multi-seed comparisons, a verified diversity mechanism, and a working few-shot DA recipe on at least one large-scale benchmark."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a clearly written thesis that honestly reports mixed evidence, but its main OOD claim is contradicted by its own tables. On PACS, WA+grad averages 88.46, below DiWA-60's 89.0; on VLCS, 77.86 vs 78.6. The few-shot DA results are more interesting, but they lack the baseline needed to attribute the gain.\n\nWhat the paper does well: it is well-organized, the discussion openly admits the gradient-diversity results were 'not entirely promising', and it includes several ablations (architecture, optimizer, number of models, number of shots). The t-SNE visualizations are a useful qualitative check. The digit-task results consistently show adaptation-after-WA beating single-model adaptation, and SAM helps across the board.\n\nThe soft spots are structural. The main novelty — gradient similarity as a diversity regularizer — is already in Teney et al. [85], which the paper cites; moving it into weight averaging is a straightforward transfer. There is no direct evidence that the regularizer actually increases weight-space diversity: Algorithm 1 never reports cosine similarities or weight angles. The DiWA comparison is unmatched in model count (40 vs 60 on PACS, 30 vs 60 on VLCS) and the direction is unfavorable. VisDA-C is a collapse (49.41 vs 80.66 for d-SNE), which the paper acknowledges but does not explain. And the few-shot setup has no no-WA control, so you cannot separate WA from SAM from the combination. The thesis is an honest exploration, but the central empirical claim fails on the numbers, and the proposed mechanism is unverified.\n\nWho gets value from this: someone working on weight averaging who wants a survey of the recipe space and a cautionary example of why matched baselines matter. It does not deserve a serious referee as-is; the primary contribution is not supported. If the author re-ran with matched model counts, verified the diversity mechanism, and either fixed VisDA-C or dropped it, a short workshop paper could be salvageable.","headline":"An honest but incremental thesis whose central OOD claim is not supported by its own tables; the few-shot DA results are more interesting but lack a no-WA control.","tokens_in":22992,"tokens_out":3181,"would_cite":false,"duration_ms":30888,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that explicit gradient diversity improves weight-averaged out-of-distribution generalization, and that weight averaging plus SAM substantially boosts few-shot domain adaptation.","keywords":["weight averaging","out-of-distribution generalization","few-shot domain adaptation","gradient similarity","sharpness-aware minimization","model diversity","domain generalization","covariate shift"],"falsifier":"Train the same paired models with a sweep of the gradient-similarity penalty weight and measure both the cosine angle between the final weight vectors and the averaged model's OOD accuracy; if accuracy does not rise as weight-space angle grows, the diversity mechanism is not what drives the gain. A matched-model-count comparison of WA+grad against DiWA on PACS and VLCS (using 60 averaged models with identical hyperparameter ranges) would also settle whether the regularizer adds anything beyond plain weight averaging.","tokens_in":22059,"feed_emoji":"⚖️","tokens_out":11249,"duration_ms":98697,"temperature":0.7,"pith_summary":"This paper sets out to strengthen weight averaging as a tool for distribution shift. Its first claim is that explicitly encouraging diversity among the models being averaged — by adding a cosine-similarity penalty on their gradients — improves out-of-distribution generalization over plain weight averaging. Its second claim is that combining weight averaging with sharpness-aware minimization (SAM) gives large gains in few-shot domain adaptation, particularly when the averaged model is adapted to the target domain with a few labeled examples. The experiments cover in-distribution CIFAR100, the public OOD benchmarks PACS and VLCS, digit datasets (MNIST, SVHN, USPS, MNIST-M), and VisDA-C, and the reported gains are largest on the digit adaptation tasks. If the claims hold, practitioners get a way to make averaged models more transferable without any extra cost at inference time.","feed_headline":"Weight averaging plus SAM lifts few-shot domain adaptation","feed_subtitle":"Gradient-diverse weight averaging is reported to improve out-of-distribution accuracy on PACS and VLCS.","key_machinery":"The central machinery is the gradient-similarity regularizer: for each pair of models being trained for averaging, the loss of each model gains the term $\\mathrm{cossim}(\\nabla_{h_{k1}} f_{\\theta_{k1}}(x), \\nabla_{h_{k2}} f_{\\theta_{k2}}(x))$, the cosine similarity between the two models' gradients with respect to their shared feature layers, which the paper uses to push the paired models toward different solutions before their weights are averaged. The other load-bearing ingredient is the weight-averaging protocol itself: all models start from a shared initialization (an ImageNet-pretrained ResNet-50 with a linear-probed final layer, or a CNN trained to at least 85% accuracy on the source domain), each run uses a different combination of learning rate, weight decay, SAM radius, and dropout drawn from a small random search, and the final model is the uniform average of all weights. For few-shot adaptation, the pipeline adds a fine-tuning step on $k$ samples per class of the target domain, and the paper finds that fine-tuning the already-averaged model works better than fine-tuning each model before averaging.","core_discovery":"On the paper's own terms, the discovery is that weight averaging's gains can be driven by explicit model diversity rather than only by the number of models averaged. The OOD method trains pairs of models from a shared initialization and adds the cosine similarity between the pair's gradients to each model's classification loss; averaging all trained weights then yields a model that the paper reports as competitive with prior weight-averaging baselines on PACS and VLCS. For few-shot domain adaptation, the paper shows that averaging source-trained models and then fine-tuning the averaged model on k labeled target examples per class outperforms adapting each model before averaging, and that using SAM as the optimizer yields the better results on the digit benchmarks. On the digits datasets the averaged pipeline exceeds the FADA, CCSA, and d-SNE baselines on several transfer tasks, while on VisDA-C it falls well short of those baselines; the paper identifies this gap as an open issue for further analysis.","pith_inferences":["One reading of the reported numbers is that the gradient-similarity penalty is not yet the right diversity mechanism: WA+grad's averages on PACS and VLCS sit slightly below DiWA's with far fewer models, and Section 6 itself concedes the results were 'not entirely promising.' A testable alternative is to regularize on the weight-space angle directly, or to anneal the penalty during training.","The large VisDA-C gap (49.41 vs 80.66 for d-SNE) suggests that WA+fine-tuning may need a stronger adaptation step for large domain gaps; applying the gradient-diversity regularizer during the adaptation phase, rather than only during source training, is a natural next experiment.","Because the strongest few-shot gains appear on digit-to-digit transfer, the recipe may be a cheap and strong baseline for small-scale domain adaptation; comparing WA+SAM against self-supervised or feature-alignment methods on the same tasks would clarify where the gain comes from."],"forward_implications":["If the gradient-similarity mechanism works, then explicit diversity regularization becomes a lever for improving weight-averaged models under covariate shift, complementing the implicit diversity from hyperparameter variation.","If WA+SAM is as effective as reported on digits, few-shot domain adaptation can be done by averaging a handful of source models and fine-tuning the average, avoiding the need to fine-tune each model separately.","The finding that adaptation after averaging beats adaptation before averaging implies that preserving the common features learned by the ensemble is critical; averaged models are a better starting point for target fine-tuning than any single adapted model.","The ablation showing diminishing returns beyond roughly 20 averaged models suggests that moderate compute budgets suffice to capture most of the OOD gain.","The paper's in-distribution CIFAR100 results imply that the same recipe helps standard i.i.d. accuracy, so the method is not only a distribution-shift tool."],"supporting_citations":[{"why":"Supplies the DiWA weight-averaging method and the shared-initialization plus mild-hyperparameter-search conditions the paper builds on.","marker":"[72]"},{"why":"Documents the correlation between model angle and accuracy gain that motivates explicit diversity regularization.","marker":"[93]"},{"why":"Introduces SAM, the sharpness-aware optimizer the paper mates with weight averaging for few-shot adaptation.","marker":"[22]"},{"why":"Establishes stochastic weight averaging and flat minima, the foundation for averaging weights.","marker":"[40]"},{"why":"Provides SWAD, a flat-minima baseline the paper compares against on the OOD benchmarks.","marker":"[13]"},{"why":"Provides the linear-probing initialization used as the shared starting point for the averaged models.","marker":"[49]"},{"why":"Supplies the d-SNE few-shot domain adaptation baseline that the paper compares against on digits and VisDA-C.","marker":"[96]"},{"why":"Supplies the FADA few-shot adversarial domain adaptation baseline.","marker":"[59]"}],"fun_headline_variants":["Gradient diversity drives weight averaging gains","Weight averaging plus SAM improves few-shot adaptation","Explicit diversity in weight averaging boosts OOD","Gradient-similarity regularizer strengthens WA","Weight averaging with SAM: better few-shot transfer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that forcing paired models to have dissimilar gradients will push their learned weights into positions where the averaged model is more accurate on out-of-distribution data, rather than simply making each individual model worse.","fun_headline_variants_meta":{"raw":{"variants":["Gradient diversity drives weight averaging gains","Weight averaging plus SAM improves few-shot adaptation","Explicit diversity in weight averaging boosts OOD","Gradient-similarity regularizer strengthens WA","Weight averaging with SAM: better few-shot transfer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000221,"raw_usage":{"total_tokens":1455,"prompt_tokens":957,"completion_tokens":498,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":430}},"tokens_in":573,"tokens_out":498,"duration_ms":5103,"temperature":1.0,"reasoning_tokens":430,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:30:25.644402+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same paired models with a sweep of the gradient-similarity penalty weight and measure both the cosine angle between the final weight vectors and the averaged model's OOD accuracy; if accuracy does not rise as weight-space angle grows, the diversity mechanism is not what drives the gain. A matched-model-count comparison of WA+grad against DiWA on PACS and VLCS (using 60 averaged models with identical hyperparameter ranges) would also settle whether the regularizer adds anything beyond plain weight averaging.","supporting_citations":[{"cited_title":"Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time","cited_arxiv_id":null,"evidence_quote":"Documents the correlation between model angle and accuracy gain that motivates explicit diversity regularization."},{"cited_title":"d-sne: Domain adaptation using stochastic neighborhood embedding","cited_arxiv_id":null,"evidence_quote":"Supplies the d-SNE few-shot domain adaptation baseline that the paper compares against on digits and VisDA-C."},{"cited_title":"Few-shot adversarial domain adaptation","cited_arxiv_id":null,"evidence_quote":"Supplies the FADA few-shot adversarial domain adaptation baseline."}],"review_version":1}