{"id":"e31877e6-63ee-45b4-b1d7-1c06ed5a17d3","arxiv_id":"2501.04958","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"IADA is claimed to improve embryo-stage classification across four imaging modalities, but the supporting table lacks variance and includes AUC drops that contradict the advertised improvements across all scenarios.","lead":"This paper proposes Imbalance-Aware Domain Adaptation (IADA), which combines class-specific attention, reweighted adversarial alignment, and adaptive thresholds to classify embryo images across different microscope types. The authors report accuracy gains over a single prior method, but the evidence is weakened by missing error bars, an unverified code link, and AUC results that partly contradict the abstract.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains likely inflated by hyperparameter selection on target test sets; the protocol in Sections 6.1 and 6.3 makes the central claim unsupported.","rationale":"The reader's stated weakest assumption is unreported seed-to-seed variability. That is a real problem, but I judge it secondary. Seed variability widens the error bars around point estimates; target-set hyperparameter selection can bias the point estimates themselves. The evidence is in the paper: Section 6.1 says λreg and λadv are chosen by line search, and Section 6.3 evaluates the searched values by target-scenario AUC on the same ED3, ED2, and ED1 test sets. For a 258-image target set, a 25.19 percentage point accuracy swing corresponds to about 65 correct images, while the ablation curves show AUC movements of 0.3–0.5 over the searched λ ranges. Selecting the favorable point on those curves is more than enough to produce the reported gains. The missing variance metrics compound the problem: without per-seed results, there is no way to know whether even a correctly held-out evaluation would be stable. I therefore agree with the reader's REJECT verdict, but for a more decisive reason than the one listed as the weakest assumption. Secondary issues noted by the reader also stand: Section 6.2's claim of AUC improvement across all scenarios is directly contradicted by Table 1 (e.g., ED4→ED2 ResNet AUC falls from 0.8462 to 0.8434, and ED4→ED4 Xception AUC falls from 0.9415 to 0.9386), and the convergence theory in Section 5 is proven for a smooth convex objective, not for the actual IADA network. A revised version with a nested target-validation protocol, per-seed error bars, and honest reporting of Table 1 would be needed before the headline claims can be evaluated.","tokens_in":13769,"tokens_out":6892,"duration_ms":69994,"concrete_test":"Re-run the full protocol with an explicit target validation split. For each setting, hold out a portion of the target images as validation data, select λreg and λadv on that validation set only, then evaluate on the held-out target test portion. Repeat for all three target domains and all three backbones, and also report the five-seed mean and coefficient of variation promised in Section 6.1. If the ED4→ED3 ResNet-50 accuracy gap of 25.19 percentage points drops below roughly 10 points, or overlaps the baseline within error bars, the headline claim is an artifact of target-set tuning rather than a genuine generalization result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—25.19% accuracy gain and 12.56% AUC gain on low-quality target domains—requires that the target-domain test sets were not used for model selection. The paper as written violates this. Section 6.1 states that the key hyperparameters λreg and λadv are 'selected according to their performance in a line search', with 'More details can be found in Section 6.3.' Section 6.3 (Fig. 2) then plots target-scenario AUC as a function of λreg and λadv for ED4→ED3, ED4→ED2, and ED4→ED1. These are the same target test sets that Section 6.1 says are 'reserved entirely for testing.' Tuning on those AUC curves means the Table 1 numbers are selected on the evaluation data, not independent measurements of generalization. This is not a minor detail: the headline ED4→ED3 ResNet-50 gain is 0.9457 versus 0.6938, an absolute 65-image difference on a 258-image test set, and the ablation curves show AUC swings of 0.3–0.5 across the searched ranges. The claimed advantage can therefore be explained by picking the favorable λ per setting rather than by the method itself. The promised five-seed means and coefficients of variation from Section 6.1 are never reported, so there is no variance estimate to bound this selection bias. Even if the point estimates are honest, the absence of any held-out target validation protocol breaks the paper's central empirical claim as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes Imbalance-Aware Domain Adaptation (IADA), a framework for embryo-development image classification that combines class-specific attention, class-weighted adversarial alignment, and adaptive decision thresholds. It presents a theoretical analysis with generalization, convergence, and complexity bounds, and evaluates the method on one source domain (ED4, clinical time-lapse) and three target domains (ED3, ED2, ED1) of decreasing image quality, comparing against MD-Net with ResNet-50, Inception v3, and Xception. The abstract claims accuracy gains up to 25.19% and AUC gains up to 12.56%. The central empirical claim is not supported as reported because the key hyperparameters are selected on the target test sets, the promised five-seed variance statistics are absent, and some table entries contradict the abstract's blanket AUC claim.","tokens_in":14078,"tokens_out":11362,"duration_ms":101027,"significance":"If the claimed results survived a sound evaluation, IADA would be a practically useful contribution to domain adaptation under class imbalance in medical imaging, and the public code availability would aid reproducibility. The paper also attempts a formal treatment of class-imbalance-aware generalization and convergence, which is a meaningful direction. However, the current significance is limited by a test-set selection protocol that invalidates the headline numbers, by missing variance estimates on very small target test sets, and by a theoretical analysis that is not connected to the experiments in a predictive way. The positive results in some settings suggest the method may have merit, but the present evidence is insufficient to support the paper's central claims.","major_comments":[{"comment":"Section 6.1 states that the target-domain datasets are 'reserved entirely for testing,' but the same section says λreg and λadv are chosen by a line search and refers to Section 6.3. Section 6.3 (Fig. 2) plots AUC on ED4→ED3, ED4→ED2, and ED4→ED1 as functions of λreg and λadv; these are the same target test sets used in Table 1. Selecting hyperparameters from these curves makes the Table 1 results post-selection estimates rather than independent generalization measurements. This is load-bearing because the headline 25.19% accuracy gain on ED4→ED3 with ResNet-50 (0.9457 vs. 0.6938) and the large ablation swings in Fig. 2 could be produced by picking a favorable λ per setting. The experiments should be redone with hyperparameters selected on a source-only validation set or a separate target validation set, and the exact selection rule must be reported.","section":"§6.1, §6.3, Table 1, Fig. 2"},{"comment":"The text promises that all experiments are repeated with five different random seeds and that mean performance and coefficient of variation are reported, but Table 1 contains only single point estimates. With ED2 containing 69 images and ED3 containing 258 images, the 25.19 percentage-point accuracy difference corresponds to roughly 65 images and the ED4→ED2 differences to only a few images, so seed-to-seed variability could be decisive. The authors should report the five-seed means and standard deviations or CVs, and ideally confidence intervals, before any comparative claim can be assessed.","section":"§6.1, Table 1"},{"comment":"The blanket claim of AUC improvement is contradicted by the table. For Inception v3, IADA has AUC 0.3750 versus MD-Net's 0.6442 on ED4→ED2 and 0.4543 versus 0.6025 on ED4→ED1, and Xception on ED4→ED4 is also slightly below MD-Net (0.9386 vs. 0.9415). The 'up to 12.56%' AUC improvement is therefore a selected favorable case rather than a verified general result. The abstract should be qualified by architecture and setting, or the authors should provide a statistically valid aggregated comparison.","section":"Abstract, Table 1"},{"comment":"The theoretical results do not provide a predictive account of the experiments. Section 6.3 explains the ablation curves using d_i(H) from Theorem 5.1 and the gradient bound G from Theorem 5.7, but neither quantity is measured, and Theorem 5.7's Assumption 5.5 (β-smoothness and μ-strong convexity) is not satisfied by the deep networks used in the experiments. The claim that a high λadv degrades performance 'because G increases' is therefore not tested. The authors should either measure the relevant quantities, derive a falsifiable prediction from the bounds, or explicitly present the theory as independent optimization-theoretic guarantees rather than as an explanation of the experimental behavior.","section":"§5, §6.3"},{"comment":"Theorem 5.15 is mathematically incorrect as stated. Minimizing Σ_i 1/b_i subject to Σ_i b_i = B gives all batch sizes equal, b_i = B/C, not the square-root dependence on min(πs_i, πt_i) claimed in the theorem. The Lagrangian derivation in the proof does not yield the stated formula, so the theorem and its proof need to be corrected or the optimization problem restated.","section":"§5, Theorem 5.15"}],"minor_comments":[{"comment":"The phrase 'The code is made public available' should be 'The code is made publicly available.'","section":"Abstract"},{"comment":"The caption labels both panels as '(a)' and says both show λreg; the second panel is discussed in the text as varying λadv and should be labeled '(b)' accordingly.","section":"Fig. 2"},{"comment":"Several references are duplicated: [8] and [21] are the same Rubio et al. paper, [5] and [22] are the same Dimitriadis et al. paper, and [9] and [23] are the same Zaninovic and Rosenwaks paper; these should be merged.","section":"References"},{"comment":"Reference [38] cites a flower-classification tutorial rather than the original Inception-v3 architecture paper by Szegedy et al.; please cite the primary source.","section":"References"},{"comment":"Equation (12) introduces λ1, λ2, λ3 and the terms Lcons and Ldiv without defining them or connecting them to the λreg reported in Section 6.1; please specify the objective actually implemented.","section":"§4.4, §6.1"},{"comment":"Only MD-Net is compared in Table 1; given the abstract's reference to 'existing methods' in the plural, at least one additional standard domain-adaptation or class-imbalance baseline should be included for a meaningful comparison.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The main barrier to publication is the evaluation protocol: the headline experimental claims are unsupported because hyperparameters are selected on the target test sets and because the promised variance statistics are absent. I recommend that the editor require the authors to redo the experiments with a properly isolated validation set and report seed-level results before any acceptance decision. If such re-analysis is not possible, the paper should be rejected. The reference list also contains a noticeable number of author self-citations; this is not by itself a problem, but it may be worth checking that they are all necessary."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a plausible engineering combination—class-specific attention, class-weighted adversarial alignment, adaptive thresholds—applied to embryo staging across four imaging modalities. That part is fine and the problem is real. But the central empirical claim is not supported as stated. The hyperparameters λreg and λadv are selected by line search on the same ED3/ED2/ED1 target sets that Section 6.1 calls 'reserved entirely for testing.' Section 6.3 plots AUC on those exact target scenarios against λ values. So the Table 1 numbers are cherry-picked per setting, not honest generalization measures. The promised five-seed means and %CV never appear. And the abstract's 'AUC improvements up to 12.56%' is contradicted by Table 1: Inception v3 on ED4→ED2 AUC 0.375 vs 0.644 for the baseline, and on ED4→ED1 0.454 vs 0.603. That's not a minor typo—it is the opposite of 'robust generalization.'\n\nThe theory is standard: Theorem 5.1 is a class-weighted domain adaptation bound, and Theorem 5.7 is the usual smooth-strongly-convex convergence rate. The proof of Theorem 5.1 has a sketchy step where |πt εt − πs εs| becomes min(πs, πt)d(H) without careful handling, but it's recognizably in the Ben-David lineage. It is not connected to the actual IADA objective (attention, adversarial training, adaptive thresholds), so it functions as narrative rather than prediction.\n\nWhat the paper does well: it follows the MD-Net protocol, reports on four imaging modalities with a real clinical problem, and the class-imbalance-aware attention idea is sensible. With a proper held-out validation split, error bars, and corrected abstract claims, this could be a valid extension paper for the embryo-AI community.\n\nWho is this for? Researchers working on domain adaptation for embryo or other medical imaging who want a recipe that combines known ingredients. It deserves a serious referee—not because the current results are trustworthy, but because the flaws are fixable and the underlying idea is reasonable.","headline":"Useful engineering combination for embryo domain adaptation, but the empirical protocol is broken: target test sets were used for hyperparameter selection, and the abstract's AUC claim is contradicted by Table 1.","tokens_in":14609,"tokens_out":2213,"would_cite":false,"duration_ms":20580,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims a single training objective can correct both image-quality shifts and class imbalance, improving embryo-stage classification by up to 25.19 percentage points over a strong adversarial baseline.","keywords":["domain shift","class imbalance","domain adaptation","embryo development assessment","medical imaging","adversarial training","adaptive threshold","class-specific attention"],"falsifier":"Re-run the ED4→ED3 transfer with ResNet-50 for five seeds and compute each seed's accuracy: if the spread of IADA accuracies overlaps the MD-Net range, or if the standard error of the 0.9457 − 0.6938 = 0.2519 difference exceeds roughly 0.05, the headline improvement is not statistically distinguishable.","tokens_in":13516,"feed_emoji":"🔬","tokens_out":5752,"duration_ms":53371,"temperature":0.7,"pith_summary":"The paper tries to show that the two obstacles to deploying embryo-assessment models—domain shift (moving from clinical time-lapse systems to cheaper microscopes) and class imbalance (blastocyst versus non-blastocyst proportions varying from 28.9% to 81.2%)—are best solved together, not separately. It proposes IADA, which couples class-specific attention in the feature extractor, class-weighted adversarial domain alignment, and adaptive classification thresholds, and proves a generalization bound that splits the target error into class-proportion mismatch plus class-weighted domain discrepancy. On four imaging modalities, IADA reports accuracy gains up to 25.19 percentage points and AUC gains up to 12.56 percentage points over the MD-Net baseline, with balanced precision and recall. A sympathetic reader would care because if the claim holds, inexpensive portable and smartphone microscopes could produce embryo assessments close to clinical time-lapse quality, with minority classes not sacrificed.","feed_headline":"Imbalance-aware adaptation lifts embryo grading accuracy by up to 25%","feed_subtitle":"The method keeps minority-class performance high while adapting embryo images across four microscope types.","key_machinery":"The load-bearing object is the three-part IADA training objective (10): classification with weighted focal loss, adversarial domain alignment with class-specific instance weights, and regularization. The mechanism that carries the argument is the coupling of three formulas: the class-attention feature combination $z_i = \\sum_c \\alpha_c(x_i) f_c(x_i)$ (Eq. 3), the class-weighted adversarial loss $L_{adv}$ (Eqs. 4–6), and the adaptive threshold decision $\\hat{y} = \\arg\\max_c (C_\\psi(z)_c - \\tau_c)$ (Eq. 8), with $\\tau_c = \\beta \\log(n^s_c / \\min_k n^s_k) + \\gamma$. Theorem 5.1 is the identity that ties the story together: target error is bounded by source error plus the $\\ell^1$ difference of class proportions plus class-minimum-weighted domain discrepancies plus ideal joint error, so imbalance and shift are not separable nuisances but additive terms in the same bound.","core_discovery":"The central claim is that a domain-adaptation framework can be made imbalance-aware by construction. IADA's feature extractor computes per-class feature adapters and attends over them, its discriminator is trained with class-balanced sampling and instance weights inverse to source class frequency, and its classifier subtracts learnable class-dependent thresholds before the argmax. The paper derives a generalization bound $\\epsilon_t(h) \\leq \\epsilon_s(h) + \\sum_i |\\pi^s_i-\\pi^t_i| + \\sum_i \\min(\\pi^s_i,\\pi^t_i)d_i(H) + \\lambda$, arguing that both class-proportion shifts and domain discrepancy contribute additively to target error, and gives convergence and complexity results for the training objective. Experiments compare IADA to MD-Net on ResNet-50, Inception v3, and Xception across transfers ED4→ED4, ED4→ED3, ED4→ED2, and ED4→ED1; the largest reported win is ED4→ED3 with ResNet-50, where accuracy rises from 0.6938 to 0.9457.","pith_inferences":["The same additive-bound logic suggests a stress test beyond embryo images: in any clinically imbalanced classification task, the framework should show its largest gains exactly when source and target class proportions diverge most; if not, the mechanism's specific contribution is weaker than claimed.","The paper does not use target-domain labels, so IADA's adaptive thresholds can only be tuned to source frequencies; an implied extension is to replace the source-frequency threshold with an online estimate of target proportions, which the bound suggests would further reduce error.","Because the target test sets are very small (69 and 258 images), the advertised percentage-point differences should be read as provisional; a direct extension is to pool or enlarge the target sets and check whether the 12.56-point AUC improvement survives."],"forward_implications":["If the reported numbers hold, models trained on one clinical time-lapse system can be transferred to portable and smartphone microscopes while keeping accuracy above 0.84 in most tested settings, instead of degrading toward random-level performance.","Minority-class performance in low-quality domains improves: in ED4→ED2 with Xception, IADA reaches perfect recall while holding precision near 0.89, which matters when non-blastocyst embryos are the rare class.","The generalization bound implies that domain adaptation alone cannot erase the cost of label shift; deployments must also measure or estimate target class proportions, since $\\sum_i |\\pi^s_i-\\pi^t_i|$ appears additively.","Because the balanced-domain corollary removes the proportion-difference term, the framework predicts that deliberately balancing source data or reweighting it toward target proportions should shrink the domain gap, a testable design rule."],"supporting_citations":[{"why":"Supplies the MD-Net baseline, the embryo dataset split across ED1–ED4, and the experimental protocol the paper follows.","marker":"[4]"},{"why":"Provides the gradient-reversal adversarial mechanism that IADA's balanced domain alignment builds on.","marker":"[3]"},{"why":"ResNet-50 backbone used as the primary architecture and ablation base.","marker":"[37]"},{"why":"Inception v3 backbone used to test multi-scale feature extraction with adaptive thresholds.","marker":"[38]"},{"why":"Xception backbone used to test depthwise-separable convnets with class-weighted attention.","marker":"[39]"},{"why":"SMOTE oversampling baseline for imbalanced learning that motivates the class-balance component.","marker":"[6]"},{"why":"ADASYN, an adaptive synthetic sampling method, included as an imbalance-handling comparison point.","marker":"[12]"}],"fun_headline_variants":["Imbalance-aware domain adaptation boosts embryo grading up to 25%","New method tackles domain shift and data imbalance in embryo AI","IADA improves embryo grading accuracy by 25% across modalities","Domain-shift fix for embryo imaging: accuracy up 25%","Embryo grading AI: imbalance-aware adaptation gains 25%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The performance claims stand on the assumption that the unreported seed-to-seed variability of Table 1 is small, because the section promises five random seeds and coefficient of variation yet reports single numbers on tiny target sets (69 and 258 images).","fun_headline_variants_meta":{"raw":{"variants":["Imbalance-aware domain adaptation boosts embryo grading up to 25%","New method tackles domain shift and data imbalance in embryo AI","IADA improves embryo grading accuracy by 25% across modalities","Domain-shift fix for embryo imaging: accuracy up 25%","Embryo grading AI: imbalance-aware adaptation gains 25%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000221,"raw_usage":{"total_tokens":1457,"prompt_tokens":961,"completion_tokens":496,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":408}},"tokens_in":577,"tokens_out":496,"duration_ms":4729,"temperature":1.0,"reasoning_tokens":408,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:21:04.260433+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the ED4→ED3 transfer with ResNet-50 for five seeds and compute each seed's accuracy: if the spread of IADA accuracies overlaps the MD-Net range, or if the standard error of the 0.9457 − 0.6938 = 0.2519 difference exceeds roughly 0.05, the headline improvement is not statistically distinguishable.","supporting_citations":[{"cited_title":"Adaptive adversarial neural networks for the analysis of lossy and domain-shifted datasets of medical images,","cited_arxiv_id":null,"evidence_quote":"Supplies the MD-Net baseline, the embryo dataset split across ED1–ED4, and the experimental protocol the paper follows."},{"cited_title":"Domain-adversarial training of neural networks,","cited_arxiv_id":null,"evidence_quote":"Provides the gradient-reversal adversarial mechanism that IADA's balanced domain alignment builds on."},{"cited_title":"Learning from class-imbalanced data: Review of methods and applications,","cited_arxiv_id":null,"evidence_quote":"ResNet-50 backbone used as the primary architecture and ablation base."},{"cited_title":"Inception-v3 for flower classification,","cited_arxiv_id":null,"evidence_quote":"Xception backbone used to test depthwise-separable convnets with class-weighted attention."},{"cited_title":"Smote: Synthetic minority over-sampling technique,","cited_arxiv_id":null,"evidence_quote":"SMOTE oversampling baseline for imbalanced learning that motivates the class-balance component."},{"cited_title":"Adasyn: Adaptive synthetic sampling approach for imbalanced learning,","cited_arxiv_id":null,"evidence_quote":"ADASYN, an adaptive synthetic sampling method, included as an imbalance-handling comparison point."}],"review_version":1}