{"id":"4a5f3553-e0fd-45d3-8462-fd85d579e4c7","arxiv_id":"2608.10437","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MammoMix combines three domain-specialized YOLOS detectors with gating and confidence calibration, improving mammogram lesion detection on heterogeneous datasets compared to one model trained on all data.","lead":"MammoMix trains separate breast-lesion detectors on three mammography datasets and fuses their outputs with a learned gate and a calibration step. It reports better average precision than a single combined model on the two more diverse datasets, at the cost of higher compute.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The integrated MammoMix system is never evaluated and reliability is never measured; the central claim is unsupported by the reported experiments.","rationale":"The reader's weakest_assumption focuses on two component-level assumptions: that the CNN gate can visually separate domains, and that assigning IoU=0 to out-of-domain validation images is an accurate model of out-of-domain performance. Both are genuine concerns, but the more load-bearing problem is upstream: no experiment runs the full MammoMix pipeline with gate, experts, calibration, and fusion all active, and no reliability metric is measured. The reader's rationale does mention that the integrated MammoMix model is never run and calibration error is never measured, so there is partial overlap, but the formal weakest_assumption field targets different premises. The identical rows in Table 2 are best read not as evidence of fabrication but as a symptom that Simple MoE is hard routing rather than the adaptive ensemble promised in the abstract; this supports the same conclusion that the claimed system is absent. Because the central claim cannot be assessed from the submitted material, the reader's REJECT verdict remains appropriate. A conditional accept would require the missing end-to-end experiment and calibration metrics before the claim could be evaluated, so no verdict change is recommended.","tokens_in":10891,"tokens_out":6025,"duration_ms":54231,"concrete_test":"Inspect the public repository (https://github.com/tommyngx/MammoMix) for an inference configuration that chains all four components: the CNN gate, the three YOLOS experts, the three Random-Forest calibrators, and Soft NMS plus Score Voting, as in Figure 1. Run that configuration on the CSAW, DDSM, and DMID test splits from Table 1 and report mAP@50-95 and expected calibration error for each dataset. If no such configuration exists, or if calibration error is not computed, then the manuscript's claim that the integrated MammoMix framework improves both accuracy and reliability is not supported by the experiments.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and conclusion claim that MammoMix—expert specialization plus gating plus calibrated ensemble fusion—outperforms baselines in both accuracy and reliability. To support that claim, the paper would need to evaluate the full pipeline described in Sections 4.2–4.4 and Figure 1: three YOLOS experts, a per-image CNN gate, the three Random-Forest calibrators, and Soft NMS with Score Voting, all operating together. No such integrated model is reported. Simple MoE (Section 4.3) is hard selection of a single expert via argmax, not adaptive weighting or fusion, and MoCAE (Section 4.4) runs all three experts without the CNN gate, fusing calibrated outputs via Refining NMS. The two components are never combined, so there is no experimental result for the actual MammoMix framework. Furthermore, reliability is claimed throughout but no calibration metric—expected calibration error, reliability diagram, or Brier score—appears anywhere; mAP is an accuracy measure, not a reliability measure. The near-identical values between 'Individually' and 'Simple MoE' in Table 2 are plausibly explained by the hard gate routing each test image to its dataset-specific expert, so they do not themselves indicate copying, but they reinforce that Simple MoE is not the claimed adaptive ensemble. The decisive gap is that the central claim describes a system that was not run.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MammoMix, a Mixture-of-Experts framework for mammogram lesion detection. It trains three YOLOS experts on three datasets (CSAW, DDSM, DMID), a CNN gate intended to route each image to the appropriate expert, and a MoCAE calibration module that fits Random Forest regressors to map confidence scores to expected IoU, followed by Soft NMS and Score Voting fusion. The experiments compare a monolithic combined-training model (MoMo), a simple hard-routing MoE (Simple MoE), and the calibration-based ensemble (MoCAE) against DETR and RT-DETRv2 baselines on the three datasets, reporting mAP at multiple IoU thresholds. The paper claims that MammoMix improves both detection accuracy and reliability across heterogeneous datasets.","tokens_in":11064,"tokens_out":5043,"duration_ms":48134,"significance":"If the integrated MammoMix system performed as claimed, it would be a useful contribution to multi-domain medical object detection, where per-domain experts, learned routing, and calibrated fusion could improve generalization across imaging protocols. The paper uses three public mammography datasets, describes the components in reasonable detail, and makes code available. However, the significance is undercut by the fact that the full MammoMix pipeline—experts, CNN gate, calibrators, and fusion—is never evaluated as a single system, and no calibration or reliability metric is reported. The numerical results in Table 2 also contain unexplained duplicates between the 'Individually' and 'Simple MoE' rows, which raises concerns about the reliability of the reported experiments. The central claims of the paper are therefore not supported by the evidence presented.","major_comments":[{"comment":"The integrated MammoMix system described in the abstract and Figure 1—three experts, a CNN gate, three Random Forest calibrators, Soft NMS, and Score Voting operating together—is never evaluated. Table 2 reports results only for MoMo, Simple MoE, and MoCAE. Simple MoE (Section 4.3) performs hard argmax selection of one expert, while MoCAE (Section 4.4) runs all experts but does not use the CNN gate. No experiment combines the gate with calibrated fusion, so the manuscript provides no experimental result for the actual MammoMix framework. This is a load-bearing omission because the central claim that MammoMix outperforms baselines in accuracy is unsupported by the reported experiments.","section":"Section 6, Table 2; Abstract"},{"comment":"The abstract and conclusion claim that MammoMix improves both average precision and reliability, but no calibration or reliability metric appears anywhere in the paper. The reported metric is mAP at various IoU thresholds, which is an accuracy measure. Expected calibration error, reliability diagrams, Brier score, or any direct assessment of whether confidence scores reflect true precision are absent. The statement in Section 6 that MoCAE 'improves model reliability' is therefore an unsupported claim.","section":"Section 6; Abstract"},{"comment":"The rows labeled 'Individually' and 'Simple MoE' are numerically identical on CSAW (0.3292, 0.7185, 0.2106, 0.3210 for mAP50-95, mAP50, small, medium) and on DMID (0.3364, 0.6249, 0.0000 for those metrics). If Simple MoE is a hard gate that selects the dataset-specific expert, then on per-dataset test splits the output would equal the individual expert only if the gate classifies the dataset origin with perfect accuracy on the test set. Any realistic gate error would produce small numerical differences, as seen on DDSM. The exact duplicates therefore imply either an implausible oracle gate or duplication of table entries; this needs a detailed explanation and corrected results.","section":"Table 2; Section 4.3"},{"comment":"The calibration regressors assign an IoU of 0 to every out-of-domain validation image when constructing the training set, regardless of whether the expert actually produces a correct detection on that image. This modeling assumption forces the calibrator to output low calibrated scores for any expert on images from other datasets, even when the expert detects a lesion correctly. In a mixed-domain test set with real out-of-domain positives, this systematically suppresses true positives from the minority expert and biases MoCAE's fusion. The paper provides no evidence that this assumption holds, and it directly affects the validity of the MoCAE results.","section":"Section 4.4"}],"minor_comments":[{"comment":"The caption contains the typo 'across dataset'; it should read 'across datasets'.","section":"Figure 2 caption"},{"comment":"The acronym is spelled inconsistently as both 'MoCaE' and 'MoCAE' (e.g., Section 4.4 header and Section 6 text). Please standardize.","section":"Throughout"},{"comment":"The header line 'Subset mAP DETR baseline YOLOS' is confusing; the columns should be clearly grouped by dataset and model variant.","section":"Table 2"},{"comment":"The sentence 'DETR baseline outperformed all other approaches in terms of mAP@50-95 (0.1995)' is correct per Table 2, but the later sentence 'MoE variants remained competitive' understates that MoCAE is substantially worse (0.1357) on this dataset; the discussion could be more balanced.","section":"Section 6, DDSM paragraph"},{"comment":"The notation for the calibrated score uses both ˜s and ˜s^e_i; please define once and use consistently.","section":"Section 4.4"}],"recommendation":"reject","confidential_remarks":"The exact duplicate values between 'Individually' and 'Simple MoE' in Table 2 are a serious integrity concern; the authors should be asked to verify the raw results and explain the routing behavior on test data. The paper also oversells the integrated MammoMix system, which is never run as a single pipeline, and the reliability claim is unsupported by any calibration metric. These issues go beyond presentation and would require substantial new experiments and a re-analysis of the existing results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is an honest engineering paper with a real, if modest, new combination—three per-domain YOLOS experts, a CNN gate that picks a domain expert, and Random Forest calibration with soft NMS fusion. The writing is serviceable and the authors are upfront about costs and limitations. What it is not is a demonstration of MammoMix as advertised. The abstract and conclusion claim the integrated gating-plus-calibration-plus-fusion system beats baselines in accuracy and reliability, but the experiments only run the pieces separately: Simple MoE is hard argmax selection, MoCAE runs all experts without the gate. The full pipeline in Figure 1 is never evaluated. And reliability is claimed without a single calibration metric—no ECE, no reliability diagram, no Brier score; mAP is not a reliability measure.\n\nThe most damaging symptom is Table 2: 'Individually' and 'Simple MoE' have identical numbers on CSAW and DMID. The stress-test note is right that this is consistent with the gate always routing each test image to its own dataset expert, so the rows are not necessarily faked. But it does confirm that Simple MoE is not an adaptive ensemble; it is the same as oracle dataset selection, which is a much weaker claim than 'MammoMix adaptively weighs contributions.'\n\nThe out-of-domain IoU=0 trick in Section 4.4 is also a soft spot. Assigning zero IoU to all validation images from other datasets teaches the calibrator that cross-domain detections are worthless. That may be roughly true for these particular dataset pairs, but treating it as ground truth biases the calibrated scores down for any expert that could actually contribute out-of-domain, and the paper never validates the assumption.\n\nThere is some genuine value here. The domain-specialization-plus-calibration combination for mammography is a new application, the ablation—MoMo vs Simple MoE vs MoCAE—is a sensible way to isolate the pieces, and the authors note the computational and interpretability caveats. The citation pattern looks fine; MoCaE [13] is the right prior work to build on. The math is simple enough that there is no deep technical error hidden in the equations.\n\nWho is this for? Someone looking for a quick survey of how MoE components behave on public mammography datasets. It is not ready for a serious referee as a claims paper, because the central claim is unverifiable from the submitted experiments. A conditional reject with a clear path—run the full MammoMix, add ECE/reliability plots, fix the table ambiguity, and validate or drop the IoU=0 assignment—would be my call. That is a real revision, not a desk rejection.","headline":"A reasonable MoE-for-mammography engineering idea whose headline claim is unsupported because the full MammoMix system was never run and reliability was never measured.","tokens_in":11743,"tokens_out":1984,"would_cite":false,"duration_ms":18951,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MammoMix claims that training separate detectors on each mammography dataset and fusing their outputs through a learned gate plus calibrated confidence gives more accurate and reliable lesion detection than a single model trained on the…","keywords":["Mixture-of-Experts","Mammography","Lesion detection","YOLOS","Confidence calibration","Domain generalization","Object detection","Breast cancer screening"],"falsifier":"Measure the gating network's accuracy at predicting dataset origin on held-out validation images; if it is near chance, or if out-of-domain detections are not uniformly zero-IoU in a representative mixed test set, the routing and calibration premises fail.","tokens_in":10561,"feed_emoji":"🩻","tokens_out":4520,"duration_ms":40063,"temperature":0.7,"pith_summary":"This paper is trying to show that when mammography images come from several different sources, a single object detector trained on all of them together loses accuracy, while a mixture of experts—one detector per source—can do better. The authors build MammoMix, which pairs a per-source YOLOS detector with a gating network that decides which expert to trust, and a calibration module that rescales each expert's confidence to match its true precision before the boxes are fused. The result, they argue, is improved average precision and more reliable confidence scores on datasets with high variability, like CSAW and DMID. On the more homogeneous DDSM set, a strong single detector remains competitive, so the benefit is conditional on how heterogeneous the data are.","feed_headline":"Specialist detectors beat one-size-fits-all on mammograms","feed_subtitle":"Training one expert per mammography source and calibrating their scores improves detection where image data vary most.","key_machinery":"The machinery is three domain-specialized YOLOS detectors, one trained on CSAW, one on DDSM, and one on DMID, where YOLOS is a Vision Transformer object detector that treats image patches as tokens with learnable detection tokens. A separate CNN gating network classifies an input image's dataset origin and routes it to the corresponding expert, while MoCAE calibrates each expert's confidence by training a Random Forest regressor that maps a ResNet-18 image embedding plus raw confidence score to an expected IoU value, explicitly assigning zero IoU to out-of-domain validation images. The calibrated predictions from all experts are then fused through Soft NMS and Score Voting, which together consolidate overlapping detections into refined boxes.","core_discovery":"The central discovery is that expert specialization plus calibrated ensemble fusion, implemented as MammoMix, improves breast lesion detection over a monolithic model (MoMo) trained on combined data. On CSAW, Simple MoE reaches mAP@50-95 of 0.3287 versus MoMo's 0.2458; on DMID, Simple MoE reaches 0.3348 versus MoMo's 0.2660. The calibration-augmented variant MoCAE achieves the best small-lesion AP on CSAW (0.3119) and the best large-lesion AP on DMID (0.4910). On DDSM, however, the best MammoMix variant only matches MoMo (0.1973 vs. 0.1710) and DETR outranks all MammoMix variants (0.1995).","pith_inferences":["A stronger test of MammoMix would require a held-out dataset that looks visually similar to one training domain but contains different lesion distributions; if the gate routes by appearance while the expert was trained on different pathology, the system could be confidently wrong.","The practice of assigning IoU=0 to out-of-domain validation images quietly encodes the assumption that cross-domain detections are worthless, which could depress calibrated scores below their true value in real deployments with overlapping domains.","The same mixture-of-calibrated-experts recipe could transfer to other multi-center medical imaging tasks, such as chest X-ray or retinal imaging, provided each site's annotations are available for training and calibration.","The gains from calibration on small lesions suggest that miscalibration is a major failure mode specifically for small-object detection, a direction worth isolating in future studies."],"forward_implications":["On diverse or small datasets such as CSAW and DMID, domain-specialized experts with gating outperform a single YOLOS trained on the combined data, giving a recipe for multi-source medical detection without monolithic retraining.","Per-expert Random Forest calibration that maps image embeddings and raw confidence to expected IoU yields confidence scores that better mirror true detection quality, improving small-lesion detection on CSAW.","Fusion through Soft NMS and Score Voting consolidates partial detections from multiple experts, improving localization over any single expert.","For more homogeneous datasets, a strong single detector (such as DETR on DDSM) can still match or beat the MoE approach, so the advantage of MammoMix is conditional on data heterogeneity."],"supporting_citations":[{"why":"Supplies the YOLOS Transformer detector used as the backbone for every expert and for the monolithic baseline.","marker":"[4]"},{"why":"The MoCaE mixture-of-calibrated-experts approach supplies the calibration and refining NMS fusion strategy that the paper adapts.","marker":"[13]"},{"why":"Provides the CSAW-CC dataset, one of the three training domains and a key evaluation set.","marker":"[2]"},{"why":"Provides the CBIS-DDSM dataset, the largest training domain and evaluation set.","marker":"[10]"},{"why":"Provides the DMID dataset, the smallest and most variable domain that motivates the need for expert specialization.","marker":"[14]"},{"why":"Establishes the Mixture-of-Experts paradigm that the framework is built on.","marker":"[12]"},{"why":"Defines the DETR baseline that is compared against MammoMix in the experiments.","marker":"[1]"},{"why":"Documents multi-center evidence that a single model degrades on heterogeneous mammography data, motivating the problem.","marker":"[5]"}],"fun_headline_variants":["Mixture of experts boosts mammogram lesion detection","Specialists beat generalist on mammogram detection","Calibrated expert ensemble improves breast detection","Expert specialization sharpens mammogram screening","Domain-specialized experts beat single detector on mammograms"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The system depends on the gating network being able to tell which dataset an image comes from just by looking at it; if mammograms from different centers are not visually distinguishable, routing fails and the mixture collapses into a plain ensemble.","fun_headline_variants_meta":{"raw":{"variants":["Mixture of experts boosts mammogram lesion detection","Specialists beat generalist on mammogram detection","Calibrated expert ensemble improves breast detection","Expert specialization sharpens mammogram screening","Domain-specialized experts beat single detector on mammograms"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002016,"raw_usage":{"total_tokens":7863,"prompt_tokens":952,"completion_tokens":6911,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":6841}},"tokens_in":568,"tokens_out":6911,"duration_ms":42971,"temperature":1.0,"reasoning_tokens":6841,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:19:32.260204+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the gating network's accuracy at predicting dataset origin on held-out validation images; if it is near chance, or if out-of-domain detections are not uniformly zero-IoU in a representative mixed test set, the routing and calibration premises fail.","supporting_citations":[{"cited_title":"Radiology294(2), 265–272 (2020)","cited_arxiv_id":null,"evidence_quote":"Provides the CSAW-CC dataset, one of the three training domains and a key evaluation set."},{"cited_title":"Biomedical Engineering Letters14(2), 317–330 (2024)","cited_arxiv_id":null,"evidence_quote":"Provides the DMID dataset, the smallest and most variable domain that motivates the need for expert specialization."},{"cited_title":"In: European conference on computer vision","cited_arxiv_id":null,"evidence_quote":"Defines the DETR baseline that is compared against MammoMix in the experiments."},{"cited_title":"Artificial Intelligence in Medicine132, 102386 (2022)","cited_arxiv_id":null,"evidence_quote":"Documents multi-center evidence that a single model degrades on heterogeneous mammography data, motivating the problem."}],"review_version":1}