{"id":"f4aab0ad-01d2-4c59-b828-bb2dc38a031d","arxiv_id":"2412.19412","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A synthetic data engine built on generative models lets a single fine-tuned matcher outperform modality-specific methods across many real cross-modal image matching tasks.","lead":"This paper generates a huge synthetic dataset of fake infrared, depth, event, and stylized images from ordinary RGB photos, then uses it to train image-matching models that work across different camera types. The trained models beat specialized methods on many real-world cross-modal matching benchmarks, especially for visible-infrared and RGB-depth pairs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Event modality generation uses a random per-image motion to create events, so the inherited MegaDepth correspondences are not geometrically valid for RGB-Event pairs; this undermines the claimed 'accurate labels' for all modalities and the 19-case evaluation.","rationale":"The reader's weakest_assumption already flags the event simulation's random slight motion as a potential break of geometric consistency, and my analysis confirms this is the most load-bearing internal flaw. The central claim requires that all generated modalities inherit MegaDepth labels exactly; for events this is false by construction, since the random motion field is independent of the epipolar geometry that defines the labels. This is not merely an 'outside consensus' concern but an internal inconsistency in the data generation pipeline. The RGB-Event results in Tab. 2 and Tab. 5, and the count of '19 cross-modal cases,' therefore rest on invalid ground truth for one modality. However, the core contribution for RGB-IR, RGB-Depth, and RGB-Normal is supported by the ablations (Tab. 6) and by zero-shot results on real datasets, so the paper need not be rejected outright. The remedy is to either regenerate events consistently with the known pose/depth or to remove RGB-Event claims and revise the 'any modality' language. The reader's CONDITIONAL verdict remains appropriate; my concrete test would settle whether the event generation is actually broken, and if so, which claims must be narrowed.","tokens_in":31580,"tokens_out":7889,"duration_ms":75234,"concrete_test":"Regenerate the Event modality in MD-syn using the actual MegaDepth relative motion: for each RGB pair, warp the source RGB with the known depth map and relative pose to synthesize a short frame sequence, then run the event simulator of [15] to produce the event image, so that the event image is in the same coordinate frame as the RGB view. Fine-tune and evaluate MINIMA on RGB-Event exactly as in Tab. 2 and Tab. 5. If the RGB-Event AUC changes materially (e.g., the synthetic Tab. 2 numbers drop or the DSEC zero-shot gains disappear), the current 'random slight motion' generation breaks label validity, and the '19 cross-modal cases' claim must be revised. As a cheaper diagnostic, compute Sampson distances for a sample of labeled RGB-Event correspondences using the inherited MegaDepth pose; a median error far above the RGB-RGB baseline confirms the offset.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing premise is that every pseudo-modality in MD-syn inherits the MegaDepth matching labels exactly (Sec. 3.1). This is internally violated for the Event modality in Sec. 3.3: after Eq. (1), events are computed by giving 'a random slight motion' to the source RGB image. An event image generated this way is a temporal integral of brightness changes under an arbitrary motion field, not a rendering of the scene from the MegaDepth camera pose. A pixel that fires an event at location x corresponds to a scene point whose projection swept through x during the injected motion; it does not correspond to the RGB pixel at x unless the motion is zero. Therefore the inherited pose/depth labels for RGB-Event pairs are systematically offset, the synthetic RGB-Event AUC in Tab. 2 is not a valid accuracy measure, and the '19 cross-modal cases' claim is inflated. Sec. 5.6 asserts the real-pseudo gap has 'little impact' but does not address this internal geometric inconsistency.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MINIMA, a unified cross-modal image matching framework built on a synthetic data engine. Starting from MegaDepth RGB pairs with known depth/pose, the authors generate pseudo-modalities (Infrared, Depth, Event, Normal, and two artistic styles) using off-the-shelf or fine-tuned generative models, producing MD-syn with over 480M cross-modal pairs. They then fine-tune existing matchers (LightGlue, LoFTR, RoMa) on randomly selected modality pairs and evaluate on synthetic and real datasets, including zero-shot medical, remote sensing, and RGB-Event cases. The central claim is that a single matcher fine-tuned only on synthetic multimodal data can generalize across many real cross-modal matching tasks and can outperform modality-specific methods.","tokens_in":31708,"tokens_out":3650,"duration_ms":35769,"significance":"If the central claim holds, the paper makes a valuable empirical contribution: it offers a scalable recipe for cross-modal matching data without expensive multi-sensor capture, and it demonstrates gains on several real benchmarks across three matcher families. The paper is also commendable for the scale of its evaluation, its ablation studies, and the stated intent to release dataset and code. However, the validity of the claimed universal superiority depends on two load-bearing assumptions: that all generated pseudo-modalities inherit MegaDepth geometry labels exactly, and that the data-cleaning step does not circularly bias the training/evaluation. The event-modality generation appears to violate the first assumption, and the cleaning step in Appendix A.2 is described in a way that invites circularity concerns. With those points addressed, the empirical evidence for the IR/Depth/Normal cases would support publication.","major_comments":[{"comment":"The event simulation is internally inconsistent with the claimed label inheritance. After Eq. (1), the authors state that they \"give a random slight motion to compute the event responses.\" An event image generated this way records brightness changes along the trajectory of the injected random motion, not the projection of the scene point corresponding to the RGB pixel at the same location. Therefore the MegaDepth pose/depth labels for RGB-Event pairs in MD-syn are not geometrically valid, and the synthetic RGB-Event AUC values in Tab. 2 are not a valid accuracy measure. This directly affects the Sec. 3.1 claim that labels are \"directly inherited\" and the \"19 cross-modal cases\" count in the abstract. Please either generate events using the known relative motion between the two MegaDepth views without an extra random motion field, or remove event from the label-validity claims and re-evaluate the affected tables.","section":"Sec. 3.3, Event paragraph, and Tab. 2"},{"comment":"The data-cleaning step uses \"our matching model (fine-tuned on the target modality)\" to filter training pairs by recovery of the identity homography. This is circular if the model used for cleaning was trained on the same pairs, or if the final models are evaluated on the cleaned set without reporting results on the uncleaned set. Even though only 0.91% of pairs are dropped, the procedure should specify exactly which model is used, when it was trained relative to the cleaning step, and whether the same filter is applied to test pairs. Without this, the cleaned MD-syn numbers may reflect selection toward easy examples for the chosen matcher rather than intrinsic data quality.","section":"Appendix A.2, Data Cleaning"},{"comment":"The claim that MINIMA \"can significantly outperform the baselines and even surpass modality-specific methods\" is too broad as stated. In Tab. 3, MINIMALoFTR achieves AUC@10 of 30.84 on real RGB-IR, below XoFTR's 34.64. In Tab. 4, MINIMALoFTR achieves 18.65 AUC@5px on real RGB-Depth, below XoFTR's 27.24. The superiority claim is defensible for the LightGlue and RoMa pipelines on several datasets, but not for all pipelines and datasets. Please qualify the claim by pipeline category and dataset, or restrict the abstract and conclusion to the supported cases.","section":"Abstract and Sec. 5.3, Tabs. 3 and 4"},{"comment":"For Depth and Normal, the paper uses monocular estimators (DepthAnything V2 and DSINE) whose outputs are not guaranteed to be geometrically consistent with the MegaDepth camera poses or metric scale. Since RGB-Depth and RGB-Normal are two of the three modality pairs used in the final training (Sec. 5.1), the inherited-label assumption is load-bearing for these modalities as well. Unlike the infrared generator, which is validated in Appendix A.1, there is no quantitative validation of geometric consistency for depth/normal outputs. Please provide a direct comparison of generated depth/normal against MegaDepth ground-truth geometry on a held-out split, or otherwise demonstrate that label errors are small enough to be negligible.","section":"Sec. 3.3, Depth and Normal generation"}],"minor_comments":[{"comment":"The RGB-Event zero-shot evaluation uses only 100 pairs from DSEC; please state whether the reported AUCs have confidence intervals, since several method differences in Tab. 5 are only a few percentage points.","section":"Sec. 5.1 and Tab. 5"},{"comment":"The caption says \"Six Datasets of Real Cross-modal Image Pairs,\" but the left panel includes results from the synthetic MD-syn test split; please clarify which panels are synthetic and which are real.","section":"Fig. 1 caption"},{"comment":"The text says the final model uses only RGB-IR, RGB-Depth, and RGB-Normal pairs, but Tab. A2 also reports models trained with Event, Paint, and Sketch data; please clarify which rows of Tab. A2 correspond to the models used in the main tables.","section":"Sec. 5.1 and Tab. A2"},{"comment":"Several appendix figure captions are duplicated with different dataset citations (e.g., Figure A5 appears twice, once citing METU-VisTIR [45] and once [43]); please deduplicate and correct the citations.","section":"Appendix C, Figures A5-A8"},{"comment":"There are minor capitalization and reference inconsistencies, such as \"Megadepth-1500\" in Tab. A5 and inconsistent citation numbers for M3FD/PIAFusion; please proofread the final version.","section":"General"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Run the data engine idea through a serious referee: the paper is worth engaging even though its headline claims are too big. What's actually new: they turn MegaDepth RGB pairs into six pseudo-modalities (IR via fine-tuned StyleBooth, depth via DepthAnythingV2, normals via DSINE, events via simulation, two artistic styles), assemble MD-syn at 480M cross-modal pairs, and show that fine-tuning LightGlue, LoFTR, or RoMa on a subset of these pairs transfers to real RGB-IR, RGB-Depth, RGB-Event, and several remote-sensing/medical cases. The gains for LightGlue and RoMa on real RGB-IR and RGB-Depth are large, and the ablation in Tab. 6 shows synthetic data beats real data for fine-tuning. That is a useful, concrete result, and they ship code and data.\n\nThe soft spots are real but mostly fixable. First, the 'significantly outperform baselines and even surpass modality-specific methods' line only holds for some backbones. MINIMA-LoFTR loses to XoFTR on real RGB-IR and RGB-Depth, and on medical the improvements are a point or two. The RGB-Event numbers are low in absolute terms. The paper should be careful to claim the data engine works, not that every pipeline becomes SOTA. Second, the event generation is sloppy: giving the source image 'a random slight motion' before computing events means the inherited MegaDepth correspondences are not geometrically exact for RGB-Event pairs. The synthetic event AUC in Tab. 2 shouldn't be read as clean pose accuracy, and Sec. 5.6 doesn't address this. It's not fatal because event pairs aren't used in the final training mix, but the 'accurate labels' claim needs qualification. Third, the data cleaning step in A.2 uses a matching model to filter its own training data; only 0.91% of pairs are dropped, so the circularity is mild, but it should be disclosed and ideally avoided. Fourth, no error bars anywhere; some comparisons are close.\n\nNet: the core empirical claim, that synthetic pseudo-modalities transfer to real cross-modal matching, holds up for the better backbones. The paper deserves a serious referee; it should come back with tighter claims and a clearer account of the event labels.","headline":"Worth a serious referee: the data engine and MD-syn are a real contribution, but the universal-superiority claims need trimming and the event label issue needs a fix.","tokens_in":32319,"tokens_out":4120,"would_cite":true,"duration_ms":38555,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A data engine turns cheap RGB pairs into a large synthetic multimodal dataset with valid labels, and fine-tuning existing matchers on it yields one model that generalizes to real cross-modal matching, including unseen modality pairs.","keywords":["cross-modal image matching","synthetic data generation","modality invariance","data engine","MD-syn dataset","zero-shot generalization","fine-tuning matching models","multimodal perception"],"falsifier":"Measure geometric consistency of generated pairs directly: warp the generated modality image of the second view into the first view using the inherited MegaDepth depth and pose; if the reprojection error is large for event or depth modalities, the labels are corrupted. Alternatively, train on MD-syn with the event generator's random motion disabled and on a depth generator with known metric-scale error; if zero-shot AUC on real DSEC and DIODE does not fall, the reported gains do not depend on the assumed geometry preservation.","tokens_in":31311,"feed_emoji":"🧩","tokens_out":6962,"duration_ms":60660,"temperature":0.7,"pith_summary":"MINIMA's thesis is that cross-modal image matching fails for lack of data, not for lack of model design, and that the missing data can be synthesized. The paper builds a data engine that takes RGB image pairs from MegaDepth, which already have dense depth and pose labels, and generates six pseudo-modality versions of each image (infrared, depth, event, surface normal, and two artistic styles) without changing the known geometry. The resulting MD-syn dataset is used to fine-tune standard matchers that were pre-trained on RGB, yielding one model per backbone that handles many modality pairs. The paper reports large gains on real RGB-infrared, RGB-depth, and RGB-event benchmarks and on zero-shot remote sensing and medical cases, often beating methods trained specifically for one modality pair. A sympathetic reading: if the transfer holds, real multimodal capture and manual labeling can be replaced by generation from abundant RGB data.","feed_headline":"Synthetic-only data trains one matcher for 19 cross-modal cases","feed_subtitle":"Fine-tuning on generated infrared, depth, and event images beats modality-specific matchers on real data, zero-shot.","key_machinery":"MD-syn, a synthetic multimodal dataset produced by a data engine of six generative models applied per image to MegaDepth RGB pairs. Because each generator transforms a single view in place, the inherited depth and pose labels remain geometrically consistent for all generated pairs. The training recipe pairs this dataset with a two-stage procedure: pre-train on RGB MegaDepth, then fine-tune on randomly sampled cross-modal pairs, so the final model is not specialized to any one modality pair.","core_discovery":"The paper's central claim is that modality invariance can be acquired from synthetic scale: fine-tuning an off-the-shelf matcher on randomly selected pairs of generated pseudo-modalities, all inheriting MegaDepth geometry, produces a single model that matches real images across different sensors. The data engine applies per-image generators, so the relative pose and depth computed once for each RGB pair remain valid labels for every generated cross-modal pair, which is what makes label inheritance sound. Using only RGB-IR, RGB-depth, and RGB-normal synthetic pairs for fine-tuning, the resulting MINIMA models transfer to real RGB-IR, RGB-depth, RGB-event, remote sensing, and medical matching across 19 cross-modal cases, with the dense RoMa variant showing the largest improvements. The paper attributes the gains to the data engine rather than to any new matching architecture.","pith_inferences":["The method implies that the dominant cost of building cross-modal matchers shifts from sensor collection to generator fidelity, so improving style transfer and depth estimation should translate directly into matching gains.","A natural stress test is to vary generator realism while holding the training pipeline fixed; if real-world AUC tracks generator fidelity, the synthetic-to-real transfer is driven by the modality gap statistics rather than by unrelated priors.","The same label-inheritance recipe could generate pseudo-modal pairs for other geometric tasks such as optical flow or stereo, where real cross-modal ground truth is even scarcer.","The retained RGB-only performance (except LoFTR) suggests the fine-tuning acts partly as a regularizer, hinting that new modalities could be added incrementally without catastrophic forgetting."],"forward_implications":["One matcher fine-tuned on MD-syn can be deployed on many cross-modal tasks (RGB-IR, RGB-depth, RGB-event, optical-SAR, medical pairs) without per-modality retraining.","Synthetic-only training transfers to real sensors, with the reported pose-estimation AUC on METU-VisTIR increasing by over 400% for the sparse variant and similar gains on DIODE and DSEC.","Adding a new modality reduces to adding a generator to the engine, so the dataset grows without any new real multimodal capture.","Fine-tuning on a mix of synthetic modalities outperforms fine-tuning on a single one, and the models mostly retain their RGB-only matching accuracy on MegaDepth-1500, with LoFTR being the exception."],"supporting_citations":[{"why":"Source data: MegaDepth RGB pairs supply the scene diversity and the depth and pose labels that MD-syn inherits.","marker":"[27]"},{"why":"StyleBooth is the diffusion model fine-tuned on real RGB-IR pairs to generate the infrared pseudo-modality.","marker":"[16]"},{"why":"Depth Anything V2 generates the pseudo depth images from RGB.","marker":"[56]"},{"why":"DSINE generates the surface-normal pseudo-modality.","marker":"[4]"},{"why":"LightGlue is one of the three pre-trained matchers fine-tuned into MINIMA LG.","marker":"[29]"},{"why":"LoFTR is the semi-dense matcher fine-tuned into MINIMA LoFTR and used in the ablations.","marker":"[43]"},{"why":"RoMa is the dense matcher fine-tuned into MINIMA RoMa, which achieves the largest reported gains.","marker":"[11]"},{"why":"XoFTR is the modality-specific RGB-IR baseline that MINIMA is compared against and must beat on real infrared data.","marker":"[45]"},{"why":"GIM represents the alternative data-scaling route (internet videos) and is the main comparison for generalization.","marker":"[41]"}],"fun_headline_variants":["Synthetic data trains a universal matcher for 19 modality gaps","Generative data engine bridges 19 cross-sensor matching tasks","One model matches 19 sensor types, trained on simulated pairs","Train any matcher on synthetic pairs, get cross-modal powers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The generated pseudo-modalities preserve the geometry of the source RGB pairs and reproduce the appearance gap of real cross-modal sensors closely enough that the inherited matching labels stay valid and training transfers to real cameras.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic data trains a universal matcher for 19 modality gaps","Generative data engine bridges 19 cross-sensor matching tasks","One model matches 19 sensor types, trained on simulated pairs","Train any matcher on synthetic pairs, get cross-modal powers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001215,"raw_usage":{"total_tokens":5007,"prompt_tokens":958,"completion_tokens":4049,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":3977}},"tokens_in":574,"tokens_out":4049,"duration_ms":29304,"temperature":1.0,"reasoning_tokens":3977,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:37:41.467782+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure geometric consistency of generated pairs directly: warp the generated modality image of the second view into the first view using the inherited MegaDepth depth and pose; if the reprojection error is large for event or depth modalities, the labels are corrupted. Alternatively, train on MD-syn with the event generator's random motion disabled and on a depth generator with known metric-scale error; if zero-shot AUC on real DSEC and DIODE does not fall, the reported gains do not depend on the assumed geometry preservation.","supporting_citations":[{"cited_title":"Megadepth: Learning single- view depth prediction from internet photos","cited_arxiv_id":null,"evidence_quote":"Source data: MegaDepth RGB pairs supply the scene diversity and the depth and pose labels that MD-syn inherits."},{"cited_title":"Depth any- thing v2","cited_arxiv_id":null,"evidence_quote":"Depth Anything V2 generates the pseudo depth images from RGB."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DSINE generates the surface-normal pseudo-modality."},{"cited_title":"Lightglue: Local feature matching at light speed","cited_arxiv_id":null,"evidence_quote":"LightGlue is one of the three pre-trained matchers fine-tuned into MINIMA LG."},{"cited_title":"Loftr: Detector-free local feature matching with transformers","cited_arxiv_id":null,"evidence_quote":"LoFTR is the semi-dense matcher fine-tuned into MINIMA LoFTR and used in the ablations."},{"cited_title":"Roma: Robust dense feature matching","cited_arxiv_id":null,"evidence_quote":"RoMa is the dense matcher fine-tuned into MINIMA RoMa, which achieves the largest reported gains."},{"cited_title":"Xoftr: Cross-modal feature matching transformer","cited_arxiv_id":null,"evidence_quote":"XoFTR is the modality-specific RGB-IR baseline that MINIMA is compared against and must beat on real infrared data."},{"cited_title":"Gim: Learning generalizable image matcher from internet videos","cited_arxiv_id":null,"evidence_quote":"GIM represents the alternative data-scaling route (internet videos) and is the main comparison for generalization."}],"review_version":1}