{"id":"7ea63b2c-54ca-48c5-be26-4e8cf6610fd9","arxiv_id":"2411.17610","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Disjoint Relevance Mapping Networks, which forbid weight sharing across sensor modalities, reduce forgetting in incremental semantic segmentation but only slightly outperform the shared-weight RMN baseline.","lead":"The paper introduces 'modality-incremental learning', where a segmentation model learns new sensor types (RGB, depth, infrared, grayscale) one at a time, and proposes Disjoint Relevance Mapping Networks that keep each modality's neural connections separate. The authors report that this separation reduces catastrophic forgetting compared to shared-connection continual learning baselines on two automotive datasets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DRMN's mIoU advantage over RMN rests on single-run comparisons; the reported gaps (≤1 mIoU) are within typical seed noise, so the disjointness effect is not statistically established.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the empirical DRMN-vs-RMN improvement is based on single runs without error bars or significance testing. This is the single most critical point because the paper's stated contribution—that disjoint relevance maps improve learning in modality-incremental settings—stands or falls on that measured advantage. If the advantage is noise, the contribution reduces to a variant of RMN with no demonstrated benefit, and the 'complete forgetting mitigation' and 'no capacity penalty' claims are secondary. The paper is otherwise internally coherent: the disjointness mechanism is enforced by construction, the utilization measurements are plausible, and the comparisons against EWC/ILT show the approach is promising. The missing statistical support is addressable with a modest amount of additional computation, which is why the verdict remains conditional rather than reject. I agree with the reader's assessment and do not recommend changing the verdict.","tokens_in":13621,"tokens_out":5500,"duration_ms":51731,"concrete_test":"Run the RGB→IR→Gray sequence on Freiburg Thermal and the IR→RGB→Depth→Gray sequence on InfraParis with at least 5 random seeds each for RMN and DRMN, using the published hyperparameters and code. Compute the mean and standard deviation of per-task mIoU after all tasks, and apply a paired test (e.g., Wilcoxon signed-rank or paired t-test) to the difference in final-task mIoU and in average mIoU. If the DRMN advantage remains significant (p<0.05) and the effect size exceeds 0.5 mIoU on both datasets, the central claim is supported; if not, the reported gaps are within seed noise and the disjointness benefit is unverified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim is that enforcing disjoint relevance maps improves modality-incremental learning over the shared-weight RMN baseline. This claim rests entirely on Tables 1 and 3, where DRMN exceeds RMN by 0.37–1.01 mIoU on the three Freiburg sequences and by 0.10 mIoU on InfraParis. No error bars, repeated runs, or significance tests are reported (Sec. 5.2). For DeepLabV3+ with ResNet-101 trained on these datasets, seed-to-seed variation in mIoU is typically 1–2 points, comparable to or larger than the reported gaps. The supplementary Table 5 also shows that both RMN and DRMN equally mitigate forgetting on prior tasks (e.g., IR stays 55.3→55.1→55.0 for DRMN vs 55.3→55.2→55.1 for RMN), so the unique benefit of disjointness is the forward-transfer gain on new tasks—exactly the quantity that is most susceptible to optimization noise. Thus the conclusion that parameter isolation rather than run-to-run randomness drives the measured improvement is an unsupported inference. A second, less central issue is that the tested modality pairs (RGB, IR, depth, gray) are all visually dissimilar; the paper's own Sec. 4.2 concedes that similar modalities might benefit from sharing, so the claimed universal advantage of disjointness is not explored.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces modality-incremental learning (MIL), a continual learning setting in which each new task is a new sensor modality for image-based semantic segmentation while the label space stays fixed. The authors propose Disjoint Relevance Mapping Networks (DRMN), a modification of Relevance Mapping Networks (RMN) that masks out connections used by previous tasks, forcing each modality to use disjoint network connections. Experiments on Freiburg Thermal and InfraParis with DeepLabV3+/ResNet-101 compare DRMN against fine-tuning, joint training, EWC, ILT, RMN, and two intermediate variants (ORMN and PRMN). The paper reports that DRMN slightly outperforms RMN on average mIoU across task sequences, keeps prior-task performance stable, and achieves comparable overall network utilization while reducing cross-task weight overlap to zero.","tokens_in":13938,"tokens_out":3415,"duration_ms":32038,"significance":"If the central claim holds, the paper makes a useful contribution: it formalizes a practical continual learning setting (MIL), shows that a fixed-capacity network can accommodate new sensor modalities through parameter isolation, and provides a careful ablation from RMN to ORMN to PRMN to DRMN. The supplementary material is a strength: it reports per-task evolution of mIoU, robustness to the pruning threshold, and detailed network-utilization statistics, which makes the paper's mechanism transparent and its claims checkable. The main weakness is that the decisive DRMN-versus-RMN comparison rests on single-run mIoU differences of 0.10 to 1.08 points, which are within typical seed-to-seed variation for this type of segmentation model. The paper's broad claims about completely mitigating forgetting and about the superiority of disjointness are therefore not yet statistically supported.","major_comments":[{"comment":"The central empirical claim that DRMN improves over RMN in MIL rests on single-run mIoU comparisons. In Table 1, the average DRMN advantage over RMN is 0.37, 0.12, and 1.08 mIoU across the three Freiburg sequences; in Table 2 on InfraParis it is 0.10 mIoU. No error bars, number of seeds, or significance tests are reported anywhere in the paper or supplementary material. For DeepLabV3+ with ResNet-101 trained on such datasets, these gaps are comparable to or smaller than typical run-to-run variation. To support the conclusion that disjointness, rather than optimization noise, drives the improvement, the authors should provide repeated runs (at least three to five seeds) with mean and standard deviation, and ideally a significance test or effect-size estimate, for at least the RMN versus DRMN comparison on all task sequences.","section":"Sec. 5.2, Tables 1 and 2"},{"comment":"The abstract and Sec. 4.2 claim that preventing overlap in relevance maps mitigates forgetting, but the reported data do not show a forgetting advantage for DRMN over RMN. In Table 5, on the sequence IR to Gray to RGB, prior-task mIoU for DRMN is 55.30, 55.16, 54.97 (IR) and for RMN is 55.30, 55.18, 55.10 (IR); both methods are equally stable and have near-zero forgetting. The actual observed advantage of DRMN is on the newly learned modality (e.g., Gray at 70.61 vs. 68.85 and RGB at 71.19 vs. 69.46). Thus the paper should either present explicit forgetting metrics showing that DRMN improves stability relative to RMN, or reframe the contribution as improved forward transfer with equal stability, rather than reduced forgetting.","section":"Supplementary Sec. B, Table 5; Sec. 4.2"},{"comment":"The method enforces complete disjointness for every modality pair, and the paper claims this is beneficial without hurting capacity. However, all tested modality pairs (RGB, IR, depth, grayscale) are visually dissimilar, and the paper itself concedes in Sec. 4.2 and the conclusion that similar modalities might benefit from shared weights. The claimed advantage of disjointness is therefore only demonstrated for one end of the modality-similarity spectrum. Either experiments with more similar modalities (or with a controlled similarity axis) are needed to support the general claim, or the claim should be explicitly restricted to dissimilar modalities, with the adaptive overlap mechanism left as future work rather than as a caveat.","section":"Sec. 4.2 and Sec. 6"}],"minor_comments":[{"comment":"The GPU name is written as 'RTXA6000'; it should be 'RTX A6000'.","section":"Sec. 5.2"},{"comment":"The notation fout = W · Mt · fin would benefit from a brief explanation of how the relevance map multiplies the convolution weights, especially for convolutional layers, since the dimensions of W and Mt are not stated.","section":"Eq. (1)"},{"comment":"In the first sentence, 'network utlization' is a typo for 'network utilization'.","section":"Supplementary Sec. D"},{"comment":"The venue is written as 'NeurIOS'; this appears to be a typo for 'NeurIPS'.","section":"Reference [4]"},{"comment":"The paper says the Freiburg dataset offers '13 object categories' and InfraParis '20 classes'; it would help to state whether these are the same label sets used for both datasets or whether each dataset has its own evaluation protocol.","section":"Sec. 5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the authors are transparent about their method and ablations. The main risk is overclaiming from single-run experiments. If the authors provide repeated runs with variance estimates, add explicit forgetting metrics, and soften or suitably restrict the claims, the paper could become publishable. I do not see a fundamental flaw in the method itself, but the quantitative evidence as presented is not yet sufficient to support the central comparative claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper defines a new incremental learning setting, modality-incremental learning (MIL), where the task sequence is a set of different sensor modalities, and proposes DRMN, a hard-masking variant of Relevance Mapping Networks that forces disjoint task-specific connections. The idea is simple and the writing is clear. The setting is a useful practical framing for adding sensors to a deployed system, and the paper does a decent job benchmarking it against EWC, ILT, and fine-tuning on two multi-modal segmentation datasets. The ablations (ORMN, PRMN, DRMN) show a consistent trend: more separation between task masks improves average performance, which is the most persuasive part of the empirical case.\n\nThe soft spots are real, though. The headline claim is that disjoint relevance maps prevent forgetting 'completely.' Their own Table 5 shows DRMN still loses a bit on the first task (IR goes 55.30 to 54.97), and RMN shows the same small drift. That wording should be softened. More importantly, the central DRMN-versus-RMN comparison rests on single runs without error bars. The gaps are 0.37 to 1.01 mIoU on the Freiburg sequences and 0.10 on InfraParis. For segmentation models with ResNet-101 backbones, seed-to-seed variation is typically in the same range, so the claim that disjointness rather than noise drives the improvement is not statistically established. The authors should report multiple seeds or at least acknowledge this limitation.\n\nAlso, the contribution list promises to 'benchmark existing methods for DIL in this novel setting,' but the experiments only include EWC and ILT, no DIL-specific architecture methods like MDIL or DoSe. That is a missing piece of the promised evaluation.\n\nOverall: this is a coherent, useful paper with a modest new idea and a defensible experimental setup, but the central advantage is not proven beyond noise and the language oversells it. I'd send it to review—the MIL formulation alone is worth kicking around, and the authors can address the statistical concerns with additional runs. If they don't, the paper should be held to a lower-impact bar, e.g., workshop level.","headline":"MIL is a useful new framing, but the central DRMN-vs-RMN gain is within seed noise and the 'complete forgetting' claim is overstated.","tokens_in":14460,"tokens_out":2455,"would_cite":true,"duration_ms":22041,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fixed-size network can add new sensor modalities one at a time without forgetting old ones, by giving each modality its own disjoint set of neural connections.","keywords":["continual learning","modality-incremental learning","semantic segmentation","relevance mapping network","catastrophic forgetting","multi-modal perception","autonomous driving","parameter isolation"],"falsifier":"Run each Freiburg Thermal sequence (e.g., RGB→IR→Gray) at least five times with different random seeds for RMN and DRMN and compare the mIoU distributions on the final task; if the DRMN advantage (on average 0.37–1.01 points) is smaller than the typical between-seed spread, the central claim is unsupported. A complementary test: train DRMN on two intentionally similar modalities (RGB then grayscale) and check whether performance drops below RMN, which would show that zero sharing is not universally beneficial.","tokens_in":13460,"feed_emoji":"🚗","tokens_out":6603,"duration_ms":55187,"temperature":0.7,"pith_summary":"Modality-incremental learning (MIL) is a new continual learning setting defined here: a model trained on one sensor modality (say RGB) must later take on new modalities (infrared, depth, grayscale) with the same label set, without forgetting older ones. The paper argues that existing continual learning methods fail here because very different modalities fight for the same weights. It proposes Disjoint Relevance Mapping Networks (DRMN), which extend Relevance Mapping Networks by forcing each task's relevance map to exclude all weights already used by earlier tasks, so every modality learns on its own private set of connections. The central claim is that this hard separation eliminates forgetting and improves new-modality learning compared to shared-weight baselines, while keeping overall network capacity usage nearly unchanged. If true, a single fixed-size network could accommodate an expanding sensor stack in autonomous driving without replaying old data or growing the architecture.","feed_headline":"Disjoint maps let one network learn new sensors without forgetting","feed_subtitle":"Adding IR or depth usually wipes an RGB model's knowledge; this approach forces each sensor onto private connections.","key_machinery":"The central object is the relevance map $M$, a per-task set of weights multiplying the network weights $W$ at each layer, $f_{\\text{out}} = W \\cdot M_t \\cdot f_{\\text{in}}$. The original Relevance Mapping Network learns these maps alongside $W$, prunes values below a threshold $\\mu$, and freezes the selected important weights for each task. DRMN adds one step: before learning task $t$, every weight that any previous task marked relevant is set to zero in $M_t$, so the new task is restricted to connections no earlier modality used. This is parameter isolation by additive masking: network nodes can still be reused across modalities, but the individual connections cannot. The mechanism is what converts a shared-weight continual learner into a per-modality path allocator, and it also allows all tasks' relevance maps to be stored in one compact structure.","core_discovery":"The paper claims that in modality-incremental learning, where each task introduces a new sensor modality with the same set of semantic classes, enforcing a complete separation of relevant neural connections between modalities—disjoint relevance maps—mitigates catastrophic forgetting almost completely without hurting the network's usable capacity. It further claims that weight sharing, which normally aids transfer, becomes detrimental across strongly dissimilar sensing modalities, and that forcing each modality onto previously unused connections produces better final-task accuracy than letting tasks overlap in weight space. The empirical support is a consistent, though small, mIoU advantage of DRMN over the shared-weight RMN baseline on three task orders from Freiburg Thermal (0.37–1.01 points) and on InfraParis (0.10 points), together with per-task evaluations showing performance on earlier modalities stays effectively constant across the learning sequence.","pith_inferences":["If disjoint masks really eliminate forgetting, then the stability–plasticity dilemma for sensor expansion is resolved by allocation rather than by balancing losses; a natural next test is a similarity-aware overlap that allows partial sharing between modalities that are known to be close (e.g., RGB and grayscale), which the paper itself flags as future work.","The result suggests a general recipe for arbitrary input-domain shift, not just sensors: any setting where task identity is known at test time and domains are strongly distinct could use disjoint masks instead of replay or distillation buffers.","One testable extension is to pair DRMN with a small rehearsal buffer: if a few stored frames from old modalities are added to new-task training, this should close most of the remaining gap to single-task models without changing the mask structure.","The reported 'no forgetting' claim concerns mIoU stability on the tested datasets; measuring per-class and per-pixel forgetting on rare classes (e.g., small traffic objects) would show whether hard separation also protects categories that contribute little to aggregate mIoU."],"forward_implications":["A fixed-size network can add new sensor modalities sequentially without storing old sensor data or expanding parameters, because each modality leaves a permanent, frozen footprint.","Because the sensor itself supplies the task ID at inference time, the network does not need any task-identification or task-inference module.","Regularization and distillation methods that rely on shared weights (EWC, ILT) underperform in this setting, suggesting that modality shift is a distinct and more severe form of domain shift.","The disjoint masks keep total network utilization almost unchanged relative to RMNs, so capacity is not exhausted faster despite the hard separation; on InfraParis the final task learns from only about 6% of connections and still matches shared-weight RMN.","All relevance maps for a growing number of tasks can be kept in a single data structure, so the storage and loading overhead stays constant."],"supporting_citations":[{"why":"Supplies the Relevance Mapping Network baseline and the relevance-map mechanism that DRMN modifies with disjoint masks.","marker":"[25]"},{"why":"Baseline regularization method that penalizes overwriting important weights; its failure in MIL motivates the need for parameter isolation.","marker":"[27]"},{"why":"Baseline distillation method whose feature alignment across modalities causes poor results, used as a comparison point.","marker":"[31]"},{"why":"Freiburg Thermal dataset provides aligned RGB and infrared images with semantic labels and the task sequences used in the main experiments.","marker":"[39]"},{"why":"InfraParis dataset supplies RGB, infrared, depth, and derived grayscale modalities for the four-task MIL experiment.","marker":"[12]"},{"why":"DeepLabV3+ is the segmentation model whose convolutional layers are augmented with relevance maps.","marker":"[5]"}],"fun_headline_variants":["Disjoint maps stop forgetting when adding new sensors","Forcing sensor-specific connections prevents catastrophic forgetting","Private neural paths for each sensor beat shared weights","Modality-incremental learning: keep each sensor's knowledge separate","Disjoint relevance maps let segmentation networks learn new sensors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the small mIoU advantage of DRMN over RMN reflects a genuine effect of weight separation rather than run-to-run variation, since the paper reports single runs with no error bars.","fun_headline_variants_meta":{"raw":{"variants":["Disjoint maps stop forgetting when adding new sensors","Forcing sensor-specific connections prevents catastrophic forgetting","Private neural paths for each sensor beat shared weights","Modality-incremental learning: keep each sensor's knowledge separate","Disjoint relevance maps let segmentation networks learn new sensors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000147,"raw_usage":{"total_tokens":1152,"prompt_tokens":877,"completion_tokens":275,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":200}},"tokens_in":493,"tokens_out":275,"duration_ms":3003,"temperature":1.0,"reasoning_tokens":200,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:54:53.611039+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run each Freiburg Thermal sequence (e.g., RGB→IR→Gray) at least five times with different random seeds for RMN and DRMN and compare the mIoU distributions on the final task; if the DRMN advantage (on average 0.37–1.01 points) is smaller than the typical between-seed spread, the central claim is unsupported. A complementary test: train DRMN on two intentionally similar modalities (RGB then grayscale) and check whether performance drops below RMN, which would show that zero sharing is not universally beneficial.","supporting_citations":[{"cited_title":"Overcoming catastrophic forgetting in neu- ral networks","cited_arxiv_id":null,"evidence_quote":"Baseline regularization method that penalizes overwriting important weights; its failure in MIL motivates the need for parameter isolation."},{"cited_title":"Knowledge dis- tillation for incremental learning in semantic segmentation","cited_arxiv_id":null,"evidence_quote":"Baseline distillation method whose feature alignment across modalities causes poor results, used as a comparison point."},{"cited_title":"HeatNet: Bridging the Day-Night Domain Gap in Semantic Segmentation with Thermal Images","cited_arxiv_id":"2003.04645","evidence_quote":"Freiburg Thermal dataset provides aligned RGB and infrared images with semantic labels and the task sequences used in the main experiments."},{"cited_title":"Infraparis: A multi-modal and multi-task autonomous driving dataset","cited_arxiv_id":null,"evidence_quote":"InfraParis dataset supplies RGB, infrared, depth, and derived grayscale modalities for the four-task MIL experiment."}],"review_version":1}