{"id":"d09b1b56-f55e-49f3-b947-7e3d948dec36","arxiv_id":"2608.10240","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Per-sequence modality masking during training makes multi-modal sequential recommenders retain 1.0 to 3.2x more accuracy when text or images are missing at serving time.","lead":"This paper proposes Sequential Modality Dropout (SMD), a four-line training-time change that randomly erases the image or text stream of a user's entire interaction history, so recommenders learn to work when a modality is missing at serving time. It reports that SMD preserves 61% of full-modality accuracy at extreme 95% missing rates versus 22% without, across several recommender backbones and Amazon domains.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central design choice (per-sample, whole-history modality masks) is never tested against the obvious per-item masking alternative, despite the paper's own test protocol describing per-item missingness as realistic; if per-item masks match or beat SMD, the 'Sequential' contribution is…","rationale":"Reader's weakest_assumption is exactly this per-sample vs per-item design assumption, and I agree. The paper's own text creates a tension: Section 2.2 says per-item missingness is 'rarely seen in deployment,' while Section 3.1 calls Protocol 2 'matching item-level missingness in real catalogs.' The reported RQ1/RQ2 results are credible for the conditions tested, but they do not test the defining design decision. This is not an attack on the authors; it is an omitted experiment that a reviewer can require. Secondary issues also support a CONDITIONAL verdict: the abstract's 'four backbones on four Amazon domains' overstates the 11 completed cells (the main text says all 16 were run, while Appendix B says five cells were omitted for compute budget), and the RQ3 retention improvement from 90% to 98% is a ratio artifact because both absolute HR@10 values fall in Table 1c. However, the per-sample/per-item ablation is the most load-bearing for the central claim. The verdict should remain CONDITIONAL: the core SMD result may hold, but the paper should either run the ablation or soften the 'Sequential' design claim.","tokens_in":11572,"tokens_out":10065,"duration_ms":91631,"concrete_test":"Train MM-SASRec on Scientific (and repeat on Arts) under three matched conditions with identical seeds and hyperparameters: (1) no dropout, (2) SMD per-sample mask with p=0.3, (3) per-item Bernoulli mask with p=0.3 (independent mask per item and modality, applied at the same fusion point). Evaluate each checkpoint under Protocol 1 (full text removal) and Protocol 2 (per-item missingness, p_miss=0.95, 5 seeds). Compare full HR@10, no-text HR@10, retention R, and per-item HR@10. If per-item training is within about one point of SMD on Protocol 1 and better on Protocol 2, the per-sample design is not load-bearing; if SMD dominates both, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.2 (Why Per-Sample) justifies the core mechanism by asserting that real missingness clusters by whole modality streams and that per-item masking trains for a pattern 'rarely seen in deployment.' Yet Section 3.1 defines Protocol 2 as per-item independent missingness and explicitly calls it 'matching item-level missingness in real catalogs [2,11].' The paper never runs a per-item training-mask ablation. This matters because the only difference between SMD and standard modality dropout is that the mask is shared across the entire user sequence. If per-item masking at the same p=0.3 achieves comparable retention under Protocol 2 and acceptable retention under Protocol 1, then the whole-history mask provides no demonstrated benefit, and the paper's novelty claim reduces to existing per-sample modality dropout applied to sequences. Conversely, if per-sample masking clearly dominates per-item masking, the design rationale is supported. Since the central claim is about a sequential (per-sample) mask, this missing comparison is the most load-bearing gap: it leaves the method's defining design choice unvalidated against the obvious alternative.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes Sequential Modality Dropout (SMD), a training-time per-sample Bernoulli mask that zeroes an entire image or text stream across all items in a user's history at the fusion point of a multi-modal sequential recommender. The authors evaluate SMD on MM-SASRec, IISAN, MISSRec, and fMRLRec on Amazon domains from the MISSRec benchmark, measuring retention (HR@10 under modality removal divided by full-modality HR@10) and robustness gain relative to the same backbone trained without SMD. They report text-retention gains of 1.0 to 3.2x across 11 evaluated backbone-domain cells, a 2.8x improvement at 95% per-item missingness in Figure 2, statistically significant per-user paired tests, and an optional cross-modal reconstruction loss that further improves retention on a simple additive backbone under severe text missingness.","tokens_in":11646,"tokens_out":6151,"duration_ms":58980,"significance":"The method is simple, architecture-agnostic, and potentially useful in practice: the four-line plug-in is clearly described, the authors release source code and JSON outputs backing every number, full-catalog ranking and chronological splits are used, and two complementary robustness protocols plus per-user paired significance tests strengthen the empirical support. If the reported results are reproducible, SMD is a valuable addition to the multi-modal sequential recommendation toolbox. The central claim is supported for the cells that were actually run; the main weaknesses are that the defining design choice (per-sample whole-history mask vs per-item mask) is never tested against the obvious alternative, and the coverage claims in the abstract and Section 3.2 are broader than the evaluated matrix.","major_comments":[{"comment":"Section 2.2 justifies the per-sample (whole-history) mask by asserting that real missingness is whole-modality and category-structured, and that a per-item mask would train for a pattern 'rarely seen in deployment.' Section 3.1, however, defines Protocol 2 as per-item independent missingness and explicitly calls it 'matching item-level missingness in real catalogs [2,11].' The paper never compares SMD against per-item training masks at the same dropout probability. If per-item masking performs comparably under Protocol 2 and acceptably under Protocol 1, then the paper's central novelty, the sequential per-sample mask, has no demonstrated benefit over standard per-sample modality dropout applied to sequences. This ablation is load-bearing for the paper's title and for the RQ1/RQ2 claims, and it should be run across at least the MM-SASRec domains, and ideally all backbones, before the per-sample design can be accepted.","section":"Section 2.2 'Why Per-Sample' vs Section 3.1 'Evaluation Protocols'"},{"comment":"The main text says 'we ran all 16 (backbone, dataset) combinations' and states '11 of the 16 improve,' but Appendix B reports that Arts runs are omitted for IISAN, MISSRec, and fMRLRec, and Office runs are omitted for MISSRec and fMRLRec. Table 3 actually contains 10 cells, plus the MM-SASRec/Arts cell in Table 1b, for 11 evaluated cells. The abstract's 'across four backbones ... on four Amazon domains' therefore overstates the empirical coverage, and the claim that 11 of 16 improve is internally inconsistent with the appendix. The authors should either complete the missing cells or revise all coverage claims to refer explicitly to the 11 evaluated cells.","section":"Section 3.2 and Appendix B / Table 3"},{"comment":"The universal-claim wording in Section 3.2 ('SMD helps on every recommender we tested') and the conclusion ('it lifts HR@10 text retention by 1.0 to 3.2x over 11 of 16 combinations') should be reconciled with the actual number of evaluated cells. With 11 cells evaluated and all 11 improving, the manuscript should state 'all 11 evaluated cells improve by 1.0 to 3.2x' rather than using the ambiguous '11 of 16.' This is not merely a wording issue: the paper's stated coverage and its reproducibility artifact are in conflict, and the claim as written is not supported by the appendix tables.","section":"Section 3.2, Section 3.1"}],"minor_comments":[{"comment":"The abstract and the RQ3 contribution bullet present the reconstruction loss as lifting retention from 90% to 98% on Beauty, but Table 1c shows that full-modality HR@10 drops from 0.63 (SMD only) to 0.52 (SMD+recon), a 17% relative loss, and that the no-SMD full-modality baseline is 0.72. Section 3.4 does acknowledge the trade-off, but the abstract and contribution bullet should state the accuracy cost explicitly, since the paper otherwise emphasizes that SMD preserves full-modality accuracy.","section":"Table 1c / Abstract RQ3 bullet"},{"comment":"The reported slopes (-0.027 and -0.051 per unit p_miss) describe HR@10 in percentage points per percentage point of p_miss; please clarify these units in the text so the reader does not confuse them with raw metric values.","section":"Figure 2 / Table 4"},{"comment":"The 121k-user paired significance tests are reported only for MM-SASRec; Section 3.3 should state explicitly that the per-user paired-claims apply to that backbone, not to all four backbones.","section":"Section 3.3 / Table 7"},{"comment":"The sentence 'Arts runs are omitted for IISAN, MISSRec, and fMRLRec for the same compute-budget reason' refers to a compute-budget reason that is not stated earlier for Arts; please specify the omitted cells in one transparent table and give the concrete reason for each omission.","section":"Table 3 caption and Appendix B"},{"comment":"The caption says the better No-SMD/SMD value per cell is bold, but in the Instruments full column the No-SMD value is 7.14 and the SMD value is 7.13; please qualify the claim 'matching or exceeding peak HR@10 on every dataset' so readers know that one cell is a statistical tie rather than an improvement.","section":"Table 1b caption"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for the venue and the release of code and JSON outputs backing every number is commendable. The two load-bearing issues are the missing per-item masking ablation and the inconsistency between the 'ran all 16' claim and the appendix's documented omissions. Both are fixable: run the per-item training-mask comparison and correct the coverage statements. The RQ3 accuracy-cost disclosure should also be moved into the abstract bullet for that result. No concerns about author conduct or novelty disclosure beyond the above."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: SMD looks like a genuinely practical plug-in — per-sample Bernoulli masking of each modality stream, applied before fusion, helps retention almost everywhere they tested it, and they shipped code and detailed appendices. But the central design choice, masking the whole history rather than per item, is never tested against the obvious baseline. That gap matters because the paper's own Protocol 2 calls per-item missingness \"matching item-level missingness in real catalogs\" — the same justification they use for per-sample masks. You can't have it both ways. If per-item training masks perform comparably under Protocol 2 (and reasonably under Protocol 1), then the \"sequential\" part of SMD adds nothing; the method collapses to existing per-sample modality dropout. If per-item masks are worse, the design is justified. They should run that ablation with the same p=0.3 and report it. Without it, the novelty claim is unproven.\n\nWhat's good: the method is simple, architecture-agnostic, and the evaluation is honest in several ways. They disclose the full-modality trade-offs per cell (mean +0.8%, worst -4.4%). They ran per-user paired tests on 121k users and report that 2 of 24 are non-significant, rather than cherry-picking. The 11-of-16 coverage is disclosed in the appendix, though the abstract's \"on four Amazon domains\" phrasing overstates it. The retention sweep at 95% per-item missingness — 61% vs 22% — is a striking result if it holds up under per-item training masking.\n\nSoft spots, in proportion: the missing per-item ablation is the load-bearing one. The RQ3 reconstruction-loss headline (90% to 98% retention on MM-SASRec/Beauty) is a ratio artifact; both absolute HR@10 values fall (0.63 to 0.52 full, 0.57 to 0.51 no-text). It's an opt-in enhancement, so this is minor, but the presentation should not lead with the ratio. The two non-significant full-modality HR@10 tests are fine; NDCG is significant there, and they disclosed it.\n\nBottom line: worth a serious referee. The idea is simple enough that the missing baseline is the deciding factor. If the authors add it and the per-sample mask wins, this is a solid, citable empirical paper. If not, the contribution shrinks to a footnote. I'd send it to review with the request for that ablation and a corrected abstract.","headline":"Useful four-line fix for missing modalities in multi-modal sequential recommenders, but the paper never tests the per-item masking alternative that would actually justify the per-sample design.","tokens_in":12299,"tokens_out":2631,"would_cite":true,"duration_ms":25086,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Multi-modal sequential recommenders can be made robust to missing images or text by a per-sample Bernoulli mask that zeroes an entire modality stream for each user's history during training, lifting text retention from 40–73% to 79–97% on…","keywords":["sequential recommendation","multi-modal","modality dropout","missing modality","robustness","HR@10 retention","per-sample mask","cross-modal reconstruction"],"falsifier":"Train the same backbones with a per-item rather than per-sample modality mask under identical protocols and compare retention under the per-item missing-rate sweep (Protocol 2); if per-item masking matches or exceeds SMD's retention, then the whole-history masking pattern is not what drives the gains. A second concrete check: measure the spatial distribution of missing images and text in a real logged catalog—if missingness within a user's history is item-independent, SMD's training distribution does not match deployment.","tokens_in":11222,"feed_emoji":"🧩","tokens_out":7605,"duration_ms":62037,"temperature":0.7,"pith_summary":"Multi-modal sequential recommenders predict what a user will interact with next by combining image and text features, yet they are normally trained on data where every item carries both modalities. Real catalogs routinely lack one of them, and models trained on complete data lose much of their recommendation accuracy when a modality is missing at serving time. This paper proposes Sequential Modality Dropout (SMD): during training, the entire image stream or the entire text stream is randomly erased with probability p for each user's whole history, so the model learns to predict the next item without relying on any single modality. The paper claims that this four-line change lifts text retention (the fraction of full-modality HR@10 retained when a modality is removed) from 40–73% to 79–97% across four Amazon domains on a base architecture, and at an extreme 95% per-item missing rate keeps 61% of HR@10 versus 22% without SMD. If correct, any deployed multi-modal sequential recommender can be made robust to missing images or text by inserting a few lines at its fusion point.","feed_headline":"Masking whole modality streams triples recommender retention","feed_subtitle":"When 95% of items lose a modality, the model keeps 61% of its accuracy; without the trick, only 22%.","key_machinery":"The mechanism is a per-sample Bernoulli modality mask: a random binary draw per user-history sample and per modality (image and text), multiplied onto the modality embeddings before fusion, so zeroing one mask zeroes that entire modality stream across all items in the user's sequence. It is applied only during training and is not rescaled, unlike standard dropout, because its goal is invariance to a genuinely absent modality rather than variance reduction; the full-modality test input is simply the p=0 case already seen in training. The optional cross-modal reconstruction loss trains the two modality projections to predict each other through two-layer side projections, giving the model a fallback for recovering a missing modality when fusion is simple additive. The mask's per-sample design is the load-bearing choice: the paper argues that real catalog missingness clusters by category or source, so an entire modality tends to be missing for a user's history at once, which is a pattern per-item masking would not train for.","core_discovery":"The central discovery is that a per-sample Bernoulli modality mask—not a per-item mask—applied at the fusion point of any multi-modal sequential recommender during training is enough to make it robust to missing modalities at serving time. For each training sample, the image stream and the text stream are each independently zeroed with probability p, and the same mask applies to every item in the user's chronological sequence; at test time no mask is applied. Across four backbones (MM-SASRec, IISAN, MISSRec, fMRLRec) on the MISSRec Amazon benchmarks, SMD lifts HR@10 text retention from 18–94% to 56–99% on Scientific, and from 40–73% to 79–97% for MM-SASRec across four domains, with a mean full-modality HR@10 change of +0.8% over the 16 backbone-dataset cells tested (11 improve). Under a per-item missing-rate sweep, retention stays approximately flat (slope ≈ -0.027 per unit of missing rate) versus a steep drop without SMD (slope ≈ -0.051), leaving SMD at 61% retention versus 22% at a 95% per-item missing rate. An optional cross-modal reconstruction loss further raises text retention from 90% to 98% on a simple additive backbone in a severely text-missing catalog, at a small full-modality accuracy cost.","pith_inferences":["The per-sample mask may also function as a regularizer that discourages the model from relying on modality-specific shortcuts; the paper measures only retention and HR@10, leaving representation-level effects an open test.","SMD could be combined with feature-propagation or missing-modality-completion methods, since masking makes the backbone tolerant of absence while completion methods reconstruct the missing features; the paper mentions but does not test this combination.","A direct per-item-mask training variant is the natural next experiment: it would separate the benefit of masking itself from the benefit of matching whole-stream missingness, and would indicate how sensitive SMD is to a mismatch between assumed and actual missingness structure.","The same whole-stream masking could extend beyond images and text to audio, video, or structured attributes, and to corruption (noisy) rather than absence; the paper lists these as open directions."],"forward_implications":["SMD plugs into any multi-modal sequential backbone at the fusion point in four lines, with no hyperparameter changes to the host model; even fMRLRec, already at 94% text retention, rises to 99%.","With SMD, MM-SASRec keeps 61% of HR@10 when 95% of items independently lose a modality, versus 22% without SMD, and the gap widens monotonically with the missing rate.","Across all 16 backbone-dataset cells, 11 improve with a mean peak HR@10 change of +0.8%, and the two largest full-modality drops (IISAN, -4.4% and -4.1%) coincide with the largest retention gains (3.2x and 1.4x).","Per-user paired tests over 121k users on three datasets find SMD significantly better in 22 of 24 condition-dataset cells by McNemar (HR@10) or Wilcoxon (NDCG@10) tests.","The optional cross-modal reconstruction loss is worthwhile only on simple additive fusion under severe missingness (90% to 98% text retention on a 48% text-missing catalog); on dynamic fusion it costs 2.3% of full-modality HR@10 for only 3 points of retention."],"supporting_citations":[{"why":"Supplies the SASRec self-attentive sequential backbone whose additive multimodal extension forms the MM-SASRec host in the main robustness sweeps.","marker":"[9]"},{"why":"Provides the MISSRec benchmark datasets and a dynamic-fusion backbone; its reported results are the point of comparison for whether the reconstruction loss pays off.","marker":"[16]"},{"why":"Contributes the IISAN backbone (decoupled PEFT with frozen ViT/BERT), the test case showing the largest retention gain (3.2x).","marker":"[3]"},{"why":"Contributes fMRLRec, the linear-recurrent backbone whose near-redundant fusion already retains 94% of text accuracy and rises to 99% with SMD.","marker":"[17]"},{"why":"Prior modality-dropout method for rating prediction; the per-sample versus per-item design is positioned against the missing-at-random pattern it trains for.","marker":"[15]"},{"why":"Feature-propagation approach for missing modalities in recommendation, named as the complementary technique SMD can be combined with.","marker":"[11]"},{"why":"Masked modality projection whose spirit the optional cross-modal reconstruction loss borrows.","marker":"[13]"},{"why":"Establishes the no-rescaling modality-dropout convention SMD follows, drawn from audio-visual gesture recognition.","marker":"[12]"}],"fun_headline_variants":["Whole-stream dropout triples recommender retention","Modality dropout keeps 61% accuracy when 95% missing","Architecture-agnostic fix triples recommender resilience","Erase whole modality streams, keep recommender robust","Simple mask during training triples recommender retention"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's success depends on real catalog missingness clustering by whole modality streams across a user's history (for example, an entire category lacking text or an image feed going down) rather than being scattered independently per item; the paper argues for this pattern from category-level missing rates but never runs the direct comparison of per-item masking during training.","fun_headline_variants_meta":{"raw":{"variants":["Whole-stream dropout triples recommender retention","Modality dropout keeps 61% accuracy when 95% missing","Architecture-agnostic fix triples recommender resilience","Erase whole modality streams, keep recommender robust","Simple mask during training triples recommender retention"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000246,"raw_usage":{"total_tokens":1610,"prompt_tokens":1084,"completion_tokens":526,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":700,"completion_tokens_details":{"reasoning_tokens":449}},"tokens_in":700,"tokens_out":526,"duration_ms":5061,"temperature":1.0,"reasoning_tokens":449,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:10:32.118992+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same backbones with a per-item rather than per-sample modality mask under identical protocols and compare retention under the per-item missing-rate sweep (Protocol 2); if per-item masking matches or exceeds SMD's retention, then the whole-history masking pattern is not what drives the gains. A second concrete check: measure the spatial distribution of missing images and text in a real logged catalog—if missingness within a user's history is item-independent, SMD's training distribution does not match deployment.","supporting_citations":[{"cited_title":"MISSRec: Pre-training and Transferring Multi-modal Interest-aware Sequence Representation for Recommendation","cited_arxiv_id":"2308.11175","evidence_quote":"Provides the MISSRec benchmark datasets and a dynamic-fusion backbone; its reported results are the point of comparison for whether the reconstruction loss pays off."},{"cited_title":"IISAN: Efficiently Adapting Multimodal Representation for Sequential Recommendation with Decoupled PEFT","cited_arxiv_id":"2404.02059","evidence_quote":"Contributes the IISAN backbone (decoupled PEFT with frozen ViT/BERT), the test case showing the largest retention gain (3.2x)."}],"review_version":1}