{"id":"84866b42-b8df-40e0-8941-691a6ce7f27d","arxiv_id":"2602.13748","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A two-stage training pipeline—unified schema warm-up on heterogeneous supervision followed by task-specific fine-tuning—improves low-resource multimedia event extraction on M2E2 across three VLMs.","lead":"RMPL trains a vision-language model for multimedia event extraction in two stages: a warm-up on text, image, and relation supervision, then task-specific fine-tuning. On the M2E2 benchmark it reports F1 gains for event mention and argument role extraction across three VLM backbones and all modality settings.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The core transfer claim rests on an undocumented ACE/SWiG/imSitu-to-M2E2 label mapping (Sec 5.1); if that projection is lossy or M2E2-specific, the reported gains may be an artifact of the mapping rather than of the progressive-learning paradigm.","rationale":"The paper's experimental core is the Baseline/Warm-up/RMPL comparison in Table 2. All three conditions share the same VLM and evaluation protocol, so the internal comparison is relatively clean. The weakest point is upstream: external supervision must be projected onto M2E2's label space, and that projection is asserted but not specified. Since the method is explicitly proposed for low-resource MEE, this projection is not a minor implementation detail; it defines what the model is trained to predict. If most SWiG activities (e.g., 'eating', 'driving') have no natural M2E2 type, the visual branch either trains on a tiny filtered subset or on noisy many-to-one labels; either scenario undermines the transfer claim. I agree with the reader that this is the weakest assumption. The missing mapping also blocks reproducibility, and the test-set-based selection of mixing ratios in Sec 5.4.2 compounds the risk. A concrete release-and-retrain check would settle whether the reported gains survive an independently specified mapping. I therefore keep the reader's CONDITIONAL verdict; the concern does not by itself refute the empirical tables, but it makes the central claim unverifiable as written.","tokens_in":15979,"tokens_out":9832,"duration_ms":91326,"concrete_test":"Ask the authors to release the full mapping dictionaries (ACE subtype to M2E2 type; SWiG/imSitu activity/role to M2E2 type/role) and the number of training instances retained after filtering. Then rerun Table 2's image-only and multimedia conditions using only training instances whose mapped label is unambiguous (or using a naive baseline mapping, e.g., exact verb/activity match), holding all other training choices fixed. If the F1 gains over Baseline collapse or fail to be consistent across the three backbones, the undocumented projection, not the progressive-learning paradigm, is doing the work.","verdict_should_be":"UNCHANGED","load_bearing_attack":"RMPL's training signal comes entirely from ACE 2005 (33 event types, 36 roles), SWiG/imSitu (504 activities, 1,788 roles), and MNRE (23 relations), but the paper only says these are 'mapped to the same 8 event types defined in M2E2' (Sec 5.1). No mapping table, coverage count, or treatment of unmapped labels is given. This matters because the central claim is that heterogeneous supervision transfers to MEE; if the mapping silently discards most SWiG/imSitu labels or manually encodes M2E2's ontology into training targets, then the 'consistent improvements' in Table 2 could be produced by the label projection rather than by the warm-up + specialization stages. The risk is amplified by Sec 5.4.2, where the 1:1:1 supervision mixing ratio is selected by running on M2E2 itself, so the evaluation is not an out-of-sample test. Without the mapping and coverage statistics, the central claim is not independently checkable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RMPL, a two-stage training method for multimedia event extraction (MEE) under low-resource conditions where M2E2 has no training split. Stage I warm-ups a VLM under a unified generation schema using heterogeneous external supervision: ACE 2005 textual event extraction, SWiG/imSitu visual event extraction, and MNRE multimedia relation extraction. Stage II specializes the warmed-up model through supervised fine-tuning for event mention identification and argument role extraction, using the same external sources projected onto M2E2's 8 event types and 15 roles. Experiments on M2E2 with Qwen2-VL-7B, InternVL3_5-8B, and Qwen3-VL-8B report consistent F1 gains over prompt-only and warm-up-only baselines across text-only, image-only, and multimedia settings, together with ablations for relation-aware warm-up and mixing proportions.","tokens_in":16308,"tokens_out":4456,"duration_ms":44001,"significance":"If the reported gains are genuine, the paper makes a useful contribution: it shows that heterogeneous unimodal supervision plus a relation-aware warm-up and task-specific specialization can improve MEE without using M2E2 training labels. The use of three different VLM backbones under a common evaluation protocol and the component-wise ablations are strengths. However, the central transfer claim rests on an undocumented label mapping from ACE/SWiG/imSitu to M2E2, and a key hyperparameter (the 1:1:1 mixing ratio) is selected by evaluating on M2E2 itself. Until these issues are resolved, the source and magnitude of the reported gains remain uncertain.","major_comments":[{"comment":"The load-bearing claim is that supervision from ACE 2005 (33 event types, 36 roles), SWiG/imSitu (504 activities, 1,788 roles), and MNRE (23 relations) transfers to M2E2's 8 event types and 15 roles. The paper only says, 'we map both textual event types and visual activity semantics to the same 8 event types defined in M2E2.' No mapping table, coverage statistics, or treatment of unmapped labels is given. If the projection is lossy or manually encodes M2E2's ontology into the training targets, the consistent gains in Table 2 could be produced by the label projection rather than by the proposed progressive-learning paradigm. The authors should provide the full mapping, counts of how many training examples fall in each M2E2 type/role, and a discussion of discarded or ambiguous labels.","section":"Section 5.1, Table 1"},{"comment":"The supervision mixing ratio (1:1:1) is selected by comparing variants on M2E2 itself. Because M2E2 is the only evaluation benchmark, choosing the ratio on it means the reported 'consistently best' configuration is partly fitted to the test set, undermining the out-of-sample claim. The authors should either use a held-out portion of M2E2 for validation or pre-specify the ratio and report sensitivity. The same concern applies to the Stage-I/Stage-II training schedules, which are also free parameters selected for the reported configuration.","section":"Section 5.4.2, Figure 3"},{"comment":"All results appear to be single runs with no error bars, multiple seeds, or significance tests. Given the relatively small M2E2 evaluation set and the variability of VLM fine-tuning, 'consistent improvements' could be within run-to-run noise. The authors should report means and standard deviations over at least three seeds per condition, and ideally a paired significance test across the three backbones. Without this, the magnitude of the claimed gains is not statistically grounded.","section":"Tables 2-4, Experimental Settings"},{"comment":"The method is defined by its prompt templates and schema serialization, but neither the exact templates nor the code is released or included in the appendix. Eq. (1) depends on the schema control variable τ and the serialization protocol, yet the actual prompts shown in Figure 2 are illustrative. For the central claim to be independently checkable, the authors should release the prompt templates, the mapping tables, and the training/inference code, or at least provide complete template details in a supplementary appendix.","section":"Section 4.2, Figure 2, Reproducibility"}],"minor_comments":[{"comment":"Several table cells appear to have formatting errors with missing spaces or concatenated numbers, e.g., '88.5686.63' in the InternVL3_5-8B Baseline row of Table 2, and '75.02.6', '49.959.1', '42.449.945.8' in Table 3. These need to be corrected for readability.","section":"Tables 2 and 3"},{"comment":"The notation m=τ is confusing: τ is introduced as a schema control variable in Eq. (1), and m is used for modality in Eq. (2). Please use distinct symbols or define the relation between m and τ explicitly.","section":"Section 4.2, Eq. (2)"},{"comment":"The mapping between α:β:λ in Eq. (4) and the 'sampling ratios' ACE:SWiG:MNRE in Figure 3 is not made explicit. Please clarify whether α, β, λ are loss weights, sampling probabilities, or both.","section":"Section 5.4.2"},{"comment":"The comparison with prior work uses different backbone architectures and training setups. The paper correctly frames this as a 'high-level reference,' but the text says RMPL 'achieves the best overall performance'; please soften this to avoid overclaiming given the uncontrolled comparison.","section":"Section 5.3, Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is in scope and the proposed training paradigm is plausible, but the core empirical claim currently cannot be independently verified. The missing label mapping is the most serious issue; the M2E2-based selection of the mixing ratio further weakens the out-of-sample claim. If the authors can supply the mapping with coverage statistics, multi-seed results, and reproducible templates/code, the contribution would be considerably stronger. I recommend major revision rather than rejection because these issues are addressable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: RMPL is a reasonable, well-motivated recipe: warm up on text event, visual event, and multimedia relation supervision under a unified schema, then fine-tune for mention and argument extraction. It doesn't use M2E2 training labels, and it reports consistent F1 gains over prompt-only and warm-up-only baselines on three VLM backbones. The novelty is modest — the components are known — but the combination and stage-wise ordering are new enough, and the ablations suggest both the relation supervision and the warm-up stage are doing real work. That's worth credit.\n\nWhat I like: the design is simple and the empirical pattern is internally consistent. Three backbones, all showing RMPL above its own baselines, with component ablations on Qwen2-VL. The claim that heterogeneous unimodal supervision plus relational supervision helps warm-up is plausible and the ablation supports it.\n\nWhere I'd push back: (1) The label mapping from ACE 2005 and SWiG/imSitu to M2E2's 8 event types is the load-bearing transfer step, and the paper only says 'following Li et al. [1]' — no mapping table, no coverage counts, no treatment of unmapped labels. If most SWiG activities are discarded or if event labels are hand-aligned to M2E2's ontology, the reported gains could come from that projection rather than the progressive-learning paradigm. This needs to be described in a revision. (2) The supervision mixing ratio 1:1:1 is selected by running on M2E2 itself (Sec 5.4.2), so the evaluation is not fully out-of-sample. It's a modest leak, not a fatal one, but it should be acknowledged and ideally validated on a small held-out split. (3) No error bars or multiple seeds, no code or prompt templates released. For a paper whose claim is 'this recipe transfers', that's a serious reproducibility gap. (4) Minor: Table 3 has at least one malformed row (FSMEE) and the numbers in the related-work comparison are hard to cross-check.\n\nI don't think the stress-test note's worry sinks it — the mapping is inherited from Li et al., and the gains are consistent across several settings — but the paper cannot be independently checked as written.\n\nBottom line: this deserves a serious referee. I'd send it to review and ask for the mapping details, coverage statistics, seeds/error bars, and either code or full prompts. With those, it would be a solid subfield contribution.","headline":"A sensible low-resource training recipe for multimedia event extraction, with consistent gains across three backbones; the main caveat is that the label mapping and evaluation protocol are not documented enough to verify.","tokens_in":16741,"tokens_out":2286,"would_cite":false,"duration_ms":21054,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A vision-language model can learn multimedia event extraction from text-only, image-only, and relation supervision, without any annotated multimedia events.","keywords":["multimedia event extraction","vision-language models","multi-task learning","progressive learning","supervised fine-tuning","relation extraction","low-resource learning","event argument extraction"],"falsifier":"Re-annotate a held-out subsample of M2E2 as a training split and train the same VLM directly on it; if direct supervision does not beat RMPL's external-supervision pipeline, the transfer claim is weakened. Alternatively, replace the ACE/SWiG annotations with shuffled or coarse labels that preserve the task format but destroy event semantics; if RMPL still shows most of its gain, the improvement comes from format adaptation, not from event knowledge.","tokens_in":15886,"feed_emoji":"🖼️","tokens_out":4464,"duration_ms":39784,"temperature":0.7,"pith_summary":"The paper claims that multimedia event extraction can be learned without any annotated multimedia event data by training in two stages. In the first stage, a vision-language model is warmed up on a unified schema covering textual event extraction, visual event extraction, and multimedia relation extraction. In the second stage, it is fine-tuned separately for event mention identification and argument role extraction on mixed text and image supervision. On the M2E2 benchmark, this raises event-mention and argument-role F1 across three different VLM backbones and in text-only, image-only, and multimedia evaluation settings. The significance is that it offers a route around the field's core bottleneck: M2E2 is the only benchmark and has no training split.","feed_headline":"Two-stage training lifts multimedia event F1 to 92.4","feed_subtitle":"Warming up on text, image, and relation tasks then specializing beats prompt-only extraction on every backbone tested.","key_machinery":"The key mechanism is stage-wise progressive learning with a unified schema. A schema control variable selects among three supervision families (textual event extraction, visual event extraction, multimedia relation extraction), all serialized into the same autoregressive generation format. Stage I trains the model to predict these schemas from the document; Stage II fine-tunes the warmed-up model, without relation supervision, on the two MEE subtasks using mixed textual and visual data. The auxiliary relation extraction is what makes the initialization 'relation-aware', and the ablation attributes part of the final gain to it.","core_discovery":"The central discovery is that structured, event-centric representations for multimedia event extraction can be transferred from heterogeneous unimodal and relational supervision sources rather than from in-domain annotations. RMPL first trains the VLM to generate linearized event schemas for text, images, and relation triples under a shared interface, then specializes it with task-specific supervised fine-tuning. On M2E2, the full recipe outperforms both prompt-only inference and warm-up-only training for all three backbones tested; for example multimedia event-mention F1 reaches 92.44 on one backbone and argument-role F1 rises by up to 22.87 absolute points over the prompt-only baseline.","pith_inferences":["The paper's mapping from ACE 2005 event types and SWiG visual activities to M2E2's 8 event types and 15 roles is stated but not detailed; if that mapping is lossy, part of the reported gain could be an artefact of the projection rather than of the progressive-learning paradigm.","A testable extension the paper does not run: swap the relation-extraction corpus for a different multimedia relation dataset to see whether the relation-aware warm-up benefit is tied to that particular corpus's schema.","One implication left implicit is that the same staged recipe may transfer to other low-resource multimodal tasks such as multimodal named entity recognition or multimodal sentiment analysis, where heterogeneous unimodal supervision is available.","The paper does not isolate whether Stage II's gains come from supervised task format exposure or from the warm-up's representation; a variant that skips warm-up but uses the same Stage II data shows warm-up matters, but the boundary between format learning and semantic learning is not drawn."],"forward_implications":["If the claim holds, MEE can be tackled in low-resource settings using existing unimodal event datasets plus a relation-extraction corpus, removing the need to annotate multimedia events.","The same two-stage recipe could be applied to other multimodal information-extraction tasks whose benchmarks lack training splits.","Relation-aware warm-up is presented as a generally useful initialization: even when Stage II does most of the work, the warm-up stabilises and improves the final model.","The gains are consistent across three VLM backbones, suggesting the training strategy transfers rather than being tuned to one model.","Direct comparison against prior MEE methods shows best or near-best F1 across most modality settings, so the paradigm is competitive with dedicated cross-modal alignment and prompting methods."],"fun_headline_variants":["Warm-up on relations lifts multimedia event F1 to 92.44","Two-stage pretraining beats prompt-only on M2E2 by 22.87","Event extraction? Pretrain on text, images, and relations first","Relational warm-up boosts argument-role F1 by 22.87 points","Multimedia events: shared-schema pretraining outperforms prompting"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"Everything rests on the assumption that ACE 2005 textual events, SWiG visual activities, and MNRE relations can be projected onto M2E2's 8 event types and 15 roles without systematic loss; the paper asserts this mapping in Section 5.1 but provides no mapping details or coverage statistics.","fun_headline_variants_meta":{"raw":{"variants":["Warm-up on relations lifts multimedia event F1 to 92.44","Two-stage pretraining beats prompt-only on M2E2 by 22.87","Event extraction? Pretrain on text, images, and relations first","Relational warm-up boosts argument-role F1 by 22.87 points","Multimedia events: shared-schema pretraining outperforms prompting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000855,"raw_usage":{"total_tokens":3539,"prompt_tokens":718,"completion_tokens":2821,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":462,"completion_tokens_details":{"reasoning_tokens":2722}},"tokens_in":462,"tokens_out":2821,"duration_ms":18378,"temperature":1.0,"reasoning_tokens":2722,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T23:25:37.377638+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a held-out subsample of M2E2 as a training split and train the same VLM directly on it; if direct supervision does not beat RMPL's external-supervision pipeline, the transfer claim is weakened. Alternatively, replace the ACE/SWiG annotations with shuffled or coarse labels that preserve the task format but destroy event semantics; if RMPL still shows most of its gain, the improvement comes from format adaptation, not from event knowledge.","supporting_citations":[],"review_version":1}