{"id":"27391d0d-f69e-438f-8c71-697b6100ee09","arxiv_id":"2411.12790","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A memory-based editor with a multimodal scope classifier improves focused editing of one object in multi-entity images, supported by a new benchmark.","lead":"The paper introduces a benchmark and a method for updating a fact about one object in an image without disturbing answers about other objects in the same image. A reader might care because it addresses a practical weakness in multimodal AI assistants: making precise corrections without retraining.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Specificity is not well-defined: the in-scope target answers y'_in in Eq. 13 are never generated or documented, so the reported specificity gains cannot be interpreted.","rationale":"The reader's verdict is already CONDITIONAL and correctly flags the unvalidated GPT-generated scope labels and the unspecified choice of ye. My stress-test goes one step deeper: even if the GPT in/out scope labels were perfect, Eq. 13 is still undefined because the desired counterfactual answer y'_in for in-scope questions is never specified in Section 4.2 or elsewhere. This is not a disagreement with the reader but a more targeted version of the same label-validity concern. I therefore do not change the verdict; I would make the condition explicit: the authors must release the generation protocol for y'_in and validate it on a human-annotated subset. The method itself is a reasonable and clearly described extension of SERAC, and the transferability and computational-cost analyses are positive supporting evidence, but the headline metric needs a defined ground truth before the reported specificity gains can be accepted.","tokens_in":12104,"tokens_out":7969,"duration_ms":81646,"concrete_test":"Obtain (or ask authors to release) the FGVEdit hard in-visual-scope subset with the exact target y'_in used for each question. Independently generate counterfactual answers for a random 200-sample subset using a different annotator (human or a different LLM) given the edit (ie, te, ye), then recompute M^v_in and Specificity from Eq. 13. If the released labels are just the original VQA answers, or if the recomputed M^v_in drops materially (e.g., more than 5 points) relative to Table 1, the Specificity claim is an artifact of undefined or incorrect target labels.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim rests on the new Specificity metric (Eq. 12-14). Computing M^v_in in Eq. 13 requires a label y'_in for every hard in-visual-scope question under the counterfactual edit (ie, te, ye). Section 4.2 describes only how questions are partitioned into in/out-of-scope using GPT-4o-mini entailment judgments; it never describes how the edited answers y'_in are produced. The source data is VQAv2, whose answers are pre-edit ground truths. If y'_in is taken to be the original VQA answer, the metric penalizes a model that correctly applies the edit (e.g., answering 'kite' instead of the original 'stuffed elephant'), so the high MSCKE specificity numbers would be meaningless. If y'_in is instead generated by GPT-4o-mini, that generation step is undocumented and unvalidated, and the same model was already used to build the training labels for the scope classifier. In either case, the Specificity numbers in Table 1 do not currently measure what the paper claims: whether the edited model gives the correct counterfactual answer within the fine-grained visual scope. Without released data or code specifying y'_in, the headline improvement of MSCKE over SERAC (61.60 vs 31.92 on BLIP-2 OPT) is not verifiable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a 'visual-oriented fine-grained' knowledge editing task for multimodal large language models, in which an edit targets a specific entity within an image containing multiple interacting entities. It presents the FGVEdit benchmark, built from VQAv2 by using GPT-4o-mini entailment judgments to partition questions into in-scope and out-of-scope sets relative to the first question for each image. The proposed MSCKE method extends SERAC by replacing its text-only scope classifier with a multimodal classifier that fuses CLIP image and text features, routing in-scope inputs to a counterfactual model and out-of-scope inputs to a frozen base model. Experiments on BLIP-2 OPT and MiniGPT-4 report improvements on a new Specificity metric over text-only baselines, along with analyses of classifier design and transferability.","tokens_in":12380,"tokens_out":6062,"duration_ms":56407,"significance":"If the evaluation were made rigorous, the task definition would be a reasonable extension of knowledge editing beyond treating images as single entities, and the multimodal scope classifier is a natural adaptation of memory-based editing that could be practically useful. The reported transferability of the classifier across counterfactual models and base MLLMs is a valuable property. However, the central quantitative evidence is currently not verifiable: the Specificity metric's in-scope target labels are unspecified, the evaluation labels are also used to train the classifier, and no variance or significance information is reported.","major_comments":[{"comment":"The in-visual-scope target labels y'_in used in Eq. (13) are never defined or documented. Section 4.2 describes only how questions are partitioned into hard in-visual-scope and hard out-of-visual-scope using GPT-4o-mini entailment judgments; it does not state how the edited answers y'_in are produced for those in-scope questions. If y'_in is taken to be the original VQAv2 ground-truth answer, the metric would penalize a correctly edited model (e.g., one that answers 'kite' after the edit instead of 'stuffed elephant'), making the reported Specificity gains meaningless. If y'_in is generated by GPT-4o-mini, that generation step is undocumented and unvalidated. Similarly, the edit target y_e for the first question (the editing sample) is not specified; the paper says the first question serves as the editing sample but does not state what the new answer is or how it is obtained. The authors must specify the generation protocol for both y_e and y'_in, and ideally release the evaluation data so that the headline numbers in Tables 1 and 3 can be independently verified.","section":"Section 4.1, Eq. (13) and Section 4.2"},{"comment":"The scope classifier is trained with the same GPT-4o-mini-generated in/out-of-scope labels that are later used to compute the Specificity metric in Section 4.1. This creates a circularity: the classifier is trained to reproduce the very labels used to measure its success, so the reported classification improvements and the resulting Specificity gains could be an artifact of training to match the evaluator. The authors should provide a human-validated held-out set for evaluation, or at minimum report inter-annotator agreement between GPT-4o-mini and human annotators for the entailment judgments and for any generated target answers.","section":"Section 3.3 and Section 4.2"},{"comment":"The claim that MSCKE 'outperforms existing methods' is not supported on the headline Specificity metric: MEND achieves 65.85 on BLIP-2 OPT and 67.39 on MiniGPT-4, both higher than MSCKE's 61.60 and 57.20. The paper should either restrict the claim to 'outperforms text-only editing methods' or explain why MEND is not the primary comparison. As written, the abstract and introduction overstate the results relative to the table.","section":"Section 5.2, Table 1"},{"comment":"No error bars, variance, or significance tests are reported for any metric. Since the differences between MSCKE-MEND and MEND are modest (e.g., 68.38 vs 65.85 on BLIP-2 OPT) and the evaluation is performed on a single train/test split of the FGVEdit dataset, it is unclear whether the reported improvements are stable. The authors should report means and standard deviations over multiple runs or provide statistical significance tests for the key comparisons.","section":"Section 5.2, Table 1"},{"comment":"The selection rule k* = arg min_k fcls(...) is inconsistent with treating ρ as a similarity in [0,1] where values ≥ 0.5 are in-scope. If fcls outputs a similarity, the most similar editing example should be selected by arg max; if fcls outputs a distance, then the threshold and the loss in Eq. (11) need to be reinterpreted. As written, the algorithm is not implementable as stated. This appears to be an easily corrected error, but it must be fixed for the method description to be coherent.","section":"Section 3.3, Eqs. (1)-(2)"}],"minor_comments":[{"comment":"The sentence describing the training data says 'the dataset provides both in-scope inputs (i^k_in, t^k_in) and out-of-scope inputs (i^k_in, t^k_in)'; the second pair should be (i^k_out, t^k_out).","section":"Section 3.3, text after Eq. (11)"},{"comment":"The heading 'Locolity' should be 'Locality'.","section":"Table 3"},{"comment":"The caption says results are presented as 'transferred/retrained' but does not explain what the slash denotes; please clarify the notation in the caption.","section":"Table 3 caption"},{"comment":"The Locality metric is defined on text-only inputs t_l while the rest of the paper emphasizes multimodal inputs; this choice should be stated explicitly. For Generality, the comparison in Eq. (17) is to f_{θ_e}(i_e, t_e) rather than to the target y_e; this is only meaningful when reliability holds, so it would be clearer to define M_gen against y_e.","section":"Section 4.1, Eqs. (16)-(17)"},{"comment":"The paper says 'we manually construct two prompts'; the classification is performed by GPT-4o-mini, not by human annotators, so this phrasing is misleading and should be changed to something like 'we use two prompts to query GPT-4o-mini'.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The idea of extending SERAC with a multimodal scope classifier is plausible and the transferability analysis is a useful contribution, but the evaluation is currently unverifiable because the target labels for the new Specificity metric are not defined and the metric shares its labels with the classifier's training signal. The authors should be asked to provide the data generation details and an external human-validated evaluation set before the claims can be accepted. The paper also overstates its results relative to MEND; this should be corrected in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is a plausible extension of memory-based editing to multimodal fine-grained scenarios, but the headline claim rests on a metric that is not actually defined. I agree with the reader's conditional verdict, and I'd go further than the stress-test: the Specificity numbers in Table 1 are uninterpretable as reported.\n\nWhat is new: FGVEdit is the first benchmark I know of that edits knowledge about multiple interacting entities in one image. The scope classifier is a CLIP-based replacement for SERAC's text-only classifier, and the classification-accuracy comparison in Figure 4 is a direct, useful measurement that doesn't depend on the problematic metric. The transferability result (Table 3) is also a nice practical addition.\n\nThe soft spots are real. Most importantly, Eq. 13 defines M_v_in using labels y'_in for hard in-visual-scope questions, but the paper never says where these edited answers come from. If they're the original VQA answers, then a model that correctly applies the edit (e.g., \"kite\" instead of \"stuffed elephant\") is penalized. If they're generated by GPT-4o-mini, that step is undocumented and the same model already produced the scope labels used to train the classifier, so the evaluation is circular. Either way, the 61.60 vs 31.92 improvement cannot be verified. This is not a minor footnote; it's the paper's central claim.\n\nThe edit target construction is also under-specified. Taking the first VQA question as the edit sample with the original answer as ye assumes the base model initially answers incorrectly, but there's no check that yo ≠ ye. Some of the high reliability scores may be trivial.\n\nNo error bars, no human validation of the GPT-generated labels, and no released data or code make the empirical results hard to trust. That said, the conceptual contribution — a benchmark for multi-entity visual editing and a multimodal scope classifier — is real and could become a useful resource if the evaluation is cleaned up. I'd send it to review, but I'd expect heavy revision; the authors should release the dataset, specify exactly how y'_in is constructed, and add human agreement statistics.\n\nFor a reading group, it's worth an hour to talk about metric construction, but I wouldn't cite it in its current form.\n\nRecommendation: send to peer review with a request for major revision.","headline":"Useful benchmark and a sound idea, but the Specificity metric is under-specified and the headline numbers can't be interpreted as reported.","tokens_in":12904,"tokens_out":3570,"would_cite":false,"duration_ms":33753,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Replacing a text-only edit router with one that also sees the image lets multimodal LLMs update one entity in a multi-entity scene without disturbing the rest, and the paper measures a near-doubling of fine-grained specificity on two…","keywords":["fine-grained knowledge editing","multimodal large language models","multimodal scope classifier","visual question answering","specificity metric","counterfactual model","FGVEdit benchmark","entity-level editing"],"falsifier":"Take a random subset of FGVEdit specificity pairs, have several human annotators decide whether a change in the edit answer would affect the target question's answer, then recompute Specificity from the human labels; if the MSCKE versus SERAC gap (29.68 points on BLIP-2 OPT and 19.35 on MiniGPT-4) shrinks or reverses, the reported improvement is an artifact of the GPT-4o-mini labels.","tokens_in":11931,"feed_emoji":"🎯","tokens_out":5261,"duration_ms":46963,"temperature":0.7,"pith_summary":"This paper sets out to establish that knowledge editing for multimodal LLMs should operate at the level of individual entities inside an image, not at the level of the whole image, and that doing so requires the editing system to use vision, not just text. To test this, the authors build the FGVEdit benchmark from VQA pairs on images with several interacting objects, and propose MSCKE, a memory-based editor whose scope classifier fuses CLIP image and text features to decide whether a query is about the edited entity. On FGVEdit, MSCKE lifts the new Specificity score from 31.92 to 61.60 on BLIP-2 OPT and from 37.85 to 57.20 on MiniGPT-4 relative to the text-only SERAC baseline, while keeping locality at 100. The paper's point is that visual information is load-bearing for fine-grained multimodal editing, and that a lightweight multimodal router is enough to deliver it.","feed_headline":"Visual scope classifier lifts fine-grained edit specificity to 61.6","feed_subtitle":"Text-only editors reach 31.9 on the same task; adding image-text fusion to the router nearly doubles precision.","key_machinery":"The load-bearing component is the Multimodal Scope Classifier. It maps the image and text of both the query and each stored edit example into a shared CLIP feature space, aligns the two modalities, fuses them with dot-product attention so the text highlights the relevant parts of the image, and returns a cosine similarity between the query's fused representation and the edit example's fused representation. Inputs with similarity below 0.5 go to the frozen base model; inputs at or above 0.5 go to the counterfactual model, which is conditioned on the stored edit example. The classifier is trained as a binary classifier with cross-entropy loss on in-scope and out-of-scope pairs, and it can also accept text-only inputs by setting the image to null.","core_discovery":"The paper's central claim is that in multimodal knowledge editing, the question of which inputs are in scope must be decided by both the image and the text, and that a text-only scope classifier cannot reliably separate questions about the edited entity from questions about other entities in the same image. Concretely, MSCKE stores edit examples in memory, uses a multimodal scope classifier (CLIP encoders, an alignment projection, dot-product attention fusion, and a similarity threshold of 0.5) to route inputs either to the frozen base model or to a counterfactual model, and thereby achieves Specificity of 61.60 on BLIP-2 OPT and 57.20 on MiniGPT-4, versus 31.92 and 37.85 for SERAC. When the counterfactual model is replaced by MEND, Specificity rises further to 68.38 and 71.98. Reliability, locality, and generality stay at or near the level of the strongest baselines, with locality reaching 100 on both base models.","pith_inferences":["Inference: if visual routing is the key mechanism, replacing the coarse similarity threshold with entity localization (for example, attention maps or detection boxes) could make MSCKE robust on denser scenes and on entities that look alike.","Inference: the Specificity gains should be re-measured with human-validated entailment labels; the current labels come from a single GPT-4o-mini call, and a disagreement study would show how much of the reported gap is a labeling artifact.","Inference: the router design suggests a cheap deployment path for continual updates in MLLMs: keep the frozen base model, maintain a growing memory of edits, and swap only the lightweight classifier and counterfactual model when the base model changes.","A testable extension is to train the classifier on hard out-of-visual-scope pairs where text alone is ambiguous and see whether Specificity improves further on a held-out split."],"forward_implications":["On the FGVEdit benchmark, MSCKE reduces over-editing: text-only baselines disturb answers to out-of-visual-scope questions, while MSCKE keeps locality at 100 on both BLIP-2 OPT and MiniGPT-4.","The multimodal scope classifier transfers across base models and counterfactual editors without retraining, performing nearly identically to a classifier retrained per setting.","Because the classifier handles text-only inputs, the same editing framework covers both text knowledge edits and visual fine-grained edits.","The classifier adds little overhead: it uses far less inference time and memory than the base or counterfactual models, so the precision gain does not come at high compute cost.","Specificity, the new metric, shows that standard reliability, locality, and generality scores miss a failure mode where edited knowledge leaks to unrelated entities in the same image."],"supporting_citations":[{"why":"Supplies the memory-based SERAC architecture whose text-only scope classifier MSCKE replaces.","marker":"[20]"},{"why":"Provides the pre-trained CLIP image and text encoders used to build the multimodal scope classifier.","marker":"[21]"},{"why":"VQAv2 is the source of question-answer pairs from which FGVEdit constructs images with multiple entities.","marker":"[8]"},{"why":"MMEdit defines the locality data construction approach reused in FGVEdit.","marker":"[4]"},{"why":"KEBench supplies the fine-tuning setup for baseline methods.","marker":"[9]"},{"why":"MEND serves as an alternative parameter-modifying counterfactual model in the MSCKE-MEND variant.","marker":"[19]"},{"why":"Natural Questions provides out-of-scope question-answer pairs for the locality metric.","marker":"[11]"},{"why":"Supplies the implementation scaffolding for the SERAC, MEND, and IKE baselines.","marker":"[25]"}],"fun_headline_variants":["Visual-aware scope classifier nearly doubles multimodal edit precision","MSCKE specificity hits 61.6 on BLIP-2, up from 31.9","Text-only routers lag in fine-grained vision edit; visual fusion fixes it","Multimodal scope classifier boosts edit precision to 61.6 from 31.9","Fine-grained visual editing: MSCKE hits 61.6 specificity, SERAC 31.9"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The Specificity metric is computed from GPT-4o-mini's entailment judgments about which questions are in or out of the visual editing scope, and there is no human validation or agreement statistic reported, so the benchmark's labels are the load-bearing premise for the reported gains.","fun_headline_variants_meta":{"raw":{"variants":["Visual-aware scope classifier nearly doubles multimodal edit precision","MSCKE specificity hits 61.6 on BLIP-2, up from 31.9","Text-only routers lag in fine-grained vision edit; visual fusion fixes it","Multimodal scope classifier boosts edit precision to 61.6 from 31.9","Fine-grained visual editing: MSCKE hits 61.6 specificity, SERAC 31.9"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000762,"raw_usage":{"total_tokens":3390,"prompt_tokens":960,"completion_tokens":2430,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":2322}},"tokens_in":576,"tokens_out":2430,"duration_ms":17944,"temperature":1.0,"reasoning_tokens":2322,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:23:25.154532+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random subset of FGVEdit specificity pairs, have several human annotators decide whether a change in the edit answer would affect the target question's answer, then recompute Specificity from the human labels; if the MSCKE versus SERAC gap (29.68 points on BLIP-2 OPT and 19.35 on MiniGPT-4) shrinks or reverses, the reported improvement is an artifact of the GPT-4o-mini labels.","supporting_citations":[{"cited_title":"Memory-based model editing at scale","cited_arxiv_id":null,"evidence_quote":"Supplies the memory-based SERAC architecture whose text-only scope classifier MSCKE replaces."},{"cited_title":"Learn- ing transferable visual models from natural language super- vision","cited_arxiv_id":null,"evidence_quote":"Provides the pre-trained CLIP image and text encoders used to build the multimodal scope classifier."},{"cited_title":"Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing","cited_arxiv_id":null,"evidence_quote":"VQAv2 is the source of question-answer pairs from which FGVEdit constructs images with multiple entities."},{"cited_title":"Can we edit multimodal large language models? In EMNLP, 2023","cited_arxiv_id":null,"evidence_quote":"MMEdit defines the locality data construction approach reused in FGVEdit."},{"cited_title":"Fast model editing at scale","cited_arxiv_id":null,"evidence_quote":"MEND serves as an alternative parameter-modifying counterfactual model in the MSCKE-MEND variant."},{"cited_title":"Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov","cited_arxiv_id":null,"evidence_quote":"Natural Questions provides out-of-scope question-answer pairs for the locality metric."}],"review_version":1}