{"id":"fbe3287e-7d2f-4645-8b4e-b76182856e6f","arxiv_id":"2412.12821","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A new eight-task benchmark with in-domain metrics KGI and KPI reveals that existing multimodal editing methods degrade on related samples, and the proposed HICE method achieves a better balance.","lead":"This paper introduces ComprehendEdit, a benchmark of eight tasks for testing how well multimodal AI models can have their knowledge corrected, plus two new metrics that measure spillover effects on related questions. It also proposes HICE, a method that balances corrections with preserving unrelated answers, and reports that existing editing methods perform poorly on the new metrics.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported T-KPI values in the E-VQA/MiniGPT-4 ablation tables (46.2) exceed the stated metric maximum ([0,25]) for that setting, so the quantitative evidence for HICE's superior KPI is internally inconsistent and unverifiable.","rationale":"The reader's weakest assumption concerns representativeness of the k=4 nearest/farthest subsample, which is a variance and precision issue. The concern I raise is more fundamental: the published numbers for the very metric that supports the central claim have an undefined or contradictory scale. This is an internal inconsistency (rule 6), not a disagreement with the field's consensus. If the T-KPI values exceed the stated maximum range, either the metric definition in Eq. (7) is not what was computed, or the reported ranges are wrong; in either case, the evidence for the paper's main comparative claim is not self-consistent. This does not automatically invalidate the benchmark dataset or the KGI/KPI idea, but it makes the quantitative evaluation impossible to trust without correction, preserving the CONDITIONAL verdict. A concrete fix would be to release a full numerical results table and reconcile the metric bounds with Eq. (7). The paper does provide a code repository and a thoughtfully constructed multi-task benchmark, which are real contributions, but the central empirical claim is currently undermined by the scaling inconsistency.","tokens_in":18141,"tokens_out":11015,"duration_ms":103880,"concrete_test":"Run the released ComprehendEdit code to recompute T-KPI and I-KPI on E-VQA with MiniGPT-4, following Eq. (7) exactly; verify whether the maximum attainable value is 25 or 100 and whether HICE's value equals 46.20. Then compare against the y-axis scale of Figure 3: if the axis maximum is 25, a bar at 46.20 cannot appear, indicating a normalization mismatch. Submit a corrected table with explicit units (proportion vs. percentage vs. count) and check whether HICE still leads on KPI under the corrected scale.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The quantitative backbone of the central claim is unreliable as reported. In the Experimental Results section, the Figure 3 caption states that on E-VQA the values of I-KGI, T-KGI, I-KPI, T-KPI lie in [0,16] for BLIP-2 OPT and [0,25] for MiniGPT-4. However, Table 2 (E-VQA, MiniGPT-4) reports T-KPI = 46.20 for HICE, and the hyperparameter ablations in Tables 9-12 report T-KPI values between 45.87 and 46.80 for the same backbone and benchmark. Since MKPI in Eq. (7) is an expectation of an indicator function, its valid range is [0,100]; a value of 46.20 is mathematically possible only on a [0,100] scale, not a [0,25] scale. Either the table values are computed with a different normalization than Eq. (7) (e.g., raw counts instead of proportions), the figure's stated maximal ranges are misprinted, or the evaluation script contains a scaling bug. No numerical tables are provided for the main results in Figure 4, so the ComprehendEdit results cannot be cross-checked against this inconsistency. Consequently, the paper's headline assertion that HICE demonstrates superior and more balanced performance on the newly introduced metrics is not currently verifiable from the published numbers.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ComprehendEdit, a multimodal knowledge editing benchmark covering eight tasks derived from five datasets (GQA, TallyQA, VSR, TextVQA, MathVista). It proposes two new metrics, Knowledge Generalization Index (KGI) and Knowledge Preservation Index (KPI), to measure editing effects on in-domain samples. It also presents Hierarchical In-Context Editing (HICE), a two-stage method that first classifies inputs as in-domain or out-of-domain and then applies in-context editing only to in-domain inputs. The authors compare FT-V, FT-L, IKE, SERAC, and MEND across conventional metrics and the new KGI/KPI metrics, reporting that existing methods struggle on the new metrics while HICE achieves more balanced performance.","tokens_in":18486,"tokens_out":7490,"duration_ms":65348,"significance":"If the results are reliable, this work fills a genuine gap: current multimodal knowledge editing evaluation (e.g., MMEdit) relies heavily on synthetic content and out-of-domain locality, ignoring in-domain side effects. The KGI/KPI metrics are a reasonable and well-motivated attempt to quantify these effects, and the proposed benchmark broadens task coverage beyond the narrow object-recognition focus of prior evaluations. The HICE baseline is clearly specified and the dataset construction process is documented in detail. However, the quantitative evidence for HICE's superiority is currently undermined by an internal inconsistency in the reported metric ranges and by the unvalidated sampling approximation used to compute KGI/KPI. The significance is therefore conditional on the authors resolving these evaluation issues.","major_comments":[{"comment":"Figure 3's caption states that I-KGI, T-KGI, I-KPI, and T-KPI lie in [0,16] for BLIP-2 OPT and [0,25] for MiniGPT-4, and Figure 4's caption states [0,40] for ComprehendEdit. However, Table 2 (E-VQA, MiniGPT-4) reports T-KPI = 46.20 for several rows and T-KPI = 46.34 for HICE, and Tables 9-12 report T-KPI values between 45.87 and 46.80 on the same backbone and benchmark. Since MKPI in Eq. (7) is an expectation of an indicator function, its valid range on a percentage scale is [0,100], and a value of 46.20 cannot lie in [0,25]. Either the figure captions misstate the metric maxima, or the table values use a different normalization (e.g., raw counts) that does not match Eq. (7). As written, the quantitative support for the claim that HICE achieves superior and more balanced KPI is not verifiable. Please correct the ranges, state the exact normalization used, and report the numerical values for Figures 3 and 4.","section":"Results, Figure 3 caption; Table 2"},{"comment":"Equations (6) and (7) define KGI and KPI as expectations over all in-domain samples in DKGI(s) and DKPI(s). The subsequent paragraph, 'Similarity-based Sampling', replaces these sets with only the k=4 nearest and k=4 farthest neighbors selected by CLIP similarity. The paper neither defines this sampled metric formally nor provides a convergence check, confidence interval, or comparison with full-set evaluation. This is load-bearing because the central finding that existing methods underperform in-domain is based on this 8-neighbor subsample. Please either define the sampled metric explicitly (e.g., KGI_k, KPI_k) and report its uncertainty, or validate the approximation by computing both full-set and sampled estimates on a subset of editing samples.","section":"Proposed Evaluation Metrics, 'Similarity-based Sampling'"},{"comment":"In 'Implementation Details', HICE's hyperparameters (λ, M, T, k0, memory ratio) are selected using a validation split of the training set. In contrast, the baseline methods (FT-V, FT-L, IKE, SERAC, MEND) are run with fixed settings taken from prior work, as listed in Appendix Tables 5-8. This asymmetry gives HICE a potential advantage, especially on the newly introduced KGI/KPI metrics for which the baselines were not tuned. To ensure a fair comparison, please tune each baseline on the same validation split of ComprehendEdit, or report both fixed and tuned baseline results so the reader can gauge the effect.","section":"Implementation Details; Ablation Study"},{"comment":"The definition of D(s) as 'samples from the same source dataset as s' is ambiguous regarding whether it includes training samples. If D(s) includes the training split, then KGI/KPI evaluation for HICE can be inflated because HICE's memory M1 and classifier are constructed from that same training split. The paper must specify whether D(s) is test-only or train+test, and must confirm that no in-domain evaluation sample overlaps with the samples used to build HICE's memory or classifier. If overlap exists, the reported KGI/KPI improvements for HICE may partly reflect memorization rather than editing generalization.","section":"Task Formulation; Similarity-based Sampling"},{"comment":"The main results in Figures 3 and 4 are presented as bar charts with single numerical labels and no error bars, standard deviations, or significance tests. Because each test sample is edited independently, the variability across editing samples is expected to be substantial. Please report means with standard deviations (e.g., across editing samples or multiple runs), or at least provide a table of the underlying numbers so that the differences between methods can be assessed quantitatively.","section":"Experimental Results (Figures 3 and 4)"}],"minor_comments":[{"comment":"The captions of Figures 5 and 6 refer to 'Eq. (11)' and 'Eq. (2)', but the main text contains equations numbered up to (9); please renumber or remove these cross-references.","section":"Appendix, Figures 5-7 captions"},{"comment":"The text mentions a '†' symbol to indicate reproduced results, but the symbol does not appear in Table 2 or its caption; please add the appropriate footnote or remove the reference.","section":"Table 2 caption"},{"comment":"There is a typo in 'performannce' (should be 'performance'); also, in the Ablation Study section, 'rephrase questions' should be 'rephrased questions'.","section":"Results section"},{"comment":"The ablation study uses '1 nearest and 1 farthest neighbor' for KGI/KPI, whereas the main experiments use k=4; please clarify whether the conclusions are sensitive to this choice and how the two settings relate.","section":"Ablation Study"},{"comment":"The phrase 'the original model answered incorrectly (exclude s)' is ambiguous: it is unclear whether s is excluded only from DKGI(s) or from both DKGI(s) and DKPI(s); please state the exclusion rule explicitly.","section":"Proposed Evaluation Metrics"},{"comment":"Table 3 is not referenced in the main text; please add a cross-reference so readers know where to find the comparison with E-VQA and KEBench.","section":"Appendix, Table 3"}],"recommendation":"major_revision","confidential_remarks":"The benchmark and the motivation for in-domain evaluation are timely, and the dataset resource could be useful to the community. However, the metric-range inconsistency and the unvalidated sampling shortcut directly affect the paper's central quantitative claim. I do not see evidence of intent to mislead, but the reporting must be corrected and the code/data should be made available for verification. I recommend asking the authors to re-run the evaluations with consistent metric definitions, provide numerical tables with error bars, and clarify the composition of the in-domain evaluation sets."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, the benchmark is real: eight tasks across five datasets, assembled with care, and the KGI/KPI idea—checking editing effects on real in-domain samples instead of synthetic rephrasings—is a legitimate gap in the MMEdit/KEBench line. Second, the numbers don't add up. The Figure 3 caption says KGI/KPI live in [0,16] on BLIP-2 and [0,25] on MiniGPT-4, but Table 2 (E-VQA, MiniGPT-4) gives T-KPI as 46.20. Since Eq. (7) is an expectation of an indicator, 46.20 is only possible on a [0,100] scale. Someone is wrong: the caption, the table, or the eval script. And because Figure 4 has no numerical table, you can't cross-check the ComprehendEdit results. That's a load-bearing problem—the headline claim 'HICE superior and balanced on KPI' rides on those numbers.\n\nWhat's actually good: the dataset construction is thoughtful (answer-type balancing, avoiding synthetic-image content shifts), the metrics are simple enough to be adopted, and HICE is a straightforward baseline combining IKE with a classifier. The empirical finding that prior methods collapse on KGI/KPI is likely true in spirit, but the current evidence is too shaky to cite.\n\nSofter issues: no error bars, no exact numbers in Figure 4, and HICE's hyperparameters are tuned on a validation split while baselines use fixed settings from prior work. Those are standard issues, fixable. The range contradiction is the one that needs a clear answer.\n\nWho's it for: the multimodal editing crowd. If the numbers get corrected, this could become a standard benchmark. Right now, I'd want the authors to release a revised version with full tables and error bars before I'd trust the comparison. It deserves a referee—not desk rejection—because the resource and framing are valuable, but acceptance should be conditional on resolving these issues.","headline":"Useful benchmark and metrics for in-domain multimodal editing evaluation, but the reported numbers contain an internal range inconsistency that makes the central HICE claim unverifiable as printed.","tokens_in":18969,"tokens_out":4100,"would_cite":false,"duration_ms":36955,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that current multimodal knowledge editing evaluations are incomplete and biased, and that new in-domain metrics reveal hidden damage to nearby knowledge that older metrics overlook.","keywords":["multimodal knowledge editing","benchmark","in-domain generalization","knowledge preservation","in-context learning","visual question answering","evaluation metrics"],"falsifier":"Run KGI and KPI exhaustively over all in-domain samples rather than only the $k=4$ nearest and farthest neighbors for several editing methods. If the relative rankings change, or if a method the paper reports as weak matches HICE under full evaluation, the central claim fails.","tokens_in":17971,"feed_emoji":"🧠","tokens_out":9124,"duration_ms":81967,"temperature":0.7,"pith_summary":"Knowledge editing for multimodal models is usually judged by whether the edited answer changes, whether rephrased questions still get the new answer, and whether unrelated questions are unchanged. The paper argues this misses what matters most: what happens to nearby, same-domain knowledge. To close that gap it builds ComprehendEdit, an eight-task benchmark of visual question answering, and adds two metrics computed on real in-domain samples—Knowledge Generalization Index (does the edit help other samples the model previously got wrong?) and Knowledge Preservation Index (does the edit preserve samples the model previously got right?). On this benchmark, methods that looked strong in earlier evaluations (fine-tuning, IKE, SERAC, MEND) score poorly on the new metrics, while the paper's baseline, Hierarchical In-Context Editing (HICE), matches prior methods on older metrics and balances the new ones. If these results hold, future editing methods need to be judged on in-domain side effects, not only on editing accuracy and out-of-domain locality.","feed_headline":"AI knowledge edits break nearby answers—new metric catches it","feed_subtitle":"Across eight visual tasks, editing methods flunk newly tested in-domain measures; a two-stage baseline restores balance.","key_machinery":"ComprehendEdit supplies the testbed: 17,932 samples across eight tasks built from five source datasets, with a 3:1 train/test split and rephrased questions plus text and multimodal locality samples attached to each editing sample. The two proposed metrics partition the in-domain samples of each edited sample by the original model's behavior: $D_{KGI}(s)$ holds samples the original model answered incorrectly and $D_{KPI}(s)$ holds samples it answered correctly, with KGI and KPI averaging edited-model accuracy over those sets. To keep this affordable, the paper samples the $k=4$ nearest and farthest neighbors of each edited sample by CLIP feature distance, yielding image-based and text-based variants I-KGI/I-KPI and T-KGI/T-KPI. HICE is the method: a pretrained text encoder $h$, a random projection $W_r$ to dimension $M=10{,}000$, a ridge classifier $W^*=(F_p^\\top F_p+\\lambda I)^{-1}F_p^\\top Y$, a text memory $M_1$ of clustered demonstrations, and a memory $M_2$ of hard external samples, so that test inputs are either answered with $k_0=16$ retrieved demonstrations appended to the prompt or passed to the original model unchanged.","core_discovery":"The paper's central claim is that the standard evaluation of multimodal knowledge editing is incomplete and that the omission changes the verdict on real methods. Editing a model to answer 'parrot' instead of 'eagle' for one bird image is only part of the task; the edit should also help neighboring bird images that were misclassified and must not flip neighboring bird images that were correct. The paper reports that current methods fail on these two behaviors: fine-tuning overfits to the edited sample, IKE's demonstrations do not transfer to neighbors, SERAC underperforms on multimodal locality, and MEND's distilled loss, which uses only out-of-domain data, leaves in-domain generalization weak. HICE is reported to restore the balance by classifying each input as in-scope or out-of-scope and applying retrieved in-context demonstrations only to in-scope inputs, leaving the model parameters unchanged.","pith_inferences":["If KGI and KPI were computed exhaustively rather than on the four nearest and farthest neighbors, the numerical rankings could shift, and HICE's reported lead might shrink or grow; the paper does not provide a convergence check.","The KGI/KPI split depends on which base model defines 'correct' and 'incorrect', so the metrics are not model-independent; a different base model would assign samples to the two subsets differently.","The same in-domain evaluation idea could be applied to language-only knowledge editing, where the distinction between generalization and preservation at the distribution level is currently less emphasized.","The observed trade-off between parametric editing methods' strong reliability and their weak in-domain preservation suggests a possible structural limit: methods that change weights to fit one fact tend to disturb neighboring facts, whereas memory or in-context methods avoid this at the cost of prompt sensitivity."],"forward_implications":["Editing methods that score well on existing metrics can still silently corrupt in-domain knowledge, so benchmarks should measure KGI and KPI alongside reliability, generality, and locality.","HICE demonstrates a parameter-free editing route: a scope classifier plus in-context demonstrations can preserve out-of-domain behavior and improve in-domain measures at the same time.","The reported KGI results imply that a correct edit does not automatically transfer to similar misclassified samples, so future methods need mechanisms for propagating edits to neighbors.","The reported KPI results imply that edits can flip previously correct answers on nearby samples, so preservation of correct in-domain behavior should be a first-class objective."],"supporting_citations":[{"why":"Defines the MMEdit benchmark and the reliability, generality, and locality metrics that ComprehendEdit extends.","marker":"Cheng et al. 2023"},{"why":"Introduces IKE, the in-context editing method whose demonstration construction HICE adapts.","marker":"Zheng et al. 2023"},{"why":"Provides MEND, a meta-learning baseline, and the least-squares classifier formulation used in HICE.","marker":"Mitchell et al. 2021"},{"why":"Provides SERAC, a memory-based two-stage baseline that motivates the scope-classification design.","marker":"Mitchell et al. 2022"},{"why":"Documents content shifts in AI-generated images, motivating the no-synthetic-content design of KGI and KPI.","marker":"Huang et al. 2024"},{"why":"Supplies CLIP features used for similarity-based sampling of nearest and farthest in-domain neighbors.","marker":"Radford et al. 2021"},{"why":"Supplies the language model used to generate rephrased questions for text generality.","marker":"Du et al. 2021"},{"why":"Supplies NQ, the text-locality dataset used for out-of-domain evaluation.","marker":"Kwiatkowski et al. 2019"},{"why":"Supplies OK-VQA, the multimodal-locality dataset used for out-of-domain evaluation.","marker":"Marino et al. 2019"}],"fun_headline_variants":["Eight-task benchmark exposes AI editing blind spot","New metrics catch AI edit side effects on neighbors","Multimodal knowledge editing: current methods flunk","Two metrics reveal editing fails in-domain samples","HICE restores balance across eight editing tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that four nearest and four farthest in-domain neighbors of each edited sample, chosen by a pretrained similarity measure, represent how editing affects the whole in-domain set; if that small sample is unrepresentative, the reported findings about which methods fail may not generalize.","fun_headline_variants_meta":{"raw":{"variants":["Eight-task benchmark exposes AI editing blind spot","New metrics catch AI edit side effects on neighbors","Multimodal knowledge editing: current methods flunk","Two metrics reveal editing fails in-domain samples","HICE restores balance across eight editing tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000239,"raw_usage":{"total_tokens":1503,"prompt_tokens":924,"completion_tokens":579,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":509}},"tokens_in":540,"tokens_out":579,"duration_ms":7177,"temperature":1.0,"reasoning_tokens":509,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:41:07.585993+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run KGI and KPI exhaustively over all in-domain samples rather than only the $k=4$ nearest and farthest neighbors for several editing methods. If the relative rankings change, or if a method the paper reports as weak matches HICE under full evaluation, the central claim fails.","supporting_citations":[],"review_version":1}