{"id":"fa221d72-3b4e-433d-ad08-9c28161edc70","arxiv_id":"2506.20070","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"FemmIR uses graph-edit-distance weak supervision over extracted object properties to rank multimodal retrieval results without any similarity labels or fine-tuning.","lead":"This paper introduces FemmIR, a retrieval framework that ranks images, videos, and text by comparing detected object properties, using edit distance between property graphs as weak supervision instead of human similarity labels. It also presents HART, a text parser for extracting human attributes, and a new dataset MuQNOL built for a missing-person search scenario.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evaluation defines relevance with the same property-edit penalties that produce the CED weak labels, so the experiments cannot establish that CED captures actual retrieval relevance.","rationale":"Read in good faith, the paper builds a coherent pipeline: property extractors, HARG construction, CED weak labels, and SimGNN inference, and the pseudocode is detailed. The weak-supervision idea is interesting, and the comparison with FGCross-Net shows gains without fine-tuning. However, the central claim is empirical: CED weak labels are enough to rank relevant multimodal results. The only test of that claim is mAP on a ground truth constructed from manually weighted property mismatches—the same semantic object that CED computes. This is not an internal inconsistency; it is a validity gap. The proposed identity-label test uses data already available in MARS and would settle whether the concern lands. Because the reader already made acceptance conditional on stronger evidence, this stress-test does not change the verdict label.","tokens_in":19847,"tokens_out":6439,"duration_ms":77403,"concrete_test":"Re-evaluate FemmIR on the existing MuQNOL/MARS image and video splits using MARS tracklet identity as the relevance ground truth: for each query tracklet, rank all gallery tracklets by FemmIR's similarity and compute mAP with same person ID as positive. Compare this identity-mAP with the property-penalty mAP in Table II, with EARS under the same ground truth, and with a simple color-histogram baseline. If identity-mAP is near chance or below the attribute-overlap baseline, CED captures the surrogate property score rather than the actual missing-person relevance; if identity-mAP is high, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that CED weak supervision is sufficient to rank cross-modal relevance—has no evaluation independent of the proposed similarity function. In Section V, the MuQNOL ground truth is constructed by ranking data samples in ascending order of mismatched properties, with hand-assigned penalties rcost(TOP-COLOR)=1, rcost(BOTTOM-COLOR)=2, rcost(GENDER)=3, and the retrieval evaluation declares a sample relevant when its CED is below 3. CED is exactly the Munkres-optimized sum of those property-mismatch costs (Algorithm 1, lines 10–26), and the same CED scores generate the weak labels used to train SimGNN. Therefore the reported mAP mainly measures how well SimGNN reproduces a hand-defined penalty rule over a fixed attribute set, not whether edit distance tracks the missing-person relevance the paper claims to model. The paper's observation that image/video mAP of 27–37% tracks property-identifier F1 of 79.59% supports this reading: the weak labels themselves inherit the errors and the semantics of the predefined property/penalty layer. An independent relevance signal is required before the headline claim is supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FemmIR, a framework for multimodal retrieval by example without similarity labels. Samples are represented as hierarchical attributed relational graphs built from property identifiers; a content edit distance (CED) computed by a Munkres assignment is used as weak supervision to train SimGNN to rank query examples against candidates. The paper also introduces HART for attribute extraction from unstructured text and a benchmark, MuQNOL, built from MARS and InciText. Experiments on a missing-person use case report mAP for cross-modal retrieval and compare with EARS and FGCross-Net.","tokens_in":20236,"tokens_out":4178,"duration_ms":44848,"significance":"If the central claim were established, reusing pretrained property identifiers plus edit-distance weak labels would be a practical way to avoid similarity-label annotation in multimodal retrieval, and MuQNOL and HART would be useful resources. The paper is explicit about its weak-supervision assumptions and describes a concrete algorithm, which is a strength. However, the evaluation does not currently support the central claim because the relevance ground truth is generated from the same property-mismatch penalties used to define CED, and the main baseline EARS outperforms FemmIR on every reported row.","major_comments":[{"comment":"The retrieval evaluation is circular with the weak-label generation. The MuQNOL ground truth is created by ranking samples by mismatched properties with hand-assigned penalties rcost(TOP-COLOR)=1, rcost(BOTTOM-COLOR)=2, rcost(GENDER)=3, and relevance is declared when CED < 3. CED is exactly the Munkres-optimized sum of these same property replacement/insertion costs in Algorithm 1 (lines 10-26), and the same CED scores are used as weak labels for training SimGNN. Consequently, the reported mAP mainly measures how well SimGNN reproduces the hand-defined penalty rule on a fixed attribute set, not whether edit distance tracks the missing-person relevance claimed. The paper should evaluate against an independent relevance signal (e.g., person identity labels in MARS/InciText or human relevance judgments) and should compare against a direct weighted-mismatch ranking baseline.","section":"Section V-A and Algorithm 1"},{"comment":"Table II contradicts the contribution claim that FemmIR \"surpasses existing multimodal knowledge retrieval methods without fine-tuning.\" EARS, an exact inference model that requires no training, achieves higher mAP than FemmIR on every row (e.g., Text-to-Text 1.00 vs 0.52; all-modality average 0.44 vs 0.33). The abstract and Section VII describe \"comparable\" performance, which is accurate, but the stronger comparative claim in the introduction and contribution list is unsupported as stated. The authors should either revise the claim to match Table II or provide a fairer comparison, for example by using identical property identifiers for both systems.","section":"Section V-e, Table II"},{"comment":"The open-world generalization claim is constrained by assumptions (I) and (II): system-specific properties are assumed known, and entity types must be consistent through the system lifetime. These assumptions are reasonable for the missing-person use case, but they are not reflected in the abstract's \"open world\" framing. More importantly, because the property set is fixed and small (gender, top/bottom color, clothing, race, height), the paper does not demonstrate that edit-distance weak supervision transfers to novel attributes or modalities. A transfer experiment or an explicit limitation statement is needed to support the general claim.","section":"Section III-A"}],"minor_comments":[{"comment":"The text uses \"Munkers\" for the Munkres algorithm; please correct the spelling for consistency with the rest of the paper.","section":"Section III-B"},{"comment":"The figure caption lists panels (a) and (c), while the body text refers only to (a) and (b); the panel labels should be made consistent.","section":"Figure 4"},{"comment":"MuQNOL is presented as a new benchmark, but no download URL or access mechanism is provided; a public release statement is needed for reproducibility.","section":"Section V-A"},{"comment":"The column headers \"Clothes Attr-only\" and \"Clothes Attr-value\" appear misaligned with the values in the precision/recall/F1 rows; please reformat the table.","section":"Table I"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new piece here is the weak-supervision mechanism: turn extracted properties into HARGs, compute a content edit distance via Munkres, and train SimGNN on those distances without any similarity labels. The HARG construction with EPL vertices is careful, and the pseudocode is detailed enough to reimplement. The MuQNOL dataset and the HART text-attribute extractor are also useful contributions; HART's POS-based heuristics show respectable F1 on the clothes attribute without training, which is a real result for a domain where annotation is scarce.\n\nThat said, the evaluation has a load-bearing flaw. The relevance ground truth is built from the same hand-assigned property-mismatch penalties that define CED: rcost(TOP-COLOR)=1, rcost(BOTTOM-COLOR)=2, rcost(GENDER)=3, and relevant means CED < 3. CED is the Munkres-optimized sum of those exact penalties, and the same CED scores generate the weak labels used to train SimGNN. So the reported mAP mostly measures how well SimGNN reproduces a hand-defined penalty rule over a fixed attribute set, not whether edit distance tracks actual retrieval relevance. An independent relevance signal is needed before the central claim is supported. This is not a nitpick; it is the main thing the paper needs to fix.\n\nThe paper also overstates its results. The abstract says FemmIR performs \"comparably\" to similar systems, but the introduction claims it \"surpasses existing multimodal knowledge retrieval methods,\" and Table II shows EARS beating FemmIR on every row. The comparison set is also thin: no error bars, no modern zero-shot baselines, and the only exact baseline is EARS. The code and dataset are not released, which matters for a paper whose contribution is partly a benchmark. The low mAP scores track the property-identifier F1, so the method is capped by its upstream extractors; that is acknowledged but not addressed.\n\nWho is this for? Readers working on label-free multimodal retrieval or graph-edit-distance weak supervision will find the architecture and the HARG encoding worth studying. As a published claim of state-of-the-art retrieval, it is not convincing. But the underlying idea is legitimate, the writing is clear, and the limitations are at least partially visible in the paper itself.\n\nRecommendation: send it to peer review, but with a referee who will insist on an evaluation that does not define relevance using the same penalties that produce the weak labels, plus code/data release and a fair baseline set. The paper deserves serious referee time, not desk rejection, because the weak-supervision mechanism and the dataset could be valuable once the evaluation is cleaned up.","headline":"The edit-distance weak supervision idea and HARG graph construction are worth attention, but the retrieval evaluation is circular and the paper's own Table II contradicts its headline claim.","tokens_in":20647,"tokens_out":2036,"would_cite":false,"duration_ms":25456,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FemmIR claims that multimodal retrieval can be taught with edit-distance weak labels instead of similarity annotations or fine-tuning.","keywords":["multimodal information retrieval","weak supervision","graph edit distance","attributed relational graph","query by example","missing person retrieval","human attribute recognition","MuQNOL"],"falsifier":"Take MuQNOL queries with gold property annotations and replace the noisy video and image property identifiers with the ground-truth attributes; if FemmIR's mAP does not rise substantially beyond 27–37%, the gap would indicate that edit-cost weak labels, rather than identifier noise, are the binding constraint. Alternatively, collect human relevance judgments on a sample of query–candidate pairs and test whether the ranking induced by CED agrees with them better than a naive matched-attribute count; if not, the edit-distance signal carries no extra information.","tokens_in":19618,"feed_emoji":"🔍","tokens_out":4291,"duration_ms":43205,"temperature":0.7,"pith_summary":"FemmIR claims that multimodal retrieval can be solved without similarity labels or fine-tuning by treating relevance as an edit cost between graphs built from extracted properties. A query example and a candidate sample are each converted into a hierarchical attributed relational graph; matching those graphs costs more for samples that differ in the properties a user cares about. Those edit costs serve as weak supervision for a graph-similarity network that learns to rank. On a new missing-person benchmark, MuQNOL, the framework retrieves images, text, and video with mean average precision comparable to an exact-join baseline and well above a fine-grained cross-modal model that was not fine-tuned. The authors claim this opens retrieval to applications with scarce annotations.","feed_headline":"No labels needed: edit distance trains multimodal search","feed_subtitle":"FemmIR ranks images, text, and video by how much their property graphs must change, with no fine-tuning.","key_machinery":"The load-bearing object is the Hierarchical Attributed Relational Graph (HARG), a multi-level tree whose leaf nodes hold property values and whose internal nodes represent entities and relations. Comparisons are computed on Entity-with-Property-in-Leaf (EPL) vertices, so that the cost of matching a person includes the costs of their clothing and relation edges. The Content Edit Distance (CED) is computed by a modified Munkres bipartite assignment over these vertices with per-property replacement and insertion costs, plus an optional cumulative pass that propagates parent costs to children to respect dependencies; the result is normalized and exponentiated to give similarity labels. Those labels train SimGNN, a graph neural network with a Neural Tensor Network interaction layer, to approximate the edit distance at inference.","core_discovery":"The paper's central claim is that weak supervision from a purpose-built graph edit distance, Content Edit Distance (CED), is enough to train a ranked cross-modal retriever without any similarity labels or application-specific fine-tuning. For each data sample, a property extractor produces attributes that are arranged into a hierarchical relational graph; the CED measures the minimal cost of converting one sample's graph into another's, where replacement costs express user priorities such as gender mismatch being worse than shirt-color mismatch. The normalized distance is converted to a similarity score via $\\mathrm{SIM} = e^{-\\mathrm{nCED}}$, and this score trains SimGNN to predict similarity from graph embeddings, so at inference time only the property graphs and the learned mapping are needed. The paper argues this works because the edit cost implicitly encodes the structure and dependencies among properties, not just matched attribute counts, and demonstrates it on the MuQNOL missing-person retrieval task.","pith_inferences":["The same weak-supervision mechanism should transfer to other entity-centric domains, such as product search or surveillance, as long as a finite property vocabulary and consistent entity types can be fixed in advance.","Because CED labels are only as good as the property graphs, improving attribute extractors, especially video and image identifiers, should translate almost linearly into better retrieval mAP, a prediction that can be tested by swapping in stronger identifiers on MuQNOL.","The cumulative Munkres variant suggests that relation-aware edit costs matter most when properties are interdependent; ablating the cumulative pass would isolate how much of the ranking signal comes from structural dependencies rather than from simple attribute matching.","One implicit risk is that using edit cost over a small fixed attribute set defines relevance operationally, so samples that are relevant through unmodeled attributes would be misranked even if the framework itself works as claimed."],"forward_implications":["Retrieval systems can be built for new applications by plugging in existing property extractors and defining replacement costs, without collecting similarity annotations.","Performance of the retriever is bounded by the quality of the property identifiers: the paper reports that video and image identifiers with average F1 near 79.59% yield mAP scores of 27–37%, while text identifiers with higher F1 give the best mAP.","The edit-cost formulation lets users express which properties matter, such as gender outweighing top color, through replacement costs, keeping the system adaptable to different information needs.","The MuQNOL benchmark provides a testbed for weakly supervised multimodal retrieval where relevance is derived from gold property annotations rather than similarity labels."],"supporting_citations":[{"why":"Supplies the bipartite graph matching method that Content Edit Distance adapts for computing edit costs between HARGs.","marker":"[21]"},{"why":"Provides SimGNN, the graph neural network trained on CED weak labels to approximate similarity at inference.","marker":"[23]"},{"why":"MARS is the source dataset for image and video modalities with the 16 annotated attributes used to build MuQNOL.","marker":"[6]"},{"why":"EARS is the exact-inference content-based retrieval baseline that FemmIR is compared against.","marker":"[18]"},{"why":"Defines the human-attribute property set for person search and provides the color-sampling property identifier baseline.","marker":"[20]"},{"why":"Graph Convolutional Networks provide the representation-invariant node embeddings used inside the HARG encoder.","marker":"[22]"},{"why":"Neural Tensor Networks supply the interaction scoring between graph embeddings in the learned similarity model.","marker":"[25]"},{"why":"Provides the normalization scheme that converts raw edit distances into similarity scores for weak label generation.","marker":"[24]"}],"fun_headline_variants":["Edit distance trains multimodal retrieval without labels","Weak supervision via graph edit cost for cross-modal search","No-label ranking with content edit distance","Graph edit distance weak supervision retrieves media","Property edit cost weak labels for multimodal search"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that relevance between samples is fully captured by the edit cost over the user-specified property graphs, so retrieval quality is capped by how well that small attribute set, such as gender, clothing colors, and height, describes what actually makes a person relevant.","fun_headline_variants_meta":{"raw":{"variants":["Edit distance trains multimodal retrieval without labels","Weak supervision via graph edit cost for cross-modal search","No-label ranking with content edit distance","Graph edit distance weak supervision retrieves media","Property edit cost weak labels for multimodal search"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000192,"raw_usage":{"total_tokens":1366,"prompt_tokens":983,"completion_tokens":383,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":317}},"tokens_in":599,"tokens_out":383,"duration_ms":4813,"temperature":1.0,"reasoning_tokens":317,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:56:02.121323+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take MuQNOL queries with gold property annotations and replace the noisy video and image property identifiers with the ground-truth attributes; if FemmIR's mAP does not rise substantially beyond 27–37%, the gap would indicate that edit-cost weak labels, rather than identifier noise, are the binding constraint. Alternatively, collect human relevance judgments on a sample of query–candidate pairs and test whether the ranking induced by CED agrees with them better than a naive matched-attribute count; if not, the edit-distance signal carries no extra information.","supporting_citations":[{"cited_title":"Bipartite graph matching for computing the edit distance of graphs,","cited_arxiv_id":null,"evidence_quote":"Supplies the bipartite graph matching method that Content Edit Distance adapts for computing edit costs between HARGs."},{"cited_title":"Mars: A video benchmark for large-scale person re-identification,","cited_arxiv_id":null,"evidence_quote":"MARS is the source dataset for image and video modalities with the 16 annotated attributes used to build MuQNOL."},{"cited_title":"Applying machine learning and data fusion to the “missing person","cited_arxiv_id":null,"evidence_quote":"EARS is the exact-inference content-based retrieval baseline that FemmIR is compared against."},{"cited_title":"Surveillance video querying with a human-in-the-loop,","cited_arxiv_id":null,"evidence_quote":"Defines the human-attribute property set for person search and provides the color-sampling property identifier baseline."},{"cited_title":"Reasoning with neural tensor networks for knowledge base completion,","cited_arxiv_id":null,"evidence_quote":"Neural Tensor Networks supply the interaction scoring between graph embeddings in the learned similarity model."},{"cited_title":"Graph based shapes representation and recognition,","cited_arxiv_id":null,"evidence_quote":"Provides the normalization scheme that converts raw edit distances into similarity scores for weak label generation."}],"review_version":1}