{"id":"833a881b-75cb-480b-b5b2-92b627bac8ae","arxiv_id":"1908.03361","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Interactive relevance feedback on content-based image retrieval finds task-relevant flood images much more effectively than a single image query alone.","lead":"This paper tests whether letting an analyst give feedback on a few retrieved flood images can quickly pull out the images that matter for flood impact tasks. It introduces an expert-annotated dataset of 3,710 flood images and reports that five rounds of feedback lift a ranking-quality score from 55% to 87%.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Feedback images are never excluded from the evaluated ranking, so the 87%/93% NDCG gains may partly reflect memorizing labeled examples rather than discovering new relevant images.","rationale":"The reader's weakest assumption concerned external validity: simulated feedback may not represent real analyst use. My concern is more internal to the evaluation protocol: the simulated feedback images appear to remain in the corpus on which NDCG@100 is computed. Since the feedback methods are trained on those very images, their high ranks in later rounds can be caused by memorizing the labeled examples rather than by generalizing to unseen relevant images. This is a standard pitfall in relevance feedback evaluation; feedback documents should be removed from the result list before scoring, or at least reported separately. The concrete test would settle whether this effect materially changes the headline improvement. If the drop is small, the original claim stands and the paper's contribution is largely intact. If the drop is large, the absolute NDCG figures would need to be revised, though the relative comparison among feedback methods might still be informative. This does not change the reader's verdict of CONDITIONAL, but it adds a specific, testable condition that should be satisfied before the headline numbers are taken at face value.","tokens_in":12219,"tokens_out":9049,"duration_ms":106517,"concrete_test":"Recompute the Figure 5 curves while excluding from the ranked list, and from R(q) in Eq. (1), all images that were labeled in previous feedback rounds (and also the query image, if it currently remains in the corpus). If the NDCG@100 at rounds 5 and 10 drops by more than a few percentage points relative to the reported 86.9 and 92.9, the headline improvement is partly an artifact of evaluating on the feedback set. The dataset and the described protocol are sufficient to run this check.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 4.2's simulated protocol labels 10 random images from the current top 100 each round and then evaluates NDCG@100 over the full combined corpus. The text never states that these labeled images are removed from the ranked list before computing Eq. (1). Because ITML and the other metric learners are trained on constraints that include exactly those images, re-ranking the full corpus can improve NDCG by placing the known relevant feedback images high, even if no previously unseen relevant images are discovered. In a live interactive session the analyst has already seen the images they labeled, so including them in the reported NDCG@100 overstates the system's ability to surface new evidence. After five rounds, up to 50 labeled images, and after ten rounds, up to 100 labeled images, are potentially 'taught' into the ranking; the random sampling from the top 100 means many of these are already near the top, so the inflation is not negligible. This directly affects the headline numbers of 87% and 93%.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an interactive content-based image retrieval (CBIR) system for flood impact analysis, in which an analyst provides a query image and then labels a small number of retrieved images as relevant or irrelevant over several rounds, with the system updating the ranking via feedback methods such as metric learning, classification, or kernel density estimation. The authors introduce a new expert-annotated dataset of 3,710 flood images (3,435 European Flood 2013 images from Wikimedia Commons plus 275 water-pollution images), mix it with 97,085 distractor images from Flickr100k, and evaluate 611 ideal query images using NDCG@100. The central claim is that relevance feedback improves retrieval from about 55% NDCG@100 at baseline to 87% after five simulated rounds and 93% after ten rounds, with ITML providing the best early-round performance.","tokens_in":12355,"tokens_out":9285,"duration_ms":91992,"significance":"The paper has two potentially valuable contributions: a publicly released, expert-annotated flood-image retrieval dataset, and a systematic comparison of several modern feature representations and relevance-feedback methods. The experimental design is careful in several respects, including ten repetitions with different 75% subsamples, a reported maximum standard deviation of 1.2%, and paired t-tests for significance. If the evaluation protocol is sound, the result that ITML gives strong early-round feedback performance is practically useful for disaster response. However, the headline NDCG improvements rest on a simulation whose evaluation may include the very images used to train the feedback model, and on threshold choices that may use oracle information; these issues must be resolved before the claimed magnitudes can be accepted.","major_comments":[{"comment":"The simulated feedback protocol labels 10 random images from the current top 100 each round and then computes NDCG@100 over the full corpus, but the text does not state that the labeled images are removed from the ranked list before evaluation. Since the metric-learning methods are trained on constraints involving exactly those images, re-ranking can raise NDCG by placing the already-seen relevant feedback images at the top, even if no previously unseen relevant image is discovered. In a real interactive session the analyst has already inspected the labeled images, so counting them as retrieved evidence overstates the system's ability to surface new information. After five rounds this affects up to 50 images and after ten rounds up to 100 images, directly impacting the headline 87% and 93% figures. Please clarify whether such exclusion was performed; if not, the experiments should be re-run with feedback images excluded from the ranked list (or with both variants reported).","section":"Section 4.2, Eq. (1)"},{"comment":"The ITML threshold setting described in Section 4.2 relies on \"the first irrelevant retrieval result\" in the baseline ranking. Because the simulation has access to ground-truth labels for the entire corpus, this threshold may be set using oracle information that a real analyst would not have: the analyst would not know which unlabeled result is the first irrelevant one. If the threshold is instead intended to be computed only from the labeled feedback images, that should be stated explicitly. Please clarify the information used to set these thresholds and, if oracle information is used, re-evaluate ITML under a realistic setting or at least quantify the sensitivity of the results to the threshold rule.","section":"Section 4.2, ITML threshold setting"},{"comment":"The evaluation treats the single-expert relevance labels and the 611 \"ideal\" query images as ground truth, but no inter-annotator agreement is reported. The three task definitions are subjective, and the text acknowledges that only one annotation was obtained per image. Without a reliability measure such as Cohen's kappa on a subset, it is difficult to determine whether the reported NDCG differences reflect stable task-relevant distinctions or one expert's idiosyncratic judgments. Please report annotation agreement or otherwise justify label stability.","section":"Section 3.2"},{"comment":"The feedback simulation assumes that in each round the user labels 10 uniformly random images from the current top 100 and that these labels are perfectly accurate. Real analysts under time pressure are likely to select images non-uniformly and to make occasional errors, so the simulated conditions may overstate the practical benefit. A sensitivity analysis with noisy labels or a small user study would substantially strengthen the claim that the method helps hydrologists find relevant images quickly.","section":"Section 4.2, feedback simulation"}],"minor_comments":[{"comment":"The abstract uses the term \"precision\" for the reported metric, but the paper evaluates NDCG@100; please use consistent terminology.","section":"Abstract"},{"comment":"The sentence \"the average NDCG@100 can be improved from 55% yield by the baseline retrieval\" is ungrammatical; it should be reworded, for example, \"from the 55% yielded by the baseline retrieval.\"","section":"Conclusion"},{"comment":"The statement \"We have empirically found p = 2 to work well on our dataset\" indicates that the GeM pooling exponent was chosen on the evaluation data; please state whether this was done on a held-out validation set, and if not, treat p as a tuned parameter in the comparison.","section":"Section 4.1"},{"comment":"The per-task bars in Figure 4c lack numeric labels, making it hard to read the exact NDCG values; adding the values would improve clarity.","section":"Figure 4c"},{"comment":"The manuscript contains typesetting artifacts such as \"V olunteered\" in the Abstract and the \"P R E- P R I N T\" header; these should be cleaned before publication.","section":"General formatting"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical study and the dataset is a useful contribution. My main reservation is the evaluation protocol; if the authors can show that the feedback images were excluded from the evaluation, or re-run the experiments accordingly, the paper could become acceptable. I see no concerns about novelty disclosure or fit with the journal's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a solid application of known retrieval and relevance-feedback methods to a new, useful dataset, but the headline NDCG gains are inflated because the simulated feedback evaluation never removes the labeled images from the ranked list. This is the one thing to know before reading.\n\nThe real contribution is the dataset: 3,710 expert-annotated flood images with three task definitions (flooded area, inundation depth, water pollution), mixed with 97,085 distractors, along with query selections. That is a reusable benchmark for a real application. The evaluation is careful: ten repetitions, maximum standard deviation 1.2%, paired t-tests, multiple features and feedback methods. The finding that ITML helps most in early rounds while SVM catches up later is a practical, non-obvious takeaway.\n\nThe soft spot is load-bearing. In Section 4.2, each round samples 10 random images from the top 100, uses their labels to train ITML or other methods, and then computes NDCG@100 over the full ranked list. The text never says those labeled images are removed from the ranking before evaluation. In a real session, the analyst has already seen those images; a retrieval system is useful for surfacing new evidence. Including them rewards the learner for memorizing the label set, so after five rounds up to 50, and after ten up to 100, already-seen items can be placed high. That inflates the reported 55% to 87% to 93% numbers. The relative comparison between feedback methods is still fair because they all share the protocol, but the absolute claim should not be read as 'new relevant images found' performance.\n\nMinor issues: single-expert annotations without inter-annotator agreement, especially for the subjective pollution task; the simulation assumes randomly selected feedback images, which may not match real analyst behavior; the images come from Wikimedia and web searches rather than live social media streams; and the abstract says 'precision' when the metric is really NDCG@100. The GeM pooling exponent was tuned on the evaluation set, but that is a minor caveat since it doesn't drive the feedback result.\n\nWho should read this: people building disaster-response information systems or studying interactive retrieval. The dataset is worth citing. The paper deserves a serious referee, but the evaluation protocol needs to be fixed or at least discussed, and absolute numbers re-reported, before the central claim is trustworthy.","headline":"Useful flood-image retrieval benchmark, but the simulated feedback evaluation leaks labeled images into the ranked list, so the headline NDCG numbers overstate real interactive performance.","tokens_in":12931,"tokens_out":3217,"would_cite":true,"duration_ms":31663,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Relevance feedback lifts flood-image retrieval from 55% to 87%","keywords":["content-based image retrieval","relevance feedback","volunteered geographic information","flood impact analysis","information-theoretic metric learning","social media images","disaster management","image retrieval dataset"],"falsifier":"Run a controlled user study in which hydrologists perform the same retrieval task with the proposed interface on a recent flood event, then measure the average NDCG@100 of the top 100 results after five feedback rounds; if it is close to the 55% baseline instead of near 87%, the simulation overstates the benefit. A complementary check is to replay the same queries with label noise, such as 10 to 20 percent flipped relevance labels, and see whether the improvement collapses.","tokens_in":11991,"feed_emoji":"🌊","tokens_out":8559,"duration_ms":81467,"temperature":0.7,"pith_summary":"The paper claims that an interactive image-retrieval loop, in which an analyst starts from one example image and then marks a few retrieved images as relevant or irrelevant, can find task-relevant flood images far more effectively than a one-shot similarity search. In a simulated evaluation with 611 expert-chosen queries over a newly introduced dataset of 3,710 expert-annotated flood and water-pollution images mixed with 97,085 distractors, the best feedback configuration raises average NDCG@100 from 55% at baseline to 87% after five feedback rounds and 93% after ten. This matters because flood analysts cannot manually inspect the large stream of social media images, and because relevance is task-dependent: flooded area, inundation depth, and water pollution require different visual cues that are hard to express as keywords or fixed classes. The paper therefore proposes a general interactive mechanism rather than a task-specific classifier.","feed_headline":"Five rounds of feedback lift flood-image search from 55% to 87%","feed_subtitle":"Mark a few images and the system re-ranks, with no keywords or per-task classifiers needed","key_machinery":"The load-bearing component is information-theoretic metric learning (ITML), which learns a Mahalanobis distance metric from pairwise similarity constraints. Each feedback round converts the analyst's labels on ten images into constraints, and ITML updates a positive semidefinite matrix that reshapes the feature space so that relevant images become closer and irrelevant images are pushed farther apart, with regularization toward the Euclidean metric to avoid overfitting in the high-dimensional feature space. The re-ranking for the next round uses this learned metric. Retrieval features come from deep convolutional networks: Deep R-MAC and VGG16 with partial mean pooling, both averaged over multiple image scales. The evaluation harness is the expert-annotated flood dataset combined with 97,085 Flickr100k distractor images.","core_discovery":"The central discovery is that relevance feedback in the form of information-theoretic metric learning turns a weak flood-image search into a strong one within a few rounds of user effort. With Deep R-MAC features and multi-scale averaging, a single-query nearest-neighbor baseline reaches an average NDCG@100 of 55%; after the user labels ten images per round for five rounds, the ITML-based re-ranking reaches 87%, and after ten rounds it reaches 93%, with at most 1.2% standard deviation across repetitions. The paper also contributes a benchmark of 3,710 images annotated by domain experts for three information objectives and ideal query images for each task, and it demonstrates that ITML gives the best early-round performance while an SVM-based classifier only becomes competitive from round seven onward.","pith_inferences":["A natural extension is active learning: instead of labeling ten random top-100 images, the system could pick the most informative images to label; the large early-round gains in this study suggest that would cut the number of clicks needed even further.","The dataset includes region-level annotations of important image areas that the current system ignores; using them as spatial attention or regional pooling could help when relevance hinges on a small part of the image, such as a partially submerged traffic sign.","Because no flood-specific classifier is trained, the same interactive protocol could transfer to other time-critical disaster imagery, such as wildfires or earthquakes, and to other kinds of volunteered geographic information.","The 55% baseline uses a single query image; starting an analyst with several seed images at round zero might shorten the path to high NDCG, a variation the paper does not test."],"forward_implications":["After five rounds of ten labels each, ranking quality reaches 87% NDCG@100, so an analyst can expect most relevant flood images to appear within the first 100 results.","The same interactive loop covers flooded-area, inundation-depth, and water-pollution tasks without retraining, since the query image and feedback define the notion of relevance.","ITML's strong early-round performance means the method is usable by analysts who will not provide many feedback rounds; SVM-based refinement only overtakes it after about seven rounds.","The released dataset gives the community an expert-annotated benchmark with ideal query images for each task, enabling direct comparison of future flood-image retrieval and feedback methods.","Because retrieval uses only image content, photographs with sparse or missing text captions can be found, avoiding the misses of keyword-based social-media filtering."],"supporting_citations":[{"why":"Supplies information-theoretic metric learning, the feedback method that produces the reported 87% and 93% NDCG@100 results.","marker":"Davis et al (2007)"},{"why":"Supplies Deep R-MAC features, the best-performing baseline retrieval representation in the comparison.","marker":"Gordo et al (2017)"},{"why":"Supplies the Flickr100k image pool used as non-flood distractors in the evaluation.","marker":"Philbin et al (2007)"},{"why":"Defines the NDCG@100 metric used for all retrieval-quality measurements.","marker":"Järvelin and Kekäläinen (2002)"},{"why":"Provides the kernel-density-estimation feedback baseline and feature-weighting method compared against ITML.","marker":"Deselaers et al (2008)"},{"why":"Provides partial mean pooling, the best aggregation for VGG16 features in the comparison.","marker":"Zhi et al (2016)"},{"why":"Supplies the SVM feedback classifier that becomes competitive with ITML in later feedback rounds.","marker":"Cortes and Vapnik (1995)"}],"fun_headline_variants":["Five rounds of user feedback raise flood image retrieval to 87%","ITML re-ranking lifts flood image precision to 87% in five rounds","No keywords or classifiers: feedback re-ranks flood images to 87%","From 55% to 87%: five feedback rounds improve flood image search"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that simulated ground-truth feedback on randomly chosen top-100 images matches how real flood analysts would actually label images in practice; if real feedback is noisier or differently selected, the 87% and 93% figures may not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Five rounds of user feedback raise flood image retrieval to 87%","ITML re-ranking lifts flood image precision to 87% in five rounds","No keywords or classifiers: feedback re-ranks flood images to 87%","From 55% to 87%: five feedback rounds improve flood image search"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000531,"raw_usage":{"total_tokens":2523,"prompt_tokens":877,"completion_tokens":1646,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":1563}},"tokens_in":493,"tokens_out":1646,"duration_ms":13596,"temperature":1.0,"reasoning_tokens":1563,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:15:20.938105+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled user study in which hydrologists perform the same retrieval task with the proposed interface on a recent flood event, then measure the average NDCG@100 of the top 100 results after five feedback rounds; if it is close to the 55% baseline instead of near 87%, the simulation overstates the benefit. A complementary check is to replay the same queries with label noise, such as 10 to 20 percent flipped relevance labels, and see whether the improvement collapses.","supporting_citations":[{"cited_title":"International Journal of Computer Vision 124(2):237--254, doi:10.1007/s11263-017-1016-8","cited_arxiv_id":null,"evidence_quote":"Supplies Deep R-MAC features, the best-performing baseline retrieval representation in the comparison."}],"review_version":1}