{"id":"c4da5c05-97ac-476d-9cf8-be72ca48fb36","arxiv_id":"1908.03651","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A hybrid CNN detector plus a Gaussian-sigmoid scoring function maps watermarks to a human-correlated distraction score for image ranking.","lead":"The authors train a two-part neural network to find watermarks in photos and then fit a simple formula that turns the detected watermarks into a single 'distraction score.' The system is designed to help image-ranking services demote pictures with obtrusive watermarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ranking evidence for the distraction score is likely in-sample: Table 8's 10k evaluation overlaps the 8k training set, and the held-out Table 10 drops to near chance for the 1-vs-0 pair.","rationale":"Reader's weakest_assumption targets the Gaussian-plus-sigmoid functional form and the omission of opacity. I agree that is a genuine limitation, but the more decisive issue is evident from the paper's own tables: the ranking evaluation may not be out-of-sample. Section 5 says the scoring function is fit on 8k images and its MSE is computed on a 2k validation split, yet Table 8 reports pairwise accuracy on 10k images without saying the training split was excluded. In the only clearly held-out full-distribution table (Table 10), accuracy for the hardest pairwise distinction, score 1 vs score 0, is 57.95%, close to chance. Table 9's high accuracy is obtained after deleting predicted negatives, a selection that cannot be justified for the stated image-ranking application. I would keep the reader's CONDITIONAL verdict, because a clean out-of-sample recomputation could settle the concern, but the paper does not currently provide that evidence. I do not base the concern on disagreement with the community or on the absence of code; the issue is internal to the reported evaluation protocol.","tokens_in":9404,"tokens_out":8034,"duration_ms":84183,"concrete_test":"Recompute Table 8 strictly on the 2k validation split of the scoring dataset (and, for the full system, on all validation images without dropping predicted negatives). If the 1-vs-0 pairwise accuracy on that held-out split is near 57.95% rather than 92.08%, or if the 3-vs-1 accuracy is near 86.40% rather than 95.36%, then Table 8's high numbers were inflated by training-set evaluation and the ranking claim fails. Also state explicitly whether the 10k images in Table 8 include the 8k training images.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the fitted score is 'correlated with human perception' and 'validated in an image ranking setup' (Abstract) rests on the pairwise accuracies in Table 8. Section 5 states that the scoring function was fit on an 8k subset of the 10k-image scoring dataset and evaluated for MSE on the remaining 2k, but Table 8 is captioned 'Evaluated on 10k images' with no statement that the 8k training portion was excluded. If those 10k images include the training split, the 92-97% accuracies are in-sample and do not establish generalization. The only clearly held-out full-distribution ranking result, Table 10, is substantially weaker: score-1 vs score-0 pairwise accuracy is 57.95%, barely above chance, and the 3-vs-1 accuracy drops from 95.36% (Table 8) to 86.40%. Table 9 reaches 96% only after removing all images predicted not to contain watermarks, which removes exactly the hard pairs a deployed ranking system must order. The burden is on the paper to show Table 8 is out-of-sample; as written, the empirical support for the headline claim is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-tower CNN system for watermark detection, combining a DeepLab-based semantic segmentation model with a binary image classifier, and a nonlinear scoring function that maps detected watermark masks to a single 0-1 'distraction score' via a Gaussian-weighted pixel sum followed by a sigmoid. The authors report pixel- and image-level metrics for the hybrid detector, estimate image precision under a sparse 10% positive-example assumption, compare against a text detector, and evaluate the scoring function with pairwise ranking accuracies. The central claims are that the hybrid model substantially improves image precision on real-world sparse data and that the fitted distraction score is correlated with human perception and validated in an image ranking setup.","tokens_in":9671,"tokens_out":5969,"duration_ms":63203,"significance":"If the ranking claim were established, the distraction score would be a practically useful tool for downranking watermarked images in large-scale search and mapping applications, and the hybrid detection architecture offers a sensible way to trade recall for precision in sparse positive settings. The scoring function is explicitly fitted to human ratings and the paper reports an out-of-sample MSE on a 2k-image validation split, which is a strength. The hybrid detector's ePrecision improvement from 26.13% to 79.51% under the 10%-positive assumption is also a clear, falsifiable quantitative claim. However, the ranking validation is currently not convincing: the only clearly held-out full-distribution ranking table shows near-chance accuracy for the score-1 vs score-0 pair, and the main ranking table is ambiguous about whether training images are included. The paper is therefore a useful engineering contribution but needs a substantially clearer and more complete evaluation to support its headline claim.","major_comments":[{"comment":"The pairwise ranking accuracies in Table 8 are captioned 'Evaluated on 10k images,' but Section 5 states that the scoring function was fit on an 8k subset and validated on the remaining 2k. As written, Table 8 appears to include the 8k training images, so the 92-97% accuracies may be in-sample and do not establish generalization. This is load-bearing for the abstract's claim that the method is 'validated in an image ranking setup.' Please either clarify explicitly that the 10k evaluation excludes the 8k fitting split, or report the pairwise accuracies on the 2k validation set only. The held-out numbers in Table 10 are much lower, so the distinction is material.","section":"Section 5, Table 8"},{"comment":"On the validation set, the score-1 vs score-0 pairwise accuracy is 57.95%, close to chance. This is the most important distinction for the stated application of demoting images with distracting watermarks, because score 0 means 'no watermark' and score 1 means 'small watermark near edges.' The paper does not discuss this near-chance result or explain why the fitted Gaussian-sigmoid scoring function fails to separate these classes. If Table 8 is in-sample, then the only out-of-sample evidence for the ranking claim is Table 10, which does not support the claim that the score is correlated with human perception in a ranking setup. Please analyze this failure and, if possible, revise the scoring function or the evaluation to address it.","section":"Section 5, Table 10"},{"comment":"The scoring function maps watermarks to a score using only a Gaussian-weighted sum of the segmentation label and a single sigmoid. It does not account for watermark opacity, contrast, content, or interaction with the underlying scene, all of which contribute to perceptual obstructiveness. The paper itself lists opacity as future work, but this limitation directly affects the central claim that the score represents 'visual obstructiveness.' The held-out ranking results suggest that even for the limited size/location features, the score does not reliably separate small edge watermarks from no watermark. Please provide a quantitative analysis of where the scoring function fails on the validation set, and state clearly which perceptual factors are and are not captured by the current score.","section":"Section 3.4"}],"minor_comments":[{"comment":"The caption says '1.7k randomly sampled images from the validation set,' but the validation set is described as 2k images. Please clarify why 1.7k images were used and whether the 300 excluded images were excluded before or after the random sample.","section":"Section 5, Table 10 caption"},{"comment":"The text states that for a scoring function based on normalized watermark area, maximizing pairwise ranking accuracy is equivalent to maximizing IOU and minimizing image false positives/negatives. However, the actual scoring function introduced in Section 3.4 is a Gaussian-weighted sum passed through a sigmoid, not a normalized area. The equivalence argument should be updated to match the final scoring function or explicitly framed as a motivating approximation.","section":"Section 3.3"},{"comment":"The Gaussian function g(i,j) is not fully specified: it is not stated whether the Gaussian is normalized to have a peak value of 1 or unit integral. This matters because the fitted values of sigma and alpha depend on the normalization. Also, the caption says 'isometric' but the correct term is 'isotropic.'","section":"Section 3.4 and Figure 5"},{"comment":"The dataset split is described as 80/10/10 of 200k images, but Section 4.3 says the semantic segmentation model is trained on 160k watermarked images. Please clarify whether the 160k training images are all watermarked, and how the image classification model's training set is constructed from the same 200k images.","section":"Section 4.1"},{"comment":"The ePrecision formula says iTP and iFP are computed from a balanced dataset of 50% watermarked images, but the table captions say the metrics are computed on a dataset with 62.5% watermarked images. Please clarify which dataset was used; this affects the reproducibility of the estimated precision numbers.","section":"Section 4.2 and Tables 4/6"},{"comment":"The Flickr results are only qualitative. It would strengthen the cross-domain claim to report at least a small quantitative measure, such as detection or scoring accuracy on a labeled subset of Flickr images.","section":"Section 4.6 and Tables 11/12"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a concise industry report on a proprietary dataset, with no code or data release, which limits reproducibility but is not by itself a reason to reject. The decisive issue is the evaluation protocol for the ranking claim: if Table 8 includes the training split, the paper's main validation is missing. The authors should be asked to recompute and clearly report held-out pairwise accuracy on the full validation set, including the score-1 vs score-0 pairs, and to discuss why that accuracy is near chance. If the held-out numbers remain close to chance, the central claim will need to be substantially weakened or the scoring function revised."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you care about watermark detection for image ranking, but read the evaluation tables carefully before trusting the headline numbers.\n\nThe genuinely new piece is the distraction score: a Gaussian-weighted area of detected watermark pixels run through a fitted sigmoid, with the fit done on human ratings. The two-tower hybrid (segmentation gated by a binary classifier) is a sensible engineering choice for sparse-positive real-world data, and the estimated image precision gain (79.5% vs 26.1% at 10% watermark prevalence) is plausible and consistent with the false-positive counts. The scoring function is simple, but that is not a flaw; it is calibrated on held-out human scores with an MSE of 0.041, which is a legitimate out-of-sample check.\n\nThe soft spot is the ranking evaluation. Table 8, captioned \"Evaluated on 10k images,\" is almost certainly the full scoring dataset. Section 5 says that dataset was split into 8k for fitting and 2k for validation, and Table 8 does not state that the 8k was removed. So the 92–97% pairwise accuracies are likely in-sample and tell you little about generalization. Table 10, which is explicitly on a held-out validation sample, is much weaker: the 1-vs-0 pair sits at 57.95%, barely above chance, and the 3-vs-1 drops to 86.4%. Table 9 reaches 96% only after deleting images the model predicts as negative, which conveniently removes the hardest pairs. The paper does include Table 10, so the authors are not hiding the weak result, but the abstract's claim of validation in an image ranking setup leans on Table 8.\n\nThere are also smaller issues: opacity is explicitly left for future work, so the score ignores a factor that plausibly matters; the proprietary data and lack of code mean the results are not independently reproducible. None of these are fatal, but together they mean the empirical support for the headline claim is not established as written.\n\nWho is this for? Someone building practical watermark scoring for photo ranking; the detector comparison and the scoring idea are worth knowing. It does not deliver a new scientific capability, and the ranking evidence needs a redo on a clean held-out set.\n\nIf I were the editor, I would send it to review rather than desk-reject, but I would expect the reviewers to demand a held-out ranking evaluation with all images kept in. The core idea is fine; the proof is incomplete.","headline":"A practical watermark-detection plus scoring pipeline whose headline ranking accuracy is largely in-sample; the held-out ranking results are far weaker than advertised.","tokens_in":10167,"tokens_out":1254,"would_cite":false,"duration_ms":15858,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that watermark distraction can be reduced to a single fitted score based on Gaussian-weighted pixel area, and that a hybrid segmenter-classifier makes this score precise enough for real-world image ranking.","keywords":["watermark detection","semantic segmentation","image classification","perceptual image scoring","image ranking","hybrid model","deep learning"],"falsifier":"Take pairs of images whose predicted scores are equal but whose watermarks differ in opacity or contrast, such as a faint large watermark versus a small solid one; if human raters systematically rank one type above the other, the area-only scoring assumption fails. Concretely, collect a few hundred such pairs and check whether the model's pairwise ranking accuracy drops toward chance.","tokens_in":9199,"feed_emoji":"🖼️","tokens_out":4880,"duration_ms":51248,"temperature":0.7,"pith_summary":"This paper tries to turn the messy visual nuisance of watermarks into a single number that tracks how much people are bothered by them. It argues that two ingredients are enough: reliable detection of watermark pixels across varied real-world images, and a scoring formula that weights a watermark's area by where it sits in the frame. The authors report that in a sparse real-world setting, where under 10% of images carry watermarks, the hybrid model would return correct detections about 79.5% of the time versus about 26.1% for segmentation alone. If the score holds, image-ranking systems could automatically demote heavily watermarked photos.","feed_headline":"A single score now ranks how much a watermark distracts","feed_subtitle":"Hybrid detection plus a Gaussian-weighted area rule matches human rankings in pairwise tests on real photos.","key_machinery":"The load-bearing object is the hybrid segmentation label $L(i,j) = w_I \\cdot S(i,j)$, where $w_I$ is the binary image-level classification (thresholded) and $S(i,j)$ is the pixel-level segmentation. This masking suppresses false-positive pixels from text-like objects such as signs and menus that the segmentation tower alone would flag. The scoring engine is then the Gaussian-weighted pixel sum $G(\\sigma,L) = \\sum_{i,j} g(i,j) L(i,j)$ followed by the fitted sigmoid $\\text{Score}(I) = 1/(1 + e^{-\\lambda(G(\\sigma,L) - \\alpha)})$, with $\\sigma$, $\\lambda$, and $\\alpha$ optimized by mean squared error against human scores. The Gaussian encodes the assumption that center watermarks distract more, and the sigmoid encodes saturating human response to growing watermark area.","core_discovery":"The central discovery is that a watermark's perceptual impact can be captured by a two-stage pipeline: a two-tower CNN (one semantic segmentation tower, one binary classification tower) whose outputs are combined by masking, followed by a fitted nonlinear scoring function. The scoring function computes a Gaussian-weighted sum of detected watermark pixels centered at the image center, then maps it through a sigmoid with parameters fit by least squares to human ratings. The fitted standard deviation and steepness produce a very steep response: once a watermark covers more than a small central region, the score jumps toward 1. Pairwise ranking accuracy reaches 95.36% for images with very distracting watermarks versus ignorable ones, and the hybrid model keeps pixel-level quality while sharply reducing image-level false positives.","pith_inferences":["If opacity were added as a third input to the scoring function, the fitted sigmoid's steepness suggests human ratings saturate quickly; opacity may separate mid-range scores better than area alone.","The same Gaussian-weighted-area score could apply to other localized visual obstructions such as subtitles, UI overlays, or timestamp labels, where center-bias and area plausibly dominate distraction.","A direct test is to compare human rankings of images with matching Gaussian-weighted area but different watermark opacity or contrast; any systematic disagreement would indicate the scoring function needs an opacity term.","The strong center bias implies ranking gains concentrate on center watermarks, so corner watermarks may need separate treatment in quality-sensitive ranking tasks."],"forward_implications":["Image-ranking systems can use the score directly to demote images with large or central watermarks; the paper reports 95.36% pairwise ranking accuracy for very distracting versus ignorable watermarks.","Because the segmentation tower learns from positive examples and the classifier filters negatives, the hybrid model keeps pixel IOU near the segmentation model while raising estimated image precision from 26.13% to 79.51% in a dataset with 10% watermarked images.","The scoring function generalizes to unseen watermark varieties, including rotated text, logos, and both transparent and solid watermarks, as shown on public images outside the training set.","Watermarks covering most of the frame receive scores near 1, and small edge watermarks receive scores near 0, matching the intended ranking behavior for demotion."],"supporting_citations":[{"why":"Provides the atrous-convolution segmentation architecture that the paper adapts for watermark pixel detection and image classification.","marker":"[7]"},{"why":"Supplies the OCR text detector used as an alternative detection baseline and shown to be unsuitable due to high false positives.","marker":"[2]"},{"why":"Establishes the prior watermark-removal setting for transparent, predetermined watermark types that this work contrasts with general detection.","marker":"[12]"},{"why":"Provides out-of-domain public images used for qualitative evaluation of detection and scoring on unseen watermarks.","marker":"[1]"}],"fun_headline_variants":["Watermark distraction scored by CNN and human perception","One score ranks watermark distraction for image search","Gaussian-weighted watermark score matches human rankings","Two-tower CNN plus sigmoid yields watermark distraction score","Quantifying watermark distraction: from pixels to a single score"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole score rests on the assumption that human distraction is fully determined by the area of detected watermark pixels weighted by one fixed center-biased Gaussian and passed through one global sigmoid, with opacity, content, contrast, and scene interaction playing no role.","fun_headline_variants_meta":{"raw":{"variants":["Watermark distraction scored by CNN and human perception","One score ranks watermark distraction for image search","Gaussian-weighted watermark score matches human rankings","Two-tower CNN plus sigmoid yields watermark distraction score","Quantifying watermark distraction: from pixels to a single score"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000139,"raw_usage":{"total_tokens":1075,"prompt_tokens":780,"completion_tokens":295,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":396,"completion_tokens_details":{"reasoning_tokens":222}},"tokens_in":396,"tokens_out":295,"duration_ms":3416,"temperature":1.0,"reasoning_tokens":222,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:06:43.043914+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take pairs of images whose predicted scores are equal but whose watermarks differ in opacity or contrast, such as a faint large watermark versus a small solid one; if human raters systematically rank one type above the other, the area-only scoring assumption fails. Concretely, collect a few hundred such pairs and check whether the model's pairwise ranking accuracy drops toward chance.","supporting_citations":[{"cited_title":"https://cloud","cited_arxiv_id":null,"evidence_quote":"Supplies the OCR text detector used as an alternative detection baseline and shown to be unsuitable due to high false positives."},{"cited_title":"Dekel, M","cited_arxiv_id":null,"evidence_quote":"Establishes the prior watermark-removal setting for transparent, predetermined watermark types that this work contrasts with general detection."},{"cited_title":"http://www.flickr.com","cited_arxiv_id":null,"evidence_quote":"Provides out-of-domain public images used for qualitative evaluation of detection and scoring on unseen watermarks."}],"review_version":1}