{"id":"5ccc6f4e-02d5-45f0-ad78-c653df660095","arxiv_id":"2501.00740","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"RORem trains an SDXL-based object remover on a 200K-pair dataset grown by iterative human feedback and a learned discriminator, surpassing prior methods by roughly 18 points in human-judged success rate.","lead":"A new object-removal model, RORem, is trained on a 200K-pair dataset built by iteratively combining human annotation with an automated discriminator that filters the model's own outputs. It reports an 18 percentage point higher human-judged removal success rate than prior methods, and a distilled version runs in four diffusion steps (about half a second per image).","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 18-point margin rests on a five-volunteer, non-method-blinded user study with no agreement or variance statistics, and the paper's Dphi corroboration is miscalibrated on key baselines; the margin needs an independent blinded re-test.","rationale":"The paper's headline is a human-judged success-rate margin, and the only evidence for it is a small, unreported, non-blinded user study with no variance or agreement statistics. If that study is biased or noisy, the central claim has no independent support, because the automatic Dphi metric is trained on the same human judgments and is demonstrably miscalibrated for several key baselines in Table 2. This is therefore the most load-bearing weakness. The reader's conditional verdict already captures this concern, and my stress-test does not move the verdict: the pipeline, dataset, and released artifacts are credible enough that the concern should be resolved by an independent blinded re-evaluation rather than by rejection. The concrete re-test above would settle whether the 18-point margin is real or an artifact of the evaluation protocol.","tokens_in":18231,"tokens_out":6577,"duration_ms":64556,"concrete_test":"Recruit at least 30 naive annotators (e.g., MTurk) who are not authors; present the same 500 test pairs per resolution with each method's output shown individually in randomized order with method identity hidden; include 10% repeated images as attention checks; compute Fleiss' kappa, per-method success rates, and a bootstrap 95% CI for the difference RORem minus the best baseline. If the lower CI bound at 512 exceeds 10 points and kappa > 0.6, the headline margin is credible; if the CI includes 0 or kappa < 0.4, the central claim is not established. Also report Dphi success rates on the same pairs; if the Dphi-human discrepancy for PPT/DesignEdit remains above 5 points, Dphi should not be cited as corroboration.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim ('more than 18% success-rate gain') is established only by the user study in Sec. 4.1 and Appendix A: five volunteers, 500 pairs per resolution, multiple-choice selection of successful methods, no blinding of method identity, no inter-annotator agreement, no per-method variance, and no confidence intervals. Because the volunteer pool is unreported and may overlap with the authors' lab, a systematic preference for RORem's output style would directly manufacture the reported 20.4-point (512) and 17.8-point (1024) margins. The paper's secondary corroboration via the Dphi discriminator does not rescue the claim: in Table 2, Dphi-vs-human deviations are 6.8 points for Lama at 512, 7.8 points for PPT at 1024, and 5.4 points for CLIPAway at 1024, contradicting the Sec. 4.2 statement that 'the deviation is less than 3% in most cases.' Dphi is also trained partly on RORem outputs and refined with labels from the same human protocol, so it is not an independent confirmation. A smaller but related issue: the abstract's 'more than 18%' is not true at 1024, where the margin is 17.8 points.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RORem, a diffusion-based object remover trained with a human-in-the-loop data generation pipeline. Starting from two open-source datasets (RORD and Mulan) with about 60K triplets, the authors fine-tune an SDXL inpainting model, use it to generate candidate removal pairs from OpenImages, filter them through human annotation and a learned discriminator Dphi, iteratively retrain, and finally obtain 201,134 training pairs. A four-step distilled variant (RORem-4S) runs in about 0.5 seconds per image. The central claim, stated in the abstract, is that RORem improves the object removal success rate over previous methods by more than 18%, based on Table 2's human-judged success rates (76.2% vs 55.8% at 512x512; 70.2% vs 52.4% at 1024x1024).","tokens_in":18503,"tokens_out":6362,"duration_ms":56708,"significance":"If the reported success-rate gains are confirmed, RORem would be a substantial practical advance in object removal reliability, and the released dataset, code, and model would be a useful community resource. The paper also demonstrates a credible pipeline for semi-supervised data collection with human feedback and a strong distilled efficient variant. However, the headline claim is currently supported mainly by a small, possibly unblinded user study, and the paper's secondary Dphi corroboration is not independent; these issues must be addressed before the central claim can be considered established.","major_comments":[{"comment":"The headline claim of 'more than 18%' improvement is established solely by the user study in Sec. 4.1 and Appendix A, which used five volunteers, each annotating 500 pairs per resolution, and reports only the averaged success rate without inter-annotator agreement, per-method variance, or confidence intervals. Because the volunteer recruitment and their relationship to the authors are not described, and because the interface in Fig. 9 shows the editing results of all methods side by side without blinding, the measured 20.4-point (512) and 17.8-point (1024) margins may be systematically biased toward RORem. Please provide a blinded, preregistered user study with a larger and independent rater pool, report agreement statistics (e.g., Fleiss' kappa) and per-method confidence intervals, and describe how volunteers were recruited and instructed.","section":"Sec. 4.1 and Appendix A"},{"comment":"The abstract's 'more than 18%' is inaccurate for the 1024x1024 resolution, where Table 2 shows a margin of 70.2% - 52.4% = 17.8 percentage points. In addition, Sec. 4.2 states that Dphi's success rates deviate from human annotation by less than 3% in most cases, but Table 2 shows deviations of 6.8 points for Lama at 512 (55.4 vs 48.6), 7.8 points for PPT at 1024 (46.8 vs 54.6), and 5.4 points for CLIPAway at 1024 (23.8 vs 29.2). Please rephrase the quantitative claims and re-analyze the Dphi alignment, or restrict the claim to the 512-resolution result.","section":"Abstract and Sec. 4.2"},{"comment":"The Dphi metric is not an independent corroboration of the human study. Dphi is trained on labels from the same human-feedback protocol, partly on RORem outputs (Table 4), and its validation in Table 3 and Appendix B uses a test set of RORem-generated outputs only. Therefore, the Dphi success-rate columns in Table 2 for the baseline methods are not validated, and the statement that Dphi 'effectively aligns with human preferences' overstates the evidence. Please either validate Dphi on held-out human labels for each method separately or remove the Dphi-based success-rate comparisons from the main table.","section":"Sec. 4.3 and Table 3"}],"minor_comments":[{"comment":"The test protocol applies a substantial mask dilation (kernel sizes 50 and 100 for 512 and 1024 resolutions) to accommodate methods like PPT, but the manuscript does not state whether RORem was trained with masks of comparable dilation. If RORem was trained on tighter masks, the shared dilated mask could disadvantage RORem or, conversely, the dilation could mask RORem's sensitivity to fine masks. Please clarify the mask preprocessing for training versus testing and, if necessary, evaluate with both original and dilated masks.","section":"Sec. 4.1"},{"comment":"The relationship between the training-time human annotation (10 annotators) and the final evaluation (5 volunteers) should be clarified; currently the text in Sec. 4.1 and Appendix A could be misread as the same group, leading to confusion about whether the final evaluation is independent.","section":"Sec. 3.2 and Appendix A"},{"comment":"Table 4 contains a typo: 'dowmsample' should be 'downsample', and the column header 'Baseline' should be 'Baselines'. Please correct these in the final version.","section":"Table 4"},{"comment":"The phrase 'four diffussion steps' should be corrected to 'four diffusion steps'. Additionally, the rank of LoRA layers (64 for distillation, 4 for Dphi) should be mentioned explicitly in the main text to avoid ambiguity.","section":"Sec. 3.4"},{"comment":"Because the test set is sampled from OpenImages and filtered with the same keyword and mask-size rules as the training data, the evaluation is in-distribution for RORem. The paper should acknowledge this as a limitation and ideally include an additional out-of-distribution test set to ensure the reported improvement is not an artifact of distribution matching.","section":"Sec. 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's main contribution is a model with a claimed large improvement in human-judged success rate, but that claim is not yet statistically grounded. I would require a larger, blinded user study with agreement statistics and a corrected treatment of the Dphi metric before accepting. The artifact release and the semi-supervised pipeline are valuable, so I am not recommending rejection, but the central claim needs substantially stronger evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The main thing you should know: this is a solid, useful object-removal paper whose value is the 200K-pair dataset and the human-in-the-loop bootstrap that built it, not a conceptual breakthrough. The headline 18-point gain over the second-best method is plausible but the evidence for it has a specific soft spot that needs an independent re-test.\n\nWhere it earns credit: The data-generation pipeline is clearly described and the numbers in Table 1 are internally consistent — the success rate climbs from 7.6% to 76.2% as the dataset grows. The discriminator is a sensible RLHF-style preference proxy, and the final four-step distilled model at 0.5s is a practical addition. The paper also acknowledges its failure cases (fingers, small faces). If the dataset and code are actually released, that is a real contribution for the subfield.\n\nSoft spots: The user study is small — five volunteers, 1000 samples each — and no inter-annotator agreement, per-method variance, or confidence intervals are reported. The appendix says display order is shuffled, so volunteers are not told which method is which, but the pool is unreported and could be lab-biased. That makes the central 76.2-vs-55.8 margin less solid than it looks. The secondary Dphi metric mostly agrees, but not uniformly: deviations from human scores at 1024 reach 7.8 points for PPT and 5.4 for CLIPAway, so the paper's claim that deviation is 'less than 3% in most cases' is misleading. And the abstract's 'more than 18%' is simply wrong at 1024, where the margin is 17.8 points — minor, but careless.\n\nThe test set is drawn from the same OpenImages distribution and filtered with the same rules as training, so the in-distribution result is real but generalizability beyond that distribution remains open. That is a standard limitation in this area, not a fatal one.\n\nVerdict: The central claim holds up in the sense that RORem appears to be a strong, robust object remover on the tested distribution. The margin over prior art is likely large, but its exact size needs an independent, ideally larger, blinded evaluation with agreement statistics. This paper deserves a serious referee: the dataset release alone justifies engaging with it, and the evaluation gaps are addressable in revision. I would send it to review with a request to tighten the user study and fix the 'more than 18%' wording.","headline":"Solid dataset-plus-pipeline paper whose headline 18% margin is plausible but rests on a small, under-reported user study; worth reviewing for the 200K-pair dataset alone.","tokens_in":19068,"tokens_out":2961,"would_cite":true,"duration_ms":27063,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Object remover trained with human feedback beats rivals by 18 points.","keywords":["object removal","image inpainting","human-in-the-loop","semi-supervised learning","diffusion models","training data generation","discriminator","model distillation"],"falsifier":"Recruit a fresh panel of at least 20 annotators who are blind to method identity and method order, have them judge the same 500 pairs at each resolution, and compute the mean success-rate gap between RORem and the runner-up with a 95% confidence interval; if the interval includes zero or the gap is materially below 18 points, the paper's central performance claim fails.","tokens_in":17983,"feed_emoji":"🖼️","tokens_out":5784,"duration_ms":49301,"temperature":0.7,"pith_summary":"The paper argues that the low success rate of object removal models comes from a training-data problem: self-supervised inpainting teaches the model to reconstruct whatever was masked, so at test time it hesitates between redrawing the object and restoring the background. The fix is a semi-supervised, human-in-the-loop pipeline that builds a large dataset of true removal triplets (source image, mask, clean removal result). Starting from 60K triplets from existing datasets, an initial model proposes removals, humans filter the good ones, and a discriminator trained on those judgments automates further filtering; several rounds produce over 200K pairs. Fine-tuning an SDXL inpainting model on this data yields RORem, which human judges mark as successful 76.2% of the time at 512x512 and 70.2% at 1024x1024, about 18 points above the next-best method. A distilled four-step version runs in about half a second per image, making the reliability gain usable in practice.","feed_headline":"Object remover trained with human feedback beats rivals by 18 points","feed_subtitle":"A 200K-pair dataset built by humans and a learned judge lifts removal success to 76 percent.","key_machinery":"The load-bearing object is the training triplet (source image, mask, true removal result), which removes the self-supervised ambiguity. Around it the paper builds a data flywheel: an initial SDXL inpainting model proposes removals, human annotators label each output as success or failure, and a discriminator built from the down/middle blocks of SDXL with trainable LoRA layers learns to predict those labels, scoring candidate pairs above a 0.9 threshold for automatic inclusion. The same triplet formulation is used in every training round, with the source image's masked region blanked before concatenation, and the final fast model is obtained by distilling the trained RORem into four diffusion steps using a latent-consistency-style objective with LoRA layers.","core_discovery":"The central claim is that object removal reliability is bottlenecked by the ambiguity of self-supervised random-mask training, and that the bottleneck can be broken by supplying the model with ground-truth removal pairs at scale. The paper shows that a model trained on 60K pairs from existing video and synthetic datasets succeeds less than half the time, but that each round of human filtering plus discriminator-automated filtering adds tens of thousands of high-quality triplets and steadily raises success rate from 7.6% to 76.2% as the training set grows past 200K. The resulting RORem outperforms seven prior methods on human-judged success rate at both 512x512 and 1024x1024, and its distilled variant keeps most of the gain while cutting inference time by roughly 88% at 512x512.","pith_inferences":["The same triplet-plus-discriminator flywheel should transfer to other image-editing tasks such as object insertion, replacement, or style harmonization, because the paper's mechanism only assumes that a preference discriminator can imitate human quality judgments.","A stronger test of the method would be an independent user study with more annotators, inter-annotator agreement, and confidence intervals; the paper reports averages only, so the 18-point claim is as strong as those five volunteers' consistency.","The released discriminator could serve as a cheap automated evaluation metric for future object removers, provided it is periodically re-calibrated on new failure modes, since the paper shows it aligns with human labels within about 3% on its own method.","A direct extension would test RORem on images outside the OpenImages sampling distribution (e.g., user-captured photos with unusual backgrounds), since the test set is filtered by the same keyword and mask-size rules as the training data."],"forward_implications":["At 512x512, human judges rate RORem as successful on 76.2% of cases versus 55.8% for the second-best method, and at 1024x1024 the margin is 70.2% versus 52.4%.","The 200K-pair dataset and trained model are released publicly, letting downstream applications fine-tune on top of a reliable removal prior rather than starting from ambiguous self-supervised weights.","The four-step distilled variant runs in 0.50 seconds per image at 512x512 and 0.83 seconds at 1024x1024, with success rate drops of only 1.4 and 2.8 points.","Success rate rises monotonically with dataset size across the annotation rounds, from 38.6% after initialization to 76.2% after the final high-resolution fine-tuning."],"supporting_citations":[{"why":"Supplies 15K real-world removal triplets extracted from fixed-camera videos, the largest clean source for initial training.","marker":"[47]"},{"why":"Adds category-diverse synthetic removal pairs from COCO and LAION images, complementing RORD's mostly-human scenes.","marker":"[56]"},{"why":"Provides the source images and masks for the test sets and for the human and automated annotation loops that grow the dataset.","marker":"[20]"},{"why":"The pre-trained SDXL inpainting model that is fine-tuned in every round and whose blocks seed the discriminator backbone.","marker":"[42]"},{"why":"Supplies the latent consistency distillation objective used to compress RORem to four diffusion steps.","marker":"[36]"},{"why":"The strongest CNN baseline for large-mask inpainting, used as a comparison point in the user study and metrics.","marker":"[54]"},{"why":"The second-best method at 512x512 that RORem's 18-point margin is measured against.","marker":"[72]"},{"why":"The second-best training-free baseline at 1024x1024, used in the high-resolution comparison.","marker":"[17]"}],"fun_headline_variants":["Human-in-the-loop boosts object removal success by 18%","RORem: object removal trained with human feedback, 76% success","From 7.6% to 76%: human-guided data beats self-supervised","Object remover learns from human picks, beats rivals by 18 points","Semi-supervised training with human filter lifts removal success"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline 18-point advantage rests on five volunteers' yes/no judgments of 500 image pairs per resolution, with no reported inter-rater agreement, variance, or confidence intervals; if those judgments are noisy or biased toward the authors' model, the margin is not established.","fun_headline_variants_meta":{"raw":{"variants":["Human-in-the-loop boosts object removal success by 18%","RORem: object removal trained with human feedback, 76% success","From 7.6% to 76%: human-guided data beats self-supervised","Object remover learns from human picks, beats rivals by 18 points","Semi-supervised training with human filter lifts removal success"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000694,"raw_usage":{"total_tokens":3155,"prompt_tokens":973,"completion_tokens":2182,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":2086}},"tokens_in":589,"tokens_out":2182,"duration_ms":13705,"temperature":1.0,"reasoning_tokens":2086,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:43:54.528943+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recruit a fresh panel of at least 20 annotators who are blind to method identity and method order, have them judge the same 500 pairs at each resolution, and compute the mean success-rate gap between RORem and the runner-up with a 95% confidence interval; if the interval includes zero or the gap is materially below 18 points, the paper's central performance claim fails.","supporting_citations":[{"cited_title":"Rord: A real-world object removal dataset","cited_arxiv_id":null,"evidence_quote":"Supplies 15K real-world removal triplets extracted from fixed-camera videos, the largest clean source for initial training."},{"cited_title":"Mulan: A multi layer anno- tated dataset for controllable text-to-image generation","cited_arxiv_id":null,"evidence_quote":"Adds category-diverse synthetic removal pairs from COCO and LAION images, complementing RORD's mostly-human scenes."},{"cited_title":"The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale","cited_arxiv_id":null,"evidence_quote":"Provides the source images and masks for the test sets and for the human and automated annotation loops that grow the dataset."},{"cited_title":"Resolution-robust large mask inpainting with fourier convolutions","cited_arxiv_id":null,"evidence_quote":"The strongest CNN baseline for large-mask inpainting, used as a comparison point in the user study and metrics."}],"review_version":1}