Pith. sign in

REVIEW 4 major objections 4 minor 24 references

OpenRR-1k: A Scalable Dataset for Real-World Reflection Removal

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces OpenRR-1k, a dataset of 1,000 pixel-aligned real-world transmission-reflection image pairs built by cleaning smartphone AI outputs, and shows that fine-tuning existing reflection removal models on it raises PSNR by…

desk verdict Novel scalable protocol for real-world reflection data, but the ground-truth labels are unvalidated and the evaluation is partly circular; deserves review with major revision. read the letter →

arxiv 2506.08299 v1 pith:TNTI7PBL submitted 2025-06-10 cs.CV

Kangning Yang , Ling Ouyang , Huiming Sun , Jie Cai , Lan Fu , Jiaming Ding , Chiu Man Ho , Zibo Meng This is my paper
classification cs.CV
keywords reflectionremovalsingleimagereal-worlddatasetpixel-alignedpairscollectionprotocolfine-tuningbenchmarkevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the main bottleneck in single-image reflection removal is data, not model architecture, and proposes a cheap way to get that data: photograph real reflections, run each photo through a commercial AI reflection-removal tool, then manually clean up residual reflections in an image editor. The result is OpenRR-1k, 1,000 perfectly pixel-aligned pairs of blended and transmission images drawn from everyday scenes with varied lighting and glass surfaces. The authors show that after fine-tuning on this dataset, all five tested methods improve, with their own baseline gaining over 7 dB PSNR on the OpenRR-1k test set and also improving on older benchmarks. If the labels are trustworthy, the claim that matters is that the field's data bottleneck can be removed at low cost and at scale.

What carries the argument

The central mechanism is the clean-the-blend collection protocol: instead of capturing a ground-truth transmission image by removing glass or covering the scene with black cloth, the protocol starts with a real blended photo, applies a commercial AI reflection-removal tool to get a rough clean image, and then uses manual image-editing cleanup to remove residual reflections and artifacts. This guarantees pixel-level alignment because the ground truth is edited from the exact same image as the input, and it permits arbitrary real-world capture, yielding diversity in reflections, scenes, and lighting. The paper also introduces a NAFNet-based baseline with an expanded bottleneck of 12 middle blocks, used both to benchmark the dataset and to demonstrate the fine-tuning gains.

What would settle it

Select a random subset of OpenRR-1k test pairs, revisit the depicted locations, photograph the same scenes without any glass present, align those physically clean captures with the published transmission images, and measure their difference; if a substantial fraction of pairs show residuals or details that the no-glass captures lack (for example, PSNR well below 30 dB), the claim of high-quality, perfectly aligned ground truth fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that high-quality, perfectly aligned, in-the-wild reflection removal training data can be produced without physically removing glass or blocking light: take real photos containing reflections, use an off-the-shelf AI reflection removal tool to produce an initial clean image, then manually edit away residual reflections with professional image editing software. Because the transmission image is derived from the same photograph as the blended image, the pairs are aligned by construction, and because no capture restrictions are imposed, the data reflect natural diversity in lighting, glass type, and scene content. Using this protocol the authors assembled OpenRR-1k with 1,000 pairs split 800/100/100 for training, validation, and testing. Benchmarking four existing reflection removal methods plus their own NAFNet-based baseline, they find that all methods perform noticeably worse on OpenRR-1k than on older datasets when using pretrained weights, but after fine-tuning on OpenRR-1k's training set performance improves across almost all methods and benchmarks, with their baseline's PSNR rising from 24.15 dB to 31.93 dB on the OpenRR-1k test set.

Load-bearing premise

The dataset's labels are trustworthy as ground truth only if the commercial AI tool plus manual cleanup truly recovers the reflection-free scene, and the paper provides no independent verification that residual reflections or hallucinated details are absent from the transmission images.

Editorial extensions

If this is right

  • Existing single-image reflection removal methods generalize poorly to OpenRR-1k's real-world images without fine-tuning, so their strong scores on older synthetic and semi-synthetic benchmarks have overstated real-world robustness.
  • Fine-tuning on OpenRR-1k improves all five tested methods on the OpenRR-1k test set and often on older benchmarks, indicating the dataset provides useful training signal, not just a harder evaluation set.
  • Because the protocol needs no specialized equipment, controlled lighting, or physical manipulation of glass, it can be crowdsourced and scaled beyond 1,000 pairs.
  • The fixed 80/10/10 split gives the community a standard benchmark for comparing future real-world reflection removal methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A consequence the authors leave implicit: since the ground truth is produced by a commercial AI tool plus manual editing, the dataset's quality ceiling is set by that tool's ability to reconstruct the true scene, and any systematic hallucination or smoothing it introduces could be baked into the labels.
  • A testable extension would be to independently verify label fidelity by physically capturing the same scenes without glass for a subset of pairs and measuring agreement with the published transmission images.
  • The protocol could transfer to other layered-image tasks such as dehazing or shadow removal wherever a commercial or heuristic cleaner produces a plausible first estimate that human annotators can refine.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a scalable protocol for constructing real-world reflection-removal training data. Instead of physically capturing clean transmission images, the protocol runs OPPO's commercial AI reflection-removal tool on in-the-wild blended images and then manually cleans residual artifacts with Photoshop/MeituPic, yielding 1,000 pixel-aligned transmission-reflection pairs (OpenRR-1k). The authors benchmark four existing SIRR methods plus a NAFNet-based baseline under two strategies: S1 (off-the-shelf or baseline training without OpenRR-1k) and S2 (fine-tuning on OpenRR-1k train). They report that S2 improves performance, most dramatically on the OpenRR-1k validation/test sets, with the baseline's PSNR increasing by over 7 dB, and conclude that the dataset improves generalization in challenging real-world environments. The dataset is publicly released and the paper includes comparisons on Nature, Real, and SIR2.

Significance. If the label-quality concerns are resolved, OpenRR-1k would be a useful contribution: it is larger and more diverse than Nature/Real, its pairs are aligned by construction, and its release plus the S1/S2 benchmark provide a common evaluation protocol. The paper honestly reports that all S1 methods underperform the input image on OpenRR-1k, which is a thought-provoking negative result. The external benchmark gains after fine-tuning (e.g., DSRNet's Nature PSNR rising from 25.07 to 26.33) offer non-circular evidence that the data may transfer. However, the central claim of 'high-quality ground truth' is not validated: the labels are generated by a commercial AI model plus subjective manual cleanup, with no independent quantitative error analysis, no inter-annotator checks, and no comparison against physically captured clean transmissions. Because the training labels, test labels, and the AI tool itself all come from the same source, the dataset's value for benchmarking and training is currently not established.

major comments (4)
  1. [Section 3.1] The ground-truth transmission images are produced solely by OPPO's AI reflection-removal software followed by manual cleanup in Photoshop/MeituPic. The paper asserts that 'the final processed images are of high quality and suitable for training and evaluation purposes,' but it provides no quantitative validation: no comparison against independently captured clean transmission images, no measurement of residual reflections, no inter-annotator agreement on the three subjective criteria (Cleanliness, Artifacts, Overall Image Quality). Since the dataset's claimed high quality is load-bearing for both the training and benchmarking claims, this missing validation is a central issue. Please add a physically captured ground-truth subset (e.g., black-cloth or two-shot captures), report residual-reflection statistics, and quantify annotator agreement.
  2. [Table 2 and Section 4.3] On OpenRR-1k val/test, the Input Image row (PSNR 26.37/26.20, SSIM 0.943/0.939, LPIPS 0.077/0.078) is better than every S1 method, whereas on Nature/Real/SIR2 the input PSNRs are much lower (20.46/19.07/22.76). This asymmetry is consistent with the OpenRR-1k labels being much closer to the input than true reflection-free transmissions, for example because the AI tool under-removes reflections or the manual cleanup favors conservative edits. The paper's interpretation that existing methods 'perform poorly' on true real-world data is only one possible explanation; the alternative is that the labels are biased toward the input. Please disambiguate these hypotheses, for instance by measuring residual reflection energy in the labels or by validating on a physically captured subset.
  3. [Sections 4.2 and 4.3] The S2 versus S1 comparison is not a controlled experiment. For the proposed baseline, S1 uses 60 training epochs, while S2 adds 100 additional epochs on OpenRR-1k; for the pretrained methods, S2 fine-tuning also adds extra optimization. Thus the reported improvements, including the headline 'PSNR increased by over 7 dB' on OpenRR-1k, could be due to additional training steps rather than to the dataset's content. Moreover, the OpenRR-1k test labels are generated by the same OPPO+manual pipeline as the training labels, making the gain partly circular: a model can improve by reproducing the label-generation distribution. Please include a control that trains S1 for the same total number of epochs without OpenRR-1k data, and report gains on external benchmarks separately with confidence intervals or paired significance tests.
  4. [Section 3.1] The protocol claims 'perfectly aligned' / 'pixel-level alignment' because no physical capture setup is needed. However, the OPPO AI tool and the subsequent manual editing can modify image content (inpainting, over-smoothing, color shifts), not just remove reflections. Alignment of the glass-rendering geometry is therefore not the only possible source of misalignment; content-level changes can make the label inconsistent with the transmission layer. Please provide evidence that the manual and AI editing do not introduce systematic content changes, for example by reporting per-pixel alignment statistics on a subset with physically captured ground truth or by showing that the edited labels preserve the original texture frequencies.
minor comments (4)
  1. [Throughout] There are several typographical issues, including 'DA TASET' in the title and 'Speficially' in Section 3.1; please proofread the manuscript.
  2. [Table 1] The table reports 'Pair Number' and average resolution for previous datasets, but the split information for RRW is not shown; adding the train/test split and license information would improve comparability.
  3. [Section 3.2] The 80/10/10 split is mentioned, but the paper does not state whether the split was random, stratified by scene category, or balanced by lighting condition; please clarify to help future users reproduce the benchmark.
  4. [Section 4.2] For the S2 experiments, the fine-tuning protocol for the pretrained methods is not described in enough detail (learning rates, number of epochs, data augmentation); please provide the exact fine-tuning settings so that the comparison is reproducible.

Circularity Check

1 steps flagged · score 6.0 of 10

Partial circularity: OpenRR-1k ground truth is generated by the same OPPO AI+manual pipeline used to create training labels, so the internal benchmark partly measures fit to that pipeline; external benchmarks provide partial independent support.

  1. self definitional [Sec. 3.1 (Dataset Collection Protocol); Sec. 4.1/4.3 (S1 vs S2, Table 2, Eq. 1)]
    "We adopted the OPPO smartphone’s AI-based reflection removal software to obtain the initial reflection removal results. ... After precise manual adjustments, the final processed images are of high quality and suitable for training and evaluation purposes. ... S2: We fine-tuned the proposed baseline model and four comparison methods with the OpenRR-1ktrain dataset ... our baseline model achieves the best results on multiple datasets, including OpenRR-1k val (PSNR: 31.74, SSIM: 0.965, LPIPS: 0.040) and OpenRR-1k test (PSNR: 31.93, SSIM: 0.964, LPIPS: 0.038)."

    The dataset's ground-truth transmission image is defined as the output of OPPO's AI reflection-removal software plus subjective manual cleanup, with no independent capture of a clean transmission image. The training loss (Eq. 1) and the Table 2 evaluation metrics compare model outputs against this same kind of T on OpenRR-1k val/test. Because the task is reflection removal, the reference standard is itself a reflection-removal result: a model that learns to reproduce the OPPO+manual input-output mapping will score high on the internal benchmark essentially by construction. The reported >7 dB S2-vs-S1 gain on OpenRR-1k therefore partly reflects fitting the label-generation pipeline rather than recovering true physical transmissions.

full rationale

There is no load-bearing self-citation or imported uniqueness theorem; references [13] and [14] are background material. The main circularity is self-definitional: OpenRR-1k's 'ground-truth' transmission images are produced by an unvalidated commercial AI reflection-removal tool plus manual editing, a pipeline explicitly chosen to avoid glass removal or black cloth. The internal evaluation on OpenRR-1k val/test therefore measures agreement with that same pipeline, and the headline fine-tuning gain partially reduces to reproducing the label-generation mapping. However, the paper also reports consistent S2 improvements on Nature, Real, and SIR2, whose ground truths come from independent physical capture setups, so the central claim that fine-tuning on OpenRR-1k improves reflection removal does not reduce entirely to the self-defined labels. On the strength of those external benchmarks, the circularity is partial; the unvalidated label generation remains a serious correctness risk but does not by itself make the paper's derivation wholly circular. Score 6 reflects the partial self-reference of the internal benchmark, not an accusation of intentional circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim rests entirely on the unverified correctness of the AI-plus-manual label generation process. No independent physical ground truth, such as polarimetric or time-lapse capture, is used to validate the labels. The dataset diversity and the manual quality bar are also asserted without quantitative support.

assumptions (3)
  • domain assumption Off-the-shelf OPPO AI reflection removal plus manual editing produces accurate, artifact-free transmission ground truth.
    Section 3.1 states this without any external validation; all dataset labels depend on it.
  • domain assumption The three manual quality criteria (cleanliness, no artifacts, overall quality) are sufficient to guarantee label correctness.
    Section 3.1; these are subjective and no inter-annotator agreement is reported.
  • domain assumption The collected images are diverse and representative of real-world reflection scenarios.
    Section 3.2; the social media scraping and no-restriction capture are not described in enough detail to assess coverage or selection bias.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OpenRR-1k: A Scalable Dataset for Real-World Reflection Removal." pith.science (2026). https://pith.science/paper/TNTI7PBL

@misc{pith2026250608299,
  author       = {Pith},
  title        = {Pith review of: OpenRR-1k: A Scalable Dataset for Real-World Reflection Removal},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TNTI7PBL}},
  note         = {Machine review of arXiv:2506.08299}
}
read the original abstract

Reflection removal technology plays a crucial role in photography and computer vision applications. However, existing techniques are hindered by the lack of high-quality in-the-wild datasets. In this paper, we propose a novel paradigm for collecting reflection datasets from a fresh perspective. Our approach is convenient, cost-effective, and scalable, while ensuring that the collected data pairs are of high quality, perfectly aligned, and represent natural and diverse scenarios. Following this paradigm, we collect a Real-world, Diverse, and Pixel-aligned dataset (named OpenRR-1k dataset), which contains 1,000 high-quality transmission-reflection image pairs collected in the wild. Through the analysis of several reflection removal methods and benchmark evaluation experiments on our dataset, we demonstrate its effectiveness in improving robustness in challenging real-world environments. Our dataset is available at https://github.com/caijie0620/OpenRR-1k.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 22 canonical work pages

  1. [1]

    OpenRR-1k: A Scalable Dataset for Real-World Reflection Removal

    INTRODUCTION Single image reflection removal (SIRR) is a critical task in im- age processing, focusing on recovering the true scene behind reflections from reflective surfaces (e.g., transparent glasses). This is crucial in fields such as autonomous driving, medi- cal imaging [1, 2], and augmented reality. Over the years, various techniques have been prop...

    work page Pith review arXiv 2025

  2. [2]

    RELA TED WORK Currently, public datasets supporting SIRR can be divided into two categories: fully-synthetic and semi-synthetic datasets. For fully-synthetic datasets, this technique typically involves selecting two clean images (without reflections) and combin- ing them with different coefficients to create a synthetic image with reflections, which serve...

    work page

  3. [3]

    Dataset Collection Protocol As shown in Fig

    METHODOLOGY 3.1. Dataset Collection Protocol As shown in Fig. 1, our proposed data collection protocol consists of two main steps. The first step involves using a proven off-the-shelf tool to initially remove reflections from the images. We adopted the OPPO smartphone’s AI-based reflection removal software to obtain the initial reflection removal results....

    work page 2017

  4. [4]

    conv2 2”, “conv3 2

    EXPERIMENTS 4.1. Experiment setting To conduct a comprehensive benchmark evaluation on the OpenRR-1k dataset, we proposed a new NAFNet-based base- line model by adapting the widely-used restoration architec- ture introduced in [18]. For better representation learning ability, we expand the network’s bottleneck capacity by in- creasing the number of middle...

    work page

  5. [5]

    Our pipeline provides researchers with a more convenient way to collect diverse, high-quality true real- world data samples at low cost

    CONCLUSION In this paper, we propose a novel reflection removal pipeline that addresses a long-standing challenge in this field from a new perspective. Our pipeline provides researchers with a more convenient way to collect diverse, high-quality true real- world data samples at low cost. Using this pipeline, we con- structed the OpenRR-1k dataset and cond...

    work page

  6. [6]

    Automatic acetowhite lesion segmenta- tion via specular reflection removal and deep attention network,

    Zijie Yue, Shuai Ding, Xiaojian Li, Shanlin Yang, and Youtao Zhang, “Automatic acetowhite lesion segmenta- tion via specular reflection removal and deep attention network,” IEEE Journal of Biomedical and Health In- formatics, vol. 25, no. 9, pp. 3529–3540, 2021

    work page 2021

  7. [7]

    Deep learning based end-to-end specular reflection re- moval for medical endoscopic images,

    Chi-Sheng Shih, Yu-Cheng Liao, and Ching-Ting Tan, “Deep learning based end-to-end specular reflection re- moval for medical endoscopic images,” in Proceedings of the 2023 International Conference on Research in Adaptive and Convergent Systems, 2023, pp. 1–9

    work page 2023

  8. [8]

    A generic deep architecture for sin- gle image reflection removal and image smoothing,

    Qingnan Fan, Jiaolong Yang, Gang Hua, Baoquan Chen, and David Wipf, “A generic deep architecture for sin- gle image reflection removal and image smoothing,” in Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 3238–3247

    work page 2017

Show all 24 references
  1. [9]

    Single image reflection removal through cas- caded refinement,

    Chao Li, Yixiao Yang, Kun He, Stephen Lin, and John E Hopcroft, “Single image reflection removal through cas- caded refinement,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 3565–3574

    2020

  2. [10]

    Ro- bust single image reflection removal against adversarial attacks,

    Zhenbo Song, Zhenyuan Zhang, Kaihao Zhang, Wenhan Luo, Zhaoxin Fan, Wenqi Ren, and Jianfeng Lu, “Ro- bust single image reflection removal against adversarial attacks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 24688–24698

    2023

  3. [11]

    Benchmarking single-image reflec- tion removal algorithms,

    Renjie Wan, Boxin Shi, Ling-Yu Duan, Ah-Hwee Tan, and Alex C Kot, “Benchmarking single-image reflec- tion removal algorithms,” in Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 3922–3930

    2017

  4. [12]

    Single im- age reflection separation with perceptual losses,

    Xuaner Zhang, Ren Ng, and Qifeng Chen, “Single im- age reflection separation with perceptual losses,” inPro- ceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 4786–4794

    2018

  5. [13]

    Polarized re- flection removal with perfect alignment in the wild,

    Chenyang Lei, Xuhua Huang, Mengdi Zhang, Qiong Yan, Wenxiu Sun, and Qifeng Chen, “Polarized re- flection removal with perfect alignment in the wild,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 1750–1758

    2020

  6. [14]

    A categorized reflection removal dataset with diverse real- world scenes,

    Chenyang Lei, Xuhua Huang, Chenyang Qi, Yankun Zhao, Wenxiu Sun, Qiong Yan, and Qifeng Chen, “A categorized reflection removal dataset with diverse real- world scenes,” in Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition , 2022, pp. 3040–3048

    2022

  7. [15]

    Revisiting single image reflection removal in the wild,

    Yurui Zhu, Xueyang Fu, Peng-Tao Jiang, Hao Zhang, Qibin Sun, Jinwei Chen, Zheng-Jun Zha, and Bo Li, “Revisiting single image reflection removal in the wild,” in Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, 2024, pp. 25468– 25478

    2024

  8. [16]

    Benchmarking ultra-high-definition image reflection removal,

    Zhenyuan Zhang, Zhenbo Song, Kaihao Zhang, Zhaoxin Fan, and Jianfeng Lu, “Benchmarking ultra-high-definition image reflection removal,” arXiv preprint arXiv:2308.00265, 2023

    2023 arXiv

  9. [17]

    Robust sepa- ration of reflection from multiple images,

    Xiaojie Guo, Xiaochun Cao, and Yi Ma, “Robust sepa- ration of reflection from multiple images,” in Proceed- ings of the IEEE conference on computer vision and pat- tern recognition, 2014, pp. 2187–2194

    2014

  10. [18]

    NTIRE 2025 challenge on single image reflection removal in the wild: Datasets, methods and results,

    Kangning Yang, Jie Cai, Ling Ouyang, Florin- Alexandru Vasluianu, Radu Timofte, Jiaming Ding, Huiming Sun, Lan Fu, Jinlong Li, Chiu Man Ho, Zibo Meng, et al., “NTIRE 2025 challenge on single image reflection removal in the wild: Datasets, methods and results,” in Proceedings o...

    2025

  11. [19]

    Survey on single-image reflection removal using deep learning techniques,

    Kangning Yang, Huiming Sun, Jie Cai, Lan Fu, Ji- aming Ding, Jinlong Li, Chiu Man Ho, and Zibo Meng, “Survey on single-image reflection removal using deep learning techniques,” arXiv preprint arXiv:2502.08836, 2025

    2025 arXiv

  12. [20]

    Single image reflection removal exploiting misaligned training data and network enhancements,

    Kaixuan Wei, Jiaolong Yang, Ying Fu, David Wipf, and Hua Huang, “Single image reflection removal exploiting misaligned training data and network enhancements,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 8178–8187

    2019

  13. [21]

    Single image reflection separation via component synergy,

    Qiming Hu and Xiaojie Guo, “Single image reflection separation via component synergy,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 13138–13147

    2023

  14. [22]

    Two-stage single image reflec- tion removal with reflection-aware guidance,

    Yu Li, Ming Liu, Yaling Yi, Qince Li, Dongwei Ren, and Wangmeng Zuo, “Two-stage single image reflec- tion removal with reflection-aware guidance,” Applied Intelligence, vol. 53, no. 16, pp. 19433–19448, 2023

    2023

  15. [23]

    Simple baselines for image restoration,

    Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun, “Simple baselines for image restoration,” in Eu- ropean Conference on Computer Vision. Springer, 2022, pp. 17–33

    2022

  16. [24]

    The pascal visual object classes (voc) challenge,

    Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman, “The pascal visual object classes (voc) challenge,” Interna- tional journal of computer vision , vol. 88, no. 2, pp. 303–338, 2010

    2010

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.