{"id":"6a754e09-b651-4e2d-8ede-11119e324d68","arxiv_id":"1909.00968","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A U-Net inpainting model with learnable forward and reverse attention maps improves irregular-hole filling over partial convolution and other state-of-the-art methods.","lead":"This paper proposes learnable bidirectional attention maps for image inpainting, replacing the handcrafted mask renormalization of partial convolution with parameters learned end-to-end. The method reports sharper, more coherent filled regions than previous CNN inpainting models on Paris StreetView and Places.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1's PConv baseline is copied from [17] under a different training, mask, and loss protocol; the 0.2-0.4 dB PSNR margins may not survive a fair rerun.","rationale":"The reader's weakest assumption is exactly the load-bearing point: the quantitative advantage over PConv depends on numbers copied from a different experimental setup. I agree with this identification. The margins are small (0.19-0.42 dB PSNR, up to 0.019 SSIM), and LBAM is actually worse than PConv* on mean l1 in the (0.1,0.2] and (0.2,0.3] bins, so the claimed advantage is not consistent across metrics. The user study (63.2% vs 15.2%) is subjective and does not guarantee a fair PConv baseline either, since it is not stated whether PConv was retrained under the same protocol. The paper does provide useful evidence: detailed ablations isolating forward vs reverse attention, mask-updating visualizations, and a user study, which support the internal consistency of the method. But these do not compensate for the missing same-protocol PConv comparison. A verification rerun of PConv under the paper's exact protocol would settle the central claim. Until then, CONDITIONAL remains the appropriate verdict.","tokens_in":16336,"tokens_out":17517,"duration_ms":171609,"concrete_test":"Retrain the PConv model of [17] from scratch on the same 10-category Places subset used here, with the same 256x256 random crops, same mask generation (including the 12,000 masks from [17]), same loss function (Eq. 18) and hyperparameters as LBAM, and evaluate on the same test images and mask-ratio bins as Table 1. Report PSNR, SSIM, and mean l1 per bin, ideally over at least three seeds to obtain error bars. If PConv under this protocol reaches or exceeds LBAM's numbers, the 0.2-0.4 dB advantage vanishes and the state-of-the-art claim is unsupported; if LBAM retains the margin, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim that LBAM outperforms partial convolution rests on Table 1's PConv* row, whose caption states the numbers are taken from [17]. Those numbers were produced by the original PConv model trained on the full Places2 dataset, with its own loss weights (e.g., L_hole + 6*L_valid + 0.05*L_perceptual + 120*L_style + 0.1*L_TV), its own test split, and its own mask sampling. The present paper trains LBAM on only 10 selected Places365 categories, evaluates on a different test set (500 images per category), and uses a different objective (Eq. 18: L1 over the entire image, WGAN-GP with lambda=10, perceptual, and style loss with weight 120). It also mixes its own 18,000 random masks with 12,000 masks from [17] for training and testing. Because PSNR/SSIM differences are only 0.2-0.4 dB (PSNR) and 0.002-0.019 (SSIM), and mean l1 is actually worse for LBAM in the two lowest mask-ratio bins, these protocol differences are large enough to flip the ranking. The paper's own ablations on Paris StreetView do not include PConv as a same-protocol control, so the contribution of the bidirectional attention maps cannot be disentangled from the change in training data and loss. Without rerunning PConv under identical conditions, the claimed superiority over partial convolution is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces learnable bidirectional attention maps (LBAM) for image inpainting with irregular holes. The core idea is to generalize partial convolution (PConv) by replacing handcrafted mask renormalization and hard 0-1 mask updating with learned attention maps g_A and mask-updating functions g_M, and by introducing reverse attention maps on the decoder so that the decoder focuses on filling holes rather than reconstructing known regions. The authors show that, when bias is removed, the mask-convolution step of PConv can be replaced by standard convolution because the features at unfilled mask positions are zero by construction (Eqs. 6-7). They adopt a U-Net with forward attention on the encoder and reverse attention on the decoder, trained with L1, WGAN-GP, perceptual, and style losses. Experiments on Paris StreetView and a 10-category subset of Places365 compare against PatchMatch, Global&Local, Context Attention, and PConv, including a user study in which LBAM is preferred 63.2% of the time. Ablations show the effects of the learned attention maps, reverse attention, activation functions, and the adversarial loss. The authors acknowledge failure cases in the supplementary material and promise to release code and models.","tokens_in":16675,"tokens_out":8927,"duration_ms":85205,"significance":"If the quantitative results are reproducible, LBAM is a useful incremental improvement over partial convolution for irregular-hole inpainting. The paper's analytical observation that PConv without bias reduces to standard convolution plus learned renormalization is elegant and likely to be reused. The systematic ablation, including the comparison of activation functions and the reverse attention maps, provides a good picture of what the components contribute. The user study and the explicit failure cases add credibility to the perceptual claims. The main weakness is that the central quantitative comparison with PConv relies on numbers copied from the original paper under a different protocol, so the claimed margins (0.2-0.4 dB PSNR) may not be significant or even real.","major_comments":[{"comment":"The PConv* comparison is not protocol-matched. The caption states that PConv* results are taken from [17], but the present paper trains on 50,000 images from 10 Places categories and tests on a different 500-image-per-category split, generates 18,000 random masks mixed with 12,000 masks from [17], and optimizes the objective in Eq. (18) with WGAN-GP and loss weights λ1=1, λ2=0.1, λ3=0.05, λ4=120. The original PConv was trained on the full Places2 dataset with different loss weights (e.g., L_hole + 6*L_valid + 0.05*L_perceptual + 120*L_style + 0.1*L_TV) and its own mask sampling. Since the PSNR margins are only 0.19-0.42 dB and the mean l1 is worse for LBAM at the two lowest mask-ratio bins (1.12 vs 1.09 and 1.93 vs 1.88), these protocol differences are large enough to reverse the ranking. The authors should rerun PConv under their exact training and evaluation protocol, or provide evidence that PConv's numbers are insensitive to the dataset subset, mask distribution, and loss changes.","section":"Table 1"},{"comment":"The ablation studies do not isolate the contribution of the bidirectional attention maps against a same-protocol PConv baseline. The variants Ours(unlearned) and Ours(3×3) use fixed 1/16 or 3×3 mask filters with hard activation functions, which approximate PConv-like behavior, but they share the paper's U-Net backbone (14 layers, 4×4 stride-2 kernels, no bias, batch normalization) and training loss. It is therefore unclear how much of the gain over PConv in Table 1 is due to the attention modules rather than to backbone, data, or loss changes. Please add a direct PConv control trained and evaluated under the same protocol, or argue explicitly why the current variants control for the relevant variables.","section":"Section 4.2, Table 2"},{"comment":"The quantitative claims lack error bars and statistical testing. The reported PSNR/SSIM values are single estimates on one test set, with margins as small as 0.002 SSIM or 0.19 dB PSNR. Without standard deviations across multiple runs or bootstrap confidence intervals, it is not possible to determine whether the observed differences are meaningful. This is particularly important after a protocol-matched PConv rerun is obtained, since the central claim of superiority over PConv rests on these small margins.","section":"Section 4.1, Tables 1 and 2"}],"minor_comments":[{"comment":"The sentence 'we generate 18,000 masks with random shape, and 12,000 masks from [17] for training and testing' is ambiguous about whether the same mask set is used for both training and testing and how the masks are partitioned; please clarify.","section":"Section 4.1"},{"comment":"The caption for Figure 11 lists '(e) 3×3 filter' and then also '(e) our full LBAM model'; the second entry should be labeled '(d)' for the 3×3 variant and '(e)' for the full model.","section":"Section 4.2, Figure 11 caption"},{"comment":"The paper reports λ4=120 for the style loss, which is an order of magnitude larger than typical style-loss weights in inpainting papers; a brief sentence on how these weights were selected or their sensitivity would be useful.","section":"Section 3.5"},{"comment":"The text acknowledges that Ours(w/o Ladv) outperforms Ours(full) on PSNR/SSIM and justifies the adversarial loss only by qualitative inspection of the supplementary figures; given that the user study supports the qualitative claim, this trade-off between distortion metrics and perceptual quality should be stated more explicitly in the main text.","section":"Section 4.2"},{"comment":"The paper cites gated convolution [35] in the related work but does not include it as an experimental baseline; since that method also targets irregular holes and was available at the time of submission, a sentence explaining its omission or adding the comparison would strengthen the state-of-the-art claim.","section":"Related Work and Experiments"}],"recommendation":"major_revision","confidential_remarks":"The paper is technically sound in its internal derivations, and the main issue is experimental comparability rather than methodological circularity. I would advise the editor to require a same-protocol rerun of PConv before acceptance, as the current practice of copying numbers from [17] into Table 1 is the weak link in the quantitative claim. The self-citations are benign, and the explicit failure cases in the supplementary material are a positive sign."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the mechanism is genuinely new and the ablations suggest it works, but the headline claim that LBAM beats partial convolution is not actually demonstrated. Table 1's PConv* numbers are copied from [17], and PConv was trained on full Places2 while LBAM was trained on 10 selected categories, with different loss weights, test split, and mask sampling. With PSNR gains of only 0.2–0.4 dB, protocol differences can easily flip the ranking. That's the main thing to know.\n\nWhat's good: Section 3.1's reinterpretation of PConv as standard convolution plus attention is clean and, as far as I can see, correct. The learnable gA and gM generalize the fixed 1/9 filter and hard thresholding, and the reverse attention path on the decoder is not in the prior work they cite. The ablation study is reasonably thorough—five variants on Paris, plus visualizations of the learned masks that show the intended behavior. The user study is small but at least it's a direct comparison of outputs.\n\nSoft spots, in order of severity. First, the Table 1 comparison is load-bearing and not same-protocol. The caption admits PConv* is taken from [17]. They mix 18k random masks with 12k PConv masks; the PConv row presumably used the original paper's mask protocol. The l1 numbers are actually worse for LBAM in the two lowest mask-ratio bins, which doesn't help their case. They never run PConv under their own protocol; the Paris ablations in Table 2 have no PConv baseline at all. Second, no error bars or significance tests. Third, the adversarial loss doesn't help on PSNR/SSIM—Ours(w/o Ladv) is better on both—and the justification that it improves visual quality rests only on a small user study. Fourth, no released code despite the abstract promise.\n\nWho gets value from this: anyone working on mask-aware convolutions for inpainting or free-form editing. The reverse attention idea is reusable. It's not foundational, but it's a reasonable engineering contribution.\n\nRecommendation: this deserves peer review. The mechanism is coherent, the experiments are broadly reproducible in principle, and the main flaw is fixable: rerun PConv under the same protocol and add error bars. I'd send it out with a note to the referee to focus on whether the same-protocol comparison holds up. In my own work I'd cite it for the reverse attention idea, but I'd treat the SOTA claim with caution.","headline":"Reverse attention is a real idea, but the paper's main claim against partial convolution isn't established because the baseline numbers come from a different training protocol.","tokens_in":17183,"tokens_out":2232,"would_cite":true,"duration_ms":23681,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that replacing partial convolution's hand-crafted mask renormalization and mask updating with learnable bidirectional attention maps yields sharper, more coherent, and more visually plausible image inpainting for…","keywords":["image inpainting","learnable attention maps","bidirectional attention","partial convolution","irregular holes","U-Net","feature renormalization","adversarial training"],"falsifier":"Rerun partial convolution under this paper's exact training protocol — the same Places subset, mask generation, loss weights, and epoch count — and compare PSNR and SSIM on the same test masks. If the gap disappears or reverses on masks with ratio above 0.3, the claimed advantage over partial convolution is not established.","tokens_in":16184,"feed_emoji":"🖼️","tokens_out":5763,"duration_ms":52217,"temperature":0.7,"pith_summary":"The paper proposes learnable bidirectional attention maps (LBAM) for image inpainting with irregular holes. It argues that replacing partial convolution's fixed mask renormalization and mask-updating rules with learned attention maps, and adding reverse attention maps that focus the decoder on the missing region, produces sharper and more coherent results. On Paris StreetView and Places, the authors report higher PSNR and SSIM than existing methods at high mask ratios, along with a 63.2% user preference. If correct, this makes learned attention a practical alternative to the hand-crafted rules in partial convolution networks.","feed_headline":"Learnable attention maps beat partial convolution on irregular holes","feed_subtitle":"A U-Net that learns forward and reverse mask attention reports sharper fills on large irregular holes.","key_machinery":"The central object is the learnable attention map, a module that renormalizes convolution features by multiplying them with an attention map produced by an asymmetric Gaussian-shaped activation, and updates the mask with a ReLU-based function where the exponent is a hyperparameter. The attention map's parameters are layer-specific and learned end-to-end, so the network decides how much to trust intermediate filled regions rather than trusting all of them equally. The reverse attention map applies the same mechanism to the decoder using the complementary mask, restricting the decoder's task to the hole. These maps are inserted into every U-Net layer — forward on the encoder, reverse on the decoder — and this bidirectional learned renormalization is what carries the claimed improvement.","core_discovery":"The paper claims that the three hand-crafted components of partial convolution — the fixed mask-convolution filter, the hard 0-1 mask updating, and the all-one decoder mask — can be replaced by learned counterparts. Its forward attention map learns feature renormalization and mask propagation through an asymmetric Gaussian activation function and a ReLU-based mask update, while a reverse attention map on the decoder side, driven by the complementary mask, lets the decoder concentrate on filling the hole. The result is a U-Net whose encoder and decoder features are both renormalized by learned masks, trained end-to-end with pixel reconstruction, perceptual, style, and adversarial losses. The paper argues that this yields visually sharper and more coherent inpaintings, particularly for masks covering more than 30% of the image.","pith_inferences":["Editorial inference: the same learnable-attention recipe could be applied to other masked-input tasks such as image extrapolation, deblurring with structured masks, or video inpainting, where mask confidence needs to propagate across time.","Editorial inference: because the attention maps are layer-specific and trained end-to-end, the architecture might transfer to free-form editing with user-provided masks without retraining the mask-update rules, though the paper does not test this.","Editorial inference: the asymmetric Gaussian activation suggests the network learns a smooth confidence profile around hole boundaries; this could be probed by freezing the learned parameters and visualizing the effective receptive field of the mask branch."],"forward_implications":["On irregular masks covering 30–50% of an image, LBAM reports higher PSNR and SSIM than partial convolution, contextual attention, global-and-local, and PatchMatch on the Places benchmark.","Because the decoder is focused on the hole via reverse attention, the model avoids wasting capacity reconstructing known regions, which the paper shows in feature visualizations and in higher-quality results on real-world object removal.","The learned mask update makes it feasible to train with an adversarial loss, which the paper reports is difficult with partial convolution alone; the adversarial loss trades a small drop in PSNR and SSIM for better visual quality.","The model runs in about 70 ms per 256×256 image, roughly three times faster than the global-and-local method and five times faster than contextual attention."],"supporting_citations":[{"why":"Provides the partial-convolution baseline that LBAM extends and the hand-crafted mask-updating rule it replaces.","marker":"[17]"},{"why":"Contextual attention baseline for irregular-hole inpainting, used in qualitative and quantitative comparisons.","marker":"[36]"},{"why":"Global-and-local baseline for image completion, used in comparisons and user study.","marker":"[10]"},{"why":"PatchMatch exemplar-based baseline, used in comparisons and user study.","marker":"[2]"},{"why":"Places dataset supplies the training and testing images for the main quantitative comparison.","marker":"[40]"},{"why":"Paris StreetView dataset supplies the second benchmark and the ablation study.","marker":"[6]"},{"why":"Wasserstein GAN with gradient penalty is used for the adversarial loss that improves visual quality.","marker":"[9]"},{"why":"Perceptual loss on VGG-16 features is used in the training objective.","marker":"[12]"},{"why":"Style loss via Gram matrices is used to recover detailed textures.","marker":"[7]"}],"fun_headline_variants":["Learnable bidirectional attention maps improve irregular hole filling","U-Net learns mask renormalization for sharper inpainting","Bidirectional attention maps beat partial convolution on holes","End-to-end learnable attention maps for hole filling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The quantitative advantage over partial convolution rests on Table 1's partial-convolution numbers being copied from the original partial-convolution paper instead of rerun under this paper's training data and mask protocol; if the training setups differ, the 0.2 to 0.4 dB PSNR differences may not reflect a real advantage.","fun_headline_variants_meta":{"raw":{"variants":["Learnable bidirectional attention maps improve irregular hole filling","U-Net learns mask renormalization for sharper inpainting","Bidirectional attention maps beat partial convolution on holes","End-to-end learnable attention maps for hole filling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000757,"raw_usage":{"total_tokens":3325,"prompt_tokens":868,"completion_tokens":2457,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":484,"completion_tokens_details":{"reasoning_tokens":2402}},"tokens_in":484,"tokens_out":2457,"duration_ms":18472,"temperature":1.0,"reasoning_tokens":2402,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:30:17.531446+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun partial convolution under this paper's exact training protocol — the same Places subset, mask generation, loss weights, and epoch count — and compare PSNR and SSIM on the same test masks. If the gap disappears or reverses on masks with ratio above 0.3, the claimed advantage over partial convolution is not established.","supporting_citations":[{"cited_title":"Percep- tual losses for real-time style transfer and super-resolution","cited_arxiv_id":null,"evidence_quote":"Perceptual loss on VGG-16 features is used in the training objective."},{"cited_title":"Gatys, Alexander S","cited_arxiv_id":null,"evidence_quote":"Style loss via Gram matrices is used to recover detailed textures."},{"cited_title":"Reda, Kevin Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro","cited_arxiv_id":null,"evidence_quote":"Provides the partial-convolution baseline that LBAM extends and the hand-crafted mask-updating rule it replaces."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contextual attention baseline for irregular-hole inpainting, used in qualitative and quantitative comparisons."},{"cited_title":"Globally and locally consistent image completion","cited_arxiv_id":null,"evidence_quote":"Global-and-local baseline for image completion, used in comparisons and user study."},{"cited_title":"PatchMatch: A randomized correspon- dence algorithm for structural image editing","cited_arxiv_id":null,"evidence_quote":"PatchMatch exemplar-based baseline, used in comparisons and user study."},{"cited_title":"Places: A 10 million image database for scene recognition","cited_arxiv_id":null,"evidence_quote":"Places dataset supplies the training and testing images for the main quantitative comparison."},{"cited_title":"What makes paris look like paris?Com- munications of the ACM, pages 103–110, 2015","cited_arxiv_id":null,"evidence_quote":"Paris StreetView dataset supplies the second benchmark and the ablation study."},{"cited_title":"Improved training of wasserstein gans","cited_arxiv_id":null,"evidence_quote":"Wasserstein GAN with gradient penalty is used for the adversarial loss that improves visual quality."}],"review_version":1}