Pith. sign in

REVIEW 3 major objections 5 minor 20 references

Fine-tuning BiRefNet on 1,228 curated anime images yields ToonOut, which matches or slightly beats the best closed-source background-removal service on anime imagery.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 22:58 UTC pith:MOIV6KGA

load-bearing objection Useful released artifacts (dataset, weights, code) and an honest write-up, but the headline performance claim rests on a small, self-curated test set that doesn't establish parity with closed-source models. the 3 major comments →

arxiv 2509.06839 v1 pith:MOIV6KGA submitted 2025-09-08 cs.CV cs.LG

ToonOut: Fine-tuned Background-Removal for Anime Characters

Matteo Muratori , Jo\"el Seytre This is my paper
classification cs.CV cs.LG
keywords background removalanimedichotomous image segmentationBiRefNetfine-tuningPixel Accuracydataset release
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to show that a general-purpose background-removal network, which is strong on photos but weak on anime, can be made competitive in the anime domain through a small amount of domain-specific fine-tuning. The authors collect and annotate 1,228 generated anime images covering faces, emotions, poses, actions, and standalone items, then fine-tune the open BiRefNet model on them. On their test set, the resulting model ToonOut raises Pixel Accuracy from 95.3% to 99.5%, cutting the error rate from 4.7% to 0.5% and slightly edging out the strongest closed-source service tested. They also introduce a Pixel Accuracy metric that counts a prediction 'correct' when its alpha value differs from ground truth by at most 10 after one erosion, and they release code, model weights, and the dataset.

Core claim

On the paper's own terms, the central claim is that fine-tuning an open segmentation model on a deliberately curated anime dataset closes the gap to--and slightly surpasses--closed-source background-removal services for anime-style images. The evidence is the test-set comparison: ToonOut scores 99.5% Pixel Accuracy, 95.6% Boundary IoU, and 99.4% Weighted F-measure, against 99.2%/95.2%/99.3% for the strongest closed-source baseline, 97.8%/92.4%/98.8% for the other open baseline, and 95.3%/88.5%/97.8% for the un-fine-tuned BiRefNet. The authors attribute the gain to the dataset's coverage of hard cases--complex hair, transparency, poses with objects--and to resuming from a BiRefNet checkpoint

What carries the argument

The load-bearing object is BiRefNet, a Bilateral Reference Network for high-resolution dichotomous image segmentation: a Localization Module finds the salient object using global context, and a Reconstruction Module produces the alpha mask using hierarchical image patches as source reference and gradient maps as target reference, with dual supervision from auxiliary gradients and ground truth. The fine-tuning recipe carries the result: resume a checkpoint saved after 244 epochs, train 46 epochs at batch size 2 with learning rate 1e-5 (halved at epochs 20 and 40), gradient clipping at 100, and loss L = 10*L_SSIM + 90*L_MAE + 0.25*L_IoU plus a binary-cross-entropy gradient loss. The third piec

Load-bearing premise

The test set is representative of the intended domain: the 126 test images come from the same generated anime distribution as the training images and were deliberately selected to include cases where the baseline model fails; if real-world anime images differ enough from that distribution, the reported margins may not persist.

What would settle it

Compare ToonOut against the closed-source baseline on a held-out set of anime images created outside the dataset-generation pipeline--for example, frames from existing anime productions or hand-drawn illustrations--and compute Pixel Accuracy and Boundary IoU. If ToonOut's Pixel Accuracy no longer reaches or exceeds the closed-source model's on that set, the paper's claim that it matches the best closed-source models on anime background removal would fail outside its own distribution.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • On the paper's test set, ToonOut reduces Pixel Accuracy error from 4.7% to 0.5% and Boundary-IoU error from 11.5% to 4.0% relative to the original BiRefNet.
  • The fine-tuned model edges out the strongest closed-source service tested on anime images (99.5% vs 99.2% Pixel Accuracy) while keeping code, weights, and data open.
  • The largest relative gain is on 'action' images (characters interacting with objects), where baseline BiRefNet scores 76.8% Pixel Accuracy and ToonOut reaches 99.0%; the smallest gain is on standalone 'items'.
  • The public release makes it possible for others to reproduce the result and to fine-tune further on their own anime distributions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the test set is drawn from the same synthetic distribution as training and was filtered to include hard cases for the baseline, so the headline margin over closed-source models is likely to shrink on real-world anime screenshots or hand-drawn art; an out-of-distribution benchmark would test this.
  • Editorial extension: the Pixel Accuracy tolerance (alpha difference <=10 after one erosion) may not fully penalize soft transparency gradients around hair, so a human-ranking study or perceptual metric would complement the numbers.
  • Editorial extension: the recipe is light enough (1,228 images, 46 epochs, two GPUs) that the same fine-tune-and-release pattern looks transferable to other niche visual styles such as comics, pixel art, or watercolor, provided enough diverse examples are collected.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces ToonOut, a fine-tuned version of BiRefNet for background removal in anime-style images. The authors collect a custom dataset of 1,228 synthetic anime images generated with SDXL Yamer's Anime, curate it partly by prioritizing images where BiRefNet performs poorly, and fine-tune BiRefNet on this dataset. They also propose a new Pixel Accuracy metric. On a 126-image test split of the same curated pool, ToonOut reports PA 99.5% vs. Photoroom 99.2%, BIoU 95.6% vs. 95.2%, and claims to 'match the best closed-source models.' The code, weights, and dataset are released.

Significance. If the performance claim held on representative anime content, this would be a useful demonstration that domain-specific fine-tuning of an open DIS model can compete with closed-source APIs, and the released dataset/weights would be a practical resource. However, the current evaluation does not support the central claim because the test set is drawn from the same curated synthetic distribution as the training data and is deliberately enriched for baseline failures. The small sample size and absence of uncertainty intervals make the reported margins over Photoroom non-compelling. The introduced Pixel Accuracy metric also has a definitional flaw. The open-sourcing of code, weights, and data is a strength worth acknowledging, but the evidence for the headline claim is not yet convincing.

major comments (3)
  1. [Sec. 2.1, Table 3] The test set is not representative of the target domain. The 126 test images are a 10% split of the same pool that was generated with SDXL Yamer's Anime and explicitly filtered to prioritize cases where the BiRefNet baseline performs poorly (with only up to 20% easy cases). Therefore the test set is enriched for examples on which the base model is weak, and it comes from exactly the same synthetic distribution used for training. This selection and distribution shift inflates the apparent improvement of a BiRefNet fine-tune. To support the claim of matching closed-source models on anime background removal, an evaluation on an external, independently collected set of anime images (e.g., real illustrations, screenshots, user-uploaded content) is necessary.
  2. [Table 1, Table 2] The quantitative evidence for 'matching the best closed-source models' is weak. The overall test set consists of only 126 images, and the margins over Photoroom are tiny: PA 99.5% vs. 99.2% (0.3 pp) and BIoU 95.6% vs. 95.2% (0.4 pp). No confidence intervals or significance tests are reported. Moreover, Table 2 shows that ToonOut is actually worse than Photoroom on Pose (BIoU 96.4 vs. 96.7), Factory (BIoU 96.0 vs. 96.8), and Items (PA 96.6 vs. 98.3); the aggregate advantage is driven almost entirely by the 15-image Action subset. With per-subset sizes as low as 8-25, these numbers are within plausible sampling error. The claim should be tempered or supported by per-image statistical analysis.
  3. [Sec. 4.2, Eq. (2)] The newly introduced Pixel Accuracy metric is not well-posed as written. Eq. (2) computes the error rate relative to 'Total Number of Foreground Pixels,' not the total number of pixels. False positives on background pixels are therefore never penalized: a model that predicts an all-foreground mask would achieve 100% PA on any image. The definition must either use all pixels in the denominator or justify why background misclassifications are irrelevant. In addition, the displayed formula appears to square the complement (an exponent 2 appears after the parenthesis), which is inconsistent with the reported PA values: a 4.7% error rate would yield 95.3% PA if not squared but approximately 90.8% if squared as printed. Table 1 reports 95.3%, so the formula or the table needs correction. Since PA is the headline metric, this is a load-bearing issue.
minor comments (5)
  1. [Sec. 4.2] The description of the erosion step is underspecified. Please state the kernel size and shape (e.g., 3x3 square) used for the 'erode the error mask once' operation, as this directly affects PA scores.
  2. [Sec. 4.2] The phrase 'foreground pixels denote the pixels with α > 128' should clarify whether α refers to ground-truth alpha or predicted alpha. The metric definition and reproducibility depend on this.
  3. [References] Reference [10] (Margolin et al., 'How to Evaluate Foreground Maps?') lists the DOI/URL as arxiv.org/pdf/1708.00786, which is the same URL as reference [6] (Structure-measure). Please correct the reference for Margolin et al.
  4. [Sec. 4.1] The selection of PA as the primary metric is based on a human-judged ranking on the validation set from the same distribution. This is not inherently wrong, but it should be disclosed prominently that the metric was chosen post hoc, and all candidate metrics should be reported to avoid cherry-picking concerns.
  5. [Table 1] The column 'F-measure' (standard F-measure) is somewhat redundant with 'Weighted F-measure.' Please clarify the threshold or averaging method used for the F-measure column.

Circularity Check

0 steps flagged

No circular derivation: the fine-tuning result is an empirical, held-out evaluation; the main risks are test-set representativeness and metric selection, neither of which equates the output to the input.

full rationale

The paper does not claim a first-principles derivation; it reports an empirical fine-tuning result. The central comparison (BiRefNet 95.3% -> ToonOut 99.5% Pixel Accuracy, Table 1) is measured on a held-out 10% split of the authors' own dataset, not on the training portion. Eq. (1) is a loss function and Eq. (2) is an evaluation metric; neither is defined in terms of the model's outputs or of the reported improvements. The dataset is admittedly generated synthetically and filtered to prioritize images where the BiRefNet baseline performs poorly (Section 2.1), which creates a real external-validity concern: the test set is not a random sample of anime images and may inflate the apparent improvement. However, this is a sampling/representativeness limitation, not a circularity: the baseline and ToonOut scores are empirical measurements on those images, and ToonOut's predictions are not read off from the test labels or from the curation criterion. The paper itself flags a related limitation for the Items category in Table 2, but that is a data-quantity issue, not a circular step. The only self-referential item is the mention of toongether [8] as an application, which is not load-bearing for any claim. No fitted parameter is renamed as a prediction, no uniqueness theorem from the authors is invoked, and no ansatz is smuggled in via self-citation. The paper is open-sourcing model, code, and data, so the empirical claims are externally checkable; circularity score is therefore 0.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The central performance claim rests on three pillars the authors provide: a synthetic dataset that is assumed to represent anime content, manual annotations assumed to be correct, and a self-designed metric assumed to track human perception. The model weights and loss configuration come from tuned hyperparameters, not from a derivation.

free parameters (5)
  • Loss weights lambda_1, lambda_2, lambda_3 = 10, 90, 0.25
    Chosen by 'extensive empirical experimentation' (Section 3.2); affect training and final performance.
  • Learning rate schedule = 1e-5, halved after 20 and 40 epochs
    Standard hyperparameter selection described in Section 3.2.
  • Pixel Accuracy tolerance delta = 10
    Chosen as 'visually indistinguishable' without quantitative evidence (Section 4.2).
  • Erosion parameters for Pixel Accuracy = unspecified
    OpenCV erode is referenced but kernel size and iterations are not given, making the metric non-reproducible.
  • Dataset curation threshold = up to 20% 'easy' images
    Section 2.1: inclusion was biased toward hard cases for the baseline, affecting the test distribution.
axioms (4)
  • domain assumption The 1,228 SDXL-generated anime images are representative of the target anime domain.
    Section 2.1: images are generated with Yamer's Anime SDXL and filtered, but no external validation of representativeness.
  • domain assumption The manually annotated alpha masks are accurate ground truth, including transparent edges.
    Section 2.1 and acknowledgements: annotations are manual, but no protocol or inter-annotator agreement is reported.
  • domain assumption The newly introduced Pixel Accuracy metric aligns with human visual quality.
    Section 4.1: only a small validation exercise on the authors' own preferences supports this.
  • domain assumption The pretrained BiRefNet checkpoint is a valid starting point for fine-tuning.
    Section 3.2: training resumes from a checkpoint after 244 epochs of the original model; no ablation of starting point.

pith-pipeline@v1.3.0-alltime-deepseek · 4947 in / 9565 out tokens · 93557 ms · 2026-08-04T22:58:00.902070+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of ToonOut: Fine-tuned Background-Removal for Anime Characters." pith.science (2026). https://pith.science/paper/MOIV6KGA

@misc{pith2026250906839,
  author       = {Pith},
  title        = {Pith review of: ToonOut: Fine-tuned Background-Removal for Anime Characters},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MOIV6KGA}},
  note         = {Machine review of arXiv:2509.06839}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

While state-of-the-art background removal models excel at realistic imagery, they frequently underperform in specialized domains such as anime-style content, where complex features like hair and transparency present unique challenges. To address this limitation, we collected and annotated a custom dataset of 1,228 high-quality anime images of characters and objects, and fine-tuned the open-sourced BiRefNet model on this dataset. This resulted in marked improvements in background removal accuracy for anime-style images, increasing from 95.3% to 99.5% for our newly introduced Pixel Accuracy metric. We are open-sourcing the code, the fine-tuned model weights, as well as the dataset at: https://github.com/MatteoKartoon/BiRefNet.

Figures

Figures reproduced from arXiv: 2509.06839 by Jo\"el Seytre, Matteo Muratori.

Figure 1
Figure 1. Figure 1: Closed source [7] and open source models ([16], [2], [20]) are inadequate on our anime characters test set. We fine-tune BiRefNet [20] and the resulting model ToonOut yields better outputs for our use-case. Abstract While state-of-the-art background removal models excel at realistic imagery, they frequently underper￾form in specialized domains—such as anime-style con￾tent, where complex features like hair … view at source ↗
Figure 2
Figure 2. Figure 2: Examples of predictions made by ToonOut on images contained in our test datasets. Our datasets cover a variety of characters in challenging poses, their interactions with items, and standalone objects. results. This challenge motivated our work to improve back￾ground removal for images of anime characters and items. We aimed to select a popular DIS model and enhance its capabilities on anime content. Our p… view at source ↗
Figure 3
Figure 3. Figure 3: Example of images contained in the dataset, covering different characters, poses and items. We generated images using the anime-specialized checkpoint Yamer’s Anime [17] of Stable Diffusion XL [12, 3]. Generated images were rigorously fil￾tered to ensure high quality: we discarded images with anatomical inconsistencies, unclear foreground￾background boundaries, and artifacts that would re￾sult in visually … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

20 extracted references · 16 canonical work pages · 5 internal anchors

  1. [1]

    CascadePSP: Toward Class-Agnostic and Very High-Resolution Segmentation via Global and Local Refinement

    Ho Kei Cheng (HKUST) et al. “CascadePSP: Toward Class-Agnostic and Very High- Resolution Segmentation via Global and Local Refinement”. In: (2020).doi:https : //arxiv.org/pdf/2005.02551

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  2. [2]

    Bria AI.BRIA Background Removal v2.0.url: https://huggingface.co/briaai/RMBG-2.0

    work page

  3. [3]

    Springer, 2024

    Chris Bishop.Deep Learning. Springer, 2024. isbn: 978-3-031-45467-7

    work page 2024

  4. [4]

    Boundary IoU: Improving Object-Centric Image Segmentation Evaluation

    Bowen Cheng et al. “Boundary IoU: Improv- ing Object-Centric Image Segmentation Evalu- ation”. In: (2021).doi:https://arxiv.org/ pdf/2103.16562

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  5. [5]

    Enhanced-alignment Measure for Binary Foreground Map Evalua- tion

    Deng-Ping Fan et al. “Enhanced-alignment Measure for Binary Foreground Map Evalua- tion”. In: (2018).doi:https : / / arxiv . org / pdf/1805.10421

    Pith/arXiv arXiv 2018

  6. [6]

    Structure-measure: A New Way to Evaluate Foreground Maps

    Deng-Ping Fan et al. “Structure-measure: A New Way to Evaluate Foreground Maps”. In: (2017).doi:https://arxiv.org/pdf/1708. 00786

    work page 2017

  7. [7]

    photoroom.com/it

    Photoroom Inc.Photoroom.url:https://www. photoroom.com/it

    work page

  8. [8]

    kartoon.ai.Toongether: Comic Creation, by Kartoon AI.url:https://toongether.ai

    work page

  9. [9]

    Komiko.AI Background Removal.url:https: //komiko.app/background-removal

    work page

  10. [10]

    How to Evaluate Foreground Maps?

    Ran Margolin, Lihi Zelnik-Manor, and Ayellet Tal. “How to Evaluate Foreground Maps?” In: (2014).doi:https://arxiv.org/pdf/1708. 00786

    work page 2014

  11. [11]

    OpenCV.erode.url:https://docs.opencv. org / 3 . 4 / db / df6 / tutorial _ erosion _ dilatation.html

    work page

  12. [12]

    SDXL: Improving La- tent Diffusion Models for High-Resolution Image Synthesis

    Dustin Podell et al. “SDXL: Improving La- tent Diffusion Models for High-Resolution Image Synthesis”. In: (2023).doi:https : / / arxiv . org/pdf/2307.01952

    Pith/arXiv arXiv 2023

  13. [13]

    On Mean Absolute Error for Deep Neural Network Based Vector-to-Vector Regression

    Jun Qi et al. “On Mean Absolute Error for Deep Neural Network Based Vector-to-Vector Regres- sion”. In: (2020).doi:https : / / arxiv . org / pdf/2008.07281

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  14. [14]

    Boundary-Aware Segmenta- tion Network for Mobile and Web Applications

    Xuebin Qin et al. “Boundary-Aware Segmenta- tion Network for Mobile and Web Applications”. In: (2021), pp. 6–7.doi:https://arxiv.org/ pdf/2101.04704

    Pith/arXiv arXiv 2021

  15. [15]

    Highly Accurate Dichotomous Image Segmentation

    Xuebin Qin et al. “Highly Accurate Dichoto- mous Image Segmentation”. In: (2022).doi: https://arxiv.org/pdf/2203.03041

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  16. [16]

    co / spaces / skytnt / anime-remove-background

    skytnt.Anime Remove Background.url: https : / / huggingface . co / spaces / skytnt / anime-remove-background

    work page

  17. [17]

    Yamer.SDXL Yamer’s Anime.url:https:// civitai.com/models/76489

    work page

  18. [18]

    Multi-view Aggregation Net- work for Dichotomous Image Segmentation

    Qian Yu et al. “Multi-view Aggregation Net- work for Dichotomous Image Segmentation”. In: (2024).doi:https://arxiv.org/pdf/2404. 07445

    work page 2024

  19. [19]

    Gradient-Induced Co-Saliency Detection

    Zhao Zhang et al. “Gradient-Induced Co- Saliency Detection”. In: (2020).doi:https:// arxiv.org/pdf/2004.13364

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  20. [20]

    Bilateral Reference for High- Resolution Dichotomous Image Segmentation

    Peng Zheng et al. “Bilateral Reference for High- Resolution Dichotomous Image Segmentation”. In: (2024).doi:https : / / arxiv . org / pdf / 2401.03407. 6

    Pith/arXiv arXiv 2024