Semantic filtering through deep source separation on microscopy images

Avelino Javer; Jens Rittscher

REVIEW 4 major objections 6 minor 11 references

Semantic filtering through deep source separation on microscopy images

T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Training a U-Net on synthetic image pairs where only the target object layer is shared teaches it to output exactly that layer in real microscopy images, matching fully supervised detectors without dense annotations.

desk verdict A useful Noise2Noise extension for microscopy layer separation, with a real evaluation gap: the paper never directly tests whether the trained filter actually outputs the target layer. read the letter →

arxiv 1909.00691 v1 pith:T3P6C24I submitted 2019-09-02 cs.CV

Avelino Javer , Jens Rittscher This is my paper · ORCID

classification cs.CV

keywords semanticfilteringdeepsourceseparationmicroscopyimageanalysissynthetictrainingdataNoise2NoiseU-Netcellsegmentationhistology

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that semantic segmentation of microscopy images can be achieved without densely labelled training data. The idea extends Noise2Noise: train a network on pairs of images that share only the target layer, so that everything else varies, and the network is forced to output just that layer. Synthetic pairs are built by adding crops of the target class to varying background and non-target objects, and the authors demonstrate transfer to fluorescence co-cultures, histology images, and C. elegans videos. If correct, this makes high-content imaging analysis far cheaper and preserves texture and morphology in the output.

What carries the argument

The load-bearing identity is the additive image model $x = \ell_t + \ell_u + b$, where $\ell_t$ is the target-object layer, $\ell_u$ the non-target object layer, and $b$ the background. Synthetic training pairs are generated by randomly placing augmented crops of each class into background patches, with the target layer identical across the pair and all other layers changed. The U-Net, trained with L1 loss, is forced to predict the only invariant component, effectively computing a median over the changing layers; this is the direct analogue of Noise2Noise's noise cancellation, applied to semantic content instead of noise.

What would settle it

A direct check: take a real image set with dense pixel-level ground truth for the target class, run the synthetic-trained filter, and compare its output mask against that ground truth with IoU; if the IoU is substantially below a supervised U-Net trained on the same data, the transfer claim is falsified. A cheaper signal is to evaluate the filter on images from a different microscope or staining protocol than the crops were taken from; large performance drops would indicate the model memorized the synthetic crop distribution rather than the semantic layer.

Watch

Extended reading notes

Core claim

The central claim is that a U-Net trained with an L1 loss on pairs of synthetic images generated from the additive model $x = \ell_t + \ell_u + b$, where $\ell_t$ is held fixed while $\ell_u$ and $b$ vary, converges to a semantic filter that returns $\ell_t$ alone. Because the network must predict a plausible output from two inputs whose only common content is the target objects, the loss-minimizing prediction averages out the changing non-target layers, leaving the invariant target layer. The paper demonstrates this in four settings: moving C. elegans worms removed from static-camera videos with zero manual labels, out-of-focus microglia removed while in-focus morphology is preserved, overlapping hepatocyte/fibroblast co-cultures resolved with F1 scores at or above the original pixel-based classifier, and astrocyte detection on histology where the filter reaches F1 0.72 against a fully supervised Faster R-CNN baseline at 0.77 while requiring far fewer training annotations.

Load-bearing premise

The load-bearing premise is that real microscopy images are well approximated by additively combining a few hundred representative crops of the target class, non-target objects, and background, so that a network trained on those synthetic pairs behaves the same on genuine images.

Editorial extensions

If this is right

Time-lapse microscopy of moving specimens needs no manual labels: pairs of frames at a fixed time lag train the filter to remove moving objects, and subtracting its output recovers them, even when specimen motility is impaired.
For a new assay, gathering a few hundred isolated-object crops via thresholding or simple segmentation is enough to train a class-specific separator; no dense pixel masks are required.
Because the output is a soft image of the target layer rather than a mask of detected instances, it retains texture and can resolve overlapping objects from different classes that confound single-pixel annotations.
On the histology benchmark, the synthetic-trained filter matches the best fully supervised detector when only 100 training images are available, indicating that annotation savings grow in low-data regimes.
The filter output can be used directly for phenotype comparisons: on the co-culture data it separates hepatocyte concentration conditions with a smaller p-value and larger Z-factor than the original pixel-based classification.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the transfer is as robust as the paper suggests, crop-exemplar collection could replace dense annotation across many microscopy assays, but the paper does not demonstrate cross-dataset transfer such as training on one stain and applying to another; testing that would be a natural next step.
The additive formation model implies the filter should degrade gracefully as overlap fraction grows; a controlled stress test varying overlap from zero to near-complete would map the failure boundary and may motivate a non-additive extension.
The output layer is a soft intensity map, so it could be plugged directly into instance-segmentation or tracking pipelines, potentially improving counting in crowded cultures without changing the training scheme.
Because the same architecture is reused everywhere, the contribution likely lies in the data-generation recipe rather than the network design; applying the recipe to 3D volumes or moving-camera video are untested extensions implied by the method.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

What you should know: this paper applies Noise2Noise-style pair training to semantic filtering in microscopy. The training pairs are synthetic, with the target object layer held constant and the background/other objects varied. That is a genuine new application, and it gives a practical route to segmentation-like outputs without dense labels. But the central mechanism is never directly verified: there is no pixel-level evaluation on synthetic test images where the target layer is known. All quantitative scores come from thresholding the network output and matching detections to real-image annotations that are themselves noisy.

What is actually good: the synthetic image generation model is simple and clearly specified. The C. elegans video experiment is a nice natural demonstration of the training scheme, and the authors are honest that a median projection would work for healthy worms. The BBBC042 results show graceful degradation as training data shrink, which matters for the claimed high-content screening use case. The outputs in the figures do look like plausible cell layer extractions, and the idea of retaining texture while resolving overlaps is sensible.

Where the soft spots are: the abstract overclaims "state-of-the-art segmentation" when the headline BBBC042 F1 is 0.72 versus 0.77 for Faster-RCNN with augmentations. BBBC026 comparisons use numbers reported in the source paper rather than re-run baselines, the microglia section has no quantitative evaluation, and there are no error bars anywhere. The bigger issue is the missing synthetic-domain check. A detection proxy can pass even if the output is not a clean target layer; a constant offset or residual background could still yield acceptable F1, especially when the annotations contain false negatives, which the authors themselves invoke in Section 3.4. So the evidence does not clearly distinguish source separation from a foreground-enhancing transform. This is fixable: evaluate on held-out synthetic images against the known target layer.

Who this is for: anyone working on label-efficient microscopy segmentation or high-content screening. The paper deserves a serious referee. The core idea is credible and the recipe is clear, but the evaluation needs additional baselines, error bars, and a direct synthetic-domain experiment before the strong claims are warranted.

Referee Report

4 major / 6 minor

Summary. The paper proposes 'semantic filtering' as an extension of Noise2Noise. A U-Net is trained on pairs of images in which the target object layer l_t is fixed while the remaining object layer l_u and background b vary, so that the network learns to output only l_t. For most experiments the pairs are synthesized under an assumed additive model x = l_t + l_u + b, using crops extracted by classical segmentation. The method is demonstrated on C. elegans videos (where frame pairs separated by a time lag provide the training signal), on microglia co-cultures (qualitatively), and on two public datasets: BBBC026 hepatocyte/fibroblast co-cultures and BBBC042 astrocyte histology, with localization F1 as the main quantitative metric. The paper claims state-of-the-art segmentation results with little or no manual annotation.

Significance. If the semantic-filtering mechanism were verified, this would be a useful contribution: it replaces dense pixel labels with a handful of representative crops, uses a simple additive synthesis model, and the C. elegans experiment demonstrates a genuinely annotation-free training signal. The core idea of training on pairs with an invariant target layer is elegant, and the central loss is parameter-free. However, the evidence is not yet sufficient for the claims. The synthetic-pipeline output is never compared with a known target layer, the BBBC042 F1 (0.72) is below the re-run FasterRCNN baseline (0.77), and several comparisons rely on reported numbers without error bars. The potential is real, but the evaluation needs strengthening before acceptance.

major comments (4)

[Section 2.2 / Section 3 (all synthetic-pipeline experiments)] The central mechanism claim is not directly tested. For the synthetic-pipeline experiments (microglia, BBBC026, BBBC042), the network output is never compared pixelwise to a known target layer l_t on held-out synthetic images. All quantitative evaluation goes through detection proxies: Otsu thresholding, connected components, and bounding-box matching against annotations that are themselves noisy (BBBC042 annotator agreement F1 0.77-0.82; BBBC026 single-pixel labels). A foreground-enhancing transform that does not actually separate layers can still produce acceptable F1, especially because the authors invoke annotation false negatives to explain extra detections (Fig. 5). The C. elegans experiment (Section 3.1) does compare with ground-truth segmentations, but for the synthetic pipeline, which is the paper's main methodological contribution, no such comparison is reported. Please add a controlled synthetic test set with known l_t and l_u, and report pixel-level separation metrics (e.g., IoU or MSE between output and l_t, and suppression of l_u). This is the load-bearing experiment for the paper's title and main claim.
[Abstract / Section 3.4] The abstract claims 'state-of-the-art segmentation results', but the reported BBBC042 detection F1 is 0.72, below the re-run FasterRCNN with augmentations (0.77), and the evaluation is localization, not segmentation. No error bars, confidence intervals, or statistical tests are supplied for any experiment, so the statement in Section 3.4 that performance with 100 or fewer training images is 'the same as the top model' cannot be assessed. Please correct the abstract and either add uncertainties and significance tests or explicitly frame the claim as 'comparable to fully supervised detectors' rather than 'state-of-the-art'.
[Section 3.3] The BBBC026 comparison is not apples-to-apples. The hepatocyte and fibroblast F1 scores are compared with numbers 'reported in the dataset source paper [6]' rather than re-running the baseline under the same evaluation protocol. The assignment of detected connected components to the dataset's single-pixel labels is not described, and no per-fold variability is reported even though a 5-fold cross-validation was performed. Because the claimed F1 gains are modest (0.86 vs 0.80; 0.95 vs 0.92), the comparison needs the same matching code and error bars, or it should be presented as an approximate comparison.
[Section 3.2] The microglia experiment is presented as evidence that the method removes out-of-focus cells while preserving morphology, but it contains no quantitative evaluation or ground truth. This section cannot support the paper's claim that the method 'generalizes to a broad range of experimental settings'. Please add a quantitative measure (e.g., separation accuracy on cells labeled in/out of focus, or a morphology preservation score) or explicitly mark the experiment as a qualitative demonstration.

minor comments (6)

[Section 3.1] 'We shown an example of this in Fig. 3.1' appears to reference a nonexistent figure; the intended figure number should be corrected.
[References] The text attributes FasterRCNN to reference [1], but [1] is Girshick's Fast R-CNN; the Faster R-CNN paper of Ren et al. should be cited.
[Throughout] The manuscript contains several typos: 'demostrate' (Section 1), 'outputing' (Fig. 1 caption), 'hepathocytes' (Supplementary S1), and 'less that 300 pixels' (Section 3.4).
[Section 2.2] The claim that minimizing L1 loss on noisy pairs yields the conditional median is stated without derivation or citation; a pointer to Noise2Noise or a one-line explanation would help.
[Section 3.4] The sentence 'The results are shown in Fig. 5 Right' appears inconsistent with the figure caption, where the learning curve is described on the left; please check the directional references.
[Reproducibility] No code release or data availability is mentioned for the in-house microglia and C. elegans datasets, which limits replication of the non-public experiments; please add a reproducibility statement.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the network is trained on synthetic images with known target layers and evaluated on held-out real images.

full rationale

The paper's pipeline is a supervised image-to-image training procedure on synthetic pairs x = l_t + l_u + b, where the supervision signal is the known target layer l_t constructed by placing extracted crops. The network learns to map the synthetic input to that known target layer; the target is not derived from the network's output, and no test-set labels are used during training. The extension of Noise2Noise is attributed to Lehtinen et al., an external prior work, and the only self-citation (Javer et al. 2018) is used to supply the C. elegans dataset and its published ground-truth segmentation, which is independent evidence rather than a load-bearing self-citation. The absence of a direct pixel-level evaluation on held-out synthetic images is a validation gap concerning whether the trained filter truly performs source separation on real images, but it does not make the derivation equivalent to its inputs by construction. The reported F1 scores on BBBC026 and BBBC042 compare against held-out real-image annotations, so the central transfer claim is empirically testable and not forced by the training definition. Therefore no significant circularity is present.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central method is an empirical training procedure that relies on the additivity of image layers, the Noise2Noise loss-minimization property, and synthetic-to-real transfer. No new physical entities are introduced. The numerical free parameters are hand-set thresholds and augmentations, with several thresholds left unspecified.

free parameters (5)

microglia foreground area threshold = 10^4 pixels
Connected components above 10^4 pixels are treated as foreground when extracting microglia crops (Supplementary S1).
Laplacian-of-Gaussian crop score thresholds = unspecified lower and upper bounds
Crops are bucketed into in-focus, out-of-focus, or discarded based on mean absolute LoG response (S1). The thresholds are not reported, hurting reproducibility.
synthetic object overlap fraction = 50-90%
Partial overlaps between placed objects are allowed up to 50-90% of the object, with complete overlaps avoided to prevent artifacts (Section 2.2).
minimum connected component area for BBBC042 detection = 300 pixels
Blobs below 300 pixels are discarded when converting the semantic filter output to bounding boxes (Section 3.4).
training patch size = 256x256 pixels
Patches of 256x256 pixels are used for training (Section 3.1).

assumptions (4)

domain assumption Image formation is additive: x = 𝓁t + 𝓁u + b (Section 2.2).
The synthetic training data is generated under this model. It holds well for fluorescence microscopy but only approximately for histology and bright-field, where the authors invert and complement images to approximate it.
standard math L1 loss on pairs with independently varying non-target layers makes the network converge to the conditional median of the target layer (Section 2.1, citing Lehtinen et al.).
This is the Noise2Noise optimality property that the method builds on; the paper does not re-derive it.
domain assumption Synthetic pairs transfer to real images despite being 'far from realistic' (Section 1).
The whole evaluation depends on this empirical transfer assumption; it is tested on four datasets but not guaranteed for other modalities.
domain assumption For C. elegans training, the median projection over a fixed time lag is a clean background (Section 3.1).
The authors themselves note this fails for impaired motility (unc-51), so the training set must consist of videos where worms move enough.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Semantic filtering through deep source separation on microscopy images." pith.science (2026). https://pith.science/paper/T3P6C24I

@misc{pith2026190900691,
  author       = {Pith},
  title        = {Pith review of: Semantic filtering through deep source separation on microscopy images},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T3P6C24I}},
  note         = {Machine review of arXiv:1909.00691}
}

read the original abstract

By their very nature microscopy images of cells and tissues consist of a limited number of object types or components. In contrast to most natural scenes, the composition is known a priori. Decomposing biological images into semantically meaningful objects and layers is the aim of this paper. Building on recent approaches to image de-noising we present a framework that achieves state-of-the-art segmentation results requiring little or no manual annotations. Here, synthetic images generated by adding cell crops are sufficient to train the model. Extensive experiments on cellular images, a histology data set, and small animal videos demonstrate that our approach generalizes to a broad range of experimental settings. As the proposed methodology does not require densely labelled training images and is capable of resolving the partially overlapping objects it holds the promise of being of use in a number of different applications.

Figures

Figures reproduced from arXiv: 1909.00691 by the authors.

**Figure 1.** Illustration of the training procedure. Left, the method consists in try to make a network learn the mapping between pairs of images where the only thing in common is the target objects. Once trained, the network is then capable of outputing only the signal from the objects of interest and smoothing out the rest. Right, examples of synthetic training pairs for the different datasets. For illustration purposes the fi… view at source ↗

**Figure 2.** Using consecutive frames of movies of moving objects makes possible [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** The signal from the out of focus cells is removed while preserving [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: The model is capable of resolve overlapping cells on co-cultures of [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: Performance on histological images of astrocytes cells (BBBC042). [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

11 extracted references · 8 canonical work pages

[6]

Methods 96, 6--11 (2016)

Logan, D.J., Shan, J., Bhatia, S.N., Carpenter, A.E.: Quantifying co-cultured cell phenotypes in high-throughput using pixel-based classification. Methods 96, 6--11 (2016)

work page 2016
[1]

In: Proceedings of the IEEE international conference on computer vision

Girshick, R.: Fast r-cnn. In: Proceedings of the IEEE international conference on computer vision. pp. 1440--1448 (2015)

work page 2015
[2]

Nature methods 15(9), 645 (2018)

Javer, A., Currie, M., Lee, C.W., Hokanson, J., Li, K., Martineau, C.N., Yemini, E., Grundy, L.J., Li, C., Ch’ng, Q., et al.: An open-source platform for analyzing and sharing worm-behavior data. Nature methods 15(9), 645 (2018)

work page 2018
[3]

In: PMLR

Lehtinen, J., Munkberg, J., Hasselgren, J., Laine, S., Karras, T., Aittala, M., Aila, T.: N oise2 N oise: Learning image restoration without clean data. In: PMLR. pp. 2965--2974 (2018)

work page 2018
[4]

In: Proceedings of the IEEE international conference on computer vision

Lin, T.Y., Goyal, P., Girshick, R., He, K., Doll \'a r, P.: Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision. pp. 2980--2988 (2017)

2017
[5]

Nature methods 9(7), 637--637 (2012)

Ljosa, V., Sokolnicki, K.L., Carpenter, A.E.: Annotated high-throughput microscopy image sets for validation. Nature methods 9(7), 637--637 (2012)

work page 2012
[7]

In: MICCAI

Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI. pp. 234--241. Springer (2015)

work page 2015
[8]

Scientific reports 8 (2018)

Suleymanova, I., Balassa, T., Tripathi, S., Molnar, C., Saarma, M., Sidorova, Y., Horvath, P.: A deep convolutional neural network approach for astrocyte detection. Scientific reports 8 (2018)

work page 2018

Show all 11 references

[9]

Nature methods 15(12), 1090 (2018)

Weigert, M., Schmidt, U., Boothe, T., M \"u ller, A., Dibrov, A., Jain, A., Wilhelm, B., Schmidt, D., Broaddus, C., Culley, S., et al.: Content-aware image restoration: pushing the limits of fluorescence microscopy. Nature methods 15(12), 1090 (2018)

2018
[10]

, " * write output.state after.block = add.period write

ENTRY address author booktitle chapter doi edition editor eid howpublished institution journal key month note number organization pages publisher school series title type url volume year label INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION in...
[11]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [6]

Methods 96, 6--11 (2016)

Logan, D.J., Shan, J., Bhatia, S.N., Carpenter, A.E.: Quantifying co-cultured cell phenotypes in high-throughput using pixel-based classification. Methods 96, 6--11 (2016)

work page 2016

[2] [1]

In: Proceedings of the IEEE international conference on computer vision

Girshick, R.: Fast r-cnn. In: Proceedings of the IEEE international conference on computer vision. pp. 1440--1448 (2015)

work page 2015

[3] [2]

Nature methods 15(9), 645 (2018)

Javer, A., Currie, M., Lee, C.W., Hokanson, J., Li, K., Martineau, C.N., Yemini, E., Grundy, L.J., Li, C., Ch’ng, Q., et al.: An open-source platform for analyzing and sharing worm-behavior data. Nature methods 15(9), 645 (2018)

work page 2018

[4] [3]

In: PMLR

Lehtinen, J., Munkberg, J., Hasselgren, J., Laine, S., Karras, T., Aittala, M., Aila, T.: N oise2 N oise: Learning image restoration without clean data. In: PMLR. pp. 2965--2974 (2018)

work page 2018

[5] [4]

In: Proceedings of the IEEE international conference on computer vision

Lin, T.Y., Goyal, P., Girshick, R., He, K., Doll \'a r, P.: Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision. pp. 2980--2988 (2017)

2017

[6] [5]

Nature methods 9(7), 637--637 (2012)

Ljosa, V., Sokolnicki, K.L., Carpenter, A.E.: Annotated high-throughput microscopy image sets for validation. Nature methods 9(7), 637--637 (2012)

work page 2012

[7] [7]

In: MICCAI

Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI. pp. 234--241. Springer (2015)

work page 2015

[8] [8]

Scientific reports 8 (2018)

Suleymanova, I., Balassa, T., Tripathi, S., Molnar, C., Saarma, M., Sidorova, Y., Horvath, P.: A deep convolutional neural network approach for astrocyte detection. Scientific reports 8 (2018)

work page 2018