Relation U-Net

P. Ellen Grant; Rina Bao; Sheng He; Yangming Ou

REVIEW 3 major objections 5 minor 17 references

Relation U-Net

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Relation U-Net turns the gap between union and intersection predictions into a per-image confidence score that tracks segmentation accuracy without ground truth.

desk verdict A useful per-image confidence signal for segmentation, but the evidence needs statistical rigor and a direct test of the untrained reflexivity assumption. read the letter →

arxiv 2501.09101 v1 pith:I42K5HYQ submitted 2025-01-15 eess.IV cs.CV

Sheng He , Rina Bao , P. Ellen Grant , Yangming Ou This is my paper · ORCID

classification eess.IVcs.CV

keywords RelationU-Netmedicalimagesegmentationconfidencescoreestimationdeeplearninguncertaintyquantificationwithoutgroundtruth

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Relation U-Net generalizes U-Net to take two input images and output two segmentation maps plus the 'Possible' (union) and 'Consensus' (intersection) relation maps between them. The paper's central claim is that when the same test image is fed twice, the disagreement between the estimated union and intersection maps is a per-image confidence score that ranks segmentation difficulty without any ground truth. The authors report that this score is linearly correlated with true Dice accuracy on four public medical datasets (LiTS, Hippocampus, BraTS, ISIC) and that the relation outputs match or beat vanilla U-Net in Dice. The practical payoff would be a way for clinicians to know which automated segmentations need manual review.

What carries the argument

The central mechanism is the Relation U-Net, built from a vanilla U-Net by concatenating the two input images at the input layer and adding four independent classifiers at the output layer: the segmentation heads $\hat{s}_1, \hat{s}_2$ and the relation heads $\hat{r}_p$ (union/OR) and $\hat{r}_c$ (intersection/AND). The confidence score is defined as the Dice coefficient between $\hat{r}_p$ and $\hat{r}_c$, written $C = 2|\hat{r}_p \cap \hat{r}_c|/(|\hat{r}_p| + |\hat{r}_c|)$. The argument depends on reflexivity: with $x_1 = x_2$ the ground-truth relations coincide with the segmentation itself, so the residual discrepancy of the independently predicted relation maps is interpreted as a difficulty signal. Training samples pairs of distinct images with supervision on union and intersection of their ground truths; testing feeds one image twice and reads off $C$ as an unsupervised confidence estimate.

What would settle it

Train Relation U-Net twice with different random seeds. If the confidence scores $C$ for the same test images have low rank correlation between the two training runs while the segmentation Dice stays similar, then $C$ is dominated by subnetwork variability rather than image difficulty; alternatively, on a test set of uniformly easy, high-quality images, a large spread in $C$ would show the same thing.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the consistency between two relation segmentation maps—the Possible relation $\hat{r}_p = \hat{s}_1 \vee \hat{s}_2$ and the Consensus relation $\hat{r}_c = \hat{s}_1 \wedge \hat{s}_2$—acts as a proxy for the accuracy of the segmentation of a single test image. Because the relations are reflexive, when $x_1 = x_2$ the two relation maps should both equal the segmentation; the trained network only approximates this, and the resulting discrepancy, measured as a Dice consistency score $C$, is claimed to be linearly correlated with the true Dice on the test image. The paper shows this correlation is stronger than that obtained from Monte Carlo Dropout uncertainty, and that thresholding test images by $C$ selects subsets with higher average Dice. The authors' strongest claim is that 'the discrepancy of the estimated relation segmentation maps is correlated to the segmentation accuracy of each test images.'

Load-bearing premise

The method assumes that when the same image is fed twice, the difference between the predicted union and intersection maps reflects how hard the image is to segment, rather than random variation among the independently trained prediction heads.

Editorial extensions

If this is right

On the four evaluated datasets, Relation U-Net's outputs (particularly the Possible relation $\hat{r}_p$ or the average of the two segmentation heads) reach Dice scores at least as high as vanilla U-Net, so adding relation heads does not sacrifice accuracy.
Without ground truth, the confidence score $C$ lets a user rank test images by difficulty; keeping only the images with the highest $C$ yields a subset with substantially higher average Dice.
The Pearson correlation between $C$ and true Dice is higher for Relation U-Net than for the MC Dropout baseline on all four datasets.
Using a second input image as an anchor (sampled from the training or test set) gives a small additional accuracy gain over feeding the same image twice, suggesting the relation heads can also act as an ensemble.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the same discrepancy trick should transfer to any multi-output segmentation architecture whose outputs stand in a set-theoretic relation, not just U-Net, because it only requires two independently predicted maps with known union/intersection semantics.
Editorial inference: the paper's correlation results come from in-distribution five-fold splits; whether the confidence signal survives out-of-distribution shifts (different scanner, contrast, or population) is an open question a deployment study would need to test.
Editorial inference: a direct way to strengthen the reflexivity premise would be to add identical-pair training examples with the constraint $\hat{r}_p = \hat{r}_c = \hat{s}$; this could reduce subnetwork noise and sharpen the confidence score.
Editorial inference: for clinical triage, what matters is not just linear correlation but calibration of $C$ against a clinical threshold such as Dice > 0.8; converting $C$ into a probability of acceptable segmentation would make the score directly actionable.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short take: the confidence-score idea is real and useful, but the evidence as presented is not yet tight enough to fully back it.

What's new: using the discrepancy between predicted union and intersection maps from a multi-output U-Net as a per-image confidence score, when the same image is fed twice. The relation maps themselves come from the pairwise learning literature (Wang et al.), and the MIMO structure is borrowed, but the test-time reflexivity trick is new, as far as I know. The paper runs four public datasets with five-fold CV and compares against an MC-dropout baseline. That's a reasonable first pass.

What it does well: the method is simple and cheap—one forward pass at test time gives both a segmentation and a ranking signal. Figure 5 shows the ranking visually tracks perceived difficulty. If the correlation holds, that's practically valuable for triage.

Where it's soft: the reported accuracy gains are small and inconsistent (ISIC actually drops on rp). The Pearson correlations in Fig 4(a) are only shown as bars; no numbers, no confidence intervals, no significance tests. The central theoretical justification—reflexivity of the learned relations at x1=x2—is asserted but never trained or enforced. The two relation heads share an encoder but have independent classifiers, so their discrepancy on identical input could reflect systematic head bias or random seed variation rather than segmentation difficulty. The paper doesn't check whether the confidence score is just tracking object size or boundary length. These are not fatal objections, but they are exactly the kind of thing a reviewer would want ruled out before trusting the ranking.

The stress-test note is fair. It puts the finger on the load-bearing assumption. Still, the empirical correlation across four datasets is suggestive, and the intended use case—flagging hard cases for human review—doesn't require the mechanism to be perfectly clean; the ranking just needs to be useful. So I'd send this to review, but with major-revision expectations: report numeric correlations with CIs, add a significance test, test reflexivity directly by training on some identical pairs (or enforcing it), and check confounders like object size. Code would help.

Bottom line: an incremental but practically motivated idea, with enough empirical signal to deserve referee time, and clear gaps to fix.

Referee Report

3 major / 5 minor

Summary. The paper proposes 'Relation U-Net', a U-Net variant with two input images and four output heads that predict the two input segmentation maps plus their Possible (union) and Consensus (intersection) relation maps. At test time, the same image is fed twice, and the Dice coefficient between the predicted union and intersection maps is used as a per-image confidence score. The method is evaluated with five-fold cross-validation on LiTS, Hippocampus, BraTS, and ISIC. The paper claims that Relation U-Net improves Dice over vanilla U-Net and that the estimated confidence score is linearly correlated with segmentation accuracy on test images. The central claim is that the discrepancy between the two relation maps ranks test images by difficulty without ground truth.

Significance. If established, the per-image confidence score would be practically useful for selective human review in clinical segmentation workflows. The method is simple, uses four public datasets with a standard five-fold protocol, and includes an MC Dropout baseline, which are clear strengths. The qualitative examples in Fig. 5 also support the intuitive plausibility of the approach. However, the load-bearing link between the learned relation heads and the claimed confidence score is not adequately verified, and the reported correlations are only graphical. The accuracy gains over the vanilla U-Net are also small and inconsistent. With additional experiments and proper statistical reporting, the contribution would be a useful lightweight extension; in its current form, the central claim is not fully supported.

major comments (3)

[Section 2.3 and Section 3.2] The confidence score rests on an unverified reflexivity assumption. The ground-truth relations r_p = s1 v s2 and r_c = s1 ^ s2 are reflexive by definition, but the trained heads are learned only from pairs of independently sampled distinct training images, and no loss term or training pair enforces that r_p(x,x) = r_c(x,x) = s(x). Feeding the same image twice is therefore an off-manifold input. C = Dice(r_p, r_c) can be dominated by systematic bias between the union-trained and intersection-trained heads, by random initialization, or by object-size and boundary-length effects, rather than by segmentation difficulty. To support the central claim, the authors should (i) report the distribution of C and of its correlation with Dice across multiple training seeds, (ii) compare the relation heads on identical inputs against a fixed 'head-bias' baseline, for example by adding a reflexivity-consistency loss or including identical pairs in training, and (iii) report partial correlations or stratified analyses controlling for object size and contrast. Without this, the confidence-score equation in Section 2.3 is not established.
[Section 3.4 and Fig. 4(a)] The Pearson correlation coefficients are presented only graphically, without numerical values, confidence intervals, or p-values, and no rank correlation is reported. Since the intended use is to rank test images for review, the appropriate evidence includes Spearman or Kendall coefficients, per-fold and per-seed intervals, and the distribution of differences against the MC Dropout baseline. Fig. 4(b) would also benefit from error bars or shaded confidence bands over the five folds. The current reporting makes it impossible to assess whether the claimed linear correlation is statistically reliable.
[Tables 1 and 2] The claimed accuracy advantage over vanilla U-Net is not consistent or statistically supported. In Table 1, the Possible relation r_p is worse than U-Net on ISIC (86.08 vs 87.89) and only marginally better on LiTS and Hippocampus. In Table 2, the x1 = x2 condition is worse than U-Net on Hippocampus (82.59 vs 82.76) and BraTS (82.27 vs 82.63). All reported differences are within one standard deviation of the five-fold means. Formal paired tests across folds, or a larger number of training runs, are needed before claiming 'better accuracy than vanilla U-Net' in the abstract.

minor comments (5)

[Section 2.3] The formula C = 2|r_p ^ r_c|/(r_p + r_c) is informal for binary maps; please define whether |.| denotes the number of positive pixels and clarify the denominator notation.
[Section 2.3 and Table 1] Notation alternates among r_c, sc, and r_p; please use one consistent symbol for the Consensus relation map in text, equations, and tables.
[References] Reference [6] lists the dataset as 'LIST' and the first author as 'Patrick ilic'; the correct names are 'LiTS' and 'Patrick Bilic'.
[Section 3.3.2 and Table 2] The choice of 200 anchor images is not justified; please report sensitivity to n or explain why 200 forward passes are sufficient for a stable estimate.
[Fig. 4(a)] The caption does not clearly identify which curves correspond to which methods, and the grayscale rendering makes the comparison difficult to read in print.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the confidence score is an internal-consistency measure evaluated on held-out data, not fitted to the claimed accuracy.

full rationale

The paper's central claim is that the discrepancy between the Possible and Consensus relation maps, C = Dice(rp, rc), correlates with segmentation accuracy on test images without ground truth. This is an empirical, internal-consistency heuristic, not a prediction derived from fitted parameters. The network outputs rp and rc independently for the same input image, and C is computed directly from those outputs; no constant in the confidence formula is fit to test-set accuracies. The correlation with accuracy is measured on held-out folds, not enforced by construction. The relation definitions (union and intersection) are taken from external prior work [4], and the self-citations [5,13] are used only as architectural or contextual references, not as load-bearing evidence for the main claim. The paper does assert reflexivity for the ground-truth logical relations, but it explicitly acknowledges that the learned subnetworks do not produce exactly identical rp and rc for the same input; that residual discrepancy is precisely what C measures. Whether this discrepancy tracks segmentation difficulty is an empirical assumption about the learned subnetworks, not a circular reduction: the network is never trained to make C correlate with Dice, and the correlation is an observed outcome rather than an input. The concern that head bias, seed variance, or object-size confounding could drive the correlation is a validity and robustness issue, not a circularity issue. Overall, the derivation is self-contained against external benchmarks and does not reduce to its own inputs.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the reflexivity of logical relations, the assumption that independently trained relation heads produce meaningful disagreement on identical inputs, and the Dice-based discrepancy being a valid difficulty proxy. No new physical entities are introduced; the confidence score is a derived metric, not a postulated entity. Test-time ensembling uses 200 anchors, a hand-chosen hyperparameter with no sensitivity analysis.

free parameters (2)

Number of anchor images per test image at inference (n=200) = 200
Chosen without sensitivity analysis; used for averaging in Table 2 when the second input is sampled from train or test. Higher n likely stabilizes estimates but costs more forward passes.
MC Dropout forward passes for baseline (n=200) = 200
Baseline comparison uses 200 stochastic forward passes; matched to Relation U-Net's 200 anchors but would need sensitivity analysis to ensure fair comparison.

assumptions (3)

standard math Possible (union) and Consensus (intersection) relations between binary segmentation maps are reflexive, so for x1=x2 the ground-truth relations equal the segmentation s1=s2.
Logical set operations satisfy this; the paper cites prior work and invokes it in Section 2.3 as the basis for using the relation outputs at test time.
domain assumption The two relation prediction heads are trained independently enough that their disagreement on an identical input reflects segmentation difficulty rather than random initialization or training noise.
Relation U-Net is trained only on pairs of distinct images, yet the confidence score depends on the discrepancy of the union and intersection maps for the same image; no direct evidence is given that this discrepancy is difficulty-driven.
ad hoc to paper Dice similarity between the predicted union and intersection maps is a suitable proxy for per-image segmentation accuracy.
The paper defines the confidence score as this Dice value and asserts its correlation with accuracy; this is the empirical claim under test, not an independently established proxy.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Relation U-Net." pith.science (2026). https://pith.science/paper/I42K5HYQ

@misc{pith2026250109101,
  author       = {Pith},
  title        = {Pith review of: Relation U-Net},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I42K5HYQ}},
  note         = {Machine review of arXiv:2501.09101}
}

read the original abstract

Towards clinical interpretations, this paper presents a new ''output-with-confidence'' segmentation neural network with multiple input images and multiple output segmentation maps and their pairwise relations. A confidence score of the test image without ground-truth can be estimated from the difference among the estimated relation maps. We evaluate the method based on the widely used vanilla U-Net for segmentation and our new model is named Relation U-Net which can output segmentation maps of the input images as well as an estimated confidence score of the test image without ground-truth. Experimental results on four public datasets show that Relation U-Net can not only provide better accuracy than vanilla U-Net but also estimate a confidence score which is linearly correlated to the segmentation accuracy on test images.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 15 canonical work pages

[1]

output-only

INTRODUCTION Many deep learning-based methods for medical image seg- mentation [1] have been proposed to improve the accuracy. Most deep learning models are “output-only” models, which only output a single segmentation map given the input image. In general, the average accuracy on the evaluation dataset is reported. However, in practice the average accura...

work page
[2]

Relation U-Net

METHOD 2.1. Relation U-Net Fig. 1 shows the framework of the Relation U-Net. Given two different input images xi, i = 1, 2 their corresponding bina- rized segmentation maps si in the training set, we use logical operations to define the relations based on segmentation maps arXiv:2501.09101v1 [eess.IV] 15 Jan 2025 Input Single image Segmentation Model Sing...

work page Pith review arXiv 2025
[3]

Datasets We use four public datasets to evaluate the proposed method, including LiTS [6], Hippocampus [7], BraTS [8] and ISIC [9]

EXPERIMENTAL RESULTS 3.1. Datasets We use four public datasets to evaluate the proposed method, including LiTS [6], Hippocampus [7], BraTS [8] and ISIC [9]. The LiTS dataset contains contrastenhanced CT images. The Hippocampus dataset contains MRI images with the ante- rior and posterior of the hippocampus. For BraTS, we use the brain MRIs on 2018 Brain T...

work page 2018
[4]

CONCLUSION This paper proposed a simple generalization of the segmenta- tion neural network which can not only provide the segmen- tation result, but also an estimated confidence score to eval- uate segmentation accuracies for each test image when the ground-truth is not available. The segmentation network re- ceives multiple two input images and outputs ...

work page
[5]

U-Net: Convolutional networks for biomedical image segmentation,

Olaf Ronneberger, Philipp Fischer, and Thomas Brox, “U-Net: Convolutional networks for biomedical image segmentation,” in MICCAI 2015, 2015, pp. 234–241

work page 2015
[6]

Confi- dence calibration and predictive uncertainty estimation for deep medical image segmentation,

Alireza Mehrtash, William M Wells, Clare M Tem- pany, Purang Abolmaesumi, and Tina Kapur, “Confi- dence calibration and predictive uncertainty estimation for deep medical image segmentation,” IEEE TMI, vol. 39, no. 12, 2020

work page 2020
[7]

Es- timating example difficulty using variance of gradients,

Chirag Agarwal, Daniel D’souza, and Sara Hooker, “Es- timating example difficulty using variance of gradients,” in CVPR, 2022, pp. 10368–10378

work page 2022
[8]

Pairwise learning for medical image seg- mentation,

Renzhen Wang, Shilei Cao, Kai Ma, Yefeng Zheng, and Deyu Meng, “Pairwise learning for medical image seg- mentation,” MIA, vol. 67, pp. 101876, 2021

work page 2021

Show all 17 references

[9]

Deep relation learning for regression and its appli- cation to brain age estimation,

Sheng He, Yanfang Feng, P Ellen Grant, and Yangming Ou, “Deep relation learning for regression and its appli- cation to brain age estimation,” IEEE TMI, vol. 41, no. 9, pp. 2304–2317, 2022

2022
[10]

The liver tumor segmentation benchmark (LIST),

Patrick ilic, Patrick Christ, Hongwei Bran Li, Eugene V orontsov, Avi Ben-Cohen, Georgios Kaissis, Adi Sze- skin, Colin Jacobs, Gabriel Efrain Humpire Mamani, Gabriel Chartrand, et al., “The liver tumor segmentation benchmark (LIST),” MIA, vol. 84, pp. 102680, 2023

2023
[11]

The medical seg- mentation decathlon,

Michela Antonelli, Annika Reinke, Spyridon Bakas, Keyvan Farahani, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, Olaf Ron- neberger, Ronald M Summers, et al., “The medical seg- mentation decathlon,” Nature communications, vol. 13, no. 1, pp. 4128, 2022

2022
[12]

The multimodal brain tumor image seg- mentation benchmark (brats),

Bjoern H Menze, Andras Jakab, Stefan Bauer, Jayashree Kalpathy-Cramer, Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, et al., “The multimodal brain tumor image seg- mentation benchmark (brats),” IEEE TMI, vol. 34, no. 10, pp. 1993–2...

1993
[13]

Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (ISIC),

Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al., “Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imagin...

2018 arXiv
[14]

Exploring uncertainty measures in deep net- works for multiple sclerosis lesion detection and seg- mentation,

Tanya Nair, Doina Precup, Douglas L Arnold, and Tal Arbel, “Exploring uncertainty measures in deep net- works for multiple sclerosis lesion detection and seg- mentation,” MIA, vol. 59, pp. 101557, 2020

2020
[15]

Automatic segmentation variability estimation with segmentation priors,

Leo Joskowicz, D Cohen, N Caplan, and Jacob Sosna, “Automatic segmentation variability estimation with segmentation priors,” MIA, vol. 50, pp. 54–64, 2018

2018
[16]

Quantifying and leveraging predictive uncertainty for medical image assessment,

Florin C Ghesu, Bogdan Georgescu, Awais Mansoor, Youngjin Yoo, Eli Gibson, RS Vishwanath, Abishek Balachandran, James M Balter, Yue Cao, Ramandeep Singh, et al., “Quantifying and leveraging predictive uncertainty for medical image assessment,” MIA, vol. 68, pp. 101855, 2021

2021
[17]

Segmentation ability map: Interpret deep features for medical image segmentation,

Sheng He, Yanfang Feng, P Ellen Grant, and Yangming Ou, “Segmentation ability map: Interpret deep features for medical image segmentation,” MIA, vol. 84, pp. 102726, 2023

2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

output-only

INTRODUCTION Many deep learning-based methods for medical image seg- mentation [1] have been proposed to improve the accuracy. Most deep learning models are “output-only” models, which only output a single segmentation map given the input image. In general, the average accuracy on the evaluation dataset is reported. However, in practice the average accura...

work page

[2] [2]

Relation U-Net

METHOD 2.1. Relation U-Net Fig. 1 shows the framework of the Relation U-Net. Given two different input images xi, i = 1, 2 their corresponding bina- rized segmentation maps si in the training set, we use logical operations to define the relations based on segmentation maps arXiv:2501.09101v1 [eess.IV] 15 Jan 2025 Input Single image Segmentation Model Sing...

work page Pith review arXiv 2025

[3] [3]

Datasets We use four public datasets to evaluate the proposed method, including LiTS [6], Hippocampus [7], BraTS [8] and ISIC [9]

EXPERIMENTAL RESULTS 3.1. Datasets We use four public datasets to evaluate the proposed method, including LiTS [6], Hippocampus [7], BraTS [8] and ISIC [9]. The LiTS dataset contains contrastenhanced CT images. The Hippocampus dataset contains MRI images with the ante- rior and posterior of the hippocampus. For BraTS, we use the brain MRIs on 2018 Brain T...

work page 2018

[4] [4]

CONCLUSION This paper proposed a simple generalization of the segmenta- tion neural network which can not only provide the segmen- tation result, but also an estimated confidence score to eval- uate segmentation accuracies for each test image when the ground-truth is not available. The segmentation network re- ceives multiple two input images and outputs ...

work page

[5] [5]

U-Net: Convolutional networks for biomedical image segmentation,

Olaf Ronneberger, Philipp Fischer, and Thomas Brox, “U-Net: Convolutional networks for biomedical image segmentation,” in MICCAI 2015, 2015, pp. 234–241

work page 2015

[6] [6]

Confi- dence calibration and predictive uncertainty estimation for deep medical image segmentation,

Alireza Mehrtash, William M Wells, Clare M Tem- pany, Purang Abolmaesumi, and Tina Kapur, “Confi- dence calibration and predictive uncertainty estimation for deep medical image segmentation,” IEEE TMI, vol. 39, no. 12, 2020

work page 2020

[7] [7]

Es- timating example difficulty using variance of gradients,

Chirag Agarwal, Daniel D’souza, and Sara Hooker, “Es- timating example difficulty using variance of gradients,” in CVPR, 2022, pp. 10368–10378

work page 2022

[8] [8]

Pairwise learning for medical image seg- mentation,

Renzhen Wang, Shilei Cao, Kai Ma, Yefeng Zheng, and Deyu Meng, “Pairwise learning for medical image seg- mentation,” MIA, vol. 67, pp. 101876, 2021

work page 2021