Shape Distribution Matters: Shape-specific Mixture-of-Experts for Amodal Segmentation under Diverse Occlusions

Chen Hui; Jeonghaeng Lee; Sanghoon Lee; Weisi Lin; Yujia Liu; Zhixuan Li

REVIEW 2 major objections 4 minor 6 references

Shape Distribution Matters: Shape-specific Mixture-of-Experts for Amodal Segmentation under Diverse Occlusions

T0 review · 2 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ShapeMoE claims that modeling each object's shape as a Gaussian embedding and routing it to a specialized lightweight expert yields state-of-the-art amodal segmentation on COCOA-cls, D2SA, and KINS, with the largest gains on occluded…

desk verdict A plausible, well-ablated architecture with real-looking gains, but test-set hyperparameter tuning and missing variance keep the headline numbers from being fully trustworthy. read the letter →

arxiv 2508.01664 v1 pith:R65CGISX submitted 2025-08-03 cs.CV

Zhixuan Li , Yujia Liu , Chen Hui , Jeonghaeng Lee , Sanghoon Lee , Weisi Lin This is my paper

classification cs.CV

keywords amodalinstancesegmentationmixture-of-expertsshape-awareroutingGaussianshapedistributionoccludedregionsparseexpertselectionSAMhyper-network

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that amodal segmentation under diverse occlusions improves when each object is routed, by its shape, to a specialized expert rather than processed by one generic decoder. It claims that a single model cannot capture both rigid and deformable shapes, and that naive Mixture-of-Experts routing without modeling shape distribution leads to mismatched assignments. ShapeMoE encodes each visible mask as a Gaussian distribution in a learned latent shape space, samples a shape code from it, and uses a sparse router to select one of four lightweight hyper-network experts in a SAM-style decoder. Across COCOA-cls, D2SA, and KINS it reports consistently higher mIoU for full amodal masks and especially occluded regions than fully supervised and zero-shot methods. If the paper is right, explicit shape-distribution modeling is a productive direction for occlusion reasoning rather than just scaling a single decoder.

What carries the argument

The mechanism is a shape-specialized sparse mixture-of-experts built on a Gaussian shape embedding. A Shape Distribution Encoder maps the mask embedding to parameters $\mu$ and $\sigma$; the router samples $l_o = \mu + \mathrm{Softplus}(\sigma) \odot \eta$ with $\eta \sim N(0,1)$, computes expert scores $s = W l_o$, keeps the top-1 expert via TopK and softmax, and activates only that expert's hyper-network, the lightweight weight-generating stage of a SAM-style mask decoder. A coefficient-of-variation-squared loss forces balanced expert usage across the four experts.

What would settle it

Retrain ShapeMoE with a multimodal latent distribution, for example a Gaussian mixture, in place of the single Gaussian $N(\mu, \sigma^2)$ and compare mIoUocc on COCOA-cls; if accuracy does not drop, the Gaussian assumption is not the source of the reported gains.

Watch

Extended reading notes

Core claim

The central discovery is that explicit, distribution-based shape modeling changes how well a mixture-of-experts segmenter handles occlusion. The paper claims that encoding each object's visible mask as a Gaussian distribution $N(\mu, \sigma^2)$ in a learned latent shape space, sampling a shape code from that distribution, and routing the code to one of four lightweight hyper-network experts yields consistently higher amodal segmentation accuracy than a single generic decoder or a softmax-routed MoE. On COCOA-cls, D2SA, and KINS, ShapeMoE reports mIoUfull of 89.53, 92.40, and 89.82 and mIoUocc of 30.68, 37.95, and 49.70, surpassing both fully supervised and zero-shot baselines, with the largest relative gains on occluded-region IoU.

Load-bearing premise

The load-bearing premise is that one Gaussian per object, estimated from the visible mask, faithfully represents the object's amodal shape, so that small or ambiguous visible regions still give routing information that matches the correct expert.

Editorial extensions

If this is right

Four experts with top-1 routing give the best results on all three datasets; more experts (8 or 16) or more selected experts per sample slightly hurt accuracy.
Removing the expert balancing loss consistently lowers performance (for example, COCOA-cls mIoUfull drops from 89.53 to 87.61), so balanced expert utilization is part of the method's effectiveness.
Routing from a probabilistic shape embedding, rather than a fixed feature, lets unseen shapes be projected into the same latent space and assigned to a suitable expert.
The expert is the decoder hyper-network rather than the full mask decoder, so the added shape capacity costs little extra computation at inference.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the router is trained only on masks seen with their visible regions, routing quality is likely to degrade as visible area shrinks; a direct test would stratify mIoUocc by visible-mask size, which the paper does not report.
The Gaussian assumption is untested against multimodal alternatives; replacing the single Gaussian with a Gaussian-mixture or flow-based prior could sharpen expert specialization for categories with bimodal shapes (rigid versus deformable), an extension the paper leaves open.
The reported gains over foundation-model baselines suggest that adding a shape-routed expert set to an off-the-shelf segmenter could be a cheap way to adapt it to occlusion-heavy domains, though the paper does not demonstrate such transfer.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The gist: ShapeMoE is a well-put-together architecture paper with a genuinely new idea in this subfield. Encoding each object as a Gaussian in a latent shape space and routing to one of four hyper-network experts is a sensible, cheap way to specialize the SAM decoder. The ablations do real work: 4 experts beat 1, top-1 routing beats top-k, and the balancing loss helps. The mIoU gains over C2F-Seg and SAMBA are large, especially on COCOA-cls and KINS, and the occluded-region numbers, where reported, are impressive.

Now the soft spots. The stress-test concern holds up. Section 4.4 tunes the number of experts, the selected-expert number, and the balancing loss on all three datasets, and for KINS that is the same test split shown in Table 1. That conflates model selection with evaluation. The margins inside the ablations are small (0.34 to 1.67 mIoUfull), and with three runs averaged and no standard deviations reported, those differences are within plausible seed noise. The large gaps to C2F-Seg and SAMBA probably survive, but the final numbers should be treated as optimistic until a clean validation split is used. Also, the Gaussian assumption for shape distributions is never tested against alternative distributions or external shape labels. That is a modeling bet, not a fatal one, but a diagnostic would help.

What is missing is standard stuff: code, error bars, and a clear statement of which split was used for picking K=4 and k=1. The related-work coverage looks fine, and the self-citation pattern is normal for the area. No sign of circularity: the router is trained with segmentation and balancing losses, and the Gaussian is an internal latent variable.

Bottom line: this paper deserves a serious referee. It is a strong subfield advance if the numbers hold, and the idea is unlikely to be scooped in a trivial way. I would accept it into peer review with the requirement that the authors release code, report variance, and either use a held-out validation set for hyperparameter selection or explicitly acknowledge the current procedure as a limitation. If I worked on amodal segmentation, I would cite it.

Referee Report

2 major / 4 minor

Summary. The paper proposes ShapeMoE, a sparse mixture-of-experts framework for amodal instance segmentation. The method encodes each object's visible mask into a Gaussian embedding via a shape distribution encoder, routes each instance to a small number of shape-specialized hyper-network experts through a sparse router, and supervises the amodal mask with cross-entropy plus an expert-balancing loss. Experiments on COCOA-cls, D2SA, and KINS compare ShapeMoE with prior amodal segmentation methods and report consistent improvements in mIoUfull and mIoUocc, with the largest gains on occluded regions. The authors also present ablations over the total number of experts, the number of selected experts, and the balancing loss.

Significance. If the empirical claims hold, ShapeMoE would be a useful contribution: it applies sparse MoE to amodal segmentation with a shape-aware routing signal, and the expert design is parameter-efficient because it replicates only the lightweight hyper-network rather than the full mask decoder. The paper also provides interpretability claims through shape-to-expert correspondence. The core idea is plausible and the reported gains over strong baselines such as C2F-Seg and SAMBA are substantial, particularly on COCOA-cls. However, the evaluation as presented is weakened by test-set hyperparameter selection and the absence of variance estimates, and the paper does not yet provide code or sufficient experimental details to independently reproduce the numbers.

major comments (2)

[Section 4.3 and Section 4.4, Tables 2 and 3] The central SOTA claim is compromised by using the KINS test set for model selection. Table 1 reports ShapeMoE on the KINS test set, and then Table 2 and Table 3 use the same KINS split to choose the number of experts (K=4), the number of selected experts (k=1), and whether to keep the balancing loss. Reported numbers on a split that was used to select hyperparameters are optimistically biased and cannot be treated as unbiased estimates of generalization. Please either use a held-out validation split for these choices, report the selection procedure explicitly, or re-evaluate the final configuration on a separate test set.
[Section 4.2, Tables 1-4] The paper states that all experiments are repeated three times and averaged, but no standard deviations, confidence intervals, or significance tests are reported anywhere. This matters because several key margins are small: the KINS mIoUfull gap over SAMBA is 1.35 points, and the improvement from 1 expert to 4 experts in Table 2 is 1.67 points on KINS and 0.34 points on COCOA-cls. Without variance information, these differences cannot be distinguished from run-to-run noise, especially with only three runs. Please report per-seed results, standard deviations, or statistical tests for the main tables.

minor comments (4)

[Section 2.1] The related-work paragraph introduces a method named VEAL without a citation or explanation, and then says 'Based on our Shape-MoE, shape distributions are learned...' This appears to be either a leftover from an earlier draft or an unsupported claim; please clarify the relationship between VEAL and the proposed method.
[Section 4.1] The text says 'all methods are trained and evaluated on the official training and validation sets of each dataset,' but Section 4.3 evaluates on the KINS test set. Please reconcile this discrepancy and state explicitly which split is used for KINS.
[Section 3.5] The justification for replacing the full decoder with only the hyper-network is qualitative. It would strengthen the paper to include an ablation where the full decoder is replicated as experts, to quantify the claimed efficiency-accuracy trade-off.
[Section 4.4, Tables 2-4] The ablation tables do not include the metric being reported (presumably mIoUfull), nor any indication of the validation split used. Adding metric labels and split information would make the tables self-contained.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: ShapeMoE's amodal mask predictions are supervised by external ground-truth masks, and its routing and balancing losses do not encode the benchmark outcomes by construction.

full rationale

Walking the derivation chain in Sections 3.2–3.6, the pipeline is: the visible mask is embedded (em), a Shape Distribution Encoder predicts Gaussian parameters (mu, sigma) via Eq. (1), a latent shape representation is sampled via Eq. (2), the Shape-Aware Sparse Router computes a sparse routing distribution via Eqs. (3)–(4), and the selected hyper-network expert predicts the amodal mask. The only supervision for the final prediction is the cross-entropy loss LCE against the ground-truth amodal mask Ma in Eq. (5), plus the CV^2 balancing loss on routing probabilities. Nothing in these equations defines the predicted amodal mask as an algebraic or statistical transform of the ground-truth mask, and no fitted parameter is renamed as a prediction. The Gaussian shape-distribution assumption in Section 3.3 is an internal modeling ansatz, not an input that is recovered as output; it is not validated against alternative distributions, but that is a modeling-robustness concern, not circularity. The ablations in Section 4.4 are used to select hyperparameters such as the number of experts and selected experts, and the same KINS test split is used both for that selection and for reporting final numbers; this is a test-set-contamination and statistical-significance concern, not a circular derivation, because the reported metric values are still measured against external ground-truth masks rather than being forced to equal the hyperparameter values. No load-bearing self-citation chain is present: the method is trained end-to-end and compared against external benchmarks, and the arguments for expert specialization are supported by the reported external metrics. The paper is self-contained against external ground-truth amodal masks, so the appropriate circularity score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 1 invented entities

The central empirical claim rests on several unverified modeling assumptions rather than a derivation: the Gaussian shape prior, the informativeness of the visible mask embedding, and the claim that the hyper-network is the shape-critical stage. The only tuned quantities reported are K and k, chosen by validation and test-set ablations. The latent dimension and loss weights are unreported, which increases the burden on independent reproduction.

free parameters (4)

total number of experts K = 4
Best value in Table 2 across COCOA-cls, D2SA, and KINS; values 1, 2, 4, 8, 16 were tested.
selected experts per sample k = 1
Best value in Table 3 with the total number of experts fixed at 4.
latent shape dimension d = not specified
Hidden dimension of the sampled shape representation lo and the routing matrix W; chosen by hand but never reported.
LCV2 loss weight = not specified
Equation 5 sums LCE and LCV2, but no weighting coefficient is reported.

assumptions (3)

domain assumption Object shapes follow a Gaussian distribution in the learned latent space.
Stated in Section 3.3 and used in Eqs. 1 and 2; no empirical justification or comparison to alternative distributions is provided.
domain assumption The visible mask embedding contains enough shape information to predict the Gaussian parameters.
The Shape Distribution Encoder in Section 3.3 takes only the mask embedding em as input. For heavily occluded objects, the visible mask may be too sparse to support reliable routing.
domain assumption The hyper-network is the component that determines shape-specific mask prediction, so duplicating it creates shape-specialized experts.
Motivated in Section 3.5 as the reason for replicating only the hyper-network; no ablation duplicates the two-way transformer instead, or tests both components together.

invented entities (1)

Latent Gaussian shape distribution per object
purpose: Compact shape representation used by the router for expert selection
The distribution is learned from the visible mask embedding and never validated against external shape labels or an independent shape benchmark. The paper claims unseen shapes project into the same space, but does not provide evidence for this.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Shape Distribution Matters: Shape-specific Mixture-of-Experts for Amodal Segmentation under Diverse Occlusions." pith.science (2026). https://pith.science/paper/R65CGISX

@misc{pith2026250801664,
  author       = {Pith},
  title        = {Pith review of: Shape Distribution Matters: Shape-specific Mixture-of-Experts for Amodal Segmentation under Diverse Occlusions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R65CGISX}},
  note         = {Machine review of arXiv:2508.01664}
}

read the original abstract

Amodal segmentation targets to predict complete object masks, covering both visible and occluded regions. This task poses significant challenges due to complex occlusions and extreme shape variation, from rigid furniture to highly deformable clothing. Existing one-size-fits-all approaches rely on a single model to handle all shape types, struggling to capture and reason about diverse amodal shapes due to limited representation capacity. A natural solution is to adopt a Mixture-of-Experts (MoE) framework, assigning experts to different shape patterns. However, naively applying MoE without considering the object's underlying shape distribution can lead to mismatched expert routing and insufficient expert specialization, resulting in redundant or underutilized experts. To deal with these issues, we introduce ShapeMoE, a shape-specific sparse Mixture-of-Experts framework for amodal segmentation. The key idea is to learn a latent shape distribution space and dynamically route each object to a lightweight expert tailored to its shape characteristics. Specifically, ShapeMoE encodes each object into a compact Gaussian embedding that captures key shape characteristics. A Shape-Aware Sparse Router then maps the object to the most suitable expert, enabling precise and efficient shape-aware expert routing. Each expert is designed as lightweight and specialized in predicting occluded regions for specific shape patterns. ShapeMoE offers well interpretability via clear shape-to-expert correspondence, while maintaining high capacity and efficiency. Experiments on COCOA-cls, KINS, and D2SA show that ShapeMoE consistently outperforms state-of-the-art methods, especially in occluded region segmentation. The code will be released.

Figures

Figures reproduced from arXiv: 2508.01664 by the authors.

**Figure 2.** Given an input image and a visible mask, ShapeMoE performs amodal segmentation through the following stages. (1) [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Architecture of the Shape Distribution Encoder. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Qualitative results of the proposed ShapeMoE. Four representative cases are shown across various object categories, [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

6 extracted references · 4 canonical work pages

[6]

International Journal of Computer Vision , 129: 3195–3215

Visiting the invisible: Layer-by-layer completed scene decomposition. International Journal of Computer Vision , 129: 3195–3215. Zhu, Y .; Tian, Y .; Metaxas, D.; and Doll´ar, P. 2017. Seman- tic amodal segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 1464–1472

work page 2017
[755]

Liu, Z.; Qiao, L.; Chu, X.; Ma, L.; and Jiang, T

Springer. Liu, Z.; Qiao, L.; Chu, X.; Ma, L.; and Jiang, T. 2025. To- wards Efficient Foundation Model for Zero-shot Amodal Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 20254–20264. Muhadi, N. A.; Abdullah, A. F.; Bejo, S. K.; Mahadi, M. R.; and Mijic, A. 2020. Image segmentation methods for flood mon...

work page 2025
[2020]

InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3784–3792

Self-supervised scene de-occlusion. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3784–3792. Zhang, Z.; Chen, A.; Xie, L.; Yu, J.; and Gao, S. 2019. Learning semantics-aware distance map with semantics lay- ering network for amodal instance segmentation. In ACM International Conference on Multimedia, 2124–2132. Zheng...

work page 2019
[2021]

In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 35, 2995–3003

Amodal segmentation based on visible region seg- mentation and shape prior. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 35, 2995–3003. Xiong, Y .; Varadarajan, B.; Wu, L.; Xiang, X.; Xiao, F.; Zhu, C.; Dai, X.; Wang, D.; Sun, F.; Iandola, F.; et al. 2024. Ef- ficientSAM: Leveraged masked image pretraining for effi- cient seg...

work page 2024
[2023]

In Proceedings of the AAAI Conference on Artifi- cial Intelligence, volume 37, 313–321

Amodal instance segmentation via prior-guided ex- pansion. In Proceedings of the AAAI Conference on Artifi- cial Intelligence, volume 37, 313–321. Chen, T.; Lu, A.; Zhu, L.; Ding, C.; Yu, C.; Ji, D.; Li, Z.; Sun, L.; Mao, P.; and Zang, Y . 2024. Sam2-adapter: Evaluating & adapting segment anything 2 in downstream tasks: Camouflage, shadow, medical image s...

arXiv 2024
[2024]

arXiv preprint arXiv:2412.10859

Duet: Dual clustering enhanced multivariate time se- ries forecasting. arXiv preprint arXiv:2412.10859. Ravi, N.; Gabeur, V .; Hu, Y .-T.; Hu, R.; Ryali, C.; Ma, T.; Khedr, H.; R¨adle, R.; Rolland, C.; Gustafson, L.; et al. 2024. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714. Riquelme, C.; Puigcerver, J.; Mustafa, B.; Neuma...

arXiv 2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [6]

International Journal of Computer Vision , 129: 3195–3215

Visiting the invisible: Layer-by-layer completed scene decomposition. International Journal of Computer Vision , 129: 3195–3215. Zhu, Y .; Tian, Y .; Metaxas, D.; and Doll´ar, P. 2017. Seman- tic amodal segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 1464–1472

work page 2017

[2] [755]

Liu, Z.; Qiao, L.; Chu, X.; Ma, L.; and Jiang, T

Springer. Liu, Z.; Qiao, L.; Chu, X.; Ma, L.; and Jiang, T. 2025. To- wards Efficient Foundation Model for Zero-shot Amodal Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 20254–20264. Muhadi, N. A.; Abdullah, A. F.; Bejo, S. K.; Mahadi, M. R.; and Mijic, A. 2020. Image segmentation methods for flood mon...

work page 2025

[3] [2020]

InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3784–3792

Self-supervised scene de-occlusion. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3784–3792. Zhang, Z.; Chen, A.; Xie, L.; Yu, J.; and Gao, S. 2019. Learning semantics-aware distance map with semantics lay- ering network for amodal instance segmentation. In ACM International Conference on Multimedia, 2124–2132. Zheng...

work page 2019

[4] [2021]

In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 35, 2995–3003

Amodal segmentation based on visible region seg- mentation and shape prior. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 35, 2995–3003. Xiong, Y .; Varadarajan, B.; Wu, L.; Xiang, X.; Xiao, F.; Zhu, C.; Dai, X.; Wang, D.; Sun, F.; Iandola, F.; et al. 2024. Ef- ficientSAM: Leveraged masked image pretraining for effi- cient seg...

work page 2024

[5] [2023]

In Proceedings of the AAAI Conference on Artifi- cial Intelligence, volume 37, 313–321

Amodal instance segmentation via prior-guided ex- pansion. In Proceedings of the AAAI Conference on Artifi- cial Intelligence, volume 37, 313–321. Chen, T.; Lu, A.; Zhu, L.; Ding, C.; Yu, C.; Ji, D.; Li, Z.; Sun, L.; Mao, P.; and Zang, Y . 2024. Sam2-adapter: Evaluating & adapting segment anything 2 in downstream tasks: Camouflage, shadow, medical image s...

arXiv 2024

[6] [2024]

arXiv preprint arXiv:2412.10859

Duet: Dual clustering enhanced multivariate time se- ries forecasting. arXiv preprint arXiv:2412.10859. Ravi, N.; Gabeur, V .; Hu, Y .-T.; Hu, R.; Ryali, C.; Ma, T.; Khedr, H.; R¨adle, R.; Rolland, C.; Gustafson, L.; et al. 2024. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714. Riquelme, C.; Puigcerver, J.; Mustafa, B.; Neuma...

arXiv 2024