REVIEW 4 minor 41 references
Sampling Gaussian frame embeddings before optimal-transport pseudo-labeling breaks the local-optimum trap that traps deterministic unsupervised action segmenters.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-11 07:27 UTC pith:35GK6FLM
load-bearing objection Clean, reproducible fix: Gaussian frame embeddings + GCN sampling before OT give solid gains over ASOT/VASOT without changing the OT math.
Learning Probabilistic Embeddings for Unsupervised Action Segmentation
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Modeling each frame embedding as a diagonal Gaussian and sampling from it before every optimal-transport pseudo-label step, rather than feeding a single deterministic vector into the transport, prevents the joint representation-and-clustering loop from collapsing into a poor local optimum and produces measurably better unsupervised action segmentations.
What carries the argument
Probabilistic embeddings: a GCN predicts per-frame mean and log-variance; the re-parameterization trick draws Monte-Carlo samples that become the cost matrix for fused Kantorovich–Gromov-Wasserstein optimal transport; the resulting soft assignments supervise an uncertainty-aware cross-entropy loss.
Load-bearing premise
The number of action classes for every activity is known in advance and set exactly to the ground-truth count; if that number must be discovered from data the reported gains may shrink.
What would settle it
Run the identical GCN-plus-sampling pipeline on the same four datasets while forcing K to be estimated by a standard model-selection criterion (e.g., silhouette or BIC on the learned means) instead of the ground-truth class count; if the MoF and F1 lifts over the deterministic baseline disappear, the central claim fails.
If this is right
- Any existing OT-based unsupervised segmenter can be upgraded simply by swapping its deterministic encoder for a mean-and-variance GCN and sampling before transport.
- The same sampling-before-clustering idea can be dropped into other joint-representation-and-clustering loops that currently over-fit early pseudo-labels.
- Because only the mean is used at inference, the accuracy gains come with zero extra test-time cost.
- The learned per-frame variance itself becomes a free uncertainty map that correlates with segmentation errors.
Where Pith is reading between the lines
- The same Gaussian-sampling trick may stabilize other EM-style video pipelines that alternate between assignment and representation learning, such as unsupervised action detection or multi-view clustering.
- If the Monte-Carlo sample count is annealed with training progress (more samples early when uncertainty is high), further gains may appear without raising asymptotic cost.
- The method’s reliance on a known K suggests a natural next experiment: replace the fixed action prototypes with a Dirichlet-process or Chinese-restaurant prior and measure how much of the reported lift survives.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes PEOT, which replaces deterministic frame embeddings in OT-based unsupervised temporal action segmentation with probabilistic embeddings modeled as diagonal Gaussians. A GCN predicts per-frame mean and log-variance; samples are drawn via the reparameterization trick before computing the fused Kantorovich–Gromov-Wasserstein OT pseudo-labels that supervise a cross-entropy loss (Eqs. 5–11). At inference only the mean is used. The method is evaluated on Breakfast, YTI, 50Salads (Eval) and Desktop Assembly, reporting MoF/F1/mIoU after activity-level Hungarian matching. Controlled ablations (Tables 2–3) show that grafting the same probabilistic layer onto ASOT and VASOT yields consistent gains (up to +12.2 MoF / +12.1 F1), and PEOT is best or second-best on 9 of 12 metrics.
Significance. If the reported gains hold, the work supplies a simple, drop-in improvement for the dominant joint-representation-and-OT paradigm. The ablations cleanly isolate the probabilistic component from the GCN architecture, multi-seed statistics appear in the supplement, and public code is provided. These elements make the contribution immediately usable by the community and strengthen the case that modeling embedding uncertainty mitigates early overfitting to noisy pseudo-labels.
minor comments (4)
- Abstract and several body paragraphs contain missing spaces (e.g., “proposetolearnaprobabilisticembedding”). A global pass for typesetting is needed.
- Sec. 4.1 states that K is set to the ground-truth number of actions per activity. While this is standard in the literature, a short remark on sensitivity to over-/under-estimated K would help readers who must choose K without labels.
- Fig. 2 caption and the surrounding text use both “covariance” and “variance”; the model predicts a diagonal covariance, so consistent terminology would avoid confusion.
- Table 1 reports literature numbers while Table 2 recomputes ASOT/VASOT with public code; a one-sentence note explaining the modest numerical differences would improve transparency.
Circularity Check
No significant circularity; purely empirical ML paper whose gains are measured against external ground-truth via Hungarian matching.
full rationale
The paper proposes replacing deterministic frame embeddings with Gaussians (sampled via reparameterization before OT pseudo-label estimation) inside an otherwise standard joint representation-learning + OT pipeline (ASOT/VASOT). All claims are empirical: MoF/F1/mIoU improvements are obtained by matching predicted segments to held-out ground-truth labels with the Hungarian algorithm on four public datasets. No equation, uniqueness theorem, or fitted constant is redefined as a “prediction”; hyperparameters follow the cited baselines and are ablated; self-citations are limited to ordinary prior methods used as controls. The known-K assumption is field-standard and does not create a definitional loop. The derivation chain is therefore self-contained against external benchmarks.
Axiom & Free-Parameter Ledger
free parameters (3)
- number of Monte-Carlo samples M =
3
- GCN neighborhood size =
3 frames
- OT hyper-parameters (α, r, ρ, λ) =
dataset-specific (e.g. α=0.4, r=0.04 on Breakfast)
axioms (3)
- domain assumption The number of action classes K equals the ground-truth count for each activity.
- ad hoc to paper Frame features are well-modeled by independent diagonal Gaussians whose parameters can be predicted from a local temporal graph.
- domain assumption The fused KOT+GWOT objective of ASOT is a suitable surrogate for temporal action segmentation.
invented entities (1)
-
PEOT probabilistic embedding layer (GCN-mean + GCN-var + reparameterized sampling)
no independent evidence
read the original abstract
This paper concerns the problem of unsupervised temporal action segmentation for long, untrimmed videos. Recent successful approaches follow a joint representation learning and clustering paradigm, where optimal transport (OT) is adopted to produce pseudo labels for learning frame representations. These approaches alternate between estimating pseudo labels using OT and optimizing the parameters with gradient descent during training, where OT is used for obtaining the final temporal action segmentation. A major limitation of these works is that they learn a deterministic embedding for frame representations. The iterative procedure between learning deterministic embeddings based on pseudo labels and estimating pseudo labels from the learned embedding can thus get quickly stuck in a local optimum. As an alternative, we thus propose to learn a probabilistic embedding for frame representations. The embeddings are modeled by Gaussian distributions and we sample from the distributions before estimating the pseudo labels. We evaluate our approach on several challenging temporal action segmentation datasets and achieve results comparable to, and in some cases, better than the state of the art. Compared to baselines with deterministic embeddings, our approach improves MoF up to 20.7\% and F1-score up to 19.0\%. Our code is available at https://github.com/derkbreeze/PEOT.
Figures
Reference graph
Works this paper leans on
-
[1]
In: Proceedings of the IEEE conference on computer vision and pattern recognition
Alayrac, J.B., Bojanowski, P., Agrawal, N., Sivic, J., Laptev, I., Lacoste-Julien, S.: Unsupervised learning from narrated instruction videos. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4575–4583 (2016) 16 S. Li et al
2016
-
[2]
In: Proceedings of the IEEE/CVF International Conference on Computer Vision
Ali, A.S., Mahmood, S.A., Saeed, M., Konin, A., Zia, M.Z., Tran, Q.H.: Joint self-supervised video alignment and action segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10807–10818 (2025)
2025
-
[3]
In: Proceedings of the IEEE/CVF International Con- ference on Computer Vision
Bueno-Benito, E., Dimiccoli, M.: Clot: Closed loop optimal transport for unsuper- vised action segmentation. In: Proceedings of the IEEE/CVF International Con- ference on Computer Vision. pp. 10719–10729 (2025)
2025
-
[4]
Mathematics of computation87(314), 2563– 2609 (2018)
Chizat, L., Peyré, G., Schmitzer, B., Vialard, F.X.: Scaling algorithms for unbal- anced optimal transport problems. Mathematics of computation87(314), 2563– 2609 (2018)
2018
-
[5]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
De Plaen, H., De Plaen, P.F., Suykens, J.A., Proesmans, M., Tuytelaars, T., Van Gool, L.: Unbalanced optimal transport: A unified framework for object de- tection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3198–3207 (2023)
2023
-
[6]
Journal ofthe royalstatistical society: seriesB (method- ological)39(1), 1–22 (1977)
Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data viathe emalgorithm. Journal ofthe royalstatistical society: seriesB (method- ological)39(1), 1–22 (1977)
1977
-
[7]
In: Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition
Farha, Y.A., Gall, J.: Ms-tcn: Multi-stage temporal convolutional network for ac- tion segmentation. In: Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition. pp. 3575–3584 (2019)
2019
-
[8]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Huang, Y., Sugano, Y., Sato, Y.: Improving action segmentation via graph-based temporal reasoning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 14024–14034 (2020)
2020
-
[9]
Kendall, A., Gal, Y.: What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems30(2017)
2017
-
[10]
In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
Khan, H., Haresh, S., Ahmed, A., Siddiqui, S., Konin, A., Zia, M.Z., Tran, Q.H.: Timestamp-supervised action segmentation with graph convolutional networks. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 10619–10626. IEEE (2022)
2022
-
[11]
arXiv preprint arXiv:1412.6980 (2014)
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Pith/arXiv arXiv 2014
-
[12]
arXiv preprint arXiv:1312.6114 (2013)
Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)
Pith/arXiv arXiv 2013
-
[13]
arXiv preprint arXiv:1609.02907 (2016)
Kipf, T.: Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016)
Pith/arXiv arXiv 2016
-
[14]
In: Proceedings of the IEEE conference on computer vision and pattern recognition
Kuehne, H., Arslan, A., Serre, T.: The language of actions: Recovering the syn- tax and semantics of goal-directed human activities. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 780–787 (2014)
2014
-
[15]
Naval research logistics quarterly2(1-2), 83–97 (1955)
Kuhn, H.W.: The hungarian method for the assignment problem. Naval research logistics quarterly2(1-2), 83–97 (1955)
1955
-
[16]
In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition
Kukleva, A., Kuehne, H., Sener, F., Gall, J.: Unsupervised learning of action classes with continuous temporal embedding. In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition. pp. 12066–12074 (2019)
2019
-
[17]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Kumar, S., Haresh, S., Ahmed, A., Konin, A., Zia, M.Z., Tran, Q.H.: Unsuper- vised action segmentation by joint representation learning and online clustering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20174–20185 (2022)
2022
-
[18]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Li, J., Todorovic, S.: Action shuffle alternating learning for unsupervised action segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12628–12636 (2021) Learning Probabilistic Embeddings for Unsupervised Action Segmentation 17
2021
-
[19]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Li, S., Kong, Y., Rezatofighi, H.: Learning of global objective for network flow in multi-object tracking. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8855–8865 (2022)
2022
-
[20]
Advances in Neural Information Processing Systems36, 18046–18075 (2023)
Nguyen, K., Ho, N.: Energy-based sliced wasserstein distance. Advances in Neural Information Processing Systems36, 18046–18075 (2023)
2023
-
[21]
arXiv preprint arXiv:1810.00319 (2018)
Oh, S.J., Murphy, K., Pan, J., Roth, J., Schroff, F., Gallagher, A.: Modeling uncer- tainty with hedged instance embedding. arXiv preprint arXiv:1810.00319 (2018)
Pith/arXiv arXiv 2018
-
[22]
In: International conference on machine learning
Peyré, G., Cuturi, M., Solomon, J.: Gromov-wasserstein averaging of kernel and distance matrices. In: International conference on machine learning. pp. 2664–2672. PMLR (2016)
2016
-
[23]
Engineering Applications of Artificial Intelli- gence148(2025)
Romeo, L., Marani, R., Perri, A.G., Gall, J.: Multi-modal temporal action segmen- tation for manufacturing scenarios. Engineering Applications of Artificial Intelli- gence148(2025)
2025
-
[24]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Sarfraz, S., Murray, N., Sharma, V., Diba, A., Van Gool, L., Stiefelhagen, R.: Temporally-weighted hierarchical clustering for unsupervised action segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11225–11234 (2021)
2021
-
[25]
In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
Sener, F., Yao, A.: Unsupervised learning and segmentation of complex activities from video. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 8368–8376 (2018)
2018
-
[26]
In: Proceedings of the IEEE/CVF international conference on computer vision
Shi, Y., Jain, A.K.: Probabilistic face embeddings. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 6902–6911 (2019)
2019
-
[27]
In: Proceedings of the AAAI Conference on Artificial Intelligence
Spurio, F., Bahrami, E., Francesca, G., Gall, J.: Hierarchical vector quantization for unsupervised action segmentation. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 39, pp. 6996–7005 (2025)
2025
-
[28]
In: Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing
Stein, S., McKenna, S.J.: Combining embedded accelerometers with computer vi- sion for recognizing food preparation activities. In: Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing. pp. 729–738 (2013)
2013
-
[29]
In: European Conference on Computer Vision
Sun, J.J., Zhao, J., Chen, L.C., Schroff, F., Adam, H., Liu, T.: View-invariant probabilistic embedding for human pose. In: European Conference on Computer Vision. pp. 53–70. Springer (2020)
2020
-
[30]
In: 2021 IEEE International Conference on Image Processing (ICIP)
Swetha, S., Kuehne, H., Rawat, Y.S., Shah, M.: Unsupervised discriminative em- bedding for sub-action learning in complex activities. In: 2021 IEEE International Conference on Image Processing (ICIP). pp. 2588–2592. IEEE (2021)
2021
-
[31]
Notes of Course at University of Cambridge3(2018)
Thorpe, M.: Introduction to optimal transport. Notes of Course at University of Cambridge3(2018)
2018
-
[32]
In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision
Tran, Q.H., Mehmood, A., Ahmed, M., Naufil, M., Zafar, A., Konin, A., Zia, Z.: Permutation-aware activity segmentation via unsupervised frame-to-segment alignment. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 6426–6436 (2024)
2024
-
[33]
Advances in neural information pro- cessing systems30(2017)
Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information pro- cessing systems30(2017)
2017
-
[34]
In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision
VidalMata, R.G., Scheirer, W.J., Kukleva, A., Cox, D., Kuehne, H.: Joint visual- temporal embedding for unsupervised learning of actions in untrimmed sequences. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 1238–1247 (2021)
2021
-
[35]
arXiv preprint arXiv:1412.6623 (2014) 18 S
Vilnis, L., McCallum, A.: Word representations via gaussian embedding. arXiv preprint arXiv:1412.6623 (2014) 18 S. Li et al
Pith/arXiv arXiv 2014
-
[36]
In: Pro- ceedings of the IEEE international conference on computer vision
Wang, H., Schmid, C.: Action recognition with improved trajectories. In: Pro- ceedings of the IEEE international conference on computer vision. pp. 3551–3558 (2013)
2013
-
[37]
In: Proceedings of the European conference on computer vision (ECCV)
Wang, X., Gupta, A.: Videos as space-time region graphs. In: Proceedings of the European conference on computer vision (ECCV). pp. 399–417 (2018)
2018
-
[38]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Xu, M., Gould, S.: Temporally consistent unbalanced optimal transport for un- supervised action segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14618–14627 (2024)
2024
-
[39]
watching
Yang, Y., Li, Y., Fermuller, C., Aloimonos, Y.: Robot learning manipulation action plans by “watching” unconstrained videos from the world wide web. In: Proceedings of the AAAI conference on artificial intelligence. vol. 29 (2015)
2015
-
[40]
In: European Conference on Computer Vision
Zatsarynna, O., Bahrami, E., Farha, Y.A., Francesca, G., Gall, J.: Gated temporal diffusion for stochastic long-term dense anticipation. In: European Conference on Computer Vision. pp. 454–472. Springer (2024)
2024
-
[41]
In: Proceedings of the IEEE/CVF international conference on computer vision
Zeng, R., Huang, W., Tan, M., Rong, Y., Zhao, P., Huang, J., Gan, C.: Graph convolutional networks for temporal action localization. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 7094–7103 (2019) Learning Probabilistic Embeddings for Unsupervised Action Segmentation Supplementary Materials Shuai Li1, Duc Manh Vu1, and Ju...
2019
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.