A masked cluster-prediction transformer over four sign-language streams sets state-of-the-art results on multiple ASL translation and recognition benchmarks using only public pre-training data.
Testing MediaPipe Holistic for Linguistic Analysis of Nonmanual Markers in Sign Languages
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Advances in Deep Learning have made possible reliable landmark tracking of human bodies and faces that can be used for a variety of tasks. We test a recent Computer Vision solution, MediaPipe Holistic (MPH), to find out if its tracking of the facial features is reliable enough for a linguistic analysis of data from sign languages, and compare it to an older solution (OpenFace, OF). We use an existing data set of sentences in Kazakh-Russian Sign Language and a newly created small data set of videos with head tilts and eyebrow movements. We find that MPH does not perform well enough for linguistic analysis of eyebrow movement - but in a different way from OF, which is also performing poorly without correction. We reiterate a previous proposal to train additional correction models to overcome these limitations.
fields
cs.CL 1years
2024 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
SHuBERT: Self-Supervised Sign Language Representation Learning via Multi-Stream Cluster Prediction
A masked cluster-prediction transformer over four sign-language streams sets state-of-the-art results on multiple ASL translation and recognition benchmarks using only public pre-training data.