Skeleton-based Gait Index Estimation with LSTMs

Huu Hung Huynh; Jean Meunier; Trong Nguyen Nguyen

REVIEW 3 major objections 5 minor 17 references

Skeleton-based Gait Index Estimation with LSTMs

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A gait index built entirely from LSTM auto-encoder reconstruction errors separates normal from abnormal walking with an equal-error rate of 0.056 per sequence.

desk verdict A modest, clearly written application of LSTM autoencoders to gait scoring, with a plausible result undermined by a very small test set and no per-subject analysis. read the letter →

arxiv 1908.07416 v1 pith:ERDNKKVH submitted 2019-08-17 cs.CV

Trong Nguyen Nguyen , Huu Hung Huynh , Jean Meunier This is my paper · ORCID

classification cs.CV

keywords gaitindexestimationskeleton-basedanalysisLSTMautoencoderabnormaldetectionreconstructionerrorunsupervisedanomalyKinectskeletonequalrate

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a way to turn a skeleton video into a single number that measures how abnormal a person's gait is. The system is a stack of three LSTM auto-encoders, one for each spatial axis, trained only on normal walking sequences. Each auto-encoder reconstructs short segments of the selected 17 joints, and its mean-squared error is treated as a weak gait index; the three weak indices are combined by a weighted sum, then averaged over the whole 1200-frame sequence. On a dataset of nearly one hundred thousand skeletons, the resulting index separates normal from abnormal walking with an equal-error rate of 0.056 per sequence, against 0.139 for a one-class SVM baseline and 0.250 for an HMM baseline. The paper's point is that unsupervised reconstruction error, with no hand-crafted gait features, can serve as a competitive gait quality score.

What carries the argument

The central object is the LSTM auto-encoder with peephole connections: an encoder LSTM compresses a temporal sequence of joint coordinates into a latent representation, and a decoder LSTM reconstructs the input in reverse order from that representation. The mean-squared error between input and reconstruction is the weak gait index, under the assumption that normal patterns are reconstructed better than abnormal ones. Two auxiliary mechanisms carry the argument: the axis-wise decomposition (17 selected joints, three separate auto-encoders for X/Y/Z), and the weighted combination $w_k = e_k^{-1} / \sum_k e_k$ in which a model with smaller training MSE contributes more. A dropout layer on the encoder input slightly improves the score, and averaging per-segment indices over long sequences converts the noisy weak indices into the final gait index.

What would settle it

Run the same protocol on a held-out cohort with several subtle abnormal gaits (for instance, a 2-centimeter sole lift), and compute per-sequence weighted reconstruction indices; if the ROC AUC for separating normal from abnormal is not near the reported 0.969, or if mildly abnormal gaits overlap normal indices, the central assumption is falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the reconstruction error of an LSTM auto-encoder trained exclusively on normal gait is a usable gait index. An abnormal gait is expected to lie outside the learned distribution of normal motion and therefore to be reconstructed poorly, producing a higher index. The authors realize this with three independently trained peephole-LSTM auto-encoders processing the X, Y, and Z coordinate streams separately; per-axis MSEs are weak indices, and the final index is the weighted sum $w_k = e_k^{-1} / \sum_k e_k$ over axes. Averaging per-segment indices over a trial yields an AUC of 0.969 and an EER of 0.056, lower than the reimplemented HMM and one-class SVM baselines. The conclusion is that an automatic, unsupervised index can outperform recent hand-crafted gait analysis methods on this task.

Load-bearing premise

The load-bearing premise is that an auto-encoder trained only on normal gait will reconstruct abnormal gait poorly; if an abnormal walking pattern is close enough to normal motion in the skeleton stream, its reconstruction error will look normal and the index will not flag it.

Editorial extensions

If this is right

Abnormal-gait screening can be run without collecting or labeling abnormal samples at training time, since only normal sequences are needed.
The per-segment design supports online or streaming assessment, with a smoothed index obtained by averaging over a longer walk.
Because the recurrent auto-encoders accept variable-length inputs, the same pipeline can adapt to different walking speeds or camera frame rates.
The comparison suggests that a purely reconstruction-based, unsupervised index can beat hand-crafted-feature baselines on the abnormal-gait detection task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same reconstruction-error recipe could transfer to other movement-quality domains, such as rehabilitation exercise form or fall risk screening, because it only requires a stream of body joints and a definition of normal motion; this extension is not tested in the paper.
The weak Z-axis index hints that depth-channel noise or treadmill positioning dominates reconstruction error; a reader might expect a joint model or per-axis adaptive weighting to improve robustness.
With only four subjects in the test set, the reported 0.056 EER is likely optimistic for unseen populations; a realistic deployment would need validation across body shapes, speeds, and camera viewpoints.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a competent and clearly written application of a standard idea. Three axis-wise LSTM autoencoders are trained on normal gait only; reconstruction error is the weak index; the three errors are combined with weights inversely proportional to training MSE. That weighted sum plus dropout gives an EER of 0.056 per sequence, versus 0.139 for a reimplemented one-class SVM and 0.250 for an HMM. The core idea is not new — autoencoder reconstruction error as an anomaly score is textbook — but the specific combination of axis-wise decomposition and inverse-training-error weighting for skeleton gait index is a reasonable twist, and the paper is honest about what it does. What it does well: the method is simple to reproduce from the description, the training details are concrete (T=12, 256 hidden units, 100 epochs, dropout 0.5), and the paper reports per-segment and per-sequence results with AUC, EER, precision, specificity, sensitivity, accuracy, and F1. The observations that the Z-axis model is weaker and that weighting and temporal smoothing help are internally coherent. Comparing against their own earlier HMM work is a legitimate baseline choice, not a citation problem. Soft spots, in proportion: the evaluation is the whole ballgame and it is thin. Nine subjects total, four in the test set. The per-sequence EER of 0.056 is computed from four normal sequences and 32 abnormal sequences, so the ROC is essentially a step function and one or two subjects could be driving the pooled AUC. There are no error bars, no per-subject breakdown, and no released code or data. The load-bearing assumption — that an autoencoder trained on normal gaits from five subjects will produce higher reconstruction error on abnormal gaits from unseen subjects — is not independently checked. Because each axis is normalized to [0,1], distance-to-camera is partly mitigated, but subject-specific joint-coordinate patterns remain a plausible confound. The stress-test concern is real rather than manufactured; the missing per-subject table is the natural fix. Baseline reimplementation uncertainty is a minor issue for a conference paper. Bottom line: for someone in rehabilitation or fall-risk screening who wants a cheap gait abnormality score from a single depth camera, this is a useful existence proof and worth engaging with. If it were submitted today it would deserve peer review, but with a request for a larger evaluation and per-subject reporting. My verdict would be conditional acceptance, not a clean accept.

Referee Report

3 major / 5 minor

Summary. The manuscript proposes a gait-index estimator based on three LSTM autoencoders, one per coordinate axis, trained on normal gait skeleton sequences from five subjects. The reconstruction MSE of each autoencoder serves as a weak per-segment index; the three indices are combined with weights inversely proportional to training-set errors, and a per-sequence index is obtained by averaging segment indices. On a test set of four subjects (one normal and eight abnormal gaits each), the paper reports an EER of 0.056 per sequence, which it compares with reimplemented HMM and one-class SVM baselines and claims to outperform recent works. The central claim is empirical, not derivational, and rests on the assumption that reconstruction error generalizes from the training subjects to unseen subjects.

Significance. If the result holds, the method is a useful unsupervised alternative to hand-crafted gait features: it is simple, fully specified (LSTM equations, joint selection, weighting), and reports standard metrics (AUC, EER, accuracy, F1). The three-axis decomposition and inverse-error weighting are clearly described, and the experimental protocol (subject split, overlap parameters, segment lengths) is explicit enough to reproduce. However, the significance is conditional: the per-sequence result is based on only four normal test sequences, no per-subject or confidence-interval analysis is given, and the baselines are reimplementations rather than the original published methods. These gaps prevent the reported superiority from being established at the level claimed in the abstract.

major comments (3)

[Section 3.1 and Table 2] The per-sequence EER of 0.056 reported in Table 2 is computed from only four normal sequences (one per test subject) and 32 abnormal sequences. The ROC curve is therefore a step function whose EER threshold is determined by roughly two to three sequences, so the difference from the baseline EER of 0.139 is within the sampling noise of this test design. Please report per-subject ROC curves, bootstrap confidence intervals, or a leave-one-subject-out evaluation to support the comparative claim.
[Section 3.2] The load-bearing premise, stated as 'An input of abnormal gait is expected to provide a bad reconstruction compared with normal gait', is not directly verified for unseen subjects. Because the autoencoder is trained only on normal gaits from five subjects, a new subject's normal sequences may produce high reconstruction errors due to body proportions, joint-angle biases, or camera distance rather than gait abnormality. The paper provides no per-subject analysis, so the pooled AUC of 0.969 could be driven by subject identity. Please provide per-subject index distributions and demonstrate that, for each test subject, normal sequences are separated from abnormal ones.
[Section 3.3] The comparison with related methods is not a comparison with the original published works. The paper states that it reimplemented [3] as a one-class SVM and [7] as an HMM, whereas the original [3] used a binary SVM with a temporal trigger and supervised training. The abstract's claim of outperforming 'some recent works' is therefore an overstatement if these baselines are not the published methods. Please either reimplement the original methods or explicitly frame the results as 'our method vs. our reimplemented one-class/HMM baselines', and temper the conclusion accordingly.

minor comments (5)

[Abstract and Section 3.1] The abstract describes the dataset as 'nearly one hundred thousand skeletons'; this refers to the number of frames, not the number of independent subjects (nine), which is the sample size that limits the conclusions. Please rephrase to avoid overstatement.
[Equation (7)] The notation e^{-1}_k is confusing; writing 1/e_k would be clearer and would avoid confusion with matrix inversion.
[Section 2.1] The normalization step ('the data range along each axis is normalized by scaling it into [0, 1]') does not state whether the scaling is per sequence or computed globally over the training set. This affects whether subject-specific distance information is removed and should be clarified for reproducibility.
[Section 3.2 and Figure 6] The t-SNE visualizations of LSTM weights are difficult to quantify, and the statement that dropout 'seemed to improve' the mapping is not supported numerically. A comparison of weight statistics (e.g., mean and variance of gate weights) before and after dropout would be more convincing.
[Section 3.2] The manuscript does not report hyperparameters such as the optimizer, learning rate, weight initialization, or mini-batch size. Please include them for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reconstruction-error gait index is an empirical quantity, the combination weights use only training errors, and the self-citations are non-load-bearing baselines.

full rationale

The paper's central object is an empirical reconstruction error: Section 2.3 defines the gait index as the MSE between an input skeleton sequence and the autoencoder output, and Section 3.2 applies that definition as an abnormality score after training only on normal gait sequences from five subjects. No equation or fitted parameter is defined in terms of the final test index. The weighted combination in Eq. 7 computes weights from each model's training-set MSE, not from test labels or from the final per-sequence index, so it is not a fitted-input-called-prediction case. The statement that abnormal gait 'is expected to provide a bad reconstruction' is an unvalidated generalization about the method, not a circular derivation; it concerns external validity with four test subjects, which is a correctness risk, not a circularity risk. Citations to the authors' own prior works [7] and [8] appear only as reimplemented comparison baselines in Table 2 and as a prior depth-camera study, respectively; neither supplies the method's architecture, its uniqueness, or its empirical claim. No self-citation is load-bearing, and no ansatz is smuggled in through citation. The LSTM autoencoder design is standard and independently applicable. Therefore the derivation chain is self-contained and the appropriate circularity score is 0.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central claim depends on a cluster of domain assumptions about the relationship between reconstruction error and gait abnormality, the sufficiency of the joint selection, and the representativeness of the small dataset. The only standard background is the LSTM formulation. No new physical or conceptual entities are introduced.

free parameters (4)

hidden units per LSTM = 256
Chosen empirically without stated ablation (Section 2.2).
sequence length T = 12
Chosen for training sub-sequences with overlap 6 (Section 3.2).
dropout retention probability = 0.5
Chosen for the input dropout layer (Section 3.2).
training epochs = 100
Fixed for all three autoencoders (Section 3.2).

assumptions (6)

domain assumption Reconstruction error from a normal-only autoencoder is higher for abnormal gaits in unseen subjects.
Invoked in Section 3.2 to define the index; not independently verified.
domain assumption The 17 selected joints (discarding neck, mid-spine, and hand clusters) retain sufficient information for gait analysis.
Justified by intuition in Section 2.1, no quantitative comparison.
domain assumption Training separate autoencoders per spatial axis improves the gait index.
Stated as an expectation in Section 2.3.
domain assumption The 9-subject treadmill dataset is representative enough to evaluate gait index generalization.
Dataset described in Section 3.1; no external validation.
domain assumption The reimplementations of HMM [7] and one-class SVM [3] faithfully represent the original methods.
Needed for the comparison in Section 3.3; original code not used.
standard math LSTM update equations with peephole connections (Eq. 1-6) are correct and standard.
Standard RNN formulation used without proof.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Skeleton-based Gait Index Estimation with LSTMs." pith.science (2026). https://pith.science/paper/ERDNKKVH

@misc{pith2026190807416,
  author       = {Pith},
  title        = {Pith review of: Skeleton-based Gait Index Estimation with LSTMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ERDNKKVH}},
  note         = {Machine review of arXiv:1908.07416}
}

read the original abstract

In this paper, we propose a method that estimates a gait index for a sequence of skeletons. Our system is a stack of an encoder and a decoder that are formed by Long Short-Term Memories (LSTMs). In the encoding stage, the characteristics of an input are automatically determined and are compressed into a latent space. The decoding stage then attempts to reconstruct the input according to such intermediate representation. The reconstruction error is thus considered as a weak gait index. By combining such weak indices over a long-time movement, our system can provide a good estimation for the gait index. Our experiments on a large dataset (nearly one hundred thousand skeletons) showed that the index given by the proposed method outperformed some recent works on gait analysis.

Figures

Figures reproduced from arXiv: 1908.07416 by the authors.

**Figure 2.** The LSTM architecture with peephole connections. The [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Our auto-encoder that uses two LSTMs for the encoding and decoding stages. An output [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: The overlapping split of each normal gait sequence in the training set. The term [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Training loss of the three auto-encoders. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 6.** Figure 6: Visualizing distributions of LSTM weights that are related to input connections using [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 15 canonical work pages

[3]

Automatic detection of abnormal gait

Christian Bauckhage, John K Tsotsos, and Frank E Bunn. Automatic detection of abnormal gait. Image and Vision Computing, 27(1):108–115, 2009. 7 Published as a conference paper at 2018 IEEE/ACIS 17th Int. Conf. on Computer and Information Science. The ﬁnal version is available at: https://doi.org/10.1109/ICIS.2018.8466522

work page arXiv 2009
[7]

Skeleton-based abnormal gait detection

Trong-Nguyen Nguyen, Huu-Hung Huynh, and Jean Meunier. Skeleton-based abnormal gait detection. Sensors, 16(11):1792, 2016. ISSN 1424-8220. doi: 10.3390/s16111792. URL http://www.mdpi.com/1424-8220/16/11/1792

work page doi:10.3390/s16111792 2016
[1]

Andriluka, S

M. Andriluka, S. Roth, and B. Schiele. Pictorial structures revisited: People detection and articulated pose estimation. In 2009 IEEE Conference on Computer Vision and Pattern Recog- nition, pp. 1014–1021, June 2009. doi: 10.1109/CVPR.2009.5206754

work page arXiv 2009
[2]

Bauckhage, J

C. Bauckhage, J. K. Tsotsos, and F. E. Bunn. Detecting abnormal gait. In The 2nd Canadian Conference on Computer and Robot Vision (CRV’05), pp. 282–288, May 2005. doi: 10.1109/ CRV .2005.32

work page 2005
[4]

Gers, Nicol N

Felix A. Gers, Nicol N. Schraudolph, and Jürgen Schmidhuber. Learning precise tim- ing with lstm recurrent networks. J. Mach. Learn. Res. , 3:115–143, March 2003. ISSN 1532-4435. doi: 10.1162/153244303768966139. URL https://doi.org/10.1162/ 153244303768966139

work page doi:10.1162/153244303768966139 2003
[5]

Graves, A

A. Graves, A. r. Mohamed, and G. Hinton. Speech recognition with deep recurrent neural networks. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 6645–6649, May 2013. doi: 10.1109/ICASSP.2013.6638947

work page arXiv 2013
[6]

Visualizing data using t-sne.Journal of machine learning research, 9(Nov):2579–2605, 2008

Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne.Journal of machine learning research, 9(Nov):2579–2605, 2008

2008
[8]

Assessment of gait normal- ity using a depth camera and mirrors

Trong-Nguyen Nguyen, Huu-Hung Huynh, and Jean Meunier. Assessment of gait normal- ity using a depth camera and mirrors. In 2018 IEEE EMBS International Conference on Biomedical Health Informatics (BHI) , pp. 37–41, Las Vegas, NV , USA, March 2018. doi: 10.1109/BHI.2018.8333364

work page arXiv 2018

Show all 17 references

[9]

Online quality assessment of human movement from skeleton data

Adeline Paiement, Lili Tao, Sion Hannuna, Massimo Camplani, Dima Damen, and Majid Mirmehdi. Online quality assessment of human movement from skeleton data. In Proceedings of British Machine Vision Conference, 2014

2014
[10]

Depth energy image for gait symmetry quantiﬁcation

Caroline Rougier, Edouard Auvinet, Jean Meunier, Max Mignotte, and Jacques A De Guise. Depth energy image for gait symmetry quantiﬁcation. In 2011 Annual International Confer- ence of the IEEE Engineering in Medicine and Biology Society , pp. 5136–5139. IEEE, 2011

2011
[11]

Senior, and Françoise Beaufays

Hasim Sak, Andrew W. Senior, and Françoise Beaufays. Long short-term memory recur- rent neural network architectures for large scale acoustic modeling. In INTERSPEECH 2014, 15th Annual Conference of the International Speech Communication Association, Singapore, September 14-18...

2014
[12]

Shotton, A

J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, and A. Blake. Real-time human pose recognition in parts from single depth images. In CVPR 2011, pp. 1297–1304, June 2011. doi: 10.1109/CVPR.2011.5995316

2011
[13]

Dropout: A simple way to prevent neural networks from overﬁtting

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhut- dinov. Dropout: A simple way to prevent neural networks from overﬁtting. Journal of Ma- chine Learning Research, 15:1929–1958, 2014. URL http://jmlr.org/papers/v15/ srivastava14a.html

1929
[14]

Unsupervised learning of video representations using lstms

Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using lstms. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Re...

2015
[15]

Lstm neural networks for language modeling

Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. Lstm neural networks for language modeling. In Thirteenth Annual Conference of the International Speech Communication Association, pp. 194–197, 2012. URL http://www.isca-speech.org/archive/ interspeech_2012/i12_0194.html

2012
[16]

A comparative study of pose representation and dynamics modelling for online motion quality assessment

Lili Tao, Adeline Paiement, Dima Damen, Majid Mirmehdi, Sion Hannuna, Massimo Cam- plani, Tilo Burghardt, and Ian Craddock. A comparative study of pose representation and dynamics modelling for online motion quality assessment. Computer Vision and Image Un- derstanding, 148:13...

2016
[17]

Recurrent neural network regularization

Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329, 2014. 8

2014 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [3]

Automatic detection of abnormal gait

Christian Bauckhage, John K Tsotsos, and Frank E Bunn. Automatic detection of abnormal gait. Image and Vision Computing, 27(1):108–115, 2009. 7 Published as a conference paper at 2018 IEEE/ACIS 17th Int. Conf. on Computer and Information Science. The ﬁnal version is available at: https://doi.org/10.1109/ICIS.2018.8466522

work page arXiv 2009

[2] [7]

Skeleton-based abnormal gait detection

Trong-Nguyen Nguyen, Huu-Hung Huynh, and Jean Meunier. Skeleton-based abnormal gait detection. Sensors, 16(11):1792, 2016. ISSN 1424-8220. doi: 10.3390/s16111792. URL http://www.mdpi.com/1424-8220/16/11/1792

work page doi:10.3390/s16111792 2016

[3] [1]

Andriluka, S

M. Andriluka, S. Roth, and B. Schiele. Pictorial structures revisited: People detection and articulated pose estimation. In 2009 IEEE Conference on Computer Vision and Pattern Recog- nition, pp. 1014–1021, June 2009. doi: 10.1109/CVPR.2009.5206754

work page arXiv 2009

[4] [2]

Bauckhage, J

C. Bauckhage, J. K. Tsotsos, and F. E. Bunn. Detecting abnormal gait. In The 2nd Canadian Conference on Computer and Robot Vision (CRV’05), pp. 282–288, May 2005. doi: 10.1109/ CRV .2005.32

work page 2005

[5] [4]

Gers, Nicol N

Felix A. Gers, Nicol N. Schraudolph, and Jürgen Schmidhuber. Learning precise tim- ing with lstm recurrent networks. J. Mach. Learn. Res. , 3:115–143, March 2003. ISSN 1532-4435. doi: 10.1162/153244303768966139. URL https://doi.org/10.1162/ 153244303768966139

work page doi:10.1162/153244303768966139 2003

[6] [5]

Graves, A

A. Graves, A. r. Mohamed, and G. Hinton. Speech recognition with deep recurrent neural networks. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 6645–6649, May 2013. doi: 10.1109/ICASSP.2013.6638947

work page arXiv 2013

[7] [6]

Visualizing data using t-sne.Journal of machine learning research, 9(Nov):2579–2605, 2008

Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne.Journal of machine learning research, 9(Nov):2579–2605, 2008

2008

[8] [8]

Assessment of gait normal- ity using a depth camera and mirrors

Trong-Nguyen Nguyen, Huu-Hung Huynh, and Jean Meunier. Assessment of gait normal- ity using a depth camera and mirrors. In 2018 IEEE EMBS International Conference on Biomedical Health Informatics (BHI) , pp. 37–41, Las Vegas, NV , USA, March 2018. doi: 10.1109/BHI.2018.8333364

work page arXiv 2018