REVIEW 3 major objections 5 minor 34 references
Sifting through the haystack -- efficiently finding rare animal behaviors in large-scale datasets
T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Rare animal behaviors can be found by anomaly scores, not random search
desk verdict Useful applied pipeline for rare-behavior labeling, but the central random-sampling comparison is confounded by a doubled training-set size. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the anomaly-score split. The STG-NF model (spatio-temporal graph normalizing flow) maps graph sequences of animal keypoints—or of planar accelerations—to a latent Gaussian via invertible transformations, and its negative log-likelihood serves as the anomaly score for each clip. The pipeline draws normal samples from the center of the anomaly-score distribution, pseudo-labels them without review, and draws abnormal samples from the high-score tail for human review; this two-pool split is what turns an unsupervised ranker into a training-set builder. The downstream classifier is a shallow ST-GCN (spatio-temporal graph convolutional network), and the paper uses a sliding window of $f=8$ frames to score long pose sequences.
What would settle it
On a labeled dataset with known rare behaviors, compute the precision of the highest-scoring anomaly tail for the true rare class; if that precision equals the base rate of the rare class, then the anomaly-score ranking carries no information and the pipeline cannot beat random labeling. The FishLarvae1 eye-coordinate case is a concrete place to run this check.
Extended reading notes
Core claim
The paper's central claim is that anomaly scores from a spatio-temporal graph normalizing flow carry enough information about animal motion to organize human annotation. For an unlabeled dataset of clipped pose or acceleration sequences, the model assigns each clip a likelihood-based anomaly score; samples near the mean of the score distribution are pseudo-labeled normal, while the high-scoring tail is sent to a human reviewer who keeps the true rare behaviors. A shallow ST-GCN classifier trained on this mix is claimed to outperform random sampling at the same labeling effort, with the largest advantage under high rarity and with performance that stays nearly constant as rarity increases. The paper demonstrates this on a synthetic kinematic dataset and on three published biological datasets, including one accelerometry dataset, and reports that the pipeline still halves annotation effort when the behavior is not rare.
Load-bearing premise
The load-bearing premise is that 'unusual' as measured by the anomaly detector lines up with 'rare and worth finding,' and that clips near the average score can be trusted as normal without review; the FishLarvae1 case, where the detector missed a fine-grained difference and pipeline AuPRC dropped to 0.27, shows where that premise gives way.
Editorial extensions
If this is right
- At a fixed labeling budget, the pipeline yields a training set of $N_{\mathrm{reviewed}}$ human-reviewed anomalies plus $N_{\mathrm{pseudo}}$ pseudo-labeled normals, roughly twice the size of what random sampling yields for the same review effort.
- For rare behaviors at or below 1% of the data, classifier performance stays nearly flat as rarity increases, while random sampling loses roughly 0.17 AuPRC for every order-of-magnitude increase in rarity.
- Starting from unlabeled pose or acceleration data, a researcher needs no pre-existing rare examples and no assumption about the number or type of rare behavior classes.
- The same graph representation transfers from pose keypoints to tri-axial accelerometry by constructing planar two-channel acceleration graphs, so the pipeline is not tied to one recording modality.
- Even when the target behavior is common enough that random sampling performs as well, the anomaly-guided approach still halves the annotation effort.
Reading between the lines
- A testable extension is to insert a nuisance-anomaly filter: the PoseR cleaning step shows the anomaly detector is especially sensitive to skeleton-estimation flicker, so separating tracking artifacts from behavioral anomalies could sharpen the tail for true rare behaviors.
- The FishLarvae1 result suggests an explicit benchmark where the rare behavior is encoded in a small subset of keypoints (the eye coordinates); a synthetic version of that setting would test whether local keypoint attention is needed for the anomaly score to remain a useful rarity signal.
- Because pseudo-labeling the score mean rules out hard positives, an adaptive loop that re-ranks the dataset with the first trained classifier and proactively reviews near-boundary samples could extend the pipeline's usefulness when the behavior is less rare.
- The synthetic dataset's simple kinematic rule (frequency/amplitude swap) makes it a reusable controlled testbed for comparing any future rare-behavior sampling scheme.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a pipeline for efficiently discovering rare animal behaviors in large unlabeled pose/acceleration datasets. The pipeline trains an unsupervised graph-based anomaly detector (STG-NF) on the full unlabeled data, uses the anomaly scores to direct human review toward the high-score tail, pseudo-labels samples near the score mean as normal, and trains an ST-GCN binary classifier on the resulting dataset. The authors evaluate the method on a synthetic dataset and on three real biological datasets (FishLarvae1, PoseR, Meerkat) across induced rarity levels, reporting that it consistently outperforms random sampling by an average of roughly 64-70% in AuPRC. The paper includes publicly available code, synthetic data, and raw experimental outputs, and it provides a statistical analysis of the results. The central comparison, however, is confounded because the proposed method trains on twice as many samples as the random-sampling baseline for the same human-review budget.
Significance. If the reported improvement over random sampling were established, the pipeline would be a practically valuable tool for ethologists and ecologists who need to train rare-behavior classifiers without existing labeled examples. The paper's strengths include its focus on a realistic and underserved problem, the use of multiple real-world datasets, the inclusion of a controlled synthetic experiment, and the release of code and data. The methodological confound described below, however, prevents the current results from supporting the paper's central claim. The contribution is therefore not yet established, though the general approach is plausible and worth pursuing.
major comments (3)
- [Section 3.2] The comparison to random sampling is confounded. The paper explicitly states in Section 3.2 that for a labeling budget of N_reviewed, the proposed method trains on N_reviewed reviewed tail samples plus N_reviewed pseudo-labeled center samples, yielding a training set of size 2*N_reviewed, while random sampling trains on only N_reviewed samples. All reported improvements (the abstract's 70%, Section 5.1's 64.82% +/- 2.36, Table 1, and Figures 5-6) therefore compare classifiers trained on datasets of different sizes and with different labeling protocols. The advantage could be driven entirely by the extra pseudo-labeled normal samples or by the doubled training-set size, rather than by the anomaly-score-guided selection of which samples to review. To support the central claim, please add a control experiment in which random sampling is augmented with the same pseudo-labeling procedure (e.g., N_reviewed randomly sampled and reviewed samples plus N_reviewed pseudo-labeled samples from the anomaly-score center, or from the full dataset), trained under identical conditions. Without such a control, the contribution of the anomaly-detection component is not established.
- [Section 5.2] The result on FishLarvae1 shows a small and likely non-significant improvement (AuPRC 0.27 +/- 0.067 vs. 0.21 +/- 0.022 for random sampling, Table 1). The paper attributes this to the anomaly detector failing on fine-grained eye-coordinate differences (Section 6), which is an honest limitation. However, this weakens the blanket claim that the method 'consistently outperformed traditional random sampling.' Please report per-dataset effect sizes with confidence intervals or formal significance tests, and temper the abstract and conclusion accordingly, noting that the advantage is dataset-dependent and may be minimal when the anomaly detector does not align with the behavior of interest.
- [Section 5.1] The slope comparison in Section 5.1 (random method slope = 0.17 +/- 0.05; proposed method slope = -0.04 +/- 0.06) is also subject to the same confound. Random sampling's stronger dependence on rarity could be a consequence of its smaller effective training set rather than of the sampling strategy itself. Recompute these slopes in the proposed control setup (with equal training-set size and pseudo-labeling for both methods) before drawing conclusions about rarity robustness.
minor comments (5)
- [Abstract] The abstract states an 'average improvement of 70%' while Section 5.1 reports '64.82% +/- 2.36'; please reconcile these numbers.
- [Section 3.1] The sliding window size f=8 is mentioned for segmentation, but the FishLarvae1 dataset contains variable-length clips; please clarify how variable-length sequences are handled during training and inference.
- [Section 4.3] The paper states that STG-NF models were 'trained for 4 epochs as we found it was enough for models to converge,' but also that no validation set was used (Section S1.2). Please describe the convergence check and how hyperparameters were selected without a validation set.
- [Figure 3] The term 'behaviorSD' is used without definition in the caption; please define it or refer to the text where it is introduced.
- [Supplementary S1.2.3] The Meerkat dataset is split randomly rather than by individual, which may leak individual identity across train and test; this limitation is acknowledged in the supplementary but should also be stated in the main text.
Circularity Check
No significant circularity: the pipeline is an empirical evaluation with held-out labels, and the sole self-citation (STG-NF) is an independently published, code-released building block.
full rationale
This paper is an empirical evaluation of a labeling pipeline; it does not derive a formal result or prediction from fitted inputs. The central comparison (anomaly-tail review plus pseudo-labeled normal sampling versus random review) is measured by held-out AuPRC on fixed test sets with ground-truth labels. The claim of 70% improvement is a reported mean over experiments, not a quantity forced by construction. The only self-referential element is the reliance on STG-NF [11], co-authored by two of the present authors, as the anomaly scorer. That model is an independently published ICCV 2023 method with public code, and the present paper uses it as a fixed building block rather than fitting it to the outcome; under the review rules this is real evidence and does not raise the circularity score. The paper explicitly discloses that its training set is twice the size of the random baseline (N_reviewed + N_pseudolabeled = 2*N_reviewed), which raises a legitimate confound about whether the gain comes from extra pseudo-labeled normal samples rather than anomaly-guided review, but that is a methodological or correctness concern, not circularity: no tested quantity is defined in terms of the quantity it is claimed to predict, and no uniqueness theorem or ansatz is smuggled in via self-citation. Therefore the honest finding is no significant circularity.
Assumptions & free parameters
free parameters (1)
- Sampling pool thresholds =
mean +/- 0.25*sigma; mean + 0.05*sigma; mean + 2*sigma
assumptions (3)
- domain assumption The anomaly detector's density estimate is accurate enough that scores rank unusual behaviors above common ones.
- domain assumption The behavior of interest is a kinematic or motion pattern, not an appearance-based one.
- domain assumption Pseudo-labeling center samples as normal without human review is safe.
Cite this review
Pith. "Pith review of Sifting through the haystack -- efficiently finding rare animal behaviors in large-scale datasets." pith.science (2026). https://pith.science/paper/LZN2FNWC
@misc{pith2026241203452,
author = {Pith},
title = {Pith review of: Sifting through the haystack -- efficiently finding rare animal behaviors in large-scale datasets},
year = {2026},
howpublished = {\url{https://pith.science/paper/LZN2FNWC}},
note = {Machine review of arXiv:2412.03452}
}
read the original abstract
In the study of animal behavior, researchers often record long continuous videos, accumulating into large-scale datasets. However, the behaviors of interest are often rare compared to routine behaviors. This incurs a heavy cost on manual annotation, forcing users to sift through many samples before finding their needles. We propose a pipeline to efficiently sample rare behaviors from large datasets, enabling the creation of training datasets for rare behavior classifiers. Our method only needs an unlabeled animal pose or acceleration dataset as input and makes no assumptions regarding the type, number, or characteristics of the rare behaviors. Our pipeline is based on a recent graph-based anomaly detection model for human behavior, which we apply to this new data domain. It leverages anomaly scores to automatically label normal samples while directing human annotation efforts toward anomalies. In research data, anomalies may come from many different sources (e.g., signal noise versus true rare instances). Hence, the entire labeling budget is focused on the abnormal classes, letting the user review and label samples according to their needs. We tested our approach on three datasets of freely-moving animals, acquired in the laboratory and the field. We found that graph-based models are particularly useful when studying motion-based behaviors in animals, yielding good results while using a small labeling budget. Our method consistently outperformed traditional random sampling, offering an average improvement of 70% in performance and creating datasets even when the behavior of interest was only 0.02% of the data. Even when the performance gain was minor (e.g., when the behavior is not rare), our method still reduced the annotation effort by half.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...
-
[2]
Yahaya Idris Abubakar, Alice Othmani, Patrick Siarry, and Aznul Qalid Md Sabri. A systematic review of rare events detection across modalities using machine learning and deep learning. IEEE Access , 2024
work page 2024
-
[3]
Ubnormal: New benchmark for supervised open-set video anomaly detection
Andra Acsintoae, Andrei Florescu, Mariana-Iuliana Georgescu, Tudor Mare, Paul Sumedrea, Radu Tudor Ionescu, Fahad Shahbaz Khan, and Mubarak Shah. Ubnormal: New benchmark for supervised open-set video anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 20143--20153, 2022
work page 2022
-
[4]
Assessing the determinants of larval fish strike rates using computer vision
Shir Bar, Liraz Levy, Shai Avidan, and Roi Holzman. Assessing the determinants of larval fish strike rates using computer vision. Ecological Informatics , 77:102195, 2023
work page 2023
-
[5]
Mvtec ad--a comprehensive real-world dataset for unsupervised anomaly detection
Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtec ad--a comprehensive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9592--9600, 2019
work page 2019
-
[6]
On model evaluation under non-constant class imbalance
Jan Brabec, Tom \'a s Kom \'a rek, Vojt e ch Franc, and Luk \'a s Machlica. On model evaluation under non-constant class imbalance. In International Conference on Computational Science , pages 74--87. Springer, 2020
work page 2020
-
[7]
A novel biomechanical approach for animal behaviour recognition using accelerometers
Pritish Chakravarty, Gabriele Cozzi, Arpat Ozgul, and Kamiar Aminian. A novel biomechanical approach for animal behaviour recognition using accelerometers. Methods in Ecology and Evolution , 10(6):802--814, 2019
work page 2019
-
[8]
The philosophy of outliers: Reintegrating rare events into biological science
Chelsea N Cook, Angela R Freeman, James C Liao, and Lisa A Mangiamele. The philosophy of outliers: Reintegrating rare events into biological science. Integrative and Comparative Biology , 61(6):2191--2198, 2021
work page 2021
Show all 34 references
-
[9]
Nice: Non-linear independent components estimation
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516 , 2014
2014 arXiv
-
[10]
Improving the precision of estimates of the frequency of rare events
Philip M Dixon, Aaron M Ellison, and Nicholas J Gotelli. Improving the precision of estimates of the frequency of rare events. Ecology , 86(5):1114--1123, 2005
2005
-
[11]
Deep autoencoder-based behavioral pattern recognition outperforms standard statistical methods in high-dimensional zebrafish studies
Adrian J Green, Lisa Truong, Preethi Thunga, Connor Leong, Melody Hancock, Robyn L Tanguay, and David M Reif. Deep autoencoder-based behavioral pattern recognition outperforms standard statistical methods in high-dimensional zebrafish studies. bioRxiv , 2023
2023
-
[12]
Normalizing flows for human pose anomaly detection
Or Hirschorn and Shai Avidan. Normalizing flows for human pose anomaly detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 13545--13554, 2023
2023
-
[13]
Surface defect saliency of magnetic tile
Yibin Huang, Congying Qiu, and Kui Yuan. Surface defect saliency of magnetic tile. The Visual Computer , 36(1):85--96, 2020
2020
-
[14]
Probabilistic models of larval zebrafish behavior reveal structure on many scales
Robert Evan Johnson, Scott Linderman, Thomas Panier, Caroline Lei Wee, Erin Song, Kristian Joseph Herrera, Andrew Miller, and Florian Engert. Probabilistic models of larval zebrafish behavior reveal structure on many scales. Current Biology , 30(1):70--82, 2020
2020
-
[15]
Russell V. Lenth. emmeans: Estimated Marginal Means, aka Least-Squares Means , 2024. R package version 1.10.5-0900001, https://rvlenth.github.io/emmeans/
2024
-
[16]
Coarse-to-fine animal pose and shape estimation
Chen Li and Gim Hee Lee. Coarse-to-fine animal pose and shape estimation. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 11757--11768. Curran Associates, Inc., 2021
2021
-
[17]
Future frame prediction for anomaly detection--a new baseline
Wen Liu, Weixin Luo, Dongze Lian, and Shenghua Gao. Future frame prediction for anomaly detection--a new baseline. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6536--6545, 2018
2018
-
[18]
Unsupervised quantification of naturalistic animal behaviors for gaining insight into the brain
Michael H McCullough and Geoffrey J Goodhill. Unsupervised quantification of naturalistic animal behaviors for gaining insight into the brain. Current Opinion in Neurobiology , 70:89--100, 2021
2021
-
[19]
Mark, Deva Ramanan, and Kayvon Fatahalian
Ravi Teja Mullapudi, Fait Poms, William R. Mark, Deva Ramanan, and Kayvon Fatahalian. Learning rare category classifiers on a tight labeling budget. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 8423--8432, October 2021
2021
-
[20]
Poser-a deep learning toolbox for decoding animal behavior
Pierce N Mullen, Beatrice Bowlby, Holly C Armstrong, and Maarten F Zwart. Poser-a deep learning toolbox for decoding animal behavior. BioRxiv , pages 2023--04, 2023
2023
-
[21]
Simple behavioral analysis (simba)--an open source toolkit for computer classification of complex social behaviors in experimental animals
Simon RO Nilsson, Nastacia L Goodwin, Jia Jie Choong, Sophia Hwang, Hayden R Wright, Zane C Norville, Xiaoyu Tong, Dayu Lin, Brandon S Bentzley, Neir Eshel, et al. Simple behavioral analysis (simba)--an open source toolkit for computer classification of complex social behavior...
2020
-
[22]
Pytorch: An imperative style, high-performance deep learning library
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu ...
2019
-
[23]
Active learning for anomaly and rare-category detection
Dan Pelleg and Andrew Moore. Active learning for anomaly and rare-category detection. Advances in neural information processing systems , 17, 2004
2004
-
[24]
The precision-recall plot is more informative than the roc plot when evaluating binary classifiers on imbalanced datasets
Takaya Saito and Marc Rehmsmeier. The precision-recall plot is more informative than the roc plot when evaluating binary classifiers on imbalanced datasets. PloS one , 10(3):e0118432, 2015
2015
-
[25]
Precrec: fast and accurate precision--recall and roc curve calculations in r
Takaya Saito and Marc Rehmsmeier. Precrec: fast and accurate precision--recall and roc curve calculations in r. Bioinformatics , 33(1):145--147, 2017
2017
-
[26]
Real-world anomaly detection in surveillance videos
Waqas Sultani, Chen Chen, and Mubarak Shah. Real-world anomaly detection in surveillance videos. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6479--6488, 2018
2018
-
[27]
Automatic recording of rare behaviors of wild animals using video bio-loggers with on-board light-weight outlier detector
Kei Tanigaki, Ryoma Otsuka, Aiyi Li, Yota Hatano, Yuanzhou Wei, Shiho Koyama, Ken Yoda, and Takuya Maekawa. Automatic recording of rare behaviors of wild animals using video bio-loggers with on-board light-weight outlier detector. PNAS nexus , 3(1):pgad447, 2024
2024
-
[28]
Perspectives in machine learning for wildlife conservation
Devis Tuia, Benjamin Kellenberger, Sara Beery, Blair R Costelloe, Silvia Zuffi, Benjamin Risse, Alexander Mathis, Mackenzie W Mathis, Frank van Langevelde, Tilo Burghardt, et al. Perspectives in machine learning for wildlife conservation. Nature communications , 13(1):1--15, 2022
2022
-
[29]
Keypoint-moseq: parsing behavior by linking point tracking to pose dynamics
Caleb Weinreb, Jonah E Pearl, Sherry Lin, Mohammed Abdal Monium Osman, Libby Zhang, Sidharth Annapragada, Eli Conlin, Red Hoffmann, Sofia Makowska, Winthrop F Gillis, et al. Keypoint-moseq: parsing behavior by linking point tracking to pose dynamics. Nature Methods , 21(7):132...
2024
-
[30]
Daniel Salzman, Dora Angelaki, Andr\' e s Bendesky, The International Brain Laboratory The International Brain Laboratory, John P Cunningham, and Liam Paninski
Anqi Wu, Estefany Kelly Buchanan, Matthew Whiteway, Michael Schartner, Guido Meijer, Jean-Paul Noel, Erica Rodriguez, Claire Everett, Amy Norovich, Evan Schaffer, Neeli Mishra, C. Daniel Salzman, Dora Angelaki, Andr\' e s Bendesky, The International Brain Laboratory The Intern...
2020
-
[31]
Spatial temporal graph convolutional networks for skeleton-based action recognition
Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton-based action recognition. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018
2018
-
[32]
Graph regularized flow attention network for video animal counting from drones
Pengfei Zhu, Tao Peng, Dawei Du, Hongtao Yu, Libo Zhang, and Qinghua Hu. Graph regularized flow attention network for video animal counting from drones. IEEE Transactions on Image Processing , 30:5339--5351, 2021
2021
-
[33]
Visualization of regression models using visreg
Patrick Breheny and Woodrow Burchett. Visualization of regression models using visreg. The R Journal , 9(2):56--71, 2017
2017
-
[34]
Russell V. Lenth. emmeans: Estimated Marginal Means, aka Least-Squares Means , 2024. R package version 1.10.2
2024
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.