Pith. sign in

REVIEW 5 major objections 5 minor 39 references

Agent-centric generative models—in which a network senses and acts from an animal's own reference frame—capture the distribution of social courtship behavior in groups of Drosophila, and the choice of egocentric representation is the main d

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-01 12:24 UTC pith:XNI5ZQ3R

load-bearing objection Solid infrastructure paper with a good library and honest evaluation, but the headline claim about capturing the behavior distribution outruns the paper's own discriminator and probe-transfer evidence. the 5 major comments →

arxiv 2607.19548 v1 pith:XNI5ZQ3R submitted 2026-07-21 cs.LG

Agent-Centric Animal Pose Forecasting

Eyrun Eyjolfsdottir , Kristin Branson This is my paper
classification cs.LG
keywords agent-centric behavior modelsegocentric representationgenerative pose forecastingDrosophila courtship behaviorautoregressive multi-agent rolloutoutput discretizationbehavior distribution evaluationlinear probes of internal representations
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes agent-centric generative modeling of animal behavior: instead of predicting world-frame trajectories, a transformer is given egocentric sensory features and asked to output egocentric movements, so that social interaction emerges from independently sensing agents rather than a centralized group model. The paper claims that, with a library of composable invertible operations to translate between world, egocentric, and machine-learning representations, this approach captures the distribution of social courtship behavior in groups of Drosophila—feature statistics, walking gait, chasing and wing extension, and condition differences. The authors also conclude, from systematic ablations, that the agent-centric representation itself, rather than architecture, is the main driver of accuracy, with output discretization necessary to avoid collapse to no movement. For a reader, the significance is a reusable recipe and toolset for building biologically constrained generative models that can be interrogated as hypotheses about how animals compute behavior.

Core claim

On the paper's own terms, the discovery is that an autoregressive transformer, constrained to receive only hand-crafted egocentric sensory approximations (a visual depth map of nearby flies, social and arena touch, proprioceptive pose) and to emit egocentric movement, learns to generate multi-agent Drosophila behavior whose feature distributions, gait patterns, chasing and wing-extension frequencies, and condition-dependent differences match real data at multiple timescales—and does so largely because of the agent-centric representation. Replacing it with world-frame keypoints degrades almost every metric, while removing handcrafted joint-angle parameterization barely matters; output discret

What carries the argument

The central mechanism is the Operation abstraction: composable, invertible transformations (global/local velocity, egocentric roll, z-scoring, discretization, sensory and pose encoders) that connect world-frame keypoints, egocentric sensory inputs, and ML-ready labels, chained together and stored as a data-flow graph. Because every forward operation has an inverse and the chain is recorded, autoregressive rollout can invert predicted egocentric movements back to world pose, update all agents, recompute each agent's sensory inputs for the next step, and cleanly swap representations—making the agent-centric comparison the paper's main experimental lever.

Load-bearing premise

The load-bearing premise is that the linear probes trained and validated only on real trajectories (§6.6, §E.3) remain accurate on simulated hidden states; if they do not transfer, the reported frequencies of courtship behaviors in simulated flies are unsupported.

What would settle it

A concrete check: take model rollouts, obtain ground-truth behavior labels independently (manual annotation or a rule-based detector like the paper's walk detector), train the same linear probes on simulated hidden states, and re-count behavior frequencies; a large drop in probe accuracy on simulated states would falsify the transfer assumption underlying the behavior-frequency results.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Behavioral fidelity is attributable to representation, not architecture: the agent-centric egocentric setting is the dominant factor, and the world-frame keypoint variant performs worst on nearly every metric, including probe accuracy.
  • Output discretization is a practical necessity for behavior cloning of stochastic, multi-modal behavior: without it, models collapse to predicting no movement and simulations freeze.
  • Forecasting alone serves as a representation-learning objective: linear probes on an intermediate transformer layer decode known behavior categories and fly-type conditions, so next-step prediction can act as unsupervised behavior representation learning.
  • Closed-loop social interaction can emerge without physics simulation: simulated flies approach, chase, avoid collisions, and remain in the arena purely through recomputed sensory feedback.
  • The same operation-based pipeline transfers to a new domain (synthetic rat navigation with boundary- and head-direction sensory inputs) by adding only a new sensory operation, offering a recipe for agent-centric generative models in other species.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Our inference: the probe-transfer caveat noted in §§6.6 and E.3 means the simulated courtship-behavior frequencies should be read as upper bounds on fidelity; an easy test is to retrain probes on simulated hidden states with labels from an independent detector or manual annotation.
  • Our inference: the paper's own failure mode with independent per-dimension sampling during rollout suggests joint quantization of full poses (e.g., vector-quantized tokens over the whole body) as the natural next step for long-horizon realism.
  • Our inference: if the egocentric-representation result generalizes beyond flies, it hints that world-frame joint prediction in multi-agent human pose forecasting may be part of why interactive accuracy suffers; agent-centric sensory gating could improve those systems too.
  • Our inference: since probes succeed with no behavior-label supervision during training, future work could mine hidden states for unannotated behavioral motifs—the paper names this goal as open but does not execute it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces an agent-centric framework for generative modeling of animal pose trajectories, along with a Python library (AnimalPoseForecasting) that composes and inverts transformations between world-frame, egocentric, sensory, and ML-friendly representations. The authors train transformer models on multi-fly Drosophila courtship data, compare several input/output representation variants, and evaluate them through qualitative videos, feature-distribution Wasserstein distances, real-vs-simulated discriminators, behavior-frequency comparisons, and linear probes. They also demonstrate transfer to a synthetic rat domain (RatInABox). The central claims are that the agent-centric models capture the distribution of social behavior in courting Drosophila and that agent-centric representations are the most important modeling factor.

Significance. If the claims hold, this is a valuable contribution: a reusable library for a non-trivial representational problem, a systematic ablation of representation choices under matched architecture, and a quantitative evaluation toolkit for generative behavior models. The composable/invertible operation abstraction is a genuine contribution, and the open-source release makes the infrastructure widely usable. The empirical study is anchored to real data rather than only model outputs, and the multi-scale evaluation (gait, motifs, long-timescale frequencies) is more thorough than typical in pose forecasting. The RatInABox adaptation supports the generality argument. However, the central scientific claim is currently stronger than the evidence supports, for the reasons detailed below.

major comments (5)
  1. [§6.6 and §E.3] The behavior-frequency evaluation for courting, chasing, and wing extension uses linear probes trained on real hidden states and then applies them to simulated hidden states. The text itself concedes these classifiers 'are only trained and validated on real trajectory data, and may not generalize well to simulated data' (§6.6), and §E.3 adds that the probe used differs per variant. Because these RE values are the main quantitative support for the claim that rollouts reproduce longer-timescale social behavior, the central claim requires either validation of probe transfer (e.g., simulated labels from a synthetic oracle, or a domain-shift calibration on held-out real frames) or a clearly qualified conclusion.
  2. [§6.5] The real-vs-simulated discriminator results are in direct tension with the abstract's claim that models 'capture the distribution of social behavior.' The velocity discriminator reaches 0.92 accuracy within the first four simulated frames and 0.97 overall (Fig. 6b), and the pose-based discriminator is 0.65 even at 64 frames. A simple MLP can thus almost perfectly separate real from simulated instantaneous movement. The paper acknowledges this in §6.5 but does not reconcile it with the central claim. Please either temper the abstract/conclusion or provide a quantitative argument for why near-perfect discriminability is consistent with 'capturing the distribution.'
  3. [§6.2 and §6.8] All comparisons between model variants are based on single training runs, with no repeated-seed error bars or significance tests. The attribution that agent-centric representations are 'most important' (KEYPOINTS worst on nearly every metric) and the conclusions about discretization, context length, and pose parameterization could be driven by run-to-run variance. Please report means and variances over at least 3–5 seeds for the key variants (REFERENCE, KEYPOINTS, NO DISCRETIZATION, SHORTER CONTEXT) or otherwise justify the stability of the reported differences.
  4. [§6.2 and §6.8] The KEYPOINTS variant changes both input and output representations (world-frame keypoints instead of egocentric sensory features and egocentric pose velocity). The conclusion in §6.8 that 'agent-centric representations enabled by our library is most important' is therefore supported only by a confounded ablation. To support the attribution, the authors should include intermediate variants (e.g., egocentric input with world-frame keypoint output, or world-frame input with egocentric output) to isolate the contribution of the input vs. output representation.
  5. [§D] According to §D, rollouts are seeded from training trajectories. This means the model's context window contains frames seen during training, which is non-standard for generative model evaluation and may yield optimistic results if the model has memorized these sequences. The paper should either evaluate rollouts from held-out validation/test prompts, or explain why training-set prompts are appropriate for the distributional claim.
minor comments (5)
  1. [§6.6] The parenthetical caveat is missing a closing parenthesis: 'trained and validated on real trajectory data (§ 6.7, and may not generalize well to simulated data.' Please fix.
  2. [Figure 6b] The axis labels 'Velocity discr. accuracy' and 'Dist. to prompt (fr. range)' are informal; please spell out 'Discriminator accuracy' and 'Distance to prompt (frame range)' for clarity.
  3. [Reference [33]] The author list for the FlyMABe2022 dataset reference appears malformed: 'Catherine Schretter Kristin Branson, Alice A. Robie' should be formatted consistently with the other references.
  4. [Affiliation] The affiliation line reads 'Ashburn, V A 20147'; the state abbreviation should be 'VA'.
  5. [§6.2] The description of the 'previous frame's output' as an input modality is slightly ambiguous; clarify whether this is the model's own output under teacher forcing or self-feeding at test time.

Circularity Check

0 steps flagged

No significant circularity: the central evaluation anchors to held-out real data and external behavior labels; the §6.6 probe-transfer caveat is a validity risk, not a circular reduction.

full rationale

The paper's core derivation chain is not circular. Models are trained to forecast next-frame egocentric movement from sensory inputs, and the main claims are evaluated by comparing simulated rollouts against real-data distributions (feature histograms, real-vs-sim discriminators, behavior frequencies from real labels, and linear probes on hidden states trained to external MABe labels). None of these target quantities is used to construct the training loss or the model outputs, so there is no equation-level reduction of a 'prediction' to a fitted input. The only mild self-referential elements are self-citations [11,12] for a sensory nonlinearity and evaluation criteria; these are design precedents, not load-bearing uniqueness or derivation steps. The §6.6/E.3 caveat that behavior-frequency classifiers are trained and validated only on real trajectory data and may not transfer to simulated hidden states is a real external-validity limitation of the frequency comparison, but it is not circularity: an independent probe applied to new inputs can fail without making the target definitionally equal to the training fit. Similarly, bin edges fit on training data are ordinary preprocessing, not a forced prediction of the evaluation histograms. Thus no circular step meets the required evidentiary standard.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central claim rests on domain assumptions about sensory sufficiency and pose completeness rather than on free mathematical postulates. The fitted discretization bins and hand-set sensory constants are the main input choices; they are data-driven or hand-picked, not derived from theory. No new physical entities are introduced.

free parameters (4)
  • Discretize bin edges (fly, per feature) = K=25 bins; edges optimized by coordinate descent on first-order Markov-model likelihood over training velocities
    A.3.1; determines all discrete output tokens and is therefore load-bearing for the model's multimodal sampling behavior.
  • Sensory ρ nonlinearity constants (vision: γ=0.6, c=(2R)^-γ; touch: γ=1.3, c=0.311) = hand-set
    A.1; these constants define the numeric sensory values that are the network's only observations of other flies and the arena; changing them changes the task.
  • Per-fly scale parameters (6 body-part lengths) = per-fly median over all frames
    A.2; needed to invert the Pose operation and reconstruct absolute keypoints from local pose predictions; identity-dependent.
  • RatInABox Discretize bin edges = equal-frequency quantile bins from 10,000 training episodes
    C.3; used for all three global-velocity outputs in the rat domain demonstration.
axioms (5)
  • domain assumption Each animal's behavior is generated by a stationary policy mapping its egocentric sensory observations to egocentric movements, and social behavior emerges from independent agents sensing/responding to each other.
    Introduced in §4; the entire agent-centric framework rests on this biological constraint being sufficient for behavior.
  • domain assumption The hand-crafted sensory features (72-bin visual depth map, arena ceiling height at 19 keypoints, 95 pairwise body-part distances) are a sufficient approximation of what a fly perceives.
    §4.1 and §A.1 define these features as the model's only inputs; if flies use information beyond these, the model cannot capture it.
  • domain assumption The 19 tracked keypoints and 29-feature hand-crafted pose decomposition are sufficient to represent behaviorally meaningful pose.
    §A.2 defines the Pose operation; the model's output space is limited to these features.
  • standard math Autoregressive factorization of the joint pose distribution p(x_1:T)=∏ p(x_t|x_<t) is a valid generative model for behavior.
    Used throughout §4.3 rollout; standard sequence-modeling assumption.
  • domain assumption Linear probes trained on real hidden states transfer to simulated hidden states for behavior-frequency counting.
    §6.6 and §E.3 use probes trained on real data to count behaviors in simulations, with an explicit caveat that they may not generalize.

reviewed 2026-08-01 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Agent-Centric Animal Pose Forecasting." pith.science (2026). https://pith.science/paper/XNI5ZQ3R

@misc{pith2026260719548,
  author       = {Pith},
  title        = {Pith review of: Agent-Centric Animal Pose Forecasting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XNI5ZQ3R}},
  note         = {Machine review of arXiv:2607.19548}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Understanding animal behavior at an algorithmic level -- what animals attend to, how they form internal models and plans, and how this maps to action -- remains a central challenge in neuroscience and ethology. Data-driven generative models offer a path toward this understanding. We introduce a framework for training agent-centric autoregressive models of animal behavior from tracked pose, applicable to single animals and to groups in which each agent senses and responds to its conspecifics. Our models input egocentric sensory observations and output egocentric movements, mirroring the biological constraint that animals observe and act on the world from their own reference frame. Social behavior emerges from agents independently sensing and responding to one another. This agent-centric formulation requires managing many parallel representations of the same data, along with ML-specific transformations like discretization. We release a general-purpose library focused on the composable sequences of operations that translate between these representations. We show that trained models capture the distribution of social behavior in groups of courting Drosophila, and our library includes quantitative tools for measuring fit. We demonstrate how the library supports systematic comparison across input and output representations and that it adapts straightforwardly to a new domain.

Figures

Figures reproduced from arXiv: 2607.19548 by Eyrun Eyjolfsdottir, Kristin Branson.

Figure 1
Figure 1. Figure 1: Our animal pose forecasting framework and library. a Unlike standard world-frame approaches, we model behavior in each agent’s egocentric frame. b Hand-designed sensory features summarize what each agent perceives. c A deep network predicts egocentric movement, paralleling the animal’s own sensory-to-motor mapping; agents coordinate through sensing rather than centralized control. d Our AnimalPoseForecasti… view at source ↗
Figure 2
Figure 2. Figure 2: (a) MABe 2022 fly behavior data video frame and trajectories. (b) RatInABox synthetic [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Left: 1000 simulations of 128 frames each, for two agents. The true trajectory of each fly is highlighted in white. Predicted trajectories are colored by which fly they end up near, or gray if they end up far from any fly. Right: Relative leg-tip positions for example walking bouts detected in real data and in data simulated from the REFERENCE model. to chase a fly, or whether to produce a courtship song. … view at source ↗
Figure 4
Figure 4. Figure 4: Relative leg-tip positions for example walking bouts for ground truth and for simulated data from all model variants. one another, with simulated flies approaching and chasing other flies. They stay within the arena and avoid colliding with other flies even though, unlike in physics simulators [29], physical collisions are not part of our simulation environment [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: a. distribution of three selected pose velocity features, real vs. simulated from each model variant, with Wasserstein distance reported across all velocity features. Thin vertical lines indicate the linear portion of the symlog transform. b. Same, for position/pose features, instead of velocity. motion, performs slightly worse than our REFERENCE model, while the NO HANDCRAFTED POSE model performs better. … view at source ↗
Figure 6
Figure 6. Figure 6: a. Real-vs-simulated discrimination accuracy using position and pose (top) and velocity (bottom) features over the first 64 (top) and 512 (bottom) simulated frames. b. Discrimination accuracy for data simulated with the REFERENCE model, using velocity features at increasing distance from the prompt. Discretized Global Global 0.58 Global + left_wing 0.61 Wing angles 0.63 Back leg angles 0.63 Discretized fea… view at source ↗
Figure 7
Figure 7. Figure 7: a. Real-vs-simulated discrimination accuracy using individual per-frame pose-velocity features. b. Discrimination accuracy using subsets of pose-velocity features. c. Illustration of example features: global velocity features, selected static pose features. error (back-leg tip angle velocities): considered independently they reach 0.59, and jointly 0.63, a far smaller gap than for the wing angles. 6.6 Beha… view at source ↗
Figure 8
Figure 8. Figure 8: a) Evolution of a) selected feature means and b) behavior class frequency for increasing distance to prompt [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Left: binary classification performance on held-out data, for all model variants. MABe labels are split into animal-type labels and behavior labels. Right: frequency of these labels in real vs. simulated data, with red indicating higher frequency in simulated data and blue indicating lower. LOCAL POSE match "wing extension" frequency well for the first 64 frames, but the frequency increases greatly as dist… view at source ↗
Figure 10
Figure 10. Figure 10: Quantitative comparison of model variants. Red indicates performance worse than the REFERENCE model, blue indicates performance better. For all metrics apart from the probe score, lower is better. from the prompt ( [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: RatInABox results. Top: Trajectories generated from the "real" RatInABox model from the same initial conditions. One trajectory is highlighted in black, all others are colored by time. Bottom: Rollouts generated from our trained forecasting model. The black trajectory is the real trajectory highlighted above, colored trajectories are simulations from the trained model. behavior of groups of courting Droso… view at source ↗
Figure 12
Figure 12. Figure 12: Data flow modifications of variants relative to Reference model. RatInABox RatInABoxSensory GlobalVelocity [PITH_FULL_IMAGE:figures/full_fig_p020_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Data flow for RatInABox dataset. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

39 extracted references · 19 linked inside Pith

  1. [1]

    Jennifer J. Sun, Markus Marks, Andrew Ulmer, Dipam Chakraborty, Brian Geuther, Edward Hayes, Heng Jia, Vivek Kumar, Sebastian Oleszko, Zachary Partridge, Milan Peelman, Alice Robie, Catherine E. Schretter, Keith Sheppard, Chao Sun, Param Uttarwar, Julian M. Wagner, Eric Werner, Joseph Parker, Pietro Perona, Yisong Yue, Kristin Branson, and Ann Kennedy. MA...

    Pith/arXiv arXiv 2023

  2. [2]

    Ratinabox, a toolkit for modelling locomotion and neuronal activity in continuous environments

    Tom M George, Mehul Rastogi, William de Cothi, Claudia Clopath, Kimberly Stachenfeld, and Caswell Barry. Ratinabox, a toolkit for modelling locomotion and neuronal activity in continuous environments. Elife, 13:e85274, 2024

    2024

  3. [3]

    Human motion trajectory prediction: a survey.The International Journal of Robotics Research, 39(8): 895–935, June 2020

    Andrey Rudenko, Luigi Palmieri, Michael Herman, Kris M Kitani, Dariu M Gavrila, and Kai O Arras. Human motion trajectory prediction: a survey.The International Journal of Robotics Research, 39(8): 895–935, June 2020. ISSN 1741-3176. doi: 10.1177/0278364920917446. URL http://dx.doi.org/ 10.1177/0278364920917446

    doi:10.1177/0278364920917446 2020

  4. [4]

    Review of pedestrian trajectory prediction methods: Comparing deep learning and knowledge-based approaches, 2022

    Raphael Korbmacher and Antoine Tordeux. Review of pedestrian trajectory prediction methods: Comparing deep learning and knowledge-based approaches, 2022. URLhttps://arxiv.org/abs/2111.06740

    Pith/arXiv arXiv 2022

  5. [5]

    Salim, and Yang Song

    Renhao Huang, Hao Xue, Maurice Pagnucco, Flora D. Salim, and Yang Song. Vision-based multi- future trajectory prediction: A survey.IEEE Transactions on Neural Networks and Learning Systems, 36(8):13691–13708, August 2025. ISSN 2162-2388. doi: 10.1109/tnnls.2025.3550350. URL http: //dx.doi.org/10.1109/TNNLS.2025.3550350

    arXiv 2025

  6. [6]

    Social lstm: Human trajectory prediction in crowded spaces

    Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social lstm: Human trajectory prediction in crowded spaces. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 961–971, 2016. 17

    2016

  7. [7]

    Choy, Philip H

    Namhoon Lee, Wongun Choi, Paul Vernaza, Christopher B. Choy, Philip H. S. Torr, and Manmohan Chandraker. Desire: Distant future prediction in dynamic scenes with interacting agents, 2017. URL https://arxiv.org/abs/1704.04394

    Pith/arXiv arXiv 2017

  8. [8]

    Social gan: Socially acceptable trajectories with generative adversarial networks

    Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 2255–2264, 2018

    2018

  9. [9]

    The trajectron: Probabilistic multi-agent trajectory modeling with dynamic spatiotemporal graphs, 2019

    Boris Ivanovic and Marco Pavone. The trajectron: Probabilistic multi-agent trajectory modeling with dynamic spatiotemporal graphs, 2019. URLhttps://arxiv.org/abs/1810.05993

    Pith/arXiv arXiv 2019

  10. [10]

    Agentformer: Agent-aware transformers for socio-temporal multi-agent forecasting, 2021

    Ye Yuan, Xinshuo Weng, Yanglan Ou, and Kris Kitani. Agentformer: Agent-aware transformers for socio-temporal multi-agent forecasting, 2021. URLhttps://arxiv.org/abs/2103.14023

    Pith/arXiv arXiv 2021

  11. [11]

    Learning recurrent representations for hierarchical behavior modeling, 2016

    Eyrun Eyjolfsdottir, Kristin Branson, Yisong Yue, and Pietro Perona. Learning recurrent representations for hierarchical behavior modeling, 2016. URLhttps://arxiv.org/abs/1611.00094

    Pith/arXiv arXiv 2016

  12. [12]

    Evaluation metrics for behaviour modeling.arXiv preprint arXiv:2007.12298, 2020

    Daniel Jiwoong Im, Iljung Kwak, and Kristin Branson. Evaluation metrics for behaviour modeling.arXiv preprint arXiv:2007.12298, 2020

    Pith/arXiv arXiv 2007

  13. [13]

    Recurrent network models for kinematic tracking

    Katerina Fragkiadaki, Sergey Levine, and Jitendra Malik. Recurrent network models for kinematic tracking. CoRR, abs/1508.00271, 2015. URLhttp://arxiv.org/abs/1508.00271

    Pith/arXiv arXiv 2015

  14. [14]

    Quaternet: A quaternion-based recurrent model for human motion.CoRR, abs/1805.06485, 2018

    Dario Pavllo, David Grangier, and Michael Auli. Quaternet: A quaternion-based recurrent model for human motion.CoRR, abs/1805.06485, 2018. URLhttp://arxiv.org/abs/1805.06485

    Pith/arXiv arXiv 2018

  15. [15]

    Tripod: Human trajectory and pose dynamics forecasting in the wild

    Vida Adeli, Mahsa Ehsanpour, Ian Reid, Juan Carlos Niebles, Silvio Savarese, Ehsan Adeli, and Hamid Rezatofighi. Tripod: Human trajectory and pose dynamics forecasting in the wild. In2021 IEEE/CVF International Conference on Computer Vision (ICCV), page 13370–13380. IEEE, October 2021. doi: 10.1109/iccv48922.2021.01314. URLhttp://dx.doi.org/10.1109/ICCV48...

    arXiv 2021

  16. [16]

    Somoformer: Multi-person pose forecasting with transformers, 2022

    Edward Vendrow, Satyajit Kumar, Ehsan Adeli, and Hamid Rezatofighi. Somoformer: Multi-person pose forecasting with transformers, 2022. URLhttps://arxiv.org/abs/2208.14023

    Pith/arXiv arXiv 2022

  17. [17]

    Black, and Javier Romero

    Julieta Martinez, Michael J. Black, and Javier Romero. On human motion prediction using recurrent neural networks, 2017. URLhttps://arxiv.org/abs/1705.02445

    Pith/arXiv arXiv 2017

  18. [18]

    Best practices for 2-body pose forecasting, 2023

    Muhammad Rameez Ur Rahman, Luca Scofano, Edoardo De Matteis, Alessandro Flaborea, Alessio Sampieri, and Fabio Galasso. Best practices for 2-body pose forecasting, 2023. URL https://arxiv. org/abs/2304.05758

    Pith/arXiv arXiv 2023

  19. [19]

    Learning trajectory dependencies for human motion prediction, 2020

    Wei Mao, Miaomiao Liu, Mathieu Salzmann, and Hongdong Li. Learning trajectory dependencies for human motion prediction, 2020. URLhttps://arxiv.org/abs/1908.05436

    Pith/arXiv arXiv 2020

  20. [20]

    H3wb: Human3.6m 3d wholebody dataset and benchmark

    Yue Zhu, Nermin Samet, and David Picard. H3wb: Human3.6m 3d wholebody dataset and benchmark. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 20166–20177, October 2023

    2023

  21. [21]

    Alvinn: An autonomous land vehicle in a neural network.Advances in neural information processing systems, 1, 1988

    Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network.Advances in neural information processing systems, 1, 1988

    1988

  22. [22]

    Foster, Adam Block, and Dipendra Misra

    Dylan J. Foster, Adam Block, and Dipendra Misra. Is behavior cloning all you need? understanding horizon in imitation learning, 2024. URLhttps://arxiv.org/abs/2407.15007

    Pith/arXiv arXiv 2024

  23. [23]

    Decision transformer: Reinforcement learning via sequence modeling.Advances in neural information processing systems, 34:15084–15097, 2021

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling.Advances in neural information processing systems, 34:15084–15097, 2021

    2021

  24. [24]

    Offline reinforcement learning as one big sequence modeling problem, 2021

    Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem, 2021. URLhttps://arxiv.org/abs/2106.02039

    Pith/arXiv arXiv 2021

  25. [25]

    Multi-agent reinforcement learning is a sequence modeling problem, 2022

    Muning Wen, Jakub Grudzien Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi-agent reinforcement learning is a sequence modeling problem, 2022. URLhttps://arxiv.org/ abs/2205.14953

    Pith/arXiv arXiv 2022

  26. [26]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control, 2023

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov...

    Pith/arXiv arXiv 2023

  27. [27]

    Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    The arcade learning environment: An evaluation platform for general agents.Journal of artificial intelligence research, 47:253–279, 2013

    Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents.Journal of artificial intelligence research, 47:253–279, 2013

    2013

  29. [29]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026–5033. IEEE, 2012

    2012

  30. [30]

    Whole-body physics simulation of fruit fly locomotion.Nature, 643(8074):1312–1320, 2025

    Roman Vaxenburg, Igor Siwanowicz, Josh Merel, Alice A Robie, Carmen Morrow, Guido Novati, Zinovia Stefanidi, Gert-Jan Both, Gwyneth M Card, Michael B Reiser, et al. Whole-body physics simulation of fruit fly locomotion.Nature, 643(8074):1312–1320, 2025

    2025

  31. [31]

    A virtual rodent predicts the structure of neural activity across behaviours.Nature, 632(8025):594–602, 2024

    Diego Aldarondo, Josh Merel, Jesse D Marshall, Leonard Hasenclever, Ugne Klibaite, Amanda Gellis, Yuval Tassa, Greg Wayne, Matthew Botvinick, and Bence P Ölveczky. A virtual rodent predicts the structure of neural activity across behaviours.Nature, 632(8025):594–602, 2024

    2024

  32. [32]

    Connectome- constrained networks predict neural activity across the fly visual system.Nature, 634(8036):1132–1140, 2024

    Janne K Lappalainen, Fabian D Tschopp, Sridhama Prakhya, Mason McGill, Aljoscha Nern, Kazunori Shinomiya, Shin-ya Takemura, Eyal Gruntman, Jakob H Macke, and Srinivas C Turaga. Connectome- constrained networks predict neural activity across the fly visual system.Nature, 634(8036):1132–1140, 2024

    2024

  33. [33]

    Catherine Schretter Kristin Branson, Alice A. Robie. FlyMABe2022 (revision c705992), 2026. URL https://huggingface.co/datasets/kristinbranson/FlyMABe2022

    2026

  34. [34]

    The fly disco: hardware and software for optogenetics and fine-grained fly behavior analysis.bioRxiv, pages 2024–11, 2024

    Alice A Robie, Adam L Taylor, Catherine E Schretter, Mayank Kabra, and Kristin Branson. The fly disco: hardware and software for optogenetics and fine-grained fly behavior analysis.bioRxiv, pages 2024–11, 2024

    2024

  35. [35]

    Mapping the neural substrates of behavior.Cell, 170(2):393–406, 2017

    Alice A Robie, Jonathan Hirokawa, Austin W Edwards, Lowell A Umayam, Allen Lee, Mary L Phillips, Gwyneth M Card, Wyatt Korff, Gerald M Rubin, Julie H Simpson, et al. Mapping the neural substrates of behavior.Cell, 170(2):393–406, 2017

    2017

  36. [36]

    Multi-person 3d motion prediction with multi-range transformers

    Jiashun Wang, Huazhe Xu, Medhini Narasimhan, and Xiaolong Wang. Multi-person 3d motion prediction with multi-range transformers. In M. Ranzato, A. Beygelzimer, Y . Dauphin, P.S. Liang, and J. Wortman Vaughan, editors,Advances in Neural Information Processing Systems, volume 34, pages 6036–6049. Cur- ran Associates, Inc., 2021. URL https://proceedings.neur...

    2021

  37. [37]

    Gordon, and J

    Stéphane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. No-regret reductions for imitation learning and structured prediction.CoRR, abs/1011.0686, 2010. URLhttp://arxiv.org/abs/1011.0686

    Pith/arXiv arXiv 2010

  38. [38]

    Alvinn: An autonomous land vehicle in a neural network

    Dean Pomerleau. Alvinn: An autonomous land vehicle in a neural network. In D.S. Touretzky, editor, Proceedings of (NeurIPS) Neural Information Processing Systems, pages 305 – 313. Morgan Kaufmann, December 1989

    1989

  39. [39]

    Insect walking.Annual review of entomology, 11(1):103–122, 1966

    Donald M Wilson. Insect walking.Annual review of entomology, 11(1):103–122, 1966. AOperationdetails Fig. 12 shows the operations involved in computing the input and output representations from the tracked world-frame keypoints of the flies, where the world-frame scale is millimeters and the origin is the center of the arena. Fig. 13 shows the operations f...

    1966

This paper was first reviewed by deepseek-v4-flash on August 1, 2026.