{"id":"60416e7d-3f81-4f06-bc30-acc74946151a","arxiv_id":"2607.19548","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Agent-centric transformers trained through a composable library reproduce several marginal statistics of courting fly behavior, but discriminators still separate simulated from real flies.","lead":"The authors model each animal as an agent that senses and moves in its own egocentric frame, and release a library of composable operations for training and rolling out such models. Applied to courting fruit flies, the models reproduce some behavioral statistics, but a trained discriminator can still tell simulated from real movement.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Behavior-frequency evidence relies on real-trained probes applied to simulated hidden states; §6.6/E.3 admit this transfer is unvalidated, so the central claim's main quantitative support is not yet established.","rationale":"Good-faith reading: the paper is a systems/benchmark contribution; the library and systematic variant comparisons are valuable independent of the strong claim. I focused on the strongest quantitative evidence for 'captures the distribution of social behavior': the behavior-pattern frequency analysis (§6.6, Figs. 8b/9b). That evidence uses linear probes trained on real hidden states (§6.7/E.4) and applied to simulated hidden states without a transfer check; §6.6 explicitly warns 'may not generalize well to simulated data,' and §E.3 notes each variant uses its own probe. This is the least secure condition in the central claim. Other metrics—feature histograms (§6.4), discriminator (§6.5), and probe MCC on real data (§6.7)—are informative, but they do not establish that the model produces the right frequency of courtship behaviors; the discriminator actually shows simulated velocity statistics are distinguishable from real (§6.5). The frequency result is thus the key missing piece. I do not think this warrants rejection: the paper is transparent about the caveat, and the library + evaluation suite is a real contribution. But the claim should remain conditional until probe transfer is either validated or the frequency analysis is redone on simulated data with ground-truth labels. This matches the reader's weakest assumption, so agreement = agree and verdict unchanged (CONDITIONAL).","tokens_in":21659,"tokens_out":4970,"duration_ms":48351,"concrete_test":"Expert-annotate a sample of rollouts from the REFERENCE model (e.g., 10 sessions × 512 frames) for courting/chasing/wing extension, following the MABe label definitions. Train/apply the same linear probes to simulated hidden states and compute MCC against these annotations on simulated frames; compare with probe-vs-manual MCC on real validation frames. If simulated MCC is close to real MCC, the transfer assumption holds; if it drops substantially (e.g., toward chance), recompute Fig. 9b using manual labels or retrained probes and re-evaluate the 'captures distribution of social behavior' claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 6.6 reports relative frequency errors for walking, courting, chasing, and wing extension using a handcrafted walk detector and binary classifiers (§6.7) trained on real trajectory hidden states. The text immediately flags that these classifiers 'are only trained and validated on real trajectory data, and may not generalize well to simulated data,' and §E.3 adds that the classifiers differ across model variants. Because the frequency comparison is the main quantitative evidence that rollouts reproduce longer-timescale social behaviors (Fig. 8b/9b), the central claim that models 'capture the distribution of social behavior' depends on an unvalidated transfer assumption. Rollout hidden states are generated from the model and can be far from the real-data manifold—especially for variants like DISCRETIZE ALL or deep into the rollout—so a probe that is accurate on real validation frames can have arbitrary or biased predictions on simulated frames. If transfer fails, the reported RE values do not measure behavior-frequency fidelity; they measure a mixture of model fidelity and probe domain shift. The authors' own caveat makes this the least secure step in the argument.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces an agent-centric framework for generative modeling of animal pose trajectories, along with a Python library (AnimalPoseForecasting) that composes and inverts transformations between world-frame, egocentric, sensory, and ML-friendly representations. The authors train transformer models on multi-fly Drosophila courtship data, compare several input/output representation variants, and evaluate them through qualitative videos, feature-distribution Wasserstein distances, real-vs-simulated discriminators, behavior-frequency comparisons, and linear probes. They also demonstrate transfer to a synthetic rat domain (RatInABox). The central claims are that the agent-centric models capture the distribution of social behavior in courting Drosophila and that agent-centric representations are the most important modeling factor.","tokens_in":22044,"tokens_out":7543,"duration_ms":69077,"significance":"If the claims hold, this is a valuable contribution: a reusable library for a non-trivial representational problem, a systematic ablation of representation choices under matched architecture, and a quantitative evaluation toolkit for generative behavior models. The composable/invertible operation abstraction is a genuine contribution, and the open-source release makes the infrastructure widely usable. The empirical study is anchored to real data rather than only model outputs, and the multi-scale evaluation (gait, motifs, long-timescale frequencies) is more thorough than typical in pose forecasting. The RatInABox adaptation supports the generality argument. However, the central scientific claim is currently stronger than the evidence supports, for the reasons detailed below.","major_comments":[{"comment":"The behavior-frequency evaluation for courting, chasing, and wing extension uses linear probes trained on real hidden states and then applies them to simulated hidden states. The text itself concedes these classifiers 'are only trained and validated on real trajectory data, and may not generalize well to simulated data' (§6.6), and §E.3 adds that the probe used differs per variant. Because these RE values are the main quantitative support for the claim that rollouts reproduce longer-timescale social behavior, the central claim requires either validation of probe transfer (e.g., simulated labels from a synthetic oracle, or a domain-shift calibration on held-out real frames) or a clearly qualified conclusion.","section":"§6.6 and §E.3"},{"comment":"The real-vs-simulated discriminator results are in direct tension with the abstract's claim that models 'capture the distribution of social behavior.' The velocity discriminator reaches 0.92 accuracy within the first four simulated frames and 0.97 overall (Fig. 6b), and the pose-based discriminator is 0.65 even at 64 frames. A simple MLP can thus almost perfectly separate real from simulated instantaneous movement. The paper acknowledges this in §6.5 but does not reconcile it with the central claim. Please either temper the abstract/conclusion or provide a quantitative argument for why near-perfect discriminability is consistent with 'capturing the distribution.'","section":"§6.5"},{"comment":"All comparisons between model variants are based on single training runs, with no repeated-seed error bars or significance tests. The attribution that agent-centric representations are 'most important' (KEYPOINTS worst on nearly every metric) and the conclusions about discretization, context length, and pose parameterization could be driven by run-to-run variance. Please report means and variances over at least 3–5 seeds for the key variants (REFERENCE, KEYPOINTS, NO DISCRETIZATION, SHORTER CONTEXT) or otherwise justify the stability of the reported differences.","section":"§6.2 and §6.8"},{"comment":"The KEYPOINTS variant changes both input and output representations (world-frame keypoints instead of egocentric sensory features and egocentric pose velocity). The conclusion in §6.8 that 'agent-centric representations enabled by our library is most important' is therefore supported only by a confounded ablation. To support the attribution, the authors should include intermediate variants (e.g., egocentric input with world-frame keypoint output, or world-frame input with egocentric output) to isolate the contribution of the input vs. output representation.","section":"§6.2 and §6.8"},{"comment":"According to §D, rollouts are seeded from training trajectories. This means the model's context window contains frames seen during training, which is non-standard for generative model evaluation and may yield optimistic results if the model has memorized these sequences. The paper should either evaluate rollouts from held-out validation/test prompts, or explain why training-set prompts are appropriate for the distributional claim.","section":"§D"}],"minor_comments":[{"comment":"The parenthetical caveat is missing a closing parenthesis: 'trained and validated on real trajectory data (§ 6.7, and may not generalize well to simulated data.' Please fix.","section":"§6.6"},{"comment":"The axis labels 'Velocity discr. accuracy' and 'Dist. to prompt (fr. range)' are informal; please spell out 'Discriminator accuracy' and 'Distance to prompt (frame range)' for clarity.","section":"Figure 6b"},{"comment":"The author list for the FlyMABe2022 dataset reference appears malformed: 'Catherine Schretter Kristin Branson, Alice A. Robie' should be formatted consistently with the other references.","section":"Reference [33]"},{"comment":"The affiliation line reads 'Ashburn, V A 20147'; the state abbreviation should be 'VA'.","section":"Affiliation"},{"comment":"The description of the 'previous frame's output' as an input modality is slightly ambiguous; clarify whether this is the model's own output under teacher forcing or self-feeding at test time.","section":"§6.2"}],"recommendation":"major_revision","confidential_remarks":"This is a solid systems/ML paper with a useful library and a rich empirical study. The main blockers are the unvalidated probe transfer in the behavior-frequency evaluation and the tension between the discriminator results and the abstract's strong claim. The other issues (single seeds, confounded KEYPOINTS ablation, training-set prompts) are fixable with additional experiments or careful rephrasing. The paper's central contribution—the compositional library and the systematic comparison—is worth publishing after these points are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work on generative models of behavior. The real contribution is the AnimalPoseForecasting library and the systematic comparison of representation choices. The agent-centric framing is not brand new—it extends the authors' earlier fly work—but this is the first multi-animal full-pose transformer setup, and the library's composable/invertible operations make the design space explorable. The RatInABox adaptation shows it transfers to another domain.\n\nWhat's good: the evaluation suite is thoughtful, with feature histograms, discriminators, probes, and behavior frequencies. The code is released. The paper is honest about its own limitations—it explicitly notes that the behavior classifiers 'may not generalize well to simulated data.' That honesty is to its credit.\n\nThe soft spots are real but not fatal. First, the velocity discriminator at 0.97 overall and 0.92 within the first four rollout frames means a trained classifier can tell real from simulated frames almost perfectly. That directly undercuts the abstract's 'capture the distribution of social behavior.' Second, the behavior-frequency numbers—the main evidence for longer-timescale fidelity—depend on linear probes trained on real hidden states and applied to simulated hidden states with no validation of transfer. Since simulated states can fall off the real-data manifold, especially deep in rollouts, those relative-error values could be measuring probe domain shift as much as model fidelity. Third, there are no repeated-seed error bars and no external baselines, so it's hard to know how much variability to attribute to the model choices. The KEYPOINTS variant changes both input and output, so the agent-centric conclusion is not as cleanly identified as the text implies.\n\nNone of this sinks the paper. The library is useful, the comparisons are systematic, and the limitations are addressable. But the authors should either soften the central claim or add transfer validation and error bars before someone relies on the behavior-frequency numbers.\n\nI'd send it to peer review. The infrastructure and experiments deserve referee time, and the probe-transfer issue is exactly what a good reviewer should push them to fix.","headline":"Solid infrastructure paper with a good library and honest evaluation, but the headline claim about capturing the behavior distribution outruns the paper's own discriminator and probe-transfer evidence.","tokens_in":22445,"tokens_out":2615,"would_cite":true,"duration_ms":23800,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Agent-centric generative models—in which a network senses and acts from an animal's own reference frame—capture the distribution of social courtship behavior in groups of Drosophila, and the choice of egocentric representation is the main d","keywords":["agent-centric behavior models","egocentric representation","generative pose forecasting","Drosophila courtship behavior","autoregressive multi-agent rollout","output discretization","behavior distribution evaluation","linear probes of internal representations"],"falsifier":"A concrete check: take model rollouts, obtain ground-truth behavior labels independently (manual annotation or a rule-based detector like the paper's walk detector), train the same linear probes on simulated hidden states, and re-count behavior frequencies; a large drop in probe accuracy on simulated states would falsify the transfer assumption underlying the behavior-frequency results.","tokens_in":21613,"feed_emoji":"🪰","tokens_out":7356,"duration_ms":60442,"temperature":0.7,"pith_summary":"The paper proposes agent-centric generative modeling of animal behavior: instead of predicting world-frame trajectories, a transformer is given egocentric sensory features and asked to output egocentric movements, so that social interaction emerges from independently sensing agents rather than a centralized group model. The paper claims that, with a library of composable invertible operations to translate between world, egocentric, and machine-learning representations, this approach captures the distribution of social courtship behavior in groups of Drosophila—feature statistics, walking gait, chasing and wing extension, and condition differences. The authors also conclude, from systematic ablations, that the agent-centric representation itself, rather than architecture, is the main driver of accuracy, with output discretization necessary to avoid collapse to no movement. For a reader, the significance is a reusable recipe and toolset for building biologically constrained generative models that can be interrogated as hypotheses about how animals compute behavior.","feed_headline":"Agent-centric fly pose models capture courtship behavior distributions","feed_subtitle":"Library turns tracked pose into egocentric inputs and shows representation choice, not architecture, drives behavioral accuracy.","key_machinery":"The central mechanism is the Operation abstraction: composable, invertible transformations (global/local velocity, egocentric roll, z-scoring, discretization, sensory and pose encoders) that connect world-frame keypoints, egocentric sensory inputs, and ML-ready labels, chained together and stored as a data-flow graph. Because every forward operation has an inverse and the chain is recorded, autoregressive rollout can invert predicted egocentric movements back to world pose, update all agents, recompute each agent's sensory inputs for the next step, and cleanly swap representations—making the agent-centric comparison the paper's main experimental lever.","core_discovery":"On the paper's own terms, the discovery is that an autoregressive transformer, constrained to receive only hand-crafted egocentric sensory approximations (a visual depth map of nearby flies, social and arena touch, proprioceptive pose) and to emit egocentric movement, learns to generate multi-agent Drosophila behavior whose feature distributions, gait patterns, chasing and wing-extension frequencies, and condition-dependent differences match real data at multiple timescales—and does so largely because of the agent-centric representation. Replacing it with world-frame keypoints degrades almost every metric, while removing handcrafted joint-angle parameterization barely matters; output discret","pith_inferences":["Our inference: the probe-transfer caveat noted in §§6.6 and E.3 means the simulated courtship-behavior frequencies should be read as upper bounds on fidelity; an easy test is to retrain probes on simulated hidden states with labels from an independent detector or manual annotation.","Our inference: the paper's own failure mode with independent per-dimension sampling during rollout suggests joint quantization of full poses (e.g., vector-quantized tokens over the whole body) as the natural next step for long-horizon realism.","Our inference: if the egocentric-representation result generalizes beyond flies, it hints that world-frame joint prediction in multi-agent human pose forecasting may be part of why interactive accuracy suffers; agent-centric sensory gating could improve those systems too.","Our inference: since probes succeed with no behavior-label supervision during training, future work could mine hidden states for unannotated behavioral motifs—the paper names this goal as open but does not execute it."],"forward_implications":["Behavioral fidelity is attributable to representation, not architecture: the agent-centric egocentric setting is the dominant factor, and the world-frame keypoint variant performs worst on nearly every metric, including probe accuracy.","Output discretization is a practical necessity for behavior cloning of stochastic, multi-modal behavior: without it, models collapse to predicting no movement and simulations freeze.","Forecasting alone serves as a representation-learning objective: linear probes on an intermediate transformer layer decode known behavior categories and fly-type conditions, so next-step prediction can act as unsupervised behavior representation learning.","Closed-loop social interaction can emerge without physics simulation: simulated flies approach, chase, avoid collisions, and remain in the arena purely through recomputed sensory feedback.","The same operation-based pipeline transfers to a new domain (synthetic rat navigation with boundary- and head-direction sensory inputs) by adding only a new sensory operation, offering a recipe for agent-centric generative models in other species."],"fun_headline_variants":["For fly behavior, egocentric input beats world-frame keypoints","Agent-centric pose model reproduces fly courtship without world coordinates","Representation choice, not architecture, key to fly pose model accuracy","New library enables agent-centric animal pose forecasting"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the linear probes trained and validated only on real trajectories (§6.6, §E.3) remain accurate on simulated hidden states; if they do not transfer, the reported frequencies of courtship behaviors in simulated flies are unsupported.","fun_headline_variants_meta":{"raw":{"variants":["For fly behavior, egocentric input beats world-frame keypoints","Agent-centric pose model reproduces fly courtship without world coordinates","Representation choice, not architecture, key to fly pose model accuracy","New library enables agent-centric animal pose forecasting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00042,"raw_usage":{"total_tokens":1975,"prompt_tokens":700,"completion_tokens":1275,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":444,"completion_tokens_details":{"reasoning_tokens":1207}},"tokens_in":444,"tokens_out":1275,"duration_ms":9127,"temperature":1.0,"reasoning_tokens":1207,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T12:24:37.457701+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: take model rollouts, obtain ground-truth behavior labels independently (manual annotation or a rule-based detector like the paper's walk detector), train the same linear probes on simulated hidden states, and re-count behavior frequencies; a large drop in probe accuracy on simulated states would falsify the transfer assumption underlying the behavior-frequency results.","supporting_citations":[],"review_version":1}