pith. sign in

arxiv: 2604.25949 · v1 · submitted 2026-04-21 · 💻 cs.RO

FalconApp: Rapid iPhone Deployment of End-to-End Perception via Automatically Labeled Synthetic Data

Yan Miao , Will Shen , Sayan Mitra This is my paper

Pith reviewed 2026-05-10 01:43 UTC · model grok-4.3

classification 💻 cs.RO
keywords synthetic datapose estimationmask detectionmobile deploymentGaussian splattingperception modelsiPhone app6-DoF estimation
0
0 comments X

The pith

FalconApp converts a short iPhone video of a rigid object into a deployable mask-detection and 6-DoF pose model using automatically labeled synthetic data.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper demonstrates an end-to-end mobile pipeline that captures a brief handheld video of an object, reconstructs it as a 3D asset, generates photorealistic synthetic images with ground-truth labels, trains perception models, and returns a working module to the phone. This matters because reliable robotics perception normally requires large manually annotated real-world datasets that are slow and expensive to create. If the pipeline works, users can quickly produce custom perception for new objects without expert labeling or external servers. A sympathetic reader would see it as a practical way to lower the data barrier for object-specific robotic tasks on consumer hardware.

Core claim

FalconApp produces usable perception models with about 20 minutes of synthetic-data generation and training per object on average, around 30 ms end-to-end on-device latency on iPhone, and better overall pose accuracy than a PnP baseline on 4 / 5 objects in both simulation and real-world evaluation.

What carries the argument

The photorealistic auto-labeling workflow that reconstructs an editable GSplat asset from the user video, composites it with diverse backgrounds, and renders synthetic images carrying ground-truth masks and poses for training.

Load-bearing premise

A short handheld capture of a rigid object is sufficient to produce an accurate editable 3D asset whose rendered images train models that generalize to real-world captures.

What would settle it

Real-world evaluation on the five tested objects showing the trained models achieve lower pose accuracy than the PnP baseline or exceed 30 ms end-to-end latency on iPhone.

Figures

Figures reproduced from arXiv: 2604.25949 by Sayan Mitra, Will Shen, Yan Miao.

Figure 1
Figure 1. Figure 1: FalconApp Pipeline: From a 2-minute iPhone video of an object, FalconApp reconstructs a photorealistic GSplat, composites it with diverse background in FalconGym 2.0 [1], automatically synthesizes ground-truth masks and pose under randomized viewpoints, produces a full perception module in ∼20 min and returns it to the app frontend for live inference with an overall latency of ∼30 ms. Abstract— Reliable pe… view at source ↗
Figure 2
Figure 2. Figure 2: FalconApp frontend interface: the four views, from left to right, show iPhone app installation, TCP connection to the backend, recording mode (Stage 1), and inference mode (Stage 4) in the pipeline of [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Demonstration of Automatically labeled Synthetic Data: FalconApp produces photorealistic synthetic images and automatically generates labels for objects with diverse geometry and appearance to train a perception module [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Perception Architecture: an encoder extracts features that feed into a mask prediction head; the predicted mask conditions a gated-attention pose head to regress the relative 6-DoF target pose. During training, we use FalconGym 2.0 to re-render targets at predicted poses and apply reprojection loss to improve geometric consistency and pose estimation. C. FalconApp Backend: Perception Module Training Each u… view at source ↗
read the original abstract

Reliable perception for robotics depends on large-scale labeled data, yet real-world datasets rely on heavy manual annotation and are time-consuming to produce. We present FalconApp, an iPhone app with an end-to-end frontend-backend pipeline that turns a short handheld capture of a rigid object into a perception module for mask detection and 6-DoF pose estimation. Our core contribution is a rapid mobile deployment pipeline paired with a photorealistic auto-labeling workflow: from a user-captured video of an object, FalconApp reconstructs an editable GSplat asset, composites it with diverse photorealistic backgrounds, renders synthetic images with ground-truth masks and poses, trains the perception module, and deploys it back to the iPhone frontend. Experiments across five rigid objects with diverse geometry and appearance show that FalconApp produces usable perception models with about 20 minutes of synthetic-data generation and training per object on average, around 30 ms end-to-end on-device latency on iPhone, and better overall pose accuracy than a PnP baseline on 4 / 5 objects in both simulation and real-world evaluation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 3 minor

Summary. The manuscript presents FalconApp, an iPhone app implementing an end-to-end pipeline that converts a short handheld video capture of a rigid object into a deployable perception module for mask detection and 6-DoF pose estimation. The pipeline reconstructs an editable GSplat asset from the capture, composites it with photorealistic backgrounds to auto-generate synthetic training images with ground-truth labels, trains the model, and deploys it on-device. Experiments on five objects with varied geometry and appearance report an average of 20 minutes for synthetic data generation and training per object, approximately 30 ms end-to-end on-device latency, and superior pose accuracy relative to a PnP baseline on 4 out of 5 objects in both simulation and real-world evaluations.

Significance. If the results hold under more detailed scrutiny, the work offers a practical advance in mobile robotics perception by automating the creation of custom, labeled datasets and models from minimal real-world input. This could substantially reduce the time and expertise needed for deploying object-specific perception on consumer devices, with relevance to AR, robotics, and rapid prototyping scenarios where manual annotation is a bottleneck.

major comments (2)
  1. Experiments section: The central claim of better overall pose accuracy than the PnP baseline on 4/5 objects (both sim and real) is load-bearing for the paper's contribution, yet the manuscript provides no specific metrics (e.g., ADD-S, rotation/translation error distributions, or success thresholds), test set sizes, or data split protocols. This omission prevents verification of the reported improvements and generalization from synthetic to real data.
  2. Method / pipeline description: The assumption that a short handheld capture suffices to produce an accurate, editable GSplat asset whose renderings train models that generalize to real captures is central to the workflow, but the manuscript lacks quantitative reconstruction quality metrics (e.g., PSNR, coverage analysis) or ablation on capture length/variation to support this.
minor comments (3)
  1. Abstract: Include at least one concrete metric (e.g., mean pose error or success rate) alongside the qualitative claim of 'better overall pose accuracy' to strengthen the summary of results.
  2. Figures: Pipeline overview figures would benefit from clearer annotations distinguishing real capture, GSplat reconstruction, synthetic rendering, and on-device inference stages.
  3. Related work: Ensure citations cover recent mobile deployment and synthetic data methods for 6-DoF pose estimation to contextualize the contribution fully.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive and detailed review of our manuscript. We appreciate the emphasis on the need for greater quantitative detail to support the central claims. Below we respond point-by-point to the major comments and indicate the revisions we will make.

read point-by-point responses

  1. Referee: Experiments section: The central claim of better overall pose accuracy than the PnP baseline on 4/5 objects (both sim and real) is load-bearing for the paper's contribution, yet the manuscript provides no specific metrics (e.g., ADD-S, rotation/translation error distributions, or success thresholds), test set sizes, or data split protocols. This omission prevents verification of the reported improvements and generalization from synthetic to real data.

    Authors: We agree that the absence of these specific metrics limits the ability to verify the reported improvements. In the revised manuscript we will expand the Experiments section to report ADD-S scores, mean and distributional statistics for rotation and translation errors, success rates at standard thresholds (e.g., 10°/5 cm), the exact sizes of the simulation and real-world test sets, and the data-splitting protocols employed. These additions will directly substantiate the claim of superior pose accuracy on 4/5 objects and clarify the degree of sim-to-real generalization. revision: yes

  2. Referee: Method / pipeline description: The assumption that a short handheld capture suffices to produce an accurate, editable GSplat asset whose renderings train models that generalize to real captures is central to the workflow, but the manuscript lacks quantitative reconstruction quality metrics (e.g., PSNR, coverage analysis) or ablation on capture length/variation to support this.

    Authors: We acknowledge that quantitative evidence for reconstruction quality would strengthen the pipeline description. We will add PSNR values computed on held-out views for each of the five GSplat reconstructions and include a basic coverage analysis (e.g., percentage of surface points observed). Regarding ablation on capture length and variation, we will describe the standardized capture protocol used across objects and, where possible with existing recordings, provide supplementary analysis of how modest changes in capture duration affect downstream perception accuracy. Any ablation that would require new data collection will be noted as a limitation and direction for future work. revision: partial

Circularity Check

0 steps flagged

No significant circularity; pipeline uses independent real-world evaluation

full rationale

The described pipeline begins with real handheld video captures of rigid objects, reconstructs GSplat assets, composites photorealistic backgrounds, renders synthetic images with automatically generated ground-truth labels, trains perception models for mask detection and pose estimation, and deploys to iPhone. Evaluation occurs on separate real-world tests and simulation benchmarks, with comparisons to a PnP baseline showing improvements on 4/5 objects. No derivation step reduces a claimed prediction or result to a fitted parameter or self-citation by construction; the workflow is externally falsifiable via held-out real captures and does not rely on self-definitional loops or imported uniqueness theorems.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The approach rests on domain assumptions about reconstruction quality and synthetic-to-real transfer rather than new free parameters or invented entities.

axioms (2)
  • domain assumption Rigid objects can be accurately reconstructed from short handheld video using Gaussian Splatting to produce an editable asset.
    Invoked as the starting point of the auto-labeling workflow in the abstract.
  • domain assumption Compositing the reconstructed asset with diverse photorealistic backgrounds produces synthetic images whose ground-truth labels enable models that generalize to real captures.
    Central to the claim that 20-minute synthetic data suffices for usable performance.

pith-pipeline@v0.9.0 · 5494 in / 1604 out tokens · 62177 ms · 2026-05-10T01:43:13.435207+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

12 extracted references · 12 canonical work pages

  1. [1]

    Performance-guided refine- ment for visual aerial navigation using editable gaussian splatting in falcongym 2.0,

    Y . Miao, E. Yuceel, G. Fainekoset al., “Performance-guided refine- ment for visual aerial navigation using editable gaussian splatting in falcongym 2.0,” inProceedings of IEEE International Conference on Robotics and Automation (ICRA), 2026

    work page 2026

  2. [2]

    Mask r-cnn,

    K. He, G. Gkioxari, P. Doll ´aret al., “Mask r-cnn,” in2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2980– 2988

    work page 2017

  3. [3]

    Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes,

    Y . Xiang, T. Schmidt, V . Narayananet al., “Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes,” inRobotics: Science and Systems (RSS), 2018. [Online]. Available: https://github.com/yuxng/PoseCNN

    work page 2018

  4. [4]

    Imagenet: A large-scale hierar- chical image database,

    J. Deng, W. Dong, R. Socheret al., “Imagenet: A large-scale hierar- chical image database,” in2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 248–255

    work page 2009

  5. [5]

    Bdd100k: A diverse driving dataset for heterogeneous multitask learning,

    F. Yu, H. Chen, X. Wanget al., “Bdd100k: A diverse driving dataset for heterogeneous multitask learning,” inIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020

    work page 2020

  6. [6]

    CARLA: An open urban driving simulator,

    A. Dosovitskiy, G. Ros, F. Codevillaet al., “CARLA: An open urban driving simulator,” inProceedings of the 1st Annual Conference on Robot Learning, 2017, pp. 1–16

    work page 2017

  7. [7]

    Design and use paradigms for gazebo, an open-source multi-robot simulator,

    N. Koenig and A. Howard, “Design and use paradigms for gazebo, an open-source multi-robot simulator,” in2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566), vol. 3, 2004, pp. 2149–2154 vol.3

    work page 2004

  8. [8]

    Nerf: Representing scenes as neural radiance fields for view synthesis,

    B. Mildenhall, P. P. Srinivasan, M. Tanciket al., “Nerf: representing scenes as neural radiance fields for view synthesis,”Commun. ACM, vol. 65, no. 1, p. 99–106, Dec. 2021. [Online]. Available: https://doi.org/10.1145/3503250

    work page doi:10.1145/3503250 2021

  9. [9]

    3d gaussian splatting for real-time radiance field rendering,

    B. Kerbl, G. Kopanas, T. Leimk ¨uhleret al., “3d gaussian splatting for real-time radiance field rendering,”ACM Transactions on Graphics, vol. 42, no. 4, July 2023. [Online]. Available: https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/

    work page 2023

  10. [10]

    Falcongym: A photorealistic simulation framework for zero-shot sim-to-real vision-based quadrotor navigation,

    Y . Miao, W. Shen, and S. Mitra, “Falcongym: A photorealistic simulation framework for zero-shot sim-to-real vision-based quadrotor navigation,” in2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2025, pp. 17 154–17 161

    work page 2025

  11. [11]

    Pose estimation for augmented reality: A hands-on survey,

    E. Marchand, H. Uchiyama, and F. Spindler, “Pose estimation for augmented reality: A hands-on survey,”IEEE Transactions on Visu- alization and Computer Graphics, vol. 22, no. 12, pp. 2633–2651, 2016

    work page 2016

  12. [12]

    Falconwing: An ultra-light indoor fixed-wing uav platform for vision-based autonomy,

    Y . Miao, W. Shen, H. Cuiet al., “Falconwing: An ultra-light indoor fixed-wing uav platform for vision-based autonomy,” 2025. [Online]. Available: https://arxiv.org/abs/2505.01383

    work page arXiv 2025