FoodTrack: Estimating Handheld Food Portions with Egocentric Video

Ervin Wang; Yuhao Chen

REVIEW 3 major objections 6 minor 16 references

FoodTrack: Estimating Handheld Food Portions with Egocentric Video

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FoodTrack estimates a handheld sandwich’s volume from egocentric video with roughly 7% absolute error, improving on a 16.4% best-case prior baseline.

desk verdict A promising but very preliminary pipeline: one sandwich and a fragile metric-depth scale factor make the 7.01% headline a possibility, not a result. read the letter →

arxiv 2505.04055 v1 pith:L2Q46IMK submitted 2025-05-07 cs.CV

Ervin Wang , Yuhao Chen This is my paper · ORCID

classification cs.CV

keywords egocentricvideofoodvolumeestimation3Dreconstructiondietarymonitoringmonoculardepthscalinghandheldportionsize

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FoodTrack is a proposed end-to-end pipeline for measuring the volume of a handheld food item from a single egocentric video captured with camera glasses, without fixed camera angles or assumptions about bite size. The paper reports an absolute percentage error of about 7.01% on a sandwich—345 mL estimated versus 371 ± 1 mL measured by water displacement—compared with a previous approach’s best-case error of 16.40% under stricter data-collection conditions. The core strategy is to reconstruct a 3D mesh of the food with a neural tracking method, then impose physical scale using one monocular depth estimate. If the approach generalizes, everyday dietary monitoring could measure actual food volume rather than counting bites.

What carries the argument

The load-bearing mechanism is a scale factor $R = D/f_x$ applied to a mesh whose shape is produced by a neural 6-DOF tracking and reconstruction routine (called BundleSDF in the paper). A monocular metric-depth model estimates the absolute depth $D$ in meters at the food mask’s centroid and the camera’s focal length $f_x$; because volume scales as the cube of a linear dimension, the mesh volume is multiplied by $R^3$. The preceding steps—super-resolution, zero-shot segmentation, video object segmentation, temporally consistent depth maps, and replacement of the default feature matcher with a learned matcher—all serve to make the reconstructed mesh reliable enough for this scaling to be meaningful.

What would settle it

Take a food object of known volume, run FoodTrack at three different camera distances, and compare estimated against measured volume; because the scale factor enters as $(D/f_x)^3$, a constant depth bias should make the volume error about three times the depth error and the three estimates should disagree by roughly that factor. If the estimates do not behave that way, the depth-cubing mechanism is not what drives the reported accuracy.

Watch

Extended reading notes

Core claim

The paper claims that the volume of an occluded, hand-held food object can be recovered from egocentric video by combining object tracking with a single depth-based scale factor. Concretely, a 15-second rotating-sandwich video yields a reconstructed mesh whose volume, after scaling by the cube of the ratio between estimated depth and focal length, comes to 345 mL against a measured 371 ± 1 mL. The authors present this as a preliminary single-example demonstration, not as a statistical evaluation.

Load-bearing premise

The entire absolute volume rests on one monocular depth estimate from one frame: if that depth or the focal length is off by a few percent, the volume error is roughly tripled, and the paper reports no test of this scaling across poses or objects.

Editorial extensions

If this is right

If the result holds on more objects, portion logging could be done from wearable camera footage with no special plate setup, scale reference, or bite counting.
The single-example error of about 7% is in a range that would matter for nutrition monitoring, so a larger multi-food evaluation is a direct next step.
Because scaling uses only one frame’s depth and focal length, the method could in principle work even when the object is heavily occluded by hands for most of the video.
The authors’ stated future direction of tracking the food relative to the hand could turn this from whole-item volume into per-bite portion estimates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The cubed scale factor means the method’s practical accuracy is dominated by the absolute accuracy of one monocular depth estimate: a 2% depth error would become roughly a 6% volume error, so robustness must be tested across camera distances and poses.
A natural testable extension is to film the same food at several distances with a known-size reference object in view and check whether the $D/f_x$ scaling reproduces the measured volumes.
The paper’s own note that BundleSDF struggles with rotationally invariant silhouettes suggests the method may fail selectively on round or symmetric foods, which are common in real diets.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the paper is worth reading, not because the 7.01% is established, but because it is an honest, clearly described first step toward a practical egocentric food-volume system. The numerical claim is a single trial and the scale factor is fragile, so treat the headline as a hypothesis.

The genuinely new thing is the specific pipeline: Aria egocentric video, super-resolution, zero-shot segmentation, Cutie masking, BundleSDF with LightGlue, and then a single-frame Depth-Pro estimate to convert mesh pixels to metric scale. Each component is published and the paper says so clearly. The 7.01% error is internally consistent with the measured 371±1 mL and estimated 345 mL. The authors also admit in Section 4 that this is one example. That honesty matters.

Where it is soft: first, the comparison to the 16.40% number from Rahman et al. is not a controlled benchmark. Different food, different capture protocol, and a single trial versus a mean absolute error. Second, the scale factor is load-bearing. Equation 8 sets R = D/fx, with both D and fx taken from Depth-Pro applied to one frame at the mask centroid. Volume multiplies by R^3. A 2% depth error is roughly a 6% volume error, which is about the size of the claimed improvement. The paper gives no validation of Depth-Pro on Aria pinhole images, no repeated depth estimates, and no error bars. Third, the abstract says the method is "robust to hand occlusions" and "flexible with varying camera and object poses", but the evidence is one rotating sandwich. That is an overstatement in proportion to the evidence.

None of this makes the paper wrong. The pipeline is plausible, the depth-scaling trick is an honest attempt to solve a real problem, and the limitation is stated. I am not convinced that 7.01% will hold across foods and hands, but it is a legitimate empirical result on one object.

This is a workshop-level contribution, useful to people working on dietary monitoring and egocentric 3D reconstruction. I would not cite it as evidence of accuracy, but I would cite it as a proof-of-concept. I would send it to peer review rather than desk-reject, because the method is clearly presented and the authors do not overclaim in the main text. A good referee would ask for a larger dataset and an ablation of the Depth-Pro scale.

Referee Report

3 major / 6 minor

Summary. The paper proposes FoodTrack, a pipeline for estimating the volume of handheld food items from egocentric video captured with Project Aria glasses. The pipeline combines super-resolution (ResShift), prompt-based segmentation (Grounded SAM), video object segmentation (Cutie), depth estimation (ChronoDepth), and BundleSDF-based 3D reconstruction, with LightGlue replacing BundleSDF's default matcher. The reconstructed mesh is projected into pixel space and scaled to metric units using a scale factor R = D/fx, where D is a single Depth-Pro depth estimate at the mask centroid and fx is the focal length. The reported preliminary result is a 7.01% absolute percentage error on one sandwich (estimated 345 mL versus measured 371 ± 1 mL), which the authors compare favorably with the 16.40% MAPE of Rahman et al. [1]. The paper explicitly acknowledges that the analysis is based on a single example.

Significance. If the reported accuracy were robust, the approach would be a useful contribution because it avoids gesture-based bite-size assumptions and does not require fixed camera angles or multiple viewpoints. The scale estimation is not circular: the depth-based scale factor comes from an external monocular depth model and focal length, not from fitting to the ground-truth volume, which is a strength. The significance is currently limited by three factors. First, the headline 7.01% figure rests on a single trial with no error bars or repeated measurements. Second, the metric scale depends on a single, unvalidated monocular depth estimate, and the volume scales as the cube of that depth, so the accuracy claim is fragile. Third, the comparison to Rahman et al. [1] is not controlled, involving a different object, recording protocol, and metric (single error versus a mean). The novelty is mostly an engineering combination of existing components, and the paper does not yet provide enough evidence to establish a general capability.

major comments (3)

[Section 4, Eq. (9)] The headline result is based on a single sandwich trial. The 7.01% absolute percentage error is the difference between one estimated volume (345 mL) and one measured volume (371 ± 1 mL) on a single 15-second video. There are no repeated recordings, no multiple objects, and no uncertainty propagation from the segmentation, depth, or mesh reconstruction stages. The paper itself admits this limitation in Section 4, but the abstract and introduction nevertheless state the method 'achieves' 7.01% and 'improves upon' the prior 16.40% MAPE. As written, the result is a single anecdote and cannot support a general accuracy claim.
[Section 2, Eq. (8)] The metric scale is determined entirely by a single Depth-Pro depth estimate D taken at the mask centroid of the last frame used to construct the mesh, together with the focal length fx, and the mesh volume is multiplied by (D/fx)^3. The paper provides no validation that this D is accurate for the Aria pinhole frames, and a 2% depth error translates to roughly a 6% volume error, which is comparable to the claimed 7.01% accuracy. Because the sandwich has non-negligible thickness and is filmed close-up, a single centroid depth may not represent the whole object. The manuscript should include a sensitivity analysis (e.g., perturbing D by ±2–5%) and a cross-check of the estimated depth-scale against the known physical dimensions of the sandwich; without that, the 7.01% number cannot be considered reliable.
[Section 4, comparison to [1]] The comparison to Rahman et al. [1] is not controlled. The 16.40% MAPE comes from a different food object, different camera and lighting setup, and a different evaluation protocol (mean over trials versus a single error on one trial). The claim that FoodTrack 'improves upon' the previous approach is therefore not supported by the presented data. A fair comparison would require either applying the baseline to the same or comparable data, or reporting multiple trials of the proposed method and computing a MAPE under matched conditions.

minor comments (6)

[Abstract and Section 1] The abstract and introduction use 'absolute percentage loss' while Section 4 uses 'absolute percentage error'; please standardize the terminology to 'absolute percentage error'.
[Abstract] There is a missing space in 'a16.40%'; it should read 'a 16.40%'.
[Section 2, Eq. (3)] The camera intrinsic matrix K is written as a 4x4 matrix, but a pinhole intrinsic matrix is conventionally 3x3. Please clarify the indexing and, if K is augmented for homogeneous coordinates, explain how the projection matrix is derived.
[Section 2, Eq. (5)] The normalization step is described in words but the formula uses S1 and the relation between 'same percent width' and the variables wOP and wIP is not fully explicit. Please define S1 and state the exact scaling factor.
[Section 2, Eqs. (6)–(7)] The variable L is used in the transformation to 'un-normalized pixel space' but is not defined; presumably it is the square image side length in pixels. Please define it explicitly.
[General] The paper does not state whether the code, the captured Aria video, or the reconstructed mesh will be released. For reproducibility, please include a data/code availability statement.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the metric volume estimate rests on an external monocular depth model, not on fitted ground-truth values.

full rationale

The paper's derivation chain is self-contained with respect to its target output. The volume estimate is computed by first reconstructing a mesh in pixel coordinates via BundleSDF, then converting to metric volume using Eq. (8), R = D/fx, where D and fx come from the external Depth-Pro model applied to the last image. No parameter is fitted to the measured sandwich volume of 371 ± 1 mL; the ground-truth volume is used only for evaluation in Eq. (9). The scale factor is therefore an independent external measurement, not a quantity derived from the target. The comparison to Rahman et al. is not circular either, although it is a single uncontrolled trial with a different object and protocol, which is a correctness and generalizability concern rather than a circularity concern. The paper's own admission that the analysis is based on a single example further confirms that no hidden fitting to the target is being claimed. No self-citations appear in the argument, and no known result is merely renamed. The main fragility is the cubed sensitivity of (D/fx)^3 to depth error, but that is an accuracy risk, not a circular derivation.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central volume estimate rests on the accuracy of external models (segmentation, depth, reconstruction) and on the pinhole projection assumptions. No parameters are fitted to the target volume, but the single-frame depth choice is a hand-picked reference.

free parameters (1)

reference frame for depth scaling = last image used to construct the mesh
Depth D and focal length fx are taken from a single chosen frame; choosing another frame would change the scale and thus the volume estimate.

assumptions (5)

domain assumption The transformed Aria video can be modeled by a pinhole camera with known intrinsics K.
Equations (2)-(4) rely on pinhole projection to map the mesh to the image plane.
domain assumption Depth-Pro provides an accurate metric depth D and focal length fx for the object at the mask centroid.
The volume scale is computed as R = D/fx and applied as R^3; any error in D or fx is cubed in the volume.
domain assumption BundleSDF with super-resolution and LightGlue produces a mesh whose shape is faithful to the true object up to scale.
The method rescales the reconstructed mesh in x/y pixels and z by the same factor, so shape distortion directly biases volume.
domain assumption Water displacement is a valid ground-truth measurement of the sandwich volume.
The 371 mL reference value used in Eq. (9) is obtained by water displacement.
domain assumption The super-resolution step does not change the geometric proportions of the object in pixel space.
The pixel-width normalization in Eq. (5) assumes the RGB image dimensions reflect the true projection.

how reviews work

0 comments

Cite this review

Pith. "Pith review of FoodTrack: Estimating Handheld Food Portions with Egocentric Video." pith.science (2026). https://pith.science/paper/L2Q46IMK

@misc{pith2026250504055,
  author       = {Pith},
  title        = {Pith review of: FoodTrack: Estimating Handheld Food Portions with Egocentric Video},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L2Q46IMK}},
  note         = {Machine review of arXiv:2505.04055}
}

read the original abstract

Accurately tracking food consumption is crucial for nutrition and health monitoring. Traditional approaches typically require specific camera angles, non-occluded images, or rely on gesture recognition to estimate intake, making assumptions about bite size rather than directly measuring food volume. We propose the FoodTrack framework for tracking and measuring the volume of hand-held food items using egocentric video which is robust to hand occlusions and flexible with varying camera and object poses. FoodTrack estimates food volume directly, without relying on intake gestures or fixed assumptions about bite size, offering a more accurate and adaptable solution for tracking food consumption. We achieve absolute percentage loss of approximately 7.01% on a handheld food object, improving upon a previous approach that achieved a 16.40% mean absolute percentage error in its best case, under less flexible conditions.

Figures

Figures reproduced from arXiv: 2505.04055 by the authors.

**Figure 3.** RGB image of an egocentric scene with two visible [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗

**Figure 4.** Binary mask of sandwich generated by Cutie from Fig. [PITH_FULL_IMAGE:figures/full_fig_p002_4.png] view at source ↗

Figures from the paper (3 more)

**Figure 5.** Figure 5: Depth image generated by ChronoDepth from Fig. [PITH_FULL_IMAGE:figures/full_fig_p002_5.png]

**Figure 6.** Figure 6: Processing pipeline for portion estimation. [PITH_FULL_IMAGE:figures/full_fig_p003_6.png]

**Figure 7.** Figure 7: Example of projected and scaled mesh beside its respec [PITH_FULL_IMAGE:figures/full_fig_p003_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 10 canonical work pages

[1]

A comparative anal- ysis of sensor-, geometry-, and neural-based methods for food volume estimation

Lubnaa Abdur Rahman, Ioannis Papathanail, Lorenzo Brigato, and Stavroula Mougiakakou. A comparative anal- ysis of sensor-, geometry-, and neural-based methods for food volume estimation. In Proceedings of the 8th Interna- tional Workshop on Multimedia Assisted Dietary Manage- ment, pages 21–29, 2023. 1, 4

work page 2023
[2]

Depth pro: Sharp monocular metric depth in less than a second

Aleksei Bochkovskii, Ama ¨el Delaunoy, Hugo Germain, Marcel Santos, Yichao Zhou, Stephan R Richter, and Vladlen Koltun. Depth pro: Sharp monocular metric depth in less than a second. arXiv preprint arXiv:2410.02073, 2024. 3

arXiv 2024
[3]

Putting the object back into video object segmentation

Ho Kei Cheng, Seoung Wug Oh, Brian Price, Joon-Young Lee, and Alexander Schwing. Putting the object back into video object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3151–3161, 2024. 1

2024
[4]

Project aria: A new tool for egocentric multi-modal ai research

Jakob Engel, Kiran Somasundaram, Michael Goesele, Albert Sun, Alexander Gamino, Andrew Turner, Arjang Talattof, Arnie Yuan, Bilal Souti, Brighid Meredith, et al. Project aria: A new tool for egocentric multi-modal ai research. arXiv preprint arXiv:2308.13561, 2023. 1

arXiv 2023
[5]

Valida- tion of sensor-based food intake detection by multicamera video observation in an unconstrained environment

Muhammad Farooq, Abul Doulah, Jason Parton, Megan A McCrory, Janine A Higgins, and Edward Sazonov. Valida- tion of sensor-based food intake detection by multicamera video observation in an unconstrained environment. Nutri- ents, 11(3):609, 2019. 1

work page 2019
[6]

Automatic ingestion monitor: a novel wearable device for monitoring of ingestive behavior

Juan M Fontana, Muhammad Farooq, and Edward Sazonov. Automatic ingestion monitor: a novel wearable device for monitoring of ingestive behavior. IEEE Transactions on Biomedical Engineering, 61(6):1772–1779, 2014. 1

work page 2014
[7]

Lightglue: Local feature matching at light speed

Philipp Lindenberger, Paul-Edouard Sarlin, and Marc Polle- feys. Lightglue: Local feature matching at light speed. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17627–17638, 2023. 2

2023
[8]

A novel mobile structured light system in food 3d reconstruc- tion and volume estimation

Sepehr Makhsous, Hashem M Mohammad, Jeannette M Schenk, Alexander V Mamishev, and Alan R Kristal. A novel mobile structured light system in food 3d reconstruc- tion and volume estimation. Sensors, 19(3):564, 2019. 1

work page 2019

Show all 16 references

[9]

Traditional self- reported dietary instruments are prone to inaccuracies and new approaches are needed

Michele N Ravelli and Dale A Schoeller. Traditional self- reported dietary instruments are prone to inaccuracies and new approaches are needed. Frontiers in nutrition , 7:90,
[10]

Grounded sam: Assembling open-world models for diverse visual tasks

Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159 ,

arXiv
[11]

Learning tem- porally consistent video depth from video diffusion priors

Jiahao Shao, Yuanbo Yang, Hongyu Zhou, Youmin Zhang, Yujun Shen, Matteo Poggi, and Yiyi Liao. Learning tem- porally consistent video depth from video diffusion priors. arXiv preprint arXiv:2406.01493, 2024. 1

2024 arXiv
[12]

A new video dataset for rec- ognizing intake gestures in a cafeteria setting

Zeyu Tang and Adam Hoover. A new video dataset for rec- ognizing intake gestures in a cafeteria setting. In 2022 26th International Conference on Pattern Recognition (ICPR) , pages 4399–4405. IEEE, 2022. 1

2022
[13]

Eating activ- ity monitoring in home environments using smartphone- based video recordings

Chunzhuo Wang, T Sunil Kumar, Gilles Markvoort, J ´er´emy Caby, Hans Hallez, and Bart Vanrumste. Eating activ- ity monitoring in home environments using smartphone- based video recordings. In 2022 International Conference on Digital Image Computing: Techniques and Applications...

2022
[14]

Eat-radar: Con- tinuous fine-grained intake gesture detection using fmcw radar and 3d temporal convolutional network with attention

Chunzhuo Wang, T Sunil Kumar, Walter De Raedt, Guido Camps, Hans Hallez, and Bart Vanrumste. Eat-radar: Con- tinuous fine-grained intake gesture detection using fmcw radar and 3d temporal convolutional network with attention. IEEE Journal of Biomedical and Health Informatics , 2023. 1

2023
[15]

Bundlesdf: Neural 6-dof tracking and 3d reconstruction of unknown objects

Bowen Wen, Jonathan Tremblay, Valts Blukis, Stephen Tyree, Thomas M ¨uller, Alex Evans, Dieter Fox, Jan Kautz, and Stan Birchfield. Bundlesdf: Neural 6-dof tracking and 3d reconstruction of unknown objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patte...

2023
[16]

Resshift: Efficient diffusion model for image super- resolution by residual shifting

Zongsheng Yue, Jianyi Wang, and Chen Change Loy. Resshift: Efficient diffusion model for image super- resolution by residual shifting. Advances in Neural Infor- mation Processing Systems, 36, 2024. 1

2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

A comparative anal- ysis of sensor-, geometry-, and neural-based methods for food volume estimation

Lubnaa Abdur Rahman, Ioannis Papathanail, Lorenzo Brigato, and Stavroula Mougiakakou. A comparative anal- ysis of sensor-, geometry-, and neural-based methods for food volume estimation. In Proceedings of the 8th Interna- tional Workshop on Multimedia Assisted Dietary Manage- ment, pages 21–29, 2023. 1, 4

work page 2023

[2] [2]

Depth pro: Sharp monocular metric depth in less than a second

Aleksei Bochkovskii, Ama ¨el Delaunoy, Hugo Germain, Marcel Santos, Yichao Zhou, Stephan R Richter, and Vladlen Koltun. Depth pro: Sharp monocular metric depth in less than a second. arXiv preprint arXiv:2410.02073, 2024. 3

arXiv 2024

[3] [3]

Putting the object back into video object segmentation

Ho Kei Cheng, Seoung Wug Oh, Brian Price, Joon-Young Lee, and Alexander Schwing. Putting the object back into video object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3151–3161, 2024. 1

2024

[4] [4]

Project aria: A new tool for egocentric multi-modal ai research

Jakob Engel, Kiran Somasundaram, Michael Goesele, Albert Sun, Alexander Gamino, Andrew Turner, Arjang Talattof, Arnie Yuan, Bilal Souti, Brighid Meredith, et al. Project aria: A new tool for egocentric multi-modal ai research. arXiv preprint arXiv:2308.13561, 2023. 1

arXiv 2023

[5] [5]

Valida- tion of sensor-based food intake detection by multicamera video observation in an unconstrained environment

Muhammad Farooq, Abul Doulah, Jason Parton, Megan A McCrory, Janine A Higgins, and Edward Sazonov. Valida- tion of sensor-based food intake detection by multicamera video observation in an unconstrained environment. Nutri- ents, 11(3):609, 2019. 1

work page 2019

[6] [6]

Automatic ingestion monitor: a novel wearable device for monitoring of ingestive behavior

Juan M Fontana, Muhammad Farooq, and Edward Sazonov. Automatic ingestion monitor: a novel wearable device for monitoring of ingestive behavior. IEEE Transactions on Biomedical Engineering, 61(6):1772–1779, 2014. 1

work page 2014

[7] [7]

Lightglue: Local feature matching at light speed

Philipp Lindenberger, Paul-Edouard Sarlin, and Marc Polle- feys. Lightglue: Local feature matching at light speed. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17627–17638, 2023. 2

2023

[8] [8]

A novel mobile structured light system in food 3d reconstruc- tion and volume estimation

Sepehr Makhsous, Hashem M Mohammad, Jeannette M Schenk, Alexander V Mamishev, and Alan R Kristal. A novel mobile structured light system in food 3d reconstruc- tion and volume estimation. Sensors, 19(3):564, 2019. 1

work page 2019