OmniFood-Bench: Evaluating VLMs for Nutrient Reasoning and Personalized Health Advice

Jingpu Yang; Miao Fang; Qian Jiang; Zhecheng Shi; Zirui Song

VLMs name dishes well but fail at mass estimation and often give unsafe diet advice.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 07:40 UTC pith:2V2DFG6T

load-bearing objection Useful hierarchical food-VLM benchmark with clear MAPE numbers; the safety-hallucination claim rests on unvalidated nutrient-to-advice labels. the 2 major comments →

arxiv 2607.08423 v2 pith:2V2DFG6T submitted 2026-07-09 cs.AI

OmniFood-Bench: Evaluating VLMs for Nutrient Reasoning and Personalized Health Advice

Qian Jiang , Zhecheng Shi , Jingpu Yang , Zirui Song , Miao Fang This is my paper

classification cs.AI

keywords Vision-Language ModelsFood ComputingNutritional ReasoningPortion Size EstimationSafety AlignmentDietary AdviceBenchmark

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that vision-language models used for food and diet advice suffer from a systemic information asymmetry: they can see a plate but cannot reliably recover the physical mass or chemical composition that determine real nutritional risk. Existing food benchmarks stop at dish naming or recipe retrieval and therefore never test the full chain a dietitian must perform. OmniFood-Bench closes that gap by scoring models on three progressive stages—ingredient and cooking-method recognition, gram-level portion and macronutrient estimation, and disease-specific intake advice—using a curated subset of real-world meal images. Across six leading models the authors find a clear Semantic-Physical Gap: semantic recognition is strong, yet mass estimation errors routinely exceed 50 percent and safety advice for diabetes, kidney disease and hyperlipidemia is frequently wrong or overly permissive. The result is offered as a concrete trustworthiness standard for any autonomous agent that might be deployed as a personal health assistant.

Core claim

State-of-the-art vision-language models close the gap on naming dishes and ingredients yet exhibit catastrophic failure on mass estimation and frequently hallucinate benign recommendations for high-risk clinical profiles; the authors term this the Semantic-Physical Gap and show it is the dominant bottleneck for safe dietary agents.

What carries the argument

OmniFood-Bench: a three-layer hierarchical evaluation that forces models to traverse from basic perception (ingredients, cooking method) through quantitative estimation (portion size and nutritional profile, scored by MAPE) to safety-critical advisory (three-class intake decisions for diabetes, obesity, lipids and kidney disease).

Load-bearing premise

The three-class safety labels for each disease can be correctly derived solely from the protein, fat and carbohydrate gram weights already in the source dataset by applying fixed numerical ranges drawn from 'authoritative standards,' without further clinical validation.

What would settle it

Re-annotate a random subset of the 1,208 images with independent clinical dietitians using published nutrient cut-offs for each disease; if the majority of model-generated advice then matches the new human labels, the claimed safety gap shrinks or disappears.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Useful hierarchical food-VLM benchmark with clear MAPE numbers; the safety-hallucination claim rests on unvalidated nutrient-to-advice labels.

read the letter

The real addition here is the progressive chain: ingredients/cooking → portion mass + macros (MAPE) → three-class disease advice. That is new relative to Food-101, Nutrition5k and the MM-Food-100K source. Tables I–II give clean numbers across four domains and six models; the MAPE collapse with ingredient variety (Fig. 4) and the glazed-pork case study make the “Semantic-Physical Gap” concrete. Open models are competitive on perception, everyone fails hard on mass. Code and data are promised, which is good.

The soft spot is exactly the one the stress-test flags. Section III.A says the Normal/Controlled/Avoid labels for diabetes, lipids, kidney, obesity were produced by applying “authoritative health and hygiene standards” to the existing protein/fat/carb grams, with “precise numerical ranges.” No cut-offs, no ADA/KDIGO citations, no clinician check, no inter-rater numbers. Table III accuracies (26–46 %) and the “safety hallucination” diagnosis therefore score models against an unvalidated proxy. The mass/nutrient MAPE results stand on their own; the medical-advice half of the headline claim does not until those thresholds are published and justified. Secondary issues: proprietary models run on only 496 of 1 208 samples with selection criteria unstated, and no error bars or significance tests.

No circular math, no invented physics—just an empirical resource with one load-bearing annotation gap. Useful for anyone building or auditing food VLMs; less so for clinical AI until the labels are fixed. I would send it to referees with a clear request for the exact ranges and a clinical validation note. Worth engaging once that is addressed.

Referee Report

2 major / 0 minor

Summary. The paper introduces OmniFood-Bench, a hierarchical benchmark built from a curated 1,208-sample subset of MM-Food-100K. It evaluates six VLMs (gpt-5.1, gemini-3-flash, claude-sonnet-4, qwen3-vl-8B, InternVL3.5-8B, Llama-3.2-11B-Vision) on three progressive tasks: Basic Perception (cooking-method accuracy and ingredient match rate), Quantitative Estimation (MAPE for portion size and macronutrient grams), and Safety-Critical Advisory (3-class Normal/Controlled/Avoid recommendations for lipids, obesity, kidney disease, and diabetes). Results show relatively strong semantic recognition (especially on raw produce) but high MAPE on mass/nutrients (often >50%, up to 185–346% on raw items) and low advisory accuracy (roughly 20–46%). The authors diagnose a “Semantic-Physical Gap” and “Dense Quantitative Failure,” illustrated by a Sweet-and-Sour-Pork case study, and argue that current VLMs are not yet trustworthy for dietary health agents.

Significance. If the reported numbers hold, the work supplies a useful, multi-stage stress test that goes beyond Food-101-style classification and laboratory Nutrition5k settings. The progressive pipeline (perception → mass/nutrient regression → disease-specific advice) and the public release of code/data make the failure modes of mass estimation and multi-component scenes concrete and reproducible. The MAPE tables and complexity-vs-accuracy plots (Fig. 4) are independently valuable for the food-computing and VLM communities even if the clinical-label construction is later refined. The safety-advisory results, however, currently rest on an unvalidated proxy and therefore cannot yet be treated as a definitive measure of biomedical risk.

major comments (2)

Section III.A and Task III: the three-class safety labels (Normal / Controlled / Avoid) for diabetes, obesity, hyperlipidemia and kidney disease are stated to be derived solely by applying “authoritative health and hygiene standards” and “precise numerical ranges” to the protein/fat/carbohydrate gram weights already present in MM-Food-100K. No cut-off values, no cited clinical guidelines (ADA, KDIGO, etc.), no inter-rater reliability, and no clinician review are reported. Because Table III and the “Safety Hallucinations” claim treat these labels as ground truth, the low accuracies (e.g., 26–41 % on Diabetes) and the headline safety conclusion are not yet medically grounded. Either publish the exact thresholds with literature support or re-label a clinically reviewed subset; otherwise the safety half of the Semantic-Physical Gap claim remains an unvalidated proxy.
Section III.A and Experimental Setup: proprietary models are evaluated on only 496 of the 1,208 samples while open-source models use the full set. The selection criteria for the 496-sample subset are never stated, nor are any statistical tests or confidence intervals supplied for the accuracy/MAPE differences across models or domains (Tables I–III). Without transparency on subset construction and without error bars, claims that one model “achieves the highest accuracy” or that open-source models are “closing the gap” cannot be assessed for robustness.

Circularity Check

0 steps flagged

No circularity: purely empirical VLM evaluation against fixed external ground truth from MM-Food-100K; no derivation, fit, or self-referential claim reduces to its inputs.

full rationale

OmniFood-Bench is a zero-shot evaluation benchmark, not a first-principles derivation or predictive model. Ground-truth portion weights, macronutrient grams, cooking methods, and ingredients are taken directly from the independent MM-Food-100K source (Section III.A). The three-class safety labels (Normal/Controlled/Avoid) are constructed by applying fixed numerical ranges to those same grams; models are then scored by classification accuracy against the resulting labels (Tables I–III, Section IV). No parameter is fitted to any subset of the evaluation data and then re-presented as a prediction. There are no uniqueness theorems, ansatzes imported via self-citation, or self-definitional loops. Framing phrases such as “Semantic-Physical Gap” are post-hoc descriptive labels for the observed accuracy drop-offs; they do not enter the scoring equations. The evaluation chain is therefore self-contained against an external dataset and exhibits zero circularity under the stated criteria.

Axiom & Free-Parameter Ledger

1 free parameters · 3 axioms · 2 invented entities

Benchmark paper rests on domain assumptions about food nutrition and clinical mapping rather than free parameters or new physical entities. The coined terms package empirical observations; the main unstated load is the mapping from nutrient grams to clinical intake classes.

free parameters (1)

nutrient-to-intake-class thresholds
Exact gram cut-offs that map protein/fat/carb weights onto Normal/Controlled/Avoid for each disease are never listed or justified numerically; they are treated as given by 'authoritative standards'.

axioms (3)

domain assumption Visual appearance of prepared food is systematically asymmetric to its intrinsic nutritional composition (Systemic Information Asymmetry).
Stated in the introduction as the motivating premise for the entire hierarchical evaluation.
ad hoc to paper Nutrient gram weights already present in MM-Food-100K, when thresholded, correctly encode clinical safety for diabetes, obesity, hyperlipidemia and kidney disease.
Section III.A constructs the three-class labels this way; no external clinical source or validation is supplied.
domain assumption Zero-shot prompting with a standardized (but unpublished) template is a fair test of real-world dietary-agent capability.
Experimental setup (Section IV.A) treats zero-shot accuracy/MAPE as the primary measure.

invented entities (2)

Semantic-Physical Gap no independent evidence
purpose: Label the observed discrepancy between high semantic recognition accuracy and poor mass/nutrient estimation plus unsafe advice.
Coined for the paper’s findings; supported by the reported tables but not independently measured outside this evaluation suite.
Dense Quantitative Failure no independent evidence
purpose: Name the sharp drop in portion-estimation accuracy as the number of distinct components rises.
Descriptive label for the trend in Fig. 4; no external falsifiable handle.

pith-pipeline@v1.1.0-grok45 · 15256 in / 2571 out tokens · 51545 ms · 2026-07-10T07:40:56.339273+00:00 · methodology

0 comments

read the original abstract

The rapid integration of Large Vision-Language Models (VLMs) into critical infrastructure promises to revolutionize personalized healthcare and dietary management. However, in the domain of food systems, autonomous agents face a unique and persistent challenge: the "Systemic Information Asymmetry" between visual appearance and intrinsic nutritional composition. Existing benchmarks primarily focus on coarse-grained classification tasks, such as food category recognition, which fail to evaluate the intricate reasoning chain required for real-world dietary management -- specifically, the ability to traverse from identifying hidden ingredients to estimating physical mass, and finally synthesizing safety-critical medical advice. In this paper, we introduce OmniFood-Bench, a comprehensive benchmark constructed from the MM-Food-100K dataset. Unlike previous works, OmniFood-Bench evaluates VLMs across three progressive capabilities: Basic Perception (Ingredients & Cooking Methods), Quantitative Reasoning (Portion Size & Nutritional Profiling), and Safety-Critical Advisory (Disease-Specific Recommendations). We evaluate six state-of-the-art VLMs, including gpt-5.1, gemini-3-flash, and qwen3-vl-8B. Our extensive experiments reveal a startling "Semantic-Physical Gap": while models achieve near-human accuracy in naming dishes, they exhibit catastrophic failure in mass estimation and frequently hallucinate benign advice for high-risk diabetic profiles. This work establishes a rigorous standard for trustworthiness in autonomous agents deployed for public health. The code and datasets are available in: https://anonymous.4open.science/r/OmniFood-Bench-7D0B