{"id":"f5eb995b-e1a8-45a7-aa35-afd65c32ed85","arxiv_id":"2505.08747","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper introduces the FastFood dataset and a visual-ingredient feature fusion method that substantially improves automatic nutrition estimation from food images.","lead":"A new fast-food image dataset with 84,446 images and per-category nutrition labels is introduced, together with a method that fuses visual features with ingredient information to predict calories, fat, carbs, and protein. The approach lowers prediction error by roughly a third to a half compared to image-only baselines on both the new dataset and the existing Nutrition5k benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"FastFood's image-level 70/20/10 split keeps every category in both train and test, so the 48% MAE gain may reflect category memorization rather than ingredient-based nutrition estimation; a category-disjoint split is the decisive check.","rationale":"The reader's weakest assumption (portion-size mismatch) is plausible but secondary: official fast-food portions are relatively standardized, and the authors report manual filtering, so label noise is likely bounded. The more decisive issue is the evaluation structure. With category-level labels, an image-level split tests recognition-memorization rather than generalization to unseen dishes. Since the method's test-time pipeline feeds LMM-predicted ingredients into the model, it has an especially direct route to category identity; the visual baselines can also exploit category overlap, but the relative claim that ingredient fusion helps needs to be separated from knowing the category helps, and a category-disjoint split does exactly that. It is a single, feasible experiment that settles whether the central number is an artifact. If the category-disjoint result still shows a large improvement, the paper's claim is substantially strengthened. I therefore keep the conditional verdict rather than moving to rejection; the concern is a required condition, not a demonstrated failure.","tokens_in":15707,"tokens_out":8365,"duration_ms":97077,"concrete_test":"Run a category-disjoint version of Table 1: assign whole FastFood categories to train/validation/test (70/20/10 of the 908 categories) so no category appears in more than one split, retrain ResNet101 and ResNet101+VIF2 with the same hyperparameters, and report caloric/fat/carb/protein MAE. If the 48.1% caloric-MAE reduction collapses or becomes small, the headline improvement is an artifact of category overlap rather than ingredient-feature fusion; if it survives, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that ingredient-feature fusion significantly improves nutrition prediction, with Table 1's reduction from 118.04 to 61.26 kcal MAE as the headline result. The FastFood evaluation does not yet establish that this gain reflects nutrition estimation rather than category lookup. Section 3 assigns each fast-food item a single official nutrition label and then splits the expanded image set 70/20/10 by image, not by category. Under that split, essentially every category appears in both training and test, so the model can memorize a label per dish and recognize the dish at test time. Ingredient features, especially the LMM-predicted ingredients used at inference, are a very strong cue for exactly this category identity. A random image-level split also allows near-duplicate crawled images to cross the train/test boundary, making the absolute MAE and the 48.1% relative improvement optimistic. On Nutrition5k, the stronger Protocol 2 similarly selects the 'optimal frame per video' post hoc, an oracle procedure that does not reflect a deployable estimator. Until the FastFood comparison is repeated with disjoint categories, and the frame-selection rule is fixed without test-set information, the paper's central claim is conditional rather than demonstrated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents FastFood, a dataset of 84,446 images across 908 fast-food categories with per-category ingredient and nutrition annotations, and proposes VIF2, a model-agnostic method that fuses visual features with ingredient features for nutrition regression. Ingredients are embedded with the CLIP text encoder and projected into the visual feature space; during training the ingredient lists are perturbed by synonym replacement and sampling, and during testing ingredients are predicted by LMMs with augmentation and majority voting. Experiments compare ResNet, InceptionV3, and ViT backbones with and without VIF2 on FastFood and Nutrition5k, and include ablations on fusion layer, threshold, LMM choice, and ground-truth ingredients.","tokens_in":15940,"tokens_out":6105,"duration_ms":55244,"significance":"The dataset is a potentially useful resource, and the method is cleanly described and model-agnostic. The paper does several things right: it ships an internal control experiment with ground-truth ingredients (Table 3) to bound the effect of ingredient prediction, it evaluates across four backbones and two datasets, and the fusion mechanism is simple enough to reproduce. If the FastFood comparison were run under a category-disjoint split and the Nutrition5k oracle frame selection were removed or relabeled, the evidence would be substantially strengthened. As it stands, the central quantitative claim is plausible but not yet demonstrated.","major_comments":[{"comment":"The FastFood 70/20/10 split is performed over images rather than over the 908 food categories, and each category carries a single nutrition label. Under this split, essentially every category appears in both training and test, so a model can memorize one label per dish and recognize the dish at test time; ingredient features, especially the LMM-predicted ingredients used at inference, are a very strong cue for exactly this category identity. The 48.1% caloric-MAE reduction therefore does not yet establish that VIF2 improves nutrition estimation rather than category lookup. Please re-run the FastFood experiments with a category-disjoint split, and additionally verify that near-duplicate crawled images do not cross the train/test boundary.","section":"Section 3, Table 1"},{"comment":"Selecting the 'optimal frame per video' after observing the test video is an oracle procedure that uses test-set information; the 23.32 kcal caloric MAE and any state-of-the-art comparisons built on it are not a deployable result. Please either replace Protocol 2 with a fixed frame-selection rule, such as a sharpness heuristic chosen on validation data, or report it explicitly as an oracle upper bound, and remove claims of state-of-the-art performance that depend on it.","section":"Section 5.1, Testing Protocol 2"},{"comment":"All results are single runs without error bars or significance tests, so the word 'significantly' in the conclusion is not supported statistically. Please report means and standard deviations over at least three seeds, or equivalent paired tests, for the main comparisons.","section":"Tables 1-3"}],"minor_comments":[{"comment":"The y-axis of Figure 9 shows an average MAE of about 19.37 at threshold 4, while the text says the lowest value is 61.26; 61.26 is the caloric MAE from Table 1, not the plotted average. Please correct the statement or the figure caption.","section":"Section 5.4, Figure 9"},{"comment":"The sentence 'For the other five models' should read 'the other three models', since only ResNet50, ResNet101, InceptionV3, and ViT are evaluated.","section":"Section 5.1, Implementation details"},{"comment":"The label 'Testing Protocol 1' is repeated for the VIF2 block; merge the headings or relabel the second block to avoid confusion.","section":"Table 2"},{"comment":"The sentence 'Since the evaluation protocol for these methods is unclear, we do not include them in the comparison tables. Under a fair comparison...' is self-contradictory; specify exactly which prior methods are compared and under which protocol.","section":"Section 5.2"},{"comment":"The caption says 'large language model (LMM)', but the method uses a large multimodal model; please correct the terminology.","section":"Figure 7 caption"}],"recommendation":"major_revision","confidential_remarks":"The category-disjoint FastFood split is the decisive check for the headline claim. If the VIF2 advantage collapses when categories are held out, the paper currently does not support its central conclusion. Please request this experiment before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Didn't have to get past page one to know the dataset is the real news. FastFood—84k images, 908 categories, official nutrition labels, and semi-automatic ingredient annotations—is a genuinely useful addition to food computing. The manual filtering of crawled images to match portion sizes is more careful than most web-scraped efforts. The VIF2 fusion is simple and model-agnostic: average CLIP text embeddings of ingredients, project into the visual space, and add them to the feature map (or prepend a token in ViT). It's a sensible idea, and they test it across four backbones and two datasets. The ground-truth ingredient experiment (Table 3) is a nice touch—it gives an upper bound on what the fusion can do.\n\nThe problem is that the central quantitative claim isn't supported by the evaluation as written. The FastFood split is by image, not by category: 70/20/10 on the 84k images means every category appears in both training and test. With a single nutrition label per category, the model can simply memorize the label for each dish and then recognize the dish at test time. Ingredient features, especially LMM-predicted ingredient lists, are a strong cue for exactly that dish identity. So the 118.04 to 61.26 kcal MAE drop is likely part real estimation and part category lookup, and we don't know the split. A category-disjoint split is the decisive experiment, and it's missing.\n\nNutrition5k's Protocol 2 has a related issue: the 'optimal frame per video' is selected after the fact, using the test set. That is an oracle, not a usable estimator, and it shouldn't be used to claim SOTA. The threshold tau for ingredient voting is picked off a performance curve, also without a clearly separate validation set. And there are no error bars anywhere.\n\nNone of this sinks the paper. The dataset alone is worth the read, and the method is a reasonable baseline that others will compare against. But I'd treat the headline numbers as provisional until the split and frame-selection are fixed. A serious referee should ask for exactly those two things. I'd send it out rather than desk-reject.","headline":"The FastFood dataset is a genuinely useful new resource, but the headline 48% error reduction is not yet trustworthy because the train/test split is by image rather than category, and Nutrition5k's best numbers use an oracle frame selection.","tokens_in":16448,"tokens_out":2924,"would_cite":true,"duration_ms":29956,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Integrating ingredient names with visual features halves calorie-estimation error on a new 84,446-image fast-food dataset.","keywords":["nutrition estimation","visual-ingredient feature fusion","fast food dataset","ingredient recognition","multi-task regression","large multimodal models","food computing","mean absolute error"],"falsifier":"Measure calorie MAE on a held-out test set of visually identical menu items photographed at visibly different portion sizes than the official label, or swap in randomly sampled ingredient lists while keeping the image fixed; if the model's predictions barely respond to the ingredient changes or collapse toward category mean calories, the reported benefit of ingredient fusion is not robust.","tokens_in":1650,"feed_emoji":"🍔","tokens_out":1861,"duration_ms":72367,"temperature":0.7,"pith_summary":"This paper argues that nutrition values can be predicted much more accurately from a food image when the model is also told what ingredients the food contains. To support this, it introduces FastFood, a dataset of 84,446 images across 908 fast-food categories with standardized nutrition labels and semi-automatically curated ingredient lists. It then proposes a model-agnostic Visual-Ingredient Feature Fusion (VIF2) method that encodes ingredient names with a text encoder, projects them into visual feature space, and fuses them into the image representation used for regression. The headline result is that adding ingredient features to a ResNet101 baseline cuts caloric mean absolute error on FastFood from 118.04 to 61.26 kcal, a 48.1 percent reduction, with similar gains on fat, carbohydrate, and protein. If correct, this means ingredient information provides a compact prior that visual appearance alone lacks, offering a practical route to nutrition estimation from ordinary RGB images rather than depth sensors or precise portion estimation.","feed_headline":"Adding ingredients to food images cuts calorie error in half","feed_subtitle":"A new 84,446-image fast-food dataset and a fusion method drop caloric MAE from 118 to 61 kcal.","key_machinery":"The load-bearing object is the Visual-Ingredient Feature Fusion (VIF2) module. For each image, the ingredient list is converted to a single vector by averaging CLIP text-encoder embeddings of the ingredient names, then mapped with a linear layer and ReLU into the visual feature space. In convolutional backbones, this vector is broadcast and added channel-wise to an intermediate feature map — block2 for ResNet and the second pooling layer for InceptionV3; in ViT, it is inserted as an extra input token alongside the class and patch tokens. The fused representation feeds separate two-layer heads that regress calories, fat, carbohydrates, and protein under a summed MAE loss. At test time, an off-the-shelf large multimodal model generates ingredient lists from augmented views of each image, and majority voting above a threshold filters out hallucinated ingredients before the fusion step.","core_discovery":"The central claim is that ingredient-aware feature fusion substantially improves food nutrition prediction across different visual backbones. The paper builds VIF2, which embeds each ingredient name with a pre-trained CLIP text encoder, averages the embeddings, passes them through a linear ReLU projector, and fuses the resulting vector into the image features — channel-wise addition for convolutional networks and an extra token in the input sequence for ViT. Four task-specific heads then regress calories, fat, carbohydrates, and protein under a summed mean-absolute-error loss. On FastFood, ResNet101 + VIF2 reduces caloric MAE from 118.04 to 61.26 kcal (relative error from 29.84 percent to 15.49 percent), and on Nutrition5k it reduces caloric MAE from 103.03 to 83.75. When ground-truth ingredient lists are supplied at test time instead of LMM predictions, FastFood caloric MAE falls further to 44.71 kcal. The paper concludes that ingredient information is a decisive signal for nutrition estimation and that VIF2 works with ResNet, InceptionV3, and ViT backbones alike.","pith_inferences":["A natural stress test would hold out entire food categories during training: if the large FastFood gain persists only for categories whose ingredients were seen before, the fusion may be learning category-level nutrition priors rather than a general ingredient-to-nutrition mapping.","The ingredient embedding is a length-normalized average that treats every ingredient equally, so weighting ingredients by predicted amount or portion size is a testable extension that could further reduce error and would use the weight column already present in FastFood.","The same fusion recipe could be carried to cafeteria, restaurant, or home-cooked dishes wherever an LMM can supply ingredient lists, but the paper's experiments do not yet demonstrate that transfer beyond fast food and cafeteria data.","The reported failure on visually mixed foods such as a mac-and-cheese tray suggests that injecting ingredient features at only one intermediate layer may be insufficient for well-blended dishes; a multi-scale fusion variant is a concrete next experiment."],"forward_implications":["Adding ingredient names to a single RGB image substantially lowers error on all four nutrition targets; on FastFood, caloric MAE falls by 48.1 percent from 118.04 to 61.26 kcal.","The gain transfers to a different food domain: on Nutrition5k, ResNet101 + VIF2 reduces caloric MAE by 18.7 percent and carbohydrate MAE from 12.91 to 5.64.","VIF2 is model-agnostic: ResNet50, ResNet101, InceptionV3, and ViT all improve with ingredient fusion, so the module can be added to existing image encoders without redesigning them.","Better test-time ingredient predictions translate directly into better nutrition estimates, since ground-truth ingredients lower FastFood caloric MAE to 44.71 kcal; progress in ingredient recognition should therefore keep improving nutrition estimation.","FastFood provides a new training and evaluation resource with 84,446 images, 908 categories, standardized nutrition labels, and ingredient annotations, enabling future work on nutrition estimation without relying on depth data."],"supporting_citations":[{"why":"It supplies the Nutrition5k benchmark, the video-frame sampling protocol, and the per-nutrient regression-head design the paper adapts.","marker":"[39]"},{"why":"It provides the ResNet backbones used as image-only baselines and as the convolutional architectures into which ingredient features are fused.","marker":"[15]"},{"why":"It provides the InceptionV3 baseline and fusion target used for the convolutional comparison.","marker":"[37]"},{"why":"It provides the ViT architecture in which the ingredient vector is injected as an extra transformer token.","marker":"[8]"},{"why":"It supplies the LLaVA1.6 large multimodal model used at test time to generate ingredient lists from augmented images.","marker":"[23]"},{"why":"It is the DPF-Nutrition method whose monocular-image caloric MAE on Nutrition5k the paper compares itself against.","marker":"[14]"},{"why":"It is the RGB-D fusion nutrition method cited as a prior state-of-the-art comparator for the Nutrition5k results.","marker":"[35]"}],"fun_headline_variants":["Ingredient fusion halves calorie error in food images","Visual-ingredient feature fusion cuts calorie MAE to 61 kcal","New 84K-image dataset and fusion method improve nutrition AI","Ingredient-aware image model reduces calorie estimation error","Ingredient fusion aids nutrition prediction across backbones"],"cache_read_input_tokens":18688,"weakest_assumption_plain":"The paper assumes that every image in a given FastFood category shows the same portion size as the official brand nutrition label, so any real-world variation in portion size or product recipe enters the training and test data as label noise that manual filtering cannot fully remove.","fun_headline_variants_meta":{"raw":{"variants":["Ingredient fusion halves calorie error in food images","Visual-ingredient feature fusion cuts calorie MAE to 61 kcal","New 84K-image dataset and fusion method improve nutrition AI","Ingredient-aware image model reduces calorie estimation error","Ingredient fusion aids nutrition prediction across backbones"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001243,"raw_usage":{"total_tokens":5119,"prompt_tokens":982,"completion_tokens":4137,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":4059}},"tokens_in":598,"tokens_out":4137,"duration_ms":32144,"temperature":1.0,"reasoning_tokens":4059,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:46:25.842769+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure calorie MAE on a held-out test set of visually identical menu items photographed at visibly different portion sizes than the official label, or swap in randomly sampled ingredient lists while keeping the image fixed; if the model's predictions barely respond to the ingredient changes or collapse toward category mean calories, the reported benefit of ingredient fusion is not robust.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the Nutrition5k benchmark, the video-frame sampling protocol, and the per-nutrient regression-head design the paper adapts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the ResNet backbones used as image-only baselines and as the convolutional architectures into which ingredient features are fused."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the LLaVA1.6 large multimodal model used at test time to generate ingredient lists from augmented images."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It is the DPF-Nutrition method whose monocular-image caloric MAE on Nutrition5k the paper compares itself against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It is the RGB-D fusion nutrition method cited as a prior state-of-the-art comparator for the Nutrition5k results."}],"review_version":1}