{"id":"63cebf28-cfa8-4255-b5b7-bfec5bcb793f","arxiv_id":"2411.08715","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A retrieval-augmented LMM with stochastic retrieval sampling and self-consistency voting improves recipe generation from food images on Recipe1M.","lead":"This paper builds a system that turns a photo of food into a full recipe by pulling similar recipes from a database and feeding them to a vision-language model. It reports small but consistent gains over prior systems on the Recipe1M benchmark, though the evaluation has some methodological caveats.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training-time retrieval may leak the ground-truth recipe into the prompt; the SOTA claim depends on ruling this out.","rationale":"The reader's weakest_assumption concerns test-time retrieval relevance. My concern is more fundamental and upstream: during training the datastore contains the answer itself. If the retriever returns it, the model is trained with the ground truth in the prompt, which can artificially produce the reported gains and the claimed hallucination reduction. This is not a disagreement with the reader's concern; both are about the retrieval mechanism, but the self-retrieval leak is more directly checkable and, if present, invalidates the central claim. The paper gives no indication that the query item was excluded, and the supplementary qualitative examples do not resolve this because they show other recipes, not the top-ranked list. I recommend conditional acceptance pending the self-retrieval check.","tokens_in":27868,"tokens_out":4479,"duration_ms":43903,"concrete_test":"Sample 100 training images; for each, run the retriever against the full training datastore and check whether the image's own ground-truth recipe appears in the top-50 retrieved sets. If it does, retrain the generator with that recipe excluded from all retrieved candidates (and the datastore filtered for the query image), then re-evaluate on the same 5,000 test samples with the same S=11 voting. If the BLEU/SacreBLEU/ROUGE-L gap over LLaVA-FT shrinks materially or reverses, the central claim is invalidated.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (SOTA recipe generation, hallucination reduction) rests on retrieval augmentation being genuinely informative. In Section 4.1.2 the authors use the Recipe1M training set as the external datastore M, and Section 4.1.3 retrieves the top 50 recipes for each image during training. Yet the paper never states that the ground-truth recipe for a training image is excluded from the retrieved candidates. Since the retriever [57] was trained to match image-recipe pairs, the same image's own recipe is a likely top result. If it appears in the top-K, the model is trained with the answer in the context (\"Search results are only for referring...\"), which can trivially reduce hallucination and inflate BLEU/SacreBLEU/ROUGE-L. At test time the ground truth is not in the datastore, so the benefit disappears, and the reported SOTA numbers overstate the method's real contribution. The absence of any mention of self-recipe exclusion in Sections 3.2.1, 4.1.3, or the ablations makes this a concrete gap, not a style issue.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a retrieval-augmented large multimodal model for generating recipes from food images. It uses an off-the-shelf cross-modal retriever (Salvador et al. [57]) over the Recipe1M training set as a datastore. During training, Stochastic Diversified Retrieval Augmentation (SDRA) randomly samples three retrieved ingredient sets and one retrieved instruction set to insert into the prompt; during inference, Self-Consistency Ensemble Voting generates S candidate instruction sets using the top-S retrieved recipes and selects the candidate with the highest average pairwise cosine similarity. Experiments on a fixed 5,000-sample random subset of Recipe1M report BLEU 30.11, SacreBLEU 6.42, and ROUGE-L 38.93 for instruction generation, and IOU 33.62 / F1 50.32 for ingredient recognition, which the paper claims are state-of-the-art. The paper also presents ablations for the retrieval pool size K, the voting number S, and the way retrieved information is concatenated, together with qualitative examples and supplementary analyses of retrieval successes and failures.","tokens_in":28089,"tokens_out":6672,"duration_ms":55860,"significance":"If the reported numbers were obtained under a clean protocol, the paper would make a useful contribution: it shows that retrieval augmentation with a frozen LLaVA backbone and LoRA tuning improves recipe generation metrics, and that multi-candidate self-consistency voting yields consistent improvements as S grows. The internal ablations in Tables 3 and 5 are a genuine strength, and the supplementary analysis of retrieval failures is unusually candid and helpful. However, the headline SOTA claim is currently not supported because the experimental protocol has two load-bearing weaknesses: the training-time retrieval may include the ground-truth recipe of the training image, and the final configuration (K=50, S=11) is selected on the same 5,000-sample test subset that is later reported as the result. The baseline comparisons in Table 1 are also not apples-to-apples. If these protocol issues are fixed, the method itself remains plausible and worth publishing.","major_comments":[{"comment":"The training-time retrieval setup does not rule out leakage of the ground-truth recipe. Section 4.1.1 states that the Recipe1M training set is used as the external datastore M, and Section 4.1.3 states that during training the top 50 recipes are retrieved for each image. Because the datastore contains the exact image-recipe pair used as the training query, and because the retriever [57] was trained to rank image-recipe matches, the ground-truth recipe of the training image is a plausible top-50 candidate. The paper never states that this recipe is excluded from the retrieved set. If it is not excluded, the model is trained with a near-answer copy in the prompt, which can inflate BLEU/SacreBLEU/ROUGE-L and reduce hallucination without any real retrieval benefit; at test time the ground truth is not in the datastore, so the reported SOTA gap would be an artifact of this asymmetry. Please report the fraction of training queries for which the ground-truth recipe appears in the top-50, and re-run training with the ground-truth recipe removed from the datastore for each query.","section":"4.1.1, 4.1.3"},{"comment":"The final configuration is selected on the same test subset used for the headline results. Section 4.1.1 fixes a 5,000-sample random subset of the test set for all experiments; Section 4.3.1 uses this subset to choose K=50 (Table 3), and Section 4.3.2 uses it to choose S=11 (Table 5), with the selection criterion being the same BLEU/SacreBLEU/ROUGE-L metrics that are later reported in Table 1. Thus the 'Ours' row is not an independent evaluation of a pre-specified model; it is a test-set-fitted value. A validation split should be used for model selection, and Table 1 should report results on an untouched test subset, or the paper should explicitly describe the numbers as test-set-selected and avoid claiming SOTA relative to baselines that did not receive this selection.","section":"4.3.1, 4.3.2, Tables 3 and 5"},{"comment":"The comparison with prior published systems is not apples-to-apples. The paper evaluates on a random 5,000-sample subset of the Recipe1M test set, while the baseline numbers in Table 1 are taken from prior papers that use the full test set or different splits and preprocessing (e.g., [11, 56, 65]). For example, under 'FIRE' BLEU is reported as '—' and SacreBLEU/ROUGE-L numbers may have been computed under different tokenization or ingredient-clustering settings. The only fully controlled baseline is LLaVA-FT, which the authors fine-tuned themselves. To support a SOTA claim, all compared methods should be re-evaluated on the same 5,000-sample subset with the same evaluation script, or the differences should be shown not to be an artifact of subset selection.","section":"Table 1"},{"comment":"The paper's stated motivation is hallucination reduction, but no quantitative hallucination metric is reported. Section 4.2.3 and Figure 4 provide qualitative examples, yet BLEU/SacreBLEU/ROUGE-L measure surface overlap and can reward copying retrieved text rather than factual correctness. Please add an ingredient-level precision/recall or hallucination metric (e.g., fraction of predicted ingredients unsupported by the image or ground truth) and compare it directly against LLaVA-FT, with and without retrieval, so the hallucination-reduction claim can be verified.","section":"Section 4.2, Fig. 4"}],"minor_comments":[{"comment":"Equation (3) has a duplicated conditioning term: pθ(yi|[X;R;y<i];θ) should be pθ(yi|[X;R;y<i]).","section":"Eq. (3)"},{"comment":"The voting method in Equation (4) uses cosine similarity between generated recipes, but the paper never specifies how the recipe texts are vectorized (e.g., TF-IDF, sentence embeddings, or token overlap). Please state this explicitly, since it is essential for reproducibility.","section":"Section 3.3"},{"comment":"The text says 'relative improvement' of 2.25%, 0.18%, and 1.97% over FoodLMM, but the values are absolute differences (30.11 vs 27.86, etc.). Please use the correct term.","section":"Section 4.2.1"},{"comment":"The description of 'SDRA(fixed top 1)' is confusing: 'the top 2 retrieved ingredients sets and the top 3 ingredients along with top 1 instruction' is not clear about whether 'top 3 ingredients' means a third ingredient set or the top-3 ingredient items. Please rewrite.","section":"Section 4.3.1"},{"comment":"The caption says the confusion matrix is computed for 'top 11 retrieval-augmented prediction results', while the text refers to 'seven predictions'; please make the number of candidates consistent.","section":"Fig. 6"},{"comment":"The conclusion contains a typo: 'first retrieval augmented large multimodal mode' should be 'model'.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the most serious risk is the self-recipe leakage in Section 4.1.3; the manuscript does not state whether the ground-truth recipe is excluded from the retrieved top-K during training, and I could not verify this from the paper alone. I would ask the authors to provide the retrieval top-K lists or exclusion code, or to rerun with explicit exclusion, before the SOTA claim can be accepted. The second issue is the test-set hyperparameter selection, which is easier to fix by introducing a validation split. The paper's internal ablation trends are otherwise encouraging, and the candid supplementary discussion of retrieval failures is a positive sign."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a competent application of retrieval-augmented generation and self-consistency voting to food-image recipe generation. If the method works as claimed, it is a useful practical step for a narrow subfield, but the headline SOTA numbers are less solid than they look. The internal ablations are the strongest part: a controlled LLaVA fine-tune baseline, clear gains from retrieval and voting, and a few honest failure-case analyses. That is real work.\n\nThe biggest soft spot is one the paper never addresses: during training, the datastore is the Recipe1M training set, and the retriever is trained to match image-recipe pairs. For a training image, the ground-truth recipe is almost certainly in the top-K retrieved set, likely at rank 1. The paper never says it excludes the self-recipe. If it is present, the model is trained with the answer in the context, which cheaply lowers the training loss and teaches the model to lean on retrieved text. At test, the GT is not in the datastore, so the model faces a different regime, and the reported gains may not transfer. This needs to be addressed head-on: either confirm self-recipe exclusion or re-run training with it excluded.\n\nSecond, the SOTA claim is compromised by methodology. K=50 and S=11 were picked by ablating on the same 5,000-sample test subset used for the final table, so the reported numbers are partly test-set-fitted. Baselines from prior papers were evaluated on different splits, and there are no error bars or significance tests. The gaps over FoodLMM are small (a couple of BLEU points), so without matched baselines the \"significant outperformance\" claim is not established.\n\nThere are also minor issues: no code release, and the \"first retrieval-augmented LMM for recipe generation\" claim is not backed by a serious survey. But the method itself is plausible, the controlled baseline is fair as far as it goes, and the failure cases show the authors are thinking about limitations.\n\nWho this is for: people working on food computing or recipe generation will get useful ideas. RAG researchers will find it a straightforward application. I would send it to review—it deserves a serious referee—but I would expect major revision, specifically around the self-recipe leakage and the evaluation protocol. Right now, the central claim needs stronger support.","headline":"A plausible RAG-for-recipes paper with honest ablations but a likely training-time leakage issue and test-set-tuned SOTA numbers.","tokens_in":28607,"tokens_out":2632,"would_cite":false,"duration_ms":22458,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T45","68T50"],"pacs":[],"model":"deepseek-v4-flash","headline":"Retrieval-augmented recipes beat prior models on Recipe1M","keywords":["retrieval-augmented generation","recipe generation","food computing","large multimodal models","hallucination reduction","self-consistency voting","cross-modal retrieval","Recipe1M"],"falsifier":"Replace the retrieved recipes with random recipes from the Recipe1M training set while keeping the rest of the pipeline identical; if the gains over the fine-tuned LLaVA baseline shrink or reverse, the method's improvement depends on retrieval relevance rather than on the extra textual context or the voting mechanism alone.","tokens_in":27690,"feed_emoji":"🍳","tokens_out":5208,"duration_ms":39778,"temperature":0.7,"pith_summary":"Retrieval-augmented recipe generation proposes to fix the hallucination problem in large multimodal models when writing cooking instructions from food images. The paper builds a generator on frozen LLaVA with a trainable LoRA adapter, and at both training and inference time feeds it recipes retrieved from the Recipe1M training set that are semantically similar to the input image. A second component, self-consistency ensemble voting, generates several candidate recipes conditioned on different retrieved recipes and picks the one that agrees most with the others. On a fixed 5,000-image test subset of Recipe1M, the method reports state-of-the-art BLEU 30.11, SacreBLEU 6.42, and RougeL 38.93 for recipe generation, and F1 50.32 for ingredient recognition.","feed_headline":"Retrieval-augmented recipes beat prior models on Recipe1M","feed_subtitle":"Stochastic retrieval plus self-consistency voting lifts BLEU to 30.11 and ingredient F1 to 50.32%.","key_machinery":"The load-bearing mechanism has two parts. Stochastic Diversified Retrieval Augmentation (SDRA) builds a 'recipe demonstration' by randomly sampling multiple retrieved ingredient lists and one retrieved instruction list from the top-K retrieval results, giving the generator diverse but relevant context. Self-Consistency Ensemble Voting uses agreement among candidate recipes generated from different retrieved contexts as a confidence signal: the candidate with the highest average pairwise cosine similarity to the other candidates is chosen as the final output. The voting step is what converts the diversity created by SDRA into a single more reliable prediction.","core_discovery":"The paper's central claim is that retrieval augmentation can reduce hallucination in LMM-based recipe generation enough to beat all prior published systems on Recipe1M. The proposed architecture is an image-to-recipe retriever followed by a frozen LLaVA generator fine-tuned with LoRA. During training, Stochastic Diversified Retrieval Augmentation randomly samples three of the top-50 retrieved ingredient lists and one retrieved instruction list, concatenates them with fixed prompts telling the model that the search results are for reference only, and optimizes standard next-token prediction. During inference, Self-Consistency Ensemble Voting generates S candidate recipes by conditioning on the top-S retrieved recipes separately, computes pairwise cosine similarity among the candidates, and returns the candidate with the highest average agreement. The authors report that this combination outperforms two-stage methods such as InverseCooking, prior LMM-based methods such as FoodLMM, and a fine-tuned LLaVA baseline, and that ingredient recognition also improves without a dedicated recognition head.","pith_inferences":["The voting mechanism is a form of unsupervised confidence estimation; if it generalizes, any multimodal task that can be conditioned on multiple retrieved or sampled contexts could use the same agreement-based selection without extra supervision.","The paper's own failure cases show that when retrieval is bad, it introduces wrong ingredients and steps; a natural test is to condition on retrieval quality per sample and measure how much the voting gain varies with it.","Automatic n-gram metrics may understate the true gain, since the paper notes cases where 'pasta' and 'spaghetti' are counted as mismatches; a human-evaluation study of recipe plausibility and hallucination rate would likely be more informative than the reported BLEU/SacreBLEU deltas."],"forward_implications":["If the central claim is right, retrieval augmentation plus self-consistency voting is an effective hallucination remedy for LMMs in vertical generation tasks, not just for recipe generation.","The monotonic improvement with voting candidate count S up to 11 implies a simple compute-quality trade-off: using S around 5 already captures most of the gain, per the paper's own analysis.","Because the generator is frozen except for a LoRA adapter, the approach is a cheap way to specialise a large multimodal model to a new structured-output domain.","The method also lifts ingredient recognition metrics, suggesting that the retrieved ingredients act as a knowledge source that compensates for the model's limited food-domain vocabulary.","The fixed prompt that search results are 'only for referring' is a practical recipe for telling an LMM to use retrieval as context rather than as a literal answer."],"supporting_citations":[{"why":"Recipe1M dataset supplies the image-recipe pairs for training and the test subset used for all reported metrics.","marker":"[66]"},{"why":"LLaVA is the frozen generator backbone the paper fine-tunes with LoRA.","marker":"[44]"},{"why":"LoRA provides the parameter-efficient fine-tuning method that keeps the large generator trainable at low cost.","marker":"[26]"},{"why":"The off-the-shelf cross-modal recipe retrieval model returns the semantically similar recipes that SDRA samples from.","marker":"[57]"},{"why":"FoodLMM is the prior LMM-based recipe generation baseline that the paper's method improves on.","marker":"[70]"},{"why":"InverseCooking represents the two-stage recipe generation approach that the paper's single-stage method is compared against.","marker":"[56]"},{"why":"FIRE is the dedicated ingredient-recognition baseline the paper surpasses on ingredient F1 and IOU.","marker":"[11]"},{"why":"LISA weights initialise the generator, providing the pre-trained vision-language reasoning capability.","marker":"[33]"}],"fun_headline_variants":["Recipe generation gets retrieval boost to cut hallucinations","Retrieval + self-consistency voting lifts recipe generation to SOTA","SDRA and voting reduce recipe hallucinations, beat existing models","Image-to-recipe: retrieval augmentation reduces errors, sets benchmark","Retrieval-augmented LMM slashes recipe hallucinations, hits SOTA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that the off-the-shelf retriever returns recipes that are semantically relevant to the input image; if retrieval is often off-target, the added context can inject wrong ingredients and make hallucinations worse, as the paper's own supplementary failure cases acknowledge.","fun_headline_variants_meta":{"raw":{"variants":["Recipe generation gets retrieval boost to cut hallucinations","Retrieval + self-consistency voting lifts recipe generation to SOTA","SDRA and voting reduce recipe hallucinations, beat existing models","Image-to-recipe: retrieval augmentation reduces errors, sets benchmark","Retrieval-augmented LMM slashes recipe hallucinations, hits SOTA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000314,"raw_usage":{"total_tokens":1789,"prompt_tokens":961,"completion_tokens":828,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":740}},"tokens_in":577,"tokens_out":828,"duration_ms":7844,"temperature":1.0,"reasoning_tokens":740,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:25:29.368191+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the retrieved recipes with random recipes from the Recipe1M training set while keeping the rest of the pipeline identical; if the gains over the fine-tuned LLaVA baseline shrink or reverse, the method's improvement depends on retrieval relevance rather than on the extra textual context or the voting mechanism alone.","supporting_citations":[{"cited_title":"Learning cross-modal embeddings with ad- versarial networks for cooking recipes and food images","cited_arxiv_id":null,"evidence_quote":"Recipe1M dataset supplies the image-recipe pairs for training and the test subset used for all reported metrics."},{"cited_title":"Visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"LLaVA is the frozen generator backbone the paper fine-tunes with LoRA."},{"cited_title":"Lora: Low-rank adaptation of large language models","cited_arxiv_id":null,"evidence_quote":"LoRA provides the parameter-efficient fine-tuning method that keeps the large generator trainable at low cost."},{"cited_title":"Revamping cross-modal recipe retrieval with hierarchical transformers and self-supervised learning","cited_arxiv_id":null,"evidence_quote":"The off-the-shelf cross-modal recipe retrieval model returns the semantically similar recipes that SDRA samples from."},{"cited_title":"Inverse cooking: Recipe generation from food images","cited_arxiv_id":null,"evidence_quote":"InverseCooking represents the two-stage recipe generation approach that the paper's single-stage method is compared against."},{"cited_title":"Fire: Food image to recipe generation","cited_arxiv_id":null,"evidence_quote":"FIRE is the dedicated ingredient-recognition baseline the paper surpasses on ingredient F1 and IOU."}],"review_version":1}