{"id":"3557667e-9b67-48df-8497-5466843c8e82","arxiv_id":"2607.25820","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Injecting BERT-encoded, LLM-generated ingredient labels into segmentation features and decoder queries raises FoodSeg103 mIoU from 51.9 (Mask2Former baseline) to 54.4 with LIM-F and 55.0 with LIM-Q.","lead":"Two new modules inject LLM-generated ingredient names into a food segmentation model and lift FoodSeg103 mIoU from 51.9 to 55.0 (LIM-Q). Because the labels are generated from the same test image, the reported state-of-the-art gain may be partly due to this extra hint rather than the modules themselves.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation confounds LLM-derived test-image labels with LIM module design; no label-conditioned baseline, so attribution of mIoU gains to LIM cross-attention is not established.","rationale":"The reader's weakest assumption identifies the same evaluation gap: the LIM variants receive image-specific LLM ingredient labels that the baselines do not, so the reported mIoU gains conflate the extra input signal with the proposed fusion architecture. I agree this is the most load-bearing concern. The paper's own Section 4.1 describes the LLM input as the image plus the class list, and no experiment isolates the label signal from the module design. Table 2's dramatic rare-class improvements are consistent with a label-prior effect rather than with the cross-attention mechanism. This does not make the method internally inconsistent; it means the empirical attribution is not yet established. Because the reader already issued a CONDITIONAL verdict on these grounds, my stress-test does not move the verdict — the same control experiment would resolve the issue. I therefore recommend keeping the reader's conditional assessment rather than upgrading or rejecting.","tokens_in":8473,"tokens_out":3081,"duration_ms":32116,"concrete_test":"Train a Mask2Former (Swin-L) baseline on FoodSeg103 with the same dynamically generated LLM ingredient labels from §4.1, injected as a learned embedding added to the initial decoder queries (or as a per-image class-bias added to the classification logits), without any LIM cross-attention modules. Use the same 240k iterations, optimizer, and loss weights (e.g., 5,5,4 or 5,5,6). If this simple label-conditioned baseline reaches approximately 54–55 mIoU, the gains attributed to LIM-F/LIM-Q are mostly due to the label signal rather than the module architecture. If it remains near the 51.9 Mask2Former baseline, the cross-attention mechanism is validated as the cause.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 states that the LLM input included the food image and the full list of 104 FoodSeg103 classes, and that the model was asked to infer likely ingredients. Every LIM variant therefore receives an image-specific semantic prior (e.g., 'date is present') that is not given to the Mask2Former, K-Net, BEiT, or Swin-TUNA baselines. The central claim — that LIM-F and LIM-Q cross-attention modules cause the mIoU improvements — is confounded with this extra label signal. The paper reports no ablation in which the same LLM-derived labels are injected into the baseline through a trivial mechanism, such as a learned query bias, a per-image class-prior logit adjustment, or a label-conditioned classifier. Table 2 makes the confound concrete: rare classes like 'date' jump from 1.28 to 88.93 mIoU, which is exactly the pattern expected from a label prior that tells the model which rare classes are present, rather than from the specific cross-attention design. The 49.8 vs 49.0 abstract/table discrepancy is secondary but reinforces that the quantitative report needs more care. Until a label-conditioned control is run, the module-level attribution is underdetermined.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two plug-and-play language-injection modules, LIM-F and LIM-Q, for food image segmentation. Ingredient labels are generated from the input image by an LLM (GPT-o4 mini), encoded by BERT, and injected either into multi-scale visual features (LIM-F) or into Mask2Former decoder queries (LIM-Q). On FoodSeg103 with a Swin-L backbone, LIM-Q reaches a reported mIoU of 55.0, LIM-F reaches 54.4, and the paper also reports gains on rare classes and moderate training-memory overhead.","tokens_in":8789,"tokens_out":3662,"duration_ms":36513,"significance":"If the module-level attribution were established, this would be a practical contribution: it avoids paired image-text pretraining, is architecture-agnostic, and reports modest memory costs. Strengths include comparisons with several strong baselines, ablations of classification-loss weights and BERT fine-tuning, and qualitative examples. However, the evaluation currently cannot separate the effect of the LIM cross-attention design from the effect of the privileged, image-specific ingredient-presence signal that only the LIM variants receive. Because a trivial label-conditioning baseline is missing, the central claim that the LIM modules themselves cause the observed mIoU gains is underdetermined.","major_comments":[{"comment":"The LLM input is \"a food image and the full list of 104 ingredient class names\" (Section 4.1), so every LIM variant receives an image-specific semantic prior (e.g., \"date is present\") that is not provided to the Mask2Former, K-Net, BEiT, or Swin-TUNA baselines. The reported rare-class jumps, especially date from 1.28 to 88.93 in Table 2, are exactly the pattern expected from a class-presence prior rather than from the specific LIM cross-attention modules. The paper needs a control that injects the same LLM labels into the baseline through a trivial mechanism (e.g., per-image class-prior logit adjustment, a label-conditioned classifier head, or a query-bias initialization) and reports mIoU and per-class results. Without such a control, the gains cannot be attributed to LIM-F/LIM-Q.","section":"§4.1, Table 2"},{"comment":"The abstract claims an improvement from 47.7 to 49.8 mIoU for the CNN-based K-Net architecture, but Table 1 reports 49.0 (+2.7%) for K-Net + LIM-F. This internal inconsistency needs correction. In addition, no error bars, multiple seeds, or significance tests are reported; the per-class numbers in Table 2 are single runs with very large swings. The quantitative claims should be made statistically reliable.","section":"Abstract vs. Table 1"},{"comment":"LLM-generated labels are a central input to the method, but the description is insufficient for reproduction: the model name \"GPT-o4 mini\", access date, exact prompt template, sampling parameters, and the actual generated-label outputs are not provided, and no code is released. At minimum, the authors should provide the full prompt, temperature/token settings, a sample of LLM outputs, and a cached label file, so the language-guidance input can be verified and reused.","section":"§4.1, Reproducibility"}],"minor_comments":[{"comment":"The model name \"GPT-o4 mini\" appears twice; if GPT-4o mini is intended, please correct the typo. The footnote \"accessed in early May, 2025\" should include the exact model version and snapshot.","section":"§3.1, §4.1"},{"comment":"Table 3 reports only a subset of loss-weight settings. For completeness, please include the setting (5,5,2) with BERT unfreezing for LIM-F and LIM-Q, so the effect of BERT unfreezing is not aliased with the loss-weight change.","section":"Table 3"},{"comment":"The caption reports final mIoU values 54.44 and 54.96, which are consistent with Table 1 but would be clearer if they matched exactly; round consistently to one decimal place.","section":"Figure 4"},{"comment":"The related-work discussion lists LLM vision systems (MiniGPT-4, LLaVA, KOSMOS-2) without clarifying how they differ from the proposed use of an LLM as a label generator; a sentence distinguishing label generation from end-to-end multimodal pretraining would help.","section":"§2.3"}],"recommendation":"major_revision","confidential_remarks":"The central empirical claim is plausible but not yet isolated. The missing label-conditioned control is the key blocker; without it, the paper cannot claim that LIM modules, rather than the extra semantic prior, drive the improvements. The internal 49.8 vs 49.0 inconsistency and the absence of error bars also need attention. The paper is within scope for the journal, but I would not accept it before these issues are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, the idea is actually practical: instead of curating image-text pairs, the authors use an LLM to generate ingredient labels from the input food image and inject them into a segmentation model via two lightweight cross-attention modules (LIM-F and LIM-Q). That removes a real bottleneck and the modules are clearly described. Second, the evaluation is confounded in a way that matters. In Section 4.1, the LLM receives the test image plus the full list of 104 FoodSeg103 classes and is asked to infer which ingredients are present. So every LIM variant gets a per-image semantic prior—essentially a class-presence hint—that the Mask2Former, K-Net, BEiT, and Swin-TUNA baselines do not receive. The reported gains, especially the rare-class jump for 'date' from 1.28 to 88.93 in Table 2, look exactly like what you'd expect from a class-presence prior rather than from the specific cross-attention design. The paper never runs the obvious control: inject the same LLM label list through a trivial mechanism (class-prior bias, label-conditioned classifier head, or simple query bias) and see if the gains persist.\n\nThat confound is the main soft spot, and it is load-bearing. The modules may be perfectly good, but as written the attribution of the 51.9 to 55.0 improvement to LIM-Q is not established. I'd also flag the internal inconsistency: the abstract claims 49.8 mIoU for the CNN-based LIM-F while Table 1 reports 49.0. And there are no error bars, no multiple seeds, and no code. Those are secondary, but they make the numbers harder to trust.\n\nNow the credit. The work does more than cherry-pick results: it compares three integration variants, shows convergence curves, reports GPU memory overhead, and ablate the classification loss weight and BERT unfreezing. The writing is honest about what the LLM input is, which is how I could diagnose the confound so quickly. This is not a sloppy paper; it's an under-controlled one.\n\nWho should read it? People working on food segmentation or on injecting LLM-derived side information into vision models. It's a good starting point for a workshop paper or a short conference paper—after the control experiment is run. The methodological idea, avoiding paired data via LLM labels, is worth discussing even if the specific numbers don't survive unchanged.\n\nMy recommendation: yes, send it to peer review, because the idea is interesting and the modules are plausible. But reviewers should push hard for a label-conditioned control. Without it, the central claim remains underdetermined, and I would not accept the paper in its current form.","headline":"Clean idea and a real confound: the LLM sees the test image and leaks a per-image ingredient prior, so the mIoU gains are not cleanly attributable to the LIM modules.","tokens_in":750,"tokens_out":1846,"would_cite":false,"duration_ms":34310,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Injecting LLM ingredient labels into Mask2Former reaches 55.0 mIoU on FoodSeg103","keywords":["food image segmentation","ingredient labels","large language models","multimodal fusion","Mask2Former","LIM-F","LIM-Q","FoodSeg103"],"falsifier":"Train the same Mask2Former backbone with the LLM ingredient labels provided as a simple per-image class-prior weighting of the classification loss, or as input tokens appended to queries, but without the LIM cross-attention structure. If this simpler conditioning matches the 54.4–55.0 mIoU, the module-level claim collapses. Conversely, replacing the LLM labels with random labels should drop performance if the modules genuinely use semantic content.","tokens_in":8360,"feed_emoji":"🍽️","tokens_out":3405,"duration_ms":29573,"temperature":0.7,"pith_summary":"The paper proposes two lightweight modules, LIM-F and LIM-Q, that inject ingredient labels generated by a large language model into the visual segmentation pipeline. On the FoodSeg103 benchmark, the best configuration reaches 55.0 mIoU, surpassing prior state-of-the-art results by a noticeable margin. Crucially, the method works without paired image-text pretraining or curated recipe corpora, relying on LLM-generated labels from the input image itself. If correct, this suggests that semantic text cues can be effectively plugged into standard segmentation architectures at a modest training-memory cost.","feed_headline":"LLM ingredient labels lift food segmentation to 55.0 mIoU","feed_subtitle":"Two plug-in modules fuse BERT-coded LLM labels into Mask2Former, beating prior best by 4.4 points.","key_machinery":"The two modules are the central mechanism: LIM-F (Language Injection Module for Features) applies cross-attention at each stage of a multi-level encoder, using visual features as queries and BERT-encoded ingredient embeddings as keys and values, with residual connections and feed-forward networks. LIM-Q (Language Injection Module for Queries) operates inside a Transformer decoder, performing cross-attention between learnable query tokens and BERT-encoded ingredient labels at the start of every decoding round, followed by self-attention and a feed-forward network. Both avoid pre-aligned image-text pairs; the ingredient labels are produced per image by an LLM prompted with the image and the fu","core_discovery":"The authors claim that LLM-derived ingredient labels, encoded by BERT and fused via two simple cross-attention modules, substantially improve fine-grained food segmentation. LIM-Q, injected into the Mask2Former decoder before each round of query updates, yields 55.0 mIoU on FoodSeg103 with a Swin-L backbone, compared with 51.9 for the baseline Mask2Former and 50.6 for the previous best method, Swin-TUNA. LIM-F, which fuses text into multi-level visual features, achieves 54.4 mIoU under the same backbone and also improves a CNN-based K-Net pipeline from 47.7 to 49.0 mIoU. The authors attribute the gains to improved discrimination of visually similar and rare ingredients, and they show the ext","pith_inferences":["LLM label quality becomes a hidden variable: if the LLM mislabels an ingredient, the module may reinforce an error; the paper does not assess this failure mode.","A class-prior baseline that simply up-weights rare classes might capture part of the same gains, since the LLM labels effectively provide a per-image class prior.","The approach could be extended to open-vocabulary food segmentation if the LLM produces labels beyond the fixed 104 classes.","The faster convergence of LIM-Q relative to LIM-F suggests query-level injection is a more direct channel for semantic guidance; future work might explore even earlier or hierarchical injection."],"forward_implications":["The plug-and-play design means ingredient semantics can be added to existing segmentation architectures without changing backbones or annotation formats.","Gains are largest for rare and visually ambiguous ingredients, which matters for nutrition tracking where minor ingredients are easily missed.","Eliminating paired image-text pretraining lowers the barrier for applying language-guided segmentation to other domains with scarce aligned data.","The modest memory increase (at most 3.8 GB) makes the approach feasible on a single GPU for training."],"fun_headline_variants":["LLM labels plug into segmenters, hit 55.0 mIoU on FoodSeg103","Two cross-attention modules fuse LLM labels for SOTA food segmentation","LLM-derived labels push food segmentation to 55.0 mIoU","Plug-and-play text injection lifts food segmentation by 4.4 points"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The evaluation assumes that LLM-generated ingredient labels for the test image are a fair system input, and that the mIoU increase over baselines without those labels is attributable to the LIM modules, rather than to the label information alone or to a per-image class prior.","fun_headline_variants_meta":{"raw":{"variants":["LLM labels plug into segmenters, hit 55.0 mIoU on FoodSeg103","Two cross-attention modules fuse LLM labels for SOTA food segmentation","LLM-derived labels push food segmentation to 55.0 mIoU","Plug-and-play text injection lifts food segmentation by 4.4 points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000692,"raw_usage":{"total_tokens":3026,"prompt_tokens":861,"completion_tokens":2165,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":2079}},"tokens_in":605,"tokens_out":2165,"duration_ms":15728,"temperature":1.0,"reasoning_tokens":2079,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T01:22:31.930566+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same Mask2Former backbone with the LLM ingredient labels provided as a simple per-image class-prior weighting of the classification loss, or as input tokens appended to queries, but without the LIM cross-attention structure. If this simpler conditioning matches the 54.4–55.0 mIoU, the module-level claim collapses. Conversely, replacing the LLM labels with random labels should drop performance if the modules genuinely use semantic content.","supporting_citations":[],"review_version":1}