{"id":"0bc63065-3fc0-487e-b3f8-6bd1467cf15b","arxiv_id":"2411.11285","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Apple instance segmentation models trained solely on LLM-generated synthetic images with automatic SAM annotations transfer to real orchard images, though key reported metrics are inconsistent and no real-data baseline is provided.","lead":"This paper trains an apple segmentation model using only AI-generated orchard images, automatically labeled by a detection model and the Segment Anything Model, then tests it on real orchard photos. It claims this removes the need for collecting and manually labeling field images, which could cut cost and time for agricultural AI.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No recall measurement for zero-shot detections leaves training-label completeness—and hence the claimed field transfer—unverified.","rationale":"The reader's REJECT verdict is justified by multiple internal inconsistencies, and I find the weakest assumption correctly identified: the proposed method's only training signal is the automatic annotation, so its completeness is the load-bearing condition for every downstream claim. The reported synthetic Dice/IoU cannot establish completeness because the matching protocol for Eqs. (7)–(8) is not specified; if only detected masks are paired with manual masks, missed apples are invisible in the headline metrics. The abstract's Dice/IoU values also contradict §III (0.9513/0.9303 vs. 0.88/0.86), and the conclusion misattributes trained-model metrics in Table II to the zero-shot base model, further obscuring what the base detector alone achieves. None of this requires assuming misconduct; it means the evidence as reported is insufficient. A recall audit on the 40 manually labeled synthetic images would settle the question directly: high recall would support the pipeline, while low recall would confirm that the model's field performance is not attributable to complete high-quality labels. I would not soften the reader's REJECT; the paper needs corrected metrics, a defined matching protocol, and a real-data or no-training baseline before the central claim can be assessed.","tokens_in":25357,"tokens_out":9006,"duration_ms":92073,"concrete_test":"On the 40 manually annotated LLM-generated images, rerun the proposed automatic annotation pipeline (YOLO11 zero-shot detection at confidence 0.3, then SAMv2 masks on the boxes). Compute detection-level recall: the fraction of manual apple instances matched by at least one auto-detection at IoU≥0.5, and recompute mask Dice/IoU counting every unmatched manual apple as an all-zero predicted mask. If per-image recall falls below roughly 0.85, or if including unmatched apples drops Dice/IoU by more than 5 points, the auto-labeled training set is incomplete and the field validation cannot support the claimed elimination of manual annotation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that a segmenter trained only on synthetic, auto-annotated images accurately segments apples in a real orchard depends on the auto-annotation step producing complete training labels. The paper never reports recall of the zero-shot YOLO11 detections on the synthetic set. In §II.E, automatic and manual masks are compared on 40 LLM-generated images, but Eqs. (7)–(8) average Dice/IoU over mask pairs without a stated matching protocol; an apple missed at the confidence threshold of 0.3 has no predicted mask and may never enter the reported IoU average. The §III synthetic Dice/IoU (0.88/0.86, versus 0.9513/0.9303 in the abstract) therefore cannot tell us how many DALL-E apples were never labeled. If the COCO-pretrained detector misses a material fraction of apples, the training set contains systematic false negatives, and the field mask mAP@50 of 0.833 cannot be attributed to complete labels. The reported real-image annotation recall of 0.61 in §III shows the pipeline can miss apples outside the synthetic domain, making the unmeasured synthetic recall load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an entirely synthetic pipeline for apple instance segmentation: DALL-E generates 524 (later stated as 501) orchard images; a COCO-pretrained YOLO11 detector proposes boxes at confidence 0.3; SAMv2 converts each box into a mask; the masks are saved as YOLO-format labels; and five YOLO11 instance-segmentation variants are trained on these labels. Performance is reported on 40 LLM-generated validation images and on 42 real images from a commercial orchard. The authors claim the auto-annotations reach Dice 0.9513 / IoU 0.9303, that all trained configurations segment apples accurately, and that the best field result is YOLO11m-seg with mask precision 0.902 and mask mAP@50 0.833, concluding that field imaging and manual annotation can be eliminated.","tokens_in":25534,"tokens_out":8093,"duration_ms":74509,"significance":"If the central claim were fully supported, the result would be practically valuable for agricultural AI and for other domains where labeled imagery is scarce, because a pipeline that generates and auto-labels training data at scale would reduce cost and accelerate deployment. The manuscript has concrete strengths: a reproducible recipe, a public dataset link, and an independent field-validation set of 42 real images collected with a Kinect camera. That field validation is the only non-circular evidence and should be preserved. However, the current quantitative claims are undermined by internal inconsistencies, a missing baseline, and an unmeasured detection recall in the auto-labeling step; these issues must be resolved before the central claim can be accepted.","major_comments":[{"comment":"The method is not zero-shot as claimed. The YOLO11 base model is pretrained on MS-COCO, whose 80 categories include 'apple', so apples are not an unseen class for the detector. The claimed 'zero-shot detection' is actually detection of a class already present in the pre-training vocabulary. This is load-bearing because the paper's framing and title rest on zero-shot generalization, and the experiment provides no evidence that the pipeline would work for a class absent from COCO. Please rename the approach (e.g., COCO-pretrained detection) or demonstrate genuine zero-shot behavior on a non-COCO class.","section":"Abstract, §II.C, Fig. 3"},{"comment":"The abstract and conclusion report Dice 0.9513 and IoU 0.9303 for the automatically generated annotations, while Section III reports Dice 0.88 and IoU 0.86 for the synthetic dataset. These values are not reconciled anywhere. Since auto-annotation accuracy is the foundation for the training labels, the manuscript must state which numbers are final, which images they were computed on, and what matching protocol was used to compute them.","section":"Abstract vs. §III and Fig. 5(c)"},{"comment":"The evaluation of automatic versus manual masks never reports detection recall on the synthetic set. Equations (7) and (8) average Dice/IoU over mask pairs, but no matching rule is stated, and an apple missed at the 0.3 confidence threshold simply produces no mask and may therefore be excluded from the average. The training set thus contains an unknown number of false negatives, and the field mAP@50 of 0.833 cannot be attributed to complete supervision. The paper does report a real-image annotation recall of 0.61 in Section III, which shows that miss rates can be substantial outside the synthetic domain; the synthetic-set recall must be measured and reported.","section":"§II.E, Eqs. (5)-(8)"},{"comment":"No baseline model trained on real orchard images is included. Without a real-data-trained comparator, the field mask mAP@50 of 0.833 cannot be judged as 'accurate' or as evidence that synthetic-only training suffices. In addition, the 40-image synthetic test set is drawn from the same DALL-E distribution as the training images, and its pseudo-labels were generated by the same YOLO11/SAM pipeline used at training time, making the ~0.92 synthetic scores largely circular. The 42-image field set is the only independent evidence, and it is too small for reliable comparisons across five configurations without confidence intervals or statistical testing.","section":"Tables I-III, §III.C"},{"comment":"The dataset size and split are inconsistent and under-specified. Section II.B says 524 images were generated, while Section II.E and Section IV refer to 501 images; the manuscript never states how many images were used for training, validation, and testing, nor whether the 40 manually annotated images used in Tables I and II were held out from training. This ambiguity prevents interpretation of the reported numbers and makes the experiments difficult to reproduce. Please specify exact splits and reconcile the image counts.","section":"§II.B, §II.E, §IV"},{"comment":"The conclusion bullet reports mask precision 0.92, recall 0.851, and mask mAP@50 0.92 for the 'zero-shot base model,' but these numbers match the trained YOLO11x-seg results in Table II rather than any zero-shot detector. A COCO-pretrained YOLO11 produces bounding boxes, not masks; the masks in the pipeline come from SAMv2. This conflates the detector, the annotator, and the trained segmenter, and it should be corrected for the results to be interpretable.","section":"Conclusion, 'Zero-Shot Base Model Performance' bullet"}],"minor_comments":[{"comment":"Section II.A contains a duplicated verbatim paragraph describing the study site and data acquisition; remove one copy.","section":"§II.A"},{"comment":"Equation (12) writes '109 per image' where 10^9 is intended, and Eq. (3) contains broken formatting in 'conf idencedetectionsi'; please fix the LaTeX and the notation.","section":"Eqs. (3), (12)"},{"comment":"Reported inference times are inconsistent: Section III says 1,986.4 ms (1.9 s) per image on average, while Section III.A and Fig. 8a state 4.4 seconds for YOLO11n-seg and Fig. 10 reports times in milliseconds. Clarify the units and whether these are per-image or per-batch measurements.","section":"§III, §III.A, Fig. 8, Fig. 10"},{"comment":"There are several typos and style issues, e.g., 'techiques' (§I), 'deminstrating' (§I), 'innivation' (§I), and 'pf' for 'of' (§II.G); please correct them.","section":"Throughout"},{"comment":"The text alternates between 'SAM', 'SAMv2', and 'SAM 2'; use the official model name consistently and define it when first introduced.","section":"§II.D, Fig. 2a"},{"comment":"The bar chart in Fig. 5(c) is difficult to read in the provided version; ensure the figure is legible and add numeric labels to the bars so the reported values can be checked against the text.","section":"Fig. 5(c)"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline case. The core recipe (LLM-generated images plus auto-annotation from a COCO-pretrained detector and SAM) is interesting, and the field validation, though small, is a genuine positive. My main worries are that the 'zero-shot' framing is factually incorrect because COCO contains the apple class, and that the quantitative foundation is shaky: abstract/results numbers conflict, no detection recall is reported for the synthetic labels, and no real-data-trained baseline is provided. I would be willing to accept a revised version that adds the missing recall and a real-data baseline, removes the zero-shot terminology, reconciles the numbers, and clarifies the data split. If the recall cannot be computed or the baseline shows synthetic training to be substantially worse than real-data training, the paper should be rejected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper has a real experimental kernel but the current manuscript overstates it. The core idea—use DALL-E to generate orchard images, auto-annotate with COCO-pretrained YOLO11 plus SAM, train YOLO11-seg on the result—is coherent, and the field validation on 42 real orchard images (mask mAP@50 0.833 for YOLO11m-seg) is a legitimate independent test. That is worth something. The previous detection paper (their ref [69]) didn't do segmentation or field validation of a trained segmenter, so this is a genuine extension, and they provide a GitHub link to the generated dataset.\n\nThe problems are serious and mostly in presentation and experimental design. First, the abstract and conclusion report Dice 0.9513 and IoU 0.9303 for the automatic annotations, but Section III reports 0.88 and 0.86 for the synthetic set and 0.91 precision / 0.61 recall on real images. Those numbers don't match, and the conclusion attributes metrics to the \"zero-shot base model\" that actually belong to trained models. That is a trust-breaker. Second, \"zero-shot\" is wrong: apple is a COCO class, so a COCO-pretrained YOLO11 is not zero-shot for apples. It is a pretrained detector used without fine-tuning. Third, there is no baseline trained on real images, so the \"viable alternative\" claim is unquantified—0.833 mAP@50 may be fine, but we don't know relative to standard practice.\n\nThe stress-test note about recall is correct and important. The paper never reports recall of the zero-shot detections on the synthetic set. If the detector misses apples at the 0.3 confidence threshold, those apples never get masks and never enter the reported Dice/IoU average. The real-image annotation recall of 0.61 shows the pipeline can miss apples, so unmeasured synthetic recall is load-bearing. That needs to be fixed with a recall measurement on a manually annotated synthetic subset.\n\nMinor issues: dataset size is stated as 524 in Methods and 501 in Results; the validation is a single orchard, one cultivar, one season; and the writing has duplicated paragraphs. None of these are fatal on their own, but they add up.\n\nNet: the paper deserves a serious referee, not a desk reject. The synthetic-to-real transfer result is worth examining, but the authors need to correct the metric inconsistency, rename the zero-shot framing, add a real-data baseline, and report synthetic recall. As written, I wouldn't cite it; after those revisions, it could be a useful data point for agricultural AI.","headline":"Real synthetic-to-real transfer result, but metric inconsistencies, a mislabeled 'zero-shot' framing, and a missing real-data baseline mean the paper needs major revision before it is citable.","tokens_in":26108,"tokens_out":4016,"would_cite":false,"duration_ms":37500,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a YOLO11 instance-segmentation model for apples can be trained exclusively on LLM-generated synthetic images with zero-shot automatic annotations and still segment apples in a real commercial orchard.","keywords":["YOLO11","SAMv2","Segment Anything Model","zero-shot automatic annotation","synthetic image generation","instance segmentation","apple orchard","LLM-generated datasets"],"falsifier":"A decisive check is to count missed apples and measure mask overlap on a held-out sample of the synthetic images against manual annotations: if the zero-shot detector at threshold 0.3 misses a large share of visible apples, or the SAMv2 masks overlap manual masks by less than roughly 0.9 IoU, then the training labels are systematically biased and the reported field mAP would not survive a recall-centered evaluation.","tokens_in":25111,"feed_emoji":"🍎","tokens_out":9594,"duration_ms":89521,"temperature":0.7,"pith_summary":"The paper claims that a usable apple instance-segmentation model can be built from text prompts alone: DALL-E generates orchard images, a COCO-pretrained YOLO11 detector finds apples without any apple-specific training, and SAMv2 converts the boxes into pixel masks that serve as labels. No field imaging and no manual annotation enter the training process; real orchard images are used only for validation. On 42 validation images from a commercial orchard, the best configuration reaches a mask precision of 0.902 and a mask mAP@50 of 0.833, and the auto-generated masks agree with manual masks at a Dice coefficient of 0.9513. If this transfers to other crops and objects, the dominant cost of applied segmentation, collecting and labeling data, could be replaced by prompt engineering and compute.","feed_headline":"Synthetic images alone segment real orchard apples at 0.902 precision","feed_subtitle":"A YOLO11-SAM pipeline built from DALL-E images and auto labels matches manual-annotation quality, then works in the field.","key_machinery":"The load-bearing mechanism is the YOLO11-to-SAMv2 annotation cascade. A COCO-pretrained YOLO11 model, run at confidence threshold 0.3, acts as a zero-shot apple proposal generator on synthetic images; each proposed bounding box is fed to SAMv2 as a prompt, and SAMv2's mask decoder outputs a per-instance mask. The masks are normalized to image dimensions and saved in YOLO label format, creating the training set that teaches the final YOLO11-seg models. The same cascade is then evaluated by comparing its annotations against manual masks and by the transfer performance of the trained segmenters on real orchard images.","core_discovery":"In the paper's own terms, the discovery is that a fully synthetic, fully automatic data pipeline is sufficient supervision for instance segmentation in a real agricultural scene. The zero-shot YOLO11 detector, run at confidence threshold 0.3, proposes apple boxes in LLM images; SAMv2 turns each box into a mask; and every YOLO11-seg configuration trained on those masks recognizes and delineates apples. YOLO11m-seg performs best on field data, achieving mask precision 0.902 and mask mAP@50 (mean average precision at 50% mask overlap) 0.833 on 42 commercial-orchard images, while the auto-annotation step itself matches manual labels with a Dice coefficient of 0.9513 and IoU of 0.9303.","pith_inferences":["Beyond the paper, the method's transfer limit is probably set by the zero-shot detector: objects far from COCO categories would get few or no boxes, so SAM would never annotate them.","The paper leaves implicit that its own failure images (occluded apples missed, foliage falsely masked) could be fed back as corrective examples in new text prompts to generate more realistic training data.","A testable extension is to feed the trained segmenter's masks back into the annotation pipeline as proposals for a second round, which would scale the training data without any new labels."],"forward_implications":["Apple instance segmentation can be developed without any field imaging or manual labeling during training; the only real images needed are for validation.","Because the pipeline uses a general-purpose detector and a promptable segmenter, the same recipe can be repeated for other fruits or objects by changing the text prompt and the target label.","Automatic annotations are close enough to manual ones (Dice 0.9513, IoU 0.9303) to play the role of ground truth in training, so annotation cost collapses to inference cost.","Even the smallest configuration, YOLO11n-seg, runs at 3.8 ms per image, which is fast enough for on-robot fruit perception if the quality holds in deployment."],"supporting_citations":[{"why":"Supplies the 524 DALL-E-generated synthetic orchard images and the earlier detection pipeline that this work extends from detection to instance segmentation.","marker":"[69]"},{"why":"Supplies the Segment Anything model, whose promptable mask decoder converts YOLO11 bounding boxes into pixel-level training labels.","marker":"[74]"},{"why":"Provides the zero-shot detection concept the authors cite to justify running a COCO-pretrained YOLO11 model on apples without apple-specific training data.","marker":"[75]"}],"fun_headline_variants":["Zero-shot LLM-generated data replaces field imaging for apple segmentation","Synthetic images alone yield 0.902 precision on real apples","Auto-annotated synthetic orchard beats manual: 0.95 Dice overlap","LLM + SAM: zero-shot pipeline trains apple segmenter without real data","Synthetic data, auto labels: field apples at 0.902 precision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes that a COCO-pretrained YOLO11 detector at confidence 0.3 finds almost all apples in DALL-E images, and that SAMv2's masks inside those boxes are clean enough to serve as training labels; missed apples or imprecise boxes would inject errors that the final model inherits.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot LLM-generated data replaces field imaging for apple segmentation","Synthetic images alone yield 0.902 precision on real apples","Auto-annotated synthetic orchard beats manual: 0.95 Dice overlap","LLM + SAM: zero-shot pipeline trains apple segmenter without real data","Synthetic data, auto labels: field apples at 0.902 precision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000767,"raw_usage":{"total_tokens":3463,"prompt_tokens":1069,"completion_tokens":2394,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":685,"completion_tokens_details":{"reasoning_tokens":2298}},"tokens_in":685,"tokens_out":2394,"duration_ms":15985,"temperature":1.0,"reasoning_tokens":2298,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:42:04.189022+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive check is to count missed apples and measure mask overlap on a held-out sample of the synthetic images against manual annotations: if the zero-shot detector at threshold 0.3 misses a large share of visible apples, or the SAMv2 masks overlap manual masks by less than roughly 0.9 IoU, then the training labels are systematically biased and the reported field mAP would not survive a recall-centered evaluation.","supporting_citations":[{"cited_title":"Synthetic meets authentic: Leveraging llm generated datasets for yolo11 and yolov10-based apple detection through machine vision sensors,","cited_arxiv_id":null,"evidence_quote":"Supplies the 524 DALL-E-generated synthetic orchard images and the earlier detection pipeline that this work extends from detection to instance segmentation."},{"cited_title":"Segment anything,","cited_arxiv_id":null,"evidence_quote":"Supplies the Segment Anything model, whose promptable mask decoder converts YOLO11 bounding boxes into pixel-level training labels."},{"cited_title":"Zero-shot object detection,","cited_arxiv_id":null,"evidence_quote":"Provides the zero-shot detection concept the authors cite to justify running a COCO-pretrained YOLO11 model on apples without apple-specific training data."}],"review_version":1}