{"id":"bfa3facb-86fb-44ee-a4b3-83a0818a3c9c","arxiv_id":"2411.16219","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SAM-generated masks train a panoptic segmentation model that counts banana defects exactly in 36.2% of images and estimates relative defect size with r=0.96, at roughly one-tenth the annotation cost.","lead":"Banana inspectors can train a defect-detection model using only cheap bounding box clicks because SAM fills in the pixel-level masks, and the final model counts bruises and scars about as well as one trained on hand-drawn masks. The work shows a practical low-annotation route for fresh-produce grading, but also pinpoints where the current pipeline fails.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Count and size claims rest on a single annotator's ground truth; without an inter-annotator baseline, the 36.2% exact-count and r=0.96 numbers cannot be interpreted as practical grading accuracy.","rationale":"The strongest claim has two parts: (1) SAM labels can replace hand labels for training with negligible loss, and (2) the resulting mask statistics are practically useful for grading. Part (1) is well supported: Table 1 shows the SAM-trained/human-validated row matches the fully-supervised row on PQ, AP, AR, and defect IoU within small margins, with five-fold error bars. Part (2) is where the argument is least secure. The paper's evidence for count and size accuracy is agreement with a single expert, but the paper repeatedly acknowledges that defect boundaries and the decision to annotate minor blemishes are subjective; it even recommends multi-annotator agreement as a baseline in Section 5. Since the ground truth itself is noisy, the reported numbers are uninterpretable without knowing the annotator ceiling. The relative-size proxy concern raised by the reader is real but secondary: the paper explicitly frames relative size as a proxy, and since the goal is to automate visual inspection rather than measure physical dimensions, pixel-ratio agreement with human visual estimates is a defensible target. A multi-annotator study would settle the more fundamental question of whether the evaluation target itself is stable, and would also show whether the model's tendency to over-predict counts is an annotation-policy artifact. Verdict remains CONDITIONAL because the concern is addressable and does not invalidate the core training comparison.","tokens_in":12414,"tokens_out":11862,"duration_ms":115397,"concrete_test":"Recruit two additional annotators to label defect counts and masks on a random subset of at least 50 images using the same protocol (Section 2.2). Compute pairwise human-human exact-count agreement and size correlation using the same matching rule (minimum 0.5 IoU, Section 3.3). Compare with the model-human values (36.2%, 76.2%, r=0.96). If human-human agreement is substantially higher than model-human, the model's count/size performance is worse than it appears; if similar, the model is near the annotation-noise ceiling and the reported numbers are meaningful.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that the method provides 'practical estimates of defect number and relative size' is supported only by agreement with one expert's masks. Section 2.2 states 'The images were annotated by a single expert.' Figures 4 and 5 are computed against that single annotation set. The paper itself flags the problem: Section 5 recommends 'involve multiple annotators and quantify their agreement... using inter-annotator disagreement as a baseline for evaluating machine learning models,' and Appendix D attributes the failed four-class categorization partly to 'ambiguous annotation.' Without human-human agreement numbers, we do not know whether the reported 36.2% exact count / 76.2% within-one and r=0.96 are near the ceiling of achievable agreement or far below it. The r=0.96 is additionally computed after discarding unmatched predictions (Section 3.3), which can inflate correlation. These issues do not weaken the central training comparison in Table 1 (SAM masks vs hand masks as training labels), but they do weaken the grading-utility component of the claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a weakly supervised pipeline for defect-based grading of fresh produce: SAM is prompted with coarse annotations (bounding boxes for defects, points for bananas) to generate dense panoptic masks, which are then used to train a Maskformer panoptic segmentation model. Experiments on 476 field images with 1,440 banana defects compare training on SAM-generated masks versus fully hand-annotated masks, evaluated against human-annotated ground truth. The central result is that both training label sources achieve nearly identical panoptic quality (PQ 0.779) on human validation. The paper also reports defect-count accuracy (36.2% exact, 76.2% within one) and a relative-size correlation of r=0.96 against a single expert's annotations, and it claims the annotation workload is reduced by at least a factor of 10.","tokens_in":12591,"tokens_out":5901,"duration_ms":53528,"significance":"If the central result holds, this is a practical contribution to low-data agricultural inspection: it demonstrates that foundation-model-generated masks can substitute for expensive manual mask annotation, with a sound experimental design that separates training and validation label sources. The paper is honest about failure cases (small defects, thin scars), includes a useful ablation of SAM backbone sizes, and releases code. However, the grading-utility claims (count and size accuracy) are currently supported only by agreement with a single annotator, the size correlation excludes non-matchable defects, the count and size metrics lack cross-validation variance, and the annotation-reduction factor is unmeasured. These issues do not undermine the central training comparison but need to be addressed before the broader claims are fully supported.","major_comments":[{"comment":"The definition of relative defect size is inconsistent. Section 2.6 defines it as defect pixels divided by foreground banana pixels, whereas Section 3.3 states that the denominator is 'the aggregated number of pixels assigned to either the foreground banana class or any defect.' Please clarify which denominator was used to compute Figure 5 and reconcile the text, since the correlation value depends on this choice.","section":"Section 2.6 vs Section 3.3"},{"comment":"The Pearson correlation r=0.96 is computed only on defect pairs matched with IoU >= 0.5, and non-matchable predicted defects are excluded. This selection can inflate the correlation. Report the fraction of predicted and annotated defects excluded, include an analysis that accounts for unmatched predictions (e.g., a matching-penalized metric or a sensitivity analysis at different IoU thresholds), and quantify the systematic underestimation visible from the regression slope deviating from the diagonal.","section":"Section 3.3, Figure 5"},{"comment":"The count accuracy percentages (36.2% exact, 76.2% within one) are reported without the cross-validation variance that is provided for Table 1. Please report these metrics per fold as mean and standard deviation, and specify which model configuration and postprocessing setting were used to produce Figure 4, so the reader can assess the robustness of the count claim.","section":"Section 3.3, Figure 4"},{"comment":"The statement that manual annotation workload is reduced 'by at least a factor of 10' is an estimate without supporting measurement. Provide annotation-time data for bounding-box/point prompts versus full mask annotation, or revise the claim to clearly indicate that it is an assumption. Since the abstract and discussion advertise this reduction, it should be substantiated or appropriately qualified.","section":"Section 4"},{"comment":"All quantitative claims rest on annotations from a single expert, and the paper itself notes that defect boundaries and categories are often ambiguous. Without an inter-annotator agreement baseline, the reported counting and size accuracies cannot be interpreted as practical grading accuracy. The paper should present the count and size results as agreement with one expert rather than as absolute accuracy, and should state this limitation in the abstract and conclusion, not only in the future-work section.","section":"Sections 2.2, 4, 5"}],"minor_comments":[{"comment":"The row with SAM-generated masks for both training and validation (PQ 0.782) is a self-consistency check rather than a direct comparison to human-validated rows. The text describing it as a 'slightly higher PQ score' may mislead; please clarify that the primary comparison for label-source equivalency is between the SAM-train/Anno-val row and the Anno-train/Anno-val row.","section":"Section 3.2, Table 1"},{"comment":"The postprocessing step 3 states that instances within a distance of 2d pixels are merged, but this follows from a dilation by d pixels only implicitly. State explicitly that two instances are merged if their dilated masks overlap, i.e., if their original boundaries are within 2d pixels.","section":"Section 2.5"},{"comment":"The x-axis bin labels are truncated in the rendered figure (e.g., '[10 , 1024²)' appears to be missing the exponent 4). Please update the figure or its caption so that all bin boundaries are legible.","section":"Figure 2"},{"comment":"The green and orange color coding in the count confusion matrix is not visible in grayscale reproductions. Add a legend or use distinct hatching to indicate the exact-match and within-one regions.","section":"Figure 4"},{"comment":"The text reports n=793 matched defect pairs for the size analysis, but the dataset contains 1,440 annotated defects. Please define 'non-matchable' precisely and report how many annotated and predicted defects were excluded, so the reader can gauge the coverage of the correlation analysis.","section":"Section 3.3"},{"comment":"The pixel-ratio proxy for relative defect size is acknowledged as a simplification, but the paper does not discuss the varying camera distances and perspectives across the dataset. Add a sentence noting that these variations affect the pixel ratio and that the proxy is only meaningful for images taken under similar acquisition conditions.","section":"Section 2.6"}],"recommendation":"major_revision","confidential_remarks":"The core training comparison (SAM-generated vs hand-annotated masks) is well designed and appears sound, so the paper is not fatally flawed. The requested revisions concern the grading-utility claims: they need either additional experiments (cross-validation variance for count/size, annotation-time measurement) or careful reframing (single-annotator ground truth, exclusion of unmatched defects). Given that the paper is currently accepted at a CVPR workshop, the authors may be able to make these changes in a short revision cycle. One data-level limitation—the lack of a second annotator—cannot be fully fixed without new data, but the authors should at least temper the abstract and discussion claims accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper's central claim is solid: a MaskFormer trained on SAM-generated masks from coarse bounding-box prompts reaches essentially the same panoptic quality on human ground truth as one trained on fully hand-annotated masks (PQ 0.779 vs 0.779 with postprocessing, Table 1). That is a genuinely useful empirical result for low-data agricultural vision, and the three-way train/val design (human-human, SAM-SAM, SAM-train/human-val) is the right way to measure error propagation. The paper also earns credit for explicitly showing where SAM fails—small defects, thin scars—and for being honest that four-class categorization does not work.\n\nWhat is new here is not the architecture. Combining SAM pseudo-labels with MaskFormer training is straightforward, and the paper misses box-supervised baselines that already train instance segmentation from boxes. The novelty is the domain dataset and the careful broken-down evaluation. The 476-image banana defect set, with 1,440 instances, is a real contribution, even if the dataset itself is not released.\n\nNow the soft spots. The stress-test concern is the important one. The count and size numbers are computed against a single expert's annotations. The paper itself recommends multi-annotator agreement as future work, but that makes the reported 36.2% exact count and r=0.96 hard to interpret right now—without a human-human baseline, we cannot tell whether the model is near the ceiling of achievable agreement or well below it. The r=0.96 is also computed after discarding non-matchable defects, which can inflate the correlation. The relative-size proxy (defect pixels divided by foreground banana pixels from an uncalibrated photo) is a reasonable practical choice, but it is never checked against physical severity, so the grading-utility claim is only as strong as that proxy. Also minor: the SAM-SAM row (PQ 0.782) is self-consistency, not evidence against the human-val setup, and the \"at least 10x less annotation\" estimate is not measured.\n\nThese are addressable issues. The central training comparison is not weakened by the single-annotator problem, because both training-label sources are evaluated on the same human ground truth. A serious referee would want sensitivity analyses, an inter-annotator study, and better reporting of count accuracy variance. I would send this to review, not desk-reject it. I would not cite the size-correlation number in my own work until the matching bias and annotator baseline are addressed.","headline":"The central SAM-vs-human training comparison is honest and holds up, but the grading-utility numbers (count accuracy, size correlation) rest on a single annotator and need an inter-annotator baseline before they mean what the paper suggests.","tokens_in":13160,"tokens_out":1347,"would_cite":false,"duration_ms":128488,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training on SAM-generated masks matches hand-annotated training for banana defect grading.","keywords":["weakly supervised segmentation","panoptic segmentation","Segment Anything Model","defect grading","banana surface defects","low-data regime","foundation models","agricultural computer vision"],"falsifier":"Photograph the same set of banana bunches with fixed physical defects from several distances and angles and check whether the relative defect-size ratio stays within the tolerance required for grading; if the ratio swings across viewpoints for the same physical defect, the size proxy fails and the reported r=0.96 does not translate into usable grading decisions.","tokens_in":12178,"feed_emoji":"🍌","tokens_out":6498,"duration_ms":55002,"temperature":0.7,"pith_summary":"This paper tries to establish that a panoptic segmentation model can be trained on dense masks auto-generated by the Segment Anything Model (SAM) from coarse inputs—bounding boxes around defects and points on bananas—instead of on painstaking hand-drawn masks. Working with 476 field photos of banana bunches containing 1440 annotated defects, the authors find that training on SAM masks yields panoptic quality (PQ 0.779) identical to training on fully manual masks when both are evaluated against human ground truth, while cutting annotation workload by at least a factor of ten. They also show the trained model predicts the exact number of defects in 36.2% of images, is off by at most one in 76.2%, and estimates relative defect sizes with correlation r=0.96 against human annotation. The value of establishing this is practical: if it holds, defect-based grading of fresh produce becomes feasible in decentralized supply chains where large labeled datasets do not exist. The paper is candid about boundaries, reporting that SAM misses very small defects, over-segments long thin scars, and that four-class defect categorization is not reliable on this data.","feed_headline":"Auto masks match hand labels in banana defect grading","feed_subtitle":"Training on SAM-generated masks cuts annotation effort 10x with equal panoptic quality on 476 field images.","key_machinery":"The machinery is a two-stage weak-supervision pipeline. First, SAM—the promptable Segment Anything Model—turns sparse prompts (bounding boxes around defects, click points on bananas) into dense instance masks without any training, with a ViT-L or ViT-H image encoder as the promptable backbone; this converts coarse annotations into pixel-level labels. Second, a MaskFormer model, pretrained on the ADE20k scene-parsing dataset, is trained on these SAM-generated masks to output panoptic predictions labeling Foreground Banana, Background Banana, and Defect instances. A postprocessing step then merges defect instances of the same class whose dilated masks overlap within a 2d-pixel distance (d=5), which notably improves defect AP. Finally, grading metrics are derived from the panoptic output: defect count from instance enumeration, and relative defect size from the ratio of defect pixels to foreground-banana pixels.","core_discovery":"The central claim is that SAM-generated masks are a drop-in substitute for hand-annotated masks in training a MaskFormer panoptic segmentation model for banana surface defects. With the same postprocessing, the model trained on SAM-L masks scores PQ 0.779 against hand-annotated validation, exactly matching the PQ 0.779 of the model trained on fully annotated masks, while defect IoU drops only from 56.3% to 56.2% and defect AP from 22.3% to 21.7%. A larger SAM backbone (ViT-H) improves SAM-to-human mask alignment but does not improve downstream segmentation metrics, implying the training model absorbs some mask noise. The derived grading quantities are also close: exact defect count in 36.2% of images, within one defect in 76.2%, and a Pearson correlation of r=0.96 between predicted and annotated relative defect sizes across 793 paired defect instances. The four-way defect categorization (old/new bruise/scar) is reported as unreliable and is therefore not recommended on this dataset.","pith_inferences":["A natural next test is to calibrate the relative-size proxy against physical measurements (e.g., ruler or 3D scans) at multiple camera distances and angles; the paper does not provide that calibration, and the proxy is only faithful if viewpoint variation stays within grading tolerance.","SAM's failure below roughly 100 pixels of defect area implies a sensitivity floor: very small blemishes will be systematically missed, which sets a lower bound on what this pipeline can grade.","Because the paper uses a single annotator, the reported human ground truth itself contains ambiguity; multi-annotator agreement studies would show how much of the remaining error is model error versus annotation noise.","If the same weak-supervision recipe works for other commodities, the practical consequence is that grading pilots can be launched from a few hundred smartphone photos instead of large curated datasets."],"forward_implications":["Annotation effort for dense panoptic labels drops by at least a factor of 10: only boxes and points are needed, not pixel-wise masks.","A model trained on fewer than 500 field images with SAM-generated labels reaches the same panoptic quality as one trained on fully manual labels, so the approach is viable in low-data, decentralized settings.","The postprocessing merge step is a cheap, model-agnostic way to improve defect detection accuracy and can be applied to any semantic segmentation output.","Defect count and relative size, the two quantities most grading schemes need, are recoverable from panoptic masks with the reported accuracy.","The approach transfers, in principle, to other produce types whose grading depends on countable, sizable blemishes."],"supporting_citations":[{"why":"Supplies the MaskFormer architecture used as the panoptic segmentation model.","marker":"[4]"},{"why":"Supplies the Segment Anything Model that generates dense masks from coarse prompts.","marker":"[15]"},{"why":"Defines panoptic segmentation and the PQ metric that anchors the evaluation.","marker":"[14]"},{"why":"Provides the scene-parsing dataset on which MaskFormer is pretrained.","marker":"[30]"},{"why":"Defines the average-precision and average-recall metrics used to evaluate defect instances.","marker":"[18]"},{"why":"Provides the connected-components implementation used in the postprocessing merge step.","marker":"[20]"},{"why":"Provides the open-source labeling tool used to create the human annotations that form ground truth.","marker":"[24]"}],"fun_headline_variants":["SAM masks match hand labels for banana defect grading","Training on SAM masks cuts annotation effort for produce defects","Auto masks rival manual masks in banana defect segmentation","Segment Anything masks enable low-annotation produce grading","Zero-shot SAM masks yield same panoptic quality as manual"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that relative defect size—defect pixels divided by foreground banana pixels in one uncalibrated 2D photo—faithfully reflects physical defect severity; without calibration, the same physical defect can look different at different distances, angles, and occlusions.","fun_headline_variants_meta":{"raw":{"variants":["SAM masks match hand labels for banana defect grading","Training on SAM masks cuts annotation effort for produce defects","Auto masks rival manual masks in banana defect segmentation","Segment Anything masks enable low-annotation produce grading","Zero-shot SAM masks yield same panoptic quality as manual"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000292,"raw_usage":{"total_tokens":1691,"prompt_tokens":921,"completion_tokens":770,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":696}},"tokens_in":537,"tokens_out":770,"duration_ms":8264,"temperature":1.0,"reasoning_tokens":696,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:21:23.075717+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Photograph the same set of banana bunches with fixed physical defects from several distances and angles and check whether the relative defect-size ratio stays within the tolerance required for grading; if the ratio swings across viewpoints for the same physical defect, the size proxy fails and the reported r=0.96 does not translate into usable grading decisions.","supporting_citations":[{"cited_title":"Schwing, and Alexander Kirillov","cited_arxiv_id":null,"evidence_quote":"Supplies the MaskFormer architecture used as the panoptic segmentation model."},{"cited_title":"Panoptic segmen- tation","cited_arxiv_id":null,"evidence_quote":"Defines panoptic segmentation and the PQ metric that anchors the evaluation."},{"cited_title":"Scene Pars- ing through ADE20K Dataset","cited_arxiv_id":null,"evidence_quote":"Provides the scene-parsing dataset on which MaskFormer is pretrained."},{"cited_title":"Microsoft coco: Com- mon objects in context","cited_arxiv_id":null,"evidence_quote":"Defines the average-precision and average-recall metrics used to evaluate defect instances."},{"cited_title":"Open Source Computer Vision Library,","cited_arxiv_id":null,"evidence_quote":"Provides the connected-components implementation used in the postprocessing merge step."},{"cited_title":"Label Studio: Data labeling software, 2020","cited_arxiv_id":null,"evidence_quote":"Provides the open-source labeling tool used to create the human annotations that form ground truth."}],"review_version":1}