{"id":"4824e015-dcef-4022-a5ac-978abec75386","arxiv_id":"2505.16513","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"An evaluation of RGB-based deep learning models for plastic sorting shows they often attend to backgrounds and physical appearance, prompting caution about their real-world accuracy.","lead":"This paper evaluates how well modern computer vision models sort plastic by type using new image datasets from recycling plants and other sources. It finds that these models often rely on surface clues like shape and color, and sometimes look at backgrounds such as conveyor belts, suggesting limits for real-world optical sorting.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The conclusion that RGB optical sorting has limited success at MRFs is not supported by the presented experiments: the key MRF evidence comes from a 70-image training set and ImageNet-pretrained models that were never fine-tuned, with no measured sorting accuracy.","rationale":"The most load-bearing step in the paper is the inference from activation visualizations to a general limitation of optical sorting. That inference fails at a specific technical point: the models whose heatmaps are shown for the MRF setting were not trained for the MRF classification task. Xception and ResNet-50 are introduced as ImageNet-pretrained models applied to MRF data (Section 4.1.1, items 4-5); their attention falling on the conveyor belt is expected under domain shift and does not measure what a fine-tuned sorter would learn. The only MRF-trained detector uses 70 images with no reported quantitative accuracy, while the strong results on cleaner datasets (96% web-scraped, 80% mAP US Plastics) cut the other way. Without an end-to-end sorting metric or a comparison to NIR-based systems, the abstract's conclusion is a hypothesis, not a finding. The proposed check—fine-tuning on a proper MRF split and measuring held-out accuracy—would settle whether the central claim has any empirical basis. This concern is distinct from but aligned with the reader's weakest assumption: the reader emphasized the equation of optical recognition with RGB CNNs and the absence of end-to-end evaluation; I further emphasize that the MRF heatmaps come from models that were never fine-tuned on the target domain, so they cannot even serve as evidence about a trained RGB sorter.","tokens_in":7587,"tokens_out":3887,"duration_ms":31293,"concrete_test":"Fine-tune a Mask R-CNN (or equivalent detector) on a larger split of the MRF dataset—e.g., 600 training and 200 validation images from the 800 collected—and report mAP, per-class accuracy, and confusion matrices on held-out MRF images. If the fine-tuned model attains high accuracy on deformed and contaminated plastics, the paper's central conclusion is unsupported; if it fails, the conclusion would gain empirical support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that optic recognition methods have limited success in accurate sorting of real-world plastics at MRFs because they rely on physical properties such as color and shape. For this claim to hold, the experiments must show that a properly trained optical-vision system fails on the MRF task. That condition is not met. In Section 4.1.1, Mask R-CNN on the MRF dataset is trained on 50 images and validated on 20; no mAP or accuracy is reported for this setting. The Xception and ResNet-50 gradient-activation experiments (Section 4.1.1, items 4-5) use ImageNet-pretrained checkpoints without any MRF fine-tuning; their Grad-CAM maps concentrating on conveyor belts and cans reflect domain shift and do not characterize a trained optical sorter. The OFF ResNet-34 result (71.8% accuracy) is on manually cleaned food-label images, not an MRF stream, and the US Plastics Mask R-CNN reaches about 80% mAP, which weakens the broad claim. No comparison with deployed NIR/optical sorting systems or end-to-end sorting rates is provided. The paper's own acknowledged limitations—clean web/catalog images, small MRF set—make the generalization from qualitative heatmaps to 'limited success' of optical sorting untenable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper curates a collection of more than 20,000 images of plastics from web sources, a plastics manufacturer's catalog, the Open Food Facts database, and a local Materials Recovery Facility (MRF). It applies several modern computer-vision models—Mask R-CNN, ResNet-34, Xception, and ResNet-50—to classify or segment plastics, and interprets their behavior using Grad-CAM, feature maps, and confusion matrices. The central claim, stated in the abstract and Section 6, is that optic recognition methods have limited success in accurate sorting of real-world plastics at MRFs because they rely on physical properties such as color and shape, and the paper recommends multi-modal sensing as a remedy.","tokens_in":7780,"tokens_out":3483,"duration_ms":29090,"significance":"If established, the claim that RGB-based optical sorting is fundamentally limited at MRFs would be of considerable practical importance, motivating investment in multi-modal or spectral approaches. The compiled datasets could also serve as a useful community resource. However, the experiments as presented do not support the broad conclusion: the MRF evaluation relies on an extremely small training set and on off-the-shelf pretrained models that were never fine-tuned, no end-to-end sorting accuracy is measured, and several results (80% mAP on the US Plastics dataset, 71.8% accuracy on Open Food Facts) actually indicate that optical models can perform reasonably well when training data are adequate. The paper's own acknowledged limitations—clean catalog images, a small MRF set, and manual cleaning—undercut the generalization from qualitative heatmaps to a verdict on optical sorting technology.","major_comments":[{"comment":"The Mask R-CNN experiment on the MRF dataset is trained on only 50 images and validated on 20, and the paper reports no detection or segmentation accuracy (e.g., mAP, precision/recall) for this setup. Without any quantitative performance measure, the reader cannot assess whether the model 'successfully identifies' the plastic objects, and the claim that optical recognition has limited success at MRFs rests entirely on a statistically insignificant sample with no error bars.","section":"Section 4.1.1, item 1"},{"comment":"The Xception and ResNet-50 Grad-CAM experiments use ImageNet-pretrained checkpoints without any fine-tuning on MRF images. The observation that these models attend to conveyor belts and metal cans is an expected symptom of domain shift, not a property of a trained optical sorter. Interpreting these heatmaps as evidence that 'optic recognition methods rely on physical properties such as color and shape' is therefore not justified; a valid test would require models trained or fine-tuned on the target task.","section":"Section 4.1.1, items 4 and 5"},{"comment":"The Open Food Facts result (71.8% accuracy with ResNet-34) is obtained on manually cleaned images of food packaging, and the paper itself acknowledges that this dataset is limited to food products and contains no deformed plastics. This is not a proxy for MRF conditions, and the 71.8% accuracy is a moderate result that does not support the broad conclusion about limited success at MRFs.","section":"Section 4.1.1, item 3 and Section 4, item 3"},{"comment":"The central conclusion—that optic recognition methods have limited success in accurate sorting of real-world plastics at MRFs—is not supported by the presented evidence because the paper never measures end-to-end sorting accuracy, never compares with deployed NIR/optical sorting systems, and bases its qualitative interpretation on a handful of selected Grad-CAM images. In contrast, the Mask R-CNN trained on the US Plastics dataset reaches approximately 80% mAP (Section 4.1.1, item 2), which demonstrates that an optical system can perform well when sufficient training data are available; this weakens the paper's sweeping conclusion and suggests the authors should restrict their claims to the specific under-trained models they tested.","section":"Section 5 and Abstract"}],"minor_comments":[{"comment":"The affiliation 'NIT T richy' contains a typo; it should be 'NIT Tiruchirappalli' or similar.","section":"Author affiliations"},{"comment":"Reference [8] is mis-cited: the entry 'W. Yoshida, R. J. Dolan, and K. J. Friston, Game theory of mind' does not correspond to SPP-net; the proper SPP-net citation (He et al., 2015) appears to be missing.","section":"References"},{"comment":"The description of the manual cleaning of Open Food Facts images is vague; please specify how images were selected and how many were discarded.","section":"Section 4.1.1, item 3"},{"comment":"The sentence 'We trained 70 (50 train and 20 validation) images' is ambiguous: it should state whether this is the total number of images or per class, and it should list the number of plastic categories.","section":"Section 4.1.1, item 1"},{"comment":"The paper uses inconsistent spellings such as 'Grad-CAM' and 'Grad CAM'; please standardize the terminology.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The paper has a useful dataset-collection effort and an honest acknowledgment of limitations, but the central claim is not supported by the experiments as designed. The MRF experiment is far too small, the pretrained-model Grad-CAM analysis is not a valid evaluation of a trained optical sorter, and no comparison with deployed systems or end-to-end sorting accuracy is provided. A revision that reframes the paper as a dataset contribution or as a study of off-the-shelf model limitations on a specific benchmark might be suitable for a workshop or a more specialized venue, but in its current form the manuscript does not meet the bar for a journal publication. The reference mis-citation should also be corrected regardless of venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper is a useful dataset-collection effort wrapped in a conclusion the experiments don't support. The compiled datasets from the MRF, Open Food Facts, and US Plastics Corp. are real work, and the authors are honest that clean catalog images don't reflect real recycling streams. The observation that CNNs key on conveyor belts and product labels is not new, but documenting it on an actual MRF conveyor is a reasonable data point.\n\nThe central claim, though, is not supported. The Mask R-CNN on the MRF dataset was trained on 50 images and validated on 20, with no reported mAP or accuracy. The Xception and ResNet-50 Grad-CAM experiments use ImageNet-pretrained checkpoints with no fine-tuning; of course they attend to conveyor belts and cans—that's domain shift, not evidence about a trained optical sorter. Meanwhile, the same kind of architecture reaches 80% mAP on the US Plastics data and 71.8% accuracy on the OFF data, which suggests the MRF difficulty may be about data volume and domain adaptation rather than an inherent limit of RGB-based recognition. The paper never measures end-to-end sorting throughput or compares against NIR-based systems, so the phrase \"limited success\" is an interpretation of heatmaps, not a measured outcome.\n\nThe paper's own acknowledged limitations—clean web/catalog images, small MRF set—make the generalization from qualitative activation maps to a verdict on commercial optical sorting untenable. That is a load-bearing flaw, so the current version should not be accepted as is.\n\nThat said, the datasets are the real contribution, and the paper deserves a serious referee rather than a desk reject. The problem is practically important and the data collection is non-trivial. I'd send it to review with the expectation of heavy revision: either narrow the claims to what the evidence actually shows, or add the missing experiments (fine-tuned models on a larger MRF set, measured sorting accuracy, comparison with NIR). If the authors release the datasets, this could become a useful benchmark paper.","headline":"Useful dataset work wrapped in an unsupported conclusion: the broad claim that RGB optic sorting fails at MRFs doesn't follow from 50 training images and untuned ImageNet models.","tokens_in":8312,"tokens_out":2225,"would_cite":false,"duration_ms":19934,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper finds that RGB-only computer vision sorts plastics by appearance, not material identity, so real-world sorting needs extra modalities.","keywords":["plastic sorting","optical recognition","deep learning","Grad-CAM","instance segmentation","materials recovery facility","plastic classification","multimodal sensing"],"falsifier":"Take a single mixed-plastics conveyor, sort with an RGB-only detector, a near-infrared sensor, and a fused system, and audit ground-truth resin codes by hand: if the RGB-only detector matches the NIR sensor on deformed and contaminated items, the paper's central claim fails. Alternatively, retrain on images with labels and colors removed; high accuracy with attention on the plastic body would contradict the claim that models rely on those cues.","tokens_in":7377,"feed_emoji":"♻️","tokens_out":7963,"duration_ms":66448,"temperature":0.7,"pith_summary":"Using 20,000+ images from clean catalogs, web sources, consumer-packaging photos, and a working materials recovery facility, the paper trains standard deep-learning detectors and classifiers and inspects where their attention falls. It finds that models key on physical appearance—object shape, color, packaging labels, and even conveyor-belt background—rather than on the material itself, so accuracy drops sharply from clean images to real-world streams. The paper's central claim is that optic (RGB-image) recognition methods have limited success in accurate sorting of real-world plastics at MRFs precisely because they rely on such physical properties. If true, this reframes the sorting problem: reliable resin identification needs additional modalities, such as near-infrared spectroscopy, fused with vision.","feed_headline":"Optical sorting models chase color and shape, not plastic type","feed_subtitle":"Grad-CAM heatmaps show attention on belts and labels; the fix is fusing vision with spectroscopy.","key_machinery":"The argument is carried by pairing object-detection and classification models with attribution visualizations: Mask R-CNN for instance segmentation, ResNet-34, ResNet-50, and Xception for classification, and Grad-CAM and intermediate feature maps to localize the image regions that drive each prediction. Grad-CAM (gradient-weighted class activation mapping) produces heatmaps of the pixels most responsible for a classification, and those heatmaps are the evidence that the model is keying on conveyor belts, labels, or shape rather than polymer identity.","core_discovery":"On the paper's own terms, the central discovery is an empirical diagnosis of why modern optical sorting models underperform on real materials-recovery streams. Trained models reach roughly 80% mean average precision on clean factory-catalog images and about 96% classification accuracy on clean web-scraped plastics, but accuracy falls to 71.8% on consumer-packaging photos, and the MRF data expose the mechanism: Grad-CAM and feature maps show the network attending to conveyor belts, beverage labels, metal cans nearby, and shape outlines instead of the plastic polymer surface. From this the authors conclude that optic recognition, as embodied by these RGB-based CNN models, does not reliably identify plastic type in contaminated, deformed, mixed streams, and they recommend combining vision with spectroscopic modalities.","pith_inferences":["Editorial — the same attention-shortcut diagnosis likely applies to other recycling domains (glass color sorting, e-waste) where visual appearance only weakly correlates with material value.","Editorial — a quantitative test the authors did not run is to compare end-to-end sorting accuracy of an RGB-only CNN against a co-located NIR sensor on the same conveyor; the attention maps predict RGB will lose.","Editorial — the paper's evidence is consistent with, but does not prove, the stronger claim that no amount of RGB training data can close the gap, because resin chemistry is not visually encoded; a training-data scaling study would be needed to test that.","Editorial — if recycling facilities already use NIR for resin typing, the practical implication is not to replace optics but to use vision for object localization and NIR for material classification."],"forward_implications":["Clean-dataset accuracy claims for plastic sorters should not be read as real-world performance; the same models lose more than 20 percentage points on consumer-packaging images and focus on background cues on MRF footage.","Any RGB-only sorting deployment at a materials recovery facility should be validated with attribution analysis before being trusted for resin-type decisions.","Spectroscopic sensors (near-infrared, Raman, FTIR) become the natural complement; vision can localize objects while the spectral channel identifies material.","The compiled multi-source image collection gives later work a concrete benchmark for measuring whether a model classifies by material identity rather than by appearance."],"supporting_citations":[{"why":"The commercial optical-sorting deployment whose practical setting the paper's conclusion targets.","marker":"[1]"},{"why":"A second commercial optical-sorting system used as a real-world reference in the paper's motivation.","marker":"[2]"},{"why":"Supplies the Mask R-CNN instance-segmentation architecture whose feature maps the paper analyzes on both clean catalog and MRF images.","marker":"[7]"},{"why":"Prior work reporting high shape-based PET/PET-G classification accuracy, the kind of appearance-driven result the paper argues does not transfer to real streams.","marker":"[22]"},{"why":"Provides the multi-modal spectral database and cross-modal classification results that ground the paper's recommendation to fuse vision with spectroscopy.","marker":"[25]"},{"why":"Supplies Grad-CAM, the attribution method used to show that models attend to conveyor belts, labels, and other non-material regions.","marker":"[26]"}],"fun_headline_variants":["AI sorters fail on real plastics: they see belt, label, shape","Optic sorting models learn belt and label, not polymer type","Why robot pickers miss plastic type: they focus on color and shape","Grad-CAM shows sorters eye belts, cans, labels—not polymers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the RGB-image CNNs trained here represent 'optic recognition' generally, so that attention maps focusing on conveyor belts, labels, and shape are enough to show optical sorting is unreliable; the paper does not test NIR sensors or measure end-to-end sorting accuracy.","fun_headline_variants_meta":{"raw":{"variants":["AI sorters fail on real plastics: they see belt, label, shape","Optic sorting models learn belt and label, not polymer type","Why robot pickers miss plastic type: they focus on color and shape","Grad-CAM shows sorters eye belts, cans, labels—not polymers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000179,"raw_usage":{"total_tokens":1335,"prompt_tokens":1013,"completion_tokens":322,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":242}},"tokens_in":629,"tokens_out":322,"duration_ms":3294,"temperature":1.0,"reasoning_tokens":242,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:58:20.238481+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a single mixed-plastics conveyor, sort with an RGB-only detector, a near-infrared sensor, and a fused system, and audit ground-truth resin codes by hand: if the RGB-only detector matches the NIR sensor on deformed and contaminated items, the paper's central claim fails. Alternatively, retrain on images with labels and colors removed; high accuracy with attention on the plastic body would contradict the claim that models rely on those cues.","supporting_citations":[{"cited_title":"AMP robotics recycling reimagined","cited_arxiv_id":null,"evidence_quote":"The commercial optical-sorting deployment whose practical setting the paper's conclusion targets."},{"cited_title":"Grey parrot waste intelligence","cited_arxiv_id":null,"evidence_quote":"A second commercial optical-sorting system used as a real-world reference in the paper's motivation."},{"cited_title":"Mask r-cnn,","cited_arxiv_id":null,"evidence_quote":"Supplies the Mask R-CNN instance-segmentation architecture whose feature maps the paper analyzes on both clean catalog and MRF images."},{"cited_title":"Advancing plastic waste classification and recycling efficiency: Integrating image sensors and deep learning algorithms,","cited_arxiv_id":null,"evidence_quote":"Prior work reporting high shape-based PET/PET-G classification accuracy, the kind of appearance-driven result the paper argues does not transfer to real streams."},{"cited_title":"Cross-modal generative models for multi- modal plastic sorting,","cited_arxiv_id":null,"evidence_quote":"Provides the multi-modal spectral database and cross-modal classification results that ground the paper's recommendation to fuse vision with spectroscopy."},{"cited_title":"Grad-cam: Visual explana- tions from deep networks via gradient-based localization,","cited_arxiv_id":null,"evidence_quote":"Supplies Grad-CAM, the attribution method used to show that models attend to conveyor belts, labels, and other non-material regions."}],"review_version":1}