REVIEW 2 major objections 6 minor 27 references
MDWD: A Street-Level Dataset for Municipal Solid Waste Detection in Dense Urban Environments
T0 review · 2 major / 6 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read This paper introduces MDWD, a street-level benchmark of 3,697 images and 11,461 labeled waste instances across five municipal waste streams, and shows that detectors trained on it reach high accuracy (RF-DETR-M mAP50 94.49%, F1 93.56%).
desk verdict Solid, well-documented dataset paper with a real gap to fill; the benchmark numbers are conditional on annotation consistency that is asserted but unmeasured. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the dataset itself, with its five-category taxonomy tied directly to Malta's color-coded collection system. The annotation protocol — manual bounding boxes, a '~20% visible' rule, single lead-annotator QA pass, and exclusion of burst captures and video frames — determines the label quality. The evaluation protocol fixes an 80/10/10 split, offline augmentation (ten variants per training image), and a shared training recipe for YOLO-family models, with RF-DETR trained on the same augmented data, so that reported scores measure dataset learnability rather than fine-tuned architecture differences.
What would settle it
Re-annotate a random subset of, say, 200 MDWD images with at least two independent annotators and compute inter-annotator agreement (e.g., IoU and category kappa); also run a duplicate/near-duplicate search (perceptual hash or embedding similarity) across the train/test split. If agreement is low or duplicates appear across splits, the benchmark scores are not a trustworthy signal.
Extended reading notes
Core claim
MDWD is a public benchmark of 3,697 street-level images annotated with 11,461 axis-aligned bounding boxes across five waste categories: Mixed Waste (black bags), Organic Waste (white bags), Recyclable Material (gray/green bags), Orange CMD (distinctive orange bags), and Other Waste (a long-tail residual class). The paper's central claim is that this dataset occupies a position none of the reviewed waste datasets does — street-level context, instance-level localization, and multiple operationally defined domestic waste streams — and that the benchmark results (RF-DETR-M: mAP50 94.49%, F1 93.56%; all models above 83% F1 on test) show the annotations are learnable across CNN-based, attention-ce
Load-bearing premise
The reported mAP and F1 numbers depend on two unverified premises: that the 11,461 manual labels are correct and consistent, and that the 80/10/10 split contains no duplicate or near-duplicate images of the same waste pile.
Editorial extensions
If this is right
- A public, green-field benchmark now exists for instance-level municipal waste detection from street imagery, letting future work compare detectors on a common protocol.
- Because RF-DETR and YOLO variants all exceed 83% test F1, the dataset is learnable enough that accuracy differences below a few points likely reflect architecture choice rather than annotation noise.
- The weakest class, Other Waste (mAP50 93.2, recall 83.5), gives a concrete target for long-tail waste categories.
- Validation-test gaps are small, so the split appears stable enough for model selection.
- Pre-trained checkpoints (COCO for YOLO, Objects365 for RF-DETR) transfer well, meaning the task aligns with generic object detection priors.
Reading between the lines
- If the dataset is adopted as a benchmark, the next test is whether detectors trained on it transfer to other cities with different bag colors and collection schemes; the paper does not address geographic transfer.
- The absence of an inter-annotator agreement statistic means the 11,461 boxes' reliability is untested; a natural follow-up is a label-quality study on a random subset.
- The fixed augmentation pipeline and single seed make the baseline reproducible, but also mean sensitivity to augmentation choices is unexplored.
- The orange CMD class's high precision/low recall pattern suggests a deployment system could use it for targeted collection alerts rather than general detection.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper introduces MDWD, a street-level benchmark for municipal solid waste detection comprising 3,697 high-resolution images and 11,461 manually annotated instances across five classes that mirror Malta's color-coded municipal collection streams (Mixed Waste, Organic Waste, Recyclable Material, Orange CMD, and Other Waste). The authors report an 80/10/10 train–validation–test split, offline augmentation, and a cross-architecture baseline study covering YOLO11, YOLO12, YOLO26, and RF-DETR at several scales. RF-DETR-M achieves the strongest test performance (mAP50 94.49%, F1 93.56%), and all evaluated models exceed 83% test F1. The paper claims that no existing dataset simultaneously provides street-level imagery, instance-level bounding boxes, and categorization into multiple operationally defined domestic waste streams, and that the results demonstrate the dataset supports effective detector training across CNN-based, attention-centric, and transformer-based paradigms.
Significance. If the underlying annotations and split are reliable, MDWD is a genuinely useful resource: it addresses a real gap, is publicly released on Roboflow, uses an operationally grounded taxonomy, and provides a fixed split with consistent validation-to-test gaps across ten models. The cross-architecture comparison is a practical strength, and the small val-test deltas in Table IV are encouraging evidence of split stability. However, the benchmark's value rests on the correctness of 11,461 manual boxes and on the independence of the train/test partitions, neither of which is currently measured. The stress-test concern about annotation consistency lands: the absence of inter-annotator agreement or label-noise statistics means the numbers in Table IV cannot yet be read as a trustworthy benchmark signal. With added annotation-consistency and duplicate-image analyses, this would be a solid benchmark contribution.
major comments (2)
- [Section III.A / Table IV] The ground-truth boxes are the benchmark signal, but their correctness is asserted rather than measured. The QA pass by a single lead annotator is described, yet no inter-annotator agreement statistic or label-noise estimate is reported. The protocol's own decision rules are most difficult exactly where the paper acknowledges visual ambiguity: 'approximately 20% visible' is subjective, and the boundaries between Mixed Waste (black bags) and Recyclable Material (gray/green bags), as well as the residual Other Waste class, are noted to be visually similar or heterogeneous. If the lead annotator imposed idiosyncratic color or boundary preferences, every metric in Table IV and the Section IV.A claim that MDWD 'supports effective detector training' would be compromised. Add an inter-annotator agreement study on a stratified sample (e.g., per-class kappa) or a label-error audit; without it the
- [Section III.A / III.B] Split independence is asserted but not verified. The text states that each image depicts a distinct waste pile and that burst captures and video frames were excluded, but no duplicate or near-duplicate analysis is reported. Since images were collected opportunistically on foot and from moving vehicles, the same pile could plausibly appear in more than one image with a slightly different viewpoint; if any such near-duplicates cross the 80/10/10 boundary, the small validation-to-test gaps in Table IV would reflect leakage rather than generalization. Report a pairwise image-similarity screen (e.g., perceptual hashing or feature matching) across partitions, or provide acquisition metadata demonstrating no same-pile overlap. This is load-bearing for the benchmark's generalization claim.
minor comments (6)
- [Section III.C / Table IV] The sentence 'Detector architecture and model scale constitute the sole independent variables' is overstated because RF-DETR was trained on the Roboflow cloud platform defaults with Objects365 pretraining, while the YOLO models use custom Ultralytics settings with COCO pretraining. This conflates architecture with infrastructure, optimizer, and pretraining. The paper already acknowledges this in the following paragraph; please qualify the statement explicitly or soften it.
- [Table IV] All results appear to come from a single training run (YOLO seed 42; RF-DETR seed not reported). For a benchmark paper, reporting variance over two or three seeds, or at least noting the single-run status in the table caption, would let readers gauge the stability of the reported differences.
- [Section IV.B] The class-level analysis for RF-DETR-M is reported only as unnumbered prose (e.g., 'mAP50 98.3', 'false negative count 46'). A table with per-class mAP50, precision, recall, and counts would improve reproducibility and make the long-tail discussion easier to verify.
- [Section IV.A] Precision, recall, and F1 are computed with 'a fixed-threshold matching procedure (confidence and IoU thresholds of 0.50)', while mAP50 and mAP50:95 follow each framework's native COCO-style evaluation. Please clarify whether the fixed threshold is applied to the same detections used for mAP and state the confidence threshold explicitly; this affects comparability of the P/R/F1 columns.
- [Section III.B / Table II] Vertical flipping is applied with 50% probability. For street-level waste imagery this creates unnatural gravity orientation and may introduce artifacts not present in deployment. This is not blocking, but a brief justification or ablation would strengthen confidence in the augmentation choice.
- [Various] Minor textual issues: Table IV has a formatting error in the YOLO26-L Precision T column ('97.18' with no space before the next value), and the class distribution caption in Figure 1 reports percentages that should be cross-checked against the 11,461 total (e.g., 27.7% for Mixed Waste).
Circularity Check
No significant circularity: benchmark numbers are measured on a held-out test split, and the dataset's categories and gap claim rest on external operational and literature anchors rather than on the paper's own outputs.
full rationale
The paper's derivation chain is self-contained as a benchmark contribution: it constructs a dataset with an externally anchored five-class taxonomy (Malta's color-coded municipal collection streams), performs manual annotation, fixes an 80/10/10 split before training, trains multiple detectors under a shared protocol, and reports measured test-set metrics. No fitted parameter is renamed as a prediction: the mAP/F1 values in Table IV are evaluations on a held-out test partition, not quantities solved for during the derivation. The central claim that MDWD 'supports effective detector training across a broad range of architectural paradigms' is directly supported by the reported test F1-scores, all above 83%, and is not a restatement of the dataset's definition. The gap claim is argued against named external datasets (TrashNet, TACO, GIGO, SODA, GVP, etc.), and while two citations involve overlapping authors (SODA [7] and the UAV litter review [8]), they are used as prior work and motivation, not as a load-bearing proof of the present results. The absence of an inter-annotator agreement statistic and the lack of duplicate-image analysis are reproducibility and validity concerns, but they do not constitute circularity: even if the labels were noisy, the reported results would still be measurements rather than conclusions forced by definition. There is no self-definitional step, no fitted input called a prediction, and no uniqueness theorem imported from the authors' prior work. Accordingly, the paper merits a circularity score of 0.
Assumptions & free parameters
free parameters (4)
- Motion blur augmentation strength =
up to 40 px
- Annotation visibility threshold =
20% visible
- Photometric augmentation ranges =
hue ±19, sat ±29%, brightness ±17%, exposure ±10%
- Precision/Recall/F1 matching thresholds =
confidence 0.50, IoU 0.50
assumptions (4)
- domain assumption Manual annotations are correct ground truth
- domain assumption Train/test split is leakage-free (each image a distinct waste pile)
- domain assumption COCO/Objects365 pretrained weights transfer to street-level waste imagery
- standard math COCO-style mAP and fixed-threshold P/R/F1 correctly summarize detector quality here
invented entities (1)
-
Five-class waste taxonomy with residual 'Other Waste' bucket
independent evidence
Cite this review
Pith. "Pith review of MDWD: A Street-Level Dataset for Municipal Solid Waste Detection in Dense Urban Environments." pith.science (2026). https://pith.science/paper/VZAY37FY
@misc{pith2026260800257,
author = {Pith},
title = {Pith review of: MDWD: A Street-Level Dataset for Municipal Solid Waste Detection in Dense Urban Environments},
year = {2026},
howpublished = {\url{https://pith.science/paper/VZAY37FY}},
note = {Machine review of arXiv:2608.00257}
}
read the original abstract
Automated visual monitoring of urban environments is a growing Computer Vision research area, but municipal solid waste detection remains under-represented in dedicated benchmark resources. Existing waste-related datasets predominantly address individual litter detection, aerial imagery, or image-level classification, and none simultaneously provide street-level imagery, instance-level localization, and categorization of domestic waste streams within a structured municipal collection context. This paper introduces the Maltese Domestic Waste Dataset (MDWD), a street-level benchmark comprising 3,697 high-resolution images and 11,461 manually annotated instances across five domestic waste categories representative of Malta's municipal collection system. The dataset captures substantial variation in location, illumination, object scale, occlusion, and urban context. To establish reproducible baselines, a cross-architecture benchmark is conducted across multiple generations of the YOLO family and a transformer-based detector. On the test set, RF-DETR-M achieves the strongest overall performance with an mAP50 of 94.49% and an F1-score of 93.56%, whilst smaller-capacity variants maintain competitive accuracy at substantially reduced parameter counts. These results indicate that MDWD supports effective training across both compact real-time detectors and transformer-based models, establishing a benchmark for future research in vision-based municipal waste monitoring.
Figures
Reference graph
Works this paper leans on
-
[1]
Exploring big data applications in sustainable urban infrastructure: A review,
D. V . Ogunkan and S. K. Ogunkan, “Exploring big data applications in sustainable urban infrastructure: A review,”Urban Governance, vol. 5, no. 1, pp. 54–68, 2025. [Online]. Available: https: //www.sciencedirect.com/science/article/pii/S2664328625000038
2025
-
[2]
Augmenting urban planning with computer vision: A review of the state-of-the-art,
R. Marasinghe, T. Yigitcanlar, X. Fu, and S. J. Quan, “Augmenting urban planning with computer vision: A review of the state-of-the-art,” Sustainable Cities and Society, vol. 138, p. 107209, 2026
2026
-
[3]
Municipal waste statistics,
Eurostat, “Municipal waste statistics,” Eurostat Statistics Explained, 2026, accessed: 2026-06-10. [Online]. Available: https://ec.europa.eu/e urostat/statistics-explained/index.php?title=Municipal_waste_statistics
2026
-
[4]
Municipal solid waste management and its impact: a review,
M. Adnan, A. Jha, S. Kumaret al., “Municipal solid waste management and its impact: a review,”International Journal of Advanced Research in Engineering and Technology (IJARET), vol. 11, no. 5, pp. 685–693, 2020
2020
-
[5]
Performance analysis of yolo, faster r-cnn, and detr for automated personal protective equipment detection,
R. Naufaldihanif, D. Kurniawan, and K. D. Tania, “Performance analysis of yolo, faster r-cnn, and detr for automated personal protective equipment detection,”Journal of Applied Informatics and Computing, vol. 9, no. 6, p. 3810–3820, Dec. 2025. [Online]. Available: https://jurnal.polibatam.ac.id/index.php/JAIC/article/view/11593
2025
-
[6]
Garbage vulnerable point monitoring using iot and computer vision,
R. Kumar, A. Lall, S. Chaudhari, M. Kale, and A. Vattem, “Garbage vulnerable point monitoring using iot and computer vision,” in2025 IEEE 11th World Forum on Internet of Things (WF-IoT), 2025, pp. 1–6
2025
-
[7]
Soda: A dataset for small object detection in uav captured imagery,
D. Pisani, D. Seychell, C. J. Debono, and M. Schembri, “Soda: A dataset for small object detection in uav captured imagery,” in2024 IEEE International Conference on Image Processing (ICIP), 2024, pp. 151–157
2024
-
[8]
Litter detection from aerial imagery: A review of uav-based approaches and deep learning techniques,
M. Bartolo, G. Hili, D. Seychell, M. Montebello, C. J. Debono, S. Formosa, and K. Makantasis, “Litter detection from aerial imagery: A review of uav-based approaches and deep learning techniques,” Multimedia Tools and Applications, vol. 85, p. 232, 2026. [Online]. Available: https://doi.org/10.1007/s11042-026-21440-1
Show all 27 references
-
[9]
A dataset for detection and segmentation of underwater marine debris in shallow waters,
A. Ðuraš, B. J. Wolf, A. Ilioudi, I. Palunko, and B. De Schutter, “A dataset for detection and segmentation of underwater marine debris in shallow waters,”Scientific Data, vol. 11, p. 921, 2024
2024
-
[10]
Taco: Trash annotations in context for litter detection,
P. F. Proença and P. Simões, “Taco: Trash annotations in context for litter detection,” 2020. [Online]. Available: https://arxiv.org/abs/2003.06975
2020 arXiv
-
[11]
A multi-level approach to waste object segmentation,
T. Wang, Y . Cai, L. Liang, and D. Ye, “A multi-level approach to waste object segmentation,”Sensors, vol. 20, no. 14, 2020. [Online]. Available: https://www.mdpi.com/1424-8220/20/14/3816
2020
-
[12]
Classification of trash for recyclability status,
M. Yang and G. Thung, “Classification of trash for recyclability status,” CS229 project report, vol. 2016, no. 1, p. 3, 2016
2016
-
[13]
Bottle detection in the wild using low-altitude unmanned aerial vehicles,
J. Wang, W. Guo, T. Pan, H. Yu, L. Duan, and W. Yang, “Bottle detection in the wild using low-altitude unmanned aerial vehicles,” in2018 21st International Conference on Information Fusion (FUSION), 2018, pp. 439–444
2018
-
[14]
Gigo, garbage in, garbage out: An urban garbage classification dataset,
M. Sukel, S. Rudinac, and M. Worring, “Gigo, garbage in, garbage out: An urban garbage classification dataset,” inMultiMedia Modeling, D.-T. Dang-Nguyen, C. Gurrin, M. Larson, A. F. Smeaton, S. Rudinac, M.- S. Dao, C. Trattner, and P. Chen, Eds. Cham: Springer International Pu...
2023
-
[15]
Intelligent computer vision applications for automated urban infrastructure monitoring,
M. Ashraf and R. Pavel, “Intelligent computer vision applications for automated urban infrastructure monitoring,”ISCSITR-International Journal of Computer Applications (ISCSITR-IJCA), vol. 7, no. 1, pp. 1–7, 2026. [Online]. Available: https://iscsitr.in/index.php/ISCSITR-IJC A...
2026
-
[16]
Towards general urban monitoring with vision-language models: A review, evaluation, and a research agenda,
A. Torneiro, J. D. Monteiro, P. Novais, P. Rangel Henriques, and N. F. Rodrigues, “Towards general urban monitoring with vision-language models: A review, evaluation, and a research agenda,”arXiv preprint arXiv:2510.12400, 2025. [Online]. Available: https://arxiv.org/abs/2510.12400
2025
-
[17]
A multimodal vision: Language framework for intelligent detection and semantic interpretation of urban waste,
V . M. Jonuzi and I. Mishkovski, “A multimodal vision: Language framework for intelligent detection and semantic interpretation of urban waste,”Informatics, vol. 13, no. 4, 2026. [Online]. Available: https://www.mdpi.com/2227-9709/13/4/57
2026
-
[18]
Harnessing deep learning for sustainable waste classification: An innovative approach using mobilenetv2 on a diverse bag dataset,
E. Jain and R. Kumar, “Harnessing deep learning for sustainable waste classification: An innovative approach using mobilenetv2 on a diverse bag dataset,” in2024 International Conference on Cybernation and Computation (CYBERCOM), 2024, pp. 335–339
2024
-
[19]
Enhancing waste management with automated garbage bag classification with resnet50v2,
——, “Enhancing waste management with automated garbage bag classification with resnet50v2,” in2024 Eighth International Conference on Parallel, Distributed and Grid Computing (PDGC), 2024, pp. 150– 154
2024
-
[20]
Yolov11: An overview of the key architectural enhancements,
R. Khanam and M. Hussain, “Yolov11: An overview of the key architectural enhancements,” 2024. [Online]. Available: https: //arxiv.org/abs/2410.17725
2024 arXiv
-
[21]
Yolov12: Attention-centric real-time object detectors,
Y . Tian, Q. Ye, and D. DOERMANN, “Yolov12: Attention-centric real-time object detectors,” inAdvances in Neural Information Processing Systems, D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen, Eds., vol. 38. Curran Associates, Inc., 2025, pp. 78...
2025
-
[22]
Ultralytics yolo26: Unified real-time end-to-end vision models,
G. Jocher, J. Qiu, M. Liu, S. Lyu, F. C. Akyon, and M. E. Kalfaoglu, “Ultralytics yolo26: Unified real-time end-to-end vision models,” 2026. [Online]. Available: https://arxiv.org/abs/2606.03748
2026 arXiv
-
[23]
Rf- detr: Neural architecture search for real-time detection transformers,
I. Robinson, P. Robicheaux, M. Popov, D. Ramanan, and N. Peri, “Rf- detr: Neural architecture search for real-time detection transformers,”
-
[24]
Sam 3: Segment anything with concepts,
N. Carion, L. Gustafson, Y .-T. Hu, S. Debnath, R. Hu, D. Suris, C. Ryali, K. V . Alwala, H. Khedr, A. Huang, J. Lei, T. Ma, B. Guo, A. Kalla, M. Marks, J. Greer, M. Wang, P. Sun, R. Rädle, T. Afouras, E. Mavroudi, K. Xu, T.-H. Wu, Y . Zhou, L. Momeni, R. Hazra, S. Ding, S. Va...
2026 arXiv
-
[25]
V-jepa 2: Self-supervised video models enable understanding, prediction and planning,
M. Assran, A. Bardes, D. Fan, Q. Garrido, R. Howes, Mojtaba, Komeili, M. Muckley, A. Rizvi, C. Roberts, K. Sinha, A. Zholus, S. Arnaud, A. Gejji, A. Martin, F. R. Hogan, D. Dugas, P. Bojanowski, V . Khalidov, P. Labatut, F. Massa, M. Szafraniec, K. Krishnakumar, Y . Li, X. Ma,...
2025 arXiv
-
[26]
Ranking regions of visual saliency in rgb-d content,
D. Seychell and C. J. Debono, “Ranking regions of visual saliency in rgb-d content,” in2018 International Conference on 3D Immersion (IC3D), 2018, pp. 1–8
2018
-
[2026]
Available: https://arxiv.org/abs/2511.09554
[Online]. Available: https://arxiv.org/abs/2511.09554
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.