{"id":"8e6f9690-21ff-495b-8f4a-ae53b20c2931","arxiv_id":"2507.18966","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"On a private NSW Police vehicle-image dataset, YOLO detection models with plate-grouped majority voting reach 93.70% make, 82.86% shape, 85.19% colour, and 94.86% bright/dark colour accuracy.","lead":"This paper tests three YOLO computer-vision models on more than 100,000 real-world vehicle images captured by NSW Police, extracting make, shape, and colour. The authors find that grouping images of the same vehicle and applying majority voting materially improves accuracy, giving a baseline for searching large police image archives.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Detection-vs-classification claim is confounded: the paper never states whether YOLO-Classification saw full frames or vehicle-only crops, so the make/shape advantage may reflect input localisation, not architecture.","rationale":"The paper's central contribution is an applied baseline: MVI consistently improves performance, and detection models are claimed to beat classification-only models on make and shape. The MVI result is well supported across all four tasks. The weaker of the two headline claims is the detection-versus-classification comparison, and it is the one most exposed to a concrete technical confound: the paper never states what input representation YOLO-Classification received. The localisation filter in Section 2.5.2 and the bounding-box annotations described in Section 2.5.3 imply detection models operated on localised vehicle regions; if the classification baseline instead saw entire cluttered frames, the comparison is not apples-to-apples. This is the most load-bearing concern because it targets a headline claim and can be settled experimentally. The reader's label-reliability concern is real but applies uniformly to all models and is less likely to reverse the MVI conclusion. A crop-controlled re-run would settle the preprocessing issue, and a separate annotation-agreement subsample would address label noise; the paper should report both before the comparative claim is taken as established.","tokens_in":16312,"tokens_out":5230,"duration_ms":57496,"concrete_test":"Retrain YOLO-Classification (small, large, x-large) on the make and shape datasets using images cropped to the ground-truth bounding box or the YOLO-selected largest box, keeping all hyperparameters and the MVI protocol identical, and compare SVI and MVI accuracy with Tables 2 and 3. If the detection-vs-classification gap disappears or reverses under identical cropped inputs, the claimed benefit of detection models is an artifact of localisation, not an architectural advantage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.5.2 describes a YOLO-based localisation filter that retains the largest bounding box to select the relevant vehicle, and Section 2.5.3 says each image was stored with ground-truth bounding boxes for detection training. The paper never states whether YOLO-Classification models received vehicle-only crops or the full scene. If YOLO-Classification was trained and evaluated on full frames while YOLO-v11 and YOLO-World were trained and evaluated on localised object regions, the reported make and shape advantages in Tables 2 and 3 are an input-distribution effect, not evidence that localisation-aware detection is superior. The MVI result is less affected, but the abstract's second headline claim rests on this comparison. The performance gaps are also small (roughly 1.3 to 4.1 points) and no error bars or repeated runs are reported, so a preprocessing confound is plausible and would reverse the conclusion.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks three YOLO-based approaches (YOLO-v11, YOLO-World, and YOLO-Classification) for extracting vehicle make, shape, colour, and bright/dark colour from a large real-world MANPR image dataset collected by NSW Police. The authors build task-specific training sets with over 100,000 images each, keep a shared test set of 29,937 images belonging to 1,809 number plates, and evaluate both single-view inference (SVI) and multi-view inference (MVI), where predictions are aggregated by majority voting across all images of a number plate. The main reported findings are that MVI consistently improves accuracy over SVI, that the detection-based models outperform classification-only models on make and shape, and that smaller YOLO variants perform comparably to larger ones. The paper positions this as a baseline study for real-world vehicle metadata extraction and for filtering and searching large image archives.","tokens_in":16500,"tokens_out":4666,"duration_ms":45812,"significance":"If the findings hold, the paper provides a useful and unusually realistic baseline for vehicle attribute extraction in law-enforcement image archives. Its strengths are the scale of the real-world dataset, the consistent test split across tasks, the number-plate-based grouping enabling multi-view inference, and the systematic comparison across model families and sizes. The MVI idea is practical and directly relevant to the MANPR use case, and the with/without fine-tuning comparison for YOLO-World is informative. However, the significance is currently limited by the absence of uncertainty quantification, by an unresolved preprocessing confound in the detection-versus-classification comparison, and by unmeasured ground-truth label noise.","major_comments":[{"comment":"The central comparison between detection and classification is confounded because the paper never states whether YOLO-Classification models were trained and evaluated on full frames or on vehicle-only crops. Section 2.5.2 describes a YOLO-based localisation filter that retains the largest bounding box, and Section 2.5.3 states that each image is stored with ground-truth bounding boxes for detection training, but the corresponding input for YOLO-Classification is not specified. If YOLO-Classification saw full scenes while the detection models operated on localised vehicle regions, the reported make and shape advantages in Tables 2 and 3 would reflect an input-distribution difference rather than a benefit of the detection architecture. Please state the exact preprocessing for each model family and, if the current protocol is as described, add a classification-on-crops baseline to make the comparison fair.","section":"Sections 2.5.2, 2.5.3 and Tables 2, 3"},{"comment":"No confidence intervals, repeated runs, or significance tests are reported, yet several headline differences are small. For example, the best make MVI accuracy is 93.70% for YOLO-World small (Table 2) versus 92.43% for YOLO-Classification x-large, and the best shape MVI accuracy is 82.86% for YOLO-v11 x-large (Table 3) versus 82.81% for YOLO-World small. Claims such as \"detection models consistently outperformed classification models\" and \"smaller YOLO variants perform comparably\" are not statistically supported without uncertainty estimates. Please provide confidence intervals, repeated-seed results, or significance tests, and account for the fact that images from the same number plate are not independent.","section":"Tables 2 to 5 and Section 3.3"},{"comment":"Ground-truth quality is not established. Labels were produced by recruited annotators and reviewed once by a different annotator, but no inter-annotator agreement, adjudication protocol, or cross-check against vehicle registration data is reported. Because every accuracy figure in Tables 2 to 5 inherits the noise in these labels, the lack of label-quality measures weakens the baseline and the comparative conclusions. Please report inter-annotator agreement or a validated subset, and discuss how label noise might affect the make, shape, and colour comparisons.","section":"Section 2.5.1 (Labelling Process)"},{"comment":"The conclusion misstates the results. It says detection-based models outperformed classification models on make and shape, \"achieving accuracies of up to 94.86% and 82.86%, respectively,\" but 94.86% is the colour-binary accuracy of YOLO-Classification (Table 5), not a make-detection result, and the best make accuracy is 93.70% (Table 2). This misattribution should be corrected in the conclusion and, if the abstract relies on the same phrasing, in the abstract as well.","section":"Section 4 (Conclusion)"},{"comment":"The treatment of no-detection outputs is not quantified. Section 3.3.5 argues that a no-detection output is often preferable in policing contexts, but the frequency of no-detection cases is never reported, nor is it stated how such cases are counted in the accuracy denominators for detection models. This matters for the fairness of the detection-versus-classification comparison, since classification models always produce a label while detection models may produce none. Please report detection rates and the exact handling of missing detections in the SVI and MVI accuracy calculations.","section":"Section 3.3.5 and Table 6"}],"minor_comments":[{"comment":"The spelling of YOLO-World is inconsistent (\"YOLO World\" appears in several places); please standardise it.","section":"Throughout"},{"comment":"The text says fine-tuning improved accuracy \"by up to 90 percentage points,\" but the largest improvement in Table 6 is from 1.80% to 93.70% for make small, which is 91.9 percentage points; please correct the figure.","section":"Section 3.3.4, Table 6"},{"comment":"The phrase \"the top YOLO-Classification model achieving 85.19% accuracy using MVI\" is correct, but the surrounding ranking sentence is ambiguous about whether colour or binary-colour is being discussed; please clarify.","section":"Section 3.3.2"},{"comment":"The MVI description in Figure 3's caption says the AI predicts \"Mercedes\" three times and \"No Detection\" one time, yielding a final classification of \"Mercedes\"; it would be helpful to state explicitly how ties and no-detection votes are handled in the majority-voting procedure.","section":"Section 2.4 and Figure 3"},{"comment":"The reference format with superscript numbers is nonstandard and hard to read; please switch to a numbered citation style consistent with the journal's guidelines.","section":"References and formatting"},{"comment":"Please add a short statement about whether the task-specific test sets are identical across all four tasks or only share the same image set; Table 1 suggests the image counts are identical, but the wording is not fully explicit.","section":"Section 2.5.3 and Table 1"}],"recommendation":"major_revision","confidential_remarks":"This is a useful applied baseline, but the two headline claims (detection beats classification on make/shape, and MVI is necessary) rest on comparisons that currently lack uncertainty quantification and on an unspecified preprocessing protocol for the classification models. The conclusion also contains a clear misattribution of the 94.86% result. I think the paper is potentially salvageable with added clarity and statistical support, but the authors need to either confirm that YOLO-Classification received the same localised input as the detection models or re-run the comparison with matched inputs, and they should add confidence intervals or significance tests. I would also encourage the editor to ask for inter-annotator agreement or a label-validation subset, since the entire accuracy scale depends on the reliability of the manual annotations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honest applied baseline with a real police dataset; the multi-view inference result is solid, but the detection-versus-classification claim is confounded by an unstated preprocessing choice.\n\nThe paper does two useful things. It documents a large in-house dataset (MANPR) from NSW Police Highway Patrol vehicles, with multiple images per plate, and it shows that grouping predictions by number plate and taking a majority vote (MVI) improves accuracy by four to six points across all four attribute tasks and all YOLO variants. That is a practical, deployable finding. The observation that small YOLO models match large ones is also useful for real-time use, and the zero-shot comparison (fine-tuning lifts accuracy by tens of points) is a fair caution for practitioners.\n\nThe soft spots are real but manageable. The headline claim that detection models beat classification models on make and shape rests on a comparison the paper never fully specifies. Section 2.5.2 describes a YOLO-based localisation filter that keeps the largest bounding box, but it is never stated whether the YOLO-Classification models saw full frames or vehicle-only crops. If the classifiers saw full frames and the detectors saw localised regions, the make/shape gap is an input-distribution effect, not an architecture effect. This needs to be clarified before the abstract's second sentence can be taken at face value. The MVI result does not depend on this comparison, so the main operational finding stands.\n\nThe accuracy figures also rest on labels from a single annotation pass with no inter-annotator agreement, and the gaps between models are small with no confidence intervals or repeated runs. In particular, the conclusion misattributes 94.86% to make/shape, when that figure is the colour-binary result; the make best is 93.70% and shape is 82.86%. That is an editorial error, but it should be fixed.\n\nWho benefits: practitioners building search and filtering tools over police or traffic image archives, and anyone looking for baseline numbers on unconstrained vehicle imagery. The dataset is private, so replication is impossible, but the MVI idea transfers. I would send this to a serious referee rather than desk-reject, asking specifically for the classification input pipeline, label reliability analysis, and error bars.","headline":"Honest applied baseline with a real police dataset; the multi-view inference result is solid, but the detection-versus-classification claim is confounded by an unstated preprocessing choice.","tokens_in":16992,"tokens_out":5337,"would_cite":false,"duration_ms":45854,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Object detection outperforms image classification for vehicle make and shape","keywords":["vehicle metadata extraction","YOLO","object detection","multi-view inference","majority voting","automatic number plate recognition","vehicle make recognition","real-world imagery"],"falsifier":"Re-annotate a random sample of the 1,809 test number plates with two independent annotators who are blind to each other and to the model outputs, then recompute all single-view and multi-view accuracies against the re-annotated majority labels; if the original and re-annotated labels disagree on more than a small fraction of plates, or if the multi-view gain over single-view inference shrinks or disappears under the new labels, the reported comparison is not measuring model skill alone.","tokens_in":16134,"feed_emoji":"🚗","tokens_out":9272,"duration_ms":87109,"temperature":0.7,"pith_summary":"This paper argues that vehicle metadata from messy, real-world patrol-camera images should be extracted by object detectors that first locate the car, rather than by classifiers that label the whole frame. It tests YOLO-v11, YOLO-World, and YOLO-Classification on police plate-reader imagery, with separate datasets of more than 100,000 images for make, shape, and colour, and a shared test set of 29,937 images from 1,809 number plates. The paper reports that detectors beat classification-only models on make and shape, and that grouping images by number plate and applying majority voting consistently improves accuracy by several points over single-image predictions. If this is right, investigators could search plate-reader archives by make, colour, or shape without querying registration databases.","feed_headline":"Detection beats classification for car make and shape","feed_subtitle":"Majority voting over number-plate image groups raises usable accuracy to 94.86 percent on real patrol-camera data.","key_machinery":"The mechanism is the reformulation of attribute recognition as object detection: the model predicts a bounding box around the vehicle and the class label in one forward pass, so localisation forces the network to attend to the car rather than background clutter. On top of this, multi-view inference (MVI) groups predictions by number plate, reliable because the plate reader links images of the same vehicle, and majority voting produces one label per plate. Task-specific training sets (make, shape, colour, colour-binary) give each model the visual cues most relevant to its task, while a shared test set keeps comparisons fair.","core_discovery":"The paper's central result is that localisation-aware YOLO detectors (YOLO-v11 and YOLO-World) consistently outperform YOLO-Classification on fine-grained attributes, reaching 93.70% top-1 accuracy for make and 82.86% for shape under multi-view inference, while classification-only models do slightly better on colour (85.19%) and on the simplified bright/dark colour task (94.86%). Multi-view inference, meaning majority voting over the images belonging to each number plate, improves accuracy in every experiment compared with single-image inference, with gains of roughly three to six percentage points. The authors conclude that multi-view inference is necessary for usable performance on this kind of data, that detection models are better suited to cluttered scenes where localisation helps, and that small YOLO variants perform about as well as x-large ones, making real-time deployment feasible. Fine-tuning is essential: without it, YOLO-World scored at most 2.3% on make, 38.4% on shape, and 15.4% on colour.","pith_inferences":["Going beyond the paper, weighting per-image votes by detection confidence or image quality would likely add a point or two, since not all views of a plate are equally informative.","Because ground truth came from a single annotator pass, some of the multi-view gain may come from voting away annotator mistakes rather than from true visual signal; re-annotating the test set with independent labelers would separate those effects.","The same detection-plus-voting recipe could transfer to untested attributes the paper lists, such as vehicle damage, accessories, or text, and to other camera networks with a reliable frame-grouping signal.","Whether these accuracies hold on other police camera deployments is open; running the same protocol on a public multi-camera vehicle benchmark would show how much of the difficulty is specific to this archive."],"forward_implications":["Patrol-camera archives can be searched by make, colour, or shape without bulk queries to registration databases, narrowing the images an investigator must review.","Detection-based pipelines should be the default for fine-grained attributes in cluttered, unconstrained imagery.","Small YOLO variants are accurate enough for real-time or in-vehicle deployment, since they match larger variants on these tasks.","Expected accuracy tracks how visually separable the classes are: bright/dark colour is easiest, shape is hardest.","Zero-shot open-vocabulary detection is not usable on this data without fine-tuning, so deployment should plan for supervised training."],"supporting_citations":[{"why":"Introduces YOLO as a single-stage detector predicting boxes and classes in one pass, the architectural basis for the detection models.","marker":"[38]"},{"why":"Supplies the YOLO11 closed-set detector used as the main detection baseline.","marker":"[44]"},{"why":"Presents YOLO-World, the open-vocabulary detector tested zero-shot and after fine-tuning.","marker":"[45]"},{"why":"Provides the YOLOv8 vehicle localisation used to clean the dataset by keeping the largest bounding box.","marker":"[59]"},{"why":"A part-based vehicle-recognition framework that combines predictions by majority voting, the precedent for multi-view inference.","marker":"[8]"},{"why":"A real-time vehicle make and model recognition system on uncontrolled road images, used to frame the operational task.","marker":"[1]"}],"fun_headline_variants":["Detection models edge out classifiers for car make and shape","Majority voting lifts vehicle attribute accuracy to 94.86%","Small YOLO variants rival large ones for real-time car metadata","Localization-aware detection wins for vehicle make and shape","Multi-view inference ups car recognition accuracy on patrol data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the human-assigned labels used as ground truth are correct, since the labelling process used a single reviewing pass with no reported inter-annotator agreement and no cross-check against registration records, so any systematic label error is inherited by every accuracy figure.","fun_headline_variants_meta":{"raw":{"variants":["Detection models edge out classifiers for car make and shape","Majority voting lifts vehicle attribute accuracy to 94.86%","Small YOLO variants rival large ones for real-time car metadata","Localization-aware detection wins for vehicle make and shape","Multi-view inference ups car recognition accuracy on patrol data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000565,"raw_usage":{"total_tokens":2733,"prompt_tokens":1057,"completion_tokens":1676,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":673,"completion_tokens_details":{"reasoning_tokens":1594}},"tokens_in":673,"tokens_out":1676,"duration_ms":12953,"temperature":1.0,"reasoning_tokens":1594,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:03:20.271389+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of the 1,809 test number plates with two independent annotators who are blind to each other and to the model outputs, then recompute all single-view and multi-view accuracies against the re-annotated majority labels; if the original and re-annotated labels disagree on more than a small fraction of plates, or if the multi-view gain over single-view inference shrinks or disappears under the new labels, the reported comparison is not measuring model skill alone.","supporting_citations":[{"cited_title":"& Farhadi, A","cited_arxiv_id":null,"evidence_quote":"Introduces YOLO as a single-stage detector predicting boxes and classes in one pass, the architectural basis for the detection models."},{"cited_title":"& Qiu, J","cited_arxiv_id":null,"evidence_quote":"Supplies the YOLO11 closed-set detector used as the main detection baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Presents YOLO-World, the open-vocabulary detector tested zero-shot and after fine-tuning."},{"cited_title":"Yolov8: Ultralytics official implementation","cited_arxiv_id":null,"evidence_quote":"Provides the YOLOv8 vehicle localisation used to clean the dataset by keeping the largest bounding box."},{"cited_title":"& Barshooi, A","cited_arxiv_id":null,"evidence_quote":"A part-based vehicle-recognition framework that combines predictions by majority voting, the precedent for multi-view inference."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"A real-time vehicle make and model recognition system on uncontrolled road images, used to frame the operational task."}],"review_version":2}