{"id":"af11301f-6b87-42e4-a9f0-2979e9e60c36","arxiv_id":"2411.19143","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Co-Learning reports 36.5 mAP on AI City Challenge Track 2 using 10% labeled data and a label-alignment step, yet lacks the fully-supervised baseline needed to justify its parity claim.","lead":"The paper introduces Co-Learning, a semi-supervised teacher-student framework for object detection from roadside cameras using only 10% labeled data. Its key experiment reports that a text-based annotation-alignment step raises mean average precision from 23.0 to 36.5 on AI City Challenge Track 2, but no fully-supervised baseline is provided to support the headline claim.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1's reported mAP values do not match the per-class APs shown, and the 'Oracle' is not a fully-supervised baseline; the headline parity claim rests on internally inconsistent numbers.","rationale":"The reader's rejection is well-founded, but the most load-bearing concern is more direct than the stated weakest assumption about label derivation. The headline claim is an empirical parity claim, and its only quantitative support is Table 1; that table cannot be reproduced from its own per-class AP values. This is an internal inconsistency, not a disagreement with community consensus, and it is checkable from the paper alone. The label-space circularity raised by the reader is a genuine risk, but the arithmetic failure is prior: even a clean experimental design would still require the reported mAPs to be correct. The missing fully-supervised baseline is also critical: the 'Oracle' is trained on pseudo-labels, not human boxes, so there is no reference point for 'comparable performance as fully-supervised solutions.' The contradiction between 'fully annotated dataset' in Section 3.3 and '10% of annotated data' in Section 3.4 further undermines the 10%-label claim. No code, no released derived labels, and no detailed per-class evaluation protocol are provided, so the setup cannot be independently reconstructed. These issues together leave the central claim without credible support; the reader's REJECT verdict stands, and the proposed recomputation would either confirm the arithmetic failure or force disclosure of the actual mAP definition and baseline.","tokens_in":6529,"tokens_out":8384,"duration_ms":69323,"concrete_test":"Recompute Table 1 by summing the seven per-class AP values in each row and dividing by 7. If the results are 26.1, 16.7, and 27.2, the printed mAPs are wrong; if the authors used a different class set or weighting, they must release the complete per-class AP list and the exact mAP formula. As a further check on the parity claim, run a standard Faster R-CNN with the same ResNet-50 backbone on 100% of the actual human box annotations and compare its mAP to 36.5.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative evidence for the abstract's parity claim is Table 1, but the table is internally inconsistent. Averaging the seven per-class AP values listed for each row gives Oracle 26.1, w/o Annotation-Alignment 16.7, and w/ Annotation-Alignment 27.2, not the printed 36.1, 23.0, and 36.5. The caption defines the column as mAP (mean average precision), so the printed numbers are either computational errors or the result of an unstated class set or weighting. In either case, the 36.5 mAP that anchors the comparison cannot be taken at face value. Separately, Section 3.3 says the aligned model is 'further trained on the fully annotated dataset' while Section 3.4 says only 10% of annotations are used; if the first statement is literal, the 'using 10% labeled data' claim is false. Finally, the 'Oracle' is defined as a model trained only on pseudo-labels, not a fully-supervised detector trained on human box annotations, so even a correct 36.5 would not by itself establish comparability with fully-supervised solutions.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Co-Learning, a teacher-student semi-supervised object detection framework for roadside camera data, evaluated on the 7th AI City Challenge Track 2 dataset. The method combines three modules—dynamic pseudo-label assignment, pseudo-feature alignment, and pseudo-label refinement—with an annotation-alignment strategy that maps natural-language vehicle descriptions to a smaller set of box-level class labels via stemming, lemmatization, and synonym clustering. Using 10% labeled data, the authors report a mean average precision (mAP) of 36.5 with annotation alignment versus 23.0 without it, and claim that this is comparable to fully-supervised solutions. The central evidence for this claim is Table 1, which compares an 'Oracle' model trained on pseudo-labels with variants with and without annotation alignment.","tokens_in":6779,"tokens_out":4258,"duration_ms":33890,"significance":"If the reported results were reliable, the paper would offer a useful contribution to semi-supervised object detection on edge-roadside cameras, particularly the idea of aligning noisy text-derived annotations into a consistent label taxonomy. However, the evaluation is not currently trustworthy: the headline mAP values in Table 1 are internally inconsistent with the per-class APs, the 'Oracle' baseline is not a fully-supervised detector, the annotation-alignment comparison appears circular with respect to the label space, and no ablations are provided for the three proposed modules. The paper's central claim of parity with fully-supervised solutions is therefore unsupported. The paper does address a relevant practical problem and proposes a coherent pipeline, but the experimental validation as presented is insufficient to establish the claimed contribution.","major_comments":[{"comment":"The printed mAP values in Table 1 are inconsistent with the per-class AP values in the same table. Averaging the seven per-class APs gives Oracle 26.1, w/o Annotation-Alignment 16.7, and w/ Annotation-Alignment 27.2, whereas the table reports 36.1, 23.0, and 36.5. Because the caption defines mAP as mean average precision, the headline numbers appear to be computational errors or to rely on an unstated class set or weighting. The 36.5 mAP that anchors the abstract's parity claim therefore cannot be taken at face value.","section":"Table 1, Section 3.3"},{"comment":"The abstract claims Co-Learning 'achieves comparable performance as fully-supervised solutions using 10% labeled data,' but no fully-supervised baseline is reported. The 'Oracle' row in Table 1 is defined as a model trained only on pseudo-labels, not as a detector trained on human box annotations. Moreover, Section 3.3 states that the aligned model is 'further trained on the fully annotated dataset,' while Section 3.4 states that only 10% of annotations are used; the two statements are contradictory. The parity claim is therefore unsupported and needs either a proper fully-supervised comparison or a revision of the claim.","section":"Abstract, Section 3.3, Section 3.4"},{"comment":"The comparison 'w/ Annotation-Alignment' versus 'w/o Annotation-Alignment' is confounded by the label space used for evaluation. The paper applies stemming, lemmatization, and synonym clustering to define a new label taxonomy (e.g., 'red van', 'van', and 'blue van' are all mapped to 'van'), but it does not specify whether both rows are evaluated on the original label space, the aligned label space, or each on its own training label space. If the w/ model is evaluated on the aligned taxonomy while the w/o model is evaluated on the original taxonomy, the reported improvement is an artifact of label remapping. A fixed held-out label space for evaluation must be used.","section":"Section 3.1, Table 1"},{"comment":"The paper attributes the result to three modules (dynamic pseudo-label assignment, pseudo-feature alignment, and pseudo-label refinement) but provides no ablation or individual evaluation of these modules. Without ablations, the contribution of each module to the reported mAP cannot be assessed, and the central method claim is not substantiated.","section":"Section 3.3"}],"minor_comments":[{"comment":"The caption states 'We reported AP0.5 as the mean averaged precision mAP,' but Section 3.4 refers simply to mAP; please define the evaluation protocol precisely, including the IoU threshold, the class set, and the averaging method.","section":"Table 1 caption, Section 3.4"},{"comment":"The subscript formatting in the standardized text format is corrupted in the manuscript; the definition of the three attribute parts should be presented readably.","section":"Section 3.1"},{"comment":"The heading 'Semi-supverised Object Detection' contains a typo.","section":"Section 2.2"},{"comment":"No error bars, confidence intervals, or multiple-seed results are reported; the 0.4% improvement over the Oracle is small enough that it could arise from randomness, so an uncertainty estimate is needed.","section":"Table 1, Section 3.3"},{"comment":"The dataset split is underspecified: it is not stated how the 10% labeled subset was selected, how many images or frames are used, or how the text descriptions are paired with video frames.","section":"Section 3.4"}],"recommendation":"reject","confidential_remarks":"The paper appears to originate from a workshop venue (EAmSI24), and the arXiv version is not at the standard of a full journal submission. The central numerical evidence in Table 1 is internally inconsistent, which calls into question all subsequent conclusions. Before any resubmission, the authors should recompute and verify their mAP values, add a genuine fully-supervised baseline, define a single fixed evaluation label space, and provide ablations. The current manuscript cannot support the abstract's headline claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: the abstract's claim that Co-Learning matches fully-supervised detection with 10% labels is not supported by the paper's own experiments. Table 1's mAP column doesn't match the per-class APs — averaging the listed values gives roughly 27.2 for the aligned model, not 36.5. The 'Oracle' is trained only on pseudo-labels, not on human box annotations, so even a correct 36.5 wouldn't establish parity with fully-supervised training. And Section 3.3 says the model was 'further trained on the fully annotated dataset' while Section 3.4 says only 10% of annotations were used. That's a direct contradiction.\n\nWhat's worth credit: the pipeline targets a real use case — roadside cameras with scarce box annotations — and the idea of deriving box-level labels from natural-language descriptions via stemming and synonym clustering is a sensible data-curation direction. The authors cite the relevant SSOD literature and use a standard Faster R-CNN base. The annotation-alignment concept is legitimate, even if the execution is flawed.\n\nThe soft spots are major, not minor. The Table 1 inconsistency alone makes the headline quantitative claim unreliable. There is no fully-supervised baseline, no error bars, and no ablation isolating the three proposed modules. The 0.4 mAP improvement over the pseudo-label oracle is within noise. The annotation alignment is evaluated on the same label space it defines, so part of the gain may just be label remapping. The method details are too vague to reproduce — the dynamic assignment, feature alignment, and refinement modules are named but not specified. The circularity concern is real: self-training with teacher-generated pseudo-labels is inherently self-referential, and the paper adds a second alignment loop on top.\n\nI don't think this paper is coherent on its own terms. The central evidence is internally inconsistent, and the contradictions suggest the experiments were not carefully checked. It could be salvaged if the authors fix the table, add a true fully-supervised baseline, clarify what data the final model sees, and release code. As it stands, I would not send it to peer review.\n\nRecommendation: desk reject, with an invitation to revise and resubmit after fixing the numbers and the experimental setup.","headline":"The paper's central parity claim is undercut by its own inconsistent numbers and a mislabeled oracle baseline.","tokens_in":7309,"tokens_out":2446,"would_cite":false,"duration_ms":37650,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"With 10% labeled frames, a teacher–student detector matches fully supervised roadside detection.","keywords":["semi-supervised learning","object detection","teacher-student network","pseudo-label refinement","annotation alignment","roadside cameras","text-derived annotations"],"falsifier":"Run the with-alignment and without-alignment models against a single human-verified label set and score both on the same label vocabulary; if the aligned model's advantage over the oracle disappears or shrinks to noise, the reported 36.5 mAP result is an artifact of annotation remapping rather than evidence for the semi-supervised learning method.","tokens_in":6352,"feed_emoji":"🚗","tokens_out":9802,"duration_ms":86727,"temperature":0.7,"pith_summary":"The paper is trying to establish that semi-supervised learning can make roadside object detection practical when labels are scarce: a teacher–student framework called Co-Learning trains on 10% labeled frames while treating the other 90% as unlabeled, and the authors report that it reaches performance comparable to a fully supervised solution. On the AI City Challenge Track 2 dataset, with Faster R-CNN and a ResNet-50 backbone, the aligned Co-Learning model scores 36.5 mean average precision (mAP), versus 36.1 for an oracle model trained only on pseudo-labels and 23.0 for the same framework without annotation alignment. The paper attributes the gain to attacking label inconsistency: pseudo-label assignment, feature alignment, and threshold refinement stabilize what the student learns, while stemming, lemmatization, and synonym clustering standardize the text-derived box labels. A reader should care because labeling every frame from dense roadside camera networks is expensive, and a method that works with 10% of the labels lowers the barrier to deploying detectors at the edge.","feed_headline":"One detector matches full-data mAP with only 10% labels","feed_subtitle":"Co-Learning standardizes noisy text-derived labels so roadside cameras can train effectively with a tenth of the annotation effort.","key_machinery":"The load-bearing object is a teacher-student network with mutual learning, in which a teacher generates pseudo-labels from unlabeled frames and a multi-head student trains on them under exponential-moving-average coupling. Three sub-modules carry the method: dynamic pseudo-label assignment, which replaces static IoU matching so noisy box boundaries do not misassign anchors; pseudo-feature alignment, which lets each classification feature pick a suitable regression feature vector; and pseudo-label refinement, which adaptively adjusts the score threshold to keep the pseudo-ground-truth count stable early in training. The supporting machinery is annotation alignment, built from stemming, lemmatization, and semantic-role labeling: free-text vehicle descriptions are reduced to color, type, and motion attributes and then collapsed into a representative class (for example, 'red van', 'van', and 'blue van' all become 'van'). Together these pieces are meant to make the student learn from a consistent, less noisy label set rather than from contradictory pseudo-boxes.","core_discovery":"Co-Learning's central claim is that the bottleneck in semi-supervised roadside detection is label inconsistency rather than labeled-data scarcity. Using 10% of the AI City Challenge Track 2 frames as labeled data and 90% as unlabeled data, a Faster R-CNN teacher-student detector with ResNet-50 achieves 36.5 mAP (AP@0.5) when annotation alignment is applied, exceeding the 36.1 mAP of the oracle model that can access all pseudo-labels; without alignment the same pipeline drops to 23.0 mAP. The authors conclude that consistent pseudo-labels—maintained by dynamic thresholds, cross-teacher/student agreement, and a standardized label vocabulary derived from natural-language descriptions—are what allow a small labeled set to approach supervised-level detection.","pith_inferences":["The comparison in the paper is against an oracle trained only on pseudo-labels, so the abstract's 'fully supervised' phrasing is stronger than the experiment: a human-verified label ceiling would let the 36.5 mAP result be measured against true full supervision.","Part of the 0.4 mAP gain over the oracle may come from the evaluation label vocabulary changing between the aligned and unaligned conditions; scoring both variants on one fixed label set would isolate the learning contribution from the relabeling.","A natural extension is to sweep the labeled fraction (for example, 1%, 5%, 20%) to see whether annotation alignment matters more or less as supervision becomes scarcer.","The text-standardization recipe is not tied to road scenes and could be probed on other weakly labeled visual domains, such as surveillance or aerial imagery, where natural-language descriptions accompany the frames."],"forward_implications":["If Co-Learning works as reported, roadside deployments could annotate only 10% of frames and still obtain detectors whose mAP is on par with models trained on the complete pseudo-label set.","The framework implies that label consistency, not the sheer volume of unlabeled data, is the main lever in semi-supervised roadside object detection.","The dynamic assignment and feature-alignment modules should reduce sensitivity to pseudo-box boundary noise, making the method applicable to dense traffic scenes where static IoU assignment is fragile.","The authors position the pipeline as ready to transfer to edge devices such as Jetson Orin and Xavier, where the 10%-label regime is practical under limited compute."],"supporting_citations":[{"why":"Supplies the 7th AI City Challenge Track 2 dataset, whose natural-language descriptions the pipeline converts into box-level annotations.","marker":"[4]"},{"why":"Provides Faster R-CNN, the base two-stage detector used for both teacher and student networks.","marker":"[5]"},{"why":"Establishes the consistency-based semi-supervised object detection approach that teacher-student pseudo-label training builds on.","marker":"[12]"},{"why":"Represents the two-stage unbiased teacher SSOD line that the pseudo-label modules extend.","marker":"[14]"},{"why":"Inspires the stemming and lemmatization technique used to derive consistent box-level labels from natural-language vehicle descriptions.","marker":"[20]"},{"why":"Semantic Role Labeling is used to extract color, type, and motion attributes that feed the standardized annotation format.","marker":"[23]"},{"why":"ResNet-50 is the fixed feature-extraction backbone for all experiments, setting the common base for the reported comparisons.","marker":"[26]"}],"fun_headline_variants":["10% labels match full-data mAP in roadside detection","Roadside AI: label consistency, not volume, unlocks detection","Co-Learning: 10% labeled frames achieve full-data detection","10% labels, full mAP: Co-Learning for road cameras"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that box labels created from natural-language text—via stemming, lemmatization, and synonym clustering—are reliable enough to serve as ground truth for both training and evaluation; if those labels are systematically wrong, or if the two variants are scored on different label vocabularies, the reported improvement is an artifact of relabeling rather than evidence for the semi-supervised method.","fun_headline_variants_meta":{"raw":{"variants":["10% labels match full-data mAP in roadside detection","Roadside AI: label consistency, not volume, unlocks detection","Co-Learning: 10% labeled frames achieve full-data detection","10% labels, full mAP: Co-Learning for road cameras"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000862,"raw_usage":{"total_tokens":3700,"prompt_tokens":868,"completion_tokens":2832,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":484,"completion_tokens_details":{"reasoning_tokens":2759}},"tokens_in":484,"tokens_out":2832,"duration_ms":21250,"temperature":1.0,"reasoning_tokens":2759,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:28:58.725522+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the with-alignment and without-alignment models against a single human-verified label set and score both on the same label vocabulary; if the aligned model's advantage over the oracle disappears or shrinks to noise, the reported 36.5 mAP result is an artifact of annotation remapping rather than evidence for the semi-supervised learning method.","supporting_citations":[{"cited_title":"Naphade, S","cited_arxiv_id":null,"evidence_quote":"Supplies the 7th AI City Challenge Track 2 dataset, whose natural-language descriptions the pipeline converts into box-level annotations."},{"cited_title":"Jeong, S","cited_arxiv_id":null,"evidence_quote":"Establishes the consistency-based semi-supervised object detection approach that teacher-student pseudo-label training builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Inspires the stemming and lemmatization technique used to derive consistent box-level labels from natural-language vehicle descriptions."},{"cited_title":"Palmer, D","cited_arxiv_id":null,"evidence_quote":"Semantic Role Labeling is used to extract color, type, and motion attributes that feed the standardized annotation format."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ResNet-50 is the fixed feature-extraction backbone for all experiments, setting the common base for the reported comparisons."}],"review_version":1}