{"id":"1ceb8ad3-9485-46cf-a49b-7db0030fe148","arxiv_id":"1908.02454","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Adaptive supervision, switching from cheap center-click labels to strong bounding-box labels based on confidence or accuracy gains, reduces annotation time for active object detection.","lead":"The paper proposes an active learning framework for object detection that starts by asking annotators for cheap center clicks instead of full bounding boxes, and only requests expensive box annotations for images where the model is uncertain. In experiments on PASCAL VOC and a wheat-head detection dataset, the method reaches the same accuracy in about 15-30% less annotation time than standard active learning.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ImageNet-derived annotation timing model is unvalidated on VOC/Wheat; cost ratio changes could erase the reported savings","rationale":"The reader identified the ImageNet timing model as the weakest assumption. I agree because the paper's primary contribution is measured in annotation hours, not mAP alone. The conversion from label counts to hours is entirely based on external ImageNet statistics, and the paper explicitly acknowledges it had no VOC/Wheat timing data. The single-run experiments, simulated weak labels, and arbitrary thresholds are also concerns, but they weaken the evidence without necessarily changing the ranking; the timing model directly determines the quantitative magnitude of the claimed savings. If the cost ratio of weak-to-strong annotation is different on these datasets, the reported 15–30% savings could disappear or invert. This is a load-bearing, testable assumption. Since the reader already marked the paper CONDITIONAL, and my concern is the same, no verdict change is needed.","tokens_in":10135,"tokens_out":6904,"duration_ms":67186,"concrete_test":"Run a sensitivity analysis on Eq. (3) for all three datasets: recompute annotation time with weak per-object cost c in {1, 2, 5, 10, 15}s and per-image overhead o in {0, 4, 7.8, 12}s, and determine whether soft switch still reaches the target mAP sooner than standard PBAL for every combination. Additionally, collect real annotation times on a random 100-image subset of VOC 2007 and the Wheat dataset using a center-click tool and a bounding-box tool (e.g., via Amazon Mechanical Turk) and recompute Figures 4–6 with the measured medians.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 (Eq. 3) converts every label into annotation time using ImageNet median costs from Su et al. and Papadopoulos et al.: 34.5s per box, 7.8s per-image overhead, and 3.0s per center-click. The paper's quantitative claim—e.g., '30% savings' on VOC 2007 (Figure 4a)—is computed from the ratio of these costs. The text explicitly states the VOC/Wheat annotation times were unavailable and assumes similarity to ImageNet, but provides no sensitivity analysis or validation. If real center-click costs on VOC/Wheat are higher (e.g., 10s rather than 3s) or if the per-image overhead differs, the savings shrink or reverse. Since the abstract's headline is a reduction in annotation cost, not just mAP, this unvalidated conversion is the most load-bearing assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an adaptive supervision framework for active learning in object detection. Instead of always querying bounding-box annotations, the framework first queries weak labels (center clicks) for actively selected images, generates pseudo-labels by matching predicted boxes to click locations, and uses a switching rule to request strong labels when the model is uncertain (soft switch) or when the mAP improvement has saturated (hard switch). The authors evaluate two switching variants on PASCAL VOC 2007, VOC 2012, and a wheat head counting dataset, using three query functions, and report 15-30% savings in annotation time to reach a given mAP compared with a standard pool-based active learning baseline. The main conclusion is that adaptive supervision reduces annotation cost.","tokens_in":10328,"tokens_out":6172,"duration_ms":60504,"significance":"If the reported savings are reliable, the paper introduces a simple and model-agnostic way to reduce annotation cost in detection, and it provides a useful comparison of weak-versus-strong supervision in an active learning loop. The soft-switch mechanism is particularly appealing because it requires no extra architecture and no access to a validation set. The authors also include a passive-learning ablation that isolates the effect of adaptive supervision. However, the quantitative claims rest on unvalidated annotation-time constants and single-run experiments, and the hard-switch variant appears to use test-set information; these issues must be addressed before the savings can be trusted. The comparison to 'state of the art' active learning methods is not supported by the experiments, which only compare to generic query baselines.","major_comments":[{"comment":"The hard-switch condition is defined on validation mAP in Section 3.4, but Section 4.2 states 'we switch to strong supervision when the test mAP increase in the last episode is less than 30%...'. If test mAP is used, the annotation budget is being allocated using test-set information, which is unavailable in real applications and inflates the reported hard-switch savings; if a validation set is used, it is not described in the setup. Please clarify and, if the test set was used, re-run the hard-switch experiments with a held-out validation split and report only test mAP at the end.","section":"Section 4.2 / Algorithm 1"},{"comment":"The annotation-time conversion is the basis for the headline savings, but the constants (34.5 s/box, 3.0 s/click, 7.8 s/image overhead) are taken from ImageNet and applied without change to PASCAL VOC and Wheat, with no sensitivity analysis. The text acknowledges that VOC/Wheat annotation times were unavailable, yet the reported percentages (e.g., 30% savings in Figure 4a) are computed directly from the ratio of these constants. Please provide a sensitivity analysis over the click-to-box cost ratio (e.g., 3.0s to 10s, per-image overhead 0-15s) and, ideally, a small-scale annotation timing study on VOC/Wheat to confirm the ratio.","section":"Section 3.3, Eq. (3)"},{"comment":"All results appear to be from a single run per condition. Given the stochastic nature of active learning (initial pool, sampling, training), differences of a few hours of annotation time (e.g., Figure 5a showing a 2.5-hour gap between hard and soft switch) may be within run-to-run variance. Please report mean and standard deviation over at least 3-5 independent runs and state whether the observed savings are statistically significant.","section":"Sections 4.2-4.4"},{"comment":"Weak labels are simulated by adding 'small zero mean Gaussian random noise' to ground-truth center positions, but the noise scale is not reported, making the experiments irreproducible. More importantly, simulated clicks do not model real annotator behavior such as systematic bias or occasional errors. Please report the noise standard deviation and include either a sensitivity analysis or a small real-annotator validation of center clicks on one of the datasets.","section":"Section 4.1"},{"comment":"The abstract claims the framework trains detectors with 'much lesser annotation costs than the state of the art active learning approaches for object detection', but the experiments compare only against a standard PBAL baseline with three generic query functions (max-margin, avg-entropy, least-confident) implemented by the authors. No comparison to previously published active learning methods for object detection (e.g., localization-aware active learning, Kao et al. 2018) is provided. Please either include such comparison or temper the 'state of the art' claim to 'standard pool-based active learning'.","section":"Abstract / Section 4"}],"minor_comments":[{"comment":"The per-image overhead of 7.8 s is added for both strong and weak annotation; please clarify whether this overhead applies to click annotations or adjust the formula.","section":"Eq. (3)"},{"comment":"The hard-switch ratio dn/dmax requires division by dmax; please specify the behavior when dmax=0, for example in the first episode or if mAP does not improve between consecutive episodes.","section":"Section 3.4"},{"comment":"The threshold values γ=0.3 and δ=0.75/0.85 are reported without a sensitivity analysis; a short study on how the savings vary with these thresholds would increase confidence in their generality.","section":"Section 4.2"},{"comment":"References [4] and [5] are duplicate citations (Brust et al., Active learning for deep object detection); please deduplicate.","section":"References"},{"comment":"The text contains a typo: 'highnumber' should be 'high number'.","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"The central idea is attractive and the soft-switch mechanism is simple and plausibly effective. The most serious concern is the apparent use of test mAP for the hard-switch decision, which, if confirmed, would invalidate the hard-switch savings. The cost-model issue is also important because the quantitative claims are expressed in annotation time. I believe both concerns are fixable within the scope of a revision, so major_revision rather than reject seems appropriate. The missing comparison to previously published active learning methods should also be addressed before the 'state of the art' claim is used."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"One thing to know about arXiv:1908.02454: it proposes a simple, sensible way to cut annotation cost in active learning for object detection — query center clicks first, then ask for full bounding boxes only on images where the model's own confidence is low. Across VOC 2007, VOC 2012, and a wheat head dataset, and with three different query strategies, the soft-switch variant reaches the same mAP with roughly 15–30% less annotation time than the all-strong baseline. The idea is genuinely new in this combination: prior AL methods for detection query only strong labels, and weak supervision methods don't do active sample selection. The mechanism is transparent, model-agnostic, and easy to graft onto existing detectors.\n\nThe paper also does some things right. The ablation on passive learning shows the gain is not solely from active sampling. The authors explicitly note that annotation times for VOC/Wheat are unavailable and use ImageNet medians; that's honest, but they don't stress how load-bearing that choice is. The writing is clear and the experiments cover a real application domain.\n\nThe soft spots are real but mostly addressable. The headline numbers are annotation-time savings, and those savings are computed entirely from a timing model (34.5s per box, 3s per click, 7.8s overhead per image) taken from earlier ImageNet studies. If the true cost of center clicks relative to boxes on VOC/Wheat is higher than 3s, or the overhead structure differs, the percentages shift. No sensitivity analysis is given. Weak labels are simulated by adding Gaussian noise to ground-truth centers, not collected from people, so the 3s cost is not actually exercised. All experiments are single-run with no error bars, so the 17–30% figures are point estimates. Thresholds γ and δ are hand-picked, with no robustness study. And the abstract calls the baseline 'state of the art,' but the comparison is to standard PBAL with three classic query strategies, not to recent tuned AL detectors. No code or data is released, which makes reproduction harder.\n\nThat said, the central qualitative result holds up: adaptive supervision reaches the target mAP with less annotation time than all-strong PBAL, consistently, and it is not forced by construction. The exact savings should be treated as indicative, not measured. This is a useful paper for anyone trying to reduce labeling cost in detection, especially in domains like agriculture or medicine where boxes are expensive. It deserves a serious referee. A revision that adds error bars, a sensitivity analysis of the timing model, and ideally real click annotations would move it from promising to solid.","headline":"A simple and sensible way to interleave weak and strong labels in active detection, with consistent but not rigorously quantified savings.","tokens_in":10801,"tokens_out":2343,"would_cite":true,"duration_ms":25576,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Active learning that asks for center clicks before full boxes reaches target accuracy with roughly 15–30% less annotation time.","keywords":["active learning","object detection","weak supervision","center clicks","adaptive supervision","pseudo-labeling","annotation cost","Faster R-CNN"],"falsifier":"Measure actual annotation times for full boxes versus center clicks on PASCAL VOC 2007 with the same annotator pool, then re-plot the curves using real times; if center clicking costs more than roughly a small fraction of a full box, for instance 10 seconds per click rather than 3 seconds, the reported 30% savings at mAP around 0.55 will largely or entirely disappear.","tokens_in":9916,"feed_emoji":"🎯","tokens_out":5248,"duration_ms":54362,"temperature":0.7,"pith_summary":"The paper tries to establish that active learning for object detection does not need to query full bounding boxes at every round. It proposes an adaptive supervision framework in which the oracle first supplies cheap center-click labels, the detector's own predictions are filtered by those clicks to form pseudo-labels, and a switching rule raises the supervision level to full boxes only where the model is unsure. The central result is that this schedule reaches the same mAP as standard pool-based active learning with roughly 15–30% less annotation time on PASCAL VOC 2007, VOC 2012, and a wheat-head detection dataset. If true, the value is that annotation budgets can be spent on data that actually needs strong labels, without changing the detector architecture.","feed_headline":"Click-first active learning cuts detection annotation cost by up to 30%","feed_subtitle":"Queries weak center-click labels first, saving full bounding boxes for the images the model is least sure about.","key_machinery":"The load-bearing object is the adaptive supervision module, a small decision layer on top of an off-the-shelf detector. It has two named variants: a hard (inter-episode) switch, which checks whether the latest mAP improvement has dropped to at most gamma times the best improvement so far, and a soft (intra-episode) switch, which checks each image's average prediction confidence against a threshold delta and routes low-confidence images to strong annotations. The other half of the machinery is pseudo-labeling from center clicks: given click locations, the model keeps, for each click, the predicted box whose center is closest and tags it with the highest-probability class; this converts weak labels into training boxes in one forward pass, with no architectural change.","core_discovery":"The central discovery is that interleaving weak supervision into active learning is not just a cost-saving add-on: it is a way to spend the annotation budget better, because weakly supervised pseudo-labels let the detector train on more images per hour and strong labels are reserved for exactly the images where the model is least confident. Concretely, with the soft-switch variant, each actively sampled image gets pseudo-labels from its center clicks unless the model's average prediction confidence falls below a threshold, in which case the image is sent for full bounding-box annotation. The paper reports that this soft switch beats both standard active learning and the hard-switch variant across three active query metrics and three datasets, e.g. reaching mAP around 0.55 on VOC 2007 in about 24.6 hours instead of around 35 hours.","pith_inferences":["If the claim holds, a natural extension is to tune the switching thresholds online as a function of remaining budget rather than fixing them before training; a budget-aware scheduler could make the trade-off between label cost and model confidence a design choice rather than a constant.","The method's reliance on the detector's own predicted boxes for pseudo-labels suggests that early episodes may be fragile if the initial model is very weak; a testable extension is to measure how the annotation-time savings vary with the size of the initial labeled seed.","The same confidence-based routing idea could transfer to other expensive annotation tasks such as instance segmentation or keypoint labeling, where the decision to ask for a strong label might be made per object or per structure instead of per image."],"forward_implications":["Soft-switch adaptive supervision reaches a fixed mAP with less annotation time than standard pool-based active learning for all three active query methods tested (avg-entropy, max-margin, least-confident) on all three datasets.","Hard switching gives smaller but consistent savings, roughly 7–15%, and remains useful when weak and strong annotation cannot be requested in the same round.","The savings persist in passive learning with random sampling, with soft switch saving about 13% of annotation time to reach mAP 0.53 on VOC 2007, so the gain is not purely an artifact of active query selection.","Because the framework uses only detector outputs and requires no change to the model architecture, the same adaptive module transfers across query methods and datasets."],"supporting_citations":[{"why":"Supplies the center-click weak supervision method and the 3.0 s median per-object click time that the framework uses as its weak-labeling primitive.","marker":"[18]"},{"why":"Supplies the median bounding-box and per-image annotation times (25.5 s, 9.0 s, 7.8 s) used to convert labels into annotation-time costs.","marker":"[26]"},{"why":"Faster R-CNN is the object detector that the framework wraps, providing the bounding-box predictions whose centers are matched to the click locations.","marker":"[21]"},{"why":"Provides the max-margin active sampling query and serves as a deep active-learning baseline for object detection.","marker":"[4]"},{"why":"Provides the average-entropy and least-confident query techniques, along with the deep active-learning baseline that the proposed framework is compared against.","marker":"[22]"},{"why":"Supplies the Wheat dataset used for the real-world agriculture evaluation, which has many object instances per image.","marker":"[17]"},{"why":"Defines PASCAL VOC 2007, the main evaluation dataset and source of the mAP numbers used to compare annotation-time savings.","marker":"[7]"},{"why":"Defines PASCAL VOC 2012, the second evaluation dataset used to confirm the savings.","marker":"[8]"}],"fun_headline_variants":["Weak labels first: active detection learns cheaper","Adaptive supervision: cheaper object detection training","Center-click first: active learning saves 30% annotation cost","Smarter labeling: weak first, strong when needed","Active detection with adaptive supervision cuts labeling cost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported percentage savings rest on the assumption that the ImageNet timing ratios, 34.5 seconds per bounding box, 3 seconds per center click, and 7.8 seconds per image, are realistic for PASCAL VOC and Wheat; if clicks are relatively slower on those images, the headline savings shrink.","fun_headline_variants_meta":{"raw":{"variants":["Weak labels first: active detection learns cheaper","Adaptive supervision: cheaper object detection training","Center-click first: active learning saves 30% annotation cost","Smarter labeling: weak first, strong when needed","Active detection with adaptive supervision cuts labeling cost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000206,"raw_usage":{"total_tokens":1337,"prompt_tokens":828,"completion_tokens":509,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":444,"completion_tokens_details":{"reasoning_tokens":437}},"tokens_in":444,"tokens_out":509,"duration_ms":5613,"temperature":1.0,"reasoning_tokens":437,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:43:55.976563+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure actual annotation times for full boxes versus center clicks on PASCAL VOC 2007 with the same annotator pool, then re-plot the curves using real times; if center clicking costs more than roughly a small fraction of a full box, for instance 10 seconds per click rather than 3 seconds, the reported 30% savings at mAP around 0.55 will largely or entirely disappear.","supporting_citations":[{"cited_title":"Training object class detectors with click supervision","cited_arxiv_id":"1704.06189","evidence_quote":"Supplies the center-click weak supervision method and the 3.0 s median per-object click time that the framework uses as its weak-labeling primitive."},{"cited_title":"Crowdsourcing annotations for visual object detec- tion","cited_arxiv_id":null,"evidence_quote":"Supplies the median bounding-box and per-image annotation times (25.5 s, 9.0 s, 7.8 s) used to convert labels into annotation-time costs."},{"cited_title":"Faster r-cnn: To- wards real-time object detection with region proposal networks","cited_arxiv_id":null,"evidence_quote":"Faster R-CNN is the object detector that the framework wraps, providing the bounding-box predictions whose centers are matched to the click locations."},{"cited_title":"Namboodiri","cited_arxiv_id":null,"evidence_quote":"Provides the average-entropy and least-confident query techniques, along with the deep active-learning baseline that the proposed framework is compared against."},{"cited_title":"Ear density estimation from high resolution rgb imagery using deep learning technique","cited_arxiv_id":null,"evidence_quote":"Supplies the Wheat dataset used for the real-world agriculture evaluation, which has many object instances per image."},{"cited_title":"Everingham, L","cited_arxiv_id":null,"evidence_quote":"Defines PASCAL VOC 2007, the main evaluation dataset and source of the mAP numbers used to compare annotation-time savings."},{"cited_title":"Everingham, L","cited_arxiv_id":null,"evidence_quote":"Defines PASCAL VOC 2012, the second evaluation dataset used to confirm the savings."}],"review_version":1}