{"id":"e47bb787-086d-4759-9c1d-27094941f375","arxiv_id":"2604.23290","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Real crowd-sourced text annotations are used to test eight active learning techniques, showing their behavior under actual noisy labeling and refusals unlike simulated oracles.","lead":"This paper collects real annotations from crowd workers on text samples from three benchmark classification datasets and evaluates eight active learning algorithms paired with deep neural networks. It examines how these methods perform when annotators give wrong labels or refuse to label, providing more realistic insights than typical simulated-oracle studies.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"No significant objection identified","rationale":"The reader's weakest assumption correctly flags scope limitations, but these do not constitute a load-bearing risk to the narrower claim that the study illuminates performance on the specific real annotations gathered. Because the paper supplies the raw data, the analysis is falsifiable and reproducible; no hidden modeling assumption (e.g., perfect oracle, specific noise model) is required for the reported trends to be meaningful.","tokens_in":1750,"tokens_out":303,"duration_ms":101336,"concrete_test":"Re-run the eight AL procedures on the released annotation files using the exact train/validation splits and DNN architectures described in the methods section; if the relative ordering of the methods by final test accuracy (or label efficiency) remains stable under 5 random seeds, the headline empirical observations hold.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that empirical evaluation of eight active learning methods (paired with DNNs) on pre-collected crowd annotations from three benchmark text datasets sheds light on behavior when oracles can err or refuse. The setup uses static annotations to simulate the oracle responses during AL selection; the modest claim of providing insights plus public release of the annotation data does not rest on any internally inconsistent or unstated assumption that would falsify the reported observations. Representativeness of MTurk-style workers and coverage of all possible AL variants are limitations on scope, not on the validity of the concrete results obtained from the collected labels.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.3","summary":"The paper collects annotations for text samples from three benchmark classification datasets via a crowd-sourcing platform, then uses these static annotations to simulate oracles that can err or refuse labels. It evaluates the performance of eight common active learning algorithms paired with deep neural networks under these real-world conditions and releases the collected annotations publicly at a GitHub repository.","tokens_in":1845,"tokens_out":417,"duration_ms":29493,"significance":"If the experimental protocol is fully documented and the results are reproducible, the work offers concrete insights into how standard active learning methods behave when oracles exhibit realistic imperfections, moving beyond purely simulated noise models. The public release of the crowd-sourced annotation data is a clear strength that can support follow-on research and benchmarking in noisy active learning.","major_comments":[{"comment":"The description of the empirical evaluation (abstract and §5) provides no details on the active learning simulation protocol: how refusals are handled during query selection, how multiple or conflicting annotations per instance are resolved into a single oracle response, or how the static dataset is replayed across AL iterations. These choices are load-bearing for the central claim that the study captures real-world annotation challenges.","section":"§5 (Experiments)"},{"comment":"No information is given on the number of independent runs, random seeds, statistical significance tests, or variance measures used to compare the eight active learning techniques. Without these, the reported performance differences cannot be assessed for reliability.","section":"§5 (Experiments)"}],"minor_comments":[{"comment":"The eight active learning techniques should be explicitly listed with citations in the introduction or methods section for clarity.","section":"Introduction"},{"comment":"The paper would benefit from a brief discussion of how the chosen crowd-sourcing platform and worker pool relate to other real-world annotation settings.","section":"Data Collection"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback. The comments highlight areas where additional detail will strengthen the reproducibility and clarity of our experimental protocol. We address each point below and will revise the manuscript accordingly.","responses":[{"response":"We agree that these protocol details are essential. The current manuscript describes the collection of annotations but does not fully specify the replay mechanics in the AL loop. In revision we will add a dedicated paragraph (and pseudocode) in §5 clarifying: (i) refusals are treated as 'no label' and the instance remains in the unlabeled pool for potential future queries; (ii) when multiple annotations exist for an instance, we use majority vote among non-refusal labels (ties broken randomly); (iii) the static annotation set is replayed deterministically—each queried instance receives the pre-collected label (or refusal) without re-sampling or model-based simulation. This will make explicit how real-world noise and abstention are injected.","revision_made":"yes","referee_comment":"[§5 (Experiments)] The description of the empirical evaluation (abstract and §5) provides no details on the active learning simulation protocol: how refusals are handled during query selection, how multiple or conflicting annotations per instance are resolved into a single oracle response, or how the static dataset is replayed across AL iterations. These choices are load-bearing for the central claim that the study captures real-world annotation challenges."},{"response":"We acknowledge the omission. The experiments were executed with multiple independent runs using fixed random seeds for model initialization, data shuffling, and query selection. In the revision we will report the exact number of runs, the seed values, the variance (standard deviation) across runs for all curves, and any statistical comparisons performed. If the referee prefers, we can also include pairwise significance tests in the updated tables/figures.","revision_made":"yes","referee_comment":"[§5 (Experiments)] No information is given on the number of independent runs, random seeds, statistical significance tests, or variance measures used to compare the eight active learning techniques. Without these, the reported performance differences cannot be assessed for reliability."}],"tokens_in":1365,"tokens_out":467,"duration_ms":32072,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"This paper collects annotations from actual crowd workers on three benchmark text classification datasets and then runs eight standard active learning methods paired with deep neural networks against those labels. The annotations are released on GitHub. That is the concrete step forward. Most noisy-oracle active learning work still relies on machine learning models to fake errors, which misses how real people refuse labels or err in uneven ways. Using pre-collected human labels gives a more direct look at those issues during selection and training.","headline":"Real crowd annotations on text data, released publicly, is the main thing worth knowing here.","tokens_in":2306,"tokens_out":159,"would_cite":true,"duration_ms":50898,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Real crowd-sourced annotations show how eight active learning methods handle label noise and refusals on text data.","keywords":["active learning","crowdsourcing","noisy labels","text classification","deep neural networks","empirical evaluation","imperfect oracles","label abstention"],"falsifier":"Repeating the full set of experiments with annotations collected from a different crowd platform or different workers that reverses the relative performance of the eight algorithms would undermine the reported findings.","tokens_in":2658,"feed_emoji":"📊","tokens_out":594,"duration_ms":28096,"temperature":0.7,"pith_summary":"The paper gathers text annotations directly from crowd workers on three standard classification datasets through a platform. These real labels contain mistakes and instances where workers decline to answer. It then runs eight common active learning algorithms paired with deep neural networks on the collected data. The analysis checks how the methods behave when the oracle is imperfect, unlike the perfect oracles in theory or the machine-simulated noise in earlier studies. This matters because it can indicate which techniques are more practical when human labelers introduce errors or stop providing labels.","feed_headline":"Real crowd labels expose active learning differences under noise","feed_subtitle":"Eight methods tested on human annotations from three text datasets reveal effects of errors and refusals unlike machine simulations.","key_machinery":"The empirical evaluation that uses real crowd-sourced annotations incorporating human errors and abstentions to test active learning techniques instead of simulated oracles.","core_discovery":"By collecting actual annotations from crowd-sourced workers on benchmark text datasets and evaluating eight active learning techniques with deep networks on them, the work reveals the impact of incorrect labels and label refusals on algorithm performance, offering evidence that differs from results obtained with simulated oracles.","pith_inferences":["Similar experiments on image or audio data could check whether the observed effects of noise and abstention hold in other modalities.","Active learning variants that estimate individual worker reliability might reduce the impact of errors and refusals.","Larger-scale replications with more datasets would test if the performance patterns remain stable across different annotation conditions."],"forward_implications":["Active learning methods may select less useful samples or require more queries when faced with inconsistent human labels.","Worker refusals to provide labels can slow progress more than label errors alone in some techniques.","Deep neural networks trained with active learning may need modifications to account for real annotation variability before deployment.","The released dataset supports further testing of noisy active learning strategies on text tasks."],"fun_headline_variants":["Crowd annotations show active learning noise impacts","Real labels reveal active learning method variances","Eight active learning techniques evaluated with crowd noise","Real-world data shows active learning sensitivity to errors"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The crowd annotations obtained from the platform represent the full range of real-world labeling problems and the eight chosen techniques cover the main active learning approaches in use.","fun_headline_variants_meta":{"raw":{"variants":["Crowd annotations show active learning noise impacts","Real labels reveal active learning method variances","Eight active learning techniques evaluated with crowd noise","Real-world data shows active learning sensitivity to errors"]},"model":"grok-4.3","cost_usd":0.006872,"raw_usage":{"total_tokens":3118,"prompt_tokens":685,"num_sources_used":0,"completion_tokens":53,"cost_in_usd_ticks":68715500,"prompt_tokens_details":{"text_tokens":685,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2380,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":685,"tokens_out":53,"duration_ms":37928,"temperature":1.0,"reasoning_tokens":2380,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-08T08:14:45.391273+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Repeating the full set of experiments with annotations collected from a different crowd platform or different workers that reverses the relative performance of the eight algorithms would undermine the reported findings.","supporting_citations":[],"review_version":1}