{"id":"70ad8b12-ae08-4699-b28e-6d4cf86cc46a","arxiv_id":"2501.07727","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"On realistic, high-cardinality, imbalanced text tasks, weak supervision beats supervised learning until supervised training has over 1,000 labeled examples.","lead":"A new benchmark, BOXWRENCH, tests weak supervision on harder, more realistic text tasks with many classes, imbalanced labels, and domain expertise. It finds that supervised learning often needs over 1,000 hand labels to match weak supervision, much higher than older benchmarks suggested.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Supervised baseline is handicapped by fixed 6,000 steps with no early stopping, inflating crossover points; re-running with early stopping could move crossovers below 1,000.","rationale":"The reader's weakest_assumption already identifies the early-stopping asymmetry as part of the concern, and my stress-test agrees that it is the most load-bearing issue. The central claim depends entirely on where the supervised and WS curves cross. An unfair baseline moves those crossing points in a known direction: fixed 6,000-step fine-tuning on tiny labeled sets overfits, and early stopping on those same sets would improve supervised accuracy. Since the paper does not report any control experiment with early stopping for the supervised models, the magnitude of the inflation is unknown. The concern is not that WS is useless or that the datasets are uninteresting; it is that the quantitative headline ('1000+ labels') may be an artifact of protocol. The proposed test is cheap because the code and LF assets are released, and it directly settles whether the crossover values survive a symmetric protocol. This supports the reader's CONDITIONAL verdict rather than a rejection.","tokens_in":18069,"tokens_out":5728,"duration_ms":56925,"concrete_test":"Re-run the supervised, WS, and CFT curves for Banking77, Claude9, and ChemProt with supervised fine-tuning using early stopping on a held-out portion of the validation set (or the same validation set used for WS early stopping), matching the WS hyperparameter search budget and using at least 5 seeds. If the supervised curve rises enough that crossovers for Banking77 or ChemProt fall below 1,000 labels, or a crossover appears on Claude9, the headline claim is weakened; if crossovers remain above 1,000, the concern is refuted.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The crossover comparison is skewed by an asymmetric training protocol in Section 3.4: weakly supervised and CFT runs use early stopping on the validation set, while supervised runs are fine-tuned for a fixed 6,000 steps with no early stopping. On small validation sets (50-1,000 examples), 6,000 RoBERTa fine-tuning steps is far into overfitting territory, so the supervised curves in Figures 3-5 are systematically depressed. This directly inflates the reported crossover points. The effect is likely largest on imbalanced Claude9, where macro-F1 is sensitive to collapse onto the majority class, and on Banking77 at low label counts. Because the paper's headline claim is exactly that crossovers exceed 1,000 labels, this asymmetry is load-bearing: if supervised models are allowed the same early-stopping or any standard model-selection procedure, crossovers could move below 1,000 or Claude9 could cross over, and the 'stronger than you think' conclusion would be substantially weakened. Following Zhu et al. [49] does not make the fixed-6,000-step rule a fair baseline; this paper argues against that prior result and should not handicap the comparison. The paper's own released code makes the re-run feasible.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces BOXWRENCH, a weak supervision benchmark consisting of five text classification settings (Banking77, ChemProt, Claude9, MASSIVE18/60, and Amazon31) chosen to be more realistic than earlier WS suites in class cardinality, class imbalance, domain expertise, and cross-lingual LF reuse. Against Zhu et al.'s \"Weaker Than You Think\" finding, the authors measure crossover points where fine-tuning on clean validation labels overtakes weak supervision (WS) or continuous fine-tuning (CFT). They report crossovers beyond 1,000 clean labels on Amazon31 and Banking77, no crossover on Claude9, an increase in the ChemProt F1 crossover to about 1,600 after improved LFs, and crossovers above 1,000 on multilingual MASSIVE18 variants. The paper releases code, labeling functions, and (for most tasks) datasets.","tokens_in":18277,"tokens_out":10049,"duration_ms":107291,"significance":"If the crossover results hold under a fair comparison, BOXWRENCH would be a genuinely useful benchmark: it moves WS evaluation toward high-cardinality, imbalanced, domain-specific tasks; it publishes a documented LF-development procedure; it reproduces the opposing Zhu et al. results on existing benchmarks; and it demonstrates an interesting cross-lingual LF-reuse mechanism with DeepL. The public code release is a real strength and makes the control experiments requested below feasible. I would not yet endorse the headline \"stronger than you think,\" because the two label-efficiency asymmetries discussed below directly affect the magnitude and even the existence of the reported crossover points.","major_comments":[{"comment":"The supervised baseline is trained for a fixed 6,000 steps with no early stopping, while weakly supervised and CFT models use early stopping on the validation set. On clean sets of 50-1,000 examples, a fixed 6,000-step RoBERTa fine-tune is very likely to be in the overfitting regime, so the supervised curves in Figures 3-5 may be systematically depressed. Table 2 is consistent with this concern: ChemProt supervised F1_macro is essentially flat (0.405-0.412) as the validation size grows from 100 to 1,607. This asymmetry is load-bearing because the paper's headline is that crossovers exceed 1,000 labels and that Claude9 shows no crossover with 200 labels. Following Zhu et al. [49] does not make the fixed schedule fair: this paper argues against that work's conclusion and should not inherit a protocol that handicaps the supervised side. Please rerun supervised and CFT with matched model selection (e.g., early stopping on the same validation labels for all methods), or provide evidence that 6,000 fixed steps is not overfitting for these datasets.","section":"Section 3.4, Figures 3-5"},{"comment":"The label-efficiency comparison undercounts WS's clean-label consumption. Section 3.3 describes a 250-example LF development set for Amazon31 and a 24-example development set for Claude9, and states that final LFs are evaluated on the original validation set; the same validation labels are also used for WS early stopping and hyperparameter selection (Section 3.4). The crossover analysis in Section 4.1 compares supervised training on Dval against WS/CFT, counting only Dval labels as the \"label cost.\" The clean development labels and the clean labels used to select LFs are not charged to WS. A fair comparison should either add these labels to the supervised budget (e.g., supervised with nval+250 labels for Amazon31 or nval+24 for Claude9) or explicitly report a sensitivity analysis showing that the >1,000 crossovers are robust under this accounting. Without this, the reported crossover points conflate WS's algorithmic value with the extra clean labels consumed during LF development and selection.","section":"Section 3.3 and Sections 3.2/4.1"},{"comment":"Crossover points are reported as point estimates without uncertainty intervals. The curves in Figures 3-5 have no error bars, and Table 2 reports means and standard deviations only at discrete validation fractions, not at the crossover. Because fine-tuning is stochastic and several of the clean sets are tiny (12, 25, 50, 100, and 200 labels for Claude9), the claims \"beyond 1,000,\" \"no crossover,\" and \"around 1,600\" need confidence intervals or bootstrap distributions over seeds. As written, the reader cannot tell whether, for example, the Amazon31 crossover is statistically distinct from 800 or whether Claude9's \"no crossover\" would survive perturbing the supervised curve by one standard error. Please report per-point standard errors and, if feasible, uncertainty on the interpolated crossover locations.","section":"Section 4.1 and Figures 3-5"}],"minor_comments":[{"comment":"The dataset name \"ChemPort\" in the Table 2 caption is a typo for \"ChemProt.\"","section":"Table 2 caption"},{"comment":"Amazon31 is no longer publicly available and is explicitly not released with BOXWRENCH; because Amazon31 is one of the two datasets supporting the >1,000 crossover claim, the main text should mark it as non-reproducible and, if possible, release the processed version or make it available on request.","section":"Appendix B and Section 3.3"},{"comment":"Please specify how crossover points are computed from the validation-size grid (e.g., linear interpolation, first crossing, or visual inspection). The current text says only that the curves \"intersect.\"","section":"Section 3.1"},{"comment":"The paper notes in Limitations that WS hyperparameters were not thoroughly tuned, following Zhu et al.; this choice is conservative for WS and should be stated alongside the main crossover results rather than only in the appendix/limitations, especially because the fixed 6,000-step rule for supervised training is a separate and non-conservative choice.","section":"Section 5, limitation (3)"},{"comment":"The abstract's \"1000+\" claim is too broad: MASSIVE60's crossover is reported as exceeding only 500, and some ChemProt accuracy improvements are described as \"smaller but consistent.\" A more precise summary of where the 1,000-label claim does and does not hold would help readers.","section":"Abstract and Section 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is an empirical benchmarking study, and the two main issues (matched stopping criteria and full label-budget accounting for LF development/selection) are correctable within the manuscript's scope. The stress-test concern about the fixed 6,000-step supervised baseline lands: the protocol asymmetry is load-bearing for the headline crossover claims. I do not see evidence of misconduct; the limitations section is candid, but it does not address these particular asymmetries. If the authors rerun with matched model selection and re-account the label budget, the paper could become a solid benchmark contribution. Amazon31's unavailability is an additional reproducibility concern for the editor to weigh."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, BOXWRENCH is a real step up from WRENCH and AutoWS-Bench-101: higher-cardinality tasks, imbalanced classes, domain-heavy LFs, and a multilingual LF-reuse experiment that is simple and clever. The authors also reproduce Zhu et al. on the old tasks, which is the right way to frame a challenge. Second, the central number—that supervised learning needs 1,000+ labels to match WS on Amazon31 and Banking77, and never catches up on Claude9—rests on a comparison that handicaps the supervised baseline. WS runs early stopping on the validation set; supervised runs are fine-tuned for a fixed 6,000 steps with no early stopping (Section 3.4). At small validation sizes, 6,000 RoBERTa steps is far into overfitting territory. The stress-test note is right: re-running supervised with early stopping could pull those crossovers below 1,000. This is not a nit; it directly bears on the abstract's claim.\n\nThe paper does several things well. The LF design process is documented, and the released assets let others rebuild or improve the LFs. The ChemProt experiment—improving LFs raises the F1 crossover from ~800 to ~1,600—is a clean demonstration that LF quality matters. The MASSIVE translation experiment is practical and useful. And the authors are transparent that Amazon31 was retracted; still, a headline result hinging on an unavailable dataset weakens reproducibility.\n\nTwo more soft spots, both secondary. The LFs are selected using the same validation set that defines the supervised baseline (Section 3.3). That means WS gets to peek at clean labels during LF development, then again during early stopping and model selection, while the supervised model only sees them at training time. The development sets (250 for Amazon31, 24 for Claude9) are not charged to the WS label budget. None of this looks like fraud—it is common practice in parts of the WS literature—but it makes the reported crossovers upper bounds rather than fair estimates.\n\nMy overall take: the benchmark is worth having, and the qualitative conclusion—WS looks more valuable on harder tasks than on the toy benchmarks—survives the critique, at least directionally. The quantitative crossovers need to be re-estimated under a fair protocol. A serious referee should engage, with the expectation of a revision that re-runs supervised baselines with early stopping and reports sensitivity to the supervised step budget.\n\nWho should read this: anyone building WS benchmarks or deciding whether to deploy WS versus 50 clean labels. I would not cite the crossover numbers as-is, but I would cite the benchmark and the LF assets. Recommendation: send to peer review. The benchmark and reproducible assets earn referee time; the training-protocol asymmetry is fixable and should be demanded.","headline":"A genuinely harder WS benchmark that deserves review, but the headline claim of 1,000+ label crossovers is inflated by an unfair supervised baseline.","tokens_in":18835,"tokens_out":3625,"would_cite":true,"duration_ms":32250,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On realistic high-cardinality, imbalanced tasks, weak supervision keeps an advantage over supervised learning until clean labels exceed 1,000, and on one task never loses within 200 labels.","keywords":["weak supervision","labeling functions","label models","benchmark","text classification","crossover point","class imbalance","multilingual transfer"],"falsifier":"Re-run the crossover analysis with a single unified label budget that charges the weak-supervision pipeline for the development labels used to write labeling functions and the validation labels used for early stopping and selection; if supervised training then catches up to weak supervision with fewer than 1,000 labels on Amazon31 or Banking77, the paper's headline claim fails.","tokens_in":17854,"feed_emoji":"📊","tokens_out":8852,"duration_ms":79487,"temperature":0.7,"pith_summary":"BOXWRENCH is a new benchmark for weak supervision (WS), built from text-classification tasks that have many classes, imbalanced label distributions, and genuine domain expertise requirements. The paper's central claim is that on these realistic tasks, supervised learning needs more than 1,000 clean labels to match the better of weak supervision or continuously fine-tuned weak supervision, whereas earlier benchmark studies suggested 50 labels were enough. Two datasets exhibit crossover points beyond 1,000 clean labels, one never crosses within its 200-label validation budget, and carefully rewritten labeling functions raise a fourth dataset's crossover to about 1,600. The same labeling functions written for English are reusable in other languages through translation, yielding crossovers above 1,000 without writing new functions. If the claim holds, weak supervision is a substantially more practical tool for label-hungry applications than recent critiques concluded.","feed_headline":"Supervised learning needs 1,000+ labels to catch weak supervision","feed_subtitle":"On realistic, imbalanced and multilingual tasks, weak supervision holds its lead until clean-label budgets pass the thousand mark.","key_machinery":"The load-bearing measure is the crossover point: the amount of clean validation labels at which a supervised-only model's test performance matches or overtakes the better of the weakly supervised or continuously fine-tuned models (where the latter is a model first trained on weak labels and then further fine-tuned on the clean validation labels). The argument is carried by a controlled pipeline in which labeling functions -- small heuristic rules that vote a label or abstain -- are written by a documented development-set procedure, aggregated by simple label models such as majority vote, and used to fine-tune pretrained language models; validation labels are used for early stopping and for selecting the final labeling functions. Because the same pipeline is run on existing simple benchmarks and on the new harder tasks, the crossover differences are attributed to task realism and labeling-function quality rather than to implementation choices.","core_discovery":"The paper's discovery is that the perceived weakness of weak supervision is an artifact of overly simple benchmark tasks and low-quality labeling functions, not a property of the method. On the new BOXWRENCH tasks, the number of clean labels at which supervised-only training overtakes weak supervision -- the crossover point -- is 1,000 or more on Amazon31 and Banking77, and on Claude9 no crossover occurs within the available 200 validation labels, with continuous fine-tuning ending about 5% higher in macro-F1. Improving the ChemProt labeling functions by adding entity-distance features and pruning low-accuracy rules raises the F1 crossover from roughly 800 to roughly 1,600. In the multilingual setting, English labeling functions applied after machine translation reach crossovers above 1,000 on Chinese and Japanese variants, and the gain persists even when only a fraction of the target-language unlabeled data is available.","pith_inferences":["Editorial inference: an apples-to-apples label budget would charge the weak-supervision pipeline for its development set (250 examples on Amazon31, 24 on Claude9) and for validation labels used in early stopping and labeling-function selection; those charges would pull the crossover points down, and the paper does not quantify how far.","Editorial inference: the translation-based labeling-function reuse recipe should be testable on non-parallel multilingual corpora and on languages without dedicated pretrained encoders; if it holds, weak supervision becomes a cheap route to low-resource text classification.","Editorial inference: the paper's appendix finds that LLM-generated labeling functions have higher coverage but lower precision than the hand-written ones; a natural next step is a pipeline that combines LLM suggestion with the paper's careful validation and pruning procedure.","Editorial inference: the crossover-point framing gives the field a single number to compare datasets and labeling-function sets, which could make future weak-supervision benchmarks directly comparable."],"forward_implications":["On Amazon31 and Banking77, supervised-only training does not catch up to weak supervision until the clean-label budget passes 1,000 examples.","On Claude9, no crossover occurs within the 200 available validation labels, and the continuously fine-tuned weak-supervision model stays about 5% higher in macro-F1.","Improving the ChemProt labeling functions raises the F1 crossover from roughly 800 to roughly 1,600 clean labels, showing that labeling-function quality directly controls how long weak supervision stays ahead.","Reusing English labeling functions via translation yields crossovers above 1,000 on the Chinese and Japanese variants of the voice-assistant task, with no new labeling functions written in the target languages.","The simplest aggregation strategies (majority vote and an expectation-maximization-based label model) matched or beat more complex label models across the new datasets, so the conclusion does not depend on a sophisticated aggregation trick."],"supporting_citations":[{"why":"Supplies the prior finding that 50 clean labels can match weak supervision, which the paper reproduces and then contradicts on harder tasks.","marker":"[49]"},{"why":"Provides the existing benchmark suite and pipeline format whose low crossover points the paper extends with realistic tasks.","marker":"[46]"},{"why":"Supplies the original ChemProt labeling functions that the paper improves with entity-distance features, raising the F1 crossover.","marker":"[45]"},{"why":"Provides the labeling-function development procedure and development-set sizes used to write the new LFs.","marker":"[23]"},{"why":"Supplies the parallel multilingual corpus used to test translating and reusing English labeling functions across languages.","marker":"[10]"},{"why":"Supplies an expectation-maximization label model that, with majority vote, performed best in the paper's ablations.","marker":"[8]"},{"why":"Supplies the programmatic weak-supervision formulation used to build the pipeline.","marker":"[35]"}],"fun_headline_variants":["Weak supervision beats supervised until 1,000+ labels","Crossover point for supervised? Often over 1,000 labels","Weak supervision stays strong on realistic benchmarks","Supervised needs 1,000+ labels to best weak supervision","On many real tasks, weak supervision never gets beaten"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison counts only the clean labels used to train the supervised end model, while the weak-supervision pipeline also spends clean labels on designing its labeling functions and on validation-based early stopping and selection; if those labels are charged to weak supervision, the reported crossover points shrink.","fun_headline_variants_meta":{"raw":{"variants":["Weak supervision beats supervised until 1,000+ labels","Crossover point for supervised? Often over 1,000 labels","Weak supervision stays strong on realistic benchmarks","Supervised needs 1,000+ labels to best weak supervision","On many real tasks, weak supervision never gets beaten"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000182,"raw_usage":{"total_tokens":1309,"prompt_tokens":941,"completion_tokens":368,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":288}},"tokens_in":557,"tokens_out":368,"duration_ms":4371,"temperature":1.0,"reasoning_tokens":288,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:36:20.620594+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the crossover analysis with a single unified label budget that charges the weak-supervision pipeline for the development labels used to write labeling functions and the validation labels used for early stopping and selection; if supervised training then catches up to weak supervision with fewer than 1,000 labels on Amazon31 or Banking77, the paper's headline claim fails.","supporting_citations":[{"cited_title":"activat”, “increas","cited_arxiv_id":null,"evidence_quote":"Supplies the prior finding that 50 clean labels can match weak supervision, which the paper reproduces and then contradicts on harder tasks."},{"cited_title":"Wrench: A comprehensive benchmark for weak supervision, 2021","cited_arxiv_id":null,"evidence_quote":"Provides the existing benchmark suite and pipeline format whose low crossover points the paper extends with realistic tasks."},{"cited_title":"Fine-tuning pre-trained language model with weak supervision: A contrastive-regularized self-training approach","cited_arxiv_id":null,"evidence_quote":"Supplies the original ChemProt labeling functions that the paper improves with entity-distance features, raising the F1 crossover."},{"cited_title":"Characterizing the impacts of semi- supervised learning for weak supervision","cited_arxiv_id":null,"evidence_quote":"Provides the labeling-function development procedure and development-set sizes used to write the new LFs."},{"cited_title":"Massive: A 1m-example multilingual natural language understanding dataset with 51 typologically-diverse languages, 2022","cited_arxiv_id":null,"evidence_quote":"Supplies the parallel multilingual corpus used to test translating and reusing English labeling functions across languages."},{"cited_title":"Ratner, Stephen H","cited_arxiv_id":null,"evidence_quote":"Supplies the programmatic weak-supervision formulation used to build the pipeline."}],"review_version":1}