{"id":"3b2a935c-f5f8-48cf-baf0-b3cad7400406","arxiv_id":"2501.12332","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"RAC ranks label descriptions by semantic similarity, then runs iterative binary LLM checks one label at a time, improving zero-shot labelling accuracy and enabling a precision-coverage trade-off.","lead":"This paper proposes Retrieval Augmented Classification (RAC), which labels text by checking one candidate label at a time with an open-source 7B language model, starting with the label most similar to the query. RAC improves labelling accuracy on high-cardinality datasets and can abstain on uncertain cases, trading coverage for precision.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The retriever ceiling for Truncated RAC is never measured: only an average MRR@1 of 0.691 is reported, but with n=5 the relevant quantity is recall@5, and without it the F1-vs-coverage tradeoff may be a selection artifact.","rationale":"RAC is a simple, practical method, and the improvements on public benchmarks are consistent across the reported settings: Table 3 shows L+D outperforming L in all eight RAC configurations. The paper also includes a label-distillation check and an internal use case, which support the usefulness of the framework. I do not see an internal inconsistency in the algorithm itself. The most load-bearing unexamined condition is the one the reader identified: Truncated RAC cannot label an example correctly unless the true label survives retrieval. The paper's only retrieval metric is a single average MRR@1 of 0.691 for L+D; that number is insufficient for n=5, and F1 on labelled examples alone can create a spurious tradeoff by dropping the hard tail. I considered the alternative concern that the 'All info in prompt' baseline is not shown and scores only 4.0 F1, which is suspiciously low; however, the Full RAC versus Random Retriever comparison in Table 4 partially controls for the benefit of the iterative binary setup, so the retriever ceiling is the more decisive issue. A per-dataset recall@5 computation would settle whether the reported F1 improvement is a genuine property of the method or a selection artifact. This does not warrant rejecting the paper, but it does justify keeping the conditional verdict until the missing retrieval analysis is provided.","tokens_in":10133,"tokens_out":7516,"duration_ms":78914,"concrete_test":"Run the retrieval step exactly as described (all-mpnet-base-v2, L+D label representations) on the four public test sets and compute per-dataset recall@5 (the fraction of examples whose true label is in the top 5) and MRR@5. If recall@5 is at least 0.9 on every dataset, the retriever is not the bottleneck and the existing conclusions stand; if it is materially below the reported Truncated coverage (e.g., 88.7% for Banking77), then a large share of labelled examples cannot be correct, and Truncated RAC's F1 advantage should be reinterpreted as the effect of abstaining from the hard tail rather than as improved label quality. As a tighter check, recompute the Table 3 Truncated (L+D) F1 restricted to examples whose true label is in the top 5; the drop relative to the reported F1 measures how much retrieval misses cost.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim as stated in Table 3 ('Truncated RAC improves F1 while covering roughly 70-89% of examples') depends on the retrieved candidate list containing the true label within the n=5 steps. This is a hard ceiling: if the true label is outside the top n, no LLM quality can label the example correctly. The paper reports only a single averaged MRR@1 of 0.691 for L+D (Table 2) and says MRR for other n 'shows consistent trends' without giving recall@5 or per-dataset numbers. MRR@1 does not bound recall@5, and an average across four datasets can mask low recall on high-cardinality sets such as Banking77 and Amazon. Moreover, F1 is computed only on examples the LLM decides to label; if abstention is concentrated in examples whose true label is missing from the candidate list, Truncated RAC's F1 advantage over Full RAC is a selection effect, not evidence that truncation improves label quality. Without recall@5, the 'trade-off' claim is not established; it may simply be precision gained by discarding the hard tail.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies zero-shot automatic labelling with open-source 7B-scale LLMs under privacy and cost constraints. The authors observe that providing label names and descriptions improves binary classification but that a single multi-class prompt with all label schema degrades on high-cardinality tasks (e.g., 4.0 F1 on Banking77). They propose Retrieval Augmented Classification (RAC): label descriptions are embedded offline, a retriever ranks labels per example, and the LLM performs a series of binary classifications over the ranked list, stopping when it accepts a label. A truncated variant stops after n rejections and abstains. The paper evaluates RAC on AGNews, DBpedia, Amazon, and Banking77, reporting that label descriptions improve F1 in both Full and Truncated RAC, that Truncated RAC improves F1 at reduced coverage, and that label distillation on truncated labels is competitive on low-cardinality datasets. They also apply the method to an internal 61-class consumer banking dataset and show that combining top-1 retrieval with multi-inference self-consistency reaches up to 74.7 macro-F1 at 35.5% coverage.","tokens_in":10401,"tokens_out":9994,"duration_ms":88811,"significance":"If the empirical claims hold, the paper makes a useful practical contribution: it shows that a simple retrieval-augmented binary decomposition can make open-source 7B models usable for in-house labelling, and it documents real latency/coverage trade-offs. The paper is commendable for evaluating on multiple public datasets, reporting latency, including an industry case study, and testing label distillation. The central claims are nevertheless under-supported in several places: the retriever ceiling for truncated RAC is not measured, the all-info baseline is absent from the main table, and design choices are selected on the test sets without uncertainty quantification. These issues are fixable with additional analysis and experiments.","major_comments":[{"comment":"The paper's central trade-off claim for Truncated RAC is not supported without per-dataset recall@n. Truncated RAC only offers the top n labels, so the retriever's recall@n is an upper bound on the fraction of examples for which correct labelling is possible; the paper reports only an averaged MRR@1 (0.691 for L+D, Table 2) and a statement that other n 'show consistent trends'. Because F1 is computed only on labelled examples (Section 4.1), the improved F1 of Truncated over Full RAC in Table 3 (e.g., Banking77 73.4 vs 68.3 macro) could simply reflect a selection effect in which abstention removes examples whose true label is absent from the candidate list, rather than an improvement in label quality. Please report recall@n per dataset, an oracle ceiling for Truncated RAC (assuming the LLM always accepts the true label when it is in the top n), and a comparison of Full RAC F1 computed on the same subset of examples that Truncated RAC labels.","section":"Section 4.1/4.2, Tables 2 and 3"},{"comment":"The abstract and introduction claim that naively providing all label descriptions in a single prompt degrades performance on high-cardinality tasks, motivating RAC. However, the all-info-in-prompt baseline is reported only for Banking77 (Table 4); Table 3 does not include this baseline for AGNews, DBpedia, or Amazon. Without these numbers, the claimed improvement of RAC over the naive single-prompt approach is not established across the public benchmark suite. Please add the all-info baseline to Table 3 or provide the numbers for all datasets.","section":"Section 3 and Table 4"},{"comment":"The paper tunes the LLM (Mistral vs Llama), prompting style (CoT vs no-CoT), self-consistency strategy and inference count, and the truncation steps n (2 or 5) using the same public test splits on which the final results are reported. This selection process can inflate the reported F1 numbers and makes the cross-dataset consistency claims fragile. Please either use a separate validation split for all design choices or provide bootstrap confidence intervals and significance tests for the key comparisons in Tables 3 and 4.","section":"Sections 2.3 and 4"},{"comment":"For the public datasets, label descriptions are generated by Mistral-7B from an unspecified 'small quantity' of training examples and then used by Mistral-7B for classification. Footnote 2 acknowledges a possible bias but no experiment quantifies it. Since the L+D improvement is a central message, please include at least one comparison with non-Mistral-generated descriptions (e.g., human-written or GPT-4-written) on a public dataset. In addition, specify the number of training examples per class used for description generation, and the number of self-consistency inferences used in the main RAC experiments (Section 2.3 mentions majority voting but does not state the vote count for Table 3).","section":"Section 2.1 and Appendix C"}],"minor_comments":[{"comment":"Throughout the manuscript, 'COT' and 'CoT' are used inconsistently; please standardize to 'CoT'.","section":"Global"},{"comment":"The sentence 'When we use LLM to make a binary classification, we only output a prediction when no-COT and COT approaches output the same label' is ambiguous; please clarify whether the main RAC experiments require agreement between the two prompting variants or use a single variant.","section":"Section 2.3"},{"comment":"The row and column headers are unclear, in particular what 'BS' and 'Both' denote; please restructure the table.","section":"Appendix B, Table 8"},{"comment":"Please report the number or fraction of training examples used for label distillation, since the coverage of the truncated labelling on the training split is not stated.","section":"Section 4.3.2"},{"comment":"The number of training examples used to generate each label description is not given; please specify this quantity, as it is a free parameter that affects description quality.","section":"Section 2.1"},{"comment":"Reporting only the MRR@1 averaged over datasets obscures large differences in cardinality; please include per-dataset MRR values.","section":"Table 2"},{"comment":"The source code and configuration details are not provided; a reproducibility statement or a public code release would greatly help readers.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a practically important problem and the proposed method is simple to adopt. The main concern is empirical support for the truncated-RAC trade-off: the missing recall@n analysis and the absence of the all-info baseline from the main table make the core claims weaker than the abstract suggests. These are fixable with additional experiments and analysis. The internal dataset results cannot be independently verified, but that is typical for industry papers. No concerns about citation practices or novelty disclosure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"RAC is a sensible combination of existing ideas—label schema prompts, retrieval over label descriptions, and binary one-vs-rest classification—and the paper provides a genuinely practical framework for zero-shot labelling with 7B open-source models. The latency study and label distillation experiments are pluses; the internal use case with SME-written descriptions is concrete and honestly reported, including cases where the method did not meet business requirements.\n\nThe central empirical claim, that Truncated RAC trades coverage for higher F1, is plausible but not yet established. The paper reports only MRR@1 averaged over datasets (0.691 for L+D) and says the trend is consistent for other n, but with truncation at n=5 the relevant quantity is recall@5. Without per-dataset recall@5, the F1-vs-coverage trade-off could be selection: the model abstains on examples whose true label is missing from the retrieved list, so the F1 gain may simply reflect discarding the hard tail. This is not a trivial omission because it bears directly on the headline contribution. It is also easily fixed.\n\nOther soft spots are more minor but real. The prompt strategy and step counts are chosen on the same test splits used for the final numbers; there are no error bars; no code or data are released. The label descriptions for public datasets are generated by Mistral-7B and then evaluated with Mistral-7B, which the authors acknowledge in a footnote. The all-label baseline prompt is not shown, which makes the 4.0 F1 result on Banking77 hard to inspect.\n\nNone of this kills the paper. The improvements are consistent across datasets, and the method is simple enough to reproduce independently. The paper would be stronger with recall@5 numbers, per-dataset retrieval metrics, and a clearer statement of how many configuration choices were made on the same test set.\n\nThis is an applied paper for practitioners who need privacy-preserving labelling with modest compute, and a useful datapoint for researchers working on label-schema integration. It deserves a serious referee—send it to review, but expect revision. I would want the retrieval ceiling measured before publication.","headline":"A useful applied method with a plausible central claim, but the truncation trade-off is not actually measured because recall@5 is missing.","tokens_in":10923,"tokens_out":2536,"would_cite":true,"duration_ms":24220,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Retrieval-ordered binary checks lift labeling F1 from 4 to 68 on Banking77.","keywords":["automatic labelling","large language models","label schema","retrieval augmented classification","zero-shot classification","open-source LLM","coverage tradeoff","intent classification"],"falsifier":"Compute the retriever's recall@n (the fraction of examples whose true class appears in the top-n retrieved candidates) on a dataset; if that recall is materially below Truncated RAC's reported coverage on the same classes, then the LLM's abstention rate is set by retrieval misses rather than by the model's confidence, and improving the retriever alone should raise F1.","tokens_in":9958,"feed_emoji":"🏷️","tokens_out":5635,"duration_ms":45956,"temperature":0.7,"pith_summary":"The paper argues that open-source 7B language models can label text accurately if the classification problem is restructured: instead of asking the model to pick from all classes at once, retrieve the most semantically related class descriptions, then ask the model a yes/no question for each candidate in order, stopping at the first yes. The authors show that providing both label names and descriptions systematically helps, but only when classes are presented one at a time; presenting all 77 classes at once collapses to 4.0 macro-F1 on Banking77, while their full pipeline reaches 68.3 and a truncated version that only probes the top-ranked labels reaches 73.4 at 88.7% coverage. This matters because it makes privacy-preserving, in-house labeling with open models practical for high-cardinality business taxonomies.","feed_headline":"Label-by-label checks lift open LLM labeling F1 from 4 to 68","feed_subtitle":"Truncating candidate labels to the top ranks pushes F1 to 73.4 at 88.7% coverage with a 7B open model.","key_machinery":"Retrieval Augmented Classification (RAC): each class is embedded as a label name plus a ~50-word description; at inference time the text to label is embedded and the classes are sorted by cosine similarity. The LLM then runs binary prompts ('does this sample belong to class X?') in that order, accepting the first class that returns True. Truncated RAC only probes the top n classes and abstains if none are accepted, turning the retriever's ranking into a precision/coverage dial.","core_discovery":"The central claim is that RAC, which dynamically integrates label schema by ranking label descriptions and performing binary LLM classification one label at a time, improves labelling F1 over a single all-label prompt, and that Truncated RAC trades coverage for higher F1. The paper reports that Full RAC (L+D) reaches 68.3 macro-F1 on Banking77 while Truncated RAC reaches 73.4 macro-F1 at 88.7% coverage, against a 4.0 F1 all-info-in-prompt baseline.","pith_inferences":["Because the retriever is an off-the-shelf sentence transformer, swapping in a stronger retriever or fine-tuning it on label descriptions should directly raise the ceiling of Truncated RAC, since the LLM never sees classes beyond the top n.","The paper's abstention mechanism can serve as a cheap confidence signal: coverage itself is a proxy for confidence, so deployments could set a coverage threshold to match business tolerance for unlabeled data.","A natural extension, which the authors note in passing, is to return all accepted labels within the n probes, turning RAC into a multi-label classifier; a testable question is whether this variant maintains precision when label co-occurrence is high."],"forward_implications":["Label descriptions are worth generating even when not provided, since they improve both retrieval ranking and binary classification in every tested configuration.","Truncated RAC can label the majority of a dataset (>70%) with substantially higher accuracy than Full RAC, making abstention a viable route to quality.","The quality-coverage knob lets practitioners set the F1/coverage tradeoff per business need, as demonstrated on a 61-class internal banking dataset where micro-F1 ranges from 61.5 to 82.2 depending on truncation and self-consistency.","Labels produced by Truncated RAC can be distilled into a small classifier that generalizes beyond the labeled subset even with noisy, partial training data, approaching Truncated RAC's performance on several public benchmarks."],"supporting_citations":[{"why":"Supplies the label-schema integration strategy (label names plus descriptions) that RAC builds on.","marker":"[9]"},{"why":"Documents the 'lost in the middle' long-context failure that motivates replacing the all-label prompt with one-label-at-a-time binary questions.","marker":"[19]"},{"why":"Provides the multiclass-to-binary reduction that RAC applies to turn classification into a sequence of binary decisions.","marker":"[1]"},{"why":"Contributes the retrieval-augmented classification idea that RAC adapts to zero-shot LLM labelling.","marker":"[20]"},{"why":"Another retrieval-augmented classification baseline that supports the design of ranking labels before classification.","marker":"[12]"},{"why":"Supplies the prompt templates used to generate label names and descriptions from training examples.","marker":"[29]"},{"why":"Defines the Banking77 dataset, the high-cardinality benchmark where the all-info baseline collapses to 4.0 F1.","marker":"[5]"},{"why":"Introduces Mistral-7B, the open-source model used for all main experiments.","marker":"[13]"}],"fun_headline_variants":["Open LLMs label better when given one label at a time","Ranked label schemas lift open LLM labeling F1 from 4 to 68","Per-label retrieval boosts open-source LLM labeling accuracy","Truncated label list ups open LLM F1 to 73 with 88% coverage","Dynamic label integration turns open LLMs into strong labelers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The semantic retriever must place the true label inside the top few candidates for most examples, because Truncated RAC never offers labels beyond the top n; the paper's own MRR@1 averages 0.691, so for roughly a third of examples the correct label is not first, and for some it may not appear at all.","fun_headline_variants_meta":{"raw":{"variants":["Open LLMs label better when given one label at a time","Ranked label schemas lift open LLM labeling F1 from 4 to 68","Per-label retrieval boosts open-source LLM labeling accuracy","Truncated label list ups open LLM F1 to 73 with 88% coverage","Dynamic label integration turns open LLMs into strong labelers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0005,"raw_usage":{"total_tokens":2401,"prompt_tokens":854,"completion_tokens":1547,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":470,"completion_tokens_details":{"reasoning_tokens":1451}},"tokens_in":470,"tokens_out":1547,"duration_ms":11472,"temperature":1.0,"reasoning_tokens":1451,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:15:41.820267+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the retriever's recall@n (the fraction of examples whose true class appears in the top-n retrieved candidates) on a dataset; if that recall is materially below Truncated RAC's reported coverage on the same classes, then the LLM's abstention rate is set by retrieval misses rather than by the model's confidence, and improving the retriever alone should raise F1.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces Mistral-7B, the open-source model used for all main experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Another retrieval-augmented classification baseline that supports the design of ranking labels before classification."},{"cited_title":"Can Large Language Models Follow Concept Annotation Guidelines? A Case Study on Scientific and Financial Domains","cited_arxiv_id":"2311.08704","evidence_quote":"Supplies the label-schema integration strategy (label names plus descriptions) that RAC builds on."},{"cited_title":"Allwein, Robert E","cited_arxiv_id":null,"evidence_quote":"Provides the multiclass-to-binary reduction that RAC applies to turn classification into a sequence of binary decisions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the retrieval-augmented classification idea that RAC adapts to zero-shot LLM labelling."}],"review_version":1}