Pith. sign in

REVIEW 4 major objections 7 minor 1 cited by

Automatic Labelling with Open-source LLMs using Dynamic Label Schema Integration

T0 review · 4 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Retrieval-ordered binary checks lift labeling F1 from 4 to 68 on Banking77.

desk verdict A useful applied method with a plausible central claim, but the truncation trade-off is not actually measured because recall@5 is missing. read the letter →

arxiv 2501.12332 v1 pith:UJRYQESL submitted 2025-01-21 cs.CL cs.AIcs.LG

Thomas Walshe , Sae Young Moon , Chunyang Xiao , Yawwani Gunawardana , Fran Silavong This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords automaticlabellinglargelanguagemodelslabelschemaretrievalaugmentedclassificationzero-shotopen-sourceLLMcoveragetradeoffintent
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that open-source 7B language models can label text accurately if the classification problem is restructured: instead of asking the model to pick from all classes at once, retrieve the most semantically related class descriptions, then ask the model a yes/no question for each candidate in order, stopping at the first yes. The authors show that providing both label names and descriptions systematically helps, but only when classes are presented one at a time; presenting all 77 classes at once collapses to 4.0 macro-F1 on Banking77, while their full pipeline reaches 68.3 and a truncated version that only probes the top-ranked labels reaches 73.4 at 88.7% coverage. This matters because it makes privacy-preserving, in-house labeling with open models practical for high-cardinality business taxonomies.

What carries the argument

Retrieval Augmented Classification (RAC): each class is embedded as a label name plus a ~50-word description; at inference time the text to label is embedded and the classes are sorted by cosine similarity. The LLM then runs binary prompts ('does this sample belong to class X?') in that order, accepting the first class that returns True. Truncated RAC only probes the top n classes and abstains if none are accepted, turning the retriever's ranking into a precision/coverage dial.

What would settle it

Compute the retriever's recall@n (the fraction of examples whose true class appears in the top-n retrieved candidates) on a dataset; if that recall is materially below Truncated RAC's reported coverage on the same classes, then the LLM's abstention rate is set by retrieval misses rather than by the model's confidence, and improving the retriever alone should raise F1.

Watch

Extended reading notes

Core claim

The central claim is that RAC, which dynamically integrates label schema by ranking label descriptions and performing binary LLM classification one label at a time, improves labelling F1 over a single all-label prompt, and that Truncated RAC trades coverage for higher F1. The paper reports that Full RAC (L+D) reaches 68.3 macro-F1 on Banking77 while Truncated RAC reaches 73.4 macro-F1 at 88.7% coverage, against a 4.0 F1 all-info-in-prompt baseline.

Load-bearing premise

The semantic retriever must place the true label inside the top few candidates for most examples, because Truncated RAC never offers labels beyond the top n; the paper's own MRR@1 averages 0.691, so for roughly a third of examples the correct label is not first, and for some it may not appear at all.

Editorial extensions

If this is right

  • Label descriptions are worth generating even when not provided, since they improve both retrieval ranking and binary classification in every tested configuration.
  • Truncated RAC can label the majority of a dataset (>70%) with substantially higher accuracy than Full RAC, making abstention a viable route to quality.
  • The quality-coverage knob lets practitioners set the F1/coverage tradeoff per business need, as demonstrated on a 61-class internal banking dataset where micro-F1 ranges from 61.5 to 82.2 depending on truncation and self-consistency.
  • Labels produced by Truncated RAC can be distilled into a small classifier that generalizes beyond the labeled subset even with noisy, partial training data, approaching Truncated RAC's performance on several public benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the retriever is an off-the-shelf sentence transformer, swapping in a stronger retriever or fine-tuning it on label descriptions should directly raise the ceiling of Truncated RAC, since the LLM never sees classes beyond the top n.
  • The paper's abstention mechanism can serve as a cheap confidence signal: coverage itself is a proxy for confidence, so deployments could set a coverage threshold to match business tolerance for unlabeled data.
  • A natural extension, which the authors note in passing, is to return all accepted labels within the n probes, turning RAC into a multi-label classifier; a testable question is whether this variant maintains precision when label co-occurrence is high.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper studies zero-shot automatic labelling with open-source 7B-scale LLMs under privacy and cost constraints. The authors observe that providing label names and descriptions improves binary classification but that a single multi-class prompt with all label schema degrades on high-cardinality tasks (e.g., 4.0 F1 on Banking77). They propose Retrieval Augmented Classification (RAC): label descriptions are embedded offline, a retriever ranks labels per example, and the LLM performs a series of binary classifications over the ranked list, stopping when it accepts a label. A truncated variant stops after n rejections and abstains. The paper evaluates RAC on AGNews, DBpedia, Amazon, and Banking77, reporting that label descriptions improve F1 in both Full and Truncated RAC, that Truncated RAC improves F1 at reduced coverage, and that label distillation on truncated labels is competitive on low-cardinality datasets. They also apply the method to an internal 61-class consumer banking dataset and show that combining top-1 retrieval with multi-inference self-consistency reaches up to 74.7 macro-F1 at 35.5% coverage.

Significance. If the empirical claims hold, the paper makes a useful practical contribution: it shows that a simple retrieval-augmented binary decomposition can make open-source 7B models usable for in-house labelling, and it documents real latency/coverage trade-offs. The paper is commendable for evaluating on multiple public datasets, reporting latency, including an industry case study, and testing label distillation. The central claims are nevertheless under-supported in several places: the retriever ceiling for truncated RAC is not measured, the all-info baseline is absent from the main table, and design choices are selected on the test sets without uncertainty quantification. These issues are fixable with additional analysis and experiments.

major comments (4)
  1. [Section 4.1/4.2, Tables 2 and 3] The paper's central trade-off claim for Truncated RAC is not supported without per-dataset recall@n. Truncated RAC only offers the top n labels, so the retriever's recall@n is an upper bound on the fraction of examples for which correct labelling is possible; the paper reports only an averaged MRR@1 (0.691 for L+D, Table 2) and a statement that other n 'show consistent trends'. Because F1 is computed only on labelled examples (Section 4.1), the improved F1 of Truncated over Full RAC in Table 3 (e.g., Banking77 73.4 vs 68.3 macro) could simply reflect a selection effect in which abstention removes examples whose true label is absent from the candidate list, rather than an improvement in label quality. Please report recall@n per dataset, an oracle ceiling for Truncated RAC (assuming the LLM always accepts the true label when it is in the top n), and a comparison of Full RAC F1 computed on the same subset of examples that Truncated RAC labels.
  2. [Section 3 and Table 4] The abstract and introduction claim that naively providing all label descriptions in a single prompt degrades performance on high-cardinality tasks, motivating RAC. However, the all-info-in-prompt baseline is reported only for Banking77 (Table 4); Table 3 does not include this baseline for AGNews, DBpedia, or Amazon. Without these numbers, the claimed improvement of RAC over the naive single-prompt approach is not established across the public benchmark suite. Please add the all-info baseline to Table 3 or provide the numbers for all datasets.
  3. [Sections 2.3 and 4] The paper tunes the LLM (Mistral vs Llama), prompting style (CoT vs no-CoT), self-consistency strategy and inference count, and the truncation steps n (2 or 5) using the same public test splits on which the final results are reported. This selection process can inflate the reported F1 numbers and makes the cross-dataset consistency claims fragile. Please either use a separate validation split for all design choices or provide bootstrap confidence intervals and significance tests for the key comparisons in Tables 3 and 4.
  4. [Section 2.1 and Appendix C] For the public datasets, label descriptions are generated by Mistral-7B from an unspecified 'small quantity' of training examples and then used by Mistral-7B for classification. Footnote 2 acknowledges a possible bias but no experiment quantifies it. Since the L+D improvement is a central message, please include at least one comparison with non-Mistral-generated descriptions (e.g., human-written or GPT-4-written) on a public dataset. In addition, specify the number of training examples per class used for description generation, and the number of self-consistency inferences used in the main RAC experiments (Section 2.3 mentions majority voting but does not state the vote count for Table 3).
minor comments (7)
  1. [Global] Throughout the manuscript, 'COT' and 'CoT' are used inconsistently; please standardize to 'CoT'.
  2. [Section 2.3] The sentence 'When we use LLM to make a binary classification, we only output a prediction when no-COT and COT approaches output the same label' is ambiguous; please clarify whether the main RAC experiments require agreement between the two prompting variants or use a single variant.
  3. [Appendix B, Table 8] The row and column headers are unclear, in particular what 'BS' and 'Both' denote; please restructure the table.
  4. [Section 4.3.2] Please report the number or fraction of training examples used for label distillation, since the coverage of the truncated labelling on the training split is not stated.
  5. [Section 2.1] The number of training examples used to generate each label description is not given; please specify this quantity, as it is a free parameter that affects description quality.
  6. [Table 2] Reporting only the MRR@1 averaged over datasets obscures large differences in cardinality; please include per-dataset MRR values.
  7. [General] The source code and configuration details are not provided; a reproducibility statement or a public code release would greatly help readers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical measurements, not derived quantities, and no load-bearing self-citation or definitional equivalence is present.

full rationale

The paper's core claims — that RAC improves labelling F1 over a single all-label prompt, and that Truncated RAC trades coverage for higher F1 — are reported as measurements on test splits, not as quantities derived from fitted parameters. No parameter is fitted to a subset of data and then renamed as a prediction. The F1 and MRR numbers in Tables 2-5 are direct evaluations. The preliminary selection of Mistral-7B over Llama-7B and of majority voting is a model-selection choice on the same benchmarks, which raises a data-snooping concern but is not circularity: the reported results are not forced by construction. The most relevant self-referential point is acknowledged by the authors in Section 2.3, footnote 2: 'The fact that we generate label descriptions from Mistral 7B might bias this comparison, for which we leave the thorough investigation for future work.' This is a genuine confound — the label descriptions used for public benchmarks are generated by the same model family that is then evaluated — but it is not a circular reduction: the descriptions are produced from training examples, not from the labels being predicted, and the internal dataset uses SME-written descriptions (Section 5), providing an independent grounding for RAC's usefulness. The Truncated RAC F1/coverage tradeoff is also a measurement, though F1 is computed only on examples the model chooses to label, so abstention on hard examples could inflate F1; the paper does not report recall@5, which is an evidence gap for the tradeoff claim, not circularity. Finally, the method builds on explicitly cited prior work ([1] for binary decomposition, [20, 12] for retrieval-augmented classification) rather than on a self-citation chain or an imported uniqueness theorem. Overall, the derivation chain is self-contained as an empirical study, and no step reduces to its own inputs by definition.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim is empirical, so the ledger is dominated by assumptions about the retriever and LLM behavior rather than free parameters. The main free parameters are the truncation step and self-consistency counts, which are dataset-specific and tuned on the same test sets. No new entities are introduced.

free parameters (4)
  • truncation step count n = 2 for AGNews, 5 for others
    Chosen per dataset to balance label quality and coverage; results in Tables 3 and 5 depend on this choice.
  • self-consistency inference count = 1, 3, or 5
    Number of repeated inferences that must agree before a label is accepted; used on internal dataset (Table 5).
  • retrieval embedding model = all-mpnet-base-v2
    Off-the-shelf sentence transformer used without fine-tuning; retrieval quality directly controls which labels the LLM sees.
  • number of training examples for description generation = not specified (small quantity)
    Public dataset label descriptions are generated by Mistral-7B from a small number of training examples; the amount is not stated and may affect description quality.
assumptions (3)
  • domain assumption Semantic similarity between query and label description ranks the true label sufficiently high.
    RAC only offers labels in retrieval order, so the true label must appear in the candidate list for correct labelling. Reported MRR@1 is 0.691 on average, so this assumption fails for a meaningful fraction of examples.
  • domain assumption The LLM's binary accept/reject decisions are reliable and early stopping does not systematically reject the correct label while accepting wrong ones.
    The iterative procedure assigns the first label the LLM accepts; over-acceptance of broad categories is acknowledged in the error analysis (Section 4.2).
  • standard math Macro-F1 computed only on labelled samples is a valid measure of label quality for comparing configurations with different coverage.
    Truncated RAC excludes hard examples, which inflates F1; the paper does not correct for this selection effect.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automatic Labelling with Open-source LLMs using Dynamic Label Schema Integration." pith.science (2026). https://pith.science/paper/UJRYQESL

@misc{pith2026250112332,
  author       = {Pith},
  title        = {Pith review of: Automatic Labelling with Open-source LLMs using Dynamic Label Schema Integration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UJRYQESL}},
  note         = {Machine review of arXiv:2501.12332}
}
read the original abstract

Acquiring labelled training data remains a costly task in real world machine learning projects to meet quantity and quality requirements. Recently Large Language Models (LLMs), notably GPT-4, have shown great promises in labelling data with high accuracy. However, privacy and cost concerns prevent the ubiquitous use of GPT-4. In this work, we explore effectively leveraging open-source models for automatic labelling. We identify integrating label schema as a promising technology but found that naively using the label description for classification leads to poor performance on high cardinality tasks. To address this, we propose Retrieval Augmented Classification (RAC) for which LLM performs inferences for one label at a time using corresponding label schema; we start with the most related label and iterates until a label is chosen by the LLM. We show that our method, which dynamically integrates label description, leads to performance improvements in labelling tasks. We further show that by focusing only on the most promising labels, RAC can trade off between label quality and coverage - a property we leverage to automatically label our internal datasets.

Figures

Figures reproduced from arXiv: 2501.12332 by the authors.

Figure 1
Figure 1. Retrieval Augmented Classification (RAC) during inference: the (bottom) retrieval part retrieved most [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Prompt used to generate a category label using the text from samples belonging to the same class. Prompt [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Prompt used to generate a category description using the text from samples belonging to the same class. [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Prompt used to classify a sample against a category described by the label and description using No-COT [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Prompt used to classify a sample against a category described by the label and description using COT [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. BELLE: A Bi-Level Multi-Agent Reasoning Framework for Multi-Hop Question Answering

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A question-type-aware, bi-level multi-agent debate that selects and combines existing QA operators outperforms fixed single-method baselines on four multi-hop benchmarks.

Reference graph

Works this paper leans on

46 extracted references · 24 canonical work pages · cited by 1 Pith paper

  1. [1]

    Allwein, Robert E

    Erin L. Allwein, Robert E. Schapire, and Yoram Singer. 2000. Reducing multiclass to binary: A uni- fying approach for margin classifiers. In Proc. 17th International Conf. on Machine Learning, pages 9–

    work page 2000

  2. [2]

    Simran Arora, Avanika Narayan, Mayee F Chen, Laurel Orr, Neel Guha, Kush Bhatia, Ines Chami, Frederic Sala, and Christopher Ré. 2022. Ask me anything: A simple strategy for prompting language models. arXiv preprint arXiv:2210.02441

    arXiv 2022

  3. [3]

    Emanuele Bastianelli, Andrea Vanzo, Pawel Swieto- janski, and Verena Rieser. 2020. Slurp: A spoken language understanding resource package. arXiv preprint arXiv:2011.13205

    arXiv 2020

  4. [4]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901

    work page 2020

  5. [5]

    Iñigo Casanueva, Tadas Tem ˇcinas, Daniela Gerz, Matthew Henderson, and Ivan Vuli´c. 2020. Efficient intent detection with dual sentence encoders. arXiv preprint arXiv:2003.04807

    arXiv 2020

  6. [6]

    Aakanksha Chowdhery, Sharan Narang, Jacob De- vlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311

    arXiv 2022

  7. [7]

    Elizamary de Souza Nascimento, Iftekhar Ahmed, Edson Oliveira, Márcio Piedade Palheta, Igor Stein- macher, and Tayana Conte. 2019. Understanding development process of machine learning systems: Challenges and solutions. In 2019 acm/ieee interna- tional symposium on empirical software engineering and measurement (esem), pages 1–6. IEEE

    work page 2019

  8. [8]

    Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, et al. 2022. Massive: A 1m-example multilin- gual natural language understanding dataset with 51 typologically-diverse languages. arXiv preprint arXiv:2204.08582

    arXiv 2022

Show all 46 references
  1. [9]

    Marcio Fonseca and Shay B Cohen. 2023. Can large language models follow concept annotation guide- lines? a case study on scientific and financial do- mains. arXiv preprint arXiv:2311.08704

    2023 arXiv

  2. [10]

    Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence em- beddings. arXiv preprint arXiv:2104.08821

    2021 arXiv

  3. [11]

    Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd- workers for text-annotation tasks. arXiv preprint arXiv:2303.15056

    2023 arXiv

  4. [12]

    Ahmet Iscen, Alireza Fathi, and Cordelia Schmid

  5. [13]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al

  6. [14]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199– 22213

    2022

  7. [15]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gon- zalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serv- ing with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Pri...

    2023

  8. [16]

    Morgan Kaufmann, San Francisco, CA

  9. [17]

    arXiv preprint arXiv:2310.06825

    Mistral 7b. arXiv preprint arXiv:2310.06825

    arXiv

  10. [18]

    Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Na- man Goyal, Heinrich Küttler, Mike Lewis, Wen- tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented gener- ation for knowledge-intensive NLP tasks. CoRR...

    2020 arXiv

  11. [19]

    Dong-Ho Lee, Jay Pujara, Mohit Sewak, Ryen W White, and Sujay Kumar Jauhar. 2023. Making large language models better data creators. arXiv preprint arXiv:2310.20111

    2023 arXiv

  12. [20]

    Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, et al. 2015. Dbpedia–a large- scale, multilingual knowledge base extracted from wikipedia. Semantic web, 6(2):167–195

    2015

  13. [21]

    Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han

  14. [22]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. Lost in the middle: How lan- guage models use long contexts

    2023

  15. [23]

    Alexander Long, Wei Yin, Thalaiyasingam Ajan- than, Vu Nguyen, Pulak Purkait, Ravi Garg, Alan Blair, Chunhua Shen, and Anton van den Hengel

  16. [24]

    OpenAI. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    2023 arXiv

  17. [25]

    Neoklis Polyzotis, Sudip Roy, Steven Euijong Whang, and Martin Zinkevich. 2018. Data lifecycle challenges in production machine learning: a survey. ACM SIGMOD Record, 47(2):17–28

    2018

  18. [26]

    Ad- vances in Neural Information Processing Systems , 35:462–477

    Generating training data with language mod- els: Towards zero-shot language understanding. Ad- vances in Neural Information Processing Systems , 35:462–477

  19. [27]

    Anders Giovanni Møller, Jacob Aarup Dalsgaard, Arianna Pera, and Luca Maria Aiello. 2023. Is a prompt and a few samples all you need? using gpt-4 for data augmentation in low-resource classification tasks. arXiv preprint arXiv:2304.13861

    2023 arXiv

  20. [28]

    Ranjita Naik, Varun Chandrasekaran, Mert Yuk- sekgonul, Hamid Palangi, and Besmira Nushi

  21. [29]

    arXiv preprint arXiv:2310.07088

    Diversity of thought improves reasoning abilities of large language models. arXiv preprint arXiv:2310.07088

    arXiv

  22. [30]

    Alexander Sorokin and David Forsyth. 2008. Utility data annotation with amazon mechanical turk. In 2008 IEEE computer society conference on computer vision and pattern recognition workshops, pages 1–8. IEEE

    2008

  23. [31]

    Xiaofei Sun, Xiaoya Li, Jiwei Li, Fei Wu, Shangwei Guo, Tianwei Zhang, and Guoyin Wang. 2023. Text classification via large language models

    2023

  24. [32]

    Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446

    2021 arXiv

  25. [33]

    Alexander Ratner, Stephen H Bach, Henry Ehren- berg, Jason Fries, Sen Wu, and Christopher Ré. 2017. Snorkel: Rapid training data creation with weak su- pervision. In Proceedings of the VLDB Endowment. International Conference on Very Large Data Bases, volume 11.3, page 269. NI...

    2017

  26. [34]

    Timo Schick and Hinrich Schütze. 2020. It’s not just size that matters: Small language models are also few-shot learners. arXiv preprint arXiv:2009.07118

    2020 arXiv

  27. [35]

    Chirag Shah, Ryen W White, Reid Andersen, Georg Buscher, Scott Counts, Sarkar Snigdha Sarathi Das, Ali Montazer, Sathish Manivannan, Jennifer Neville, Xiaochuan Ni, et al. 2023. Using large language models to generate, validate, and apply user intent taxonomies. arXiv preprint...

    2023 arXiv

  28. [36]

    Ruida Wang, Wangchunshu Zhou, and Mrinmaya Sachan. 2023. Let’s synthesize step by step: Iterative dataset synthesis with large language models by ex- trapolating errors from small models. arXiv preprint arXiv:2310.13671

    2023 arXiv

  29. [37]

    Shuohang Wang, Yang Liu, Yichong Xu, Chen- guang Zhu, and Michael Zeng. 2021. Want to reduce labeling cost? GPT-3 can help. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4195–4205, Punta Cana, Dominican Re- public. Association for Computationa...

    2021

  30. [38]

    Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239

    2022 arXiv

  31. [39]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Ham- bro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  32. [40]

    Tina Tseng, Amanda Stent, and Domenic Maida

  33. [42]

    Lewis Tunstall, Nils Reimers, Unso Eun Seo Jo, Luke Bates, Daniel Korat, Moshe Wasserblat, and Oren Pereg. 2022. Efficient few-shot learning with- out prompts. arXiv preprint arXiv:2209.11055

    2022 arXiv

  34. [45]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models

    2023

  35. [46]

    explain your answer step-by-step

    Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classi- fication. Advances in neural information processing systems, 28. A Llama 2 7B preliminary results L L+D No CoT 71.7 72.3 CoT 58.4 58.6 Table 7: Macro-averaged zero-shot class...

    2015

  36. [2020]

    arXiv preprint arXiv:2009.11654

    Best practices for managing data annotation projects. arXiv preprint arXiv:2009.11654

    2009 arXiv

  37. [2022]

    In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pages 6959–6969

    Retrieval augmented classification for long-tail visual recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pages 6959–6969

  38. [2023]

    In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19295–19304

    Improving image recognition by retrieving from web-scale image-text data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19295–19304

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.