Automated Collection of Evaluation Dataset for Semantic Search in Low-Resource Domain Language

Anastasia Zhukova; Bela Gipp; Christian E. Matt

REVIEW 3 major objections 5 minor 24 references

Automated Collection of Evaluation Dataset for Semantic Search in Low-Resource Domain Language

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An ensemble of text encoders plus an LLM re-ranker improves automated relevance scoring for low-resource domain German.

desk verdict A sensible pipeline for low-resource test-collection building, but the headline gains are partly fitted to the same seven datasets and one dataset outright contradicts the 'all metrics' claim. read the letter →

arxiv 2412.10008 v1 pith:XNS4TKJJ submitted 2024-12-13 cs.CL

Anastasia Zhukova , Christian E. Matt , Bela Gipp This is my paper · ORCID

classification cs.CL

keywords semanticsearchensemblelearninglow-resourcelanguagedomain-specificGermanLLMannotationtestcollectionrelevancescoringre-ranking

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that an ensemble of off-the-shelf text encoders, combined with an LLM re-ranker, can build test collections for semantic search in a low-resource domain language without large-scale human annotation. Working in domain-specific German from the process industry, the authors generate queries with an LLM, retrieve candidate documents using an average of cosine-similarity scores across several encoders, and then re-rank each pair by merging the encoder score with an LLM relevance score. On seven plant shift-book datasets, the combined score agreed with a human annotator nearly four times better than either component alone, and it improved macro F1 by about 1.5 times. If the result holds, automated test-collection construction becomes practical for specialized languages where trained annotators and domain models are scarce.

What carries the argument

The load-bearing mechanism is a two-stage scoring pipeline: (1) retrieval by an ensemble of encoders, where each query and its LLM-generated paraphrases are embedded, cosine similarity is averaged over paraphrases per encoder and then over encoders (Eqs. 1–2), and the average is thresholded into a 1–3 relevance score (Eq. 3); and (2) an LLM re-ranker that independently assigns a 0–3 relevance score, after which a hand-derived fusion rule (Eqs. 4–5) combines the two scores, zeroing any pair the LLM deems irrelevant and weighting the more confident source otherwise. The fusion rule is the central object; it is what converts two biased scorers into a single score with higher agreement.

What would settle it

Take the same pipeline with the fixed fusion rule and apply it to a held-out set of queries and documents from a different plant or subdomain. If the combined score's Krippendorff's alpha drops to the level of, or below, the best single component, or if re-fitting the bins per dataset gives a large improvement, the reported gains are artifacts of the tuned rule rather than of the ensemble principle. A simpler check is to test whether the combined score still beats components when the threshold values in Eq. (3) and bins in Eq. (5) are perturbed; if performance is highly sensitive, the rule is overfit.

Watch

Extended reading notes

Core claim

The central discovery is that the disagreement between two imperfect scorers — an ensemble of 'weak' bi-encoders trained on common knowledge and a generative LLM prompted with domain examples — can be resolved into a single relevance score that tracks human judgment far better than either source alone. The paper attributes the gain to the complementarity of the two views: the encoders over-assign the lowest relevance class, the LLM over-assigns the highest, and the combination rule (Eqs. 4–5) weights each source most where the other is least reliable. Across 205 queries and 26,747 verified query-document pairs from seven datasets, the combined method raises average Krippendorff's alpha from roughly 11–27 (component baselines) to 40.5 and macro F1 from about 28–36 to 42.7.

Load-bearing premise

The fusion weights and thresholds in Eqs. (4)–(5) were derived by inspecting the score distribution of the seven datasets, and the paper assumes this hand-tuned rule will generalize to unseen queries and other datasets.

Editorial extensions

If this is right

Test collections for semantic search can be produced largely automatically in low-resource domain languages, with manual verification reduced to checking a smaller candidate set.
The same pipeline should transfer to other narrow domains and languages where at least a few German- or multilingual-supporting encoders and an LLM with domain prompting are available.
Providing specific worked examples in the LLM prompt (few-shot) matters more than the choice of vague instructions: it improves all metrics, especially detection of irrelevant pairs.
Using several diverse encoders rather than a single strong encoder is a viable route when no domain-specialized model exists.
The combined scoring method can serve as a pre-annotation step for human-in-the-loop annotation, cutting the cost of building evaluation data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The fusion rule and its thresholds are tuned to the observed score distribution; on a new domain the same rule might underperform, but a learned combination (e.g., a small regression on a handful of labeled pairs) could adapt the weights automatically.
If the method's success depends mainly on diversity between the encoder ensemble and the LLM, then swapping GPT-4o for a smaller open multilingual model may preserve much of the gain at lower cost — a testable prediction the paper leaves open.
The fourfold improvement in inter-coder agreement gives human annotators a much smaller set of genuinely ambiguous pairs to adjudicate, which could shift the bottleneck from annotation volume to disagreement resolution.
The reported gains are averages over seven datasets with wide variance (e.g., dataset E shows negative alpha for several methods), so the robust claim is about the aggregate, not every domain.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read it. The paper tackles a real bottleneck—building semantic-search test collections in low-resource domain languages—and proposes a pragmatic pipeline: an ensemble of off-the-shelf encoders plus GPT-4o re-ranking, with a hand-tuned fusion rule. The qualitative claim that combining diverse scorers helps is credible; the average gains across the seven datasets are consistent, and the paper is honest about the empirical derivation of its fusion rule. That said, the headline numbers are shakier than the prose admits.

The fusion rule (Eqs 4–5) was derived from the same experiments used to report the improvement, so the effect size is partly a fitting artifact. More concretely, Table 1 shows Source E where the combined method's Krippendorff's alpha is -24.91 versus 81.60 for the ensemble alone—a direct counterexample to the claim that the combined method 'outperformed the baselines in all metrics.' The pooled average masks a harmful failure on one dataset, and the paper does not analyze why. There is also only one human annotator and no confidence intervals, which limits the strength of any inter-coder agreement claim.

I would not call the paper unsound. The central direction is sensible, the evaluation is fairly thorough within its constraints, and the confusion matrices and per-class recall are useful. The authors are also transparent about limitations like GDPR restrictions and the lack of public code and prompts. But before this can be adopted as a benchmark-building tool, the overfitting concern needs to be addressed and per-dataset results reported transparently.

If the authors can add held-out validation or a clear development/test split, this could be a solid contribution. As it stands, it is a legitimate new application rather than a conceptual breakthrough. I'd bring it to a reading group as a good case study in evaluation-set construction and the pitfalls of tuning on your own test data.

Referee Report

3 major / 5 minor

Summary. The paper proposes an end-to-end pipeline for automatically creating evaluation test collections for semantic search in low-resource, domain-specific German (process-industry shift logs). The pipeline generates queries with GPT-4o, retrieves candidate documents using an ensemble of three text encoders, and re-ranks query-document pairs by combining encoder similarity scores with a GPT-4o relevance judgment via a hand-designed fusion rule (Eqs. 4–5). The method is evaluated on seven plant shift-book datasets by comparing automated relevance scores with those of a single human annotator, using Krippendorff's alpha, macro precision/recall/F1, and nDCG. The authors report that the combined method outperforms the individual baselines on average, with a nearly fourfold increase in Krippendorff's alpha and a 1.5-fold F1 improvement.

Significance. If the claimed gains are robust, the paper would make a useful practical contribution: a semi-automated pipeline for building domain-specific semantic-search test collections with reduced annotation effort, applied to a genuinely low-resource domain (German process-industry language). The paper's strengths include a concrete, real-world application, a clearly described pipeline, and a honest discussion of limitations. However, the central empirical claim currently rests on a fusion rule that is explicitly tuned on the same data used for evaluation, a failure on one of the seven sources is concealed by the pooled average, and the gold standard is a single human annotator. These issues prevent the current evidence from supporting a generalizable methodology claim.

major comments (3)

[Section 4.2, Eqs. (4)–(5), Fig. 3] The fusion rule and its thresholds are described as 'empirically derived from our experiments' and are chosen after inspecting the score distribution in Fig. 3. Because the same seven datasets are used both to set these thresholds/weights and to report the improvement of the combined method, the measured gain is at least partly a fitting artifact. The paper should provide a held-out evaluation (e.g., leave-one-source-out cross-validation) or pre-specified, non-adaptive thresholds to support the claim that the approach generalizes beyond the seven tuned datasets.
[Table 1, Source E; Section 4.1 (Results)] The text states that the combined approach 'outperformed the baselines in all metrics,' but Table 1 shows that on Source E the combined method yields Krippendorff's alpha of -24.91, far worse than the ensemble's 81.60 and essentially no better than GPT-4o-SE (-23.67). The pooled average alone (40.50 vs. 10.92) drives the 'nearly fourfold' improvement claim. The authors should report per-source results explicitly, discuss why the fusion rule fails so severely on Source E, and demonstrate that the average result is not an artifact of a few easy sources.
[Section 4.1 (Evaluation setup)] The gold-standard annotations come from a single human annotator (one native German speaker). The paper repeatedly refers to 'inter-coder agreement,' but with only one human rater the metric measures agreement between the automated system and that one person, not inter-coder reliability in the usual sense. There is no human–human agreement baseline, so it is unclear how much of the measured disagreement reflects system error versus annotator subjectivity. Additional annotators, or at least a clear discussion of this limitation, are needed before the reported alpha values can be interpreted as evidence of reliable annotation.

minor comments (5)

[Section 4.1 (Metrics)] The text says Krippendorff's alpha 'yields a value between 0 and 1,' but Table 1 contains negative values (e.g., -24.91). Please correct this to note that alpha can be negative when agreement is worse than chance.
[Eq. (3)] The cosine-similarity thresholds (0.5, 0.6, 0.7) are introduced without justification. Please state whether they are standard practice, are derived from preliminary experiments, or are arbitrary, and discuss their impact on results.
[Section 4.1 (Experimental setup)] The selection of 28–30 queries per source and 'up to 1000 relevant documents' is described only briefly. It is unclear whether this selection was based on system scores or human relevance judgments; please clarify to rule out selection bias.
[References] The reference to 'Liu and Zsu (2009)' appears to be a misspelling of M. Tamer Özsu (Encyclopedia of Database Systems). Please correct.
[Section 7 (Ethic considerations)] The paper states that code and datasets that can be shared are linked, but no URLs or repository names appear in the text. Please add concrete links or a footnote where the resources are described.

Circularity Check

1 steps flagged · score 6.0 of 10

Empirically derived fusion rule in Eqs. 4-5 is fitted on the same seven datasets used for Table 1; the reported fourfold agreement gain is in-sample, and Source E shows the fit fails.

fitted input called prediction [Section 4.2, Eqs. (4)-(5); Figure 3; Table 1]
"Figure 3 shows that most scores were either annotated by the ensemble as 1 or by the GPT-4o as 3. Hence, when computing the combined score, we give more weight to the GPT scores when the score is 3 or to the ensemble scores when it is 1; otherwise, we compute their average. Moreover, GPT-4o tends to re-rank the fourth of the ensemble-positive scores as 0. Therefore, we keep the re-ranking score of GPT-4o. The bins(x) function was empirically derived from our experiments."

The fusion weights and bin thresholds are chosen after inspecting Figure 3, which is built from the same seven datasets whose manual labels then appear in Table 1. Since Eq. (4) is 'empirically derived from our experiments,' the reported 4x alpha gain is an in-sample fit, not a prediction on new data. Source E shows the rule fails badly (combined alpha -24.91 vs ensemble 81.60), so the pooled average is not evidence of generalization.

full rationale

The central claim—that combining encoder ensemble scores with GPT-4o reranking improves agreement with human labels—is supported by Table 1, but the fusion rule itself is explicitly 'empirically derived from our experiments' (Eqs. 4-5, Section 4.2), using the same seven datasets and the same score distribution shown in Figure 3. Therefore the reported 4x improvement in Krippendorff's alpha is an in-sample measure of the hand-tuned rule rather than an out-of-sample prediction. Source E illustrates the risk: the combined rule gives alpha -24.91 where the ensemble alone gives 81.60. This is a partial circularity (test-set tuning), not a definitional collapse: the manual annotations are external ground truth, the encoder ensemble is evaluated separately, and no load-bearing self-citation is used. Score 6 reflects that the headline quantitative gain is partly a fitting artifact without held-out validation or per-dataset analysis.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several hand-set thresholds and an empirically derived fusion rule, plus the assumption that a single human annotator provides reliable ground truth. No new physical or conceptual entities are introduced.

free parameters (5)

cosine similarity thresholds in Eq 3 = 0.5, 0.6, 0.7
Chosen by hand to map cosine similarity to relevance scores 1-3; no principled basis is given.
bins thresholds in Eq 5 = 1.0, 2.0, 2.6
Admitted to be empirically derived from the experiments (Section 4.2), so the fusion rule is tuned to the test data.
combination weights in Eq 4 = 2/3, 1/3, 1/2 depending on branch
Chosen based on the observed score distribution in Figure 3; no independent justification.
minimum document length for query generation = 100 chars (single query), 300 chars (multiple queries)
Hand-set thresholds that determine which documents become queries and how many queries are generated.
retrieval threshold conditions = cos.simd >= 0.5 and at least two relevant docs per query
Hand-set criteria that decide which documents are retrieved for annotation; affects the evaluation set composition.

assumptions (4)

domain assumption Cosine similarity to a 1-3 relevance scale is a valid proxy for query-document relevance in this domain.
The thresholds in Eq 3 are asserted without external validation; the relevance labels ultimately come from one human annotator.
domain assumption A single human annotator's relevance scores are a reliable ground truth for evaluation.
Inter-coder agreement is computed between the automated system and one annotator; there is no human-human agreement estimate or multiple coders.
domain assumption The relevance scores from encoders and the LLM are commensurable and can be combined arithmetically.
Eq 4 averages scores across different scoring mechanisms without calibration, assuming they live on the same 0-3 scale.
ad hoc to paper The score distribution observed in Figure 3 is stable across domains, justifying the weights in Eq 4.
The weighting scheme is empirically derived from the experiments and is not shown to transfer to other languages or datasets.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Automated Collection of Evaluation Dataset for Semantic Search in Low-Resource Domain Language." pith.science (2026). https://pith.science/paper/XNS4TKJJ

@misc{pith2026241210008,
  author       = {Pith},
  title        = {Pith review of: Automated Collection of Evaluation Dataset for Semantic Search in Low-Resource Domain Language},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XNS4TKJJ}},
  note         = {Machine review of arXiv:2412.10008}
}

read the original abstract

Domain-specific languages that use a lot of specific terminology often fall into the category of low-resource languages. Collecting test datasets in a narrow domain is time-consuming and requires skilled human resources with domain knowledge and training for the annotation task. This study addresses the challenge of automated collecting test datasets to evaluate semantic search in low-resource domain-specific German language of the process industry. Our approach proposes an end-to-end annotation pipeline for automated query generation to the score reassessment of query-document pairs. To overcome the lack of text encoders trained in the German chemistry domain, we explore a principle of an ensemble of "weak" text encoders trained on common knowledge datasets. We combine individual relevance scores from diverse models to retrieve document candidates and relevance scores generated by an LLM, aiming to achieve consensus on query-document alignment. Evaluation results demonstrate that the ensemble method significantly improves alignment with human-assigned relevance scores, outperforming individual models in both inter-coder agreement and accuracy metrics. These findings suggest that ensemble learning can effectively adapt semantic search systems for specialized, low-resource languages, offering a practical solution to resource limitations in domain-specific contexts.

Figures

Figures reproduced from arXiv: 2412.10008 by the authors.

**Figure 2.** A proposed methodology with ensembles of (L)LM encoders used to retrieve the most relevant documents, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** The distribution of the relevance scores pro [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: The confusion matrices of the annotated vs. automated relevance scores for four methods: an ensemble of [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 7 canonical work pages

[1]

Aguda, Suchetha Siddagangappa, Elena Kochkina, Simerjot Kaur, Dongsheng Wang, and Charese Smiley

Toyin D. Aguda, Suchetha Siddagangappa, Elena Kochkina, Simerjot Kaur, Dongsheng Wang, and Charese Smiley. 2024. https://aclanthology.org/2024.lrec-main.885 Large language models as financial data annotators: A study on effectiveness and efficiency . In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources...

2024
[2]

Jonas Becker. 2024. https://arxiv.org/abs/2410.22932 Multi-agent large language models for conversational task-solving . Preprint, arXiv:2410.22932

arXiv 2024
[3]

Chenhui Chu and Rui Wang. 2018. https://aclanthology.org/C18-1111 A survey of domain adaptation for neural machine translation . In Proceedings of the 27th International Conference on Computational Linguistics, pages 1304--1319, Santa Fe, New Mexico, USA. Association for Computational Linguistics

work page 2018
[4]

David Farr, Nico Manzonelli, Iain Cruickshank, Kate Starbird, and Jevin West. 2024. https://arxiv.org/abs/2410.13006 Llm chain ensembles for scalable and accurate data annotation . Preprint, arXiv:2410.13006

arXiv 2024
[5]

Hedderich, Lukas Lange, Heike Adel, Jannik Str \"o tgen, and Dietrich Klakow

Michael A. Hedderich, Lukas Lange, Heike Adel, Jannik Str \"o tgen, and Dietrich Klakow. 2021. https://doi.org/10.18653/v1/2021.naacl-main.201 A survey on recent approaches for natural language processing in low-resource scenarios . In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human ...

doi:10.18653/v1/2021.naacl-main.201 2021
[6]

Krippendorff

K. Krippendorff. 2013. https://books.google.de/books?id=s_yqFXnGgjQC Content Analysis: An Introduction to Its Methodology . SAGE Publications

work page 2013
[7]

Bhawesh Kumar, Jonathan Amar, Eric Yang, Nan Li, and Yugang Jia. 2024. https://arxiv.org/abs/2405.06093 Selective fine-tuning on llm-labeled data may reduce reliance on human annotation: A case study using schedule-of-event table detection . Preprint, arXiv:2405.06093

arXiv 2024
[8]

Minzhi Li, Taiwei Shi, Caleb Ziems, Min-Yen Kan, Nancy Chen, Zhengyuan Liu, and Diyi Yang. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.92 C o A nnotating: Uncertainty-guided work allocation between human and large language models for data annotation . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14...

doi:10.18653/v1/2023.emnlp-main.92 2023

Show all 24 references

[9]

Tamer Zsu

Ling Liu and M. Tamer Zsu. 2009. Encyclopedia of Database Systems, 1st edition. Springer Publishing Company, Incorporated

2009
[10]

Yuxuan Lu, Bingsheng Yao, Shao Zhang, Yun Wang, Peng Zhang, Tun Lu, Toby Jia-Jun Li, and Dakuo Wang. 2023. https://arxiv.org/abs/2311.09825 Human still wins over llm: An empirical study of active learning on domain-specific annotation tasks . Preprint, arXiv:2311.09825

2023 arXiv
[11]

Ibomoiye Domor Mienye and Yanxia Sun. 2022. https://doi.org/10.1109/ACCESS.2022.3207287 A survey of ensemble learning: Concepts, algorithms, applications, and prospects . IEEE Access, 10:99129--99149

2022
[12]

Nicholas Pangakis, Samuel Wolken, and Neil Fasching. 2023. https://arxiv.org/abs/2306.00176 Automated annotation with generative ai requires validation . Preprint, arXiv:2306.00176

2023 arXiv
[13]

Yu Shang, Yu Li, Fengli Xu, and Yong Li. 2024. https://arxiv.org/abs/2402.02563 Synergy-of-thoughts: Eliciting efficient reasoning in hybrid language models . Preprint, arXiv:2402.02563

2024 arXiv
[14]

AAAI Press Staff, Pater Patel Schneider, Sunil Issar, J Scott Penberthy, George Ferguson, Hans Guesgen, Francisco Cruz, and Marc Pujol-Gonzalez . 2023. https://doi.org/10.36190/2023.15 From humans to machines: Can chatgpt-like llms effectively replace human annotators in nlp t...

2023 doi
[15]

Mirac Suzgun and Adam Tauman Kalai. 2024. https://arxiv.org/abs/2401.12954 Meta-prompting: Enhancing language models with task-agnostic scaffolding . Preprint, arXiv:2401.12954

2024 arXiv
[16]

Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. 2024. https://arxiv.org/abs/2402.13446 Large language models for data annotation: A survey . Preprint, arXiv:2402.13446

2024 arXiv
[17]

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672

2024 arXiv
[18]

Yang, Damian Dailisan, Marcin Korecki, Carina I

Joshua C. Yang, Damian Dailisan, Marcin Korecki, Carina I. Hausladen, and Dirk Helbing. 2024. https://ojs.aaai.org/index.php/AIES/article/view/31758 Llm voting: Human choices and ai collective decision-making . Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society,...

2024
[19]

Zhangyue Yin, Qiushi Sun, Cheng Chang, Qipeng Guo, Junqi Dai, Xuanjing Huang, and Xipeng Qiu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.936 Exchange-of-thought: Enhancing large language model capabilities through cross-model communication . In Proceedings of the 2023 C...

2023 doi
[20]

Yiming Zhu, Peixian Zhang, Ehsan-Ul Haq, Pan Hui, and Gareth Tyson. 2023. https://arxiv.org/abs/2304.10145 Can chatgpt reproduce human-generated labels? a study of social computing tasks . Preprint, arXiv:2304.10145

2023 arXiv
[21]

Anastasia Zhukova, Felix Hamborg, and Bela Gipp. 2021. ANEA : Automated (named) entity annotation for German domain-specific texts. In Proceedings of the 2nd Workshop on Extraction and Evaluation of Knowledge Entities from Scientific Documents (EEKE 2021) co-located with JCDL ...

2021
[22]

Matt, and Bela Gipp

Anastasia Zhukova, Lukas von Sperl, Christian E. Matt, and Bela Gipp. 2024. https://doi.org/10.1007/s10115-024-02212-5 Generative user-experience research for developing domain-specific natural language processing applications . Knowledge and Information Systems, 66:7859–7889

2024 doi
[23]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[24]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Aguda, Suchetha Siddagangappa, Elena Kochkina, Simerjot Kaur, Dongsheng Wang, and Charese Smiley

Toyin D. Aguda, Suchetha Siddagangappa, Elena Kochkina, Simerjot Kaur, Dongsheng Wang, and Charese Smiley. 2024. https://aclanthology.org/2024.lrec-main.885 Large language models as financial data annotators: A study on effectiveness and efficiency . In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources...

2024

[2] [2]

Jonas Becker. 2024. https://arxiv.org/abs/2410.22932 Multi-agent large language models for conversational task-solving . Preprint, arXiv:2410.22932

arXiv 2024

[3] [3]

Chenhui Chu and Rui Wang. 2018. https://aclanthology.org/C18-1111 A survey of domain adaptation for neural machine translation . In Proceedings of the 27th International Conference on Computational Linguistics, pages 1304--1319, Santa Fe, New Mexico, USA. Association for Computational Linguistics

work page 2018

[4] [4]

David Farr, Nico Manzonelli, Iain Cruickshank, Kate Starbird, and Jevin West. 2024. https://arxiv.org/abs/2410.13006 Llm chain ensembles for scalable and accurate data annotation . Preprint, arXiv:2410.13006

arXiv 2024

[5] [5]

Hedderich, Lukas Lange, Heike Adel, Jannik Str \"o tgen, and Dietrich Klakow

Michael A. Hedderich, Lukas Lange, Heike Adel, Jannik Str \"o tgen, and Dietrich Klakow. 2021. https://doi.org/10.18653/v1/2021.naacl-main.201 A survey on recent approaches for natural language processing in low-resource scenarios . In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human ...

doi:10.18653/v1/2021.naacl-main.201 2021

[6] [6]

Krippendorff

K. Krippendorff. 2013. https://books.google.de/books?id=s_yqFXnGgjQC Content Analysis: An Introduction to Its Methodology . SAGE Publications

work page 2013

[7] [7]

Bhawesh Kumar, Jonathan Amar, Eric Yang, Nan Li, and Yugang Jia. 2024. https://arxiv.org/abs/2405.06093 Selective fine-tuning on llm-labeled data may reduce reliance on human annotation: A case study using schedule-of-event table detection . Preprint, arXiv:2405.06093

arXiv 2024

[8] [8]

Minzhi Li, Taiwei Shi, Caleb Ziems, Min-Yen Kan, Nancy Chen, Zhengyuan Liu, and Diyi Yang. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.92 C o A nnotating: Uncertainty-guided work allocation between human and large language models for data annotation . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14...

doi:10.18653/v1/2023.emnlp-main.92 2023