DNB-AI-Project at SemEval-2025 Task 5: An LLM-Ensemble Approach for Automated Subject Indexing

Lisa Kluge; Maximilian K\"ahler

REVIEW 3 major objections 5 minor 39 references

DNB-AI-Project at SemEval-2025 Task 5: An LLM-Ensemble Approach for Automated Subject Indexing

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An ensemble of off-the-shelf LLMs, without fine-tuning, ranks first in expert-judged subject indexing.

desk verdict A workmanlike shared-task system paper whose quantitative results are solid; the qualitative first-place claim is real but statistically shakier than the abstract suggests. read the letter →

arxiv 2504.21589 v1 pith:RA7TJFPO submitted 2025-04-30 cs.CL cs.AIcs.DL

Lisa Kluge , Maximilian Kähler This is my paper · ORCID

classification cs.CLcs.AIcs.DL

keywords automatedsubjectindexingLLMensemblefew-shotpromptingcontrolledvocabularymappingSemEval-2025Task5qualitativeevaluationopen-weightLLMstagging

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that automated subject indexing for a national technical library's catalog can be done competitively without fine-tuning, by combining several off-the-shelf open-weight LLMs into an ensemble. On the SemEval-2025 Task 5 benchmark, the system placed fourth in the official quantitative ranking but first in a qualitative ranking conducted by subject indexing experts. The authors present this as evidence that few-shot prompting across diverse models and prompts, followed by mapping to a controlled vocabulary and LLM-based reranking, can rival supervised machine learning approaches. The paper also documents ablations showing the selected ensemble outperforms single-model and single-prompt strategies on the development test set.

What carries the argument

The load-bearing mechanism is the ensemble vote over model×prompt combinations: seven open-weight LLMs (Llama 3.2-3B, Llama 3.1-70B, Mistral-7B v0.1 and Instruct v0.3, Mixtral-8x7B, OpenHermes-2.5, Teuken-7B) are each prompted with 8–12 German, English, or mixed-language examples of intellectually indexed records, prompting them to suggest keywords for new records. A mapping stage embeds both the free keywords and the target vocabulary with BGE-M3 and finds nearest neighbours in a Weaviate vector store, storing similarity scores. A summarise stage sums these similarities across all model×prompt combinations and normalises to an ensemble score; a separate Llama-3.1-8B-Instruct model then rates each suggested term's relevance to the record on a 0–10 scale. The final ranking is a weighted combination $s_{fin} = \alpha \times s_{ens} + (1-\alpha) \times s_{rel}$ with $\alpha = 0.3$, so the relevance rating dominates the vote frequency.

What would settle it

Take a fresh 1,000-document sample from the same pool, re-run the two-stage ensemble selection (Monte-Carlo sampling then chain removal), and compare the re-selected ensemble's scores on the official test set with the paper's reported ones; if the re-selected ensemble performs materially worse, the selection procedure is not stable and the reported results depend on the particular dev-opt draw.

Watch

Extended reading notes

Core claim

The central discovery is that an ensemble of 20 model×prompt combinations, selected from 135 candidates, each generating free keywords via few-shot prompting, achieves the best expert-judged subject indexing result among all submitted systems while requiring no fine-tuning and no large training corpus. The keywords are mapped to the GND-Subjects-all vocabulary using BGE-M3 embeddings, scored by summed mapping similarities, then reranked by a separate LLM that rates each term's relevance to the record. The authors argue that this shows the combination of diverse open-weight LLMs can approximate the performance of supervised systems on quantitative metrics and surpass them on qualitative usefulness. The system's fourth-place quantitative ranking, they note, partially reflects its fixed output-list length of about 18 terms, which is disadvantageous for recall at high k thresholds.

Load-bearing premise

The system's official performance assumes the 20 model–prompt combinations selected for best precision-recall balance on a 1,000-document dev-opt sample generalise to the full official test set; if that small sample misrepresents the test distribution, the reported fourth-place quantitative and first-place qualitative ranks would not hold.

Editorial extensions

If this is right

A zero-fine-tuning ensemble of open-weight LLMs can produce subject terms that domain experts consider more relevant than those of supervised systems, suggesting libraries without large in-house training sets can still deploy automated indexing.
The fixed output length of the ensemble's suggested term lists limits recall@k comparisons; systems that can produce longer ranked lists will be advantaged on that metric, so shared-task evaluation should account for output-list length.
The mapping stage's extension of the target vocabulary with named entities from the DNB catalogue was needed to control false positives, indicating that vocabulary coverage is a critical design choice for LLM-based indexing.
Performance differences by record type (worst on Articles) and language (better on German) point to concrete improvements: adding more Article examples to prompts and providing English instructions or an English vocabulary.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A robustness check is missing: the 20-combination ensemble was selected on a 1,000-document dev-opt sample, so the reported official scores assume that selection transfers to the 27,987-document test set. Repeating selection on another small sample could reveal how much of the result depends on that particular tuning split.
The same pipeline could be dropped into other controlled-vocabulary annotation tasks, such as MeSH tagging of biomedical papers, where the target vocabulary is larger and fine-tuning data scarce; the mapping and rerank stages are vocabulary-agnostic.
The inference-cost table shows the 70B model alone consumed more than 17 GPU-hours on the test set; if deployment cost matters, one could test whether a smaller agreement-preserving subset of models reaches similar expert ratings.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The quantitative backbone is trustworthy. Fourth place in the official all-subjects track is externally evaluated, and the ablation in Appendix A.2 shows the top-20 ensemble beats the single-model-single-prompt baseline (F1 0.420 vs 0.380). That is a credible result, and the authors are honest that the ensemble adds less at the F1-optimal operating point than in the high-precision and high-recall regions. The system description is clear, the code is public, and the error analysis is a genuine attempt to understand failure modes rather than a gloss.

What is actually new is narrow but real: the specific ensemble of seven open-weight LLMs with 15 prompts, the vocabulary extension with 109,382 named entities to fix the false-positive mapping problem, and the two-stage Monte Carlo plus chain selection of the ensemble subset. These are engineering contributions, not conceptual breakthroughs, and the paper does not oversell them.

The soft spot is the qualitative first-place claim. Table 3 shows a 2.8-percentage-point F1@5 margin over DUTIR831, but the paper never states how many documents were in the expert evaluation, how many annotators judged them, or whether there is any inter-annotator agreement. No significance test is reported. Given that the qualitative protocol deliberately removes 'technically correct but irrelevant' terms, a small annotation sample could easily flip the ranking. That is the headline claim in the abstract, so the missing statistics matter.

Two smaller issues. First, the ensemble subset (20 of 135) was chosen by PR-AUC on a 1,000-document dev-opt sample; the paper gives no evidence that this selection is stable across samples. That is a common weakness in shared-task systems, and the authors acknowledge the tuning, but it should be flagged as a limitation rather than implied to be part of the method's general behavior. Second, there is a small internal inconsistency: Section 4.1 says alpha was tuned on dev-opt, while Appendix A.4 reports the choice on dev-test. Minor, but a careful referee will ask for a fix.

Overall, this is a solid system paper for the shared-task venue. It deserves a serious referee: the official results and ablation give it enough empirical grounding, and the limitations are stated honestly. I would not cite it as a methodological advance, but I would recommend accepting it with light revision requests about the qualitative evaluation metadata and the alpha discrepancy.

Referee Report

3 major / 5 minor

Summary. This paper describes the DNB-AI-Project system submitted to SemEval-2025 Task 5 (LLMs4Subjects). The system generates subject-term candidates by few-shot prompting several open-weight LLMs with multiple prompt templates, maps the generated free keywords to an extended GND vocabulary via BGE-M3 embeddings and vector search, aggregates mapped terms into an ensemble score, uses Llama-3.1-8B-Instruct to produce a relevance score, and combines both scores with a weighted average (α=0.3). A selection procedure on a 1,000-document dev-opt sample reduced 135 model×prompt combinations to 20 for the final run. On the official all-subjects test set, the system is reported as fourth in the quantitative ranking (by average recall) and first in the qualitative expert ranking (case 2, F1@5 = 0.412). Ablations on a dev-test sample indicate that the top-20 ensemble improves F1 over one-prompt-one-model (0.420 vs 0.380). The paper argues that competitive subject indexing can be achieved without fine-tuning or large training corpora.

Significance. If the reported results are reliable, the paper offers a useful data point: an ensemble of off-the-shelf open-weight LLMs with few-shot prompting can compete with supervised systems on a controlled-vocabulary indexing task, and the qualitative evaluation suggests that expert raters prefer its output. The paper is transparent in releasing code and in reporting compute costs, and the ablation study gives some support for the ensemble benefit. The central limitation is statistical: the headline qualitative first place rests on a 2.8-point F1@5 margin with no reported evaluation size, annotator details, agreement measure, or significance test, and the configuration itself was selected on a small dev sample. These issues are addressable, but they currently leave the strongest claim under-supported.

major comments (3)

[§5.2, Table 3] The paper's headline claim of 'best result in the qualitative ranking' rests on a 0.028 difference in F1@5 (0.412 vs 0.384 for DUTIR831) under the case-2 protocol. The manuscript does not report the number of documents in the qualitative evaluation, the number and expertise of the annotators, or inter-annotator agreement, nor does it provide a significance test. Given the small margin, these omissions are load-bearing: the difference may be within sampling noise. Please add the evaluation-protocol details (or cite the specific section of the task overview where they are defined) and apply a pairwise significance test (e.g., McNemar or bootstrap over documents) to the F1@5 difference between the top systems.
[§4.1, Appendix A.4] Section 4.1 says dev-opt was used to tune both the model×prompt selection and the combine weight α, but Appendix A.4 and Table 6 report PR-AUC values on dev-test for different α and state that α=0.3 was chosen from these dev-test results. This contradicts the stated split. If α was in fact tuned on dev-test, then dev-test is not an untouched evaluation set, and the ablations in Table 5 and Figure 2 should be interpreted accordingly. Please clarify which sample was used for each tuning decision and, if dev-test was used for tuning, either re-run the affected ablations on a true hold-out or explicitly label dev-test as an additional development set.
[§4.5, Appendix A.2] Section 4.5 describes selecting 20 of 135 model×prompt combinations by PR-AUC on a 1,000-document dev-opt sample, using a Monte-Carlo-like sampling followed by iterative removal. The appendix reports only point estimates for the chosen strategy (Table 5: F1 0.420 vs 0.380 for one-prompt-one-model). Since the official submission depends on this selection, the paper should report the stability of the selection (e.g., bootstrap over dev-opt or repeated selection runs) and provide confidence intervals for the dev-test F1 values. Without this, it is unclear whether the reported ensemble benefit is robust to the selection procedure.

minor comments (5)

[§6, throughout] The text contains several typos: 'appoach' should be 'approach', 'enourmous' should be 'enormous', 'ressources' should be 'resources', 'involed' should be 'involved', and 'comparision' should be 'comparison'.
[Appendix A.4, Table 6] The caption abbreviates the second ensemble as 'one-prompt-all-models (1M-AP)', which is inconsistent with the column header '1M-AP' and with the earlier usage in the paper; this should be 'one-model-all-prompts (1M-AP)'.
[Appendix A.8, Table 9] The percentages in Table 9 are not clearly aligned with the row and column semantics; consider adding row/column totals or a more explicit caption that defines 'Found', 'Close', 'Distant', and 'Difficult'.
[§5.2] The term 'qualitative ranking' and the 'case 2' protocol should be defined more carefully in the paper, or the exact section of the task-overview paper that defines them should be cited, so that readers can interpret the qualitative result without consulting external materials.
[§4.2] Please clarify whether the 109,382 named entities from the DNB catalogue are selected by a deterministic rule and whether the extended vocabulary is released publicly for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: official external evaluation anchors the central claims.

full rationale

The paper's central claims—fourth in the quantitative ranking and first in the qualitative expert ranking at SemEval-2025 Task 5—are determined by the task organizers on the official test set and by external expert judgments, not by any quantity fitted or defined in this paper. The tuned components (α = 0.3 in Eq. 1 and the selection of 20 model×prompt combinations out of 135 on dev-opt) are hyperparameters chosen for the submitted configuration; they are not re-labelled predictions, and the official metrics do not reduce to them by construction. The ensemble score s_ens and relevance score s_rel are combined in Eq. 1 as a ranking method, not as a derivation of the benchmark outcome. Self-citations to Kluge and Kähler (2024) are used only to position the work and to describe a previously used lemma-overlap prompt-sampling measure; none of the load-bearing claims depends on an unverified self-citation chain. The paper itself flags practical limitations such as list length and inference cost, but these are caveats about the system's deployability, not circular reductions. The paper is therefore self-contained against external benchmarks, and no circular step is present.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The system's performance rests on a small number of tuned hyperparameters (alpha, ensemble subset, token limits) and on domain assumptions about text representation, embedding quality, and evaluation validity. No new theoretical entities are introduced.

free parameters (4)

combine weight alpha = 0.3
In Equation 1, s_fin = alpha*s_ens + (1-alpha)*s_rel. Tuned on dev-opt (Section 4.5, Appendix A.4); the chosen value 0.3 gave the best PR-AUC on dev-test.
mapping similarity threshold = not specified
Section 3.2 states that matches with a low similarity score can be discarded with a tunable threshold, but the value used is not reported anywhere in the paper.
ensemble subset = 20 of 135 model x prompt combinations
The final ensemble was selected by Monte Carlo sampling and iterative chain removal on dev-opt (Section 4.5, Appendix A.2). This selection is a data-driven model choice that directly affects the system's outputs.
generation token limits = min 24, max 100 tokens
Section 4.3 states the number of keywords generated was controlled by setting min/max tokens; these hyperparameters affect the length and recall of candidate sets and are available in the repository.

assumptions (4)

domain assumption Concatenation of title and abstract is sufficient text representation for subject indexing.
Section 4.1: 'For both input texts and prompts, we used the concatenation of title and abstract as text representation.' The system never sees full text, so subjects only inferable from full text will be missed, as the error analysis in Section 5.3 acknowledges.
domain assumption BGE-M3 embeddings provide reliable semantic similarity between free keywords and controlled vocabulary terms.
Section 3.2: The map stage embeds keywords and vocabulary with BGE-M3 and performs nearest-neighbor search; mapping quality depends entirely on this model, which is taken as given.
domain assumption The official gold standard and the qualitative expert evaluation are valid measures of indexing quality.
The central claims of fourth quantitative and first qualitative are defined by the shared task's evaluation; no independent validation of these measures is provided in the paper.
domain assumption The dev-opt sample is representative of the test distribution.
Section 4.1: dev-opt is a stratified random sample of 1,000 documents from the union of development sets; hyperparameters and the ensemble subset are optimized on it, and generalization to the official test set is assumed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of DNB-AI-Project at SemEval-2025 Task 5: An LLM-Ensemble Approach for Automated Subject Indexing." pith.science (2026). https://pith.science/paper/RA7TJFPO

@misc{pith2026250421589,
  author       = {Pith},
  title        = {Pith review of: DNB-AI-Project at SemEval-2025 Task 5: An LLM-Ensemble Approach for Automated Subject Indexing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RA7TJFPO}},
  note         = {Machine review of arXiv:2504.21589}
}

read the original abstract

This paper presents our system developed for the SemEval-2025 Task 5: LLMs4Subjects: LLM-based Automated Subject Tagging for a National Technical Library's Open-Access Catalog. Our system relies on prompting a selection of LLMs with varying examples of intellectually annotated records and asking the LLMs to similarly suggest keywords for new records. This few-shot prompting technique is combined with a series of post-processing steps that map the generated keywords to the target vocabulary, aggregate the resulting subject terms to an ensemble vote and, finally, rank them as to their relevance to the record. Our system is fourth in the quantitative ranking in the all-subjects track, but achieves the best result in the qualitative ranking conducted by subject indexing experts.

Figures

Figures reproduced from arXiv: 2504.21589 by the authors.

**Figure 2.** Precision-Recall curves for different model [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

**Figure 3.** Precision-Recall Balance of single prompt [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 11 canonical work pages

[1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Mehdi Ali, Michael Fromm, Klaudia Thellmann, Jan Ebert, Alexander Arno Weber, Richard Rutmann, Charvi Jain, Max Lübbering, Daniel Steinigen, Johannes Leveling, Katrin Klug, Jasper Schulze Buschhoff, Lena Jurkschat, Hammam Abdelwahab, Benny Jörg Stein, Karl-Heinz Sylla, Pavel Denisov, Nicolo' Brandizzi, Qasid Saleem, Anirban Bhowmick, Lennard Helmer, Chels...

arXiv 2024
[4]

https://weaviate.io/blog/hybrid-search-explained Hybrid search explained [online]

Erika Cardenas. https://weaviate.io/blog/hybrid-search-explained Hybrid search explained [online]. 2025

work page 2025
[5]

Jianlv Chen , Shitao Xiao , Peitian Zhang , Kun Luo , Defu Lian , and Zheng Liu . 2024. https://doi.org/10.48550/arXiv.2402.03216 Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation . arXiv e-prints, page arXiv:2402.03216

arXiv doi:10.48550/arxiv.2402.03216 2024
[6]

Lingjiao Chen , Matei Zaharia , and James Zou . 2023. https://doi.org/10.48550/arXiv.2305.05176 Frugalgpt: How to use large language models while reducing cost and improving performance . arXiv e-prints, page arXiv:2305.05176

arXiv doi:10.48550/arxiv.2305.05176 2023
[7]

Karel D'Oosterlinck, Omar Khattab, Fran c ois Remy, Thomas Demeester, Chris Develder, and Christopher Potts. 2024. http://arxiv.org/abs/2401.12178 In-context learning for extreme multi-label classification . arXiv preprint arXiv:2401.12178

arXiv 2024
[8]

Jennifer D'Souza, Sameer Sadruddin, Holger Israel, Mathias Begoin, and Diana Slawig. 2025. https://aclanthology.org/2025.semeval2025-1.139 Semeval-2025 task 5: Llms4subjects - llm-based automated subject tagging for a national technical library's open-access catalog . In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025),...

work page 2025

Show all 39 references

[9]

Nicolai Erbs, Iryna Gurevych, and Marc Rittberger. 2013. http://download.hrz.tu-darmstadt.de/pub/FB20/Dekanat/Publikationen/UKP/NeKeyphrasesDLib2013.pdf Bringing order to digital libraries: From keyphrase extraction to index term assignment . D-Lib Magazine, 19(9/10):1--16

2013
[10]

Chenhao Fang, Xiaohan Li, Zezhong Fan, Jianpeng Xu, Kaushiki Nag, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. 2024. https://doi.org/10.1145/3626772.3661357 Llm-ensemble: Optimal large language model ensemble method for e-commerce product attribute value extraction . In P...

2024
[11]

Koraljka Golub. 2021. https://doi.org/10.1080/01639374.2021.2012311 Automated subject indexing: An overview . Cataloging & Classification Quarterly, 59(8):702--719

2021
[12]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. http://arxiv.org/abs/2407.21783 The llama 3 herd of models

2024 arXiv
[13]

Bairu Hou, Joe O’Connor, Jacob Andreas, Shiyu Chang, and Yang Zhang. 2023. Promptboosting: Black-box text classification with ten forward passes. In International Conference on Machine Learning, pages 13309--13324. PMLR

2023
[14]

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

2023 arXiv
[15]

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

2024 arXiv
[16]

Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023 b . https://doi.org/10.18653/v1/2023.acl-long.792 LLM -blender: Ensembling large language models with pairwise ranking and generative fusion . In Proceedings of the 61st Annual Meeting of the Association for Computational Ling...

2023 doi
[17]

Sujay Khandagale, Han Xiao, and Rohit Babbar. 2020. https://doi.org/10.1007/s10994-020-05888-2 Bonsai: diverse and shallow trees for extreme multi-label classification . Machine Learning, 109(11):2099--2119

2020 doi
[18]

Joshi , Hanna Moazam , Heather Miller , Matei Zaharia , and Christopher Potts

Omar Khattab , Arnav Singhvi , Paridhi Maheshwari , Zhiyuan Zhang , Keshav Santhanam , Sri Vardhamanan , Saiful Haq , Ashutosh Sharma , Thomas T. Joshi , Hanna Moazam , Heather Miller , Matei Zaharia , and Christopher Potts . 2023. https://doi.org/10.48550/arXiv.2310.03714 Dsp...

2023 doi arXiv
[19]

Lisa Kluge and Maximilian K \"a hler. 2024. https://aclanthology.org/2024.konvens-main.16/ Few-shot prompting for subject indexing of G erman medical book titles . In Proceedings of the 20th Conference on Natural Language Processing (KONVENS 2024), pages 141--148, Vienna, Aust...

2024
[20]

Ruslan Kuprieiev, skshetry, Peter Rowlands, Dmitry Petrov, Paweł Redzyński, Casper da Costa-Luis, David de la Iglesia Castro, Alexander Schepanovski, Ivan Shcheklein, Gao, Batuhan Taskaya, Jorge Orpinel, Fábio Santos, Daniele, Ronan Lamy, Aman Sharma, Zhanibek Kaimuldenov, Dan...

2025 doi
[21]

Gonzalez, Hao Zhang, and Ion Stoica

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

2023
[22]

Wanhae Lee, Minki Chun, Hyeonhak Jeong, and Hyunggu Jung. 2023. https://doi.org/10.1145/3581754.3584126 Toward keyword generation through large language models . In Companion Proceedings of the 28th International Conference on Intelligent User Interfaces, IUI '23 Companion, pa...

2023
[23]

Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. 2023. http://arxiv.org/abs/2311.08692 Routing to the expert: Efficient reward-guided ensemble of large language models

2023 arXiv
[24]

Malkov and D

Yu A. Malkov and D. A. Yashunin. 2016. https://doi.org/10.1109/TPAMI.2018.2889473 Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs . IEEE Transactions on Pattern Analysis and Machine Intelligence, 42:824--836

2016
[25]

Reza Yousefi Maragheh, Chenhao Fang, Charan Chand Irugu, Parth Parikh, Jason Cho, Jianpeng Xu, Saranyan Sukumar, Malay Patel, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. 2023. https://doi.org/10.1109/BigData59044.2023.10386476 Llm-take : Theme-aware keyword extraction us...

2023
[26]

Olena Medelyan. 2009. https://hdl.handle.net/10289/3513 Human-competitive automatic topic indexing . Ph.D. thesis, The University of Waikato, New Zealand

2009
[27]

Rajvardhan Patil and Venkat Gudivada. 2024. https://doi.org/10.3390/app14052074 A review of current trends, techniques, and challenges in large language models (llms) . Applied Sciences, 14(5)

2024 doi
[28]

Youri Peskine, Damir Koren c i \'c , Ivan Grubisic, Paolo Papotti, Raphael Troncy, and Paolo Rosso. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.267 Definitions matter: Guiding gpt for multi-label classification . In Findings of the Association for Computational Lingu...

2023 doi
[29]

Zhang , Andrew Wang , and Jimmy Ba

Silviu Pitis , Michael R. Zhang , Andrew Wang , and Jimmy Ba . 2023. https://doi.org/10.48550/arXiv.2304.05970 Boosted prompt ensembles for large language models . arXiv e-prints, page arXiv:2304.05970

2023 doi arXiv
[30]

Stephen Robertson and Hugo Zaragoza. 2009. https://doi.org/10.1561/1500000019 The probabilistic relevance framework: Bm25 and beyond . Foundations and Trends® in Information Retrieval, 3(4):333--389

2009 doi
[31]

Osma Suominen. 2019. https://liberquarterly.eu/article/view/10732 Annif: Diy automated subject indexing using multiple algorithms . LIBER Quarterly: The Journal of the Association of European Research Libraries, 29(1):1--25

2019
[32]

Selim Furkan Tekin, Fatih Ilhan, Tiansheng Huang, Sihao Hu, and Ling Liu. 2024. http://arxiv.org/abs/2410.03953 Llm-topla: Efficient llm ensemble by maximising diversity

2024 arXiv
[33]

Martin Toepfer and Christin Seifert. 2020. https://link.springer.com/article/10.1007/s00799-018-0240-3 Fusion architectures for automatic subject indexing under concept drift: Analysis and empirical results on short texts . International Journal on Digital Libraries, 21(2):169--189

2020 doi
[34]

Fouad Trad and Ali Chehab . 2024. https://doi.org/10.48550/arXiv.2412.00166 To ensemble or not: Assessing majority voting strategies for phishing detection with large language models . arXiv e-prints, page arXiv:2412.00166

2024 doi arXiv
[35]

Fanqi Wan , Xinting Huang , Deng Cai , Xiaojun Quan , Wei Bi , and Shuming Shi . 2024. https://doi.org/10.48550/arXiv.2401.10491 Knowledge fusion of large language models . arXiv e-prints, page arXiv:2401.10491

2024 doi arXiv
[36]

Hongyi Wang , Felipe Maia Polo , Yuekai Sun , Souvik Kundu , Eric Xing , and Mikhail Yurochkin . 2023. https://doi.org/10.48550/arXiv.2310.01542 Fusing models with complementary expertise . arXiv e-prints, page arXiv:2310.01542

2023 doi arXiv
[37]

Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Shaochen Zhong, Bing Yin, and Xia Hu. 2024. https://doi.org/10.1145/3649506 Harnessing the power of llms in practice: A survey on chatgpt and beyond . ACM Trans. Knowl. Discov. Data, 18(6)

2024 doi
[38]

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong W...

2023 arXiv
[39]

Yaxin Zhu and Hamed Zamani. 2024. http://arxiv.org/abs/2311.09649 Icxml : An in-context learning framework for zero-shot extreme multi-label classification . arXiv preprint arXiv:2311.09649

2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Mehdi Ali, Michael Fromm, Klaudia Thellmann, Jan Ebert, Alexander Arno Weber, Richard Rutmann, Charvi Jain, Max Lübbering, Daniel Steinigen, Johannes Leveling, Katrin Klug, Jasper Schulze Buschhoff, Lena Jurkschat, Hammam Abdelwahab, Benny Jörg Stein, Karl-Heinz Sylla, Pavel Denisov, Nicolo' Brandizzi, Qasid Saleem, Anirban Bhowmick, Lennard Helmer, Chels...

arXiv 2024

[4] [4]

https://weaviate.io/blog/hybrid-search-explained Hybrid search explained [online]

Erika Cardenas. https://weaviate.io/blog/hybrid-search-explained Hybrid search explained [online]. 2025

work page 2025

[5] [5]

Jianlv Chen , Shitao Xiao , Peitian Zhang , Kun Luo , Defu Lian , and Zheng Liu . 2024. https://doi.org/10.48550/arXiv.2402.03216 Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation . arXiv e-prints, page arXiv:2402.03216

arXiv doi:10.48550/arxiv.2402.03216 2024

[6] [6]

Lingjiao Chen , Matei Zaharia , and James Zou . 2023. https://doi.org/10.48550/arXiv.2305.05176 Frugalgpt: How to use large language models while reducing cost and improving performance . arXiv e-prints, page arXiv:2305.05176

arXiv doi:10.48550/arxiv.2305.05176 2023

[7] [7]

Karel D'Oosterlinck, Omar Khattab, Fran c ois Remy, Thomas Demeester, Chris Develder, and Christopher Potts. 2024. http://arxiv.org/abs/2401.12178 In-context learning for extreme multi-label classification . arXiv preprint arXiv:2401.12178

arXiv 2024

[8] [8]

Jennifer D'Souza, Sameer Sadruddin, Holger Israel, Mathias Begoin, and Diana Slawig. 2025. https://aclanthology.org/2025.semeval2025-1.139 Semeval-2025 task 5: Llms4subjects - llm-based automated subject tagging for a national technical library's open-access catalog . In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025),...

work page 2025