Pith. sign in

REVIEW 4 major objections 4 minor 28 references

On a new Greek book-search benchmark, weighted hybrid retrieval — sparse BM25 plus two multilingual dense models — outperforms every standalone retriever, and multilingual embeddings beat Greek-specific ones.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 07:55 UTC pith:VZAKPAF3

load-bearing objection CUP is a genuinely useful new Greek retrieval benchmark; the headline hybrid gain is probably inflated by tuning on the test queries, but the qualitative ordering of methods is likely robust. the 4 major comments →

arxiv 2607.21274 v1 pith:VZAKPAF3 submitted 2026-07-23 cs.CL cs.AI

A Comparative Evaluation of Embeddings and LLMs in a Greek Book Publisher Setting - The CUP Dataset

Katerina Papantoniou , Panagiotis Papadakos , Theodore Patkos , Dimitris Garefalakis , Nikos Vardakis , Dimitris Plexousakis This is my paper
classification cs.CL cs.AI
keywords Greek information retrievalbook searchhybrid retrievalmultilingual embeddingsBM25dense retrievalLLM-assisted retrievalretrieval benchmark
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces CUP, a benchmark of 868 Greek book catalog records and 104 expert-annotated queries with graded relevance judgments, spanning keyword, natural-language, noisy, cross-lingual, and named-entity searches. Its central claim is that no single retriever is best: weighted fusion of BM25 with the two strongest multilingual dense models reaches nDCG@9 of 0.673 and MRR of 0.810, clearly above the best dense model (0.582 nDCG@9) and BM25 (0.544). Multilingual embeddings consistently outperform Greek-specific models, and LLM-generated table-of-contents summaries improve retrieval over raw TOCs, while LLM post-filtering buys early precision at a large latency cost. The paper's contribution is a realistic testbed for Greek retrieval and a concrete recipe: use hybrid fusion plus multilingual encoders, and keep BM25 for exact named-entity queries.

Core claim

The paper's discovery is that lexical and semantic signals are complementary on this corpus, and that score-level weighted fusion captures that complementarity. The winning combination weights sparse BM25 at 0.4 and a dense component — itself a 0.6/0.4 blend of the two top multilingual dense models — at 0.6, reaching nDCG@9 of 0.673 and MRR of 0.810. This beats BM25 (0.544 nDCG@9) and the best standalone dense model (0.582). Per-query-type results show BM25 remains near-perfect for exact author-name queries while dense and hybrid systems carry natural-language, noisy, cross-lingual, and concept queries. LLM-generated TOC summaries improve retrieval over raw TOCs, and LLM post-filtering impro

What carries the argument

The load-bearing object is the benchmark itself: 868 records with five fields (title, author, categories+tags, content, TOC; plus LLM-generated TOC summaries) and 104 graded, expert-annotated queries partitioned into eight query types. The mechanism that produces the headline result is score-level weighted fusion, s0.4 + (d10.6 + d20.4)0.6 — a weighted sum of BM25 and two dense retrievers — along with field-aware prefixes that let embeddings encode which field a passage came from. The benchmark's design, including noisy, cross-lingual, and concept queries, is what lets the comparison expose complementarity rather than a single winner.

Load-bearing premise

The method ranking assumes the greedy fusion weights (Section 4) were not tuned on the 104 evaluation queries and that the four experts' relevance labels (Section 3) are mutually consistent; if either fails, the hybrid advantage over BM25 and dense retrieval may not reproduce.

What would settle it

Take the 104 queries, tune the hybrid weights on a random half, and evaluate on the held-out half; if the hybrid margin over BM25 and the best dense model disappears or reverses, the headline ordering is a tuning artifact. Independently, a second round of expert labeling on a subset of queries would test whether the graded judgments are reliable enough to support the comparison.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Practitioners building Greek book or catalog search should expect hybrid score fusion to beat any single retriever; the paper's weights and formula are a concrete starting point.
  • Multilingual embeddings should be preferred over Greek-specific ones for retrieval, even with Greek-domain text, because scale and cross-lingual training outweigh language-specific adaptation.
  • Exact named-entity and author queries should be routed to BM25, or at least included in the fusion, since it achieves perfect nDCG@9 there.
  • LLM-generated TOC summaries are a viable enrichment for sparse structured metadata and improve retrieval over raw TOCs; prompt adherence matters and can distort results.
  • LLM post-filtering can lift early precision (MRR 0.821) but at per-query latency that makes it impractical for real-time use.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: if the fusion weights generalize beyond the 104 queries, a similar sparse+dense hybrid should transfer to other morphologically rich, low-resource languages, since the complementarity mechanism is not Greek-specific.
  • The large spread between Greek-specific and multilingual encoders suggests the bottleneck is pretraining data scale rather than domain adaptation; fine-tuning a Greek-specific model on CUP and seeing whether the gap closes would test this.
  • The failure of one Greek-focused LLM due to prompt leakage implies the reported LLM-assisted gains partly measure instruction-following rather than summarization quality; future benchmarks could control for this by measuring lexical overlap between summaries and source fields.
  • Because the noisy-query category shows the largest hybrid gain, the benchmark suggests hybrid fusion is especially valuable for user-generated or typo-laden queries, which could be tested on other noisy IR collections.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces CUP, a Greek-language book retrieval benchmark built from 868 Crete University Press catalog records and 104 expert-annotated queries with graded relevance judgments. It evaluates BM25, 13 sentence-transformer embedding models (including Greek-specific and multilingual), weighted and RRF hybrid retrievers, and five LLMs used for TOC summarization and post-retrieval filtering/reranking. Headline results include: a weighted hybrid s0.4+(d10.6+d20.4)0.6 reaching nDCG@9=0.673 and MRR=0.810, versus 0.582/0.748 for the best dense model and 0.544/0.652 for BM25; multilingual embeddings outperforming Greek-specific models; LLM TOC summarization improving TOC-only retrieval; and LLM post-filtering improving early precision at high inference cost. The paper also presents per-query-type analysis showing complementary strengths of lexical, dense, and hybrid methods.

Significance. If the reported numbers hold, CUP fills a real gap: a Greek IR benchmark with graded relevance judgments, realistic publisher metadata, and diverse query types. The paper is a direct empirical comparison, not a fitted derivation, and the dataset and detailed results are promised publicly. The per-query analysis is useful for practitioners. However, the central quantitative claim - that weighted hybrid retrieval is best - rests on fusion weights tuned on the same 104 queries used for evaluation, and the relevance-label reliability is not quantified. These issues affect the generalizability of the numeric ordering, even though the benchmark itself remains a valuable resource.

major comments (4)
  1. [Section 4, Table 2] The best hybrid configuration s0.4+(d10.6+d20.4)0.6 is selected by "greedy search" over fusion weights, but no held-out development split, cross-validation, or selection-corrected evaluation protocol is described. With only 104 evaluation queries, the reported nDCG@9=0.673 is the maximum of a set of weighted/RRF fusions evaluated on the same queries used to report all metrics. The 0.091 gain over nemotron* may therefore be partially a selection artifact. Please provide a train/dev/test split, nested evaluation, or at minimum a complete statement of the candidate weight grid and a correction for model selection. Without this, the "hybrid retrieval performs best" claim is not fully supported.
  2. [Section 4] The statement that the best hybrid gains are "statistically significant over most metrics (p < 0.01)" is not verifiable: no test is named, no multiple-comparison correction is described, and the same 104 queries were used for both weight tuning and significance testing. State the test used (e.g., paired bootstrap, Wilcoxon), the exact comparisons, and whether the p-values were corrected across the many models and hybrids evaluated. As written, the significance claim is not interpretable and should be removed or made precise.
  3. [Section 3] No inter-annotator agreement or adjudication procedure is reported for the graded relevance judgments. The text says 4 domain experts examined the corpus, but it does not say whether each query was independently judged by all annotators, how disagreements were resolved, or how consistent the labels were. Since every retrieval metric in Table 2 depends on these labels, the benchmark's reliability is unknown. Report agreement (e.g., Fleiss' kappa or Krippendorff's alpha) on the full set or a subset, and describe the adjudication process.
  4. [Section 4, Table 2] The conclusion that "multilingual embeddings outperform Greek-specific models" is confounded by model type and era: the two Greek-specific encoders (st-Greek-BERT 2023, xlm-r 2020) are not retrieval-tuned sentence encoders, whereas the multilingual models include recent retrieval-specialized models such as BGE-M3, Nemotron, and Qwen3. The comparison is valid as a practical benchmark, but the wording overgeneralizes. Please soften the claim to "the evaluated Greek-specific sentence encoders underperform current multilingual retrieval embeddings" or add a retrieval-trained Greek-specific baseline.
minor comments (4)
  1. [Table 2] The header "ndnDCGcg@20" appears to be a typo for "nDCG@20". Please correct.
  2. [Section 4] The sentence "TOCs are summarized using five LLMs (see below), and we use the krikri ones for the best method retrieval evaluation" is ambiguous. Clarify whether the main hybrid retrieval results use the raw TOC field or the Krikri-summarized TOC, and justify the choice of Krikri over EuroLLM.
  3. [Section 4] BM25 parameters (k1, b) are not specified. Since BM25 is a baseline for all hybrid comparisons, please state the parameter values used or note that the rank_bm25 defaults were used.
  4. [References] The text says metrics were computed using "rank-eval [2]", but reference [2] is the ranx library. Align the citation with the package name, or cite the appropriate rank-eval reference if a different tool was used.

Circularity Check

0 steps flagged

No circularity: the paper is an empirical benchmark evaluation; claims are direct measurements, not derivations from fitted values or self-citations.

full rationale

This paper does not contain a derivation chain in the sense the circularity check targets. It introduces the CUP benchmark and reports measured retrieval scores for sparse, dense, hybrid, and LLM-based methods. Claims such as 'The best results are obtained by weighted hybrid retrieval...' and the reported nDCG@9/MRR values are computed directly from the annotated queries and retrieved lists, not derived from the method's definition. The hybrid weights were 'tuned via greedy search' (Section 4) on the same 104 evaluation queries, which is a potential test-set-selection / overfitting concern for the soundness and generalizability of the numerical comparisons; however, it does not make the reported ranking circular by construction: the paper does not fit a parameter and then present a closely related quantity as an independent prediction, nor does it define any target metric in terms of the tuned weights. There are no load-bearing self-citations, no imported uniqueness theorem, no ansatz smuggled via citation, and no renaming of a known result. The absence of a held-out split and the lack of inter-annotator agreement are experimental-validity issues, not circular-reasoning loops. Accordingly, the correct circularity score is 0.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

Central empirical claims rely on the annotation protocol and on tuning decisions. Most of the scientific content is contributed by expert labels and the choice of fusion weights; those are the quantities the reader should scrutinize.

free parameters (1)
  • Hybrid fusion weights (s, d1, d2) = s=0.4; dense blend d1=0.6, d2=0.4; dense weight=0.6
    Chosen by greedy search in §4. If tuned on the same 104 queries used for evaluation, the best-hybrid nDCG@9=0.673 and MRR=0.810 are optimistically biased.
axioms (4)
  • domain assumption Expert relevance judgments are treated as ground truth.
    §3 says four experts assigned graded labels after examining the corpus, but no inter-annotator agreement or adjudication is reported; noisy labels would change method rankings.
  • domain assumption Greedy fusion weights were selected without overfitting to the evaluation queries.
    §4 states weights are 'tuned via greedy search' with no train/validation split; if tuning used the same 104 queries, reported hybrid gains are inflated.
  • domain assumption Pretrained models are used according to their intended prompt/format conventions.
    §4 says recommended prompting/formatting is followed; meltemi's failure to follow the prompt shows results are sensitive to this choice.
  • domain assumption The significance test behind p<0.01 is valid.
    §4 reports p<0.01 without naming the test or accounting for tuned parameters; invalid testing would weaken the claim that hybrid gains generalize.

pith-pipeline@v1.3.0-alltime-deepseek · 8847 in / 13191 out tokens · 121597 ms · 2026-08-01T07:55:50.841450+00:00 · methodology

0 comments
read the original abstract

We present CUP, a Greek book retrieval benchmark consisting of 868 catalog records and 104 expert-annotated queries with graded relevance judgments. We evaluate sparse (BM25), dense (sentence-transformers), hybrid, and LLM-assisted retrieval methods in this book-search setting. Multilingual embeddings outperform Greek-specific models, while hybrid retrieval performs best overall. A query-level analysis shows that BM25 excels at named-entity queries, while dense and hybrid methods improve natural-language, noisy, cross-lingual, and concept queries. Field-aware prompting has model-specific effects, while LLM TOC summarization improves TOC-only retrieval and LLM post-filtering improves early-stage retrieval at a high cost. Overall, CUP enables real-world evaluation of Greek retrieval across lexical, semantic, noisy, and cross-lingual queries.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

28 extracted references · 1 canonical work pages

  1. [1]

    Ahmed Aly, Ahmed Ibrahim, and Rasha Kashef. 2026. RAG-Rec: Retrieval Augmented Gen- eration for Robust Personalized Recommendation Systems. Procedia Comput. Sci. 275 (2026), 1–8

    2026

  2. [2]

    Elias Bassani. 2022. ranx: A blazing-fast python library for ranking evaluation and comparison. In ECIR. Springer, 259–264

    2022

  3. [3]

    Luiz Bonifacio, Israel Campiotti, Roberto de Alencar Lotufo, and Rodrigo Frassetto Nogueira

  4. [4]

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. BGE M3- Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. https://arxiv.org/abs/2402.03216

    Pith/arXiv arXiv 2024

  5. [5]

    Chlapanis, Dimitrios Galanis, Nikolaos Aletras, and Ion Androutsopoulos

    Odysseas S. Chlapanis, Dimitrios Galanis, Nikolaos Aletras, and Ion Androutsopoulos. 2025. GreekBarBench: A Challenging Benchmark for Free-Text Legal Reasoning and Citations. https://arxiv.org/abs/2505.17267

    arXiv 2025

  6. [6]

    Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzemiński, Genta Indra Winata, et al. 2025. MMTEB: Massive Multilingual Text Embedding Benchmark. https://arxiv.org/abs/2502.13595

    arXiv 2025

  7. [7]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, et al. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Re- inforcement Learning. https://arxiv.org/abs/2501.12948

    Pith/arXiv arXiv 2025

  8. [8]

    Konstantinos Kogkalidis, Stergios Chatzikyriakidis, Eirini Giannikouri, Vasiliki Katsouli, Christina Klironomou, Christina Koula, Dimitris Papadakis, Thelka Pasparaki, Erofili Psaltaki, and Efthymia Sakellariou. 2024. OYXOY: A Modern NLP Test Suite for Modern Greek. In EACL. 311–322

    2024

  9. [9]

    Zhiyu Li, Yanfang Chen, Xuan Zhang, and Xun Liang. 2023. BookGPT: A General Framework for Book Recommendation Empowered by LLM. Electronics 12, 22 (2023), 4654

    2023

  10. [10]

    Guerreiro, Ricardo Rei, Duarte M

    Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, Pierre Colombo, Barry Haddow, José G. C. de Souza, Alexandra Birch, and André F. T. Martins. 2025. EuroLLM: Multilingual Language Models for Europe. Procedia Comput. Sci. 255 (2025), 53–62...

    doi:10.1016/j.procs.2025.02.260 2025

  11. [11]

    Moreira, Radek Osmulski, Mengyao Xu, Ronay Ak, Benedikt Schifferer, and Even Oldridge

    Gabriel de Souza P. Moreira, Radek Osmulski, Mengyao Xu, Ronay Ak, Benedikt Schifferer, and Even Oldridge. 2024. NV-Retriever: Improving Text Embedding Models with Effective Hard-Negative Mining. https://arxiv.org/abs/2407.15831

    Pith/arXiv arXiv 2024

  12. [12]

    Zach Nussbaum and Brandon Duderstadt. 2025. Training Sparse Mixture Of Experts Text Embedding Models. https://arxiv.org/abs/2502.07972

    Pith/arXiv arXiv 2025

  13. [13]

    Dimitris Papadopoulos, Katerina Metropoulou, Nikolaos Papadakis, and Nikolaos Matsatsinis

  14. [14]

    Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In EMNLP. ACL. http://arxiv.org/abs/1908.10084

    Pith/arXiv arXiv 2019

  15. [15]

    Dimitris Roussis, Leon Voukoutis, Georgios Paraskevopoulos, Sokratis Sofianopoulos, Prokopis Prokopidis, Vassilis Papavassileiou, Athanasios Katsamanis, Stelios Piperidis, and Vassilis Katsouros. 2025. Krikri: Advancing Open Large Language Models for Greek. InEMNLP Findings 2025. 5012–5033. doi:10.18653/v1/2025.findings-emnlp.268

    doi:10.18653/v1/2025.findings-emnlp.268 2025

  16. [16]

    Alexandros Tassios, Stergios Tegos, Christos Bouas, Konstantinos Manousaridis, Maria Pa- poutsoglou, Maria Kaltsa, Eleni Dimopoulou, Thanassis Mavropoulos, Stefanos Vrochidis, and Georgios Meditskos. 2025. LLM Performance in Low-Resource Languages: Selecting an Opti- mal Model for Migrant Integration Support in Greek. Future Internet 17, 6 (2025), 235

    2025

  17. [17]

    Jianqiang Wang. 2025. Building Recommender Systems Using LLMs . Springer

    2025

  18. [18]

    Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual E5 Text Embeddings: A Technical Report. https://arxiv.org/abs/2402.05672

    Pith/arXiv arXiv 2024

  19. [19]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, et al

  20. [20]

    Puxuan Yu, Luke Merrick, Gaurav Nuti, and Daniel Campos. 2024. Arctic-Embed 2.0: Multi- lingual Retrieval Without Compromise. https://arxiv.org/abs/2412.04506

    Pith/arXiv arXiv 2024

  21. [21]

    Dimitrios Zaikis, Stylianos Kokkas, and Ioannis Vlahavas. 2023. DACL: A Domain-Adapted Contrastive Learning Approach to Low Resource Language Representations for Document Clustering Tasks. In Engineering Applications of Neural Networks . Springer Nature, 585–598

    2023

  22. [22]

    Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. 2023. MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages. TACL 11 (2023), 1114–1131

    2023

  23. [23]

    Yang Zhang, Mersin Konomi, Christos Xypolopoulos, Konstantinos Divriotis, Konstantinos Skianis, Giannis Nikolentzos, Giorgos Stamou, Guokan Shang, and Michalis Vazirgiannis

  24. [24]

    Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. 2025. Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models. https://arxiv.org/ abs/2506.05176

    Pith/arXiv arXiv 2025

  25. [2021]

    https: //arxiv.org/abs/2108.13897

    mMARCO: A Multilingual Version of MS MARCO Passage Ranking Dataset. https: //arxiv.org/abs/2108.13897

    Pith/arXiv arXiv

  26. [2022]

    InWorkshop on DL for Low-Resource NLP

    FarFetched: Entity-centric Reasoning and Claim Validation for the Greek Language based on Textually Represented Environments. InWorkshop on DL for Low-Resource NLP. ACL, 180–191. doi:10.18653/v1/2022.deeplo-1.19

    work page doi:10.18653/v1/2022.deeplo-1.19 2022

  27. [2025]

    https://arxiv.org/abs/2505.09388

    Qwen3 Technical Report. https://arxiv.org/abs/2505.09388

    Pith/arXiv arXiv

  28. [2026]

    https://arxiv.org/abs/2602.05150

    GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greek. https://arxiv.org/abs/2602.05150

    arXiv