REVIEW 3 major objections 5 minor 66 references
mFollowIR: a Multilingual Benchmark for Instruction Following in Retrieval
T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read The paper introduces mFollowIR, the first multilingual benchmark for instruction following in retrieval, and reports that English instruction-trained retrievers transfer well across languages but fully multilingual performance drops…
desk verdict Useful multilingual instruction-following benchmark, but p-MRR may reward keyword matching rather than semantic instruction-following, so the main headline needs a lexical baseline to be convincing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a paired-edit reranking task evaluated with p-MRR, a metric that compares how a document's rank changes when the instruction is edited. Each query comes with an original narrative and an edited narrative; the edit makes roughly half of the previously relevant documents newly non-relevant. p-MRR computes whether the model moved each such document down in the ranked list: a positive score means it demoted the document as instructed, zero means no change, and a negative score means it moved up. This isolates instruction-following ability from standard relevance matching, because a model that merely matches keywords can still rank the documents highly despite the edit.
What would settle it
Re-annotate a random sample of the 123 queries with fresh judges per language and measure agreement on which previously relevant documents become non-relevant under each edited instruction, because low agreement would mean the p-MRR ground truth is too unstable to support the reported conclusions.
Extended reading notes
Core claim
The central claim is that mFollowIR provides the first multilingual benchmark for instruction following in retrieval, and that current models mostly fail this test unless the instructions are in English. Using paired narrative edits, the authors isolate instruction-following from keyword matching: a good model should demote documents that become non-relevant under the edited instruction. On the cross-lingual setting, the best instruction-trained English retriever reaches an average p-MRR of 10.4, while most other models score near or below zero. In the multilingual setting, the best average p-MRR drops to 5.2, roughly half the cross-lingual best, showing a clear gap that the paper attributes to English-centric training data.
Load-bearing premise
The benchmark's validity rests on the annotators' judgments that specific documents become non-relevant under each edited instruction; no inter-annotator agreement is reported, so if those judgments are noisy the p-MRR scores do not cleanly measure instruction following.
Editorial extensions
If this is right
- English instruction-training data transfers to cross-lingual retrieval, so instruction-following ability is not fully language-specific in retrieval models.
- The multilingual gap (best p-MRR 5.2 versus 10.4 cross-lingual) indicates that multilingual instruction data, rather than model architecture alone, is the main missing ingredient.
- Cross-encoders and models with 7B or more parameters follow instructions better than smaller bi-encoders, establishing a practical baseline for future multilingual retrieval models.
- mFollowIR provides a reranking task that can be used to train or select instruction-following models for non-English search, not just to evaluate them.
- Because high nDCG does not imply high p-MRR, retrieval quality and instruction-following ability should be reported as separate dimensions in multilingual evaluation.
Reading between the lines
- The same annotation recipe could be applied to low-resource languages to test whether the cross-lingual transfer observed from English training extends beyond Russian, Chinese, and Persian.
- Because p-MRR measures only relative rank changes on a fixed pooled list, pairing mFollowIR with recall-oriented metrics could distinguish instruction-following from retrieval coverage.
- The observed cross-lingual versus multilingual gap suggests that generating multilingual instruction-following training data may be a high-leverage research direction, which the paper calls for but does not build.
- Since the benchmark is precision-focused reranking, improving mFollowIR scores could directly reduce distraction in multilingual retrieval-augmented generation systems.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces mFollowIR, a multilingual benchmark for instruction-following in retrieval, built on TREC NeuCLIR narratives for Persian, Chinese, and Russian. Human annotators edit the narratives to make them more specific, mark previously relevant documents that become newly non-relevant, translate the altered narratives into each language, and pool documents from NeuCLIR 2022/2023 systems. The authors evaluate 17 bi-encoders and cross-encoders in two settings: cross-lingual (English queries/instructions with non-English documents) and multilingual (non-English queries/instructions and documents). The primary metric is p-MRR, which measures whether newly non-relevant documents drop in the ranked list after the narrative edit; nDCG@20 is reported as a standard retrieval metric. The central findings are that instruction-trained English retrievers, especially Promptriever, achieve the highest p-MRR in the cross-lingual setting, while all models perform substantially worse in the multilingual setting.
Significance. If the benchmark is valid, mFollowIR fills a clear gap: there is currently no multilingual benchmark for instruction-following in retrieval, and the paper provides a new resource with human annotations, translations, and a broad model evaluation. The authors release the data and code, follow standard IR practices for pooling and significance testing, and evaluate both bi-encoders and cross-encoders. The observation that English instruction-training transfers only partially to multilingual settings is a useful and falsifiable finding. However, the benchmark's validity rests on the p-MRR metric cleanly isolating instruction-following from lexical matching, and on the reliability of single-annotator relevance re-judgments and single-translator narrative edits. The current manuscript does not provide sufficient evidence for either, which limits the strength of the central claim until these points are addressed.
major comments (3)
- [Section 4.1 with Section 3.1, Table 1] The paper claims in Section 4.1 that p-MRR 'sidesteps the issue of the impact of lexical overlap,' but the dataset construction does not support this. The narrative edits in Section 3.1 and Table 1 add explicit constraint terms (e.g., 'has to be related to chemicals'), and the newly non-relevant documents are, by construction, those that fail the added constraint. A model that up-weights documents containing the added terms will demote these documents and receive positive p-MRR without any semantic instruction understanding. The paper includes no lexical baseline (e.g., BM25 with the same query/instruction concatenation) against which to calibrate p-MRR. Consequently, the headline result in Table 3—Promptriever-Llama3.1 achieving 10.4 average p-MRR—may reflect added-keyword matching rather than instruction-following ability.
- [Sections 3.1 and 3.2] The validity of mFollowIR as ground truth rests on human judgments that are not shown to be reliable. Section 3.1 states that 'Two annotators with native English proficiency performed this annotation task' but does not report that any item was double-annotated or provide an inter-annotator agreement score; it is unclear whether each edit and relevance re-judgment was performed by one annotator. Section 3.2 states that each language's altered narrative was translated by a single native or fluent speaker, with no back-translation or independent verification. If these judgments are noisy or inconsistent, then cross-language differences in p-MRR in Tables 3 and 4 could reflect annotation or translation artifacts rather than model capability. An agreement measure on a subsample and a translation quality check are needed to support the benchmark's validity claim.
- [Section 3.4 and Table 2] The paper does not specify how the 123 queries (40 Persian, 43 Chinese, 40 Russian) were selected from the NeuCLIR 2022/2023 topics. The caption of Table 2 says 'We use a subset of the queries,' but no selection criteria are given. Without knowing why queries were excluded—for example, whether some were dropped because a clean half-split edit could not be found or because relevance judgments were too sparse—readers cannot assess possible selection bias or the generalizability of the benchmark to the full NeuCLIR collection.
minor comments (5)
- [Section 4.1, Equation (1)] The p-MRR formula does not specify how to handle documents that do not appear in one of the two ranked lists; please state the convention (e.g., assigning rank 1001) or excluding such documents from the computation.
- [Section 5] The significance-testing procedure should state the alpha level and whether any multiple-comparison correction was applied across the 17 models.
- [Section 3.1] The text says 'Two annotators with native English proficiency performed this annotation task' but then refers to 'the annotator' in the singular; please clarify whether each query was annotated by one person or by both.
- [References] Several references are incomplete: [21] has no year, [28] has a placeholder '(????)', and [60] is missing its venue. Please complete them.
- [Table 1] The table caption appears twice in the typeset version; please remove the duplication.
Circularity Check
No significant circularity: the p-MRR metric and edit-based protocol are defined in-paper (Eq. 1, Section 3.1), and self-citations to FollowIR supply precedent and the metric name but are not load-bearing evidence.
full rationale
The paper's central claim is a benchmark construction and an empirical evaluation, not a derivation from first principles. The load-bearing components are: (a) NeuCLIR narratives and qrels from external TREC tracks, (b) fresh human annotation of narrative edits and newly non-relevant documents (Section 3.1), (c) translations by fluent speakers (Section 3.2), and (d) p-MRR, which is fully specified in Equation (1) as a rank comparison before and after an edit. Although the paper says 'We construct this dataset following previous work in instruction following in retrieval [56]' and calls p-MRR '(from FollowIR)', the metric itself is defined in the paper and does not depend on the citation for its content. The empirical observation that some models have high nDCG with low p-MRR is supported by the paper's own Table 3, not imported from self-citation. There is no fitted parameter renamed as a prediction, no uniqueness theorem invoked from the authors' prior work, and no ansatz smuggled in via citation. A possible concern is that p-MRR may reward lexical demotion of documents containing added constraint terms, but that is a construct-validity or correctness risk, not circularity: the metric is explicitly and definitionally rank movement, and the paper does not claim that its definition is derived from some independent result. The self-citations to FollowIR and Promptriever are real prior work with stated methods, and the benchmark's labels are externally grounded in NeuCLIR. Thus the circularity burden is low; the score reflects only a minor self-citation of the FollowIR evaluation protocol.
Assumptions & free parameters
assumptions (4)
- domain assumption NeuCLIR narratives are valid complex relevance instructions and the pooled top-1000 document lists are a fair reranking candidate set.
- domain assumption A single annotator's split of relevant documents into still-relevant and newly-non-relevant is treated as correct ground truth.
- domain assumption Translations preserve the semantic content of the edited narratives across Russian, Chinese, and Persian.
- domain assumption p-MRR correctly isolates instruction-following from keyword matching.
Cite this review
Pith. "Pith review of mFollowIR: a Multilingual Benchmark for Instruction Following in Retrieval." pith.science (2026). https://pith.science/paper/IA6GLWGJ
@misc{pith2026250119264,
author = {Pith},
title = {Pith review of: mFollowIR: a Multilingual Benchmark for Instruction Following in Retrieval},
year = {2026},
howpublished = {\url{https://pith.science/paper/IA6GLWGJ}},
note = {Machine review of arXiv:2501.19264}
}
read the original abstract
Retrieval systems generally focus on web-style queries that are short and underspecified. However, advances in language models have facilitated the nascent rise of retrieval models that can understand more complex queries with diverse intents. However, these efforts have focused exclusively on English; therefore, we do not yet understand how they work across languages. We introduce mFollowIR, a multilingual benchmark for measuring instruction-following ability in retrieval models. mFollowIR builds upon the TREC NeuCLIR narratives (or instructions) that span three diverse languages (Russian, Chinese, Persian) giving both query and instruction to the retrieval models. We make small changes to the narratives and isolate how well retrieval models can follow these nuanced changes. We present results for both multilingual (XX-XX) and cross-lingual (En-XX) performance. We see strong cross-lingual performance with English-based retrievers that trained using instructions, but find a notable drop in performance in the multilingual setting, indicating that more work is needed in developing data for instruction-based multilingual retrievers.
Figures
Reference graph
Works this paper leans on
-
[1]
Abe, K., Shinden, K., Kato, M.P.: Kasys at the trec 2022 neuclir track. In: TREC (2022)
work page 2022
-
[2]
In: Workshop of the Cross-Language Evaluation Forum for European Languages, pp
Agirre, E., Nunzio, G.M.D., Ferro, N., Mandl, T., Peters, C.: Clef 2008: Ad hoc track overview. In: Workshop of the Cross-Language Evaluation Forum for European Languages, pp. 15–37, Springer (2008)
work page 2008
-
[3]
Challenges in Information-Seeking QA: Unanswerable Questions and Paragraph Retrieval
Asai, A., Choi, E.: Challenges in information-seeking qa: Unanswerable ques- tions and paragraph retrieval. arXiv preprint arXiv:2010.11915 (2020)
work page Pith review arXiv 2020
-
[4]
Asai, A., Kasai, J., Clark, J.H., Lee, K., Choi, E., Hajishirzi, H.: Xor qa: Cross-lingual open-retrieval question answering. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies, pp. 547–564 (2021)
work page 2021
-
[5]
arXiv preprint arXiv:2211.09260 (2022)
Asai, A., Schick, T., Lewis, P., Chen, X., Izacard, G., Riedel, S., Ha- jishirzi, H., Yih, W.t.: Task-aware retrieval with instructions. arXiv preprint arXiv:2211.09260 (2022)
arXiv 2022
-
[6]
In: European Conference on Information Retrieval, pp
Bassani, E.: ranx: A blazing-fast python library for ranking evaluation and comparison. In: European Conference on Information Retrieval, pp. 259–264, Springer (2022)
work page 2022
-
[7]
Bendersky, M., Croft, W.B.: Discovering key concepts in verbose queries. In: Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (2008), URL https://api.semanticscholar. org/CorpusID:2512107 14 O. Weller et al
work page 2008
-
[8]
In: WSCD ’09 (2009), URL https://api.semanticscholar
Bendersky, M., Croft, W.B.: Analysis of long queries in a large scale search log. In: WSCD ’09 (2009), URL https://api.semanticscholar. org/CorpusID:7989387
work page 2009
Show all 66 references
-
[9]
In: Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, pp
Bendersky, M., Metzler, D., Croft, W.B.: Parameterized concept weighting in verbose queries. In: Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, pp. 605– 614 (2011)
2011
-
[10]
arXiv preprint arXiv:2108.13897 (2021)
Bonifacio, L., Jeronymo, V., Abonizio, H.Q., Campiotti, I., Fadaee, M., Lotufo, R., Nogueira, R.: mmarco: A multilingual version of the ms marco passage ranking dataset. arXiv preprint arXiv:2108.13897 (2021)
2021 arXiv
-
[11]
In: ACM Sigir forum, vol
Broder, A.: A taxonomy of web search. In: ACM Sigir forum, vol. 36, pp. 3–10, ACM New York, NY, USA (2002)
2002
-
[12]
Information retrieval 10, 491–508 (2007)
Buckley, C., Dimmick, D., Soboroff, I., Voorhees, E.: Bias and the limits of pooling for large collections. Information retrieval 10, 491–508 (2007)
2007
-
[13]
ACM Transactions on Intelligent Systems and Technology 15(3), 1–45 (2024)
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al.: A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology 15(3), 1–45 (2024)
2024
-
[14]
arXiv preprint arXiv:2409.18110 (2024)
Chen, H.T., Choi, E.: Open-world evaluation for retrieving diverse perspec- tives. arXiv preprint arXiv:2409.18110 (2024)
2024 arXiv
-
[15]
Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., Liu, Z.: Bge m3- embedding: Multi-lingual, multi-functionality, multi-granularity text embed- dings through self-knowledge distillation (2024)
2024
-
[16]
Transactions of the Association for Computational Linguistics 8, 454–470 (2020)
Clark, J.H., Choi, E., Collins, M., Garrette, D., Kwiatkowski, T., Nikolaev, V., Palomaki, J.: Tydi qa: A benchmark for information-seeking question an- swering in ty pologically di verse languages. Transactions of the Association for Computational Linguistics 8, 454–470 (2020)
2020
-
[17]
arXiv preprint arXiv:2003.07820 (2020)
Craswell, N., Mitra, B., Yilmaz, E., Campos, D., Voorhees, E.M.: Overview of the TREC 2019 deep learning track. arXiv preprint arXiv:2003.07820 (2020)
2020 arXiv
-
[18]
In: Trec, vol
Dang, H.T., Kelly, D., Lin, J., et al.: Overview of the trec 2007 question answering track. In: Trec, vol. 7, p. 63 (2007)
2007
-
[19]
In: Workshop of the Cross-Language Evaluation Forum for European Lan- guages, pp
Ferro, N., Peters, C.: Clef 2009 ad hoc track overview: Tel and persian tasks. In: Workshop of the Cross-Language Evaluation Forum for European Lan- guages, pp. 13–35, Springer (2009)
2009
-
[20]
In: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp
Gupta, M., Bendersky, M.: Information retrieval with verbose queries. In: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1121–1124 (2015)
2015
-
[21]
Huang, Z., Yu, P., Allan, J.: Umass at trec 2023 neuclir track (????)
2023
-
[22]
arXiv preprint arXiv:2112.09118 (2021)
Izacard, G., Caron, M., Hosseini, L., Riedel, S., Bojanowski, P., Joulin, A., Grave, E.: Unsupervised dense information retrieval with contrastive learn- ing. arXiv preprint arXiv:2112.09118 (2021)
2021 arXiv
-
[23]
arXiv preprint arXiv:2303.16145 (2023) mFollowIR: Multilingual Instruction Following in Retrieval 15
Jeronymo, V., Lotufo, R., Nogueira, R.: Neuralmind-unicamp at 2022 trec neuclir: Large boring rerankers for cross-lingual retrieval. arXiv preprint arXiv:2303.16145 (2023) mFollowIR: Multilingual Instruction Following in Retrieval 15
2023 arXiv
-
[24]
arXiv preprint arXiv:2310.06825 (2023)
Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., Casas, D.d.l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al.: Mistral 7b. arXiv preprint arXiv:2310.06825 (2023)
2023 arXiv
-
[25]
In: TREC (2022)
Ju, J.H., Chen, W.C., Chang, H.T., Lin, C.W., Tsai, M.F., Wang, C.J.: Cfda & clip at trec 2022 neuclir track. In: TREC (2022)
2022
-
[26]
In: TREC (2022)
Kamalloo, E., Alfonso-Hermelo, D., Rezagholizadeh, M.: Huawei noah’s ark lab at trec neuclir 2022. In: TREC (2022)
2022
-
[27]
arXiv preprint arXiv:2004.04906 (2020)
Karpukhin, V., O˘ guz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., Yih, W.t.: Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906 (2020)
2020 arXiv
-
[28]
Lassance, C., Pradeep, R., Lin, J.: Naverloo@ trec deep learning and (????)
-
[29]
Lawrie, D., MacAvaney, S., Mayfield, J., McNamee, P., Oard, D.W., Sol- daini, L., Yang, E.: Overview of the TREC 2022 NeuCLIR track (2023)
2023
-
[30]
Lawrie, D., MacAvaney, S., Mayfield, J., McNamee, P., Oard, D.W., Sol- daini, L., Yang, E.: Overview of the trec 2023 neuclir track (2024)
2024
-
[31]
arXiv preprint arXiv:2409.15700 (2024)
Li, C., Qin, M., Xiao, S., Chen, J., Luo, K., Shao, Y., Lian, D., Liu, Z.: Making text embedders few-shot learners. arXiv preprint arXiv:2409.15700 (2024)
2024 arXiv
-
[32]
In: International Conference on Machine Learning, pp
Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H.W., Tay, Y., Zhou, D., Le, Q.V., Zoph, B., Wei, J., et al.: The flan collection: Designing data and methods for effective instruction tuning. In: International Conference on Machine Learning, pp. 22631–22648, PMLR (2023)
2023
-
[33]
arXiv preprint arXiv:2310.08319 (2023)
Ma, X., Wang, L., Yang, N., Wei, F., Lin, J.: Fine-tuning llama for multi- stage text retrieval. arXiv preprint arXiv:2310.08319 (2023)
2023 arXiv
-
[34]
arXiv preprint arXiv:2402.09906 (2024)
Muennighoff, N., Su, H., Wang, L., Yang, N., Wei, F., Yu, T., Singh, A., Kiela, D.: Generative representational instruction tuning. arXiv preprint arXiv:2402.09906 (2024)
2024 arXiv
-
[35]
arXiv preprint arXiv:2210.07316 (2022)
Muennighoff, N., Tazi, N., Magne, L., Reimers, N.: Mteb: Massive text em- bedding benchmark. arXiv preprint arXiv:2210.07316 (2022)
2022 arXiv
-
[36]
In: European Conference on Information Retrieval, pp
Nair, S., Yang, E., Lawrie, D., Duh, K., McNamee, P., Murray, K., Mayfield, J., Oard, D.W.: Transfer learning approaches for building cross-language dense retrieval models. In: European Conference on Information Retrieval, pp. 382–396, Springer (2022)
2022
-
[37]
CoRR abs/1611.09268 (2016), URL http://arxiv.org/ abs/1611.09268
Nguyen, T., Rosenberg, M., Song, X., Gao, J., Tiwary, S., Majumder, R., Deng, L.: MS MARCO: A human generated machine reading comprehen- sion dataset. CoRR abs/1611.09268 (2016), URL http://arxiv.org/ abs/1611.09268
2016 arXiv
-
[38]
arXiv preprint arXiv:2402.01613 (2024)
Nussbaum, Z., Morris, J.X., Duderstadt, B., Mulyar, A.: Nomic em- bed: Training a reproducible long context text embedder. arXiv preprint arXiv:2402.01613 (2024)
2024 arXiv
-
[39]
arXiv preprint arXiv:2402.14334 (2024)
Oh, H., Lee, H., Ye, S., Shin, H., Jang, H., Jun, C., Seo, M.: Instructir: A benchmark for instruction following of information retrieval models. arXiv preprint arXiv:2402.14334 (2024)
2024 arXiv
-
[40]
Weller et al
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P.F., Leike, 16 O. Weller et al. J., Lowe, R.: Training lang...
2022
-
[41]
Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, Semih Yavuz: Sfr-embedding-2: Advanced text embedding with multi-stage training (2024)
2024
-
[42]
In: International Conference on Learning Representations (2022), URL https: //openreview.net/forum?id=9Vrb9D0WI4
Sanh, V., Webson, A., Raffel, C., Bach, S., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Raja, A., Dey, M., Bari, M.S., Xu, C., Thakker, U., Sharma, S.S., Szczechla, E., Kim, T., Chhablani, G., Nayak, N., Datta, D., Chang, J., Jiang, M.T.J., Wang, H., Manica, M., She...
2022
-
[43]
In: International Conference on Machine Learning, pp
Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E.H., Sch¨ arli, N., Zhou, D.: Large language models can be easily distracted by irrelevant con- text. In: International Conference on Machine Learning, pp. 31210–31227, PMLR (2023)
2023
-
[44]
In: 30th Text REtrieval Conference
Soboroff, I.: Overview of trec 2021. In: 30th Text REtrieval Conference. Gaithersburg, Maryland (2021)
2021
-
[45]
In: Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, pp
Soboroff, I., Robertson, S.: Building a filtering test collection for trec 2002. In: Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, pp. 243–250 (2003)
2003
-
[46]
de Souza P. Moreira, G., Osmulski, R., Xu, M., Ak, R., Schifferer, B., Oldridge, E.: NV-Retriever: Improving text embedding models with effective hard-negative mining (2024), URL https://arxiv.org/abs/2407.15831
2024 arXiv
-
[47]
Su, H., Shi, W., Kasai, J., Wang, Y., Hu, Y., Ostendorf, M., Yih, W.t., Smith, N.A., Zettlemoyer, L., Yu, T.: One embedder, any task: Instruction-finetuned text embeddings (2022), URL https://arxiv.org/abs/2212.09741
2022 arXiv
-
[48]
Sun, W., Shi, Z., Wu, J., Yan, L., Ma, X., Liu, Y., Cao, M., Yin, D., Ren, Z.: Mair: A massive benchmark for evaluating instructed retrieval
-
[49]
In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in In- formation Retrieval, pp
Thomas, P., Spielman, S., Craswell, N., Mitra, B.: Large language mod- els can accurately predict searcher preferences. In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in In- formation Retrieval, pp. 1930–1940 (2024)
2024
-
[50]
arXiv preprint arXiv:2302.13971 (2023)
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozi` ere, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and ef- ficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)
2023 arXiv
-
[51]
In: SIGIR, ACM (2018)
Van Gysel, C., de Rijke, M.: Pytrec eval: An extremely fast python interface to trec eval. In: SIGIR, ACM (2018)
2018
-
[52]
Na- ture methods 17(3), 261–272 (2020) mFollowIR: Multilingual Instruction Following in Retrieval 17
Virtanen, P., Gommers, R., Oliphant, T.E., Haberland, M., Reddy, T., Cour- napeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., et al.: Scipy 1.0: fundamental algorithms for scientific computing in python. Na- ture methods 17(3), 261–272 (2020) mFollowIR: Multili...
2020
-
[53]
In: TREC (1996)
Voorhees, E., Harman, D.: Overview of the fifth text retrieval conference (trec-5). In: TREC (1996)
1996
-
[54]
arXiv preprint arXiv:2212.03533 (2022)
Wang, L., Yang, N., Huang, X., Jiao, B., Yang, L., Jiang, D., Majumder, R., Wei, F.: Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533 (2022)
2022 arXiv
-
[55]
arXiv preprint arXiv:2401.00368 (2023)
Wang, L., Yang, N., Huang, X., Yang, L., Majumder, R., Wei, F.: Im- proving text embeddings with large language models. arXiv preprint arXiv:2401.00368 (2023)
2023 arXiv
-
[56]
arXiv preprint arXiv:2403.15246 (2024)
Weller, O., Chang, B., MacAvaney, S., Lo, K., Cohan, A., Van Durme, B., Lawrie, D., Soldaini, L.: FollowIR: Evaluating and teaching information re- trieval models to follow instructions. arXiv preprint arXiv:2403.15246 (2024)
2024 arXiv
-
[57]
Conference of the European Chapter of the Association for Computational Linguistics (2024), URL https://api.semanticscholar
Weller, O., Lawrie, D.J., Durme, B.V.: Nevir: Negation in neural informa- tion retrieval. Conference of the European Chapter of the Association for Computational Linguistics (2024), URL https://api.semanticscholar. org/CorpusID:258676146
2024
-
[58]
arXiv preprint arXiv:2011.08115 (2020)
Weller, O., Lourie, N., Gardner, M., Peters, M.E.: Learning from task de- scriptions. arXiv preprint arXiv:2011.08115 (2020)
2020 arXiv
-
[59]
arXiv preprint arXiv:2409.11136 (2024)
Weller, O., Van Durme, B., Lawrie, D., Paranjape, A., Zhang, Y., Hessel, J.: Promptriever: Instruction-trained retrievers can be prompted like language models. arXiv preprint arXiv:2409.11136 (2024)
2024 arXiv
-
[60]
arXiv preprint arXiv:2404.08118 (2024)
Yang, E., Lawrie, D., Mayfield, J.: HLTCOE at TREC 2023 NeuCLIR track. arXiv preprint arXiv:2404.08118 (2024)
2024 arXiv
-
[61]
arXiv preprint arXiv:2310.01558 (2023)
Yoran, O., Wolfson, T., Ram, O., Berant, J.: Making retrieval- augmented language models robust to irrelevant context. arXiv preprint arXiv:2310.01558 (2023)
2023 arXiv
-
[62]
arXiv preprint arXiv:2406.17744 (2024)
Yuan, W., Kulikov, I., Yu, P., Cho, K., Sukhbaatar, S., Weston, J., Xu, J.: Following length constraints in instructions. arXiv preprint arXiv:2406.17744 (2024)
2024 arXiv
-
[63]
Transactions of the Associ- ation for Computational Linguistics 11, 1114–1131 (09 2023), ISSN 2307- 387X
Zhang, X., Thakur, N., Ogundepo, O., Kamalloo, E., Alfonso-Hermelo, D., Li, X., Liu, Q., Rezagholizadeh, M., Lin, J.: MIRACL: A Multilingual Re- trieval Dataset Covering 18 Diverse Languages. Transactions of the Associ- ation for Computational Linguistics 11, 1114–1131 (09 202...
2023
-
[64]
Zhang, X., Zhang, Y., Long, D., Xie, W., Dai, Z., Tang, J., Lin, H., Yang, B., Xie, P., Huang, F., Zhang, M., Li, W., Zhang, M.: mgte: Generalized long-context text representation and reranking models for multilingual text retrieval (2024), URL https://arxiv.org/abs/2407.19669
2024 arXiv
-
[65]
arXiv preprint arXiv:2405.02714 (2024)
Zhao, X., Chen, T., Chen, S., Zhang, H., Wu, T.: Beyond relevance: Evaluate and improve retrievers on perspective awareness. arXiv preprint arXiv:2405.02714 (2024)
2024 arXiv
-
[66]
arXiv preprint arXiv:2311.07911 (2023)
Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., Hou, L.: Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911 (2023)
2023 arXiv
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.