Pith. sign in

REVIEW 5 minor 33 references

Artifact Sharing for Information Retrieval Research

T0 review · 0 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single serialization format with a small metadata file gives Information Retrieval researchers a flexible way to share pre-built indexes, caches, and other artifacts across different hosting platforms.

desk verdict A genuinely useful engineering contribution for sharing IR artifacts, with a working public implementation; the fallback adapter concern is real but minor. read the letter →

arxiv 2505.05434 v1 pith:S7NJXL6U submitted 2025-05-08 cs.IR

Sean MacAvaney This is my paper · ORCID
classification cs.IR
keywords ReproducibilityArtifactreuseGreenIRInformationretrievalSerializationformatMetadataPre-builtindexesResearchinfrastructure
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that Information Retrieval research lacks a consensus way to share computational artifacts other than code and trained models, and that this gap blocks reproducibility and slows collaboration. It presents a system in which any artifact—a built search index, a cache, a corpus graph—can be packed into one compressed archive carrying a small JSON metadata file, uploaded to any supported host, and later downloaded and loaded as a ready-to-use Python object. The paper argues that this design is flexible enough to cover virtually every artifact type and hosting option, and demonstrates it with 14 artifact classes and 7 hosts, including over 100 artifacts already shared through one platform and use in a cross-institutional collaboration. A sympathetic reader would take the contribution to be a practical, low-overhead standard that makes sharing pre-built indexes as routine as sharing trained models.

What carries the argument

The Artifact Serialization File is the central mechanism: a TAR archive compressed with LZ4, optionally containing a pt_meta.json file with required type and format fields and an optional package_hint. This format carries the argument because the type/format pair lets the system automatically pick the correct artifact class, and the package_hint tells a user which extension to install when no class is available. Registration happens through Python entry points, so new artifact classes and new hosts can be added without changing the core package, and segmented archives handle hosts with per-file size limits. When the metadata file is absent, metadata-adapter entry points inspect file contents as a fallback.

What would settle it

Take a set of legacy pre-built indexes in several common formats, created before the system existed, remove any metadata file from the archive, and attempt to load each through the one-command interface; if any is assigned the wrong artifact class or fails to load, the interoperability claim for pre-existing artifacts is falsified for that case.

Watch

Extended reading notes

Core claim

The central claim is that a single serialization format, built on a compressed TAR file with an optional pt_meta.json metadata file, can unify artifact sharing across the field. The metadata file names the artifact's type and format and suggests a Python package to load it; the system uses those fields to select the matching artifact class, which hands back an object that is immediately ready for retrieval or other use. For artifacts created before this metadata existed, content-inspecting adapters try to infer the type and format as a fallback. The paper reports that the system currently supports 14 artifact classes written in Python, Java, C++, and Rust, and 7 hosts including general research repositories, a model-sharing hub, institutional repositories, peer-to-peer transfer, and several IR-specific index repositories, and that it has already been used to share over 100 artifacts and to support cross-institutional collaborations.

Load-bearing premise

The load-bearing premise is that the metadata-based type/format detection, especially the fallback content-inspection adapters, correctly identifies artifacts that lack the new pt_meta.json file; the paper itself notes in Section 3.1 that these checks are potentially error-prone and more expensive than loading metadata.

Editorial extensions

If this is right

  • Researchers can share pre-built indexes and caches directly from their own toolkits, without asking a platform maintainer to host them.
  • Reproducibility studies can validate intermediate steps, not just final rankings, by loading the exact index a paper used.
  • Collaborations can exchange work-in-progress artifacts privately or peer-to-peer, reducing ad hoc file transfers.
  • Reusing already-built indexes and caches lowers the computational and energy cost of IR research, a goal the paper ties to green IR.
  • A common format across index engines makes it easier to compare systems that use different underlying implementations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If this format becomes widely adopted, artifact discoverability could come to resemble model hubs: searchable tags, generated documentation, and access control would make pre-built indexes first-class research objects.
  • The same serialization-plus-metadata pattern could be exported to other data-heavy research fields that lack a sharing consensus, such as precomputed embeddings or knowledge graphs.
  • The fallback adapters are the most fragile part of the design; measuring their error rate on a corpus of legacy artifacts would tell whether the low-overhead claim holds for pre-existing data.
  • A natural stress test is scale: segmenting archives works for per-file limits, but hosts that cap total repository size or bandwidth will still bound what a single researcher can share.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

0 major / 5 minor

Summary. This demonstration paper introduces a system for sharing IR research artifacts, integrated into PyTerrier. The core contribution is an artifact serialization file (a compressed TAR archive with optional pt_meta.json metadata) that can be uploaded to and downloaded from multiple hosts (HuggingFace Hub, Zenodo, CIFF Hub, TIREx, and arbitrary URLs) and instantiated as typed artifact classes for sparse indexes, dense indexes, auxiliary structures, and caches. The paper illustrates the workflow with short code examples, claims broad flexibility and interoperability, and positions the system relative to prior efforts such as PyTerrier, Pyserini, TIREx, BM25S, and CIFF.

Significance. If the system works as demonstrated, it addresses a real gap in IR practice: the lack of a consensus mechanism for sharing non-model artifacts such as built indexes. The design is modular and sensible, using Python entry points for extensible artifact classes and hosts, and it builds on existing platforms rather than requiring a new infrastructure. The paper's central claim is supported by a public implementation, a Google Colab notebook, and a live demonstration, all of which strengthen the credibility of the demonstration. The main caveat is that interoperability with legacy artifacts, achieved via content-inspecting metadata adapters, is explicitly acknowledged as potentially error-prone and is not evaluated; this is a limitation but not a refutation of the central claim.

minor comments (5)
  1. [Section 1 and Conclusion] Section 1 states that the system currently supports 13 artifact classes, while Table 1 lists 14 artifact classes and the Conclusion says 'provides 14 Artifact Classes'; please reconcile this discrepancy.
  2. [Section 3.1] The metadata-adapter fallback is acknowledged as 'potentially error-prone'; since this is the only path for loading artifacts that lack pt_meta.json, adding a brief evaluation of adapter precision or a more explicit qualifier on the interoperability claim would strengthen the paper.
  3. [Section 3.3] In the Magic-Wormhole paragraph, 'PyTrrier' should be 'PyTerrier'.
  4. [Keywords and Conclusion] The keyword 'Reproduciblity' is misspelled; the Conclusion also contains 'envirionmental' instead of 'environmental'.
  5. [Section 3.1] The phrase 'a artifact.tar.lz4.json file' should be 'an artifact.tar.lz4.json file'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an implementation/demonstration whose claims are backed by public code and live examples, not by a self-referential derivation.

full rationale

This is a systems and demonstration paper, not a derivation or empirical prediction paper. The central claim—that the artifact-sharing format provides flexible and interoperable sharing—is supported by the described implementation, public repository, Google Colab notebook, and a live demonstration, all of which are externally checkable. When the paper defines its own serialization format with a required metadata file, the fact that loading works for artifacts carrying pt_meta.json is true by construction; that is legitimate for a format specification, not circularity. The only genuinely weak link, the fallback metadata-adapter path for legacy artifacts, is explicitly disclosed in Section 3.1: the adapters 'match an artifact to its corresponding metadata by inspecting its contents,' and the paper concedes these checks 'are potentially error-prone and are more expensive than loading the metadata from a file.' That is an acknowledged engineering limitation and an unmeasured risk, not a circular step: the fallback does not define the artifact class in terms of the target claim, and metadata-bearing artifacts bypass it entirely. The self-citations (PyTerrier, TIREx, and related use cases such as [1, 8, 26-28]) are contextual prior work and integration points; none is invoked as a uniqueness theorem or as the sole evidence that the system works. No fitted parameter is renamed as a prediction, no equation reduces to its own input, and no alternative is excluded by citation. Accordingly, the appropriate finding is no significant circularity, score 0.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

No free parameters, axioms, or invented entities are introduced. The claim is an implementation claim about a software system, relying on standard software engineering practices (TAR/LZ4 archiving, Python entry points) and the availability of third-party hosting platforms. None of these are new theoretical postulates that require an axiom ledger entry.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Artifact Sharing for Information Retrieval Research." pith.science (2026). https://pith.science/paper/S7NJXL6U

@misc{pith2026250505434,
  author       = {Pith},
  title        = {Pith review of: Artifact Sharing for Information Retrieval Research},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S7NJXL6U}},
  note         = {Machine review of arXiv:2505.05434}
}
read the original abstract

Sharing artifacts -- such as trained models, pre-built indexes, and the code to use them -- aids in reproducibility efforts by allowing researchers to validate intermediate steps and improves the sustainability of research by allowing multiple groups to build off one another's prior computational work. Although there are de facto consensuses on how to share research code (through a git repository linked to from publications) and trained models (via HuggingFace Hub), there is no consensus for other types of artifacts, such as built indexes. Given the practical utility of using shared indexes, researchers have resorted to self-hosting these resources or performing ad hoc file transfers upon request, ultimately limiting the artifacts' discoverability and reuse. This demonstration introduces a flexible and interoperable way to share artifacts for Information Retrieval research, improving both their accessibility and usability.

Figures

Figures reproduced from arXiv: 2505.05434 by the authors.

Figure 1
Figure 1. An overview of the distributed design for sharing [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 10 canonical work pages

  1. [1]

    Xuejun Chang, Debabrata Mishra, Craig Macdonald, and Sean MacAvaney. 2024. Neural Passage Quality Estimation for Static Pruning. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024 , Grace Hui Yang, Hongning Wang, Sam Han, Claudia Hauff, Guido ...

    arXiv 2024

  2. [2]

    Ryan Clancy, Nicola Ferro, Claudia Hauff, Jimmy Lin, Tetsuya Sakai, and Ze Zhong Wu. 2019. The SIGIR 2019 Open-Source IR Replicability Challenge (OSIRRC 2019). In Proceedings of the 42nd International ACM SIGIR Confer- ence on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019 , Benjamin Piwowarski, Max Chevalie...

    work page arXiv 2019

  3. [3]

    Kaustubh D. Dhole. 2024. PyTerrier-GenRank: The PyTerrier Plugin for Reranking with Large Language Models. CoRR abs/2412.05339 (2024). https://doi.org/10. 48550/ARXIV.2412.05339 arXiv:2412.05339

    arXiv doi:10.48550/arxiv.2412.05339 2024

  4. [4]

    Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2024. The Faiss library. CoRR abs/2401.08281 (2024). https://doi.org/10.48550/ARXIV. 2401.08281 arXiv:2401.08281

    arXiv doi:10.48550/arxiv.2401.08281 2024

  5. [5]

    European Organization For Nuclear Research and OpenAIRE. 2013. Zenodo. https://doi.org/10.25495/7GXK-RD71

    doi:10.25495/7gxk-rd71 2013

  6. [6]

    Maik Fröbe, Jan Heinrich Reimer, Sean MacAvaney, Niklas Deckers, Simon Reich, Janek Bevendorff, Benno Stein, Matthias Hagen, and Martin Potthast. 2023. The In- formation Retrieval Experiment Platform. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2023, Taipei, Taiwan, July 23-27, ...

    arXiv 2023

  7. [7]

    Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar. 2020. Accelerating Large-Scale Inference with Anisotropic Vector Quantization. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event (Proceedings of Machine Learning Research, Vol. 119) . PMLR, 3887–389...

    work page 2020

  8. [8]

    Hrishikesh Kulkarni, Nazli Goharian, Ophir Frieder, and Sean MacAvaney. 2024. LexBoost: Improving Lexical Document Retrieval with Nearest Neighbors. In Proceedings of the ACM Symposium on Document Engineering 2024, DocEng 2024, San Jose, CA, USA, August 20-23, 2024 . ACM, 16:1–16:10. https://doi.org/10.1145/ 3685650.3685658

    work page arXiv 2024

Show all 33 references
  1. [9]

    Hrishikesh Kulkarni, Sean MacAvaney, Nazli Goharian, and Ophir Frieder. 2023. Lexically-Accelerated Dense Retrieval. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2023, Taipei, Taiwan, July 23-27, 2023...

    2023

  2. [10]

    Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Frassetto Nogueira. 2021. Pyserini: A Python Toolkit for Repro- ducible Information Retrieval Research with Sparse and Dense Representations. In SIGIR ’21: The 44th International ACM SIGIR Con...

    2021

  3. [11]

    Mackenzie, Chris Kamphuis, Craig Macdonald, Antonio Mallia, Michal Siedlaczek, Andrew Trotman, and Arjen P

    Jimmy Lin, Joel M. Mackenzie, Chris Kamphuis, Craig Macdonald, Antonio Mallia, Michal Siedlaczek, Andrew Trotman, and Arjen P. de Vries. 2020. Supporting Interoperability Between Open-Source Search Engines with the Common Index File Format. In Proceedings of the 43rd Internati...

    2020

  4. [12]

    Xing Han Lù. 2024. BM25S: Orders of magnitude faster lexical search via eager sparse scoring. CoRR abs/2407.03618 (2024). https://doi.org/10.48550/ARXIV. 2407.03618 arXiv:2407.03618

    2024 doi arXiv

  5. [13]

    Sean MacAvaney and Craig Macdonald. 2022. A Python Interface to PISA!. In SIGIR ’22: The 45th International ACM SIGIR Conference on Research and Develop- ment in Information Retrieval, Madrid, Spain, July 11 - 15, 2022 , Enrique Amigó, Pablo Castells, Julio Gonzalo, Ben Carter...

    2022

  6. [14]

    Sean MacAvaney and Craig Macdonald. 2025. On Precomputation and Caching in Information Retrieval Experiments with Pipeline Architectures. In Proceedings of the 2nd International Workshop on Open Web Search (WOWS) co-located with 47th European Conference on Information Retrieva...

    2025

  7. [15]

    Dhole, Leif Azzopardi, Maik Fröbe, Marc Bertin, Nishchal Prasad, Saber Zerhoudi, Shuai Wang, Shubham Chatterjee, Thomas Jänich, Udo Kruschwitz, Xi Wang, and Zijun Long

    Sean MacAvaney, Adam Roegiest, Aldo Lipani, Andrew Parry, Björn Engelmann, Christin Katharina Kreutz, Chuan Meng, Erlend Frayling, Eugene Yang, Ferdinand Schlatt, Guglielmo Faggioli, Harrisen Scells, Iana Atanassova, Jana Friese, Janek Bevendorff, Javier Sanz-Cruzado, Johanne ...

    2024

  8. [16]

    Sean MacAvaney, Nicola Tonellotto, and Craig Macdonald. 2022. Adaptive Re- Ranking with a Corpus Graph. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, Atlanta, GA, USA, October 17-21, 2022, Mohammad Al Hasan and Li Xiong (Eds.). ...

    2022

  9. [17]

    Craig Macdonald, Jinyuan Fang, Andrew Parry, Craig Macdonald, and Zaiqiao Meng. 2025. Constructing and Evaluating Declarative RAG Pipelines in PyTerrier. In Proceedings of the 48th International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2...

    2025

  10. [18]

    Craig Macdonald, Nicola Tonellotto, Sean MacAvaney, and Iadh Ounis. 2021. PyTerrier: Declarative Experimentation in Python from BM25 to Dense Retrieval. In CIKM ’21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Austr...

    2021

  11. [19]

    Mackenzie, and Torsten Suel

    Antonio Mallia, Michal Siedlaczek, Joel M. Mackenzie, and Torsten Suel. 2019. PISA: Performant Indexes and Search for Academia. In Proceedings of the Open- Source IR Replicability Challenge co-located with 42nd International ACM SIGIR Conference on Research and Development in ...

    2019

  12. [20]

    Antonio Mallia, Torsten Suel, and Nicola Tonellotto. 2024. Faster Learned Sparse Retrieval with Block-Max Pruning. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 20...

    2024

  13. [21]

    Jan Heinrich Merker, Janek Bevendorff, Maik Fröbe, Tim Hagen, Harrisen Scells, Matti Wiegmann, Benno Stein, Matthias Hagen, and Martin Potthast. 2025. Web- Scale Retrieval Experimentation with chatnoir-pyterrier. In Advances in Infor- mation Retrieval - 47th European Conferenc...

    2025

  14. [22]

    Morris and Alexander M

    John X. Morris and Alexander M. Rush. 2024. Contextual Document Embed- dings. CoRR abs/2410.02525 (2024). https://doi.org/10.48550/ARXIV.2410.02525 arXiv:2410.02525

    2024 doi arXiv

  15. [23]

    Blaise Munyampirwa, Vihan Lakshman, and Benjamin Coleman. 2024. Down with the Hierarchy: The ’H’ in HNSW Stands for "Hubs". CoRR abs/2412.01940 (2024). https://doi.org/10.48550/ARXIV.2412.01940 arXiv:2412.01940

    2024 doi arXiv

  16. [24]

    Thong Nguyen, Sean MacAvaney, and Andrew Yates. 2023. A Unified Framework for Learned Sparse Retrieval. In Advances in Information Retrieval - 45th European Conference on Information Retrieval, ECIR 2023, Dublin, Ireland, April 2-6, 2023, Proceedings, Part III (Lecture Notes i...

    2023

  17. [25]

    Iadh Ounis, Gianni Amati, Vassilis Plachouras, Ben He, Craig Macdonald, and Douglas Johnson. 2005. Terrier Information Retrieval Platform. In Advances in Information Retrieval, 27th European Conference on IR Research, ECIR 2005, Santiago de Compostela, Spain, March 21-23, 2005...

    2005 doi

  18. [26]

    Mandeep Rathee, Sean MacAvaney, and Avishek Anand. 2025. Guiding Retrieval Using LLM-Based Listwise Rankers. In Advances in Information Retrieval - 47th European Conference on Information Retrieval, ECIR 2025, Lucca, Italy, April 6-10, 2025, Proceedings, Part I (Lecture Notes ...

    2025

  19. [27]

    Mandeep Rathee, Sean MacAvaney, and Avishek Anand. 2025. Quam: Adaptive Retrieval through Query Affinity Modelling. In Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining, WSDM 2025, Hannover, Germany, March 10-14, 2025, Wolfgang Nejdl, Sör...

    2025

  20. [28]

    Mandeep Rathee, V Venktesh, Sean MacAvaney, and Avishek Anand. 2025. Break- ing the Lens of the Telescope: Online Relevance Estimation over Large Re- trieval Sets. In Proceedings of the 48th International ACM SIGIR conference on SIGIR ’25, July 13–18, 2025, Padua, Italy MacAva...

    2025

  21. [30]

    Voorhees

    Ellen M. Voorhees. 2020. Coopetition in IR research. SIGIR Forum 54, 2 (2020), 1:1–1:3. https://doi.org/10.1145/3483382.3483384

    2020

  22. [31]

    Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R

    Ellen M. Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R. Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. 2020. TREC-COVID: constructing a pandemic information retrieval test collection. SIGIR Forum 54, 1 (2020), 1:1–1:12. https://doi.org/1...

    2020

  23. [32]

    Murdick, Devvret Rishi, Jerry Sheehan, Zhihong Shen, Brandon Stilson, Alex D

    Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Darrin Eide, Kathryn Funk, Rodney Kinney, Ziyang Liu, William Mer- rill, Paul Mooney, Dewey A. Murdick, Devvret Rishi, Jerry Sheehan, Zhihong Shen, Brandon Stilson, Alex D. Wade, Kuansan Wang, Chris ...

    2020 arXiv

  24. [33]

    Xiao Wang, Sean MacAvaney, Craig Macdonald, and Iadh Ounis. 2022. An Inspection of the Reproducibility and Replicability of TCT-ColBERT. In SIGIR ’22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 - 15,...

    2022

  25. [34]

    Peilin Yang, Hui Fang, and Jimmy Lin. 2018. Anserini: Reproducible Ranking Baselines Using Lucene. ACM J. Data Inf. Qual. 10, 4 (2018), 16:1–16:20. https: //doi.org/10.1145/3239571

    2018 doi

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.