Pith. sign in

REVIEW 2 major objections 4 minor 63 references

The paper claims that per-domain contamination in multi-domain retrieval can be certified at inference time without query labels, with abstention instead of silent violation when the router is too weak.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 05:45 UTC pith:UIPH7ZKR

load-bearing objection C3R is a genuinely new and honestly-reported combination of RCPS and router-error correction that certifies per-domain contamination with abstention, limited mainly by the usual exchangeability assumption and the lack of any shift detection. the 2 major comments →

arxiv 2607.14157 v1 pith:UIPH7ZKR submitted 2026-07-14 cs.LG cs.AIcs.IRcs.SYeess.SY

Certified Domain Consistency for Multi-Domain Retrieval: Label-Free Per-Domain Contamination Control with Conformal Risk Guarantees

Jayakumar Manoharan This is my paper
classification cs.LG cs.AIcs.IRcs.SYeess.SY
keywords multi-domain retrievalcontaminationconformal risk controllabel-free inferenceper-domain certificateretrieval-augmented generationsoft demotioncertified guarantees
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper seeks to certify, per domain and without any query-time label, an upper bound on the fraction of top-ranked retrieved documents whose source domain differs from the query's domain. Existing conformal risk control only certifies a marginal average, which empirically leaves the most contaminated domain far over budget. The proposed layer splits calibration data into two independent parts, uses the first to estimate how often the inferred-domain router misroutes and how impure its inferred groups are, and uses the second to pick per-domain demotion thresholds under conformal risk control. The result is a finite-sample transfer bound from the inferred domain to the true domain whose slack is fully estimable; when the requested budget lies below the router's error floor, the layer abstains rather than violating. A sympathetic reader would care because wrong-domain evidence in retrieval-augmented pipelines can carry operational or legal authority.

Core claim

The central claim is Theorem 1: with probability at least 1−γ over both calibration splits, simultaneously for every domain c, the expected contamination of top-K results conditioned on the query's true domain is at most (1−ρ̄_c)·α_c/(1−π̄_c)+ρ̄_c·B, where α_c is the level at which group-level conformal risk control certifies the inferred-domain risk, ρ̄_c and π̄_c are one-sided upper bounds on the router's misrouting rate and group impurity, and B is the loss bound. All slack terms are computed from data, so a weak router widens the certificate visibly rather than causing a silent violation. Inverting the bound yields a budget for the internal run, and when that budget is non-positive the m

What carries the argument

The load-bearing mechanism is a two-split latent-group risk-control scheme. Split D1 yields one-sided binomial upper bounds on the router's per-domain misrouting rate ρ_c and inferred-group impurity π_c; split D2, independent of D1, runs conformal risk control inside each inferred-domain group to choose the largest demotion threshold whose risk is certified below a rescaled budget. Demotion is soft—a continuous posterior-overlap mismatch score m(q,d) with a per-domain threshold—so it removes the fixed error floor of hard label filtering while sharing the irreducible router-error floor ρ̄_c B. The budget-rescaling corollary inverts the certificate, mapping a requested target to the internal r

Load-bearing premise

The load-bearing premise is that calibration and deployment queries are exchangeable draws from one fixed query pool, with the frozen domain router not breaking exchangeability; if the deployment stream's domain mix or style shifts, the certificate silently loses validity.

What would settle it

Hold out a genuinely fresh query stream drawn from a different domain distribution than the calibration pool, serve it through the method, and count how often the realized per-domain contamination exceeds the issued certificate; a violation rate above the nominal γ (e.g., >0.1) would falsify the exchangeability assumption. A second check: recompute certificates after deliberately corrupting split independence (e.g., reusing D1 to pick thresholds); if violations appear, the non-circularity claim is the cause.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the central inequality is correct, a deployer can publish for each domain a certificate of the form 'expected contamination ≤ α' that holds simultaneously across domains at level 1−γ, with no query-time domain label.
  • Marginal conformal risk control fails exactly on the domains the certificate is meant to protect: in the paper's experiments it violates the most-contaminated domain in 100% of resampled calibrations, while the certified layer never violates.
  • Where budgets are feasible, soft demotion retains far more recall than the strongest calibrated hard filter at equal certified contamination—up to roughly six-fold on a contaminated domain at a tight budget—and ties on clean domains.
  • Contamination is shown to be orthogonal to ranking quality: a stronger reranker improves ranking but leaves contamination essentially unchanged, so the certified layer is a necessary complement rather than a proxy.
  • The certificate transfers to genuinely different corpora, including public US federal regulations, with zero violations across 1000 resampled calibrations at every point of a contamination spectrum.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Extension: because the certificate bounds provenance mismatch, and wrong-authority errors are a subset of cross-domain provenance mismatches, the certificate also upper-bounds the rate at which retrieved evidence cites the wrong authority—so it can serve as an audit signal even when downstream readers are robust.
  • Extension: the certificate turns router quality into an operational dial: improving the domain probe tightens every feasible certificate and converts abstaining domains into servable ones, giving a quantitative target for probe investment.
  • Extension: the same two-split transfer argument could extend to other bounded retrieval losses beyond contamination—for example, jointly certifying contamination and relevance—or to online recalibration under distribution shift, both of which the paper flags as future work.
  • Extension: the abstention behavior suggests a testable deployment policy: a domain that returns nothing at a requested budget is informative, and one could monitor the abstention rate as a drift detector for the query stream.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. This paper proposes C3R, a post-hoc control layer for multi-domain retrieval that certifies per-domain contamination, CSCR@K (Eq. 1), without query-time labels. The core is a two-split calibration: D1 bounds the router's misrouting and impurity rates via Clopper-Pearson; D2 calibrates per-inferred-domain demotion thresholds via Hoeffding-Bentkus RCPS. Theorem 1 derives the transfer bound Eq. (2), and Eq. (3) inverts it into a target budget with abstention below the router floor. Experiments on BEIR-MIX and three replication pools report zero certificate violations over 1000 resampled calibrations, contrast with marginal CRC violations, and show soft demotion retaining more Recall@10 than calibrated cascades at equal certified contamination. An LLM-judged probe suggests wrong-authority grounding rises with contamination and falls under control. The paper is explicit that the empirical 0/1000 is a stability result, not population coverage, and that population validity rests on Theorem 1.

Significance. Conditional on the proof of Theorem 1, this is a valuable and clean extension of conformal risk control to a latent test-time group with heterogeneous budgets and fully estimable router-error slack. The contribution is well positioned relative to group-conditional coverage and observed-group risk control. Strengths include explicit assumptions, an explicit Bonferroni/union-bound treatment, exact Clopper-Pearson intervals, candid reporting of abstention and of the stability-vs-population distinction, a public benchmark/code artifact, and a replication on an independent federal-regulations pool. The main theorem appears internally coherent. The chief weaknesses are an overreaching Lemma 1 and the silent loss of certificate validity under distribution shift; both are fixable within the manuscript's scope.

major comments (2)
  1. [Section 5, Lemma 1; Section B] The lemma claims soft demotion "removes the classifier kept-set-error floor" and is certifiable at any positive budget. The proof's empty-served-set case requires the most aggressive threshold to make the served set empty, but the implemented threshold grid is tau in [0,1] (Section B), and at tau=0 candidates with m(q,d)=0 remain served, including double-misclassification cases. Thus the floor is removed only in a zero-recall/abstaining limit, not at positive recall within the stated operating range. This overstates a contribution that motivates C2; please tighten the statement/proof or revise "breaks" to "lowers".
  2. [Section 4, Assumption 1; Section 7] The certificate (2) is conditional on single-pool exchangeability: both the D2 RCPS step and the D1 Clopper-Pearson estimates are population statements for the fixed calibration pool. The 1000-resample evaluation and the Figure 3 simulation draw calibration and test from the same pool, so they never exercise a shift in query stream, domain mix, or corpus. Because the abstract and Section 1 frame C3R for high-stakes deployment, the conditionality should be stated in the claims, and a shifted-prior stress test or a simple shift-detection monitor would materially strengthen the paper.
minor comments (4)
  1. [Section 4] The text says "C+1 group risk-control statements" but only C per-group RCPS statements are used; the "+1" is unexplained. Please reconcile with the 3C+1 Bonferroni split.
  2. [Algorithm 1, Serve step] "Emit the certificate (2)" should explicitly state that the certificate is a population-level per-true-domain bound, not a per-query guarantee, since y(q) is unknown at serving time.
  3. [Table 6] For abstaining domains (SciFact, TREC-COVID), C3R viol = 0.00 is tautological because the vacuous certificate B=1 is issued. Please mark these rows as "abstain" rather than "certified" in the violation columns, or report violation rates only over resamples where a non-vacuous certificate is issued.
  4. [Section B] The calibration protocol says the threshold grid is tau in {0, 0.02, ..., 1}; Lemma 1's proof should be consistent with this grid or explicitly extend it.

Circularity Check

0 steps flagged

No significant circularity: Theorem 1's bound is derived from independent splits D1/D2 with fully estimable slack; the resampling result is explicitly labeled a stability check.

full rationale

I reviewed the derivation chain for the main certificate (Eq. 2, Section 4 and Appendix A). The proof partitions the true-domain risk by the router outcome, uses the D2 RCPS statement E[L|g-hat=c] <= alpha_c, converts it to the true-domain subset via pi_c, and separately bounds the misrouted mass via rho_c; the unknown rho_c and pi_c are then replaced by one-sided Clopper-Pearson upper bounds computed on the disjoint split D1. The passage 'The estimates rho_bar_c, pi_bar_c are computed on D1, while tau_c (hence the certified alpha_c) is computed on D2; by Assumption 3 these are independent... This is the step that makes the slack estimable rather than circular' directly addresses the main possible circularity, and the claim is correct: no quantity is fit on the same data that is then used to certify it. Equation (3) is the exact algebraic inverse of (2), not an independent reuse of the target budget. The empirical 0/1000 result is explicitly scoped as a stability result ('the 1000 draws resample calibration splits of the same data, so 0/1000 is a stability result - the procedure does not violate across splits... not a fresh estimate of population coverage'), and population validity is honestly rested on Theorem 1 plus a controlled simulation rather than on the resampling. The assumptions (exchangeability, bounded loss, split independence) are stated and do not include the target bound; Assumption 1 is a standard modeling assumption, not a self-definition. No load-bearing self-citation is present: the conformal machinery (RCPS, Hoeffding-Bentkus, Clopper-Pearson) is standard external work. I therefore find no step where a prediction reduces by construction to an input, and no self-citation chain forcing the result.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

The central bound rests on standard conformal machinery plus two structural assumptions: exchangeability of calibration/test queries and independence of the two calibration splits. No new physical or metaphysical entities are postulated; the ambig bucket is an internal routing group, not an entity requiring independent evidence. Hand-set knobs (probe temperature, ambiguity threshold, budgets, threshold grid) affect feasibility and recall but not the validity argument.

free parameters (4)
  • probe temperature T
    Temperature scaling of the logistic domain probe; fitted on held-out query-style data and controls posterior sharpness, hence router errors rho, pi. Feasibility and certificate width depend on it.
  • ambiguity threshold u
    Posterior confidence threshold above which argmax is routed to a domain and below which the query goes to the ambig bucket. Chosen by hand in Section 5; affects routing and feasibility.
  • per-domain target budgets alpha*_c = 0.05 for most-contaminated domain, 0.15 elsewhere on BEIR-MIX; 0.40 on Sector-Bench
    Hand-set experimental budgets used in the inversion (3). Not fitted, but they determine which domains certify and which abstain.
  • demotion threshold grid spacing = 0.02 over [0,1], 51 points
    Discretizes tau selection in RCPS calibration. Not load-bearing for validity, but it can miss optimal thresholds.
axioms (6)
  • domain assumption Exchangeability of calibration and test queries from a single fixed query pool; conditioning on the frozen inferred-domain function preserves exchangeability (Assumption 1)
    Section 4 Assumption 1. Required for both the RCPS group guarantee and the Clopper-Pearson bounds to transfer to deployment. If the query stream shifts, the certificate loses validity.
  • domain assumption D1, D2, and the probe-training split are independent and disjoint (Assumption 3)
    Section 4 Assumption 3. This is what makes the router-error slack estimable rather than circular. Design choice, checkable by construction.
  • standard math CSCR lies in [0,1] and the monotone running-maximum envelope upper-bounds the served loss, so RCPS applies to the envelope
    Section 4. Mathematical definition of the metric and the envelope; not empirical.
  • standard math Clopper-Pearson one-sided binomial bounds and the Hoeffding-Bentkus RCPS p-value are valid at their stated confidence levels
    Appendix B. Standard finite-sample tools from the cited conformal literature, invoked for every reported certificate.
  • domain assumption Source-corpus provenance is the true domain, and wrong provenance is a conservative proxy for harm
    Section 3 and Section 6. If within-domain wrong authority exists, or if source datasets are not clean domains, CSCR under-covers harm. The authors acknowledge this explicitly.
  • domain assumption The LLM judge's labels are a valid automatic proxy for wrong-authority grounding
    Section 6 downstream probe. Authors flag it as proxy-only and call for human annotation; the headline certificate does not depend on this assumption.

pith-pipeline@v1.3.0-alltime-deepseek · 25236 in / 16636 out tokens · 172477 ms · 2026-08-02T05:45:34.427206+00:00 · methodology

0 comments
read the original abstract

Retrieval over corpora that mix several domains often returns relevant but wrong-domain evidence that ranking metrics miss and that conformal risk control bounds only marginally, under-covering the worst domains. This work introduces C3R, a drop-in control layer that, from an inferred domain posterior and no query-time label, certifies a per-domain contamination budget where feasible and otherwise abstains rather than silently violating; on the hardest domains it guarantees a reduction, not a tight bound. The core is a two-split scheme built on risk-controlling prediction sets, whose finite-sample transfer bound crosses from the inferred to the true domain with fully estimable slack, supports heterogeneous budgets, and inverts for deployment. Population validity rests on this bound and a controlled simulation; across a thousand resampled calibrations the certificate never violates (a stability result) while marginal control violates the most-contaminated domain in every draw, and soft demotion retains more recall than the strongest calibrated cascade at equal certified contamination. The method replicates across open testbeds including an independent one from public federal regulations, and an LLM-judged downstream probe indicates wrong-authority grounding rises with contamination and falls under control. The layer is frozen-stack and reranker-agnostic.

Figures

Figures reproduced from arXiv: 2607.14157 by Jayakumar Manoharan.

Figure 1
Figure 1. Figure 1: (a) Cross-domain contamination is sharply asymmetric: SciFact (0.63) and NFCorpus (0.56) exceed a marginal budget (0.29, [PITH_FULL_IMAGE:figures/full_fig_p013_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Two-split calibration: D1 bounds the router errors 𝜌,¯ 𝜋¯; the independent D2 sets the demotion threshold 𝜏. Independence (D1 ⊥ D2) makes the transfer bound non-circular. 0.0 0.1 0.2 0.3 0.4 0.5 probe error ρ (strict domain) 0.0 0.2 0.4 0.6 0.8 1.0 violation rate (strict domain) γ=0.1 C3R (ours) Naive transfer Marginal CRC Weighted [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Simulation: C3R’s strict-domain violation rate stays at [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Recall@10 retained where the budget is certified, vs. contamination budget 𝛼. On the contaminated domain (SciFact) C3R retains 2-6× more than the strongest calibrated cascade and the hard cascade cannot certify; on the clean domain (FiQA) all methods tie. The uncertified classifier filter (★) sits at 0.57 contamination with no guarantee. 0.0 0.1 0.2 0.3 0.4 0.5 0.6 misrouting ρ̄ 0.0 0.1 0.2 0.3 0.4 0.5 0.6… view at source ↗
Figure 5
Figure 5. Figure 5: Vacuity frontier: certificate width as a function of router error [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Empirical sample complexity: feasibility onset matches the theoretical [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

63 extracted references · 18 linked inside Pith

  1. [1]

    Angelopoulos

    Anastasios N. Angelopoulos. 2026. Conformal Risk Control for Non-Monotonic Losses.arXiv preprint arXiv:2602.20151(2026)

    arXiv 2026

  2. [2]

    Angelopoulos and Stephen Bates

    Anastasios N. Angelopoulos and Stephen Bates. 2023. Conformal Prediction: A Gentle Introduction.Foundations and Trends in Machine Learning16, 4 (2023), 494-591. https://doi.org/10.1561/2200000101

    doi:10.1561/2200000101 2023

  3. [3]

    Angelopoulos, Stephen Bates, Emmanuel J

    Anastasios N. Angelopoulos, Stephen Bates, Emmanuel J. Candès, Michael I. Jordan, and Lihua Lei. 2021. Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control. (2021). arXiv:2110.01052

    Pith/arXiv arXiv 2021

  4. [4]

    Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster

    Anastasios N. Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. 2024. Conformal Risk Control. InInternational Conference on Learning Representations (ICLR)

    2024

  5. [5]

    Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, and Wen tau Yih. 2023. Task-Aware Retrieval with Instructions. InFindings of the Association for Computational Linguistics (ACL)

    2023

  6. [6]

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. InInternational Conference on Learning Representations (ICLR)

    2024

  7. [7]

    Konstantina Bairaktari, Jiayun Wu, and Zhiwei Steven Wu. 2025. Kandinsky Conformal Prediction: Beyond Class- and Covariate-Conditional Coverage.arXiv preprint arXiv:2502.17264(2025)

    Pith/arXiv arXiv 2025

  8. [8]

    Candès, Aaditya Ramdas, and Ryan J

    Rina Foygel Barber, Emmanuel J. Candès, Aaditya Ramdas, and Ryan J. Tibshirani. 2021. The Limits of Distribution-Free Conditional Predictive Inference.Information and Inference: A Journal of the IMA10, 2 (2021)

    2021

  9. [9]

    Osbert Bastani, Varun Gupta, Christopher Jung, Georgy Noarov, Ramya Ramalingam, and Aaron Roth. 2022. Practical Adversarial Multivalid Conformal Prediction. InAdvances in Neural Information Processing Systems 35 (NeurIPS). Manuscript submitted to ACM Certified Domain-Consistency for Multi-Domain Retrieval: Label-Free Per-Domain Contamination Control with ...

    2022

  10. [10]

    Angelopoulos, Lihua Lei, Jitendra Malik, and Michael I

    Stephen Bates, Anastasios N. Angelopoulos, Lihua Lei, Jitendra Malik, and Michael I. Jordan. 2021. Distribution-Free, Risk-Controlling Prediction Sets.J. ACM68, 6 (2021)

    2021

  11. [11]

    Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. 2023. LEACE: Perfect Linear Concept Erasure in Closed Form. InAdvances in Neural Information Processing Systems (NeurIPS)

    2023

  12. [12]

    Aabesh Bhattacharyya and Rina Foygel Barber. 2024. Group-Weighted Conformal Prediction.arXiv preprint arXiv:2401.17452(2024)

    Pith/arXiv arXiv 2024

  13. [13]

    Vera Boteva, Demian Gholipour, Artem Sokolov, and Stefan Riezler. 2016. A Full-Text Learning to Rank Dataset for Medical Information Retrieval. In Advances in Information Retrieval - 38th European Conference on IR Research (ECIR) (LNCS, Vol. 9626). Springer, 716-722. https://doi.org/10.1007/978- 3-319-30671-1_58

    doi:10.1007/978- 2016

  14. [14]

    Debashish Chakraborty, Eugene Yang, Daniel Khashabi, Dawn Lawrie, and Kevin Duh. 2025. Principled Context Engineering for Retrieval-Augmented Generation: Statistical Guarantees via Conformal Prediction.arXiv preprint arXiv:2511.17908(2025)

    arXiv 2025

  15. [15]

    Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S. Weld. 2020. SPECTER: Document-level Representation Learning using Citation-informed Transformers. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL). Association for Computational Linguistics, 2270-2282. https://doi.org/10.18653/v1/2020.acl-main.207

    doi:10.18653/v1/2020.acl-main.207 2020

  16. [16]

    Cormack, Charles L

    Gordon V. Cormack, Charles L. A. Clarke, and Stefan Büttcher. 2009. Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods. InProceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). ACM, 758-759. https://doi.org/10.1145/1571941.1572114

    arXiv 2009

  17. [17]

    Angelopoulos, Asaf Gendler, and Yaniv Romano

    Bat-Sheva Einbinder, Stephen Bates, Anastasios N. Angelopoulos, Asaf Gendler, and Yaniv Romano. 2024. Label Noise Robustness of Conformal Prediction.Journal of Machine Learning Research(2024)

    2024

  18. [18]

    Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. 2024. RAGAs: Automated Evaluation of Retrieval Augmented Generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations (EACL). Association for Computational Linguistics, 150-158

    2024

  19. [19]

    Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2023. Precise Zero-Shot Dense Retrieval without Relevance Labels. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL). arXiv:2212.10496

    Pith/arXiv arXiv 2023

  20. [20]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2023. Retrieval-Augmented Generation for Large Language Models: A Survey.arXiv preprint(2023). arXiv:2312.10997

    Pith/arXiv arXiv 2023

  21. [21]

    Yonatan Geifman and Ran El-Yaniv. 2017. Selective Classification for Deep Neural Networks. InAdvances in Neural Information Processing Systems 30 (NeurIPS)

    2017

  22. [22]

    Cherian, and Emmanuel J

    Isaac Gibbs, John J. Cherian, and Emmanuel J. Candès. 2025. Conformal Prediction with Conditional Guarantees.Journal of the Royal Statistical Society Series B(2025)

    2025

  23. [23]

    Kim, Omer Reingold, and Guy N

    Úrsula Hébert-Johnson, Michael P. Kim, Omer Reingold, and Guy N. Rothblum. 2018. Multicalibration: Calibration for the (Computationally- Identifiable) Masses. InProceedings of the 35th International Conference on Machine Learning (ICML) (PMLR, Vol. 80). 1939-1948

    2018

  24. [24]

    Sebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. 2021. Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling. InProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). arXiv:2104.06967

    Pith/arXiv arXiv 2021

  25. [25]

    Verspoor, and Timothy Baldwin

    Doris Hoogeveen, Karin M. Verspoor, and Timothy Baldwin. 2015. CQADupStack: A Benchmark Data Set for Community Question-Answering Research. InProceedings of the 20th Australasian Document Computing Symposium (ADCS). ACM, 3:1-3:8. https://doi.org/10.1145/2838931.2838934

    arXiv 2015

  26. [26]

    Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised Dense Information Retrieval with Contrastive Learning.Transactions on Machine Learning Research (TMLR)(2022)

    2022

  27. [27]

    Gautier Izacard and Edouard Grave. 2021. Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL). Association for Computational Linguistics, 874-880

    2021

  28. [28]

    Mintong Kang, Nezihe Merve Gürel, Ning Yu, Dawn Song, and Bo Li. 2024. C-RAG: Certified Generation Risks for Retrieval-Augmented Language Models. InInternational Conference on Machine Learning (ICML)

    2024

  29. [29]

    Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 6769-6781

    2020

  30. [30]

    Omar Khattab and Matei Zaharia. 2020. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). ACM, 39-48. https: //doi.org/10.1145/3397271.3401075

    arXiv 2020

  31. [31]

    Hyunji Lee, Luca Soldaini, Arman Cohan, Minjoon Seo, and Kyle Lo. 2025. RouterRetriever: Routing over a Mixture of Expert Embedding Models. In AAAI Conference on Artificial Intelligence

    2025

  32. [32]

    Tibshirani, and Larry Wasserman

    Jing Lei, Max G’Sell, Alessandro Rinaldo, Ryan J. Tibshirani, and Larry Wasserman. 2018. Distribution-Free Predictive Inference for Regression.J. Amer. Statist. Assoc.113, 523 (2018), 1094-1111. https://doi.org/10.1080/01621459.2017.1307116

    arXiv 2018

  33. [33]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. InAdvances in Neural Information Processing Systems 33 (NeurIPS). 9459-9474. Manuscript submitt...

    2020

  34. [34]

    Rui Luo and Zhixin Zhou. 2025. Conditional Conformal Risk Adaptation.arXiv preprint arXiv:2504.07611(2025)

    Pith/arXiv arXiv 2025

  35. [35]

    Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018. WWW’18 Open Challenge: Financial Opinion Mining and Question Answering. InCompanion Proceedings of The Web Conference 2018 (WWW ’18 Companion). ACM, 1941-1942. https://doi.org/10.1145/3184558.3192301

    arXiv 2018

  36. [36]

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computatio...

    2023

  37. [37]

    Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A Human Generated Machine Reading Comprehension Dataset. InCoCo@NIPS. arXiv:1611.09268

    Pith/arXiv arXiv 2016

  38. [38]

    Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Y Zhao, Yi Luan, Keith B Hall, Ming-Wei Chang, and Yinfei Yang. 2022. Large Dual Encoders Are Generalizable Retrievers. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP). arXiv:2112.07899

    Pith/arXiv arXiv 2022

  39. [39]

    Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage Re-ranking with BERT.arXiv preprint(2019). arXiv:1901.04085

    Pith/arXiv arXiv 2019

  40. [40]

    Jaakkola, and Regina Barzilay

    Victor Quach, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn, Tommi S. Jaakkola, and Regina Barzilay. 2024. Conformal Language Modeling. In International Conference on Learning Representations (ICLR)

    2024

  41. [41]

    Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection. InAnnual Meeting of the Association for Computational Linguistics (ACL)

    2020

  42. [42]

    Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP-IJCNLP). arXiv:1908.10084

    Pith/arXiv arXiv 2019

  43. [43]

    Stephen Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Beyond.Foundations and Trends in Information Retrieval3, 4 (2009), 333-389. https://doi.org/10.1561/1500000019

    doi:10.1561/1500000019 2009

  44. [44]

    Yaniv Romano, Evan Patterson, and Emmanuel J. Candès. 2019. Conformalized Quantile Regression. InAdvances in Neural Information Processing Systems 32 (NeurIPS). 3538-3548

    2019

  45. [45]

    Yaniv Romano, Matteo Sesia, and Emmanuel J. Candès. 2020. Classification with Valid and Adaptive Coverage. InAdvances in Neural Information Processing Systems 33 (NeurIPS)

    2020

  46. [46]

    Matteo Sesia, Y. X. Rachel Wang, and Xin Tong. 2025. Adaptive Conformal Classification with Noisy Labels.Journal of the Royal Statistical Society Series B(2025)

    2025

  47. [47]

    Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2024. REPLUG: Retrieval- Augmented Black-Box Language Models. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT). Association for Computational Linguistics, 8371-8384

    2024

  48. [48]

    Effrosyni Sokli, Pranav Kasela, Georgios Peikos, and Gabriella Pasi. 2024. Investigating Mixture of Experts in Dense Retrieval.arXiv preprint arXiv:2412.11864(2024)

    Pith/arXiv arXiv 2024

  49. [49]

    Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. InNeurIPS Datasets and Benchmarks

    2021

  50. [50]

    Tibshirani, Rina Foygel Barber, Emmanuel J

    Ryan J. Tibshirani, Rina Foygel Barber, Emmanuel J. Candès, and Aaditya Ramdas. 2019. Conformal Prediction Under Covariate Shift. InAdvances in Neural Information Processing Systems 32 (NeurIPS). 2526-2536

    2019

  51. [51]

    Yiteng Tu, Weihang Su, Yujia Zhou, Qingyao Ai, and Yiqun Liu. 2025. RbFT: Robust Fine-tuning for Retrieval-Augmented Generation against Retrieval Defects. InProceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). arXiv:2501.18365

    Pith/arXiv arXiv 2025

  52. [52]

    Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang

    Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R. Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. 2021. TREC-COVID: Constructing a Pandemic Information Retrieval Test Collection.ACM SIGIR Forum54, 1 (2021), 1-12. https: //doi.org/10.1145/3451964.3451965

    arXiv 2021

  53. [53]

    Vladimir Vovk. 2012. Conditional Validity of Inductive Conformal Predictors. InProceedings of the Asian Conference on Machine Learning (ACML) (PMLR, Vol. 25). 475-490

    2012

  54. [54]

    2005.Algorithmic Learning in a Random World

    Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. 2005.Algorithmic Learning in a Random World. Springer, New York

    2005

  55. [55]

    David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or Fiction: Verifying Scientific Claims. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 7534-7550. https://doi.org/10.18653/v1/2020.emnlp-main.609

    doi:10.18653/v1/2020.emnlp-main.609 2020

  56. [56]

    Kexin Wang, Nandan Thakur, Nils Reimers, and Iryna Gurevych. 2022. GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval. InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT). Association for Computational Linguistics, 2345-2360

    2022

  57. [57]

    Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text Embeddings by Weakly-Supervised Contrastive Pre-training.arXiv preprint arXiv:2212.03533(2022)

    Pith/arXiv arXiv 2022

  58. [58]

    Orion Weller, Benjamin Van Durme, Dawn Lawrie, Ashwin Paranjape, Yuhao Zhang, and Jack Hessel. 2024. Promptriever: Instruction-Trained Retrievers Can Be Prompted Like Language Models.arXiv preprint arXiv:2409.11136(2024). Manuscript submitted to ACM Certified Domain-Consistency for Multi-Domain Retrieval: Label-Free Per-Domain Contamination Control with C...

    Pith/arXiv arXiv 2024

  59. [59]

    Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. 2023. C-Pack: Packed Resources for General Chinese Embeddings.arXiv preprint arXiv:2309.07597(2023)

    Pith/arXiv arXiv 2023

  60. [60]

    Jiarui Zhang, Xiangyu Liu, and Yong Hu. 2025. RAGRouter: Learning to Route Queries to Multiple Retrieval-Augmented Language Models.arXiv preprint arXiv:2505.23052(2025)

    arXiv 2025

  61. [61]

    Yao Zhang and Emmanuel J. Candès. 2024. Posterior Conformal Prediction.arXiv preprint arXiv:2409.19712(2024)

    Pith/arXiv arXiv 2024

  62. [62]

    Zexuan Zhong, Ziqing Huang, Alexander Wettig, and Danqi Chen. 2023. Poisoning Retrieval Corpora by Injecting Adversarial Passages. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 13764-13775

    2023

  63. [63]

    Wei Zou, Runpeng Geng, Binghui Wang, and Jinyuan Jia. 2025. PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models. In34th USENIX Security Symposium (USENIX Security). 3827-3844. arXiv:2402.07867. A PROOF OF THE PER-DOMAIN TRANSFER CERTIFICATE We prove the per-domain bound(2); simultaneity over domains follows...

    Pith/arXiv arXiv 2025