Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

MixLoRA-DSI: Dynamically Expandable Mixture-of-LoRA Experts for Rehearsal-Free Generative Retrieval over Dynamic Corpora

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MixLoRA-DSI keeps a generative retrieval model's index current over growing corpora by adding LoRA experts only when router energy scores flag out-of-distribution tokens, achieving sublinear parameter growth and beating full-model update…

desk verdict Solid empirical contribution on PEFT-based continual learning for generative retrieval, but the OOD expansion mechanism is not causally demonstrated and the sublinear-growth claim outruns the evidence. read the letter →

arxiv 2507.09924 v1 pith:BHJFGL6F submitted 2025-07-14 cs.IR cs.AIcs.CLcs.LG

Tuan-Luc Huynh , Thuy-Trang Vu , Weiqing Wang , Trung Le , Dragan Gašević , Yuan-Fang Li , Thanh-Toan Do This is my paper
classification cs.IRcs.AIcs.CLcs.LG
keywords generativeretrievaldynamiccorporacontinuallearningmixtureofexpertslow-rankadaptationout-of-distributiondetectionrehearsal-freeresidualquantization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Generative retrieval stores a document collection in the parameters of a sequence-to-sequence model, so adding documents normally means retraining the whole model, which is costly and makes it forget earlier material. MixLoRA-DSI tries a cheaper path: freeze the transformer, route tokens through a small mixture of low-rank adapters, and add a new adapter only when the router's energy scores show that a significant share of the incoming corpus looks genuinely new. On NQ320k and MSMARCO, the authors report that this expansion rule yields sublinear parameter growth, with average performance and forgetting that beat full-model continual-learning baselines while training far fewer parameters (0.9M versus 235.4M on NQ320k). The insight matters because it points to a way to keep model-based search indexes current without storing or replaying old documents.

What carries the argument

Three components carry the method. The first is energy-based expansion: each MixLoRA layer's router is treated as a classifier over its LoRA experts, and the energy score $E(x;R) = -\log\sum_i \exp(\langle R_i,x\rangle)$ serves as the out-of-distribution signal; the model keeps exponential-moving-average thresholds of in-distribution energy and adds a new expert to a layer when more than $\delta$ of the incoming corpus's pseudo-query tokens exceed the threshold. The second is the improved router: a top-$k$ cosine classifier replaces the softmax router and is trained with an auxiliary loss $L_\mathrm{aux}$ that aligns the newly added router weight with docid hidden states and penalizes similarity between new and old router weights, removing the recency bias that otherwise sends most tokens to the newest expert. The third is a continual-learning toolkit for residual-quantization docids, which encode each document as a sequence of codes from learned codebooks: a decoding mask restricts each generation step to the active RQ codebook, and a slow-learner gradient scale plus KL-divergence regularizer keeps the docid posterior close to the previous model's, allowing the large docid output space to be updated without destructive shifts.

What would settle it

Vary the expansion threshold across a wide range (for example, from 0% to 50% of tokens flagged per layer) on a stream of deliberately constructed corpora that alternate between rephrased old topics and entirely new topics, and compare the resulting expert count with retrieval quality on old and new queries; if quality and forgetting remain unchanged across thresholds, the energy-score trigger is not what produces the reported gains.

Watch

Extended reading notes

Core claim

The central claim is that continual indexing of a generative retrieval model over dynamic corpora can be treated as an out-of-distribution detection problem rather than a capacity-allocation problem. MixLoRA-DSI starts from a model whose decoder feed-forward layers are replaced by a mixture of LoRA experts with a cosine-classifier router; before each new corpus arrives, it scans the corpus's generated queries, computes per-token energy scores from each layer's router, and expands a layer by one expert only when the fraction of out-of-distribution tokens exceeds a threshold. The router replacement and an auxiliary loss that keeps new expert weights distinct remove the recency bias of softmax MoE routing, while residual-quantization docids are protected by a decoding mask and a slow-learner KL regularizer. The reported outcome is that MixLoRA-DSI achieves the best average performance among continual-learning generative retrieval models on NQ320k and a comparable stability-plasticity trade-off on MSMARCO, using 0.9M trainable parameters where full-model updates use 235.4M.

Load-bearing premise

The load-bearing assumption is that the router's single energy number faithfully tells whether a token carries genuinely new information, and that the hand-picked threshold for acting on that number works across corpora; if either fails, the model will either never add capacity when it needs to or will add experts on redundant material.

Editorial extensions

If this is right

  • A generative search index can ingest newly added documents without retraining from scratch or replaying old ones, removing a major cost barrier for live collections.
  • Parameter growth stays sublinear in the number of corpus updates: on NQ320k the full system uses 0.9M trainable parameters, while full-model CLGR baselines use 235.4M.
  • The cosine-classifier router plus auxiliary loss prevents the recency bias that makes standard MoE routers dump everything on the newest expert, so earlier LoRA experts keep contributing.
  • RQ-based docid continual-learning strategies (masking, slow learner, KL regularization) let the large docid output space be updated without destructive shifts, avoiding the scaling limits of atomic docids.
  • Training, inference, and storage footprints stay near a single checkpoint; no Fisher-information estimation or replay buffer is needed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the same energy-threshold expansion rule could be applied to any frozen-backbone mixture-of-LoRA system that ingests new data, not just generative retrieval; the router energy score is a general budget signal for when to grow capacity.
  • Inference: since $\delta$ is hand-set per dataset and variant (1%–5% on NQ320k, 0.01%–10% on MSMARCO), the calibration of this threshold is an open knob; a validation-based rule for setting $\delta$ from observed OOD rates would likely make the mechanism more portable.
  • Inference: combining OOD-triggered expansion with a small rehearsal buffer of the most OOD-heavy documents is a natural next test that might close the remaining gap to full-model updates on MSMARCO.
  • Inference: because OOD scoring runs on pseudo-queries, the fidelity of the pseudo-query generator is a hidden variable; replacing it with a better generator could reduce unnecessary expansion and further lower parameter growth.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes MixLoRA-DSI, a parameter-efficient fine-tuning (PEFT) framework for rehearsal-free generative retrieval over dynamic corpora. It replaces standard MoE routers in selected decoder layers with a top-2 cosine classifier router and an auxiliary loss, and it introduces a layer-wise, token-wise out-of-distribution (OOD) detection mechanism that uses router energy scores to decide when to add new LoRA experts. The paper also proposes continual-learning strategies for RQ-based docids, including decoding masks, slow-learner gradient scaling, and KL regularization against the previous model. Experiments on NQ320k, MSMARCO, and LongEval (appendix) compare against full-model-update baselines (BASE, DSI++, CLEVER) and PEFT baselines (PromptDSI, CorpusBrain++, Naive Expansion), reporting retrieval metrics, forgetting, parameter counts, and memory footprints. The central claim is that the OOD-driven expansion strategy yields sublinear parameter growth while maintaining retrieval effectiveness.

Significance. The paper addresses a timely and practically relevant problem: continually updating generative retrieval models without rehearsal or full retraining. The experimental design is careful in several respects: it includes strong baselines, ablates each proposed component on two datasets (Tables 4 and 9), adds a third dataset (LongEval) in the appendix, compares memory footprints (Table 3), and reports detailed implementation settings. The router improvements (cosine classifier plus auxiliary loss) and the RQ-based docid continual-learning strategies appear to give consistent gains in the ablations. However, the central novelty, the energy-based OOD gate that enables sublinear parameter growth, is not yet convincingly validated. The gate fires exactly once on NQ320k and never on MSMARCO, no sensitivity analysis is provided for the hand-set threshold δ, and the comparisons that isolate the gate are not capacity-matched. If the mechanism is confirmed with proper sensitivity and control experiments, the paper would be a solid contribution; as it stands, the main claim outruns the evidence.

major comments (4)
  1. [Section 3.1 and Table 5] The OOD expansion trigger is validated only indirectly, and the affirmative evidence is thin: on NQ320k the gate fires exactly once (before D1, with # LoRA growing from 10 to 15 in Table 5), and on MSMARCO Section 4.2 states that MixLoRA-DSI "coincides with the non-expansion variant," with Table 2 showing 0.6M parameters. With a single expansion event across all experiments, the data do not establish that the energy-score criterion reliably identifies when new expert capacity is needed, nor that the criterion generalizes to a much larger and more heterogeneous corpus. Please add a sensitivity analysis over δ (the layer-wise OOD query threshold) on both datasets, report the number and layer locations of expansions for each setting, and test a more permissive δ or additional incremental corpora on MSMARCO so that the gate's firing behavior can be observed rather than inferred from a single non-event.
  2. [Section 4.1 and Table 1] The comparison between MixLoRA-DSI and MixLoRA-DSI-OOD is not capacity-matched: the two variants use 0.9M vs. 1.6M trainable parameters. Consequently, the claim that the OOD gate "maintains over 98% of AP4 and FWT4" with 60% of the parameters conflates the gate's decision quality with the effect of a smaller final model. I request an ablation that holds the final parameter budget fixed while varying only the expansion criterion, for example energy-score-triggered expansion versus periodic or random expansion to the same final expert count. Without such a control, Table 4's OOD row and the "98%" claim do not demonstrate that the energy gate is the reason sublinear growth is safe.
  3. [Appendix E.3] The expansion threshold δ is hand-set per dataset and per variant (1% on the NQ pre-trained variant, 5% on MixLoRA-DSI-PT, 0.01% on the MSMARCO pre-trained variant, and 10% on the MSMARCO MixLoRA-DSI-PT variant), and no sensitivity analysis is reported. Since δ fully governs when expansion occurs, the sublinear-growth result may depend on a favorable threshold choice. Appendix B.2 tunes α_1 and α_2 but not δ. Please provide a sweep over δ that reports AP, BWT, and final parameter count for each setting, and justify the chosen values, for both NQ320k and MSMARCO.
  4. [Section 3.1 and Table 5] The premise that the router's energy score is a valid OOD signal for "significant novel information" is assumed rather than directly tested. Table 5 reports only the percentage of OOD queries per layer, with no correlation to downstream retrieval gains, no analysis of whether newly added LoRAs actually receive high routing weight on OOD tokens, and no comparison against alternative novelty signals (e.g., increase in retrieval loss, router entropy, or random expansion with the same budget). I ask for router-utilization statistics for newly added experts and an ablation that replaces the energy score with one alternative criterion, so that the mechanism's causal role in expansion quality is evidenced rather than assumed.
minor comments (5)
  1. [Table 1] There is a formatting error in the CLEVER(n=512) row: "35.0/53.690.6/84.8" should be "35.0/53.6 90.6/84.8"; several other cells also have inconsistent spacing (e.g., "55.2 /35.9"), which should be corrected.
  2. [Table 5] The header "# LoRA10 15 15 15 10 15 18 19" is missing a space and should read "# LoRA 10 15 15 15 10 15 18 19"; please also align the columns for readability.
  3. [Section 3.3 and Eq. (7)] When no expansion occurs during continual indexing, it is unclear how the auxiliary router loss Laux is computed, since there is no "new router weight" R_N. Please clarify whether Laux is applied to existing experts with the last expert used as the "new" weight, or whether it is omitted entirely in non-expansion steps.
  4. [Section 4.1] The description of MixLoRA-DSI-OOD is ambiguous: "does not employ energy-based dynamic expansion" does not specify what expansion schedule it follows. Please state explicitly that this variant expands by one LoRA per layer per corpus (or otherwise describe its expansion schedule) so that the parameter counts in Tables 1 and 2 are interpretable.
  5. [Limitations] The Limitations section acknowledges that "all GR models lag behind traditional IR on MSMARCO in this realistic setting." Since the abstract emphasizes outperforming full-model update baselines, it would be helpful to mention this MSMARCO caveat in the conclusion as well, to avoid overstating the practical advantage.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: MixLoRA-DSI's results are empirical; the only minor concerns are non-load-bearing self-citations and dataset-level threshold choice.

full rationale

The paper does not derive any result from an equation that assumes the conclusion. The OOD expansion criterion (Eq. 2 energy score vs. EMA thresholds) is an independent algorithmic gate: it decides whether to add a LoRA expert, and the downstream retrieval metrics (AP4, BWT4, FWT4) are computed from standard seq2seq cross-entropy and constrained beam search, not from the gate itself. 'Sublinear parameter growth' is an empirical outcome of thresholded expansion, not a tautology: if every layer triggered at every timestep, growth would be linear, so the observed NQ320k and MSMARCO parameter counts are genuine measurements. The hand-set delta (1%/5% on NQ320k, 0.01%/10% on MSMARCO) is a hyperparameter selected on the evaluation datasets, and no sensitivity analysis is given; this is a model-selection limitation, not a circular definition. Self-citations to Huynh et al. 2024 (PromptDSI) are used for dataset-split simulation and for the heuristic of applying PEFT to the first five decoder layers; both are architectural and methodological choices, and they are not load-bearing for the central claim. The MSMARCO case, where MixLoRA-DSI coincides with the non-expansion variant, is explicitly acknowledged and weakens the generality of the OOD gate, but it is an empirical null result rather than a circular step. Overall, the derivation chain is self-contained against external benchmarks.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the empirical behavior of the router's energy scores as an OOD signal, the hand-set expansion threshold delta, and the hyperparameters alpha_1 and alpha_2. No new physical or theoretical entities are introduced beyond the trainable LoRA experts.

free parameters (3)
  • delta (OOD query threshold per layer) = 1% (NQ pre-trained), 5% (NQ-PT), 0.01% (MSMARCO pre-trained), 10% (MSMARCO-PT)
    Controls when a new LoRA expert is added; set differently per dataset and variant to obtain best trade-off, so it is a hand-tuned hyperparameter.
  • alpha_1 (auxiliary router loss weight) = 1.0
    Chosen after grid search over {0.05, 0.1, 0.5, 1.0}; balances AP and BWT.
  • alpha_2 (KL divergence weight) = 0.1
    Chosen after grid search; larger values reduce forgetting but hurt AP.
assumptions (4)
  • domain assumption Queries and docids across corpora share structural patterns; new docids frequently overlap in latent space with previously indexed ones.
    Used to justify that energy-based OOD detection at the router can identify which layers need expansion (Section 3.1).
  • domain assumption The energy score from the router is a reliable indicator of out-of-distribution tokens.
    The expansion decision is based entirely on this score (Eq. 2).
  • domain assumption RQ-based docids with masking and slow-learner updates prevent catastrophic forgetting in the output vocabulary.
    Underlies the continual learning strategies in Section 3.2 and the KL loss.
  • domain assumption T5-base pretrained checkpoint and the RIPOR initialization provide a strong base for all models.
    All models share this backbone; results may not transfer to larger or different backbones (Limitations).

how reviews work

0 comments
Cite this review

Pith. "Pith review of MixLoRA-DSI: Dynamically Expandable Mixture-of-LoRA Experts for Rehearsal-Free Generative Retrieval over Dynamic Corpora." pith.science (2026). https://pith.science/paper/BHJFGL6F

@misc{pith2026250709924,
  author       = {Pith},
  title        = {Pith review of: MixLoRA-DSI: Dynamically Expandable Mixture-of-LoRA Experts for Rehearsal-Free Generative Retrieval over Dynamic Corpora},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BHJFGL6F}},
  note         = {Machine review of arXiv:2507.09924}
}
read the original abstract

Continually updating model-based indexes in generative retrieval with new documents remains challenging, as full retraining is computationally expensive and impractical under resource constraints. We propose MixLoRA-DSI, a novel framework that combines an expandable mixture of Low-Rank Adaptation experts with a layer-wise out-of-distribution (OOD)-driven expansion strategy. Instead of allocating new experts for each new corpus, our proposed expansion strategy enables sublinear parameter growth by selectively introducing new experts only when significant number of OOD documents are detected. Experiments on NQ320k and MS MARCO Passage demonstrate that MixLoRA-DSI outperforms full-model update baselines, with minimal parameter overhead and substantially lower training costs.

Figures

Figures reproduced from arXiv: 2507.09924 by the authors.

Figure 1
Figure 1. Continual learning performance of different [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of MixLoRA-DSI. Before training, the model scans the new corpus to perform energy-based [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Token routing analysis of the original MoE [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. A Parametric Memory Head for Continual Generative Retrieval

    cs.IR 2026-04 unverdicted novelty 7.0 of 10

    A product-key parametric memory head with selective sparse updates mitigates catastrophic forgetting in generative retrieval models during sequential addition of new documents.

Reference graph

Works this paper leans on

58 extracted references · 32 canonical work pages · cited by 1 Pith paper

  1. [1]

    Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. 2019. Task-free continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11254--11263

    work page 2019

  2. [2]

    Randy Ardywibowo, Zepeng Huo, Zhangyang Wang, Bobak J Mortazavi, Shuai Huang, and Xiaoning Qian. 2022. Varigrow: Variational architecture growing for task-agnostic continual learning based on bayesian novelty. In International Conference on Machine Learning, pages 865--877. PMLR

    work page 2022

  3. [3]

    Jimmy Lei Ba. 2016. Layer normalization. arXiv preprint arXiv:1607.06450

    arXiv 2016

  4. [4]

    Artem Babenko and Victor Lempitsky. 2014. Additive quantization for extreme vector compression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 931--938

    work page 2014

  5. [5]

    Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, and 1 others. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268

    arXiv 2016

  6. [6]

    Michele Bevilacqua, Giuseppe Ottaviano, Patrick Lewis, Scott Yih, Sebastian Riedel, and Fabio Petroni. 2022. Autoregressive search engines: Generating substrings as document identifiers. Advances in Neural Information Processing Systems, 35:31668--31683

    work page 2022

  7. [7]

    Yinqiong Cai, Keping Bi, Yixing Fan, Jiafeng Guo, Wei Chen, and Xueqi Cheng. 2023. L2r: Lifelong learning for first-stage retrieval with backward-compatible representations. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pages 183--192

    work page 2023

  8. [8]

    Jiangui Chen, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Wei Chen, Yixing Fan, and Xueqi Cheng. 2023. Continual learning for generative retrieval over dynamic corpora. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pages 306--315

    work page 2023

Show all 58 references
  1. [9]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...

    2019 doi

  2. [10]

    Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, and 1 others. 2024. Loramoe: Alleviating world knowledge forgetting in large language models via moe-style plugin. In Proceedings of the 62nd Annual Meeting of ...

    2024

  3. [11]

    William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1--39

    2022

  4. [12]

    Tobias Fink, Florina Piroi, Alaa El-Ebshihy, Jüri Keller, Romain Devaud, David Iommi, Petra Galuščáková, Gabriela Gonzalez-Saez, Philippe Mulhem, Lorraine Goeuriot, and et al. 2025. https://doi.org/10.48436/th5h0-g5f51 Longeval 2025 web retrieval collection

    2025 doi

  5. [13]

    Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. 2020. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412

    2020 arXiv

  6. [14]

    Luyu Gao, Yunyi Zhang, Jiawei Han, and Jamie Callan. 2021. https://doi.org/10.18653/v1/2021.repl4nlp-1.31 Scaling deep contrastive learning batch size under memory limited setup . In Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021), pages 316-...

    2021 doi

  7. [15]

    Spyros Gidaris and Nikos Komodakis. 2018. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4367--4375

    2018

  8. [16]

    Maarten Grootendorst. 2022. Bertopic: Neural topic modeling with a class-based tf-idf procedure. arXiv preprint arXiv:2203.05794

    2022 arXiv

  9. [17]

    Jiafeng Guo, Changjiang Zhou, Ruqing Zhang, Jiangui Chen, Maarten de Rijke, Yixing Fan, and Xueqi Cheng. 2024. Corpusbrain++: A continual generative pre-training framework for knowledge-intensive language tasks. arXiv preprint arXiv:2402.16767

    2024 arXiv

  10. [18]

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790--2799. PMLR

    2019

  11. [19]

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lo RA : Low-rank adaptation of large language models . In International Conference on Learning Representations

    2022

  12. [20]

    Tuan-Luc Huynh, Thuy-Trang Vu, Weiqing Wang, Yinwei Wei, Trung Le, Dragan Gasevic, Yuan-Fang Li, and Thanh-Toan Do. 2024. Promptdsi: Prompt-based rehearsal-free instance-wise incremental learning for document retrieval. arXiv preprint arXiv:2406.12593

    2024 arXiv

  13. [21]

    Jeff Johnson, Matthijs Douze, and Herv \'e J \'e gou. 2019. Billion-scale similarity search with GPUs . IEEE Transactions on Big Data, 7(3):535--547

    2019

  14. [22]

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.550 Dense passage retrieval for open-domain question answering . In Proceedings of the 2020 Conference on Empiric...

    2020 doi

  15. [23]

    Chaeeun Kim, Soyoung Yoon, Hyunji Lee, Joel Jang, Sohee Yang, and Minjoon Seo. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.755 Exploring the practicality of generative retrieval on dynamic corpora . In Proceedings of the 2024 Conference on Empirical Methods in Natural La...

    2024 doi

  16. [24]

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, and 1 others. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national aca...

    2017

  17. [25]

    Weinberger

    Varsha Kishore, Chao Wan, Justin Lovelace, Yoav Artzi, and Kilian Q. Weinberger. 2023. Incdsi: Incrementally updatable document retrieval. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org

    2023

  18. [26]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019 doi

  19. [27]

    Soochan Lee, Junsoo Ha, Dongsu Zhang, and Gunhee Kim. 2020. https://openreview.net/forum?id=SJxSOJStPr A neural dirichlet process mixture model for task-free continual learning . In International Conference on Learning Representations

    2020

  20. [28]

    Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691

    2021 arXiv

  21. [29]

    Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190

    2021 arXiv

  22. [30]

    Yongqi Li, Nan Yang, Liang Wang, Furu Wei, and Wenjie Li. 2023. Multiview identifiers enhanced generative retrieval. arXiv preprint arXiv:2305.16675

    2023 arXiv

  23. [31]

    Yongqi Li, Nan Yang, Liang Wang, Furu Wei, and Wenjie Li. 2024. Learning to rank in generative retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 8716--8723

    2024

  24. [32]

    Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini : A Python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th Annual International ACM SIGIR Con...

    2021

  25. [33]

    Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. 2020. Energy-based out-of-distribution detection. Advances in neural information processing systems, 33:21464--21475

    2020

  26. [34]

    Yuxuan Liu, Tianchi Yang, Zihan Zhang, Minghui Song, Haizhen Huang, Weiwei Deng, Feng Sun, and Qi Zhang. 2024. Asi++: Towards distributionally balanced end-to-end generative retrieval. arXiv preprint arXiv:2405.14280

    2024 arXiv

  27. [35]

    Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101

    2017 arXiv

  28. [36]

    Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109--165. Elsevier

    1989

  29. [37]

    Sanket Mehta, Jai Gupta, Yi Tay, Mostafa Dehghani, Vinh Tran, Jinfeng Rao, Marc Najork, Emma Strubell, and Donald Metzler. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.510 DSI ++: Updating transformer memory with new documents . In Proceedings of the 2023 Conference on Em...

    2023 doi

  30. [38]

    Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. 2019. https://openreview.net/forum?id=H1xwNhCcYm Do deep generative models know what they don't know? In International Conference on Learning Representations

    2019

  31. [39]

    Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rockt \"a schel, and Sebastian Riedel. 2021. https://doi.org/10.18653/v1/2021.naacl-main.200 KIL...

    2021 doi

  32. [40]

    Ronak Pradeep, Kai Hui, Jai Gupta, Adam Lelkes, Honglei Zhuang, Jimmy Lin, Donald Metzler, and Vinh Tran. 2023. How does generative retrieval scale to millions of passages? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1305--1321

    2023

  33. [41]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1--67

    2020

  34. [42]

    Stephen Robertson, Hugo Zaragoza, and 1 others. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval , 3(4):333--389

    2009

  35. [43]

    Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. https://openreview.net/forum?id=B1ckMDqlg Outrageously large neural networks: The sparsely-gated mixture-of-experts layer . In International Conference on Learning ...

    2017

  36. [44]

    Reza Shokri and Vitaly Shmatikov. 2015. Privacy-preserving deep learning. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pages 1310--1321

    2015

  37. [45]

    James Seale Smith, Junjiao Tian, Shaunak Halbe, Yen-Chang Hsu, and Zsolt Kira. 2023. A closer look at rehearsal-free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2409--2419

    2023

  38. [46]

    Weiwei Sun, Lingyong Yan, Zheng Chen, Shuaiqiang Wang, Haichao Zhu, Pengjie Ren, Zhumin Chen, Dawei Yin, Maarten Rijke, and Zhaochun Ren. 2023. Learning to tokenize for generative retrieval. Advances in Neural Information Processing Systems, 36

    2023

  39. [47]

    Yubao Tang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Wei Chen, and Xueqi Cheng. 2024. https://openreview.net/forum?id=2xTkeyJFJb Generative retrieval meets multi-graded relevance . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  40. [48]

    Yi Tay, Vinh Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, and 1 others. 2022. Transformer memory as a differentiable search index. Advances in Neural Information Processing Systems, 35:21831--21843

    2022

  41. [49]

    Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and 1 others. 2022. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision, pages 631...

    2022

  42. [50]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and ...

    2020 doi

  43. [51]

    Xun Wu, Shaohan Huang, and Furu Wei. 2024. https://openreview.net/forum?id=uWvKBCYh4S Mixture of lo RA experts . In The Twelfth International Conference on Learning Representations

    2024

  44. [52]

    Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Ping Hu, Dong Wang, Huchuan Lu, and You He. 2024. Boosting continual learning of vision-language models via mixture-of-experts adapters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23219--23230

    2024

  45. [53]

    Hansi Zeng, Chen Luo, Bowen Jin, Sheikh Muhammad Sarwar, Tianxin Wei, and Hamed Zamani. 2024 a . Scalable and effective generative information retrieval. In Proceedings of the ACM on Web Conference 2024, pages 1441--1452

    2024

  46. [54]

    Hansi Zeng, Chen Luo, and Hamed Zamani. 2024 b . Planning ahead in generative retrieval: Guiding autoregressive generation through simultaneous decoding. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 469--480

    2024

  47. [55]

    Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. 2023. Slca: Slow learner with classifier alignment for continual learning on a pre-trained model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19148--19158

    2023

  48. [56]

    Peitian Zhang, Zheng Liu, Yujia Zhou, Zhicheng Dou, Fangchao Liu, and Zhao Cao. 2024. Generative retrieval via term set generation. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 458--468

    2024

  49. [57]

    Yujia Zhou, Jing Yao, Zhicheng Dou, Ledell Wu, Peitian Zhang, and Ji-Rong Wen. 2022. Ultron: An ultimate retriever on corpus with a model-based indexer. arXiv preprint arXiv:2208.09257

    2022 arXiv

  50. [58]

    Shengyao Zhuang, Houxing Ren, Linjun Shou, Jian Pei, Ming Gong, Guido Zuccon, and Daxin Jiang. 2022. Bridging the gap between indexing and retrieval for differentiable search index with query generation. arXiv preprint arXiv:2206.10128

    2022 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.