I Don't Know: Explicit Modeling of Uncertainty with an [IDK] Token

Eden Biran; Gerard de Melo; Konstantin Dobler; Roi Cohen

REVIEW 3 major objections 5 minor 1 cited by

I Don't Know: Explicit Modeling of Uncertainty with an [IDK] Token

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that adding a special [IDK] token and shifting wrong-prediction probability mass toward it during continual pretraining lets language models abstain on answers they do not know, raising factual precision with only a small…

desk verdict A genuinely new self-supervised abstention objective, but the 'small recall loss' claim is contradicted by PopQA and the anti-false-positive regularizer has a blind spot that explains it. read the letter →

arxiv 2412.06676 v1 pith:IWYKFOPT submitted 2024-12-09 cs.LG cs.CL

Roi Cohen , Konstantin Dobler , Eden Biran , Gerard de Melo This is my paper · ORCID

classification cs.LGcs.CL

keywords hallucinationmitigationuncertaintyestimation[IDK]tokenmodelcalibrationselectivepredictioncontinualpretrainingfactualprecisionknowledgerecall

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a language model can be trained to abstain by emitting a special [IDK] token instead of a wrong answer, using only the model's own prediction errors on ordinary web text. The authors add the token to the vocabulary and modify the cross-entropy objective so that, whenever the model is wrong and uncertain, some of the probability mass meant for the correct token is redirected to [IDK]. Across several model families and factual benchmarks, IDK-tuned models answer far fewer questions incorrectly while losing only a small share of the facts the base model knew, and the gains hold against confidence-threshold, P(True), and semantic-entropy baselines. If the claim is right, it offers a self-supervised route to hallucination reduction that does not require labeled uncertainty data or answer choices.

What carries the argument

The load-bearing object is the [IDK] token plus the IDK objective, a replacement for the standard next-token cross-entropy loss. The loss is $L_{\mathrm{IDK}} = L_{\mathrm{CE}}(\hat{y}, (1-\lambda) y + \lambda \mathbf{1}_{[\mathrm{IDK}]})$, where $y$ is the gold token's one-hot target and $\mathbf{1}_{[\mathrm{IDK}]}$ is a one-hot target for [IDK]. The uncertainty factor $\lambda = \Pi(1 - p(\mathrm{gold}) / \max_i p(i))$ decides how much target mass moves to [IDK]: zero when the gold token leads the prediction, approaching one when the model's best guess is far from the gold token. An anti-false-positive term is added only when the prediction is correct, so the model learns not to reach for [IDK] when it knows the answer.

What would settle it

Run the IDK-tuned Mistral-7B on open-ended factual prompts that require multi-token answers, strip [IDK] out of the output stream, and measure the precision of the remaining text: if precision does not rise relative to the base model while the [IDK] firing rate stays high, the abstention signal is not tracking factual knowledge. A complementary check is to present the same factual question in easy and deliberately hard paraphrases and test whether the model abstains far more on the hard paraphrase even when the base model answers both correctly.

Watch

Extended reading notes

Core claim

The central discovery, stated on the paper's own terms, is that an LLM's next-token prediction errors encode enough uncertainty signal to teach the model to know when it does not know. IDK-tuning adds a randomly initialized [IDK] token and modifies the standard next-token cross-entropy loss so that wrong predictions share some target probability mass with [IDK], with the amount determined by how far the gold token is from the model's top prediction. An additional regularization term suppresses [IDK] use on correct predictions. On LAMA, TriviaQA, PopQA, and the lm-eval-harness tasks, the resulting models show large precision gains with small recall losses, for instance raising Mistral-7B's precision on the LAMA Google-RE subset from 48.1 to 71.1 while recall falls from 48.1 to 40.6. The authors also show that the effect scales with model size and that IDK-tuning fails on the smallest 70M and 160M models, whose training diverges or collapses.

Load-bearing premise

The load-bearing premise is that uncertainty signals learned from ordinary next-token prediction errors on web text transfer to factual question answering, even though most of those errors concern grammar, style, and rare word patterns rather than missing knowledge.

Editorial extensions

If this is right

On closed-book factual sentence completion, IDK-tuned Mistral-7B reaches higher precision than the base model on every reported benchmark, with F1 rising on LAMA, TriviaQA, and PopQA.
The precision gains come at a modest recall cost, and the authors report the same pattern on multiple-choice tasks from the evaluation harness.
The method transfers across architectures and sizes: BERT, Pythia models from 410M to 2.8B, and Mistral-7B all improve, with F1 growing roughly log-linearly with model scale.
Continual pretraining on The Pile with the standard loss does not reproduce the gains, so the authors attribute the improvement to the IDK objective rather than to new knowledge in the extra training text.
General language skills survive mostly intact: on three summarization benchmarks the IDK-tuned Mistral scores within a fraction of a ROUGE-L point of the base model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper evaluates only short completions; a reader might extend the same check to open-ended generation, where [IDK] would need to compete with natural-language refusals before the method becomes a deployable guardrail.
Because the loss fires on every wrong token in web text, the [IDK] token may partly encode 'hard-to-predict wording' rather than 'missing fact'; restricting the loss to entity-like tokens is a natural variant that could preserve more recall.
The collapse on 70M and 160M models hints at a minimum scale for self-taught abstention; probing that threshold would say which small deployed models can safely rely on this approach.
Since the hyperparameter $\Pi$ controls the maximum mass shifted to [IDK], it acts as a precision-recall dial that applications could tune per domain without retraining.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Roi, read this one for a clean example of a good idea fighting an overstated conclusion. The IDK objective is genuinely new: a special token, a self-supervised loss that shifts target mass toward [IDK] when the model's own uncertainty is high (measured by the gap between gold probability and top probability). No labeled data, a proper control (Mistral on The Pile with standard CE), and a sensible scaling analysis across Pythia sizes. The ablations are informative and the writing is clear. The paper earns credit for the control and for releasing checkpoints (though the code link should be checked).

The soft spot is load-bearing. The abstract claims only a small loss of encoded knowledge, but PopQA recall drops from 35.5 to 20.5, a 42% relative loss. That is not small. Worse, the regularizer they designed to prevent false abstention, L_FP-reg, only fires when lambda=0, i.e., when the model's argmax is already the gold token. The moment the model flips to [IDK] on a known fact, lambda>0 and the loss pushes even more mass onto [IDK]. So the regularizer cannot directly penalize the exact failure mode it is named for. The empirical tradeoff curves show it helps reduce the IDK error rate, but it does not stop the self-reinforcing loop. This explains the PopQA result and should force a rewrite of the central recall claim.

Other issues are more minor: no error bars or multiple seeds (they admit this in the checklist), and the exact training sample from The Pile is not specified, which hurts exact reproducibility. The confidence threshold baseline gets a threshold search on the dev set, which is fair but should be stated as such. The reader's worry about transfer from arbitrary next-token errors to factual QA is legitimate; the authors themselves note the objective might be ill-posed for some next-token predictions.

None of this makes the paper circular; the objective is trained on unlabeled data and evaluated on held-out benchmarks. The idea is plausible and likely useful for retrieval-augmented generation and hallucination mitigation. But the paper as written overstates what is shown. A serious referee should engage with it, but the authors need to temper the abstract, add seeds or bootstrap intervals, and ideally fix the regularizer so it penalizes false IDK even when IDK is the argmax. Who is this for? Anyone working on abstention, calibration, or selective prediction. I would bring it to a reading group; the stress-test point is a good teaching moment.

Referee Report

3 major / 5 minor

Summary. The paper proposes IDK-tuning: a special [IDK] token is added to an LLM's vocabulary, and the standard next-token cross-entropy objective is modified during continued pretraining so that, when the model is uncertain, probability mass in the target distribution is shifted toward [IDK]. The shift is controlled by an uncertainty factor λ (Eq. 2) and an upper bound Π, with an additional false-positive regularization term LFP-reg (Eq. 3). The method is trained on unlabeled text from The Pile and evaluated on closed-book factual sentence completion (LAMA, TriviaQA, PopQA) and multiple-choice benchmarks, across Mistral-7B, Pythia 70m–2.8B, and BERT. The authors report large precision gains with, they claim, only small recall loss, and provide ablations of Π, adaptive vs. fixed λ, and LFP-reg.

Significance. If the main claim holds, IDK-tuning is a practically attractive self-supervised route to abstention-based hallucination reduction: it requires no labeled uncertainty data, is a drop-in modification of the language-modeling loss, and is evaluated across multiple architectures and sizes. The paper includes a useful control for continued training on The Pile, a scaling analysis across the Pythia suite, and detailed ablations, and the authors commit to releasing code and checkpoints. However, the central precision–recall tradeoff claim is not uniformly supported by the reported numbers, and the proposed anti-false-positive regularizer has a structural gap that limits its ability to prevent the very failure mode it targets. These issues are load-bearing because the value of the method is precisely the claimed precision/recall tradeoff.

major comments (3)

[Section 2.2, Eqs. (2)–(4)] The LFP-reg regularizer is only active when λ = 0, which by Eq. (2) means the gold token is the argmax prediction. Therefore the model is never directly penalized for making [IDK] its top prediction on an example it actually knows. Once [IDK] is the argmax, λ > 0 and the loss switches to LIDK, whose target assigns probability λ to [IDK]; depending on the current p([IDK]) relative to that λ, the gradient can even increase mass on [IDK] rather than reduce it. The regularizer can only act before the model crosses into the false-abstention regime, so it does not directly protect against the failure mode it is named for. The PopQA result in Table 1 (recall 35.5 → 20.5) is a concrete manifestation of this gap, and the mechanism intended to limit recall loss is therefore only partially effective for exactly the cases where recall is lost.
[Abstract and Section 4.1, Table 1] The abstract and Section 4 claim 'only a small decrease in recall' of factual knowledge, but Table 1 shows PopQA recall dropping from 35.5 to 20.5, a 42% relative loss, and LAMA Google-RE dropping from 48.1 to 40.6. These are not uniformly small decreases. The claim appears to be a summary over datasets that masks substantial task-specific degradation. The authors should either report the tradeoff per dataset and temper the global claim, or provide an analysis of why PopQA (which tests popular, often predictive facts) is disproportionately affected. Since the paper's principal contribution is the precision–recall tradeoff, this discrepancy is central rather than a presentation issue.
[Sections 3.1 and 4.1] All main results are reported from single training runs with no error bars, confidence intervals, or multiple seeds. Given that Section 4.3 shows that the same loss can produce divergent optimization behavior for smaller Pythia models, and given that the PopQA recall drop is large, single-run results are not sufficient to establish whether the observed tradeoff is systematic or partly a consequence of optimization instability. The paper should at minimum state this limitation in the main text and, ideally, provide a small number of repeated runs for the central configurations or a sensitivity analysis around the reported numbers.

minor comments (5)

[Section 2.2, Eq. (2)] The notation in Eq. (2) uses prob(yt = [gold] | y<t, x) and max_i(prob(yt = i | y<t, x)); it would be clearer to define these as model probabilities p_gold and p_max before using them in the formula, especially because the denominator is the max over the vocabulary including [IDK].
[Section 3.2, baselines] The Confidence Threshold baseline is tuned on a development set to find the best threshold, while IDK-tuning uses a fixed Π = 1/2 with no tuning. This is a favorable comparison for the baselines, not the proposed method, but the asymmetry should be stated explicitly when interpreting the results.
[Table 3] The caption contains a grammatical error: 'for of our IDK-tuned bert-base-cased' should read 'for our IDK-tuned bert-base-cased'.
[Table 4] 'RougleL' is a typo for 'ROUGE-L'.
[Section 6, Conclusion] The conclusion acknowledges that 'we potentially apply our objective for next-token predictions where it might be ill-posed' and suggests filtering for named entities as a fix. This is an important caveat that is not reflected in the abstract or the main-results discussion; it should be stated earlier, since the transfer of uncertainty signals from arbitrary text to factual QA is a core assumption of the method.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; IDK-tuning is an externally evaluated empirical method, not a derivation that presupposes its own conclusions.

full rationale

The central claim is an empirical training outcome, not a derivation. No equation reduces a reported result to an input of the method. IDK-tuning trains on The Pile with a modified cross-entropy objective, and precision/recall are measured on external closed-book QA benchmarks (LAMA, TriviaQA, PopQA, and lm-eval-harness datasets). The uncertainty factor lambda depends on the model's own softmax distribution, but that is the mechanism being tested, not a circular reuse of the evaluation signal. The Pile control model (Mistral-7B-v0.1 trained with regular cross-entropy on the same data) rules out continued pretraining as the source of the observed gains. The hyperparameter Pi is fixed without tuning, and no test-set fitting occurs. Self-citations in the paper (Cohen et al. 2023a, 2023b, 2024) support background statements about knowledge storage, self-contradiction, and prompting-based uncertainty and are not load-bearing for the central result. The skeptical concern that LFP-reg never directly penalizes [IDK]-as-top-prediction on known facts is a substantive question about the robustness of the precision-recall tradeoff, but it does not make the claim circular. The derivation chain is self-contained: the method is trained, the control is in place, and the evaluation is external.

Assumptions & free parameters 1 free parameters · 3 assumptions · 1 invented entities

The central claim rests primarily on hyperparameter Π (set by hand) and on three domain assumptions: that next-token uncertainty generalizes to factual uncertainty, that the continued pretraining does not add knowledge, and that [IDK] probability can be read as a refusal signal. The [IDK] token itself is a new vocabulary entity with direct empirical evaluation.

free parameters (1)

Π (upper bound on [IDK] probability mass in target) = 0.5
Set by hand without tuning (Section 2.2); controls the maximum fraction of target probability shifted to [IDK]. The paper states 'In practice, we do not tune this and set Π = 1/2.'

assumptions (3)

domain assumption Uncertainty in next-token prediction on arbitrary text is a valid proxy for factual uncertainty
The [IDK] token is trained on all next-token errors from The Pile, including non-factual tokens. The paper relies on this transfer to improve factual QA, and only partially addresses it via evaluations.
domain assumption Continuing pretraining on 1B tokens from The Pile does not introduce new factual knowledge that explains the results
The paper tests this with a control model trained with standard CE on the same data (Table 1), but this control is itself subject to training noise and does not fully rule out interactions.
domain assumption The [IDK] token probability mass can be treated as a refusal signal at inference time without further calibration
Evaluation measures precision and recall by treating any non-[IDK] generation as an answer, which assumes the model's use of [IDK] maps cleanly to a binary abstain decision.

invented entities (1)

[IDK] special token independent evidence
purpose: Provides an explicit, trainable abstention symbol in the LM vocabulary so the model can express uncertainty in its output.
The token's behavior is evaluated on external factual benchmarks and its probability mass is shown to track prediction correctness on those benchmarks, giving it a falsifiable handle outside the paper.

how reviews work

0 comments

Cite this review

Pith. "Pith review of I Don't Know: Explicit Modeling of Uncertainty with an [IDK] Token." pith.science (2026). https://pith.science/paper/IWYKFOPT

@misc{pith2026241206676,
  author       = {Pith},
  title        = {Pith review of: I Don't Know: Explicit Modeling of Uncertainty with an [IDK] Token},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IWYKFOPT}},
  note         = {Machine review of arXiv:2412.06676}
}

read the original abstract

Large Language Models are known to capture real-world knowledge, allowing them to excel in many downstream tasks. Despite recent advances, these models are still prone to what are commonly known as hallucinations, causing them to emit unwanted and factually incorrect text. In this work, we propose a novel calibration method that can be used to combat hallucinations. We add a special [IDK] ("I don't know") token to the model's vocabulary and introduce an objective function that shifts probability mass to the [IDK] token for incorrect predictions. This approach allows the model to express uncertainty in its output explicitly. We evaluate our proposed method across multiple model architectures and factual downstream tasks. We find that models trained with our method are able to express uncertainty in places where they would previously make mistakes while suffering only a small loss of encoded knowledge. We further perform extensive ablation studies of multiple variations of our approach and provide a detailed analysis of the precision-recall tradeoff of our method.

Figures

Figures reproduced from arXiv: 2412.06676 by the authors.

**Figure 1.** Illustration of our proposed IDK objective. During continual pretraining, we shift some probability mass of wrong predictions towards a special [IDK] token. The amount of shifted probability mass depends on the uncertainty in the model’s prediction. We detail our method in Section 2. are better aligned with their quality. This calibration allows LLMs to explicitly express uncertainty, allowing them to caveat their r… view at source ↗

**Figure 2.** Average performance on closed-book factual sentence completion benchmarks of [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 4.** Tradeoff between IDK recall and IDK error rate for different parameter combinations. We annotate each data point with its corresponding Π value. bert-base-cased results [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Scene Generation at Absolute Scale: Utilizing Semantic and Geometric Guidance From Text for Accurate and Interpretable 3D Indoor Scene Generation
cs.CV 2026-03 unverdicted novelty 5.0 of 10

A text-to-3D pipeline that predicts a global semantic-geometric layout, conditions panoramic and video diffusion on it, and fuses views with 3D Gaussian Splatting into an absolute-scale navigable indoor scene.

Reference graph

Works this paper leans on

18 extracted references · 15 canonical work pages · cited by 1 Pith paper

[1]

Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper

Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: For all claims made in the abstract and introduction, we provide experimental results that back these claims. Guidelines: • The answer NA means that the abstract and introduction do not include the ...

work page
[2]

Limitations

Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: See Section 7. Guidelines: • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separa...

work page
[3]

Guidelines: • The answer NA means that the paper does not include theoretical results

Theory Assumptions and Proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] 18 Justification: We do not include new theoretical results that warrant proofs. Guidelines: • The answer NA means that the paper does not include theoretical results. • All the theorems, ...

work page
[4]

Guidelines: • The answer NA means that the paper does not include experiments

Experimental Result Reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: We provide the formulation...

work page
[5]

Guidelines: • The answer NA means that paper does not include experiments requiring code

Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? 19 Answer: [No] Justification: We will publish all datasets, code, and model checkpoints with camera-ready version. Guidelines: • The an...

work page
[6]

See also our answer to question 4

Experimental Setting/Details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: We provide these details in Section 3.1. See also our answer to question 4. Guidelines: • The answer NA means...

work page
[7]

Our large-scale continual training experiments are, unfortunately, too expensive to repeat multiple times with different random seeds

Experiment Statistical Significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [No] Justification: Our evaluations are done via prompting rather than fine-tuning (see Section 3.2), yielding no source of randomness to aggregate int...

work page
[8]

Guidelines: • The answer NA means that the paper does not include experiments

Experiments Compute Resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We provide those details in APPENDIX (see Appendix B). Guidelines: • The answer NA means that the paper...

work page

Show all 18 references

[9]

One potential impact is discussed in Appendix A

Code Of Ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethicshttps://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: We carefully read the NeurIPS Code of Ethics document and made sure it’s aligned...
[10]

Guidelines: • The answer NA means that there is no societal impact of the work performed

Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss the potential social impacts in Appendix A. Guidelines: • The answer NA means that there is no...
[11]

We believe that the resulting checkpoints do not warrant additional safeguards

Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: We only continu...
[12]

Guidelines: • The answer NA means that the paper does not use existing assets

Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: In Section 3, we...
[13]

Additionally, in Appendix C, we provide the complete prompt we used in order to create our closed-booked sentence completion dataset as discussed in Section 3.2

New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: Yes, we properly explained each of the new assets we introduced. Additionally, in Appendix C, we provide the complet...
[14]

Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

Crowdsourcing and Research with Human Subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...
[15]

Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...
[2021]

doi: 10.18653/v1/2021.emnlp-main.619

Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.619. URL https://aclanthology.org/2021.emnlp-main.619. Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and ...

2021 arXiv
[2022]

URL https://aclanthology.org/2022.tacl-1.43

doi: 10.1162/tacl_a_00486. URL https://aclanthology.org/2022.tacl-1.43. Amos Azaria and Tom Mitchell. The internal state of an LLM knows when it’s lying. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Com- putational Linguistics: EMNLP 2...

2022 arXiv
[2023]

Where was Michael Jackson born?

Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.322. URL https://aclanthology.org/2023.findings-acl.322. Yi Tay, Vinh Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, et al. Transformer memory as ...

2023 doi

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper

Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: For all claims made in the abstract and introduction, we provide experimental results that back these claims. Guidelines: • The answer NA means that the abstract and introduction do not include the ...

work page

[2] [2]

Limitations

Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: See Section 7. Guidelines: • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separa...

work page

[3] [3]

Guidelines: • The answer NA means that the paper does not include theoretical results

Theory Assumptions and Proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] 18 Justification: We do not include new theoretical results that warrant proofs. Guidelines: • The answer NA means that the paper does not include theoretical results. • All the theorems, ...

work page

[4] [4]

Guidelines: • The answer NA means that the paper does not include experiments

Experimental Result Reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: We provide the formulation...

work page

[5] [5]

Guidelines: • The answer NA means that paper does not include experiments requiring code

Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? 19 Answer: [No] Justification: We will publish all datasets, code, and model checkpoints with camera-ready version. Guidelines: • The an...

work page

[6] [6]

See also our answer to question 4

Experimental Setting/Details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: We provide these details in Section 3.1. See also our answer to question 4. Guidelines: • The answer NA means...

work page

[7] [7]

Our large-scale continual training experiments are, unfortunately, too expensive to repeat multiple times with different random seeds

Experiment Statistical Significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [No] Justification: Our evaluations are done via prompting rather than fine-tuning (see Section 3.2), yielding no source of randomness to aggregate int...

work page

[8] [8]

Guidelines: • The answer NA means that the paper does not include experiments

Experiments Compute Resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We provide those details in APPENDIX (see Appendix B). Guidelines: • The answer NA means that the paper...

work page