Continual Learning Using Only Large Language Model Prompting

Bing Liu; Jiabao Qiu; Zixuan Ke

REVIEW 4 major objections 6 minor 15 references

Continual Learning Using Only Large Language Model Prompting

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Prompts alone can teach an LLM new classes with almost no forgetting.

desk verdict A genuinely new black-box CIL setup with a plausible summary-based method, but the headline 'very large margin' claim collapses against the paper's own zero-shot baseline on two of four datasets. read the letter →

arxiv 2412.15479 v1 pith:D3SI6ZAD submitted 2024-12-20 cs.CL cs.AI

Jiabao Qiu , Zixuan Ke , Bing Liu This is my paper · ORCID

classification cs.CLcs.AI

keywords continuallearningclass-incrementallargelanguagemodelsin-contextincrementalsummarizationcatastrophicforgettingprompting

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes CLOB, a continual learning paradigm in which a large language model is used purely as a black box: no parameters are tuned and no trainable components are added. Within this paradigm it introduces CIS, which compresses each class into a three-sentence summary, updates those summaries incrementally as new samples arrive, and classifies test samples by comparing them with the summaries through prompted confidence scores. The authors claim that CIS suffers almost no catastrophic forgetting and outperforms parameter-updating continual learning baselines by a large margin on four text classification benchmarks. If true, continual learning could run entirely through an API, with no access to model weights and no stored raw data.

What carries the argument

The load-bearing mechanism is incremental summarization, implemented by three prompted components: the Reflector ($M_r$) generates an initial three-sentence class summary from a handful of examples (Eq. 1); the Updator ($M_u$) merges new samples of an old class into the existing summary without exceeding three sentences (Eq. 2); and the Solver ($M_s$) assigns confidence scores to a test sample against chunked sets of class summaries, then re-ranks the top-$k$ candidates in a second prompt. This replaces stored raw data with a growing 'summary base,' and replaces parameter-based catastrophic forgetting with a much smaller prompt-based forgetting that the paper measures as the gap between incremental and single-batch summarization.

What would settle it

Run CIS on a long-document dataset where each document exceeds the context window; if the required chunk-then-summarize procedure loses the class signal and accuracy collapses, the core mechanism breaks in that regime. A cheaper check: replace each class summary with a summary of a different class and see whether the solver's confidence output changes—if accuracy stays high, the solver is not actually using summary content.

Watch

Extended reading notes

Core claim

The central discovery is that class-incremental learning can be done with no model updates at all: an LLM prompted to write a compact verbal summary of each class, then prompted to update that summary when more examples arrive, retains enough class knowledge to classify new samples accurately. The summary serves as a lossy but sufficient representation of the class; because only summaries are stored, the token limit of the LLM is respected and the model's internal parameters are never touched. Compared with EWC, LAMOL, and VAG, which fine-tune parameters or replay generated data, CIS reports substantially higher final accuracy, and it is close to or above joint prompting and fine-tuning upper bounds on several datasets.

Load-bearing premise

The method assumes that a three-sentence text summary of a handful of examples carries enough class-defining information, and that rewriting the summary with new examples keeps old knowledge intact instead of drifting.

Editorial extensions

If this is right

A developer can deploy continual learning on any LLM reachable through an API, with no weight access and no GPU training, so the approach scales to models too large to fine-tune.
Because raw training data is discarded after summarization, CIS offers a privacy-friendly CL pipeline: only textual summaries persist.
The blurry-boundary (online streaming) formulation means CIS can ingest samples as they arrive, even when task arrivals interleave.
The chunked two-step solver shows a practical way to perform classification over many classes under a fixed context window.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the summary mechanism proves robust at larger scale, it would suggest that the knowledge needed for classification is highly compressible, and that prompt-based compression could replace rehearsal buffers in other continual learning settings.
A natural extension is relation classification or long-document classification, where the paper itself notes that chunk-then-summarize pipelines remain untested; the method's viability there depends on whether summaries preserve cross-sentence dependencies.
The confidence-based solver may implicitly perform out-of-distribution detection; if so, CIS could be combined with explicit OOD scoring to reject unknown classes rather than forcing a choice among known ones.
The method's reliance on summary quality suggests a testable extension: feeding the summaries back into the LLM and asking it to write a discriminator between confusable classes could recover accuracy when summaries are coarse.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Interesting paper. The setup is genuinely new: class-incremental learning over a black-box LLM, using only prompts, with blurry boundaries, no parameter updates, and no stored raw data. The incremental summarization idea (CIS) is simple and plausible; the ablation showing blurry vs non-blurry accuracy about equal is a decent sign that the summary update is not catastrophically forgetting. On Banking-77 and CLINC-80, CIS clearly beats the zero-shot baseline, so the mechanism does extract useful class knowledge on intent-type datasets. Credit where due: this is a practical capability for API-only deployment, and the paper is clearly written.

The soft spots are mostly about the empirical claims. The abstract and conclusion say CIS 'outperforms baselines by a very large margin,' but the paper's own Table 2 contradicts that. With Llama-3.1, the Joint zero-shot baseline—class names, no examples—beats CIS on DBpedia-14 (93.36 vs 92.07–92.26) and crushes it on Reuters-14 (92.55 vs 83.97–84.61). On those two topic-classification datasets, the learned summaries are actively worse than using no summary at all. The comparison to VAG, which is the main support for the 'large margin' claim, is acknowledged as unfair (different backbone, full-data training). So the headline claim needs to be re-scoped to the datasets/backbones where CIS actually wins, and the zero-shot baseline needs to be treated as a primary reference point, not an afterthought.

Two smaller issues. The top-k (k=5) and the prompt templates were selected on the evaluation datasets (CLINC and DBpedia) with no validation split; that's mild tuning, but it should at least be disclosed and ideally justified with a held-out split. The theoretical justification is a sketch—citing Kim et al. for the WP/OOD condition and then asserting summaries help OOD is not an argument. That is fine as a heuristic, but don't call it theory. Also, no code or data is released; for a method whose whole point is reproducibility in a constrained API setting, a code link would help.

Bottom line: the paradigm is worth taking seriously and deserves a proper referee. The method is not consistently better than zero-shot prompting, and the authors need to fix the overclaiming and redo the baselines, but the idea is new, simple, and useful for a real deployment scenario. I'd send it to review, expecting major revision.

Referee Report

4 major / 6 minor

Summary. The paper proposes CLOB, a continual learning (CL) paradigm in which a large language model is treated as a black box and all learning is done through verbal prompting, and CIS, a method that incrementally maintains per-class text summaries generated and updated by the LLM. CIS is evaluated in a class-incremental setting with blurry task boundaries on four text classification datasets (Banking-77, CLINC-80, DBpedia-14, Reuters-14) using three LLMs. The reported results show that CIS substantially outperforms parameter-updating CL baselines (EWC, LAMOL, VAG) and exhibits minimal forgetting relative to a non-blurry setting.

Significance. If the claims held, the contribution would be significant: it would demonstrate that class-incremental learning can be performed with no parameter updates, no stored raw data, and only prompt-based summaries, which is attractive for API-only LLMs. The paper provides a genuinely new problem formulation and a simple, reasonably detailed method, with ablations over three LLMs and several blurry settings. However, the central empirical claim is weakened by the paper's own zero-shot baseline, which outperforms CIS on two of four datasets, and by the acknowledged confounded comparison with VAG. The strengths are the clear paradigm description, the explicit prompts, and the attempt to measure prompt-based forgetting; the weaknesses are the overstated conclusion and insufficient discussion of the zero-shot failure mode.

major comments (4)

[Abstract, Section 3.1, Table 2, Conclusion] The claim that 'CIS outperforms baselines by a very large margin' is not supported by the Joint zero-shot baseline in Table 2. On DBpedia-14, zero-shot (using only class names, no training data) achieves 93.36% while CIS (Llama) achieves 92.07–92.26%; on Reuters-14, zero-shot achieves 92.55% while CIS achieves 83.97–84.61%. Thus on two of four datasets, the incremental summaries actively reduce accuracy compared to using no data at all, and the margin is particularly large on Reuters. The paper should either rescope the headline claim to comparisons with parameter-updating CL baselines, provide a detailed explanation for why zero-shot is not an appropriate baseline for this setting, or add failure analysis showing when summaries are lossy or misleading.
[Section 3.1, 'CIS outperforms baselines' paragraph] The primary comparison against VAG is acknowledged as 'somewhat unfair' because VAG uses BART while CIS uses Llama-3.1. Since the cross-backbone comparison cannot separate the effect of the method from the effect of the much stronger backbone, the statement that CIS 'dramatically outperforms' VAG is not a controlled result. A fair comparison would require running CIS on the same backbone as VAG (or a parameter-updating method on Llama), or at least reporting a same-backbone parameter-updating baseline; without this, the superiority claim over state-of-the-art CL methods remains unsubstantiated.
[Section 2.1, 'Theoretical Justification'] The theoretical justification is not actually theoretical: the statement that 'anything that does not belong to the class is dissimilar to the summary' is an empirical assumption, and the citation to Kim et al. is not used to derive any testable condition. The claim that the method depends only on OOD detection is also presented without connecting it to the actual confidence-based solver, which does not perform OOD detection. This paragraph should be clearly labeled as intuition or replaced with a more careful argument (e.g., under what conditions on summary fidelity the method could be expected to work).
[Section 3, Baselines] The 'Joint: zero-shot' baseline is grouped under 'Joint systems' and described as an upper bound, but zero-shot is not an upper bound: it uses no training data and no summaries, and it outperforms CIS on two datasets. Mislabeling this baseline can mislead readers about what the comparison shows. The paper should distinguish upper-bound references (Joint prompting and fine-tuning with all data) from reference points that require no learning (zero-shot), and should treat the zero-shot results as a serious benchmark for any method that claims to learn from a handful of examples.

minor comments (6)

[Table 2] The column structure of Table 2 is difficult to parse because the header line combines multiple nested labels (e.g., 'Joint (Llama)' with 'Zero-shot', 'Prompting', 'Fine-tuning' and then separate columns for '7 samples' and 'full data'). Please restructure the table so each column has a single unambiguous label (e.g., 'Joint zero-shot', 'Joint prompting (7 samples)', 'Joint fine-tuning (7 samples)', 'Joint fine-tuning (full data)').
[Section 3.1, Ablations] The comparison between CIS (GPT) and CIS (classify - GPT) is described qualitatively ('In general, using classification in step 2 gives poorer results'), but the differences are sometimes within one standard deviation (e.g., Banking-77 3/4-blurry: 85.20±2.02 vs 85.58±0.57). Please either provide statistical significance tests or soften the claim accordingly.
[Section 3, Implementation details] The choice of k=5 is justified only by 'We also tried other k values with k=5 being the best.' Please report the range of k values tried and the corresponding accuracies; otherwise the reader cannot assess the sensitivity of the method to this hyperparameter.
[Appendix C] The random-order experiments are limited to DBpedia-14, and the GPT-3.5 results in Appendix C are higher than the main-table results, which the authors attribute to 'the update of OpenAI's system.' This is an uncontrolled confound; please note this caveat in the main text or present the appendix results as illustrative rather than as a controlled robustness study.
[Section 3, Datasets] There is a typo: 'DPpedia-14' should be 'DBpedia-14'.
[Section 5, Limitations] The limitations section is candid about long-document handling and vision applications, which is appreciated. Please also mention the small numbers of training (7) and test (50) samples per class as a budget-related limitation that may affect the generalizability of the reported accuracies.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: CIS summaries are generated from class data and tested on held-out samples; accuracy claims rest on external benchmarks, not on self-citations.

full rationale

None of the load-bearing claims reduce to their inputs by construction. CIS's class summaries are produced by Eq. (1) from training samples of each class and updated by Eq. (2) from later samples; classification in Eq. (3) compares a held-out test sample against these summaries. The reported accuracies (Tables 1-2) are measured on held-out data, so the empirical claim is not a fitted parameter renamed as prediction. The only in-house citation in the derivation chain is the 'Theoretical Justification' invoking Kim et al. (2022, 2023) for the necessary/sufficient conditions of CIL; this is motivational framing, not the source of the accuracy numbers, so it is not load-bearing for the paper's central claim. The weakness that Joint zero-shot outperforms CIS on DBpedia-14 and Reuters-14 (Table 2) is a support problem for the abstract's 'very large margin' claim, not a circularity. Score 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No new physical entities; the only constructed artifacts are textual summaries, which are outputs of the LLM rather than independent postulates. The main free choices are top-k and summary length, plus prompt templates selected on two datasets.

free parameters (2)

top-k in Step 2 = 5
Selected as best among tried values; used to pick candidate classes before the final confidence prompt (Section 3, Implementation details).
Summary length cap = 3 sentences, <100 tokens
Hand-set constraint in reflector/updator prompts (Appendix A); affects how much class knowledge is retained and whether the token limit is met.

assumptions (3)

domain assumption The CIL theory of Kim et al. (2022, 2023): good within-task prediction and OOD detection per task are necessary and sufficient for good CIL.
Invoked in 'Theoretical Justification' (Section 2) to argue summaries suffice; not proved in this paper.
domain assumption The LLM's summaries and confidence scores are reliable, stable verbal representations of class knowledge.
Core to CIS; if summaries lose discriminative information or drift on update, the method fails (Section 2.1, Eq. 1/2).
domain assumption The model is a black box that responds deterministically to prompts at temperature 0.
Experiments set temperature to 0 (Section 3); the method assumes repeatable outputs.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Continual Learning Using Only Large Language Model Prompting." pith.science (2026). https://pith.science/paper/D3SI6ZAD

@misc{pith2026241215479,
  author       = {Pith},
  title        = {Pith review of: Continual Learning Using Only Large Language Model Prompting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D3SI6ZAD}},
  note         = {Machine review of arXiv:2412.15479}
}

read the original abstract

We introduce CLOB, a novel continual learning (CL) paradigm wherein a large language model (LLM) is regarded as a black box. Learning is done incrementally via only verbal prompting. CLOB does not fine-tune any part of the LLM or add any trainable parameters to it. It is particularly suitable for LLMs that are accessible via APIs. We also propose a new CL technique, called CIS, based on incremental summarization that also overcomes the LLM's input length limit. Experiments show CIS outperforms baselines by a very large margin.

Figures

Figures reproduced from arXiv: 2412.15479 by the authors.

**Figure 1.** (1) Left: Overview of CIS in CLOB. (2) Right: Prompts used in each component of learning. Full prompts can be found in Appendix A. Some example summaries are given in Appendix B. sample x can be from any learned class, testing is done in two steps. In Step 1, we divide all classes into multiple chunks. Each chunk consists of summaries of k classes in the summary base (no saving of any of their data) that can fit wi… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 13 canonical work pages

[2]

Top-1: building; 2: village; 3: natural_place

work page
[3]

He has been a member of the Ottawa Senators for his entire career which began with the 1997–98 season

Top-1: building; 2: village; 3: natural_place Test sample 2: Chris Phillips (born March 9 1978) is a Canadian professional ice hockey player for the Ottawa Senators of the National Hockey League (NHL). He has been a member of the Ottawa Senators for his entire career which began with the 1997–98 season. He also serves as their alternate captain and is reg...

work page 1978
[5]

In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies

Overcoming catastrophic forgetting during domain adaptation of seq2seq language generation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. Qingbin Liu, Xiaoyan Yu, Shizhu He, Kang Liu, and Jun Zhao. 2021. Lifelong intent detection via multi- strategy reba...

arXiv 2022
[11]

Top-1: athlete; 2: office_holder; 3: company

work page
[12]

Top-1: athlete; 2: company; 3: office_holder

work page
[13]

Top-1: athlete; 2: office_holder; 3: company Test sample 3: The Himalayan agama (Paralaudakia hi- malayana) is an agamid lizard found in Central Asia and South Asia. Original top-3 output classes from step 1 in the experiments for the paper: Top-1: animal; 2: natural_place; 3: plant Top-3 output classes from the new ex- periments with 3 random orderings o...

work page
[14]

Top-1: plant; 2: animal; 3: natural_place

work page
[15]

Top-1: animal; 2: natural_place; 3: village

work page

Show all 15 references

[16]

Top-1: animal; 2: natural_place; 3: plant
[2016]

Overcoming catastrophic forgetting in neural networks. CoRR. Hyunseo Koh, Dahyun Kim, Jung-Woo Ha, and Jonghyun Choi. 2022. Online continual learning on class incremental blurry task configuration with anytime inference. Stefan Larson, Anish Mahendran, Joseph J. Peper, Christo...

2022
[2017]

Proceedings of the National Academy of Sciences, 114(13):3521–3526

Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521–3526. James Kirkpatrick, Razvan Pascanu, Neil C. Rabi- nowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Ag- ni...
[2021]

Advances in Neu- ral Information Processing Systems, 34

Achieving forgetting prevention and knowl- edge transfer in continual learning. Advances in Neu- ral Information Processing Systems, 34. Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. 2023. Continual pre- training of language models. In The Eleve...

2023
[2022]

CoRR, abs/2210.02406

Decomposed prompting: A modular approach for solving complex tasks. CoRR, abs/2210.02406. Gyuhak Kim, Changnan Xiao, Tatsuya Konishi, Zix- uan Ke, and Bing Liu. 2022. A theoretical study on solving continual learning. In Advances in Neural Information Processing Systems. Gyuha...

2022 arXiv
[2023]

In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- ume 2: Short Papers) , pages 1263–1276, Toronto, Canada

Class-incremental learning based on label gen- eration. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- ume 2: Short Papers) , pages 1263–1276, Toronto, Canada. Association for Computational Linguistics. Zhenmei Shi, Junyi Wei, ...

2023
[2024]

Transactions:

Large language models as optimizers. In The Twelfth International Conference on Learning Representations. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large la...

2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [2]

Top-1: building; 2: village; 3: natural_place

work page

[2] [3]

He has been a member of the Ottawa Senators for his entire career which began with the 1997–98 season

Top-1: building; 2: village; 3: natural_place Test sample 2: Chris Phillips (born March 9 1978) is a Canadian professional ice hockey player for the Ottawa Senators of the National Hockey League (NHL). He has been a member of the Ottawa Senators for his entire career which began with the 1997–98 season. He also serves as their alternate captain and is reg...

work page 1978

[3] [5]

In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies

Overcoming catastrophic forgetting during domain adaptation of seq2seq language generation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. Qingbin Liu, Xiaoyan Yu, Shizhu He, Kang Liu, and Jun Zhao. 2021. Lifelong intent detection via multi- strategy reba...

arXiv 2022

[4] [11]

Top-1: athlete; 2: office_holder; 3: company

work page

[5] [12]

Top-1: athlete; 2: company; 3: office_holder

work page

[6] [13]

Top-1: athlete; 2: office_holder; 3: company Test sample 3: The Himalayan agama (Paralaudakia hi- malayana) is an agamid lizard found in Central Asia and South Asia. Original top-3 output classes from step 1 in the experiments for the paper: Top-1: animal; 2: natural_place; 3: plant Top-3 output classes from the new ex- periments with 3 random orderings o...

work page

[7] [14]

Top-1: plant; 2: animal; 3: natural_place

work page

[8] [15]

Top-1: animal; 2: natural_place; 3: village

work page