{"id":"0b56f314-76af-48c4-a1a8-a362b7b8a0a1","arxiv_id":"2412.15479","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"CIS performs class-incremental learning on black-box LLMs by incrementally summarizing each class in text and classifying new samples by prompting for confidence scores against those summaries.","lead":"This paper presents a way to do continual learning on a large language model without touching its internals: it stores a short summary of each class and asks the model to update and compare those summaries through prompts alone. The method, called CIS, is tested on four text-classification datasets and is offered for situations where the model is only reachable through an API.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CIS's headline claim is contradicted by its own zero-shot baseline on DBpedia and Reuters, where no-training class names beat the learned summaries.","rationale":"The central claim of the paper is empirical: prompting-only incremental summarization beats baselines. The paper's own Table 2 contains a direct counterexample: a zero-shot baseline that uses no training data outperforms CIS on DBpedia-14 and Reuters-14 with the same backbone. This is more decisive than the reader's mechanism-level worry about summary lossiness because it is an observed failure, not a hypothetical one. The concern is load-bearing because it attacks the headline result, not just a component. However, the reader already flagged this contradiction in their rationale and assigned CONDITIONAL; our finding reinforces that verdict rather than changing it. The mechanism-level concern is real but secondary: even if summaries are lossy, the empirical claim could still hold on other datasets; here the empirical claim fails on half the datasets. Therefore the verdict remains CONDITIONAL: the paper needs re-scoping and fairer baselines before the claim can be accepted.","tokens_in":11785,"tokens_out":5416,"duration_ms":44929,"concrete_test":"Re-run the paper's CIS and zero-shot protocols with identical prompts and three seeds on DBpedia-14 and Reuters-14 using Llama-3.1, and compute paired differences and 95% confidence intervals. If zero-shot remains significantly better (e.g., >5 points on Reuters), the claim 'outperforms baselines' cannot be maintained. Additionally, add a CIS-ablated condition in which summaries are replaced by class names, to directly test whether summarization adds value over no learning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim ('CIS outperforms baselines by a very large margin') is not supported by its own Table 2. Using Llama-3.1, the Joint zero-shot baseline—which uses only class names and no training examples—achieves 93.36% on DBpedia-14 and 92.55% on Reuters-14, while CIS achieves 92.07–92.26% and 83.97–84.61% respectively. Thus on two of four datasets, the incremental summaries actively degrade accuracy relative to using no data at all. The authors acknowledge the comparison to VAG is 'somewhat unfair' (different backbone, full-data training), so the only fair baselines are zero-shot and Joint prompting; CIS is not consistently better than either, and it is substantially worse than zero-shot on Reuters. This contradicts the unqualified abstract/conclusion claim and indicates the summarization mechanism fails to preserve class-discriminative knowledge on these datasets. The paper should either rescope the claim to the datasets/backbones where CIS wins, or explain why zero-shot is not an appropriate baseline.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CLOB, a continual learning (CL) paradigm in which a large language model is treated as a black box and all learning is done through verbal prompting, and CIS, a method that incrementally maintains per-class text summaries generated and updated by the LLM. CIS is evaluated in a class-incremental setting with blurry task boundaries on four text classification datasets (Banking-77, CLINC-80, DBpedia-14, Reuters-14) using three LLMs. The reported results show that CIS substantially outperforms parameter-updating CL baselines (EWC, LAMOL, VAG) and exhibits minimal forgetting relative to a non-blurry setting.","tokens_in":11962,"tokens_out":4977,"duration_ms":37617,"significance":"If the claims held, the contribution would be significant: it would demonstrate that class-incremental learning can be performed with no parameter updates, no stored raw data, and only prompt-based summaries, which is attractive for API-only LLMs. The paper provides a genuinely new problem formulation and a simple, reasonably detailed method, with ablations over three LLMs and several blurry settings. However, the central empirical claim is weakened by the paper's own zero-shot baseline, which outperforms CIS on two of four datasets, and by the acknowledged confounded comparison with VAG. The strengths are the clear paradigm description, the explicit prompts, and the attempt to measure prompt-based forgetting; the weaknesses are the overstated conclusion and insufficient discussion of the zero-shot failure mode.","major_comments":[{"comment":"The claim that 'CIS outperforms baselines by a very large margin' is not supported by the Joint zero-shot baseline in Table 2. On DBpedia-14, zero-shot (using only class names, no training data) achieves 93.36% while CIS (Llama) achieves 92.07–92.26%; on Reuters-14, zero-shot achieves 92.55% while CIS achieves 83.97–84.61%. Thus on two of four datasets, the incremental summaries actively reduce accuracy compared to using no data at all, and the margin is particularly large on Reuters. The paper should either rescope the headline claim to comparisons with parameter-updating CL baselines, provide a detailed explanation for why zero-shot is not an appropriate baseline for this setting, or add failure analysis showing when summaries are lossy or misleading.","section":"Abstract, Section 3.1, Table 2, Conclusion"},{"comment":"The primary comparison against VAG is acknowledged as 'somewhat unfair' because VAG uses BART while CIS uses Llama-3.1. Since the cross-backbone comparison cannot separate the effect of the method from the effect of the much stronger backbone, the statement that CIS 'dramatically outperforms' VAG is not a controlled result. A fair comparison would require running CIS on the same backbone as VAG (or a parameter-updating method on Llama), or at least reporting a same-backbone parameter-updating baseline; without this, the superiority claim over state-of-the-art CL methods remains unsubstantiated.","section":"Section 3.1, 'CIS outperforms baselines' paragraph"},{"comment":"The theoretical justification is not actually theoretical: the statement that 'anything that does not belong to the class is dissimilar to the summary' is an empirical assumption, and the citation to Kim et al. is not used to derive any testable condition. The claim that the method depends only on OOD detection is also presented without connecting it to the actual confidence-based solver, which does not perform OOD detection. This paragraph should be clearly labeled as intuition or replaced with a more careful argument (e.g., under what conditions on summary fidelity the method could be expected to work).","section":"Section 2.1, 'Theoretical Justification'"},{"comment":"The 'Joint: zero-shot' baseline is grouped under 'Joint systems' and described as an upper bound, but zero-shot is not an upper bound: it uses no training data and no summaries, and it outperforms CIS on two datasets. Mislabeling this baseline can mislead readers about what the comparison shows. The paper should distinguish upper-bound references (Joint prompting and fine-tuning with all data) from reference points that require no learning (zero-shot), and should treat the zero-shot results as a serious benchmark for any method that claims to learn from a handful of examples.","section":"Section 3, Baselines"}],"minor_comments":[{"comment":"The column structure of Table 2 is difficult to parse because the header line combines multiple nested labels (e.g., 'Joint (Llama)' with 'Zero-shot', 'Prompting', 'Fine-tuning' and then separate columns for '7 samples' and 'full data'). Please restructure the table so each column has a single unambiguous label (e.g., 'Joint zero-shot', 'Joint prompting (7 samples)', 'Joint fine-tuning (7 samples)', 'Joint fine-tuning (full data)').","section":"Table 2"},{"comment":"The comparison between CIS (GPT) and CIS (classify - GPT) is described qualitatively ('In general, using classification in step 2 gives poorer results'), but the differences are sometimes within one standard deviation (e.g., Banking-77 3/4-blurry: 85.20±2.02 vs 85.58±0.57). Please either provide statistical significance tests or soften the claim accordingly.","section":"Section 3.1, Ablations"},{"comment":"The choice of k=5 is justified only by 'We also tried other k values with k=5 being the best.' Please report the range of k values tried and the corresponding accuracies; otherwise the reader cannot assess the sensitivity of the method to this hyperparameter.","section":"Section 3, Implementation details"},{"comment":"The random-order experiments are limited to DBpedia-14, and the GPT-3.5 results in Appendix C are higher than the main-table results, which the authors attribute to 'the update of OpenAI's system.' This is an uncontrolled confound; please note this caveat in the main text or present the appendix results as illustrative rather than as a controlled robustness study.","section":"Appendix C"},{"comment":"There is a typo: 'DPpedia-14' should be 'DBpedia-14'.","section":"Section 3, Datasets"},{"comment":"The limitations section is candid about long-document handling and vision applications, which is appreciated. Please also mention the small numbers of training (7) and test (50) samples per class as a budget-related limitation that may affect the generalizability of the reported accuracies.","section":"Section 5, Limitations"}],"recommendation":"major_revision","confidential_remarks":"The paper comes from a well-known group in continual learning and the proposed paradigm is interesting, but the empirical claims need substantial revision. The most serious issue is the zero-shot baseline: on DBpedia and Reuters, using only class names without any training examples beats CIS, which contradicts the abstract and conclusion. This is not a matter of a missing citation or a minor wording issue; it changes the contribution's significance. I recommend that the editor require the authors to either (a) explicitly limit their claims to comparisons with parameter-updating CL methods and add a discussion of the zero-shot gap, or (b) provide a new mechanism or analysis that explains and ideally fixes the lossiness of summaries on those datasets. I would not reject the paper outright because the paradigm and the summary-updating idea are novel and could be valuable with appropriately scoped claims, but in its current form the central claim is overstated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Interesting paper. The setup is genuinely new: class-incremental learning over a black-box LLM, using only prompts, with blurry boundaries, no parameter updates, and no stored raw data. The incremental summarization idea (CIS) is simple and plausible; the ablation showing blurry vs non-blurry accuracy about equal is a decent sign that the summary update is not catastrophically forgetting. On Banking-77 and CLINC-80, CIS clearly beats the zero-shot baseline, so the mechanism does extract useful class knowledge on intent-type datasets. Credit where due: this is a practical capability for API-only deployment, and the paper is clearly written.\n\nThe soft spots are mostly about the empirical claims. The abstract and conclusion say CIS 'outperforms baselines by a very large margin,' but the paper's own Table 2 contradicts that. With Llama-3.1, the Joint zero-shot baseline—class names, no examples—beats CIS on DBpedia-14 (93.36 vs 92.07–92.26) and crushes it on Reuters-14 (92.55 vs 83.97–84.61). On those two topic-classification datasets, the learned summaries are actively worse than using no summary at all. The comparison to VAG, which is the main support for the 'large margin' claim, is acknowledged as unfair (different backbone, full-data training). So the headline claim needs to be re-scoped to the datasets/backbones where CIS actually wins, and the zero-shot baseline needs to be treated as a primary reference point, not an afterthought.\n\nTwo smaller issues. The top-k (k=5) and the prompt templates were selected on the evaluation datasets (CLINC and DBpedia) with no validation split; that's mild tuning, but it should at least be disclosed and ideally justified with a held-out split. The theoretical justification is a sketch—citing Kim et al. for the WP/OOD condition and then asserting summaries help OOD is not an argument. That is fine as a heuristic, but don't call it theory. Also, no code or data is released; for a method whose whole point is reproducibility in a constrained API setting, a code link would help.\n\nBottom line: the paradigm is worth taking seriously and deserves a proper referee. The method is not consistently better than zero-shot prompting, and the authors need to fix the overclaiming and redo the baselines, but the idea is new, simple, and useful for a real deployment scenario. I'd send it to review, expecting major revision.","headline":"A genuinely new black-box CIL setup with a plausible summary-based method, but the headline 'very large margin' claim collapses against the paper's own zero-shot baseline on two of four datasets.","tokens_in":12532,"tokens_out":2949,"would_cite":true,"duration_ms":23204,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Prompts alone can teach an LLM new classes with almost no forgetting.","keywords":["continual learning","class-incremental learning","large language models","in-context learning","incremental summarization","catastrophic forgetting","prompting"],"falsifier":"Run CIS on a long-document dataset where each document exceeds the context window; if the required chunk-then-summarize procedure loses the class signal and accuracy collapses, the core mechanism breaks in that regime. A cheaper check: replace each class summary with a summary of a different class and see whether the solver's confidence output changes—if accuracy stays high, the solver is not actually using summary content.","tokens_in":11553,"feed_emoji":"💬","tokens_out":5478,"duration_ms":40400,"temperature":0.7,"pith_summary":"The paper proposes CLOB, a continual learning paradigm in which a large language model is used purely as a black box: no parameters are tuned and no trainable components are added. Within this paradigm it introduces CIS, which compresses each class into a three-sentence summary, updates those summaries incrementally as new samples arrive, and classifies test samples by comparing them with the summaries through prompted confidence scores. The authors claim that CIS suffers almost no catastrophic forgetting and outperforms parameter-updating continual learning baselines by a large margin on four text classification benchmarks. If true, continual learning could run entirely through an API, with no access to model weights and no stored raw data.","feed_headline":"Prompts alone can teach an LLM new classes with almost no forgetting","feed_subtitle":"Incremental three-sentence summaries let a black-box LLM keep learning text classes without storing raw data.","key_machinery":"The load-bearing mechanism is incremental summarization, implemented by three prompted components: the Reflector ($M_r$) generates an initial three-sentence class summary from a handful of examples (Eq. 1); the Updator ($M_u$) merges new samples of an old class into the existing summary without exceeding three sentences (Eq. 2); and the Solver ($M_s$) assigns confidence scores to a test sample against chunked sets of class summaries, then re-ranks the top-$k$ candidates in a second prompt. This replaces stored raw data with a growing 'summary base,' and replaces parameter-based catastrophic forgetting with a much smaller prompt-based forgetting that the paper measures as the gap between incremental and single-batch summarization.","core_discovery":"The central discovery is that class-incremental learning can be done with no model updates at all: an LLM prompted to write a compact verbal summary of each class, then prompted to update that summary when more examples arrive, retains enough class knowledge to classify new samples accurately. The summary serves as a lossy but sufficient representation of the class; because only summaries are stored, the token limit of the LLM is respected and the model's internal parameters are never touched. Compared with EWC, LAMOL, and VAG, which fine-tune parameters or replay generated data, CIS reports substantially higher final accuracy, and it is close to or above joint prompting and fine-tuning upper bounds on several datasets.","pith_inferences":["If the summary mechanism proves robust at larger scale, it would suggest that the knowledge needed for classification is highly compressible, and that prompt-based compression could replace rehearsal buffers in other continual learning settings.","A natural extension is relation classification or long-document classification, where the paper itself notes that chunk-then-summarize pipelines remain untested; the method's viability there depends on whether summaries preserve cross-sentence dependencies.","The confidence-based solver may implicitly perform out-of-distribution detection; if so, CIS could be combined with explicit OOD scoring to reject unknown classes rather than forcing a choice among known ones.","The method's reliance on summary quality suggests a testable extension: feeding the summaries back into the LLM and asking it to write a discriminator between confusable classes could recover accuracy when summaries are coarse."],"forward_implications":["A developer can deploy continual learning on any LLM reachable through an API, with no weight access and no GPU training, so the approach scales to models too large to fine-tune.","Because raw training data is discarded after summarization, CIS offers a privacy-friendly CL pipeline: only textual summaries persist.","The blurry-boundary (online streaming) formulation means CIS can ingest samples as they arrive, even when task arrivals interleave.","The chunked two-step solver shows a practical way to perform classification over many classes under a fixed context window."],"supporting_citations":[{"why":"Supplies the theoretical result that good within-task prediction and OOD detection are necessary and sufficient for good CIL, which motivates representing each class as a single summary.","marker":"Kim et al. 2022"},{"why":"Extends the theoretical grounding for CIL learnability, supporting the claim that summary-based OOD comparison suffices.","marker":"Kim et al. 2023"},{"why":"Provides VAG, the state-of-the-art parameter-updating CIL baseline that CIS is primarily compared against.","marker":"Shao et al. 2023"},{"why":"Provides the Banking-77 intent classification dataset used in evaluation.","marker":"Casanueva et al. 2020"},{"why":"Provides the CLINC-80 intent and out-of-scope dataset used in evaluation.","marker":"Larson et al. 2019"},{"why":"Provides the DBpedia-14 ontology classification dataset.","marker":"Lehmann et al. 2014"},{"why":"Provides the Reuters-14 text categorization dataset.","marker":"Lewis 1997"},{"why":"Defines the online/streaming CIL setting that CIS operates in.","marker":"Guo et al. 2022"}],"fun_headline_variants":["Prompt-only continual learning: no fine-tuning, just summaries","Black-box LLM learns new classes via prompts and summaries","Incremental summaries let LLMs learn without parameter updates","CIS: prompt-based continual learning beats fine-tuning","No model updates: LLM prompted to summarize classes incrementally"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a three-sentence text summary of a handful of examples carries enough class-defining information, and that rewriting the summary with new examples keeps old knowledge intact instead of drifting.","fun_headline_variants_meta":{"raw":{"variants":["Prompt-only continual learning: no fine-tuning, just summaries","Black-box LLM learns new classes via prompts and summaries","Incremental summaries let LLMs learn without parameter updates","CIS: prompt-based continual learning beats fine-tuning","No model updates: LLM prompted to summarize classes incrementally"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000228,"raw_usage":{"total_tokens":1385,"prompt_tokens":767,"completion_tokens":618,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":383,"completion_tokens_details":{"reasoning_tokens":537}},"tokens_in":383,"tokens_out":618,"duration_ms":5606,"temperature":1.0,"reasoning_tokens":537,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:23:06.074421+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CIS on a long-document dataset where each document exceeds the context window; if the required chunk-then-summarize procedure loses the class signal and accuracy collapses, the core mechanism breaks in that regime. A cheaper check: replace each class summary with a summary of a different class and see whether the solver's confidence output changes—if accuracy stays high, the solver is not actually using summary content.","supporting_citations":[],"review_version":1}