{"id":"9561e9f5-213d-43b7-a97f-0f9c54de4080","arxiv_id":"2601.04465","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A single token embedding trained on concept definitions can directionally steer a frozen LLM's behavior, but hallucination gains come mainly from abstention and new factual details are not reliably stored.","lead":"This paper introduces 'concept tokens'—new special tokens whose embeddings are trained on natural-language definitions with a frozen language model, then used to steer behavior such as hallucination and recasting. The method shows a directional effect in tests, but hallucination control mainly increases abstentions and novel factual details remain unreliable.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The directional-steering claim is plausible and internally consistent, but the hallucination headline rests on single runs without confidence intervals; the asserted-vs-baseline gap (31.2 vs 28.7) could be noise.","rationale":"The reader identifies the weakest assumption as semantic compositionality—whether the model treats tc as a meaningful operator. I find this concern less load-bearing than the reader does: the paper's own experiments partially answer it. If tc were treated as an unknown token or a literal string, the prompt variations in §4.1 and §4.2 that differ only by negation or assertion would not produce such consistent directional shifts. The recasting experiment is especially telling: replacing the word 'recasting' with tc reverses the correction style (62.33% recasting vs 16.74% explicit correction for the word), showing the embedding carries different behavioral content than a simple lexical synonym. The towers experiment's next-token ambiguity (Buenos Aires vs Montevideo) is a limitation on factual specificity, which the paper explicitly acknowledges in the Limitations section and Appendix A.2, not a refutation of the steering claim. My central concern is therefore statistical robustness, not compositionality. All §4.1 results are single runs with no error bars, and the directional sub-claim about assertion relies on a small percentage gap. This is a correctness risk that the reader's verdict already flags ('no confidence intervals or significance tests'), but it deserves to be the primary caveat. The recasting results, with wider margins, are more convincing and partially compensate. Verdict remains CONDITIONAL, as the reader recommended; the concrete test would settle which parts of the headline are robust.","tokens_in":15205,"tokens_out":2375,"duration_ms":27129,"concrete_test":"Re-run the three hallucination conditions (§4.1.1: tc negated, no instruction, tc asserted) on the same 1000-instance HotpotQA subset with 3-5 independent seeds of the judge, or bootstrap-resample the 1000 instances, and compute 95% confidence intervals for the hallucination, abstention, and precision rates. If the asserted-vs-baseline hallucination gap (31.2 vs 28.7) and the precision drop (34.59 vs 46.65) fall outside the intervals, the directional claim is strengthened; if either is within noise, the claim should be softened to 'directional but not statistically established.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that a single optimized embedding, trained only on definitions with the model frozen, can directionally steer behavior. The evidence supports this claim in direction: in §4.1, negating the hallucination token reduces hallucinations from 28.7% to 21.9% and asserting it raises them to 31.2%, with precision dropping from 46.65 to 34.59 when asserted. In §4.2, asserting the recasting token raises recasting from 23.26% to 62.33%, while negating it drops recasting to 20.47%, and the follow-up rate is 98.04% vs 63.07% for the in-context baseline. These are coherent and the paper honestly notes that the main effect in §4.1 is increased abstention, not precision gain. The weakest point is statistical: every §4.1 number comes from a single run on a 1000-instance HotpotQA subset, with no confidence intervals, significance tests, or seed variance. The most load-bearing directional sub-claim—that asserting the token increases hallucinations relative to baseline—depends on a 2.5-point gap (31.2 vs 28.7) that could easily fall within sampling noise. The negated-vs-baseline gap (21.9 vs 28.7) is larger, but the paper itself interprets it as mainly an abstention effect. The recasting results are more robust because the margins are much wider, but the overall claim 'steering works' is disproportionately dependent on the hallucination table, which is under-powered. This is a legitimate correctness risk, not a fatal flaw: the paper's own limitations section acknowledges the single-model, constrained-compute setup, and the directional pattern is plausible. Thus the CONDITIONAL verdict stands, but the specific claim about asserting the hallucination token should be treated as exploratory until error bars are provided.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Concept Tokens, a method for adding a new special token to a frozen pretrained LLM and learning only its input embedding from a corpus of natural-language definitions of a target concept. At inference, the token is inserted into prompts and can be asserted or negated to steer behavior. The authors evaluate the method in three settings: (1) reducing hallucinations in closed-book QA on a 1000-instance HotpotQA subset, where negating the hallucination token lowers hallucinated answers (28.7% to 21.9%) but mainly by increasing abstentions, while asserting it raises hallucinations to 31.2% and lowers precision; (2) inducing recasting in a second-language teaching dialog, where asserting the recasting token raises recasting from 23.26% to 62.33% on erroneous student answers and preserves follow-up questions (98.04%) better than the in-context definition baseline (63.07%); and (3) a qualitative study with the Eiffel Tower and a fictional 'Austral Tower', showing the embeddings activate a coherent semantic theme but do not reliably store novel factual details. The paper argues that concept tokens provide a compact, definition-driven control signal for frozen LLMs.","tokens_in":15614,"tokens_out":4000,"duration_ms":46306,"significance":"If validated, Concept Tokens would be a lightweight and interpretable alternative to in-context prompting for behavior steering, with potential applications in low-resource or privacy-sensitive settings. The method is simple, does not require labeled behavioral examples, and the authors provide code and data. The recasting experiment is particularly suggestive: the concept token achieves comparable or superior compliance with secondary instructions compared to a long in-context definition, a result that is practically relevant. The paper is also unusually candid about limitations, including the abstention-driven hallucination reduction and the under-determination of factual details in the towers study. However, the central claims rest on evidence that is currently under-powered (single run, one model, one quantization) and lacks a crucial control for the mere presence of a newly added token. The significance would be considerably strengthened by a more rigorous statistical treatment and by controls that isolate the contribution of the learned embedding.","major_comments":[{"comment":"The key directional claims for hallucination steering rest on single-run percentages from a 1000-instance subset. The asserted-token vs. no-instruction gap (31.2% vs. 28.7%) is small; with n=1000 the standard error is about 1.4 points, so a 2.5-point difference is not statistically meaningful. The negated-token gap (21.9% vs. 28.7%) is larger, but the paper itself attributes it primarily to increased abstention. Please provide confidence intervals or significance tests (e.g., bootstrap, multiple seeds, or multiple randomly selected subsets), and specify how the 1000 instances were selected (random, stratified, first N?). Without this, the hallucination-steering claim is not adequately supported.","section":"§4.1, Table 1"},{"comment":"There is no control for the effect of introducing a new special token into the vocabulary. The observed behavioral changes could in principle be caused by the model's reaction to any novel token (or to an out-of-vocabulary string), rather than by the semantics learned from the definitional corpus. Please include a control condition with (a) an untrained/randomly initialized embedding for the same token, or (b) a token trained on a definition of an unrelated concept (e.g., 'gravity') using the same protocol and prompt frames. This control is necessary to attribute the directional effects to the learned concept representation.","section":"§4.1.1 / §4.2.1"},{"comment":"The central hypothesis—'the best possible embedding ... is the one that better captures the original concept'—is not formally defined or independently tested. The paper later shows (towers experiment) that the embedding does not reliably capture factual details, and the §4.1 results show that the hallucination token mainly affects abstention behavior. Please operationalize what 'captures the concept' means in a testable way (e.g., a predicted ordering of behavioral outcomes across asserted, negated, and baseline conditions) and specify which observations would falsify it. As stated, the hypothesis is close to a definition and is not informative about the mechanism.","section":"§3, central hypothesis"},{"comment":"The evaluation pipeline has several free parameters that are not systematically varied or reported with sensitivity analysis: training hyperparameters (learning rate, epochs) are chosen on a small hand-selected development set (10 prompts) without reporting the chosen values for each concept; the definitional corpora are generated by GPT-5 with no human verification of their quality or diversity; and the HotpotQA subset is described only as '1000 instances from the validation set' without selection criteria. Since the method's core claim is that definition-only supervision can steer behavior, the results should demonstrate robustness across at least a few corpus designs and subset choices, or explicitly report the selection and hyperparameter values so readers can assess the degrees of freedom.","section":"Appendix C / §4.1.1"}],"minor_comments":[{"comment":"Please state whether the 1000 instances were randomly sampled and whether any filtering was applied (e.g., by question type or answer length). This affects the generalizability of the reported percentages.","section":"§4.1.1, HotpotQA subset"},{"comment":"The follow-up-question rate is reported for only two methods (importantly, the two that differ most in prompt length). It would be useful to also report the follow-up rate for 'No instruction' and 'Recasting mention' to understand the baseline tendency to ask follow-ups.","section":"§4.2.2, Table 3"},{"comment":"The notation for the concept token embedding alternates between 'e_c' and 'ec' in the text; unify to avoid confusion.","section":"§3, notation"},{"comment":"In the recasting prompts, the concept-token condition and the 'recasting mention' condition differ only by replacing the token with the word 'recasting'. However, the system prompt already contains the word 'technique' and the word 'correct mistakes', which may prime the model. Consider a control that uses an unrelated word in the same position.","section":"Appendix E.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable exploratory study, but the evidence for the central 'steering works' claim is currently too thin to support acceptance. The authors are honest about limitations, and the recasting result is promising. The missing random-token control and the absence of any statistical significance assessment are the main barriers. I would be willing to look at a revision that adds these controls and analyzes the variance, even if the authors cannot run larger experiments; a careful bootstrap analysis on the existing 1000 instances would already be a significant improvement."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read the full preprint. The central claim—a single embedding trained only on definitions can directionally steer a frozen model—holds up directionally, but the strength of evidence varies a lot across experiments. The recasting result is solid: asserting the token raises recasting from 23 to 62 percent on error-containing answers, and the token strategy preserves follow-up questions at 98 percent versus 63 for the in-context definition baseline. That trade-off is the most useful finding in the paper. The hallucination experiment is weaker: negating the token cuts hallucinations from 28.7 to 21.9 percent, mostly by increasing abstentions, and precision is flat. The asserted token's increase to 31.2 percent versus the 28.7 baseline is within sampling noise; treat that sub-claim as exploratory until error bars appear.\n\nWhat's new: this extends the memory-token line from compressing a single sequence to learning a multi-definition concept embedding, and the directional prompting trick is a nice practical idea. The comparison against in-context definitions, especially the follow-up question metric, is a useful contribution. The paper is well-written, ships full prompts, annotation guidelines, and code/data, and the limitations section is honest.\n\nSoft spots: all behavior results come from one 4-bit Llama 3.1 8B with greedy decoding and single runs—no confidence intervals or significance tests. The 1000-instance HotpotQA subset's selection is unspecified. Hyperparameters were tuned on a small behavior-probing development set; that's a mild risk but the eval sets are separate. The towers experiment is qualitative but appropriately cautious: the token activates known concepts well but doesn't store novel facts reliably. The 'semantically compositional operator' assumption is untested, but the directional results are real even if the mechanism is unclear.\n\nWho this is for: people working on cheap steering for small on-prem models, and the recasting+follow-up trade-off is relevant for tutoring systems. It deserves a serious referee; send it to review as a solid empirical paper, likely with major revisions for statistical rigor. I agree with the conditional verdict.","headline":"Directional steering mostly works, but the hallucination evidence is underpowered; the recasting experiment is the convincing part.","tokens_in":16116,"tokens_out":3247,"would_cite":false,"duration_ms":33875,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single special token, trained only on text definitions of a concept, can steer a frozen large language model to suppress or amplify that concept's behavior.","keywords":["concept tokens","behavioral embeddings","language model steering","hallucination","recasting","frozen model","definitional corpus","instruction following"],"falsifier":"Train two concept tokens from the same prompt template—one on definitions of 'hallucinations' and one on definitions of an unrelated concept (e.g., 'uncertainty')—and run the same closed-book question-answering evaluation. If both tokens produce identical shifts in hallucination and abstention rates, the effect is not concept-specific. Alternatively, probe the model's internal activations at the token position with a linear probe: if the embedding does not move the representation along a direction associated with 'hallucination' (as identified by dictionary learning), the semantic-reading assu","tokens_in":15079,"feed_emoji":"🎛️","tokens_out":5292,"duration_ms":52867,"temperature":0.7,"pith_summary":"This paper proposes Concept Tokens: new special tokens whose embeddings are learned from natural-language definitions of a target concept while the language model is kept frozen. The central claim is that such a token becomes a compact, directional control signal: asserting the token in a prompt amplifies the associated behavior, negating it suppresses the behavior. Evidence comes from two behavioral tasks: hallucination in closed-book question answering (negation cuts hallucinated answers from 28.7% to 21.9%, mostly by increasing abstentions; assertion raises them to 31.2%) and recasting in second-language teaching (assertion raises recasting from 23.3% to 62.3% on erroneous answers). A qualitative study suggests the token acts as a semantic attractor rather than a reliable store of novel facts. If the claim is right, concept tokens offer a lightweight way to add concepts and steer behavior without fine-tuning weights.","feed_headline":"One token, trained on definitions, steers a frozen LLM","feed_subtitle":"Assert it to induce a behavior, negate it to suppress: hallucination drops from 28.7% to 21.9%, recasting jumps to 62%.","key_machinery":"The key mechanism is the concept token embedding itself: a new row in the model's input embedding matrix, trained by backpropagating the next-token cross-entropy loss on an 'instantiated definitional corpus'—definitions where every mention of the concept is replaced by the token. Because the model is frozen, all learning must fit into the single embedding vector, and the diverse definitions act as competing constraints that force a generalized representation. The same token is then used directionally in prompts: asserted (as an instruction to generate the concept) or negated (as an instruction not to generate it).","core_discovery":"The paper's central discovery is that a single newly added input embedding, optimized with the standard language-modeling objective on a set of definitions of a concept (with mentions replaced by the token), can act as a learned behavioral switch in a frozen pretrained model. The embedding is trained to satisfy multiple, sometimes conflicting, definitional constraints, so it must generalize rather than memorize. At inference, inserting the token in an instruction ('Generate <token>') increases the target behavior, while negating it ('Do not generate <token>') suppresses it. This directional effect appears in two independent tasks—hallucination avoidance and recasting of learner errors—and th","pith_inferences":["If the directional effect is mediated by the embedding's position in the model's semantic space, then combining several concept tokens in one prompt might steer behavior along multiple independent dimensions (the paper lists this as future work; we note it would be the natural next test).","The paper's next-token observation in the towers experiment—geographically proximate cities competing for the token's location—suggests that a single vector is low-capacity; pairing concept tokens with a fact-retrieval mechanism could fix the factual gaps while keeping the behavioral steering.","The result that negation mainly increases abstentions implies that 'hallucination' as learned from definitions is a behavior to avoid rather than a calibration signal; one could test whether training the token on 'uncertainty' definitions produces more selective abstention.","A cautious extension: the same method could be used to inject concepts like 'safety' or 'toxicity' as tokens, but the paper gives no evidence about whether negating such a token would actually reduce unsafe outputs or just shut down generation."],"forward_implications":["Negating a concept token can reduce a target behavior like hallucination, but the paper shows this works mainly by increasing abstention, not by improving the precision of attempted answers.","Asserting a concept token can induce a complex behavior (recasting an erroneous learner answer) with a single token, outperforming the word 'recasting' in the prompt and nearly matching a full in-context definition.","Concept tokens preserve instruction-following better than long in-context definitions: the recasting token kept follow-up questions at 98.04% versus 63.07% for the definitional baseline.","Concept tokens can add a novel concept to a frozen model, but the resulting representation is a semantic theme rather than a reliable fact table; factual details remain prone to hallucination.","The approach works on a quantized, instruction-tuned model, suggesting it is feasible for smaller on-premise deployments."],"fun_headline_variants":["Single concept token, learned from definitions, steers frozen LLM","Frozen LLM behavior toggled by a single embedding from definitions","Learned from definitions, one token controls LLM behavior","A definition-trained token that flips LLM behavior","Behavioral switch: one embedding learned from concept definitions"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that a single optimized embedding is read by the frozen model as a meaningful semantic operator—so that 'do not generate <token>' actually suppresses the concept rather than merely producing a surface pattern—and no activation-level evidence is provided to confirm this.","fun_headline_variants_meta":{"raw":{"variants":["Single concept token, learned from definitions, steers frozen LLM","Frozen LLM behavior toggled by a single embedding from definitions","Learned from definitions, one token controls LLM behavior","A definition-trained token that flips LLM behavior","Behavioral switch: one embedding learned from concept definitions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000245,"raw_usage":{"total_tokens":1367,"prompt_tokens":734,"completion_tokens":633,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":478,"completion_tokens_details":{"reasoning_tokens":559}},"tokens_in":478,"tokens_out":633,"duration_ms":6848,"temperature":1.0,"reasoning_tokens":559,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T12:00:58.208029+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train two concept tokens from the same prompt template—one on definitions of 'hallucinations' and one on definitions of an unrelated concept (e.g., 'uncertainty')—and run the same closed-book question-answering evaluation. If both tokens produce identical shifts in hallucination and abstention rates, the effect is not concept-specific. Alternatively, probe the model's internal activations at the token position with a linear probe: if the embedding does not move the representation along a direction associated with 'hallucination' (as identified by dictionary learning), the semantic-reading assu","supporting_citations":[],"review_version":1}