{"id":"c63468c2-29b8-4ab3-ac86-64269161384b","arxiv_id":"2502.01540","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LLM similarity ratings over number pairs are best explained by combining Levenshtein string edit distance with a log-linear numerical distance, indicating entangled string and numeric representations.","lead":"Large language models judge how similar two numbers are using a blend of text-like digit overlap and math-like value closeness. The paper shows this across six models, that context shifts the balance, and that it can cause incorrect choices in a simple decision task.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Behavioral 'entanglement' may be a task heuristic: internal probes are trained on similarity-prompt residuals with the very metrics they confirm, so representation-level evidence is missing.","rationale":"The paper is honest: limitations are stated, code is released, and the evidence is convergent across six models, multiple contexts, and a decision task. My read does not overturn the conditional verdict. The concern I focus on is not that the behavioral R2 values are wrong, but that their interpretation as evidence about 'representation' is underdetermined. The similarity prompt is a task, and LLMs are known to be highly sensitive to task framing; the internal probes, while useful, are trained on the same task-conditioned representations and the same metrics, so they cannot break the underdetermination. A single additional experiment, using pairwise distances from a neutral context without the word 'similar,' would separate a stable representation from a task heuristic. If that experiment fails, the central claim should be softened from 'LLMs learn representational spaces that blend...' to 'when asked to judge similarity, LLMs behave as if...'. If it succeeds, the current manuscript's conditional acceptance is easily justified. I therefore keep the reader's CONDITIONAL verdict unchanged and recommend adding this control.","tokens_in":14682,"tokens_out":5156,"duration_ms":67679,"concrete_test":"Measure the pairwise distances between number tokens in the residual stream at the last number-token position using a neutral, non-similarity context (e.g., 'The values are 685 and 791.' followed by a simple next-token continuation), without any similarity instruction, for Llama-3.1-8b. Then fit the same combined regression of these pairwise distances on Levenshtein and Log-Linear similarity and compare R2 and unique-variance contributions to the behavioral results in Figure 2B. If Levenshtein contributes no unique variance, or the combined R2 is negligible, the entanglement found in Sections 5.1 and 5.2 is specific to the similarity prompt, and the representation-level conclusion should be weakened. If Levenshtein still contributes independent variance, the representation claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that elicited similarity judgments reflect an entangled internal representation of numbers. The weakest load-bearing link is the readout assumption in Section 3: the 'How similar are the two numbers?' prompt invites the model to adopt an ad hoc comparison rule, and the paper's internal evidence does not close this gap. In Sections 4.4 and 5.2, the probes are trained on the residual of the last token after the full similarity prompt ('Rating:'), a representation already conditioned on the similarity instruction and the rating task. A linear probe can decode any feature useful for producing that rating; the fact that Levenshtein and Log-Linear distances are decodable is therefore consistent with a surface heuristic such as 'compare digit overlap and length,' not with a stable number representation. The probe targets are also exactly the two theoretical metrics used to define the matrices, so the correlations in Table 2 mostly show that the probe can be trained to detect those metrics in task-conditioned activations. This is not circular for the behavioral regression, but it means Section 5.2 does not independently validate the behavioral readout. Furthermore, all three experiments, including the decision scenario, use the word 'similar' or 'most similar,' so they share the same semantic framing. Without a prompt-independent representation measurement, the 'representation' claim rests on one model's similarity-prompt behavior.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks whether large language models (LLMs) represent numbers using a mixture of string-like (Levenshtein edit distance) and numerical (Log-Linear magnitude) codes. The authors elicit pairwise similarity ratings for integers 0–999 from six LLMs, regress the ratings on the two theoretical distances, and report that a combined linear model achieves an average R^2 of 0.726 in the default context. They further manipulate context with int()/str() wrappers and number-base changes, probe the internal activations of Llama-3.1-8b to decode the two distances, and construct a test-tube selection scenario where a string-distance bias produces objectively wrong choices. The paper concludes that LLM number representations are an entangled blend of string and numerical structure that can be reduced but not eliminated by context.","tokens_in":14952,"tokens_out":5561,"duration_ms":49374,"significance":"If the result holds, this is a valuable and methodologically interesting contribution. The behavioral regression is carefully executed with bootstrap confidence intervals and a linear-distance control, the context manipulations (int/str, base 4/8) are strong controls, and the decision scenario gives the work practical relevance. The paper also provides code, model versions, and an appendix with temperature robustness checks, which supports reproducibility. The central finding—that elicited similarity judgments over integers are well described by a combination of Levenshtein and Log-Linear distances—is falsifiable and re-runnable. The main risk is that the paper interprets this finding as evidence about internal representations, and that interpretation rests on load-bearing assumptions that are not fully validated.","major_comments":[{"comment":"The two predictor distances (Levenshtein and Log-Linear) are highly correlated for numbers, but the paper reports only R^2 for the combined and separate models, not partial correlations or a variance-partitioning analysis. Without partial R^2 values, beta weights, or a commonality analysis, the claim that the two distances contribute uniquely is underdetermined. Please report standardized regression coefficients with confidence intervals and a decomposition of the combined R^2 into unique and shared components.","section":"4.3 and 5.1"},{"comment":"The internal-probe experiment does not independently confirm an entangled representation. The probes are trained on the last-token residual after the full similarity prompt ('Rating:') and their targets are exactly the two theoretical metrics used in the behavioral regression. A linear probe can decode any feature that is useful for producing the rating, so the correlations in Table 2 are consistent with a task-specific response heuristic (e.g., 'compare digit overlap and length') rather than with a stable internal number representation. To support the representational claim, the probes should be trained on activations from a neutral context that does not mention similarity (e.g., the number tokens alone or next-token prediction), or the claims in Section 5.2 should be restricted to task-conditioned decodability.","section":"4.4 and 5.2"},{"comment":"The method assumes that the 'How similar are the two numbers?' prompt is a faithful readout of internal number representation. All of the paper's controls—int()/str(), base changes, and the decision scenario—still use the word 'similar' (or 'most similar'), so they may all tap the same surface heuristic. A concrete validation would be to use a non-similarity paradigm, such as asking the model to choose the numerically nearest number or to rate 'which number is closer in value'; if the dual-distance structure persists without the word 'similar', the readout assumption is strengthened. Without such a control, the abstract's claim about 'representational spaces' should be tempered.","section":"3 and 5.1"},{"comment":"The statement that all integers in 0–999 are 'represented as unique tokens' in all six models is an empirical tokenizer claim, but no tokenizer analysis or evidence is provided. If some three-digit numbers are multi-token in these models, the Levenshtein-distance effect could partly reflect token-boundary artifacts rather than character-level string similarity. Please verify this claim with the actual tokenizers and report the fraction of single-token versus multi-token numbers in the 0–999 range for each model.","section":"4.1"}],"minor_comments":[{"comment":"The caption contains a typo: 'faction' should be 'fraction'.","section":"Figure 5 caption"},{"comment":"The example triplet construction says the digits are sampled from {2,...,9}, but the first example is '3, 3, 1 → 331', which includes 1; the text is inconsistent and should be corrected.","section":"4.5"},{"comment":"The claim that the probes 'nicely dissociate' the integer and string subspaces is overstated: the string probe correlates 0.527 with the Log-Linear measure, and the integer probe correlates 0.393 with Levenshtein, so the dissociation is partial. The correlations are also not specified as Pearson; please state the correlation type.","section":"5.2 and Table 2"},{"comment":"The int()/str() manipulation changes the surface string by adding parentheses and function names, so part of the observed context effect may be a formatting effect rather than type semantics. Consider a control that wraps the numbers in neutral characters (e.g., 'Number: (NUM)') to isolate the semantic effect.","section":"Appendix A"},{"comment":"The linear ℓ1-distance control is reported only in the appendix, but it is an important control; consider mentioning it more prominently in the main text, possibly with a sentence in Section 5.1 explaining that the Log-Linear metric outperforms linear magnitude.","section":"5.1 and Appendix E"}],"recommendation":"major_revision","confidential_remarks":"The behavioral regression is solid and the paper is likely to make a useful contribution after revision. The main issue is the interpretation: the paper's title and abstract claim a representational result, but the internal-probe evidence is not independent, and the readout assumption is untested. If the authors can add a neutral-context probe or a non-similarity behavioral control, the central claim would be much stronger. The collinearity of the predictors should also be addressed with partial statistics. I would not reject the paper, but it needs a substantive revision rather than minor edits."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Marcus,\n\nThe paper gives you something real: a simple behavioral protocol (pairwise similarity over 0–999) that shows LLM number similarity judgments are a blend of Levenshtein edit distance and a log-linear magnitude distance, with context (int() vs str()) shifting the weights. That pattern holds across six models with bootstrap CIs, and the decision scenario turns it into an error rate. As a behavioral description of prompted similarity judgments, I think it lands.\n\nThe newness is legitimate. Prior work on number representation in LLMs went after arithmetic circuits or linear encoding of value; nobody had elicited similarity ratings over integer pairs and decomposed them into string and magnitude components. The context manipulation is a good control, and the base-4/base-8 extension strengthens the claim that the string component is about digit form, not magnitude.\n\nSoft spots are real but not fatal. First, the two predictor distances are correlated, and the paper reports only per-predictor R^2, not partial or unique variance. So the 'entangled' decomposition is not as clean as the text implies; the combined fit could be mostly driven by one component. That is fixable. Second, the readout assumption: 'How similar are these two numbers?' invites an ad hoc comparison rule, and the internal probes do not close that gap—they are trained on the last-token residual after the similarity prompt, so decodability of the two metrics is consistent with a task heuristic. The probes are described as 'internal representation' evidence, but they are really evidence about task-conditioned activations. The paper mostly hedges this in limitations, but the abstract and discussion lean on entanglement. Third, probes run only on Llama-3.1-8b; zero temperature; all experiments share the 'similar' qualifier. Minor.\n\nI don't think the stress-test kills the paper. The behavioral result stands on its own, and the decision scenario shows downstream error. The representation-level story needs a prompt-independent measurement or a softened claim. That is conditional-acceptance territory.\n\nWho is this for? Cognitive scientists and interpretability folks who want a behavioral assay for number representations. It deserves a serious referee. I'd send it out.","headline":"Behavioral result is solid and new; the 'representation' claim is softer than advertised, so conditional acceptance is right.","tokens_in":15454,"tokens_out":1530,"would_cite":true,"duration_ms":14458,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Large language models represent numbers as a blend of digit-string and magnitude structure, and that blend causes measurable errors in quantitative decisions.","keywords":["number representation","large language models","similarity judgments","Levenshtein edit distance","log-linear distance","string bias","tokenization","quantitative reasoning"],"falsifier":"Run the same similarity protocol in the int() condition at zero temperature across the six models; if any model's Levenshtein regression weight over the Log-Linear-only model falls within a 95% confidence interval that includes zero, the claim that the string component is irreducible fails.","tokens_in":14506,"feed_emoji":"🔢","tokens_out":8278,"duration_ms":67268,"temperature":0.7,"pith_summary":"Large language models must treat the same digit sequence as both a number and a string, and this paper asks what representation actually arises. The authors elicited pairwise similarity ratings over integers 0 to 999 from six modern large language models and found that the resulting similarity patterns are well described by a weighted combination of Levenshtein edit distance and a psychological Log-Linear numerical distance. Explicitly telling the model whether the tokens are int() or str() pushed the pattern in the predicted direction but did not remove the mixture. The same entanglement shows up in the model's internal embeddings, and a test-tube concentration choice shows that the string bias can cause wrong answers in a realistic quantitative decision. If correct, this means models do not have a clean 'number sense': numerical competence is entangled with surface string structure, and context only partially resolves the ambiguity.","feed_headline":"LLMs store numbers as strings and magnitudes; the mix causes errors","feed_subtitle":"Similarity ratings from six models mix edit distance and magnitude; context can reduce but not remove the blend.","key_machinery":"The paper's central instrument is the similarity-judgment paradigm: a neutral prompt that asks the model to rate pairwise similarity, converted into a symmetric similarity matrix over the integers 0 through 999. The theoretical yardsticks are the Levenshtein edit distance, the minimum number of character insertions, deletions, or substitutions between two digit strings, and the Log-Linear distance $d_{\\mathrm{Log}}(x,y) = 1 - \\exp(-|\\log(x+\\epsilon) - \\log(y+\\epsilon)|)$ with $\\epsilon = 10^{-4}$, a psychological magnitude measure with logarithmic compression. The argument runs by linear regression of the elicited matrix on the two normalized distances, and by linear probes that decode each distance from the last-token residual of Llama-3.1-8b. The combination of the two distances is what carries the 'entanglement' claim.","core_discovery":"On its own terms, this paper establishes that the representation of integer tokens inside large language models is an entanglement of a string-like code and a magnitude code. Across six models, similarity matrices elicited by asking 'How similar are the two numbers?' are regressed by a linear combination of Levenshtein edit distance and Log-Linear numerical distance, explaining on average 72.6% of the variance in the default prompt; each component contributes on its own, and replacing Log-Linear with absolute difference lowers the fit. Writing int() or str() around the tokens shifts the weights in opposite directions, but the Levenshtein contribution never disappears. Internal linear probes on Llama-3.1-8b decode both distances from the embeddings, and in the integer-probe subspace the string-induced sub-diagonals persist. In a forced-choice scenario with compound concentrations, a model with string bias can choose the numerically wrong tube: for three-digit concentrations the maximal error rate is 36.9% for Llama-3.1-8b (with GPT-4o at 0.02%), and for five-digit numbers the maximal error rises to 47.0%.","pith_inferences":["A direct extension the paper does not test: applying the same protocol to reasoning-oriented models or chain-of-thought settings should shrink the Levenshtein component if the string bias is a fast surface heuristic; if it does not, the blend is more deeply structural.","The similarity protocol could be used to probe tokenization effects directly: numbers with group separators (e.g., '1,234' vs '1234') should shift the edit-distance component if token boundaries drive the string effect.","The concentration-choice task could be turned into a cheap behavioral calibration instrument: the ratio of Levenshtein to Log-Linear regression weights might predict a model's error rate on magnitude comparisons, giving a testable predictor beyond the paper's reported results."],"forward_implications":["Explicit type disambiguation via int() or str() shifts but does not abolish the entanglement: the Levenshtein component stays above zero in every tested context, so prompt design can mitigate but not remove string bias.","In decision tasks where magnitude matters, string similarity between digit strings can override numerically closer options, and the error rate grows with digit count for the most biased models.","The representational blend is visible at the embedding level, not only in prompted answers, so it is a property of the learned representation rather than a quirk of the similarity prompt.","Replacing the Log-Linear distance with a simple absolute difference degrades the fit, supporting a logarithmically compressed magnitude code in these models, paralleling human approximate number representations."],"supporting_citations":[{"why":"Supplies the edit-distance metric that forms the string component of the predicted similarity.","marker":"Levenshtein (1966)"},{"why":"Supplies the Log-Linear psychological distance that forms the numerical component.","marker":"Piantadosi (2016)"},{"why":"Establishes the similarity-based approach to characterizing psychological representations that the prompting protocol adapts.","marker":"Shepard (1980)"},{"why":"Provides the Bayesian grounding for using similarity judgments to infer representational distance.","marker":"Tenenbaum & Griffiths (2001)"},{"why":"Connects similarity matrices to underlying representational structure, motivating the probe analysis.","marker":"Kriegeskorte et al. (2008)"},{"why":"Shows LLM embeddings encode number value linearly in arithmetic contexts, the result this paper contrasts with its log-linear-plus-string finding.","marker":"Zhu et al. (2025)"}],"fun_headline_variants":["LLMs see numbers as strings and magnitudes, causing errors","Blended number representations lead to numeric mistakes in LLMs","String-like coding of numbers persists even with context","LLMs mix edit distance and magnitude when judging numbers","LLMs' string bias leads to wrong numeric picks up to 47%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole argument assumes that asking a model 'how similar are these two numbers?' reports the model's internal representation of numbers, rather than a task-specific habit of comparing strings that the prompt itself triggers.","fun_headline_variants_meta":{"raw":{"variants":["LLMs see numbers as strings and magnitudes, causing errors","Blended number representations lead to numeric mistakes in LLMs","String-like coding of numbers persists even with context","LLMs mix edit distance and magnitude when judging numbers","LLMs' string bias leads to wrong numeric picks up to 47%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001041,"raw_usage":{"total_tokens":4393,"prompt_tokens":974,"completion_tokens":3419,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":3337}},"tokens_in":590,"tokens_out":3419,"duration_ms":23002,"temperature":1.0,"reasoning_tokens":3337,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T15:00:20.662294+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same similarity protocol in the int() condition at zero temperature across the six models; if any model's Levenshtein regression weight over the Log-Linear-only model falls within a 95% confidence interval that includes zero, the claim that the string component is irreducible fails.","supporting_citations":[],"review_version":1}