{"id":"801e9089-f49e-4019-b4d6-0283cfdc9667","arxiv_id":"2611.08070","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Zero-shot weight reading by Claude Opus 5.5 is dominated by local edge arithmetic plus memorized dataset templates, with no verified evidence of weight-flow tracing.","lead":"A controlled audit gave a chatbot the raw weights of small image classifiers as plain text and checked whether it follows the network's computation or leans on arithmetic at the edges plus memories of famous datasets. The model's arithmetic held up, while its class names came from familiar templates, including wrong ones on label-permuted and satellite networks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"The negative half of the central claim (no weight-flow tracing) rests on self-reports and failed partial checks, yet the paper's own verification shows this model's weight-reading reports can be wrong, and the decisive forward-pass test was not run.","rationale":"The reader's weakest assumption matches mine: the negative half of the central claim is licensed by self-reports and by the absence of the forward-pass test, and both are acknowledged by the authors (Sections 5.5, 7, and 8). I agree with that reading and with the CONDITIONAL verdict. The positive evidence is genuinely strong: verified optimizer-step inference from batch-norm statistics, output sign structure recovering the vehicle/animal split, the reordered-label failure reproducing exactly the template-matching error pattern, and convergent prior-driven mislabeling across four datasets. The stress-test sharpens the negative-claim concern in two ways beyond the reader's statement. First, the paper's own verification data show that this model's reports about its weight reading are unreliable (the SAT-6 'blue' feature misread and the Cats vs Dogs conv2 indexing errors in Section 5.5), so self-reports cannot carry the negative attribution. Second, the partial-tracing record is mixed, not uniformly failed: 3 of 4 channel-to-output claims were correct after full recomputation, so the model performs some multi-layer path computations and the boundary between local arithmetic and flow tracing is empirically blurred. Neither point refutes the hedged behavioral claim, which the evidence supports; both mean the title-level capability question is underdetermined. The decisive test is the paper's own proposed forward-pass flow test; down-scaling it to ~1,000-2,000 weights makes exact propagation feasible in a no-tools reply while preserving the task family. The other conditions the reader lists, artifacts available only on request, single runs without significance tests, and scoring by the same vendor's tooling, are real and also block ACCEPT, but they are secondary to the evidentiary gap in the negative claim. No ad hominem and no manufactured concern: the authors' explicit statement that 'no session was asked to compute a forward pass' is itself the strongest support for keeping the verdict conditional.","tokens_in":20260,"tokens_out":28440,"duration_ms":175348,"concrete_test":"Run the forward-pass flow test that Section 8 lists first, on a network small enough that exact forward propagation is arithmetically tractable within the model's reply budget: roughly 1,000-2,000 weights (e.g., 10x10 input, one or two narrow conv layers, one dense layer), in the same plain-text weight format and no-tools setting as the paper. Give one concrete input tensor per trial over 5 held-out inputs and require explicit intermediate activations and final logits, computed layer by layer; score against the true forward pass, not just top-1 class. If the model reproduces logits and intermediates by explicit propagation, the negative half of the central claim fails and the deflationary account should be rejected at least in that regime. If the model cannot trace even ~1,000 weights, the negative claim gains direct support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is compound: a positive half (local arithmetic plus dataset priors) and a negative half (not weight-flow tracing). The positive half is well supported by verified checks: batch-norm running statistics yield the correct ~3-step estimate, output-layer sign patterns recover the CIFAR-10 vehicle/animal split, and the reordered-label run reproduces the template-matching error pattern, with convergent mislabeling across KMNIST, fast food, and SAT-6. The negative half rests on (i) the model's self-reports that it read the edges closely and sampled the middle sparsely and (ii) several failed partial-tracing checks (Section 5.5). The paper concedes these are self-reports, not measurements, and that no session was asked to compute a forward pass (Sections 5.5 and 7). What remains unresolved: this model's weight-reading self-reports are independently shown unreliable within the paper, as the SAT-6 session described as 'blue' a feature whose blueness correlation is negative and a Cats vs Dogs session quoted real values attached to the wrong conv2 channels (Section 5.5). If reports about what the model read can be wrong, then reports about what it did not trace ('I did not trace these features through to the outputs'; 'did not trace conv3 or fc1 in detail') cannot license the negative attribution. The partial-tracing record is also mixed rather than negative: 3 of 4 channel-to-output claims named the correct top output when recomputed across all 78 hidden units, including the green feature's link to output 4. The model demonstrably performs some multi-layer path computations, so the line between local arithmetic and flow tracing is empirically blurred, and the negative half is underdetermined without the forward-pass flow test that Section 8 names 'the most direct test of the central claim.' The hedged conclusion is supported; the capability-level reading of the title question is not settled.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"This paper audits the reading side of \"weight-space programming\" (WSP) claims by Takehi et al. [36], testing Claude Opus 5.5 on small convolutional classifiers (about 125,000 parameters) trained by the author on five image datasets. Each network's complete weights are provided as plain text to a fresh, tool-less session, with no dataset or class names, no code execution, and no forward passes.","tokens_in":20618,"tokens_out":17922,"duration_ms":89874,"significance":"If the account holds, this is a valuable deflationary control for WSP claims and a practical audit recipe: withhold the dataset name, permute the labels, vary dataset familiarity and network accuracy, and separate local boundary statistics from circuit-level understanding. The positive half of the claim is well supported. Quoted values were checked against the actual parameters with zero mismatches on most sessions (Appendix C); the Adam-step inference was verified against true training history; the reordered-label control behaves exactly as prior-driven semantics would predict, although it is a single run; and the scoring rules were set before the prompts were sent for all but the first two runs. There is no circularity in the design: the networks, weights, and training histories are external ground truth, and nothing is fitted to produce the conclusion. The paper is also unusually candid, disclosing that the tracing evidence is indirect, that no forward-pass test was run, that configurations are mostly single runs, and that scores lack a second rater.","major_comments":[{"comment":"The negative half of the central claim — \"better described as local arithmetic plus dataset priors than as weight-flow tracing\" (abstract) and \"we found no evidence that it traces the intermediate computations\" (§1) — is load-bearing, yet the paper itself rates it only \"Supported indirectly\" (Table 7), concedes in §5.5 that the tracing-related quotes \"are self-reports, not measurements of an internal process,\" concedes in §7 that \"no session was asked to compute a forward pass,\" and states in §8 that a forward-pass flow test \"is the most direct test of the central claim.\" The gap is material, not merely presentational, because the paper's own verification shows that this model's verbal reports about the weights can be wrong: a SAT-6 feature the session called \"blue\" has a negative correlation with blueness, and a Cats vs Dogs session quoted real weight values attached to the wrong conv2 input channels (§5.5). If reports about what was read are unreliable, reports about what was not traced (\"I did not trace these features through to the outputs\"; \"did not trace conv3 or fc1 in detail\") cannot license the negative attribution. Please either run a forward-pass or multi-layer tracing probe and report it, or re-scope the abstract and conclusion to match the evidence, e.g., \"local arithmetic plus dataset priors, with no evidence of weight-flow tracing under this protocol.\"","section":"Abstract and §1 vs. §5.5, §7, Table 7","authors_acknowledge":"These are self-reports, not measurements of an internal process, and no session was asked to compute a forward pass, so this evidence is indirect.","authors_acknowledge_p":0.91},{"comment":"The partial-tracing record is mixed rather than negative: three of the four channel-to-output claims named the correct top output when recomputed across all 78 hidden units, including the green feature's link to output 4. In the green-feature row of Table 6, the model's channel-to-output tracing was correct — output 4, which is trees — and only the semantic label (\"frog\") was wrong. Table 7's row \"It is not following weight flows — Supported indirectly — several claims based on partial tracing failed our checks\" summarizes only the failures and omits these successes. The paper should either argue explicitly that one-hop output-layer (fc2) effects are local statistics rather than flow tracing, or acknowledge these as successful partial traces that the negative conclusion must accommodate.","section":"§5.5, Table 7","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"Figures 1, 5, and 6 are schematic illustrations, and their captions correctly say they are not literal renderings; please mark \"schematic — not data\" on the figures themselves, since Figures 5 and 6 appear inside the results sections and could otherwise be misread as measured outputs.","section":"Figures 1, 5, 6","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The Cats vs Dogs Model 2 answer satisfied both the full-pass and the neutral scoring rule (Appendix C); applying \"the stricter reading\" is disclosed but was chosen after the answer existed, so please either pre-commit the tie-breaking rule in the protocol or report both readings in Table 8.","section":"Appendix C","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The audit protocol is a stated contribution, but the data and code are \"available from the author on request\"; a public archive of prompts, transcripts, scoring logs, and checkpoints would make the audit independently repeatable.","section":"Data and Code Availability","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Table 8 asterisks the two CIFAR-10 runs that had no preset scoring rule; since these are the flagship recognition results, one sentence on how those answers would be scored under the later Appendix C criteria would help the reader gauge sensitivity to the scoring choice.","section":"Table 8","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Appendix C tallies only the items the author could locate in each answer and states that not every claim was checked; a per-session statement of how many claims were made versus how many were checked would make the verification coverage interpretable.","section":"Appendix C","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper alternates between the hedged phrasing \"we found no evidence that it generalizes that recognition into tracing\" (§1) and the unhedged comparative conclusion in the abstract; aligning the abstract, §1, and §9 with a single, evidence-matched phrasing would remove the tension identified in the major comments.","section":"Abstract and §1","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"The one thing to know: the label-permutation diagnostic is the real contribution, and it works. Permuting CIFAR-10's output labels keeps the learned structure while breaking the index-to-name map; the model finds the vehicle/animal split at the new indices but gets every interpreted class pair wrong. That cleanly separates arithmetic reading from semantic naming, and I don't know of a prior controlled test of exactly that distinction for zero-shot weight reading.\n\nThe positive half of the central claim is solid. The verification tallies are genuine evidence: quoted weight values matched the actual parameters (zero mismatches in most tallied sessions), the Adam-step inference from batch-norm statistics hit the true three-update training history, and the output-layer sign pattern / vehicle-animal split checks out. The KMNIST grayscale deduction from the channel-difference initialization bound is a nice catch, and it verifies against the weights.\n\nThe negative half — \"no evidence of weight-flow tracing\" — is where I'd hold back. It rests on the model's self-reports that it sampled the middle layers, plus a mixed set of partial-tracing checks. Mixed matters: three of four channel-to-output claims in the SAT-6 session named the correct top output when recomputed across all hidden units. And the same paper documents two cases where the model's reports about what it read were wrong: the \"blue\" feature that correlates negatively with blueness, and the Cats vs Dogs session quoting real values against the wrong conv2 channels. If its reports about what it did read are unreliable, its reports about what it did not trace are a weak basis for a negative attribution. The authors say exactly this in Sections 5.5 and 7, and they name the forward-pass test as the decisive missing experiment. So the hedged conclusion is supported; the title-level question is not settled.\n\nMinor issues, all acknowledged: mostly single runs, no significance tests, no independent rater, artifacts only on request, and the scoring tool and the model under test are both Anthropic products. The KMNIST preprocessing confound — normalization on padded images shifting the background value toward MNIST's standard — is a real worry for that one result, and they flag it. None of these are hidden.\n\nWho it's for: anyone building or evaluating weight-reading and weight-editing claims, and anyone benchmarking LLMs as network interpreters. The protocol in the appendix is reusable as-is.\n\nRecommendation: send it to peer review. A referee's main job will be deciding how much the negative claim can carry without the forward-pass test, which is exactly the kind of question worth referee time. The title is louder than the content, but the content is measured and the authors have framed their own limits correctly.","headline":"The label-permutation diagnostic is the real contribution, and it cleanly separates weight arithmetic from semantic naming; the positive half of the claim holds, but \"no flow tracing\" is underdetermined without the forward-pass test the authors themselves name.","tokens_in":21287,"tokens_out":4620,"duration_ms":31528},"pith_extraction":{"msc":["68T07"],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"Given a small network's raw weights as plain text, a general-purpose LLM performs real local arithmetic at the edges but assigns semantics from familiar dataset templates rather than tracing the computation.","keywords":["weight-space programming","LLM weight reading","dataset priors","mechanistic interpretability","local arithmetic","label permutation control","batch normalization statistics","model auditing"],"falsifier":"Run the forward-pass flow test the paper proposes: hand the same kind of session a small network and one concrete input image and ask for the exact output logits, with no tools. If the answers land close to the true logits on inputs the model has never seen, the \"no weight-flow tracing\" conclusion fails; if they do not, the deflationary account holds. A quicker check is to repeat the label-permuted CIFAR-10 run several times under the revised prompt, expecting the split at the new indices together with consistently wrong class names.","tokens_in":21588,"feed_emoji":"🧮","tokens_out":13990,"duration_ms":80510,"temperature":0.7,"pith_summary":"An LLM that is handed the raw weights of a trained network as a text file could be reading the computation, or it could be doing simple arithmetic at the network's edges and leaning on what it remembers about famous datasets. This paper argues the second, on small convolutional networks of about 125,000 parameters trained on five image datasets: the model performs genuine local calculations — batch-norm running statistics let it infer roughly three optimizer steps on a network stopped after three updates, and output-layer sign patterns recover CIFAR-10's vehicle/animal split — but its semantic judgments come from template matching, relabeling KMNIST as MNIST, calling all three fast-food networks CIFAR-10, and reading a 99.57%-accurate satellite network as CIFAR-10 animal photos. The key diagnostic is a label-permuted CIFAR-10 network: the model still found the output split at the new indices, but every class pair it interpreted was wrong, showing it matched a remembered label order rather than traced features. If the account is right, a correct dataset guess from weights is weak evidence of weight understanding, and the reading side of weight-space programming needs controls for label order, dataset familiarity, and accuracy before any claim of comprehension is made. The paper is exploratory, mostly single runs, and it concedes that no session was asked to compute a forward pass.","feed_headline":"LLM reads weights arithmetically but names datasets from memory","feed_subtitle":"Handed plain-text weights of small CNNs, it aced edge statistics but misnamed every class when labels were permuted.","key_machinery":"The argument is carried by a two-level account of weight reading. At the local level, the model does arithmetic on boundary statistics: batch-norm running means and variances combine with framework conventions (momentum 0.9, initialization bounds) to produce optimizer-step counts, and output-layer sign patterns and principal components produce class groupings — low-dimensional quantities a language model can estimate from a long list of numbers. At the semantic level, the model matches the network's outer footprint ($32\\times32$ RGB input, ten outputs) and its edge statistics against memorized templates, with CIFAR-10 as the default. The load-bearing diagnostic is the label-permutation control: permuting output labels preserves the weight structure and test accuracy while breaking the index-to-class-name mapping, so a model that finds the vehicle/animal split at the new indices but misnames every class demonstrates template matching rather than flow tracing. The batch-norm deduction is the positive anchor of the local side: running variance near $0.9^3 = 0.729$ identifies roughly three Adam steps, a calculation the paper verified against the actual parameters.","core_discovery":"On the paper's own terms, the central claim is that zero-shot weight reading by Claude Opus 5.5 is \"local arithmetic plus dataset priors,\" not weight-flow tracing. Handed every weight of a small CNN as plain text — no tools, no forward passes, no dataset or class names — the model calculates low-dimensional statistics at the network's boundary: batch-norm running variances near $0.9^3 = 0.729$ combine with framework defaults to yield an estimate of about three Adam steps on a network stopped after three updates, and the dominant output-layer sign pattern separates CIFAR-10's vehicles {0,1,8,9} from its animals {2,...,7}, with the first principal component carrying 27.7% and 28.5% of the variance in the two networks. When asked what the network was trained on, the model fills semantics from familiar templates: KMNIST becomes MNIST, all three fast-food networks become CIFAR-10, and the 99.57%-accurate SAT-6 network is read as CIFAR-10 animal photos, with the feature that fires most on trees called \"frog.\" The label-permutation control is the key test: with vehicles moved to outputs 2, 4, 5, and 7, the session still found the split at the new indices — real reading of output structure — yet all nine class pairs it interpreted were wrong, which the paper reads as matching the remembered CIFAR-10 order rather than tracing features to their visual meaning. Across sessions the model reported reading the first and last layers closely and sampling the middle, and several claims built on that partial tracing failed verification.","pith_inferences":["A general empirical signature follows from the label-permutation result: class attributions that move rigidly with index order, rather than with weight structure, mark a reader as template-driven, and the same single control could classify any claimed weight-reading or weight-editing capability even when aggregate accuracy looks high.","If the deflationary account transfers across LLMs, the reading side of weight-space programming is bounded by how well a model's training corpus covers public frameworks and checkpoints, which predicts sharp degradation on proprietary architectures or unusual layer designs regardless of model scale.","The paper's proposed forward-pass test could be standardized into a benchmark — exact logit prediction on arbitrary inputs for unseen networks — giving a clean pass/fail boundary between template matching and genuine tracing that would settle what the self-reports left open.","The verified edge statistics themselves (optimizer-step counts, grayscale detection from batch-norm running values, output correlation structure) could be built into a forward-pass-free auditing tool for third-party models, a use the paper notes in passing but does not develop."],"forward_implications":["A correct dataset name recovered from weights is weak evidence of weight understanding unless label order, dataset familiarity, and network accuracy are controlled; the paper's protocol supplies those controls.","The model's reliable skills sit at the network boundary, so weight reading should be expected to succeed on low-dimensional edge statistics and to struggle on multi-layer feature tracing and obscure datasets.","Weight-editing results are not refuted by this audit, but any edit that improves a benchmark should be shown to depend on the claimed understanding rather than on search over the weights.","The label-permutation control, the shuffled-weight control, and the familiarity and accuracy comparisons give future audits a cheap first check for whether a claimed weight-reading result is prior-driven.","The verified edge statistics — optimizer-step counts, grayscale detection, output correlation structure — can support useful audits of third-party models without amounting to circuit-level understanding."],"supporting_citations":[{"why":"The weight-space programming report whose reading-side claim this audit tests; supplies the capability being evaluated.","marker":"[36]"},{"why":"Gives the batch-normalization running-statistics formulas and momentum conventions the model uses to count optimizer steps.","marker":"[17]"},{"why":"Defines the Adam optimizer whose defaults underpin the model's inference of about three training updates.","marker":"[21]"},{"why":"Supplies the initialization bounds and default conventions the model leans on for its local arithmetic.","marker":"[31]"},{"why":"Establishes CIFAR-10's 32x32/ten-class footprint and standard label order, the template the model matches against.","marker":"[22]"},{"why":"Defines KMNIST, the dataset the model relabels as MNIST despite correctly deducing grayscale input.","marker":"[10]"},{"why":"Defines SAT-6, the satellite dataset the model reads as CIFAR-10 animal photos.","marker":"[3]"},{"why":"Weight Oracles: the contrast case of a fine-tuned weight reader with external computation, which the paper distinguishes from the untrained general model.","marker":"[18]"}],"fun_headline_variants":["LLM weight reading is local math plus dataset memory, not tracing","Claude does edge math, then names datasets from memory","Weight-space reading is arithmetic and priors, not tracing","Shuffled labels keep the split but break every class read","No forward pass: it computes, then guesses CIFAR-10 from memory"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The claim that the model does not trace weight flows rests on the model's own statements about which layers it read and a few failed spot-checks, because no session was ever asked to compute a forward pass; the paper itself says these are self-reports, not measurements of an internal process.","fun_headline_variants_meta":{"raw":{"variants":["LLM weight reading is local math plus dataset memory, not tracing","Claude does edge math, then names datasets from memory","Weight-space reading is arithmetic and priors, not tracing","Shuffled labels keep the split but break every class read","No forward pass: it computes, then guesses CIFAR-10 from memory"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000248,"raw_usage":{"cost":0.0002482116,"is_byok":false,"cost_details":{"upstream_inference_cost":0.0002482116,"upstream_inference_prompt_cost":3.54536e-05,"upstream_inference_completions_cost":0.000212758},"total_tokens":2485,"prompt_tokens":1316,"completion_tokens":1169,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":1158,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":1093}},"tokens_in":1316,"tokens_out":1169,"cost_source":"openrouter_usage","duration_ms":15561,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":1093,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-06T10:23:38.983763+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Run the forward-pass flow test the paper proposes: hand the same kind of session a small network and one concrete input image and ask for the exact output logits, with no tools. If the answers land close to the true logits on inputs the model has never seen, the \"no weight-flow tracing\" conclusion fails; if they do not, the deflationary account holds. A quicker check is to repeat the label-permuted CIFAR-10 run several times under the revised prompt, expecting the split at the new indices together with consistently wrong class names.","supporting_citations":[{"cited_title":"Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights.https://rikiyat.github.io/blog/ weight-space-programming/, November 2026","cited_arxiv_id":null,"evidence_quote":"The weight-space programming report whose reading-side claim this audit tests; supplies the capability being evaluated."},{"cited_title":"Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift","cited_arxiv_id":"1502.03167","evidence_quote":"Gives the batch-normalization running-statistics formulas and momentum conventions the model uses to count optimizer steps."},{"cited_title":"Adam: A Method for Stochastic Optimization","cited_arxiv_id":"1412.6980","evidence_quote":"Defines the Adam optimizer whose defaults underpin the model's inference of about three training updates."},{"cited_title":"PyTorch: An Imperative Style, High-Performance Deep Learning Library","cited_arxiv_id":"1912.01703","evidence_quote":"Supplies the initialization bounds and default conventions the model leans on for its local arithmetic."},{"cited_title":"Learning multiple layers of features from tiny images","cited_arxiv_id":null,"evidence_quote":"Establishes CIFAR-10's 32x32/ten-class footprint and standard label order, the template the model matches against."},{"cited_title":"Deep Learning for Classical Japanese Literature","cited_arxiv_id":"1812.01718","evidence_quote":"Defines KMNIST, the dataset the model relabels as MNIST despite correctly deducing grayscale input."},{"cited_title":"DeepSat - A Learning framework for Satellite Imagery","cited_arxiv_id":"1509.03602","evidence_quote":"Defines SAT-6, the satellite dataset the model reads as CIFAR-10 animal photos."},{"cited_title":"Weight Oracles: Reading Neural Network Weights with Language Models","cited_arxiv_id":"2610.07334","evidence_quote":"Weight Oracles: the contrast case of a fine-tuned weight reader with external computation, which the paper distinguishes from the untrained general model."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}