{"id":"5ae7b38b-6886-4af8-be8f-aafadbc2c8ba","arxiv_id":"2608.13048","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"DMDIntel ranks input tokens by how strongly their hidden-state changes project onto the dominant linear modes of a model's token trajectory, and this ranking usually matches GPT-4.1-labeled important tokens better than PCA, IG, or SHAP.","lead":"This paper applies dynamic mode decomposition to the hidden states of fine-tuned language models and ranks input tokens by their projection onto the resulting modes. The method, called DMDIntel, is tested on three text classification datasets and three model families, where it usually matches GPT-4.1-labeled important tokens slightly better than PCA, integrated gradients, and SHAP.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim of consistent superiority is contradicted by the paper's own faithfulness table: DMDIntel is third on several accuracy-drop cells, and the ranking metrics are anchored to GPT-4.1 ground truth with only moderate human agreement.","rationale":"After reading the paper in good faith, I find the core idea interesting: applying DMD to hidden-state trajectories and using mode projections for attribution is novel, and the authors include several of the right checks, namely layer sensitivity, denoising ablation, reconstruction error, and a masking-based faithfulness analysis. The problem is that those checks do not cohere with the central claim. The ranking evidence is only as good as the GPT-4.1 ground truth, and the paper's own Table 6 shows that causal faithfulness is not consistently in DMDIntel's favor. I considered whether the main issue is the post-hoc selection of the best of four DMD/HODMD variants per dataset; that is a real secondary concern, since reporting the best variant against fixed baselines inflates apparent gains, and the adaptive delay parameter d is never specified. But the decisive issue is the evaluation target: without a known causal ground truth or a consistently favorable perturbation result, the abstract's 'by far outperforms' and 'consistent' language is not justified. A synthetic-trigger experiment would settle this directly, and if it succeeds the method deserves publication; if it fails, the paper should be revised to claim only alignment with external rationales, not causal driver identification. I therefore keep the reader's CONDITIONAL verdict rather than escalating to REJECT, because the method is plausible and the required fix is a well-defined validation experiment, not a conceptual impossibility.","tokens_in":21703,"tokens_out":8157,"duration_ms":82842,"concrete_test":"Fine-tune the three LLMs on a synthetic classification task whose label depends exactly on a small set of trigger tokens (e.g., label=positive iff token A or B appears), with hyperparameters and the adaptive delay rule fixed before evaluation; then compute recall@k and RBO of DMDIntel, PCA, IG, and SHAP against this exact ground truth. If DMDIntel does not beat PCA significantly (paired bootstrap or Wilcoxon across 1,000 samples), the claim that it identifies the highest proportion of ground-truth tokens is not supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The load-bearing weakness is that the paper's central claim is tested against GPT-4.1-generated importance rankings, and the one causal check the paper includes does not support the headline. In Section 6.4, Table 6, DMDIntel is described as 'consistently first or second' in accuracy drop, but on Negative Sentiment the accuracy drops are: Llama-3.2-3B-inst: IG 12.18, PCA 13.68, DMDIntel 12.57; Qwen3-4B-inst: IG 15.77, PCA 12.68, DMDIntel 10.78; Mistral-7B-v0.3-inst: IG 26.75, PCA 19.76, DMDIntel 17.76. On HateXplain hateful, Llama: IG 50.66, PCA 56.88, DMDIntel 45.50; Qwen: IG 57.01, PCA 47.75, DMDIntel 44.31. In these cells DMDIntel ranks third, and IG or PCA gives the largest drop. The ranking metrics (matched count, RBO, recall@k) are computed against GPT-4.1 rankings whose human agreement is only RBO 0.54 to 0.70 (Section 5), so they measure alignment with another model's opinion rather than faithfulness to the fine-tuned LLM. Appendix I concedes that identifying label-intensive tokens does not imply causal dependency. Combined with the reconstruction error of 0.06 to 0.13 in Appendix F, the argument that DMD modes are 'the main drivers' of the model's decision lacks the causal support the headline requires.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DMDIntel, an input-attribution method for fine-tuned decoder-only LLMs. It extracts per-token hidden states from the MLP down_proj of a selected layer, subtracts an instruction-bias vector obtained from an empty prompt, applies DMD or HODMD to the debiased hidden-state trajectory, and ranks tokens by their projected scores onto the dominant modes. The method is evaluated on sentiment, HateXplain, and FakeEdit classification across Llama-3.2-3B, Qwen3-4B, and Mistral-7B, with PCA, IG, and SHAP as baselines. The paper claims that DMDIntel consistently identifies the highest proportion of ground-truth tokens and preserves their ranked importance, outperforming the baselines by a large margin. It also includes a token-masking fidelity analysis, layer-selection sensitivity experiments, ablations, and a reconstruction-error analysis.","tokens_in":22019,"tokens_out":8007,"duration_ms":77202,"significance":"The dynamical-systems perspective on hidden-state trajectories is a genuinely novel angle for decoder-only LLM attribution, and the single-forward-pass property makes the method practically attractive if its claims hold. The paper is also useful in its breadth of ablations: layer sensitivity (Table 7), denoising (Appendix H.1), threshold choice (Appendix H.2), top-k choice (Appendix H.3), and mode count (Appendix H.4) are all examined, and the reconstruction-error analysis in Appendix F is a constructive check. However, the evaluation as designed primarily measures agreement with GPT-4.1-generated rankings rather than faithfulness to the fine-tuned model, and the paper's own causal fidelity results do not support the strong headline. The core method is coherent and worth further study, but the manuscript's central comparative claims need substantial revision.","major_comments":[{"comment":"The claim that DMDIntel 'by far outperforms' PCA, IG, and SHAP is not supported by the reported numbers. In Table 4 (Mistral-7B-v0.3-inst, HateXplain), PCA beats DMDIntel on all three metrics (MC 2.95 vs. 2.82, RBO 0.30 vs. 0.26, Recall 0.67 vs. 0.65). In Table 3 (Qwen3-4B-inst, Negative Sentiment), PCA has a higher matched count (5.46 vs. 5.43) and equal recall, and the RBO advantage for DMDIntel is only 0.01. These are marginal or reversed differences, not the 'by far outperforms' stated in the abstract and introduction. The claims should be rewritten to match the actual effect sizes, or the evaluation should be extended to demonstrate a robust advantage.","section":"Abstract; Section 1; Tables 3-4"},{"comment":"The ranking metrics are computed against GPT-4.1-generated token importance rankings whose human agreement is only RBO 0.54-0.70 (Section 5). This measures alignment with another model's opinion, not faithfulness to the fine-tuned LLM's decision. The masking experiment in Section 6.4, Table 6, is the only causal check, and it contradicts the 'main drivers' language: for Qwen3-4B-inst Negative Sentiment, DMDIntel's accuracy drop is 10.78 versus IG 15.77 and PCA 12.68; for Llama-3.2-3B-inst on HateXplain hateful, DMDIntel drops 45.50 versus IG 50.66 and PCA 56.88. In these and other cells DMDIntel ranks third, contrary to the text's statement that it 'consistently ranks first or second.' The causal evidence should be made the primary evaluation, and the abstract and introduction should be reconciled with these results.","section":"Section 5; Section 6.4; Table 6"},{"comment":"The main results report the best of four DMD configurations (DMD or HODMD crossed with amplitude-based or time-averaged-amplitude mode ranking), and the best variant changes by dataset and model. This post hoc selection, without a validation split or correction for multiple comparisons, inflates the reported performance. The full comparison in Appendix D shows many configurations within 0.01-0.02 of each other. The chosen variant should be justified a priori, or the full variant table should be in the main text and treated as a sensitivity analysis rather than as the method's default performance.","section":"Section 6; Appendix D"},{"comment":"No statistical significance tests, confidence intervals, or repeated-run variability are reported, yet differences as small as 0.01 in RBO are treated as evidence of superiority. Since each evaluation uses 1,000 samples per dataset and the metrics are averages, the paper should provide bootstrap confidence intervals or multiple fine-tuning seeds to establish that the observed advantages are not noise. This is especially important because several headline comparisons in Tables 3 and 4 are within 0.01-0.03 on RBO and recall.","section":"Section 6; Tables 3-4"},{"comment":"The paper itself concedes that 'identifying label-intensive tokens does not imply causal dependency' and that non-labeled contextual tokens may be relevant to the decision trajectory. This directly limits the Introduction's claim that the identified tokens are 'the main drivers' of the model's input processing. The abstract and introduction should be revised to reflect this limitation, and the fidelity-by-masking analysis should be presented as the appropriate test of the causal claim rather than as a secondary check.","section":"Appendix I"}],"minor_comments":[{"comment":"In Table 15, the Llama-3.2-3B-inst FakeEdit row reports RBO@20 = 3.43 for DMDIntel, which is outside the valid [0,1] range and appears to be a typo for 0.34.","section":"Table 15"},{"comment":"The method name is formatted inconsistently as 'DMDINTEL' and 'DMDIntel', and there are spacing errors such as 'DMDINTELwhichuses' in the abstract; these should be cleaned up.","section":"Abstract; Section 1"},{"comment":"The RBO formula as written should be checked: the term X_k/k * p^k is not the standard RBO extrapolated agreement, and the expression as shown is not obviously bounded between 0 and 1.","section":"Section 5, Eq. (8)"},{"comment":"The GPT-4.1 ground truth is attributed to Achiam et al. (2023), which is the GPT-4 technical report; a GPT-4.1-specific citation or model-card reference should be provided.","section":"References"},{"comment":"Algorithm 1 hard-codes the cosine threshold tau = 0.25, and the sensitivity analysis only tests tau = 0.5; a small sweep or a data-driven justification for the fixed value would make the layer-selection heuristic more robust.","section":"Section 4.3; Appendix H.2"},{"comment":"The reconstruction errors in Table 9 range up to 0.132, which is described as 'quite accurate'; a comparison against the reconstruction error of PCA or a random subspace would help calibrate this claim.","section":"Appendix F"}],"recommendation":"major_revision","confidential_remarks":"The main blockers are the mismatches between the headline claims and the paper's own Tables 3, 4, and 6, together with the reliance on GPT-4.1-generated ground truth as the primary evaluation target. I would ask the authors to reposition the contribution as a fast, lightweight attribution method with modest gains, report the full variant comparison transparently, and make the causal fidelity analysis the basis for any 'main drivers' claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on DMDIntel. The genuinely new thing is using DMD on the hidden-state trajectory of a decoder-only LLM to rank input tokens, with a bias-subtraction step and a heuristic layer picker. That's a real idea, and the paper does more experiments than most interpretability papers—three models, three datasets, plus ablations.\n\nBut the headline doesn't survive contact with the paper's own numbers. 'By far outperforms' is contradicted by Tables 3 and 4: the gains are often small, PCA wins several cells (e.g., Mistral on HateXplain RBO 0.30 vs 0.26), and in the fidelity analysis of Table 6, DMDIntel is third on several accuracy-drop cells—IG and PCA both beat it on Negative Sentiment for Qwen and Mistral. The ranking metrics are computed against GPT-4.1 ground truth whose human agreement is only RBO 0.54–0.70, so the headline measures alignment with another model's opinion, not faithfulness to the fine-tuned LLM. The paper does include one causal check, the masking experiment, and it doesn't support the 'consistently first or second' claim. Appendix I honestly concedes that identifying label-intensive tokens does not imply causal dependency—good that they say it, but it should be in the main text.\n\nThere are also reproducibility gaps: they report the best of four variant configurations, the chosen variant switches per dataset, the HODMD delay parameter is called 'adaptive' but never specified, and no code or data is released. Appendix Table 15 has an RBO of 3.43 for FakeEdit, which is impossible—likely a typo, but it shouldn't ship.\n\nCredit where it's due: the bias-subtraction idea is sound and the layer-selection heuristic is plausible; the qualitative results show DMDIntel is often better than IG and SHAP in practice, and the cross-model stability is a nice point. The reconstruction errors (0.06–0.13) suggest the linear surrogate isn't crazy. So the method is worth pursuing, but the evaluation needs to be substantially cleaned up: pre-register the protocol, report all variants, specify the delay rule, release artifacts, and calibrate the abstract.\n\nWho's this for? Interpretability researchers who care about cheap, single-pass attribution and like dynamical-systems framing. It deserves a serious referee—the novelty is real—but I'd only accept after major revision. Recommendation: send to peer review, and insist the authors address the above before it can be trusted.","headline":"DMDIntel is a genuinely novel single-pass attribution method, but its 'by far outperforms' claim is unsupported by its own tables and the evaluation needs substantial cleanup before the results can be trusted.","tokens_in":22605,"tokens_out":2791,"would_cite":false,"duration_ms":24174,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Dynamic mode decomposition of a fine-tuned LLM's token-by-token hidden-state trajectory can rank the input tokens that drive a classification decision, outperforming PCA, integrated gradients, and SHAP on the paper's benchmarks.","keywords":["dynamic mode decomposition","LLM interpretability","input attribution","hidden-state trajectory","token ranking","higher-order DMD","text classification","explainable AI"],"falsifier":"Score the same attribution methods against two alternative ground truths: human annotator rankings on the paper's own samples, and causal ground truth from token-deletion or token-perturbation experiments. If DMDIntel's advantage over PCA, integrated gradients, and SHAP shrinks, reverses, or disappears under either alternative, the central claim is refuted; the paper's reported human-to-annotator-model RBO of 0.54-0.70 makes this check decisive. A cheaper version: mask the top-k tokens from DMDIntel, from a random baseline, and from the annotator model, and compare accuracy drops—if DMDIntel's drops are indistinguishable from random, the attribution is not faithful.","tokens_in":21431,"feed_emoji":"🧠","tokens_out":7426,"duration_ms":64397,"temperature":0.7,"pith_summary":"This paper sets out to show that a fine-tuned decoder-only LLM can be interpreted by treating its per-token hidden states as a dynamical trajectory and decomposing that trajectory into a handful of spatio-temporal modes. The proposed method, DMDIntel, takes the MLP down-projection vectors for each token, subtracts an 'instruction bias' captured from an empty prompt, selects a decoder layer by a cosine-similarity heuristic, and applies dynamic mode decomposition (or its higher-order variant) to obtain modes ranked by amplitude. Token importance is then the summed magnitude of that token's state change projected onto the top five modes. Across three text-classification datasets and three model families, the paper reports that this ranking recovers more ground-truth tokens than PCA, integrated gradients, and SHAP, with comparable or better rank overlap, and does so in a single forward pass. The reason to care is that a parameter-free, post-hoc attribution built on sequential information flow, if reliable, would be much cheaper and more interpretable than gradient- or perturbation-based baselines.","feed_headline":"One forward pass ranks LLM key tokens via DMD modes","feed_subtitle":"Trajectories of hidden states decompose into modes that beat PCA, integrated gradients, and SHAP at finding key tokens.","key_machinery":"The machinery is the eigendecomposition of a best-fit linear operator on token-indexed hidden states. Treating the sequence of down-projection MLP outputs as $h_{t+1} \\approx A h_t$, the method builds a snapshot matrix, removes the instruction bias, and (for HODMD) stacks delayed snapshots into a Hankel matrix before an SVD-based fit; the eigenmodes $\\phi_i$ of $A$ are the low-dimensional spatio-temporal patterns, and their eigenvalues encode growth, decay, and oscillation. Modes are ranked by initial amplitude or by time-averaged amplitude, and each token's state change $\\Delta h_t = h_t - h_{t-1}$ is projected onto the top-$k$ modes, with the sum of absolute projections giving the token's attribution score. A layer-selection heuristic picks the decoder layer at the steepest ascent of the average cosine similarity between token vectors and the instruction-bias vector below a threshold of 0.25, which is meant to balance input-token fidelity against instruction-following bias.","core_discovery":"The central claim is that the dominant DMD/HODMD modes of the debiased hidden-state trajectory serve as reliable indicators of which input tokens drive a fine-tuned LLM's classification output, and that ranking tokens by projection onto these modes outperforms PCA, integrated gradients, and SHAP on the paper's matched-count, rank-biased overlap, and recall-at-k metrics. The choice of decomposition is task-dependent: HODMD with time-averaged amplitude ranking wins on sentiment, where meaning is spread across multi-word phrases, while standard DMD with initial-amplitude ranking wins on HateXplain and FakeEdit, where single trigger words dominate. The paper also claims that the linear surrogate is accurate—reconstruction error is of order $10^{-2}$—and that the attribution is stable under one-layer shifts of the selected decoder layer. A token-masking fidelity experiment shows that removing DMDIntel's top tokens lowers accuracy and confidence at rates comparable to the stronger baselines.","pith_inferences":["Because the paper scores against one annotator model's rankings, the strongest next test is to score against direct human rankings or against causal tests such as deletion of the attributed tokens; the reported human-to-annotator-model agreement of RBO 0.54-0.70 suggests the two standards could disagree on which method wins.","The low linear-reconstruction error of the surrogate operator implies DMD modes could support counterfactual experiments, such as predicting the hidden-state trajectory after deleting a token, which would turn the attribution ranking into a directly testable causal claim.","The pipeline is framed for single-token-output classification; extending it to multi-token generation, as the authors note, requires a different snapshot formulation because generated tokens re-enter the input auto-regressively, and that extension would determine whether dynamical attribution works beyond classification."],"forward_implications":["If DMDIntel's attributions are faithful, token-level explanations for fine-tuned decoder-style classifiers can be produced non-intrusively, with a single forward pass, avoiding the many forward and backward passes that gradient and perturbation methods require.","The task-dependent winner between standard DMD and HODMD suggests that the appropriate temporal memory length is a property of the task: localized single tokens call for near-Markovian dynamics, while phrase-level sentiment calls for delay embedding.","The layer-selection heuristic and instruction-bias subtraction generalize across the three model families tested, indicating the pipeline may transfer to other fine-tuned decoder-only models without per-model tuning.","Layer-sensitivity results, with maximum recall deviation around 0.02 across $\\ell^*\\pm 1$, imply the method does not rest on a fragile choice of decoder layer."],"supporting_citations":[{"why":"Introduces dynamic mode decomposition, the core technique applied to hidden-state snapshots.","marker":"(Schmid, 2010)"},{"why":"Supplies higher-order DMD with delay-embedded Hankel matrices, the variant used for sentiment classification.","marker":"(Le Clainche and Vega, 2017)"},{"why":"Provides the amplitude-based mode ranking criterion used for standard DMD attribution.","marker":"(Rowley et al., 2009)"},{"why":"Provides time-averaged amplitude ranking, which the paper uses for HODMD-based attribution.","marker":"(Tissot et al., 2014)"},{"why":"Motivates instruction-bias subtraction by showing DMD's sensitivity to noise in the data.","marker":"(Hemati et al., 2017)"},{"why":"Source of the annotator model used to generate ground-truth token rankings for evaluation.","marker":"(Achiam et al., 2023)"},{"why":"Defines integrated gradients, one of the baseline attribution methods compared against.","marker":"(Sundararajan et al., 2017)"},{"why":"Defines SHAP, another baseline attribution method compared against.","marker":"(Lundberg and Lee, 2017)"},{"why":"Defines Rank-Biased Overlap, the metric central to evaluating ranked attribution quality.","marker":"(Webber et al., 2010)"}],"fun_headline_variants":["DMD beats SHAP and IG at ranking LLM key tokens in one pass","DMD modes outperform PCA, IG, and SHAP for LLM token ranking","DMDIntel: Find LLM key tokens better than SHAP and IG","Why DMD beats PCA, IG, and SHAP on LLM attribution","DMD modes reveal LLM key tokens, beating SHAP and IG"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the external annotator model's token rankings are a valid proxy for what truly drives the fine-tuned classifier's decision; the paper's human agreement with that proxy is only moderate (RBO 0.54-0.70), and no experiment causally verifies that the ranked tokens are the ones whose removal changes the prediction.","fun_headline_variants_meta":{"raw":{"variants":["DMD beats SHAP and IG at ranking LLM key tokens in one pass","DMD modes outperform PCA, IG, and SHAP for LLM token ranking","DMDIntel: Find LLM key tokens better than SHAP and IG","Why DMD beats PCA, IG, and SHAP on LLM attribution","DMD modes reveal LLM key tokens, beating SHAP and IG"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001211,"raw_usage":{"total_tokens":4929,"prompt_tokens":835,"completion_tokens":4094,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":451,"completion_tokens_details":{"reasoning_tokens":3992}},"tokens_in":451,"tokens_out":4094,"duration_ms":24381,"temperature":1.0,"reasoning_tokens":3992,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:52:50.938704+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Score the same attribution methods against two alternative ground truths: human annotator rankings on the paper's own samples, and causal ground truth from token-deletion or token-perturbation experiments. If DMDIntel's advantage over PCA, integrated gradients, and SHAP shrinks, reverses, or disappears under either alternative, the central claim is refuted; the paper's reported human-to-annotator-model RBO of 0.54-0.70 makes this check decisive. A cheaper version: mask the top-k tokens from DMDIntel, from a random baseline, and from the annotator model, and compare accuracy drops—if DMDIntel's drops are indistinguishable from random, the attribution is not faithful.","supporting_citations":[{"cited_title":"Noack , title =","cited_arxiv_id":null,"evidence_quote":"Provides time-averaged amplitude ranking, which the paper uses for HODMD-based attribution."}],"review_version":1}