{"id":"92c3af79-b3a1-413c-bc06-28a0da9aada0","arxiv_id":"2506.10378","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"From Open LLM Leaderboard data grouped by base model, the authors recover a three-factor ordering of LLM capabilities and claim instruction-following causally supports math reasoning.","lead":"This paper applies a causal representation learning method to public LLM benchmark scores and claims that model capabilities form a causal chain: general reasoning, then instruction following, then mathematics. It matters because it proposes a guide for which capabilities to strengthen during fine-tuning, though the causal evidence is not fully convincing.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed z2→z3 causal edge is never identified: HCA recovers only a transitive closure under a triangular parametrization, so the reported direction rests on an untested modeling assumption.","rationale":"Agreeing with the reader, I focus on the one assumption that bears the whole causal direction. The paper is transparent about the gap: Section C.1 says HCA only identifies the transitive closure and the authors deliberately skip the edge-existence test. Without that test, a fitted nonzero coefficient in the triangular B_k is not evidence for a direct edge. The post hoc semantic alignment compounds the problem: z3 is chosen to maximize R² with MATH, so the chain's interpretation is not a holdout prediction. The independent contribution—the low-rank heterogeneous-base-model analysis, the matrix-completion remark, and the reproducible code release—is real, but it does not rescue the directional claim. The proposed constrained refit is cheap and would settle whether z2→z3 survives when not assumed by the parametrization. Since the reader already rejected, no verdict adjustment is needed.","tokens_in":57826,"tokens_out":6920,"duration_ms":87417,"concrete_test":"Refit HCA on the same four domains (Llama-3-8B, Llama-3.1-8B, Qwen2.5-7B, Qwen2.5-14B) with the off-diagonal entry of B_k corresponding to z2→z3 constrained to zero for all k, re-optimizing the remaining triangular entries and the permutation search; compare the resulting MIC and rank-1 row-residual errors (Table 4a) with the unconstrained 0.04 fit. If the MIC does not materially increase (say ≤0.05), the z2→z3 edge is an artifact of the triangular parametrization and the central causal claim is unsupported. Run the same zero-constraint test for z1→z2 and z1→z3 to see which edges survive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing causal direction 'instruction following (z2) → math (z3)' is not actually identified by HCA. In Section C.1 the authors state that HCA 'only recovers a transitive closure Ḡ of the true graph G' and that, 'for simplicity and due to the fundamental inexactness of our model', they do not test which edges of Ḡ exist; they simply constrain each B_k to be triangular. The edge weights in Figure 7 are fitted parameters in that triangular parametrization, not discoveries. Theorem 1 certifies recovery only up to left multiplication by a lower-triangular matrix (equivalently, up to the transitive closure of the total order); it does not establish that the sparse chain z1→z2→z3, rather than a denser graph with the same closure, generated the data. The semantic labels are then assigned by OLS maximizing R² against individual benchmarks (Section 4.1), so the abstract's 'clear causal direction' is a post hoc reading of an unverified edge. A constrained fit with the z2→z3 coefficient forced to zero could produce essentially the same MIC (0.04), which would leave no empirical content to the directional claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a causal representation learning framework, Hierarchical Component Analysis (HCA), to recover latent capability factors from benchmark performance data of LLMs. The model assumes a shared linear mixing matrix across base models, with latent factors following a linear structural causal model per base model. Applied to Open LLM Leaderboard data for four base models, the authors report a three-node causal structure with direction from general problem-solving (z1), through instruction-following (z2), to mathematical reasoning (z3), supported by a minimal inexactness coefficient (MIC) of 0.04 and by SFT experiments. The paper also includes analyses of base-model heterogeneity, matrix completion, and MMLU subtasks.","tokens_in":58040,"tokens_out":9340,"duration_ms":94776,"significance":"The idea of using multi-domain heterogeneity to identify latent causal capabilities is timely and addresses a real gap in causal evaluation of LLMs. The HCA algorithm is novel and comes with theoretical identifiability results, and the code is publicly released. The additional experiments on base-model-controlled matrix completion and the stability analysis are useful contributions. However, the central claim of a sparse causal direction is not supported by the identification result (which recovers only a transitive closure) or by the empirical evidence; this substantially weakens the paper's contribution as a causal discovery result.","major_comments":[{"comment":"The identifiability guarantee of HCA only identifies the latent factors up to a lower-triangular transformation, i.e., up to the transitive closure of the true graph. In Section C.1 the authors explicitly state that HCA 'only recovers a transitive closure Ḡ of the true graph G' and that, 'for simplicity and due to the fundamental inexactness of our model', they do not test which edges exist, instead constraining each Bk to be upper-triangular. Consequently, the graph in Figure 7 includes all edges consistent with a total order, such as an edge z1→z3, and the sparse chain z1→z2→z3 claimed in the abstract is not identified. Theorem 1 itself is stated and proved for a complete DAG with all lower-triangular edges, so it certifies recovery only of the total order, not of sparsity. The 'clear causal direction' in the abstract is therefore a post hoc reading of an unverified modeling assumption.","section":"§C.1, §C.2, Theorem 1, Figure 7"},{"comment":"The semantic labels of the latent factors are assigned by an OLS procedure that selects, for each factor, the benchmark maximizing R². The reported R²=1.00 for z3 versus MATH Lvl 5 is an in-sample fitting outcome, not a predictive validation. This circularity means the strong correlations cannot be used as evidence for the causal interpretation of the factors, and the apparent near-perfect alignment with MATH is partly a consequence of the label-selection procedure.","section":"§4.1, Figure 8, Table 3"},{"comment":"The SFT experiments do not provide direct interventional evidence for the specific z2→z3 edge. Fine-tuning on IFEval is an intervention on the whole model, not on the isolated latent factor z2, and the measured MATH improvements are small and inconsistent: Gemma-2-9B shows a decrease from 0.13 to 0.12 after IFEval SFT, and Llama-3-8B stays at 0.05. The 'Instruct' rows in Table 2 are observational leaderboard entries subject to selection bias, so they cannot serve as controlled causal evidence.","section":"§4.2, Table 2"},{"comment":"The text states that 'running our algorithm on the subset of {1,2,4,5}, with Qwen2-7B excluded, achieves a minimal MIC of 0.04.' Figure 18, however, shows size-3 domain subsets with MIC values as low as 0.02. This inconsistency undermines the reported model selection: if other subsets yield smaller MIC, then the choice of {1,2,4,5} and the associated causal graph are not justified as the best-fitting model.","section":"§4.1, Figure 18"}],"minor_comments":[{"comment":"There are typos: 'wuth' should be 'with' in Definitions 1 and 2, and 'varisbles' should be 'variables' in Section 3.1. Also, 'benhmark' in Appendix G.3 should be 'benchmark'.","section":"§3 (Definitions 1, 2) and §3.1"},{"comment":"The text defines Sinv = {1,2,4,5,6} as 'the index set of these seven models', but the set has only five elements; this should say 'five models'.","section":"§2.1"},{"comment":"The OLS formula is presented with 'P' instead of a summation symbol: 'zi ≈ P j<i ajzj +γ BxB +c'. The description of the adjustment could be more precise about what is being residualized and why.","section":"§4.1"},{"comment":"The caption labels 'Explained Variance Ratio' and 'Cumulative Explained Variance' in a way that appears to be swapped relative to the plot content; please check the labeling.","section":"Figure 2a caption"},{"comment":"The sensitivity analysis shows that the z1 factor is not stable when the domain set is expanded, and the authors themselves note that 'it is hard to argue which one makes more sense.' This instability is not flagged in the main text, where z1 is described as 'foundational general capability.'","section":"Appendix H"}],"recommendation":"reject","confidential_remarks":"The paper has valuable components, particularly the formalization of multi-domain CRL for capability discovery and the released code. However, the central causal claim is not identified by the method: HCA recovers only a transitive closure under an imposed total order, and the abstract overstates the result as a 'clear causal direction'. The SFT evidence is not a direct intervention test, and the semantic label fitting is circular. These issues are load-bearing and would require a fundamental re-scoping of the claims, so I recommend rejection rather than major revision. If the authors reframe the work as a hypothesis-generation study about latent factor structure without claiming directional causation, it might be suitable for a more exploratory venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe short version: this paper makes a serious, transparent attempt to push causal representation learning into LLM benchmarking, but the headline causal claim is not supported by the algorithm. The authors admit in Section C.1 that HCA only recovers the transitive closure of the true graph and that they don't test which edges exist; they simply fit upper-triangular weight matrices. So the edge weights in Figure 7 are fitted parameters, not discoveries. The abstract's 'clear causal direction' is a post hoc reading.\n\nWhat's genuinely useful: the low-rank structure of benchmark performance and the base-model heterogeneity observation are worth reporting, and the matrix-completion detour is nice. The HCA algorithm is a reasonable variation of LiNGCReL with an inexactness relaxation, and the authors are unusually candid about its limitations—they report MIC, unmixing errors, and a sensitivity analysis. The SFT experiments are a good-faith attempt at external validation, even if the results are mixed.\n\nThe core problem is that Theorem 1 requires an exact SCM, but the data is only alpha-inexact with MIC 0.04. The theorem certifies recovery only up to the transitive closure, i.e., the total order, not the sparse edges. Since the authors then impose a triangular structure, the specific z2→z3 edge is an assumption, not a discovery. A constrained fit with that coefficient forced to zero could plausibly give the same MIC, which would remove the empirical content from the directional claim. The semantic labels are also assigned by maximizing R2 against individual benchmarks, so the R2=1.00 alignment with MATH is a fitting outcome, not a prediction. The domain subset is selected post hoc to minimize MIC. These are load-bearing issues, not minor quibbles.\n\nWho is this for? Anyone working on causal representation learning or LLM evaluation. It's a good case study of how easy it is to overclaim causal structure from fitted latent variable models. I would bring it to reading group for the discussion. I likely won't cite it as a causal discovery, but the HCA algorithm could be cited in work on inexact linear SCMs.\n\nMy recommendation: send it to peer review with the expectation of major revisions. The methodological kernel is worth preserving, and the heterogeneity analysis is solid. But the causal-direction claim needs to be reframed as a modeling assumption, and the paper should test whether sparser graphs fit equally well. This deserves a serious referee, not a desk reject.","headline":"A transparent, useful look at low-rank structure and heterogeneity in LLM benchmarks, but the claimed causal hierarchy is an artifact of the triangular parametrization, not a discovery.","tokens_in":58623,"tokens_out":4552,"would_cite":false,"duration_ms":49643,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Language-model benchmark results trace a causal chain from general capability through instruction-following to math.","keywords":["causal representation learning","language model evaluation","latent capability factors","hierarchical structural causal model","benchmark heterogeneity","instruction following","mathematical reasoning","identifiability"],"falsifier":"Take one of the four base models, fine-tune it only on instruction-following examples that contain no mathematical content, and compare its MATH Lvl 5 score against a control model fine-tuned on an equal number of non-instruction paragraphs; the claimed causal chain predicts a significantly larger math gain in the instruction group, so a null or reversed result would refute it.","tokens_in":57565,"feed_emoji":"🧠","tokens_out":13582,"duration_ms":149550,"temperature":0.7,"pith_summary":"This paper tries to establish that the six benchmark scores of fine-tuned language models are generated by three latent capability factors, and that those factors are causally ordered rather than merely correlated. To show this, it develops Hierarchical Component Analysis (HCA), a causal representation learning method that treats the base model as a shared confounder and recovers latent factors that are linear in the benchmarks and structurally related across base-model domains. Applied to public leaderboard data on over 1,500 models built from four base models, the method finds a compact three-node chain, with general problem-solving feeding instruction-following and instruction-following feeding mathematical reasoning. A sympathetic reading is that evaluation results are therefore not a flat leaderboard: they encode an actionable hierarchy, and math-score gains should be interpreted with the mediating instruction-following capability in mind.","feed_headline":"LLM skills form a causal chain ending in math ability","feed_subtitle":"Benchmark data from 1,500+ models places instruction-following between general reasoning and math.","key_machinery":"Hierarchical Component Analysis (HCA) is the central object. It starts with independent component analysis (ICA) on each base-model domain to obtain an unmixing matrix $M_k$ that maps observed benchmarks to independent sources, and it exploits the decomposition $M_k = P_k B_k H$, where $B_k$ is the domain-specific lower-triangular structural weight matrix and $H$ is the shared unmixing inverse of the mixing matrix $G$. The algorithm's key step is row-residual extraction: after guessing an order, it projects each row of $M_k$ onto the span of earlier rows and requires the projected residuals across domains to be rank 1, recovering the next row of $H$ as the top singular vector; it then searches over row permutations and fits the best lower-triangular $B_k$'s, selecting the configuration with the smallest maximum inexactness coefficient. The paper proves that under an exact linear SCM with non-Gaussian sources and enough generic domains, $H$ is identifiable up to a left lower-triangular factor for $K \\ge d$, and it relaxes the setting to $\\alpha$-inexact SCMs measured by MIC, with the four-domain fit achieving MIC 0.04.","core_discovery":"The paper's central claim is that, within the four base models that share an invariant subspace, observed benchmark vectors $x$ are approximately $G z$ for a shared mixing matrix $G$ and three latent capability factors $z_1, z_2, z_3$, and that within each base model those factors follow a linear structural causal model with one common directed graph: $z_1$ influences $z_2$ and $z_2$ influences $z_3$. The recovered factors align tightly with benchmarks, $z_1$ to BBH and MMLU-Pro (general problem solving), $z_2$ to IFEval (instruction following), and $z_3$ to MATH Lvl 5 (advanced mathematical reasoning), with the $z_3$ alignment reaching $R^2 = 1.00$. The paper supports the causal reading with an intervention-style comparison: fine-tuning on IFEval raises both IFEval and MATH scores while leaving other benchmarks roughly unchanged, and this transfer is stronger for Qwen models than for Llama models, matching the recovered edge weights. The identifiability guarantee holds only up to ancestor mixtures and recovers the graph as its transitive closure, so the direct claim is the ancestor ordering rather than the presence of every individual edge.","pith_inferences":["Beyond the paper: applying HCA to newer or larger base-model families would test whether the recovered chain is a stable property of post-training or specific to this leaderboard cohort.","Beyond the paper: a clean mediation test would fine-tune only on instruction-following data stripped of mathematical content and compare MATH gains against a token-matched control, separating genuine mediation from generic fine-tuning effects.","Beyond the paper: if the hierarchy is a structural law, math gains from instruction-following should eventually saturate as the general capability node becomes the bottleneck, a predicted nonlinearity the current data do not estimate.","Beyond the paper: the recovered ancestor ordering suggests a curriculum for post-training, general capability first, then instruction-following, then math specialization, which the paper motivates but does not test."],"forward_implications":["The same post-training recipe can no longer be assumed to transfer across base models, because the recovered causal graphs have different edge weights for different base models.","Math-score gains should not be read purely as mathematical reasoning gains, since part of the effect is predicted to run through instruction-following; fine-grained and conditional scores are needed.","Scaling pretraining compute is the primary lever for the general capability node, whereas instruction-following and math are more responsive to fine-tuning, so budget allocation between pretraining and post-training can target whichever node is the bottleneck.","The HCA pipeline offers a no-retraining evaluation route: recover latent factors from existing leaderboard data, then test causal hypotheses with small targeted fine-tuning runs instead of full retraining sweeps."],"supporting_citations":[{"why":"Establishes the linear causal representation identifiability theory and the LiNGCReL algorithm that HCA adapts and makes robust to inexact SCMs.","marker":"[JS24]"},{"why":"Provides the observational scaling-law baseline in which benchmark performance is a linear transform of latent capability factors recovered by PCA, which Hypothesis 1 refines.","marker":"[RMH24]"},{"why":"Supplies the independent component analysis used in HCA's first step to obtain per-domain unmixing matrices.","marker":"[HHH+09]"},{"why":"Gives the non-Gaussian ICA uniqueness result that underwrites the decomposition $M_k = P_k B_k H$.","marker":"[EK04]"},{"why":"Provides the structural causal model and DAG framework within which the hierarchical capability hypothesis is formalized.","marker":"[Pea95]"},{"why":"Extends the observational scaling-law approach across model families, the multi-benchmark setting this paper makes causal.","marker":"[PSC+24]"},{"why":"Documents the Qwen2.5 base models whose fine-tuned derivatives form two of the four analyzed domains.","marker":"[YYZ+24]"},{"why":"Documents the Llama-3 and Llama-3.1 base models forming the other two domains.","marker":"[GDJ+24]"}],"fun_headline_variants":["LLM capabilities form causal chain: problem-solving to instruction to math","Causal analysis of 1500+ LLMs reveals ordered skill chain","General problem-solving feeds instruction-following, then math","Latent skills chain: problem-solving, instruction-following, then math"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the six benchmark scores of all models built from the four base models are, up to a small residual, a fixed linear mixture of the same three latent factors, and that inside each base model those factors follow one shared causal chain with nearly independent noise sources.","fun_headline_variants_meta":{"raw":{"variants":["LLM capabilities form causal chain: problem-solving to instruction to math","Causal analysis of 1500+ LLMs reveals ordered skill chain","General problem-solving feeds instruction-following, then math","Latent skills chain: problem-solving, instruction-following, then math"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000966,"raw_usage":{"total_tokens":4135,"prompt_tokens":997,"completion_tokens":3138,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":613,"completion_tokens_details":{"reasoning_tokens":3062}},"tokens_in":613,"tokens_out":3138,"duration_ms":24404,"temperature":1.0,"reasoning_tokens":3062,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:29:22.047317+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one of the four base models, fine-tune it only on instruction-following examples that contain no mathematical content, and compare its MATH Lvl 5 score against a control model fine-tuned on an equal number of non-instruction paragraphs; the claimed causal chain predicts a significantly larger math gain in the instruction group, so a null or reversed result would refute it.","supporting_citations":[],"review_version":1}