Pith. sign in

REVIEW 3 major objections 5 minor 16 references

The paper claims that the superficial alignment hypothesis can be made precise as a statement about program length: pre-trained models make many complex tasks solvable with programs of only thousands of bits.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 22:43 UTC pith:6XS25O6L

load-bearing objection A genuinely useful formalization and a solid set of Pareto curves, but the paper's evidence for the SAH rests on an unmeasured baseline and a confounded post-training comparison. the 3 major comments →

arxiv 2602.15829 v2 pith:6XS25O6L submitted 2026-02-17 cs.LG

Operationalising the Superficial Alignment Hypothesis via Task Complexity

Tom\'as Vergara-Browne , Darshan Patil , Ivan Titov , Siva Reddy , Tiago Pimentel , Marius Mosbach This is my paper
classification cs.LG
keywords superficial alignment hypothesistask complexityprogram lengthalgorithmic information theorypost-trainingLLM adaptationPareto frontierKolmogorov complexity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that the dispute over whether post-training merely 'surfaces' pre-trained knowledge can be replaced with a measurable question: how many bits of program are needed to reach a given score on a task. It defines task complexity C(Tδ) as the length of the shortest program that solves a task at performance δ, and conditional task complexity C(Tδ|θ) when the program may read a model's weights. The superficial alignment hypothesis then becomes the claim that many complex tasks have small C(Tδ|θ) for pre-trained θ. The paper estimates these quantities on three tasks and three open models, finding strong performance reachable with programs as small as a few thousand bits, and that post-training collapses the program length needed to reach the same performance by several orders of magnitude. A sympathetic reader would take this as a framework for making the SAH debate empirical rather than verbal.

Core claim

The paper's central claim is that the superficial alignment hypothesis is best stated as Definition 3.7: there are many complex tasks T—tasks with high unconditional complexity C(Tδ)—of practical interest for which a pre-trained model θ is (b,δ)-adaptable with a relatively small program size b. Operationally, b is an upper bound on C(Tδ|θ), the length of the shortest program that, given access to θ, reaches performance δ. Empirically, for GSM8K math reasoning, FLORES translation, and IFEval instruction following on SmolLM3 3B and Olmo3 7B/32B, the authors construct programs from three strategy families—data subset fine-tuning, low-rank adapters, and prompts—and plot length-performance Pareto

What carries the argument

The central object is task complexity, C(Tδ), defined as the minimum length in bits of a program that achieves score at least δ on a task, alongside its conditional version C(Tδ|θ), in which the program receives model weights θ as an extra input. From these definitions the paper derives (b,δ)-adaptability and states the SAH as the existence of many high-complexity tasks with small conditional complexity. The empirical work operationalizes the metric by constructing concrete programs from three adaptation strategies—data subset fine-tuning, parameter-efficient adapters, and prompt-based inference control—then measuring each program's length and achieved score and reading off the Pareto fronti

Load-bearing premise

The load-bearing premise is that the three tasks have high unconditional complexity C(Tδ), which the paper never measures; if GSM8K, FLORES, and IFEval are actually easy tasks with short unconditional programs, then short conditional programs show adaptability but not that pre-training contains the needed knowledge, and the post-training comparison is further strained by the acknowledged overlap between task data and post-training corpora.

What would settle it

Find or construct a program of comparable length—say, under a megabyte—that reaches the same accuracy on GSM8K, FLORES, or IFEval from scratch without access to any language-model weights; if such a program exists, the claimed reduction from C(Tδ) to C(Tδ|θ) is an artifact of the tasks being easy rather than of pretraining. A softer but decisive test: apply the same methods to a held-out task whose data provably never appears in pre- or post-training corpora; if kilobyte programs fail to transfer, the post-training collapse reflects memorization rather than surface alignment.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • The three classic arguments for the superficial alignment hypothesis—few data, few parameters, and prompt-only control—become different routes to the same object, a short program Pθ, and can be compared on a single bit-length scale.
  • If the estimates are representative, the marginal information needed to adapt an open pre-trained model to a practical task is tiny compared with its pre-training data, giving the phrase 'alignment is superficial' a concrete quantitative meaning.
  • The result that post-training collapses the complexity of reaching a fixed performance level gives an information-theoretic reading of post-training: it does not add capabilities so much as make access to them cheaper.
  • The framework converts critiques of the SAH into quantitative claims: 'performance does not saturate with few examples' becomes a statement about the performance level δ at which complexity rises sharply, and 'linear probing is insufficient' becomes a statement about one suboptimal program family.
  • A standard measurement suggests itself for future models: report a length-performance Pareto curve for benchmark tasks, making 'how adaptable is this model?' a concrete empirical quantity.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The strongest next test of the framework would be estimating or bounding the unconditional complexity C(Tδ) for these tasks; without that, the evidence is one-sided and the claim that short conditional programs reveal pre-existing knowledge remains an inference rather than a measurement.
  • A testable extension of the post-training result would use held-out tasks that provably never appear in post-training corpora, since the paper acknowledges GSM8K- and IFEval-derived data are present there and the reported collapse could be partially due to memorization.
  • The same length-performance measurement could be applied to multimodal, tool-use, or agentic tasks, where the program space is richer and the Pareto frontier may look very different from the three text tasks studied here.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes an algorithmic-information-theoretic operationalisation of the superficial alignment hypothesis (SAH). It defines task complexity C(Tδ) as the length of the shortest program achieving performance δ on task T, and conditional task complexity C(Tδ|θ) given a model's weights θ. The SAH is formalised (Definition 3.7) as the claim that many complex tasks T with high C(Tδ) are (b,δ)-adaptable to a pre-trained model θ with small b. The paper derives basic properties, connects the data, parametric, and inference-control views as different strategies for constructing short programs, and estimates upper bounds on C(Tδ|θ) for GSM8K, FLORES en-fr, and IFEval using SmolLM3 and OLMo3 models. It reports programs as small as ~4–5 kb achieving strong performance and a post-training collapse in the program length needed to reach a given performance. The central interpretation is that these results provide evidence for the SAH.

Significance. If correct, the paper would give the SAH debate a quantitative common language in bits, unifying previously orthogonal evidence and enabling direct comparison of adaptation methods. The formal definitions are carefully grounded in Kolmogorov complexity, the uncomputability of the quantities is acknowledged, the program-length measurement protocol is reproducible, and the code release is valuable. However, the empirical evidence for the SAH as defined depends on unconditional task complexity, which is never measured, and the post-training comparison is confounded by data overlap. The framework itself is a useful contribution even if the current experiments do not fully validate the SAH.

major comments (3)
  1. [Definition 3.7, §6.1, §9] Definition 3.7 restricts the SAH to tasks with high unconditional complexity C(Tδ). Section 9 explicitly states that C(Tδ) is not estimated and that the formalisation relies solely on conditional complexity. Consequently, the short conditional programs found in §6.1 (e.g., 4358 bits for OLMo3-32B on GSM8K) are upper bounds on C(Tδ|θ), but without a lower bound or estimate of C(Tδ) they cannot distinguish 'the model contains task knowledge' from 'the task is intrinsically easy' (e.g., a finite test set can be hard-coded). Appendix C only asserts that the tasks 'plausibly' have high complexity. This is the load-bearing gap: the abstract and §6.1 interpret the results as evidence for the SAH, but the defining condition of the hypothesis is untested. Please either provide a measurable proxy or lower bound for C(Tδ) for at least some tasks, or substantially weaken the SAH-supporting claims to
  2. [§7, footnote 9] The comparison between pre-trained and post-trained checkpoints is confounded because GSM8K- and IFEval-derived data appear in the post-training corpora of both model families (acknowledged in footnote 9). The post-training collapse in program length may therefore reflect memorisation of task data rather than a general reduction in adaptation complexity. This directly affects the 'post-training collapses complexity' claim. I recommend evaluating the same procedure on held-out tasks or model families whose post-training data does not overlap with the evaluation tasks, or at least quantifying the contribution of memorisation (e.g., by testing on rephrased versions of the same problems).
  3. [§5.1, §6, Appendix D] The Pareto curves are constructed by sweeping hyperparameters (learning rates, LoRA ranks, subset sizes, etc.) and selecting non-dominated points based on the same test-set evaluations used to report performance. This selection-on-the-test-set procedure yields upward-biased estimates of the score for the task distribution; the reported short programs are valid only for the finite test sample, not necessarily for the distribution p in Definition 3.1. Since the central quantitative claims (kilobyte-length programs) are based on these curves, the protocol should use a held-out validation split for hyperparameter selection and report test performance for the selected points, or otherwise account for the multiplicity.
minor comments (5)
  1. [Appendix C] Typo: 'set pf analysed tasks' should be 'set of analysed tasks'.
  2. [§3.3, Eq. (7)] The notation xδ in Equation (7) is used without definition; clarify that it denotes a distortion/performance parameter for the finite string x.
  3. [Figure 3] The reference sizes ('100 lines Python', 'ImageNet image', 'BERT-base') are not defined in the caption; specify the exact bit counts used for these reference lines.
  4. [Definition 3.7] The definition is labelled 'Informal'; consider formalising the notion of 'relatively small b' and 'relatively high δ' or explicitly stating it as a working postulate rather than a definition.
  5. [§7] The statement that 'program size has minimal effect on performance beyond a size of 10^4 bits' is stronger than Figure 5 shows; suggest softening to 'smaller effect'.

Circularity Check

1 steps flagged

Partial experimental circularity in the post-training complexity-collapse result; the core conditional-complexity metric is self-contained, but the SAH high-complexity premise is unmeasured.

specific steps
  1. fitted input called prediction [Section 7, footnote 9; Appendix E; abstract and Section 7 headline claim]
    "We note that the training data for both GSM8K and IFEval were included in the post-training of both SmolLM3 and Olmo3 (see Appendix E), which could partially explain this effect."

    The headline finding is that 'post-training collapses the complexity of reaching this same performance' (Section 7). But the post-trained checkpoints θ_post were constructed by training on data from the same tasks being measured: Appendix E states that SmolLM3 post-training data 'include[s] data of verifiable instruction following based on rules of IFEval, and also synthetic questions sourced from GSM8K training set,' and Olmo3 post-training similarly includes these tasks. Thus C(Tδ|θ_post) is computed with task-relevant information already placed inside the conditioning model. The short programs found after post-training are not an independent estimate of how pre-training stores task knowledge; they inherit the task data that was put into θ_post. The paper's own footnote concedes this ove

full rationale

The core metric is not circular: C(Tδ|θ) is upper-bounded by concrete, measured programs, and the Appendix A inequalities are standard Kolmogorov-complexity relations. No load-bearing self-citation chain is present; prior work by the same authors is used only as background or for auxiliary compression/injectivity claims. However, one part of the empirical story is partially circular: the post-training comparison in Section 7 uses checkpoints whose post-training corpora contain GSM8K- and IFEval-derived data, so the 'complexity collapse' is partly written into θ_post rather than independently discovered, as acknowledged in footnote 9 and Appendix E. Separately, Definition 3.7 requires high unconditional complexity C(Tδ), and Section 9 admits that C(Tδ) is not estimated, so the SAH interpretation rests on an unmeasured premise; this is a support gap, not a definitional equivalence. Overall score 4 reflects one partly circular comparison while the central conditional-complexity measurements remain independent evidence.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 1 invented entities

The central empirical interpretation depends on the unmeasured premise that the tasks are unconditionally complex, on an accounting convention that excludes library code from program length, and on standard assumptions about test-set approximation and arithmetic coding. The post-training comparison further depends on the overlap between post-training corpora and the evaluation tasks, which the paper acknowledges but does not control for.

free parameters (4)
  • Subset-training sample size and learning rate = n swept over powers of 2 from 8 to 32768 plus full dataset; lr ∈ {1e-4, 1e-5}
    Pareto points for the data view depend on these choices; no single fitted value is reported.
  • LoRA rank and learning rate = lr ∈ {1e-4, 1e-5}; rank swept (Appendix D)
    Parametric-view program lengths and performances are determined by these hyperparameters.
  • Bayesian-LoRA scale learning rate and rank initialization/pruning = scale lr ∈ {1e-2, 2e-3}; rank init 1 or 2 with pruning
    B-LoRA adapts ranks and quantization levels per layer, directly setting adapter weight file size.
  • URIAL hand-designed prompt and ICL example counts = URIAL uses n=2 with an undisclosed hand-designed prompt; ICL n swept to memory limit
    Inference-control programs' lengths and scores depend on these choices, but the prompt text is not included in the paper.
axioms (6)
  • ad hoc to paper The studied tasks (GSM8K, FLORES en-fr, IFEval) have high unconditional complexity C(Tδ) at the target performances.
    Needed for Definition 3.7's 'complex tasks' condition; never measured because C(Tδ) is uncomputable; §9 admits relying solely on conditional complexity. If false, low C(Tδ|θ) is not evidence of superficiality.
  • domain assumption All Python libraries are part of the universal machine and their code is not counted in program length.
    Section 5.1 and Appendix B. This keeps lengths small but is an unenforced convention; a library that implemented the whole solution would collapse the metric, though authors state they avoid this.
  • domain assumption Finite test-set averages approximate the task expectation over p in the score definition.
    Appendix C. Standard practice, but no confidence intervals around reported scores.
  • standard math Arithmetic coding with model θ achieves a compressed size equal to the model's negative log-likelihood of the data.
    Appendix B; standard arithmetic-coding property; used to estimate len(D'_c) and κ_c.
  • standard math Task complexity is uncomputable, so empirical program lengths are upper bounds on C(Tδ|θ).
    Corollary A.5 and §9; the paper correctly labels all measurements as upper bounds.
  • domain assumption Post-trained checkpoints have seen GSM8K- and IFEval-derived data during post-training.
    Appendix E and §7 footnote 9; acknowledged confound for the claim that post-training collapses complexity.
invented entities (1)
  • Task complexity C(Tδ|θ) and information I(Tδ;θ) independent evidence
    purpose: Formalise the superficial alignment hypothesis as a reduction in shortest-program length and compare adaptation strategies on one axis.
    These are formal metrics rather than physical entities. They have an empirical handle because upper bounds are measured by real programs; exact values are uncomputable.

pith-pipeline@v1.3.0-alltime-deepseek · 22085 in / 18029 out tokens · 163394 ms · 2026-08-02T22:43:56.776386+00:00 · methodology

0 comments
read the original abstract

The superficial alignment hypothesis (SAH) posits that large language models learn most of their knowledge during pre-training, and that post-training merely surfaces this knowledge. The SAH, however, lacks a precise definition, which has led to (i) different and seemingly orthogonal arguments supporting it, and (ii) important critiques to it. We propose a new metric called task complexity: the length of the shortest program that achieves a target performance on a task. In this framework, the SAH simply claims that pre-trained models drastically reduce the complexity of achieving high performance on many tasks. Our definition unifies prior arguments supporting the SAH, interpreting them as different strategies to find such short programs. Experimentally, we estimate the task complexity of mathematical reasoning, machine translation, and instruction following; we then show that these complexities can be remarkably low when conditioned on a pre-trained model. Further, we find that pre-training enables access to strong performances on our tasks, but it can require programs of gigabytes of length to access them. Post-training, on the other hand, collapses the complexity of reaching this same performance by several orders of magnitude. Overall, our results highlight that task adaptation often requires surprisingly little information -- often just a few kilobytes.

Figures

Figures reproduced from arXiv: 2602.15829 by Darshan Patil, Ivan Titov, Marius Mosbach, Siva Reddy, Tiago Pimentel, Tom\'as Vergara-Browne.

Figure 1
Figure 1. Figure 1: Pareto curve of program length vs. performance for Olmo3-7B on GSM8K. We argue that prior works about the su￾perficial alignment hypothesis can be seen as proposing different approaches to find short programs to solve a task, and we find that these different views inform different regions of this Pareto curve. 1. Introduction The superficial alignment hypothesis (SAH) states that a large language model’s “… view at source ↗
Figure 2
Figure 2. Figure 2: Pseudo-code of the programs P constructed by strategies in Section 4. Each program includes its compressed data or parameters, and its size is usually dominated by such terms. We explain the details of how we measure program size in our experiments in Appendix B. Together, the length (b) and performance (δ) of these pro￾grams P allow us to estimate the points on a Pareto-optimal length–performance (b, δ) c… view at source ↗
Figure 3
Figure 3. Figure 3: The program length vs. performance Pareto curves for SmolLM3 and Olmo3 in our three analysed tasks: mathematical reasoning, machine translation, and instruction following. We test each method described in Section 4, but show only the optimal lengths and performances. To provide some intuition about these program lengths, we also mark the size in bits of more commonly known references. These models provide … view at source ↗
Figure 4
Figure 4. Figure 4: Performance of the linear projection fine-tuning proposed by Chen et al. (2025) in comparison to our estimated Pareto curve. The performance of the method is far from the Pareto curve. this update. Liu et al. (2024), for example, compress the difference between a pre-trained and fine-tuned model to ap￾proximately 1 bit per parameter, with minimal performance impacts on their analysed tasks. They interpret … view at source ↗
Figure 5
Figure 5. Figure 5: Estimated complexity curves conditioned on pre- and post-trained checkpoints of SmolLM3 and Olmo3 for math (GSM8K) and instruction (IFEval) tasks. Pre-training allows access of strong performance in these tasks. Post-training improves these performances, but more importantly, it greatly reduces the complexity to reach such performance. 8. Conclusion The superficial alignment hypothesis has lacked a precise… view at source ↗
Figure 6
Figure 6. Figure 6: Length–performance Pareto curve of SmolLM3 3B, Olmo3 7B, and Olmo3 32B in Math (GSM8K), Translation (FLORES) and Instruction (IFEval). Different views of superficiality inform different regions of the Pareto curve, with some variance depending on the choice of model and task. 10 3 10 5 10 7 10 9 10 11 20 40 60 Math Accuracy (%) SmolLM3 10 3 10 5 10 7 10 9 10 11 20 40 60 Olmo3 7B 10 3 10 5 10 7 10 9 10 11 2… view at source ↗
Figure 7
Figure 7. Figure 7: Performance of the linear (head-only) fine-tuning approach proposed by Chen et al. (2025) and our estimated Pareto curves in Math (GSM8K), Translation (FLORES) and Instruction (IFEval). 17 [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Estimated complexity curves conditioned on pre- and post-trained checkpoints of SmolLM3 and Olmo3 for FLORES. 10 3 10 5 10 7 10 9 10 11 20 40 60 80 GSM8K Accuracy (%) (Math) SmolLM3 10 3 10 5 10 7 10 9 10 11 20 40 60 80 Olmo3 7B 10 3 10 5 10 7 10 9 10 11 20 40 60 80 Olmo3 32B 10 3 10 5 10 7 10 9 10 11 10 20 30 40 FLORES BLEU (Translation) 10 3 10 5 10 7 10 9 10 11 10 20 30 40 10 3 10 5 10 7 10 9 10 11 10 2… view at source ↗
Figure 9
Figure 9. Figure 9: The program lengths vs. performance Pareto curves for SmolLM3 and Olmo3 in GSM8K and FLORES. We test each method described in Section 4, and show their message lengths and performances obtained. This is an extended version of [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: shows results using this method with SmolLM3 and Olmo3 7B. After extensive experimentation, we concluded that it is not straightforward to push the Pareto frontier with this strategy. It remains unclear whether this indicates the tightness of our Pareto frontier, or just the lack of success of this specific approach. We release these results to encourage future research to target this question. 10 3 10 5 … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 10 linked inside Pith

  1. [4]

    Gr¨unwald, P

    URL https://openreview.net/forum? id=tcsZt9ZNKD. Gr¨unwald, P. D.The Minimum Description Length Principle. MIT press, 2007. Han, Z., Gao, C., Liu, J., Zhang, J., and Zhang, S. Q. Parameter-efficient fine-tuning for large models: A com- prehensive survey. Transactions on Machine Learning Research, 2024. URL https://openreview.net/ forum?id=lIsCS8b6zj. Hewi...

    Pith/arXiv arXiv 2007

  2. [8]

    Morris, J

    URL https://openreview.net/forum? id=Igm9bbkzHC. Morris, J. X., Sitawarin, C., Guo, C., Kokhlikyan, N., Suh, G. E., Rush, A. M., Chaudhuri, K., and Mahlou- jifar, S. How much do language models memorize? arXiv preprint arXiv:2505.24832, 2025. URL https: //arxiv.org/abs/2505.24832. Morris, J. X., Mireshghallah, N., Ibrahim, M., and Mahlou- jifar, S. Learni...

    Pith/arXiv arXiv 2025

  3. [11]

    11 Operationalising the Superficial Alignment Hypothesis via Task Complexity Pezeshkpour, P

    URL https://openreview.net/forum? id=EjInprGpk9. 11 Operationalising the Superficial Alignment Hypothesis via Task Complexity Pezeshkpour, P. Measuring and modifying factual knowl- edge in large language models. In 2023 interna- tional conference on machine learning and applications (ICMLA), pp. 831–838. IEEE, 2023. URL https:// ieeexplore.ieee.org/docume...

    arXiv 2023

  4. [13]

    Qi, X., Zeng, Y ., Xie, T., Chen, P.-Y ., Jia, R., Mittal, P., and Henderson, P

    URL https://openreview.net/forum? id=cr5EQRJlRn. Qi, X., Zeng, Y ., Xie, T., Chen, P.-Y ., Jia, R., Mittal, P., and Henderson, P. Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview. net/forum?id=hTEGyKf0dZ. Raghavendra, ...

    Pith/arXiv arXiv 2024

  5. [168]

    naacl-long.168/

    URL https://aclanthology.org/2025. naacl-long.168/. 9 Operationalising the Superficial Alignment Hypothesis via Task Complexity Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training ver- ifiers to solve math word problems. arXiv preprint arXiv:2110.1416...

    Pith/arXiv arXiv 2025

  6. [254]

    emnlp-main.254/

    URL https://aclanthology.org/2020. emnlp-main.254/. Pimentel, T., Valvoda, J., Maudslay, R. H., Zmigrod, R., Williams, A., and Cotterell, R. Information- theoretic probing for linguistic structure. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. (eds.), Pro- ceedings of the 58th Annual Meeting of the Associa- tion for Computational Linguistics,...

    doi:10.18653/v1/2020.acl-main.420 2020

  7. [346]

    naacl-long.346/

    URL https://aclanthology.org/2025. naacl-long.346/. Lambert, N. Reinforcement learning from human feedback. arXiv preprint arXiv:2504.12501, 2025. URL https: //arxiv.org/abs/2504.12501. Lambert, N., Morrison, J., Pyatkin, V ., Huang, S., Ivison, H., Brahman, F., Miranda, L. J. V ., Liu, A., Dziri, N., Lyu, X., Gu, Y ., Malik, S., Graf, V ., Hwang, J. D., ...

    Pith/arXiv arXiv 2025

  8. [353]

    acl-long.353/

    URL https://aclanthology.org/2021. acl-long.353/. Lin, B. Y ., Ravichander, A., Lu, X., Dziri, N., Sclar, M., Chandu, K., Bhagavatula, C., and Choi, Y . The unlock- ing spell on base LLMs: Rethinking alignment via in- context learning. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: //openreview.net/forum?id=wxJ0eXw...

    2021

  9. [1025]

    emnlp-main.1025/

    URL https://aclanthology.org/2025. emnlp-main.1025/. Nikolaou, G., Mencattini, T., Crisostomi, D., Santilli, A., Panagakis, Y ., and Rodol`a, E. Language models are injective and hence invertible. In The Fourteenth In- ternational Conference on Learning Representations ,

    2025

  10. [1948]

    Solomonoff, R

    URL https://ieeexplore.ieee.org/ document/6773024. Solomonoff, R. J. A formal theory of inductive inference. part i. Information and control, 7(1):1–22, 1964. Sutter, D., Minder, J., Hofmann, T., and Pimentel, T. The non-linear representation dilemma: Is causal abstraction enough for mechanistic interpretability? In The Thirty- ninth Annual Conference on ...

    arXiv 1964

  11. [2020]

    doi: 10.18653/v1/2020.emnlp-main.14

    Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.14. URL https:// aclanthology.org/2020.emnlp-main.14/. Wallace, E., Watkins, O., Wang, M., Chen, K., and Koch, C. Estimating worst-case frontier risks of open-weight LLMs. arXiv preprint arXiv:2508.03153, 2025. URL https://arxiv.org/abs/2508.03153. Yadav, P., Choshen, L., Raffel, ...

    Pith/arXiv arXiv 2020

  12. [2022]

    Huang, C., Ye, P., Zheng, S., Wang, X., Bai, L., Chen, T., and Ouyang, W

    URL https://openreview.net/forum? id=nZeVKeeFYf9. Huang, C., Ye, P., Zheng, S., Wang, X., Bai, L., Chen, T., and Ouyang, W. Dynamic base model shift for delta compression. arXiv preprint arXiv:2505.11344 , 2025. URL https://arxiv.org/abs/2505.11344. Karan, A. and Du, Y . Reasoning with sampling: Your base model is smarter than you think. arXiv preprint ar...

    Pith/arXiv arXiv 2025

  13. [2024]

    Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L

    URL https://openreview.net/forum? id=jznbgiynus. Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. QLoRA: Efficient finetuning of quantized LLMs. In Thirty-seventh Conference on Neural Information Pro- cessing Systems, 2023. URL https://openreview. net/forum?id=OUIFPHEgJU. Gao, L., la Tour, T. D., Tillman, H., Goh, G., Troll, R., Radford, A., S...

    2023

  14. [2025]

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J

    URL https://openreview.net/forum? id=jf472MdwsP. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. URL https: //papers.nips.cc/paper/2020/hash/ 1457c0d6bfcb4967418b...

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.05209 1901

  15. [2026]

    URL https://openreview.net/forum? id=0kHbD6ad07. Olmo, T., :, Ettinger, A., Bertsch, A., Kuehl, B., Graham, D., Heineman, D., Groeneveld, D., Brahman, F., Timbers, F., Ivison, H., Morrison, J., Poznanski, J., Lo, K., Soldaini, L., Jordan, M., Chen, M., Noukhovitch, M., Lambert, N., Walsh, P., Dasigi, P., Berry, R., Malik, S., Shah, S., Geng, S., Arora, S....

    Pith/arXiv arXiv 2025

  16. [8856]

    Ye, Y ., Huang, Z., Xiao, Y ., Chern, E., Xia, S., and Liu, P

    URL https://openreview.net/forum? id=CovLQwu611. Ye, Y ., Huang, Z., Xiao, Y ., Chern, E., Xia, S., and Liu, P. LIMO: Less is more for reasoning. In Conference on Language Modeling (COLM) , 2025. URL https: //openreview.net/forum?id=T2TZ0RY4Zk. Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y ., Kwok, J. T., Li, Z., Weller, A., and Liu, W. Meta- Math...

    Pith/arXiv arXiv 2025