Pith. sign in

REVIEW 5 major objections 4 minor 1 cited by

Beyond Surface Structure: A Causal Assessment of LLMs' Comprehension Ability

T0 review · 5 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that most evaluated LLMs genuinely depend on core semantics when answering, and proposes a causal metric, ADCE, that separates deep-structure comprehension from surface-structure sensitivity.

desk verdict A useful relative measure of deep vs surface sensitivity, but the leap from positive ADCE to 'deep structure comprehension' is not yet secured. read the letter →

arxiv 2411.19456 v1 pith:G4FMCDUQ submitted 2024-11-29 cs.CL cs.AI

Yujin Han , Lei Xu , Sirui Chen , Difan Zou , Chaochao Lu This is my paper · ORCID
classification cs.CLcs.AI
keywords largelanguagemodelsdeepstructurecomprehensionsurfacecausalmediationanalysisdirecteffectspuriouscorrelationLLMevaluationprobabilityofcausation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the debate over whether LLMs truly understand language has been one-sided. Earlier work showed that models are sensitive to surface phrasing and concluded that their success relies on surface structure; the paper counters that surface sensitivity does not rule out genuine dependence on core semantics. It casts deep-structure comprehension as the direct causal effect of deep structure on outputs and surface-structure comprehension as an indirect causal effect, then proposes estimable proxies, ADCE and AICE. Applied to twelve LLMs on five benchmarks, ADCE is positive for most models and grows with accuracy, which the authors read as evidence that deep-structure comprehension is real and widespread. The metric is also shown to diagnose reliance on spurious surface correlations better than accuracy does.

What carries the argument

The central object is the approximated direct causal effect (ADCE), defined as the difference between the total effect of an intervention that changes both deep and surface structure and an approximated indirect causal effect (AICE) that changes only surface structure, with output changes measured by indicator functions. It is estimated in three steps: infer correct answers on the full dataset, generate paired interventions by masking core semantic tokens or by minimally rephrasing with an LLM agent equipped with a self-check, and re-run the model to compare outputs. The causal graph treats deep structure d and surface structure s as parallel mediators between the input x and the answer Y, so the path x→d→Y is the direct effect and x→s→Y is the indirect effect. Theorem 1 decomposes ADCE into weighted probabilities of sufficiency and necessity, which is the mechanism by which the paper claims ADCE is a bidirectional evaluation rather than a one-directional accuracy score.

What would settle it

Take a pool of correctly answered questions and build two versions of the surface-only intervention: one that keeps phrasing as close as possible to the changed-semantics version, as the paper does, and one deliberately rephrased into very different wording. If ADCE changes substantially between the two conditions, then AICE is contaminated by surface-state differences and does not isolate the deep-structure effect; the central claim would then hold only for the paper's specific intervention templates, not for deep-structure comprehension in general.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that most evaluated LLMs have a positive approximated direct causal effect (ADCE): intervening on the core semantics of a previously solved problem makes the model deviate from the correct answer, whereas intervening only on presentation format leaves answers mostly intact. Closed-source models show stronger dependence on deep structure than on surface structure, while open-source models show the opposite pattern, but their surface sensitivity decreases as the model gets larger. Theoretically, ADCE is shown to be a weighted combination of the probability of sufficiency and the probability of necessity of deep-structure changes for output changes, so it measures whether deep structure is both sufficient and necessary for the model's answer rather than merely whether the answer is correct. A randomly initialized baseline yields zero ADCE, supporting the interpretation that the effect is tied to learned comprehension rather than to the intervention procedure itself.

Load-bearing premise

The measurement only works if the hand-built interventions really separate meaning from wording: changing a number or an answer-changing phrase must alter core semantics, and the surface-only version must leave meaning untouched while looking almost identical in form; the paper argues for this but validates the approximation on synthetic data rather than on the actual language interventions.

Editorial extensions

If this is right

  • If ADCE is accepted, high accuracy alone should no longer be treated as evidence of comprehension; a model with high accuracy but near-zero ADCE is likely solving via surface cues.
  • Benchmark design can use ADCE as a secondary score: tasks where models show high accuracy but low ADCE are tasks that permit surface-level shortcuts.
  • The scale trend implies that larger open-source models become more semantically grounded, and this is measurable even when their accuracy is already high.
  • Because ADCE is computed from a model's outputs without needing weights or training data, it can be applied to closed-source APIs as well as open models.
  • The fine-tuning result suggests that activating task-relevant knowledge, not just adding parameters, is what produces deep-structure dependence, and ADCE can detect this activation where accuracy may not.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I infer that the approximation error of AICE is unmeasured for natural-language interventions, since the paper validates it only on synthetic data; a natural next step would be to generate several independent rephrasings per sample and check whether ADCE stays stable.
  • I infer that the accuracy–ADCE gap could be repurposed as a dataset diagnostic: a benchmark whose best models have high accuracy but low ADCE is likely solvable through surface patterns, which would predict poor transfer to novel distributions.
  • I infer that the closed-source versus open-source difference is likely entangled with model scale and post-training, so a fair test would compare models matched on accuracy or parameter count.
  • I infer that the sufficiency–necessity interpretation could be tested directly by varying the strength of deep-structure change and asking whether output change follows the predicted probability of necessity.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes a causal-mediation framework for measuring whether LLMs rely on 'deep structure' (core semantics) versus 'surface structure' (linguistic form). It defines deep-structure comprehension through a formal invariance criterion (Eq. 1), reformulates it as a direct causal effect (DCE), and introduces estimable surrogates ADCE and AICE that replace the unobservable oracle indirect effect with observable intervention arms. The paper then applies ADCE and AICE to 12 LLMs on five benchmarks, reporting that most models have positive ADCE, that ADCE correlates with accuracy, that closed-source models rely more on deep structure while open-source models are more surface-sensitive, and that ADCE behaves differently from accuracy under spurious correlations. The main theoretical result (Theorem 1) expresses ADCE as a weighted combination of probabilities of sufficiency and necessity under a monotonicity assumption. The algebraic decomposition and the reported experiments are internally consistent, but the central interpretation—that positive ADCE establishes deep-structure comprehension—rests on unverified assumptions about the intervention arms and on a stronger formal definition than the experiments actually test.

Significance. If the proposed metrics were validated, they would provide a useful complement to accuracy for evaluating whether LLM outputs depend on core semantics, and the breadth of the evaluation across model families and tasks is a genuine strength. The paper also includes a synthetic-data check, a spurious-correlation case study, and noise-robustness experiments, which are commendable attempts to support the method. However, the gap between the formal definition of comprehension and the operationalized ADCE statistic is substantial, and the approximation that AICE stands in for the oracle indirect effect is not validated on real language interventions. The contribution is therefore potentially valuable as a sensitivity-analysis tool, but the current manuscript overstates what positive ADCE proves about 'comprehension' in the sense of Eq. (1).

major comments (5)
  1. [§2, Eq. (1) and §4.2, Fig. 4] Equation (1) defines deep-structure comprehension as the requirement that surface-only interventions leave the output unchanged and deep interventions always change it. However, Figure 4 shows that all tested models suffer substantial accuracy drops under surface-only interventions (for example, Llama-3-8b drops from the 100% baseline of correctly answered samples to roughly 50% accuracy on several strategies). This means no tested model satisfies Eq. (1), so positive ADCE cannot be read as evidence for the comprehension criterion stated in Eq. (1). At most, ADCE measures a relative sensitivity between two intervention arms, and the sign of ADCE can be driven by differences in intervention strength between the TE and AICE arms rather than by dependence on deep structure.
  2. [§3.3, Algorithm 2 and Table 1] The AICE control arm is not shown to match the surface component of the TE arm in difficulty, grammaticality, or type of perturbation. In the Mask strategy, the TE sample 'What is <mask> times 20?' masks a numeral, while the AICE sample 'What <mask> 50 times 20?' masks a function word; these differ not only in deep structure but also in which token is masked and in the grammaticality of the resulting question. For the Rephrase strategy, Algorithm 3's self-check verifies only that the rephrased question changes or preserves the answer according to Claude-3.5-Sonnet; it does not verify surface-form equivalence between s(T=1) and s(T=0). As a result, ADCE = TE − AICE can be positive even for a model whose outputs depend only on surface form whenever the TE surface change is more disruptive than the AICE surface change. No sensitivity analysis over the free parameter k in Algorithm 2 is provided.
  3. [Appendix E, Eqs. (15)-(16), Fig. 11] The synthetic validation uses a linear scalar SCM with known d and s and a logistic outcome model, and it checks the algebraic estimator on that SCM. This does not validate the key approximation that replaces the unobservable ICE with the observable AICE in Eq. (5): the paper provides no formal error bound and no sensitivity analysis for the substitution of s(T=1) by s(T=0), and the synthetic model does not instantiate the actual Mask or Rephrase linguistic interventions used in the experiments. The claim that AICE approximates the oracle ICE is therefore unverified for the real LLM setting.
  4. [Theorem 1 and §4.3, Fig. 5 and Table 8] Theorem 1's interpretation of ADCE as a weighted combination of probabilities of sufficiency and necessity relies on the monotonicity of the indicator outcome with respect to T. Under that assumption, ADCE is a nonnegative combination of nonnegative quantities and hence cannot be negative. Yet Section 4.3 reports negative ADCE for Llama-3-8b on Analytic Entailment (e.g., −0.071 for the pre-trained model in Table 8). These negative values are direct evidence that the monotonicity assumption fails for those model-task cells, so the Theorem's PS/PN interpretation does not apply to the full evaluation. The paper labels the negative values an 'anomaly' but does not qualify the theoretical claim or test monotonicity.
  5. [§4.2, footnote 3 and Fig. 5] The random-weight baseline has zero accuracy, so there are no correctly answered samples on which to perform the interventions defined in Algorithm 2; its ADCE is zero by construction rather than by the estimation procedure. Consequently, comparing other models' ADCE to this baseline does not validate the estimator's ability to detect zero causal effect. The linear regression in Fig. 5 also lacks confidence intervals and per-point uncertainty, which is especially important because the number of models per task is small and ADCE values for individual models are estimated from limited intervention samples.
minor comments (4)
  1. [Throughout] There are numerous typos and inconsistent capitalizations, including 'esitimate' in §3.2, 'suck as' in §3.3, 'Entaiment' in Table 3, 'empoly' in Appendix B.1, 'intrested' in Figure 10, and mixed 'Llama'/'LLama' spellings; a careful proofread is needed.
  2. [Appendix C.2, Theorem 2] Theorem 2 states the result for δDCE, but the proof and the main-text Theorem 1 concern δADCE; the appendix should be aligned with the main theorem to avoid confusion.
  3. [Fig. 4 and Fig. 5] The figures show only point estimates and no error bars or confidence intervals, despite the fact that ADCE and accuracy are computed from small, repeatedly intervened samples (2 or 4 intervention samples per original correct sample); reporting variance would strengthen the empirical claims.
  4. [§4.5, Fig. 8] The spurious-correlation experiment reports accuracy and ADCE for majority and minority groups, but it does not provide error bars or the number of evaluation samples per group; adding these would make the comparison between accuracy and ADCE more convincing.

Circularity Check

1 steps flagged · score 6.0 of 10

ADCE on Rephrase tasks is partly forced for Claude-3.5-Sonnet because the same model both generates the interventions under answer-change/preservation self-checks and is then evaluated on them.

  1. fitted input called prediction [Section 3.3 (Rephrase data generation); Algorithm 3 in Appendix F.1; Section 4.1 model list; Section 4.2 interpretation]
    "For rephrasing, we use Claude-3.5-Sonnet [31] and design a self-checking mechanism. Claude re-answers rephrased questions to verify deep structure alteration and preservation. ... if (T = “Alter” and y′ ̸= y) or (T = “Preserve” and y′ = y) then selfCheckFlag ← True ... We test 12 leading models ... Claude (Claude-3-Sonnet, Claude-3.5-Sonnet). Positive ADCE values suggest that intervening deep structure causes LLMs to deviate from correct answers on previously solved problems, highlighting the models’ reliance on deep structure for accurate problem-solving."

    On the two Rephrase-based tasks (Analytic Entailment and CommonsenseQA), Algorithm 3 constructs the T=1 input only when the generator's own predicted answer differs from the original label, and the T=0 input only when its predicted answer is preserved. Claude-3.5-Sonnet is both the generator and one of the evaluated models. Its TE is therefore measured on questions selected to change that model's answer, and its AICE on questions selected to keep that model's answer fixed; the positive ADCE for this model on these tasks is enforced by the selection rule rather than measured independently.

full rationale

The algebraic core of the paper—ADCE as a difference of observable answer-change indicators (Eq. 5) and its decomposition into weighted PS/PN terms (Theorem 1)—is a genuine derivation from definitions and does not reduce to a fitted parameter. The Mask-based evaluations also provide independent, non-circular evidence: masking numbers or operators versus function words is a rule-based intervention not fitted to any evaluated model. The circularity is confined to the Rephrase pipeline, where the intervention data are generated by Claude-3.5-Sonnet under self-check constraints on answer change/preservation, and Claude-3.5-Sonnet is itself among the evaluated models. Because the headline positive-ADCE claim aggregates over several tasks and models, it retains independent content; however, the two Rephrase tasks' results for the generating model, and the comparative closed-source conclusion that uses those results, are partially forced by construction. Other concerns raised by the manuscript, such as the unverified equivalence of surface-only interventions in AICE and the fact that Figure 4 shows large accuracy drops under surface-only interventions (which violates the formal Eq. 1 criterion), are construct-validity and correctness risks rather than circularity, and are not scored as circular steps here.

Assumptions & free parameters 3 free parameters · 5 assumptions · 1 invented entities

The central claim depends on several hand-chosen intervention design elements (word pools, k, rephrasing agent) and on untestable assumptions about latent structure and monotonicity. The paper provides no formal error bounds for the key approximation and no uncertainty quantification for the empirical comparisons.

free parameters (3)
  • k, number of nearest non-core tokens masked in AICE = 1
    Hand-chosen in Algorithm 2; determines the surface-similarity between TE and AICE and hence the magnitude of ADCE; no sensitivity analysis is reported.
  • Intervention word pools (Mask tasks) = Task-specific lists, e.g., digits and operator for 2-digit multiplication
    These lists define what counts as deep structure; ADCE values and all conclusions depend on this hand-selected partition.
  • Rephrase agent and self-check tolerance = Claude-3.5-Sonnet, up to 10 iterations
    The generated paraphrases for T=1 and T=0 depend on the chosen LLM and prompts; this is a design choice affecting the validity of the approximation.
assumptions (5)
  • domain assumption Input x decomposes into independent latent components d and s, both mediators to output Y (x to d to Y, x to s to Y).
    Section 2 and Figure 3; the entire causal graph and estimand definitions rest on this decomposition, but d and s are unobservable and non-manipulable.
  • ad hoc to paper T=0 interventions alter s while preserving d; T=1 interventions alter both d and s.
    Equation 2 and Section 3.3; the validity of ADCE as a deep-structure effect requires that the masking and rephrasing procedures realize exactly this separation.
  • ad hoc to paper AICE in Equation 5 approximates the oracle ICE in Equation 3.
    Section 3.2; the approximation is asserted to hold by maximizing surface similarity, validated only on the simplified synthetic SCM in Appendix E.
  • domain assumption Sequential ignorability and consistency hold (no unmeasured confounders).
    Appendix B.1; asserted because Figure 3 contains all relevant variables, which cannot be checked for latent d and s.
  • domain assumption Yhat is monotonic with respect to T in Theorem 1.
    Appendix C.1 and C.2; the PS/PN decomposition requires monotonicity, which is not verified on real data.
invented entities (1)
  • Parallel latent mediators d (deep structure) and s (surface structure)
    purpose: To define DCE and ICE and the ADCE and AICE evaluation metrics
    The paper labels these unobservable constructs and operationalizes them through hand-designed word-level interventions, so the only evidence for their causal role is internal to the metric; no external falsifiable handle is provided outside the paper's own intervention protocol.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Surface Structure: A Causal Assessment of LLMs' Comprehension Ability." pith.science (2026). https://pith.science/paper/G4FMCDUQ

@misc{pith2026241119456,
  author       = {Pith},
  title        = {Pith review of: Beyond Surface Structure: A Causal Assessment of LLMs' Comprehension Ability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G4FMCDUQ}},
  note         = {Machine review of arXiv:2411.19456}
}
read the original abstract

Large language models (LLMs) have shown remarkable capability in natural language tasks, yet debate persists on whether they truly comprehend deep structure (i.e., core semantics) or merely rely on surface structure (e.g., presentation format). Prior studies observe that LLMs' performance declines when intervening on surface structure, arguing their success relies on surface structure recognition. However, surface structure sensitivity does not prevent deep structure comprehension. Rigorously evaluating LLMs' capability requires analyzing both, yet deep structure is often overlooked. To this end, we assess LLMs' comprehension ability using causal mediation analysis, aiming to fully discover the capability of using both deep and surface structures. Specifically, we formulate the comprehension of deep structure as direct causal effect (DCE) and that of surface structure as indirect causal effect (ICE), respectively. To address the non-estimability of original DCE and ICE -- stemming from the infeasibility of isolating mutual influences of deep and surface structures, we develop the corresponding quantifiable surrogates, including approximated DCE (ADCE) and approximated ICE (AICE). We further apply the ADCE to evaluate a series of mainstream LLMs, showing that most of them exhibit deep structure comprehension ability, which grows along with the prediction accuracy. Comparing ADCE and AICE demonstrates closed-source LLMs rely more on deep structure, while open-source LLMs are more surface-sensitive, which decreases with model scale. Theoretically, ADCE is a bidirectional evaluation, which measures both the sufficiency and necessity of deep structure changes in causing output variations, thus offering a more comprehensive assessment than accuracy, a common evaluation in LLMs. Our work provides new insights into LLMs' deep structure comprehension and offers novel methods for LLMs evaluation.

Figures

Figures reproduced from arXiv: 2411.19456 by the authors.

Figure 1
Figure 1. Surface structure interventions cause subtle accuracy degradation rela￾tive to the obvious accuracy decline from deep structure changes. While sensitivity to surface-level interventions shows a lack of robustness to superficial changes, it does not necessarily preclude an understanding of deep structure. To ascertain whether LLMs are merely surface structure learners, a com￾parative analysis of their understanding o… view at source ↗
Figure 2
Figure 2. Approximated DCE (ADCE) quantifies LLMs’ deep structure comprehension, while [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. For the four intervention strategies, LLM accuracy drops from 100% when surface structures are altered while deep structures re￾main unchanged in initially correct samples. We propose that an LLM, fθ, acting as a “deep thinker”, should not only provide correct answers but also fundamentally depend on deep structure for responses. Formally, let Dc ⊆ D be a subset of questions correctly answered by fθ. An LLM fθ posse… view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: Deep structure understanding in LLMs via ADCE. Positive ADCE demonstrate the existence [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: ADCE pre- and post- SFT. SFT activates entailment knowledge, en￾abling the model to exhibit deep struc￾ture causal effects on outcomes, as cap￾tured by proposed ADCE. To investigate LLMs’ failure, we explore the potential pre￾requisites for deep structure comprehension…
Figure 7
Figure 7. Figure 7: Comparing deep vs. surface structure. δADCE represents ADCE of deep structure on output, while δAICE denotes AICE of surface structure on output. Closed-source models exhibit a greater reliance on deep structure for outputs. Open-source models (e.g. LLama-2) are more s…
Figure 8
Figure 8. Figure 8: Spurious correlation results in LLama-3. In majority groups with spurious correlations, [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Typical mediation analysis graph with treatment (A), mediator (M) and outcome (Y ). x d s y 1 1 c2 c1 ϵs ϵd ϵy [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 11
Figure 11. Figure 11: Comparison of True Causal Effects (True CE of [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 13
Figure 13. Figure 13: SFT on LLama-3-70b [PITH_FULL_IMAGE:figures/full_fig_p027_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Other Mind: How Language Models Exhibit Human Temporal Cognition

    cs.AI 2025-07 conditional novelty 6.0 of 10

    Larger LLMs develop a subjective 'present' around the current date, and their year similarity judgments follow a logarithmic Weber-Fechner compression, with supporting neural and representational evidence.

Reference graph

Works this paper leans on

76 extracted references · 38 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  3. [3]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    arXiv 2023

  4. [4]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1– 113, 2023

    2023

  5. [5]

    Palm 2 technical report

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023

    arXiv 2023

  6. [6]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    arXiv 2023

  7. [7]

    Deep structure, surface structure, and semantic interpretation

    Noam Chomsky, Danny Steinberg, and Leon Jakobovits. Deep structure, surface structure, and semantic interpretation. 1971, pages 183–216, 1971

    work page 1971

  8. [8]

    Evaluating large language models at evaluating instruction following

    Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following. arXiv preprint arXiv:2310.07641, 2023

    arXiv 2023

Show all 76 references
  1. [9]

    Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization

    Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, et al. Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization. arXiv preprint arXiv:2306.05087, 2023

    2023 arXiv

  2. [10]

    Chateval: Towards better llm-based evaluators through multi-agent debate

    Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201, 2023

    2023 arXiv

  3. [11]

    Learning beyond pattern matching? assaying mathematical understanding in llms

    Siyuan Guo, Aniket Didolkar, Nan Rosemary Ke, Anirudh Goyal, Ferenc Huszár, and Bernhard Schölkopf. Learning beyond pattern matching? assaying mathematical understanding in llms. arXiv preprint arXiv:2405.15485, 2024

    2024 arXiv

  4. [12]

    A peek into token bias: Large language models are not yet genuine reasoners

    Bowen Jiang, Yangxinyu Xie, Zhuoqun Hao, Xiaomeng Wang, Tanwi Mallick, Weijie J Su, Camillo J Taylor, and Dan Roth. A peek into token bias: Large language models are not yet genuine reasoners. arXiv preprint arXiv:2406.11050, 2024

    2024 arXiv

  5. [13]

    A causal framework to quantify the robustness of mathematical reasoning with language models

    Alessandro Stolfo, Zhijing Jin, Kumar Shridhar, Bernhard Schölkopf, and Mrinmaya Sachan. A causal framework to quantify the robustness of mathematical reasoning with language models. arXiv preprint arXiv:2210.12023, 2022

    2022 arXiv

  6. [14]

    Do large code models understand programming concepts? a black-box approach

    Ashish Hooda, Mihai Christodorescu, Miltos Allamanis, Aaron Wilson, Kassem Fawaz, and Somesh Jha. Do large code models understand programming concepts? a black-box approach. arXiv preprint arXiv:2402.05980, 2024

    2024 arXiv

  7. [15]

    Does reasoning emerge? examining the probabilities of causation in large language models

    Javier González and Aditya V Nori. Does reasoning emerge? examining the probabilities of causation in large language models. arXiv preprint arXiv:2408.08210, 2024. 11

    2024 arXiv

  8. [16]

    Emergent representations of program semantics in language models trained on programs

    Charles Jin and Martin Rinard. Emergent representations of program semantics in language models trained on programs. In Forty-first International Conference on Machine Learning

  9. [17]

    A general approach to causal mediation analysis

    Kosuke Imai, Luke Keele, and Dustin Tingley. A general approach to causal mediation analysis. Psychological methods, 15(4):309, 2010

    2010

  10. [18]

    Identification, inference and sensitivity analysis for causal mediation effects

    Kosuke Imai, Luke Keele, and Teppei Yamamoto. Identification, inference and sensitivity analysis for causal mediation effects. 2010

    2010

  11. [19]

    Causal mediation analysis

    Raymond Hicks and Dustin Tingley. Causal mediation analysis. The Stata Journal, 11(4):605– 619, 2011

    2011

  12. [20]

    Direct and indirect effects

    Judea Pearl. Direct and indirect effects. In Probabilistic and causal inference: the works of Judea Pearl, pages 373–392. 2001

    2001

  13. [21]

    Causal inference using potential outcomes: Design, modeling, decisions

    Donald B Rubin. Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 100(469):322–331, 2005

    2005

  14. [22]

    Estimating causal effects of treatments in randomized and nonrandomized studies

    Donald B Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of educational Psychology, 66(5):688, 1974

    1974

  15. [23]

    Conceptual issues concerning mediation, interven- tions and composition

    Tyler J VanderWeele and Stijn Vansteelandt. Conceptual issues concerning mediation, interven- tions and composition. Statistics and its Interface, 2(4):457–468, 2009

    2009

  16. [24]

    The consistency statement in causal inference: a definition or an assumption? Epidemiology, 20(1):3–5, 2009

    Stephen R Cole and Constantine E Frangakis. The consistency statement in causal inference: a definition or an assumption? Epidemiology, 20(1):3–5, 2009

    2009

  17. [25]

    A tutorial for conducting causal mediation analysis with the twangmediation package

    Donna L Coffman, Megan S Schuler, Daniel F McCaffrey, Katherine E Castellano, Haoyu Zhou, Brian Vegetabile, and Beth Ann Griffin. A tutorial for conducting causal mediation analysis with the twangmediation package. 2021

    2021

  18. [26]

    Clarify- ing causal mediation analysis: Effect identification via three assumptions and five potential outcomes

    Trang Quynh Nguyen, Ian Schmid, Elizabeth L Ogburn, and Elizabeth A Stuart. Clarify- ing causal mediation analysis: Effect identification via three assumptions and five potential outcomes. Journal of Causal Inference, 10(1):246–279, 2022

    2022

  19. [27]

    A three-way decomposition of a total effect into direct, indirect, and interactive effects

    Tyler J VanderWeele. A three-way decomposition of a total effect into direct, indirect, and interactive effects. Epidemiology, 24(2):224–232, 2013

    2013

  20. [28]

    Mediation analysis in epidemiology: methods, interpretation and bias

    Lorenzo Richiardi, Rino Bellocco, and Daniela Zugna. Mediation analysis in epidemiology: methods, interpretation and bias. International journal of epidemiology , 42(5):1511–1519, 2013

    2013

  21. [29]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models

    BIG bench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023

    2023

  22. [30]

    Quantifying and mitigating uni- modal biases in multimodal large language models: A causal perspective

    Meiqi Chen, Yixin Cao, Yan Zhang, and Chaochao Lu. Quantifying and mitigating uni- modal biases in multimodal large language models: A causal perspective. arXiv preprint arXiv:2403.18346, 2024

    2024 arXiv

  23. [31]

    Announcing Claude 3 Sonnet, 03 2024

    Anthropic. Announcing Claude 3 Sonnet, 03 2024

    2024

  24. [32]

    Models, reasoning and inference

    Judea Pearl et al. Models, reasoning and inference. Cambridge, UK: CambridgeUniversityPress, 19(2):3, 2000

    2000

  25. [33]

    why should i trust you?

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135–1144, 2016

    2016

  26. [34]

    Recognition in terra incognita

    Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), pages 456–473, 2018

    2018

  27. [35]

    Fairness without demographics in repeated loss minimization

    Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Fairness without demographics in repeated loss minimization. In International Conference on Machine Learning, pages 1929–1938. PMLR, 2018. 12

    1929

  28. [36]

    Distributionally robust losses against mixture covariate shifts

    John C Duchi, Tatsunori Hashimoto, and Hongseok Namkoong. Distributionally robust losses against mixture covariate shifts. Under review, 2(1), 2019

    2019

  29. [37]

    Nuanced metrics for measuring unintended bias with real data for text classification

    Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion proceedings of the 2019 world wide web conference, pages 491–500, 2019

    2019

  30. [38]

    Wilds: A benchmark of in-the-wild distribution shifts

    Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, p...

    2021

  31. [39]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  32. [40]

    Commonsenseqa: A ques- tion answering challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A ques- tion answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937, 2018

    2018 arXiv

  33. [41]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  34. [42]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    arXiv

  35. [43]

    Mixtral of experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  36. [44]

    Causal parrots: Large language models may talk causality but are not causal

    Matej Zeˇcevi´c, Moritz Willig, Devendra Singh Dhami, and Kristian Kersting. Causal parrots: Large language models may talk causality but are not causal. arXiv preprint arXiv:2308.13067, 2023

    2023 arXiv

  37. [45]

    Can large language models infer causation from correlation? arXiv preprint arXiv:2306.05836, 2023

    Zhijing Jin, Jiarui Liu, Zhiheng Lyu, Spencer Poff, Mrinmaya Sachan, Rada Mihalcea, Mona Diab, and Bernhard Schölkopf. Can large language models infer causation from correlation? arXiv preprint arXiv:2306.05836, 2023

    2023 arXiv

  38. [46]

    Does fine-tuning llms on new knowledge encourage hallucinations? arXiv preprint arXiv:2405.05904, 2024

    Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning llms on new knowledge encourage hallucinations? arXiv preprint arXiv:2405.05904, 2024

    2024 arXiv

  39. [47]

    Physics of language models: Part 3.1, knowledge storage and extraction

    Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction. arXiv preprint arXiv:2309.14316, 2023

    2023 arXiv

  40. [48]

    Lima: Less is more for alignment

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024

    2024

  41. [49]

    Language models don’t always say what they think: unfaithful explanations in chain-of-thought prompting

    Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. Language models don’t always say what they think: unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems, 36, 2024

    2024

  42. [50]

    The moderator–mediator variable distinction in so- cial psychological research: Conceptual, strategic, and statistical considerations

    Reuben M Baron and David A Kenny. The moderator–mediator variable distinction in so- cial psychological research: Conceptual, strategic, and statistical considerations. Journal of personality and social psychology, 51(6):1173, 1986

    1986

  43. [51]

    Introduction to statistical mediation analysis

    David MacKinnon. Introduction to statistical mediation analysis. Routledge, 2012

    2012

  44. [52]

    Applying causal mediation analysis to personality disorder research

    Glenn D Walters. Applying causal mediation analysis to personality disorder research. Person- ality Disorders: Theory, Research, and Treatment, 9(1):12, 2018. 13

    2018

  45. [53]

    Causal indicator models: Identification, estimation, and testing

    Kenneth A Bollen and Walter R Davis. Causal indicator models: Identification, estimation, and testing. Structural Equation Modeling: A Multidisciplinary Journal, 16(3):498–522, 2009

    2009

  46. [54]

    Marginal structural models for the estimation of direct and indirect effects

    Tyler J VanderWeele. Marginal structural models for the estimation of direct and indirect effects. Epidemiology, 20(1):18–26, 2009

    2009

  47. [55]

    Fairness in decision-making—the causal explanation formula

    Junzhe Zhang and Elias Bareinboim. Fairness in decision-making—the causal explanation formula. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018

    2018

  48. [56]

    Fair inference on outcomes

    Razieh Nabi and Ilya Shpitser. Fair inference on outcomes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018

    2018

  49. [57]

    Asymptotic and resampling strategies for assessing and comparing indirect effects in multiple mediator models

    Kristopher J Preacher and Andrew F Hayes. Asymptotic and resampling strategies for assessing and comparing indirect effects in multiple mediator models. Behavior research methods , 40(3):879–891, 2008

    2008

  50. [58]

    Introduction to mediation, moderation, and conditional process analysis: a regression-based approach, 2014

    Jocelyn H Bolin. Introduction to mediation, moderation, and conditional process analysis: a regression-based approach, 2014

    2014

  51. [59]

    Mediation analysis with multiple mediators.Epi- demiologic methods, 2(1):95–115, 2014

    Tyler VanderWeele and Stijn Vansteelandt. Mediation analysis with multiple mediators.Epi- demiologic methods, 2(1):95–115, 2014

    2014

  52. [60]

    An introduction to causal mediation analysis

    Xu Qin. An introduction to causal mediation analysis. Asia Pacific Education Review, pages 1–15, 2024

    2024

  53. [61]

    Identifiability and exchangeability for direct and indirect effects

    James M Robins and Sander Greenland. Identifiability and exchangeability for direct and indirect effects. Epidemiology, 3(2):143–155, 1992

    1992

  54. [62]

    Probabilities of causation: Bounds and identification

    Jin Tian and Judea Pearl. Probabilities of causation: Bounds and identification. Annals of Mathematics and Artificial Intelligence, 28(1):287–313, 2000

    2000

  55. [63]

    Sargur N. Srihari. Causality in artificial intelligence. University at Buffalo, The State University of New York, 2021. Course CSE674, Chapter 21

    2021

  56. [64]

    Causality

    Judea Pearl. Causality. Cambridge university press, 2009

    2009

  57. [65]

    Estimating average marginal effects in nonseparable structural systems

    Susanne Schennach, Halbert White, and Karim Chalak. Estimating average marginal effects in nonseparable structural systems. Technical report, cemmap working paper, 2007

    2007

  58. [66]

    Interpreting and understanding logits, probits, and other nonlinear probability models

    Richard Breen, Kristian Bernt Karlson, and Anders Holm. Interpreting and understanding logits, probits, and other nonlinear probability models. annual review of sociology, 44(1):39–54, 2018

    2018

  59. [67]

    Identification of average marginal effects in fixed effects dynamic discrete choice models

    Victor Aguirregabiria and Jesús M Carro. Identification of average marginal effects in fixed effects dynamic discrete choice models. Review of Economics and Statistics, pages 1–46, 2024

    2024

  60. [68]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022

    2022

  61. [69]

    Finetuned language models are zero-shot learners

    Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021

    2021 arXiv

  62. [70]

    Exploring length gen- eralization in large language models

    Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. Exploring length gen- eralization in large language models. Advances in Neural Information Processing Systems , 35:38546–38556, 2022

    2022

  63. [71]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    2005 arXiv

  64. [72]

    Synthetic and natural noise both break neural machine translation

    Yonatan Belinkov and Yonatan Bisk. Synthetic and natural noise both break neural machine translation. arXiv preprint arXiv:1711.02173, 2017

    2017 arXiv

  65. [73]

    Training on synthetic noise improves robustness to natural noise in machine translation

    Vladimir Karpukhin, Omer Levy, Jacob Eisenstein, and Marjan Ghazvininejad. Training on synthetic noise improves robustness to natural noise in machine translation. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), pages 42–47, 2019. 14

    2019

  66. [74]

    Eda: Easy data augmentation techniques for boosting performance on text classification tasks

    Jason Wei and Kai Zou. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. arXiv preprint arXiv:1901.11196, 2019

    1901 arXiv

  67. [75]

    Towards robustness to label noise in text classification via noise modeling

    Siddhant Garg, Goutham Ramakrishnan, and Varun Thumbe. Towards robustness to label noise in text classification via noise modeling. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 3024–3028, 2021

    2021

  68. [76]

    Preserve

    Tingting Wu, Xiao Ding, Minji Tang, Hao Zhang, Bing Qin, and Ting Liu. Noisywikihow: A benchmark for learning with real-world noisy labels in natural language processing. arXiv preprint arXiv:2305.10709, 2023. 15 A More Examples of Surface and deep structure In this section, w...

    2023 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.