The dissertation presents docode, a causal interpretability method for neural code models, and uses a case study to show that some correlations between code properties and model performance are confounded rather than causal.
Probing Pretrained Models of Source Code
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Deep learning models are widely used for solving challenging code processing tasks, such as code generation or code summarization. Traditionally, a specific model architecture was carefully built to solve a particular code processing task. However, recently general pretrained models such as CodeBERT or CodeT5 have been shown to outperform task-specific models in many applications. While pretrained models are known to learn complex patterns from data, they may fail to understand some properties of source code. To test diverse aspects of code understanding, we introduce a set of diagnosting probing tasks. We show that pretrained models of code indeed contain information about code syntactic structure and correctness, the notions of identifiers, data flow and namespaces, and natural language naming. We also investigate how probing results are affected by using code-specific pretraining objectives, varying the model size, or finetuning.
fields
cs.SE 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Towards a Science of Causal Interpretability in Deep Learning for Software Engineering
The dissertation presents docode, a causal interpretability method for neural code models, and uses a case study to show that some correlations between code properties and model performance are confounded rather than causal.