REVIEW 4 major objections 6 minor 43 references
The impact of extracurricular education on socioeconomic mobility in Japan: an application of causal machine learning
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper claims that private tutoring in Japan has a modest positive causal effect on children's educational attainment and income relative to their parents, but the effect is concentrated among lower-income families while higher-income…
desk verdict A clean causal-forest application to Japanese shadow education whose headline ATEs are likely biased by post-treatment controls; worth reviewing, not worth taking at face value. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the causal forest, a non-parametric machine learning estimator of conditional average treatment effects that combines many causal trees grown to maximize between-leaf differences in treatment effects, with sample-splitting honesty and residualization of outcomes and treatment on covariates. It is paired with doubly robust augmented inverse probability weighting, so that the average treatment effects remain consistent if either the outcome model or the propensity score model is correctly specified. The outcomes are constructed as intergenerational transmission scores: the standardized difference between the respondent's education, income, or occupational prestige and that of their parents, with parental wealth estimated from ownership of 19 consumer durables.
What would settle it
Compare siblings where one received private tutoring and the other did not: if the within-family estimate of tutoring's effect on educational attainment is near zero while the paper's ATE is about 0.19 standard deviations, the unconfoundedness assumption would be implausible and the causal claim would fail.
Extended reading notes
Core claim
The central discovery is that private tutoring in Japan raises intergenerational educational attainment by about 0.19 standard deviations and income by about 0.11 standard deviations on average, while its effect on occupational prestige is statistically indistinguishable from zero. The average treatment effect on the treated is smaller than the population average, and conditional average treatment effects decline with parental income: children in the lower and middle income quintiles receive the largest benefits, while children in richer families see little or negative gains. Because participation is higher among richer families, the aggregate effect on social mobility is minimal. The paper uses a causal forest to estimate these heterogeneous effects and identifies parental income as the dominant driver of treatment effect heterogeneity.
Load-bearing premise
The load-bearing premise is unconfoundedness: that with the observed controls, including 9th-grade academic performance and secondary-school type, private tutoring is as good as randomly assigned, so any remaining correlation between tutoring and later outcomes is causal.
Editorial extensions
If this is right
- If the estimates are right, expanding access to private tutoring alone will not narrow Japan's intergenerational gaps, because those who participate are already advantaged and benefit less.
- Targeted subsidies for tutoring aimed at lower-income families could produce larger mobility gains than universal programs.
- Studies that estimate a single average effect of shadow education may hide opposing effects across income groups, which would explain inconsistent findings in the literature.
- Parental income, not tutoring itself, is the main lever on intergenerational mobility in Japan, according to the variable importance results.
- Causal machine learning can recover meaningful treatment effect heterogeneity in social stratification data even when the average effect is small.
Reading between the lines
- A policy extension the author leaves implicit: because the estimated benefits fall with parental income, a fixed tutoring budget would be most effective if targeted at the poorest households, and the paper's quintile results suggest the return to such targeting could be substantial.
- The treatment is defined as any private tutoring lasting more than six months, but intensity and duration vary widely; a dose-response extension could test whether the concentrated benefit among lower-income families grows with tutoring hours.
- Re-estimating the analysis with the other shadow education arms, cram schools and correspondence courses, could reveal whether the income-gradient heterogeneity is specific to private tutoring or common to all paid extracurricular education.
- Because the outcome is measured relative to one's own parents, the paper's logic implies that if higher-income families adopt tutoring en masse, the effect on rank-based social mobility could shrink even if absolute attainment rises.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper estimates the effect of private tutoring on intergenerational transmission of educational attainment, income, and occupational prestige in Japan, using the 2015 SSM survey and a causal-forest framework. The headline empirical claims are small positive average effects of tutoring on educational attainment (ATE 0.19±0.07) and income (0.11±0.08), a smaller average effect among the treated, and heterogeneous effects concentrated among lower-income families, so that the overall mobility impact is minimal. The paper contributes a causal-machine-learning application to shadow education in a Japanese setting, but the central estimates rest on an unconfoundedness assumption implemented with covariates measured during or after the treatment period, and the main figures are absent from the manuscript.
Significance. If the estimates were identified, the paper would provide useful evidence on shadow education and intergenerational mobility in Japan, and it is one of relatively few applications of causal forests to this topic. The manuscript includes a reproducible code appendix, documents post-stratification weighting and multiple imputation, and reports internal calibration and best-linear-projection checks; these are practical strengths. However, the causal interpretation is currently not supported by the identification strategy, and the quantitative claims cannot be fully checked because the key figures are missing.
major comments (4)
- [§2.1.2, Assumption 2; Table 3.1; Appendix code line 9] The unconfoundedness assumption is implemented with covariates that are not all pre-treatment. The treatment is private tutoring for more than six months during elementary or secondary school, while the causal-forest design matrix includes Academic Performance (self-rated 9th-grade rank, score_c) and School Type (school_c). For respondents tutored during elementary or early secondary school, the 9th-grade rank and secondary-school type are realized after treatment has begun; they are mediators or colliders rather than pre-treatment confounders. Conditioning on them can block part of the treatment effect and open collider bias, so the reported ATEs, the smaller ATT, and the income-gradient CATE pattern are not identified as the causal effect of tutoring. The paper should either re-estimate with covariates that are unambiguously pre-treatment or provide a substantive timing argument that these variables are pre-treatment for the entire treated sample, together with a sensitivity analysis distinguishing the two estimands.
- [§4.1 and §4.2, Figures 4.1, 4.2, 4.4, 4.5] The main quantitative evidence is not present in the manuscript. Figures 4.1, 4.2, 4.4, and 4.5 are referenced in the text, but only their captions appear; the actual plots are missing. The ATE/ATT values quoted in §4.1 and the CATE-by-income patterns in §4.2 therefore cannot be verified from the manuscript. Please include the figures or replace them with tables reporting point estimates, standard errors, and confidence intervals for the ATE, ATT, and subgroup analyses.
- [§2.1.2 and §6] No sensitivity analysis for the key unconfoundedness assumption is provided. The assumption is acknowledged to be untestable, but the paper does not quantify how large an unobserved confounder would need to be to overturn the ATE or the lower-income CATE pattern. Given that tutoring participation is strongly related to family resources, a formal sensitivity analysis (for example, the Cinelli–Hazlett robustness value or a related bounds approach) is needed before the headline causal conclusions can be considered robust.
- [Appendix code lines 19–30] The variable-selection step uses the full sample twice: a first causal forest estimates variable importance, variables with above-mean importance are selected, and a second causal forest is refit on the same data. The confidence intervals reported from the second forest do not account for this selection procedure, so they may understate uncertainty. The manuscript should either use sample splitting or cross-fitting for the selection step or report how sensitive the ATEs and CATEs are to the choice of the importance threshold.
minor comments (6)
- [§2.1.1] There is a typo in the paragraph defining the ATE and ATT: an extra 's' appears at the end of 'average treatment effect on the treated (ATT).s'.
- [§2.1.2] The acronym SUTVA is consistently misspelled as 'SUTV A' in the assumption heading.
- [Table 3.1 and Table 3.2] The variable list refers to Age Cohort, while the descriptive statistics table reports Age as a continuous variable; the appendix code creates an age-group variable from age_c. This inconsistency should be reconciled.
- [§3.2 and Figure 3.2] Figures 3.1 and 3.2, which are described in the text, are likewise not included in the manuscript; only their captions are present.
- [Table 4.1] The note says 'Results with strong significance,' but the income row for mean.forest.prediction is only marginally significant (p<0.1); the wording should be adjusted.
- [§2.1.2, Assumption 1] The claim that SUTVA holds because the SSM sample is sparse is not a sufficient argument; non-interference could still fail through peer effects or local tutoring markets. This should be reworded as an assumption to be defended substantively rather than inferred from sample size.
Circularity Check
Partial circularity: the ranked-CATE quintile graph re-sorts the model's own predictions, while the core ATE/ATT and parental-income CATEs are not circular.
-
fitted input called prediction
[Section 4.2, Figure 4.5; Appendix code (lines 67-90)]
"Figure 4.5 shows the estimated ATE of private tutoring on three socioeconomic outcomes across quintiles, ranked by the CATE from the Causal Forest model to highlight distinct subgroups. [...] Appendix: ranking <- cut(tau.hat, tau.hat.quantiles, ...); aipw.scores <- (tau.hat + W[idx]/e.hat * (Y[idx] - mu.hat.1) - (1 - W[idx])/(1 - e.hat) * (Y[idx] - mu.hat.0))"
The quintile bins are formed from tau.hat, the causal forest's own CATE predictions, and the AIPW scores used to estimate each bin's ATE contain tau.hat as their leading term. Conditional on X, the expectation of the AIPW score is tau.hat(X), so averaging within bins sorted by tau.hat mechanically reproduces the sorted in-sample fit: lower quintiles have more negative average tau.hat by construction. The appendix also uses full-sample tau.hat for both ranking and scoring, so the monotone pattern in Figure 4.5 is not an independent estimate of who benefits; it is a re-arrangement of the model's own output.
full rationale
The paper's headline ATE and ATT estimates come from a standard causal forest implemented in the external grf package, with cross-fitting built into average_treatment_effect; those numbers do not reduce to the model's inputs. The parental-income CATE analysis in Figure 4.4 is also a genuine subgroup analysis, because the subgroups are defined by inc_p rather than by the estimated CATE. The main circular element is Figure 4.5: the text presents the quintile-ranked ATE as a substantive finding, but the appendix constructs the ranking from tau.hat and then scores each observation with an AIPW expression whose conditional mean is tau.hat; hence the lower-quintile pattern is forced by sorting the fit itself. There is no load-bearing self-citation chain or imported uniqueness theorem. The use of 9th-grade rank and school type as covariates is a potential post-treatment bias concern, but that is a validity issue, not a circularity of the kind defined here.
Assumptions & free parameters
free parameters (2)
- Causal forest hyperparameters =
tuned='all' (cross-validated)
- Variable importance selection threshold =
mean(varimp)
assumptions (5)
- standard math SUTVA: no interference between units and no hidden treatment variations.
- domain assumption Unconfoundedness: treatment is independent of potential outcomes given observed covariates X.
- domain assumption Overlap: 0 < P(W=1|X) < 1 for all X.
- ad hoc to paper Covariates (school type, 9th-grade academic performance) are pre-treatment.
- domain assumption Intergenerational transmission outcomes (standardized child-parent differences) are valid measures of mobility.
Cite this review
Pith. "Pith review of The impact of extracurricular education on socioeconomic mobility in Japan: an application of causal machine learning." pith.science (2026). https://pith.science/paper/L4GKLDPM
@misc{pith2026250607421,
author = {Pith},
title = {Pith review of: The impact of extracurricular education on socioeconomic mobility in Japan: an application of causal machine learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/L4GKLDPM}},
note = {Machine review of arXiv:2506.07421}
}
read the original abstract
This paper explores the socioeconomic impacts of extracurricular education, specifically private tutoring, on social mobility in Japan. Using data from the 2015 National Survey on Social Stratification and Social Mobility (SSM), we employed a causal machine learning approach to evaluate this educational intervention on income, educational attainment, and occupational prestige. Our research suggests that while shadow education holds the potential for positive socioeconomic impacts, its benefits are undermined by the economic disparities among households, resulting in minimal overall improvement. This highlights the complex mechanisms between individual demographics and educational interventions, revealing promising machine learning applications in this field.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Abdullah, A., Doucouliagos, H., and Manning, E. (2013). Does education reduce income inequality? a meta-regression analysis. Journal of Economic Surveys . First published: 19 December 2013
work page 2013
-
[2]
and Imbens, G
Athey, S. and Imbens, G. (2016). Recursive partitioning for heterogeneous causal effects. Proceedings of the National Academy of Sciences , 113(27):7353--7360
2016
-
[3]
Athey, S., Tibshirani, J., and Wager, S. (2019). Generalized random forests . The Annals of Statistics , 47(2):1148 -- 1178
work page 2019
-
[4]
Athey, S. and Wager, S. (2019). Estimating treatment effects with causal forests: An application
work page 2019
-
[5]
Austin, P. C. (2011). An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behavioral Research , 46(3):399--424. Published online 2011 Jun 8
work page 2011
-
[6]
Becker, G. S. and Tomes, N. (1979). An equilibrium theory of the distribution of income and intergenerational mobility. Journal of Political Economy , 87(6):1153--1189
work page 1979
-
[7]
Belloni, A., Chernozhukov, V., and Hansen, C. (2014). High-dimensional methods and inference on structural and treatment effects. Journal of Economic Perspectives , 28(2):29--50
work page 2014
-
[8]
Black, S. E. and Devereux, P. J. (2011). Recent developments in intergenerational mobility. Handbook of labor economics , 4:1487--1541
work page 2011
Show all 43 references
-
[9]
Blanden, J., Gregg, P., and Machin, S. (2005). Intergenerational mobility in europe and north america. Report supported by the Sutton Trust, Centre for Economic Performance, London School of Economics
2005
-
[10]
Bray, M. (2007). The Shadow Education System: Private Tutoring and Its Implications for Planners . Fundamentals of educational planning, 61. UNESCO IIEP, Paris
2007
-
[11]
J., and Roscigno, V
Buchmann, C., Condron, D. J., and Roscigno, V. J. (2010). Shadow Education, American Style: Test Preparation, the SAT and College Enrollment . Social Forces , 89(2):435--461
2010
-
[12]
Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, J. (2018). Double/debiased machine learning for treatment and structural parameters . The Econometrics Journal , 21(1):C1--C68
2018
-
[13]
Chetty, R., Hendren, N., Kline, P., and Saez, E. (2014). Where is the land of opportunity? the geography of intergenerational mobility in the united states. The quarterly journal of economics , 129(4):1553--1623
2014
-
[14]
Cockx, B., Lechner, M., and Bollens, J. (2023). Priority to unemployed immigrants? a causal machine learning evaluation of training in belgium. Labour Economics , 80:102306
2023
-
[15]
Coqueret, G. (2021). Machine learning in finance: From theory to practice. Quantitative Finance , 21(1):9--10
2021
-
[16]
Corak, M. (2013). Income inequality, equality of opportunity, and intergenerational mobility. Journal of Economic Perspectives , 27(3):79--102
2013
-
[17]
Entrich, S. R. (2015). The decision for shadow education in japan: Students’ choice or parents’ pressure? Social Science Japan Journal , 18(2):193--216
2015
-
[18]
and Pritchett, L
Filmer, D. and Pritchett, L. H. (2001). Estimating wealth effects without expenditure data---or tears: An application to educational enrollments in states of india. Demography , 38(1):115--132
2001
-
[19]
Fredricks, J. A. and Eccles, J. S. (2006). Is extracurricular participation associated with beneficial outcomes? concurrent and longitudinal relations. Developmental psychology , 42(4):698
2006
-
[20]
Fukai, T., Ichimura, H., and Kawata, K. (2021). Describing the impacts of covid-19 on the labor market in japan until june 2020. The Japanese Economic Review , 72(3):439--470
2021
-
[21]
Gregorio, J. D. and Lee, J.-W. (2002). Education and income inequality: New evidence from cross-country data. Review of Income and Wealth , 48(3):395--416
2002
-
[22]
Holland, P. W. (1986). Statistics and causal inference. Journal of the American Statistical Association , 81(396):945--960
1986
-
[23]
HU, Y., FAN, W., and DING, W. (2016). Does shadow education aggravate inequality of educational outcomes. The Eurasia Proceedings of Educational and Social Sciences , 4:11–32
2016
-
[24]
Imbens, G. W. and Rubin, D. B. (2015). Causal Inference for Statistics, Social, and Biomedical Sciences . Number 9780521885881 in Cambridge Books. Cambridge University Press
2015
-
[25]
and Macmillan, L
Jerrim, J. and Macmillan, L. (2015). Income inequality, intergenerational mobility, and the great gatsby curve: Is education the key? Social Forces , 94(2):505--533
2015
-
[26]
Kang, J. D. Y. and Schafer, J. L. (2007). Demystifying Double Robustness: A Comparison of Alternative Strategies for Estimating a Population Mean from Incomplete Data . Statistical Science , 22(4):523 -- 539
2007
-
[27]
Kanomata, N., Tanabe, S., and Takenoshita, H. (2008). Ssm occupational code and international measures of occupational status: Conversion into egp class schema, siops and isei. Problems in Measurement and Analysis in Social Surveys (2005 SSM Survey Series 12) , pages 69--94
2008
-
[28]
and Wataru, Y
Masahiro, N. and Wataru, Y. (2023). Two types of class attainment mediated by education: The effects of high school rank and type on class destination (in japanese). Japanese Sociological Review , 74(1):34--50
2023
-
[29]
Ministry of Education, Culture, S. S. and Technology (2008). Survey Concerning Out-of-School Learning Activities of Schoolchildren (In Japanese) . MEXT
2008
-
[30]
Mitchell, T. M. (1997). Machine Learning . McGraw-Hill
1997
-
[31]
Nakamura, K., Kaneda, T., and Tanaka, H. (2023). Inequality of extracurricular educational opportunities: Measuring the inequality of paid learning opportunities using the kakwani coefficient (in japanese). Working Paper 350, School of Economics, University of Toyama. Technica...
2023
-
[32]
one hundred year history of the education system
Nakano, H. (1973). Ministry of education "one hundred year history of the education system". The Japanese Journal of Educational Research , 40(1):52--54
1973
-
[33]
Nakazawa, W. (2010). Latent class analysis of intergenerational educational attainment (in japanese). Japanese Sociological Review , 61(2):112--129
2010
-
[34]
M., Rotnitzky, A., and Zhao, L
Robins, J. M., Rotnitzky, A., and Zhao, L. P. (1994). Estimation of regression coefficients when some regressors are not always observed. Journal of the American statistical Association , 89(427):846--866
1994
-
[35]
Rosenbaum, P. R. (1987). Model-based direct adjustment. Journal of the American statistical Association , 82(398):387--394
1987
-
[36]
Rosenbaum, P. R. and Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika , 70(1):41--55
1983
-
[37]
Rubin, D. B. (1979). Using multivariate matched sampling and regression adjustment to control bias in observational studies. Journal of the American Statistical Association , 74(366a):318--328
1979
-
[38]
Rubin, D. B. (1987). Multiple Imputation for Nonresponse in Surveys . Wiley Series in Probability and Statistics. John Wiley & Sons, Inc., New York
1987
-
[39]
Ryabov, I. (2020). Intergenerational transmission of socio-economic status: The role of neighborhood effects. Journal of Adolescence , 80:84--97
2020
-
[40]
and Chernozhukov, V
Semenova, V. and Chernozhukov, V. (2020). Debiased machine learning of conditional average treatment effects and other causal functions. The Econometrics Journal , 24(2):264--289
2020
-
[41]
Shulruf, B. (2010). Do extra-curricular activities in schools improve educational outcomes? a critical review and meta-analysis of the literature. International Review of Education , 56:591--612
2010
-
[42]
and Simmons, R
Thompson, R. and Simmons, R. (2013). Social mobility and post-compulsory education: revisiting boudon’s model of social opportunity. British Journal of Sociology of Education , 34(5-6):744--765
2013
-
[43]
Tibshirani, J., Athey, S., Sverdrup, E., and Wager, S. (2024). grf: Generalized Random Forests . R package version 2.3.2
2024
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.