Pith. sign in

REVIEW 4 major objections 6 minor 43 references

The impact of extracurricular education on socioeconomic mobility in Japan: an application of causal machine learning

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read This paper claims that private tutoring in Japan has a modest positive causal effect on children's educational attainment and income relative to their parents, but the effect is concentrated among lower-income families while higher-income…

desk verdict A clean causal-forest application to Japanese shadow education whose headline ATEs are likely biased by post-treatment controls; worth reviewing, not worth taking at face value. read the letter →

arxiv 2506.07421 v1 pith:L4GKLDPM submitted 2025-06-09 econ.GN q-fin.EC

classification econ.GNq-fin.EC
keywords privatetutoringshadoweducationsocialmobilityintergenerationaltransmissioncausalmachinelearningheterogeneoustreatmenteffectsforestJapan
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish whether private tutoring, a form of shadow education, helps children move up the socioeconomic ladder relative to their parents in Japan. Using the 2015 National Survey on Social Stratification and Social Mobility and a causal machine learning method, it estimates a small positive average effect of tutoring on children's educational attainment and income, but essentially no effect on occupational prestige. The effect among those who actually receive tutoring is smaller than the population average, and the gains are concentrated among children from lower-income families. Since higher-income families are more likely to pay for tutoring, the overall contribution to social mobility is minimal. The paper argues this pattern explains the mixed findings in the existing literature and points toward targeted rather than universal interventions.

What carries the argument

The central mechanism is the causal forest, a non-parametric machine learning estimator of conditional average treatment effects that combines many causal trees grown to maximize between-leaf differences in treatment effects, with sample-splitting honesty and residualization of outcomes and treatment on covariates. It is paired with doubly robust augmented inverse probability weighting, so that the average treatment effects remain consistent if either the outcome model or the propensity score model is correctly specified. The outcomes are constructed as intergenerational transmission scores: the standardized difference between the respondent's education, income, or occupational prestige and that of their parents, with parental wealth estimated from ownership of 19 consumer durables.

What would settle it

Compare siblings where one received private tutoring and the other did not: if the within-family estimate of tutoring's effect on educational attainment is near zero while the paper's ATE is about 0.19 standard deviations, the unconfoundedness assumption would be implausible and the causal claim would fail.

Watch

Extended reading notes

Core claim

The central discovery is that private tutoring in Japan raises intergenerational educational attainment by about 0.19 standard deviations and income by about 0.11 standard deviations on average, while its effect on occupational prestige is statistically indistinguishable from zero. The average treatment effect on the treated is smaller than the population average, and conditional average treatment effects decline with parental income: children in the lower and middle income quintiles receive the largest benefits, while children in richer families see little or negative gains. Because participation is higher among richer families, the aggregate effect on social mobility is minimal. The paper uses a causal forest to estimate these heterogeneous effects and identifies parental income as the dominant driver of treatment effect heterogeneity.

Load-bearing premise

The load-bearing premise is unconfoundedness: that with the observed controls, including 9th-grade academic performance and secondary-school type, private tutoring is as good as randomly assigned, so any remaining correlation between tutoring and later outcomes is causal.

Editorial extensions

If this is right

  • If the estimates are right, expanding access to private tutoring alone will not narrow Japan's intergenerational gaps, because those who participate are already advantaged and benefit less.
  • Targeted subsidies for tutoring aimed at lower-income families could produce larger mobility gains than universal programs.
  • Studies that estimate a single average effect of shadow education may hide opposing effects across income groups, which would explain inconsistent findings in the literature.
  • Parental income, not tutoring itself, is the main lever on intergenerational mobility in Japan, according to the variable importance results.
  • Causal machine learning can recover meaningful treatment effect heterogeneity in social stratification data even when the average effect is small.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A policy extension the author leaves implicit: because the estimated benefits fall with parental income, a fixed tutoring budget would be most effective if targeted at the poorest households, and the paper's quintile results suggest the return to such targeting could be substantial.
  • The treatment is defined as any private tutoring lasting more than six months, but intensity and duration vary widely; a dose-response extension could test whether the concentrated benefit among lower-income families grows with tutoring hours.
  • Re-estimating the analysis with the other shadow education arms, cram schools and correspondence courses, could reveal whether the income-gradient heterogeneity is specific to private tutoring or common to all paid extracurricular education.
  • Because the outcome is measured relative to one's own parents, the paper's logic implies that if higher-income families adopt tutoring en masse, the effect on rank-based social mobility could shrink even if absolute attainment rises.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper estimates the effect of private tutoring on intergenerational transmission of educational attainment, income, and occupational prestige in Japan, using the 2015 SSM survey and a causal-forest framework. The headline empirical claims are small positive average effects of tutoring on educational attainment (ATE 0.19±0.07) and income (0.11±0.08), a smaller average effect among the treated, and heterogeneous effects concentrated among lower-income families, so that the overall mobility impact is minimal. The paper contributes a causal-machine-learning application to shadow education in a Japanese setting, but the central estimates rest on an unconfoundedness assumption implemented with covariates measured during or after the treatment period, and the main figures are absent from the manuscript.

Significance. If the estimates were identified, the paper would provide useful evidence on shadow education and intergenerational mobility in Japan, and it is one of relatively few applications of causal forests to this topic. The manuscript includes a reproducible code appendix, documents post-stratification weighting and multiple imputation, and reports internal calibration and best-linear-projection checks; these are practical strengths. However, the causal interpretation is currently not supported by the identification strategy, and the quantitative claims cannot be fully checked because the key figures are missing.

major comments (4)
  1. [§2.1.2, Assumption 2; Table 3.1; Appendix code line 9] The unconfoundedness assumption is implemented with covariates that are not all pre-treatment. The treatment is private tutoring for more than six months during elementary or secondary school, while the causal-forest design matrix includes Academic Performance (self-rated 9th-grade rank, score_c) and School Type (school_c). For respondents tutored during elementary or early secondary school, the 9th-grade rank and secondary-school type are realized after treatment has begun; they are mediators or colliders rather than pre-treatment confounders. Conditioning on them can block part of the treatment effect and open collider bias, so the reported ATEs, the smaller ATT, and the income-gradient CATE pattern are not identified as the causal effect of tutoring. The paper should either re-estimate with covariates that are unambiguously pre-treatment or provide a substantive timing argument that these variables are pre-treatment for the entire treated sample, together with a sensitivity analysis distinguishing the two estimands.
  2. [§4.1 and §4.2, Figures 4.1, 4.2, 4.4, 4.5] The main quantitative evidence is not present in the manuscript. Figures 4.1, 4.2, 4.4, and 4.5 are referenced in the text, but only their captions appear; the actual plots are missing. The ATE/ATT values quoted in §4.1 and the CATE-by-income patterns in §4.2 therefore cannot be verified from the manuscript. Please include the figures or replace them with tables reporting point estimates, standard errors, and confidence intervals for the ATE, ATT, and subgroup analyses.
  3. [§2.1.2 and §6] No sensitivity analysis for the key unconfoundedness assumption is provided. The assumption is acknowledged to be untestable, but the paper does not quantify how large an unobserved confounder would need to be to overturn the ATE or the lower-income CATE pattern. Given that tutoring participation is strongly related to family resources, a formal sensitivity analysis (for example, the Cinelli–Hazlett robustness value or a related bounds approach) is needed before the headline causal conclusions can be considered robust.
  4. [Appendix code lines 19–30] The variable-selection step uses the full sample twice: a first causal forest estimates variable importance, variables with above-mean importance are selected, and a second causal forest is refit on the same data. The confidence intervals reported from the second forest do not account for this selection procedure, so they may understate uncertainty. The manuscript should either use sample splitting or cross-fitting for the selection step or report how sensitive the ATEs and CATEs are to the choice of the importance threshold.
minor comments (6)
  1. [§2.1.1] There is a typo in the paragraph defining the ATE and ATT: an extra 's' appears at the end of 'average treatment effect on the treated (ATT).s'.
  2. [§2.1.2] The acronym SUTVA is consistently misspelled as 'SUTV A' in the assumption heading.
  3. [Table 3.1 and Table 3.2] The variable list refers to Age Cohort, while the descriptive statistics table reports Age as a continuous variable; the appendix code creates an age-group variable from age_c. This inconsistency should be reconciled.
  4. [§3.2 and Figure 3.2] Figures 3.1 and 3.2, which are described in the text, are likewise not included in the manuscript; only their captions are present.
  5. [Table 4.1] The note says 'Results with strong significance,' but the income row for mean.forest.prediction is only marginally significant (p<0.1); the wording should be adjusted.
  6. [§2.1.2, Assumption 1] The claim that SUTVA holds because the SSM sample is sparse is not a sufficient argument; non-interference could still fail through peer effects or local tutoring markets. This should be reworded as an assumption to be defended substantively rather than inferred from sample size.

Circularity Check

1 steps flagged · score 6.0 of 10

Partial circularity: the ranked-CATE quintile graph re-sorts the model's own predictions, while the core ATE/ATT and parental-income CATEs are not circular.

  1. fitted input called prediction [Section 4.2, Figure 4.5; Appendix code (lines 67-90)]
    "Figure 4.5 shows the estimated ATE of private tutoring on three socioeconomic outcomes across quintiles, ranked by the CATE from the Causal Forest model to highlight distinct subgroups. [...] Appendix: ranking <- cut(tau.hat, tau.hat.quantiles, ...); aipw.scores <- (tau.hat + W[idx]/e.hat * (Y[idx] - mu.hat.1) - (1 - W[idx])/(1 - e.hat) * (Y[idx] - mu.hat.0))"

    The quintile bins are formed from tau.hat, the causal forest's own CATE predictions, and the AIPW scores used to estimate each bin's ATE contain tau.hat as their leading term. Conditional on X, the expectation of the AIPW score is tau.hat(X), so averaging within bins sorted by tau.hat mechanically reproduces the sorted in-sample fit: lower quintiles have more negative average tau.hat by construction. The appendix also uses full-sample tau.hat for both ranking and scoring, so the monotone pattern in Figure 4.5 is not an independent estimate of who benefits; it is a re-arrangement of the model's own output.

full rationale

The paper's headline ATE and ATT estimates come from a standard causal forest implemented in the external grf package, with cross-fitting built into average_treatment_effect; those numbers do not reduce to the model's inputs. The parental-income CATE analysis in Figure 4.4 is also a genuine subgroup analysis, because the subgroups are defined by inc_p rather than by the estimated CATE. The main circular element is Figure 4.5: the text presents the quintile-ranked ATE as a substantive finding, but the appendix constructs the ranking from tau.hat and then scores each observation with an AIPW expression whose conditional mean is tau.hat; hence the lower-quintile pattern is forced by sorting the fit itself. There is no load-bearing self-citation chain or imported uniqueness theorem. The use of 9th-grade rank and school type as covariates is a potential post-treatment bias concern, but that is a validity issue, not a circularity of the kind defined here.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The causal estimates rest entirely on the standard identifying assumptions plus the untested 'pre-treatment' status of covariates. The paper introduces no new entities. All outcome measures are constructed from survey data using standard scale-score differences.

free parameters (2)
  • Causal forest hyperparameters = tuned='all' (cross-validated)
    Chosen by cross-validation within the same data; data-driven values for mtry, min.node.size, honesty.fraction, etc.
  • Variable importance selection threshold = mean(varimp)
    The code selects covariates whose variable importance is above the mean, a data-dependent modeling choice.
assumptions (5)
  • standard math SUTVA: no interference between units and no hidden treatment variations.
    Invoked in Section 2.1.2 to justify treating potential outcomes as independent across respondents.
  • domain assumption Unconfoundedness: treatment is independent of potential outcomes given observed covariates X.
    Assumed in Section 2.1.2; not testable, and likely violated by unmeasured motivation and ability.
  • domain assumption Overlap: 0 < P(W=1|X) < 1 for all X.
    Required for common support; checked via SMD before and after weighting in Section 3.2.
  • ad hoc to paper Covariates (school type, 9th-grade academic performance) are pre-treatment.
    The paper calls all explanatory variables 'pre-treated' (Section 3.1.2), but private tutoring during secondary school can affect both school type and 9th-grade grades.
  • domain assumption Intergenerational transmission outcomes (standardized child-parent differences) are valid measures of mobility.
    Outcomes defined in Equation 3.1; assumes comparability of standardized scores across generations and valid parental income proxy from PCA.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The impact of extracurricular education on socioeconomic mobility in Japan: an application of causal machine learning." pith.science (2026). https://pith.science/paper/L4GKLDPM

@misc{pith2026250607421,
  author       = {Pith},
  title        = {Pith review of: The impact of extracurricular education on socioeconomic mobility in Japan: an application of causal machine learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L4GKLDPM}},
  note         = {Machine review of arXiv:2506.07421}
}
read the original abstract

This paper explores the socioeconomic impacts of extracurricular education, specifically private tutoring, on social mobility in Japan. Using data from the 2015 National Survey on Social Stratification and Social Mobility (SSM), we employed a causal machine learning approach to evaluate this educational intervention on income, educational attainment, and occupational prestige. Our research suggests that while shadow education holds the potential for positive socioeconomic impacts, its benefits are undermined by the economic disparities among households, resulting in minimal overall improvement. This highlights the complex mechanisms between individual demographics and educational interventions, revealing promising machine learning applications in this field.

Figures

Figures reproduced from arXiv: 2506.07421 by the authors.

Figure 3
Figure 3. [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗
Figure 3.1
Figure 3.1. presents the differences in participation rates across different types of extracurric￾ular education in Japan. Specially, the average participation rate in private tutoring before high school is 9.7%. However, there is notable variation among prefectures. Tokyo leads with a highest 18.8% participation rate, while Iwate records the lowest at 2.2%. In three urban centers, Tokyo’s participation rate stands at 18.8%, Os… view at source ↗
Figure 3.2
Figure 3.2. Standardized mean difference of for different outcomes. Left: ITI; middle: ITE; right: ITOP [PITH_FULL_IMAGE:figures/full_fig_p019_3_2.png] view at source ↗
Figures from the paper (5 more)
Figure 4.1
Figure 4.1. Figure 4.1: Estimated (conditional) individual level treatment effects for different out￾comes. Left: ITI; middle: ITE; right: ITOP [PITH_FULL_IMAGE:figures/full_fig_p020_4_1.png]
Figure 4.2
Figure 4.2. Figure 4.2: 95% CI of ATE and ATT for the participation of private tutoring in Japan. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_4_2.png]
Figure 4.3
Figure 4.3. Figure 4.3: shows the results of variable importance analysis, providing insights into which factors contribute most to the socioeconomic mobility. Among these covariates, parental in￾come (inc p) emerges as the most influential variable. This finding aligns with existing resear…
Figure 4.4
Figure 4.4. Figure 4.4: Estimated conditional average treatment effect (CATE) by the most important variable (inc p) subgroups. Error bars represent the 95% confidence intervals. subgroups. It reveals a significant negative impact for most subgroups, except for Q5 in income. More negative C…
Figure 4.5
Figure 4.5. Figure 4.5: Estimated ATE by quintile ranking for income, educational attainment, and occupational prestige. Error bars represent the 95% confidence intervals [PITH_FULL_IMAGE:figures/full_fig_p022_4_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 38 canonical work pages

  1. [1]

    Abdullah, A., Doucouliagos, H., and Manning, E. (2013). Does education reduce income inequality? a meta-regression analysis. Journal of Economic Surveys . First published: 19 December 2013

  2. [2]

    and Imbens, G

    Athey, S. and Imbens, G. (2016). Recursive partitioning for heterogeneous causal effects. Proceedings of the National Academy of Sciences , 113(27):7353--7360

  3. [3]

    Athey, S., Tibshirani, J., and Wager, S. (2019). Generalized random forests . The Annals of Statistics , 47(2):1148 -- 1178

  4. [4]

    and Wager, S

    Athey, S. and Wager, S. (2019). Estimating treatment effects with causal forests: An application

  5. [5]

    Austin, P. C. (2011). An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behavioral Research , 46(3):399--424. Published online 2011 Jun 8

  6. [6]

    Becker, G. S. and Tomes, N. (1979). An equilibrium theory of the distribution of income and intergenerational mobility. Journal of Political Economy , 87(6):1153--1189

  7. [7]

    Belloni, A., Chernozhukov, V., and Hansen, C. (2014). High-dimensional methods and inference on structural and treatment effects. Journal of Economic Perspectives , 28(2):29--50

  8. [8]

    Black, S. E. and Devereux, P. J. (2011). Recent developments in intergenerational mobility. Handbook of labor economics , 4:1487--1541

Show all 43 references
  1. [9]

    Blanden, J., Gregg, P., and Machin, S. (2005). Intergenerational mobility in europe and north america. Report supported by the Sutton Trust, Centre for Economic Performance, London School of Economics

  2. [10]

    Bray, M. (2007). The Shadow Education System: Private Tutoring and Its Implications for Planners . Fundamentals of educational planning, 61. UNESCO IIEP, Paris

  3. [11]

    J., and Roscigno, V

    Buchmann, C., Condron, D. J., and Roscigno, V. J. (2010). Shadow Education, American Style: Test Preparation, the SAT and College Enrollment . Social Forces , 89(2):435--461

  4. [12]

    Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, J. (2018). Double/debiased machine learning for treatment and structural parameters . The Econometrics Journal , 21(1):C1--C68

  5. [13]

    Chetty, R., Hendren, N., Kline, P., and Saez, E. (2014). Where is the land of opportunity? the geography of intergenerational mobility in the united states. The quarterly journal of economics , 129(4):1553--1623

  6. [14]

    Cockx, B., Lechner, M., and Bollens, J. (2023). Priority to unemployed immigrants? a causal machine learning evaluation of training in belgium. Labour Economics , 80:102306

  7. [15]

    Coqueret, G. (2021). Machine learning in finance: From theory to practice. Quantitative Finance , 21(1):9--10

  8. [16]

    Corak, M. (2013). Income inequality, equality of opportunity, and intergenerational mobility. Journal of Economic Perspectives , 27(3):79--102

  9. [17]

    Entrich, S. R. (2015). The decision for shadow education in japan: Students’ choice or parents’ pressure? Social Science Japan Journal , 18(2):193--216

  10. [18]

    and Pritchett, L

    Filmer, D. and Pritchett, L. H. (2001). Estimating wealth effects without expenditure data---or tears: An application to educational enrollments in states of india. Demography , 38(1):115--132

  11. [19]

    Fredricks, J. A. and Eccles, J. S. (2006). Is extracurricular participation associated with beneficial outcomes? concurrent and longitudinal relations. Developmental psychology , 42(4):698

  12. [20]

    Fukai, T., Ichimura, H., and Kawata, K. (2021). Describing the impacts of covid-19 on the labor market in japan until june 2020. The Japanese Economic Review , 72(3):439--470

  13. [21]

    Gregorio, J. D. and Lee, J.-W. (2002). Education and income inequality: New evidence from cross-country data. Review of Income and Wealth , 48(3):395--416

  14. [22]

    Holland, P. W. (1986). Statistics and causal inference. Journal of the American Statistical Association , 81(396):945--960

  15. [23]

    HU, Y., FAN, W., and DING, W. (2016). Does shadow education aggravate inequality of educational outcomes. The Eurasia Proceedings of Educational and Social Sciences , 4:11–32

  16. [24]

    Imbens, G. W. and Rubin, D. B. (2015). Causal Inference for Statistics, Social, and Biomedical Sciences . Number 9780521885881 in Cambridge Books. Cambridge University Press

  17. [25]

    and Macmillan, L

    Jerrim, J. and Macmillan, L. (2015). Income inequality, intergenerational mobility, and the great gatsby curve: Is education the key? Social Forces , 94(2):505--533

  18. [26]

    Kang, J. D. Y. and Schafer, J. L. (2007). Demystifying Double Robustness: A Comparison of Alternative Strategies for Estimating a Population Mean from Incomplete Data . Statistical Science , 22(4):523 -- 539

  19. [27]

    Kanomata, N., Tanabe, S., and Takenoshita, H. (2008). Ssm occupational code and international measures of occupational status: Conversion into egp class schema, siops and isei. Problems in Measurement and Analysis in Social Surveys (2005 SSM Survey Series 12) , pages 69--94

  20. [28]

    and Wataru, Y

    Masahiro, N. and Wataru, Y. (2023). Two types of class attainment mediated by education: The effects of high school rank and type on class destination (in japanese). Japanese Sociological Review , 74(1):34--50

  21. [29]

    Ministry of Education, Culture, S. S. and Technology (2008). Survey Concerning Out-of-School Learning Activities of Schoolchildren (In Japanese) . MEXT

  22. [30]

    Mitchell, T. M. (1997). Machine Learning . McGraw-Hill

  23. [31]

    Nakamura, K., Kaneda, T., and Tanaka, H. (2023). Inequality of extracurricular educational opportunities: Measuring the inequality of paid learning opportunities using the kakwani coefficient (in japanese). Working Paper 350, School of Economics, University of Toyama. Technica...

  24. [32]

    one hundred year history of the education system

    Nakano, H. (1973). Ministry of education "one hundred year history of the education system". The Japanese Journal of Educational Research , 40(1):52--54

  25. [33]

    Nakazawa, W. (2010). Latent class analysis of intergenerational educational attainment (in japanese). Japanese Sociological Review , 61(2):112--129

  26. [34]

    M., Rotnitzky, A., and Zhao, L

    Robins, J. M., Rotnitzky, A., and Zhao, L. P. (1994). Estimation of regression coefficients when some regressors are not always observed. Journal of the American statistical Association , 89(427):846--866

  27. [35]

    Rosenbaum, P. R. (1987). Model-based direct adjustment. Journal of the American statistical Association , 82(398):387--394

  28. [36]

    Rosenbaum, P. R. and Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika , 70(1):41--55

  29. [37]

    Rubin, D. B. (1979). Using multivariate matched sampling and regression adjustment to control bias in observational studies. Journal of the American Statistical Association , 74(366a):318--328

  30. [38]

    Rubin, D. B. (1987). Multiple Imputation for Nonresponse in Surveys . Wiley Series in Probability and Statistics. John Wiley & Sons, Inc., New York

  31. [39]

    Ryabov, I. (2020). Intergenerational transmission of socio-economic status: The role of neighborhood effects. Journal of Adolescence , 80:84--97

  32. [40]

    and Chernozhukov, V

    Semenova, V. and Chernozhukov, V. (2020). Debiased machine learning of conditional average treatment effects and other causal functions. The Econometrics Journal , 24(2):264--289

  33. [41]

    Shulruf, B. (2010). Do extra-curricular activities in schools improve educational outcomes? a critical review and meta-analysis of the literature. International Review of Education , 56:591--612

  34. [42]

    and Simmons, R

    Thompson, R. and Simmons, R. (2013). Social mobility and post-compulsory education: revisiting boudon’s model of social opportunity. British Journal of Sociology of Education , 34(5-6):744--765

  35. [43]

    Tibshirani, J., Athey, S., Sverdrup, E., and Wager, S. (2024). grf: Generalized Random Forests . R package version 2.3.2

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.