REVIEW 4 major objections 5 minor 16 references
Transparent and Fair Profiling in Employment Services: Evidence from Switzerland
T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read Interpretable models can profile long-term unemployment risk almost as accurately as black-box models, with far greater transparency.
desk verdict A credible proof-of-concept that explainable boosting machines come close to XGBoost for long-term unemployment profiling, but the headline needs error bars and a leakage check before I'd trust the gap. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the Explainable Boosting Machine (EBM), a generalized additive model whose prediction function is a sum of univariate feature functions plus a limited number of pairwise interactions. Because the entire prediction function can be visually inspected, EBM is globally interpretable by design, and local explanations are exact rather than approximate. The argument is carried by demonstrating that this additive structure retains most of the predictive power of tree-based black-box ensembles; the experiments hinge on a moving-window cross-validation over years 2014–2018 with 2019 held out as test, and on a post-processing threshold method that equalizes false-positive rates ac
What would settle it
Obtain the same Swiss administrative data, conduct a strict audit of each feature's collection time, and exclude any feature whose values could have been known at the prediction time; if EBM's test-year AUC then falls substantially below XGBoost's (e.g., a gap above 0.02), the paper's 'nearly as good' claim is refuted.
Extended reading notes
Core claim
The paper's central claim is that the accuracy-interpretability tradeoff is real but small: in a realistic time-series validation on Swiss public-employment data, the interpretable EBM comes within a hair's breadth of XGBoost's performance while remaining globally interpretable. It further claims that interpretability can be actively enhanced—by removing unimportant features via backward selection, by smoothing numerical feature functions with cubic splines to eliminate overfitting artifacts, and by applying group-fairness post-processing to equalize false positives across age groups—without incurring more than minor predictive losses. The paper presents this as a proof-of-concept that inter
Load-bearing premise
The load-bearing premise is that any temporal data leakage affects all compared models equally, so the near-parity between EBM and XGBoost would survive a clean-data re-run; if leakage is model-specific, the measured AUC gap could be misleading.
Editorial extensions
If this is right
- Public employment services that currently rely on black-box models such as random forests could adopt EBM and obtain comparable or better predictive performance with complete transparency.
- EBM's global feature functions allow caseworkers to justify individual risk scores to jobseekers, supporting algorithmic recourse and trust.
- Sparse EBMs retaining only 30 or 45 main features keep most of the full model's AUC, indicating that profiling can be made simpler without large accuracy losses.
- Smoothing numerical feature functions removes overfitting artifacts and reduces one form of individual unfairness, since similar jobseekers then receive similar feature contributions.
- Fairness post-processing can equalize false-positive rates across age groups while preserving interpretability, though the true-positive rate drops from 80% to 66%—an explicit accuracy-fairness tradeoff.
Reading between the lines
- If the 0.003 AUC gap generalizes, the practical choice between EBM and XGBoost on tabular social data should be driven by transparency needs, not accuracy; the common belief that black-box models are needed for high stakes may be largely unfounded.
- The stated leakage risk suggests a concrete extension: re-running the comparison on a subset of features whose collection time is fully documented would test whether the near-parity holds once all outcome information is verifiably removed.
- The fairness experiment is deliberately minimal (one protected attribute, one metric); a natural next step is evaluating intersectional attributes such as age and nationality together, using in-processing methods rather than only threshold post-processing.
- Because the underlying data are not public, independent replication would require obtaining the same data from the authorities; a parallel benchmark on public administrative datasets with similar structure could provide a cost-effective external check on the near-parity claim.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper empirically compares interpretable and black-box machine learning models for long-term unemployment (LTU) risk prediction using Swiss administrative unemployment data from 2014–2019. The models are logistic regression (LR), explainable boosting machines (EBM), random forests (RF), gradient boosting (GB), and XGBoost (XGB). Using a moving-window cross-validation design, the paper reports AUC for years 2015–2019, with 2019 held out as the test year. The main empirical claim is that EBM, an inherently interpretable model, achieves predictive performance close to XGB (test 2019 AUC: EBM 0.7593 vs. XGB 0.7619, Table 5). The paper also investigates sparsity via backward selection, smoothing of EBM feature functions, and post-hoc fairness mitigation using the Hardt et al. (2016) equalized-odds post-processing method. The paper argues that interpretable profiling is a viable, accountable alternative to black-box profiling. The manuscript acknowledges in Sec. 6.2 that the temporal order of data collection is unknown for at least some features, creating a possible data leakage risk, but asserts without empirical evidence that leakage would affect all models qualitatively similarly.
Significance. If the central empirical finding is robust, the paper makes a useful contribution to the interpretable-ML and public-employment-services literature by showing that an interpretable additive model can approach the accuracy of a state-of-the-art tree ensemble on a large administrative dataset. The study has several strengths: it uses full administrative records with roughly 165,000 observations per year, employs a time-series-aware evaluation scheme, and releases code for reproduction on GitHub. The fairness analysis, while exploratory, demonstrates a concrete post-processing approach compatible with interpretable models. The significance is qualified by the unresolved leakage question and the lack of uncertainty quantification, both of which are load-bearing for the headline claim. The paper does not present a new algorithm or derivation; its value is as an applied empirical comparison.
major comments (4)
- [§6.2, App. A.1; features beitragsmonate_vor_rf, taggeld_anspr_bins, vers_verdienst] The leakage concern is acknowledged in Sec. 6.2 but the paper's response is an untested assertion: 'data leakage would affect all models, even though there may be quantitative differences.' This is load-bearing because the headline result is a quantitative AUC difference (test 2019: EBM 0.7593 vs. XGB 0.7619, Table 5). Features such as beitragsmonate_vor_rf, taggeld_anspr_bins, and vers_verdienst are derived from the monthly insurance-payment system (asal, App. A.1), and the outcome is also defined from monthly benefit receipt over the first 12 months. If the measurement windows of these features overlap the outcome window, they can directly encode the outcome. The claim that such leakage affects all models comparably is not supported by any experiment. Please provide a sensitivity analysis, e.g., rerun the comparisons on a feature set restricted to variables determined before the unempl
- [§5.1, Table 5, Fig. 1] All AUC comparisons are reported as point estimates without confidence intervals, bootstrap errors, or significance tests. For example, the validation-fold EBM-minus-XGB gaps are 0.0069, 0.0065, 0.0076, and 0.0090, while the test gap is 0.0026. It is impossible to assess whether 'nearly as well' is a statistically and practically meaningful statement. Moreover, the test conclusion rests on a single year (2019). Please provide uncertainty quantification for the AUC differences (e.g., bootstrap or DeLong intervals) and explicitly state whether the observed test-year gap is distinguishable from zero at the available sample size. If possible, also evaluate on additional held-out years or report repeated moving-window test evaluations.
- [Appendix B, §5.1] The model comparison is not fully controlled. XGB hyperparameters were tuned with Optuna, while EBM's key hyperparameter, the number of interactions, was manually set to a low value (30) for interpretability reasons. RF and GB hyperparameters are taken from an unpublished thesis (Gasser 2022). The manuscript states a 'fair' comparison, but this is not established: a manually constrained EBM could understate or overstate the gap depending on how interactions affect performance. Please report a sensitivity analysis over EBM interaction counts (e.g., 0, 10, 30, 60) and state whether the near-parity conclusion holds. Hyperparameters borrowed from non-public sources should be fully documented in the appendix or code so the results are reproducible without access to the thesis.
- [§5.4, Fig. 6, Abstract] The abstract claims that fairness mitigation can enhance transparency and fairness 'with only minor losses in performance.' The reported mitigation reduces TPR from 80% to 66% (Fig. 6d), a 14-percentage-point drop, which the paper itself describes as making the predictor 'not as efficient.' This is not a minor loss at the original operating point unless some other performance measure, such as accuracy, is the criterion. The paper reports that accuracy increases after mitigation, but the relevant trade-off depends on the cost structure of false negatives versus false positives, which is not specified. In addition, the post-processing equalizes FPR under a balanced-accuracy constraint, while the text and figures emphasize normalized false positives; the mismatch between the targeted fairness quantity and the displayed quantity should be clarified.
minor comments (5)
- [§3.1] The phrase 'train-test data (years 2018-2019)' is confusing; the test set is only year 2019 according to Table 2. Please rephrase to 'test data (year 2019)'.
- [§5.3] The text says 'smoothing was applied to individual, off-the-shelf numerical feature functions of one sparse EBM-30 in the first fold' but does not say whether the smoothing parameters were selected on that fold's validation data or on the training data. Please state this explicitly to avoid an optimistic-bias concern.
- [§5.2, Fig. 8] The backward-selection figure caption says 'the rightmost model, with 57 main features and 30 interactions, is the full EBM,' but the main text refers to the full EBM with 57 features in Sec. 4.4 as well. Please ensure the labeling is consistent between the figure, caption, and text.
- [§2.2] The claim that this is the first empirical investigation of interpretable LTU profiling should be softened or qualified, since the paper builds on Gasser (2022, 2023) and there is related work on EBM for social outcomes. A more precise statement would be 'the first comparison of EBM to black-box models in LTU profiling with Swiss administrative data.'
- [Appendix D.1] The data availability statement says data can be obtained from SECO, but the manuscript relies on two non-public theses (Gasser 2022, 2023) for parts of the data construction and hyperparameters. Please ensure all necessary preprocessing decisions are described in the paper or in the released code, so that a third party with data access can reproduce the clean dataset without consulting the theses.
Circularity Check
No circularity: the EBM-vs-XGB result is a held-out empirical comparison, not a derived quantity.
full rationale
The paper's central claim is an empirical benchmark result: EBM reaches AUC 0.7593 vs XGB 0.7619 on the held-out 2019 test set (Table 5). This is a measured comparison on data not used for training or hyperparameter tuning, not a quantity derived from a fitted parameter, a definition, or an equation. The outcome is defined directly from benefit records, and the features come from separate administrative sources; no construction makes one model's test score equal to another's or to a training fit. The acknowledged possibility of temporal data leakage (Sec. 6.2) is a data-quality limitation that could affect the comparison, but the paper does not build the leakage assumption into the reported AUCs, so this is not circularity. The paper cites the author's own prior work (e.g., Räz 2021, 2024a) and the non-public Gasser theses for outcome definition, hyperparameters, and interpretability framing; these are inputs to the experiment, not conclusions that the experiment is claimed to derive. Consequently, no load-bearing step reduces to its own input, and the central comparison is self-contained as an empirical evaluation.
Assumptions & free parameters
free parameters (5)
- EBM interaction count =
30
- EBM learning rate =
0.0014
- LR L1 regularization =
0.1
- Smoothing parameter lambda =
Per feature
- TPR threshold =
0.8
assumptions (5)
- domain assumption A jobseeker is LTU if they receive unemployment benefits during each of the first 12 months of unemployment.
- domain assumption The administrative features are temporally compatible with the outcome despite unknown data-collection order.
- standard math ROC-AUC is the appropriate metric for model comparison.
- domain assumption Equalized false-positive rates across age groups is a relevant fairness criterion.
- ad hoc to paper Local fluctuations in EBM feature functions are artifacts of overfitting.
Cite this review
Pith. "Pith review of Transparent and Fair Profiling in Employment Services: Evidence from Switzerland." pith.science (2026). https://pith.science/paper/4IBED5AE
@misc{pith2026250911847,
author = {Pith},
title = {Pith review of: Transparent and Fair Profiling in Employment Services: Evidence from Switzerland},
year = {2026},
howpublished = {\url{https://pith.science/paper/4IBED5AE}},
note = {Machine review of arXiv:2509.11847}
}
read the original abstract
Long-term unemployment (LTU) is a challenge for both jobseekers and public employment services. Statistical profiling tools are increasingly used to predict LTU risk. Some profiling tools are opaque, black-box machine learning models, which raise issues of transparency and fairness. This paper investigates whether interpretable models could serve as an alternative, using administrative data from Switzerland. Traditional statistical, interpretable, and black-box models are compared in terms of predictive performance, interpretability, and fairness. It is shown that explainable boosting machines, a recent interpretable model, perform nearly as well as the best black-box models. It is also shown how model sparsity, feature smoothing, and fairness mitigation can enhance transparency and fairness with only minor losses in performance. These findings suggest that interpretable profiling provides an accountable and trustworthy alternative to black-box models without compromising performance.
Figures
Figures from the paper (11 more)
Reference graph
Works this paper leans on
-
[5]
Arbeitsmarkt / Arbeitslosenversicherung
For eachn, a fraction of0.5interactions of the total number of main effects were added; the rightmost model, with 57 main features and 30 interactions, is the full EBM. Figure 9: Global explanation (feature importance plot) generated by EBM interface. It shows the (average) feature importance of the 15 most important features for fold 1 (trained on data f...
2014
-
[10]
Interpretml: A unified framework for machine learning interpretability.arXiv preprint arXiv:1909.09223. Ravikumar, P., J. Lafferty, H. Liu, and L. Wasserman
arXiv 1909
-
[11]
InProceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21
Group Fairness: Independence Revisited. InProceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21. New York, NY, USA: Association for Computing Machinery, pp. 129–137. ———. 2024a. ML interpretability: Simple isn’t easy.Studies in History and Philosophy of Science103: 159–67. ArXiv:2211.13617. ———. 2024b. Reliability...
arXiv 2021
-
[12]
Opening the black box of Deep Neural Networks via Information. ArXiv:1703.00810. Van den Berg, G. J., M. Kunaschk, J. Lang, G. Stephan, and A. Uhlendorff
-
[13]
Gam changer: Editing generalized additive models with interactive visualization.arXiv preprint arXiv:2112.03245. Weerts, H., M. Dudík, R. Edgar, A. Jalali, R. Lutz, and M. Madaio
-
[14]
InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems
Explaining it your way-findings from a co-creative design workshop on designing XAI applications with AI end-users from the public sector. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems. pp. 1–14. Zezulka, S.andK.Genin.2024. Fromthefairdistributionofpredictionstothefairdistribution of social goods: Evaluating the impact of ...
2024
-
[2015]
Barocas, S., M
Identification of latest trends and current developments in methods to profile jobseekers in European Public Employment Services . Barocas, S., M. Hardt, and A. Narayanan. 2022.Fairness and Machine Learning. fairml- book.org. Biran, O. and C. Cotton
2022
-
[2016]
Advances in Neural Information Processing Systems
Equality of Opportunity in Supervised Learning. Advances in Neural Information Processing Systems. Hastie, T. and R. Tibshirani. 1990.Generalized Additive Models. No. 43 in Monographs on Statistics and Applied Probability. Chapman & Hall/CRC. Hastie, T., R. Tibshirani, and J. Friedman. 2009.The Elements of Statistical Learning. Springer Series in Statisti...
1990
Show all 16 references
-
[2017]
The Stanford Encyclopedia of Philosophy
Justice. The Stanford Encyclopedia of Philosophy. Molnar, C. 2020.Interpretable machine learning. Lulu.com. 24 Nori, H., S. Jenkins, P. Koch, and R. Caruana
2020
-
[2018]
ArXiv:1606.03490
The Mythos of Model Interpretability.Queue16(3): 31–57. ArXiv:1606.03490. Lou, Y., R. Caruana, J. Gehrke, and G. Hooker
-
[2019]
InProceedings of the 25th ACM SIGKDD inter- national conference on knowledge discovery & data mining
Optuna: A next-generation hyperparameter optimization framework. InProceedings of the 25th ACM SIGKDD inter- national conference on knowledge discovery & data mining. pp. 2623–2631. Allhutter, D., A.Mager, F.Cech, F.Fischer, andG.Grill.2020. DERAMS-ALGORITHMUS. Eine Soziotechn...
2020
-
[2020]
Evaluating time series forecasting models: An empirical study on performance estimation methods.Machine Learning109(11): 1997–
1997
-
[2021]
Körtner, J
Fairness in algorithmic profiling: A German case study.arXiv preprint arXiv:2108.04134. Körtner, J. and G. Bonoli
-
[2022]
” beruf int. Y N berufskl_3 # jobs in class 3 “
Proceed with caution.Canadian Journal of Philosophy52(1): 6–25. 26 Appendix A Data A.1 Raw Datasets The following is a qualitative description of the raw datasets used to construct the clean dataset.The same raw datasets have been used previously in (Gasser, 2022, 2023). For a...
2022
-
[2023]
Gasser, M
(in) Accuracy in Algorithmic Profiling of the Unemployed– An Exploratory Review of Reporting Standards.Social Policy and Society: 1–14. Gasser, M. 2022.Interpretable rule-ensemble methods in the analysis of long-term unemploy- ment. Semester paper, ETH Zürich. 23 ———. 2023.Fai...
2022
-
[2024]
Goodman, B
Long-term fairness inquiries and pursuits in machine learning: A survey of notions, methods, and challenges.arXiv preprint arXiv:2406.06736. Goodman, B. and S. Flaxman
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.