REVIEW 3 major objections 6 minor 1 cited by
Understanding Bias in Machine Learning
T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Bias enters a machine learning model through the data it is trained on, and it can be made visible with interactive visualizations.
desk verdict A clear, honest tutorial on dataset bias with interactive demos, but the demos are not reproducible from the text and the paper adds no research contribution beyond the artifact. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The demonstration is built on two visual inspection tools. A parallel-coordinates plot (each feature is a vertical axis; each applicant is a polyline connecting their score on each axis) lets the reader brush value ranges and see which combinations of features occur together in the training and test data. Layer-wise relevance propagation (LRP) computes, by a backward pass through the network, how much each input feature contributed to the model's decision for a given applicant, displayed as a relative relevance bar chart. These two tools carry the argument: the plots show the bias in the data, and the LRP attributions show the bias in the learned model.
What would settle it
Train the same synthetic hiring task with several different network architectures and random seeds, hold the biased data fixed, and check whether the LRP relevance patterns (for example, university importance under sample selection bias) still appear in every run; if the patterns vanish in most runs, the paper's demonstration does not support its general conclusion.
Extended reading notes
Core claim
The paper's central claim is that bias enters a machine learning model through the data used to build it, and that this is not a rare corner case: any of three identifiable dataset distortions can be picked up by the model. In the synthetic hiring example, when a skill (Pytorch) was never high in the training data, the model became unreliable for applicants with high Pytorch scores; when a university was spuriously correlated with the invite label, the model treated university as a relevant feature; and when the reject class dominated the training set, the model's predictions skewed toward rejection even on a balanced test set. The paper argues that such effects can be made visible and understandable through interactive visualization of the data and explainability analysis of the model, making it possible for domain experts to recognize and counter the bias.
Load-bearing premise
The demonstrations assume that the neural network and the explanation scores shown in the plots are representative of machine learning models generally.
Editorial extensions
If this is right
- If training data is biased, the resulting model's predictions can be biased even when the test population has changed; the Pytorch example shows a model failing on a pattern it never saw.
- Models can pick up spurious correlations, such as university identity, and treat them as relevant indicators rather than the semantically meaningful skills.
- Class imbalance in the training set can shift the model's decision boundary so that its rejection rate is distorted relative to the true label distribution.
- The same inspection techniques (parallel coordinates plus LRP) can be used to detect bias in the training data and in the model before deployment.
Reading between the lines
- The paper's demonstration uses one unspecified network; the three biases should appear for any model class, but the exact relevance patterns are likely architecture-dependent, so the qualitative claim is more robust than the specific numbers.
- In practice, the three bias types often compound, so the clean separation used in the demos may not hold; detection tools would need to disentangle confounded sources.
- A testable extension is to train a deliberately debiased model (for example, resampling to balance classes or removing the university feature) and check whether LRP attributions drop the spurious feature, turning the visual explanation into a diagnostic for mitigation.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This short paper argues—correctly and in an accessible way—that machine learning models can inherit bias from biased training data. The authors define three types of bias (covariate shift, sample selection bias, and imbalance bias), introduce a synthetic hiring scenario in which the ground-truth label is based on a simple two-of-four-skills rule, and present interactive parallel-coordinate and LRP-relevance visualizations intended to show how each bias type distorts the predictions and feature attributions of a trained neural network. The paper closes with a discussion of countermeasures and a list of further reading.
Significance. The paper addresses a topic of clear importance: understanding and detecting bias in machine learning systems. Its accessible framing and the use of interactive visualizations as a pedagogical device are commendable, and the central qualitative claim is well supported by prior literature cited in Section 5. However, the manuscript's own demonstration is not independently verifiable: it omits the data-generation details, the network architecture, the training procedure, and the exact LRP rule, and it provides no quantitative evaluation. If these gaps are filled, the paper could serve as a useful educational resource; as written, it reads more like an extended abstract for a workshop demonstration than a standalone journal article. The authors should be credited for clearly separating the three bias types and for explicitly stating that a model may or may not pick up a given bias, a nuance that is often missing in popular treatments.
major comments (3)
- [§3.1–3.3, Figures 1–7] The demonstration's load-bearing assumption is that the displayed behavior of the neural network and its LRP attributions faithfully represent what a typical model trained on the biased synthetic data would do. The paper does not specify the network architecture, activation functions, layer sizes, training procedure, hyperparameters, the exact LRP propagation rule, or the data-generation parameters (e.g., the number of samples, score distributions, or the university–skill correlation). Without these details, the plots in Sections 3.1–3.3 cannot be reproduced or independently assessed, and the observed relevance patterns could be idiosyncratic to one arbitrary configuration. Please provide a complete description of the model and data in an appendix, and include a stable URL or DOI for the code and interactive plots.
- [§3.1, side note] The side note in Section 3.1 states that the neural network 'did not pick up' the intended two-skill rule even for some in-distribution applicants. This admission is important: if the network fails to learn the ground-truth rule in the unbiased case, then the differences in LRP relevance attributed to covariate shift in Figures 2 and 3 could instead be effects of underfitting, misoptimization, or an unsuitable LRP rule. The paper should report the model's training and test accuracy on the unbiased data, a confusion matrix or quantitative comparison of model decisions against the ground-truth HR rule for each bias scenario, and a sanity check of the LRP attributions on correctly classified examples.
- [§4, Conclusions] The conclusion states that 'we have shown that if the data is biased, the resulting model may be biased, too.' As a scientific claim, this is not established by the preceding demonstration: the bias is injected into the synthetic data by design, and the observed model behavior is described only qualitatively. The claim is independently supported by the literature cited in Section 5, so it is not incorrect, but the paper should explicitly position the synthetic experiments as illustrative rather than as a proof or systematic study. Adding a small quantitative experiment with multiple random seeds and architectures would also strengthen the robustness claims.
minor comments (6)
- [Abstract] The sentence 'we develop interactive plots to visualizing the bias' should read 'to visualize the bias.'
- [§2, first bullet] The definition of covariate shift as 'one of the features is not covered uniformly in the dataset' is imprecise and could be confused with missing data. Please define it as a change in the marginal distribution P(X) between the training and test/deployment populations, while P(Y|X) remains the same.
- [§3, Figures and text] Several places use 'Lets' instead of 'Let's' (e.g., 'Lets take a simplified...', 'Lets make this more concrete'), and 'HRs decision' should be 'HR's decision.'
- [Figure 7 caption] The caption of Figure 7 reads 'Interactive Visualization of Relevance of each Feature,' but the figure actually shows pie charts of invite/reject ratios and correct/incorrect decisions. Please correct the caption to match the content.
- [§3] The interactive plots are not self-contained in a static paper: the reader is directed to a workshop homepage or to 'run our python code' without a direct URL or repository identifier. Please include a stable link and, ideally, a supplementary PDF with screenshots or static versions of the key states of the plots.
- [References] Reference [7] is incomplete ('In CVPR.' with no year or venue details), and some other entries lack pages or years. Please complete all references for reproducibility of the literature survey.
Circularity Check
No significant circularity: the paper is an illustrative demo with no derivation chain, and its single self-citation is not load-bearing.
full rationale
This paper makes no formal derivation or prediction; it is a pedagogical illustration. Section 1 defines bias as entering the model through biased training data, and Sections 3.1–3.3 construct synthetic hiring data with deliberately injected covariate shift, sample selection bias, and imbalance, then show a neural network trained on that data reflects the injected bias in its decisions and LRP attributions. Because the bias is inserted by design, the demonstration is an example of the definition rather than an independent test, which makes the illustration partly self-fulfilling; however, this is not a circular argument in the technical sense. The central claim ('if the data is biased, the resulting machine learning model may be biased, too', Section 4) is independently supported by external prior work cited in Section 5, such as references [4,5,6,7,8,9]. The only self-citation is reference [21], a co-author's ACCV paper on contrastive backpropagation, and it appears in a list of backpropagation-based attribution methods without carrying any load for the paper's argument. The side note in Section 3.1 admits the network failed to learn the intended two-skill rule for some in-distribution applicants, which is a limitation of the demo's credibility but not a circular step. Similarly, the omitted network architecture and training details are reproducibility concerns, not evidence of circularity. Overall, the paper is self-contained as a demonstration and does not reduce its conclusions to its inputs by construction.
Assumptions & free parameters
free parameters (5)
- Invitation threshold =
score > 0.7 in at least two of four skills
- Training invite/reject ratio =
10% invite, 90% reject
- Covariate shift cutoff for Pytorch =
no training applicants above 0.7
- University-skills correlation =
University10/9/3 concentrated among high stats and Python applicants
- Neural network architecture and training setup =
unspecified
assumptions (3)
- domain assumption A neural network with sufficient capacity can approximate the rule 'invite when at least two skills are above 0.7' when the training data covers the relevant region.
- domain assumption Layer-wise Relevance Propagation attributions correctly reflect the features the neural network actually uses for each decision.
- domain assumption The three-fold taxonomy of bias types (covariate shift, sample selection bias, imbalance bias) is sufficient for the tutorial's claims.
Cite this review
Pith. "Pith review of Understanding Bias in Machine Learning." pith.science (2026). https://pith.science/paper/TQD4O5G6
@misc{pith2026190901866,
author = {Pith},
title = {Pith review of: Understanding Bias in Machine Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/TQD4O5G6}},
note = {Machine review of arXiv:1909.01866}
}
read the original abstract
Bias is known to be an impediment to fair decisions in many domains such as human resources, the public sector, health care etc. Recently, hope has been expressed that the use of machine learning methods for taking such decisions would diminish or even resolve the problem. At the same time, machine learning experts warn that machine learning models can be biased as well. In this article, our goal is to explain the issue of bias in machine learning from a technical perspective and to illustrate the impact that biased data can have on a machine learning model. To reach such a goal, we develop interactive plots to visualizing the bias learned from synthetic data.
Figures
Figures from the paper (4 more)
Forward citations
Cited by 1 Pith paper
-
The CRAFT principles for the responsible use of large language models in policymaking
Proposes the CRAFT principles (Control, Rigour, Accountability, Fairness, Transparency) as a practical framework for responsible LLM use in policymaking.
Reference graph
Works this paper leans on
-
[1]
Parallel coordinates: a tool for visualizing multi-dimensional geometry
Alfred Inselberg and Bernard Dimsdale. Parallel coordinates: a tool for visualizing multi-dimensional geometry. In Proceedings of the 1st conference on Visualiza- tion ’90, pages 361–378. IEEE Computer Society Press, 1990
work page 1990
-
[2]
Layerwise relevance propagation(lrp). http://heatmapping.org/. Accessed: 2019- 08-30
work page 2019
-
[3]
On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation
Sebastian Bach, Alexander Binder, Gr´ egoire Montavon, Frederick Klauschen, Klaus-Robert M¨ uller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one , 10(7):e0130140, 2015. 12 Jindong Gu 1,2, Daniela Oelke 2
work page 2015
-
[4]
Gender shades: Intersectional accuracy dispar- ities in commercial gender classification
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy dispar- ities in commercial gender classification. In Conference on fairness, accountability and transparency, pages 77–91, 2018
work page 2018
-
[5]
Man is to computer programmer as woman is to homemaker? debiasing word embeddings
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in neural information processing systems , pages 4349–4357, 2016
work page 2016
-
[6]
Men also like shopping: Reducing gender bias amplification using corpus-level con- straints
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Men also like shopping: Reducing gender bias amplification using corpus-level con- straints. arXiv preprint arXiv:1707.09457 , 2017
arXiv 2017
-
[7]
Antonio Torralba, Alexei A Efros, et al. Unbiased look at dataset bias. In CVPR
-
[8]
Undoing the damage of dataset bias
Aditya Khosla, Tinghui Zhou, Tomasz Malisiewicz, Alexei A Efros, and Antonio Torralba. Undoing the damage of dataset bias. In European Conference on Com- puter Vision , pages 158–171. Springer, 2012
work page 2012
Show all 21 references
-
[9]
A deeper look at dataset bias
Tatiana Tommasi, Novi Patricia, Barbara Caputo, and Tinne Tuytelaars. A deeper look at dataset bias. In Domain adaptation in computer vision applications , pages 37–55. Springer, 2017
2017
-
[10]
Why should i trust you?: Explaining the predictions of any classifier
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should i trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining , pages 1135–1144. ACM, 2016
2016
-
[11]
Causal interpretations of black-box models
Qingyuan Zhao and Trevor Hastie. Causal interpretations of black-box models. Journal of Business & Economic Statistics , (just-accepted):1–19, 2019
2019
-
[12]
Permutation importance: a corrected feature importance measure
Andr´ e Altmann, Laura Tolo¸ si, Oliver Sander, and Thomas Lengauer. Permutation importance: a corrected feature importance measure. Bioinformatics, 26(10):1340– 1347, 2010
2010
-
[13]
Visualizing and understanding convolutional networks
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV, pages 818–833. Springer, 2014
2014
-
[14]
Visualiz- ing deep neural network decisions: Prediction difference analysis
Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. Visualiz- ing deep neural network decisions: Prediction difference analysis. arXiv preprint arXiv:1702.04595, 2017
2017 arXiv
-
[15]
Deep inside convo- lutional networks: Visualising image classification models and saliency maps
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convo- lutional networks: Visualising image classification models and saliency maps. In ICLR, 2013
2013
-
[16]
Ried- miller
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Ried- miller. Striving for simplicity: The all convolutional net. In ICLR, 2014
2014
-
[17]
Axiomatic attribution for deep networks
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In ICML, 2017
2017
-
[18]
Smoothgrad: removing noise by adding noise
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Vi´ egas, and Martin Wat- tenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825, 2017
2017 arXiv
-
[19]
Grad-cam: Visual explanations from deep networks via gradient-based localization
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedan- tam, Devi Parikh, Dhruv Batra, et al. Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV, pages 618–626, 2017
2017
-
[20]
Learning important features through propagating activation differences
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In ICML, 2017
2017
-
[21]
Understanding individual decisions of cnns via contrastive backpropagation
Jindong Gu, Yinchong Yang, and Volker Tresp. Understanding individual decisions of cnns via contrastive backpropagation. In ACCV, 2018
2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.