REVIEW 4 major objections 5 minor 32 references
Machine Teaching by Domain Experts: Towards More Humane,Inclusive, and Intelligent Machine Learning Systems
T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper argues that machine learning systems should be built by domain experts teaching concepts directly, rather than by ML engineers mediating through labeled data.
desk verdict A well-written vision essay that names MOLA and lays out useful challenges, but the open-source analogy undercuts itself: the 'more intelligent' claim is a hope, not a result. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing distinction is between training and teaching. Training shows a machine input-output examples so it can induce patterns; teaching transfers knowledge through high-level constructs such as concepts, definitions, demonstrations, procedures, and tests. The paper names four technical requirements for making teaching work for domain experts—transparency, explainability, multiple teaching patterns, and fast re-training (under five seconds)—and, for community-scale learning, introduces the MOLA (Massive Open Learning AI system), which must handle contradictions, simultaneity, varying levels of expertise, and evaluation consistency. This combination is what would let domain experts replace ML intermediaries and build intelligent systems themselves.
What would settle it
A concrete test: build an expert-centric teaching system for a realistic domain, such as a conversational assistant for diabetic nutrition, and measure whether domain experts can successfully teach it without ML support. If expert-taught performance falls clearly below a conventionally trained baseline, or if experts abandon the tool because re-training exceeds the five-second limit, the central claim weakens. A second test: run a MOLA experiment with a community of experts on a knowledge domain with genuine disagreements and see whether contradictory teachings can be resolved into a coherent, reliable system; persistent incoherence would falsify the community-learning thesis.
Extended reading notes
Core claim
The central claim is that shifting the center of gravity of machine learning development from ML experts to domain experts can lead to better ML systems. The paper proposes that machine teaching—transferring knowledge to a machine through declarative concepts, exemplars, definitions, demonstrations, procedures, and tests, rather than through labeled examples—can allow domain experts to teach ML systems without mediation. It then extends this to communities: a Massive Open Learning AI system (MOLA) learns directly from a large, diverse, mostly unmediated community of experts, and must explicitly handle contradictory facts and opinions instead of hiding them. The paper argues that expert-centric systems built this way are more humane (experts are no longer reduced to labeling data), more inclusive (community knowledge reflects multiple points of view), and potentially more intelligent (knowledge transfer is richer and can cover complex, changing domains such as health, law, and economics).
Load-bearing premise
The claim stands on machine teaching technology actually meeting the paper's own requirements—transparency, explainability, multiple teaching patterns, and re-training in under five seconds—so that domain experts can teach without ML intermediaries; the paper itself calls this a formidable challenge for current ML and MT technologies.
Editorial extensions
If this is right
- Labeled data becomes a secondary resource: the primary knowledge source shifts to the experts who hold it, and ML systems can be updated by the people with domain knowledge rather than by ML engineers.
- Communities of experts could build and maintain AI systems in the manner of open source software, producing higher quality and resilience through diverse contributors.
- ML systems that must handle contradictory expert opinions would be forced to represent disagreement explicitly, making their uncertainty more visible than today's models.
- Complex, diverse, and fast-changing domains—health, law, economics, technical support—would become viable targets for intelligent systems where data-hungry deep learning is impractical.
Reading between the lines
- A testable next step is a bounded MOLA prototype in one specialty, such as diabetic nutrition advice, to compare whether community-taught systems capture more expert knowledge than a conventionally trained model on the same task.
- If expert-centric teaching matures, the central design problem may shift from algorithms to community governance—how to weight conflicting teachers, prevent vocal subgroups from dominating, and manage authorship of aggregated knowledge.
- The proposal implies new evaluation metrics: instead of a single held-out accuracy, systems would need measures of how faithfully they reflect the range and distribution of expert views, which do not yet exist.
- One could use the teaching interface itself as a measurement instrument, logging the concepts and corrections experts provide across domains to test whether a general teaching language can be standardized.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper is a vision/position piece arguing that current deep learning is approaching limits and that expert-centric machine teaching—where domain experts teach ML systems directly using concepts, definitions, and other high-level constructs, without ML-expert mediation—can make ML development more humane, more inclusive, and possibly more intelligent. The paper distinguishes teaching from training, lists four technical requirements for direct expert access, introduces the notion of Massive Open Learning AI (MOLA) as a community-built, openly taught ML system, illustrates the idea with a hypothetical nutritionist community building a diabetes assistant, and discusses challenges such as handling contradictions, simultaneity, varying expertise, and evaluation consistency. The paper concludes that MOLAs may unlock intelligent systems in complex domains such as health, law, and economics. It contains no experiments or formal analyses; its argument is based on analogy, illustrative examples, and citations of prior work.
Significance. If the paper's central claims could be substantiated, the work would be significant for human-centered ML and interactive machine learning: it reframes the development process from ML-experts to domain experts and offers a concrete vision of community-built ML systems. The paper is commendably honest about the technical gap: Section 3.3 explicitly calls the required machine-teaching experience a 'formidable challenge' for current ML and MT technologies. It also engages with the relevant literature on interactive machine learning, machine teaching, and open-source community dynamics, and it acknowledges risks such as sabotage (Tay), bias, and hijacking of community processes. The main value of the paper lies in posing a plausible research agenda rather than in demonstrating a result; its broad claims about 'more intelligent' systems are not yet supported.
major comments (4)
- [Section 3.3] The central premise of the paper depends on machine-teaching technology satisfying the listed requirements (transparency, explainability, multiple teaching patterns, and re-training in under 5 seconds), yet the paper itself states that this is a 'formidable challenge for both current ML and MT technologies.' The manuscript offers no concrete technical roadmap, no citation of a system that jointly meets these requirements, and no evidence that the gap is closing. Consequently, the conclusion in Section 8 that MOLAs 'may be key to unlock the deployment of intelligent systems in complex, diverse domains' rests on an acknowledged but unresolved feasibility assumption. The paper should either temper this claim explicitly to a hypothetical, or provide a substantive argument, with references or preliminary results, that the four requirements are jointly achievable.
- [Sections 4 and 7] The analogy between MOLA communities and open-source software communities is not sufficiently grounded. Section 4 concedes that, unlike Wikipedia pages, the knowledge of an ML system 'is much more complicated to represent, change, and assess.' Yet the quality mechanisms of open-source communities—inspectability, modularity, forkability, revertibility, and testability—all presuppose exactly those properties. The paper never explains how model knowledge can be made inspectable or revertible enough for community review, and Section 6 lists contradictions, simultaneity, and evaluation consistency as unsolved problems, with no proposed mechanism. Section 7 then asserts community-based development yields 'high quality and reliability,' citing Linux and Bitcoin. That is an analogy, not an argument; without a concrete mechanism for attribution, inspection, and rollback of community contributions, the claim that MOLAs will be more intelligent is unsupported at the point where the paper's novelty is strongest.
- [Sections 1, 5, and 8] The claim that expert-centric and community-built systems will be 'more intelligent' is never defined or operationalized. The paper convincingly argues for humaneness and inclusiveness, but 'improved intelligence' is asserted rather than analyzed. The nutritionist scenario in Section 5 is explicitly hypothetical and says 'let us forget for a moment the technical difficulties,' so it cannot serve as evidence. Section 7 attributes capabilities to communities (consensus on real needs, role of motivation, group understanding of the machine, high quality and reliability) without citing empirical studies connecting these properties to ML system performance. To make the central claim load-bearing, the authors need to either define 'intelligence' here in terms of measurable capabilities (e.g., handling evolving knowledge, diverse viewpoints, or rare edge cases) or explicitly reframe the paper as proposing testable hypotheses rather than asserting an outcome.
- [Section 3.2] The supporting evidence for unmediated direct access is thin. The example of intent-action chatbots is cited as a success because 'hundreds of thousands of chatbots' were created by non-ML-experts, but this is a measure of adoption, not of system quality or intelligence. The paper itself notes the intent-action model tends to 'make difficult the scale up of such efforts, thus limiting depth of knowledge and quality.' This evidence therefore does not support the stronger claim that direct expert access leads to better ML systems; at most it suggests ease of use. The manuscript should distinguish accessibility from capability when using this example.
minor comments (5)
- [Section 1] There are several typos: 'LTSMs' should be 'LSTMs', 'an revolutionary technology' should read 'a revolutionary technology', and 'We are now almost close to a decade' is awkwardly phrased.
- [Section 2] The sentence 'or get if from human beings in “labeling farms”' contains a typo: 'get if' should be 'get it' or 'obtain it'.
- [Section 3.1] The phrase 'Teaching machines withing the context of a community' contains a typo: 'withing' should be 'within'.
- [Section 5] The hypothetical community is described as nutritionists, but later paragraphs refer to 'the physicians’ language' and 'a sub-group of physicians keeps monitoring the quality of the advice.' If the example is about nutritionists, the terms should be 'nutritionists' or 'clinicians' to avoid confusion.
- [References] Some reference entries are incomplete: for example, the Amershi et al. 2015 entry and several conference proceedings entries lack full page ranges or publisher information. A consistency pass over the reference list is needed.
Circularity Check
No significant circularity: the paper is a position/argument piece with no equations, no fitted parameters, and no load-bearing self-citation.
full rationale
This is a position paper arguing by analogy and literature review rather than by formal derivation. There are no equations, no fitted parameters, and no quantities that are defined in terms of one another, so the prediction-style circularity patterns do not apply. The teaching/training distinction is stipulated, but the claim that teaching is more humane is not entailed by the stipulation alone; it is supported by independent arguments about the dehumanizing nature of labeling (Section 2) and by cited empirical work on expert preferences. The MOLA argument relies on an analogy to open-source communities, which may be contestable as an empirical or engineering claim, but it is not circular: the definition of MOLA does not itself assert the conclusion that MOLAs will be more intelligent. The paper explicitly concedes that machine teaching faces a 'formidable challenge' (Section 3.3) and lists unresolved technical challenges (Section 6), which further confirms that the claims are aspirational arguments rather than results derived from their own assumptions. No self-citation chain is load-bearing; the cited machine-teaching works are not by the author and are used as external support. Thus, on the specific circularity criteria, the honest finding is no significant circularity.
Assumptions & free parameters
assumptions (5)
- domain assumption Teaching via concepts is more effective and humane than training via examples.
- domain assumption Domain experts prefer teaching and can do it well with appropriate interfaces.
- domain assumption Open source software communities produce high quality, reliable systems.
- ad hoc to paper Machine teaching technology can be advanced to satisfy transparency, explainability, multiple teaching patterns, and fast re-training.
- domain assumption Real-world knowledge is often contradictory and should be explicitly represented.
invented entities (1)
-
MOLA (Massive Open Learning AI)
Cite this review
Pith. "Pith review of Machine Teaching by Domain Experts: Towards More Humane,Inclusive, and Intelligent Machine Learning Systems." pith.science (2026). https://pith.science/paper/5JT5T4HF
@misc{pith2026190808931,
author = {Pith},
title = {Pith review of: Machine Teaching by Domain Experts: Towards More Humane,Inclusive, and Intelligent Machine Learning Systems},
year = {2026},
howpublished = {\url{https://pith.science/paper/5JT5T4HF}},
note = {Machine review of arXiv:1908.08931}
}
read the original abstract
This paper argues that a possible way to escape from the limitations of current machine learning (ML) systems is to allow their development directly by domain experts without the mediation of ML experts. This could be accomplished by making ML systems interactively teachable using concepts, definitions, and similar high level knowledge constructs. Pointing to the recent advances in machine teaching technology, we list key technical challenges specific for such expert-centric ML systems, and suggest that they are more humane and possibly more intelligent than traditional ML systems in many domains. We then argue that ML systems could also benefit greatly from being built by a community of experts as much as open source software did, creating more inclusive systems, in terms of enabling different points-of-view about the same corpus of knowledge. Advantages of the community approach over current ways to build ML systems, as well as specific challenges this approach raises, are also discussed in the paper.
Figures
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...
-
[2]
B., and Kulesza, T
Amershi, S., Cakmak, M., Knox, W. B., and Kulesza, T. (2014). Power to the people: The role of humans in interactive machine learning. AI Magazine , 35(4):105--120
2014
-
[3]
Amershi, S., Chickering, M., Drucker, S., et al. (2015). Modeltracker: Redesigning performance analysis tools for machine learning. In Proc. of CHI'15 , pages 337--346. ACM
work page 2015
-
[4]
Aroyo, L. and Welty, C. (2015). Truth is a lie: Crowd truth and the seven myths of human annotation. AI Magazine , 36(1):15--24
work page 2015
-
[5]
Azaria, A., Krishnamurthy, J., and Mitchell, T. M. (2016). Instructable intelligent personal agent. In Proc. of AAAI'16
work page 2016
-
[6]
Blackwell, A. F. (2015). Interacting with an inferred world: the challenge of machine learning for humane computer interaction. In Proc. of The Fifth Decennial Aarhus Conference on Critical Alternatives , pages 169--180. Aarhus University Press
work page 2015
-
[7]
Darwiche, A. (2018). Human-level intelligence or animal-like abilities? CACM , 61:56--67
work page 2018
-
[8]
Dudley, J. J. and Kristensson, P. O. (2018). A review of user interface design for interactive machine learning. ACM Transactions on Interactive Intelligent Systems (TiiS) , 8(2):8
work page 2018
Show all 32 references
-
[9]
Fails, J. A. and Olsen Jr, D. R. (2003). Interactive machine learning. In Proc. of IU'03 , pages 39--45. ACM
2003
-
[10]
Fan, Y., Tian, F., Qin, T., Li, X.-Y., and Liu, T.-Y. (2018). Learning to teach. In Proc. of ICML'18
2018
-
[11]
R., and Trueman, D
Fiebrink, R., Cook, P. R., and Trueman, D. (2011). Human model evaluation in interactive supervised learning. In Proc. of CHI'11
2011
-
[12]
and Grill, T
Flexer, A. and Grill, T. (2016). The problem of limited inter-rater agreement in modelling music similarity. Journal of New Music Research , 45(3):239--251
2016
-
[13]
Fogarty, J., Tan, D., Kapoor, A., and Winder, S. (2008). Cueflik: interactive concept learning in image search. In Proc. of CHI'08 , pages 29--38. ACM
2008
-
[14]
Gurari, D., Theriault, D., Sameki, M., et al. (2015). How to collect segmentations for biomedical images? a benchmark evaluating the performance of experts, crowdsourced non-experts, and algorithms. In 2015 IEEE Winter Conference on Applications of Computer Vision , pages 1169--1176
2015
-
[15]
Hancock, B., Varma, P., Wang, S., Bringmann, M., Liang, P., and R \'e , C. (2018). Training classifiers with natural language explanations. arXiv preprint arXiv:1805.03818
2018 arXiv
-
[16]
Hood, D., Lemaignan, S., and Dillenbourg, P. (2015). When children teach a robot to write: An autonomous teachable humanoid which uses simulated handwriting. In Proc. of HRI'15 , pages 83--90
2015
-
[17]
T., Morrison, C
Kaochar, T., Peralta, R. T., Morrison, C. T., Fasel, I. R., Walsh, T. J., and Cohen, P. R. (2011). Towards understanding how humans teach robots. In Proc. of UMAP'11 , pages 347--352
2011
-
[18]
Kulesza, T., Burnett, M., Wong, W.-K., and Stumpf, S. (2015). Principles of explanatory debugging to personalize interactive machine learning. In Proc. of IUI'15 , pages 126--137. ACM
2015
-
[19]
Leu, G., Lakshika, E., Tang, J., Merrick, K., and Barlow, M. (2017). Machine education-the way forward for achieving trust-enabled machine agents. In Proc. of the NIPS'17 Workshop on Teaching Machines, Robots, and Humans
2017
-
[20]
B., Rehg, J
Liu, W., Dai, B., Humayun, A., Tay, C., Yu, C., Smith, L. B., Rehg, J. M., and Song, L. (2017). Iterative machine teaching. In Proc. of ICML'17 , pages 2149--2158
2017
-
[21]
Mitchell, T., Cohen, W., Hruschka, E., et al. (2015). Never-ending learning. In Proc. of AAAI'15
2015
-
[22]
and Nagy, P
Neff, G. and Nagy, P. (2016). Automation, algorithms, and politics | talking to bots: symbiotic agency and the case of tay. International Journal of Communication , 10:17
2016
-
[23]
Quintana-Amate, S., Bermell-Garcia, P., and Tiwari, A. (2015). Transforming expertise into knowledge-based engineering tools: A survey of knowledge sourcing in the context of engineering design. Knowledge-Based Systems , 84:89--97
2015
-
[24]
Roy, S., Kieson, E., Abramson, C., and Crick, C. (2018). Using human reinforcement learning models to improve robots as teachers. In Proc. of HRI'18 , pages 225--226. ACM
2018
-
[25]
Sarkar, A. (2015). Confidence, command, complexity: metamodels for structured interaction with machine intelligence. In Proc. of the PPIG'15
2015
-
[26]
E., and Belpaeme, T
Senft, E., Lemaignan, S., Baxter, P. E., and Belpaeme, T. (2017). Leveraging human inputs in interactive machine learning for human robot interaction. In Proc. of HRI'17 , pages 281--282
2017
-
[27]
Y., Amershi, S., Chickering, D
Simard, P. Y., Amershi, S., Chickering, D. M., et al. (2017). Machine teaching: A new paradigm for building machine learning systems. arXiv preprint arXiv:1707.06742
2017 arXiv
-
[28]
Stumpf, S., Rajaram, V., Li, L., et al. (2009). Interacting meaningfully with machine learning systems: Three experiments. International Journal of Human-Computer Studies , 67(8):639--662
2009
-
[29]
Tecuci, G., Marcu, D., Boicu, M., and Schum, D. A. (2016). Knowledge Engineering: Building Cognitive Assistants for Evidence-based Reasoning . Cambridge University Press
2016
-
[30]
Thomaz, A. L. and Breazeal, C. (2008). Teachable robots: Understanding human teaching behavior to build more effective robot learners. Artificial Intelligence , 172(6-7):716--737
2008
-
[31]
Zhu, X. (2015). Machine teaching: An inverse problem to machine learning and an approach toward optimal education. In Proc. of AAAI'15
2015
-
[32]
write newline
" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.