Pith. sign in

REVIEW 3 major objections 4 minor 3 cited by

Human-in-the-loop or AI-in-the-loop? Automate or Collaborate?

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that many human-in-the-loop systems are misnamed: when the human holds final decision authority and the AI assists, the system is AI-in-the-loop, and evaluation must be human-centered.

desk verdict A clear, honest blue-sky paper with a useful "AI2L" label; the binary taxonomy is underspecified on control, but it deserves a serious referee rather than a desk reject. read the letter →

arxiv 2412.14232 v1 pith:UBO45G5L submitted 2024-12-18 cs.HC

Sriraam Natarajan , Saurabh Mathur , Sahil Sidheekh , Wolfgang Stammer , Kristian Kersting This is my paper · ORCID
classification cs.HC
keywords human-in-the-loopAI-in-the-loophuman-AIcollaborationevaluationmetricsdecisionauthorityinteractivemachinelearninghuman-centeredautomationvs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Many systems described as human-in-the-loop (HIL) put the machine in charge and use people as labelers or advisors; this paper argues that a large share of them should instead be called AI-in-the-loop ($AI^{2}$L), where the human expert holds decision authority and the AI assists. The paper claims this mislabeling is not cosmetic: it leads to evaluating the wrong thing, measuring model accuracy when the relevant outcome is the quality of the human's decision, and it can produce abstraction errors during design and deployment. The authors propose that designers first decide whether they are automating a well-defined subtask (HIL) or collaborating inside a human decision process ($AI^{2}$L), and then adopt evaluation regimes that fit the actual control structure. If right, the paper gives the field a sharper vocabulary and a reason to make human-centered metrics, including ablation studies, interpretability, fairness, and end-goal impact, standard for collaborative systems.

What carries the argument

The load-bearing object is a two-way taxonomy built on the locus of decision-making authority and control. HIL (automate) is defined as an autonomous AI agent that may seek human input; $AI^{2}$L (collaborate) is defined as an intervention in a human decision-making process, where the AI presents synthesized information, possible actions, and consequences and the human chooses. The distinction does the argument's work: it predicts different sources of bias (model and data bias versus human interpretation bias), different trust problems (human-teacher credibility versus system transparency), and different evaluation requirements (AI-centered metrics versus human-centered, end-goal-aligned metrics). The paper's key move is to make this control-based classification the first step in system design and to show, through its table of examples, that many real subtasks in medicine, driving, logistics, manufacturing, finance, and education sort cleanly into either automate (HIL) or collaborate ($AI^{2}$L).

What would settle it

Run a reliability study in which two independent coders apply the paper's definitions to the systems listed in its examples; any substantial disagreement on the HIL versus $AI^{2}$L label would show that "who is in control" is not an operational test.

Watch

Extended reading notes

Core claim

The central claim is that 'human-in-the-loop' and 'AI-in-the-loop' name opposite control configurations. In a true HIL system the AI is the decision-maker and the human supplies corrections, labels, advice, or adversarial input to steer it; in an $AI^{2}$L system the human is the decision-maker and the AI is an assistive component that summarizes evidence, proposes options, and flags risks. Because the two configurations differ in who holds authority, they also differ in where bias enters and in what counts as success. The paper therefore contends that treating $AI^{2}$L systems as HIL produces wrong evaluation (accuracy-centered rather than human-outcome-centered), wrong trust modeling (credibility of the human teacher rather than transparency of the system), and abstraction errors in deployment. Its constructive proposal is to classify systems by control before design, and for $AI^{2}$L systems to evaluate with ablation, interpretability, fairness, and impact on the human's actual decision.

Load-bearing premise

The taxonomy assumes that a clear line can be drawn between "the human is in control" and "the AI is in control" in real systems, even though control is often shared, shifting, or context-dependent.

Editorial extensions

If this is right

  • Adopting the distinction means moving AI^2L evaluation away from accuracy, precision, and recall as the primary yardstick and toward human-centered metrics such as ablation studies, interpretability, fairness, and measured impact on the human's decision outcome.
  • Designers who currently build "HIL" systems for collaborative tasks would be forced to ask up front whether they are automating a well-defined subproblem (HIL) or intervening in an open-ended human decision process (AI^2L), changing what gets abstracted away.
  • The paper claims the framework transfers beyond supervised learning to reinforcement learning, planning, continual learning, and foundation models; in particular, large language models with thumbs-up or thumbs-down feedback are HIL-style oversight systems, not true collaborators, unless they acquire a theory of mind.
  • A correct HIL or AI^2L classification would reduce abstraction errors of the kind that occur when a complex, context-dependent domain is treated as a neat automation problem.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the control-based taxonomy is right, a natural test is to replace the binary with a graded "control profile" measuring how much discretion the human actually exercises per subtask; the paper's own admission that domains are nested suggests the binary is a heuristic, not a law.
  • A concrete extension: for AI^2L systems, model evaluation should be supplemented by decision-level causal analysis that estimates the AI's marginal effect on human choices, not just its predictive accuracy.
  • The paper's framing implies that benchmark suites for human-AI collaboration should record the human's final decision and downstream outcome, not only the AI suggestion; existing datasets that log only model outputs would be insufficient for AI^2L evaluation.
  • Applied to foundation models, the paper's argument points to a research agenda in which models are trained to represent user goals and uncertainty rather than merely to follow instructions, because only then do they shift from HIL-style reactiveness to AI^2L-style collaboration.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This blue-sky position paper argues that many systems commonly called human-in-the-loop (HIL) are in fact AI-in-the-loop (AI²L) systems: the human retains decision-making authority and the AI serves as a supporting component, whereas in true HIL systems the AI is the decision-maker and the human supplies labels, feedback, or advice. The authors contend that existing evaluation practice overemphasizes machine-learning metrics such as accuracy and precision, which is appropriate for HIL systems but inadequate for AI²L systems, and they advocate human-centric evaluation that accounts for the human expert's role, the human-AI interaction, and end goals such as health outcomes. The paper motivates the distinction with two introductory examples, a comparison figure, a table of domain examples, and a discussion of differences in control, bias, and evaluation.

Significance. If the proposed distinction were made precise and well-grounded, it would be a genuinely useful corrective: the field does often conflate systems in which the AI is the primary actor with systems in which it is a subordinate assistant, and the paper's connection to abstraction errors and human-centric evaluation is timely. The strengths of the paper as a position piece are its clear Figure 1, the concrete Table 1 examples across six domains, and its alignment with prior work on explainable/advisable AI and on the limitations of static benchmarks. However, the paper's central contribution is a taxonomy, and the taxonomy currently rests on an unoperationalized notion of control; the manuscript itself concedes that the categories are nested and granularity-dependent. This limits the paper's actionable impact and, as presented, weakens the prescriptive evaluation claims. The paper is a reasonable blue-sky argument but needs a sharper formalization before it can serve as a reliable framework for practitioners.

major comments (3)
  1. [Discussion (nested domains paragraph)] The paper's own Discussion states that 'the appropriate problem domains for the HIL and AI2L systems are typically not separated but nested' and that 'zooming in on a domain would result in a HIL problem and zooming out would give us an AI2L problem.' This makes the HIL/AI2L classification a function of analysis granularity rather than an intrinsic property of the system. Since the paper's central prescriptions—AI-centered metrics for HIL versus human-centric evaluation for AI²L—depend on a stable classification, this concession undermines the load-bearing claim. Please provide a decision procedure for selecting the level of abstraction at which the classification is made, or state an explicit rule for when a system should be evaluated as HIL versus AI²L despite the nested relationship.
  2. [Introduction and Figure 1 caption] The central notion of 'decision-making authority and control' is never defined operationally. The two introductory examples are classified by an intuitive reading of who 'decides,' and Table 1's fraud-detection row is classified as 'Automate (HIL)' even though the description says 'AI analyzes the data and flags suspicious activities; humans confirm'—a configuration that could equally be read as human-in-control if confirmation is decisive. Without explicit criteria (e.g., who has veto power, who bears final accountability, whether the system can operate without the human), the taxonomy cannot be reliably applied by other researchers, and the claim that a system should be evaluated human-centrically or AI-centrically is not testable. Please add operational criteria for control and apply them consistently to all Table 1 examples.
  3. [Abstract and the AI-in-the-loop evaluation paragraph] The paper asserts that 'existing evaluation methods often overemphasize the machine (learning) component's performance, neglecting the human expert's critical role,' but provides no survey, taxonomy of metrics, or quantitative evidence to support this empirical premise. This premise motivates the entire AI²L evaluation agenda, so it is load-bearing rather than a decorative observation. Please support it with a structured review of typical evaluation protocols in the cited application areas, or at minimum with several concrete cases where a published 'HIL' system is evaluated using AI-centered metrics that obscure the human contribution. Without such support, the paper's central diagnostic claim remains an assertion.
minor comments (4)
  1. [Throughout (terminology)] Using 'HIL' to denote a system in which the AI is in control and 'AI²L' to denote a system in which the human is in control is counterintuitive and will likely confuse readers, since 'human-in-the-loop' commonly implies human oversight. Please add an explicit note contrasting the paper's usage with the common reading, or consider relabeling the categories (e.g., 'AI-centered' versus 'human-centered').
  2. [Abstract] The sentence 'calling them HIL would be a misnomer, as they are quite the opposite, namely AI-in-the-loop systems, where the human is in control of the system' is self-contradictory on its face: if the human is in control, then a human is literally in the loop. Clarify that 'AI-in-the-loop' is meant to describe the AI's subordinate role rather than who holds control.
  3. [Table 1] The heading 'Collaborate (AI2L)?' contains a stray question mark, and the notation is inconsistent between 'AI²L' and 'AI 2L' across the paper; please standardize the symbol and its spacing.
  4. [References] The reference 'Wang, G. 2019' is incomplete, with no title or venue; please supply the full citation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the HIL/AI2L taxonomy is a definitional position argument, and its self-citations are illustrative, not load-bearing.

full rationale

This is a blue-sky position paper that proposes a conceptual taxonomy rather than deriving predictions from fitted data. The central distinction between HIL and AI2L is definitional, based on 'the role of decision-making authority and control' (Introduction) and illustrated with two examples; the recommendations about evaluation follow as normative consequences of that definitional framing, not as results that reduce to their own inputs. There are no equations, no fitted parameters, and no quantity is fit to a subset and then reported as a prediction. The paper's many self-citations (e.g., Schramowski et al. 2020; Stammer et al. 2024; Mathur et al. 2024) are used to point at existing interactive, knowledge-intensive, or explanatory learning work, but the central claim does not depend on any of them; no uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via citation. The paper also explicitly grounds AI2L in earlier human-AI symbiosis and explainable-advisable AI literature by others, which is ordinary related-work positioning rather than a circular derivation. The Discussion's admission that HIL and AI2L problem domains are 'not separated but nested' and that 'zooming in on a domain would result in a HIL problem and zooming out would give us an AI2L problem' is a limitation about the operationalization of control, not evidence that the taxonomy was assumed in order to prove itself. Overall, the argument is self-contained as a position statement, so no circular step can be exhibited.

Assumptions & free parameters 0 free parameters · 4 assumptions · 1 invented entities

The paper's contribution rests on a domain assumption about who controls decisions and an untested empirical claim about current evaluation practice. It introduces no new quantitative parameters or physical entities; the only invented entity is the AI2L label.

assumptions (4)
  • domain assumption The distinction between HIL and AI2L can be made based on who holds decision-making authority and control.
    Used throughout the paper, starting with the two examples in the Introduction and Figure 1; no operational measure of control is provided.
  • domain assumption Existing evaluation methods overemphasize the machine learning component and neglect the human expert's role.
    Asserted in the Abstract and repeated in the Discussion; supported only by citation to van Amsterdam et al. (2024), not by a survey or analysis.
  • domain assumption For AI2L systems, human-centric metrics such as ablations and outcome-based evaluation are more appropriate than accuracy, precision, or recall.
    Stated in the AI-in-the-loop section; prescriptive and untested.
  • domain assumption Machine learning systems treat inputs and outputs as non-human by default, making them agnostic to the human role.
    Inherited from interactive machine learning literature (Fails and Olsen; Amershi et al.) and used to motivate the AI2L reframing.
invented entities (1)
  • AI^2L (AI-in-the-loop) category
    purpose: To name systems where a human is in control and AI assists, distinct from HIL systems where AI is in control.
    Coined in this paper; it is a conceptual label with no falsifiable prediction attached, derived by renaming an existing distinction between automation and collaboration.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Human-in-the-loop or AI-in-the-loop? Automate or Collaborate?." pith.science (2026). https://pith.science/paper/UBO45G5L

@misc{pith2026241214232,
  author       = {Pith},
  title        = {Pith review of: Human-in-the-loop or AI-in-the-loop? Automate or Collaborate?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UBO45G5L}},
  note         = {Machine review of arXiv:2412.14232}
}
abstract

Human-in-the-loop (HIL) systems have emerged as a promising approach for combining the strengths of data-driven machine learning models with the contextual understanding of human experts. However, a deeper look into several of these systems reveals that calling them HIL would be a misnomer, as they are quite the opposite, namely AI-in-the-loop ($AI^2L$) systems, where the human is in control of the system, while the AI is there to support the human. We argue that existing evaluation methods often overemphasize the machine (learning) component's performance, neglecting the human expert's critical role. Consequently, we propose an $AI^2L$ perspective, which recognizes that the human expert is an active participant in the system, significantly influencing its overall performance. By adopting an $AI^2L$ approach, we can develop more comprehensive systems that faithfully model the intricate interplay between the human and machine components, leading to more effective and robust AI systems.

Figures

Figures reproduced from arXiv: 2412.14232 by the authors.

Figure 1
Figure 1. A comparison of human in the loop (left) and AI in the loop (right) systems. In human-in-the-loop systems, AI systems [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Recalibrating the Compass: Integrating Large Language Models into Classical Research Methods

    cs.AI 2025-05 accept novelty 4.0 of 10

    LLMs extend, rather than replace, classical social science methods, with a proposed three-tier bias framework for LLM-augmented surveys.

  2. Amplifying Minority Voices: AI-Mediated Devil's Advocate System for Inclusive Group Decision-Making

    cs.HC 2025-02 conditional novelty 4.0 of 10

    An LLM-powered devil's advocate that paraphrases minority members' private dissents as its own messages could reduce social pressure and increase opinion diversity in group decisions, but the paper provides no user st...

  3. Emphasizing Deliberation and Critical Thinking in an AI Hype World

    cs.HC 2025-07 unverdicted novelty 3.0 of 10

    An HCI researcher argues that resisting AI solutionism requires slow, deliberate use and critical thinking rather than outright bans.

Reference graph

Works this paper leans on

65 extracted references · 56 canonical work pages · cited by 3 Pith papers

  1. [1]

    C.; and Dietterich, T

    Altendorf, E.; Restificar, A. C.; and Dietterich, T. G. 2005. Learning from Sparse Data by Exploiting Monotonicity Constraints. In UAI , 18--26. AUAI Press

    work page 2005

  2. [2]

    B.; and Kulesza, T

    Amershi, S.; Cakmak, M.; Knox, W. B.; and Kulesza, T. 2014. Power to the People: The Role of Humans in Interactive Machine Learning . AI Magazine, 35(4): 105--120

    work page 2014

  3. [3]

    T.; and Mooney, R

    Baffes, P. T.; and Mooney, R. J. 1996. A novel application of theory refinement to student modeling. In Proceedings of the thirteenth national conference on Artificial intelligence-Volume 1, 403--408

    work page 1996

  4. [4]

    Beer, S. 1967. Cybernetics and management. London: English Universities Press

    work page 1967

  5. [5]

    Boutilier, C. 2002. A POMDP Formulation of Preference Elicitation Problems. In AAAI/IAAI , 239--246. AAAI Press / The MIT Press

    work page 2002

  6. [6]

    I.; Domshlak, C.; Hoos, H

    Boutilier, C.; Brafman, R. I.; Domshlak, C.; Hoos, H. H.; and Poole, D. 2004. CP-nets: A Tool for Representing and Reasoning with Conditional Ceteris Paribus Preference Statements. J. Artif. Intell. Res., 21: 135--191

    work page 2004

  7. [7]

    Chen, L.; and Pu, P. 2012. Critiquing-based recommenders: survey and emerging trends. User Model. User Adapt. Interact., 22(1-2): 125--150

    work page 2012

  8. [8]

    P.; Tong, Y.; and Ji, Q

    de Campos, C. P.; Tong, Y.; and Ji, Q. 2008. Constrained Maximum Likelihood Learning of Bayesian Networks for Facial Action Recognition. In ECCV (3) , volume 5304 of Lecture Notes in Computer Science, 168--181. Springer

    work page 2008

Show all 65 references
  1. [9]

    DeJong, G.; and Mooney, R. J. 1986. Explanation-Based Learning: An Alternative View. Mach. Learn., 1(2): 145--176

    1986

  2. [10]

    Delfosse, Q.; Sztwiertnia, S.; Rothermel, M.; Stammer, W.; and Kersting, K. 2024. Interpretable Concept Bottlenecks to Align Reinforcement Learning Agents. NeurIPS

    2024

  3. [11]

    A.; and Olsen Jr, D

    Fails, J. A.; and Olsen Jr, D. R. 2003. Interactive Machine Learning . In International Conference on Intelligent User Interfaces, 39--45

    2003

  4. [12]

    L.; and Shavlik, J

    Fung, G.; Mangasarian, O. L.; and Shavlik, J. W. 2002. Knowledge-Based Support Vector Machine Classifiers. In NIPS , 521--528. MIT Press

    2002

  5. [13]

    Griffith, S.; Subramanian, K.; Scholz, J.; Jr., C. L. I.; and Thomaz, A. L. 2013. Policy Shaping: Integrating Human Feedback with Reinforcement Learning. In NIPS , 2625--2633

    2013

  6. [14]

    Heckerman, D. 1998. A tutorial on learning with Bayesian networks. Learning in graphical models, 301--354

    1998

  7. [15]

    Johnson, B.; and Menzies, T. 2024. AI Over-Hype: A Dangerous Threat (and How to Fix It). IEEE Software, 41(6): 131--138

    2024

  8. [16]

    Kambhampati, S. 2020. Challenges of Human-Aware AI Systems AAAI Presidential Address. AI Mag. , 41(3): 3--17

    2020

  9. [17]

    Kambhampati, S.; Sreedharan, S.; Verma, M.; Zha, Y.; and Guan, L. 2022. Symbols as a Lingua Franca for Bridging Human-AI Chasm for Explainable and Advisable AI Systems. In AAAI , 12262--12267. AAAI Press

    2022

  10. [18]

    Kokel, H.; Odom, P.; Yang, S.; and Natarajan, S. 2020. A Unified Framework for Knowledge Intensive Gradient Boosting: Leveraging Human Experts for Noisy Sparse Domains. In AAAI , 4460--4468. AAAI Press

    2020

  11. [19]

    P.; Shabbeer, A.; Maclin, R.; and Shavlik, J

    Kunapuli, G.; Bennett, K. P.; Shabbeer, A.; Maclin, R.; and Shavlik, J. W. 2010. Online Knowledge-Based Support Vector Machines. In ECML/PKDD (2) , volume 6322 of Lecture Notes in Computer Science, 145--161. Springer

    2010

  12. [20]

    W.; and Natarajan, S

    Kunapuli, G.; Odom, P.; Shavlik, J. W.; and Natarajan, S. 2013. Guiding Autonomous Agents to Better Behaviors through Human Advice. In ICDM , 409--418. IEEE Computer Society

    2013

  13. [21]

    LeCun, Y.; Bengio, Y.; and Hinton, G. 2015. Deep learning. Nature, 521(7553): 436--444

    2015

  14. [22]

    Lipton, Z. C. 2018. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue, 16(3): 31--57

    2018

  15. [23]

    W.; Torrey, L.; Walker, T.; and Wild, E

    Maclin, R.; Shavlik, J. W.; Torrey, L.; Walker, T.; and Wild, E. W. 2005. Giving Advice about Preferred Actions to Reinforcement Learners Via Knowledge-Based Kernel Regression. In AAAI , 819--824. AAAI Press / The MIT Press

    2005

  16. [24]

    Mathur, S.; Antonucci, A.; and Natarajan, S. 2024. Knowledge Intensive Learning of Credal Networks. In UAI. PMLR

    2024

  17. [25]

    Mathur, S.; Gogate, V.; and Natarajan, S. 2023. Knowledge Intensive Learning of Cutset Networks. In UAI , volume 216 of Proceedings of Machine Learning Research, 1380--1389. PMLR

    2023

  18. [26]

    McCarthy, J. 1959. Programs with common sense. In Proc. Teddington Conference on the Mechanization of Thought Processes, 1959, 75--91

    1959

  19. [27]

    McCarthy, J. 1968. Programs with Common Sense. In Semantic information processing. The MIT Press

    1968

  20. [28]

    J.; Acklin, D.; and Scheuerman, J

    Michael, C. J.; Acklin, D.; and Scheuerman, J. 2020. On interactive machine learning and the potential of cognitive feedback. arXiv preprint arXiv:2003.10365

    2020 arXiv

  21. [29]

    Mitchell, T. 1997. Machine Learning. New York, NY, USA: McGraw-Hill, Inc., 1 edition. ISBN 0070428077, 9780070428072

    1997

  22. [30]

    Mohla, S.; Bagh, B.; and Guha, A. 2021. A material lens to investigate the gendered impact of the ai industry. In IJCAI 2021 Workshop on AI for Social Good

    2021

  23. [31]

    Mosqueira - Rey, E.; Hern \' a ndez - Pereira, E.; Alonso - R \' os, D.; Bobes - Bascar \' a n, J.; and Fern \' a ndez - Leal, \' A . 2023. Human-in-the-loop machine learning: a state of the art. Artif. Intell. Rev., 56(4): 3005--3054

    2023

  24. [32]

    H.; and Raedt, L

    Muggleton, S. H.; and Raedt, L. D. 1994. Inductive Logic Programming: Theory and Methods. J. Log. Program., 19/20: 629--679

    1994

  25. [33]

    Natarajan, S.; Kersting, K.; Khot, T.; and Shavlik, J. 2015. Boosted statistical relational learners: From benchmarks to data-driven medicine. Springer

    2015

  26. [34]

    Natarajan, S.; Picado, J.; Khot, T.; Kersting, K.; R \' e , C.; and Shavlik, J. W. 2014. Effectively Creating Weakly Labeled Training Examples via Approximate Domain Knowledge. In ILP , volume 9046 of Lecture Notes in Computer Science, 92--107. Springer

    2014

  27. [35]

    Y.; Harada, D.; and Russell, S

    Ng, A. Y.; Harada, D.; and Russell, S. 1999. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, volume 99, 278--287

    1999

  28. [36]

    B.; and Natarajan, S

    Odom, P.; Khot, T.; Porter, R. B.; and Natarajan, S. 2015. Knowledge-Based Probabilistic Logic Learning. In AAAI , 3564--3570. AAAI Press

    2015

  29. [37]

    Odom, P.; and Natarajan, S. 2016. Active Advice Seeking for Inverse Reinforcement Learning. In AAMAS , 512--520. ACM

    2016

  30. [38]

    Odom, P.; and Natarajan, S. 2018. Human-Guided Learning for Probabilistic Logic Models. Frontiers Robotics AI , 5: 56

    2018

  31. [39]

    Raedt, L. D. 2008. Logical and relational learning. Cognitive Technologies. Springer. ISBN 978-3-540-20040-6

    2008

  32. [40]

    H.; Ehrenberg, H

    Ratner, A.; Bach, S. H.; Ehrenberg, H. R.; Fries, J. A.; Wu, S.; and R \' e , C. 2017. Snorkel: Rapid Training Data Creation with Weak Supervision. Proc. VLDB Endow. , 11(3): 269--282

    2017

  33. [41]

    S.; Hughes, M

    Ross, A. S.; Hughes, M. C.; and Doshi - Velez, F. 2017. Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations. In Sierra, C., ed., Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence ( IJCAI ) , 2662--2670

    2017

  34. [42]

    Rudin, C. 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell., 1(5): 206--215

    2019

  35. [43]

    Schramowski, P.; Stammer, W.; Teso, S.; Brugger, A.; Herbert, F.; Shao, X.; Luigs, H.; Mahlein, A.; and Kersting, K. 2020. Making deep neural networks right for the right scientific reasons by interacting with their explanations. Nat. Mach. Intell., 2(8): 476--486

    2020

  36. [44]

    D.; danah boyd; Friedler, S

    Selbst, A. D.; danah boyd; Friedler, S. A.; Venkatasubramanian, S.; and Vertesi, J. 2019. Fairness and Abstraction in Sociotechnical Systems. In FAT , 59--68. ACM

    2019

  37. [45]

    Settles, B. 2009. Active Learning Literature Survey. Computer Sciences Technical Report 1648, University of Wisconsin--Madison

    2009

  38. [46]

    Y.; Amershi, S.; Chickering, D

    Simard, P. Y.; Amershi, S.; Chickering, D. M.; Pelton, A. E.; Ghorashi, S.; Meek, C.; Ramos, G.; Suh, J.; Verwey, J.; Wang, M.; et al. 2017. Machine teaching: A new paradigm for building machine learning systems. arXiv preprint arXiv:1707.06742

    2017 arXiv

  39. [47]

    Sreedharan, S.; Kulkarni, A.; and Kambhampati, S. 2022. Explainable human-AI interaction: A planning perspective. Springer Nature

    2022

  40. [48]

    Stammer, W.; Schramowski, P.; and Kersting, K. 2021. Right for the Right Concept: Revising Neuro-Symbolic Concepts by Interacting With Their Explanations. In CVPR , 3619--3629. Computer Vision Foundation / IEEE

    2021

  41. [49]

    Stammer, W.; W \"u st, A.; Steinmann, D.; and Kersting, K. 2024. Neural Concept Binder. NeurIPS

    2024

  42. [50]

    Steinwart, I.; and Christmann, A. 2008. Support Vector Machines, Book. Information Science and Statistics

    2008

  43. [51]

    Teso, S.; Alkan, \" O .; Stammer, W.; and Daly, E. 2023. Leveraging explanations in interactive machine learning: An overview. Frontiers Artif. Intell., 6

    2023

  44. [52]

    Teso, S.; and Kersting, K. 2019. Explanatory Interactive Machine Learning. In Conitzer, V.; Hadfield, G. K.; and Vallor, S., eds., Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society( AIES ) , 239--245

    2019

  45. [53]

    D.; Viappiani, P.; Teso, S.; Lepri, B.; and Passerini, A

    Toni, G. D.; Viappiani, P.; Teso, S.; Lepri, B.; and Passerini, A. 2024. Personalized Algorithmic Recourse with Preference Elicitation. Trans. Mach. Learn. Res

    2024

  46. [54]

    G.; and Shavlik, J

    Towell, G. G.; and Shavlik, J. W. 1994. Knowledge-based artificial neural networks. Artificial intelligence, 70(1-2): 119--165

    1994

  47. [55]

    van Amsterdam, W. A. C.; de Jong, P. A.; Verhoeff, J. J. C.; Leiner, T.; and Ranganath, R. 2024. From algorithms to action: improving patient care requires causality. BMC Medical Informatics Decis. Mak. , 24(3): 111

    2024

  48. [56]

    A.; and Witten, I

    Ware, M.; Frank, E.; Holmes, G.; Hall, M. A.; and Witten, I. H. 2001. Interactive machine learning: letting users build classifiers. Int. J. Hum. Comput. Stud., 55(3): 281--292

    2001

  49. [57]

    Wiener, N. 1988. The human use of human beings: Cybernetics and society. 320. Da capo press

    1988

  50. [58]

    W.; and Elkan, C

    Wiewiora, E.; Cottrell, G. W.; and Elkan, C. 2003. Principled Methods for Advising Reinforcement Learning Agents. In ICML , 792--799. AAAI Press

    2003

  51. [59]

    S.; and Kersting, K

    W \"u st, A.; Stammer, W.; Delfosse, Q.; Dhami, D. S.; and Kersting, K. 2024. Pix2Code: Learning to Compose Neural Visual Concepts as Programs. Uncertainty in Artificial Intelligence (UAI)

    2024

  52. [60]

    N.; Wang, W.; and Wang, S

    Yadavally, A.; Nguyen, T. N.; Wang, W.; and Wang, S. 2023. (Partial) Program Dependence Learning. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), 2501--2513. IEEE

    2023

  53. [61]

    Yang, S.; and Natarajan, S. 2013. Knowledge Intensive Learning: Combining Qualitative Constraints with Causal Independence for Parameter Learning in Probabilistic Models. In ECML/PKDD (2) , volume 8189 of Lecture Notes in Computer Science, 580--595. Springer

    2013

  54. [62]

    Zahedi, Z.; and Kambhampati, S. 2021. Human-AI symbiosis: A survey of current approaches. arXiv preprint arXiv:2103.09990

    2021 arXiv

  55. [63]

    Zahedi, Z.; Verma, M.; Sreedharan, S.; and Kambhampati, S. 2023. Trust-Aware Planning: Modeling Trust Evolution in Iterated Human-Robot Interaction. In HRI , 281--289. ACM

    2023

  56. [64]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  57. [65]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.