Pith. sign in

REVIEW 3 major objections 4 minor 80 references

Human-in-the-Loop User Feedback Affects Perceived Accuracy and Trust, but Task Subjectivity Matters

T0 review · 3 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Task subjectivity—not the act of giving feedback—decides whether users trust an AI more or less

desk verdict A careful, transparently reported three-experiment extension, but the title and abstract claim that task subjectivity is the moderator goes beyond the evidence; worth reviewing, needs revisions. read the letter →

arxiv 2607.17548 v1 pith:6JABUSXQ submitted 2026-07-20 cs.HC cs.AI

Donald R. Honeycutt , Mahsan Nourani , Eric D. Ragan This is my paper · ORCID
classification cs.HCcs.AI
keywords trustinautomationhuman-in-the-loopperceivedaccuracyfeedbacksubjectivityinteractivemachinelearninguserexplainableAItaskcontext
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that task subjectivity determines whether asking users for feedback helps or hurts their trust in an intelligent system. In an object-detection context where feedback corrects objectively wrong outputs, the authors find that interactive feedback leads users to perceive lower accuracy and to trust the system less, and that believing the system updates from their feedback does not offset the effect. In a text-classification context where feedback expresses subjective judgments about which words matter, no such negative bias appears; instead users in all conditions perceive the system as getting more accurate over time even though its true accuracy is constant. The paper argues the objective framing turns feedback into 'fix my mistakes,' while the subjective framing turns it into 'learn my thinking,' and recommends that human-in-the-loop features be framed as user agency rather than error correction.

What carries the argument

The central object is task subjectivity—whether the feedback has a single objectively correct answer (a face is or is not in a box) or is open to judgment (which words best justify a topic label). The experiments operationalize it by moving the same feedback interaction from bounding-box correction in images to explanation-word re-ranking in text, while holding the simulated system's true accuracy at 80 percent and telling participants the model was updating when it was not. The mechanism the authors propose to explain their results is error salience: objective errors are obvious and memorable, so correcting them makes the system feel worse than it is; subjective feedback makes users compare

What would settle it

Use one task and one interface in which the same feedback action is described to randomly assigned participants either as fixing objectively wrong outputs or as tuning the system to their personal judgment, keeping explanations, examples, and true accuracy identical; if the trust difference appears in both framings or in neither, task subjectivity is not the causal driver.

Watch

Extended reading notes

Core claim

The paper's central claim is that the meaning users assign to giving feedback—correcting an objectively wrong output versus expressing a subjective judgment—determines whether human-in-the-loop interaction helps or hurts their view of an intelligent system. In an object-detection task where bounding boxes either do or do not contain a face, participants who gave interactive corrections rated the system as less accurate over time and trusted it less than did participants who only gave a yes/no accuracy check; whether they believed the system was updating from their feedback made no difference. In a text-classification task where participants adjusted which highlighted words best explained a t

Load-bearing premise

The load-bearing premise is that the different results came from whether the feedback felt objective or subjective, but the studies never directly controlled or measured that feeling, and the objective and subjective tasks also differed in explanation style, example order, and participant pool—a limitation the paper itself acknowledges.

Editorial extensions

If this is right

  • In domains where system errors are obviously wrong to users, adding interactive error-correction feedback can lower perceived accuracy and trust even if the model genuinely improves from the feedback.
  • Telling users their feedback is being used does not remove the negative bias; the negative effect comes from the act of correcting errors, not from uncertainty about whether updates happen.
  • In subjective domains, explanation-based feedback can avoid the distrust penalty, but it comes with a different risk: users may come to believe the system is improving over time when its accuracy is flat, which can lead to over-reliance.
  • Designers who want human-in-the-loop features without biasing user trust should frame feedback as adjusting the system toward the user's judgment, not as correcting objective errors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A clean test of the paper's explanation would hold the task and interface constant and randomly frame the identical feedback action as either correcting errors or expressing preference; the paper's own limitation section says subjectivity was not directly controlled for.
  • If error salience is the underlying driver, then interface choices that reduce how long users dwell on each error—such as batching corrections or showing aggregate accuracy alongside mistakes—might blunt the negative bias in objective tasks.
  • The pattern suggests a calibration dilemma: making feedback feel subjective may increase satisfaction and perceived accuracy, but if perceived accuracy outruns true accuracy, designers have simply traded distrust for automation bias.
  • Because all three studies used a simulated, non-updating system and one-session tasks, the findings most directly apply to first impressions; whether the effects persist after users see real model updates or across longer use remains untested.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper reports three between-subjects experiments on how providing human-in-the-loop (HITL) feedback affects trust and perceived accuracy. Experiment 1 (n=107) uses a simulated object-detection task with binary vs. interactive feedback and an update-belief manipulation; Experiments 2 and 3 (n=144 and n=94) use a text-classification task with no, decision-based, or explanation-based feedback. The paper's central claim is that in objective-feedback contexts HITL feedback lowers trust and perceived accuracy, while in subjective-feedback contexts no such negative bias occurs, and that the two contexts lead to different perceived accuracy trajectories over time.

Significance. If the central claim were fully supported, the paper would provide a useful design guideline for interactive ML: framing feedback as objective error correction can bias users negatively, whereas subjective feedback may preserve or improve perceptions. The studies have genuine strengths: the simulated systems hold true accuracy constant, avoiding the common confound of improved system performance; Experiment 3 provides a replication with a different population and counterbalanced ordering; effect sizes are reported; and the authors are transparent about the subjectivity limitation in §6.4. However, the headline causal claim is not established by the reported statistics and design, for reasons detailed below. The empirical pattern is interesting and worth reporting, but the paper currently overstates what can be concluded.

major comments (3)
  1. [§3.2.1, §3.2.3, Abstract] The abstract states that in a context with an objectively correct answer, HITL feedback 'lowered both participants' trust in the system and their perception of system accuracy.' Experiment 1 does not support this. For perceived accuracy, the feedback-type effect was F(1,103)=3.71, p=0.050, which the authors themselves report as non-significant; for trust, the two-way ANOVA found no significant effects of feedback type or feedback usage. The only significant feedback-type effect was the retrospective perception-of-change measure (§3.2.2), which is not the same as lower perceived accuracy or lower trust. Additionally, the abstract's 'regardless of whether the system accuracy improved in response to their feedback' is not tested in this paper: system accuracy was constant in all conditions and participants were deceived about updates. The abstract and the conclusions in §7 should be reworde
  2. [§6.4, §4.2.1, §5.1] The central claim that task subjectivity causally moderates the effect of HITL feedback is based on comparing separate studies that differ on many dimensions. Experiment 1 used bounding-box correction on images; Experiments 2 and 3 used highlighted-word editing in text. The studies also differ in trials per round (30 vs. 15), stimulus ordering (fixed vs. counterbalanced), and participant population (MTurk vs. University of Florida students), and only the text condition includes a visible explanation of the system's reasoning. The authors acknowledge in §6.4 that 'subjectivity was not a variable that was directly controlled for' and that the text case 'has more of an element of explainability.' Without a direct manipulation of perceived subjectivity within a constant paradigm, or at minimum a measurement of perceived subjectivity, the observed cross-experiment differences cannot be attrib
  3. [§4.3.1, §5.3.1] The conclusion that 'no such negative bias was observed' in the subjective contexts rests on non-significant main effects of feedback condition on perceived accuracy (Experiment 2: F(2,137)=0.951, p=0.389; Experiment 3: F(2,91)=0.871, p=0.422). A non-significant p-value is not evidence for the absence of an effect unless accompanied by equivalence testing or a Bayes-factor analysis. Because the paper's central comparison is between a significant effect in one study and null effects in two others, the asymmetry should be quantified rather than inferred from p-values alone.
minor comments (4)
  1. [§4.2.1] The text says 'As in Experiment 3, this decision to deceive participants...' but Experiment 3 is introduced later; this should likely refer to Experiment 2 or Experiment 1.
  2. [§7] Typo: 'without into their systems' should read 'without negatively biasing their users' or similar.
  3. [Abstract, §7] The use of 'distrust' versus 'mistrust' is confusing. In the subjective context participants perceived the system as improving over time; describing this as 'mistrust' seems inconsistent with the usual meaning of mistrust as insufficient trust. Clarify the intended distinction.
  4. [Figure 3] In the provided manuscript version, Figure 3 appears to contain duplicate panels. Please verify the final figure.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's conclusions are empirical cross-experiment comparisons, not derivation-from-data; the reuse of prior self-published data is transparent, and the acknowledged subjectivity confound is a validity limitation, not a definitional loop.

full rationale

This paper contains no equations, no fitted parameters, and no formal model from which results are derived. Its central claim is an empirical moderator hypothesis: that objective feedback produces negative trust/accuracy bias while subjective feedback does not. That claim is inferred from three between-subjects experiments, not constructed from the input measures. The self-citation to the authors' prior HCI paper [26] is transparent data reuse: the paper states, "For the conditions that mirrored conditions from [26]—binary feedback without update and interactive feedback with update—the original data was retained and no new data was collected." This is not a definitional loop; the current paper adds new conditions and two new experiments, and the analyses are reported with standard inferential statistics. The paper's own limitation section weakens the causal moderator claim, but this is an internal-validity threat rather than circularity: "However, these were separate studies and subjectivity was not a variable that was directly controlled for. Additionally, the more subjective text case has more of an element of explainability than the more objective image case—a bounding box around a face is less of an explanation than it is part of the classification—which may have also contributed to the differences in observed effects." That statement acknowledges confounds (explanation format, domain, population, task length), not a reduction of the conclusion to its inputs. The abstract's stronger wording also somewhat overstates Experiment 1's own statistics (trust ANOVA non-significant, perceived-accuracy effect p = 0.050), but overstatement of evidence is not circularity. No step in the paper reduces to a definition, a fitted-input-as-prediction, or an imported uniqueness theorem. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No mathematical derivations or fitted models; the 'free parameters' are hand-selected experimental constants. The central claim rests on assumptions about measurement validity, deception effectiveness, and the causal role of subjectivity.

free parameters (2)
  • Constant simulated system accuracy = 80% in all rounds of all three experiments
    Hand-selected to balance credibility against visible errors; not fitted, but perceived-accuracy judgments are made relative to this number.
  • Participant quality-check threshold = 75% correct on error/no-error judgments
    Hand-selected exclusion rule; 4, 5, and 10 participants removed in Experiments 1-3. The cutoff is arbitrary but not fitted to the outcome.
assumptions (4)
  • domain assumption Perceived accuracy ratings and trust scales measure the same latent trust construct (following Yin et al. [76]).
    Used to interpret percentage estimates as trust-relevant outcomes; if invalid, accuracy-estimate results do not speak to trust.
  • domain assumption Participants believed the simulated system updated from their feedback.
    Only checked in Experiment 1 (mean 4.71/7 agreement); Experiments 2 and 3 rely on a 45-second pause plus instruction without a manipulation check.
  • ad hoc to paper The image task is objectively correct and the word-selection task is subjective.
    This classification carries the entire "subjectivity matters" interpretation, but perceived subjectivity was not measured and the tasks differ in many other ways (§6.4).
  • domain assumption Excluded low-accuracy participants are careless responders rather than a systematically different population.
    The 75% error-judgment threshold is applied post hoc; it could alter feedback-condition samples.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Human-in-the-Loop User Feedback Affects Perceived Accuracy and Trust, but Task Subjectivity Matters." pith.science (2026). https://pith.science/paper/6JABUSXQ

@misc{pith2026260717548,
  author       = {Pith},
  title        = {Pith review of: Human-in-the-Loop User Feedback Affects Perceived Accuracy and Trust, but Task Subjectivity Matters},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6JABUSXQ}},
  note         = {Machine review of arXiv:2607.17548}
}
read the original abstract

While ML can produce complex models beyond those that a human could produce manually, incorporating human input can often improve performance beyond purely data-driven models. While this feedback could come from system designers or domain experts, in many cases, the end users who regularly use the system will naturally develop an understanding of its flaws and desire the ability to change the system's behavior based on their knowledge. While soliciting feedback from end users can result in significant model improvement over time, introducing these feedback techniques can also affect several human factors-such as trust or perception of system accuracy-that are not yet fully understood and have different effects reported in the existing literature. Therefore, we sought to build on the existing research to further explore how the act of providing feedback can affect user understanding of an intelligent system and its accuracy in different contexts. We present three controlled experiments that study the effects of interactive feedback collections on user impressions in domains with objective and subjective feedback. The results show that in a context where there is an objectively correct answer, providing HITL feedback lowered both participants' trust in the system and their perception of system accuracy, regardless of whether the system accuracy improved in response to their feedback. However, when the feedback being provided involved subjective opinion, no such negative bias was observed. Furthermore, in the objective context, participants distrusted the system over time, whereas participants in the subjective context mistrusted the system over time. These results highlight the importance of considering the effects of allowing different types of end-user feedback on user trust when designing intelligent systems.

Figures

Figures reproduced from arXiv: 2607.17548 by the authors.

Figure 1
Figure 1. In the interactive feedback conditions of Experiment 1, participants could delete existing bounding boxes or click-and-drag to create new ones. In this example, the left image shows a system error, and the right shows a version after interactive correction.1 RQ1: Does belief that user feedback is being used to dynamically update system behavior affect user trust or perceived system accuracy? RQ2: Does providing more… view at source ↗
Figure 2
Figure 2. An overview of the experimental procedure shared across the three experiments. Each participant interacts with only one experiment. After each round of the task, participants rate system accuracy to capture how perception of accuracy changes over time. This is repeated for three task rounds before moving on to post-study questionnaires. 3.1.2 Experimental Design. Because our main metrics—perception of model accuracy… view at source ↗
Figure 3
Figure 3. Perceived system accuracy (Percentage) across the three task rounds (R1, R2, R3) [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Experiment 1 post-study questionnaire results. [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Overview of the Experiment 2 and 3 interface. The system highlights the top words within the text sample (A) that contribute to the classification (B). Participants in the explanation-based feedback condition could change the highlights by clicking on the words. The sy…
Figure 7
Figure 7. Figure 7: Averaged responses to the three trust scale statements described in Section 3.2.3 on a 7-point Likert scale. [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

80 extracted references · 7 linked inside Pith

  1. [26]

    Soliciting human-in-the-loop user feedback for interactive machine learning reduces user trust and impressions of model accuracy

    HONEYCUTT, D., NOURANI, M.,ANDRAGAN, E. Soliciting human-in-the-loop user feedback for interactive machine learning reduces user trust and impressions of model accuracy. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (2020), vol. 8, pp. 63–72

    2020

  2. [1]

    B.,ANDKULESZA, T

    AMERSHI, S., CAKMAK, M., KNOX, W. B.,ANDKULESZA, T. Power to the people: The role of humans in interactive machine learning. Ai Magazine 35, 4 (2014), 105–120

    2014

  3. [2]

    D., CHERNOVA, S., VELOSO, M.,ANDBROWNING, B

    ARGALL, B. D., CHERNOVA, S., VELOSO, M.,ANDBROWNING, B. A survey of robot learning from demonstration. Robotics and autonomous systems 57, 5 (2009), 469–483

    2009

  4. [3]

    Machine learning forecasts of risk to inform sentencing decisions

    BERK, R.,ANDHYATT, J. Machine learning forecasts of risk to inform sentencing decisions. Federal Sentencing Reporter 27, 4 (2015), 222–228

    2015

  5. [4]

    R.,ANDMAXWELL, W

    BERTRAND, A., BELLOUM, R., EAGAN, J. R.,ANDMAXWELL, W. How cognitive biases affect xai-assisted decision-making: A systematic review. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society (2022), pp. 78–91

    2022

  6. [5]

    Does projection into use improve trust and exploration? an example with a cruise control system

    CAHOUR, B.,ANDFORZY, J.-F. Does projection into use improve trust and exploration? an example with a cruise control system. Safety science 47, 9 (2009), 1260–1270

    2009

  7. [6]

    CAKMAK, M.,ANDTHOMAZ, A. L. Eliciting good teaching from humans for machine learners. Artificial Intelligence 217 (2014), 198–215

    2014

  8. [7]

    Explanatory and actionable debugging for machine learning: A tableqa demonstration

    CHO, M., LEE, G.,ANDHWANG, S.-W. Explanatory and actionable debugging for machine learning: A tableqa demonstration. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (2019), pp. 1333–1336

    2019

Show all 80 references
  1. [8]

    D., MACH, M.,ANDSIN ˇCÁK, P

    ˇCÍK, I., RASAMOELINA, A. D., MACH, M.,ANDSIN ˇCÁK, P. Explaining deep neural network using layer-wise relevance propagation and integrated gradients. In 2021 IEEE 19th World Symposium on Applied Machine Intelligence and Informatics (SAMI) (2021), IEEE, pp. 000381–000386

    2021

  2. [9]

    Improving generalization with active learning

    COHN, D., ATLAS, L.,ANDLADNER, R. Improving generalization with active learning. Machine learning 15, 2 (1994), 201–221

    1994

  3. [10]

    A., GHAHRAMANI, Z.,ANDJORDAN, M

    COHN, D. A., GHAHRAMANI, Z.,ANDJORDAN, M. I. Active learning with statistical models. Journal of artificial intelligence research 4 (1996), 129–145

    1996

  4. [11]

    J., SIMMONS, J

    DIETVORST, B. J., SIMMONS, J. P.,ANDMASSEY, C. Algorithm aversion: people erroneously avoid algorithms after seeing them err. Journal of experimental psychology: General 144, 1 (2015), 114

    2015

  5. [12]

    J., SIMMONS, J

    DIETVORST, B. J., SIMMONS, J. P.,ANDMASSEY, C. Overcoming algorithm aversion: People will use imperfect algorithms if they can (even slightly) modify them. Management Science 64, 3 (2018), 1155–1170

    2018

  6. [13]

    Towards a rigorous science of interpretable machine learning

    DOSHI-VELEZ, F.,ANDKIM, B. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608 (2017)

    2017 arXiv

  7. [14]

    E., GALLANT, S

    EHRMANN, D. E., GALLANT, S. N., NAGARAJ, S., GOODFELLOW, S. D., EYTAN, D., GOLDENBERG, A.,ANDMAZWI, M. L. Evaluating and reducing cognitive load should be a priority for machine learning in healthcare. Nature medicine 28, 7 (2022), 1331–1333

    2022

  8. [15]

    Incremental learning of concept drift in nonstationary environments

    ELWELL, R.,ANDPOLIKAR, R. Incremental learning of concept drift in nonstationary environments. IEEE Transactions on Neural Networks 22, 10 (2011), 1517–1531

    2011

  9. [16]

    Constructing explainable classifiers from the start—enabling human-in-the loop machine learning

    ESTIVILL-CASTRO, V., GILMORE, E.,ANDHEXEL, R. Constructing explainable classifiers from the start—enabling human-in-the loop machine learning. Information 13, 10 (2022), 464

    2022

  10. [17]

    A.,ANDOLSENJR, D

    FAILS, J. A.,ANDOLSENJR, D. R. Interactive machine learning. In Proceedings of the 8th international conference on Intelligent user interfaces (2003), pp. 39–45

    2003

  11. [18]

    Incremental learning., 2009

    GENG, X.,ANDSMITH-MILES, K. Incremental learning., 2009

    2009

  12. [19]

    V., ZHANG, Y., BELLAMY, R.,ANDMUELLER, K

    GHAI, B., LIAO, Q. V., ZHANG, Y., BELLAMY, R.,ANDMUELLER, K. Explainable active learning (xal): An empirical study of how local explanations impact annotator experience. arXiv preprint arXiv:2001.09219 (2020)

    2001 arXiv

  13. [20]

    GODDARD, K., ROUDSARI, A.,ANDWYATT, J. C. Automation bias: a systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association 19, 1 (2012), 121–127

    2012

  14. [21]

    Interactive recommender systems: A survey of the state of the art and future research challenges and opportunities

    HE, C., PARRA, D.,ANDVERBERT, K. Interactive recommender systems: A survey of the state of the art and future research challenges and opportunities. Expert Systems with Applications 56 (2016), 9–27. Manuscript submitted to ACM 24 Donald R. Honeycutt, Mahsan Nourani, and Eric D. Ragan

    2016

  15. [22]

    A.,ANDBASHIR, M

    HOFF, K. A.,ANDBASHIR, M. Trust in automation: Integrating empirical evidence on factors that influence trust. Human factors 57, 3 (2015), 407–434

    2015

  16. [23]

    Measuring trust in the xai context

    HOFFMAN, R., MUELLER, S., KLEIN, G.,ANDLITMAN, J. Measuring trust in the xai context

  17. [24]

    R., JOHNSON, M., BRADSHAW, J

    HOFFMAN, R. R., JOHNSON, M., BRADSHAW, J. M.,ANDUNDERBRINK, A. Trust in automation. IEEE Intelligent Systems 28, 1 (2013), 84–88

    2013

  18. [25]

    C., PINTEA, C.-M.,ANDPALADE, V

    HOLZINGER, A., PLASS, M., HOLZINGER, K., CRI ¸ SAN, G. C., PINTEA, C.-M.,ANDPALADE, V. Towards interactive machine learning (iml): applying ant colony algorithms to solve the traveling salesman problem with the human-in-the-loop approach. In International Conference on Availab...

    2016

  19. [27]

    L., UMADA, T., AHMED, N

    IUZZOLINO, M. L., UMADA, T., AHMED, N. R.,ANDSZAFIR, D. A. In automation we trust: investigating the role of uncertainty in active learning systems. arXiv preprint arXiv:2004.00762 (2020)

    2004 arXiv

  20. [28]

    M.,ANDDRURY, C

    JIAN, J.-Y., BISANTZ, A. M.,ANDDRURY, C. G. Foundations for an empirically determined scale of trust in automated systems. International journal of cognitive ergonomics 4, 1 (2000), 53–71

    2000

  21. [29]

    How do different levels of user control affect cognitive load and acceptance of recommendations? In Jin, Y .,Cardoso, B

    JIN, Y., CARDOSO, B.,ANDVERBERT, K. How do different levels of user control affect cognitive load and acceptance of recommendations? In Jin, Y .,Cardoso, B. and Verbert, K., 2017, August. How do different levels of user control affect cognitive load and acceptance of recommend...

    2017

  22. [30]

    A., SCHWEIGER, D

    KORSGAARD, M. A., SCHWEIGER, D. M.,ANDSAPIENZA, H. J. Building commitment, attachment, and trust in strategic decision-making teams: The role of procedural justice. Academy of Management journal 38, 1 (1995), 60–84

    1995

  23. [31]

    Openimages: A public dataset for large-scale multi-label and multi-class image classification

    KRASIN, I., DUERIG, T., ALLDRIN, N., FERRARI, V., ABU-EL-HAIJA, S., KUZNETSOVA, A., ROM, H., UIJLINGS, J., POPOV, S., KAMALI, S., MALLOCI, M., PONT-TUSET, J., VEIT, A., BELONGIE, S., GOMES, V., GUPTA, A., SUN, C., CHECHIK, G., CAI, D., FENG, Z., NARAYANAN, D.,ANDMURPHY, K. Ope...

    2017

  24. [32]

    Principles of explanatory debugging to personalize interactive machine learning

    KULESZA, T., BURNETT, M., WONG, W.-K.,ANDSTUMPF, S. Principles of explanatory debugging to personalize interactive machine learning. In Proceedings of the 20th international conference on intelligent user interfaces (2015), pp. 126–137

    2015

  25. [33]

    Explanatory debugging: Supporting end-user debugging of machine-learned programs

    KULESZA, T., STUMPF, S., BURNETT, M., WONG, W.-K., RICHE, Y., MOORE, T., OBERST, I., SHINSEL, A.,ANDMCINTOSH, K. Explanatory debugging: Supporting end-user debugging of machine-learned programs. In 2010 IEEE Symposium on Visual Languages and Human-Centric Computing (2010), IEE...

    2010

  26. [34]

    Too much, too little, or just right? ways explanations impact end users’ mental models

    KULESZA, T., STUMPF, S., BURNETT, M., YANG, S., KWAN, I.,ANDWONG, W.-K. Too much, too little, or just right? ways explanations impact end users’ mental models. In 2013 IEEE Symposium on visual languages and human centric computing (2013), IEEE, pp. 3–10

    2013

  27. [35]

    M., OBERST, I.,ANDKO, A

    KULESZA, T., WONG, W.-K., STUMPF, S., PERONA, S., WHITE, R., BURNETT, M. M., OBERST, I.,ANDKO, A. J. Fixing the program my computer learned: Barriers for end users, challenges for the machine. In Proceedings of the 14th international conference on Intelligent user interfaces (...

    2009

  28. [36]

    Active learning query strategies for classification, regression, and clustering: a survey

    KUMAR, P.,ANDGUPTA, A. Active learning query strategies for classification, regression, and clustering: a survey. Journal of Computer Science and Technology 35, 4 (2020), 913–945

    2020

  29. [37]

    The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale

    KUZNETSOVA, A., ROM, H., ALLDRIN, N., UIJLINGS, J., KRASIN, I., PONT-TUSET, J., KAMALI, S., POPOV, S., MALLOCI, M., KOLESNIKOV, A., DUERIG, T.,ANDFERRARI, V. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale....

    2020

  30. [38]

    D.,ANDSEE, K

    LEE, J. D.,ANDSEE, K. A. Trust in automation: Designing for appropriate reliance. Human factors 46, 1 (2004), 50–80

    2004

  31. [39]

    J., MCALLISTER, D

    LEWICKI, R. J., MCALLISTER, D. J.,ANDBIES, R. J. Trust and distrust: New relationships and realities. Academy of management Review 23, 3 (1998), 438–458

    1998

  32. [40]

    D.,ANDGALE, W

    LEWIS, D. D.,ANDGALE, W. A. A sequential algorithm for training text classifiers. In SIGIR’94 (1994), Springer, pp. 3–12

    1994

  33. [41]

    Human-in-the-loop data integration

    LI, G. Human-in-the-loop data integration. Proceedings of the VLDB Endowment 10, 12 (2017), 2006–2017

    2017

  34. [42]

    Explanations for human-on-the-loop: A probabilistic model checking approach

    LI, N., ADEPU, S., KANG, E.,ANDGARLAN, D. Explanations for human-on-the-loop: A probabilistic model checking approach. In Proceedings of the IEEE/ACM 15th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (2020), pp. 181–187

    2020

  35. [43]

    Measuring human-computer trust

    MADSEN, M.,ANDGREGOR, S. Measuring human-computer trust. In 11th australasian conference on information systems (2000), vol. 53, Citeseer, pp. 6–8

    2000

  36. [44]

    E., SHADBOLT, N

    MIDDLETON, S. E., SHADBOLT, N. R.,ANDDEROURE, D. C. Capturing interest through inference and visualization: Ontological user profiling in recommender systems. In Proceedings of the 2nd international conference on Knowledge capture (2003), pp. 62–69

    2003

  37. [45]

    MOHSENI, S., ZAREI, N.,ANDRAGAN, E. D. A survey of evaluation methods and measures for interpretable machine learning. ACM Transactions on Interactive Intelligent Systems (2018)

    2018

  38. [46]

    Human-in-the-loop machine learning: A state of the art

    MOSQUEIRA-REY, E., HERNÁNDEZ-PEREIRA, E., ALONSO-RÍOS, D., BOBES-BASCARÁN, J.,ANDFERNÁNDEZ-LEAL, Á. Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review 56, 4 (2023), 3005–3054

    2023

  39. [47]

    NOURANI, M., HASHKY, A.,ANDRAGAN, E. D. User profiling in human-ai design: an empirical case study of anchoring bias, individual differences, and ai attitudes. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (2024), vol. 12, pp. 137–146

    2024

  40. [48]

    R., BLOCK, J

    NOURANI, M., HONEYCUTT, D. R., BLOCK, J. E., ROY, C., RAHMAN, T., RAGAN, E. D.,ANDGOGATE, V. Investigating the importance of first Manuscript submitted to ACM Human-in-the-Loop User Feedback Affects Perceived Accuracy and Trust, but Task Subjectivity Matters 25 impressions and...

    2020

  41. [49]

    NOURANI, M., KABIR, S., MOHSENI, S.,ANDRAGAN, E. D. The effects of meaningful and meaningless explanations on trust and perceived system accuracy in intelligent systems. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (2019), vol. 7, pp. 97–105

    2019

  42. [50]

    T.,ANDRAGAN, E

    NOURANI, M., KING, J. T.,ANDRAGAN, E. D. The role of domain expertise in user trust and the impact of first impressions with intelligent systems. In Eighth AAAI Conference on Human Computation and Crowdsourcing (2020)

    2020

  43. [51]

    E., HONEYCUTT, D

    NOURANI, M., ROY, C., BLOCK, J. E., HONEYCUTT, D. R., RAHMAN, T., RAGAN, E. D.,ANDGOGATE, V. On the importance of user backgrounds and impressions: Lessons learned from interactive ai applications. ACM Transactions on Interactive Intelligent Systems 12, 4 (2022), 1–29

    2022

  44. [52]

    D., RUOZZI, N.,ANDGOGATE, V

    NOURANI, M., ROY, C., RAHMAN, T., RAGAN, E. D., RUOZZI, N.,ANDGOGATE, V. Don’t explain without verifying veracity: An evaluation of explainable ai with video activity recognition. arXiv preprint arXiv:2005.02335 (2020)

    2005 arXiv

  45. [53]

    Humans and automation: Use, misuse, disuse, abuse

    PARASURAMAN, R.,ANDRILEY, V. Humans and automation: Use, misuse, disuse, abuse. Human factors 39, 2 (1997), 230–253

    1997

  46. [54]

    User-controllable personalization: A case study with setfusion

    PARRA, D.,ANDBRUSILOVSKY, P. User-controllable personalization: A case study with setfusion. International Journal of Human-Computer Studies 78 (2015), 43–67

    2015

  47. [55]

    Systemer: A human-in-the-loop system for explainable entity resolution

    QIAN, K., POPA, L.,ANDSEN, P. Systemer: A human-in-the-loop system for explainable entity resolution

  48. [56]

    L.,ANDMEI, V

    RANI, N., CHU, S. L.,ANDMEI, V. R. Investigating the effects of different levels of user control on the effectiveness of context-aware recommender systems for web-based search. In CHI Conference on Human Factors in Computing Systems Extended Abstracts (2022), pp. 1–6

    2022

  49. [57]

    M., LING, K., TASSONE, R

    RASHID, A. M., LING, K., TASSONE, R. D., RESNICK, P., KRAUT, R.,ANDRIEDL, J. Motivating participation by displaying the value of contribution. In Proceedings of the SIGCHI conference on Human Factors in computing systems (2006), pp. 955–958

    2006

  50. [58]

    why should i trust you?

    RIBEIRO, M. T., SINGH, S.,ANDGUESTRIN, C. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining (2016), pp. 1135–1144

    2016

  51. [59]

    R., BLOCK, J

    ROY, C., NOURANI, M., HONEYCUTT, D. R., BLOCK, J. E., RAHMAN, T., RAGAN, E. D., RUOZZI, N.,ANDGOGATE, V. Explainable activity recognition in videos: Lessons learned. Applied AI Letters 2, 4 (2021), e59

    2021

  52. [60]

    From theories to queries: Active learning in practice

    SETTLES, B. From theories to queries: Active learning in practice. In Active Learning and Experimental Design workshop In conjunction with AISTATS2010 (2011), pp. 1–18

    2011

  53. [61]

    Online structured prediction via coactive learning

    SHIVASWAMY, P.,ANDJOACHIMS, T. Online structured prediction via coactive learning. arXiv preprint arXiv:1205.4213 (2012)

    2012 arXiv

  54. [62]

    Building trust in artificial intelligence, machine learning, and robotics

    SIAU, K.,ANDWANG, W. Building trust in artificial intelligence, machine learning, and robotics. Cutter Business Technology Journal 31, 2 (2018), 47–53

    2018

  55. [63]

    Y., DONG, J., DUFFY, V

    STEPHANIDIS, C., SALVENDY, G., ANTONA, M., CHEN, J. Y., DONG, J., DUFFY, V. G., FANG, X., FIDOPIASTIS, C., FRAGOMENI, G., FU, L. P.,ET AL. Seven hci grand challenges. International Journal of Human–Computer Interaction 35, 14 (2019), 1229–1269

    2019

  56. [64]

    Integrating rich user feedback into intelligent user interfaces

    STUMPF, S., SULLIVAN, E., FITZHENRY, E., OBERST, I., WONG, W.-K.,ANDBURNETT, M. Integrating rich user feedback into intelligent user interfaces. In Proceedings of the 13th international conference on Intelligent user interfaces (2008), pp. 50–59

    2008

  57. [65]

    M., HORSTMANN, A

    SZCZUKA, J. M., HORSTMANN, A. C., SZYMCZYK, N., STRATHMANN, C., ARTELT, A., MAVRINA, L.,ANDKRÄMER, N. Let me explain what i did or what i would have done: An empirical study on the effects of explanations and person-likeness on trust in and understanding of algorithms. In Proc...

    2024

  58. [66]

    why should i trust interactive learners?

    TESO, S.,ANDKERSTING, K. " why should i trust interactive learners?" explaining interactive queries of classifiers to users. arXiv preprint arXiv:1805.08578 (2018)

    2018 arXiv

  59. [67]

    Explanatory interactive machine learning

    TESO, S.,ANDKERSTING, K. Explanatory interactive machine learning. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society (2019), pp. 239–245

    2019

  60. [68]

    Support vector machine active learning for image retrieval

    TONG, S.,ANDCHANG, E. Support vector machine active learning for image retrieval. In Proceedings of the ninth ACM international conference on Multimedia (2001), pp. 107–118

    2001

  61. [69]

    Explainable artificial intelligence for predictive maintenance applications using a local surrogate model

    TORCIANTI, A.,ANDMATZKA, S. Explainable artificial intelligence for predictive maintenance applications using a local surrogate model. In 2021 4th International Conference on Artificial Intelligence for Industries (AI4I) (2021), IEEE, pp. 86–88

    2021

  62. [70]

    VAN DENBOS, K., VERMUNT, R.,ANDWILKE, H. A. The consistency rule and the voice effect: The influence of expectations on procedural fairness judgements and performance. European Journal of Social Psychology 26, 3 (1996), 411–428

    1996

  63. [71]

    A human in the loop corrective maintenance methodology using cross domain engineering data of mechatronic systems

    VATHOOPAN, M., BRANDENBOURGER, B.,ANDZOITL, A. A human in the loop corrective maintenance methodology using cross domain engineering data of mechatronic systems. In 2016 IEEE 21st International Conference on Emerging Technologies and Factory Automation (ETFA) (2016), IEEE, pp. 1–4

    2016

  64. [72]

    Measuring and understanding trust calibrations for automated systems: A survey of the state-of-the-art and future directions

    WISCHNEWSKI, M., KRÄMER, N.,ANDMÜLLER, E. Measuring and understanding trust calibrations for automated systems: A survey of the state-of-the-art and future directions. In Proceedings of the 2023 CHI conference on human factors in computing systems (2023), pp. 1–16

    2023

  65. [73]

    K., HAN, Y., CAI, Y., OUYANG, W., DU, H.,ANDLIU, C

    WONG, K. K., HAN, Y., CAI, Y., OUYANG, W., DU, H.,ANDLIU, C. From trust in automation to trust in ai in healthcare: A 30-year longitudinal review and an interdisciplinary framework. Bioengineering 12, 10 (2025), 1070

    2025

  66. [74]

    Optimal incremental learning under covariate shift

    YAMAUCHI, K. Optimal incremental learning under covariate shift. Memetic Computing 1, 4 (2009), 271

    2009

  67. [75]

    A study on interaction in human-in-the-loop machine learning for text analytics

    YANG, Y., KANDOGAN, E., LI, Y., SEN, P.,ANDLASECKI, W. A study on interaction in human-in-the-loop machine learning for text analytics. In IUI Workshops (2019)

    2019

  68. [76]

    Understanding the effect of accuracy on trust in machine learning models

    YIN, M., WORTMANVAUGHAN, J.,ANDWALLACH, H. Understanding the effect of accuracy on trust in machine learning models. In Proceedings Manuscript submitted to ACM 26 Donald R. Honeycutt, Mahsan Nourani, and Eric D. Ragan of the 2019 CHI Conference on Human Factors in Computing Sy...

    2019

  69. [77]

    Do i trust my machine teammate? an investigation from perception to decision

    YU, K., BERKOVSKY, S., TAIB, R., ZHOU, J.,ANDCHEN, F. Do i trust my machine teammate? an investigation from perception to decision. In Proceedings of the 24th International Conference on Intelligent User Interfaces (2019), pp. 460–468

    2019

  70. [78]

    annotator rationales

    ZAIDAN, O., EISNER, J.,ANDPIATKO, C. Using “annotator rationales” to improve machine learning for text categorization. In Human language technologies 2007: The conference of the North American chapter of the association for computational linguistics; proceedings of the main co...

    2007

  71. [79]

    Baylime: Bayesian local interpretable model-agnostic explanations

    ZHAO, X., HUANG, W., HUANG, X., ROBU, V.,ANDFLYNN, D. Baylime: Bayesian local interpretable model-agnostic explanations. In Uncertainty in Artificial Intelligence (2021), PMLR, pp. 887–896

    2021

  72. [80]

    Learning under concept drift: an overview

    ŽLIOBAIT ˙E, I. Learning under concept drift: an overview. arXiv preprint arXiv:1010.4784 (2010). Manuscript submitted to ACM

    2010 arXiv

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.