Pith. sign in

REVIEW 4 major objections 4 minor 32 references

MonitrLLM: A Community-Centered Evaluation Infrastructure for Large Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An open-source extension links chat transcripts to user intent and outcomes, surfacing a 23.1% failure rate that a 4.19/5 satisfaction average hides.

desk verdict Useful infrastructure, but the pilot's headline failure statistics are partly constructed by the coding definition; the qualitative value of linking transcripts to user purpose is the real contribution. read the letter →

arxiv 2608.02409 v1 pith:OSBIAIFS submitted 2026-08-03 cs.AI cs.CY

Victor Ojewale , Ro Encarnación , Suresh Venkatasubramanian , Danaé Metaxa This is my paper · ORCID
classification cs.AIcs.CY
keywords LLMevaluationcommunity-centereduser-definedoutcomesconversationtranscriptssatisfactionratingsfailuretaxonomydatadonationinfrastructuresituateduse
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that current LLM evaluation has a structural blind spot: benchmarks measure capability on fixed tasks, conversation corpora log what users did but not what they wanted, and in-interface ratings record satisfaction without purpose. MonitrLLM is open-source infrastructure that closes that gap by linking full conversation transcripts to a user's stated task purpose, their outcome assessment, and a satisfaction rating, treating all three as primary evaluative signals. The paper's feasibility pilot—25 college students, 194 reports over two weeks—demonstrates what that linkage surfaces: a 23.1% task failure rate sitting under a 4.19/5 mean satisfaction, multi-turn conversations failing at 2.5 times the rate of single-turn ones, and failures concentrated in coding and academic work. The practical claim is that this evidence layer, not a better model or benchmark, is what is needed to see how LLMs actually fail in everyday use.

What carries the argument

The central object is the linked evaluation record. MonitrLLM's browser extension collects five fields per submission—interaction description, task purpose, outcome assessment, a conversation share link, and a 1-to-5 satisfaction rating—and the backend retrieves the full conversation transcript from that share link, storing transcript and metadata as a single record. The design treats user purpose and user-reported outcome as first-class evaluative signals rather than optional metadata; failure is coded only when the participant's outcome note or rating signals that the goal was not met, not inferred from the transcript alone. The system is reconfigurable so communities can change form field

What would settle it

Run the same pilot with an automatic random-sampling mode that logs every Nth conversation without user self-selection. If the sampled conversations show no multi-turn failure gap and a materially lower overall failure rate, the pilot's headline numbers reflect which interactions participants chose to submit rather than model behavior.

Watch

Extended reading notes

Core claim

Central claim: evaluation can only surface what the surrounding system preserves as evidence, and current systems never link what a user was trying to do to whether they got it. MonitrLLM links a full transcript to user-stated purpose, an outcome note, and a 1–5 satisfaction rating, making user-defined success a first-class signal. In a two-week pilot (25 students, 194 reports), this linkage reveals a 23.1% failure rate under mean satisfaction 4.19/5, multi-turn conversations failing at 2.5 times the single-turn rate, failed conversations averaging about twice as many user turns, and failure concentrated in coding/debugging (32.0%) and academic contexts (30.0%). These are a demonstration of

Load-bearing premise

The pilot's headline numbers assume that the interactions participants chose to submit, together with the authors' coding of outcome notes and ratings, are informative about real LLM failure rather than about which interactions happened to be reported; if that assumption fails, the 23.1% and 2.5x figures are artifacts of the reporting process.

Editorial extensions

If this is right

  • If the central claim is correct, satisfaction ratings in deployed chat interfaces are a systematically over-optimistic health metric: the pilot's 4.19/5 mean coexists with a 23.1% task-failure rate.
  • Long conversations should not be treated as engagement: the pilot finds multi-turn conversations fail at 2.5x the rate of single-turn ones, and failed conversations average about twice as many user turns, so conversation length is better read with outcome data as a difficulty signal.
  • Benchmark-based capability measurement and preference-based alignment signals, even combined with large transcript corpora, leave out the user's own goal, so failures like misinterpretation, non-convergence, and unverifiable citations stay invisible; linking the three signals is a structural complement.
  • Failure is not uniform across tasks and domains: coding/debugging (32.0%) and academic contexts (30.0%) show the highest observed failure rates in the pilot, meaning user-provided domain metadata is a relevant stratification variable for evaluation.
  • Community deployment is viable at small scale: 25 students over two weeks yielded an actionable failure taxonomy, suggesting the infrastructure itself—not the sample—is the contribution.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the multi-turn failure pattern replicates, interface designers could use turn count as a live triage signal: conversations that exceed a small number of turns are disproportionately failure states, so surfacing 'want to start over or switch tools?' after, say, the fourth turn could reduce user effort.
  • The 23.1% figure likely understates true failure among lower-satisfaction groups: the authors note that users who accept unverifiable outputs without noting the gap do not generate outcome notes that surface the pattern, so the 'dissatisfaction unspecified' residual and the single grounding-failure case both point to undercounting.
  • A stronger test of the paper's claim would pair voluntary submission with automatic random sampling; comparing failure rates between reported and sampled conversations would separate true model failure from salience-driven reporting, exactly as the paper itself flags as future work.
  • Because participants submitted only interactions they found worth reflecting on, the observed failure rate is a rate among salient interactions, not among all interactions; any downstream use of the '23.1%' number should carry that caveat.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces MonitrLLM, an open-source infrastructure (browser extension plus Django backend) that links full ChatGPT conversation transcripts to user-reported task purpose, outcome notes, and 1-to-5 satisfaction ratings. The authors argue that this triple linkage fills a gap in LLM evaluation, where benchmarks, transcript corpora, and in-interface feedback each capture only part of the picture. A two-week feasibility pilot with 25-26 college students produced 194 analyzable reports. The headline results are a 23.1% goal-failure rate despite a 4.19/5 mean satisfaction, a 2.5x higher failure rate for multi-turn than single-turn conversations, and a six-category failure taxonomy. The paper frames these as demonstrations that community audit metadata changes what evaluation can see, with the infrastructure itself as the primary contribution.

Significance. If the infrastructure and pilot hold up, this is a timely and useful contribution. The system is open source, deployable, and addresses a real gap: situated user intent and outcome assessment are rarely linked to full interaction trajectories. The qualitative examples, such as the 16-turn conversation rated 4/5 with inaccurate citations, genuinely illustrate transcript-invisible failures, and the authors are candid that the estimates are not population-level. The paper therefore provides a replicable template for community-grounded evaluation. However, the headline quantitative results are not yet credible as evidence because of circular coding, self-selection, definitional overlap, and absent reliability statistics. The infrastructure contribution is sound enough to merit revision rather than rejection.

major comments (4)
  1. [§4.2, §5.3, Abstract] The failure definition is circular with respect to the paper's central quantitative contrast. §4.2 states: 'We coded a conversation as a failure only when the participant's outcome note or satisfaction rating signaled that their goal was not met.' A low satisfaction rating is therefore sufficient for a failure label, so the 1.52-point satisfaction gap in Table 2 (failed: 3.02 vs failure-free: 4.54) is partly constructed by the coding rule. The Abstract's 'despite high satisfaction 4.19/5, 23.1% failure' is less paradoxical than claimed. Please re-code failures using transcript evidence independent of the satisfaction rating, or at minimum report the failure rate among high-satisfaction conversations, the number of low-satisfaction conversations not coded as failures, and the satisfaction distribution within each failure type. The 'dissatisfaction unspecified' category (11/45) is especial
  2. [§4.1, §5.2, §7] Self-selection and instructed reporting make the 2.5x multi-turn failure ratio non-identifiable as a property of interactions. Participants were asked to submit reports for interactions they 'considered worth reflecting on,' and §7 concedes that the observed failure rate is for salient interactions, not a population estimate. The Abstract and §5.2 nevertheless state that multi-turn conversations fail at 2.5 times the rate of single-turn exchanges, which could be driven by which interactions participants chose to submit (e.g., long difficult sessions being more report-worthy). Without a random-sampling mode or at least a reporting-bias analysis, this claim should be reworded as descriptive of the submitted corpus, not of LLM use in general. The authors already propose random sampling as future work; the current paper should not present the 2.5x figure as a standalone finding.
  3. [§4.3, Table 2] No inter-rater reliability statistic is reported. §4.3 states that the two coders independently coded all conversations and computed percentage agreement, then adjudicated disagreements to full agreement, but no numeric agreement, Cohen's kappa, or per-category agreement is given. Given the subjectivity of categories such as 'dissatisfaction unspecified' (negative outcome with no identifiable mechanism) and 'interaction friction/nonconvergence' (repeated re-prompts), the 23.1% failure rate and the failure-type frequencies in Table 2 cannot be separated from coder judgment. Please report agreement statistics and, if possible, a reliability subsample with a third coder or a preregistered codebook.
  4. [§5.2, §5.3, Table 7] The multi-turn versus single-turn comparison is partly definitional. The 'interaction friction/nonconvergence' category (10/45 failures) is defined as a multi-turn interaction failing after repeated re-prompts, and 'misinterpretation and reframing' (15/45) often involves restating constraints across turns. Thus failed conversations are selected to have high turn counts by the coding scheme, which inflates both the 2.5x follow-up failure ratio and the 2:1 turn-count ratio reported in §5.2. Please re-run the trajectory analyses excluding these definition-dependent categories, or define trajectory difficulty independently of the outcome labels, and report whether the ratios survive.
minor comments (4)
  1. [§4.1, §5, Abstract] Participant count is inconsistent: the Abstract and §4.1 say 26 college students, but §4.1 later states '194 conversations from 25 participants' and §5 says '25 college students.' Clarify whether one participant contributed zero valid reports or was excluded after data cleaning.
  2. [Tables 1 and 3] The small-n categories in Tables 1 and 3 are marked with a dagger or caution, but the failure rates for n=6 or n=3 categories are reported without confidence intervals. Consider adding exact binomial confidence intervals for all observed failure rates.
  3. [§5.3] The statement that '7 is a lower bound' on factual-error/hallucination failures is plausible but untestable as written, because users who accept unverifiable responses would not necessarily report negative outcomes. This should be phrased as an interpretive caveat rather than a quantitative bound.
  4. [§3.2, Figure 1] The paper says the extension 'activates when a user clicks on its toolbar icon to submit an interaction' and the form captures interaction description, task purpose, outcome assessment, share link, and satisfaction. It would help to clarify whether the transcript is fetched server-side from the share link, how expired or deactivated share links were detected, and whether the 12 excluded reports were distributed across participants in a way that could bias the analyses.

Circularity Check

2 steps flagged · score 4.0 of 10

Pilot's headline contrasts are partly built from its own failure-coding and taxonomy definitions; the infrastructure contribution itself is not circular.

  1. self definitional [§4.2 Analytical Approach and Coding Scheme; §5.3 A Transcript-Grounded Failure Taxonomy; Abstract]
    "We coded a conversation as a failure only when the participant’s outcome note or satisfaction rating signaled that their goal was not met — we do not infer failure from the transcript alone. ... Failure-free conversations had a mean satisfaction rating of 4.54, compared to 3.02 for conversations with any observed failure, a gap of 1.52 rating points."

    The failure label is defined using the same user-satisfaction signal that the paper later contrasts with failure. In particular, the 'dissatisfaction unspecified' category is coded when 'the user signals a negative outcome through a low rating or a negative note,' making low satisfaction sufficient to create a failure. The reported 1.52-point satisfaction gap between failed and failure-free conversations is therefore partly manufactured by the labeling rule, and the Abstract's 'despite reporting high average satisfaction (4.19/5) ... 23.1% failure rate' framing is a contrast between two quantities that are not independent. The circularity is partial, because a negative outcome note can mark failure even with a high satisfaction rating, and not every low rating need have been coded as failu

  2. self definitional [§4.2 Failure type coding / Table 7 (Appendix A.4); §5.2 Follow-up suggests failure; §5.3 Interaction friction and nonconvergence]
    "Interaction friction and nonconvergence appeared in 10 conversations (22.2%, mean rating 2.60), denoting exchanges where repeated re-prompts fail to yield a usable output. ... Multi-turn interaction fails to reach a usable result despite repeated attempts, escalating constraints, or repeated re-prompts."

    One of the failure categories is defined as a multi-turn interaction that fails: it cannot be assigned to single-turn conversations at all. All 10 'interaction friction / nonconvergence' failures therefore necessarily fall into the multi-turn bucket, mechanically inflating the reported 2.5x higher failure rate for multi-turn versus single-turn conversations. The finding is not wholly definitional—other failure types also occur in multi-turn interactions, and excluding this category would still leave an elevated multi-turn failure rate—but the headline pattern that 'extended interaction is a signal of difficulty rather than engagement' is partly built into the coding taxonomy rather than emerging independently from the data.

full rationale

The paper's core contribution—open-source infrastructure linking full transcripts to user purpose and outcome assessments—is not circular. It is a self-contained systems contribution, and the paper explicitly acknowledges the key threat of submission self-selection in its Limitations section. However, two of the pilot's headline empirical demonstrations are partially constructed by the paper's own definitions. First, failure is coded 'only when the participant's outcome note or satisfaction rating signaled that their goal was not met,' and the same satisfaction signal is then used to compute the 1.52-point satisfaction gap and to frame 'failure despite high satisfaction'; thus the central contrast is partly an artifact of the coding rule. Second, the 'interaction friction / nonconvergence' failure type is defined as multi-turn nonconvergence, so 10 of 45 failures are mechanically assigned to the multi-turn bucket, inflating the 2.5x multi-turn versus single-turn failure ratio. Neither finding is fully forced: negative outcome notes can mark failure with high satisfaction, and removing the friction category would still leave multi-turn failures elevated. The absence of inter-rater reliability statistics and the self-selected sample weaken external validity but are not circularity. Score 4 reflects partial construction of the headline empirical demonstrations, while the infrastructure argument itself retains independent content.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No numerical model parameters are fitted; the analysis is descriptive. The main researcher-chosen elements are the codebook categories and the satisfaction scale, which are measurement choices rather than fitted constants. The paper introduces no new theoretical entities; MonitrLLM is a software artifact, not a construct needing independent evidence.

assumptions (5)
  • domain assumption Participants accurately report their task purpose, outcome, and satisfaction for submitted interactions.
    The entire pilot's evidence relies on self-report in the five-field form (§3.2, §4.1); if these reports are inaccurate, the failure rate and satisfaction measures are invalid.
  • domain assumption The author-developed coding scheme and adjudication produce valid failure labels.
    Failure types were coded by two of the authors, with no inter-rater reliability statistics reported (§4.3); the categories are treated as ground truth.
  • domain assumption Voluntarily submitted, salient interactions are adequate to demonstrate the infrastructure's value.
    The pilot instructs participants to submit interactions 'worth reflecting on' (§4.1), and §7 concedes this yields a self-selected sample; the paper's inferences about failure patterns assume this is sufficient for a demonstration.
  • domain assumption ChatGPT share links fully and faithfully capture the conversation.
    Transcripts are retrieved from user-generated share links (§3.2); if share links omit or alter content, the linked-record analysis is affected.
  • domain assumption Automated translation plus native-speaker validation preserves meaning for the 9 Chinese conversations.
    Section 4.1 describes translation/cross-validation; the coding of these conversations depends on translation fidelity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MonitrLLM: A Community-Centered Evaluation Infrastructure for Large Language Models." pith.science (2026). https://pith.science/paper/OSBIAIFS

@misc{pith2026260802409,
  author       = {Pith},
  title        = {Pith review of: MonitrLLM: A Community-Centered Evaluation Infrastructure for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OSBIAIFS}},
  note         = {Machine review of arXiv:2608.02409}
}
read the original abstract

Benchmark suites assess model capability on controlled tasks; large-scale conversation corpora capture naturalistic use without user feedback; and in-interface feedback mechanisms record satisfaction without task purpose. Together, they leave a critical gap in LLM evaluation: no existing infrastructure routinely links interaction trajectories to user-defined outcomes. We introduce MonitrLLM, open-source infrastructure for community-centered LLM evaluations that links full conversation transcripts to user-reported task intent and outcome assessments, treating all three as primary evaluative signals rather than optional metadata. To demonstrate the value of this approach, we conducted a two-week feasibility pilot with 26 college students using ChatGPT, collecting 206 evaluation reports with full conversation transcripts. The findings from our pilot demonstrate the value of connecting conversation trajectories with user-reported outcomes. For instance, despite reporting high average satisfaction (4.19/5) with their LLM interactions, participants also experience a substantial 23.1% failure rate on their goal tasks. We also find that multi-turn conversations are reported as failing at 2.5 times the rate of single-turn exchanges, a pattern that reframes extended interaction as a signal of difficulty rather than engagement. We conclude by discussing the value of incorporating direct user feedback with observational data for robust LLM evaluations, and the possibilities for infrastructure that enables this goal.

Figures

Figures reproduced from arXiv: 2608.02409 by the authors.

Figure 1
Figure 1. The MonitrLLM browser extension where users [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Frequency distribution of user satisfaction ratings [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 1 canonical work pages

  1. [1]

    Ouyang, Long and Wu, Jeff and Jiang, Xu and Almeida, Diogo and Wainwright, Carroll L. and Mishkin, Pamela and Zhang, Chong and Agarwal, Sandhini and Slama, Katarina and Ray, Alex and Schulman, John and Hilton, Jacob and Kelton, Fraser and Miller, Luke and Simens, Maddie and Askell, Amanda and Welinder, Peter and Christiano, Paul and Leike, Jan and Lowe, R...

    2022

  2. [2]

    2020 , eprint=

    Fine-Tuning Language Models from Human Preferences , author=. 2020 , eprint=

    2020

  3. [3]

    2024 , eprint=

    LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset , author=. 2024 , eprint=

    2024

  4. [4]

    2023 , eprint=

    Holistic Evaluation of Language Models , author=. 2023 , eprint=

    2023

  5. [5]

    2023 , eprint=

    OpenAssistant Conversations -- Democratizing Large Language Model Alignment , author=. 2023 , eprint=

    2023

  6. [6]

    2024 , eprint=

    WildChat: 1M ChatGPT Interaction Logs in the Wild , author=. 2024 , eprint=

    2024

  7. [7]

    2022 , eprint=

    Red Teaming Language Models with Language Models , author=. 2022 , eprint=

    2022

  8. [8]

    The S hare LM Collection and Plugin: Contributing Human-Model Chats for the Benefit of the Community

    Don-Yehiya, Shachar and Choshen, Leshem and Abend, Omri. The S hare LM Collection and Plugin: Contributing Human-Model Chats for the Benefit of the Community. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations). 2025

    2025

Show all 32 references
  1. [9]

    Auditing Algorithms: Understanding Algorithmic Systems from the Outside In , year =

    Metaxa, Dana\". Auditing Algorithms: Understanding Algorithmic Systems from the Outside In , year =. Found. Trends Hum.-Comput. Interact. , month = nov, pages =. doi:10.1561/1100000083 , abstract =

    doi

  2. [10]

    2014 , url=

    Auditing Algorithms : Research Methods for Detecting Discrimination on Internet Platforms , author=. 2014 , url=

    2014

  3. [11]

    Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems , articleno =

    Park, Hyanghee and Ahn, Daehwan , title =. Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems , articleno =. 2024 , isbn =. doi:10.1145/3613904.3642785 , abstract =

    2024

  4. [12]

    2020 , eprint=

    Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing , author=. 2020 , eprint=

    2020

  5. [13]

    and Stark, Luke and Wortman Vaughan, Jennifer and Wallach, Hanna , title =

    Madaio, Michael A. and Stark, Luke and Wortman Vaughan, Jennifer and Wallach, Hanna , title =. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems , pages =. 2020 , isbn =. doi:10.1145/3313831.3376445 , abstract =

    2020

  6. [14]

    2022 , isbn =

    Sloane, Mona and Moss, Emanuel and Awomolo, Olaitan and Forlano, Laura , title =. 2022 , isbn =. doi:10.1145/3551624.3555285 , booktitle =

    2022

  7. [15]

    and Gordon, Mitchell L

    Lam, Michelle S. and Gordon, Mitchell L. and Metaxa, Dana\". End-User Audits: A System Empowering Communities to Lead Large-Scale Investigations of Harmful Algorithmic Behavior , year =. Proc. ACM Hum.-Comput. Interact. , month = nov, articleno =. doi:10.1145/3555625 , abstract =

    doi

  8. [16]

    Transactions of the Association for Computational Linguistics , author =

    The. Transactions of the Association for Computational Linguistics , author =. 2022 , note =. doi:10.1162/tacl_a_00474 , abstract =

    2022 doi

  9. [17]

    Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback

    Lai, Viet and Nguyen, Chien and Ngo, Nghia and Nguyen, Thuat and Dernoncourt, Franck and Rossi, Ryan and Nguyen, Thien. Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback. Proceedings of the 2023 Conference on E...

    2023 doi

  10. [18]

    Proceedings of the International Conference on Learning Representations (ICLR) , year=

    Measuring Massive Multitask Language Understanding , author=. Proceedings of the International Conference on Learning Representations (ICLR) , year=

  11. [19]

    2023 , eprint=

    A Survey on Evaluation of Large Language Models , author=. 2023 , eprint=

    2023

  12. [20]

    Building Safe G en AI Applications: An End-to-End Overview of Red Teaming for Large Language Models

    Purpura, Alberto and Wadhwa, Sahil and Zymet, Jesse and Gupta, Akshay and Luo, Andy and Rad, Melissa Kazemi and Shinde, Swapnil and Sorower, Mohammad Shahed. Building Safe G en AI Applications: An End-to-End Overview of Red Teaming for Large Language Models. Proceedings of the...

    2025 doi

  13. [21]

    and Lipton, Zachary C

    Feffer, Michael and Sinha, Anusha and Deng, Wesley H. and Lipton, Zachary C. and Heidari, Hoda , title =. Proceedings of the 2024 AAAI/ACM Conference on AI, Ethics, and Society , pages =. 2025 , publisher =

    2024

  14. [22]

    Yu , keywords =

    Jinqi Lai and Wensheng Gan and Jiayang Wu and Zhenlian Qi and Philip S. Yu , keywords =. Large language models in law: A survey , journal =. 2024 , issn =. doi:https://doi.org/10.1016/j.aiopen.2024.09.002 , url =

    2024 doi

  15. [23]

    Large language models in healthcare and medical domain:

    Nazi, Zabir Al and Peng, Wei , month = jul, year =. Large language models in healthcare and medical domain:. doi:10.48550/arXiv.2401.06775 , abstract =

    doi arXiv

  16. [24]

    Li, Yinheng and Wang, Shaofei and Ding, Han and Chen, Hang , month = nov, year =. Large. doi:10.1145/3604237.3626869 , booktitle =

  17. [25]

    Biases in Large Language Models: Origins, Inventory, and Discussion , year =

    Navigli, Roberto and Conia, Simone and Ross, Bj\". Biases in Large Language Models: Origins, Inventory, and Discussion , year =. doi:10.1145/3597307 , journal =

    doi

  18. [26]

    2025 , issue_date =

    Huang, Lei and Yu, Weijiang and Ma, Weitao and Zhong, Weihong and Feng, Zhangyin and Wang, Haotian and Chen, Qianglong and Peng, Weihua and Feng, Xiaocheng and Qin, Bing and Liu, Ting , title =. 2025 , issue_date =. doi:10.1145/3703155 , journal =

    2025 doi

  19. [27]

    Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems , articleno =

    Danry, Valdemar and Pataranutaporn, Pat and Groh, Matthew and Epstein, Ziv , title =. Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems , articleno =. 2025 , isbn =. doi:10.1145/3706598.3713408 , abstract =

    2025

  20. [28]

    Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency , pages =

    Li, Yuxuan and Shirado, Hirokazu and Das, Sauvik , title =. Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency , pages =. 2025 , isbn =. doi:10.1145/3715275.3732212 , abstract =

    2025

  21. [29]

    R eal H arm: A Collection of Real-World Language Model Application Failures

    Le Jeune, Pierre and Liu, Jiaen and Rossi, Luca and Dora, Matteo. R eal H arm: A Collection of Real-World Language Model Application Failures. Proceedings of the The First Workshop on LLM Security (LLMSEC). 2025

    2025

  22. [30]

    Designing and Conducting Mixed Methods Research , author =

    Choosing a. Designing and Conducting Mixed Methods Research , author =

  23. [31]

    and Pandit, Ayush and Kalicki, Colin H

    Lam, Michelle S. and Pandit, Ayush and Kalicki, Colin H. and Gupta, Rachit and Sahoo, Poonam and Metaxa, Dana\". Sociotechnical Audits: Broadening the Algorithm Auditing Lens to Investigate Targeted Advertising , year =. Proc. ACM Hum.-Comput. Interact. , month = oct, articlen...

    doi

  24. [32]

    Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems , articleno =

    Jahanbakhsh, Farnaz and Karger, David R , title =. Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems , articleno =. 2024 , isbn =. doi:10.1145/3613904.3642473 , abstract =

    2024

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.