Pith. sign in

REVIEW 4 major objections 5 minor 36 references

Redefining Research Crowdsourcing: Incorporating Human Feedback with LLM-Powered Digital Twins

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The central claim is that an LLM-powered digital twin personalized to a crowd worker can auto-answer roughly half of Likert-scale survey questions with a mean absolute error of 0.97 on a 7-point scale, improving productivity and reducing…

desk verdict Useful hybrid crowdsourcing system, but the headline accuracy is undermined by a contaminated ground-truth measurement and a tuned threshold. read the letter →

arxiv 2505.24004 v1 pith:6EM5AORF submitted 2025-05-29 cs.HC cs.CLcs.CY

Amanda Chan , Catherine Di , Joseph Rupertus , Gary Smith , Varun Nagaraj Rao , Manoel Horta Ribeiro , Andrés Monroy-Hernández This is my paper
classification cs.HCcs.CLcs.CY
keywords digitaltwincrowdworklargelanguagemodelsAIuncertaintyLikert-scalesurveyshuman-AIcollaborationdecisionfatiguesurveyresearch
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a hybrid crowdsourcing system in which each crowd worker gets an LLM-powered digital twin—a personalized model that learns the worker's answers to demographic and attitude surveys and then auto-fills future survey questions when confident. It claims this arrangement lets digital twins simulate an individual worker's survey responses with a high degree of accuracy: on a 7-point Likert scale, confident predictions have a mean absolute error of 0.97, and the system auto-answers about 51% of questions at a 75% confidence threshold. If true, this would let researchers collect survey data at higher scale while keeping humans in the loop, and would spare workers repetitive questions and decision fatigue. The paper also reports interviews with crowd workers and researchers showing support for the hybrid idea, with transparency, oversight, and agency as conditions for adoption.

What carries the argument

The load-bearing object is the digital twin: a GPT-4o model prompted with the worker's demographic responses, the worker's answers to prior learning surveys, and the model's own previous predictions, instructed to output Likert-scale answers plus a summary of the worker's profile. The system uses the token log-probability of the numeric answer as a confidence measure; answers above a 75% threshold are auto-filled, and the rest are deferred to the human. Confidence calibration is what lets the system claim both automation and quality: by deferring low-confidence questions, it keeps average error on auto-filled questions at 0.97 while still covering half the survey.

What would settle it

A comparison experiment where one group sees the twin's auto-filled answers before the reflection pass and a control group does not; if the control group's mean absolute error rises above 0.97 by a meaningful margin, the accuracy claim is inflated by anchoring.

Watch

Extended reading notes

Core claim

The central claim is that a digital twin—a large language model prompted with a worker's demographic data, prior Likert-scale answers, and the model's own earlier predictions—can stand in for that worker on a substantial share of survey questions without materially changing the data. In the testing phase, the system's confident answers (token probability above 75%) deviated from the worker's own reflection-survey answers by a mean absolute difference of 0.97 on the 7-point scale, versus 1.22 for all predictions; these confident answers covered 51.1% of testing questions. The paper interprets this as evidence that digital twins may enhance productivity and reduce decision fatigue while maintaining response quality, and that workers and researchers see the main value in a hybrid arrangement that automates repetitive questions and leaves nuanced ones to humans.

Load-bearing premise

The accuracy evaluation assumes that participants' reflection-survey answers were not shaped by having already seen the twin's auto-filled answers during testing; if seeing those answers anchored their later answers, the reported 0.97 error is artificially low.

Editorial extensions

If this is right

  • A deployed system could auto-fill about half of the Likert-scale questions in a typical survey, which would let workers spend the saved time on fewer or more meaningful tasks.
  • Researchers could adopt a hybrid pipeline in which twins handle repetitive demographic and attitudinal items while humans retain complex or open-ended questions, preserving authenticity where it matters.
  • The 75% confidence threshold gives a concrete calibration trade-off: raising it should improve per-answer accuracy but reduce coverage, while lowering it should automate more but risk larger errors.
  • If the 0.97 mean error holds at scale, the twin's answers on its confident questions are statistically close to the worker's own answers on the tested 7-point instruments, making the twin usable as a fast proxy in survey pilots or longitudinal re-checks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The accuracy result may be optimistic because participants saw the twin's auto-filled answers before giving their reflection-survey answers; an anchoring-control study would tell how much of the 0.97 error is genuine reproduction versus alignment to the displayed answer.
  • Editorial inference: The fixed 75% threshold may be far from optimal for individual workers; tuning the threshold per person or per question topic could raise the automation rate or lower the error further.
  • Editorial inference: The paper's hybrid logic points toward a division of labor—twins for stable, factual, or repeated items and humans for context-sensitive judgment—that could be tested by comparing twin-assisted and fully human responses on free-text or morally ambiguous items.
  • Editorial inference: Worker ownership and monetization of twins, which the interviews left unresolved, will likely be the deciding policy issue for adoption; the technical accuracy shown here does not answer who controls the model after a worker leaves a platform.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper proposes an LLM-powered 'digital twin' system that learns crowd workers' survey preferences from demographic data and prior Likert-scale responses, automatically fills in high-confidence testing questions, and defers low-confidence questions to the human worker. The authors report an experiment with 88 Prolific workers, plus interviews with 5 crowd workers and 4 social science researchers. The headline quantitative results are mean absolute errors of 1.22 for all testing responses and 0.97 for high-confidence responses on a 7-point scale, with 51.1% of questions auto-filled, alongside self-report data suggesting perceived time savings and reduced decision fatigue. The paper concludes that such hybrid systems may enhance productivity and maintain response quality while keeping humans in the loop.

Significance. If the accuracy result were valid, this would be a meaningful contribution to crowdsourcing and HCI: it would demonstrate that a personalized LLM can reproduce roughly half of a worker's survey answers within about one Likert point, opening a concrete design space for hybrid human-AI crowd work with human oversight. The qualitative interviews add useful stakeholder perspectives on transparency, privacy, worker agency, and task-dependent trust. The system is described concretely, and the authors are candid about several limitations. However, the central quantitative claim is not currently established because the reflection-survey ground truth is not independent of the auto-filled answers shown during the testing phase, and the confidence threshold is tuned on the same data used to report accuracy. The contribution is therefore best treated as an exploratory system demonstration with perceived-benefit data, not as a validated accuracy result.

major comments (4)
  1. [§3.4 and §5.1.1] The accuracy comparison for the 51.1% of questions auto-filled by the digital twin uses reflection-survey answers as ground truth. Participants saw these auto-filled answers during the testing phase and were then asked, in the reflection survey, to answer the same questions again; although the prediction is hidden until after the participant answers, the prior exposure can anchor or conform responses. No control condition, delayed retest, or debriefing item is reported. The reported mean absolute error of 0.97 for confident responses is therefore potentially biased downward and cannot support the 'high degree of accuracy' claim. Please report accuracy on questions that the participant answered before seeing any auto-fill (for example, the deferred set) as the primary estimate, or add a control group that does not see auto-filled answers.
  2. [Footnote 1 and §5.1.2] The 75% confidence threshold is stated to have been 'empirically determined' on the same testing data as roughly the optimal point for maintaining accuracy while enabling enough auto-answers. Because the high-confidence subset is selected by a threshold that maximizes accuracy and utility, the reported 0.97 MAE is a selected statistic rather than an out-of-sample prediction. Please either pre-register or cross-validate the threshold choice, or report results across the full confidence range and label the optimized value as exploratory.
  3. [§5.1.1 and Table 1] No baseline is reported. On a 7-point Likert scale, a mean absolute error of 1.22 (or 0.97 for the confident subset) needs a comparison point: a non-personalized LLM prompted only with demographics, a majority-class or mean-response predictor, or human self-consistency on retest. Without such a baseline, 'high degree of accuracy' is not interpretable, and the claimed value of personalization over generic LLM responses is not demonstrated.
  4. [§5.2.2 and Abstract] The productivity and decision-fatigue conclusions are based on self-report items (70.5% reporting 'saving time by automating tasks' and 52.3% reporting 'reducing decision fatigue') rather than objective measures such as task completion time, error rates, or validated fatigue scales, and there is no control condition. Please reframe these as perceived benefits, or add objective measurements, to match the strength of the claims in the abstract.
minor comments (5)
  1. [Figures 4–5 and Table 1] The reported counts are inconsistent: Figure 4 reports n=3,784 while Table 1 reports 3,776 total responses, and Figure 5 reports n=1,933 while Table 1 reports 1,931 confident responses. Please reconcile these numbers.
  2. [Appendix C] The appendix says the complete list of survey questions is in a 'Supplementary Materials' document; please include the full instrument or provide a stable link, since the evaluation depends on the exact items and their adaptation to Likert format.
  3. [§3.2 and §4.1] The learning phase is described as containing three surveys of 15–19 questions each and the testing phase as 43 questions, with a 55%/45% split; please state the exact question counts and the resulting total so readers can verify the intended balance.
  4. [Throughout] The term 'digital twin' is used for a prompted GPT-4o model; please clarify that this is a lightweight personalized prompt-based proxy rather than a continuously synchronized model, to avoid overclaiming the fidelity implied by the term.
  5. [Footnote 2] Footnote 2 concedes that the reflection survey would not be included in a real system; this underscores that the accuracy evaluation is a lab procedure, and the paper should explicitly discuss how the evaluation context limits real-world applicability.

Circularity Check

1 steps flagged · score 6.0 of 10

Confident-response MAE of 0.97 is an in-sample selected statistic because the 75% confidence threshold was empirically tuned on the same testing data.

  1. fitted input called prediction [Section 3.3, footnote 1; Section 5.1.1, Figure 5]
    ""For a given question, if the confidence is above a certain threshold (75%), the system accepts the answer and automatically fills out the survey question." Footnote: "We empirically determined the 75% threshold for accepting digital twin answers as roughly the optimal point for maintaining accuracy while enabling the digital twin to answer enough questions to be useful." Section 5.1.1: "the mean error across all testing phase responses where the LLM answered in the user’s place (i.e., it displayed high confidence) was 0.97.""

    The 75% confidence cutoff is not an independent, pre-registered decision rule; it was empirically chosen on the same testing responses that later produce the headline accuracy. The reported 0.97 mean absolute error is computed only for the subset of responses that survives this tuned cutoff, so the confident-subset accuracy is an in-sample selected statistic. The central claim that a digital twin "is able to simulate an individual crowd worker’s survey tasks with a high degree of accuracy" therefore rests on a number partly manufactured by the threshold selection rather than on an out-of-sample prediction. No held-out validation or cross-validation is reported to show that the 0.97 generalizes.

full rationale

The paper's central empirical claim is an experimental evaluation rather than a derivation from definitions, but one concrete circular reduction is present: the 75% confidence threshold was empirically tuned on the same testing data used to report the confident-subset mean absolute error of 0.97. This makes that headline accuracy a fitted, selected statistic, so I score partial circularity rather than none. The separate concern that reflection-survey answers were collected after participants had already seen the auto-filled digital twin answers during the testing phase (Sections 3.3-3.4) is a serious threat to the independence of the ground truth, but it is a measurement-validity issue, not a definitional reduction, so I flag it without counting it as a circular step. The overall testing-phase error of 1.22 is threshold-independent and provides some independent evidence, which keeps the score from rising to the 8-10 range. Self-citations in the paper are background and do not carry the load-bearing argument.

Assumptions & free parameters 1 free parameters · 2 assumptions · 0 invented entities

The central evaluation rests on a tuned confidence threshold and on an uncontaminated ground-truth reflection survey. The LLM token probabilities are treated as calibrated confidence, which is assumed without validation. No new physical or conceptual entities are introduced; 'digital twin' is a framing for the prompted LLM system.

free parameters (1)
  • confidence threshold = 75%
    Empirically determined as 'roughly the optimal point' for maintaining accuracy while enabling the digital twin to answer enough questions to be useful (footnote 1). This threshold selects the subset of questions used to report high-confidence accuracy (MAE 0.97).
assumptions (2)
  • domain assumption LLM token probabilities are calibrated confidence estimates
    The system uses the linear probability of the predicted Likert token to decide whether to auto-answer or defer (Section 3.3). If these probabilities are miscalibrated, the deferral mechanism and the reported confidence accuracy are unreliable.
  • domain assumption Reflection survey answers are uncontaminated by prior exposure to digital twin answers
    Participants saw auto-filled digital twin answers during the testing phase before answering the same questions in the reflection survey (Section 3.4). The paper assumes no anchoring effect, which is load-bearing for the accuracy claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Redefining Research Crowdsourcing: Incorporating Human Feedback with LLM-Powered Digital Twins." pith.science (2026). https://pith.science/paper/6EM5AORF

@misc{pith2026250524004,
  author       = {Pith},
  title        = {Pith review of: Redefining Research Crowdsourcing: Incorporating Human Feedback with LLM-Powered Digital Twins},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6EM5AORF}},
  note         = {Machine review of arXiv:2505.24004}
}
read the original abstract

Crowd work platforms like Amazon Mechanical Turk and Prolific are vital for research, yet workers' growing use of generative AI tools poses challenges. Researchers face compromised data validity as AI responses replace authentic human behavior, while workers risk diminished roles as AI automates tasks. To address this, we propose a hybrid framework using digital twins, personalized AI models that emulate workers' behaviors and preferences while keeping humans in the loop. We evaluate our system with an experiment (n=88 crowd workers) and in-depth interviews with crowd workers (n=5) and social science researchers (n=4). Our results suggest that digital twins may enhance productivity and reduce decision fatigue while maintaining response quality. Both researchers and workers emphasized the importance of transparency, ethical data use, and worker agency. By automating repetitive tasks and preserving human engagement for nuanced ones, digital twins may help balance scalability with authenticity.

Figures

Figures reproduced from arXiv: 2505.24004 by the authors.

Figure 1
Figure 1. Participants complete surveys to train their digital twin and test its ability to answer questions in their place. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. The testing survey interface with both digital twin [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Distribution of absolute difference error for all test [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Distribution of absolute difference error for test [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: The distribution of digital twin confidence for all [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

36 extracted references · 24 canonical work pages

  1. [1]

    AIAA. 2020. Digital Twin: Definition & Value – An AIAA and AIA Position Paper. https://www.aia-aerospace.org/publications/digital-twin-definition- value-an-aiaa-and-aia-position-paper/

    work page 2020

  2. [2]

    Argyle, Ethan C

    Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua Gubler, Christopher Rytting, and David Wingate. 2022. Out of One, Many: Using Language Models to Simulate Human Samples. https://doi.org/10.1017/pan.2023.2

    doi:10.1017/pan.2023.2 2022

  3. [3]

    Nandeesh Babanagar, Brian Sheil, Jelena Ninić, Qianbing Zhang, and Stuart Hardy

    work page

  4. [4]

    Daniel Silverman, Daniel Kent, and Christopher Gelpi. 2020. Can Factual Misper- ceptions be Corrected? An Experiment on American Public Fears of Terrorism. (June 2020). https://osf.io/a7uk3/ Publisher: OSF

    work page 2020

  5. [5]

    Michael Davern, Rene Bautista, Jeremy Freese, Pamela Herd, and Stephen L. Morgan. 2024. General Social Survey 1972-2024. gssdataexplorer.norc.org

    work page 2024

  6. [6]

    Diana-Elena Drăghici, Andreea Orîndaru, Mihaela Constantinescu, and Alina Zelezneac. 2023. Revolutionizing Marketing Research Through AI: compre- hensive review of the past, present, and future.Journal of Emerging Trends in Marketing and ManagementI, 1 (May 2023), 39–45. http://www.etimm.ase. ro/RePEc/aes/jetimm/2023/ETIMM_V01_2023_73.pdf Publisher: The B...

    work page 2023

  7. [7]

    Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. ChatGPT outperforms crowd workers for text-annotation tasks.Proceedings of the National Academy of Sciences120, 30 (2023), e2305016120

    2023

  8. [8]

    2019.Ghost work: How to stop Silicon Valley from building a new global underclass

    Mary L Gray and Siddharth Suri. 2019.Ghost work: How to stop Silicon Valley from building a new global underclass. Eamon Dolan Books

    work page 2019

Show all 36 references
  1. [9]

    Luke Hewitt, Ashwini Ashokkumar, Isaias Ghezae, and Robb Willer. 2024. Pre- dicting Results of Social Science Experiments Using Large Language Models. (Aug. 2024)

    2024

  2. [10]

    Yijun Huang, Jihan Zhang, Xi Chen, Alan H. F. Lam, and Ben M. Chen. 2024. From Simulation to Prediction: Enhancing Digital Twins with Advanced Generative AI Technologies. In2024 IEEE 18th International Conference on Control & Automation (ICCA). 490–495. https://doi.org/10.1109...

    2024

  3. [11]

    Hughes, Abigail A

    Jennifer L. Hughes, Abigail A. Camden, Tenzin Yangchen, Gabrielle P. A. Smith, Melanie M. Domenech Rodríguez, Steven V. Rouse, C. Peeper McDonald, and Stella Lopez. 2022. Guidance for Researchers When Using Inclusive Demographic Questions for Surveys: Improved and Updated Ques...

    2022 doi

  4. [12]

    IBM. 2021. What Is a Digital Twin? https://www.ibm.com/topics/what-is-a- digital-twin

    2021

  5. [13]

    Business Insider. 2024. The AI world’s most valuable resource is running out, and it’s scrambling to find an alternative: ’fake’ data. (2024). https://www. businessinsider.com/ai-synthetic-data-industry-debate-over-fake-2024-8 Ac- cessed: 2025-01-19

    2024

  6. [14]

    Tomoya Kanda, Hiroyoshi Ito, and Atsuyuki Morishima. 2022. Efficient Evalu- ation of AI Workers for the Human+AI Crowd Task Assignment. In2022 IEEE International Conference on Big Data (Big Data). 3995–4001. https://doi.org/10. 1109/BigData55660.2022.10020844

    2022

  7. [15]

    Karya. [n. d.]. Karya Institute. https://institute.karya.in/

  8. [16]

    Katherine McCabe. 2019. Public Opinion and Attributions for Health Care Costs. (May 2019). https://osf.io/3pcdm/ Publisher: OSF

    2019

  9. [17]

    Masaki Kobayashi, Kei Wakabayashi, and Atsuyuki Morishima. 2021. Human+AI Crowd Task Assignment Considering Result Quality Requirements.Proceedings of the AAAI Conference on Human Computation and Crowdsourcing9 (Oct. 2021), 97–107. https://doi.org/10.1609/hcomp.v9i1.18943

    2021 doi

  10. [18]

    Maureen Craig. 2017. Racial Majority & Minority Group Members’ Psychological and Political Reactions to Minority Population Growth. (Nov. 2017). https: //osf.io/sazxn/ Publisher: OSF

    2017

  11. [19]

    Divya Nagaraj, Priya Khandelwal, Sandra Steyaert, and Olivier Gevaert. 2023. Augmenting digital twins with federated learning in medicine.The Lancet. Digital Health5, 5 (May 2023), e251–e253. https://doi.org/10.1016/S2589-7500(23)00044- 4

    2023 doi

  12. [20]

    Vishakh Padmakumar and He He. 2024. Does Writing with Language Models Reduce Content Diversity? arXiv:2309.05196 [cs.CL] https://arxiv.org/abs/2309. 05196

    2024 arXiv

  13. [21]

    Bernstein

    Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative Agents: Interactive Simu- lacra of Human Behavior. InProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST ’23...

    2023

  14. [22]

    Zou, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Robb Willer, Percy Liang, and Michael S

    Joon Sung Park, Carolyn Q. Zou, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Robb Willer, Percy Liang, and Michael S. Bernstein. 2024. Generative Agent Simulations of 1,000 People. https://doi.org/10.48550/arXiv. 2411.10109 arXiv:2411.10109 [cs]

    2024 doi arXiv

  15. [23]

    Mark Peffley and Jon Hurwitz. 2007. Persuasion and Resistance: Race and the Death Penalty in America.American Journal of Political Science51, 4 (2007), 996–1012. https://doi.org/10.1111/j.1540-5907.2007.00293.x _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1540-59...

    2007

  16. [24]

    Qloo. [n. d.]. Qloo | Develop Personalized Experiences With Taste AI. https: //www.qloo.com

  17. [25]

    Rebecca Bucci. 2023. Accounting for the Correlation between Perceived Risks and Rewards to Crime. https://osf.io/4fsg5/

    2023

  18. [26]

    Rochelle Terman. 2020. Human Rights Shaming, Compliance, and Nationalist Backlash. (Jan. 2020). https://osf.io/q8ra3/ Publisher: OSF

    2020

  19. [27]

    2019.Bit by bit: Social research in the digital age

    Matthew J Salganik. 2019.Bit by bit: Social research in the digital age. Princeton University Press

    2019

  20. [28]

    Veniamin Veselovsky, Manoel Horta Ribeiro, Philip Cozzolino, Andrew Gordon, David Rothschild, and Robert West. 2023. Prevalence and prevention of large language model use in crowd work. https://doi.org/10.48550/arXiv.2310.15683 arXiv:2310.15683 [cs]

    2023 doi arXiv

  21. [29]

    Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West. 2023. Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks. https://arxiv.org/abs/2306.07899v1

    2023 arXiv

  22. [30]

    Angelina Wang, Jamie Morgenstern, and John P Dickerson. 2024. Large language models cannot replace human participants because they cannot portray identity groups.arXiv preprint arXiv:2402.01908(2024)

    2024 arXiv

  23. [31]

    Xinru Wang, Hannah Kim, Sajjadur Rahman, Kushan Mitra, and Zhengjie Miao

  24. [32]

    Yunlong Wang, Priyadarshini Venkatesh, and Brian Y Lim. 2022. Interpretable Directed Diversity: Leveraging Model Explanations for Iterative Crowd Ideation. InProceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI ’22). Association for Computing Machi...

    2022

  25. [33]

    digital twin

    Tongshuang Wu, Haiyi Zhu, Maya Albayrak, Alexis Axon, Amanda Bertsch, Wenxing Deng, Ziqi Ding, Bill Guo, Sireesh Gururaja, Tzu-Sheng Kuo, et al. 2023. Llms as workers in human-computational algorithms? replicating crowdsourcing pipelines with llms.arXiv preprint arXiv:2307.101...

    2023 arXiv

  26. [36]

    Did not receive demographic information

    The LLM-powered digital twin’s answers to the same survey that the human answered. The purpose of this study is for the LLM-powered digital twin to learn from the human’s responses to better mimic the human crowd worker. Therefore, the human will answer the same survey that yo...

    2025

  27. [2024]

    InProceedings of the CHI Conference on Human Factors in Computing Systems(Honolulu HI USA, 2024-05-11)

    Human-LLM Collaborative Annotation Through Effective Verification of LLM Labels. InProceedings of the CHI Conference on Human Factors in Computing Systems(Honolulu HI USA, 2024-05-11). ACM, 1–21. https://doi.org/10.1145/ 3613904.3641960

    2024

  28. [2025]

    2025), 106140

    Digital twins for urban underground space.Tunnelling and Underground Space Technology155 (Jan. 2025), 106140. https://doi.org/10.1016/j.tust.2024. 106140

    2025 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.