Quality Control in Open-Ended Crowdsourcing: A Survey

Hailong Sun; Jing Zhang; Lei Chai

REVIEW 2 major objections 4 minor 245 references

Quality Control in Open-Ended Crowdsourcing: A Survey

T0 review · 2 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This survey proposes a two-tiered framework that organizes quality control research for open-ended crowdsourcing into task, worker, answer, and system aspects.

desk verdict A useful survey with a promising two-tier framework for open-ended crowdsourcing quality control, undermined by a System section that doesn't follow the promised taxonomy and some citation and screening issues. read the letter →

arxiv 2412.03991 v1 pith:N3D24KV7 submitted 2024-12-05 cs.HC cs.DC

Lei Chai , Hailong Sun , Jing Zhang This is my paper · ORCID

classification cs.HCcs.DC

keywords crowdsourcingopen-endedtasksqualitycontrolsurveytwo-tieredframeworktaskmodelworkeransweraggregation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Crowdsourcing tasks with open-ended answers—such as translation, image segmentation, and story writing—accept a large or infinite space of correct responses, so standard majority-vote and probabilistic-model quality control designed for Boolean tasks does not transfer. This survey argues that quality control for these tasks is best understood through a two-tiered framework: a first tier covering the aspects of task, worker, answer, and system, and a second tier breaking each aspect into quality dimensions, evaluation metrics, and design decisions. The paper reviews the resulting literature, organizes it into this framework, and identifies which areas are mature and which are missing. If the framework is right, it gives requesters, workers, and platform developers a usable map for choosing and designing quality control methods, and it sharpens the research agenda for open-ended crowdsourcing.

What carries the argument

The central object is the two-tiered quality control framework (the taxonomy). The first tier partitions the literature by the aspect of the crowdsourcing process being optimized: task, worker, answer, and system. The second tier further classifies each aspect into quality dimensions (the attributes that are modeled, such as task design, worker expertise, answer reliability), evaluation metrics (how quality is measured, including automatic metrics, peer feedback, and expert ratings), and design decisions (the methods used to optimize quality, such as task mapping, workflow design, teaching, incentives, and aggregation algorithms). The framework's work is to make scattered papers comparable and to expose what is under-studied.

What would settle it

Re-run the literature search with a broader keyword set (for example, 'free-form annotation', 'subjective annotation', 'generative tasks', 'LLM annotation') and with two independent screeners; if this finds many relevant quality control papers that do not fit the two-tiered framework, or finds substantial work in venues outside the chosen list, the survey's coverage claim would be refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that a two-tiered framework capturing quality dimensions, evaluation metrics, and design decisions across the aspects of task, worker, answer, and system accounts for the state of quality control research in open-ended crowdsourcing. The first tier provides a holistic view of what determines answer quality, while the second tier exposes the internal structure of each aspect: which quality attributes are modeled, how quality is measured, and what design choices are made to improve it. Surveying papers from 2012 to 2023 across major venues, the survey finds that most proposed methods are task-specific, that a few cross-task approaches exist for answer aggregation and evaluation, and that system-level joint optimization of task assignment, aggregation, and workflow is comparatively rare. The survey also positions open-ended crowdsourcing relative to Boolean crowdsourcing and to the emerging role of large language models as both annotators and sources of quality problems.

Load-bearing premise

The survey's taxonomy and gap analysis rest on its literature selection: a keyword search of 22 conferences and 14 journals for 2012 to 2023, screened once by title and abstract, with no inter-rater reliability check and with extra papers added from the authors' prior knowledge.

Editorial extensions

If this is right

Researchers can position new quality control methods within the framework and immediately see which combinations of aspect, dimension, and metric are already populated.
The survey's gap analysis implies that general or cross-task quality control methods, applicable across data types, are a priority because only a few such approaches exist.
Because the framework treats quality control as a system-level problem, it points toward joint optimization of task assignment, answer aggregation, and workflow design rather than optimizing each step in isolation.
In the era of large language models, the framework can be used to design quality control for hybrid human-AI annotation pipelines, where answers may originate from crowd workers or from LLMs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The boundary between Boolean and open-ended tasks is likely a spectrum rather than a dichotomy, so a graded version of the framework might better predict when Boolean methods such as majority voting or probabilistic graphical models can be adapted.
The 'system' aspect is the thinnest in the survey's account, suggesting that a formal definition of system-level quality metrics and their interactions would be a natural next step, though the paper does not develop one.
As LLMs increasingly generate crowd answers, the 'worker' aspect may need to be reinterpreted: worker modeling could become model behavior modeling, shifting quality control toward prompt design, consistency checks, and output filtering—an extension the paper gestures at but leaves open.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This survey is a genuinely useful reference for the crowdsourcing quality-control community. The open-ended focus fills a real gap: Daniel et al. and Jin et al. organize general or Boolean quality control, and this paper's two-tiered framework — aspects (task, worker, answer, system) broken down into quality dimensions, evaluation metrics, and design decisions — is a reasonable way to structure the space. The task, worker, and answer sections (3–5) follow the framework consistently and cover the literature broadly, including a structured search of 147 papers plus the authors' prior knowledge.

The most serious problem is that Section 6, the System aspect, does not deliver the promised second tier. The abstract and Section 2.3 say each aspect gets quality dimensions, evaluation metrics, and design decisions. Sections 3, 4, and 5 do that. Section 6 is organized by task assignment, answer aggregation, and workflow design, and there are no System-level evaluation metrics reviewed. The section frames these as 'the three core tasks in the crowdsourcing execution process' rather than as quality dimensions. That is an internal mismatch with the stated framework. It is fixable — either restructure Section 6 or soften the claim that the framework applies 'in each aspect' — but as written it overstates what the survey delivers.

There are also smaller issues. Section 1.4 attributes the truth-inference survey [94] to 'Li et al.'; the reference is Zheng et al. (VLDB 2017). The literature selection in Section 2.4 is described, but there is no inter-rater reliability, the screening is one pass by title and abstract, and 'papers derived from the authors' prior knowledge' is vague. That means coverage could have blind spots, though nothing I saw suggests major omissions.

These flaws are addressable and do not invalidate the framework. The open-ended focus is underserved, the two-tier structure is useful, and the LLM discussion in Section 7.2 is timely.

I'd send this to peer review with a request for revision. The referee should push on the System section and on the transparency of the literature search. Anyone working on crowdsourcing quality control, especially for open-ended tasks, would want to cite this after those fixes.

Referee Report

2 major / 4 minor

Summary. This manuscript surveys quality control in open-ended crowdsourcing and proposes a two-tiered framework: the first tier identifies four aspects (task, worker, answer, system), and the second tier classifies works within each aspect into quality dimensions, evaluation metrics, and design decisions. The survey describes the literature selection method, reviews representative works for the task, worker, and answer aspects, discusses system-level concerns as cross-aspect issues, and outlines challenges and future directions, including implications of large language models for crowdsourcing.

Significance. If the proposed framework is applied consistently, the survey could serve as a useful organizing map for a research area that is increasingly important as open-ended annotation tasks and LLM-assisted crowdsourcing grow. The paper does several things well: it gives concrete examples of open-ended tasks and their answer-space properties (Table 1), it provides a clear three-part second-tier structure for the task, worker, and answer aspects, and it includes an explicit, reproducible-looking literature search procedure in Section 2.4. The discussion of LLMs in Section 7.2 is timely and connects crowdsourcing quality control to current practice. The main caveat is that the System aspect does not receive the same second-tier treatment, which creates an internal inconsistency in the central contribution.

major comments (2)

[Section 2.3 and Section 6] Section 2.3 states that "each section corresponds to one aspect in the quality model, with quality dimensions, evaluation metrics and design decisions reviewed," and the abstract promises the second tier "in each aspect." Sections 3, 4, and 5 indeed follow this structure (e.g., 3.1 Quality Dimensions, 3.2 Quality Evaluation, 3.3 Quality Control Methods). Section 6, however, is organized around "the three core tasks in the crowdsourcing execution process" (6.1 Task Assignment, 6.2 Answer Aggregation, 6.3 Workflow Design) and provides no quality-dimension or evaluation-metric subsections for the System aspect. This is an internal inconsistency in the paper's central contribution. The authors should either (a) restructure Section 6 so that the System aspect also receives the promised second-tier treatment, identifying System-level quality dimensions and evaluation metrics, or (b) revise the framework description to state explicitly that System is a cross-cutting integration layer to which the second tier does not apply in full, and adjust the abstract and Section 2.3 accordingly.
[Section 2.4 and Section 1.5] The literature selection is described as a keyword search across 22 conferences and 14 journals for 2012-2023, followed by one-pass title/abstract screening by the authors, plus "additional papers derived from the authors' prior knowledge." However, Section 1.5 claims a "systematic review of all related works." The selection process has no inter-rater reliability, no citation chaining or snowballing, and no explicit inclusion/exclusion criteria beyond excluding simple crowdsourcing tasks, so the completeness and representativeness of the corpus are not established. Since the proposed taxonomy and the gap analysis in Section 7.1 depend on which papers are included, the authors should either strengthen the methodology (e.g., dual screening, inter-rater agreement, snowballing) or temper the claim of a systematic review and add a limitations discussion in the conclusion.

minor comments (4)

[Section 1.4] The text attributes reference [94] to "Li et al.," but reference [94] is Zheng et al., "Truth inference in crowdsourcing: Is the problem solved?" (PVLDB 2017). The citation should be corrected.
[Section 1.5 and Section 7] The fourth aspect is called "context" in the contributions list (Section 1.5), "workflow" in Section 7, and "system" elsewhere (Section 2.1 and the abstract). Unify the naming to avoid confusion about the framework's first tier.
[Section 2.3] The first bullet reads "Quality model. in each aspect" — the period should be removed so the sentence reads "Quality model in each aspect refers to a collection of quality dimensions..."
[Section 5.2.1] There is a typo in "froms open-ended answers" — this should be "from open-ended answers." Other minor spacing/ligature artifacts appear in the abstract and several places (e.g., "su ffi ciently"), likely from LaTeX rendering; these should be cleaned up.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the survey's taxonomy is an external organizational scheme, and the single self-citation is not load-bearing.

full rationale

This is a survey, so the claimed contribution is an organizational taxonomy rather than a fitted or predicted quantity. The two-tiered framework (task, worker, answer, system; quality dimensions, evaluation metrics, design decisions) is proposed in Section 2.3 and used as an external lens to organize the reviewed literature; the paper does not estimate any parameter from a subset of the literature and then 'predict' the remainder, nor does it derive the taxonomy from a self-cited theorem. The quality-control definition (Definition 1) is a stipulative scoping statement, not a conclusion derived from itself. The only self-citation visible in the text, [160] (Chai, Sun, Wang), appears in the Introduction as an example of open-ended question-answering tasks and is not load-bearing for the framework; the taxonomy's categories are derived from the stated quality model and the selected papers, and the literature-selection section discloses its screening procedure and the 'authors' prior knowledge' supplement, which is a completeness limitation rather than a circular reduction. No equation in the paper (e.g., Eq. (1), imported from Whitehill et al.) is used to justify the survey's classification. The Section 6 structural mismatch identified by the skeptic is a consistency or coverage issue, not a circularity, because it does not reduce the framework's claim to its inputs. Accordingly, no specific circular step can be quoted and exhibited under the required standard.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claims of this review rest on a classification scheme, not on fitted parameters or new entities. The main assumptions are the Crosby quality definition, the task partition, and the representativeness of the literature search. No free parameters or invented entities appear.

assumptions (3)

domain assumption Quality is defined as conformance to requirements (Crosby 1979), adopted in Section 1.2.
The survey's scope is restricted to quality aspects that serve requester requirements; security and reliability are excluded by this choice.
domain assumption Open-ended crowdsourcing tasks can be partitioned into the three types (intelligent information processing, crowd social decision-making, crowd ideation) and answer spaces into three sizes (countable, large but countable, large and uncountable) as set out in Section 1.1 and Table 1.
The taxonomy's completeness depends on this partition covering the relevant task landscape.
domain assumption The literature selection in Section 2.4 (419 papers found, 147 kept, plus papers from authors' prior knowledge) is representative of quality control research in open-ended crowdsourcing.
Gap analysis and statements like 'few works focus on cross-task approaches' rest on this selection. No inter-rater reliability or formal protocol is reported.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Quality Control in Open-Ended Crowdsourcing: A Survey." pith.science (2026). https://pith.science/paper/N3D24KV7

@misc{pith2026241203991,
  author       = {Pith},
  title        = {Pith review of: Quality Control in Open-Ended Crowdsourcing: A Survey},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N3D24KV7}},
  note         = {Machine review of arXiv:2412.03991}
}

read the original abstract

Crowdsourcing provides a flexible approach for leveraging human intelligence to solve large-scale problems, gaining widespread acceptance in domains like intelligent information processing, social decision-making, and crowd ideation. However, the uncertainty of participants significantly compromises the answer quality, sparking substantial research interest. Existing surveys predominantly concentrate on quality control in Boolean tasks, which are generally formulated as simple label classification, ranking, or numerical prediction. Ubiquitous open-ended tasks like question-answering, translation, and semantic segmentation have not been sufficiently discussed. These tasks usually have large to infinite answer spaces and non-unique acceptable answers, posing significant challenges for quality assurance. This survey focuses on quality control methods applicable to open-ended tasks in crowdsourcing. We propose a two-tiered framework to categorize related works. The first tier introduces a holistic view of the quality model, encompassing key aspects like task, worker, answer, and system. The second tier refines the classification into more detailed categories, including quality dimensions, evaluation metrics, and design decisions, providing insights into the internal structures of the quality control framework in each aspect. We thoroughly investigate how these quality control methods are implemented in state-of-the-art works and discuss key challenges and potential future research directions.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

245 extracted references · 64 canonical work pages

[94]

Truth inference in crowdsourc- ing: Is the problem solved? Proceedings of the VLDB Endowment, 2017, 10(5): 541–552

Zheng Y , Li G, Li Y , Shan C, Cheng R. Truth inference in crowdsourc- ing: Is the problem solved? Proceedings of the VLDB Endowment, 2017, 10(5): 541–552

2017
[1]

Faitcrowd: Fine grained truth discovery for crowdsourced data aggregation

Ma F, Li Y , Li Q, Qiu M, Gao J, Zhi S, Su L, Zhao B, Ji H, Han J. Faitcrowd: Fine grained truth discovery for crowdsourced data aggregation. In: Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining. 2015, 745–754

2015
[2]

Whose vote should count more: Optimal integration of labels from labelers of un- known expertise

Whitehill J, Wu T f, Bergsma J, Movellan J, Ruvolo P. Whose vote should count more: Optimal integration of labels from labelers of un- known expertise. Advances in neural information processing systems, 2009, 22

2009
[3]

Latent dirichlet allocation

Blei D M, Ng A Y , Jordan M I. Latent dirichlet allocation. the Journal of machine Learning research, 2003, 3: 993–1022

2003
[4]

icrowd: An adaptive crowd- sourcing framework

Fan J, Li G, Ooi B C, Tan K l, Feng J. icrowd: An adaptive crowd- sourcing framework. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data. 2015, 1015–1030

2015
[5]

The multidimensional wisdom of crowds

Welinder P, Branson S, Perona P, Belongie S. The multidimensional wisdom of crowds. 2011

2011
[6]

Compar- ing twitter and traditional media using topic models

Zhao W X, Jiang J, Weng J, He J, Lim E P, Yan H, Li X. Compar- ing twitter and traditional media using topic models. In: European conference on information retrieval. 2011, 338–349

2011
[7]

C-reference: Improving 2d to 3d object pose estimation accuracy via crowd- sourced joint object estimation

Song J Y , Chung J J Y , Fouhey D F, Lasecki W S. C-reference: Improving 2d to 3d object pose estimation accuracy via crowd- sourced joint object estimation. Proceedings of the ACM on Human- Computer Interaction, 2020, 4(CSCW1): 1–28

2020

Show all 245 references

[8]

Eventanchor: Reducing human interactions in event annotation of racket sports videos

Deng D, Wu J, Wang J, Wu Y , Xie X, Zhou Z, Zhang H, Zhang X, Wu Y . Eventanchor: Reducing human interactions in event annotation of racket sports videos. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 2021, 1–13

2021
[9]

Aggregating complex annotations via merging and matching

Braylan A, Lease M. Aggregating complex annotations via merging and matching. In: Proceedings of the 27th ACM SIGKDD Confer- ence on Knowledge Discovery & Data Mining. 2021, 86–94

2021
[10]

Capturing ambiguity in crowd- sourcing frame disambiguation

Dumitrache A, Aroyo L, Welty C. Capturing ambiguity in crowd- sourcing frame disambiguation. In: Sixth AAAI Conference on Hu- man Computation and Crowdsourcing. 2018

2018
[11]

Multistep planning for crowdsourcing complex consensus tasks

Deng Z, Xiang Y . Multistep planning for crowdsourcing complex consensus tasks. Knowledge-Based Systems, 2021, 231: 107447

2021
[12]

Crowd- sourcing question-answer meaning representations

Michael J, Stanovsky G, He L, Dagan I, Zettlemoyer L. Crowd- sourcing question-answer meaning representations. arXiv preprint arXiv:1711.05885, 2017

2017 arXiv
[13]

Repeated labeling us- ing multiple noisy labelers

Ipeirotis P G, Provost F, Sheng V S, Wang J. Repeated labeling us- ing multiple noisy labelers. Data Mining and Knowledge Discovery, 2014, 28(2): 402–441

2014
[14]

How many workers to ask? adaptive exploration for collecting high quality labels

Abraham I, Alonso O, Kandylas V , Patel R, Shelford S, Slivkins A. How many workers to ask? adaptive exploration for collecting high quality labels. In: Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. 2016, 473–482

2016
[15]

To re (label), or not to re (label)

Lin C H, Weld D S, others . To re (label), or not to re (label). In: Second AAAI conference on human computation and crowdsourcing. 2014

2014
[16]

Online sequencing of non-decomposable macrotasks in expert crowdsourcing

Schmitz H, Lykourentzou I. Online sequencing of non-decomposable macrotasks in expert crowdsourcing. ACM Transactions on Social Computing, 2018, 1(1): 1–33

2018
[17]

Creating a system for lexical substitutions from scratch using crowdsourcing

Biemann C. Creating a system for lexical substitutions from scratch using crowdsourcing. Language Resources and Evaluation, 2013, 47(1): 97–122

2013
[18]

Sprout: Crowd-powered task design for crowd- sourcing

Bragg J, Weld D S. Sprout: Crowd-powered task design for crowd- sourcing. In: Proceedings of the 31st annual acm symposium on user interface software and technology. 2018, 165–176

2018
[19]

Crowd anatomy be- yond the good and bad: Behavioral traces for crowd worker modeling and pre-selection

Gadiraju U, Demartini G, Kawase R, Dietze S. Crowd anatomy be- yond the good and bad: Behavioral traces for crowd worker modeling and pre-selection. Computer Supported Cooperative Work (CSCW), 2019, 28(5): 815–841

2019
[20]

Crowdsourcing complex workflows under budget constraints

Tran-Thanh L, Huynh T D, Rosenfeld A, Ramchurn S D, Jennings N R. Crowdsourcing complex workflows under budget constraints. In: Twenty-Ninth AAAI Conference on Artificial Intelligence. 2015

2015
[21]

Pplib: Toward the automated generation of crowd computing programs using process recombination and auto- experimentation

Boer P M D, Bernstein A. Pplib: Toward the automated generation of crowd computing programs using process recombination and auto- experimentation. ACM Transactions on Intelligent Systems and Tech- nology (TIST), 2016, 7(4): 1–20

2016
[22]

E fficiently identifying a well-performing crowd process for a given problem

De Boer P M, Bernstein A. E fficiently identifying a well-performing crowd process for a given problem. In: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. 2017, 1688–1699

2017
[23]

Investigating dif- ferences in crowdsourced news credibility assessment: Raters, tasks, and expert criteria

Bhuiyan M M, Zhang A X, Sehat C M, Mitra T. Investigating dif- ferences in crowdsourced news credibility assessment: Raters, tasks, and expert criteria. Proceedings of the ACM on Human-Computer Interaction, 2020, 4(CSCW2): 1–26

2020
[24]

Crowdsheet: instant implementation and out-of-hand execution of complex crowdsourcing

Suzuki R, Sakaguchi T, Matsubara M, Kitagawa H, Morishima A. Crowdsheet: instant implementation and out-of-hand execution of complex crowdsourcing. In: 2018 IEEE 34th International Confer- ence on Data Engineering (ICDE). 2018, 1633–1636

2018
[25]

Cross-modal data programming enables rapid medical ma- chine learning

Dunnmon J A, Ratner A J, Saab K, Khandwala N, Markert M, Sagreiya H, Goldman R, Lee-Messer C, Lungren M P, Rubin D L, others . Cross-modal data programming enables rapid medical ma- chine learning. Patterns, 2020, 1(2): 100019

2020
[26]

Crowdweaver: visually man- aging complex crowd work

Kittur A, Khamkar S, Andr ´e P, Kraut R. Crowdweaver: visually man- aging complex crowd work. In: Proceedings of the ACM 2012 Con- ference on Computer Supported Cooperative Work. 2012, 1033–1036

2012
[27]

Quality-based pricing for crowd- sourced workers

Wang J, Ipeirotis P G, Provost F. Quality-based pricing for crowd- sourced workers. 2013

2013
[28]

Measuring crowdsourcing e ffort with error-time curves

Cheng J, Teevan J, Bernstein M S. Measuring crowdsourcing e ffort with error-time curves. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 2015, 1365– 20 1374

2015
[29]

The e ffects of performance-contingent fi- nancial incentives in online labor markets

Yin M, Chen Y , Sun Y A. The e ffects of performance-contingent fi- nancial incentives in online labor markets. In: Twenty-Seventh AAAI Conference on Artificial Intelligence. 2013

2013
[30]

Complex crowdsourcing task al- location strategies employing supervised and reinforcement learning

Cui L, Zhao X, Liu L, Yu H, Miao Y . Complex crowdsourcing task al- location strategies employing supervised and reinforcement learning. International Journal of Crowd Science, 2017

2017
[31]

Task assignments in complex collaborative crowdsourcing

He W, Cui L, Huang C. Task assignments in complex collaborative crowdsourcing. In: CCF Conference on Computer Supported Coop- erative Work and Social Computing. 2018, 574–580

2018
[32]

Storia: Summarizing social media con- tent based on narrative theory using crowdsourcing

Kim J, Monroy-Hernandez A. Storia: Summarizing social media con- tent based on narrative theory using crowdsourcing. In: Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. 2016, 1018–1027

2016
[33]

Wearwrite: Crowd-assisted writing from smartwatches

Nebeling M, To A, Guo A, Freitas d A A, Teevan J, Dow S P, Bigham J P. Wearwrite: Crowd-assisted writing from smartwatches. In: Pro- ceedings of the 2016 CHI conference on human factors in computing systems. 2016, 3834–3846

2016
[34]

The knowledge accelerator: Big picture thinking in small pieces

Hahn N, Chang J, Kim J E, Kittur A. The knowledge accelerator: Big picture thinking in small pieces. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 2016, 2258–2270

2016
[35]

Communitycrit: inviting the public to improve and evaluate urban design ideas through micro-activities

Mahyar N, James M R, Ng M M, Wu R A, Dow S P. Communitycrit: inviting the public to improve and evaluate urban design ideas through micro-activities. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 2018, 1–14

2018
[36]

Exploring trade-o ffs between learning and productivity in crowdsourced history

Wang N C, Hicks D, Luther K. Exploring trade-o ffs between learning and productivity in crowdsourced history. Proceedings of the ACM on Human-Computer Interaction, 2018, 2(CSCW): 1–24

2018
[37]

Crafting policy discussion prompts as a task for newcomers

McInnis B, Leshed G, Cosley D. Crafting policy discussion prompts as a task for newcomers. Proceedings of the ACM on Human- Computer Interaction, 2018, 2(CSCW): 1–23

2018
[38]

A task decomposition framework for surveying the crowd contextual insights

Allahbakhsh M, Arbabi S, Shirazi M, Motahari-Nezhad H R. A task decomposition framework for surveying the crowd contextual insights. In: 2015 IEEE 8th International Conference on Service- Oriented Computing and Applications (SOCA). 2015, 155–162

2015
[39]

Cascade: Crowdsourcing taxonomy creation

Chilton L B, Little G, Edge D, Weld D S, Landay J A. Cascade: Crowdsourcing taxonomy creation. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2013, 1999– 2008

2013
[40]

Crowdsourcing in the field: A case study using local crowds for event reporting

Agapie E, Teevan J, Monroy-Hern ´andez A. Crowdsourcing in the field: A case study using local crowds for event reporting. In: Third AAAI Conference on Human Computation and Crowdsourcing. 2015

2015
[41]

Chorus: a crowd-powered conversational assistant

Lasecki W S, Wesley R, Nichols J, Kulkarni A, Allen J F, Bigham J P. Chorus: a crowd-powered conversational assistant. In: Proceedings of the 26th annual ACM symposium on User interface software and technology. 2013, 151–162

2013
[42]

Crowdia: Solving mysteries with crowd- sourced sensemaking

Li T, Luther K, North C. Crowdia: Solving mysteries with crowd- sourced sensemaking. Proceedings of the ACM on Human-Computer Interaction, 2018, 2(CSCW): 1–29

2018
[43]

Reviewing versus doing: Learn- ing and performance in crowd assessment

Zhu H, Dow S P, Kraut R E, Kittur A. Reviewing versus doing: Learn- ing and performance in crowd assessment. In: Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. 2014, 1445–1455

2014
[44]

Microtalk: Using argu- mentation to improve crowdsourcing accuracy

Drapeau R, Chilton L, Bragg J, Weld D. Microtalk: Using argu- mentation to improve crowdsourcing accuracy. In: Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. 2016, 32–41

2016
[45]

Smartcrowd: a workflow framework for complex crowdsourcing tasks

Xiong T, Yu Y , Pan M, Yang J. Smartcrowd: a workflow framework for complex crowdsourcing tasks. In: International Conference on Business Process Management. 2018, 387–398

2018
[46]

Optimized group formation for solving collaborative tasks

Rahman H, Roy S B, Thirumuruganathan S, Amer-Yahia S, Das G. Optimized group formation for solving collaborative tasks. The VLDB Journal, 2019, 28(1): 1–23

2019
[47]

Zencrowd: leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking

Demartini G, Difallah D E, Cudr ´e-Mauroux P. Zencrowd: leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking. In: Proceedings of the 21st international conference on World Wide Web. 2012, 469–478

2012
[48]

Crowd- sourcing for multiple-choice question answering

Aydin B I, Yilmaz Y S, Li Y , Li Q, Gao J, Demirbas M. Crowd- sourcing for multiple-choice question answering. In: AAAI. 2014, 2946–2953

2014
[49]

Bayesian classifier combination

Kim H C, Ghahramani Z. Bayesian classifier combination. In: Arti- ficial Intelligence and Statistics. 2012, 619–627

2012
[50]

Community- based bayesian aggregation models for crowdsourcing

Venanzi M, Guiver J, Kazai G, Kohli P, Shokouhi M. Community- based bayesian aggregation models for crowdsourcing. In: Proceed- ings of the 23rd international conference on World wide web. 2014, 155–164

2014
[51]

Clickstream analysis for crowd-based object segmentation with confidence

Heim E, Seitel A, Andrulis J, Isensee F, Stock C, Ross T, Maier-Hein L. Clickstream analysis for crowd-based object segmentation with confidence. IEEE transactions on pattern analysis and machine intel- ligence, 2017, 40(12): 2814–2826

2017
[52]

Exploring the e ffects of goal setting when training for complex crowdsourcing tasks

Rechkemmer A, Yin M. Exploring the e ffects of goal setting when training for complex crowdsourcing tasks. In: IJCAI. 2021, 4819– 4823

2021
[53]

Motivating novice crowd workers through goal setting: An investigation into the effects on complex crowdsourc- ing task training

Rechkemmer A, Yin M. Motivating novice crowd workers through goal setting: An investigation into the effects on complex crowdsourc- ing task training. In: Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. 2020, 122–131

2020
[54]

Teaching active human learners

Wang Z, Sun H. Teaching active human learners. In: Proceedings of the AAAI Conference on Artificial Intelligence. 2021, 5850–5857

2021
[55]

Machine teaching: An inverse problem to machine learning and an approach toward optimal education

Zhu X. Machine teaching: An inverse problem to machine learning and an approach toward optimal education. In: Proceedings of the AAAI Conference on Artificial Intelligence. 2015

2015
[56]

Trainbot: A con- versational interface to train crowd workers for delivering on-demand therapy

Abbas T, Khan V J, Gadiraju U, Markopoulos P. Trainbot: A con- versational interface to train crowd workers for delivering on-demand therapy. In: Proceedings of the AAAI Conference on Human Com- putation and Crowdsourcing. 2020, 3–12

2020
[57]

Toward a learning sci- ence for complex crowdsourcing tasks

Doroudi S, Kamar E, Brunskill E, Horvitz E. Toward a learning sci- ence for complex crowdsourcing tasks. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 2016, 2623–2634

2016
[58]

Cicero: Multi-turn, con- textual argumentation for accurate crowdsourcing

Chen Q, Bragg J, Chilton L B, Weld D S. Cicero: Multi-turn, con- textual argumentation for accurate crowdsourcing. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 21 2019, 1–14

2019
[59]

Leveraging peer communication to enhance crowdsourcing

Tang W, Yin M, Ho C J. Leveraging peer communication to enhance crowdsourcing. In: The World Wide Web Conference. 2019, 1794– 1805

2019
[60]

E ffective crowd annotation for relation extraction

Liu A, Soderland S, Bragg J, Lin C H, Ling X, Weld D S. E ffective crowd annotation for relation extraction. In: Proceedings of the 2016 conference of the North American chapter of the association for com- putational linguistics: human language technologies. 2016, 897–906

2016
[61]

Context trees: Crowdsourcing global un- derstanding from local views

Verroios V , Bernstein M S. Context trees: Crowdsourcing global un- derstanding from local views. In: Second AAAI Conference on Hu- man Computation and Crowdsourcing. 2014

2014
[62]

Strategies for crowdsourcing social data analysis

Willett W, Heer J, Agrawala M. Strategies for crowdsourcing social data analysis. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2012, 227–236

2012
[63]

The chal- lenge of variable effort crowdsourcing and how visible gold can help

Hettiachchi D, Schaekermann M, McKinney T J, Lease M. The chal- lenge of variable effort crowdsourcing and how visible gold can help. Proceedings of the ACM on Human-Computer Interaction, 2021, 5(CSCW2): 1–26

2021
[64]

Wikum: Bridging discussion forums and wikis using recursive summarization

Zhang A X, Verou L, Karger D. Wikum: Bridging discussion forums and wikis using recursive summarization. In: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and So- cial Computing. 2017, 2082–2096

2017
[65]

Creating better action plans for writing tasks via vocabulary- based planning

Kaur H, Williams A C, Thompson A L, Lasecki W S, Iqbal S T, Tee- van J. Creating better action plans for writing tasks via vocabulary- based planning. Proceedings of the ACM on Human-Computer Inter- action, 2018, 2(CSCW): 1–22

2018
[66]

An examination of the work practices of crowd- farms

Wang Y , Papangelis K, Saker M, Lykourentzou I, Khan V J, Cham- berlain A, Grudin J. An examination of the work practices of crowd- farms. In: Proceedings of the 2021 CHI Conference on Human Fac- tors in Computing Systems. 2021, 1–14

2021
[67]

Heteroglossia: In-situ story ideation with the crowd

Huang C Y , Huang S H, Huang T H K. Heteroglossia: In-situ story ideation with the crowd. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 2020, 1–12

2020
[68]

Com- puting crowd consensus with partial agreement

Hung N Q V , Viet H H, Tam N T, Weidlich M, Yin H, Zhou X. Com- puting crowd consensus with partial agreement. IEEE Transactions on Knowledge and Data Engineering, 2017, 30(1): 1–14

2017
[69]

Quality control in crowdsourcing based on fine-grained behavioral features

Pei W, Yang Z, Chen M, Yue C. Quality control in crowdsourcing based on fine-grained behavioral features. Proceedings of the ACM on Human-Computer Interaction, 2021, 5(CSCW2): 1–28

2021
[70]

Optimal complex task assignment in service crowdsourcing

Tang F. Optimal complex task assignment in service crowdsourcing. In: IJCAI. 2020, 1563–1569

2020
[71]

Using hierarchical skills for optimized task assignment in knowledge-intensive crowdsourc- ing

Mavridis P, Gross-Amblard D, Mikl ´os Z. Using hierarchical skills for optimized task assignment in knowledge-intensive crowdsourc- ing. In: Proceedings of the 25th International Conference on World Wide Web. 2016, 843–853

2016
[72]

Skill ontology- based model for quality assurance in crowdsourcing

Maarry K E, Balke W T, Cho H, Hwang S w, Baba Y . Skill ontology- based model for quality assurance in crowdsourcing. In: International conference on database systems for advanced applications. 2014, 376–387

2014
[73]

Crowdcog: A cognitive skill based system for heterogeneous task assignment and recommendation in crowdsourcing

Hettiachchi D, Van Berkel N, Kostakos V , Goncalves J. Crowdcog: A cognitive skill based system for heterogeneous task assignment and recommendation in crowdsourcing. Proceedings of the ACM on Human-Computer Interaction, 2020, 4(CSCW2): 1–22

2020
[74]

A review on the methods to evaluate crowd con- tributions in crowdsourcing applications

Aris H, Azizan A. A review on the methods to evaluate crowd con- tributions in crowdsourcing applications. In: International Confer- ence of Reliable Information and Communication Technology. 2019, 1031–1041

2019
[75]

Dexa: Supporting non-expert annotators with dynamic examples from ex- perts

Zlabinger M, Sabou M, Hofst ¨atter S, Sertkan M, Hanbury A. Dexa: Supporting non-expert annotators with dynamic examples from ex- perts. In: Proceedings of the 43rd International ACM SIGIR Confer- ence on Research and Development in Information Retrieval. 2020, 2109–2112

2020
[76]

V oyant: generating structured feedback on visual designs using a crowd of non-experts

Xu A, Huang S W, Bailey B. V oyant: generating structured feedback on visual designs using a crowd of non-experts. In: Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. 2014, 1433–1444

2014
[77]

Crowdsourced text sequence aggregation based on hybrid reli- ability and representation

Li J. Crowdsourced text sequence aggregation based on hybrid reli- ability and representation. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Informa- tion Retrieval. 2020, 1761–1764

2020
[78]

Modeling and aggregation of complex annota- tions via annotation distances

Braylan A, Lease M. Modeling and aggregation of complex annota- tions via annotation distances. In: Proceedings of The Web Confer- ence 2020. 2020, 1807–1818

2020
[79]

Crowd process design: how to coordinate crowds to solve complex problems

De Boer P. Crowd process design: how to coordinate crowds to solve complex problems. PhD thesis, University of Zurich, 2017

2017
[80]

Communicating context to the crowd for complex writing tasks

Salehi N, Teevan J, Iqbal S, Kamar E. Communicating context to the crowd for complex writing tasks. In: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and So- cial Computing. 2017, 1890–1901

2017
[81]

A classroom study of using crowd feedback in the iterative design process

Xu A, Rao H, Dow S P, Bailey B P. A classroom study of using crowd feedback in the iterative design process. In: Proceedings of the 18th ACM conference on computer supported cooperative work & social computing. 2015, 1637–1648

2015
[82]

Supporting esl writ- ing by prompting crowdsourced structural feedback

Huang Y C, Huang J C, Wang H C, Hsu J Y j. Supporting esl writ- ing by prompting crowdsourced structural feedback. In: Fifth AAAI Conference on Human Computation and Crowdsourcing. 2017

2017
[83]

An empirical study on short-and long-term e ffects of self-correction in crowdsourced microtasks

Kobayashi M, Morita H, Matsubara M, Shimizu N, Morishima A. An empirical study on short-and long-term e ffects of self-correction in crowdsourced microtasks. In: Sixth AAAI Conference on Human Computation and Crowdsourcing. 2018

2018
[84]

Human rationales as attribution priors for explainable stance detection

Jayaram S, Allaway E. Human rationales as attribution priors for explainable stance detection. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021, 5540– 5554

2021
[85]

Deep extreme cut: From extreme points to object segmentation

Maninis K K, Caelles S, Pont-Tuset J, Van Gool L. Deep extreme cut: From extreme points to object segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018, 616–625

2018
[86]

Large-scale interactive object seg- mentation with human annotators

Benenson R, Popov S, Ferrari V . Large-scale interactive object seg- mentation with human annotators. In: Proceedings of the IEEE /CVF Conference on Computer Vision and Pattern Recognition. 2019, 11700–11709

2019
[87]

Best of both worlds: human- machine collaboration for object annotation

Russakovsky O, Li L J, Fei-Fei L. Best of both worlds: human- machine collaboration for object annotation. In: Proceedings of the 22 IEEE conference on computer vision and pattern recognition. 2015, 2121–2131

2015
[88]

Lean crowdsourcing: Combining humans and machines in an online system

Branson S, Van Horn G, Perona P. Lean crowdsourcing: Combining humans and machines in an online system. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017, 7474–7483

2017
[89]

Eureca: Enhanced understanding of real environments via crowd assistance

Gouravajhala S R, Yim J, Desingh K, Huang Y , Jenkins O C, Lasecki W S. Eureca: Enhanced understanding of real environments via crowd assistance. In: Sixth AAAI conference on human computa- tion and crowdsourcing. 2018

2018
[90]

Rationale-based human-in-the-loop via supervised attention

Kanchinadam T, Westpfahl K, You Q, Fung G. Rationale-based human-in-the-loop via supervised attention. In: DaSH@ KDD. 2020

2020
[91]

Scalpel-cd: leveraging crowdsourcing and deep probabilistic mod- eling for debugging noisy training data

Yang J, Smirnova A, Yang D, Demartini G, Lu Y , Cudr ´e-Mauroux P. Scalpel-cd: leveraging crowdsourcing and deep probabilistic mod- eling for debugging noisy training data. In: The World Wide Web Conference. 2019, 2158–2168

2019
[92]

On human intellect and machine failures: Troubleshooting integrative machine learning sys- tems

Nushi B, Kamar E, Horvitz E, Kossmann D. On human intellect and machine failures: Troubleshooting integrative machine learning sys- tems. In: Thirty-First AAAI Conference on Artificial Intelligence. 2017

2017
[93]

Joint cognition of both human and machine for predicting criminal punishment in judicial system

Das A K, Ashrafi A, Ahmmad M. Joint cognition of both human and machine for predicting criminal punishment in judicial system. In: 2019 IEEE 4th International Conference on Computer and Commu- nication Systems (ICCCS). 2019, 36–40

2019
[95]

Resolving conflicts in het- erogeneous data by truth discovery and source reliability estimation

Li Q, Li Y , Gao J, Zhao B, Fan W, Han J. Resolving conflicts in het- erogeneous data by truth discovery and source reliability estimation. In: Proceedings of the 2014 ACM SIGMOD international conference on Management of data. 2014, 1187–1198

2014
[96]

A confidence-aware approach for truth discovery on long-tail data

Li Q, Li Y , Gao J, Su L, Zhao B, Demirbas M, Fan W, Han J. A confidence-aware approach for truth discovery on long-tail data. Pro- ceedings of the VLDB Endowment, 2014, 8(4): 425–436

2014
[97]

Learning from the wisdom of crowds by minimax entropy

Zhou D, Basu S, Mao Y , Platt J. Learning from the wisdom of crowds by minimax entropy. Advances in neural information processing sys- tems, 2012, 25

2012
[98]

Probabilistic graphical models

Sucar L E. Probabilistic graphical models. Advances in Computer Vision and Pattern Recognition. London: Springer London. doi, 2015, 10(978): 1

2015
[99]

Maximum likelihood estimation of observer error-rates using the em algorithm

Dawid A P, Skene A M. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics), 1979, 28(1): 20–28

1979
[100]

Eliciting structured knowledge from situated crowd markets

Goncalves J, Hosio S, Kostakos V . Eliciting structured knowledge from situated crowd markets. ACM Transactions on Internet Tech- nology (TOIT), 2017, 17(2): 1–21

2017
[101]

Rehumanized crowdsourcing: A labeling framework addressing bias and ethics in machine learning

Barbosa N M, Chen M. Rehumanized crowdsourcing: A labeling framework addressing bias and ethics in machine learning. In: Pro- ceedings of the 2019 CHI Conference on Human Factors in Comput- ing Systems. 2019, 1–12

2019
[102]

Language understanding in the wild: Combining crowdsourcing and machine learning

Simpson E D, Venanzi M, Reece S, Kohli P, Guiver J, Roberts S J, Jennings N R. Language understanding in the wild: Combining crowdsourcing and machine learning. In: Proceedings of the 24th international conference on world wide web. 2015, 992–1002

2015
[103]

Gleu: Automatic evaluation of sentence-level fluency

Mutton A, Dras M, Wan S, Dale R. Gleu: Automatic evaluation of sentence-level fluency. In: Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics. 2007, 344–351

2007
[104]

Bleu: a method for au- tomatic evaluation of machine translation

Papineni K, Roukos S, Ward T, Zhu W J. Bleu: a method for au- tomatic evaluation of machine translation. In: Proceedings of the 40th annual meeting of the Association for Computational Linguis- tics. 2002, 311–318

2002
[105]

Unitbox: An advanced object detection network

Yu J, Jiang Y , Wang Z, Cao Z, Huang T. Unitbox: An advanced object detection network. In: Proceedings of the 24th ACM international conference on Multimedia. 2016, 516–520

2016
[106]

Learn- ing from disagreement: A survey

Uma A N, Fornaciari T, Hovy D, Paun S, Plank B, Poesio M. Learn- ing from disagreement: A survey. Journal of Artificial Intelligence Research, 2021, 72: 1385–1470

2021
[107]

Crowdea: Multi-view idea prioritization with crowds

Baba Y , Li J, Kashima H. Crowdea: Multi-view idea prioritization with crowds. In: Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. 2020, 23–32

2020
[108]

Quality assessment for crowdsourced object annotations

Vittayakorn S, Hays J. Quality assessment for crowdsourced object annotations. In: BMVC. 2011, 1–11

2011
[109]

Efficient elicitation approaches to estimate collective crowd answers

Chung J J Y , Song J Y , Kutty S, Hong S, Kim J, Lasecki W S. Efficient elicitation approaches to estimate collective crowd answers. Proceed- ings of the ACM on Human-Computer Interaction, 2019, 3(CSCW): 1–25

2019
[110]

A dataset of crowdsourced word sequences: Collections and answer aggregation for ground truth creation

Li J, Fukumoto F. A dataset of crowdsourced word sequences: Collections and answer aggregation for ground truth creation. In: Proceedings of the First Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP. 2019, 24–28

2019
[111]

Label aggregation for crowdsourced triplet similarity comparisons

Li J, Endo L R, Kashima H. Label aggregation for crowdsourced triplet similarity comparisons. In: International Conference on Neural Information Processing. 2021, 176–185

2021
[112]

Context-based collective preference aggregation for prioritizing crowd opinions in social decision-making

Li J. Context-based collective preference aggregation for prioritizing crowd opinions in social decision-making. In: Proceedings of the ACM Web Conference 2022. 2022, 2657–2667

2022
[113]

Exploiting disagreement through open-ended tasks for capturing interpretation spaces

Timmermans B. Exploiting disagreement through open-ended tasks for capturing interpretation spaces. In: European Semantic Web Con- ference. 2016, 873–882

2016
[114]

Di fficult cases: From data to learning, and back

Klebanov B B, Beigman E. Di fficult cases: From data to learning, and back. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (V olume 2: Short Papers). 2014, 390– 396

2014
[115]

We need to consider disagreement in evaluation

Basile V , Fell M, Fornaciari T, Hovy D, Paun S, Plank B, Poesio M, Uma A, others . We need to consider disagreement in evaluation. In: 1st Workshop on Benchmarking: Past, Present and Future. 2021, 15–21

2021
[116]

Crowdtruth: Machine-human com- putation framework for harnessing disagreement in gathering anno- tated data

Inel O, Khamkham K, Cristea T, Dumitrache A, Rutjes A, Ploeg J v d, Romaszko L, Aroyo L, Sips R J. Crowdtruth: Machine-human com- putation framework for harnessing disagreement in gathering anno- tated data. In: International semantic web conference. 2014, 486–504

2014
[117]

f-brs: Rethinking back- propagating refinement for interactive segmentation

Sofiiuk K, Petrov I, Barinova O, Konushin A. f-brs: Rethinking back- propagating refinement for interactive segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- 23 nition. 2020, 8623–8632

2020
[118]

Learning to con- textually aggregate multi-source supervision for sequence labeling

Lan O, Huang X, Lin B Y , Jiang H, Liu L, Ren X. Learning to con- textually aggregate multi-source supervision for sequence labeling. arXiv preprint arXiv:1910.04289, 2019

1910 arXiv
[119]

Characterizing image segmentation behavior of the crowd

Sameki M, Gurari D, Betke M. Characterizing image segmentation behavior of the crowd. Collective Intelligence, 2015, 1–4

2015
[120]

Interactive image segmentation via backpropa- gating refinement scheme

Jang W D, Kim C S. Interactive image segmentation via backpropa- gating refinement scheme. In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition. 2019, 5297–5306

2019
[121]

Incorporating pixel proximity into answer aggregation for crowdsourced image segmentation

Yang Y , Chen P, Sun H. Incorporating pixel proximity into answer aggregation for crowdsourced image segmentation. CCF Transactions on Pervasive Computing and Interaction, 2022, 1–16

2022
[122]

A cognitive process theory of writing

Flower L, Hayes J R. A cognitive process theory of writing. College composition and communication, 1981, 32(4): 365–387

1981
[123]

The cost structure of sensemaking

Russell D M, Stefik M J, Pirolli P, Card S K. The cost structure of sensemaking. In: Proceedings of the INTERACT’93 and CHI’93 conference on Human factors in computing systems. 1993, 269–276

1993
[124]

Large high resolu- tion displays for co-located collaborative sensemaking: Display usage and territoriality

Bradel L, Endert A, Koch K, Andrews C, North C. Large high resolu- tion displays for co-located collaborative sensemaking: Display usage and territoriality. International Journal of Human-Computer Studies, 2013, 71(11): 1078–1088

2013
[125]

Apparition: Crowdsourced user interfaces that come to life as you sketch them

Lasecki W S, Kim J, Rafter N, Sen O, Bigham J P, Bernstein M S. Apparition: Crowdsourced user interfaces that come to life as you sketch them. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 2015, 1925–1934

2015
[126]

Latent structure in collab- oration: the case of reddit r /place

Rappaz J, Catasta M, West R, Aberer K. Latent structure in collab- oration: the case of reddit r /place. In: Twelfth International AAAI Conference on Web and Social Media. 2018

2018
[127]

How we write with crowds

Feldman M Q, McInnis B J. How we write with crowds. Proceedings of the ACM on Human-Computer Interaction, 2021, 4(CSCW3): 1– 31

2021
[128]

Task requirements and media choice in collaborative writing

Kraut R, Galegher J, Fish R, Chalfonte B. Task requirements and media choice in collaborative writing. Human–Computer Interaction, 1992, 7(4): 375–407

1992
[129]

Learnersourcing personal- ized hints

Glassman E L, Lin A, Cai C J, Miller R C. Learnersourcing personal- ized hints. In: Proceedings of the 19th ACM conference on computer- supported cooperative work & social computing. 2016, 1626–1636

2016
[130]

Mechanical novel: Crowdsourcing complex work through reflection and revision

Kim J, Sterman S, Cohen A A B, Bernstein M S. Mechanical novel: Crowdsourcing complex work through reflection and revision. In: Proceedings of the 2017 acm conference on computer supported co- operative work and social computing. 2017, 233–245

2017
[131]

Structuring, aggregating, and evaluating crowdsourced design critique

Luther K, Tolentino J L, Wu W, Pavel A, Bailey B P, Agrawala M, Hartmann B, Dow S P. Structuring, aggregating, and evaluating crowdsourced design critique. In: Proceedings of the 18th ACM con- ference on computer supported cooperative work & social computing. 2015, 473–485

2015
[132]

Crowdsourced explanations for humorous internet memes based on linguistic theories

Lin C C, Huang Y C, Hsu J Y j. Crowdsourced explanations for humorous internet memes based on linguistic theories. In: Second AAAI Conference on Human Computation and Crowdsourcing. 2014

2014
[133]

The sensemaking process and leverage points for an- alyst technology as identified through cognitive task analysis

Pirolli P, Card S. The sensemaking process and leverage points for an- alyst technology as identified through cognitive task analysis. In: Pro- ceedings of international conference on intelligence analysis. 2005, 2–4

2005
[134]

Expert and exceptional performance: Evidence of maximal adaptation to task constraints

Ericsson K A, Lehmann A C. Expert and exceptional performance: Evidence of maximal adaptation to task constraints. Annual review of psychology, 1996, 47(1): 273–305

1996
[135]

Context-based page unit recommendation for web-based sensemaking tasks

Cheng W H, Gotz D. Context-based page unit recommendation for web-based sensemaking tasks. In: Proceedings of the 14th interna- tional conference on Intelligent user interfaces. 2009, 107–116

2009
[136]

Semantic network analysis as a method for visual text analytics

Drieger P. Semantic network analysis as a method for visual text analytics. Procedia-social and behavioral sciences, 2013, 79: 4–17

2013
[137]

Biset: Semantic edge bundling with biclusters for sensemaking

Sun M, Mi P, North C, Ramakrishnan N. Biset: Semantic edge bundling with biclusters for sensemaking. IEEE transactions on visu- alization and computer graphics, 2015, 22(1): 310–319

2015
[138]

Supporting hand- off in asynchronous collaborative sensemaking using knowledge- transfer graphs

Zhao J, Glueck M, Isenberg P, Chevalier F, Khan A. Supporting hand- off in asynchronous collaborative sensemaking using knowledge- transfer graphs. IEEE transactions on visualization and computer graphics, 2017, 24(1): 340–350

2017
[139]

Collaboration among crowdsourcees: Towards a design theory for collaboration process design

Tavanapour N, Bittner E A C. Collaboration among crowdsourcees: Towards a design theory for collaboration process design. 2017

2017
[140]

Crowdsourcing and aggregating nested markable annotations

Madge C, Yu J, Chamberlain J, Kruschwitz U, Paun S, Poesio M. Crowdsourcing and aggregating nested markable annotations. 2019

2019
[141]

Crowdsourcing complex language resources: Playing to annotate dependency syntax

Guillaume B, Fort K, Lefebvre N. Crowdsourcing complex language resources: Playing to annotate dependency syntax. In: International Conference on Computational Linguistics (COLING). 2016

2016
[142]

Learning to speak and act in a fantasy text adventure game

Urbanek J, Fan A, Karamcheti S, Jain S, Humeau S, Dinan E, Rockt¨aschel T, Kiela D, Szlam A, Weston J. Learning to speak and act in a fantasy text adventure game. arXiv preprint arXiv:1903.03094, 2019

1903 arXiv
[143]

A survey of incentive en- gineering for crowdsourcing

Muldoon C, O’Grady M J, O’Hare G M. A survey of incentive en- gineering for crowdsourcing. The Knowledge Engineering Review, 2018, 33

2018
[144]

Supporting multilevel incentive mechanisms in crowdsourcing systems: an artifact-centric view

Scekic O, Truong H L, Dustdar S. Supporting multilevel incentive mechanisms in crowdsourcing systems: an artifact-centric view. In: Crowdsourcing, 91–111. Springer, 2015

2015
[145]

Aggregating crowdsourced image segmentations

Lee D, Das Sarma A, Parameswaran A. Aggregating crowdsourced image segmentations. HCOMP, 2018

2018
[146]

Measuring annotator agreement gen- erally across complex structured, multi-object, and free-text annota- tion tasks

Braylan A, Alonso O, Lease M. Measuring annotator agreement gen- erally across complex structured, multi-object, and free-text annota- tion tasks. In: Proceedings of the ACM Web Conference 2022. 2022, 1720–1730

2022
[147]

What can crowd computing do for the next gen- eration of ai systems? In: CSW@ NeurIPS

Gadiraju U, Yang J. What can crowd computing do for the next gen- eration of ai systems? In: CSW@ NeurIPS. 2020, 7–13

2020
[148]

A survey of image classification methods and tech- niques for improving classification performance

Lu D, Weng Q. A survey of image classification methods and tech- niques for improving classification performance. International journal of Remote sensing, 2007, 28(5): 823–870

2007
[149]

Sentiment analysis algorithms and applications: A survey

Medhat W, Hassan A, Korashy H. Sentiment analysis algorithms and applications: A survey. Ain Shams engineering journal, 2014, 5(4): 1093–1113

2014
[150]

Optical character recognition

Mithe R, Indalkar S, Divekar N. Optical character recognition. Inter- national journal of recent technology and engineering (IJRTE), 2013, 2(1): 72–75

2013
[151]

Imagenet: A large-scale hierarchical image database

Deng J, Dong W, Socher R, Li L J, Li K, Fei-Fei L. Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference 24 on computer vision and pattern recognition. 2009, 248–255

2009
[152]

Learning from crowds

Raykar V C, Yu S, Zhao L H, Valadez G H, Florin C, Bogoni L, Moy L. Learning from crowds. Journal of machine learning research, 2010, 11(4)

2010
[153]

Visual genome: Con- necting language and vision using crowdsourced dense image anno- tations

Krishna R, Zhu Y , Groth O, Johnson J, Hata K, Kravitz J, Chen S, Kalantidis Y , Li L J, Shamma D A, others . Visual genome: Con- necting language and vision using crowdsourced dense image anno- tations. International journal of computer vision, 2017, 123(1): 32–73

2017
[154]

Data augmentation approaches for improving animal audio classification

Nanni L, Maguolo G, Paci M. Data augmentation approaches for improving animal audio classification. Ecological Informatics, 2020, 57: 101084

2020
[155]

The who, what and why of knowledge mapping

Wexler M N. The who, what and why of knowledge mapping. Journal of knowledge management, 2001

2001
[156]

Incremental knowledge base construction using deepdive

Shin J, Wu S, Wang F, De Sa C, Zhang C, R ´e C. Incremental knowledge base construction using deepdive. In: Proceedings of the VLDB Endowment International Conference on Very Large Data Bases. 2015, 1310

2015
[157]

Face recognition: A literature survey

Zhao W, Chellappa R, Phillips P J, Rosenfeld A. Face recognition: A literature survey. ACM computing surveys (CSUR), 2003, 35(4): 399–458

2003
[158]

Antprophet: an intention mining system behind alipay’s intelligent customer service bot

Chen C, Zhang X, Ju S, Fu C, Tang C, Zhou J, Li X. Antprophet: an intention mining system behind alipay’s intelligent customer service bot. In: IJCAI. 2019, 6497–6499

2019
[159]

Viewpoint invariant pedestrian recognition with an ensemble of localized features

Gray D, Tao H. Viewpoint invariant pedestrian recognition with an ensemble of localized features. In: European conference on computer vision. 2008, 262–275

2008
[160]

An error consistency based approach to answer aggregation in open-ended crowdsourcing

Chai L, Sun H, Wang Z. An error consistency based approach to answer aggregation in open-ended crowdsourcing. Information Sci- ences, 2022, 608: 1029–1044

2022
[161]

Color image segmentation: advances and prospects

Cheng H D, Jiang X H, Sun Y , Wang J. Color image segmentation: advances and prospects. Pattern recognition, 2001, 34(12): 2259– 2281

2001
[162]

Using amazon mechanical turk and other compensated crowdsourcing sites

Schmidt G B, Jettingho ff W M. Using amazon mechanical turk and other compensated crowdsourcing sites. Business Horizons, 2016, 59(4): 391–400

2016
[163]

Towards fully au- tonomous driving: Systems and algorithms

Levinson J, Askeland J, Becker J, Dolson J, Held D, Kammel S, Kolter J Z, Langer D, Pink O, Pratt V , others . Towards fully au- tonomous driving: Systems and algorithms. In: 2011 IEEE intelligent vehicles symposium (IV). 2011, 163–168

2011
[164]

Enabling communication technologies for smart cities

Yaqoob I, Hashem I A T, Mehmood Y , Gani A, Mokhtar S, Guizani S. Enabling communication technologies for smart cities. IEEE Com- munications Magazine, 2017, 55(1): 112–120

2017
[165]

Task assignment for social- oriented crowdsourcing

Wu G, Chen Z, Liu J, Han D, Qiao B. Task assignment for social- oriented crowdsourcing. Frontiers of Computer Science, 2021, 15: 1–11

2021
[166]

Find truth in the hands of the few: acquiring specific knowledge with crowdsourcing

Han T, Sun H, Song Y , Fang Y , Liu X. Find truth in the hands of the few: acquiring specific knowledge with crowdsourcing. Frontiers of Computer Science, 2021, 15: 1–12

2021
[167]

Quality assessment in competition-based software crowdsourcing

Hu Z, Wu W, Luo J, Wang X, Li B. Quality assessment in competition-based software crowdsourcing. Frontiers of Computer Science, 2020, 14: 1–14

2020
[168]

Label distribution similarity-based noise correction for crowdsourcing

Ren L, Jiang L, Zhang W, Li C. Label distribution similarity-based noise correction for crowdsourcing. Frontiers of Computer Science, 2024, 18(5): 185323

2024
[169]

Attribute augmentation-based label integra- tion for crowdsourcing

Zhang Y , Jiang L, Li C. Attribute augmentation-based label integra- tion for crowdsourcing. Frontiers of Computer Science, 2023, 17(5): 175331

2023
[170]

All one needs to know about metaverse: A complete survey on technological singularity, virtual ecosystem, and research agenda

Lee L H, Braud T, Zhou P, Wang L, Xu D, Lin Z, Kumar A, Bermejo C, Hui P. All one needs to know about metaverse: A complete survey on technological singularity, virtual ecosystem, and research agenda. arXiv preprint arXiv:2110.05352, 2021

arXiv
[171]

Proceedings of the first workshop on aggregating and analysing crowdsourced annotations for nlp

Paun S, Hovy D. Proceedings of the first workshop on aggregating and analysing crowdsourced annotations for nlp. In: Proceedings of the First Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP. 2019

2019
[172]

Universal sentence encoder

Cer D, Yang Y , Kong S y, Hua N, Limtiaco N, John R S, Constant N, Guajardo-Cespedes M, Yuan S, Tar C, others . Universal sentence encoder. arXiv preprint arXiv:1803.11175, 2018

2018 arXiv
[173]

Bert: Pre-training of deep bidirectional transformers for language understanding

Devlin J, Chang M W, Lee K, Toutanova K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

2018 arXiv
[174]

Vldb 2021 crowd science challenge on aggregating crowdsourced audio tran- scriptions

Ustalov D, Pavlichenko N, Stelmakh I, Kuznetsov D. Vldb 2021 crowd science challenge on aggregating crowdsourced audio tran- scriptions. In: Proceedings of the 2nd Crowd Science Workshop: Trust, Ethics, and Excellence in Crowdsourced Data Management at Scale. 2021, 1–7

2021
[175]

Probabilistic graphical models: principles and techniques

Koller D, Friedman N. Probabilistic graphical models: principles and techniques. MIT press, 2009

2009
[176]

Highly accurate protein structure prediction with alphafold

Jumper J, Evans R, Pritzel A, Green T, Figurnov M, Ronneberger O, Tunyasuvunakool K, Bates R, ˇZ´ıdek A, Potapenko A, others . Highly accurate protein structure prediction with alphafold. Nature, 2021, 596(7873): 583–589

2021
[177]

The dynamics of micro-task crowdsourcing: The case of amazon mturk

Difallah D E, Catasta M, Demartini G, Ipeirotis P G, Cudr ´e-Mauroux P. The dynamics of micro-task crowdsourcing: The case of amazon mturk. In: Proceedings of the 24th international conference on world wide web. 2015, 238–247

2015
[178]

’it’s reducing a human being to a percentage’ perceptions of justice in al- gorithmic decisions

Binns R, Van Kleek M, Veale M, Lyngs U, Zhao J, Shadbolt N. ’it’s reducing a human being to a percentage’ perceptions of justice in al- gorithmic decisions. In: Proceedings of the 2018 Chi conference on human factors in computing systems. 2018, 1–14

2018
[179]

The future of crowd work

Kittur A, Nickerson J V , Bernstein M, Gerber E, Shaw A, Zimmerman J, Lease M, Horton J. The future of crowd work. In: Proceedings of the 2013 conference on Computer supported cooperative work. 2013, 1301–1318

2013
[180]

No workflow can ever be enough: How crowdsourcing workflows constrain complex work

Retelny D, Bernstein M S, Valentine M A. No workflow can ever be enough: How crowdsourcing workflows constrain complex work. Proceedings of the ACM on Human-Computer Interaction, 2017, 1(CSCW): 1–23

2017
[181]

The rise of crowdsourcing

Howe J, others . The rise of crowdsourcing. Wired magazine, 2006, 14(6): 1–4

2006
[182]

Failure of classical tra ffic flow theories: Stochastic high- way capacity and automatic driving

Kerner B S. Failure of classical tra ffic flow theories: Stochastic high- way capacity and automatic driving. Physica A: Statistical Mechanics and its Applications, 2016, 450: 700–747

2016
[183]

Image segmentation using deep learning: A survey

Minaee S, Boykov Y Y , Porikli F, Plaza A J, Kehtarnavaz N, Ter- 25 zopoulos D. Image segmentation using deep learning: A survey. IEEE transactions on pattern analysis and machine intelligence, 2021

2021
[184]

Cider: Consensus-based image description evaluation

Vedantam R, Lawrence Zitnick C, Parikh D. Cider: Consensus-based image description evaluation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2015, 4566–4575

2015
[185]

Hybrideval: A human-ai col- laborative approach for evaluating design ideas at scale

Mesbah S, Arous I, Yang J, Bozzon A. Hybrideval: A human-ai col- laborative approach for evaluating design ideas at scale. In: Proceed- ings of the ACM Web Conference 2023. 2023, 3837–3848

2023
[186]

Opencrowd: A human-ai collaborative approach for finding social influencers via open-ended answers aggregation

Arous I, Yang J, Khayati M, Cudr ´e-Mauroux P. Opencrowd: A human-ai collaborative approach for finding social influencers via open-ended answers aggregation. In: Proceedings of The Web Con- ference 2020. 2020, 1851–1862

2020
[187]

recaptcha: Human-based character recognition via web security measures

V on Ahn L, Maurer B, McMillen C, Abraham D, Blum M. recaptcha: Human-based character recognition via web security measures. Sci- ence, 2008, 321(5895): 1465–1468

2008
[188]

Data centric workflows for complex crowdsourcing applications

H ´elou¨et L, Singh R, Miklos Z. Data centric workflows for complex crowdsourcing applications
[189]

Spatial crowdsourcing: Challenges, techniques, and applications

Tong Y , Chen L, Shahabi C. Spatial crowdsourcing: Challenges, techniques, and applications. Proceedings of the VLDB Endowment, 2017, 10(12): 1988–1991

2017
[190]

Imagenet classification with deep convolutional neural networks

Krizhevsky A, Sutskever I, Hinton G E. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 2017, 60(6): 84–90

2017
[191]

Delving deep into rectifiers: Surpass- ing human-level performance on imagenet classification

He K, Zhang X, Ren S, Sun J. Delving deep into rectifiers: Surpass- ing human-level performance on imagenet classification. In: Proceed- ings of the IEEE international conference on computer vision. 2015, 1026–1034

2015
[192]

Imagenet large scale vi- sual recognition challenge

Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, others . Imagenet large scale vi- sual recognition challenge. International journal of computer vision, 2015, 115: 211–252

2015
[193]

Deep residual learning for image recognition

He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2016, 770–778

2016
[194]

Visualizing and understanding convolutional networks

Zeiler M D, Fergus R. Visualizing and understanding convolutional networks. In: Computer Vision–ECCV 2014: 13th European Confer- ence, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I

2014
[195]

Yolo9000: better, faster, stronger

Redmon J, Farhadi A. Yolo9000: better, faster, stronger. In: Proceed- ings of the IEEE conference on computer vision and pattern recogni- tion. 2017, 7263–7271

2017
[196]

Aggregated residual trans- formations for deep neural networks

Xie S, Girshick R, Doll ´ar P, Tu Z, He K. Aggregated residual trans- formations for deep neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2017, 1492– 1500

2017
[197]

Xception: Deep learning with depthwise separable convo- lutions

Chollet F. Xception: Deep learning with depthwise separable convo- lutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2017, 1251–1258

2017
[198]

Latte: accelerating lidar point cloud annotation via sensor fusion, one-click annotation, and track- ing

Wang B, Wu V , Wu B, Keutzer K. Latte: accelerating lidar point cloud annotation via sensor fusion, one-click annotation, and track- ing. In: 2019 IEEE Intelligent Transportation Systems Conference (ITSC). 2019, 265–272

2019
[199]

Popup: reconstructing 3d video using particle filtering to ag- gregate crowd responses

Song J Y , Lemmer S J, Liu M X, Yan S, Kim J, Corso J J, Lasecki W S. Popup: reconstructing 3d video using particle filtering to ag- gregate crowd responses. In: Proceedings of the 24th International Conference on Intelligent User Interfaces. 2019, 558–569

2019
[200]

Joint video object discovery and segmentation by coupled dynamic markov networks

Liu Z, Wang L, Hua G, Zhang Q, Niu Z, Wu Y , Zheng N. Joint video object discovery and segmentation by coupled dynamic markov networks. IEEE Transactions on Image Processing, 2018, 27(12): 5840–5853

2018
[201]

Segment- tube: Spatio-temporal action localization in untrimmed videos with per-frame segmentation

Wang L, Duan X, Zhang Q, Niu Z, Hua G, Zheng N. Segment- tube: Spatio-temporal action localization in untrimmed videos with per-frame segmentation. Sensors, 2018, 18(5): 1657

2018
[202]

Adversarial learning from crowds

Chen P, Sun H, Yang Y , Chen Z. Adversarial learning from crowds. In: Proceedings of the AAAI Conference on Artificial Intelligence. 2022, 5304–5312

2022
[203]

A review of motion planning techniques for automated vehicles

Gonz ´alez D, P ´erez J, Milan ´es V , Nashashibi F. A review of motion planning techniques for automated vehicles. IEEE Transactions on intelligent transportation systems, 2015, 17(4): 1135–1145

2015
[204]

Galaxy zoo

Fortson L, Masters K, Nichol R, Edmondson E, Lintott C, Raddick J, Wallin J. Galaxy zoo. Advances in machine learning and data mining for astronomy, 2012, 2012: 213–236

2012
[205]

Crowdsourcing in biomedicine: challenges and opportunities

Khare R, Good B M, Leaman R, Su A I, Lu Z. Crowdsourcing in biomedicine: challenges and opportunities. Briefings in bioinformat- ics, 2016, 17(1): 23–32

2016
[206]

Exploring automatic covid- 19 diagnosis via voice and symptoms from crowdsourced data

Han J, Brown C, Chauhan J, Grammenos A, Hasthanasombat A, Spathis D, Xia T, Cicuta P, Mascolo C. Exploring automatic covid- 19 diagnosis via voice and symptoms from crowdsourced data. In: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing...

2021
[207]

Predicting protein structures with a multiplayer online game

Cooper S, Khatib F, Treuille A, Barbero J, Lee J, Beenen M, Leaver- Fay A, Baker D, Popovi ´c Z, players F. Predicting protein structures with a multiplayer online game. Nature, 2010, 466(7307): 756–760

2010
[208]

Planning and executing scientifically sound community science in a public-facing institution

Nuessle T M, McNamara P A, Garneau N L. Planning and executing scientifically sound community science in a public-facing institution. Citizen Science: Theory and Practice, 2020, 5(1)

2020
[209]

Making better use of the crowd: How crowdsourcing can advance machine learning research

Vaughan J W. Making better use of the crowd: How crowdsourcing can advance machine learning research. J. Mach. Learn. Res., 2017, 18(1): 7026–7071

2017
[210]

The future of human-ai collaboration: a taxonomy of design knowledge for hybrid intelligence systems

Dellermann D, Calma A, Lipusch N, Weber T, Weigel S, Ebel P. The future of human-ai collaboration: a taxonomy of design knowledge for hybrid intelligence systems. arXiv preprint arXiv:2105.03354, 2021

2021 arXiv
[211]

A survey of crowdsourcing in medical image analysis

Ørting S, Doyle A, Hilten v A, Hirth M, Inel O, Madan C R, Mavridis P, Spiers H, Cheplygina V . A survey of crowdsourcing in medical image analysis. arXiv preprint arXiv:1902.09159, 2019

1902 arXiv
[212]

Crowd- sourcing in computer vision

Kovashka A, Russakovsky O, Fei-Fei L, Grauman K, others . Crowd- sourcing in computer vision. Foundations and Trends ® in computer graphics and Vision, 2016, 10(3): 177–243

2016
[213]

Quality control in crowdsourcing: A survey of quality attributes, as- sessment techniques, and assurance actions

Daniel F, Kucherbaev P, Cappiello C, Benatallah B, Allahbakhsh M. Quality control in crowdsourcing: A survey of quality attributes, as- sessment techniques, and assurance actions. ACM Computing Sur- veys (CSUR), 2018, 51(1): 1–40

2018
[214]

Understanding human-machine networks: a cross-disciplinary survey

Tsvetkova M, Yasseri T, Meyer E T, Pickering J B, Engen V , Walland 26 P, L¨uders M, Følstad A, Bravos G. Understanding human-machine networks: a cross-disciplinary survey. ACM Computing Surveys (CSUR), 2017, 50(1): 1–35

2017
[215]

A technical survey on statisti- cal modelling and design methods for crowdsourcing quality control

Jin Y , Carman M, Zhu Y , Xiang Y . A technical survey on statisti- cal modelling and design methods for crowdsourcing quality control. Artificial Intelligence, 2020, 287: 103351

2020
[216]

Quality is free: The art of making quality certain

Crosby P B. Quality is free: The art of making quality certain. (No Title), 1979

1979
[217]

Argument discovery via crowdsourcing

Nguyen Q V H, Duong C T, Nguyen T T, Weidlich M, Aberer K, Yin H, Zhou X. Argument discovery via crowdsourcing. The VLDB Journal, 2017, 26: 511–535

2017
[218]

Mobile crowd sensing and computing: The review of an emerging human- powered sensing paradigm

Guo B, Wang Z, Yu Z, Wang Y , Yen N Y , Huang R, Zhou X. Mobile crowd sensing and computing: The review of an emerging human- powered sensing paradigm. ACM computing surveys (CSUR), 2015, 48(1): 1–31

2015
[219]

Reliable aggrega- tion of boolean crowdsourced tasks

De Alfaro L, Polychronopoulos V , Shavlovsky M. Reliable aggrega- tion of boolean crowdsourced tasks. In: Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. 2015, 42– 51

2015
[220]

Crowdspeech and voxdiy: Benchmark datasets for crowdsourced audio transcription

Pavlichenko N, Stelmakh I, Ustalov D. Crowdspeech and voxdiy: Benchmark datasets for crowdsourced audio transcription. arXiv preprint arXiv:2107.01091, 2021

2021 arXiv
[221]

Crowdsourcing a dataset of audio captions

Lipping S, Drossos K, Virtanen T. Crowdsourcing a dataset of audio captions. arXiv preprint arXiv:1907.09238, 2019

1907 arXiv
[222]

Crowd-guided ensembles: How can we choreograph crowd workers for video segmentation? In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

Kaspar A, Patterson G, Kim C, Aksoy Y , Matusik W, Elgharib M. Crowd-guided ensembles: How can we choreograph crowd workers for video segmentation? In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 2018, 1–12

2018
[223]

Pricing mechanisms for crowdsourcing markets

Singer Y , Mittal M. Pricing mechanisms for crowdsourcing markets. In: Proceedings of the 22nd international conference on World Wide Web. 2013, 1157–1166

2013
[224]

Segmentation quality refinement in large- scale medical image dataset with crowd-sourced annotations

Cychnerski J, Dziubich T. Segmentation quality refinement in large- scale medical image dataset with crowd-sourced annotations. In: Eu- ropean Conference on Advances in Databases and Information Sys- tems. 2021, 205–216

2021
[225]

Cross-task crowdsourcing

Mo K, Zhong E, Yang Q. Cross-task crowdsourcing. In: Proceedings of the 19th ACM SIGKDD international conference on knowledge discovery and data mining. 2013, 677–685

2013
[226]

Crowddqs: Dynamic question selec- tion in crowdsourcing systems

Khan A R, Garcia-Molina H. Crowddqs: Dynamic question selec- tion in crowdsourcing systems. In: Proceedings of the 2017 ACM International Conference on Management of Data. 2017, 1447–1462

2017
[227]

Gpt-4 technical report

OpenAI R. Gpt-4 technical report. arXiv, 2023, 2303–08774

2023
[228]

Llms as workers in human- computational algorithms? replicating crowdsourcing pipelines with llms

Wu T, Zhu H, Albayrak M, Axon A, Bertsch A, Deng W, Ding Z, Guo B, Gururaja S, Kuo T S, others . Llms as workers in human- computational algorithms? replicating crowdsourcing pipelines with llms. arXiv preprint arXiv:2307.10168, 2023

2023 arXiv
[229]

Revisit- ing prompt engineering via declarative crowdsourcing

Parameswaran A G, Shankar S, Asawa P, Jain N, Wang Y . Revisit- ing prompt engineering via declarative crowdsourcing. arXiv preprint arXiv:2308.03854, 2023

2023 arXiv
[230]

Annollm: Making large language models to be better crowdsourced annotators

He X, Lin Z, Gong Y , Jin A, Zhang H, Lin C, Jiao J, Yiu S M, Duan N, Chen W, others . Annollm: Making large language models to be better crowdsourced annotators. arXiv preprint arXiv:2303.16854, 2023

2023 arXiv
[231]

Training language models to follow instructions with human feedback

Ouyang L, Wu J, Jiang X, Almeida D, Wainwright C, Mishkin P, Zhang C, Agarwal S, Slama K, Ray A, others . Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 2022, 35: 27730–27744

2022
[232]

Deep reinforcement learning from human preferences

Christiano P F, Leike J, Brown T, Martic M, Legg S, Amodei D. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 2017, 30

2017
[233]

Training a helpful and harmless assistant with reinforcement learning from human feedback

Bai Y , Jones A, Ndousse K, Askell A, Chen A, DasSarma N, Drain D, Fort S, Ganguli D, Henighan T, others . Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022

2022 arXiv
[234]

Palm 2 technical report

Anil R, Dai A M, Firat O, Johnson M, Lepikhin D, Passos A, Shakeri S, Taropa E, Bailey P, Chen Z, others . Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023

2023 arXiv
[235]

Llama 2: Open foundation and fine-tuned chat models

Touvron H, Martin L, Stone K, Albert P, Almahairi A, Babaei Y , Bashlykov N, Batra S, Bhargava P, Bhosale S, others . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

2023 arXiv
[236]

Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts

Wu T, Terry M, Cai C J. Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts. In: Proceedings of the 2022 CHI conference on human factors in com- puting systems. 2022, 1–22

2022
[237]

Crowdforge: Crowdsourc- ing complex work

Kittur A, Smus B, Khamkar S, Kraut R E. Crowdforge: Crowdsourc- ing complex work. In: Proceedings of the 24th annual ACM sympo- sium on User interface software and technology. 2011, 43–52

2011
[238]

Soylent: a word processor with a crowd inside

Bernstein M S, Little G, Miller R C, Hartmann B, Ackerman M S, Karger D R, Crowell D, Panovich K. Soylent: a word processor with a crowd inside. In: Proceedings of the 23nd annual ACM symposium on User interface software and technology. 2010, 313–322

2010
[239]

Self-consistency improves chain of thought reasoning in language models

Wang X, Wei J, Schuurmans D, Le Q, Chi E, Narang S, Chowdhery A, Zhou D. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022

2022 arXiv
[240]

Mea- suring and narrowing the compositionality gap in language models

Press O, Zhang M, Min S, Schmidt L, Smith N A, Lewis M. Mea- suring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350, 2022

2022 arXiv
[241]

Reflexion: Language agents with verbal reinforcement learning

Shinn N, Cassano F, Labash B, Gopinath A, Narasimhan K, Yao S. Reflexion: Language agents with verbal reinforcement learning. arXiv preprint arXiv:2303.11366, 2023

2023 arXiv
[242]

Artificial artificial artificial in- telligence: Crowd workers widely use large language models for text production tasks

Veselovsky V , Ribeiro M H, West R. Artificial artificial artificial in- telligence: Crowd workers widely use large language models for text production tasks. arXiv preprint arXiv:2306.07899, 2023

2023 arXiv
[243]

Crowdsourced data manage- ment: Industry and academic perspectives

Marcus A, Parameswaran A, others . Crowdsourced data manage- ment: Industry and academic perspectives. Foundations and Trends® in Databases, 2015, 6(1-2): 1–161

2015
[244]

Autopilot: work- load autoscaling at google

Rzadca K, Findeisen P, Swiderski J, Zych P, Broniek P, Kusmierek J, Nowak P, Strack B, Witusowski P, Hand S, others . Autopilot: work- load autoscaling at google. In: Proceedings of the Fifteenth European Conference on Computer Systems. 2020, 1–16

2020
[245]

A survey on task assignment in crowdsourcing

Hettiachchi D, Kostakos V , Goncalves J. A survey on task assignment in crowdsourcing. ACM Computing Surveys (CSUR), 2022, 55(3): 1–35 27 Lei Chai (Member, ACM) is cur- rently working toward the Ph.D. de- gree in software engineering with the School of Computer Science and En-...

2022

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [94]

Truth inference in crowdsourc- ing: Is the problem solved? Proceedings of the VLDB Endowment, 2017, 10(5): 541–552

Zheng Y , Li G, Li Y , Shan C, Cheng R. Truth inference in crowdsourc- ing: Is the problem solved? Proceedings of the VLDB Endowment, 2017, 10(5): 541–552

2017

[2] [1]

Faitcrowd: Fine grained truth discovery for crowdsourced data aggregation

Ma F, Li Y , Li Q, Qiu M, Gao J, Zhi S, Su L, Zhao B, Ji H, Han J. Faitcrowd: Fine grained truth discovery for crowdsourced data aggregation. In: Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining. 2015, 745–754

2015

[3] [2]

Whose vote should count more: Optimal integration of labels from labelers of un- known expertise

Whitehill J, Wu T f, Bergsma J, Movellan J, Ruvolo P. Whose vote should count more: Optimal integration of labels from labelers of un- known expertise. Advances in neural information processing systems, 2009, 22

2009

[4] [3]

Latent dirichlet allocation

Blei D M, Ng A Y , Jordan M I. Latent dirichlet allocation. the Journal of machine Learning research, 2003, 3: 993–1022

2003

[5] [4]

icrowd: An adaptive crowd- sourcing framework

Fan J, Li G, Ooi B C, Tan K l, Feng J. icrowd: An adaptive crowd- sourcing framework. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data. 2015, 1015–1030

2015

[6] [5]

The multidimensional wisdom of crowds

Welinder P, Branson S, Perona P, Belongie S. The multidimensional wisdom of crowds. 2011

2011

[7] [6]

Compar- ing twitter and traditional media using topic models

Zhao W X, Jiang J, Weng J, He J, Lim E P, Yan H, Li X. Compar- ing twitter and traditional media using topic models. In: European conference on information retrieval. 2011, 338–349

2011

[8] [7]

C-reference: Improving 2d to 3d object pose estimation accuracy via crowd- sourced joint object estimation

Song J Y , Chung J J Y , Fouhey D F, Lasecki W S. C-reference: Improving 2d to 3d object pose estimation accuracy via crowd- sourced joint object estimation. Proceedings of the ACM on Human- Computer Interaction, 2020, 4(CSCW1): 1–28

2020