Pith. Sign in

RECORD

Optimal Crowdsourced Classification with a Reject Option in the Presence of Spammers

PaperPDF from arXiv

data files

Paper breakdownThe authors' claims, conditions and measurements, as JSON Signed receiptProof this review has not been altered
X Bluesky LinkedIn Reddit Hacker News
Qunwei Li, Pramod K. Varshney

Human-Computer Interaction · Submitted 26 Oct 2017

abstract compiled · 2026-09-23 · no language-model review

The authors' main claim We present an aggregation approach using a weighted majority voting rule, where each worker's response is assigned an optimized weight to maximize crowd's classification performance.

how it is stated · asserted 1.00 read the record →

Comment (0) Read on arXiv ↗
Cite ▾
arXiv ID1710.09901 (version 1)
Pith ID2CGSHPLP, Pith's permanent record of this paper and its review
CitationPramod K. Varshney, Qunwei Li. "Optimal Crowdsourced Classification with a Reject Option in the Presence of Spammers." arXiv:1710.09901 (2017). https://pith.science/paper/2CGSHPLP
Claim paper

The compiled record

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Open the compiled record

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 3,555 input tokens.

What the abstract itself states, as probabilities: claims first 0.03 · states limitation 0.10 · human participants 0.86 · controlled comparison 0.05 · randomized assignment 0.02 · states quantitative result 0.04 · compares with prior results 0.07 · mentions code or data availability 0.02.

The abstract, sentence by sentence

arXiv listing abstract v1. Results and the main claim are underlined; .

We explore the design of an effective crowdsourcing system for an $M$-ary classification task. Crowd workers complete simple binary microtasks whose results are aggregated to give the final decision. We consider the scenario where the workers have a reject option so that they are allowed to skip microtasks when they are unable to or choose not to respond to binary microtasks. We present an aggregation approach using a weighted majority voting rule, where each worker's response is assigned an optimized weight to maximize crowd's classification performance.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#sentencerolethe authors' own workstates the main conclusionstrength
s0 We explore the design of an effective crowdsourcing system for an $M$-ary classification task. contribution 0.62 0.57 0.04 asserted 0.67
s1 Crowd workers complete simple binary microtasks whose results are aggregated to give the final decision. method 0.64 0.17 0.03 asserted 0.99
s2 We consider the scenario where the workers have a reject option so that they are allowed to skip microtasks when they are unable to or choose not to respond to binary microtasks. background 0.78 0.19 0.04 asserted 0.95
s3 We present an aggregation approach using a weighted majority voting rule, where each worker's response is assigned an optimized weight to maximize crowd's classification performance. method 0.65 0.89 0.41 asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from proof or derivation 0.60.

What was examined, and what was not

sources read
  • arXiv listing abstract v1
not examined
  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers
levels run
  • P abstract
not run
  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

The paper
Original abstract

We explore the design of an effective crowdsourcing system for an $M$-ary classification task. Crowd workers complete simple binary microtasks whose results are aggregated to give the final decision. We consider the scenario where the workers have a reject option so that they are allowed to skip microtasks when they are unable to or choose not to respond to binary microtasks. We present an aggregation approach using a weighted majority voting rule, where each worker's response is assigned an optimized weight to maximize crowd's classification performance.

Human review

How Pith checks papers → · Full audit of this paper →

Discussion

Know something about this paper? Leave a comment, or highlight any passage above to comment on it.