Pith. sign in

REVIEW 1 cited by

Cross-replication Reliability -- An Empirical Approach to Interpreting Inter-rater Reliability

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2106.07393 v1 pith:Q6P7U5JJ submitted 2021-06-11 stat.AP cs.AIcs.SI

Ka Wong , Praveen Paritosh , Lora Aroyo This is my paper · ORCID
classification stat.APcs.AIcs.SI
keywords approachframeworkreliabilitycross-replicationempiricalinterpretingmeasurereplication
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

We present a new approach to interpreting IRR that is empirical and contextualized. It is based upon benchmarking IRR against baseline measures in a replication, one of which is a novel cross-replication reliability (xRR) measure based on Cohen's kappa. We call this approach the xRR framework. We opensource a replication dataset of 4 million human judgements of facial expressions and analyze it with the proposed framework. We argue this framework can be used to measure the quality of crowdsourced datasets.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. "I Would Not Be This Version of Myself Today": Elaborating on the Effects of Eudaimonic Gaming Experiences

    cs.HC 2025-07 conditional novelty 6.0 of 10

    Eudaimonic gaming experiences can produce lasting positive effects in reflection, learning, social life, health, and career, with stress and narrative insight shaping which effects occur.

Pith tools