Pith. sign in

REVIEW 3 major objections 6 minor 39 references

Improving annotator selection in Active Learning using a mood and fatigue-aware Recommender System

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that selecting active-learning annotators with a knowledge-based recommender that factors in past accuracy, mood, and fatigue reduces annotation errors and training uncertainty, nearly matching an optimized selector.

desk verdict A transparent simulation study showing how mood/fatigue-aware annotator ranking could fit into active learning, but the headline gains are circular because the simulator and the recommender share the same effect coefficients. read the letter →

arxiv 2507.23756 v1 pith:SKU4E5N7 submitted 2025-07-31 cs.LG cs.HC

Diana Mortagua This is my paper
classification cs.LGcs.HC
keywords ActiveLearningRecommenderSystemsAnnotatorSelectionMoodFatigueQuery-AnnotatorPairsKnowledge-BasedRecommendationHuman-in-the-loop
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This study claims that the best annotator for a queried instance in active learning is not simply the most accurate one historically, but the one who is accurate, in a good mood, and not fatigued at that moment. To test this, the author builds a knowledge-based recommender system that ranks available annotators per query using their past accuracy, current mood, and fatigue level, and uses it to pick who labels each instance in a simulated active-learning loop. Simulated annotators are given chronotype-driven mood curves and fatigue that grows with annotation count, with performance effects taken from published productivity studies plus conservative assumptions. Across five datasets, the mood- and fatigue-aware selection labeled 36.78% more instances correctly than past-accuracy-only selection at a 2% fatigue penalty, and 42.34% more at a 4% penalty, with smaller training uncertainty and slightly higher model accuracy and F1. The broader claim is that internal cognitive states are usable, not noise, in query-annotator assignment.

What carries the argument

The carrier is a Knowledge-Based Recommendation System used in reverse of its usual role: the queried instance plays the user, the annotators play the items, and a weighted similarity score ranks who should label. The score combines overall past accuracy, per-label past accuracy for the labels the model is most confused about, a mood adjustment where each unit of mood above or below the annotator's average changes predicted accuracy by 6%, and a fatigue discount triggered every 50 annotations and then every 20 after that, by either 2% or 4% per level. Chronotype shapes each simulated annotator's mood curve across three daily work periods, so the recommender's predictions line up with the same process that generates the labels.

What would settle it

Run the identical active-learning loop with real annotators: record their self-reported mood and their annotation count, then compare a recommender that discounts predicted accuracy by 6% per mood unit and 2-4% per fatigue threshold against one that ignores these factors; if the mood and fatigue-aware ranker does not produce more correct labels or lower model uncertainty, the fixed effect sizes are wrong.

Watch

Extended reading notes

Core claim

The central discovery is that adding mood and fatigue as features in a knowledge-based recommender for annotator selection makes active learning more efficient than using past accuracy alone, and nearly as efficient as a clairvoyant optimizer built from the simulator. The ranking score is a weighted sum of an annotator's historical accuracy, overall and per label that confuses the model, a mood adjustment of 6% per 1-unit deviation from that annotator's average mood, and a fatigue penalty of 2% or 4% per fatigue threshold crossed. In simulations over five datasets, the proposed Test 3 approach outperforms Test 1, past accuracy only, and Test 2, accuracy plus mood, on all four metrics: correct annotations, model accuracy, F1-score, and query uncertainty. The advantage grows when the fatigue penalty is larger, and is strongest on a difficult dataset where traditional active learning flounders near 50% accuracy.

Load-bearing premise

The whole advantage rests on the assumption that human annotators really do lose about 6% accuracy per unit of mood drop and 2-4% per fatigue threshold, because the recommender in the proposed test is predicting performance with exactly those numbers.

Editorial extensions

If this is right

  • If the approach holds, active-learning pipelines that currently pick the most historically accurate annotator can instead rank annotators by current predicted performance and reduce mislabeled training data without changing the query strategy.
  • Annotation teams could use the same mechanism to schedule work: workers nearer their mood or fatigue thresholds would be assigned fewer instances, and the system would absorb the effect as a score rather than as post-hoc label noise.
  • The gains are concentrated where traditional active learning already struggles; on easy datasets the benefit is mainly lower query uncertainty, not higher accuracy.
  • Because the optimized oracle is only slightly better than the proposed recommender, the remaining gap is a practical upper bound for how much improvement could come from better cognitive-state prediction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An editor's inference: if real mood and fatigue effects are heterogeneous across annotators, the fixed 6% and 2-4% values should be replaced by per-annotator learned coefficients, which the same ranking architecture can support.
  • An editor's inference: the paper's decision to exclude speed from the mood effect means real-world cost savings could be larger than the reported accuracy gains, since a better-mood annotator also labels faster.
  • An editor's inference: using uncertainty on queried instances as an outcome metric suggests one cheap deployment test: monitor model uncertainty before and after switching to a mood and fatigue-aware selector, even without ground-truth labels.
  • An editor's inference: the chronotype-based mood simulation, if validated against real annotator logs, could let platforms personalize work periods rather than only select among available annotators at a fixed time.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The manuscript proposes a knowledge-based recommender system (RS) for selecting query-annotator pairs in Active Learning (AL), using each annotator's past accuracy, current mood, and fatigue level as ranking features. The evaluation is entirely simulation-based: five public datasets, three synthetic batches of 30 annotators, and four tests (past accuracy only; plus mood; plus mood and fatigue; and an optimization-based upper bound). The reported results show that the mood-and-fatigue-aware Test 3 yields more correct annotations, higher model accuracy/F1, and lower query uncertainty than Test 1 (past accuracy only), with gains of 36.78% and 42.34% in correct annotations for fatigue penalties of 2% and 4%, respectively. The manuscript is transparent about the heuristic nature of several parameters and calls for future validation with real annotators.

Significance. If the claimed effect sizes are externally valid, the work addresses a real gap in AL: most query-annotator selection strategies ignore transient cognitive states such as mood and fatigue. The proposed RS is simple, interpretable, and the simulation framework is internally coherent. The study is also commendable for grounding mood effects in cited empirical literature and for explicitly presenting an optimization baseline. However, the central evidence is entirely synthetic, and the generative model shares its coefficients with the recommender's prediction function; the reported gains therefore currently demonstrate self-consistency of the simulation rather than a validated property of real annotation workflows. With an added sensitivity analysis or real-annotator data, the contribution could be solid; as it stands, the significance is conditional on assumptions the manuscript itself flags as heuristic.

major comments (3)
  1. [§4.2.3, §4.2.4, §4.3, §4.4] The central claim that mood/fatigue-aware annotation selection outperforms past-accuracy-only selection is supported only by a simulation whose ground truth is generated with the same coefficients that the recommender uses. Specifically, §4.2.3 sets the true accuracy change to 6% per unit of mood deviation, and §4.2.4 sets fatigue penalties of 2% or 4% per level; §4.3 and §4.4 then construct the RS's predicted accuracy in Tests 2 and 3 using exactly these values. Consequently, the reported improvements (36.78% and 42.34% more correct annotations, §5.2 and Chapter 6) quantify how well the recommender recovers its own generative process, not how well mood and fatigue can be exploited with real annotators. This is the load-bearing limitation for Research Questions 1 and 2. I recommend adding a sensitivity analysis over plausible ranges of mood/fatigue effect sizes (including zero and interactive effects) and explicitly reframing the conclusions as conditional on the assumed effect magnitudes, or providing validation against real annotator data.
  2. [§5.1, Figures 5.1–5.14] All reported results are means over three annotator batches, but no variance measures, confidence intervals, or per-batch breakdowns are provided for the headline metrics. The accuracy differences between tests are often small (e.g., 0.96% to 3.65% in §5.2); without an indication of between-batch variability, the reader cannot assess whether these differences are reliable or within the noise of the synthetic annotator sampling. Please include per-batch values or error bars/confidence bands in the learning curves and summary figures.
  3. [§4.4, Algorithm 2, §5.2] The comparison to the 'optimized approach' (Test 4) is constructed to be an upper bound, because Algorithm 2 selects the annotator whose predicted accuracy is computed from the same labeling simulation code used to generate the ground-truth labels. The observation that Test 3 'approaches' Test 4 is therefore built into the experimental design and should not be interpreted as an empirical finding about the closeness of the RS to an achievable optimum. The text in §5.2 already notes the similarity for Test 3 and Test 4; this framing should be made more explicit in the conclusions, where 'nearly as effective as the optimized approach' appears without this caveat.
minor comments (6)
  1. [§4.4] There appear to be two subsections numbered 4.4 ('Experiment design' and 'Chapter summary'). Re-number the latter as 4.5 or merge it into the preceding section.
  2. [Figures 5.13 and 5.14] Figures 5.13 and 5.14 have identical captions ('Fashion MNIST ... with fatigue levels worsening performance by 4%'). Please check which figure corresponds to which metric and correct the captions.
  3. [§1.3] In Research Question 4, the sentence 'To address a comparison is made by using the RS account for mood and fatigue' is grammatically incomplete and should be rewritten.
  4. [§4.2.2 and §3.4] There are several typographical errors and inconsistent acronyms, including 'bellowing' for 'belonging' (§4.2.2), 'Fand and associates' for 'Fang and associates' (§3.2), and 'CFB RS' where 'CBF RS' is meant (§3.4).
  5. [§5.2 and Chapter 6] The percentages '36.78%' and '42.34%' are relative improvements over Test 1; the text sometimes reads as if they are absolute accuracy differences. Please state clearly that these are relative gains in the number of correct annotations.
  6. [General] No information is provided about code availability or reproducibility artifacts (e.g., random seeds, environment). Given the simulation-heavy methodology, making the code and seed settings public would substantially strengthen the reproducibility of the study.

Circularity Check

2 steps flagged · score 7.0 of 10

Mood/fatigue advantages are produced by using the same coefficients in the annotator simulator and the recommender, making the main 'prediction' self-fulfilling.

  1. self definitional [Section 4.2.3 and Section 4.4]
    "For this reason, we assume that each unit of mood difference from the average results in a corresponding 6% change in the specific period: an increase for upward differences and a decrease for downward differences."

    This 6% coefficient does double duty: it is the generative rule for the simulated annotators' true per-period accuracy, and Section 4.4 states that Test 2's 'predicted accuracy ... considers past accuracy values and how the annotator's mood affects their performance at the moment.' The RS is therefore not independently predicting an external phenomenon; it is applying the same deterministic mood-to-accuracy mapping that created the labels. Test 3's advantage over Test 1 is built in, because mood variation is guaranteed to move true accuracy by exactly the amount the recommender assumes.

  2. self definitional [Section 4.2.4, Section 4.4, and Section 5.2]
    "This percentage is deducted from the annotator’s actual labeling performance in each threshold."

    The same threshold penalties (2% or 4% per fatigue level) are deducted in the simulator (Section 4.2.4) and in Test 3's predicted accuracy, which Section 4.4 says to 'decrease the predicted accuracy aforementioned by 2% or 4% at each threshold value of annotations.' The Discussion states the outcome is expected: 'since fatigue levels always affect the annotator in all tests, it is expected that Test 3, that considers this factor, will better predict how the available annotator will perform.' The 36.78% and 42.34% correct-label gains are therefore the recommender matching the simulator's own data-generating process, not empirical evidence that fatigue-aware ranking helps with real annotators.

full rationale

The central comparison is a closed loop. Sections 4.2.3 and 4.2.4 define the simulated annotators' true accuracy by applying a 6% mood adjustment and a 2%/4% fatigue penalty; Section 4.4 defines the RS's predicted accuracy using the same mood adjustment and the same 2%/4% fatigue penalties. Test 3 is therefore not predicting an independent target; it is applying the generative formula to past accuracy, so its ranking is near-optimal by construction. The Discussion openly says Test 3 is expected to win because fatigue levels always affect the annotator in all tests. The 36.78%/42.34% gains in correct annotations and the small accuracy/F1 improvements are the correct consequence of giving the recommender the simulator's hidden parameters, not evidence that mood/fatigue measurements would transfer to real annotators. The thesis is transparent that fatigue values are heuristic and calls for real-annotator validation, which lowers the charge from deliberate circularity to a design-level circularity in the simulation's evidence. Test 4 is also built from the labeling simulator, so Test 3's proximity to Test 4 is unsurprising. The paper's external-validity claim—that a mood/fatigue-aware RS would beat past-accuracy-only AL with humans—is untested and depends on the assumed coefficient values. No self-citation chain or uniqueness theorem is involved. The score reflects that the central predictive advantage reduces by construction, while the internal simulation remains coherent and the limitations are acknowledged.

Assumptions & free parameters 10 free parameters · 6 assumptions · 1 invented entities

The contribution is a prototype simulation whose conclusions are determined by its own assumptions. The central claim is not independently grounded: the same numerical effects are used to create the synthetic ground truth and to build the recommender's predictions. The ledger below lists the most important chosen values and assumptions.

free parameters (10)
  • mood_effect_per_unit = 0.06
    6% accuracy change per unit mood difference on a 1-to-10 scale; chosen as a conservative half of the 12% productivity increase reported in Bellet et al. 2019 and Oswald et al. 2015. Used in both the annotator simulation (Section 4.2.3) and the RS prediction.
  • fatigue_effect_per_level = 0.02 and 0.04
    The paper found no quantitative source for fatigue effects and 'required to assume a value'; two values (2% and 4%) are tested. Directly controls the simulated accuracy decline and is also applied by the RS.
  • fatigue_threshold_initial = 50
    Heuristic number of annotations before fatigue level 1; affects when the fatigue penalty starts in every simulation.
  • fatigue_threshold_increment = 20
    Each 20 additional annotations increments fatigue level; heuristic.
  • period_length_annotations = 204
    Heuristic number of annotations per work period; the paper admits no empirical basis.
  • annotator_accuracy_mean = 75
    Mean of the normal distribution for annotator average accuracy; the authors state these values 'do not follow any research'.
  • annotator_accuracy_std = 7
    Std of the normal distribution for annotator average accuracy; not research-based.
  • label_accuracy_std = 6
    Std of per-label accuracy around the annotator's overall accuracy; chosen to simulate natural label differences.
  • initial_past_annotations_per_label = 100
    Assumed number of past annotations per label when initializing performance histories.
  • mood_average_range = 3 to 7
    Average mood per annotator sampled uniformly in this range to avoid extreme values.
assumptions (6)
  • domain assumption Mood and fatigue causally affect annotation accuracy in the exact quantitative way assumed (6% per unit mood, 2% or 4% per fatigue level).
    This is the load-bearing premise of the simulation and the RS. Invoked in Sections 4.2.3 and 4.2.4. If real annotators differ, the method's benefit collapses.
  • domain assumption Chronotype distributions and productivity windows from Breus (2016), Fischer et al. (2017) and Sládek et al. (2020) transfer to arbitrary annotation workers.
    Used to generate mood fluctuations across periods (Section 4.2.2).
  • domain assumption Annotator per-label accuracy is normally distributed around their overall accuracy and updated after each day.
    Modeling choice in Section 4.2.3.
  • domain assumption Model predictive entropy is a valid proxy for query difficulty.
    The RS uses the query type based on predictions; Section 4.1.
  • domain assumption The weighted-sum score in the knowledge-based RS is a valid ordering of annotator suitability.
    The exact weight scheme (Algorithm 1) is not fully specified in the manuscript; its validity is assumed.
  • domain assumption Random forest with uncertainty sampling is an adequate representative AL setup.
    Section 4.2.5.
invented entities (1)
  • Simulated annotator agents
    purpose: To test the RS and the AL pipeline without real human annotators, with synthetic accuracy, mood, chronotype, and fatigue.
    All results are produced by these artificial agents; no real annotators are used, so there is no independent evidence that the simulated behavior matches human labelers.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving annotator selection in Active Learning using a mood and fatigue-aware Recommender System." pith.science (2026). https://pith.science/paper/SKU4E5N7

@misc{pith2026250723756,
  author       = {Pith},
  title        = {Pith review of: Improving annotator selection in Active Learning using a mood and fatigue-aware Recommender System},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SKU4E5N7}},
  note         = {Machine review of arXiv:2507.23756}
}
read the original abstract

This study centers on overcoming the challenge of selecting the best annotators for each query in Active Learning (AL), with the objective of minimizing misclassifications. AL recognizes the challenges related to cost and time when acquiring labeled data, and decreases the number of labeled data needed. Nevertheless, there is still the necessity to reduce annotation errors, aiming to be as efficient as possible, to achieve the expected accuracy faster. Most strategies for query-annotator pairs do not consider internal factors that affect productivity, such as mood, attention, motivation, and fatigue levels. This work addresses this gap in the existing literature, by not only considering how the internal factors influence annotators (mood and fatigue levels) but also presenting a new query-annotator pair strategy, using a Knowledge-Based Recommendation System (RS). The RS ranks the available annotators, allowing to choose one or more to label the queried instance using their past accuracy values, and their mood and fatigue levels, as well as information about the instance queried. This work bases itself on existing literature on mood and fatigue influence on human performance, simulating annotators in a realistic manner, and predicting their performance with the RS. The results show that considering past accuracy values, as well as mood and fatigue levels reduces the number of annotation errors made by the annotators, and the uncertainty of the model through its training, when compared to not using internal factors. Accuracy and F1-score values were also better in the proposed approach, despite not being as substantial as the aforementioned. The methodologies and findings presented in this study begin to explore the open challenge of human cognitive factors affecting AL.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 23 canonical work pages

  1. [1]

    «A Blockchain and Machine Learning-Based Drug Supply Chain Management and Recommendation System for Smart Pharmaceutical Industry»

    [Abbas et al., 2020] Abbas, Khizar, et al. «A Blockchain and Machine Learning-Based Drug Supply Chain Management and Recommendation System for Smart Pharmaceutical Industry». Electronics, vol. 9, n.o 5, 2020, p

    work page 2020

  2. [6]

    Genetic Basis of Chronotype in Humans: Insights From Three Landmark GWAS

    Social Science Research Network, https://doi.org/10.2139/ssrn.3116094. [Calma et al., 2018b] Calma, Adrian, et al. «Active Learning With Realistic Data - A Case Study». 2018 International Joint Conference on Neural Networks (IJCNN), 2018, pp. 1–8. IEEE Xplore, https://doi.org/10.1109/IJCNN.2018.8489394 [Chakraborty, 2020] Chakraborty, Shayok. «Asking the ...

    work page arXiv 1989

  3. [8]

    Journal contribution

    Carnegie Mellon University. Journal contribution. https://doi.org/10.1184/R1/6620732.v1 [Du and Ling, 2010] Du, Jun, e Charles X. Ling. «Active Learning with Human-Like Noisy Oracle». 2010 IEEE International Conference on Data Mining, 2010, pp. 797–802. IEEE Xplore, https://doi.org/10.1109/ICDM.2010.114 [Dunker, 1945] Duncker, K. (1945) On Problem-Solving...

    work page doi:10.1184/r1/6620732.v1 1945

  4. [12]

    Active Learning and Discovery of Object Categories in the Presence of Unnameable Instances,

    arXiv.org, https://doi.org/10.48550/arXiv.1708.07747. [Harmon-Jones et al., 2013] Harmon-Jones, Eddie, et al. «Does Negative Affect Always Narrow and Positive Affect Always Broaden the Mind? Considering the Influence of Motivational Intensity on Cognitive Scope». Current Directions in Psychological Science, vol. 22, n.o 4, 2013, pp. 301–07. DOI.org (Cross...

    arXiv doi:10.48550/arxiv.1708.07747 2020

  5. [15]

    [Kline et al., 2010] Kline, Christopher E., et al

    DOI.org (Crossref), https://doi.org/10.1093/sleep/zsw048. [Kline et al., 2010] Kline, Christopher E., et al. «Circadian rhythms of psychomotor vigilance, mood, and sleepiness in the ultra-short sleep/wake protocol». Chronobiology International, vol. 27, n.o 1, 2010, pp. 161–80. DOI.org (Crossref), https://doi.org/10.3109/07420521003648604. [Ko et al., 202...

    work page doi:10.1093/sleep/zsw048 2010

  6. [19]

    Understanding Random Forests: From Theory to Practice

    [Louppe, 2015] Louppe, Gilles. Understanding Random Forests: From Theory to Practice. arXiv:1407.7502, arXiv,

    arXiv 2015

  7. [22]

    Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI

    [Munro, 2021] Munro, Robert M.. Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI

    work page 2021

  8. [30]

    69 [Settles, 2011] Settles, Burr

    Google Scholar, https://minds.wisconsin.edu/handle/1793/60660. 69 [Settles, 2011] Settles, Burr. «From theories to queries: Active learning in practice». Active learning and experimental design workshop in conjunction with AISTATS 2010, JMLR Workshop and Conference Proceedings, 2011, pp. 1–18. Google Scholar, https://proceedings.mlr.press/v16/settles11a.h...

    arXiv 2013

Show all 39 references
  1. [33]

    Why we sleep: Unlocking the power of sleep and dreams

    http://hdl.handle.net/2105/35058. [Walker, 2017] Walker, M. (2017). “Why we sleep: Unlocking the power of sleep and dreams”. Scribner. [Wallace et al., 2011] Wallace, Byron C., et al. «Who Should Label What? Instance Allocation in Multiple Expert Active Learning». Proceedings ...

    2017 doi

  2. [35]

    [Prelec, 2004] Prelec D

    www.mdpi.com, https://doi.org/10.3390/informatics5030035. [Prelec, 2004] Prelec D. A Bayesian truth serum for subjective data. Science. 2004 Oct 15;306(5695):462-6. https://www.science.org/doi/10.1126/science.1102081 [Ramírez-Moreno et al., 2021] Ramírez-Moreno, M. A., Carrill...

    2021

  3. [36]

    «A probabilistic model of active learning with multiple noisy oracles»

    [Wu et al., 2013] Wu, Weining, et al. «A probabilistic model of active learning with multiple noisy oracles». Neurocomputing, vol. 118, 2013, pp. 253–62. ScienceDirect, https://doi.org/10.1016/j.neucom.2013.02.034. [Yan et al., 2011] Y. Yan, R. Rosales, G. Fung, and J. G. Dy, ...

    2013 arXiv

  4. [37]

    Unifying recommendation and active learning for information filtering and recommender systems

    arXiv.org, https://doi.org/10.48550/arXiv.1803.04223 [Yang et al., 2020] Yang, Scott Cheng-Hsin, et al. Unifying recommendation and active learning for information filtering and recommender systems

    2020 doi arXiv

  5. [38]

    «PAC Learning from Positive Statistical Queries»

    https://doi.org/10.3390/technologies12030038 [Denis, 1998] Denis, François. «PAC Learning from Positive Statistical Queries». Algorithmic Learning Theory, edited by Michael M. Richter et al., Springer, 1998, pp. 112–26. Springer Link, https://doi.org/10.1007/3-540-49730-7_9. [...

    1998

  6. [39]

    [Zheng et al., 2010] Zheng, Yaling, et al

    arXiv.org, https://doi.org/10.48550/arXiv.1401.3836. [Zheng et al., 2010] Zheng, Yaling, et al. «Active Learning from Multiple Noisy Labelers with Varied Costs». 2010 IEEE International Conference on Data Mining, 2010, pp. 639–48. IEEE Xplore, https://doi.org/10.1109/ICDM.2010...

    2010 doi arXiv

  7. [40]

    «Introduction»

    [Zhou, 2021] Zhou, Zhi-Hua. «Introduction». Machine Learning, edited by Zhi-Hua Zhou, Springer, 2021, pp. 1–24. Springer Link, https://doi.org/10.1007/978-981-15-1967-3_1 [Zhu et al., 2020] Zhu, Yu, et al. «Addressing the Item Cold-Start Problem by Attribute-Driven Active Lear...

    2021

  8. [54]

    [Rogstadius et al., 2011] Rogstadius, Jakob, et al

    www.mdpi.com, https://doi.org/10.3390/biology8030054. [Rogstadius et al., 2011] Rogstadius, Jakob, et al. «An Assessment of Intrinsic and Extrinsic Motivation on Task Performance in Crowdsourcing Markets». Proceedings of the International AAAI Conference on Web and Social Medi...

    2011

  9. [63]

    [Sandrock et al., 2019] Sandrock C., Herde M; Calma, Adrien, «Combining Self-reported Confidences from Uncertain Annotators to Improve Label Quality»

    BioMed Central, https://doi.org/10.1186/s12911-017-0431-7. [Sandrock et al., 2019] Sandrock C., Herde M; Calma, Adrien, «Combining Self-reported Confidences from Uncertain Annotators to Improve Label Quality». 2019 International Joint Conference on Neural Networks (IJCNN), 201...

    2019

  10. [141]

    [Koorengevel et al., 2003] Koorengevel, Kathelijne M., et al

    www.mdpi.com, https://doi.org/10.3390/electronics11010141. [Koorengevel et al., 2003] Koorengevel, Kathelijne M., et al. «Mood Regulation in Seasonal Affective Disorder Patients and Healthy Controls Studied in Forced Desynchrony». Psychiatry Research, vol. 117, n.o 1, 2003, pp...

    2003 doi

  11. [257]

    [Kazai et al., 2012] Kazai, Gabriella, et al

    DOI.org (Crossref), https://doi.org/10.1086/209347. [Kazai et al., 2012] Kazai, Gabriella, et al. «An Analysis of Human Factors and Label Accuracy in Crowdsourcing Relevance Judgments». Information Retrieval, vol. 16, n.o 2, 2013, pp. 138–78. DOI.org (Crossref), https://doi.or...

    2012 doi

  12. [592]

    «A survey of active learning in collaborative filtering recommender systems»

    https://doi.org/10.3390/s20030592 [Elahi et al., 2016] Elahi, Mehdi, et al. «A survey of active learning in collaborative filtering recommender systems». Computer Science Review, vol. 20, 2016, pp. 29–50. ScienceDirect, https://doi.org/10.1016/j.cosrev.2016.05.002. [Ellis et a...

    1997

  13. [820]

    Does Happiness Increase Productivity?

    www.mdpi.com, https://doi.org/10.3390/math11040820. [Thurstone, 1927] Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review, 34(4), 273–286. https://doi.org/10.1037/h0070288 [Tokarchuk et al.,2012] Tokarchuk, Oksana, et al. «Analyzing Crowd Labor and Des...

    1927 doi

  14. [852]

    Does Employee Happiness Have an Impact on Productivity?

    www.mdpi.com, https://doi.org/10.3390/electronics9050852. [Amershi et al., 2014] Amershi, Saleema, et al. «Power to the People: The Role of Humans in Interactive Machine Learning». AI Magazine, vol. 35, n.o 4, 2014, pp. 105–20. DOI.org (Crossref), https://doi.org/10.1609/aimag...

    2014 doi

  15. [932]

    «Genetic Basis of Chronotype in Humans: Insights From Three Landmark GWAS»

    PubMed, https://doi.org/10.1126/science.1192439 [Kalmbach et al., 2017] Kalmbach, David A., et al. «Genetic Basis of Chronotype in Humans: Insights From Three Landmark GWAS». Sleep, vol. 40, n.o 2,

    2017 doi

  16. [1419]

    https://doi.org/10.1038/s41598-020-58413-9 [Staw and Barsade, 1993] Staw, Barry M., e Sigal G. Barsade. «Affect and Managerial Performance: A Test of the Sadder-but-Wiser vs. Happier-and-Smarter Hypotheses». Administrative Science Quarterly, vol. 38, n.o 2, 1993, pp. 304–31. J...

    1993 doi

  17. [1989]

    https://doi.org/10.1080/02678378908256939 [Kruger and Dunning, 1999] Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: How difficulties in recognizing one's own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77(6), 11...

    1999

  18. [1993]

    [Wooldridge, 2009] Wooldridge, Michael

    DOI.org (Datacite), https://doi.org/10.24432/C5DW2B. [Wooldridge, 2009] Wooldridge, Michael. An Introduction to MultiAgent Systems. 2nd ed

    2009 doi

  19. [1998]

    Morgan Kaufmann

    In Proceedings of the International Conference on Machine Learning (ICML), pages 359–367. Morgan Kaufmann. [McHill et al., 2018] McHill, Andrew W., et al. «Chronic Sleep Curtailment, Even without Extended (>16-h) Wakefulness, Degrades Human Vigilance Performance». Proceedings ...

    1964

  20. [2001]

    The effects of employee satisfaction, organizational citizenship behavior, and turnover on organizational effectiveness: A unit-level, longitudinal study

    “The effects of employee satisfaction, organizational citizenship behavior, and turnover on organizational effectiveness: A unit-level, longitudinal study.” Personnel Psychology, vol. 54, n.o 1, 2001, pp. 101–14. DOI.org (Crossref), https://doi.org/10.1111/j.1744-6570.2001.tb0...

    2001

  21. [2008]

    [Settles, 2009] Settles, Burr

    Active Learning with Real Annotation Costs. [Settles, 2009] Settles, Burr. Active learning literature survey

    2009

  22. [2009]

    [Pozo, 2018] Pozo, Manuel, et al

    DOI.org (Datacite), https://doi.org/10.24432/C56S3T. [Pozo, 2018] Pozo, Manuel, et al. «Exploiting Past Users’ Interests and Predictions in an Active Learning Method for Dealing with Cold Start in Recommender Systems». Informatics, vol. 5, n.o 3, 2018, p

    2018 doi

  23. [2014]

    [Fischer et al., 2017] Fischer D, Lombardi DA, Marucci-Wellman H, Roenneberg T

    ojs.aaai.org, https://doi.org/10.1609/aaai.v28i1.8993. [Fischer et al., 2017] Fischer D, Lombardi DA, Marucci-Wellman H, Roenneberg T. Chronotypes in the US - Influence of age and sex. PLoS One

    2017 doi

  24. [2015]

    [Macedo, 2025] Macedo, L

    arXiv.org, https://doi.org/10.48550/arXiv.1407.7502. [Macedo, 2025] Macedo, L. (2025). Artificial Intelligence Paradigms and Agent-Based Technologies. In: Germanakos, P., Juhasz, M., Kongot, A., Marathe, D., Sacharidis, D. (eds) Human-Centered AI: An Illustrated Scientific Que...

    2025 doi arXiv

  25. [2016]

    Active Learning in Recommendation Systems with Multi-level User Preferences

    [Bu et al., 2018] Bu, Yuheng and Small, Kevin . Active Learning in Recommendation Systems with Multi-level User Preferences. arXiv:1811.12591, arXiv,

    2018 arXiv

  26. [2017]

    e0178782

    p. e0178782. DOI.org (Crossref), https://doi.org/10.1371/journal.pone.0178782. [Fredrickson and Branigan, 2005] Fredrickson, B. L., & Branigan, C. (2005). Positive emotions broaden the scope of attention and thought-action repertoires. Cognition and Emotion, 19(3), 313–332. ht...

    2005

  27. [2018]

    «Designing Interactions for Robot Active Learners»

    arXiv.org, https://doi.org/10.48550/arXiv.1811.12591 [Cakmak et al., 2010] Cakmak, Maya, et al. «Designing Interactions for Robot Active Learners». IEEE Transactions on Autonomous Mental Development, vol. 2, n.o 2, 2010, pp. 108–18. IEEE Xplore, https://doi.org/10.1109/TAMD.20...

    2010 doi arXiv

  28. [2019]

    The Power of When: Learn the Best Time to Do Everything

    Social Science Research Network, https://doi.org/10.2139/ssrn.3470734 [Breus, 2016] Breus, Dr Michael. The Power of When: Learn the Best Time to Do Everything. Ebury Publishing,

    2016 doi

  29. [2020]

    An Active Learning Approach for Jointly Estimating Worker Performance and Annotation Reliability with Crowdsourced Data

    PsyArXiv, https://doi.org/10.31234/osf.io/jqa83 [Zhao et al., 2014] Zhao, Liyue, et al. An Active Learning Approach for Jointly Estimating Worker Performance and Annotation Reliability with Crowdsourced Data. arXiv:1401.3836, arXiv,

    2014 arXiv

  30. [2021]

    «Mood and the circadian system: Investigation of a circadian component in positive affect».Chronobiology International, vol

    [Murray et al., 2002] Murray, Greg, et al. «Mood and the circadian system: Investigation of a circadian component in positive affect».Chronobiology International, vol. 19, n.o 6, 2002, pp. 1151–69. DOI.org (Crossref), https://doi.org/10.1081/CBI-120015956 [Nagarnaik and Thomas...

    2002

  31. [2024]

    [Rodrigues et al., 2014] F

    arXiv.org, https://doi.org/10.48550/arXiv.2407.13699. [Rodrigues et al., 2014] F. Rodrigues, F. Pereira, and B. Ribeiro, ‘‘Gaussian process classification and active learning with multiple annotators,’’ in Proc. Int. Conf. Mach. Learn., Bejing, China, 2014, pp. 433–441. [Roenn...

    2014 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.