Pith. sign in

REVIEW 4 major objections 5 minor 36 references

ORIS: Online Active Learning Using Reinforcement Learning-based Inclusive Sampling for Robust Streaming Analytics System

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims ORIS, a Deep Q-Network-based inclusive sampling policy for online active learning, reduces memory-decay labeling errors from simulated oracles and improves downstream model F1, reporting up to 44.2% human and 70.1%…

desk verdict A fresh RL active-learning idea undermined by a broken oracle equation and a partially circular evaluation. read the letter →

arxiv 2411.18060 v1 pith:OROXQWYU submitted 2024-11-27 cs.LG

Rahul Pandey , Ziwei Zhu , Hemant Purohit This is my paper · ORCID
classification cs.LG
keywords onlineactivelearningdeepreinforcementhuman-in-the-loopmachinehumanlabelingerrormemorydecaystreaminganalyticsemotionrecognition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that online active learning can be made robust to human memory-decay labeling errors by treating the document-sampling decision as a reinforcement learning problem. It proposes ORIS, a Deep Q-Network-based agent whose state combines the current document embedding with how long it has been since the oracle last labeled each class, and whose reward is an Inclusivity score measuring Shannon entropy of recently sampled class labels. The central claim is that this learned sampling policy reduces labeling errors from a forgetting-prone oracle and improves the downstream ML model compared with random, uncertainty, and offline diversity sampling. Evaluated on five-class emotion recognition over Twitter and Reddit streams, ORIS reports large gains in both human labeling F1-macro and machine F1-macro across two pretrained transformer-based learners. The reason this matters is that streaming analytics systems increasingly rely on human labels, and per-class memory freshness offers a teachable, transferable way to spend a limited labeling budget.

What carries the argument

The load-bearing mechanism is a Markov Decision Process formulation of online document sampling, solved with a Deep Q-Network. The state is the concatenation of the average pre-trained word embedding of the current document and the k-averaged time-last-seen $\Delta t_{c_i}$ for every class, so the agent sees both what is being asked and how recently the oracle has labeled each class. The reward is the Inclusivity score, the Shannon entropy of class labels among the past $m$ picked documents, passed through $\rho e^{\delta(\mathrm{Inclusivity}(M)-1)}$ for picks, with a small constant reward for discards; this rewards sampling histories that touch all classes recently while penalizing one-class runs. The paper also introduces two oracle error models, a parameterized sigmoid and an exponential memory-decay function, both of which make error probability grow with time since a class was last labeled.

What would settle it

Run the same 500-label active learning protocol with human annotators instead of the simulated oracle on the same emotion streams, recording each annotator's error rate against the time since each class was last seen; if the error rate does not rise with recency gap, ORIS's claimed mechanism for reducing human error would be contradicted.

Watch

Extended reading notes

Core claim

On the paper's own terms, ORIS discovers that a reinforcement-learned sampling policy can jointly protect the human annotator and the ML model by keeping every class recently seen. The policy, trained with a Deep Q-Network against an error-free oracle and deployed with a memory-decaying oracle, selects documents whose labels are diverse enough to keep the oracle's per-class memory fresh while still giving the model a balanced training signal. The paper reports that under a budget of 500 labels, ORIS reaches 56.2-60.6% machine F1-macro and 83.6-88.4% human F1-macro on Twitter, versus 40.4-47.3% machine and 63.5-72.1% human for random sampling, and similarly large margins on Reddit, where human F1-macro reaches 98.5-100.0%.

Load-bearing premise

The paper's reported gains rest on the assumption that human annotator error probability really does grow as an exponential or sigmoid function of time since a class was last labeled; if that simulated memory-decay model does not match real annotator behavior, the measured reduction in labeling errors will not carry over to practice.

Editorial extensions

If this is right

  • ORIS can replace the sampling step of an existing online active learning pipeline with a near-real-time policy whose inference cost is comparable to random sampling, since it avoids per-sample model confidence computation and offline clustering.
  • Including sampling diversity as a reward protects the rare classes in imbalanced streams, which is where error-prone oracles hurt most.
  • The learned policy transfers from Twitter to Reddit without retraining, suggesting that keeping all classes recently seen is a general, dataset-independent strategy.
  • Performance gains appear early in the budget: machine improvement is visible at roughly 150 labels and human improvement at roughly 50 labels, which matters for low-budget deployments.
  • The method is stated to extend to other modalities such as images and videos, and to other streaming analytics tasks, because the state and reward do not depend on text-specific features.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If real annotators forget according to a different curve than the fitted sigmoid or exponential, the reward and state calibration would need to be refit; a concrete extension is to log real annotator error against time-last-seen and retrain the DQN on that fitted curve.
  • Because the time-last-seen values are updated from oracle labels that may themselves be wrong, a single erroneous label corrupts the memory estimate the agent acts on; an untested variant is to maintain the memory estimate from the model's own predictions as a noise-free channel.
  • The Inclusivity reward looks back only $m=10$ picks, so very long-range class starvation may fall outside its horizon; a testable extension is a variant with a longer memory window or an additive count-based bonus for rarely seen classes.
  • ORIS's cross-domain success suggests the policy learns to equalize class recency rather than dataset idiosyncrasies; this can be tested by varying class imbalance and budget while measuring whether both human and machine F1 remain stable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ORIS, a Deep Q-Network-based online active learning strategy that samples incoming documents to jointly improve machine model performance and reduce simulated human labeling errors caused by memory decay. The method's state representation combines document embeddings with average time-last-seen values per class, and the reward is a function of the Shannon entropy (inclusivity) of recently sampled class labels. The authors evaluate on Twitter and Reddit emotion datasets against random, uncertainty, and offline diversity sampling, reporting improvements in both machine F1-macro and simulated human F1-macro. The code is made available.

Significance. If the results were valid, the idea of using reinforcement learning to explicitly encourage inclusive sampling would be a useful contribution to online active learning for streaming analytics, especially in settings where annotator memory decay is a concern. The paper ships code and uses a standard DQN architecture, which aids reproducibility. However, the significance is substantially undercut because the evaluation is based entirely on a synthetic oracle, the key oracle equation (Eq 2) is mathematically invalid as written, and the reward function aligns with the simulated error model in a way that makes the reported human-performance gains largely a property of the simulator. No real human annotation data are presented, so the central claim that ORIS minimizes human labeling errors is not empirically supported.

major comments (4)
  1. [Section III-B, Eq 2] The 'exponential forgetting' oracle defined in Eq 2 as error probability score(c) = max(1, e^{αΔt+β}) is not a valid probability and is internally inconsistent with the reported results. With the stated hyperparameters α=0.6 and β=−19 (Section V-B1), for any Δt < ~31.7 the score is exactly 1, and for larger Δt it exceeds 1, so the oracle always (or more than always) errs. This contradicts Table III, where ORIS (δ=8) under the exponential oracle achieves human F1-macro of 100.0 on Reddit. Even if the intended expression was min(1, e^{...}), the paper must state this correction and re-run the experiments; as written, the simulation and the headline numbers are irreconcilable.
  2. [Section IV-C3 and Section V-B1] The reward function (Eq 4) directly incentivizes high entropy of the sampled class labels, while the simulated oracle's error probability (Eq 1) is minimized precisely when all classes have been seen recently. The DQN is trained with an error-free oracle and only this inclusivity reward (Section IV-D). Consequently, the reported human-performance improvements are a property of the simulator, not evidence that ORIS reduces human errors in real annotation. To support the central claim, the authors need either real human annotation experiments or a more conservative statement that the method reduces errors under the assumed memory-decay model.
  3. [Abstract and Section I] The headline improvement percentages ('up to 38.3% human & 55.7% machine performance improvement on Twitter, and up to 44.2% human & 70.1% machine performance improvement on Reddit') are not reproducible from Tables II and III unless the baseline is selectively chosen. For example, the 55.7% machine improvement on Twitter corresponds to comparing BERT Mini sigmoid ORIS (56.2) against the offline Diversity baseline (36.1), while the 38.3% human improvement comes from comparing against Uncertainty (62.7). The paper should state the reference baseline and configuration for each claimed percentage, or drop these uncorroborated numbers.
  4. [Section IV-C1] The state representation computes time-last-seen values from the oracle's provided labels, which may be erroneous. A mislabeled document will fail to update the time-last-seen of the true class, and the k-averaging strategy does not fully repair this corruption. Since the DQN policy is trained exclusively on error-free oracle labels, the state distribution at inference (which includes erroneous updates) may be off-distribution, yet this mismatch is not analyzed. The paper should either model this noise during training or provide empirical evidence that the policy remains robust.
minor comments (5)
  1. [Equation 4] There is a typo in Eq 4: an extra closing parenthesis appears in the expression for the reward.
  2. [Algorithm 1, line 10] The notation 'Di t+1' is ambiguous; it should likely be 'Dt+1' or the dataset index and stream position should be defined more clearly.
  3. [Section IV-C1] The notation for time-last-seen is inconsistent: the text alternates between '∆tt', '∆tci', and '∆tj ci', which makes the state construction harder to follow.
  4. [Table IV] Table IV reports total experiment durations, which are dominated by BERT fine-tuning, so the claim that ORIS is 'near real-time' is not directly evidenced; a per-decision or sampling-only timing would be more informative.
  5. [Section VI-C] Figure 3 is referenced but not clearly described in the text; the caption should be expanded so that the confidence intervals and markers are self-explanatory.

Circularity Check

1 steps flagged · score 6.0 of 10

The claimed reduction in human labeling errors is largely the ORIS reward objective (Eq. 4) restated through the paper's own memory-decay oracle (Eqs. 1-2); the evaluation is a self-consistent simulation, not an independent test.

  1. fitted input called prediction [Sec. IV-C3 Eq. 4 (reward) vs. Sec. III-B Eqs. 1-2 (oracle) and Sec. V-D (human performance)]
    "ComputeReward (a, M) = ρ ∗ e^{δ(Inclusivity (M )−1))) a = 1; λ a = 0 (4), where Inclusivity(M) = −Σ p_M(c_i) log2 p_M(c_i); the oracle's error is 'error probability score(c) = 1/(1+e^{−α∆tc+β}) (1)' and 'max(1, e^{α∆tc+β}) (2)'."

    The human-performance metric is the oracle's correctness, and the oracle's error is, by Eqs. 1-2, an increasing function of Δt_c, the time since class c was last labeled. The reward (Eq. 4) is the Shannon entropy of class labels in the last m sampled documents; maximizing that entropy forces all classes to be sampled repeatedly, which drives every Δt_c down and therefore drives the oracle's error probability to its minimum. Thus ORIS's high human F1 in Tables II-III is not an independent empirical finding: it is the optimization objective of the DQN restated as an evaluation metric.

full rationale

The central human-error claim is evaluated with a simulator whose error model (Eqs. 1-2) is the exact mechanism the ORIS reward (Eq. 4) was built to exploit, so the reported human-F1 superiority over baselines is largely a consequence of the reward design rather than a validation against real annotators; no real human labeling data are presented. The machine-F1 improvements still involve genuine BERT fine-tuning and are not circular, which is why the score is 6 rather than 8-10. Separately, Eq. 2 as printed ('max(1, e^{αΔt+β})') appears degenerate: with α=0.6 and β=−19 the expression equals 1 for every Δt, which is not a probability and is inconsistent with the human-F1 values near 100 in Table III; this is a correctness issue rather than a circularity issue. The sigmoid oracle is inherited from the authors' prior work [6],[7], but the paper also cites external psychology literature for exponential forgetting, so the self-citation is present but not the sole load-bearing element.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities. The key free parameters are the oracle decay parameters and reward shaping constants, all chosen by hand. The main domain assumptions are about the validity of the memory-decay model and the sufficiency of the hand-crafted state representation.

free parameters (6)
  • Oracle decay parameters (alpha, beta) for sigmoid and exponential forgetting = Sigmoid: alpha=0.3, beta=9; Exponential: alpha=0.6, beta=-19
    These parameters define the simulated human error rates in Section V-B2. They are chosen by the authors, not fitted to real annotation data, and directly determine the 'human performance' metric.
  • Reward activation delta = delta=8 or delta=16
    Controls how sharply low inclusivity scores are deactivated in the reward (Eq 4). Two values are tested; results vary with delta.
  • Reward scaling rho = rho=5
    Scales the positive reward for picking a document. Listed in Section V-B1.
  • Discard reward lambda = lambda=0.01
    Small reward for the discard action, used to shape exploration in the DQN.
  • Memory length m = m=10
    Number of past picked documents used to compute the Inclusivity (Shannon entropy) score in the reward function.
  • Time-last-seen averaging window k = k=3
    Number of most recent time-last-seen values averaged for each class in the state representation (Fig. 1).
assumptions (4)
  • domain assumption Human annotation errors follow the parameterized sigmoid or exponential decay based on time since the class was last seen (Eq 1 and Eq 2).
    This is the core model of human memory decay, inherited from psychology and prior work, but it is not validated for the specific annotator population or task in this paper.
  • domain assumption The k-averaged time-last-seen per class, computed from possibly erroneous oracle labels, is a sufficient statistic for the oracle's memory state.
    When the oracle gives a wrong label, the time-last-seen for the true class is not updated, while the erroneous class is updated. The state may therefore misrepresent the oracle's actual memory, yet the paper assumes this is reliable enough for the agent.
  • domain assumption The document embedding, computed as the mean of pretrained FastText word vectors, contains enough information for the agent to select informative documents.
    The state representation (Eq 3) uses only this mean embedding plus class recency features. No contextual or task-specific representation is used for sampling decisions.
  • standard math A Deep Q-Network with a simple feed-forward architecture (Eq 5) can learn an effective sampling policy for this MDP.
    The paper assumes standard DQN training (replay buffer, target network, epsilon-greedy) converges to a good policy with the provided hyperparameters.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ORIS: Online Active Learning Using Reinforcement Learning-based Inclusive Sampling for Robust Streaming Analytics System." pith.science (2026). https://pith.science/paper/OROXQWYU

@misc{pith2026241118060,
  author       = {Pith},
  title        = {Pith review of: ORIS: Online Active Learning Using Reinforcement Learning-based Inclusive Sampling for Robust Streaming Analytics System},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OROXQWYU}},
  note         = {Machine review of arXiv:2411.18060}
}
read the original abstract

Effective labeled data collection plays a critical role in developing and fine-tuning robust streaming analytics systems. However, continuously labeling documents to filter relevant information poses significant challenges like limited labeling budget or lack of high-quality labels. There is a need for efficient human-in-the-loop machine learning (HITL-ML) design to improve streaming analytics systems. One particular HITL- ML approach is online active learning, which involves iteratively selecting a small set of the most informative documents for labeling to enhance the ML model performance. The performance of such algorithms can get affected due to human errors in labeling. To address these challenges, we propose ORIS, a method to perform Online active learning using Reinforcement learning-based Inclusive Sampling of documents for labeling. ORIS aims to create a novel Deep Q-Network-based strategy to sample incoming documents that minimize human errors in labeling and enhance the ML model performance. We evaluate the ORIS method on emotion recognition tasks, and it outperforms traditional baselines in terms of both human labeling performance and the ML model performance.

Figures

Figures reproduced from arXiv: 2411.18060 by the authors.

Figure 1
Figure 1. Overview of the proposed ORIS architecture. The method aims to sample inclusive documents from streams using a DQN-based agent. Components [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Online Active Learning System Flow. B. Human Memory and Labeling Error The memory decay behavior of humans is widely studied in psychology literature, which shows that memory retention de￾creases exponentially with time. Prior research [8] provides a human error taxonomy that distinguishes between two classes of errors: mistakes and slips as discussed in the Introduction. Prior research showed that a similar taxonom… view at source ↗
Figure 3
Figure 3. Machine and Human performance comparison over budget exhausted for both Twitter and Reddit Dataset when using the BERT mini and mBERT [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 30 canonical work pages

  1. [1]

    Castillo, Big crisis data: social media in disasters and time-critical situations

    C. Castillo, Big crisis data: social media in disasters and time-critical situations. Cambridge University Press, 2016

    work page 2016

  2. [2]

    A survey on concept drift adaptation,

    J. Gama et al., “A survey on concept drift adaptation,” ACM Computing Surveys, vol. 46, no. 4, pp. 1–37, Apr. 2014. [Online]. Available: https://dl.acm.org/doi/10.1145/2523813

    doi:10.1145/2523813 2014

  3. [3]

    A Survey of Deep Active Learning,

    P. Ren et al. , “A Survey of Deep Active Learning,” ACM Computing Surveys, vol. 54, no. 9, pp. 180:1–180:40, Oct. 2021. [Online]. Available: https://doi.org/10.1145/3472291

    doi:10.1145/3472291 2021

  4. [4]

    Engineering Crowdsourced Stream Processing Systems,

    M. Imran et al. , “Engineering Crowdsourced Stream Processing Systems,” arXiv:1310.5463 [cs] , Aug. 2014, arXiv: 1310.5463. [Online]. Available: http://arxiv.org/abs/1310.5463

    arXiv 2014

  5. [5]

    Design Patterns for Hybrid Algorithmic- Crowdsourcing Workflows,

    C. Lofi and K. E. Maarry, “Design Patterns for Hybrid Algorithmic- Crowdsourcing Workflows,” in IEEE CBI , vol. 1, Jul. 2014, pp. 1–8, iSSN: 2378-1971

    work page 2014

  6. [6]

    Modeling human annotation errors to design bias-aware systems for social stream processing,

    R. Pandey, C. Castillo, and H. Purohit, “Modeling human annotation errors to design bias-aware systems for social stream processing,” in ASONAM, 2019, pp. 374–377

    work page 2019

  7. [7]

    Modeling and mitigating human annotation errors to design efficient stream processing systems with human-in-the-loop machine learning,

    R. Pandey, H. Purohit, C. Castillo, and V . L. Shalin, “Modeling and mitigating human annotation errors to design efficient stream processing systems with human-in-the-loop machine learning,” IJHCS, vol. 160, p. 102772, Apr. 2022. [Online]. Available: https://linkinghub. elsevier.com/retrieve/pii/S1071581922000015

    work page 2022

  8. [8]

    Human error: models and management,

    J. Reason, “Human error: models and management,” BMJ, vol. 320, no. 7237, pp. 768–770, Mar. 2000. [Online]. Available: https://www.bmj.com/content/320/7237/768

    work page 2000

Show all 36 references
  1. [9]

    Human-level control through deep reinforcement learning,

    V . Mnih et al. , “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, Feb. 2015, number: 7540 Publisher: Nature Publishing Group. [Online]. Available: https://www.nature.com/articles/nature14236

    2015

  2. [10]

    Multi-modal Active Learning From Human Data: A Deep Reinforcement Learning Approach,

    O. Rudovic, M. Zhang, B. Schuller, and R. Picard, “Multi-modal Active Learning From Human Data: A Deep Reinforcement Learning Approach,” in ICMI. Suzhou China: ACM, Oct. 2019, pp. 6–15. [Online]. Available: https://dl.acm.org/doi/10.1145/3340555.3353742

    2019

  3. [11]

    Cold-start Active Learning through Self-supervised Language Modeling,

    M. Yuan, H.-T. Lin, and J. Boyd-Graber, “Cold-start Active Learning through Self-supervised Language Modeling,” in EMNLP. Online: ACL, Nov. 2020, pp. 7935–7948. [Online]. Available: https://aclanthology.org/2020.emnlp-main.637

    2020

  4. [12]

    Feedback-driven multiclass active learning for data streams,

    Y . Cheng et al. , “Feedback-driven multiclass active learning for data streams,” in CIKM. New York, NY , USA: ACM, Oct. 2013, pp. 1311–

    2013

  5. [13]

    Online active learning with expert advice,

    S. Hao et al., “Online active learning with expert advice,”TKDD, vol. 12, no. 5, pp. 1–22, 2018

    2018

  6. [14]

    Active Learning With Drifting Streaming Data,

    I. ˇZliobait˙e, A. Bifet, B. Pfahringer, and G. Holmes, “Active Learning With Drifting Streaming Data,” IEEE Transactions on Neural Networks and Learning Systems , vol. 25, no. 1, pp. 27–39, Jan. 2014

    2014

  7. [15]

    Learning how to Active Learn: A Deep Reinforcement Learning Approach,

    M. Fang, Y . Li, and T. Cohn, “Learning how to Active Learn: A Deep Reinforcement Learning Approach,” in EMNLP. Copenhagen, Denmark: ACL, Sep. 2017, pp. 595–605. [Online]. Available: https://aclanthology.org/D17-1063

    2017

  8. [16]

    Deep Reinforcement Active Learning for Human-in-the- Loop Person Re-Identification,

    Z. Liu et al., “Deep Reinforcement Active Learning for Human-in-the- Loop Person Re-Identification,” in ICCV, Oct. 2019, pp. 6121–6130, iSSN: 2380-7504

    2019

  9. [17]

    Reinforced active learning for image segmentation,

    A. Casanova, P. O. Pinheiro, N. Rostamzadeh, and C. J. Pal, “Reinforced active learning for image segmentation,” Sep. 2019. [Online]. Available: https://openreview.net/forum?id=SkgC6TNFvr

    2019

  10. [18]

    AI-Based Request Augmentation to Increase Crowdsourcing Participation,

    J. Park et al. , “AI-Based Request Augmentation to Increase Crowdsourcing Participation,” HCOMP, vol. 7, pp. 115–124, Oct. 2019. [Online]. Available: https://ojs.aaai.org/index.php/HCOMP/article/view/ 5282

    2019

  11. [19]

    Making Better Use of the Crowd: How Crowdsourcing Can Advance Machine Learning Research,

    J. W. Vaughan, “Making Better Use of the Crowd: How Crowdsourcing Can Advance Machine Learning Research,” JMLR, vol. 18, no. 193, pp. 1–46, 2018. [Online]. Available: http://jmlr.org/papers/v18/17-234.html

    2018

  12. [20]

    Some tests of the decay theory of immediate memory,

    J. Brown, “Some tests of the decay theory of immediate memory,” Quarterly journal of experimental psychology, vol. 10, no. 1, pp. 12–21, 1958

    1958

  13. [21]

    Ebbinghaus, Memory: A Contribution to Experimental Psychology

    H. Ebbinghaus, Memory: A Contribution to Experimental Psychology . Teachers College, Columbia University, 1913, no. 3

    1913

  14. [22]

    Reflections of the environment in memory,

    J. R. Anderson and L. J. Schooler, “Reflections of the environment in memory,” Psychological science, vol. 2, no. 6, pp. 396–408, 1991

    1991

  15. [23]

    Evaluating forgetting curves,

    G. R. Loftus, “Evaluating forgetting curves,” Journal of Experimental Psychology: Learning, Memory, and Cognition , vol. 11, no. 2, pp. 397– 406, 1985, place: US Publisher: American Psychological Association

    1985

  16. [24]

    A markovian decision process,

    R. Bellman, “A markovian decision process,” Journal of mathematics and mechanics, pp. 679–684, 1957

    1957

  17. [25]

    A mathematical theory of communication,

    C. E. Shannon, “A mathematical theory of communication,” ACM SIGMOBILE Mobile Computing and Communications Review , vol. 5, no. 1, pp. 3–55, Jan. 2001. [Online]. Available: https://dl.acm.org/doi/ 10.1145/584091.584093

    2001

  18. [26]

    CARER: Contextualized affect representations for emotion recognition,

    E. Saravia et al. , “CARER: Contextualized affect representations for emotion recognition,” in EMNLP. Brussels, Belgium: ACL, Oct.-Nov. 2018, pp. 3687–3697. [Online]. Available: https://www.aclweb.org/ anthology/D18-1404

    2018

  19. [27]

    GoEmotions: A dataset of fine-grained emotions,

    D. Demszky et al., “GoEmotions: A dataset of fine-grained emotions,” in ACL. Online: ACL, Jul. 2020, pp. 4040–4054. [Online]. Available: https://aclanthology.org/2020.acl-main.372

    2020

  20. [28]

    Enriching word vectors with subword information,

    P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, “Enriching word vectors with subword information,” TACL, vol. 5, pp. 135–146, 2017

    2017

  21. [29]

    Loss in translation: Learning bilingual word mapping with a retrieval criterion,

    A. Joulin et al., “Loss in translation: Learning bilingual word mapping with a retrieval criterion,” in EMNLP, 2018

    2018

  22. [30]

    Generalization in nli: Ways (not) to go beyond simple heuristics,

    P. Bhargava, A. Drozd, and A. Rogers, “Generalization in nli: Ways (not) to go beyond simple heuristics,” 2021

    2021

  23. [31]

    Well-read students learn better: On the importance of pre-training compact models,

    I. Turc, M.-W. Chang, K. Lee, and K. Toutanova, “Well-read students learn better: On the importance of pre-training compact models,” 2020. [Online]. Available: https://openreview.net/forum?id=BJg7x1HFvB

    2020

  24. [32]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in NAACL. Minneapolis, Minnesota: ACL, Jun. 2019, pp. 4171–4186. [Online]. Available: https://aclanthology.org/N19-1423

    2019

  25. [33]

    Active Learning with Clustering,

    Z. Bod ´o, Z. Minier, and L. Csat ´o, “Active Learning with Clustering,” in ALED workshop (AISTATS 2010) . JMLR, Apr. 2011, pp. 127–139, iSSN: 1938-7228. [Online]. Available: https://proceedings.mlr.press/ v16/bodo11a.html

    2010

  26. [34]

    Off to a good start: Using clustering to select the initial training set in active learning,

    R. Hu et al., “Off to a good start: Using clustering to select the initial training set in active learning,” 2010

    2010

  27. [35]

    Cost-effective active learning for deep image classifica- tion,

    K. Wang et al., “Cost-effective active learning for deep image classifica- tion,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 27, no. 12, pp. 2591–2600, 2016

    2016

  28. [1320]

    Available: https://doi.org/10.1145/2505515.2505528

    [Online]. Available: https://doi.org/10.1145/2505515.2505528

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.