Pith. sign in

REVIEW 4 major objections 5 minor 40 references

Agents that request human feedback only on their most uncertain state-action pairs learn faster than agents given random or no feedback, and beat both baselines on a UVA/Padova blood-glucose control task.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Extending the Advise algorithm with variational crowd modelling and entropy-based query selection yields faster learning in small tabular RL tasks, especially highly constrained ones.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A solid, honest extension of Advise for crowd feedback with active query selection; the math is mostly sound and the BGL results are promising, but the empirical evidence is thinner than the abstract suggests and the prior-inconsistency stress-test doesn't hold up. the 4 major comments →

arxiv 2508.19132 v1 pith:XFLNE66J submitted 2025-08-26 cs.LG

Active Query Selection for Crowd-Based Reinforcement Learning

Jonathan Erskine , Taku Yamagata , Ra\'ul Santos-Rodr\'iguez This is my paper
classification cs.LG
keywords reinforcement learning from human feedbackactive query selectionlearning from crowdsentropy-based uncertainty samplingpolicy shapingvariational inferenceblood glucose controltabular Q-learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tackles a practical bottleneck in feedback-driven reinforcement learning: human feedback is expensive, noisy, and uneven in quality, so every query should count. It proposes a framework that extends the Advise policy-shaping algorithm in two directions: a variational crowd model that estimates each trainer's reliability online instead of assuming it, and an entropy-based active query rule that requests feedback only for the state-action pairs the agent is most uncertain about. The uncertainty measure is a new one-vs-all entropy defined on the posterior probability that an action is optimal, which peaks when the posterior matches chance across the action set. Tested on gridworld games and a Type 1 diabetes blood-glucose control task using the FDA-approved UVA/Padova simulator, the method reports faster learning than no-feedback and random-feedback baselines in most environments, with the clearest gains in constrained tasks like glucose control. The paper's value would be a general recipe for spending a limited human-feedback budget where it most changes agent behaviour.

Core claim

The paper's central claim: combining active query selection with online trainer-reliability estimation improves the sample efficiency of policy-shaping feedback. After each episode, the agent computes, for every visited state-action pair, the posterior probability that the action is optimal, fusing human-feedback and trajectory evidence via a factorised Bayesian update. That posterior feeds a one-vs-all entropy that peaks when the posterior is uniform across actions, and the agent requests binary right/wrong feedback only for the top-n highest-entropy pairs. Variational inference estimates each trainer's consistency level online, downweighting unreliable voices. Entropy-selected feedback bea

What carries the argument

The load-bearing piece is the one-vs-all (OvA) entropy: a renormalisation of the action-optimality probability, p′ = p / (p + (1−p)/(Na−1)), before applying the Shannon formula. It peaks when the posterior says every action in a state is equally plausible, falls to zero when one action is all but certain, and reduces to ordinary Shannon entropy for two-action states. The probability p comes from a factorised posterior over an optimality variable Os,a (Eq. 5): a human-feedback term built on the Advise consistency model multiplied by a trajectory term that integrates Gaussian Q-value estimates over rival actions under a conditional-independence factorisation (Eq. 8), divided by the prior. A va

Load-bearing premise

The query-selection signal rests on assuming that the estimated values of the different actions in a state are independent once one action's value is known; if they are actually correlated, the uncertainty scores are miscalibrated and entropy-based selection could be no better than random sampling.

What would settle it

Rerun the PACMAN and blood-glucose comparisons with the factorised posterior of Eq. 8 replaced by a joint Gaussian posterior over all actions in a state, preserving measured covariances between Q-estimates. If entropy-guided feedback no longer beats random feedback, the conditional-independence factorisation is the load-bearing assumption; if it still wins, the query signal is robust to correlation.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A fixed feedback budget goes further: querying only the top-n most uncertain state-action pairs from real trajectories beats randomly sampled feedback in most tested environments, so better policies can be trained with fewer human annotations.
  • Trainer reliability does not need to be known in advance: online variational estimation keeps learning stable across trainers of different true consistency levels, whereas fixing a wrong assumed level degrades performance.
  • The gains concentrate where tasks are constrained: entropy-based selection increasingly outperforms random selection as Frozen Lake maps add mandatory gates, matching the glucose-control result.
  • On the blood-glucose control task, entropy-guided agents hold blood glucose in the 70–180 mg/dL target range a larger fraction of training time than both baselines across all three virtual patient profiles.
  • In open environments with many viable solutions, random feedback is nearly as good as entropy-based feedback, so the extra computation of the posterior and entropy is justified mainly in bottlenecked or safety-critical tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct probe of the load-bearing assumption: replace the conditional-independence product in Eq. 8 with a joint Gaussian posterior that keeps the measured correlations between Q-estimates, and rerun the comparisons — if entropy-based querying stops beating random querying, the factorisation is what carried the result.
  • The OvA entropy weights all non-optimal actions symmetrically; in cost-sensitive domains such as insulin dosing, where mistimed insulin is far worse than a missed dose, a cost-weighted renormalisation of the same posterior would be a natural, testable variant.
  • Because the posterior variance shrinks with visit count, the query rule effectively targets epistemic uncertainty; as the method moves beyond tabular Q-learning, preserving that distinction will matter, or the rule will waste queries on irreducible environmental noise.
  • The feedback-density maps suggest entropy-based selection concentrates labels on bottleneck states; a testable prediction is that the size of the learning-speed advantage across environments scales with the share of queries landing on those gate states.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a framework for crowd-based RL feedback that extends the Advise algorithm in three ways: (i) it handles multiple trainers with online reliability estimation via variational inference, (ii) it introduces an entropy-based active query selection mechanism (the "One-vs-All" entropy) to choose which state-action pairs to send for human feedback, and (iii) it evaluates the approach on tabular gridworld environments (PACMAN, Taxi, Frozen Lake) and a Type 1 Diabetes blood-glucose control task using the UVA/Padova simulator. The central claim is that agents trained with feedback on uncertain trajectories learn faster than agents with random feedback or no feedback, and that the proposed method outperforms baselines on the blood-glucose task. The paper also includes a limitations section acknowledging the reliance on tabular Q-learning, the availability of expert/non-expert teachers, and the lack of robustness evaluation.

Significance. If the central claims hold, the paper would make a useful contribution to feedback-efficient RLHF by combining crowd modeling with active query selection, a combination that is underexplored. The online consistency estimation is a sensible extension of Advise, and the OvA entropy is a principled way to handle multi-action optimality uncertainty. The evaluation on a clinically relevant simulator is a strength, as is the explicit consideration of trainer reliability. However, the significance is tempered by the theoretical inconsistency in the posterior derivation (see major comments), the modest and inconsistent gains in several environments, and the acknowledged limitations to tabular settings. The contribution is incremental rather than foundational, but it could be publishable after substantial revision.

major comments (4)
  1. [Entropy-based Active Feedback, Eq. (5); Appendix A, Eq. (19)] There is an internal inconsistency in the definition of the prior P(Os,a) used in the posterior combination. The main text states "We assume a uniform prior, such that P(Os,a = 1) = 1/Na" immediately after Eq. (5). However, Appendix A defines the prior over the optimality variable as the underlying RL policy: P(Os = ea) = πR(a|s), and this same prior appears as the log πR(a|s) term in the VI update of q(Os) in Eq. (19). Bayes' rule requires the same prior in the denominator of Eq. (5) as is used in the two posterior factors. Using 1/Na together with a posterior obtained under a πR prior introduces a factor Na·πR(a|s) into the combined posterior. This systematically over-weights actions already favored by the current policy. Because the entropy computed from this posterior drives the active query selection, the bias directly affects which state-action pairs are selected for feedback. This
  2. [Table 1, Figure 3, Abstract] The abstract claims the method achieves "faster learning in most tasks," but the results in Taxi and the easy Frozen Lake variant show essentially no improvement over random sampling. In Table 1, Taxi AUC is 209.75 for AL-Entropy versus 209.19 for AL-Random (a difference of 0.3%), and Frozen Lake (1) is 38.59 versus 38.30. The text in the Results section says the Taxi results are "less conclusive," but the abstract still makes a global claim. Please either temper the claim to reflect the actual pattern, or provide statistical significance tests (e.g., confidence intervals on the AUC differences, paired tests across trials) that support the claim of faster learning in the intended settings.
  3. [Eq. (8), Appendix C, Discussion] The conditional-independence assumption in Eq. (8) is load-bearing because it is used to compute the trajectory posterior P(Os,a|τ), which in turn determines the entropy values that drive query selection. The paper acknowledges the assumption is "strong" but provides no sensitivity analysis. It is possible that when Q-value estimates for different actions are correlated, the computed uncertainties are miscalibrated and the entropy ranking may be no better than random. Since the empirical contribution is precisely the entropy-based selection, the authors should test robustness to this assumption, for example by comparing to a model that accounts for correlations (even approximately) or by reporting how the query selection changes when the assumption is relaxed. Without such an analysis, the observed gains cannot be confidently attributed to the intended entropy mechanism.
  4. [Experiments, Appendix C] The experimental protocol omits a key hyperparameter of the active query selection: the number of top-n state-action pairs queried per episode in the gridworld environments. Appendix C specifies "feedback 10% of the time on average" only for the BGL control task. For the gridworld experiments, neither the query frequency nor the value of n is reported. Because the feedback budget directly affects the comparison between AL-Entropy and AL-Random, the results are not reproducible without this information. Please report the query budget for each environment, and ideally include a sensitivity analysis over n. The promise that code will be released in a camera-ready version is not sufficient for evaluation.
minor comments (5)
  1. [Eq. (6)] The second exponent in Eq. (6) is written as -Δ(s,j), which appears to be a typo for -Δ(s,a). As written, the formula is ambiguous.
  2. [Table 4 caption] The caption says "Customised versions of the Taxi environment" but the maps are Frozen Lake variants. Please correct the caption.
  3. [Appendix C, Table 3] It is unclear whether the same set of hyperparameters (learning rate, discount, Boltzmann temperature) is used for both the oracle training and the agent training. Please clarify the oracle's exploration schedule and how the oracle policy is converted into binary feedback.
  4. [Figure 7b] The color scale and the statement "Higher values indicate more entropy-based feedback, lower values indicate more random feedback" are not fully explained. It is unclear whether the values are differences of counts, ratios, or something else. Please provide a clear caption and colorbar label.
  5. [Abstract and Introduction] The acronym "RLHF" is used, but the paper focuses on policy feedback as in Advise rather than reward modeling. The introduction could more clearly distinguish this policy-shaping setting from the dominant preference-based RLHF line, since the related work section already does.

Circularity Check

0 steps flagged

No significant circularity: active query selection is an internally specified loop using external oracle labels and independent performance metrics; self-citations are not load-bearing.

full rationale

The paper's central claim is that entropy-based active query selection improves learning speed and feedback efficiency when human feedback is provided by an independently pre-trained oracle. The derivation chain is: (1) Q-value estimates from the agent's own trajectories are converted into a posterior over action optimality (Eqs. 7-11); (2) human feedback counts and estimated trainer consistency are converted into a feedback posterior (Eq. 6, Appendix A); (3) Bayes' rule combines these into P(Os,a|h,τ) (Eq. 5); (4) OvA entropy (Eq. 12) ranks state-action pairs; (5) the oracle labels the selected pairs; (6) feedback shapes the policy (Eq. 2). Each step is defined from the previous step and from external feedback, and the evaluation metrics (return, time-in-range) are not used to fit any parameter in the selection rule. The conditional-independence assumption in Eq. 8 is explicitly acknowledged as strong, and the apparent mismatch between the uniform prior in Eq. 5 and the πR(a|s) prior in Appendix A/Eq. 19 is a potential Bayesian misspecification/correctness concern, not a circular reduction of the prediction to its inputs. The only self-citations (Santos-Rodríguez et al. 2009; Yamagata and Santos-Rodriguez 2024) are contextual or future-work references and do not carry any load-bearing step. Therefore, no circularity is present.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The method rests on a small number of hand-chosen hyperparameters, most notably the strong beta prior over trainer consistency (alpha=90, beta=10) and the base variance of the Q-estimates. The two main modeling axioms are the Gaussianity of Q(s,a) and the conditional independence across actions, both of which are acknowledged as strong. No new entities or forces are introduced.

free parameters (5)
  • Prior parameters for trainer consistency (alpha_l, beta_l) = 90, 10
    Strong prior assuming each trainer is about 90% consistent; this dominates early reliability estimates and was chosen by hand, not learned from data.
  • Baseline standard deviation of Q-estimate (sigma_base) = 5000 (BGL); not reported for gridworld
    Controls the spread of the Gaussian distribution over Q-values in Eq. 8, directly affecting posterior entropy and hence query selection.
  • Number of Monte Carlo samples (M) = not reported
    Controls approximation error in the stratified-sampling estimate of the posterior (Eq. 11).
  • Top-n queried state-action pairs per episode = not reported
    Feedback budget per episode; determines how many labels are requested and is chosen by hand.
  • Boltzmann exploration temperature (tau_b) = 1.5 (gridworld), 10.0 (BGL)
    Sets the exploration level of the agent; affects which state-action pairs are visited and hence which can be queried.
axioms (5)
  • domain assumption Q(s,a) follows a Gaussian distribution with mean Qhat(s,a) and standard deviation sigma_base / sqrt(N_{s,a})
    Used to compute the posterior P(O_{s,a}|tau) in Eq. 8; no justification is given for this distributional form.
  • domain assumption Conditional independence of Q(s,a') for all a' != a given Q(s,a)
    Used to factorize the product in Eq. 8; the paper itself calls this 'a strong assumption' and it is load-bearing for the entropy computation.
  • standard math Mean-field factorization of the posterior (Eq. 3, 16)
    Standard variational inference assumption; appropriate for the graphical model but yields approximate inference.
  • domain assumption Trainer feedback follows a binomial model with a single consistency parameter C_l (Advise model)
    Assumes each trainer has one reliability parameter and gives right/wrong feedback with fixed probability; oversimplifies complex human behavior.
  • domain assumption Optimality of an action is defined as having the highest Q-value (Eq. 7)
    Defines the ground-truth optimality variable in terms of the oracle's value function; the oracle is a pretrained Q-learning agent, so the feedback is only as good as that oracle.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Active Query Selection for Crowd-Based Reinforcement Learning." pith.science (2026). https://pith.science/paper/XFLNE66J

@misc{pith2026250819132,
  author       = {Pith},
  title        = {Pith review of: Active Query Selection for Crowd-Based Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XFLNE66J}},
  note         = {Machine review of arXiv:2508.19132}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Preference-based reinforcement learning has gained prominence as a strategy for training agents in environments where the reward signal is difficult to specify or misaligned with human intent. However, its effectiveness is often limited by the high cost and low availability of reliable human input, especially in domains where expert feedback is scarce or errors are costly. To address this, we propose a novel framework that combines two complementary strategies: probabilistic crowd modelling to handle noisy, multi-annotator feedback, and active learning to prioritize feedback on the most informative agent actions. We extend the Advise algorithm to support multiple trainers, estimate their reliability online, and incorporate entropy-based query selection to guide feedback requests. We evaluate our approach in a set of environments that span both synthetic and real-world-inspired settings, including 2D games (Taxi, Pacman, Frozen Lake) and a blood glucose control task for Type 1 Diabetes using the clinically approved UVA/Padova simulator. Our preliminary results demonstrate that agents trained with feedback on uncertain trajectories exhibit faster learning in most tasks, and we outperform the baselines for the blood glucose control task.

Figures

Figures reproduced from arXiv: 2508.19132 by Jonathan Erskine, Ra\'ul Santos-Rodr\'iguez, Taku Yamagata.

Figure 1
Figure 1. Figure 1: Standard reinforcement learning training loop and our extended method. We collect a trajectory [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Evaluation results for PACMAN and Taxi grid [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Evaluation results for the Frozen Lake grid [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Symbols for graphical model for consistency level estimation with VI algorithm. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Graphical model for consistency level estimation with VI algorithm. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: PACMAN Environment; (a) shows the starting positions of PACMAN and the ghost, as well as the two pellets which [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Taxi Feedback Comparison; comparing feedback sampled per location for random feedback selection and entropy [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Simulation results for three person profiles with five learning trials for 2,000 episodes. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 25 canonical work pages · 3 internal anchors

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Abdelkareem, Y.; Shehata, S.; and Karray, F. 2022. Advances in Preference-based Reinforcement Learning: A Review. In 2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2527–2532. IEEE

    work page 2022

  4. [4]

    Arakawa, R.; Kobayashi, S.; Unno, Y.; Tsuboi, Y.; and Maeda, S.-i. 2018. Dqn-tamer: Human-in-the-loop reinforcement learning with intractable feedback. arXiv preprint arXiv:1810.11748

    Pith/arXiv arXiv 2018

  5. [5]

    Buecheler, T.; Sieg, J.; Füchslin, R.; and Pfeifer, R. 2010. Crowdsourcing, Open Innovation and Collective Intelligence in the Scientific Method: A Research Agenda and Operational Framework. Artificial Life XII – Twelfth International Conference on the Synthesis and Simulation of Living Systems, Odense, Denmark, 679--686

    work page 2010

  6. [6]

    Chakraborty, S.; Qiu, J.; Yuan, H.; Koppel, A.; Manocha, D.; Huang, F.; Bedi, A.; and Wang, M. 2024. MaxMin- RLHF : Alignment with Diverse Human Preferences. In Forty-first International Conference on Machine Learning

    work page 2024

  7. [7]

    Chhan, D.; Novoseller, E.; and Lawhern, V. J. 2025. Crowd-PrefRL: Preference-Based Reward Learning from Crowds. arXiv:2401.10941

    Pith/arXiv arXiv 2025

  8. [8]

    F.; Leike, J.; Brown, T.; Martic, M.; Legg, S.; and Amodei, D

    Christiano, P. F.; Leike, J.; Brown, T.; Martic, M.; Legg, S.; and Amodei, D. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30

    work page 2017

  9. [9]

    Chu, Z.; Ma, J.; and Wang, H. 2020. Learning from Crowds by Modeling Common Confusions. CoRR, abs/2012.13052

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  10. [10]

    P.; and Skene, A

    Dawid, A. P.; and Skene, A. M. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics), 28(1): 20--28

    work page 1979

  11. [11]

    Ebert, S.; Fritz, M.; and Schiele, B. 2012. RALF: A reinforced active learning formulation for object class recognition. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, 3626--3633

    work page 2012

  12. [12]

    Fang, M.; Li, Y.; and Cohn, T. 2017. Learning how to Active Learn: A Deep Reinforcement Learning Approach. arXiv:1708.02383

    Pith/arXiv arXiv 2017

  13. [13]

    Feng, X.; Jiang, Z.; Kaufmann, T.; Xu, P.; H \"u llermeier, E.; Weng, P.; and Zhu, Y. 2025. DUO: Diverse, Uncertain, On-Policy Query Generation and Selection for Reinforcement Learning from Human Feedback. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, 16604--16612

    2025

  14. [14]

    N.; Jiao, J.; Zhu, B.; Gonzalez, J

    Frick, E.; Li, T.; Chen, C.; Chiang, W.-L.; Angelopoulos, A. N.; Jiao, J.; Zhu, B.; Gonzalez, J. E.; and Stoica, I. 2024. How to Evaluate Reward Models for RLHF. arXiv:2410.14872

    Pith/arXiv arXiv 2024

  15. [15]

    Goyal, A.; Islam, R.; Strouse, D.; Ahmed, Z.; Botvinick, M.; Larochelle, H.; Bengio, Y.; and Levine, S. 2023. InfoBot: Transfer and Exploration via the Information Bottleneck. arXiv:1901.10902

    Pith/arXiv arXiv 2023

  16. [16]

    L.; and Thomaz, A

    Griffith, S.; Subramanian, K.; Scholz, J.; Isbell, C. L.; and Thomaz, A. L. 2013. Policy Shaping: Integrating Human Feedback with Reinforcement Learning. In Burges, C.; Bottou, L.; Welling, M.; Ghahramani, Z.; and Weinberger, K., eds., Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc

    work page 2013

  17. [17]

    Guan, M.; Houlsby, N.; and Cesa-Bianchi, N. 2018. Who said what: Modeling individual annotator reliability and annotator bias. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32

    work page 2018

  18. [18]

    Guermazi, R.; Chaabane, I.; and Hammami, M. 2018. AECID: Asymmetric entropy for classifying imbalanced data. Information Sciences, 467: 373--397

    work page 2018

  19. [19]

    Kaufmann, T.; Weng, P.; Bengs, V.; and Hüllermeier, E. 2024. A Survey of Reinforcement Learning from Human Feedback. arXiv:2312.14925

    arXiv 2024

  20. [20]

    B.; and Stone, P

    Knox, W. B.; and Stone, P. 2012. Reinforcement learning from simultaneous human and MDP reward. 475--482. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 0981738117

    work page 2012

  21. [21]

    Kumar, A.; Hong, J.; Singh, A.; and Levine, S. 2022. When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning? arXiv:2204.05618

    Pith/arXiv arXiv 2022

  22. [22]

    Lee, K.; Smith, L.; and Abbeel, P. 2021. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. arXiv preprint arXiv:2106.05091

    Pith/arXiv arXiv 2021

  23. [23]

    A Sequential Algorithm for Training Text Classifiers

    Lewis, D. D.; and Gale, W. A. 1994. A Sequential Algorithm for Training Text Classifiers. arXiv:cmp-lg/9407020

    work page internal anchor Pith review Pith/arXiv arXiv 1994

  24. [24]

    Liu, H.; Zhuge, M.; Li, B.; Wang, Y.; Faccio, F.; Ghanem, B.; and Schmidhuber, J. 2023. Learning to Identify Critical States for Reinforcement Learning from Videos. arXiv:2308.07795

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  25. [25]

    D.; Micheletto, F.; Lv, D.; Breton, M.; Kovatchev, B.; and Cobelli, C

    Man, C. D.; Micheletto, F.; Lv, D.; Breton, M.; Kovatchev, B.; and Cobelli, C. 2014. The UVA/PADOVA type 1 diabetes simulator: New features. Journal of Diabetes Science and Technology, 8: 26--34

    work page 2014

  26. [26]

    Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C. L.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; Schulman, J.; Hilton, J.; Kelton, F.; Miller, L.; Simens, M.; Askell, A.; Welinder, P.; Christiano, P.; Leike, J.; and Lowe, R. 2022. Training language models to follow instructions with human feedback. arXiv:2203.02155

    Pith/arXiv arXiv 2022

  27. [27]

    C.; Yu, S.; Zhao, L

    Raykar, V. C.; Yu, S.; Zhao, L. H.; Valadez, G. H.; Florin, C.; Bogoni, L.; and Moy, L. 2010. Learning from crowds. Journal of Machine Learning Research, 11: 1297--1322

    work page 2010

  28. [28]

    C.; and Pereira, F

    Rodrigues, F. C.; and Pereira, F. 2018. Deep learning from crowds. In AAAI Conference on Artificial Intelligence, volume 32

    work page 2018

  29. [29]

    Santos-Rodr \' guez, R.; Guerrero-Curieses, A.; Alaiz-Rodr \' guez, R.; and Cid-Sueiro, J. 2009. Cost-sensitive learning based on Bregman divergences. Machine Learning, 76(2-3): 271--285

    work page 2009

  30. [30]

    Settles, B. 2009. Active Learning Literature Survey. Computer Sciences Technical Report 1648, University of Wisconsin--Madison

    work page 2009

  31. [31]

    Siththaranjan, A.; Laidlaw, C.; and Hadfield-Menell, D. 2024. Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF. arXiv:2312.08358

    Pith/arXiv arXiv 2024

  32. [32]

    Snow, R.; O'Connor, B.; Jurafsky, D.; and Ng, A. Y. 2008. Cheap and fast---but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, 254--263. Association for Computational Linguistics

    work page 2008

  33. [33]

    Strens, M. 2000. A Bayesian framework for reinforcement learning. volume 2000, 943--950. Posterior Sampling RL

    work page 2000

  34. [34]

    S.; and Barto, A

    Sutton, R. S.; and Barto, A. G. 1998. Reinforcement Learning. Cambridge, MA: The MIT Press

    work page 1998

  35. [35]

    Thompson, W. R. 1933. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4): 285--294

    work page 1933

  36. [36]

    U.; Cola, G

    Towers, M.; Kwiatkowski, A.; Terry, J.; Balis, J. U.; Cola, G. D.; Deleu, T.; Goulão, M.; Kallinteris, A.; Krimmel, M.; KG, A.; Perez-Vicente, R.; Pierré, A.; Schulhoff, S.; Tai, J. J.; Tan, H.; and Younis, O. G. 2024. Gymnasium: A Standard Interface for Reinforcement Learning Environments. arXiv:2407.17032

    Pith/arXiv arXiv 2024

  37. [37]

    Warnell, G.; Waytowich, N.; Lawhern, V.; and Stone, P. 2018. Deep tamer: Interactive agent shaping in high-dimensional state spaces. In Proceedings of the AAAI conference on artificial intelligence, volume 32

    work page 2018

  38. [38]

    Whitehill, J.; Ruvolo, P.; Wu, T.; Bergsma, J.; and Movellan, J. R. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems, volume 22

    work page 2009

  39. [39]

    Xie., J. 2018. Simglucose v0.2.1 https://github.com/jxx123/simglucose

    work page 2018

  40. [40]

    Yamagata, T.; and Santos-Rodriguez, R. 2024. Safe and Robust Reinforcement Learning: Principles and Practice. arXiv:2403.18539

    Pith/arXiv arXiv 2024

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.