Pith. sign in

REVIEW 6 major objections 6 minor 46 references

A Generalised and Adaptable Reinforcement Learning Stopping Method

T0 review · 6 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GRLStop makes one reinforcement-learning policy serve multiple target recalls in technology-assisted review, reaching target recall at lower cost than baselines in almost every tested configuration.

desk verdict A solid, citable extension of RLStop whose normalized reward does what it claims; the main open question is how much the policy depends on classifier calibration, and the evaluation lacks repeated-seed variance. read the letter →

arxiv 2505.01907 v2 pith:U3DRX3PF submitted 2025-05-03 cs.IR

Reem Bin-Hezam , Mark Stevenson This is my paper · ORCID
classification cs.IR
keywords reinforcementlearningtechnology-assistedreviewTARstoppinghigh-recallretrievalrewarddesigndocumentscreeningsystematicreviews
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a single reinforcement-learning stopping policy, GRLStop, can replace the per-target models required by previous RL stopping methods in technology-assisted review (TAR). Its reward function is normalised so that reaching the target recall always produces the same maximum cumulative reward, regardless of where in the ranking that target falls, and two parameters let a user tilt the policy toward recall maximisation or cost minimisation. On six TAR benchmark collections the paper reports that GRLStop is Pareto optimal in almost every tested configuration, reaching target recall at lower cost than baselines and closer to the oracle stopping point. If these results hold, RL-based stopping becomes practical for systematic reviews, legal discovery, and other high-recall screening tasks where target recall and cost sensitivity change from case to case.

What carries the argument

The load-bearing object is the normalised reward function of Eq. 1, whose cumulative form is the ratio-symmetric pair $(i/T)^m$ and $((B-i)/(B-T))^n$; it makes the reward range invariant to the position of the target recall, which is what lets a single model train across many target positions. The state is a fixed vector of length $B+2$: one element per batch (observed relevance proportion for examined batches, logistic-regression TF-IDF estimate for unexamined batches), plus the number of batches examined and the target recall. A feed-forward neural network with two hidden layers and a softmax output maps that state to STOP or CONTINUE, and the policy is trained with PPO, a policy-gradient reinforcement-learning algorithm, over parallel episodes from multiple topics.

What would settle it

Take a held-out set of topics, replace every classifier estimate in the unexamined batches with random values between 0 and 1, and compare GRLStop's stopping cost and recall to the reported results; a large degradation would show the results hinge on classifier accuracy. Separately, train GRLStop with the target-recall element of the state vector fixed to a constant and test on recall targets not seen in training; if performance is unchanged, the policy is not actually using the target recall, contradicting the explanation of how generalisation works.

Watch

Extended reading notes

Core claim

The central claim is that the stopping problem in TAR can be solved by one policy rather than one policy per target recall. GRLStop trains a feed-forward neural policy with a reward function, Eq. 1, whose cumulative value is $CR(S_i)=(i/T)^m$ before the target batch $T$ and $CR(S_i)=((B-i)/(B-T))^n$ after it; the maximum is always 1 at $i=T$, so rewards from different rankings and different target positions are comparable. At inference the same policy receives a state vector that records, for each of $B$ batches, the observed relevant proportion (for examined batches) or a classifier's estimate (for unexamined batches), together with the number of batches examined and the target recall. The paper claims this policy reaches target recall at lower cost than existing stopping baselines on six datasets, that the $m,n$ parameters visibly shift the recall/cost tradeoff in the intended direction, and that including classifier estimates significantly improves the CostDiff score at most target recalls.

Load-bearing premise

The method depends on the state vector carrying enough signal about where the remaining relevant documents are and what the target recall is; if the classifier's estimates for unexamined batches are poor, or the policy cannot infer the target from the state, the training-time reward normalisation will not transfer to new rankings.

Editorial extensions

If this is right

  • A single GRLStop model trained on one dataset can be applied to target recalls it never trained on (the paper demonstrates 0.7, 0.8, 0.9, and 1.0 with one model per dataset), removing the need to train and store one RL model per recall level.
  • Setting $m$ and $n$ steers the policy: $(m,n)=(4,0.25)$ yields higher recall and reliability at higher cost, while $(0.25,4)$ cuts cost at the expense of missed targets, so the same environment can serve recall-critical medical reviews and cost-sensitive legal review.
  • Adding classifier predictions for unexamined batches moves stopping decisions closer to the oracle on most datasets and target recalls, with statistically significant CostDiff improvements at target recalls 0.7, 0.8, and 1.0.
  • GRLStop lies on the Pareto front of recall versus cost in almost every dataset/target configuration tested, with the single exception of TR at target recall 1.0 where it is very close to the front.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the cumulative reward is a function of the ratio $i/T$ (and $(B-i)/(B-T)$), the policy is likely learning the fraction of the way to the target rather than an absolute batch count; this predicts it would also stop sensibly at an untested target such as 0.85, which the paper does not directly show.
  • Editorial inference: the reward family is a two-parameter distortion of a linear target-distance curve, meaning GRLStop can be viewed as one RL implementation of an entire continuum of stopping rules; the same normalisation idea could be exported to other sequential tasks with a latent target fraction, such as stopping active learning or qualitative coding saturation.
  • Editorial inference: the paper's ranking-quality results suggest the weakest link is classifier quality under noisy rankings; a natural testable extension is to use confidence-calibrated classifier probabilities instead of point predictions in the unexamined batches, which should improve CostDiff most on low-quality rankings.
  • Editorial inference: the comparison to RLStop in Table 1 is arguably favourable to GRLStop because a single GRLStop model is compared against RLStop models that were each trained for one target; a fairer stress test would be to train RLStop on a mixture of targets and see whether its original reward function can also generalise when trained that way.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. The paper introduces GRLStop, a reinforcement learning (RL) environment and policy for Technology Assisted Review (TAR) stopping. It extends the authors' earlier RLStop by (i) a reward function whose cumulative reward has range [0,1] regardless of the batch at which the target recall is reached, allowing a single policy to serve multiple target recalls; (ii) two reward parameters m and n that control the recall/cost tradeoff; and (iii) integration of classifier predictions for unexamined batches into the RL state. Experiments on six TAR datasets (CLEF e-Health 2017-19, TREC Total Recall, TREC Legal, and RCV1) with AutoTAR rankings compare GRLStop against ten baselines and an oracle, and also evaluate classifier inclusion, reward-objective adaptation, and ranking quality. The main claimed results are that GRLStop is Pareto optimal in almost all recall/cost scenarios, that a single model adapts across target recalls, and that classifier integration and reward adaptation are beneficial.

Significance. If the central claims hold, GRLStop would be a practical advance for TAR stopping: it removes the need to train a separate RL model per target recall and gives the practitioner explicit control over the recall/cost tradeoff. The paper's strengths include a clearly specified reward design with an analytical cumulative-reward argument, evaluation on multiple benchmark datasets against a broad set of baselines, a ranking-quality analysis, and released code (https://github.com/ReemBinHezam/GRLStop). The empirical support, however, has gaps that affect the strength of the claims: no repeated-seed variance or confidence intervals are reported for RL training, the Legal test set contains only two topics, and on the TR dataset the model does not vary its stopping decision with target recall for most settings. The central generalization mechanism (inferring the target-reach batch from noisy classifier estimates) is asserted but not directly validated.

major comments (6)
  1. [Section 5, Figure 3, Tables 1-3] All headline results are reported as single point estimates without repeated-seed variance or confidence intervals. Because the policy is trained with PPO, which is stochastic in initialization and updates, a single run is not sufficient to establish that GRLStop is 'Pareto optimal in almost every case' or 'consistently closer to the optimal Oracle results than other approaches'. Please report means and standard deviations over at least five seeds and apply a significance test to the key comparisons (e.g., CostDiff against the best baseline).
  2. [Section 3.1, Eq. (1)] The reward function depends on T, the batch at which the target recall is achieved, which is known during training but not at inference. At inference the policy receives states whose unexamined-batch values are logistic-regression estimates (Section 3.1) and must infer T from these noisy quantities. The paper never validates this inference mechanism: Section 5.2 ablates the classifier's presence, not its accuracy, and Section 5.1 shows only that stopping decisions differ across target recalls. A direct test would compare the policy's implied T against the oracle T, or vary classifier calibration and measure the effect on CostDiff.
  3. [Table 1, TR row] For the TR dataset, GRLStop produces identical recall (0.950), reliability (0.941), and cost (0.010) for target recalls 0.7, 0.8, and 0.9. This directly contradicts the Section 5.1 statement that 'results for all metrics vary depending on the target recall' and suggests that on this dataset the policy learns a single early stopping point rather than conditioning on the target recall. The paper should acknowledge and explain this case, or qualify the single-model adaptation claim.
  4. [Section 4.1, Tables 1-2] The Legal dataset contains only two test topics. Consequently, reliability values in Table 1 and Table 2 are restricted to 0.0, 0.5, and 1.0, and the large performance gap on Legal between GRLStop-balanced (recall 0.504, cost 0.010) and GRLStop-recall-obj (recall 0.998, cost 0.200) hinges on just two topics. Claims about Legal behaviour and the general observations drawn from it should be tempered or supplemented with additional topics.
  5. [Section 4.4] The training protocol for the claimed single model is under-specified. The paper does not state how target recalls are sampled or mixed across training episodes, whether a separate model is trained for each dataset (which is implied by the dataset-specific training sets in Section 4.1) or across datasets, or how the grid search and the early-stopping callback interact with the test-set evaluation. These details are needed to reproduce the single-model generalization experiment in Section 5.1.
  6. [Section 3.1, Eq. (2), and Section 4.4] The cumulative-reward analysis in Eq. (2) assumes an undiscounted sum of rewards, but PPO is trained with a discount factor of 0.99 (Section 4.4). The claimed range-invariance property and the shape of the objective do not automatically transfer to the discounted objective actually optimized by the agent. The authors should either prove the property for the discounted objective or present empirical evidence (e.g., sensitivity to the discount factor) that the learned policy is unaffected.
minor comments (6)
  1. [Abstract] The sentence 'Results showed that the proposed approach to be effective' is ungrammatical; it should be rephrased, e.g., 'Results showed that the proposed approach is effective'.
  2. [Section 4.4] The grid search over hyperparameters is not described; the paper should report which values were tried for the learning rate, entropy coefficient, clipping range, and the reward parameters m and n, or state that these were fixed from prior work.
  3. [Section 5.2] The paired t-test with Bonferroni correction is mentioned without specifying the exact metric and pooling strategy (per-topic CostDiff across all datasets, across all target recalls, or both). Please clarify the statistical procedure.
  4. [Section 5.4] The statement that results for other datasets 'display similar trends' should be supported by the actual tables, either in an appendix or as supplementary material, instead of being omitted for brevity.
  5. [Section 4.1] For RCV1, the 45 categories used for evaluation are not listed; a reference or an explicit list is needed for reproducibility.
  6. [Section 3.1, Eq. (1)] The case T = B is excluded because the denominator (B-T)^n vanishes, but it is not stated how episodes with T = B are handled during training; this boundary case should be clarified.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reward function is a design choice, target recall is a training-time input, and the main claims are tested against external baselines and an oracle.

full rationale

The paper's central claim is that a single policy trained with the normalized cumulative reward in Eq. (2) can adapt to multiple target recalls and to different recall/cost preferences. This is a design-and-test claim rather than a quantity fitted to the evaluation data. The target-recall batch T appears only during training, and the state vector includes the target recall, so the policy must generalise rather than read off T from the test outputs. The classifier supplies estimated relevance proportions for unexamined batches, but the paper does not claim to predict the classifier's accuracy as its headline result; the classifier is a component of the environment, not the predicted quantity. The evaluation compares recall, cost, reliability, and CostDiff against external baselines and an oracle on six benchmark datasets, so the stopping performance is not defined by construction in terms of the model's own parameters. Self-citations to RLStop and to the authors' point-process work are background and baseline references, and they are not used to justify the new reward function or to forbid alternative approaches. No equation in the paper reduces to its own inputs, and no fitted parameter is renamed as a prediction. The derivation chain is therefore self-contained with respect to the paper's stated claims.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests mainly on benchmark evaluation plus the design of a family of reward functions. There are no fitted physical constants or invented entities. The most important non-empirical load-bearing pieces are the hand-chosen reward exponents m and n and the assumption that PPO can learn to use the target recall in the state.

free parameters (4)
  • Reward exponent m (pre-target) = 1 in main experiments; 4 for recall-obj; 0.25 for cost-obj
    Controls cumulative reward shape before target recall is reached. Values in Section 5.3 were hand-chosen to encourage the intended behavior without being too extreme.
  • Reward exponent n (post-target) = 1 in main experiments; 0.25 for recall-obj; 4 for cost-obj
    Controls reward after target recall is exceeded; hand-chosen together with m in Section 5.3.
  • Number of batches B = 100
    Splits each ranking into 100 fixed-size batches, following previous work. This affects the granularity of stopping positions and the state vector length.
  • PPO and environment hyperparameters = Entropy coefficient 0.1 or 0.001, steps per environment 10 or 100, clipping range 0.1, learning rate 0.0003, etc.
    Selected by grid search per configuration and not fully enumerated in the paper, so exact training is not fully reproducible from the text alone.
assumptions (4)
  • domain assumption AutoTAR rankings are a fair and representative basis for comparing stopping methods on all datasets.
    All stopping methods are applied to the same AutoTAR rankings (Section 4.1). Different rankings can change stopping behavior, as the paper itself shows in Section 5.4.
  • domain assumption A logistic-regression TF-IDF classifier trained on the batches examined so far yields useful estimates of relevance in unexamined batches.
    Classifier predictions are part of the state (Section 3.1) and are the only information about unexamined documents. The paper tests the effect of removing them in Section 5.2.
  • domain assumption PPO converges to a policy that generalizes across target recalls when trained with the normalized reward in Eq. 1.
    The claim that one model serves multiple target recalls depends on this empirical convergence; no formal guarantee is given.
  • domain assumption The target-recall batch T used in the training reward is known during training but unavailable at inference, and the state encodes enough to let the policy infer when to stop.
    Eq. 1 uses T in the denominator. At test time only the state vector, including the target recall value as a component, is available.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Generalised and Adaptable Reinforcement Learning Stopping Method." pith.science (2026). https://pith.science/paper/U3DRX3PF

@misc{pith2026250501907,
  author       = {Pith},
  title        = {Pith review of: A Generalised and Adaptable Reinforcement Learning Stopping Method},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U3DRX3PF}},
  note         = {Machine review of arXiv:2505.01907}
}
read the original abstract

This paper presents a Technology Assisted Review (TAR) stopping approach based on Reinforcement Learning (RL). Previous such approaches offered limited control over stopping behaviour, such as fixing the target recall and tradeoff between preferring to maximise recall or cost. These limitations are overcome by introducing a novel RL environment, GRLStop, that allows a single model to be applied to multiple target recalls, balances the recall/cost tradeoff and integrates a classifier. Experiments were carried out on six benchmark datasets (CLEF e-Health datasets 2017-9, TREC Total Recall, TREC Legal and Reuters RCV1) at multiple target recall levels. Results showed that the proposed approach to be effective compared to multiple baselines in addition to offering greater flexibility.

Figures

Figures reproduced from arXiv: 2505.01907 by the authors.

Figure 1
Figure 1. RL Environment for TAR Stopping processes, including stopping rules which help reviewers to de￾cide when to stop assessing documents, thereby reducing the effort required to screen a collection for relevance. TAR stopping rules aim to identify when a desired level of recall (the target recall) has been reached during document review, while also minimising the number of documents examined. The problem is challenging … view at source ↗
Figure 2
Figure 2. Example cumulative reward functions produced by [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Recall vs cost. Target recall indicated by horizontal blue dashed line and Pareto front as grey line. Note that scale of [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Effect of including/excluding classifier on CostDiff. Green dashed horizontal line indicates optimal value (i.e. 0). [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 38 canonical work pages

  1. [1]

    ACM Computing Surveys 55(7), 1–38 (2022)

    Afsar, M.M., Crump, T., Far, B.: Reinforcement Learning Based Recommender Systems: A Survey. ACM Computing Surveys 55(7), 1–38 (2022)

    work page 2022

  2. [2]

    IEEE Signal Processing Magazine 34(6), 26–38 (2017)

    Arulkumaran, K., Deisenroth, M.P., Brundage, M., Bharath, A.A.: Deep Reinforce- ment Learning: A Brief Survey. IEEE Signal Processing Magazine 34(6), 26–38 (2017). https://doi.org/10.1109/MSP.2017.2743240

    arXiv 2017

  3. [3]

    Providing More Efficient Access To Government Records: A Use Case Involving Application of Machine Learning to Improve FOIA Review for the Deliberative Process Privilege

    Baron, J.R., Sayed, M.F., Oard, D.W.: Providing More Efficient Access To Gov- ernment Records: A Use Case Involving Application of Machine Learning to Improve FOIA Review for the Deliberative Process Privilege. arXiv preprint arXiv:2011.07203 (2020)

    work page Pith review arXiv 2020

  4. [4]

    In: Proceedings of the 26th Annual International Conference on Machine Learning

    Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum Learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. pp. 41–48 (2009)

    work page 2009

  5. [5]

    In: Find- ings of the Association for Computational Linguistics: EMNLP 2023

    Bin-Hezam, R., Stevenson, M.: Combining Counting Processes and Classifi- cation Improves a Stopping Rule for Technology Assisted Review. In: Find- ings of the Association for Computational Linguistics: EMNLP 2023. pp. 2603–2609. Association for Computational Linguistics, Singapore (Dec 2023). https://doi.org/10.18653/v1/2023.findings-emnlp.171

    doi:10.18653/v1/2023.findings-emnlp.171 2023

  6. [6]

    In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (2024)

    Bin-Hezam, R., Stevenson, M.: RLStop: A Reinforcement Learning Stopping Method for TAR. In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (2024)

    work page 2024

  7. [7]

    Using Chao's Estimator as a Stopping Criterion for Technology-Assisted Review

    Bron, M.P., van der Heijden, P.G., Feelders, A.J., Siebes, A.P.: Using Chao’s Esti- mator as a Stopping Criterion for Technology-Assisted Review. arXiv preprint arXiv:2404.01176 (2024)

    work page Pith review arXiv 2024

  8. [8]

    Systematic Reviews 9(1), 1–14 (2020)

    Callaghan, M.W., Müller-Hansen, F.: Statistical Stopping Criteria for Automated Screening in Systematic Reviews. Systematic Reviews 9(1), 1–14 (2020)

    work page 2020

Show all 46 references
  1. [9]

    arXiv preprint arXiv:1504.06868 (apr 2015)

    Cormack, G., Grossman, M.: Autonomy and Reliability of Continuous Active Learning for Technology-Assisted Review. arXiv preprint arXiv:1504.06868 (apr 2015)

    2015 arXiv

  2. [10]

    In: Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval

    Cormack, G.V., Grossman, M.R.: Engineering Quality and Reliability in Technology-Assisted Review. In: Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. pp. 75–84 (2016)

    2016

  3. [11]

    In: Proceedings of the 25th ACM interna- tional on conference on information and knowledge management

    Cormack, G.V., Grossman, M.R.: Scalability of Continuous Active Learning for Reliable High-Recall Text Classification. In: Proceedings of the 25th ACM interna- tional on conference on information and knowledge management. pp. 1039–1048 (2016)

    2016

  4. [12]

    In: Proceedings of The Nineteenth Text REtrieval Conference, TREC 2010, Gaithersburg, Maryland, USA, November 16-19, 2010

    Cormack, G.V., Grossman, M.R., Hedin, B., Oard, D.W.: Overview of the TREC 2010 Legal Track. In: Proceedings of The Nineteenth Text REtrieval Conference, TREC 2010, Gaithersburg, Maryland, USA, November 16-19, 2010. NIST Special Publication, vol. 500-294. National Institute of...

    2010

  5. [13]

    In: European Conference on Information Retrieval

    Di Nunzio, G.M.: A Study of an Automatic Stopping Strategy for Technologically Assisted Medical Reviews. In: European Conference on Information Retrieval. pp. 672–677. Springer (2018)

    2018

  6. [14]

    In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval

    Gray, L., Lewis, D.D., Pickens, J., Yang, E.: High Recall Retrieval Via Technology- Assisted Review. In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 2987–2988 (2024)

    2024

  7. [15]

    In: Proceedings of The Twenty-Fifth Text REtrieval Conference, TREC

    Grossman, M.R., Cormack, G.V., Roegiest, A.: TREC 2016 Total Recall Track Overview. In: Proceedings of The Twenty-Fifth Text REtrieval Conference, TREC

    2016

  8. [16]

    John Wiley & Sons (2019)

    Higgins, J.P., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M.J., Welch, V.A.: Cochrane Handbook for Systematic Reviews of Interventions. John Wiley & Sons (2019)

    2019

  9. [17]

    In: Working Notes of CLEF 2017 - Conference and Labs of the Evaluation Forum

    Hollmann, N., Eickhoff, C.: Ranking and Feedback-based Stopping for Recall- Centric Document Retrieval. In: Working Notes of CLEF 2017 - Conference and Labs of the Evaluation Forum. pp. 7–8 (2017)

    2017

  10. [18]

    Journal of the American Statistical Association 47(260), 663–685 (1952)

    Horvitz, D.G., Thompson, D.J.: A Generalization of Sampling without Replace- ment from a Finite Universe. Journal of the American Statistical Association 47(260), 663–685 (1952)

    1952

  11. [19]

    Environment International 138, 105623 (2020), https: //www.sciencedirect.com/science/article/pii/S0160412019314023

    Howard, B.E., Phillips, J., Tandon, A., Maharana, A., Elmore, R., Mav, D., Sedykh, A., Thayer, K., Merrick, B.A., Walker, V., Rooney, A., Shah, R.R.: SWIFT-Active Screener: Accelerated Document Screening through Active Learning and Inte- grated Recall Estimation. Environment I...

    2020

  12. [20]

    In: CEUR workshop proceedings

    Kanoulas, E., Li, D., Azzopardi, L., Spijker, R.: CLEF 2017 Technologically Assisted Reviews in Empirical Medicine Overview. In: CEUR workshop proceedings. vol. 1866 (2017)

    2017

  13. [21]

    In: CEUR workshop proceedings

    Kanoulas, E., Li, D., Azzopardi, L., Spijker, R.: CLEF 2018 Technologically Assisted Reviews in Empirical Medicine Overview. In: CEUR workshop proceedings. vol. 2125 (2018)

    2018

  14. [22]

    In: CEUR workshop proceedings

    Kanoulas, E., Li, D., Azzopardi, L., Spijker, R.: CLEF 2019 Technology Assisted Reviews in Empirical Medicine Overview. In: CEUR workshop proceedings. vol. 2380 (2019)

    2019

  15. [23]

    In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management (2021)

    Lewis, D., Yang, E., Frieder, O.: Certifying One-Phase Technology-Assisted Re- views. In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management (2021)

    2021

  16. [24]

    Journal of Machine Learning Research 5, 361– 397 (2004)

    Lewis, D.D., Yang, Y., Russell-Rose, T., Li, F.: RCV1: A New Benchmark Collection for Text Categorization Research. Journal of Machine Learning Research 5, 361– 397 (2004). https://doi.org/10.5555/1005332.1005345

    2004

  17. [25]

    ACM Trans

    Li, D., Kanoulas, E.: When to Stop Reviewing in Technology-Assisted Reviews: Sampling from an Adaptive Distribution to Estimate Residual Relevant Docu- ments. ACM Trans. on Information Systems 38(4), 1–36 (2020), https://doi.org/ 10.1145/3411755

    2020 doi

  18. [26]

    Encyclopedia of machine learning 2011, 231–235 (2008)

    Ling, C.X., Sheng, V.S.: Cost-Sensitive Learning and the Class Imbalance Problem. Encyclopedia of machine learning 2011, 231–235 (2008)

    2008

  19. [27]

    Journal of the Association for Information Science and Technology 70(1), 49–60 (2019)

    Losada, D.E., Parapar, J., Barreiro, A.: When to Stop Making Relevance Judgments? A Study of Stopping Methods for Building Information Retrieval Test Collections. Journal of the Association for Information Science and Technology 70(1), 49–60 (2019)

    2019

  20. [28]

    ACM Transactions on Information Systems (TOIS) 39(1), 1–34 (2020)

    Mcdonald, G., Macdonald, C., Ounis, I.: How the Accuracy and Confidence of Sensitivity Classification Affects Digital Sensitivity Review. ACM Transactions on Information Systems (TOIS) 39(1), 1–34 (2020)

    2020

  21. [29]

    Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.: Playing Atari with Deep Reinforcement Learning (2013)

    2013

  22. [30]

    Data Mining and Knowledge Discovery pp

    Molinari, A., Esuli, A.: SAL 𝜏: Efficiently Stopping TAR by Improving Priors Estimates. Data Mining and Knowledge Discovery pp. 1–34 (2023)

    2023

  23. [31]

    In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

    Montazeralghaem, A., Zamani, H., Allan, J.: A Reinforcement Learning Frame- work for Relevance Feedback. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 59–68 (2020)

    2020

  24. [32]

    Journal of Ma- chine Learning Research 22(268), 1–8 (2021), http://jmlr.org/papers/v22/20- 1364.html

    Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., Dormann, N.: Stable- Baselines3: Reliable Reinforcement Learning Implementations. Journal of Ma- chine Learning Research 22(268), 1–8 (2021), http://jmlr.org/papers/v22/20- 1364.html

    2021

  25. [33]

    In: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval

    Ren, Z., Huang, N., Wang, Y., Ren, P., Ma, J., Lei, J., Shi, X., Luo, H., Jose, J., Xin, X.: Contrastive State Augmentations for Reinforcement Learning-Based Recommender Systems. In: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Info...

    2023

  26. [34]

    In: Proceedings of the 31st Interna- tional Conference on Distributed Computing Systems workshops

    Satopaa, V., Albrecht, J., Irwin, D., Raghavan, B.: Finding a “Kneedle” in a Haystack: Detecting Knee Points in System Behavior. In: Proceedings of the 31st Interna- tional Conference on Distributed Computing Systems workshops. pp. 166–171. IEEE (2011)

    2011

  27. [35]

    arXiv preprint arXiv:1707.06347 (2017)

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347 (2017)

    2017 arXiv

  28. [36]

    Research Synthesis Methods 5(1), 31–49 (2014)

    Shemilt, I., Simon, A., Hollands, G.J., Marteau, T.M., Ogilvie, D., O’Mara-Eves, A., Kelly, M.P., Thomas, J.: Pinpointing needles in giant haystacks: use of text mining to reduce impractical screening workload in extremely large scoping reviews. Research Synthesis Methods 5(1)...

    2014

  29. [37]

    ACM Transactions on Information Systems 42(3), 1–37 (2023), https://doi.org/10.1145/3631990

    Stevenson, M., Bin-Hezam, R.: Stopping Methods for Technology-assisted Re- views Based on Point Processes. ACM Transactions on Information Systems 42(3), 1–37 (2023), https://doi.org/10.1145/3631990

    2023 doi

  30. [38]

    The MIT Press, Cambridge (2018)

    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. The MIT Press, Cambridge (2018)

    2018

  31. [39]

    John Wiley & Sons, Hoboken, New Jersey (2012)

    Thompson, S.: Sampling. John Wiley & Sons, Hoboken, New Jersey (2012)

    2012

  32. [40]

    Towers, M., Terry, J.K., Kwiatkowski, A., Balis, J.U., Cola, G.d., Deleu, T., Goulão, M., Kallinteris, A., KG, A., Krimmel, M., Perez-Vicente, R., Pierré, A., Schulhoff, S., Tai, J.J., Shen, A.T.J., Younis, O.G.: Gymnasium (Mar 2023), https://zenodo.org/ record/8127025

    2023

  33. [41]

    Machine Learning 8, 229–256 (1992)

    Williams, R.J.: Simple Statistical Gradient-Following Algorithms for Connection- ist Reinforcement Learning. Machine Learning 8, 229–256 (1992)

    1992

  34. [42]

    In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

    Xin, X., Karatzoglou, A., Arapakis, I., Jose, J.M.: Self-Supervised Reinforcement Learning for Recommender Systems. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 931–940 (2020)

    2020

  35. [44]

    In: Proceedings of the 21st ACM Symposium on Document Engineer- ing 2021 (DocEng ’21)

    Yang, E., Lewis, D.D., Frieder, O.: On Minimizing Cost in Legal Document Review Workflows. In: Proceedings of the 21st ACM Symposium on Document Engineer- ing 2021 (DocEng ’21). pp. 1–10 (2021). https://doi.org/10.1145/3469096.3469872

    2021

  36. [45]

    In: 2nd International Conference on Design of Experimental Search & Information REtrieval Systems (DESIRES 2021)

    Yang, E., Lewis, D.D., Frieder, O.: TAR on Social Media: A Framework for Online Content Moderation. In: 2nd International Conference on Design of Experimental Search & Information REtrieval Systems (DESIRES 2021). pp. 147–155 (2021)

    2021

  37. [46]

    Expert Systems with Applications 120, 57–71 (2019)

    Yu, Z., Menzies, T.: FAST2: An Intelligent Assistant for Finding Relevant Papers. Expert Systems with Applications 120, 57–71 (2019)

    2019

  38. [2016]

    NIST Special Publication, vol. 500-321. National Institute of Standards and Technology (NIST) (2016)

    2016

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.