{"id":"6f9d581e-07f4-4ec4-b810-4535cd12c94e","arxiv_id":"2505.01907","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"GRLStop is a reinforcement learning stopping rule for Technology Assisted Review whose reward function lets one model serve multiple target recall levels and user-selected recall/cost tradeoffs, and it improves or matches baselines on six TAR datasets.","lead":"Technology Assisted Review helps lawyers and medics screen huge document sets, and this paper trains a reinforcement learning agent to decide when to stop reviewing. The new method, GRLStop, uses a small set of reward-function dials so one trained model can aim for different recall targets and different cost-versus-recall preferences.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The single-model claim depends on the policy inferring the target-reach batch T from a state whose unexamined-batch features are noisy logistic-regression estimates; this link is untested and could be a major failure mode.","rationale":"The reader's weakest assumption is precisely that the state vector contains enough information for the policy to recognize when the target recall has been reached, and that PPO generalizes across target recalls and datasets. I agree this is the most load-bearing element: the entire flexibility claim of a single model for multiple target recalls rests on the policy being able to estimate the unobserved batch T from the state's classifier-derived features. The paper's experiments (Figure 3, Table 1) show that GRLStop works on the test collections, but they do not decompose the contribution of the classifier estimates to the policy's decision. The ablation in Section 5.2 only compares with and without classifier features, not with accurate versus inaccurate features. A concrete oracle-classifier test would directly quantify the sensitivity of the method to the weakest component of the state representation. I do not see an internal inconsistency or a fundamental flaw in the reward derivation; the normalized cumulative reward in Eq. 2 is correct, and the empirical results are plausible. The main risk is that the method's success is conditional on classifier calibration in a way that is not measured and may not transfer. This supports a CONDITIONAL verdict, which the reader already gave, so no adjustment is needed.","tokens_in":18215,"tokens_out":10616,"duration_ms":107891,"concrete_test":"Build an oracle-classifier variant of GRLStop where the state's unexamined-batch values are the true relevance proportions (available from the benchmark ground truth) instead of logistic-regression estimates. Run both variants on all six datasets and all target recalls, and compare recall, cost, and CostDiff, preferably with multiple seeds. If the oracle-classifier variant is substantially closer to the oracle stopping point (e.g., average CostDiff improves by more than 0.02) and the standard GRLStop misses target recall more often, then classifier estimate quality is a first-order bottleneck for the single-model claim. If the two variants are statistically indistinguishable, then the policy is robust to classifier error and the concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that one policy, trained with the normalized reward in Eq. 1, can handle multiple target recalls because the state (Section 3.1) includes the target recall and per-batch relevance proportions. However, the reward function uses T, the batch at which the target recall is actually reached, which is unknown at inference. The policy must therefore infer T from the state. The only information about unexamined batches comes from a logistic-regression classifier trained on the examined batches. Early in an episode, this classifier is trained on very few labeled documents (with B=100 and the first batch examined, about 1% of the collection), so its per-batch estimates are likely noisy and potentially biased, especially for rare relevant documents. If the classifier systematically underestimates the number of relevant documents in unexamined batches, the policy will underestimate the total relevant count and stop before the true T; overestimation will delay stopping. Section 5.1 demonstrates that GRLStop's decisions vary with target recall, but it does not isolate whether the policy is successfully inferring T or merely memorizing dataset-specific patterns of the fixed logistic-regression estimator. Section 5.2 ablates the classifier's presence, not its accuracy. Thus the empirical support for the single-model generalization claim does not establish that the approach would transfer to a new review task where the classifier's calibration could be different. This is load-bearing because if the method requires a well-calibrated classifier that is not available in practice, the claimed flexibility and Pareto optimality would degrade.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces GRLStop, a reinforcement learning (RL) environment and policy for Technology Assisted Review (TAR) stopping. It extends the authors' earlier RLStop by (i) a reward function whose cumulative reward has range [0,1] regardless of the batch at which the target recall is reached, allowing a single policy to serve multiple target recalls; (ii) two reward parameters m and n that control the recall/cost tradeoff; and (iii) integration of classifier predictions for unexamined batches into the RL state. Experiments on six TAR datasets (CLEF e-Health 2017-19, TREC Total Recall, TREC Legal, and RCV1) with AutoTAR rankings compare GRLStop against ten baselines and an oracle, and also evaluate classifier inclusion, reward-objective adaptation, and ranking quality. The main claimed results are that GRLStop is Pareto optimal in almost all recall/cost scenarios, that a single model adapts across target recalls, and that classifier integration and reward adaptation are beneficial.","tokens_in":18478,"tokens_out":9983,"duration_ms":91087,"significance":"If the central claims hold, GRLStop would be a practical advance for TAR stopping: it removes the need to train a separate RL model per target recall and gives the practitioner explicit control over the recall/cost tradeoff. The paper's strengths include a clearly specified reward design with an analytical cumulative-reward argument, evaluation on multiple benchmark datasets against a broad set of baselines, a ranking-quality analysis, and released code (https://github.com/ReemBinHezam/GRLStop). The empirical support, however, has gaps that affect the strength of the claims: no repeated-seed variance or confidence intervals are reported for RL training, the Legal test set contains only two topics, and on the TR dataset the model does not vary its stopping decision with target recall for most settings. The central generalization mechanism (inferring the target-reach batch from noisy classifier estimates) is asserted but not directly validated.","major_comments":[{"comment":"All headline results are reported as single point estimates without repeated-seed variance or confidence intervals. Because the policy is trained with PPO, which is stochastic in initialization and updates, a single run is not sufficient to establish that GRLStop is 'Pareto optimal in almost every case' or 'consistently closer to the optimal Oracle results than other approaches'. Please report means and standard deviations over at least five seeds and apply a significance test to the key comparisons (e.g., CostDiff against the best baseline).","section":"Section 5, Figure 3, Tables 1-3"},{"comment":"The reward function depends on T, the batch at which the target recall is achieved, which is known during training but not at inference. At inference the policy receives states whose unexamined-batch values are logistic-regression estimates (Section 3.1) and must infer T from these noisy quantities. The paper never validates this inference mechanism: Section 5.2 ablates the classifier's presence, not its accuracy, and Section 5.1 shows only that stopping decisions differ across target recalls. A direct test would compare the policy's implied T against the oracle T, or vary classifier calibration and measure the effect on CostDiff.","section":"Section 3.1, Eq. (1)"},{"comment":"For the TR dataset, GRLStop produces identical recall (0.950), reliability (0.941), and cost (0.010) for target recalls 0.7, 0.8, and 0.9. This directly contradicts the Section 5.1 statement that 'results for all metrics vary depending on the target recall' and suggests that on this dataset the policy learns a single early stopping point rather than conditioning on the target recall. The paper should acknowledge and explain this case, or qualify the single-model adaptation claim.","section":"Table 1, TR row"},{"comment":"The Legal dataset contains only two test topics. Consequently, reliability values in Table 1 and Table 2 are restricted to 0.0, 0.5, and 1.0, and the large performance gap on Legal between GRLStop-balanced (recall 0.504, cost 0.010) and GRLStop-recall-obj (recall 0.998, cost 0.200) hinges on just two topics. Claims about Legal behaviour and the general observations drawn from it should be tempered or supplemented with additional topics.","section":"Section 4.1, Tables 1-2"},{"comment":"The training protocol for the claimed single model is under-specified. The paper does not state how target recalls are sampled or mixed across training episodes, whether a separate model is trained for each dataset (which is implied by the dataset-specific training sets in Section 4.1) or across datasets, or how the grid search and the early-stopping callback interact with the test-set evaluation. These details are needed to reproduce the single-model generalization experiment in Section 5.1.","section":"Section 4.4"},{"comment":"The cumulative-reward analysis in Eq. (2) assumes an undiscounted sum of rewards, but PPO is trained with a discount factor of 0.99 (Section 4.4). The claimed range-invariance property and the shape of the objective do not automatically transfer to the discounted objective actually optimized by the agent. The authors should either prove the property for the discounted objective or present empirical evidence (e.g., sensitivity to the discount factor) that the learned policy is unaffected.","section":"Section 3.1, Eq. (2), and Section 4.4"}],"minor_comments":[{"comment":"The sentence 'Results showed that the proposed approach to be effective' is ungrammatical; it should be rephrased, e.g., 'Results showed that the proposed approach is effective'.","section":"Abstract"},{"comment":"The grid search over hyperparameters is not described; the paper should report which values were tried for the learning rate, entropy coefficient, clipping range, and the reward parameters m and n, or state that these were fixed from prior work.","section":"Section 4.4"},{"comment":"The paired t-test with Bonferroni correction is mentioned without specifying the exact metric and pooling strategy (per-topic CostDiff across all datasets, across all target recalls, or both). Please clarify the statistical procedure.","section":"Section 5.2"},{"comment":"The statement that results for other datasets 'display similar trends' should be supported by the actual tables, either in an appendix or as supplementary material, instead of being omitted for brevity.","section":"Section 5.4"},{"comment":"For RCV1, the 45 categories used for evaluation are not listed; a reference or an explicit list is needed for reproducibility.","section":"Section 4.1"},{"comment":"The case T = B is excluded because the denominator (B-T)^n vanishes, but it is not stated how episodes with T = B are handled during training; this boundary case should be clarified.","section":"Section 3.1, Eq. (1)"}],"recommendation":"major_revision","confidential_remarks":"The paper is a direct extension of the authors' own RLStop work, and the self-references are appropriate. The released code is a positive feature. The main risks to the claims are the absence of repeated-seed variance, the untested T-inference mechanism, and the two-topic Legal evaluation; these are fixable within the scope of the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a worthwhile extension of the authors' own RLStop. The normalized reward in Eq. 1 is a genuine idea—it makes cumulative reward range-invariant to where the target recall falls, and the m/n parameters give explicit control over undershoot vs overshoot. The paper earns its keep on that alone. The multi-dataset evaluation (six collections, AutoTAR rankings, sensible baselines) is broad, and the experiment in Table 1 makes the single-model point well: fixed-recall RLStop models degrade when applied to other targets, while GRLStop adapts.\n\nWhat's new and good: the state now includes classifier estimates for unexamined batches, and the ablation in §5.2 shows that helps on most datasets/target recalls. The ranking-quality analysis in §5.4 is a nice addition, because stopping rules are known to be sensitive to ranking quality. CostDiff is a useful summary metric that combines recall achievement and workload.\n\nSoft spots, in rough order:\n\n- No repeated-seed variance or confidence intervals around the main results. PPO is stochastic; one run per configuration means a re-run could shift the Pareto front. The paired t-test in §5.2 uses topics as units, not training runs, so it doesn't fix that. This is the biggest methodological gap.\n\n- The Legal dataset has two test topics, so any per-topic metric there is fragile. Minor, but should be stated.\n\n- 'State-of-the-art' in the contributions is stronger than the evidence. For target recall 1.0, GRLStop frequently fails to reach the target; the paper's own framing is that it trades cost, which is legitimate, but then 'state-of-the-art' should be qualified as a cost-recall tradeoff, not raw reliability.\n\n- The 'one model for multiple recalls' claim depends on the policy inferring T, the batch where target recall is actually reached, from the state. The state includes target recall and classifier estimates for unexamined batches. The classifier is retrained from examined batches only, so early in an episode it has very little data to work with. The paper shows the classifier helps on average but never measures its calibration or tests the policy under a different classifier. That is a real open question. The held-out-topic evaluation mitigates it—the policy does see new topics—but it doesn't isolate whether the policy learned a robust estimate of T or quirks of this particular logistic-regression pipeline.\n\nVerdict: worth a serious referee. I'd send it to review and ask for repeated-seed results, a Legal caveat, and one experiment with a deliberately degraded or different classifier at test time to probe how much the method leans on classifier calibration. For TAR researchers this is a real step forward; for an IR reading group it would produce a good discussion of what 'generalization' means in RL-equipped stopping rules.","headline":"A solid, citable extension of RLStop whose normalized reward does what it claims; the main open question is how much the policy depends on classifier calibration, and the evaluation lacks repeated-seed variance.","tokens_in":19046,"tokens_out":8030,"would_cite":true,"duration_ms":69304,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GRLStop makes one reinforcement-learning policy serve multiple target recalls in technology-assisted review, reaching target recall at lower cost than baselines in almost every tested configuration.","keywords":["reinforcement learning","technology-assisted review","TAR stopping","high-recall retrieval","reward design","document screening","systematic reviews"],"falsifier":"Take a held-out set of topics, replace every classifier estimate in the unexamined batches with random values between 0 and 1, and compare GRLStop's stopping cost and recall to the reported results; a large degradation would show the results hinge on classifier accuracy. Separately, train GRLStop with the target-recall element of the state vector fixed to a constant and test on recall targets not seen in training; if performance is unchanged, the policy is not actually using the target recall, contradicting the explanation of how generalisation works.","tokens_in":17987,"feed_emoji":"🎯","tokens_out":9180,"duration_ms":86191,"temperature":0.7,"pith_summary":"The paper claims that a single reinforcement-learning stopping policy, GRLStop, can replace the per-target models required by previous RL stopping methods in technology-assisted review (TAR). Its reward function is normalised so that reaching the target recall always produces the same maximum cumulative reward, regardless of where in the ranking that target falls, and two parameters let a user tilt the policy toward recall maximisation or cost minimisation. On six TAR benchmark collections the paper reports that GRLStop is Pareto optimal in almost every tested configuration, reaching target recall at lower cost than baselines and closer to the oracle stopping point. If these results hold, RL-based stopping becomes practical for systematic reviews, legal discovery, and other high-recall screening tasks where target recall and cost sensitivity change from case to case.","feed_headline":"One RL model now sets the stopping point for any recall target","feed_subtitle":"A normalized reward lets reviewers trade recall against cost without retraining for each target.","key_machinery":"The load-bearing object is the normalised reward function of Eq. 1, whose cumulative form is the ratio-symmetric pair $(i/T)^m$ and $((B-i)/(B-T))^n$; it makes the reward range invariant to the position of the target recall, which is what lets a single model train across many target positions. The state is a fixed vector of length $B+2$: one element per batch (observed relevance proportion for examined batches, logistic-regression TF-IDF estimate for unexamined batches), plus the number of batches examined and the target recall. A feed-forward neural network with two hidden layers and a softmax output maps that state to STOP or CONTINUE, and the policy is trained with PPO, a policy-gradient reinforcement-learning algorithm, over parallel episodes from multiple topics.","core_discovery":"The central claim is that the stopping problem in TAR can be solved by one policy rather than one policy per target recall. GRLStop trains a feed-forward neural policy with a reward function, Eq. 1, whose cumulative value is $CR(S_i)=(i/T)^m$ before the target batch $T$ and $CR(S_i)=((B-i)/(B-T))^n$ after it; the maximum is always 1 at $i=T$, so rewards from different rankings and different target positions are comparable. At inference the same policy receives a state vector that records, for each of $B$ batches, the observed relevant proportion (for examined batches) or a classifier's estimate (for unexamined batches), together with the number of batches examined and the target recall. The paper claims this policy reaches target recall at lower cost than existing stopping baselines on six datasets, that the $m,n$ parameters visibly shift the recall/cost tradeoff in the intended direction, and that including classifier estimates significantly improves the CostDiff score at most target recalls.","pith_inferences":["Editorial inference: because the cumulative reward is a function of the ratio $i/T$ (and $(B-i)/(B-T)$), the policy is likely learning the fraction of the way to the target rather than an absolute batch count; this predicts it would also stop sensibly at an untested target such as 0.85, which the paper does not directly show.","Editorial inference: the reward family is a two-parameter distortion of a linear target-distance curve, meaning GRLStop can be viewed as one RL implementation of an entire continuum of stopping rules; the same normalisation idea could be exported to other sequential tasks with a latent target fraction, such as stopping active learning or qualitative coding saturation.","Editorial inference: the paper's ranking-quality results suggest the weakest link is classifier quality under noisy rankings; a natural testable extension is to use confidence-calibrated classifier probabilities instead of point predictions in the unexamined batches, which should improve CostDiff most on low-quality rankings.","Editorial inference: the comparison to RLStop in Table 1 is arguably favourable to GRLStop because a single GRLStop model is compared against RLStop models that were each trained for one target; a fairer stress test would be to train RLStop on a mixture of targets and see whether its original reward function can also generalise when trained that way."],"forward_implications":["A single GRLStop model trained on one dataset can be applied to target recalls it never trained on (the paper demonstrates 0.7, 0.8, 0.9, and 1.0 with one model per dataset), removing the need to train and store one RL model per recall level.","Setting $m$ and $n$ steers the policy: $(m,n)=(4,0.25)$ yields higher recall and reliability at higher cost, while $(0.25,4)$ cuts cost at the expense of missed targets, so the same environment can serve recall-critical medical reviews and cost-sensitive legal review.","Adding classifier predictions for unexamined batches moves stopping decisions closer to the oracle on most datasets and target recalls, with statistically significant CostDiff improvements at target recalls 0.7, 0.8, and 1.0.","GRLStop lies on the Pareto front of recall versus cost in almost every dataset/target configuration tested, with the single exception of TR at target recall 1.0 where it is very close to the front."],"supporting_citations":[{"why":"It defines RLStop, the prior RL stopping method this work extends and the primary baseline for single-target training.","marker":"[6]"},{"why":"It supplies AutoTAR continuous active learning, used to create the rankings all stopping methods are evaluated on.","marker":"[9]"},{"why":"It provides the AutoStop sampling-based stopping baseline and reference implementations used for comparison.","marker":"[25]"},{"why":"It supplies the IP-H point-process stopping baseline and earlier evidence that ranking quality affects stopping performance.","marker":"[37]"},{"why":"It shows that classifier estimates of unexamined documents are useful for stopping, motivating the classifier integration in GRLStop.","marker":"[43]"},{"why":"It demonstrates the value of relevance predictions for unexamined documents in screening workflows, further supporting the classifier component.","marker":"[46]"},{"why":"It introduces Proximal Policy Optimization, the RL algorithm used to train the stopping policy.","marker":"[35]"}],"fun_headline_variants":["Single RL policy adapts to any recall target","Adaptive RL stopping: no retraining per recall target","Reinforcement learning that stops at any recall you set","Flexible RL stopping: one model for any recall target","RL stopping that bends to your recall target"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method depends on the state vector carrying enough signal about where the remaining relevant documents are and what the target recall is; if the classifier's estimates for unexamined batches are poor, or the policy cannot infer the target from the state, the training-time reward normalisation will not transfer to new rankings.","fun_headline_variants_meta":{"raw":{"variants":["Single RL policy adapts to any recall target","Adaptive RL stopping: no retraining per recall target","Reinforcement learning that stops at any recall you set","Flexible RL stopping: one model for any recall target","RL stopping that bends to your recall target"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000931,"raw_usage":{"total_tokens":3948,"prompt_tokens":869,"completion_tokens":3079,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":3003}},"tokens_in":485,"tokens_out":3079,"duration_ms":20326,"temperature":1.0,"reasoning_tokens":3003,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:06:28.117021+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of topics, replace every classifier estimate in the unexamined batches with random values between 0 and 1, and compare GRLStop's stopping cost and recall to the reported results; a large degradation would show the results hinge on classifier accuracy. Separately, train GRLStop with the target-recall element of the state vector fixed to a constant and test on recall targets not seen in training; if performance is unchanged, the policy is not actually using the target recall, contradicting the explanation of how generalisation works.","supporting_citations":[{"cited_title":"In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (2024)","cited_arxiv_id":null,"evidence_quote":"It defines RLStop, the prior RL stopping method this work extends and the primary baseline for single-target training."},{"cited_title":"Expert Systems with Applications 120, 57–71 (2019)","cited_arxiv_id":null,"evidence_quote":"It demonstrates the value of relevance predictions for unexamined documents in screening workflows, further supporting the classifier component."}],"review_version":1}