{"id":"b58190b5-3a2d-4c7c-8c63-f868b4f6e14e","arxiv_id":"2412.19877","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An active learning method that uses deep reinforcement learning (DDPG) to decide which unlabeled images to query, after pre-ranking images by margin uncertainty, reports modest accuracy improvements on CIFAR-10, SVHN, and Fashion-MNIST.","lead":"This paper presents a method that uses reinforcement learning to choose which images a human should label first when training an image classifier. It reports small accuracy gains over standard active learning baselines on three common datasets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DRAL's acceptance gating (Sec. 3.1) lets it discard oracle labels whenever reward is non-positive, so the reported 'Sizes' are not equal labeling budgets; the comparison to baselines is unfair.","rationale":"The reader's weakest assumption focused on the undefined accuracy set in the reward (test vs. validation leakage), which is a valid concern. However, the deeper and more concrete problem is the reward's interaction with data addition: Sec. 3.1 states that samples are added to the training set only when the reward is positive. This creates a fundamental asymmetry between DRAL and the baselines: DRAL may query the oracle for labels it then discards, so the 'Sizes' rows in Tables 1-3 are not comparable labeling budgets. The central claim depends on equal labeling effort, and the accepted protocol does not enforce that. This is an internal inconsistency in the experimental design, not merely a missing implementation detail. It can be settled empirically by the proposed ablation. Since the issue is serious but addressable with a revised protocol, the reader's CONDITIONAL verdict remains appropriate; my read does not change the verdict, hence UNCHANGED.","tokens_in":12722,"tokens_out":7758,"duration_ms":81636,"concrete_test":"Reproduce DRAL on CIFAR-10 with two changes: (1) remove the positive-reward gating so that all oracle-selected samples are added to the labeled set, and (2) log the number of oracle queries separately from the number of accepted training samples. Compare against RANDOM and margin sampling at equal query counts (e.g., 1000, 2000, 4000, 7000 total queries), rather than at equal accepted sizes. If DRAL's accuracy advantage shrinks or disappears under equal query budgets, the reported gains are an artifact of the gating; if it persists, the gating is not the source of the improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing flaw is the acceptance gating introduced in Sec. 3.1: 'Only when the difference is positive, the selected samples are added to the pool of labeled training data in order to further retrain the classifier.' This means DRAL discards oracle-labeled samples whenever the reward Acc(phi_t) - Acc(phi_{t-1}) is non-positive. In contrast, all baselines (RANDOM, EN, LC, MS) add every selected sample to the labeled set. The experiments report 'Sizes' 1000-7000 as though these were equal labeling budgets, but for DRAL the 'Size' can only count accepted samples; the number of oracle queries needed to reach that size is at least as large, and likely larger, because rejected batches consume label queries without growing the training set. Consequently, the central claim that DRAL 'significantly outperforms' the handcrafted strategies at the same labeling budgets is not supported by the described protocol. A secondary concern, identified by the reader, is that the accuracy Acc used in the reward is never specified as being computed on a held-out validation set; if it is the test accuracy, the policy is trained with test labels. But the acceptance gating alone is sufficient to invalidate the equal-budget comparison as currently reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DRAL, a deep reinforcement active learning method that casts sample selection as a Markov decision process. Unlabeled images are first ranked by classifier margin uncertainty, and the top n ranked feature vectors form the state. An actor network outputs accept/reject decisions for these candidates; accepted samples are labeled by the oracle, and a reward equal to the change in classifier accuracy is used to train the actor and critic via DDPG. Experiments on CIFAR-10, SVHN, and Fashion-MNIST compare DRAL against random, entropy, least-confidence, and margin-sampling baselines at labeled sizes from 1,000 to 7,000, reporting consistent but modest accuracy gains for DRAL.","tokens_in":13039,"tokens_out":4841,"duration_ms":49012,"significance":"If the comparisons were at equal labeling budgets and the reward were computed from a legitimate validation split rather than the test set, the paper would make a useful contribution: learning a selection policy over uncertainty-ranked candidates with an actor-critic DDPG framework is a reasonable and potentially adaptive alternative to handcrafted strategies. The consistent numerical gains across three benchmarks would then be meaningful, and the t-SNE visualizations provide qualitative support. However, as written, two protocol-level gaps undermine the central claim: the acceptance-gating procedure changes the effective labeling cost for DRAL relative to the baselines, and the definition of the reward accuracy is never tied to a specific data split. These issues must be resolved before the claimed superiority can be accepted.","major_comments":[{"comment":"The acceptance-gating procedure in Sec. 3.1 invalidates the equal-budget comparison. The text states that selected samples are labeled by the oracle and used in reward estimation, and that 'only when the difference is positive, the selected samples are added to the pool of labeled training data'. Rejected batches therefore consume oracle labels without increasing the training set. The 'Sizes' columns in Tables 1-3 can only count accepted samples for DRAL, whereas all baselines add every selected sample. Reaching a reported size of, say, 7,000 therefore requires DRAL to issue more oracle queries than the baselines, so the claimed 'same number of samples' comparison is not a same-cost comparison. The authors must either remove the gating, compare all methods at equal numbers of oracle queries, or report both accepted-sample counts and total query counts.","section":"Sec. 3.1, Tables 1-3"},{"comment":"The reward r(S,a) = Acc(phi_t) - Acc(phi_{t-1}) is defined in terms of 'the accuracy of the classifier', but the manuscript never states which dataset is used to compute Acc. If Acc is computed on the test set, then test labels are used to train the selection policy, and the reported test accuracy is partly a fitted quantity; this would be circular and would explain the observed gains. If Acc is computed on a held-out validation set, that fact and the construction of the validation set must be stated explicitly, together with a guarantee that the validation set is not used for early stopping or model selection in a way that leaks into the test results. Because this reward is the sole training signal for the RL policy, the missing specification is a load-bearing omission.","section":"Eq. (2), Sec. 3.2.4"},{"comment":"The claim that DRAL 'significantly outperforms' the baselines is not supported by the reported experiments. The tables show single runs with no variance estimates, no multiple seeds, and no significance tests. Several margins are small: on CIFAR-10 at size 7,000 the gap to margin sampling is 0.62 percentage points, and on Fashion-MNIST at size 7,000 it is 0.26 points. Given the modest differences, the authors should report means and standard deviations over at least several random seeds and, where appropriate, paired comparisons, before using the word 'significant'.","section":"Tables 1-3, Sec. 4.2"}],"minor_comments":[{"comment":"The left-hand side r(S,a) does not depend on the action a; the paper should clarify whether the reward is defined only for accepted samples or is taken as zero for rejected samples.","section":"Eq. (2)"},{"comment":"The description of the mini-batch budget b is contradictory: it first says 'we do not obtain b samples at once and submit them to the oracle for labeling', then says 'Once b is reached, the b samples are labeled and fed to the classifier for further retraining'; the intended accumulation-and-query process should be restated precisely.","section":"Sec. 3.2.3"},{"comment":"The network description is inconsistent: Sec. 3.2.2 describes the actor as having two convolutional layers, one pooling layer, and three fully connected layers, while Sec. 4.1 says both actor and critic consist of five fully connected layers; the actual architecture used in the experiments should be given unambiguously.","section":"Sec. 4.1"},{"comment":"The caption says 'In each iteration, 1000 new images are selected for manual annotation', while Sec. 4.1 sets the small budget b to 100; the relationship between b, the iteration count, and the 1,000-sample increments in the tables should be clarified.","section":"Fig. 2 caption"},{"comment":"The reference list contains many entries that are never cited in the body (e.g., [2], [4], [7], [16], [20], [26], and a long tail of multi-label classification references [43]-[127]); the list should be trimmed to works actually discussed in the text.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The acceptance-gating issue is the most serious problem: as written, the reported 'Sizes' are not equal labeling budgets, so the central comparison is not fair. If the authors can provide a revised protocol with equal oracle-query counts and a clearly specified validation-based reward, the paper may be salvageable; if they instead keep the current protocol or continue to leave the reward source unspecified, I would recommend rejection. I also note that the reference list contains a large number of apparently uncited entries, which may warrant editorial attention independent of the technical content."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this paper does exactly what it says: it takes the RL active learning framework from Slade and Branson [30]—MDP, DDPG, actor-critic, accuracy-difference reward—and adds a margin-based preselection step before the actor network. That is a real, if incremental, engineering contribution, and the writing is refreshingly clear about it. Second, the experimental comparison as reported is not trustworthy, for a reason the stress-test note puts its finger on correctly: the acceptance gating in Sec. 3.1 says selected samples are added to the labeled pool only when the reward is positive. Baselines label every selected sample. So at a reported \"Size\" of 7000, DRAL may have queried the oracle 8000 or 10000 times and discarded the rest. The tables are therefore not equal-budget comparisons, and the central claim of \"significantly outperforming\" the heuristics is unsupported by the protocol described.\n\nThe paper deserves credit for a few things. It cites [30] and frames its contribution as a computational improvement rather than pretending the MDP formulation is new. The t-SNE visualization in Fig. 2 is a nice qualitative touch. The method is described with enough detail that the flaw I just named is actually visible in the text, which is more than you get from many papers.\n\nNow the soft spots, in order of severity. First, the acceptance gating issue is load-bearing. It is not a minor omission; it invalidates the headline result. Second, the reward in Eq. (2) is an accuracy difference but the paper never states what set the accuracy is measured on. If it is the test set, the policy is trained with test labels, which is circular. The reader flagged this, and the paper does not resolve it. Third, there are no error bars or multiple seeds, no comparison to the closest prior method [30], and no code. These are smaller but still meaningful for a paper whose gains are on the order of one point.\n\nWho gets value from this? Someone working on RL-based active learning might read it as a data point about the effect of preselection, and the t-SNE is a decent qualitative illustration. But the quantitative results should not be cited until the budget accounting and the reward dataset are fixed.\n\nMy recommendation: this is not a desk reject, but it is a major-revision paper. The flaws are specific, addressable, and not the result of sloppy thinking. A serious referee should see it, with the clear expectation that the acceptance-gating protocol and the reward definition are fixed and the experiments rerun.","headline":"The acceptance gating in Sec. 3.1 breaks the equal-budget comparison, and the reward accuracy is never pinned down; the method is a sensible tweak on [30] but the evaluation needs a serious redo.","tokens_in":13541,"tokens_out":1695,"would_cite":false,"duration_ms":19147,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Deep reinforcement learning learns a sample-selection policy for active learning that outperforms standard handcrafted strategies on three image benchmarks.","keywords":["Active learning","Deep reinforcement learning","Image classification","Markov decision process","DDPG","Actor-critic","Sample selection","Uncertainty sampling"],"falsifier":"Compute the reward in Eq. (2) using a held-out validation split instead of the test set and rerun the reported experiments; if the accuracy gains over margin sampling shrink or reverse, the method's edge depends on test labels that are unavailable in real active learning.","tokens_in":12552,"feed_emoji":"🤖","tokens_out":7416,"duration_ms":60111,"temperature":0.7,"pith_summary":"Active learning normally relies on handcrafted rules, such as picking the most uncertain or least confident examples, which can fail as the data or task changes. This paper argues that sample selection can itself be learned: it casts active learning as a Markov decision process and trains an actor-critic agent with deep deterministic policy gradient (DDPG) to decide, for each uncertainty-ranked candidate, whether to spend a label on it. The claimed payoff is a dynamic policy that adapts to the oracle's feedback and the classifier's state, and the experiments report higher test accuracy than random, entropy, least-confidence, and margin sampling on CIFAR-10, SVHN, and Fashion-MNIST at labeling budgets from 1,000 to 7,000 images. A sympathetic reader would care because label budgets are the bottleneck in many applied vision tasks, and a learned policy could remove the need to guess which heuristic works.","feed_headline":"RL-trained policy beats handcrafted active learning on 3 benchmarks","feed_subtitle":"A DDPG actor decides which unlabeled images to label, raising accuracy at budgets from 1,000 to 7,000 samples.","key_machinery":"The key mechanism is the MDP formulation of active learning. The state $S_t$ is a matrix of deep features of the $n$ unlabeled images ranked highest by classifier margin uncertainty. An actor network with convolutional and fully connected layers maps that state to per-sample actions in $[-1,1]$, thresholded at zero to select or discard each candidate; the critic estimates the Q-value via Eq. (3); and the reward in Eq. (2), $r(S,a)=\\mathrm{Acc}(\\phi_t)-\\mathrm{Acc}(\\phi_{t-1})$, is the accuracy change of the classifier. Training uses DDPG with target actor and critic networks, soft updates via Eq. (7), and a replay buffer to stabilize learning.","core_discovery":"The paper's central claim is that a deep reinforcement learning agent, trained by DDPG over an MDP whose state is the top-$n$ margin-uncertain unlabeled images, can learn a greedy binary selection policy that outperforms fixed handcrafted active learning strategies. The agent receives a reward equal to the change in classifier accuracy before and after adding the selected samples, and only samples with positive predicted impact get labeled and added to the training set. Across three image classification benchmarks, the reported DRAL accuracy monotonically exceeds the baselines at every tested budget size, with the largest gaps typically at small budgets, and t-SNE visualizations suggest the selected samples are more evenly spread across the class distribution.","pith_inferences":["The paper leaves open whether the reported improvements survive when the reward accuracy is computed on a held-out validation split rather than the test set; that is a natural testable check of the method's validity.","The preselection step could be replaced by other cheap criteria (e.g., diversity or density) and the RL policy might compensate, which would decouple the method's gains from margin uncertainty.","A stronger comparison would pit DRAL against learning-based active learning baselines (e.g., loss-prediction methods), not only handcrafted heuristics; the current experiments do not show how it fares against other learned policies.","Since the policy is learned per dataset, an implicit extension is to test whether a policy trained on one dataset transfers to another, which the paper does not examine."],"forward_implications":["If DRAL's gains hold, active learning no longer requires choosing a heuristic in advance; the selection strategy is learned from interaction.","The same framework could be applied to other prediction tasks beyond image classification, since the state and reward are defined generically in terms of classifier features and accuracy.","The budget scaling behavior (bigger gains at small budgets) suggests learned selection is most valuable when labels are scarce.","Because the method preselects by margin uncertainty, it reduces the RL problem size and can scale to larger unlabeled pools without exhaustive per-sample evaluation."],"supporting_citations":[{"why":"Supplies the Deep Deterministic Policy Gradient algorithm used to train the actor-critic networks.","marker":"[19]"},{"why":"Prior work that first reframed active learning sample selection as a reinforcement learning problem, which DRAL extends.","marker":"[24]"},{"why":"The most recent MDP-based deep RL active learning baseline that DRAL's preselection is designed to scale beyond.","marker":"[30]"},{"why":"Provides the deep Q-learning inspiration for the greedy policy design in Eq. (4).","marker":"[31]"},{"why":"Contributes the target-network and replay-buffer training scheme that Eqs. (6)-(7) adopt.","marker":"[33]"},{"why":"Provides the ResNet-18 architecture used as the final classifier evaluated on the selected samples.","marker":"[34]"},{"why":"Defines entropy sampling, one of the handcrafted baselines that DRAL compares against.","marker":"[38]"},{"why":"Defines least-confidence sampling, a baseline and part of the uncertainty-sampling family.","marker":"[39]"},{"why":"Defines margin sampling, a core baseline and the basis of the preselection criterion.","marker":"[41]"}],"fun_headline_variants":["RL agent learns to pick images to label, beats handcrafted AL","DDPG policy outperforms fixed active learning on 3 image datasets","Reinforcement learning selects samples better than manual strategies","Adaptive active learning via RL: higher accuracy on 3 benchmarks","RL-trained sample selection outperforms classic active learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reward signal is defined as the change in classifier accuracy, but the paper never states whether that accuracy is measured on a held-out validation set or on the test set, so the learned policy's improvements could depend on labels that a real active learner would not have.","fun_headline_variants_meta":{"raw":{"variants":["RL agent learns to pick images to label, beats handcrafted AL","DDPG policy outperforms fixed active learning on 3 image datasets","Reinforcement learning selects samples better than manual strategies","Adaptive active learning via RL: higher accuracy on 3 benchmarks","RL-trained sample selection outperforms classic active learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000593,"raw_usage":{"total_tokens":2749,"prompt_tokens":884,"completion_tokens":1865,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":1779}},"tokens_in":500,"tokens_out":1865,"duration_ms":11823,"temperature":1.0,"reasoning_tokens":1779,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:49:39.074122+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the reward in Eq. (2) using a held-out validation split instead of the test set and rerun the reported experiments; if the accuracy gains over margin sampling shrink or reverse, the method's edge depends on test labels that are unavailable in real active learning.","supporting_citations":[{"cited_title":"Deep reinforced active learning for multi-class image classification","cited_arxiv_id":"2206.13391","evidence_quote":"The most recent MDP-based deep RL active learning baseline that DRAL's preselection is designed to scale beyond."}],"review_version":1}