{"id":"ba9f1063-ec98-4967-8d06-5c3729fe883b","arxiv_id":"2411.18060","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"ORIS uses a Deep Q-Network with an inclusivity reward to select diverse documents for labeling, reducing simulated annotator memory-decay errors and improving machine model performance.","lead":"This paper tests ORIS, a reinforcement learning agent that selects which incoming documents to label so that a simulated human annotator makes fewer memory-decay errors. The method improves simulated labeling accuracy and downstream classifier performance on two emotion recognition datasets, but only under a synthetic human-error model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq 2's exponential-forgetting oracle is degenerate: with reported α=0.6, β=−19, max(1,e^{αΔt+β})=1 for every Δt, so the oracle always errs, contradicting Table III's 100.0 human F1 and undermining the central claim.","rationale":"The paper's central claim is that ORIS minimizes human labeling errors and improves ML model performance. The only evidence for the human-error half of that claim is the simulated oracle in Section III-B. My concern is not merely that the synthetic oracle may fail to match real human memory decay; rather, the oracle as written cannot produce the reported results. Eq 2 yields an error score identically equal to 1 under the stated parameters, so the 'fast forgetting' oracle always errs, yet Table III reports human F1-macro of 100.0 for ORIS under that oracle. Eq 1, the 'slow forgetting' sigmoid, has the wrong sign and makes errors decrease as time since last seen increases. These internal inconsistencies mean Tables II and III cannot be reproduced from the paper's own model. The DQN is trained on error-free labels with a diversity-based reward, so its learned policy has no direct incentive to reduce oracle errors; any observed reduction is an emergent property of the simulator. Because the simulator is degenerate, the central claim lacks even internal support. The reader's verdict of REJECT is appropriate, and my concern strengthens the basis for rejection by identifying a concrete, checkable inconsistency rather than only the absence of real-human validation. I mark agreement as partial because the reader's weakest_assumption focused on the oracle not representing real human behavior, while my attack is stronger: the stated oracle equations are mathematically incapable of generating the reported human performance numbers. The code is provided, and a re-run of the exponential-oracle experiment would settle the matter; if the code silently corrects Eq 2, that correction must be disclosed, and all reported human-performance results would need re-evaluation under a valid, realistic oracle model.","tokens_in":13172,"tokens_out":5627,"duration_ms":52783,"concrete_test":"Re-implement the exponential oracle using Eq 2 exactly as printed (α=0.6, β=−19) and run the Reddit ORIS δ=8 experiment from Table III. Because max(1,e^{0.6Δt−19})=1 for every Δt, every oracle label should be incorrect, yielding human F1-macro near 0. If the reported 100.0 value appears instead, the implementation does not match the equation, demonstrating that the paper's human-performance evaluation is not reproducible as written.","verdict_should_be":"REJECT","load_bearing_attack":"Section III-B defines the exponential 'fast forgetting' oracle as Eq 2: error probability score(c) = max(1, e^{αΔt+β}), with α=0.6 and β=−19 (Sec V-B2). Since the maximum of 1 and any real number is at least 1, this is not a probability; for every possible Δt the score is exactly 1. At Δt=0, e^{0.6·0−19}=e^{−19}≈5.6e−9, so max(1,·)=1; for larger Δt it is even larger. Thus the exponential oracle always labels every document incorrectly. This is internally inconsistent with Table III, where ORIS (δ=8) under the exponential oracle on Reddit achieves human F1-macro of 100.0 (and 99.9), which is impossible if every oracle label is wrong. The sigmoid oracle in Eq 1 has the opposite problem: 1/(1+e^{−αΔt+β}) with α=0.3 and β=9 decreases toward 0 as Δt grows, meaning the oracle becomes more accurate as time since last seeing a class increases, contrary to the memory-decay premise. The central claim that ORIS minimizes human labeling errors is therefore evaluated entirely on an oracle that, as written, is either degenerate (Eq 2) or has the wrong monotonicity (Eq 1). Moreover, the DQN is trained with an error-free oracle and receives only the inclusivity reward (Eq 4), so any reported human-error reduction is a property of the simulator, and the equations needed to reproduce or interpret that reduction are internally inconsistent. No real human annotation data are presented, so there is no external check on whether the effect transfers to actual annotators.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ORIS, a Deep Q-Network-based online active learning strategy that samples incoming documents to jointly improve machine model performance and reduce simulated human labeling errors caused by memory decay. The method's state representation combines document embeddings with average time-last-seen values per class, and the reward is a function of the Shannon entropy (inclusivity) of recently sampled class labels. The authors evaluate on Twitter and Reddit emotion datasets against random, uncertainty, and offline diversity sampling, reporting improvements in both machine F1-macro and simulated human F1-macro. The code is made available.","tokens_in":13567,"tokens_out":9065,"duration_ms":73350,"significance":"If the results were valid, the idea of using reinforcement learning to explicitly encourage inclusive sampling would be a useful contribution to online active learning for streaming analytics, especially in settings where annotator memory decay is a concern. The paper ships code and uses a standard DQN architecture, which aids reproducibility. However, the significance is substantially undercut because the evaluation is based entirely on a synthetic oracle, the key oracle equation (Eq 2) is mathematically invalid as written, and the reward function aligns with the simulated error model in a way that makes the reported human-performance gains largely a property of the simulator. No real human annotation data are presented, so the central claim that ORIS minimizes human labeling errors is not empirically supported.","major_comments":[{"comment":"The 'exponential forgetting' oracle defined in Eq 2 as error probability score(c) = max(1, e^{αΔt+β}) is not a valid probability and is internally inconsistent with the reported results. With the stated hyperparameters α=0.6 and β=−19 (Section V-B1), for any Δt < ~31.7 the score is exactly 1, and for larger Δt it exceeds 1, so the oracle always (or more than always) errs. This contradicts Table III, where ORIS (δ=8) under the exponential oracle achieves human F1-macro of 100.0 on Reddit. Even if the intended expression was min(1, e^{...}), the paper must state this correction and re-run the experiments; as written, the simulation and the headline numbers are irreconcilable.","section":"Section III-B, Eq 2"},{"comment":"The reward function (Eq 4) directly incentivizes high entropy of the sampled class labels, while the simulated oracle's error probability (Eq 1) is minimized precisely when all classes have been seen recently. The DQN is trained with an error-free oracle and only this inclusivity reward (Section IV-D). Consequently, the reported human-performance improvements are a property of the simulator, not evidence that ORIS reduces human errors in real annotation. To support the central claim, the authors need either real human annotation experiments or a more conservative statement that the method reduces errors under the assumed memory-decay model.","section":"Section IV-C3 and Section V-B1"},{"comment":"The headline improvement percentages ('up to 38.3% human & 55.7% machine performance improvement on Twitter, and up to 44.2% human & 70.1% machine performance improvement on Reddit') are not reproducible from Tables II and III unless the baseline is selectively chosen. For example, the 55.7% machine improvement on Twitter corresponds to comparing BERT Mini sigmoid ORIS (56.2) against the offline Diversity baseline (36.1), while the 38.3% human improvement comes from comparing against Uncertainty (62.7). The paper should state the reference baseline and configuration for each claimed percentage, or drop these uncorroborated numbers.","section":"Abstract and Section I"},{"comment":"The state representation computes time-last-seen values from the oracle's provided labels, which may be erroneous. A mislabeled document will fail to update the time-last-seen of the true class, and the k-averaging strategy does not fully repair this corruption. Since the DQN policy is trained exclusively on error-free oracle labels, the state distribution at inference (which includes erroneous updates) may be off-distribution, yet this mismatch is not analyzed. The paper should either model this noise during training or provide empirical evidence that the policy remains robust.","section":"Section IV-C1"}],"minor_comments":[{"comment":"There is a typo in Eq 4: an extra closing parenthesis appears in the expression for the reward.","section":"Equation 4"},{"comment":"The notation 'Di t+1' is ambiguous; it should likely be 'Dt+1' or the dataset index and stream position should be defined more clearly.","section":"Algorithm 1, line 10"},{"comment":"The notation for time-last-seen is inconsistent: the text alternates between '∆tt', '∆tci', and '∆tj ci', which makes the state construction harder to follow.","section":"Section IV-C1"},{"comment":"Table IV reports total experiment durations, which are dominated by BERT fine-tuning, so the claim that ORIS is 'near real-time' is not directly evidenced; a per-decision or sampling-only timing would be more informative.","section":"Table IV"},{"comment":"Figure 3 is referenced but not clearly described in the text; the caption should be expanded so that the confidence intervals and markers are self-explanatory.","section":"Section VI-C"}],"recommendation":"reject","confidential_remarks":"The paper has a promising high-level idea, but the main quantitative results rest on a broken oracle equation and a circular alignment between the reward and the simulated error model. Even if the Eq 2 typo is corrected, the absence of any real annotation data leaves the central claim unsubstantiated. I would reconsider a revised manuscript that fixes the oracle, includes real-label validation (or at least a sensitivity analysis), and clearly specifies the baselines for improvement claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The central claim here — that ORIS cuts human labeling errors — doesn't survive contact with the paper's own equations. Eq 2 defines the exponential-forgetting oracle as max(1, e^{αΔt+β}), which is never below 1 and therefore not a probability. With the reported α=0.6 and β=−19, the error score is 1 for every Δt, so the oracle always mislabels. That contradicts Table III, where ORIS hits 100.0 human F1 under the exponential oracle. This is likely a typo (min instead of max), but as written it invalidates the evaluation.\n\nThat said, the core idea is genuinely new. Combining a DQN-based active-learning policy with a state that includes k-averaged time-last-seen per class and a reward based on Shannon entropy of recent labels is not in the cited prior work. The authors ship code, run two datasets, two BERT models, five seeds, and compare to a reasonable set of baselines. The sigmoid oracle (Eq 1) is actually fine: it increases with Δt, which models forgetting correctly. The stress-test note's claim that Eq 1 has wrong monotonicity is mistaken.\n\nThe bigger problems are evaluation and completeness. The DQN is trained with an error-free oracle and rewarded only for inclusivity — not for correctness. The human F1 improvement is therefore a property of the simulator's memory model, not a demonstrated reduction in real human error. The abstract's percentages (38.3%, 55.7%, etc.) don't match Tables II and III. And the most relevant baseline — the authors' own earlier heuristic that directly targets slips — is omitted.\n\nIn current form I'd reject. The central claim is not supported, and the Eq 2 issue needs fixing. But this is worth sending to peer review rather than desk-rejecting: the idea is plausible, the flaws are identifiable and fixable, and a corrected evaluation with real annotators could turn it into a useful contribution. It's also a good reading-group example of circularity in human-in-the-loop evaluation.","headline":"A fresh RL active-learning idea undermined by a broken oracle equation and a partially circular evaluation.","tokens_in":14089,"tokens_out":3572,"would_cite":false,"duration_ms":30451,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims ORIS, a Deep Q-Network-based inclusive sampling policy for online active learning, reduces memory-decay labeling errors from simulated oracles and improves downstream model F1, reporting up to 44.2% human and 70.1%…","keywords":["online active learning","deep reinforcement learning","human-in-the-loop machine learning","human labeling error","memory decay","streaming analytics","emotion recognition"],"falsifier":"Run the same 500-label active learning protocol with human annotators instead of the simulated oracle on the same emotion streams, recording each annotator's error rate against the time since each class was last seen; if the error rate does not rise with recency gap, ORIS's claimed mechanism for reducing human error would be contradicted.","tokens_in":12962,"feed_emoji":"🎯","tokens_out":7728,"duration_ms":62335,"temperature":0.7,"pith_summary":"The paper sets out to show that online active learning can be made robust to human memory-decay labeling errors by treating the document-sampling decision as a reinforcement learning problem. It proposes ORIS, a Deep Q-Network-based agent whose state combines the current document embedding with how long it has been since the oracle last labeled each class, and whose reward is an Inclusivity score measuring Shannon entropy of recently sampled class labels. The central claim is that this learned sampling policy reduces labeling errors from a forgetting-prone oracle and improves the downstream ML model compared with random, uncertainty, and offline diversity sampling. Evaluated on five-class emotion recognition over Twitter and Reddit streams, ORIS reports large gains in both human labeling F1-macro and machine F1-macro across two pretrained transformer-based learners. The reason this matters is that streaming analytics systems increasingly rely on human labels, and per-class memory freshness offers a teachable, transferable way to spend a limited labeling budget.","feed_headline":"RL sampler cuts labeling errors, boosts stream ML up to 70%","feed_subtitle":"Diverse, recency-aware sampling beats random and uncertainty baselines on emotion streams.","key_machinery":"The load-bearing mechanism is a Markov Decision Process formulation of online document sampling, solved with a Deep Q-Network. The state is the concatenation of the average pre-trained word embedding of the current document and the k-averaged time-last-seen $\\Delta t_{c_i}$ for every class, so the agent sees both what is being asked and how recently the oracle has labeled each class. The reward is the Inclusivity score, the Shannon entropy of class labels among the past $m$ picked documents, passed through $\\rho e^{\\delta(\\mathrm{Inclusivity}(M)-1)}$ for picks, with a small constant reward for discards; this rewards sampling histories that touch all classes recently while penalizing one-class runs. The paper also introduces two oracle error models, a parameterized sigmoid and an exponential memory-decay function, both of which make error probability grow with time since a class was last labeled.","core_discovery":"On the paper's own terms, ORIS discovers that a reinforcement-learned sampling policy can jointly protect the human annotator and the ML model by keeping every class recently seen. The policy, trained with a Deep Q-Network against an error-free oracle and deployed with a memory-decaying oracle, selects documents whose labels are diverse enough to keep the oracle's per-class memory fresh while still giving the model a balanced training signal. The paper reports that under a budget of 500 labels, ORIS reaches 56.2-60.6% machine F1-macro and 83.6-88.4% human F1-macro on Twitter, versus 40.4-47.3% machine and 63.5-72.1% human for random sampling, and similarly large margins on Reddit, where human F1-macro reaches 98.5-100.0%.","pith_inferences":["If real annotators forget according to a different curve than the fitted sigmoid or exponential, the reward and state calibration would need to be refit; a concrete extension is to log real annotator error against time-last-seen and retrain the DQN on that fitted curve.","Because the time-last-seen values are updated from oracle labels that may themselves be wrong, a single erroneous label corrupts the memory estimate the agent acts on; an untested variant is to maintain the memory estimate from the model's own predictions as a noise-free channel.","The Inclusivity reward looks back only $m=10$ picks, so very long-range class starvation may fall outside its horizon; a testable extension is a variant with a longer memory window or an additive count-based bonus for rarely seen classes.","ORIS's cross-domain success suggests the policy learns to equalize class recency rather than dataset idiosyncrasies; this can be tested by varying class imbalance and budget while measuring whether both human and machine F1 remain stable."],"forward_implications":["ORIS can replace the sampling step of an existing online active learning pipeline with a near-real-time policy whose inference cost is comparable to random sampling, since it avoids per-sample model confidence computation and offline clustering.","Including sampling diversity as a reward protects the rare classes in imbalanced streams, which is where error-prone oracles hurt most.","The learned policy transfers from Twitter to Reddit without retraining, suggesting that keeping all classes recently seen is a general, dataset-independent strategy.","Performance gains appear early in the budget: machine improvement is visible at roughly 150 labels and human improvement at roughly 50 labels, which matters for low-budget deployments.","The method is stated to extend to other modalities such as images and videos, and to other streaming analytics tasks, because the state and reward do not depend on text-specific features."],"supporting_citations":[{"why":"Supplies the initial evidence that annotation errors follow serial-order effects in social stream labeling.","marker":"[6]"},{"why":"Provides the parameterized human-error model and heuristic mitigation baseline that ORIS extends and replaces.","marker":"[7]"},{"why":"Supplies the mistakes-versus-slips error taxonomy that motivates modeling slips as memory decay.","marker":"[8]"},{"why":"The deep Q-network training procedure ORIS adapts for its sampling policy.","marker":"[9]"},{"why":"The memory-decay literature that justifies the exponential forgetting curve used in the oracle error model.","marker":"[22]"},{"why":"Twitter emotion dataset used for DQN training and online active learning evaluation.","marker":"[26]"},{"why":"Reddit emotion dataset used for cross-domain evaluation of ORIS.","marker":"[27]"},{"why":"The BERT-based language models fine-tuned as the active learner's classifier.","marker":"[32]"},{"why":"Offline diversity sampling baseline using agglomerative clustering that ORIS is compared against.","marker":"[33]"},{"why":"Uncertainty sampling baseline using dynamic confidence thresholds that ORIS is compared against.","marker":"[35]"}],"fun_headline_variants":["RL sampling reduces labeling errors, boosts stream ML","Inclusive RL sampler beats baselines on emotion streams","Deep Q-net sampler trims human error, sharpens ML","ORIS: RL-based sampling for robust streaming analytics"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's reported gains rest on the assumption that human annotator error probability really does grow as an exponential or sigmoid function of time since a class was last labeled; if that simulated memory-decay model does not match real annotator behavior, the measured reduction in labeling errors will not carry over to practice.","fun_headline_variants_meta":{"raw":{"variants":["RL sampling reduces labeling errors, boosts stream ML","Inclusive RL sampler beats baselines on emotion streams","Deep Q-net sampler trims human error, sharpens ML","ORIS: RL-based sampling for robust streaming analytics"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000167,"raw_usage":{"total_tokens":1236,"prompt_tokens":901,"completion_tokens":335,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":271}},"tokens_in":517,"tokens_out":335,"duration_ms":3900,"temperature":1.0,"reasoning_tokens":271,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:32:56.064274+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 500-label active learning protocol with human annotators instead of the simulated oracle on the same emotion streams, recording each annotator's error rate against the time since each class was last seen; if the error rate does not rise with recency gap, ORIS's claimed mechanism for reducing human error would be contradicted.","supporting_citations":[{"cited_title":"Modeling human annotation errors to design bias-aware systems for social stream processing,","cited_arxiv_id":null,"evidence_quote":"Supplies the initial evidence that annotation errors follow serial-order effects in social stream labeling."},{"cited_title":"Modeling and mitigating human annotation errors to design efficient stream processing systems with human-in-the-loop machine learning,","cited_arxiv_id":null,"evidence_quote":"Provides the parameterized human-error model and heuristic mitigation baseline that ORIS extends and replaces."},{"cited_title":"Human error: models and management,","cited_arxiv_id":null,"evidence_quote":"Supplies the mistakes-versus-slips error taxonomy that motivates modeling slips as memory decay."},{"cited_title":"Human-level control through deep reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"The deep Q-network training procedure ORIS adapts for its sampling policy."},{"cited_title":"Reflections of the environment in memory,","cited_arxiv_id":null,"evidence_quote":"The memory-decay literature that justifies the exponential forgetting curve used in the oracle error model."},{"cited_title":"CARER: Contextualized affect representations for emotion recognition,","cited_arxiv_id":null,"evidence_quote":"Twitter emotion dataset used for DQN training and online active learning evaluation."},{"cited_title":"GoEmotions: A dataset of fine-grained emotions,","cited_arxiv_id":null,"evidence_quote":"Reddit emotion dataset used for cross-domain evaluation of ORIS."},{"cited_title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,","cited_arxiv_id":null,"evidence_quote":"The BERT-based language models fine-tuned as the active learner's classifier."},{"cited_title":"Active Learning with Clustering,","cited_arxiv_id":null,"evidence_quote":"Offline diversity sampling baseline using agglomerative clustering that ORIS is compared against."},{"cited_title":"Cost-effective active learning for deep image classifica- tion,","cited_arxiv_id":null,"evidence_quote":"Uncertainty sampling baseline using dynamic confidence thresholds that ORIS is compared against."}],"review_version":1}