{"id":"7548b2ba-ff3b-4933-bd68-e061ded4315a","arxiv_id":"2501.13252","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":9,"one_line_summary":"The paper combines LDA topic models, TF-IDF expert keywords, and a four-metric Q-learning reward to select topics, then evaluates selection using the same conference papers that generated the rewards.","lead":"This paper proposes a pipeline that combines topic modeling, expert-defined aspect keywords, and Q-learning to help organizations detect emerging technology trends from text. It applies the pipeline to quantum communication papers, where the selected topics are checked against recent conference proceedings.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim of effectiveness is not established because the same conference papers (QCrypt 2023/2024) are used both to compute the RL reward and to validate topic alignment, making the reported alignment an artifact of circular evaluation.","rationale":"I identify the circular evaluation as the single most load-bearing concern. For the central claim to hold, the method must demonstrate that RL-driven topic selection identifies trends that align with expert input in a way that is not manufactured by the scoring procedure. The paper's own reward design violates this: in Section 3, Eq. 9 defines the base reward as the fraction of new documents (QCrypt 2023/2024 papers) whose cosine similarity to a topic exceeds threshold t; Eq. 8 adds an entropy bonus. This reward updates Q-values (Eq. 7) used to rank topics. Then Sections 4.1/4.2 use the same QCrypt 2023 and QCrypt 2024 documents both to 'derive rewards' and to 'confirm the topics' (Itr1-Step 11/12, Itr2-Step 11/12/15). Section 5.6 cites individual papers from those same conferences as validation. Optimizing a score and then reporting that the optimized score is high is circular; the claimed alignment with expert input is therefore not an independent finding. The reader's weakest_assumption focused on hand-set weights (lambda1-lambda4, lambda), and Table 7 does show strong sensitivity; that is a valid secondary issue. But even with fixed weights, the evaluation would remain circular. A temporal holdout test, training rewards on 2023 proceedings and validating on 2024 proceedings never used in the reward, would resolve the concern. Because the central claim is not supported, I agree with the REJECT verdict; my read does not change it.","tokens_in":32333,"tokens_out":6950,"duration_ms":70130,"concrete_test":"Hold out the QCrypt 2024 papers entirely. Run the full RL pipeline using only data through 2023 (including QCrypt 2023 for any reward computation, if desired) to select the top five topics. Then measure the cosine-similarity alignment of those selected topics with the held-out QCrypt 2024 papers, using the same threshold t=0.3 as in Eq. 9. If the held-out alignment is substantially lower than the within-sample alignment reported in Section 5.6, or no better than selecting the five topics with largest magnitude (a no-RL baseline), the claimed effectiveness is an artifact of using the validation documents in the reward. Report both numbers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is the circular evaluation of the central claim. The paper asserts that RL-selected topics 'align with expert input,' but the expert input and the validation signal are the same set of conference papers. In Section 3 (RL Process, Step 12), the modified reward (Eq. 8-9) is computed as the fraction of new documents (QCrypt 2023/2024 papers) whose cosine similarity to a topic exceeds threshold t, plus an entropy term. This reward updates the Q-values (Eq. 7), which then rank topics. In Section 4 (Itr1-Step 11 and Itr1-Step 12), the same QCrypt 2023 documents are used both to 'derive rewards' and to 'confirm the topics'; the DocCTP2 alignment matrix is the basis of both. Similarly, Iteration 2 uses QCrypt 2024 papers for both reward computation and validation (Itr2-Steps 11, 12, 15). Section 5.6 then validates selected topics by citing individual QCrypt 2023/2024 documents as evidence of alignment. Because the reward function explicitly rewards similarity to these documents, the reported 'alignment with expert input' is not an independent outcome; it is an artifact of the scoring procedure. The claim of trend identification that aligns with expert input therefore remains undemonstrated. This concern is distinct from, and more fundamental than, the sensitivity of the hand-set weights: even with fixed lambda and alpha, the evaluation is confounded.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a four-phase method that combines LDA topic modeling, aspect-based topic models built from expert-derived keywords, and Q-learning to select and refine topics for technology trend detection. The method is demonstrated on a quantum-communication corpus: a baseline LDA model (CTP1) is refined into aspect-based models (CTP2, CTP3), and RL rewards based on topic magnitude, cosine similarity, entropy, and ADNS are used to rank topics. Two iterations are run, with QCrypt 2023 and 2024 conference papers serving as the 'expert input' for computing modified rewards. The paper claims that the RL-selected topics align with expert input and emerging trends, and reports effects such as a 20% improvement in average cosine similarity after refinement.","tokens_in":32718,"tokens_out":3777,"duration_ms":35931,"significance":"If the central claim were established, the proposed framework could be a useful contribution to organizational technology scanning, combining interpretable topic models with a structured RL procedure and explicit expert involvement. The paper has some strengths worth acknowledging: it provides a detailed pseudocode (Table 1), explicit formulas for all metrics (Eqs. 1–9), a reproducible GitHub repository link, and a concrete two-iteration walkthrough with heatmaps and Q-value tables. However, the evidence presented does not currently support the abstract's claim of 'effectiveness in identifying, ranking, and tracking trends that align with expert input.' The evaluation is circular because the same conference documents are used both to compute rewards and to validate topic alignment, and the parameter sensitivity shown in Table 7 indicates that the reported topic selections depend heavily on arbitrarily chosen hyperparameters. The paper also lacks any baseline comparison or quantitative measurement protocol for its headline improvement. These issues are load-bearing for the paper's central contribution, so the manuscript requires substantial revision before it can be accepted.","major_comments":[{"comment":"The evaluation of topic alignment is circular. The modified reward in Eqs. 8–9 is computed as the average cosine similarity between each CTP2 topic and the QCrypt 2023/2024 documents, and the same documents are then used to 'confirm the topics' in Itr1-Step 11, to identify 'novel patterns' in Itr1-Step 15, and to validate selected topics in Section 5.6. Because the reward explicitly rewards similarity to these documents, high alignment between selected topics and the conference papers is an artifact of the scoring procedure rather than independent evidence of effectiveness. The paper needs a validation set that is not used in reward computation, or an external benchmark such as expert ratings or later-dated documents.","section":"§3, Step 12 (Eqs. 8–9); §4.1, Itr1-Step 11 and Itr1-Step 12; §5.6"},{"comment":"The reported conclusions depend on hand-set parameters that are not justified. The coefficients λ1=0.75, λ2=0.15, λ3=0.05, λ4=0.05 in Eq. 6 and λ=0.5 in Eq. 8 are asserted without derivation, and Table 7 shows that as α and λ increase, topics such as CTP2-T29 and CTP2-T34 appear only at higher settings while CTP2-T19 and CTP2-T32 disappear. The selected topics and Q-value rankings therefore change qualitatively with parameter choice, so the claim that the method 'identifies, ranks, and tracks trends' is not robust. The authors should either justify the parameter choices from principles or report the sensitivity of the main conclusions across a principled range.","section":"§5.1, Table 7"},{"comment":"The statement that refinements 'raised the average cosine similarity of aligned docs by 20%' is not a measurable claim as reported. There is no definition of the baseline ('aligned docs'), no sample size, no variance, and no description of how the 20% was computed. This figure appears in support of the method's effectiveness and needs a precise protocol or removal.","section":"§4.1, Itr1-Step 16"},{"comment":"Effectiveness is never compared against any baseline. The paper does not compare RL-selected topics with topics selected by LDA alone, by expert judgment alone, or by random selection, so the added value of the RL component is not demonstrated. A baseline comparison is needed to support the abstract's claim of a 'robust tool.'","section":"§4 and §5"}],"minor_comments":[{"comment":"The caption for Fig. 4 says the ADNS matrix is 'calculated using Formula 1,' but Formula 1 defines magnitude, while ADNS is defined in Formula 5; please correct the cross-reference.","section":"§4.1, Fig. 4 caption"},{"comment":"The text says 'Table 7 shows the selected topics in the two runs,' but Table 7 lists Q-values across parameter settings and does not directly show selected topics; this cross-reference should be corrected.","section":"§5.3"},{"comment":"Step 15 refers to 'CTP2_Allwords,' while later iterations use CTP3; please define the notation for all iterations to avoid confusion.","section":"Table 1, Step 15"},{"comment":"The reference to 'Bennett & Brassard (2014)' for the BB84 protocol should cite the original 1984 paper or clarify that the 2014 entry is a reprint.","section":"§1.1"}],"recommendation":"major_revision","confidential_remarks":"The circularity concern is the main obstacle to publication; it is potentially fixable by adding a genuinely independent validation set (e.g., a later conference year or expert survey not used in reward computation) and by adding baseline comparisons. The parameter sensitivity in Table 7 also needs to be addressed head-on. If the authors can provide such evidence, the paper could become a useful method demonstration; in its current form, the central claim is not supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, this paper recombines known pieces—LDA, expert keyword weighting, tabular Q-learning with a four-metric reward—into a workflow for technology scanning, and applies it to quantum communication. The specific reward composition and the two-iteration case study are new, and the pseudocode in Table 1 makes the process reproducible enough for others to reimplement. Credit where due: the authors show their parameter sensitivity in Table 7, and they list limitations (subjectivity, small data, scalability) plainly.\n\nThe soft spot is the one the stress-test flags, and it is load-bearing. The reward in Eq. 8-9 is computed from cosine similarity between CTP2 topics and QCrypt 2023/2024 papers; then the same papers are used, in Section 5.6 and iteration steps 11/12/15, as evidence that the selected topics align with expert input and emerging trends. That is circular: the 'detected trends' are partly manufactured by the scoring procedure. An independent validation set or a hold-out would fix it, but none is provided. The 20% improvement claim has no measurement protocol, baseline, or error bars; it's a single number with no provenance. The hand-set weights (λ1=0.75, etc.) and λ=0.5 are arbitrary, and Table 7 shows the topic rankings flip as α and λ change, so the reported selected topics depend on constants the authors chose without justification.\n\nI don't think this is fatal to the idea—the workflow is plausible and the authors are transparent—but this manuscript does not demonstrate effectiveness. No code or data is verified (the GitHub link is in the paper but we couldn't access it), and no baseline comparison (e.g., LDA-only or tf-idf ranking) exists.\n\nWho is this for? Someone building a competitive-intelligence pipeline who wants a concrete recipe. A serious referee could ask for hold-out validation, baselines, and code; the paper deserves that chance despite the current flaws. I'd send it to review but expect major revision.","headline":"A plausible workflow that recombines known components, but the evaluation is circular and the parameter sensitivity is underacknowledged; worth a revision, not a desk reject.","tokens_in":33174,"tokens_out":1727,"would_cite":false,"duration_ms":21500,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Q-learning agent can guide topic selection so the chosen technology trends align with expert judgment, demonstrated on quantum communication research.","keywords":["topic modeling","reinforcement learning","expert knowledge","technology landscape","quantum communication","Q-learning","trend detection","organizational learning"],"falsifier":"Re-run the same 1,048-document corpus and the QCrypt 2023/2024 validation papers through the pipeline with all four reward weights set equal and no entropy bonus; if the top-Q topics no longer center on security protocols, quantum key distribution, and entanglement, the reported alignment with expert input is an artifact of the chosen constants rather than evidence for the RL process.","tokens_in":32176,"feed_emoji":"📈","tokens_out":10262,"duration_ms":95603,"temperature":0.7,"pith_summary":"The paper is trying to establish that organizations can keep their technology radar current by closing a loop: expert-curated keywords reshape an LDA topic model, and a Q-learning agent decides which topics deserve further refinement by scoring them on a weighted reward of magnitude, similarity, entropy, and distribution shift. Applied to quantum communication publications up to 2022, with QCrypt 2023 and 2024 papers as validation documents, the method yields selected topics centered on security protocols, quantum key distribution, and entanglement. The authors report that these selections match what domain experts would flag and that Q-values evolve to prioritize them across iterations. The claim would matter if true because it would give firms a scalable way to convert unstructured literature into strategic priorities without rebuilding a topic model from scratch each time the field moves.","feed_headline":"Reinforcement learning picks the tech trends experts care about","feed_subtitle":"Topic modeling, expert keywords, and Q-learning track quantum trends across conference years.","key_machinery":"The load-bearing object is the Q-learning update $Q_{t+1}(s,a) = (1-\\alpha)Q_t(s,a) + \\alpha[R_t(s,a) + \\gamma \\max_{a'} Q_t(s',a')]$, paired with the approximate reward $R(s,a)=\\lambda_1\\cdot\\mathrm{Magnitude}+\\lambda_2\\cdot\\mathrm{Similarity}+\\lambda_3\\cdot\\mathrm{Entropy}+\\lambda_4\\cdot\\mathrm{ADNS}$ and a modified reward that adds $\\lambda\\cdot\\mathrm{Entropy}(s,a)$ to a base reward computed from cosine similarity between topics and newly arriving conference papers. The metrics are estimated from aspect-based topic models: LDA topics reweighted by TF-IDF expert keywords, with CTP1 and CTP2 serving as successive states. This machinery carries the argument by turning \"which topics deserve attention\" into a ranking by Q-values, and the paper's evidence is the stability and shift of those Q-values across two iterations of the loop.","core_discovery":"On the paper's own terms, the central discovery is that topic selection can be treated as a Markov decision process: the state is the pair of topic models (CTP1, the baseline, and CTP2, the aspect-refined model), an action is selecting and refining topics, and the reward is a weighted combination of four metrics—topic magnitude, cosine similarity, entropy change, and absolute difference in normalized sums—augmented by an entropy bonus when validating against new documents. The paper reports that the top-Q topics in both iterations, such as T19, T32, and T39, are precisely the security-protocol and QKD-focused topics that the QCrypt 2023 and 2024 papers emphasize, and that the updated Q-values rise for these selections while the keyword weights shift from classical cryptography terms toward quantum communication terms. The conclusion the authors draw is that RL-driven refinement lets the topic model track technology change rather than merely describe a static corpus.","pith_inferences":["Editorial extension: The paper treats the QCrypt proceedings as both expert input and validation; a stricter test would separate the two, using conference papers only after the selection step to see whether the RL signal, rather than shared vocabulary, drives the observed alignment.","Editorial extension: Since the parameter sweep shows topic rankings changing with the reward and learning settings, the practical claim of effectiveness is conditional on a defensible way to set those constants; without that, the method is closer to interactive scoring than to learning.","Editorial extension: The modified reward design is attributed to Devidze et al. (2021), but that work has no entry in the reference list, so the provenance of the reward adjustment cannot be checked from the bibliography as printed.","Editorial extension: A natural test would compare Q-learning selections against simple baselines, such as topics with the largest entropy increase or largest magnitude change, to measure whether the RL machinery adds predictive value over its own reward components."],"forward_implications":["If the central claim holds, an organization can keep a topic model current by running new reports and conference papers through the same reward loop instead of rebuilding the model or waiting for manual expert review of every cluster.","The two-iteration demonstration implies that the same pipeline can rank topics for a firm's exploration budget: high-Q topics like QKD and security protocols receive exploration, while stable topics receive exploitation.","Because the paper shows Q-values and keyword weights shifting from classical cryptography toward quantum communication when new documents are added, the method is positioned as an early signal of technology transitions rather than a post hoc description.","The paper argues that tuning the learning rate, discount factor, and reward weight shifts which topics are prioritized, making the framework adaptable to different strategic goals without changing the underlying topic model."],"supporting_citations":[{"why":"Supplies LDA, the baseline topic model that CTP1 is built from.","marker":"(Blei et al., 2003)"},{"why":"Provides the Q-learning and exploration-exploitation framing that shapes the RL loop.","marker":"(Sutton & Barto, 2018)"},{"why":"Gives the Q-learning update rule used to compute and revise topic Q-values.","marker":"(Watkins & Dayan, 1992)"},{"why":"Companion paper detailing data collection and the aspect-keyword weighting steps this paper reuses.","marker":"(Nazari & Weiss, 2025)"},{"why":"In-text source for the explicable reward design behind the modified reward; no matching entry appears in the reference list.","marker":"(Devidze et al., 2021)"},{"why":"Grounds the quantum communication application through the BB84 protocol, informing search keywords and domain context.","marker":"(Bennett & Brassard, 2014)"},{"why":"One of the seed references whose extracted keywords define the Phase 1 corpus of quantum communication documents.","marker":"(Cavaliere et al., 2020)"}],"fun_headline_variants":["RL sharpens topic selection for tech trend tracking","Quantum trend detection via reinforcement learning","AI-guided topic models follow expert priorities","Reinforcement learning pinpoints key tech shifts","Smart topics: RL reranks emerging tech signals"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's output depends on reward weights the authors chose by hand, and the paper's own experiments show that different weights produce different topics, so the claimed alignment with expert trends rests on those unstated choices.","fun_headline_variants_meta":{"raw":{"variants":["RL sharpens topic selection for tech trend tracking","Quantum trend detection via reinforcement learning","AI-guided topic models follow expert priorities","Reinforcement learning pinpoints key tech shifts","Smart topics: RL reranks emerging tech signals"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000155,"raw_usage":{"total_tokens":1224,"prompt_tokens":961,"completion_tokens":263,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":197}},"tokens_in":577,"tokens_out":263,"duration_ms":4143,"temperature":1.0,"reasoning_tokens":197,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:19:02.192676+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same 1,048-document corpus and the QCrypt 2023/2024 validation papers through the pipeline with all four reward weights set equal and no entropy bonus; if the top-Q topics no longer center on security protocols, quantum key distribution, and entanglement, the reported alignment with expert input is an artifact of the chosen constants rather than evidence for the RL process.","supporting_citations":[],"review_version":1}