{"id":"6fc66d76-f6db-4276-88a0-ae12f1e5ad38","arxiv_id":"2502.08496","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An expert-guided topic modeling framework that reweights documents by aspect keywords is demonstrated on quantum communication, but its adaptability claim is weakened by circular evaluation.","lead":"This paper introduces a framework that adds expert-chosen keywords into topic modeling, so rare but important terms can influence how documents are grouped. The authors test it on quantum communication papers and report better alignment with expert themes, though the validation has weaknesses.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed expert-knowledge benefit may be a tautology of the exact-stem reassignment rule: Eq. 2 matches only identical stems, and no random or non-expert baseline is run, so the reported gains do not yet show that expert weighting causes the alignment improvements.","rationale":"The reader's weakest assumption is that exact stem overlap in Eq. 2 does not measure semantic alignment, and Section 7.5 supports that concern. My stress-test goes one step further: even if exact overlap were accepted as a similarity measure, the paper gives no control showing that expert selection of the keyword lists matters. Because sim() is a delta function on identical stems, the aspect-topic scores, document reassignment, and year-over-year shifts are all direct consequences of which stems appear in the aspect lists. A random-keyword control would distinguish 'expert knowledge improves alignment' from 'any keyword reweighting reshuffles clusters.' The reported quantitative claims (intra-cluster similarity 0.61 to 0.74, reassignment 28 percent to 4.2 percent) appear only in prose in Section 6 without derivation or error bars, and 'topic coherence' is claimed but never measured for the refined models. The 2023-2024 adaptability result is also vulnerable because the aspect lists and test corpora come from the same years and overlapping conference sources, so the apparent shift could be an input artifact. These issues do not prove the method is wrong; they show the current evidence does not support the central claim. The reviewer's REJECT remains appropriate, so I mark the verdict unchanged, with the concrete permutation control as the decisive check for a revision.","tokens_in":24824,"tokens_out":5594,"duration_ms":58398,"concrete_test":"Run a permutation control on the same 1,048-document corpus: generate 100 random aspect keyword sets by shuffling the expert top-50 lists or sampling 50 stems with equal frequency from the corpus vocabulary, then execute Phases 3 and 4 exactly as in Appendix 6. Compare intra-cluster similarity, document reassignment rate, and QCrypt 2023 versus 2024 alignment against the expert-aspect results. If random aspects produce comparable gains, the Section 6 improvements are an artifact of exact-stem cosine reassignment rather than expert knowledge. Also report UMass or NPMI coherence for the initial LDA topics and the final aspect-refined topics, since the paper claims improved topic coherence but reports no coherence values.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.6, Eq. 2 defines aspect-topic relevance with sim(a_ik, t_jl) equal to 1 only when the aspect stem and the topic stem are identical. Sections 4.6 and 4.9 then assign each document to the aspect whose TF-IDF keyword vector has the highest cosine similarity to the document. With that construction, any keyword set that overlaps document vocabulary will mechanically increase within-cluster similarity for documents containing those stems, and changing the keyword lists from year to year will produce a corresponding temporal shift. The evaluation in Sections 5 and 6 reports improved alignment, intra-cluster similarity increasing from 0.61 to 0.74, and a reduced reassignment rate from 28 percent to 4.2 percent, but gives no baseline, no error bars, and no non-expert control. The aspect definitions and the QCrypt test corpus are drawn from the same years and overlapping quantum-conference sources, so the 2023 to 2024 shift may reflect the changed keyword lists rather than model adaptation. Section 7.5 itself concedes that cosine similarity may miss deeper semantic relationships. Thus the central claim that expert weighting, rather than the reassignment rule itself, boosts relevance and clarity is not yet established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an Aspect-Weighted Topic Modeling framework that integrates expert-curated aspect keywords into an LDA-based pipeline. The framework proceeds through four phases: baseline topic modeling with LDA and hierarchical clustering, construction of weighted aspect vectors from expert conference sources via TF-IDF, document-to-aspect alignment using cosine similarity with exact stem matching, and iterative refinement until document reassignment stabilizes. The method is applied to a quantum communication corpus of 1,048 documents, with QCrypt 2023 and 2024 papers used as temporal test sets. The authors report improved intra-cluster similarity, reduced reassignment rates, better alignment with expert themes, and the ability to detect a shift from theoretical to implementation-focused cryptography topics. The paper positions the contribution as a transparent, modular, and adaptable alternative to static unsupervised topic models.","tokens_in":25251,"tokens_out":3043,"duration_ms":32229,"significance":"If the claimed benefits were convincingly established, the framework would offer a practical way to inject expert knowledge into topic modeling without full retraining, with potential value for technology monitoring and weak-signal detection in fast-moving domains. The manuscript has several strengths: the method is described in detail with formal notation, pseudocode, and a public repository; the design-science framing provides traceability; and the authors clearly acknowledge limitations in Section 7.5. However, the significance is currently undercut by the evaluation design: the reported improvements lack a non-expert baseline, the temporal adaptability evidence is circular because the aspect definitions and test papers share the same years, and the claimed coherence gains are not measured with any coherence metric. These issues directly affect the central claim that expert weighting, rather than the reassignment rule itself, causes the observed improvements.","major_comments":[{"comment":"The similarity measure sim(a_ik, t_jl) in Eq. (2) is 1 only when the aspect stem and topic stem are identical, and Eq. (4) then uses cosine similarity over the shared vocabulary. With this construction, any keyword set that overlaps document vocabulary will mechanically increase within-cluster similarity for documents containing those stems. The evaluation in Section 6 reports intra-cluster similarity rising from 0.61 to 0.74 and reassignment falling from 28% to 4.2%, but it provides no baseline with random keywords, non-expert keywords, or a simple TF-IDF-only keyword set. Therefore the central claim that expert-informed weighting, rather than the exact-stem reassignment rule itself, drives the improvement is not established.","section":"Section 4.6, Eq. (2); Section 6"},{"comment":"The temporal adaptability evaluation is circular. Section 4.9 states that Aspect 1 was defined from the 2023 Quantum.Tech report and Aspect 2 from the 2024 report, while Section 5 tests the framework on QCrypt 2023 and QCrypt 2024 papers. The model aligns 2023 papers with Aspect 1 and 2024 papers with Aspect 2, which is essentially fitting the output to the input year labels. The observed 'shift from theoretical foundations to implementation challenges' therefore reflects the changed keyword lists, not model adaptation to evolving domain discourse. To support the adaptability claim, the authors need a held-out temporal split, such as defining aspects on years strictly before the test years, or a control in which aspect keywords are shuffled across years.","section":"Section 4.9; Section 5; Section 7.1"},{"comment":"Although the paper repeatedly claims improved topic coherence and interpretability, no coherence metric is reported for the refined models. The C-V scores mentioned in Section 4.4 are used only to select the number of primary topics in the baseline LDA; they are not computed for the aspect-weighted topics. The evaluation evidence consists of keyword tables, heatmaps, and the intra-cluster similarity/reassignment numbers, but these are reported without error bars, confidence intervals, or statistical tests. Without a formal coherence measure and a variance estimate, the coherence and stability claims are not quantitatively supported.","section":"Section 6; Section 4.4"},{"comment":"The authors acknowledge in Section 7.5 that cosine similarity 'might miss deeper semantic relationships' and that topic selection was manual, yet Section 8 concludes that aspect weighting 'is a simple yet effective way to boost the relevance and clarity of topic modeling outputs.' The limitation statement does not address the more fundamental issue that the evaluation cannot distinguish expert-informed weighting from the mechanical effect of stem-overlap reassignment. A revision should add a non-expert control condition and a proper semantic similarity measure, or the conclusions should be substantially softened to describe only the exact-stem re-ranking behavior.","section":"Section 7.5; Section 8"}],"minor_comments":[{"comment":"Figure 5 is referenced both as 'Coherence Scores for Various Numbers of Primary Topics' and later in Section 4.4 as 'Silhouette Scores for Various Numbers of SubTopics'; the figure numbering and captions appear inconsistent and should be corrected.","section":"Section 4.4, Figures 5 and 6"},{"comment":"The caption of Figure 3 and the component list in the text use slightly different names for the same components (e.g., 'Document Realignment via Semantic Scoring' versus 'Supervised Clustering and Refinement'), which may confuse readers.","section":"Section 4.2, Figure 3"},{"comment":"The availability statement gives the repository https://github.com/alinazari1/FineTuning/blob/main/, while Section 4.6 references a different URL (https://github.com/AspectTopicModels). Please verify and unify the repository link, and ensure the data referenced as an accompanying .xlsx file is actually accessible.","section":"Section 9 (Availability of data and materials)"},{"comment":"There are numerous typos and grammatical issues, such as 'these stands together emphasize' in Section 2, 'it is making it well-suited' in Section 7.3, and 'the authors weighted the documents more with cryptography keywords' in Section 4.6. A thorough language edit is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable design-science artifact with a clearly described pipeline and some reproducible elements, but the evaluation is not yet at the standard for the claims made. The circular temporal evaluation and the lack of a non-expert baseline are the two load-bearing issues; both are addressable within the manuscript's scope through additional experiments. I would not recommend rejection because the core mechanism is implemented as described and the framework could be validated with a proper control. However, the revision must include a non-expert or random-keyword baseline, a coherence metric computed for the refined topics, and a non-circular temporal evaluation, or the adaptability and effectiveness claims should be removed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a transparent, reproducible implementation of a guided topic-modeling idea, but the evaluation is too weak to support the headline claims. The exact-stem matching and year-matched aspects make the 'expert guidance' benefit hard to separate from the reassignment mechanics.\n\nWhat's actually new: not the components—seeded LDA, feature reweighting, semi-supervised clustering are all known—but the specific pipeline of building TF-IDF aspect vectors from conference materials, reassigning documents by cosine similarity, and iterating until convergence, applied to quantum communication. The paper ships pseudocode, formal notation, and a GitHub repo, and the authors are unusually candid about limitations (manual topic selection, subjectivity, cosine similarity missing semantics). That counts for something.\n\nWhere it's soft: the main evaluation is circular in a real sense. Aspect 1 comes from 2023 sources and Aspect 2 from 2024, then QCrypt 2023/2024 papers are used to show temporal adaptation; the model is effectively fitting to year labels. There is no coherence metric for the claimed 'improved topic coherence,' no error bars, and no non-expert baseline. Since Eq. 2 matches only identical stems, any keyword set overlapping document vocabulary will mechanically increase within-cluster similarity; without a control, the 'expert' part is not established. These are not minor quibbles—they go to the central claim. That said, the method is implemented as described and the limitations section already concedes the semantic-matching concern.\n\nWho it's for: practitioners in tech foresight or library science who want a simple, interpretable way to inject expert priorities into LDA and can live with heuristics. Methodologists won't learn much new here.\n\nMy recommendation: I'd send it to a serious referee rather than desk-reject. The flaws are concrete but fixable—a proper baseline (e.g., random keywords), a real coherence metric, and pre-registered aspects would make the evaluation honest. The paper deserves a chance to be revised, but I'd not accept it as is.","headline":"A transparent, reproducible guided-topic-modeling pipeline whose evaluation does not yet separate expert input from mechanical keyword overlap; fixable, but not acceptable as is.","tokens_in":25628,"tokens_out":2259,"would_cite":false,"duration_ms":24583,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Expert-informed aspect weighting sharpens LDA topic models by overlaying curated keyword sets onto an unsupervised baseline.","keywords":["Topic modeling","Expert-informed keywords","Weighted aspect keywords","Adaptive learning","Document relevance","Supervised document clustering","Quantum communication","Latent Dirichlet Allocation"],"falsifier":"Take an aspect concept that an expert describes only with words absent from the topic's top-100 stems (e.g., 'authenticated' instead of 'secur'). Under Eq. (2) the relevance score is exactly zero; if a human rater judges the document to be thematically relevant to the aspect, the central claim that the scoring reflects expert alignment would be refuted. A concrete check: replace one aspect's keyword list with synonyms sharing no stems and observe whether document-to-topic reassignments stay stable.","tokens_in":24617,"feed_emoji":"🔍","tokens_out":8964,"duration_ms":78461,"temperature":0.7,"pith_summary":"Standard topic models like Latent Dirichlet Allocation (LDA) are unsupervised and static: they reflect dominant word frequencies, not what domain experts consider important. This paper proposes an 'aspect-weighted' overlay: the practitioner defines a few aspects (e.g., Cryptography, QKD), represents each by its top 50 TF-IDF-weighted keywords from expert sources such as conference proceedings, scores every document by cosine similarity to these aspect vectors, and reassigns documents to the best-matching aspect-topic. Iterating this reassignment until few documents move is claimed to increase topic coherence, raise the visibility of rare strategic terms, and align topics with expert priorities. Applied to a quantum-communication corpus, the method reclassified documents into more thematically coherent clusters and tracked a shift in QCrypt conference papers from foundational theory toward implementation and classical–quantum interoperability. The paper's core claim is that expert-informed aspect weighting is a simple yet effective way to boost the relevance and clarity of topic modeling outputs, and that applying it iteratively helps align the model with shifting domain signals.","feed_headline":"Expert keyword weights sharpen topic models without retraining","feed_subtitle":"On quantum cryptography papers, the method tracks a field's shift from theory to implementation.","key_machinery":"The machinery is the 'aspect-topic model': an LDA topic model overlaid with expert-defined aspect vectors. Each aspect $A_i$ is a set of the top 50 TF-IDF-weighted keywords from an expert corpus (e.g., conference agendas), and each topic $T_j$ is its top 100 LDA terms with probabilities. Relevance between aspect $i$ and topic $j$ is computed by the weighted stem-overlap score in Eq. (2), which is nonzero only when an aspect keyword and a topic term share the same stem; documents are then aligned to aspect-topics by cosine similarity (Eq. 4), and the aspect centroids are updated iteratively until document reassignment stabilizes. This overlay is what lets expert knowledge steer topic boundaries without re-training the base LDA model.","core_discovery":"On the paper's own terms, the central discovery is that an LDA topic model can be fine-tuned without retraining by injecting expert-defined aspect vectors. Each aspect is a weighted keyword list, built by running TF-IDF on text from authoritative sources (here, Quantum.Tech conference material), and each topic's word distribution is compared with those aspect vectors using a weighted stem-overlap score $R_{ij} = \\sum_{k,l} w_{ik} v_{jl}\\, \\mathrm{sim}(a_{ik}, t_{jl})$, with similarity 1 only when stems match. Documents are then re-assigned to the aspect-topic with the highest cosine similarity, and the process loops until document reassignment nearly stops. In the quantum cryptography case study, the paper reports that the aspect-weighted model increased intra-cluster similarity for topic T19 from 0.61 to 0.74, cut the reassignment rate from 28% to 4.2%, and produced relevance scores up to 0.506 between the Cryptography aspect and T19. Testing on QCrypt 2023 and 2024 papers, it finds the model adapts to a discourse shift from foundational theory to implementation. The conclusion is that expert-informed aspect weighting is a simple yet effective way to boost the relevance and clarity of topic modeling outputs, and that iterative application helps align the model with shifting domain signals.","pith_inferences":["The success of the framework likely depends on the semantic coverage of the expert keyword lists; if the expert source is too small, top-50 TF-IDF terms may miss important concepts—an extension would be to mine keywords from multiple sources or use synonym expansion.","The proposed convergence criterion (few documents moving between clusters) could be made a formal fixed-point condition, which would make the stopping rule more principled than a heuristic threshold.","Beyond quantum communication, the same overlay should be testable in other specialist domains with an authoritative conference corpus (e.g., AI safety, synthetic biology), where weak signals and expert priority shifts matter."],"forward_implications":["A single overlay can update topic models with new expert knowledge: adding an aspect from a later conference re-weights topic–document relationships without retraining the base LDA model.","Rare but strategically important terms become more visible: weighting low-frequency expert keywords raises their influence on document clustering, which is useful for tracking post-quantum cryptography and similar niches.","The framework yields cleaner, more interpretable clusters: in the case study, documents with weak cryptographic relevance were re-assigned away from cryptography topics, shifting weights such as Doc9's T19 weight from 0.255 to 0.676.","Topic evolution can be tracked: the two-iteration comparison on QCrypt 2023/2024 papers shows the model moving from foundational protocols to classical–quantum interoperability.","The method stays transparent and auditable: each step—keyword lists, weights, relevance scores, and heatmaps—can be inspected, unlike black-box summarizers."],"supporting_citations":[{"why":"Supplies the base Latent Dirichlet Allocation model that the aspect-weighting framework overlays.","marker":"[1]"},{"why":"Provides the TF-IDF term-weighting approach used to construct aspect keyword vectors.","marker":"[16]"},{"why":"Introduces lexical priors/seed words into topic models, the conceptual anchor for expert-guided refinement.","marker":"[19]"},{"why":"Gives the cosine-similarity and relevance-scoring background for document–aspect alignment.","marker":"[36]"},{"why":"Supplies the seed-document clustering algorithm used for supervised document reassignment.","marker":"[37]"},{"why":"Justifies supervised clustering as a technique for improving weak-signal detection.","marker":"[38]"},{"why":"Frames the study as a design-science research artifact, shaping the evaluation and validation approach.","marker":"[31]"}],"fun_headline_variants":["Expert aspect weights boost topic models without retraining","Weighted keywords sharpen topic alignment in quantum papers","Fine-tune topic models with expert keyword weights","Aspect weighting lifts topic clarity as fields shift","Expert-informed weighting adapts topic models to research shifts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that exact stem overlap between the top-50 TF-IDF aspect keywords and the top-100 LDA topic terms is a good enough measure of semantic alignment: if real synonyms or paraphrases are missed, the reassignment will not actually reflect expert aspects.","fun_headline_variants_meta":{"raw":{"variants":["Expert aspect weights boost topic models without retraining","Weighted keywords sharpen topic alignment in quantum papers","Fine-tune topic models with expert keyword weights","Aspect weighting lifts topic clarity as fields shift","Expert-informed weighting adapts topic models to research shifts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000179,"raw_usage":{"total_tokens":1367,"prompt_tokens":1077,"completion_tokens":290,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":693,"completion_tokens_details":{"reasoning_tokens":218}},"tokens_in":693,"tokens_out":290,"duration_ms":3905,"temperature":1.0,"reasoning_tokens":218,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T04:52:03.072468+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take an aspect concept that an expert describes only with words absent from the topic's top-100 stems (e.g., 'authenticated' instead of 'secur'). Under Eq. (2) the relevance score is exactly zero; if a human rater judges the document to be thematically relevant to the aspect, the central claim that the scoring reflects expert alignment would be refuted. A concrete check: replace one aspect's keyword list with synonyms sharing no stems and observe whether document-to-topic reassignments stay stable.","supporting_citations":[{"cited_title":"Latent dirichlet allocation","cited_arxiv_id":null,"evidence_quote":"Supplies the base Latent Dirichlet Allocation model that the aspect-weighting framework overlays."},{"cited_title":"Term-weighting approaches in automatic text retrieval","cited_arxiv_id":null,"evidence_quote":"Provides the TF-IDF term-weighting approach used to construct aspect keyword vectors."},{"cited_title":"Incorporating lexical priors into topic models","cited_arxiv_id":null,"evidence_quote":"Introduces lexical priors/seed words into topic models, the conceptual anchor for expert-guided refinement."},{"cited_title":"Active Learning Literature Survey","cited_arxiv_id":null,"evidence_quote":"Supplies the seed-document clustering algorithm used for supervised document reassignment."},{"cited_title":"Supervised clustering-algorithms and benefits","cited_arxiv_id":null,"evidence_quote":"Justifies supervised clustering as a technique for improving weak-signal detection."},{"cited_title":"Anticipating Future Innovation Pathways Through Large Data Analysis","cited_arxiv_id":null,"evidence_quote":"Frames the study as a design-science research artifact, shaping the evaluation and validation approach."}],"review_version":1}