{"id":"be013aa6-f140-4f87-9f84-2fbe5a1e5778","arxiv_id":"2507.22912","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":11,"one_line_summary":"The authors report state-of-the-art detection of drug, weapon, and credential sales across dark web and social platforms using a semi-supervised ensemble over fine-tuned ModernBERT embeddings plus manual document features (F1 0.935).","lead":"A two-stage pipeline combining fine-tuned ModernBERT embeddings, hand-crafted document features, and a semi-supervised ensemble with entropy-based voting aims to flag illicit marketplace posts on dark web sites, Telegram, Reddit, and Pastebin. The authors report higher accuracy and F1 than standard transformer baselines on their own corpus and on DUTA and CoDA, but release no code, data, or label mappings.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-dataset F1 gains rest on an unpublished DUTA/CoDA relabeling; without mapping rules or inter-annotator agreement, the Table IX comparisons measure the authors' label definitions rather than model superiority.","rationale":"The reader's verdict of CONDITIONAL is well-matched to the paper's actual risk profile. I agree with the reader that the strongest claim is the cross-dataset generalization reported in Section IV.G (Table IX), and that its weakest point is the unpublished relabeling of DUTA and CoDA into the paper's four categories. This is not an internal inconsistency in the model architecture; it is a validity threat to the benchmark ground truth. The paper's own text explicitly acknowledges the relabeling but provides no mapping rules, guidelines, or agreement measures, so the concern is directly supported by the manuscript. I considered whether another issue, such as the metadata source-feature shortcut or the epoch-count inconsistency (text says 15, Table IV says 10), should be the primary concern. Those are real but less load-bearing: the metadata shortcut is partially mitigated by the held-out test set in Table IX, and the epoch inconsistency affects reproducibility of fine-tuning but not the direction of the comparative claims. The relabeling issue, however, determines whether the external benchmark numbers measure the model or the authors' labeling conventions. If the relabeling is idiosyncratic, the central claim of state-of-the-art generalization collapses even if the internal test-set numbers are correct. The proposed concrete test directly settles this by quantifying inter-annotator agreement and comparing label distributions. Therefore the reader's conditional verdict should stand: the paper should be accepted only after the relabeling is made reproducible and the agreement is demonstrated, or the cross-dataset claims should be withdrawn.","tokens_in":25349,"tokens_out":2148,"duration_ms":26344,"concrete_test":"Release the full relabeling mapping for DUTA and CoDA, including the original label-to-target-category correspondence rules and any exclusion criteria. Then have two independent annotators, blind to the authors' labels, relabel a stratified random sample of 200 DUTA and 200 CoDA documents using a written label guideline derived from Section I's definitions of sale, drug, weapon, and credential. Compute Cohen's kappa between the two annotators and between each annotator and the authors' relabels. If kappa is below 0.8 or the obtained label distribution differs materially from Table VIII, the cross-dataset F1 margins in Table IX are not a reliable measure of model superiority and the generalization claim should be downgraded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of Section IV.G is that the proposed pipeline outperforms DarkBERT and other baselines on DUTA and CoDA by large margins (e.g., 4.9% F1 on DUTA, 1.5% on CoDA). This claim depends on the authors' preprocessing step: \"filtering for English-language documents and relabeling samples according to the four target categories\" (Section IV.G). DUTA and CoDA were originally annotated with their own taxonomies (e.g., CoDA categories include drugs, hacking, pornography, violence, and financial pages; DUTA includes broader categories such as \"drugs\" and \"porno\"), not with the paper's four labels (sale/no-sale and drug/weapon/credential sale). No mapping rules, label guidelines, or inter-annotator agreement statistics are provided. The mapping from an original label like \"drug\" to the paper's \"drug sale\" is not automatic: a page can discuss drugs without offering them for sale, and a marketplace page can mention weapons alongside drugs. If the relabeling decisions encode the authors' understanding of what counts as a sale, then the 4-12% F1 gaps over DarkBERT on these benchmarks reflect the similarity between the author-generated labels and the author-trained model, not intrinsic generalization. This is structurally separate from the internal test-set claim: it concerns the validity of the external ground truth against which the headline state-of-the-art assertion is measured. The concern is load-bearing because the abstract explicitly cites DUTA and CoDA as evidence of generalization, and the paper provides no way for a reader to verify that the relabeled ground truth is objective or reproducible.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage pipeline for detecting and classifying illicit marketplace posts (sale vs. no-sale; drug/weapon/credential) across deep/dark web, Telegram, Reddit, and Pastebin. Features are fine-tuned ModernBERT embeddings concatenated with manually engineered layout, pattern, and metadata features. Stage 1 is a semi-supervised ensemble (XGB, RF, SVM) with a proposed MEC/MEW entropy-based weighted voting; Stage 2 consists of three semi-supervised XGB classifiers for sale-type categorization. The evaluation uses the authors' own 1,575 labeled / 20,000 unlabeled collection, plus DUTA and CoDA as test sets. The authors report top results on their own test set (Acc 0.96489, F1 0.93467, TMCC 0.95388) and state that the full pipeline outperforms BERT, ModernBERT, ALBERT, Longformer, BigBird, and DarkBERT on all three datasets.","tokens_in":25647,"tokens_out":6607,"duration_ms":78757,"significance":"If the results hold, the paper demonstrates a practically useful engineering result: combining long-context domain-adapted embeddings with structural features and self-training can improve detection over strong dark-web baselines under limited supervision. Strengths of the manuscript include a fixed held-out test set, 30 repeated runs, a Friedman test for representation ranking, validation-based hyperparameter selection, and evaluation on external corpora. The MEC/MEW weighting is a simple, clearly motivated idea. However, as detailed in the major comments, the external-benchmark comparisons and the limited-supervision framing need substantial clarification and additional evidence before the claims as stated are fully supportable.","major_comments":[{"comment":"The cross-dataset generalization claim depends on an undocumented relabeling of DUTA and CoDA. The text says only that samples were 'filtering for English-language documents and relabeling samples according to the four target categories'; no mapping rules, label guidelines, or inter-annotator agreement statistics are provided. DUTA and CoDA have their own taxonomies (e.g., CoDA includes hacking, pornography, violence, and financial categories alongside drugs), and a label such as 'drug' does not automatically map to 'drug sale.' Because the reported F1 margins over DarkBERT are 4.9 points on DUTA and 1.5 points on CoDA, the comparison may reflect the authors' label definitions rather than model superiority. Please provide the full mapping, annotator guidelines, agreement measures, and ideally a comparison on the original labels or on a shared subset.","section":"Section IV.G, Tables VIII-IX"},{"comment":"The reported optimal number of fine-tuning epochs for ModernBERT is inconsistent: Section IV.B states the grid search identified 15 epochs, while Table IV lists the optimal value as 10. Since the fine-tuned ModernBERT embeddings are a central component and the exact configuration affects reproducibility, please correct the discrepancy and state which value produced the headline results.","section":"Section IV.B and Table IV"},{"comment":"The comparison in Table IX varies several factors at once: the proposed pipeline uses manually engineered features, a two-stage hierarchical design, semi-supervised self-training, and the fine-tuned ModernBERT backbone, while each deep baseline is used as a 'standalone multi-label classifier.' The reported gaps therefore do not isolate the contribution of the model architecture, the feature set, or the semi-supervised mechanism. To support the claim that the framework 'outperforms' these baselines, the baselines should be run with the same concatenated manual features and, where feasible, with the same hierarchical wrapper; otherwise the headline comparison conflates feature engineering with model design.","section":"Section IV.G, Table IX"},{"comment":"At 5% labeled data the model achieves F1=0.49176 and TMCC=0.51877, which are essentially at the random baseline (TMCC=0.5 corresponds to random predictions), yet the text says this result 'indicate[s] the model's ability to learn useful patterns even from minimal labeled training set.' This statement is unsupported and conflicts with the abstract's 'robustness under limited supervision.' Please either remove the 5% claim, report statistical confidence intervals relative to a random-majority baseline, or reframe the limited-supervision claim to the settings where the model is clearly above chance (e.g., 25% and above).","section":"Section IV.E, Fig. 5"}],"minor_comments":[{"comment":"In the text, the supervised ensemble using the proposed weighting is labeled 'SSE WV' in the sentence 'best-performing supervised ensemble using the same weighting strategy (SSE WV, 0.89429)'; this should be 'SE WV' to match the table and to avoid confusion with the semi-supervised ensemble.","section":"Table VI and Section IV.F"},{"comment":"The list of metadata features contains 'Deep Web Source' twice; remove the duplicate.","section":"Section III.B.2(c)"},{"comment":"The pseudo-label decision rule does not specify tie-breaking when TPP_sale equals TPP_no-sale; please add a deterministic rule.","section":"Equation (7)"},{"comment":"The headline numbers (Acc 0.96489, F1 0.93467, TMCC 0.95388) are only for the authors' own multi-source test set; please state this explicitly in the abstract so readers do not infer they hold on DUTA and CoDA as well.","section":"Abstract"},{"comment":"The fine-tuning objective (e.g., masked language modeling or classification) and the exact corpus split used for fine-tuning ModernBERT are not stated; please specify these details for reproducibility.","section":"Section III.B.1"}],"recommendation":"major_revision","confidential_remarks":"The main concern is that the external-benchmark claim is currently not falsifiable because the relabeled DUTA/CoDA ground truth is not specified. If the authors provide the mapping, label guidelines, and agreement statistics, the paper could be publishable as a systems paper with a somewhat narrower claim. The novelty relative to the authors' prior work [15] should also be clarified; the new components (ModernBERT integration, MEC/MEW weighting, and the two-stage self-training design) should be clearly differentiated in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a credible engineering paper, not a paradigm shift. What is new is the multi-source corpus (deep/dark web, Telegram, Reddit, Pastebin), the two-stage semi-supervised pipeline with ModernBERT embeddings plus manual structural features, and the MEC/MEW entropy-based weighting rule. The internal held-out evaluation is clean: a fixed 315-sample test set, 30 runs, a Friedman test, and the gains over DarkBERT and the other baselines on that test set are plausible. The semi-supervised gains are consistent across stages. Credit where due: the authors are careful about validation and report variance.\n\nThe load-bearing weakness is the external benchmark comparison. DUTA and CoDA have their own label taxonomies; the paper relabels them into its four sale categories, but gives no mapping rules, no label guidelines, and no inter-annotator agreement. A 'drug' page in CoDA is not necessarily a 'drug sale' page, and a marketplace page can mention weapons without selling them. If the relabeling encodes the authors' interpretation, the Table IX F1 margins over DarkBERT measure that interpretation, not model superiority. That concern looks structurally valid to me. It does not sink the internal result, but it does mean the cross-dataset state-of-the-art claim is not verifiable as written.\n\nOther soft spots are minor by comparison. There is an internal inconsistency in the ModernBERT epoch count (text says 15, Table IV says 10). The 5%-labeled result is near chance—F1 0.49—and while it is honestly reported, the text frames it as a positive, which is a bit generous. There is no ablation isolating the manual features or the MEC/MEW weighting, so we cannot see how much each contributes. The metadata features include source platform, which is constant in both external test sets, so on the internal test set those features could be a shortcut; the paper does not address that.\n\nWho is this for? Researchers working on dark-web and illicit-marketplace detection. It is a useful engineering contribution with reproducible potential, but only if the corpus and relabeling are released. I would send it to review, not desk-reject it. The reviewer should ask for the relabeling mapping, the corpus, and the ablations. If those come through, the paper supports its claims. If not, the external-benchmark section should be downweighted.","headline":"A credible engineering contribution whose headline cross-dataset gains rest on an unpublished relabeling that needs to be public before the state-of-the-art claim is taken at face value.","tokens_in":26325,"tokens_out":2109,"would_cite":true,"duration_ms":24570,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-stage semi-supervised ensemble that combines fine-tuned ModernBERT embeddings with structural document features beats DarkBERT and five other transformer baselines at detecting and categorising illicit marketplace posts across…","keywords":["illicit marketplace detection","dark web","deep web","semi-supervised learning","ensemble learning","transformer embeddings","ModernBERT","text classification"],"falsifier":"A concrete check: two independent annotators re-label a random sample of DUTA and CoDA documents into sale, drug, weapon, and credential using only the paper's description, and both the proposed pipeline and DarkBERT are scored against the fresh labels; if agreement is low, or if DarkBERT matches or beats the proposed model on the fresh labels, the reported generalization margin is an artifact of the relabeling.","tokens_in":25065,"feed_emoji":"🕵️","tokens_out":9390,"duration_ms":76258,"temperature":0.7,"pith_summary":"This paper tries to establish that illicit marketplace content scattered across the deep/dark web, Telegram, Reddit, and Pastebin can be detected and classified into drug, weapon, and credential sales using a two-stage pipeline that works with very few labeled examples. The pipeline pairs domain-fine-tuned ModernBERT embeddings with hand-built structural features, then feeds them into a semi-supervised ensemble of XGBoost, Random Forest, and SVM whose votes are weighted by each model's prediction confidence as measured by entropy. The authors report that this system outperforms BERT, ALBERT, Longformer, BigBird, ModernBERT, and even DarkBERT, a model pretrained on dark-web text, on their own test set and on adjusted versions of two public benchmarks, DUTA and CoDA. If the claim holds, it would show that combining long-context language models, structural cues, and self-training is a practical route to monitoring underground markets under exactly the label scarcity that real monitoring faces.","feed_headline":"Two-stage model beats DarkBERT at spotting illicit sales posts","feed_subtitle":"Fine-tuned ModernBERT plus structural features lifts F1 by 1.4–6.4 points over DarkBERT on three test sets.","key_machinery":"The load-bearing mechanism is the two-stage sequential classifier. Stage one is a semi-supervised ensemble of XGBoost, Random Forest, and SVM base learners combined by weights $w_i = (\\textit{MEW}_i/\\textit{MEC}_i) / \\sum_j (\\textit{MEW}_j/\\textit{MEC}_j)$, where MEC is the mean entropy of a learner's correct predictions and MEW the mean entropy of its wrong predictions; low MEC and high MEW make a learner's vote count more. Confident pseudo-labels, with average class probability above a threshold $\\theta$, are added to the training pool across self-training iterations. Stage two uses three self-training XGB classifiers, one per sale category. The input representation is mean-pooled embeddings from ModernBERT, a long-context transformer that processes up to 8192 tokens via alternating local and global attention with rotary positional embeddings, concatenated with manually engineered features: line-width and indentation statistics, counts and relative weights of embedded items such as Bitcoin addresses, emails, IPs, URLs, images, and credit-card numbers, plus source and date metadata.","core_discovery":"The central claim is that the complete sequential pipeline, fine-tuned ModernBERT embeddings concatenated with layout, pattern-specific item, and metadata features, followed by a self-training ensemble with entropy-based weighted voting for sale detection and three semi-supervised XGB classifiers for category, achieves accuracy of 0.96489, F1-score of 0.93467, and TMCC of 0.95388 on the authors' multi-source test set. On the DUTA, CoDA, and multi-source test sets, the model reports higher mean accuracy, F1-score, and TMCC than BERT, ALBERT, Longformer, BigBird, ModernBERT, and DarkBERT (Table IX). The paper's argued explanation is that the gain comes from the task-specific design, domain adaptation of the embedding model, structural features beyond raw text, hierarchical two-stage classification, and the use of unlabeled data, rather than from any single component such as long-context capacity alone.","pith_inferences":["The cross-dataset margins over DarkBERT, about 4.0 F1 points on DUTA and 1.4 points on CoDA, are measured against the authors' own relabeling of those benchmarks; an independent annotation study would show how much of the margin is model performance and how much is label-definition choice.","The MEC/MEW weighting rule is generic: any ensemble of probabilistic classifiers could use it to gate pseudo-labels, so the mechanism is testable in other low-annotation text domains such as fraud, extremism, or abuse detection.","Because the CoDA advantage over DarkBERT is small, the decisive components may be the domain-fine-tuned embeddings and structural features rather than the self-training loop; an ablation that removes only the self-training stage would isolate this.","The framework is evaluated as sequential one-vs-rest categories, while illicit posts often advertise multiple goods; a multi-label variant that skips the sale gate could be a natural next step."],"forward_implications":["A monitoring system could start from roughly half the labeled data and still score above 0.82 on all three reported metrics, with the remaining lift coming from unlabeled documents.","Domain fine-tuning of the embedding model contributes more than raw sequence length: fine-tuned ModernBERT ranked first among thirteen representation techniques, while pretrained ModernBERT ranked below GloVe in the Friedman ranking.","Structural signals add value beyond text embeddings, so the full pipeline, not the language model alone, is what beats every LM-only baseline.","The entropy-weighted voting improves over majority voting in the semi-supervised ensemble, with accuracy 0.95250 versus 0.94080, indicating that confidence weighting is doing work beyond simple ensembling.","The model generalizes across distributions, with reported F1 of 0.86568 on adjusted DUTA and 0.94439 on adjusted CoDA after relabeling."],"supporting_citations":[{"why":"Supplies ModernBERT, the long-context embedding backbone that the paper fine-tunes on domain data.","marker":"[14]"},{"why":"Authors' earlier work that identified the manually engineered feature set and its optimal configuration.","marker":"[15]"},{"why":"Introduces DarkBERT, the strongest baseline, pretrained on dark-web text, which the proposed model must outperform.","marker":"[31]"},{"why":"Introduces the DUTA dataset, one of the two public test sets relabeled into the paper's four categories.","marker":"[19]"},{"why":"Introduces the CoDA dataset, the other public test set relabeled for the cross-dataset comparison.","marker":"[29]"},{"why":"Provides the BERT baseline showing the 512-token limit and general-domain gap the paper argues against.","marker":"[11]"},{"why":"Shannon entropy formula, used to define MEC and MEW in the weighted voting mechanism.","marker":"[34]"},{"why":"BigBird baseline representing long-context sparse-attention transformers with no domain adaptation.","marker":"[36]"}],"fun_headline_variants":["Semi-supervised ensemble outguns DarkBERT on illicit sales","Two-stage LM pipeline beats DarkBERT on dark web sales","Fine-tuned ModernBERT + ensemble tops DarkBERT by 1-6 F1","Multi-platform illicit sales detector beats DarkBERT and BERT","Fine-tuned ModernBERT plus features lifts F1 over DarkBERT"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The cross-dataset generalization claim depends on the authors' manual relabeling of DUTA and CoDA into the four target categories, and no mapping rules, label guidelines, or inter-annotator agreement are reported; if that relabeling is idiosyncratic, the reported F1 margins measure the authors' label definitions rather than model superiority.","fun_headline_variants_meta":{"raw":{"variants":["Semi-supervised ensemble outguns DarkBERT on illicit sales","Two-stage LM pipeline beats DarkBERT on dark web sales","Fine-tuned ModernBERT + ensemble tops DarkBERT by 1-6 F1","Multi-platform illicit sales detector beats DarkBERT and BERT","Fine-tuned ModernBERT plus features lifts F1 over DarkBERT"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000543,"raw_usage":{"total_tokens":2671,"prompt_tokens":1084,"completion_tokens":1587,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":700,"completion_tokens_details":{"reasoning_tokens":1511}},"tokens_in":700,"tokens_out":1587,"duration_ms":489813,"temperature":1.0,"reasoning_tokens":1511,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:56:00.101429+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: two independent annotators re-label a random sample of DUTA and CoDA documents into sale, drug, weapon, and credential using only the paper's description, and both the proposed pipeline and DarkBERT are scored against the fresh labels; if agreement is low, or if DarkBERT matches or beats the proposed model on the fresh labels, the reported generalization margin is an artifact of the relabeling.","supporting_citations":[{"cited_title":"Yazdanjue, M","cited_arxiv_id":null,"evidence_quote":"Authors' earlier work that identified the manually engineered feature set and its optimal configuration."},{"cited_title":"DarkBERT: A Language Model for the Dark Side of the Internet","cited_arxiv_id":"2305.08596","evidence_quote":"Introduces DarkBERT, the strongest baseline, pretrained on dark-web text, which the proposed model must outperform."},{"cited_title":"Classifying illegal activities on tor network based on web textual contents,","cited_arxiv_id":null,"evidence_quote":"Introduces the DUTA dataset, one of the two public test sets relabeled into the paper's four categories."},{"cited_title":"Shedding New Light on the Language of the Dark Web","cited_arxiv_id":"2204.06885","evidence_quote":"Introduces the CoDA dataset, the other public test set relabeled for the cross-dataset comparison."},{"cited_title":"Big bird: Transformers for longer sequences,","cited_arxiv_id":null,"evidence_quote":"BigBird baseline representing long-context sparse-attention transformers with no domain adaptation."}],"review_version":1}