{"id":"5079ae6a-6ff9-4628-879c-95d8a7d36e00","arxiv_id":"2608.12269","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A Word2Vec-GMM-Random Forest pipeline detects accusatory procurement comments in Ecuador's SOCE data with 0.84 precision and 0.91 recall, but those metrics are conditional on a label-selected cluster filter.","lead":"This paper builds a three-stage NLP pipeline that clusters comments from Ecuador's public procurement system and flags accusatory ones, reporting high precision and recall despite severe class imbalance. It matters because lightweight domain-adapted tools could help audit corruption in settings without large AI infrastructure.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported recall 0.91 is measured only inside the cluster-filtered subset (962 rows), while the deployed pipeline's cluster filter already discards 21 of 143 labeled accusations; end-to-end recall is capped near 0.85 and actually about 0.78.","rationale":"Read in good faith, the paper is in many ways transparent: Table VIII labels the scenario 'Cluster Filter' and gives rows evaluated (962) and total accusatory phrases (122). The methodology sections explain clustering, keyword identification, and classification. The problem is the inference drawn from those numbers. The abstract, introduction contributions, and conclusions repeatedly state that the pipeline 'achieves high precision and recall' (P=0.84, R=0.91, F1=0.87) under severe class imbalance. Those numbers are the recall of the classifier conditional on cluster membership, not the recall of the cascaded detector that a user would deploy.\n\nThe arithmetic is internal to the paper. In Table IV, the selected Word2Vec+GMM cluster has MNAPK=122 out of TNAP=143, a cluster-level recall of 85.31%. Section II-F applies the cluster filter before classification, so the 21 accusations outside this cluster are invisible. Therefore end-to-end recall is 0.853 x 0.91, about 0.776. If precision remains 0.84, the end-to-end F1 is approximately 0.81, not 0.87. The manual validation on unlabeled data (Section III-E) addresses precision only and does not measure recall. This is not a quibble about wording: the paper's stated contribution is a cascaded pipeline for detecting accusatory comments, and deployment decisions would be based on the pipeline's operating point. A deployed system that reports recall 0.91 but actually finds only about 78 percent of accusations is materially different.\n\nThe reader's weakest assumption concerns whether the keyword criterion selects the same enriched cluster in unlabeled data as in labeled data. That is a real and related risk. My concern is more basic: even granting the transfer, the headline recall is not end-to-end. I therefore partially agree with the reader; the same 'conditional on label-selected cluster filter' observation leads to a different precise failure mode.\n\nThe concrete test is straightforward and requires no new data or code access, though code release would help: recompute metrics on all 4,841 supervised rows with outside-cluster samples hard-labeled negative. The result should settle the issue. Because the fix is to report end-to-end metrics and scope the pipeline-level claim accordingly, the CONDITIONAL verdict is appropriate; nothing here requires rejection of the underlying classifier result once scoped correctly.","tokens_in":17025,"tokens_out":7800,"duration_ms":70640,"concrete_test":"Re-run the full pipeline on the 4,841-sample supervised set with the trained RF+SMOTE classifier and a hard cluster gate: assign every sample outside the keyword-identified GMM cluster to the negative class, classify only the 962 in-cluster samples, and compute positive-class precision/recall/F1 over all 4,841 rows (or report the confusion matrix restricted to the 143 accusations). If end-to-end recall is not approximately 0.78, the discrepancy should be explained by inspecting cluster assignments; if the authors intend Table VIII as classifier-only metrics, state that explicitly and report end-to-end numbers alongside. Repeat with 10 different GMM seeds to check whether the selected cluster and the 122/143 split are stable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-D (Table VIII) reports P=0.84, R=0.91, F1=0.87 for the positive class under 'Cluster Filter', i.e., on the 962 rows inside the keyword-selected cluster. That is a conditional classifier metric, not a pipeline metric. Table IV shows the Word2Vec+GMM cluster contains 122 of the 143 accusatory phrases (85.31% of TNAP), so the cluster filter alone has recall 0.853; comments outside this cluster are never scored in deployment (Section II-F). The best possible full-pipeline recall on the supervised set is therefore 122/143, and with the classifier's 0.91 recall on those 122, expected end-to-end recall is 0.853 x 0.91, approximately 0.78, giving F1 of roughly 0.81 rather than 0.87 if precision is unchanged. The abstract and contribution bullet present 0.84/0.91 as the pipeline's result without this caveat. This is independent of whether the keyword heuristic transfers to the unlabeled corpus: even a perfect cluster selector caps recall below the headline value. Table IX reports 1,892 candidates on unlabeled data, but without ground truth it cannot supply end-to-end recall.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a cascaded NLP pipeline for detecting accusatory or whistleblowing-style comments in Ecuador's public procurement system (SOCE). The pipeline embeds procurement comments with domain-trained Word2Vec, clusters them with a Gaussian Mixture Model, identifies the most accusatory cluster using a keyword-based heuristic, and then trains a Random Forest classifier (with SMOTE) on the cluster-filtered labeled subset. The authors compare Word2Vec against LLaMA and RoBERTa embeddings, report that Word2Vec+GMM gives the best cluster enrichment, and claim a precision of 0.84 and recall of 0.91 for the accusatory class. They also report a manual validation on unlabeled predictions, an analysis of computational efficiency, and a qualitative cross-country data-availability review for Latin America.","tokens_in":17315,"tokens_out":2907,"duration_ms":26185,"significance":"If the reported performance held end-to-end, this would be a practically useful, lightweight contribution to a real governance problem: detecting irregularity signals in public procurement with modest computational resources. The paper has concrete strengths: it ships reproducible code, it includes a manual validation of unlabeled predictions that partially addresses the precision question, and it provides a coherent comparison of embedding strategies with a clear discussion of anisotropy (Table V). The significance is, however, substantially tempered by the fact that the headline numbers are measured only on the cluster-filtered subset, not on the full labeled dataset, so the current framing overstates the pipeline's recall. The central idea is defensible, but the reporting and validation design need revision.","major_comments":[{"comment":"The reported precision of 0.84 and recall of 0.91 for Random Forest+SMOTE are measured only on the 962 rows inside the keyword-selected cluster, not on the full 4,841-sample labeled dataset. Since the cluster filter retains only 122 of 143 accusatory phrases (Table IV, MNAPK=122, 85.31% of TNAP), the deployed pipeline's recall on the full labeled set is capped at 122/143, and with the classifier's 0.91 recall on the filtered set, the expected end-to-end recall is approximately 0.78, with an F1 of roughly 0.81 if precision is unchanged. The abstract and the contribution bullet presenting 'precision of 0.84 and recall of 0.91' without this caveat misrepresent the pipeline-level performance. The authors should either report pipeline-level metrics computed on the full labeled set or clearly distinguish conditional classifier metrics from end-to-end metrics throughout the paper.","section":"Section III-D, Table VIII; Abstract; Section I (Contributions)"},{"comment":"The clustering configuration (GMM, k=5, Word2Vec) is selected using MNAPK, a label-dependent metric applied to the supervised set, and the same supervised set is then filtered by the selected cluster and used to train and evaluate the classifier. This creates a selection bias: the cluster filter is chosen to maximize the concentration of positive labels, and the classifier's reported recall is then measured on that enriched subset. The paper does not address this bias, e.g., with nested cross-validation or a held-out labeled test set. The manual validation on unlabeled data provides an independent precision check, but it does not yield a recall estimate for the deployed pipeline. At minimum, the authors should report the classifier's performance on the full labeled dataset after applying the cluster filter fitted only on the training folds, or otherwise quantify the optimism in the reported metrics.","section":"Section II-C, Table IV; Section II-E"},{"comment":"The keyword-based cluster identification assumes that the cluster with the highest cumulative frequency of at least four of five keywords is the accusatory cluster, with keywords chosen 'based on domain expertise and exploratory analysis' on the same data. The paper shows that the keyword distributions are consistent between the labeled and unlabeled datasets, but it does not assess the sensitivity of downstream performance to an incorrect cluster choice. Since deployment on unlabeled data relies entirely on this heuristic identifying the correct cluster, the authors should provide a concrete robustness test, for example re-running the pipeline with the second-most-keyword-rich cluster and reporting the resulting precision and recall, or an ablation varying the 'at least four of five' threshold. Without this, the claim that the pipeline is robust when transferred to new unlabeled corpora is not fully supported.","section":"Section II-D, Tables VI and VII; Section III-E"}],"minor_comments":[{"comment":"Table IX reports 1,892 candidates on the unlabeled dataset, but without ground truth this supplies no end-to-end recall estimate; the manual validation describes only precision on a random subset of positives. This limitation should be stated explicitly in the text surrounding Table IX.","section":"Section III-E, Table IX"},{"comment":"The text states that the complete Word2Vec-based pipeline required approximately 157 minutes, while the entries in Table X sum to 11 minutes. These numbers should be reconciled, or the text should clarify what is included in the 157-minute figure.","section":"Section III-F and Table X"},{"comment":"The average cosine similarity values are identical for the supervised and unsupervised datasets (0.36, 0.79, 0.91). Please confirm that this is not a typo and explain why the values coincide exactly, or label the table so that readers do not infer duplicated computations.","section":"Section III-A, Table V"},{"comment":"Section II-F refers to 'the keyword-based method described in Section III-C,' but the cluster identification method is described in Section II-D. Please correct the cross-reference.","section":"Section II-F, step 1"},{"comment":"The contribution bullet reporting 'a precision of 0.84 and recall of 0.91' should add the qualifier that these numbers are obtained after cluster filtering, to avoid the overstatement identified in the major comments.","section":"Section I, Introduction"},{"comment":"The code repository is referenced only as '<Kapak NLP Pipeline>'; please provide a working URL or explicit access instructions.","section":"Reproducible Research and Section II-E"}],"recommendation":"major_revision","confidential_remarks":"The paper's core finding—that a lightweight Word2Vec+GMM+Random Forest pipeline can achieve useful precision on accusatory-language detection—is plausible and the manual validation is a positive sign. The main barrier to acceptance is the current reporting of conditional metrics as end-to-end performance, which overstates recall by a noticeable margin. The cluster-selection procedure's dependence on labels also deserves careful treatment. The paper would be acceptable after these issues are addressed with revised metrics and a robustness analysis of the cluster-identification heuristic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things before reading this. First, it is a genuinely useful application paper: the domain-trained Word2Vec + GMM + Random Forest cascade is cheap, CPU-only, and beats LLaMA and RoBERTa embeddings for clustering noisy Spanish procurement comments. That empirical comparison is the real contribution. Second, the headline numbers (P=0.84, R=0.91, F1=0.87) are conditional on a cluster filter that was selected using the labels, and they overstate end-to-end performance.\n\nThe positive side is substantial. The annotation protocol is carefully defined, Fleiss kappa 0.72 is respectable, and they include a no-cluster-filter ablation that shows the full-set RF+SMOTE gets R=0.95 but P=0.58. That is an honest baseline. They also did a manual validation on unlabeled positives and report precision around 0.84, which supports the precision claim. The cross-country survey of procurement data accessibility in Latin America is a nice practical contribution.\n\nThe soft spots are real but fixable. The stress-test note is correct: Table VIII's 'Cluster Filter' scenario evaluates only the 962 rows in the selected cluster, which contains 122 of the 143 labeled positives. So the filter alone caps recall at 85.3%. Multiplying by the classifier's 0.91 recall gives an end-to-end recall around 0.78 and F1 around 0.81, not 0.87. The abstract and contribution bullet present 0.84/0.91 as if they were pipeline metrics. That is a reporting error, not a fatal flaw, but it matters for a governance tool where missed accusations have real costs. Also, the cluster configuration was chosen using label-dependent MNAPK and the keyword set emerged from exploratory analysis, so the 'unsupervised' pre-filter is not fully label-free. The manual validation only checks precision; it cannot recover the lost recall. And despite the 'Reproducible Research' note, the code link is a placeholder and the data are not available.\n\nMy take: this is a solid applied paper that needs major revision to report end-to-end metrics on the full labeled set (or clearly label the metrics as post-filter), pre-register or justify the cluster selection without peeking at labels, and release code and annotation data. The application is real and the Word2Vec-vs-Transformers finding is worth publishing. I would send it to serious peer review, not desk reject. It would make a good reading-group case on evaluation pitfalls.","headline":"A useful applied NLP paper whose headline metrics are conditional on a label-selected cluster filter; end-to-end recall is closer to 0.78 than 0.91.","tokens_in":17815,"tokens_out":2757,"would_cite":false,"duration_ms":23583,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A low-cost NLP pipeline can pick out whistleblowing comments in public procurement data, even when they are only 3% of the text.","keywords":["accusatory language detection","public procurement","Word2Vec","Gaussian Mixture Models","Random Forest","class imbalance","semi-supervised NLP","corruption risk"],"falsifier":"Annotate a random sample of the 1,892 unlabeled comments the model flags as accusatory: if the observed precision falls well below the claimed 0.84, or if the five keywords stop concentrating in a single Word2Vec+GMM cluster when the pipeline is retrained on a new batch of procurement comments, the central claim fails.","tokens_in":16854,"feed_emoji":"🔍","tokens_out":5988,"duration_ms":48830,"temperature":0.7,"pith_summary":"The paper claims that a lightweight, cascaded NLP pipeline can detect accusatory, whistleblowing-style comments in the noisy Spanish text of Ecuador's public procurement system, despite such comments being only about 3% of the data. The pipeline embeds comments, clusters them without labels, and automatically identifies the cluster richest in accusation-related words; a classifier then works only inside that cluster. The reported result is precision of 0.84, recall of 0.91, and F1 of 0.87 for the accusatory class, with a manual check on unlabeled data giving roughly the same precision. A sympathetic reader would care because it suggests that modest, transparent tools can support anti-corruption oversight without large language models or heavy computing infrastructure.","feed_headline":"A low-cost NLP pipeline flags procurement accusations at 84%","feed_subtitle":"Word2Vec, Gaussian clustering, and Random Forest catch rare whistleblower comments in Ecuador's contract data.","key_machinery":"The load-bearing mechanism is the cascade itself: domain-trained Word2Vec embeddings (a shallow neural word-embedding model trained directly on the procurement corpus) are averaged over each comment, grouped by a Gaussian Mixture Model into clusters, and a keyword heuristic selects the single most accusatory cluster; a Random Forest classifier with SMOTE is then trained and applied only to that cluster. The mechanism works because the Word2Vec space is more isotropic than the Transformer spaces, so distance-based clustering separates the rare accusatory pattern from the bulk of routine comments.","core_discovery":"In the paper's own terms, the discovery is that a domain-trained Word2Vec embedding, combined with Gaussian Mixture Model clustering and a Random Forest classifier, outperforms Transformer-based embeddings for detecting accusatory language in short, noisy, user-generated procurement questions. The decisive empirical pattern is that clustering first, then classifying only the selected cluster, turns a severely imbalanced task into a tractable one: the Word2Vec+GMM step concentrates 122 of 143 accusatory phrases into one cluster, and Random Forest with SMOTE on that filtered set reaches precision 0.84, recall 0.91, and F1 0.87. The same keyword-based cluster selection appears to generalize from the labeled corpus to the unlabeled corpus, which supports the paper's semi-supervised claim.","pith_inferences":["If the pipeline's precision holds on live deployment, the same cascade could be repurposed for other rare classes in noisy user text, such as hate speech, safety incident reports, or discrimination complaints, where only a small labeled seed exists.","The five-keyword heuristic is itself a testable design choice; a learned cluster-description step could make the method robust to vocabulary drift or to indirect language, which the paper acknowledges it currently misses.","The near-identical proportions of accusations in the labeled and unlabeled clusters could support an early-warning audit tool that flags procurement processes with unusually many predicted accusatory comments, a use the paper only gestures at.","A direct cross-country test on Chile or Costa Rica would isolate how much of the result depends on Ecuadorian lexical conventions; the paper's own survey suggests data availability, not modeling, is the main barrier."],"forward_implications":["Without any GPU, the full pipeline trains in about 11 minutes for the best configuration, and single predictions take 1-3 seconds, so continuous monitoring of procurement comments is feasible for oversight bodies.","Cluster filtering cuts the training set from 4,841 to 962 samples while retaining 122 of 143 accusatory phrases, shrinking compute and improving class balance at once.","The same keyword-based cluster identification applied to 92,579 unlabeled comments finds 1,892 candidate accusatory comments, a similar proportion to the labeled set, suggesting the approach transfers to unseen data.","Because the pipeline is embedding- and model-agnostic, it can be ported to other Latin American procurement systems whenever structured Q&A comments are accessible, though zero-shot validation remains future work."],"supporting_citations":[{"why":"Documents the LLM-based predecessor that achieved high recall but very low precision under class imbalance, defining the precision problem the present pipeline aims to solve.","marker":"[11]"},{"why":"Supplies the Fleiss' Kappa inter-annotator agreement measure used to validate the 5,005-sample labeled dataset.","marker":"[15]"},{"why":"Defines the LLaMA 3.2-1B embedding baseline that the paper compares against Word2Vec.","marker":"[17]"},{"why":"Provides the RoBERTa embedding baseline compared against Word2Vec in the clustering evaluation.","marker":"[18]"},{"why":"Describes mean pooling of token embeddings into sentence representations, the paper's unified embedding extraction strategy.","marker":"[24]"},{"why":"Documents Gaussian Mixture Models, the clustering algorithm that yields the best Word2Vec-based cluster separation.","marker":"[26]"},{"why":"Supports keyword-based cluster identification as effective for noisy short-text topic detection under limited supervision.","marker":"[29]"},{"why":"Introduces SMOTE, the synthetic oversampling method used to mitigate residual class imbalance after cluster filtering.","marker":"[30]"},{"why":"Defines Random Forests, the final classifier that achieves the reported precision and recall.","marker":"[32]"}],"fun_headline_variants":["Cluster first, classify later: rare procurement accusations surface","Two-step NLP: clustering then classifying catches procurement fraud","Word2Vec and GMM outperform Transformers on imbalanced procurement data","Cheap embeddings plus clustering surface rare procurement accusations","Clustering pinpoints 122 accusations, Random Forest hits 84% precision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline stands on the assumption that in the unlabeled corpus the cluster with the most occurrences of at least four of five accusation keywords is the same kind of accusation-rich cluster found in the labeled data; if the unlabeled comments use those words in different proportions, the filter selects the wrong cluster and the reported precision and recall no longer apply.","fun_headline_variants_meta":{"raw":{"variants":["Cluster first, classify later: rare procurement accusations surface","Two-step NLP: clustering then classifying catches procurement fraud","Word2Vec and GMM outperform Transformers on imbalanced procurement data","Cheap embeddings plus clustering surface rare procurement accusations","Clustering pinpoints 122 accusations, Random Forest hits 84% precision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000857,"raw_usage":{"total_tokens":3712,"prompt_tokens":924,"completion_tokens":2788,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":2704}},"tokens_in":540,"tokens_out":2788,"duration_ms":18788,"temperature":1.0,"reasoning_tokens":2704,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:10:34.418310+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Annotate a random sample of the 1,892 unlabeled comments the model flags as accusatory: if the observed precision falls well below the claimed 0.84, or if the five keywords stop concentrating in a single Word2Vec+GMM cluster when the pipeline is retrained on a new batch of procurement comments, the central claim fails.","supporting_citations":[{"cited_title":"Smart citizen control of public procure- ment in ecuador: Classification of accusatory comments from “sistema oficial de contrataci´ on p´ ublica del ecuador (soce)","cited_arxiv_id":null,"evidence_quote":"Documents the LLM-based predecessor that achieved high recall but very low precision under class imbalance, defining the precision problem the present pipeline aims to solve."},{"cited_title":"Survey article: Inter-coder agree- ment for computational linguistics,","cited_arxiv_id":null,"evidence_quote":"Supplies the Fleiss' Kappa inter-annotator agreement measure used to validate the 5,005-sample labeled dataset."},{"cited_title":"Llama 3.2-1b","cited_arxiv_id":null,"evidence_quote":"Defines the LLaMA 3.2-1B embedding baseline that the paper compares against Word2Vec."},{"cited_title":"Roberta: A robustly optimized bert pretraining approach,","cited_arxiv_id":null,"evidence_quote":"Provides the RoBERTa embedding baseline compared against Word2Vec in the clustering evaluation."},{"cited_title":"Sentence-bert: Sentence embed- dings using siamese bert-networks,","cited_arxiv_id":null,"evidence_quote":"Describes mean pooling of token embeddings into sentence representations, the paper's unified embedding extraction strategy."},{"cited_title":"Gaussian mixture models — scikit-learn 1.4.2 documentation","cited_arxiv_id":null,"evidence_quote":"Documents Gaussian Mixture Models, the clustering algorithm that yields the best Word2Vec-based cluster separation."},{"cited_title":"Word embedding based clustering to detect topics in social media,","cited_arxiv_id":null,"evidence_quote":"Supports keyword-based cluster identification as effective for noisy short-text topic detection under limited supervision."}],"review_version":1}