{"id":"5bae34a5-79be-4b86-9fd9-f0827da014f8","arxiv_id":"2507.14022","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A human-judgment weighting framework (CPC-CMS) is demonstrated for document-level sentiment classification, where ALBERT wins if runtime is ignored but not always when runtime is counted.","lead":"This paper introduces a framework that turns expert pairwise comparisons of eight evaluation metrics into weights, then selects the best sentiment classifier by a weighted score. It demonstrates the framework on three social media datasets with seven models, and shows the winning model changes when runtime is included.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Hand-set POM weights in Tables 7-8 are the load-bearing premise; no sensitivity analysis shows the ALBERT-first claim survives plausible expert variation.","rationale":"The paper's own arithmetic is coherent: B is skew-symmetric, Eqs. 42-43 yield weights that sum to 1, and the sample computation for LSVC in §5.3 reproduces the tabled score. The load-bearing weakness is elsewhere: the POM entries in Tables 7 and 8 are the only link between 'expert knowledge' and the final ranking, yet they are authored without any documented elicitation. Because the weights are computed from row sums of B, a single changed comparison (e.g., setting F1 vs MCC from -5 to -4) changes all weights and propagates to every G_i. The reported AI values (0.0747, 0.0647) certify only internal consistency, not correspondence to any external judgment. Without a sensitivity analysis, the claim 'ALBERT is best without efficiency' is a statement about the authors' chosen numbers, not about the framework's practical output. A Monte Carlo perturbation of the POM is the cheapest decisive check: if the top model is stable under ±1-unit perturbations, the concern is largely diffused; if it flips, the empirical conclusion needs qualification. Secondary issues (test set used in the selection scores, unreleased code) reinforce the need for the conditional verdict but are not the primary load-bearing issue. The framework itself might be sound; the demonstration is what rests on the unvalidated weights.","tokens_in":23923,"tokens_out":11686,"duration_ms":127536,"concrete_test":"Recompute the three case studies' final scores (Tables 17, 24, 31) under a Monte Carlo perturbation of the POM: for each independent upper-triangular entry in Tables 7 and 8, draw δ_ij from Uniform(-1,1), enforce b_ji = -b_ij, recompute weights via Eqs. 42-43 and rankings via Eq. 46, for 1000 replicates. Report the frequency with which ALBERT remains the unique top model under the no-efficiency weights, and the modal winner with efficiency, per dataset. If ALBERT is not top in at least 95% of replicates on every dataset, the paper's headline selection is not robust to plausible variation in the asserted expert judgments.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical result—ALBERT is the best model on all three datasets when efficiency is excluded, and different models win when it is included—depends entirely on the pairwise opposite matrices in Tables 7 and 8. These matrices are presented as expert knowledge judgments, but the paper reports no elicitation procedure, panel, or external validation; the entries are simply asserted (e.g., MCC vs Accuracy = 8, Efficiency vs MCC = -8). Weights are obtained from row averages via Eqs. 42-43, so any perturbation of a single b_ij changes every weight and, through Eq. 46, every model score. The Accordance Index (Eq. 41) only checks internal consistency of the matrix, not whether the numbers correspond to any real expert. Because no sensitivity analysis is given, the headline selection could be an artifact of the authors' choices rather than a property of the framework or the data. This does not impugn the framework's internal logic, but it makes the paper's strongest concrete claim unsupported as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the CPC-CMS framework, which uses cognitive pairwise comparison matrices (POMs) to derive weights for eight evaluation criteria and then ranks seven classification models (Naive Bayes, LSVC, Random Forest, Logistic Regression, XGBoost, LSTM, ALBERT) on three document-level sentiment datasets by a weighted sum of criterion scores. The headline findings are that, when the efficiency criterion is excluded, ALBERT ranks first on all three datasets, and when efficiency is included, the winner changes to XGBoost (Case 1), Random Forest (Case 2), and remains ALBERT (Case 3). The paper presents the framework as a generalizable, human-in-the-loop procedure for classification model selection.","tokens_in":24152,"tokens_out":10328,"duration_ms":112071,"significance":"If properly validated, the framework would provide a useful template for incorporating expert judgment into multi-criteria classifier selection, and the paper is commendable for giving explicit equations, full parameter tables, and three real datasets with clear experimental protocols. The computational steps are simple and reproducible, and the claimed rankings are falsifiable. However, the empirical support is incomplete: the POM weights are asserted without an elicitation procedure, the test set is used to select the winner, and timing uncertainty is not reported. These gaps do not invalidate the framework's internal logic but do weaken the paper's strongest claims as currently stated.","major_comments":[{"comment":"The pairwise entries in Tables 7 and 8 (e.g., MCC vs Accuracy = 8, Efficiency vs MCC = -8) are presented as expert knowledge, but no elicitation procedure, panel, or external validation is described, and the Accordance Index (Eq. 41) only checks internal consistency of the matrix, not its correspondence to any real expert. Because the weights are a deterministic function of these entries via Eqs. (42)-(43), and because the with-efficiency winner changes at the resulting small efficiency weight of 0.047 (Case 1: ALBERT drops from 0.811 to 0.778 while XGBoost rises from 0.788 to 0.802), the headline conclusions are potentially sensitive to plausible expert disagreement. A sensitivity analysis over the POM entries, or at least over the efficiency row, is needed to show the range over which each winner persists.","section":"Section 5.2, Tables 7 and 8, Eqs. (42)-(43), (46)"},{"comment":"The final scores used for model selection are computed from test-set metrics (Tables 15, 22, 29), and the 'best model' is the one with the highest test-set score. This is model selection on the test set: the same data are used to choose the model and to report its performance, which overstates the reliability of the selection. The authors should select the model using validation-set scores (or nested cross-validation) and then report the selected model's performance on the held-out test set as a separate confirmation.","section":"Section 6, Tables 15, 22, 29, Eq. (46)"},{"comment":"Running times are reported as single point values although Section 6.1 states that three seeds were used and the final running time is the mean of three runs; no standard deviations or per-run values are given. Because final scores of closely ranked models differ by as little as 0.013 (Case 2, without efficiency: ALBERT 0.916 vs Random Forest 0.903; with efficiency: Random Forest 0.909 vs ALBERT 0.876), timing variability alone could change the ranking. Report per-run timings and propagate the uncertainty into the weighted scores.","section":"Section 6.1, Tables 16, 23, 30"},{"comment":"In Cases 2 and 3, ALBERT attains the highest value on every one of the seven non-efficiency criteria, so the no-efficiency conclusion 'ALBERT is best' holds for any positive weight vector and does not actually exercise the CPC weighting mechanism. Only the efficiency-included results (Cases 1 and 2) demonstrate that the weights matter. The authors should either compare against a simple baseline such as equal weighting to show what the CPC weights add, or explicitly acknowledge that the no-efficiency result is mostly criterion dominance and focus the demonstration of CPC on the efficiency-included ranking.","section":"Section 5.2 vs Tables 15, 22, 29"}],"minor_comments":[{"comment":"The objective for LSVC is not fully specified: the minimization variable is missing, and the symbols theta and vartheta are used inconsistently; the equation should be written as a minimization over the weight vector.","section":"Eq. (15), Section 4.4"},{"comment":"The paired assessment scale is written as {-8/kappa, ..., 8/kappa}, but with kappa = 8 this gives entries in [-1, 1], while Tables 7 and 8 use integer entries from -8 to 8; the intended scale should be clarified.","section":"Section 5.2, first paragraph"},{"comment":"Table 5 lists tokens such as 'fear of' and 'is no' that are bigrams, although the text describes word-level TF-IDF tokenization; either the example should use unigrams or the inclusion of n-grams should be explained.","section":"Table 5, Section 3.3.1"},{"comment":"The text says the validation and test sets each contain 3715 documents (20% of the total), but Table 26 shows 3715 documents as 10% each; the percentages are inconsistent.","section":"Section 6.4, Table 26"},{"comment":"The data availability statement says the dataset is described in Section 4.1, but the datasets are actually described in Section 6; the reference should be corrected.","section":"Data availability"},{"comment":"The notation in Eq. (45) is malformed: the first row contains '... s12' with inconsistent subscripts, and the matrix is not properly typeset; the score matrix should be rewritten with clear indexing.","section":"Appendix, Table 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is a competent application of the authors' own CPC methodology to model selection, but the novelty is modest and the empirical validation has methodological gaps that are fixable in revision. The editor may wish to consider whether the journal's scope favors a purely procedural contribution without comparison to existing multi-criteria decision-making baselines; the test-set selection issue in particular should be resolved before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Briefly: this is a legitimate, if modest, extension of the authors' existing Cognitive Pairwise Comparison (CPC) method to the problem of selecting a classifier for document-level sentiment analysis. The framework itself is coherent, the mathematics of the weighted decision matrix is straightforward, and the demonstration on three datasets is transparent. The weak point is real and load-bearing: the criterion weights in Tables 7 and 8 are asserted, not elicited, and the paper shows no sensitivity analysis. Change a single POM entry by a few units and the winning model can change. The Accordance Index only checks internal consistency, not whether the numbers reflect any actual expert. Because the test-set scores are used to pick the winner, and code is not yet released, the empirical ranking should be read cautiously. That said, the soft spots are addressable rather than fatal; the framework's logic does not depend on the particular weights chosen. What the paper does well: it reports validation and test metrics with error ranges, separates performance from efficiency, and does not overclaim. It explicitly states that ALBERT wins only when time is ignored and that no single model dominates when efficiency is included. The literature discussion justifying the criteria is reasonable. The main novelty is the recipe, not the ingredients, so the audience is practitioners who want a structured way to combine multiple metrics, and MCDM researchers looking for an NLP application. It is not a conceptual breakthrough. My recommendation: this deserves peer review, not desk rejection. A serious referee should ask for a sensitivity analysis, a clearer account of how the POMs were obtained, and ideally the code. With those revisions, the paper would be a solid reference for the subfield.","headline":"A workmanlike extension of the authors' prior CPC weighting to classifier selection; the framework is coherent, but the headline ranking rests on hand-set weights with no sensitivity analysis.","tokens_in":24609,"tokens_out":1785,"would_cite":true,"duration_ms":24796,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes a human-centered framework that turns experts' pairwise importance judgments into weights for eight evaluation criteria, then uses a weighted decision matrix to select the best sentiment-analysis classifier.","keywords":["cognitive pairwise comparison","sentiment analysis","document-level sentiment analysis","classification model selection","multi-criteria decision making","Pairwise Opposite Matrix","ALBERT","evaluation criteria weighting"],"falsifier":"Re-run the three cases with the same model scores but a different documented POM that still passes the Accordance Index (for example, one that puts F1 above MCC, or one that weakens the efficiency penalty), and check whether ALBERT remains the unanimous leader without efficiency and whether the dataset-dependent winners with efficiency survive; if the winning model flips under small, consistent changes to the pairwise entries, the framework's output is not stable under legitimate expert disagreement. A complementary check is to replace the reverse min-max efficiency score with raw runtime as the criterion and see whether the rankings materially change.","tokens_in":23764,"feed_emoji":"🤖","tokens_out":5631,"duration_ms":62953,"temperature":0.7,"pith_summary":"The paper argues that choosing a sentiment-analysis classifier should not rest on a single metric or on intuition, and proposes CPC-CMS, a procedure in which experts make pairwise importance judgments among eight criteria (accuracy, precision, recall, F1-score, specificity, MCC, Kappa, and efficiency), convert those judgments into weights, and then combine each classifier's scores into one weighted score. Applied to three social-media datasets and seven classifiers, it reports that ALBERT ranks first when efficiency is ignored, but that including efficiency changes the best model by dataset (XGBoost, Random Forest, ALBERT), so no single model always dominates. If correct, the paper offers a reusable, human-centered model-selection procedure for document-level sentiment analysis and, more generally, for other classification tasks.","feed_headline":"ALBERT wins on quality; add runtime and winners change","feed_subtitle":"Seven classifiers, eight weighted criteria, three datasets: one framework makes the trade-off explicit.","key_machinery":"The load-bearing machinery is the Pairwise Opposite Matrix (POM), a matrix of expert difference judgments $b_{ij} = v_i - v_j$ on a scale from $-8$ to $+8$ indicating how much more important criterion $i$ is than criterion $j$. From the POM, weights are computed by the Row Average plus normal Utility (RAU) operator, $v_i = \\kappa + \\frac{1}{n}\\sum_j b_{ij}$ with $\\kappa = 8$, normalized by $n\\kappa$; consistency is checked with the Accordance Index, and the resulting weights feed a weighted decision matrix in which each classifier's comprehensive score is $G_i = \\sum_j w_j S_{ij}$. Efficiency is entered as a reverse min-max normalization of total training-plus-validation-plus-testing time, so the slowest model receives 0 in that criterion. This chain (expert POM, RAU weights, weighted sum) carries the whole selection argument, and the paper's rankings are direct products of it.","core_discovery":"The paper's central discovery, stated on its own terms, is that a single evaluation metric is not enough to rank classifiers for document-level sentiment analysis: on all three datasets ALBERT achieves the highest scores on the seven quality criteria, yet once the efficiency criterion (reverse min-max normalized runtime) is included, the winning model becomes XGBoost on Case 1, Random Forest on Case 2, and ALBERT on Case 3. The paper claims this shows CPC-CMS is feasible and that no single model always outperforms the others when both quality and time are weighted. The demonstration runs seven baseline models (Naive Bayes, LSVC, Random Forest, Logistic Regression, XGBoost, LSTM, ALBERT) over three social-media datasets (mental-health posts, Twitter sentiment, Reddit sentiment), with 10-fold cross-validation and three seeds.","pith_inferences":["Editorial inference: because the POM entries in Tables 7 and 8 are the authors' own judgments and no sensitivity analysis is reported, an immediate test is to resample the pairwise entries within the acceptable Accordance Index range and measure how often the winning model stays the same; the framework is sound as a procedure even if particular weights are contested.","Editorial inference: the reverse min-max normalization of efficiency gives the slowest model a score of exactly 0 by construction, which means that adding a very slow model to the candidate pool can change every model's efficiency score even if their runtimes are unchanged; the rankings are therefore relative to the candidate set.","Editorial inference: a natural domain extension is clinical or public-health text monitoring, where the trade-off between MCC, recall, and runtime is a policy choice; this framework turns that trade-off into an explicit, reviewable weight setting rather than an implicit modeling decision."],"forward_implications":["When classification quality alone matters, the framework's rankings on these three datasets point to ALBERT as the best among the seven models, with the largest margin on the mental-health and Reddit datasets and a narrower margin on the Twitter dataset.","Adding efficiency to the criteria changes the recommended model on two of the three datasets (XGBoost for Case 1, Random Forest for Case 2), so the framework implies that \"best model\" is not a property of a model alone but of the criteria weighting.","Because the final score is a weighted sum, any ranking can be traced back to the pairwise judgments, making model selection auditable rather than based on a single accuracy number.","The weighted decision matrix with the same eight criteria can be applied to other classification problems beyond sentiment analysis, which the paper states as its intended generality."],"supporting_citations":[{"why":"Supplies the cognitive pairwise comparison method used to derive criterion weights.","marker":"[8]"},{"why":"Defines the Pairwise Opposite Matrix and its cognitive prioritization operators, the core machinery of the weight derivation.","marker":"[10]"},{"why":"Provides the cognitive network process foundation for the pairwise comparison approach.","marker":"[9]"},{"why":"The mental-health dataset used as Case 1, on which ALBERT ranks first without the efficiency factor.","marker":"[16]"},{"why":"Introduces ALBERT, the transformer baseline that ranks first on quality in all three cases.","marker":"[20]"},{"why":"Establishes the advantage of MCC over F1 score and accuracy, motivating the high weight MCC receives in Tables 7 and 8.","marker":"[31]"},{"why":"Systematically analyzes classification performance measures, motivating the ordering of criteria in the pairwise matrix.","marker":"[32]"},{"why":"The Twitter sentiment dataset used as Case 2.","marker":"[36]"},{"why":"The Reddit sentiment dataset used as Case 3.","marker":"[37]"}],"fun_headline_variants":["ALBERT top on quality, but runtime reshuffles winners","Weighted criteria flip sentiment model rankings","CPC-CMS: quality vs speed changes the champion","No one model wins when time joins the score","ALBERT leads quality, but speed changes the best pick"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Pairwise Opposite Matrix entries in Tables 7 and 8 (judgments such as MCC being eight units more important than accuracy) are valid expert opinions, because every final ranking is computed directly from these numbers.","fun_headline_variants_meta":{"raw":{"variants":["ALBERT top on quality, but runtime reshuffles winners","Weighted criteria flip sentiment model rankings","CPC-CMS: quality vs speed changes the champion","No one model wins when time joins the score","ALBERT leads quality, but speed changes the best pick"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000158,"raw_usage":{"total_tokens":1214,"prompt_tokens":922,"completion_tokens":292,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":216}},"tokens_in":538,"tokens_out":292,"duration_ms":4364,"temperature":1.0,"reasoning_tokens":216,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:11:15.620660+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the three cases with the same model scores but a different documented POM that still passes the Accordance Index (for example, one that puts F1 above MCC, or one that weakens the efficiency penalty), and check whether ALBERT remains the unanimous leader without efficiency and whether the dataset-dependent winners with efficiency survive; if the winning model flips under small, consistent changes to the pairwise entries, the framework's output is not stable under legitimate expert disagreement. A complementary check is to replace the reverse min-max efficiency score with raw runtime as the criterion and see whether the rankings materially change.","supporting_citations":[{"cited_title":"Discov Artif Intell 4, 39 (2024)","cited_arxiv_id":null,"evidence_quote":"Supplies the cognitive pairwise comparison method used to derive criterion weights."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Pairwise Opposite Matrix and its cognitive prioritization operators, the core machinery of the weight derivation."},{"cited_title":"The Hong Kong Polytechnic University, Ph.D","cited_arxiv_id":null,"evidence_quote":"Provides the cognitive network process foundation for the pairwise comparison approach."},{"cited_title":"https://www.kaggle.com/datasets/suchintikasarkar/sentiment-analysis-for-mental-health/data (2024)","cited_arxiv_id":null,"evidence_quote":"The mental-health dataset used as Case 1, on which ALBERT ranks first without the efficiency factor."},{"cited_title":"https://www.kaggle.com/datasets/jp797498e/twitter-entity- sentiment-analysis (2021)","cited_arxiv_id":null,"evidence_quote":"The Twitter sentiment dataset used as Case 2."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Reddit sentiment dataset used as Case 3."}],"review_version":1}