{"id":"13a9325e-6a82-456e-92a7-b4fd1f3906d5","arxiv_id":"1908.05609","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"CUPCF averages NHSM and Pearson predictions and reports lower MAE than 19 collaborative-filtering baselines on MovieLens, though with weak experimental verification.","lead":"CUPCF is a movie recommender that averages predictions from two existing collaborative-filtering similarity measures, NHSM and Pearson correlation. On MovieLens the authors report lower prediction error than 19 single-similarity baselines, with relative MAE gains from 0.28% to 15.5%.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported Recall ≈ 0.997 with Precision ≈ 0.914 at Top-5 is impossible on MovieLens 100K; the evaluation metrics are internally inconsistent, undermining all experimental claims.","rationale":"The reader's weakest assumption concerned the comparability of external baseline MAE values. My concern is more fundamental: the paper's own reported Precision and Recall values are internally impossible under the described experimental protocol. The arithmetic of the confusion matrix implies that each user has only about 4.6 relevant test items, but the 80/20 split of MovieLens 100K gives each user roughly 20 test ratings, of which the majority are relevant. This contradiction means the evaluation pipeline is broken, not merely under-documented. If the pipeline computes Precision and Recall incorrectly, the MAE values and all improvement percentages are also suspect, because they flow from the same experimental setup. Therefore the central claim that equal-weight averaging of NHSM and Pearson predictions yields the reported improvements is not supported by the current evidence. The manuscript needs a corrected, reproducible evaluation before the claim can be judged; the current state is unverdictable rather than conditionally acceptable.","tokens_in":11547,"tokens_out":8232,"duration_ms":82602,"concrete_test":"Implement CUPCF exactly as in §3 (k=300) and run the described 5-fold CV on MovieLens 100K with the standard 80/20 split. For Fold 1, compute the confusion matrix over all items for Top-5 at T=3, including non-recommended items as predicted negative. If recall is not ≈0.998, the paper's evaluation is erroneous. Also, independently verify the MAE for the 19 baseline methods by re-running them under this same split rather than copying numbers from [18,28,29].","verdict_should_be":"UNVERDICTED","load_bearing_attack":"In Table 6 (T=3, Top-5), Fold 1 reports Precision=0.9170 and Recall=0.9982. With N=5, Precision = TP/5 = 0.917, so TP≈4.585. Recall = TP/(TP+FN) = 0.9982, so TP+FN≈4.593. Thus the total number of relevant test items per user (ratings ≥3) is only ≈4.59. Under the stated 80/20 5-fold split of MovieLens 100K, each user has about 20-21 test ratings, of which a majority (typically 60-80%) are ≥3, so the expected number of relevant test items is 12-17. For Top-5, the maximum possible recall is 5/12≈0.42, far below 0.998. A recall of 0.998 is achievable only if the denominator excludes relevant items that were not recommended (i.e., recall is computed as TP/(TP+FP), which is precision) or if the test set is radically smaller than described. This internal inconsistency means the published Accuracy/Precision/Recall numbers cannot be reproduced from the described experimental setup. Because the same pipeline produces the MAE comparisons in Fig. 3, those improvements are also suspect; a pipeline that miscomputes basic ranking metrics cannot be relied on to compute MAE baselines correctly either.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CUPCF, a user-based collaborative filtering method that computes two user-similarity matrices, one with the NHSM similarity measure and one with the Pearson correlation coefficient, selects k-nearest neighbors according to each measure, forms rating predictions through Eqs. (11) and (12), and finally combines the two predictions as an unweighted average in Eq. (13). The authors claim that on the MovieLens 100K dataset with 5-fold cross-validation, CUPCF reduces the MAE relative to 19 collaborative filtering baselines, with improvements ranging from 0.28% (CF-RA) to 15.5% (COS), and that the maximum Accuracy, Precision, and Recall are 0.91402, 0.91436, and 0.9974, respectively. The paper also argues that the combination alleviates data sparsity and cold-start problems by drawing on two different similarity measures.","tokens_in":11821,"tokens_out":6597,"duration_ms":68077,"significance":"If the empirical claims were correct, the central contribution would be a simple, parameter-free ensemble of two established similarity-based predictors that improves prediction accuracy without any fitted weights. The method as defined in Eq. (13) is transparent, and the MAE values reported across the five folds in Tables 6 and 7 are internally consistent with each other. There is no obvious circularity: the combination formula is not tuned to the test set, and the baseline comparisons are external. However, the significance of the paper is currently not established because the ranking metrics reported in Tables 6 and 7 are internally inconsistent with the described experimental setup, and the baseline MAE comparisons in Fig. 3 lack the implementation details and statistical support needed to sustain improvements as small as 0.28%. These are load-bearing issues for the central claim, not presentation problems.","major_comments":[{"comment":"The reported ranking metrics are internally inconsistent with the described experimental protocol. In Table 6 (T=3, Top-5), Fold 1 reports Precision=0.9170 and Recall=0.9982. With a Top-5 recommendation list, Precision=TP/5, so TP is about 4.585; from Recall=TP/(TP+FN), the total number of relevant test items per user would be TP+FN about 4.593. Under the stated 80/20 five-fold split of MovieLens 100K, each user has roughly 20-21 test ratings, and because ratings at or above the T=3 threshold are the majority of MovieLens ratings, the expected number of relevant test items is on the order of 12-17. The maximum possible recall for a Top-5 list would then be about 5/12, approximately 0.42, far below the reported 0.998. The reported values imply that the recall denominator was computed incorrectly, for example as TP+FP, which would make Recall equal to Precision. Because Tables 6 and 7 are the only evidence for the Accuracy, Precision, and Recall claims, those claims are invalidated. Since the same experimental pipeline produces the MAE values and the comparisons in Fig. 3, the correctness of those numbers is also called into question.","section":"§4, Tables 6 and 7, Eqs. (14)-(17)"},{"comment":"The headline improvement claims are not supported by the information provided. The baseline MAE values for CF, CF-RA, CF-Diff, CF-Rank, CF-HW, CF-MW, Pearson, RA-COS, RA-SRC, SRC, RA-CPC, CPC, Three-Segment, BCF, NHSM, PIP, COS, K-Means Leader, and K-Means are taken from references [18], [28], and [29], but the paper does not report the implementation, preprocessing, neighbor-count setting, or fold split used to obtain those baseline numbers, nor whether they were computed on the same random 80/20 folds as CUPCF. Several of the claimed gains are very small: CF-RA 0.28%, CF-Diff 0.8%, CF-MW 1.1%, and RA-CPC 1.1%. The CUPCF fold MAEs in Tables 6 and 7 range from 0.7257 to 0.7345, a spread of about 1.2%, so a claimed improvement of 0.28% is smaller than the fold-to-fold variation of the proposed method itself. Without significance tests, confidence intervals, or a shared experimental codebase, these comparisons do not demonstrate that CUPCF improves on the baselines.","section":"§4, Fig. 3 and the 19-method improvement list"},{"comment":"The proposed combination mechanism is described in the text as allowing the system to fall back on the other similarity measure when one measure cannot predict a rating, but Eq. (13) does not implement that behavior. It always computes the arithmetic mean of the two prediction values; when Eq. (11) or Eq. (12) cannot produce a meaningful prediction, the corresponding component is the active user's mean rating, as stated in the text. In that situation, the combined prediction is diluted by a constant rather than replaced by the other predictor. If the intended behavior is to select the available predictor, Eq. (13) needs a conditional or weighting scheme; as written, the stated motivation does not match the algorithm.","section":"§3.2, Eq. (13)"}],"minor_comments":[{"comment":"The summation index and the neighbor count are both written as m, while the text separately defines m as the number of neighbors and later sets k=300; this makes the formulas ambiguous.","section":"§3.2, Eqs. (11)-(12)"},{"comment":"The paper says Eq. (9) computes the 'standard variance'; the expression is the standard deviation, since it is the square root of the variance. The terminology should be corrected.","section":"§3.1, Eq. (9)"},{"comment":"Eq. (3) defines JaccardSim' as |Iu∩Iv|/(|Iu|×|Iv|), which is not the Jaccard index and is not normalized to a probability. This choice should be justified or renamed, and it explains the very small similarity values in Tables 2 and 4.","section":"§3.1, Eq. (3)"},{"comment":"Figure 3 appears in the manuscript as raw floating-point values rather than a legible chart with labeled axes and method names; the figure must be properly rendered so that the MAE comparison can be read.","section":"§4, Fig. 3"},{"comment":"The threshold T is described ambiguously: the text says 'rating of 1 to 3 indicates an extreme dislike, and the rating of 3 to 5 indicates a strong affinity,' which does not make clear whether relevant items are those with rating ≥3 or >3. This must be stated precisely for the reported Precision and Recall values to be interpretable.","section":"§4, Tables 6 and 7"}],"recommendation":"reject","confidential_remarks":"The reported Recall values in Tables 6 and 7 are mathematically impossible under the described MovieLens 100K five-fold protocol, which suggests that the evaluation code computes ranking metrics in a fundamentally incorrect way. Because the same evaluation pipeline is used for the MAE-based comparisons, I do not see how local revisions could restore confidence in the experimental claims; the paper would need to be substantially reworked with a correct, reproducible evaluation and properly matched baselines. I therefore recommend rejection rather than major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know: this paper is exactly what it says on the tin—an unweighted average of NHSM and Pearson similarity-based predictions, Eq. (13). That's a legitimate, if narrow, extension; the two similarity measures are established, and the combination isn't in the cited literature. Credit where due: the formula is well-defined, has no fitted parameters, and the fold-level MAE values (0.7257–0.7345) average to the reported 0.731. The idea of combining two neighbor-based predictors is sensible and cheap to implement.\n\nBut the evaluation section doesn't hold up. Table 6 reports Top-5 precision ~0.917 and recall ~0.998. With five recommendations, precision 0.917 implies about 4.6 true positives per user. Recall 0.998 then implies only about 4.6 relevant test items per user. Under the stated 80/20 split of MovieLens 100K, each user has roughly 20 test ratings, and the large majority of MovieLens ratings are ≥3, so you'd expect 12–17 relevant items. Maximum possible recall at Top-5 would be around 0.4. The numbers are impossible. Also, recall decreases as N grows from 5 to 30, which is the opposite of what recall should do. Something is wrong with the metric computation, the test protocol, or both. That's a load-bearing problem: the accuracy/precision/recall tables are not trustworthy, and because the same pipeline underlies the MAE baselines in Fig. 3, those improvements—several below 1.5%—are also suspect. There's no code, no baseline implementation details, no error bars, and no significance tests. The cold-start and data-sparsity claims in the abstract are never tested; no experiment isolates cold-start users or varying sparsity.\n\nThe citation pattern is fine; the self-citations are to the authors' own prior work and are not load-bearing. The related work covers the standard similarity measures.\n\nBottom line: the core idea is plausible enough for a small experiment, but the published evidence is broken. This paper should not be cited as is. A serious referee could usefully ask the authors to redo the evaluation, so it's not a no-hope desk rejection, but it needs major correction before any of the claimed numbers can be trusted.\n\nGiven the internal inconsistency, I'd send it to peer review only with the expectation of heavy revision or rejection. For your purposes: not citable, not a reading-group pick.","headline":"Simple averaging idea, but the reported recall/precision numbers are internally impossible, so the evaluation is unreliable.","tokens_in":12338,"tokens_out":6056,"would_cite":false,"duration_ms":58885,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that averaging NHSM and Pearson similarity predictions reduces recommendation error across 19 collaborative filtering baselines on the MovieLens dataset.","keywords":["Recommender Systems","Collaborative Filtering","User Preferences","Similarity Measures","CUPCF","MovieLens","Mean Absolute Error"],"falsifier":"Running the 19 baselines on the same MovieLens 100K splits, with the same k=300 nearest neighbors and the same threshold-based Accuracy, Precision, and Recall computations, and comparing their MAE to Eq. (13) would settle the central claim. If any baseline matches or beats CUPCF under identical conditions, the reported improvement percentages are artifacts of comparison rather than effects of averaging.","tokens_in":11327,"feed_emoji":"🎬","tokens_out":4888,"duration_ms":43565,"temperature":0.7,"pith_summary":"This paper proposes CUPCF, a user-based collaborative filtering predictor that averages the rating predictions of two similarity measures: NHSM, designed to handle sparse rating data, and the standard Pearson correlation. On the MovieLens 100K dataset with 5-fold cross-validation and 300 nearest neighbors, CUPCF reports a lower mean absolute error than each constituent measure and 19 collaborative filtering baselines, with relative MAE improvements from 0.28% to 15.5% and peak Accuracy, Precision, and Recall of 0.91402, 0.91436, and 0.9974. The motivation is that one similarity measure may fail to find usable neighbors under sparsity while the other still can, so averaging the two predictions gives the system a second path to a rating. The contribution is the equal-weight combination rule and its empirical demonstration on a standard benchmark.","feed_headline":"Averaging two predictors beats 19 recommendation methods","feed_subtitle":"CUPCF blends NHSM and Pearson forecasts to lower MAE on MovieLens, with gains up to 15.5 percent.","key_machinery":"The load-bearing object is Eq. (13), the equal-weight arithmetic average of two nearest-neighbor prediction formulas. NHSM_Predict (Eq. 11) uses the NHSM similarity, which combines a Jaccard-style overlap term with proximity, significance, singularity, and user rating-preference terms; Pearson_Predict (Eq. 12) uses the Pearson correlation coefficient over co-rated items. Averaging these two forecasts is what CUPCF contributes: it lets the system fall back on whichever similarity found useful neighbors, and the final prediction is simply their midpoint. All evaluation results in the paper are results of this average, not of either measure alone.","core_discovery":"The central claim is that computing CU_Predict(u,p) = (NHSM_Predict(u,p) + Pearson_Predict(u,p)) / 2 for every unrated item p produces better recommendations than using either predictor alone. NHSM_Predict and Pearson_Predict are each weighted nearest-neighbor forecasts of user u's rating of p, built from the u's nearest neighbors under the corresponding similarity. The paper argues that the combination relieves data sparsity: when the neighbor set of one similarity is too small or unreliable, the other measure's prediction can still contribute, and the final rating is the midpoint of the two forecasts. The reported support is the MovieLens 100K evaluation, where the averaged predictor's MAE is below all 19 baselines compared, with maximum improvements of 15.5% over cosine similarity and 12% over NHSM alone.","pith_inferences":["A natural extension the paper does not test is a weighted average: if the two predictors have unequal accuracy on a given user or item neighborhood, a learned combination weight might beat the fixed 1/2 midpoint; this can be tested on the same MovieLens folds.","Because the reported gains over several baselines are under 1%, the authors' protocol dependence matters: rerunning baselines under identical folds and k=300 would show how much of the improvement is due to averaging rather than to evaluation setup.","The same averaging idea could be applied to item-based collaborative filtering or to similarity measures beyond NHSM and Pearson, since nothing in Eq. (13) is specific to user-based prediction."],"forward_implications":["If the reported gains hold, recommender systems can lower their error rate by averaging two different similarity-based predictions rather than selecting one similarity measure.","The combination directly targets sparse rating matrices, since a failed neighbor search under one similarity does not leave the system without a prediction.","Averaging NHSM with Pearson yields larger relative MAE gains over weak baselines (15.5% over cosine) than over strong ones (0.28% over CF-RA), suggesting the benefit concentrates where the baseline is already close.","The Top-N recommendation lists also grow: if NHSM or Pearson alone finds fewer than N items for a user, the union of both lists can fill the recommendation set."],"supporting_citations":[{"why":"Supplies the NHSM similarity measure and its prediction formula, which CUPCF averages with Pearson.","marker":"[11]"},{"why":"Supplies the Pearson correlation coefficient used in Eq. (10) and Eq. (12).","marker":"[6]"},{"why":"Provides the GroupLens context and the MovieLens data source used for evaluation.","marker":"[25]"},{"why":"Documents the MovieLens datasets, including the 100K benchmark used in the experiments.","marker":"[26]"},{"why":"Source of several resource-allocation baselines (CF-RA, RA-COS, RA-SRC, SRC, RA-CPC, CPC) that CUPCF is compared against.","marker":"[18]"},{"why":"Source of the CF-HW and CF-MW neighbor-weighting baselines in the comparison.","marker":"[15]"},{"why":"Source of the Three-Segment, BCF, NHSM, PIP, and COS baselines whose MAE values are compared in Fig. 3.","marker":"[28]"},{"why":"Source of the K-Means Leader and K-Means baselines in the comparison.","marker":"[29]"}],"fun_headline_variants":["Averaging two similarity predictors cuts error by 15.5%","Blend of NHSM and Pearson forecasts beats 19 baselines","CUPCF: half-and-half of two similarity scores improves recommendations","Averaging NHSM and Pearson predictions reduces MovieLens error by up to 15%","CUPCF's average of two similarity metrics yields 15.5% error cut"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The improvements depend on the assumption that the 19 baseline error rates come from the same evaluation protocol as CUPCF, because several reported gains are small enough that protocol differences alone could produce them.","fun_headline_variants_meta":{"raw":{"variants":["Averaging two similarity predictors cuts error by 15.5%","Blend of NHSM and Pearson forecasts beats 19 baselines","CUPCF: half-and-half of two similarity scores improves recommendations","Averaging NHSM and Pearson predictions reduces MovieLens error by up to 15%","CUPCF's average of two similarity metrics yields 15.5% error cut"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000795,"raw_usage":{"total_tokens":3491,"prompt_tokens":928,"completion_tokens":2563,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":2461}},"tokens_in":544,"tokens_out":2563,"duration_ms":19426,"temperature":1.0,"reasoning_tokens":2461,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:36:07.967769+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Running the 19 baselines on the same MovieLens 100K splits, with the same k=300 nearest neighbors and the same threshold-based Accuracy, Precision, and Recall computations, and comparing their MAE to Eq. (13) would settle the central claim. If any baseline matches or beats CUPCF under identical conditions, the reported improvement percentages are artifacts of comparison rather than effects of averaging.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the NHSM similarity measure and its prediction formula, which CUPCF averages with Pearson."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Pearson correlation coefficient used in Eq. (10) and Eq. (12)."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the GroupLens context and the MovieLens data source used for evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of several resource-allocation baselines (CF-RA, RA-COS, RA-SRC, SRC, RA-CPC, CPC) that CUPCF is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the CF-HW and CF-MW neighbor-weighting baselines in the comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the Three-Segment, BCF, NHSM, PIP, and COS baselines whose MAE values are compared in Fig. 3."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the K-Means Leader and K-Means baselines in the comparison."}],"review_version":1}