Pith. sign in

REVIEW 3 major objections 5 minor 30 references

CUPCF: Combining Users Preferences in Collaborative Filtering for Better Recommendation

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that averaging NHSM and Pearson similarity predictions reduces recommendation error across 19 collaborative filtering baselines on the MovieLens dataset.

desk verdict Simple averaging idea, but the reported recall/precision numbers are internally impossible, so the evaluation is unreliable. read the letter →

arxiv 1908.05609 v1 pith:XEIF4Y47 submitted 2019-08-13 cs.IR cs.SI

Mostafa Khalaji , Nilufar Mohammadnejad This is my paper · ORCID
classification cs.IRcs.SI
keywords RecommenderSystemsCollaborativeFilteringUserPreferencesSimilarityMeasuresCUPCFMovieLensMeanAbsoluteError
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes CUPCF, a user-based collaborative filtering predictor that averages the rating predictions of two similarity measures: NHSM, designed to handle sparse rating data, and the standard Pearson correlation. On the MovieLens 100K dataset with 5-fold cross-validation and 300 nearest neighbors, CUPCF reports a lower mean absolute error than each constituent measure and 19 collaborative filtering baselines, with relative MAE improvements from 0.28% to 15.5% and peak Accuracy, Precision, and Recall of 0.91402, 0.91436, and 0.9974. The motivation is that one similarity measure may fail to find usable neighbors under sparsity while the other still can, so averaging the two predictions gives the system a second path to a rating. The contribution is the equal-weight combination rule and its empirical demonstration on a standard benchmark.

What carries the argument

The load-bearing object is Eq. (13), the equal-weight arithmetic average of two nearest-neighbor prediction formulas. NHSM_Predict (Eq. 11) uses the NHSM similarity, which combines a Jaccard-style overlap term with proximity, significance, singularity, and user rating-preference terms; Pearson_Predict (Eq. 12) uses the Pearson correlation coefficient over co-rated items. Averaging these two forecasts is what CUPCF contributes: it lets the system fall back on whichever similarity found useful neighbors, and the final prediction is simply their midpoint. All evaluation results in the paper are results of this average, not of either measure alone.

What would settle it

Running the 19 baselines on the same MovieLens 100K splits, with the same k=300 nearest neighbors and the same threshold-based Accuracy, Precision, and Recall computations, and comparing their MAE to Eq. (13) would settle the central claim. If any baseline matches or beats CUPCF under identical conditions, the reported improvement percentages are artifacts of comparison rather than effects of averaging.

Watch

Extended reading notes

Core claim

The central claim is that computing CU_Predict(u,p) = (NHSM_Predict(u,p) + Pearson_Predict(u,p)) / 2 for every unrated item p produces better recommendations than using either predictor alone. NHSM_Predict and Pearson_Predict are each weighted nearest-neighbor forecasts of user u's rating of p, built from the u's nearest neighbors under the corresponding similarity. The paper argues that the combination relieves data sparsity: when the neighbor set of one similarity is too small or unreliable, the other measure's prediction can still contribute, and the final rating is the midpoint of the two forecasts. The reported support is the MovieLens 100K evaluation, where the averaged predictor's MAE is below all 19 baselines compared, with maximum improvements of 15.5% over cosine similarity and 12% over NHSM alone.

Load-bearing premise

The improvements depend on the assumption that the 19 baseline error rates come from the same evaluation protocol as CUPCF, because several reported gains are small enough that protocol differences alone could produce them.

Editorial extensions

If this is right

  • If the reported gains hold, recommender systems can lower their error rate by averaging two different similarity-based predictions rather than selecting one similarity measure.
  • The combination directly targets sparse rating matrices, since a failed neighbor search under one similarity does not leave the system without a prediction.
  • Averaging NHSM with Pearson yields larger relative MAE gains over weak baselines (15.5% over cosine) than over strong ones (0.28% over CF-RA), suggesting the benefit concentrates where the baseline is already close.
  • The Top-N recommendation lists also grow: if NHSM or Pearson alone finds fewer than N items for a user, the union of both lists can fill the recommendation set.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not test is a weighted average: if the two predictors have unequal accuracy on a given user or item neighborhood, a learned combination weight might beat the fixed 1/2 midpoint; this can be tested on the same MovieLens folds.
  • Because the reported gains over several baselines are under 1%, the authors' protocol dependence matters: rerunning baselines under identical folds and k=300 would show how much of the improvement is due to averaging rather than to evaluation setup.
  • The same averaging idea could be applied to item-based collaborative filtering or to similarity measures beyond NHSM and Pearson, since nothing in Eq. (13) is specific to user-based prediction.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes CUPCF, a user-based collaborative filtering method that computes two user-similarity matrices, one with the NHSM similarity measure and one with the Pearson correlation coefficient, selects k-nearest neighbors according to each measure, forms rating predictions through Eqs. (11) and (12), and finally combines the two predictions as an unweighted average in Eq. (13). The authors claim that on the MovieLens 100K dataset with 5-fold cross-validation, CUPCF reduces the MAE relative to 19 collaborative filtering baselines, with improvements ranging from 0.28% (CF-RA) to 15.5% (COS), and that the maximum Accuracy, Precision, and Recall are 0.91402, 0.91436, and 0.9974, respectively. The paper also argues that the combination alleviates data sparsity and cold-start problems by drawing on two different similarity measures.

Significance. If the empirical claims were correct, the central contribution would be a simple, parameter-free ensemble of two established similarity-based predictors that improves prediction accuracy without any fitted weights. The method as defined in Eq. (13) is transparent, and the MAE values reported across the five folds in Tables 6 and 7 are internally consistent with each other. There is no obvious circularity: the combination formula is not tuned to the test set, and the baseline comparisons are external. However, the significance of the paper is currently not established because the ranking metrics reported in Tables 6 and 7 are internally inconsistent with the described experimental setup, and the baseline MAE comparisons in Fig. 3 lack the implementation details and statistical support needed to sustain improvements as small as 0.28%. These are load-bearing issues for the central claim, not presentation problems.

major comments (3)
  1. [§4, Tables 6 and 7, Eqs. (14)-(17)] The reported ranking metrics are internally inconsistent with the described experimental protocol. In Table 6 (T=3, Top-5), Fold 1 reports Precision=0.9170 and Recall=0.9982. With a Top-5 recommendation list, Precision=TP/5, so TP is about 4.585; from Recall=TP/(TP+FN), the total number of relevant test items per user would be TP+FN about 4.593. Under the stated 80/20 five-fold split of MovieLens 100K, each user has roughly 20-21 test ratings, and because ratings at or above the T=3 threshold are the majority of MovieLens ratings, the expected number of relevant test items is on the order of 12-17. The maximum possible recall for a Top-5 list would then be about 5/12, approximately 0.42, far below the reported 0.998. The reported values imply that the recall denominator was computed incorrectly, for example as TP+FP, which would make Recall equal to Precision. Because Tables 6 and 7 are the only evidence for the Accuracy, Precision, and Recall claims, those claims are invalidated. Since the same experimental pipeline produces the MAE values and the comparisons in Fig. 3, the correctness of those numbers is also called into question.
  2. [§4, Fig. 3 and the 19-method improvement list] The headline improvement claims are not supported by the information provided. The baseline MAE values for CF, CF-RA, CF-Diff, CF-Rank, CF-HW, CF-MW, Pearson, RA-COS, RA-SRC, SRC, RA-CPC, CPC, Three-Segment, BCF, NHSM, PIP, COS, K-Means Leader, and K-Means are taken from references [18], [28], and [29], but the paper does not report the implementation, preprocessing, neighbor-count setting, or fold split used to obtain those baseline numbers, nor whether they were computed on the same random 80/20 folds as CUPCF. Several of the claimed gains are very small: CF-RA 0.28%, CF-Diff 0.8%, CF-MW 1.1%, and RA-CPC 1.1%. The CUPCF fold MAEs in Tables 6 and 7 range from 0.7257 to 0.7345, a spread of about 1.2%, so a claimed improvement of 0.28% is smaller than the fold-to-fold variation of the proposed method itself. Without significance tests, confidence intervals, or a shared experimental codebase, these comparisons do not demonstrate that CUPCF improves on the baselines.
  3. [§3.2, Eq. (13)] The proposed combination mechanism is described in the text as allowing the system to fall back on the other similarity measure when one measure cannot predict a rating, but Eq. (13) does not implement that behavior. It always computes the arithmetic mean of the two prediction values; when Eq. (11) or Eq. (12) cannot produce a meaningful prediction, the corresponding component is the active user's mean rating, as stated in the text. In that situation, the combined prediction is diluted by a constant rather than replaced by the other predictor. If the intended behavior is to select the available predictor, Eq. (13) needs a conditional or weighting scheme; as written, the stated motivation does not match the algorithm.
minor comments (5)
  1. [§3.2, Eqs. (11)-(12)] The summation index and the neighbor count are both written as m, while the text separately defines m as the number of neighbors and later sets k=300; this makes the formulas ambiguous.
  2. [§3.1, Eq. (9)] The paper says Eq. (9) computes the 'standard variance'; the expression is the standard deviation, since it is the square root of the variance. The terminology should be corrected.
  3. [§3.1, Eq. (3)] Eq. (3) defines JaccardSim' as |Iu∩Iv|/(|Iu|×|Iv|), which is not the Jaccard index and is not normalized to a probability. This choice should be justified or renamed, and it explains the very small similarity values in Tables 2 and 4.
  4. [§4, Fig. 3] Figure 3 appears in the manuscript as raw floating-point values rather than a legible chart with labeled axes and method names; the figure must be properly rendered so that the MAE comparison can be read.
  5. [§4, Tables 6 and 7] The threshold T is described ambiguously: the text says 'rating of 1 to 3 indicates an extreme dislike, and the rating of 3 to 5 indicates a strong affinity,' which does not make clear whether relevant items are those with rating ≥3 or >3. This must be stated precisely for the reported Precision and Recall values to be interpretable.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: CUPCF's averaging formula is an unweighted mean of two independently defined predictors, and the reported gains are empirical rather than forced by construction.

full rationale

The paper's central formula (13) is an unweighted arithmetic mean of NHSM_Predict (Eq. 11) and Pearson_Predict (Eq. 12), both of which are defined from external similarity measures (Eqs. 1-10). No parameter in the combination is fitted to the benchmark results, no quantity used in the prediction is defined from the claimed outcome, and the reported error-rate improvements are empirical comparisons rather than identities. The self-citations [19]-[23] appear only in related work and are not invoked to justify Eq. (13) or the experimental claims. The internal inconsistency in the reported Precision/Recall values (e.g., Top-5 Recall approximately 0.998) is a validity/correctness concern about the evaluation procedure, not a circularity of the derivation; it does not make Eq. (13) equivalent to its inputs by construction. The derivation is therefore self-contained with respect to circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The model has no fitted coefficients; the only tuned knob is neighbor count k=300, and the evaluation threshold T=3/4 drives headline metrics. The core assumptions are that NHSM, Pearson, and equal-weight averaging are appropriate for this dataset, and that the cited baseline numbers are comparable.

free parameters (2)
  • k (number of nearest neighbors) = 300
    Used in Eq. (11) and (12); chosen without sensitivity analysis. Results may depend on this value.
  • Threshold T for relevance = 3 and 4
    Converts ratings into relevant/irrelevant classes for Accuracy, Precision, Recall. T=3 makes most recommendations 'relevant' and produces the headline recall of 0.9974.
assumptions (4)
  • domain assumption The NHSM similarity measure from Liu et al. 2014 is a valid similarity model for sparse rating matrices.
    Adopted without re-derivation; CUPCF builds on it directly in Eq. (1).
  • standard math Pearson correlation is a valid similarity signal for users with sufficient co-rated items.
    Eq. (10) is the standard centered Pearson correlation used throughout collaborative filtering literature.
  • ad hoc to paper Equal-weight averaging of two prediction functions (Eq. 13) improves or preserves rating prediction accuracy.
    No theoretical or empirical sensitivity analysis is given; this is the core methodological assumption of the proposed system.
  • domain assumption Baseline MAE values from [18], [28], [29] are directly comparable to CUPCF results.
    Fig. 3 compares across papers without demonstrating identical preprocessing, folds, neighbor counts, or implementations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CUPCF: Combining Users Preferences in Collaborative Filtering for Better Recommendation." pith.science (2026). https://pith.science/paper/XEIF4Y47

@misc{pith2026190805609,
  author       = {Pith},
  title        = {Pith review of: CUPCF: Combining Users Preferences in Collaborative Filtering for Better Recommendation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XEIF4Y47}},
  note         = {Machine review of arXiv:1908.05609}
}
read the original abstract

How to make the best decision between the opinions and tastes of your friends and acquaintances? Therefore, recommender systems are used to solve such issues. The common algorithms use a similarity measure to predict active users' tastes over a particular item. According to the cold start and data sparsity problems, these systems cannot predict and suggest particular items to users. In this paper, we introduce a new recommender system is able to find user preferences and based on it, provides the recommendations. Our proposed system called CUPCF is a combination of two similarity measures in collaborative filtering to solve the data sparsity problem and poor prediction (high prediction error rate) problems for better recommendation. The experimental results based on MovieLens dataset show that, combined with the preferences of the user's nearest neighbor, the proposed system error rate compared to a number of state-of-the-art recommendation methods improved. Furthermore, the results indicate the efficiency of CUPCF. The maximum improved error rate of the system is 15.5% and the maximum values of Accuracy, Precision and Recall of CUPCF are 0.91402, 0.91436 and 0.9974 respectively.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 29 canonical work pages

  1. [18]

    Javari A, Gharibshah J, Jalili M (2014), Recommender systems based on collaborative filtering and resource allocation, Social Network Analysis and Mining, Vol. 4, No. 1, pp. 1-11

    work page 2014

  2. [28]

    Hu F, (2018), Three-Segment Similarity Measure Model for Collaborative Filtering, In International Conference on Data Mining and Big Data, Springer, Cham, pp. 138-148

    work page 2018

  3. [29]

    Kant, S, Mahara T, Jain VK, Jain DK, Sangaiah AK, (2018), LeaderRank based k -means clustering initialization method for collaborative filtering, Computers & Electrical Engineering Vol. 69, pp. 598-609. Mostafa Khalaji received the B.Sc. degree in computer engineering (Software) from Sadra Institute of Higher Education, Tehran, Iran, in 2015, and the M.Sc...

    work page 2018

  4. [1]

    Aggarwal C (2016), Recommender systems, Springer International Publishing

    work page 2016

  5. [2]

    Roh TH, Oh KJ, Han I (2003), The collaborative filtering recommendation based on SOM cluster -indexing CBR, Expert Systems with Applications, Vol. 25, No. 3, pp. 413-423

    work page 2003

  6. [3]

    Park Y, Park S, Jung W, Lee S (2015), Reversed CF: A fast collaborative filtering algorithm using a k-nearest neighbor graph, Expert Systems with Applications, Vol. 42, No. 8, pp. 4022-4028

    work page 2015

  7. [4]

    Herlocker JL, Konstan JA, Borchers A, Riedl J (1999), An algorithmic framework for performing collaborative filtering, In 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR, pp. 230-237

    work page 1999

  8. [5]

    Sarwar BM, Karypis G, Konstan JA, Riedl J (2001), Item-based collaborative filtering recommendation algorithms, In Proceedings of the 10th international conference on world wide web, pp. 285–295

    work page 2001

Show all 30 references
  1. [6]

    Resnick P, Iacovou N, Suchak M, Bergstrom P, Riedl J (1994), GroupLens: an open architecture for collaborative filtering of netnews, in: Proceeding of the ACM Conference on Computer Supported Cooperative Work, pp. 175–186

    1994

  2. [7]

    Adomavicius G, Tuzhilin A (2005), Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions, IEEE Trans. Knowl. Data Eng, Vol. 17, No. 6, pp. 734–749

    2005

  3. [8]

    Cacheda F, Carneiro V, Fernández D, Formoso V (2011), Comparison of collaborative filtering algorithms: limitations of current techniques and proposals for scalable, high-performance recommender system, ACM Trans, Web, Vol. 5, No. 1, pp. 1–33

    2011

  4. [9]

    Ahn HJ (2008), A new similarity measure for collaborative filtering to alleviate the new user cold-starting problem, Inform, Sci, Vol. 178, No. 1, pp. 37–51

    2008

  5. [10]

    Koutrica G, Bercovitz B, Garcia H (2009), FlexRecs: expressing and combining flexible recommendations, in: Proceedings of the ACM SIGMOD International Conference on Management of Data, pp. 745–758

    2009

  6. [11]

    Liu H, Hu Z, Mian A, Tian H, Zhu X (2014), A new user similarity model to improve the accuracy of collaborative filtering, Knowledge-Based Systems, Vol. 56, pp. 156-166

    2014

  7. [12]

    Choi K, Suh Y (2013), A new similarity function for selecting neighbors for each target item in collaborative filtering", Knowledge- Based Systems, Vol. 37, pp. 146-153

    2013

  8. [13]

    Yang JM, Li KF, Zhang DF (2009), Recommendation based on rational inferences in collaborative filtering, Knowledge-Based Systems, Vol. 22, pp. 105–114

    2009

  9. [14]

    Zhang J, Lin Y, Lin M, Liu J (2016), An effective collaborative filtering algorithm based on user preference clustering, Applied Intelligence, Vol. 45, No. 2, pp. 230-240

    2016

  10. [15]

    Bellogín A, Castells P, Cantado r I (2013), Improving memory -based collaborative filtering by neighbour selection based on user preference overlap, Proceedings of the 10th Conference on Open Research Areas in Information Retrieval, pp. 145-148

    2013

  11. [16]

    Hu F (2018), Three-Segment Similarity Measure Model for Collaborative Filtering, In International Conference on Data Mining and Big Data, Springer, Cham, pp. 138-148

    2018

  12. [17]

    Nadi S, Saraee MH, Bagheri A, Davarpanah Jazi M (2011), FARS: fuzzy ant based recommender system for web users, International Journal of Computer Science Issues, Vol. 8, No. 1, pp. 203-209

    2011

  13. [19]

    Thesis, Department of Computer Engineering, K

    Khalaji M (2017), Hybrid recommender system performance improvement using a neural network and resource allocation, M.S. Thesis, Department of Computer Engineering, K. N. Toosi University of Technology, Tehran, Iran

    2017

  14. [20]

    Khalaji M, Dadkhah C (2019), FNHSM_HRS: Hybrid recommend er system based on fuzzy clustering heuristic similarity measure, 7th Iranian Fuzzy and Intelligent Systems Congress, 18 th conference on Fuzzy Systems and 17 th Conference on Intelligent Systems pp. 562-568

    2019

  15. [21]

    Khalaji M, Mohammadnejad N (2019), FCNHSMRA_HRS: Improve the performance of the movie hybrid recommender system using resource allocation approach, 4th International Conference on Researchers in Science & Engineering & International Congress o n Civil, Architecture and Urbanis...

    2019

  16. [22]

    Khalaji M

    Khodaverdi N., Dadkhah C. Khalaji M. (2019). Movie Hybrid Recommender System Based on Clustering and Popularity, International Conference on Technology and Innovation in Science, Engineering and Technology (TIET)

    2019

  17. [23]

    Khalaji M. (2019). NWS_RS: Personalizing Recommendation Based on Users Information and New Weighted Similarity Measure, Research in Computer Science (IRAN), Persian Format, Vol. 13, pp. 1-14, 2019

    2019

  18. [24]

    (2012, July)

    Guo, G. (2012, July). Resolving data sparsity and cold start in recommender systems. In International Conference on User Modeling, Adaptation, and Personalization (pp. 361-364). Springer, Berlin, Heidelberg

    2012

  19. [25]

    Konstan JA, Miller BN, Maltz D, Herlocker JL, Gordon LR, Riedl J (1997), GroupLens: applying collaborative filtering to Usene t news, Commun, ACM, Vol. 40, No. 3, pp. 77-87

    1997

  20. [26]

    Harper FM, Konstan JA. (2015). The MovieLens Datasets: History and Context. ACM Transactions on Interactive Intelligent Systems (TiiS) 5, 4, Article 19 (December 2015), 19 pages. DOI=http://dx.doi.org/10.1145/2827872

    2015 doi

  21. [27]

    Hahsler M (2011), Developing and testing top -n recommendation algorithms for 0 -1 data using recommenderlab, NSF Industry University Cooperative Research Center for Net-Centric Software and System

    2011

  22. [2017]

    He is a member of IEEE

    He is a lecturer at Islamic Azad University, Shahr-e-Qods Branch and Sadra Institute of Higher Education. He is a member of IEEE. His current research interests include Recommender Systems, Machine Learning, Social Network Analysis, and Data mining. His ORCID ID is 0000-0002-5...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.