{"id":"9d758687-69cb-40d9-9237-83bc69c03289","arxiv_id":"1909.02523","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Across MovieLens and Amazon data, nDCG and Precision are the most discriminative metrics for hyperparameter tuning, and the number of latent factors dominates other BPR-MF hyperparameters.","lead":"This paper tests which evaluation metrics best tell apart differently tuned recommender systems during cross-validation, using two movie datasets and three standard algorithms. It finds that nDCG and Precision are the most discriminative accuracy metrics and proposes a procedure for spotting which hyperparameter matters most.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 5's DP analysis fixes the target hyperparameter and varies the other two, so low DP for latent-factor rows cannot show that latent factors dominates; the dominance conclusion does not follow from the reported procedure.","rationale":"The reader already assigned CONDITIONAL, and the rationale explicitly notes that the Section 5 table direction appears to contradict the prose conclusion. My stress-test agrees with that observation and identifies it as the single most load-bearing concern because it directly undermines one of the paper's two stated contributions: the claim that latent factors dominate for BPR-MF. The reader's formal weakest_assumption focused on the 25 random pairs and the nDCG-based sub-grid, which are valid concerns about noise and bias, but the Section 5 logical flaw is more fundamental: even with perfect data, the described procedure cannot yield the stated conclusion. I do not recommend moving to REJECT because the underlying experimental data might still support the claim if reanalyzed with the correct pairing or with a method like fANOVA; the paper is short and the flaw is fixable. Thus the verdict remains CONDITIONAL, and my read does not change the reader's overall decision. Agreement is partial because the reader identified the issue in passing but did not make it the primary weakest assumption, whereas I regard it as the decisive point.","tokens_in":10416,"tokens_out":6005,"duration_ms":62793,"concrete_test":"Recompute the dominance analysis in the correct direction: hold iterations and learning rate fixed at a representative mid-grid value (e.g., iterations=16, learning rate=0.003125) and compute Section 3 DP over all pairs of configurations that differ only in latent factors. Repeat the same procedure for the other two hyperparameters by varying only iterations and only learning rate. If latent factors is dominant, the latent-factor-only DP should be the lowest, indicating more significant differences. Report the three DPs; if latent factors is not the lowest, the Table 5 interpretation is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is in Section 5 (Tables 5), where the authors claim that for BPR-MF the number of latent factors dominates iterations and learning rate. What the procedure actually computes is, for each value of the target hyperparameter, the DP of pairs of configurations that share that value and differ in the remaining two hyperparameters. For a fixed latent-factor count, a low DP means that varying iterations and learning rate still produces many significant accuracy differences; it does not measure the effect of changing latent factors. To establish dominance one would need to hold iterations and learning rate fixed and vary only latent factors, and compare that DP with the analogous DPs for the other two hyperparameters. Table 5 shows that when latent factors is fixed, the residual variation from the other parameters is highly discriminative (MovieLens latent-factor DP values range roughly 0.29–0.71, well below the iteration DP values), which, if anything, points away from the authors' conclusion. The paper also acknowledges the rows are not comparable across hyperparameters, yet the conclusion selects the lowest DP among latent-factor rows and compares it with the lowest among iteration rows. Because the dominance claim is a stated contribution and part of the abstract/conclusion, the paper's central message is not supported by the described analysis.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how evaluation metrics behave when used to select hyperparameters in cross-validation, and whether some hyperparameters are more important than others. The authors compute the Discriminative Power (DP) of nDCG, Precision, Recall, MRR, EFD, and EPC over grid-search configurations of User-kNN, Item-kNN, and BPR-MF on MovieLens-1M and Amazon Movies. They report that nDCG@N and Precision@N are the most discriminative accuracy metrics, that novelty metrics are also sensitive to hyperparameter changes, and that for BPR-MF the number of latent factors dominates the number of iterations and the learning rate. The paper proposes a procedure for isolating a single hyperparameter in models with several parameters and reports DP tables per hyperparameter value.","tokens_in":10601,"tokens_out":6750,"duration_ms":77955,"significance":"If the results were supported, the paper would make a useful practical contribution to recommender-system evaluation: it would guide practitioners in choosing a metric for cross-validation-based tuning and in prioritizing which BPR-MF hyperparameters to spend compute on. The work builds on an established DP framework from prior work, uses public datasets and standard libraries, and asks a well-motivated question. The empirical claims are specific and falsifiable. However, the significance is moderate because the findings rest on two datasets and on sampling choices that are not fully controlled, and because the central claim about latent-factor dominance appears not to follow from the reported analysis. No code or experimental seeds are provided, which limits reproducibility of the randomness-dependent steps.","major_comments":[{"comment":"The conclusion that the number of latent factors is the dominant hyperparameter for BPR-MF does not follow from the reported procedure. The text states that for the latent-factor analysis, each pair of systems is selected to \"share the number of latent factors, and differ in the number of iterations and learning rate.\" Thus a low DP value for a fixed latent-factor row means that variations in iterations and learning rate produce statistically significant metric differences while the latent-factor count is held constant; it is not a measurement of the effect of changing the latent-factor count. To establish dominance one would need to hold iterations and learning rate fixed, vary only the latent factors, and compare that conditional DP with the analogous conditional DPs for the other two parameters. The manuscript itself warns that \"the results between the two tables are not comparable,\" yet the conclusion selects the best DP values across the three hyperparameter analyses. Since this dominance claim appears in the abstract, introduction, and conclusions, the paper's stated contribution (iii) is unsupported by the analysis as written.","section":"Section 5 (Table 5)"},{"comment":"The DP values in Table 3 are computed from a single random draw of 25 pairs of hyperparameter configurations, with no seed, no repeated sampling, and no reported variance. For BPR-MF the MovieLens grid has 15×15×14 = 3,150 configurations, so the pairing universe is enormous, and 25 pairs is a very small sample. Several adjacent DP values in Table 3 are close (e.g., Item-kNN MovieLens nDCG@N 1.710 vs MRR@N 1.766; BPR-MF MovieLens nDCG@N 0.594 vs Precision@N 0.585), so the ranking could plausibly change under another draw. The central claim that nDCG@N and Precision@N are the most discriminative metrics rests on this table. The authors should either use all pairs, or report confidence intervals or repeated random draws, and must at minimum fix and state the random seed.","section":"Section 3 (Table 3)"},{"comment":"The Amazon experiments are run on a sub-grid selected by the nDCG@10 value of each hyperparameter plus two neighboring grid values. This makes the comparison across metrics unfair: the configurations are chosen because they are promising according to nDCG@10, so nDCG is evaluated on a grid tailored to its own preferences while the other metrics are evaluated on a grid selected by nDCG. This selection bias affects the Amazon portion of Table 3 and the Amazon portion of Table 5, and it also makes the Section 5 statement that the DP values show the sub-grid choice was \"a reasonable choice\" circular. A metric-independent subsampling rule, or a separate sub-grid selection for each metric with a corresponding correction, is needed before the Amazon metric ranking can be accepted.","section":"Section 2 (Amazon sub-grid)"}],"minor_comments":[{"comment":"There are small text errors: \"Intel Xenon\" should be \"Intel Xeon,\" and \"0,2000038948\" uses a comma as a decimal separator, which is inconsistent with the rest of the paper.","section":"Section 2"},{"comment":"The sentence \"we generate all possible combinations of pairs of hyperparameters and we randomly take 25 combinations\" is ambiguous: it is unclear whether the 25 pairs are drawn once per dataset/model/metric, and whether the same pairs are used for all metrics. Please clarify the sampling protocol.","section":"Section 3"},{"comment":"The column header \"Best + Std Dev\" is not self-explanatory. It should state explicitly whether the standard deviation is added to each ordered p-value before computing the DP sum, or to the final DP value; the current text describes the former but the table caption does not.","section":"Section 4 (Table 4)"},{"comment":"Table 5 is difficult to read because the MovieLens blocks contain 15, 14, and 15 values while the Amazon blocks contain only 5, 3, and 3 values, with no visual separator between datasets. A panel split or separate rows for the two datasets would greatly improve clarity.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The Section 5 issue is not a presentation problem but a correctness problem in the central argument. I would ask the authors to rerun the analysis with the correct conditioning direction (holding the other two parameters fixed and varying the target hyperparameter) and to address the random-pair sampling and the nDCG-based sub-grid selection before the paper can be considered for publication. The topic is appropriate for RecSys, but the current evidence does not support the headline conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the metric-ranking part is a useful, practical extension of Valcarce et al.'s Discriminative Power to hyperparameter tuning. The Section 5 claim that latent factors dominate BPR-MF doesn't stand up to a close reading; the procedure actually measures residual discriminative power of the other parameters, not the effect of the fixed one.\n\nWhat the paper does well: the experimental setup is solid (10-fold CV, temporal split, standard baselines, both accuracy and novelty metrics). The finding that nDCG and Precision are more discriminative than Recall and MRR across datasets is plausible and consistent with what I'd expect. Table 4, with DP ± standard deviation over folds, is a nice addition that gives a sense of stability. The citation to Valcarce et al. is correct, and the related work coverage is appropriate.\n\nWhere it gets shaky: Section 5 fixes the target hyperparameter (say, latent factors) and then computes DP over pairs that share that value and differ in the other two. So a low DP for latent-factor rows means that, given a fixed latent-factor count, iterations and learning rate still produce many significant differences. That is evidence that the other parameters are powerful within a slice, not that latent factors dominate. The paper's own text says the rows aren't comparable across hyperparameters, then compares them anyway. I checked the stress-test note on this and it holds up; the dominance conclusion is unsupported.\n\nTwo smaller concerns: the DP numbers are based on 25 random pairs with no seed and no variance, so the exact ranking is noisy; and the Amazon sub-grid was picked using nDCG@10, which biases the comparison against nDCG. Neither is fatal, but both would be easy to address in a revision.\n\nWho this is for: anyone tuning recommender systems, especially those wondering which metric to watch during model selection. The metric-ranking part is worth a read; the hyperparameter-dominance part should be either rewritten or ignored. It deserves a serious referee, but my recommendation is conditional—the authors should fix the Section 5 interpretation or change the procedure, and release seeds/code.","headline":"Useful metric comparison for tuning, but the dominant-hyperparameter claim doesn't follow from the analysis.","tokens_in":11142,"tokens_out":6057,"would_cite":true,"duration_ms":61603,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"nDCG and Precision are the best hyperparameter tuners","keywords":["hyperparameter tuning","cross-validation","discriminative power","recommender systems","nDCG","precision","BPR-MF","novelty metrics"],"falsifier":"Recompute DP for BPR-MF on MovieLens using all hyperparameter pairs from the full 3150-configuration grid (rather than 25 random pairs) and on the full Amazon grid (rather than the nDCG-selected sub-grid); if the metric ranking no longer places nDCG/Precision first, the paper's conclusion is an artifact of the sampling.","tokens_in":10169,"feed_emoji":"🎯","tokens_out":4953,"duration_ms":45421,"temperature":0.7,"pith_summary":"This paper asks which evaluation metric best distinguishes between good and bad hyperparameter settings when tuning recommender systems by cross-validation. Using 10-fold cross-validation on MovieLens-1M and Amazon Movies, it compares three models—User-kNN, Item-kNN, and BPR-MF—across grids of hyperparameters and measures accuracy and novelty metrics. The authors find that nDCG@N and Precision@N are the most discriminative accuracy metrics, meaning they most reliably detect significant performance differences between configurations. They also find that for BPR-MF the number of latent factors dominates the other hyperparameters, so tuning effort is best spent there. The paper proposes a general procedure for isolating whether a single hyperparameter matters for a given model.","feed_headline":"nDCG and Precision are the best hyperparameter tuners","feed_subtitle":"Across two movie datasets, they separate good from bad models best; for BPR-MF latent factors matter most.","key_machinery":"Discriminative Power (DP): for a given metric, the sum of p-values over paired statistical tests (Student's t-test) between hyperparameter configurations, averaged across cross-validation folds; a lower DP value means the metric is more sensitive to hyperparameter changes. The paper's per-parameter variant fixes one hyperparameter (e.g., latent factors) and randomly pairs the remaining dimensions, producing a p-value curve whose DP reveals whether that fixed parameter's values drive accuracy differences.","core_discovery":"The central discovery is that Discriminative Power (DP), computed from p-values of paired statistical tests over hyperparameter pairs, ranks nDCG@N and Precision@N as the strongest accuracy metrics for model selection among the six studied. These metrics show lower DP (sharper separation) than Recall and MRR across nearly all model-dataset combinations, and this ordering survives adding a standard-deviation band across folds. For BPR-MF, fixing each hyperparameter in turn and randomizing the others shows that the number of latent factors produces the smallest DP, indicating that variations in latent-factor count cause the most significant accuracy differences; the paper concludes that this parameter dominates the number of iterations and learning rate.","pith_inferences":["If the same DP analysis were run on non-movie domains (news, e-commerce, social), the dominance of nDCG/Precision might not hold; the paper's conclusion is demonstrated only on two movie datasets.","The Amazon sub-grid was pre-selected using nDCG@10, which could bias the Amazon comparison in nDCG's favor; a neutral test would select the sub-grid by a different metric or repeat the selection for each metric.","The 25 random pairs per model introduce sampling variance; constructing bootstrap confidence intervals around DP values would test whether the metric ordering is statistically stable.","The latent-factor dominance result suggests a two-stage tuning strategy: coarse search over latent factors first, then fine-tuning of learning rate and iterations, which the paper does not explicitly recommend but follows directly from its findings."],"forward_implications":["Practitioners can use nDCG@N or Precision@N as the primary selection metric when tuning neighborhood-based models or BPR-MF, instead of averaging many metrics.","When tuning BPR-MF, prioritizing the number of latent factors in the search (e.g., more granular values) over iterations and learning rate should yield most of the accuracy gain.","Novelty metrics (EFD, EPC) are also highly sensitive to hyperparameter changes, so tuning for accuracy alone may unintentionally drive novelty in ways a combined objective would avoid.","The per-parameter DP procedure offers a cheap pre-screening step: before a full grid search, identify which hyperparameters actually matter for the model and dataset.","The metric-DP comparisons across folds show that the best accuracy metrics also have acceptable fold-to-fold stability, supporting their use in k-fold cross-validation pipelines."],"supporting_citations":[{"why":"Defines Discriminative Power and the p-value curve procedure that the paper extends from cut-off variation to hyperparameter variation.","marker":"[44]"},{"why":"Original bootstrap-based metric evaluation methodology that the DP computation builds on.","marker":"[36]"},{"why":"Introduces BPR-MF, one of the three models whose hyperparameters are studied.","marker":"[34]"},{"why":"Defines nDCG, the metric the paper finds most discriminative.","marker":"[25]"},{"why":"Defines Expected Free Discovery, a novelty metric studied in the experiments.","marker":"[45]"},{"why":"Defines Expected Popularity Complement and the novelty/diversity evaluation framework.","marker":"[12]"}],"fun_headline_variants":["nDCG and Precision best separate models","For BPR-MF, latent factors dominate tuning","Which metric picks best? nDCG and Precision","Key hyperparameters: nDCG, Precision, latent factors","Latent factors drive BPR-MF cross-validation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusions rest on the representativeness of 25 randomly chosen hyperparameter pairs per model and, for Amazon, on a grid sub-selected using nDCG@10; if those choices are unrepresentative, the ranking of metrics could change.","fun_headline_variants_meta":{"raw":{"variants":["nDCG and Precision best separate models","For BPR-MF, latent factors dominate tuning","Which metric picks best? nDCG and Precision","Key hyperparameters: nDCG, Precision, latent factors","Latent factors drive BPR-MF cross-validation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000169,"raw_usage":{"total_tokens":1226,"prompt_tokens":868,"completion_tokens":358,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":484,"completion_tokens_details":{"reasoning_tokens":282}},"tokens_in":484,"tokens_out":358,"duration_ms":4261,"temperature":1.0,"reasoning_tokens":282,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:48:10.450568+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute DP for BPR-MF on MovieLens using all hyperparameter pairs from the full 3150-configuration grid (rather than 25 random pairs) and on the full Amazon grid (rather than the nDCG-selected sub-grid); if the metric ranking no longer places nDCG/Precision first, the paper's conclusion is an artifact of the sampling.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Discriminative Power and the p-value curve procedure that the paper extends from cut-off variation to hyperparameter variation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Original bootstrap-based metric evaluation methodology that the DP computation builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Expected Free Discovery, a novelty metric studied in the experiments."},{"cited_title":"Hurley, and Saul Vargas","cited_arxiv_id":null,"evidence_quote":"Defines Expected Popularity Complement and the novelty/diversity evaluation framework."}],"review_version":1}