{"id":"62afaef5-84d3-4dbb-bfd1-124cb8e5c503","arxiv_id":"2509.05393","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Ten existing prerequisite-detection heuristics are combined with a simple voting rule; the high precision headline is a product of threshold tuning on the AL-CPL test set.","lead":"A new unsupervised method infers which concepts belong before others in a course by voting on ten signals from Wikipedia, text, course structure, and graph hierarchy. The method reports precision up to 1.0, but those numbers come from tuning the vote threshold on the same benchmark used for evaluation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The precision claim rests on thresholds tuned on the same AL-CPL test set; held-out evaluation is needed before the central claim can be accepted.","rationale":"The reader's weakest assumption identifies the same core flaw: thresholds are tuned on AL-CPL and then the AL-CPL results are used as validation. This is the most load-bearing concern because the paper's headline contribution is an unsupervised method that 'achieves higher precision than existing methods,' and that claim rests almost entirely on the AL-CPL precision numbers in Table 4. Since θ=0.28 was chosen specifically because it maximizes precision on AL-CPL, the reported 1.0 precision is expected under threshold selection, not evidence of generalization. The Biology experiment does not rescue the claim: the voting algorithm performs worse than several of its own criteria and has no baseline comparison, so it cannot demonstrate superiority to existing methods on a second benchmark. Additional issues noted by the reader—e.g., the sign error in Eq. 15 and the consistently lower recall/F1—are real but secondary; the tuning circularity alone invalidates the primary empirical evidence. Therefore the REJECT verdict is appropriate, and my stress-test does not change it.","tokens_in":11910,"tokens_out":3775,"duration_ms":44374,"concrete_test":"Perform a controlled held-out evaluation on AL-CPL: split the data (e.g., by domain) into training/validation and test folds. Tune θ for RefD, BERTropy, and the voting algorithm on the validation fold only, then freeze thresholds and report precision/recall/F1 on the held-out test fold. Repeat across folds or with multiple random splits. If the Data Mining precision drops from 1.0 or no longer systematically exceeds the baselines, the central claim is an artifact of tuning on the test set. Also evaluate the Biology dataset with the AL-CPL-trained thresholds and compare the voting algorithm against the same baselines used for AL-CPL; if voting still underperforms individual criteria, the multi-criteria advantage is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that the voting algorithm achieves higher precision than existing methods—is supported only by Table 4, which reports performance on AL-CPL. But Section 4 states that the thresholds for RefD, BERTropy, and the voting algorithm were 'determined empirically using the same methodology, based on the AL-CPL dataset,' and that the voting threshold θ=0.28 was chosen because precision 'reaches its maximum value of 1.0 at θ=0.28' on that same dataset. Thus the precision 1.0 in Data Mining is a selected extreme of a threshold sweep on the test set, not an out-of-sample result. No held-out split, cross-validation, or separate validation set is described. The same circularity may also favor the proposed method over baselines if those baselines were not retuned on AL-CPL under the identical protocol. On the Biology dataset, where no comparable tuning is reported, the voting algorithm's precision (0.48) is lower than several individual criteria (e.g., HL-Ab=1.0, HL-A=0.91) and no baseline comparison is provided. Therefore the 'higher precision than existing methods' claim is not established; it is at best a fitted description of the tuning set.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an unsupervised, multi-criteria method for inferring prerequisite relations (PRs) among concepts in educational knowledge graphs. Ten criteria are defined, spanning document order, Wikipedia hyperlinks, DBpedia graph structure, course hierarchy, and BERTopic-based entropy. These binary criteria are aggregated with an equal-weight voting algorithm, and the final score is thresholded to decide the presence and direction of a PR. The method is evaluated on the AL-CPL dataset (Data Mining, Physics, Macroeconomics) against six baselines and on a Biology dataset against individual criteria. The central claim is that the voting algorithm achieves higher precision than existing methods while remaining unsupervised and scalable.","tokens_in":12212,"tokens_out":4407,"duration_ms":51688,"significance":"If the empirical claims were sound, the paper would make a useful applied contribution: it combines several heterogeneous signals for prerequisite inference without requiring labeled training data, and the individual criteria are concrete and mostly reproducible. The target application, CourseMapper, is a reasonable motivating context, and the authors are right that precision matters for downstream learning-path recommendations. However, the central empirical claim is not currently supported: the headline precision values are obtained by tuning thresholds on the same test set used for evaluation, the 'higher precision' claim is contradicted by the paper's own table in one domain, and the second experiment provides no baseline comparison. The paper's contribution is therefore at the level of a proposal requiring validation, not an established method.","major_comments":[{"comment":"The thresholds for RefD, BERTropy, and the voting algorithm are 'determined empirically using the same methodology, based on the AL-CPL dataset' (Section 4). The voting threshold θ=0.28 is chosen because 'precision reaches its maximum value of 1.0 at θ=0.28' on that same dataset. Table 4 then reports the precision of the voting algorithm on the same AL-CPL domains. This is circular: the reported Data Mining precision of 1.0 is a selected extreme of a threshold sweep over the ground-truth test set, not an out-of-sample estimate. No held-out split, cross-validation, or separate validation set is described. The comparison against baselines is therefore also biased, since the proposed method is tuned on the evaluation data while the baselines may not be similarly retuned. The central precision claim is not established until thresholds are fixed on held-out data or selected by nested validati","section":"Section 4 (Threshold Optimization) and Table 4"},{"comment":"The Abstract claims 'higher precision than existing methods,' and Section 4 says the voting algorithm 'consistently achieves higher precision.' Table 4 contradicts this: in Physics, the voting algorithm has precision 0.76, lower than Bagging-based AdaBoost's 0.87 and Active Learning's 0.80. In Data Mining, voting precision 1.0 beats all baselines, but in Physics the claimed advantage disappears. The paper acknowledges this in passing ('Bagging-based AdaBoost achieves higher precision'), but the abstract and the word 'consistently' overstate the result. The claim should be restricted to the settings where it actually holds, or the method should be modified/retuned and re-evaluated.","section":"Table 4, Physics row; Abstract; Section 4 comparative results"},{"comment":"On the Biology dataset, the voting algorithm achieves precision 0.48, which is lower than several individual criteria: HL-Ab 1.0, HL-A 0.91, HL-RCA 0.87. The voting algorithm's accuracy (0.40) is also the lowest or near-lowest in Table 5. Moreover, no baseline method is evaluated on this dataset. The conclusion that the approach 'achieves higher precision than baselines' on two benchmark datasets is therefore unsupported: the second dataset provides no baseline comparison, and the voting algorithm is not even the best individual criterion. The authors should either present baseline results on Biology, report per-dataset claims accurately, or acknowledge that the method's advantage is limited to the AL-CPL setting.","section":"Section 4, Experiment 2 and Table 5"}],"minor_comments":[{"comment":"The entropy equation has a double summation symbol with only one index. Please define the distribution over topics precisely and remove the spurious summation sign. Also, the text immediately after Eq. (16) says θ=1.33, while Section 4 says θ=1.4; please reconcile.","section":"Equation (15)"},{"comment":"The RefD formula divides by sums of cosine weights w(R_ci, c1) and w(R_ci, c2). If a related-concept set is empty or a weight is zero, the denominator is undefined. State the convention used in the implementation.","section":"Equations (5)-(7)"},{"comment":"There are many spacing/rendering artifacts: 'T emO', 'BER T ropy', 'Table1', and similar. These look like macro-related typos and should be cleaned before resubmission.","section":"Throughout"},{"comment":"No code or data are provided. Given that the evaluation is a central part of the paper and the threshold tuning is a crucial step, sharing the implementation and the exact data split would substantially improve reproducibility.","section":"General"}],"recommendation":"reject","confidential_remarks":"The paper has a useful practical motivation and a concrete set of criteria, but the evaluation is invalid as presented because thresholds are tuned on the same test set used for the headline results. The cross-domain claim is also not supported by the paper's own tables. A revision would require re-running all experiments with held-out threshold selection and, for Biology, adding baselines; those results may change the conclusions substantially. I would not invite a revision on the current basis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea—aggregate ten known PR signals into a single vote—is reasonable, but the paper's main evidence doesn't support it. The thresholds for RefD, BERTropy, and the voting algorithm are tuned on AL-CPL, and then the same AL-CPL numbers are presented as evaluation results. That precision 1.0 is a fitted artifact, not a validated finding. The paper itself admits the voting algorithm gets lower recall and F1 than the baselines, so the positive claim rests entirely on that circular precision number.\n\nTo its credit, the paper does a clean job of defining each criterion, linking them to prior work, and explaining the design choices. The Biology experiment, with expert-annotated pairs, is a good extension, and the transparency about preferring precision is defensible. But the Biology results don't help: the voting algorithm's precision (0.48) is below several of its own criteria (HL-Ab 1.0, HL-A 0.91, HL-RCA 0.87), and there are no baseline comparisons. So even outside the circular AL-CPL setup, the aggregate method is not an improvement over a single good criterion.\n\nTwo concrete technical issues add to the problem. Equation (15) gives EntPro(c) = -Σ P(x) log(1/P(x)), which is actually negative of the standard Shannon entropy because log(1/P) = -log P. That flips the sign and would reverse the BERTropy criterion's direction. Second, there's no code or data release, no error bars, and no description of a held-out split or cross-validation. The claim of \"maintaining scalability and adaptability\" isn't tested.\n\nThe multi-criteria voting idea is worth exploring, and the paper might be a useful basis for a future study with proper validation. But as it stands, the empirical claim fails, and the sign error undercuts one of the ten criteria. A serious referee could help the authors fix these issues, so it should not be desk-rejected. I would not cite it in its current form, but I'd bring it to a reading group to discuss the threshold-tuning pitfall.","headline":"A reasonable multi-criteria voting idea, but the headline precision is a fitted artifact from tuning thresholds on the same AL-CPL test set.","tokens_in":12714,"tokens_out":3169,"would_cite":false,"duration_ms":34233,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An unsupervised voting method over ten evidence criteria can infer which concepts must be learned first, with higher precision than existing approaches.","keywords":["prerequisite relationships","educational knowledge graphs","unsupervised learning","voting algorithm","Wikipedia hyperlinks","BERTopic entropy","MOOC","CourseMapper"],"falsifier":"Re-run the AL-CPL evaluation with a clean split: tune the RefD, BERTropy, and voting thresholds on Data Mining only, then report precision on Physics and Macroeconomics, or use cross-validation. If the voting method's precision drops far below the reported 1.0, 0.76, and 0.65, the precision advantage is an artifact of tuning on the test set.","tokens_in":11815,"feed_emoji":"🎓","tokens_out":11115,"duration_ms":106370,"temperature":0.7,"pith_summary":"The paper tries to show that prerequisite links between concepts—the “take this before that” structure of a course—can be inferred automatically and without labeled data by pooling ten heterogeneous signals and letting them vote. The method combines document order, Wikipedia hyperlink placement, DBpedia categories, link-ratio statistics, and topic entropy, then normalizes and thresholds the vote to decide both whether a prerequisite exists and in which direction. On two public datasets it reports higher precision than most existing approaches, while accepting lower recall. Because labeled prerequisite data is scarce and manual annotation is inconsistent, a reliable unsupervised method would let MOOC platforms add learning sequences to their knowledge graphs at scale.","feed_headline":"Voting on ten signals finds concept prerequisites with high precision","feed_subtitle":"Combines document, Wikipedia, graph, and text evidence to order learning concepts with no labeled data.","key_machinery":"The voting algorithm: ten binary criteria are evaluated in both directions for every concept pair; two one-hot arrays record which criteria are satisfied for c1→c2 and for c2→c1; the normalized difference of their sums is compared with an empirically set threshold (θ = 0.28) to decide both presence and direction of a prerequisite. The load-bearing idea is that false signals from any single criterion cancel out when many independent features agree.","core_discovery":"The paper's central claim is that an equal-weight vote over ten heterogeneous criteria can identify concept prerequisites in educational knowledge graphs more reliably—in terms of precision—than any single criterion or most existing baselines. It defines criteria from document order, Wikipedia hyperlink placement (articles, abstracts, related-concept abstracts, reference distance, inbound/outbound ratio), DBpedia categories and super-categories, CourseMapper channel hierarchy, and BERTopic topic entropy. For each ordered concept pair it builds a one-hot vector of satisfied criteria, sums the two directions, normalizes the difference to [-1, 1], and infers c1 as a prerequisite of c2 when the","pith_inferences":["Equal weights are a design choice; per-domain weights learned from a small labeled set could raise recall without sacrificing precision—an idea the paper lists as future work.","The high precision of the voting method suggests it could be used as a candidate generator for human-in-the-loop annotation, where only the top-scored pairs are reviewed.","The threshold θ = 0.28 was tuned on one dataset and may not transfer; testing on new domains with different concept spaces would clarify whether the method's advantage comes from the features or from the tuning.","The CMH criterion's strong showing implies that platform-internal ordering is a cheap, high-value signal for prerequisite inference, and systems that export content into flat structures lose it by construction."],"forward_implications":["Educational knowledge graphs such as CourseMapper's can be enriched with prerequisite links automatically, enabling sequence-aware learning and recommendations without manual annotation.","Because the algorithm favors precision over recall, downstream systems can treat inferred prerequisite links as high-confidence edges, while missing links remain unlabeled rather than wrong.","The method needs no training labels, so it can be applied to new courses or domains as long as concepts have Wikipedia articles or DBpedia categories.","Adding the course-hierarchy criterion (CMH) to the vote improves detection on structured content; flattening content into a single channel removes that signal.","The comparative results suggest that a precision-first, low-recall extractor is a viable trade-off when the cost of misleading a learner outweighs the cost of omitting a prerequisite."],"supporting_citations":[{"why":"Cited as a source of the AL-CPL dataset and the textbook concept-map extraction task on which the method is evaluated.","marker":"[18]"},{"why":"Defines the reference-distance criterion and serves as a precision baseline for comparison.","marker":"[9]"},{"why":"Cited as a source of the AL-CPL dataset and supplies the abstract-hyperlink criterion (HL-Ab).","marker":"[10]"},{"why":"Cited as a source of the AL-CPL dataset and supplies the active-learning baseline.","marker":"[11]"},{"why":"Supplies the three structural assumptions—transitivity, asymmetry, no self-connection—that the voting algorithm adopts.","marker":"[4]"},{"why":"Supplies the BERTopic entropy criterion and a BERT-based approach to prerequisite extraction.","marker":"[5]"},{"why":"Provides the Biology textbook dataset and the hierarchical-ordering idea behind the CMH criterion.","marker":"[17]"},{"why":"Supplies the WikiCPRL weakly supervised baseline used in the precision comparison.","marker":"[21]"},{"why":"Supplies the AdaBoost ensemble-learning baseline used in the precision comparison.","marker":"[22]"}],"fun_headline_variants":["Ten signals vote to map learning order without labels","Unsupervised voting ranks concept prerequisites precisely","Multi-criteria vote orders MOOC concepts with no labels","Precision boost: ten-criteria vote for prerequisite links","Voting over ten features reveals concept prerequisites"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The reported precision figures rest on thresholds chosen in the threshold-optimization experiments on the AL-CPL dataset and then reported on that same dataset; if that tuning counts as peeking at the test set, the precision 1.0 is a fitted artifact.","fun_headline_variants_meta":{"raw":{"variants":["Ten signals vote to map learning order without labels","Unsupervised voting ranks concept prerequisites precisely","Multi-criteria vote orders MOOC concepts with no labels","Precision boost: ten-criteria vote for prerequisite links","Voting over ten features reveals concept prerequisites"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000784,"raw_usage":{"total_tokens":3257,"prompt_tokens":663,"completion_tokens":2594,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":407,"completion_tokens_details":{"reasoning_tokens":2522}},"tokens_in":407,"tokens_out":2594,"duration_ms":18317,"temperature":1.0,"reasoning_tokens":2522,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T05:41:31.635560+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the AL-CPL evaluation with a clean split: tune the RefD, BERTropy, and voting thresholds on Data Mining only, then report precision on Physics and Macroeconomics, or use cross-validation. If the voting method's precision drops far below the reported 1.0, 0.76, and 0.65, the precision advantage is an artifact of tuning on the test set.","supporting_citations":[{"cited_title":"In: Proceedings of the 25th acm international on conference on information and knowledge management","cited_arxiv_id":null,"evidence_quote":"Cited as a source of the AL-CPL dataset and the textbook concept-map extraction task on which the method is evaluated."},{"cited_title":"In: Proceedings of the 2015 conference on empirical methods in natural language processing","cited_arxiv_id":null,"evidence_quote":"Defines the reference-distance criterion and serves as a precision baseline for comparison."},{"cited_title":"In: Proceedings of the AAAI Conference on Artificial Intelligence","cited_arxiv_id":null,"evidence_quote":"Cited as a source of the AL-CPL dataset and supplies the abstract-hyperlink criterion (HL-Ab)."},{"cited_title":"Active Learning of Strict Partial Orders: A Case Study on Concept Prerequisite Relations","cited_arxiv_id":"1801.06481","evidence_quote":"Cited as a source of the AL-CPL dataset and supplies the active-learning baseline."},{"cited_title":"ACM Computing Surveys (2025)","cited_arxiv_id":null,"evidence_quote":"Supplies the three structural assumptions—transitivity, asymmetry, no self-connection—that the voting algorithm adopts."},{"cited_title":"Mathematical Problems in Engineering2021(1), 3510402 (2021)","cited_arxiv_id":null,"evidence_quote":"Supplies the BERTopic entropy criterion and a BERT-based approach to prerequisite extraction."},{"cited_title":"In: Proceedings of the 18th Workshop on Innovative Use of NLP for Build- ing Educational Applications (BEA 2023)","cited_arxiv_id":null,"evidence_quote":"Provides the Biology textbook dataset and the hierarchical-ordering idea behind the CMH criterion."},{"cited_title":"In: Asia-Pacific Web (APWeb) and Web-Age Information Management (WAIM) Joint International Conference on Web and Big Data","cited_arxiv_id":null,"evidence_quote":"Supplies the WikiCPRL weakly supervised baseline used in the precision comparison."},{"cited_title":"In: 2020 16th International Conference on Mobility, Sensing and Networking (MSN)","cited_arxiv_id":null,"evidence_quote":"Supplies the AdaBoost ensemble-learning baseline used in the precision comparison."}],"review_version":1}