{"id":"7674ed7e-823f-4504-a938-74d66f65b7c2","arxiv_id":"2501.00461","paper_version":1,"verdict":"REJECT","confidence":"LOW","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A knowledge-graph ranker using PinSage and text embeddings reportedly beats TF-IDF for engineer recommendation, but the evaluation leaks target labels and reports impossible hit ratios.","lead":"The paper proposes a knowledge-graph and graph-neural-network ranker that recommends support engineers for customer tickets, claiming large gains over TF-IDF, Random Forest, and XGBoost. The evaluation is underspecified, contains impossible metric values, and appears to leak resolution labels, so the headline results are not supported.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation leaks the target: Section III-A includes responder IDs in the input 'Communication summary' for historical incidents, so the reported KG gains over TF-IDF likely reflect lookup of the engineers whose responses are already in the input; without a leakage-controlled temporal split, the…","rationale":"The paper's central claim is that a Knowledge Graph with PinSage and VLLM embeddings significantly improves engineer ranking over TF-IDF, Random Forest, and XGBoost. For that claim to be supported, the evaluation must measure performance on incidents for which the correct engineer is not already knowable from the input. That condition fails at the feature-construction level: Section III-A explicitly places the IDs of responding engineers inside the communication summary used as an input feature, while the same section explains that engineer responses are used to determine which engineer resolved the incident. Because the evaluation matches recommendations against historical resolution data, the target is present in the input features for every test incident. The reported top-50 improvement from 0.48 (TF-IDF) to 0.64 (KG) may simply reflect the KG's greater ability to exploit this lookup, rather than any genuine understanding of expert suitability. The reader's weakest_assumption identifies exactly this leakage, and I agree with that assessment. The paper also lacks the experimental protocol needed to assess generalization: no train/test split, no temporal split, no hyperparameter details, no error bars, no code, and no data. The internal inconsistency in Table 1—where the 100K row reports top-200 = 0.55 below top-100 = 0.59—is additional evidence that the reported numbers cannot be taken at face value. I am not objecting to the architectural idea itself; the knowledge-graph-plus-VLLM approach is plausible. The objection is to the correctness of the evidence supporting the central empirical claim. A single temporal, leakage-free holdout test would settle whether the claimed improvement is real or an artifact of target leakage.","tokens_in":5949,"tokens_out":3431,"duration_ms":35824,"concrete_test":"The decisive check is a leakage-controlled temporal holdout: rebuild the evaluation so that inputs contain only features available at prediction time. Remove all responder/user IDs and incident-ID joins from the communication summary and graph construction, train on incidents created before a cutoff date D, and test only on incidents created after D (e.g., D = 2020-01-01 within the 2019-2020 window). Report top-50/100/200 for TF-IDF and the KG/PinSage model on the same holdout. If the KG margin over TF-IDF collapses or becomes statistically negligible, leakage is confirmed. As a sanity check in the same protocol, verify that top-200 is at least top-100; the printed 100K values violate monotonicity and require correction before any comparison is meaningful.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the ranking generalize to incidents where the correct engineer is not already known. Section III-A defines the input 'Communication summary' as including 'an anonymized version of all communications to/from customers and IDs of responding engineers,' and the same section states that engineer responses are used 'to inform which engineer resolved which incident.' Processor IDs are also used for joining and are part of the data sources. Evaluation is performed on historical incidents, with recommendations matched to the engineers who actually resolved those incidents. Thus, for each test incident, the input contains the IDs of the engineers who responded to that incident, and the target is drawn from that same set. Any model with even weak memorization capacity can place a responder in the top-k by copying or ranking highly the IDs present in the input. TF-IDF also has access to this leakage, which likely explains its unusually high absolute hit ratios (0.48 top-50 on 10K examples) and makes the claimed relative improvement of the Knowledge Graph model uninterpretable. The paper provides no train/test split, no temporal split, no hyperparameters, no code, and no data, so one cannot rule out that the margin is entirely leakage. Furthermore, Table 1's 100K row reports top-200 = 0.55 below top-100 = 0.59, which is impossible for cumulative hit ratios and indicates a reporting or evaluation error independent of the leakage concern.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a learning-to-rank system for recommending engineers to resolve product-support tickets. The system builds a knowledge graph whose nodes include incidents, engineers, knowledge-base articles (KBAs), and swarms, and it uses PinSage, a graph neural network, over VLLM-generated text embeddings to produce ranked lists of engineers for incoming incidents. The authors compare the proposed system against TF-IDF, Random Forest, and XGBoost on samples of 10K and 100K historical incidents, reporting top-50, top-100, and top-200 hit ratios. Their central claim is that incorporating holistic contextual data through the knowledge graph significantly improves engineer-ranking accuracy over traditional baselines. The evaluation, however, is undermined by label leakage in the input definition, internal inconsistencies in the reported hit-ratio table, and an underspecified experimental protocol.","tokens_in":6197,"tokens_out":3234,"duration_ms":32619,"significance":"If the reported results were valid, the paper would demonstrate a practically useful application of knowledge-graph embeddings to expert matching in enterprise support, with the potential to reduce ticket-resolution times. The combination of PinSage with transformer-based text embeddings is not conceptually novel, but a rigorous evaluation in this domain could still be a useful engineering contribution. Unfortunately, the paper does not provide machine-checked proofs, reproducible code, or data, and the central empirical claim rests on a single table that contains impossible values and is likely corrupted by leakage. As presented, the work does not currently provide reliable evidence for its main hypothesis, so its significance is limited to a description of a plausible system architecture rather than a validated result.","major_comments":[{"comment":"The evaluation is subject to direct label leakage. Section III-A defines the input 'Communication summary' as including 'an anonymized version of all communications to/from customers and IDs of responding engineers,' and the same section states that engineer responses are used 'both as input features for engineers and to inform which engineer resolved which incident.' The prediction target for each historical incident is the engineer (or engineers) who actually resolved the incident, and the input for that same incident contains the IDs of engineers who responded to it. Any model can therefore place a responding engineer in the top-k by ranking the IDs that already appear in the input, and the reported hit ratios in Table 1 become uninterpretable as measures of generalization to new incidents. The authors must remove all engineer IDs and any text derived from post-incident communications from the input features for test incidents, and they should use a temporal split (e.g., train on incidents before a cutoff date and test on incidents after it) to prevent this leakage.","section":"Section III-A, Data Sources"},{"comment":"The reported hit ratios are internally inconsistent. For the 100K sample, the TF-IDF row gives top-50 = 0.35, top-100 = 0.59, and top-200 = 0.55, and the Knowledge Graph row gives top-50 = 0.70, top-100 = 0.65, and top-200 = 0.78. Since top-k hit ratio is a cumulative measure, the value must be non-decreasing in k; a top-200 ratio below the top-100 ratio (and a top-100 ratio below the top-50 ratio) is impossible. This is a load-bearing error because it calls into question the correctness of the entire evaluation pipeline, including the 10K sample results that support the central claim. The authors must correct the table and re-run the evaluation before any comparison between models can be taken seriously.","section":"Table 1, 100K examples rows"},{"comment":"The experimental protocol is underspecified to the point that the reported numbers cannot be independently validated. The text says the dataset was 'sampled randomly into 2 sets – one set containing 10K examples and the other set containing 100K examples,' but it never specifies a train/test split, the proportion of data used for training versus evaluation, hyperparameter values for the GNN or the baselines, or whether the 10K and 100K sets are independent samples or nested. There are also no error bars or confidence intervals, so the reader cannot assess the significance of the claimed improvements. Without this information, the manuscript does not provide enough detail to reproduce even the baseline numbers, let alone the proposed system.","section":"Section IV-A, Evaluation protocol"}],"minor_comments":[{"comment":"The phrase 'triple loss function' appears in Section III-B; the correct term is 'triplet loss function.'","section":"Abstract / Section III-B"},{"comment":"The sentence 'The results obtained proves that...' should read 'The results obtained prove that...'.","section":"Abstract"},{"comment":"The legend label 'Knowledge Graph withembeddings' is missing a space; it should be 'Knowledge Graph with embeddings'.","section":"Figures 3 and 4"},{"comment":"The layout of Table 1 is ambiguous about which numbers correspond to which model for the 10K and 100K samples; the rows for TF-IDF, Random Forest, XGBoost, and the proposed system are not clearly aligned, and the 'ML Model' header placement makes it difficult to read. A multi-index table or a clearer row separation would help.","section":"Table 1"},{"comment":"The phrase 'timeline analysis were limited' should be 'timeline analysis was limited,' and the sentence 'these models were very much less than TF-IDF' is informal; consider reporting exact values in the text instead of relying solely on the table.","section":"Section IV-A"}],"recommendation":"reject","confidential_remarks":"The paper reads as an engineering report rather than a validated research contribution. The central claim is supported only by a table that is internally inconsistent and likely affected by label leakage, and the manuscript does not include code, data, or a sufficiently detailed protocol. Even if the evaluation were redone, the novelty of applying PinSage with pre-trained text embeddings to a support-ticket domain is modest, so I would not recommend this paper for the research track of this venue without a substantial, rigorous experimental overhaul."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a paper that would be worth reading as a cautionary example, but not as a result. The central empirical claim—that a knowledge graph with PinSage and VLLM embeddings beats TF-IDF for engineer ranking—is not supported. The input features for historical incidents include the IDs of the engineers who responded to those incidents, and those same engineers are the prediction targets. That's leakage, and it's in Section III-A explicitly. Any model with even weak memory can put a responder in the top-k by ranking IDs already present in the input. The reported absolute hit ratios, like TF-IDF at 0.48 top-50 on 10K examples, are unusually high and consistent with this leakage rather than with a genuinely generalizable routing model.\n\nThe table itself is also broken: in the 100K row, top-200 is 0.55 while top-100 is 0.59 for TF-IDF, and for the KG model top-100 (0.65) is below top-50 (0.70). Cumulative hit ratios can't decrease as k grows. That's not a minor typo; it means the reported numbers can't be taken at face value.\n\nWhat the paper does have going for it: the problem is real, the proposed architecture (VLLM embeddings feeding a PinSage-style GNN over a graph with swarm edges) is a reasonable adaptation of recommender-system ideas to expert routing, and the authors correctly identify that traditional TF-IDF can't capture collaboration structure. The literature background is mostly appropriate. The writing is clear about the data sources and the ontology.\n\nBut the evaluation protocol is absent: no train/test split, no temporal split, no hyperparameters, no error bars, no code or data. The paper claims end-to-end training with negative sampling but gives no details. This is not a paper where a reviewer can check the numbers or reproduce the result.\n\nMy take: this is an industry work-in-progress note, not a research paper. The idea deserves a proper evaluation, but as written the central claim collapses. I would not cite it. I might bring it to a reading group as a case study in label leakage, because it's a textbook example. If the authors return with a leakage-controlled temporal split and corrected numbers, the underlying direction could be worth a look.\n\nDon't send this to peer review; desk reject and invite a resubmission with a real evaluation.","headline":"A plausible architecture for ticket routing, but the evaluation leaks the target and the headline numbers are internally inconsistent, so the central claim doesn't stand.","tokens_in":6708,"tokens_out":3138,"would_cite":false,"duration_ms":28321,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A knowledge graph that links engineers, incidents, KBAs, and swarms ranks the right engineer for a support ticket with a top-50 hit ratio of 0.64, beating TF-IDF's 0.48 on a 10K-sample test.","keywords":["learning-to-rank","knowledge graph","PinSage","graph neural network","very large language model embeddings","expert matching","support ticket resolution","top-k hit ratio"],"falsifier":"Re-run the 10K-sample benchmark with a version of the data from which all engineer IDs and response text have been removed from the input features, keeping only the incident description, component, and KBAs. If the Knowledge Graph's top-50 hit ratio falls to near the TF-IDF baseline of 0.48, label leakage is the likely cause; if it stays near 0.64, the graph context itself is doing the work.","tokens_in":5726,"feed_emoji":"🧠","tokens_out":5797,"duration_ms":51730,"temperature":0.7,"pith_summary":"The paper sets out to show that a learning-to-rank system for support-ticket resolution can be improved by representing engineers, incidents, knowledge-base articles, components, and past swarms as a knowledge graph and by generating embeddings with very large language models and a graph neural network. It claims that this holistic context captures relationships that traditional TF-IDF and tree-based models cannot, and it reports consistently higher top-k hit ratios on 10K and 100K incident samples. A sympathetic reader would take the central hypothesis to be that graph-structured context, not just text similarity, is what moves the ranking metric. If true, the result would give a practical path to faster expert assignment inside a large enterprise support organization.","feed_headline":"Knowledge graph ranks support engineers far better than TF-IDF","feed_subtitle":"Adding graph relationships among engineers, incidents, and articles lifts top-50 hits from 0.48 to 0.64 on 10K samples.","key_machinery":"The load-bearing mechanism is a heterogeneous knowledge graph whose nodes are engineers, incidents, KBAs, and components, with edges encoding relationships such as 'engineer resolved incident,' 'engineer wrote KBA,' and 'engineers swarmed together.' Text from incidents, KBAs, and components is converted into vector embeddings by very large language models; those embeddings and one-hot component vectors are fed into PinSage, a scalable graph convolutional recommender algorithm, which produces updated engineer embeddings. A ranking module then scores each engineer against a new incident using a triplet loss function and returns a ranked list. The claim is that this jointly learned representation captures expertise and collaboration structure that static similarity measures cannot.","core_discovery":"The paper claims that the proposed Knowledge Graph with PinSage and VLLM embeddings outperforms TF-IDF, Random Forest, and XGBoost on the engineer-ranking task. For the 10K-sample evaluation, top-50/100/200 hit ratios are 0.64/0.77/0.85 for the proposed system versus 0.48/0.58/0.68 for TF-IDF; for the 100K-sample evaluation the paper reports 0.70/0.65/0.78 against TF-IDF's 0.35/0.59/0.55. The paper interprets these numbers as evidence that additional contextual data around which cases an engineer has solved, including swarm participation, yields significantly better recommendations than the benchmark models. It also notes that the training time and GPU cost are substantially higher for the proposed system.","pith_inferences":["Editorial inference: the reported gap may be partly explained by the input construction, since the 'Communication summary' feature contains responding-engineer IDs and the resolution labels are derived from the same communications; a clean ablation that removes engineer-identifying text would tell whether the graph itself is responsible for the lift.","Editorial inference: the same graph-embedding-plus-VLLM design could transfer to other expert-routing problems, such as code-review assignment, medical second opinions, or legal case triage, wherever a collaboration graph and free-text records exist.","Editorial inference: a temporal split of the data (train on incidents before a cutoff, test after) would test whether the model is learning stable expertise signals or recency artifacts, and would better approximate deployment."],"forward_implications":["For the 10K-sample evaluation, the proposed system's top-50/100/200 hit ratios (0.64/0.77/0.85) exceed TF-IDF's (0.48/0.58/0.68), and the same pattern holds on the 100K sample.","A deployed version could rank engineers for a new incident even before any engineer is assigned, using incident text, component, and KBA similarity through the graph.","Swarm-aware ranking becomes feasible: when a swarm is already active, the system can use embeddings of current engineers to find complementary engineers.","Because the pipeline is end-to-end and parametric, updating KBAs, incident records, and user data would let the system be retrained rather than reweighted by hand."],"supporting_citations":[{"why":"PinSage is the graph neural network algorithm used in the ranking module to produce engineer embeddings.","marker":"[11]"},{"why":"BERT is cited as the representative very large language model used to generate text embeddings for incidents and KBAs.","marker":"[1]"},{"why":"GraphSage is cited as the inductive GNN basis that motivates the choice of PinSage for the knowledge graph.","marker":"[10]"},{"why":"Neural structured learning is cited as an alternative graph-regularized feature-generation approach that informs the architecture discussion.","marker":"[12]"}],"fun_headline_variants":["Knowledge graphs boost support engineer ranking by 33% over TF-IDF","Graph embeddings improve ticket-to-engineer matching in support","Learning to rank engineers with knowledge graphs beats TF-IDF","Knowledge graph rankings improve engineer selection for swarms","Knowledge graphs lift engineer top-50 hits from 0.48 to 0.64"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking gain depends on assuming that the 'Communication summary' input does not reveal the identity of the engineers who resolved the incident; if it does, the hit ratios are inflated by label leakage.","fun_headline_variants_meta":{"raw":{"variants":["Knowledge graphs boost support engineer ranking by 33% over TF-IDF","Graph embeddings improve ticket-to-engineer matching in support","Learning to rank engineers with knowledge graphs beats TF-IDF","Knowledge graph rankings improve engineer selection for swarms","Knowledge graphs lift engineer top-50 hits from 0.48 to 0.64"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000778,"raw_usage":{"total_tokens":3468,"prompt_tokens":1004,"completion_tokens":2464,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":2376}},"tokens_in":620,"tokens_out":2464,"duration_ms":17092,"temperature":1.0,"reasoning_tokens":2376,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:50:12.167698+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 10K-sample benchmark with a version of the data from which all engineer IDs and response text have been removed from the input features, keeping only the incident description, component, and KBAs. If the Knowledge Graph's top-50 hit ratio falls to near the TF-IDF baseline of 0.48, label leakage is the likely cause; if it stays near 0.64, the graph context itself is doing the work.","supporting_citations":[{"cited_title":"Since, the total number of engineering experts are almost constant compared to the incoming incidents, a ranking all the engineers can be done with minimal memory resources","cited_arxiv_id":null,"evidence_quote":"PinSage is the graph neural network algorithm used in the ranking module to produce engineer embeddings."},{"cited_title":"Graph Convolutional Neural Networks for Web-Scale Recommender Systems","cited_arxiv_id":"1806.01973","evidence_quote":"Neural structured learning is cited as an alternative graph-regularized feature-generation approach that informs the architecture discussion."}],"review_version":1}