{"id":"839612d8-2a5e-488c-b234-4b16b08cac0a","arxiv_id":"2505.14156","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A session graph serialized into symbolic text, plus self-supervised graph pre-training tasks, lets an LLM outperform existing session search rankers on AOL and Tiangong-ST.","lead":"The paper converts search session histories into text-based symbolic graphs and feeds them to a large language model, adding self-supervised graph tasks so the model can rank documents better. It shows a practical way to make LLMs reason about graph-structured user behavior, with gains on two search log datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The AOL SOTA claim is evaluated on a single 1,000-session subsample with no variance reporting, and it is not clear that baselines were scored on the same subsample; the reported edge over HEXA may be a sampling artifact.","rationale":"The reader's stated weakest assumption concerns whether a chronological flat serialization preserves graph topology and whether 'yes' logits are valid relevance scores. Those are reasonable methodological questions, but the empirical comparison is the load-bearing part of the central claim: even a theoretically imperfect serialization could work well in practice. The more decisive vulnerability is the AOL evaluation protocol. The paper evaluates SGR on a single randomly selected 1,000-session subsample with no confidence intervals or repeated draws, and it does not explicitly state that all baselines were evaluated on that same subsample. Because the reported improvement over HEXA is small, this single design choice could invalidate the headline result. This is a concrete, checkable concern rather than an internal inconsistency, so it does not by itself demand rejection; it strengthens the case for the reader's CONDITIONAL verdict. I therefore recommend no change to the verdict, but the condition should include rerunning the comparison on the full AOL test set or on multiple fixed subsets.","tokens_in":16162,"tokens_out":5488,"duration_ms":60948,"concrete_test":"Re-run the full Table 2 protocol on the complete AOL test set (29,369 sessions) for SGR and all baselines using identical candidate sets and the same data preprocessing, computing MAP/MRR/NDCG with trec_eval; if full evaluation is infeasible, draw 10 non-overlapping 1,000-session subsets with fixed seeds and report mean plus standard deviation for every method. If SGR's MAP margin over HEXA is not consistently positive across subsets, or if it disappears on the full test set, the SOTA claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.4 states: 'Due to computational constraints, we randomly selected 1,000 sessions from the AOL test set for evaluation.' The paper does not say whether all Table 2 baselines were re-run on exactly this same subset, how the subset was drawn, or how many random draws were used. If the HEXA, COCA, RICR, and ASE numbers are taken from prior full-test publications, while SGR is scored on a 1,000-session subsample, then the comparison is not apples-to-apples and the claimed paired t-test with Bonferroni correction is not valid. The absolute AOL MAP margin over HEXA is only about 0.016, which is small relative to likely session-level variance; a single subsample could easily create or erase this margin. The central claim that SGR is state-of-the-art on AOL therefore rests on an unverified sampling assumption, rather than on the symbolic graph design itself. The Tiangong-ST results and ablations are informative, but they do not resolve whether the AOL headline comparison is fair.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Symbolic Graph Ranker (SGR), a method that converts a session search history into a heterogeneous graph, serializes that graph into symbolic text via hand-designed grammar rules, and feeds the text to an LLM. Three self-supervised symbolic learning tasks (link prediction, node content generation, and generative contrastive learning) are used before listwise fine-tuning, where the probability of the 'yes' token is used as the relevance score. Experiments on AOL and Tiangong-ST compare SGR with BM25, MonoT5, RICR, COCA, ASE, and HEXA, reporting state-of-the-art MAP/MRR/NDCG numbers on AOL and competitive improvements on Tiangong-ST. The paper also reports ablations for each pre-training task, session-length robustness, data-efficiency results, and a perplexity curve during pre-training.","tokens_in":16355,"tokens_out":5412,"duration_ms":48894,"significance":"If the results hold, SGR is a meaningful advance: it combines the word-level semantic modeling of LLMs with the structural information of session graphs, and it does so with relatively little training data, matching a fully trained COCA with only 10% of the training data. The paper releases code, reports ablations for each of the three symbolic learning tasks, and compares against several external baselines, which are strengths. The main limitation is that the headline AOL result is evaluated on a single 1,000-session subsample, and the paper does not document whether the baselines were scored on the same subsample; this directly affects the central state-of-the-art claim. The Tiangong-ST results and the ablations partially mitigate the risk, but they do not fully resolve the AOL comparison issue.","major_comments":[{"comment":"The AOL evaluation is performed on only 1,000 randomly selected sessions from a 29,369-session test set, but the paper does not report the random seed, the number of random draws, or whether all baselines in Table 2 were re-run on exactly the same 1,000-session subset. If the baseline numbers (HEXA 0.5700 MAP, COCA 0.5649, etc.) are taken from prior full-test-set publications while SGR is scored on a subsample, the comparison is not apples-to-apples and the paired t-test with Bonferroni correction is not valid, because the paired structure across systems on the same sessions is not established. The reported margin over HEXA (0.5859 vs 0.5700 MAP, about 2.8% relative) is small enough that a single subsample could create or erase it. Please specify the evaluation protocol, report variance or confidence intervals, and either re-run all baselines on the identical subset or evaluate SGR on the full AOL test set.","section":"§4.4, Table 2"},{"comment":"The ablation 'SGR w/o SG' is described only as 'the session is represented by sequences strung together with delimiters,' with graph information omitted. The exact input format for this condition is not specified: it is unclear whether the same chronological edge order is used, whether edge-type symbols such as '<click on>' are removed, and whether the same node text is retained. Without this detail, the attribution of the performance drop to the loss of graph structure, rather than to a change in prompt format or tokenization, is not fully established. Please provide the precise template of the w/o SG input.","section":"§5.2, Table 4"},{"comment":"The pre-training link-prediction objective and the downstream ranking objective share the same input construction (the symbolic graph text followed by a query-document pair) and the same yes/no token probability. Consequently, the link-prediction pre-training is closely aligned with the downstream task, and the improvement from 'Link' in Table 3 may reflect direct task alignment rather than general graph-structure comprehension. The paper should explicitly discuss this alignment and provide evidence that the pre-training stage, particularly link prediction, teaches topology rather than only the surface form of the symbolic language. The perplexity curve in Figure 5 (right) is not sufficient evidence for graph comprehension, because training perplexity can decrease even when the model merely memorizes the symbolic format. A validation-set perplexity curve, or a transfer experiment in which link prediction is replaced by a less directly aligned task, would strengthen the interpretation.","section":"§3.4.1 and §3.5"}],"minor_comments":[{"comment":"The random selection of 1,000 AOL test sessions is not reproducible without a stated seed; please report the seed or the sampling procedure.","section":"§4.4"},{"comment":"Only AdamW, two epochs, and LoRA are mentioned as hyperparameters. Please list learning rate, LoRA rank and alpha, batch size, maximum sequence length, negative sampling ratio for link prediction, and masking strategy for node content generation, or state that they are fully specified in the released code.","section":"§4.4"},{"comment":"There is a typo in 'Concretley' in the second-to-last paragraph; it should be 'Concretely'.","section":"§2.1"},{"comment":"References [8] and [9] appear to be the same CIKM 2022 paper by Chen et al., and reference [44] is cited as 'LLaMa-7B' but actually corresponds to the Llama 2 technical report; please reconcile these citations.","section":"References"},{"comment":"The text says 'the logits of the ''yes'' answer token p(X_j)' and then uses p(X_j) in the softmax denominator; please clarify that p(X_j) is the softmax-normalized probability of the 'yes' token, not a raw logit, to avoid confusion.","section":"§3.5 and Eq. (4)"},{"comment":"The perplexity plot shows only training-time perplexity during symbolic learning; adding a validation split or downstream ranking metrics would make the claim of successful symbolic-graph comprehension more convincing.","section":"Figure 5 (right)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be the camera-ready version of a WWW 2024 paper. If this is being considered as a journal submission, the editor should verify that it contains substantial new material beyond the conference version, since the scope and claims appear identical. The main technical risk is the AOL subsampling issue; the authors should be asked to provide a complete evaluation protocol before the state-of-the-art claim can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short take: the paper's headline is that SGR beats HEXA by 1–4% on session search, but the AOL comparison rests on a single 1,000-session subsample with no seed or variance, and the paper never says whether the baselines were scored on that same subsample. That makes the SOTA claim on AOL unverified, not false.\n\nWhat's actually new: this is the first application of graph-to-text serialization to session search that I know of, and the three self-supervised text tasks are a sensible package for teaching an LLM the symbolic grammar. The ablations in Tables 3 and 4 are the strongest part: each component contributes, and the w/o SG drop is large enough to show the graph structure itself matters, not just extra text. The efficiency claim—SGR matching a fully trained COCA with 10% of the data—is worth taking seriously, though the data-efficiency experiment shares the same sampling weakness.\n\nThe soft spots: (1) The AOL evaluation. Randomly selecting 1,000 sessions from a 29,369-session test set, without reporting the seed, the number of draws, or per-session variance, is not enough to support a paired t-test against baselines whose scores may come from the full test set. The absolute MAP margin over HEXA is 0.016, small enough that sampling noise could easily flip it. This is a real flaw in the central claim, not a nitpick. (2) The perplexity plot in Figure 5 is weak evidence for graph comprehension; any training run shows PPL dropping. (3) Hyperparameters are only partially listed, and I haven't verified the code link.\n\nTiangong-ST results are cleaner (full test set) but the gains are smaller, with NDCG@1 not significant. So the method is a modest improvement, not a breakthrough. There is also a partial circularity concern: link prediction and the ranking objective use the same 'yes' logit, so pre-training is closely aligned with the downstream task. The ablations and external baselines largely mitigate this, but the framing of the pre-training tasks as general 'graph comprehension' is a bit inflated.\n\nOverall, the paper is coherent, well-organized, and honest about computational constraints. The idea is novel enough for the subfield, and the ablations give it a real empirical core. I'd send it to review, but I'd ask for a full AOL test evaluation (or a properly randomized multi-draw subsample with all baselines re-run on the same sessions) and variance estimates before trusting the SOTA claim.","headline":"Solid session-search paper with a real sampling problem under its headline AOL result.","tokens_in":16875,"tokens_out":3231,"would_cite":true,"duration_ms":29708,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By translating each search session's interaction graph into symbolic text, an LLM can re-rank documents better than the compared session search models on AOL and Tiangong-ST.","keywords":["session search","document ranking","large language models","graph-to-text","symbolic graph","self-supervised learning","link prediction","contrastive learning"],"falsifier":"If randomizing the order of the serialized edges while keeping the same nodes and text leaves ranking performance essentially unchanged, or if replacing the symbolic graph with a plain natural-language session description gives equal performance, then the claimed topological benefit is not what carries the result.","tokens_in":15974,"feed_emoji":"🔎","tokens_out":4872,"duration_ms":49301,"temperature":0.7,"pith_summary":"This paper proposes Symbolic Graph Ranker (SGR), a method that turns the interaction history of a search session into a heterogeneous graph, serializes that graph into symbolic text, and feeds the text to a large language model to score candidate documents. The authors claim that this combination gives an LLM both fine-grained word-level semantics and coarse-grained user-behavior structure, and that three self-supervised symbolic tasks—link prediction, node content generation, and generative contrastive learning—teach the LLM to read the graph format. On the AOL and Tiangong-ST benchmarks, SGR outperforms every compared baseline, including the graph-based HEXA model, and reaches the level of a fully trained COCA model using only about 10 percent of the training data. The point of the work is to show that graph structure in search sessions can be handled entirely inside a text-based LLM rather than by a separate graph encoder.","feed_headline":"LLM reads search sessions as text graphs and beats all baselines","feed_subtitle":"On AOL and Tiangong-ST, SGR tops graph-based HEXA and matches a fully trained COCA with one-tenth the data.","key_machinery":"The load-bearing object is the symbolic graph grammar SGC: a node is written as $([\\text{type}][\\text{id}], \\text{text})$, an edge is written as $SGC_v(v_1) <\\text{edge type}> SGC_v(v_2)$, and the whole session graph is the chronological concatenation of all edge strings. This flat text is what the LLM sees, so the graph is carried by edge ordering and repeated node identifiers rather than by an adjacency matrix. The three symbolic pretraining tasks (link prediction, node content generation, and generative contrastive learning) are what make the LLM sensitive to that structure, and the final ranking reuses the link-prediction format with a target query-document pair.","core_discovery":"The central claim is that session search can be reframed as symbolic link prediction in text: build a session graph with query and document nodes and three edge types (click on, query transition, document transition), translate it with a small grammar into an edge list like $(q_3, \\text{MacBook Price?}) <\\text{click on}> (d_5, \\$1{,}999)$, and have an LLM output 'yes' or 'no' for a candidate query-document link. The probability attached to 'yes' is the relevance score used in a listwise ranking loss. The authors further claim that because LLMs were pretrained on natural text rather than this symbolic language, three self-supervised symbolic tasks are needed, and that removing either the graph serialization or the symbolic pretraining degrades performance. If these claims hold, graph structure and text semantics are not competing representations but can be unified inside one LLM.","pith_inferences":["Editorial inference: the paper does not test whether the grammar itself is optimal, so replacing 'click on' and 'transfer to' with different relation names, or adding edge weights, could plausibly change performance and would be a natural ablation.","Editorial inference: because recurring query and document nodes reappear across sessions, the pretraining phase stores a form of global graph memory inside the LLM parameters; measuring how much repeated queries benefit from this cross-session memory would sharpen the paper's efficiency claim.","Editorial inference: the same symbolic-graph prompting and self-supervised tasks could transfer to adjacent problems such as conversational search or recommendation, where user actions also form heterogeneous graphs.","Editorial inference: the reliance on the 'yes' token logit invites a direct comparison against letting the LLM generate an explicit relevance label or numeric score, which the paper does not report."],"forward_implications":["If the central claim is correct, session search can be executed by an LLM reading a text serialization of the session graph, with no separate graph neural network encoder.","The reported data-efficiency result implies that a small fraction of labeled sessions may suffice to match existing full-data rankers, which matters for search logs where labeled data are expensive.","Because the three symbolic pretraining tasks are self-supervised, unlabeled session logs could be used to adapt an LLM to a new search domain before fine-tuning on click labels.","The same link-prediction framing could rank documents for any query where past clicks define a graph, extending the approach beyond the two tested benchmarks."],"supporting_citations":[{"why":"Provides the strongest graph-based baseline (HEXA) that SGR must beat on both datasets.","marker":"[46]"},{"why":"Supplies the sequence-to-sequence ranking idea of using answer-token logits as relevance scores.","marker":"[30]"},{"why":"Provides the AOL session search dataset and its evaluation setup used in the experiments.","marker":"[2]"},{"why":"Provides the Tiangong-ST dataset and its session data used for training and evaluation.","marker":"[11]"},{"why":"Provides the COCA baseline and the Tiangong-ST train/test configuration that the paper follows.","marker":"[58]"},{"why":"Provides the ASE sequential modeling baseline with generative auxiliary tasks.","marker":"[8]"},{"why":"Provides BM25 as the ad-hoc ranking baseline and the candidate document retrieval method.","marker":"[37]"}],"fun_headline_variants":[],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The assumption that serializing the session graph as a chronological edge list preserves enough topological information for the LLM to reason about node proximity, and that the model's probability of the token 'yes' is a valid relevance score.","fun_headline_variants_meta":{"error":"Client error '402 Payment Required' for url 'https://api.deepseek.com/chat/completions'\nFor more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/402"},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:38:19.760044+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If randomizing the order of the serialized edges while keeping the same nodes and text leaves ranking performance essentially unchanged, or if replacing the symbolic graph with a plain natural-language session description gives equal performance, then the claimed topological benefit is not what carries the result.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Tiangong-ST dataset and its session data used for training and evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the strongest graph-based baseline (HEXA) that SGR must beat on both datasets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the sequence-to-sequence ranking idea of using answer-token logits as relevance scores."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the AOL session search dataset and its evaluation setup used in the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the COCA baseline and the Tiangong-ST train/test configuration that the paper follows."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ASE sequential modeling baseline with generative auxiliary tasks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides BM25 as the ad-hoc ranking baseline and the candidate document retrieval method."}],"review_version":1}