{"id":"2d567396-d768-4e57-b243-fd107786e5cd","arxiv_id":"2502.10151","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A decentralized search overlay places users in a binary tree over LLM document embeddings with soft cloning and expansion rounds, retrieving more relevant documents than random or graph-diffusion baselines at equal network load.","lead":"Semantica organizes users of a peer-to-peer network into a tree built from LLM document embeddings, so that each user connects to semantically similar peers. On a 2006 AOL search log, the paper reports that this overlay finds more similar peers and retrieves more relevant documents at equal network load than random and graph-diffusion baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'same network load' comparison omits Semantica's tree-construction and expansion-round messages, so the reported 2x retrieval advantage may not survive an equal total-message budget.","rationale":"The abstract's quantitative claim ('more than two times the number of relevant documents given the same network load') is the central empirical assertion, and it rests on the experimental comparison in Section VI.B. That comparison, however, does not hold network load constant: it matches average degree (m=104) and counts only query messages, while Semantica's tree construction (O(N log N) per Section VII.A) and expansion rounds (O(rmax N)) are treated as free. With N=6,980 and rmax=10, these setup messages are on the order of 1e5, comparable to the entire query phase. The random Barabasi-Albert baseline has essentially zero setup cost, and graph diffusion's diffusion iterations are also uncounted. This means the measured 12.75% vs <6% two-hop accuracy is not achieved 'given the same network load'; it is achieved after an uncounted investment. The reader's weaker assumption about user embedding stability and document overlap is worth testing, but it primarily affects external validity; the uncounted-message issue affects the internal validity of the headline comparison itself. A message-counting re-simulation is a concrete, low-cost check that can settle the claim. If the advantage survives equal total-message accounting, the paper's central claim stands; if not, the abstract overstates the improvement. I therefore agree with the reader's CONDITIONAL verdict overall, but for a different, more specific reason than the one identified in the reader's weakest_assumption.","tokens_in":18172,"tokens_out":9395,"duration_ms":95275,"concrete_test":"Instrument the public repository's simulator to count every protocol message: tree-join and split-centroid messages, expansion-round queries/replies, and query hops for Semantica; diffusion iterations for graph diffusion; and bootstrap messages for the random baseline. Re-run Experiment 2 with a fixed total-message budget across all methods (e.g., 3e5 messages) and plot retrieval accuracy versus total messages. If Semantica no longer exceeds the baselines by roughly 2x at the same total-message cost, the 'same network load' claim in the abstract is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline retrieval claim (Abstract; Section VI.B, Fig. 6b) states Semantica retrieves more than two times the relevant documents 'given the same network load.' The experiment matches the Barabasi-Albert graph degree to Semantica's mean known-users size (m=104) and the x-axis counts only query messages. However, Semantica's tree construction costs O(N log N) messages and its expansion rounds cost O(rmax N) messages (Section VII.A); with N=6,980 and rmax=10, this adds on the order of 1e5 setup messages, roughly doubling the query-phase load. The random Barabasi-Albert baseline incurs essentially no construction cost, and graph diffusion's own diffusion setup is also not counted. The comparison therefore does not actually hold network load constant: the 12.75% vs <6% two-hop retrieval rates are measured after an uncounted setup investment. If these setup messages are charged against Semantica, the stated advantage may shrink or disappear.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Semantica proposes a decentralized semantic overlay network in which each user is represented by the mean LLM embedding of their documents, and users are organized into a hierarchical binary tree via k-means splits with soft-clustering (cloning) at split nodes. Each user maintains a known-users list gathered from nearby leaf nodes and refines a closest-users list through expansion rounds; queries are routed by chain-hopping through the closest-users lists. The paper evaluates the design on the AOL4PS dataset with BERT embeddings, reporting closest-user recall against a 50-nearest ground truth, document-retrieval accuracy under a hop limit relative to a Barabási-Albert graph and a graph-diffusion baseline, and hop-distance distributions. The headline claims are that Semantica finds up to ten times more semantically similar users and retrieves more than twice as many relevant documents at the same network load.","tokens_in":18421,"tokens_out":4516,"duration_ms":49851,"significance":"If the experimental claims held under a fair accounting of network cost, Semantica would be a useful contribution to decentralized semantic search. The design is clearly specified, the complexity analysis in Section VII is reasonable for the average case, and the authors provide publicly available code and use a real-world workload, which are concrete strengths. The central novelty—using LLM embeddings to build a semantic trie with cloning and expansion rounds—is plausible and worth pursuing. However, the evaluation as written does not yet support the two headline quantitative claims: the 'same network load' comparison omits Semantica's construction and expansion costs, the main hyperparameter is selected on the same test data without variance reporting, and the filtered dataset limits the generality of the workload assumption. These issues are load-bearing for the abstract's claims rather than cosmetic.","major_comments":[{"comment":"The claim that Semantica retrieves 'more than two times the number of relevant documents given the same network load' is not supported by the reported experiment, because the load comparison counts only query messages. Section VII.A states that tree construction costs O(N log N) messages and expansion rounds cost O(rmax N) messages; for the evaluated N = 6,980 and rmax = 10, this is on the order of 10^5 setup messages, while the Barabási-Albert baseline has essentially no construction cost. Charging these setup messages against Semantica's query budget would materially reduce the reported 12.75% versus <6% two-hop retrieval advantage. Please either present an equal-total-message comparison, or justify why setup cost can be treated as amortized/one-time and quantify that amortization.","section":"Abstract and §VI.B, Fig. 6b"},{"comment":"No error bars, confidence intervals, or multiple-seed variability are reported, despite multiple sources of randomness: the user insertion order is shuffled in Algorithm 1, k-means is stochastic, expansion rounds select random known-users, and the test/train document split is random. The parameter Δ is also selected based on the same AOL4PS dataset on which the headline results are measured (e.g., Δ = 0.003 in Fig. 6b, Δ = 0.001 in Fig. 7). Since the cloning mechanism is highly sensitive to Δ (Table II shows mean clones growing from 1.32 at Δ = 1e-3 to 7.45 at Δ = 5e-3), the central performance numbers need a sensitivity analysis and variance reporting before they can be taken as reliable.","section":"§V.A and §VI.A, Fig. 4 and Fig. 6b"},{"comment":"The evaluation filters AOL4PS from 12,907 users to 6,980 users with at least 30 unique documents, yet the workload assumption of document overlap is validated on the full dataset in Section II. The paper does not report retrieval or recall results on the filtered-out users, and it does not quantify co-occurrence or embedding similarity within the filtered subset. Because the entire benefit of leaf-node co-location depends on users having overlapping, semantically stable interests, the current experiments support the claimed advantage only for relatively active users; the paper should either extend the evaluation to the full population or explicitly scope the claim to the filtered workload.","section":"§V.A.1 and §II, Fig. 1"},{"comment":"The graph-diffusion baseline is run on a Barabási-Albert graph with m = 104, matched to Semantica's average known-users degree. This may disadvantage the baseline: graph diffusion as proposed by Giatsoglou et al. is designed to operate on an overlay whose edges are built for content routing, not on a random graph. As reported, the comparison is 'chain-hopping on a semantic graph' versus 'graph diffusion on a random graph,' which does not cleanly isolate the contribution of Semantica's tree. Please either run graph diffusion on the graph construction intended by its authors, or report graph diffusion on Semantica's known-users graph as an additional baseline.","section":"§VI.B, Fig. 6b"}],"minor_comments":[{"comment":"There is a notation inconsistency: Algorithm 2 uses the parameter ncu for the closest-users list size, but line 11 hardcodes the threshold as the '50th user,' and the text in Section V.A refers to L = 50 for leaf capacity while Algorithm 1 uses M. Please unify these names and make the threshold consistent with ncu.","section":"Algorithm 2 and §IV.B"},{"comment":"The caption of Fig. 6b says 'at 50 queries sent to the network,' but the x-axis appears to range from 2 to 600 queries; please clarify whether the x-axis is the number of query messages, the number of hops, or the number of distinct query initiations.","section":"Fig. 6 and §VI.B"},{"comment":"The prose says the query is first sent to the user whose embedding is closest to the query embedding, while Algorithm 3 starts at the query initiator itself and only then forwards to the closest known user. This discrepancy should be resolved so the experimental protocol is unambiguous.","section":"§V.B, Experiment 2"},{"comment":"There are minor grammatical issues: 'In Section II defines the operational setting' should read 'Section II defines...' and 'AOL4PS data set used [12]' in Section IX is missing a verb. These do not affect the technical content.","section":"§I and §II"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely within scope for an experimental distributed-systems or information-retrieval venue. The public code and real-workload evaluation are strengths, and the algorithmic idea is interesting. My main concern is that the headline 'same network load' claim is not currently supported by the experimental accounting, and the lack of variance reporting makes it hard to gauge stability. If the authors can revise the evaluation to include construction/expansion costs or clearly amortize them, and add robustness checks, the paper could be publishable. I would not recommend rejection, because the central design is defensible and the gaps are empirical rather than logical."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"One thing to know: this is a real new algorithm, not a repackaging, and the evaluation is honest enough that the paper deserves a serious referee. The combination—LLM embeddings, a binary semantic trie, soft cloning, expansion rounds, and chain-hop queries—is novel relative to the cited SON, DSI/De-DSI, and graph-diffusion work. The clone mechanism is the most interesting piece: it lets a user occupy multiple leaves when their embedding straddles a cluster boundary, which is exactly what makes the expansion rounds useful. The paper also ships code and data, and the complexity analysis covers insertion, splitting, cloning, expansions, and queries without hiding the worst case.\n\nThe soft spots are in the evaluation, not the design. The comparisons rest on one filtered dataset (AOL4PS, 2006, 6,980 users after filtering), the Delta threshold is tuned on that same data, and there are no error bars or multiple-seed runs. The neighbor-recall experiment computes ground truth from the same user embeddings used to build the tree, so high recall is partly a self-fulfilling property of the embeddings. And the stress-test point is fair: the 'same network load' comparison counts query messages but not Semantica's tree-construction and expansion-round messages, which are O(N log N) and O(rmax N). Charging those would erode the reported 2x retrieval advantage, possibly to near parity at two hops. The abstract's 'ten times more similar users' is also just recall-out-of-50 against a random baseline, not a meaningful accuracy win over a strong SON baseline.\n\nThat said, the paper is transparent about its assumptions—users with overlapping documents, the mean embedding as a stable fingerprint—and it acknowledges the downsides it finds, like unreachable documents and the need for rebalancing. The central argument, that a semantic tree helps place similar peers close, holds up qualitatively. What's missing is a fairer cost accounting and a robustness check against seed variation and at least one more workload.\n\nSend it to review. A good referee will ask for error bars, a second dataset, and a network-load accounting that includes setup. The algorithm is worth engaging with; the headline claims just need to be recalibrated.","headline":"Solid new algorithm for LLM-based decentralized semantic search, but the headline numbers rest on a thinner evaluation than they suggest.","tokens_in":18925,"tokens_out":2432,"would_cite":false,"duration_ms":24169,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Semantica claims that organizing peers by LLM-produced document embeddings into a semantic trie lets decentralized search find up to ten times more similar users and retrieve over twice as many relevant documents at equal network load.","keywords":["decentralized search","semantic overlay network","LLM embeddings","prefix tree","peer-to-peer","document retrieval","chain-hopping","latent semantic indexing"],"falsifier":"Take the full, unfiltered AOL4PS user set (all 12,907 users, including those with no co-occurring document) or simulate churn by replacing each user's documents every week, then measure two-hop retrieval and closest-user recall; if Semantica's recall falls to the random baseline, the result is carried by the overlap-and-stability assumption rather than by the trie.","tokens_in":1606,"feed_emoji":"🔎","tokens_out":6427,"duration_ms":102157,"temperature":0.7,"pith_summary":"Semantica claims that decentralized search can match centralized retrieval accuracy if peers are arranged by the semantic content of their documents rather than by opaque hashes. Each user is represented by the mean embedding of their documents, and a binary trie built by k-means splitting places users with similar embeddings in the same or nearby leaf nodes. Queries then hop greedily from one user to the most semantically similar known user, so relevant documents are found with few messages. On the AOL4PS workload, the paper reports initial closest-user recall above 5 out of 50 versus about 0.4 for random, near-complete recall after 20 expansion rounds, and two-hop retrieval of 12.75% of required documents against under 6% for the comparison methods at the same network load.","feed_headline":"Semantic trie finds 10x more similar peers in decentralized search","feed_subtitle":"Users are clustered by document embeddings so queries hop to relevant peers, retrieving twice as many documents at equal load.","key_machinery":"The load-bearing structure is a binary semantic trie built by recursive $k=2$-means splitting of leaf nodes with capacity $M=50$. Each split-node stores two centroids; a new user descends toward the closer centroid, and if the difference in normed Euclidean distances is below threshold $\\Delta$, the user is cloned into both subtrees. Each leaf supplies a user with a seed list of peers, expanded by breadth-first search into nearby leaves and then refined by expansion rounds that ask a random known peer for closer contacts using cosine similarity. Chain-hop queries greedily forward to the most similar known user, making the trie the mechanism that turns embedding similarity into short network paths.","core_discovery":"The central discovery is that a trie over LLM embeddings can serve as a predictive overlay: users co-located in a leaf by $k=2$-means clustering are likely to share documents, so network adjacency can be derived from semantic adjacency. The paper introduces soft cloning at cluster boundaries, expansion rounds that refine each user's closest-user list through random neighbor queries, and chain-hop querying that forwards a query to the known user with the highest cosine similarity to the query. In emulation on a filtered real-world query log, Semantica's closest-user recall starts above 5 out of 50 while random achieves about 0.4, rises above 40 after 20 expansion rounds for thresholds around $\\Delta=0.001$, and at a two-hop budget retrieves 12.75% of required documents while graph-diffusion and random baselines stay below 6%. The authors describe this as the first predictive decentralized document search to exploit pre-trained LLM embeddings without any training or retraining.","pith_inferences":["Editorial inference: the trie is effectively a distributed approximate nearest-neighbor index, so the same construction could serve other semantic-overlay tasks such as decentralized recommendation or clustering of users by interest, not just document retrieval.","Editorial inference: because embeddings are computed locally, a change in the embedding model or even its version would shift all user positions and fragment the tree, making embedding-space versioning a testable requirement the paper does not address.","Editorial inference: the paper itself notes that chain-hopping's advantage shrinks as the hop limit grows and that some documents become unreachable inside clusters, so a hybrid that mixes semantic hops with occasional random edges is a natural extension that would likely preserve both short-hop recall and global reachability.","Editorial inference: the threshold $\\Delta$ controls a tradeoff between recall and clone blow-up, and clone count grows rapidly beyond $\\Delta=0.001$ on this dataset, so a self-tuning per-node threshold based on local embedding density is a concrete testable extension."],"forward_implications":["A decentralized network can identify a large share of a user's closest semantic peers from purely local interactions: after 20 expansion rounds, more than 40 of the 50 ground-truth closest users are found for the tested thresholds.","Semantic routing gives the biggest gains in short, communication-limited regimes: with a two-hop budget, Semantica retrieves 12.75% of required documents while graph-diffusion and random baselines stay below 6%.","The protocol keeps network overhead linear in queries and hops: each chain-hop query costs at most $\\ell$ messages and $O(\\ell \\cdot (k+d))$ local computation, independent of network size.","Tree construction scales as $O(N \\log N)$ messages under balanced embeddings, with expansion rounds adding $O(r_{\\mathrm{max}} N)$ messages, making the overlay feasible at large $N$ under ordinary data distributions.","Soft clustering with a moderate threshold is what makes expansion rounds effective: with $\\Delta=0$, neighbors within a leaf are redundant, and recall does not improve with expansion."],"supporting_citations":[{"why":"It supplies the AOL4PS query log used to justify the overlap assumption and to run all experiments.","marker":"[12]"},{"why":"It provides the pre-trained transformer embeddings used to compute document and user vectors.","marker":"[14]"},{"why":"It defines the graph-diffusion baseline whose retrieval accuracy and hop behavior Semantica is compared against.","marker":"[11]"},{"why":"It establishes semantic overlay networks and the predictive value of semantic similarity, the foundation Semantica builds on.","marker":"[28]"},{"why":"It introduces embedding-guided tries for differentiable search; Semantica moves this idea to decentralized networks without training.","marker":"[29]"},{"why":"It is the prior decentralized extension of differentiable search that still requires periodic training, the comparison point for Semantica's training-free claim.","marker":"[20]"},{"why":"It supplies the trie-like DHT structure that Semantica adapts by replacing content hashes with semantic embeddings.","marker":"[26]"}],"fun_headline_variants":["Semantica: LLM embedding trie yields 10x similar peers, 2x doc hits","Decentralized search: Semantic trie overlay finds 10x peers, 2x docs","LLM-guided trie overlay: 10x peer recall, 2x retrieval at same load","Predictive overlay from LLM embeddings: 10x peers, 2x docs"],"cache_read_input_tokens":21120,"weakest_assumption_plain":"The load-bearing premise is that users in the target workload store overlapping documents and that a user's mean document embedding is a stable semantic fingerprint, since the benefit of leaf-node co-location would vanish for users with disjoint or rapidly changing interests.","fun_headline_variants_meta":{"raw":{"variants":["Semantica: LLM embedding trie yields 10x similar peers, 2x doc hits","Decentralized search: Semantic trie overlay finds 10x peers, 2x docs","LLM-guided trie overlay: 10x peer recall, 2x retrieval at same load","Predictive overlay from LLM embeddings: 10x peers, 2x docs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001462,"raw_usage":{"total_tokens":5910,"prompt_tokens":998,"completion_tokens":4912,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":614,"completion_tokens_details":{"reasoning_tokens":4812}},"tokens_in":614,"tokens_out":4912,"duration_ms":30899,"temperature":1.0,"reasoning_tokens":4812,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T19:11:29.880918+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the full, unfiltered AOL4PS user set (all 12,907 users, including those with no co-occurring document) or simulate churn by replacing each user's documents every week, then measure two-hop retrieval and closest-user recall; if Semantica's recall falls to the random baseline, the result is carried by the overlap-and-stability assumption rather than by the trie.","supporting_citations":[{"cited_title":"Aol4ps: A large-scale data set for personalized search,","cited_arxiv_id":null,"evidence_quote":"It supplies the AOL4PS query log used to justify the overlap assumption and to run all experiments."},{"cited_title":"A graph diffusion scheme for decentralized content search based on personalized pagerank,","cited_arxiv_id":null,"evidence_quote":"It defines the graph-diffusion baseline whose retrieval accuracy and hop behavior Semantica is compared against."},{"cited_title":"Peer-to-peer information retrieval using self-organizing semantic overlay networks,","cited_arxiv_id":null,"evidence_quote":"It establishes semantic overlay networks and the predictive value of semantic similarity, the foundation Semantica builds on."},{"cited_title":"Transformer memory as a differentiable search index,","cited_arxiv_id":null,"evidence_quote":"It introduces embedding-guided tries for differentiable search; Semantica moves this idea to decentralized networks without training."},{"cited_title":"De-dsi: Decentralised differentiable search index,","cited_arxiv_id":null,"evidence_quote":"It is the prior decentralized extension of differentiable search that still requires periodic training, the comparison point for Semantica's training-free claim."},{"cited_title":"Chord: a scalable peer-to-peer lookup protocol for internet applications,","cited_arxiv_id":null,"evidence_quote":"It supplies the trie-like DHT structure that Semantica adapts by replacing content hashes with semantic embeddings."}],"review_version":1}