{"id":"823eba7c-afd4-4379-a56b-ba8a50f9137c","arxiv_id":"2501.05374","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"Semantic similarity between LLM responses, with a fitted threshold of 0.5, separates same-model outputs from random responses and yields 76.5 percent held-out verification accuracy in a trusted-node setting.","lead":"The paper proposes using semantic similarity between language model outputs to verify that GPU computations in decentralized networks were performed correctly. It reports about 76 percent verification accuracy on a held-out set, but the experiments test LLM text responses, not the GPU computations the title claims to validate.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 76.5% accuracy is measured against random unrelated text, not against GPU nondeterminism, a wrong model, or adversarial outputs; the claimed verification property is untested.","rationale":"The reader's strongest claim is accurate: the paper asserts that semantic similarity, thresholded at 0.5 from Chatbot Arena training, can verify correct GPU execution. For that to hold, the embedding-distance decision rule must separate correct from incorrect computation in the deployment setting. The experiments never test that setting. The negative class is random text, and the evaluation even treats outputs of two different LLaMA models (8B and 70B) as comparable, which means a node substituting a different model would be accepted by the same logic. Section 5's intra-model similarities are close to the threshold, so the margin is small even in the favorable case. No GPU experiments, no adversarial-output experiments, and no consensus/adversary analysis are provided; Section 6.0.2 asserts that two verifier nodes are independent without an adversary model. As a proof-of-concept that semantic similarity can separate model text from random text, the paper has some preliminary evidence, but that is not the claimed result. I therefore see no reason to change the reader's REJECT verdict; the central claim is unsupported by the evidence presented.","tokens_in":8999,"tokens_out":5487,"duration_ms":53268,"concrete_test":"Re-run the Section 7 evaluation with three changes: (1) positive pairs are repeated outputs of the same model and prompt under actual GPU nondeterminism (e.g., same CUDA kernel on different GPU models/drivers, sampled with temperature > 0); (2) negative pairs are the reference model's output versus a different but competent model (e.g., 70B vs 8B, quantized vs full precision) on the same prompt with identical decoding parameters; (3) report per-prompt accuracy and the false-accept/false-reject rates at t*=0.5, not just aggregate accuracy. The most minimal version: on the existing 1000 prompts, compute the cosine similarity between 8B and 70B outputs and count how many exceed 0.5; since cross-model is currently included as a positive comparison, a large fraction exceeding the threshold would demonstrate the scheme accepts a wrong model.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that, for a fixed prompt, embedding cosine similarity separates correct from incorrect GPU computation at one global threshold. Section 7 optimizes t*=0.5 using a negative class of 'unrelated responses from the Arena dataset' and even includes cross-model comparisons (8B vs 70B) in its evaluation, so different but competent models are implicitly treated as valid. The cases that define the protocol's purpose are missing: same model/prompt across different GPUs or drivers with floating-point nondeterminism; stochastic sampling where legitimate repeats can diverge; a node running a quantized/older/smaller model, which produces semantically plausible outputs; and an adversarial node generating a wrong but topic-consistent answer. For these cases, the distance between correct and incorrect outputs can overlap the distance between two correct outputs. Section 5 already shows same-model similarity around 0.549-0.565 and cross-model similarity 0.557, only marginally above the chosen threshold, so overlap is not a remote possibility. The reported 76.5% held-out accuracy therefore estimates separability from random text, not the false-accept/reject rates relevant to trustless GPU verification.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that existing methods for verifying GPU computations in decentralized networks (exact recomputation, TEEs, FHE) are impractical, and it explores three alternative approaches: model fingerprinting, semantic similarity analysis, and GPU profiling. The empirical section reports a fingerprinting match rate of 15% exact / 25% partial on LLaMA-2-7B, and a semantic-similarity experiment on Chatbot Arena data in which a cosine-similarity threshold of 0.5 distinguishes same-model and cross-model LLM responses from random unrelated Arena responses with 76.5% held-out accuracy. On this basis the paper proposes a binary reference-node protocol and a ternary consensus protocol for trustless verification. The abstract and title claim validation of GPU computation, but no GPU execution, GPU non-determinism, or adversarial execution is experimentally studied; the reported metrics only separate LLM-generated text from random text.","tokens_in":9243,"tokens_out":3739,"duration_ms":39855,"significance":"The paper identifies a real problem: bitwise verification is unsuitable for non-deterministic GPU workloads, and the FHE/TEE alternatives have severe practical limitations. The proposed binary and ternary semantic-verification protocols are clearly described, and the use of a held-out test split for threshold evaluation is a methodological strength over simple in-sample reporting. If the underlying premise were validated—that semantic-similarity distance reliably separates correct from incorrect GPU computations—the framework would be a useful practical contribution. However, the paper does not validate that premise: the experiments contain no GPU traces, no controlled non-determinism, no wrong-model or adversarial outputs, and no GPU profiling results, so the central claim of the title and abstract is currently unsupported. The contribution is best assessed as a preliminary proposal for semantic verification of LLM outputs, not a validated method for GPU computation verification.","major_comments":[{"comment":"The central claim that semantic similarity can validate GPU computation is untested. The experiments compare LLM text responses (same model, cross-model, and random Arena responses) using sentence embeddings; they never involve GPU execution, bitwise variation from non-determinism, driver or architecture differences, quantized or swapped models, or adversarial outputs. The reported 76.5% held-out accuracy therefore estimates separability of LLM text from random text, not the false-accept/false-reject rates relevant to verifying a GPU computation. This is a load-bearing gap for the title and abstract's claim of 'Validation of GPU Computation in Decentralized, Trustless Networks'.","section":"Sections 5 and 7"},{"comment":"The threshold t*=0.5 is calibrated using same-model and cross-model (8B vs 70B) responses as positive examples and random Arena responses as negatives. The verification protocol, however, must in practice separate correct execution of a specified model from incorrect execution (e.g., wrong model, wrong weights, or non-deterministic numerical differences). Section 5 reports intra-model similarities of 0.549 and 0.565 and inter-model similarity of 0.557, all only marginally above the chosen threshold of 0.5. This leaves essentially no margin and suggests that similarity distributions for correct versus incorrect execution could overlap substantially; the paper does not report those distributions for any actual verification scenario.","section":"Section 6.0.1 and Section 5"},{"comment":"The ternary consensus protocol's claim that it 'eliminates the requirement of trust' is asserted without an adversary model. The protocol assumes two verifier nodes reach identical conclusions and that three response nodes behave independently, but no threat model specifies collusion, Byzantine verifiers, duplicated responses, or an attacker controlling multiple nodes. In particular, if an adversarial node simply copies a valid response from another node, pairwise similarity will be high and the invalid response can pass; the paper does not address this attack, nor does it define what happens if all three responses are semantically similar but all are wrong. Without these details, the trustless property is not established.","section":"Section 6.0.2"},{"comment":"The fingerprinting results reported in Section 5 (15% exact match, 25% partial match out of 60 samples) are low for a verification mechanism, and the paper explicitly notes that reliable detection often requires multiple queries, impacting economic viability. Despite this, Section 3.1 describes fingerprint verification through exact and inside match comparisons as a viable approach, and the abstract lists model fingerprinting as one of three explored methodologies. The paper should either present fingerprinting as a preliminary negative result or provide the query-count and cost analysis needed to assess its feasibility; as written, the treatment is internally inconsistent regarding whether fingerprinting is a supported contribution.","section":"Section 5 and Section 3.1"},{"comment":"GPU profiling is listed as a third verification methodology in the abstract and Section 3.3, but Section 5 explicitly states that 'its implementation requires extensive data collection and will be explored in future work,' and no profiling experiments or validation appear anywhere in the paper. The abstract's claim that the paper 'explores three verification methodologies' is therefore overstated; only semantic similarity receives empirical validation, and even that is only for LLM text, not GPU execution.","section":"Section 3.3 and Section 5"}],"minor_comments":[{"comment":"The description of the random baseline is under-specified: 'sampling three unrelated responses from the Arena dataset' should clarify whether these are responses to the same prompt, responses to different prompts, or arbitrary text, because the resulting similarity distribution depends critically on this choice.","section":"Section 7"},{"comment":"Figure 6 lacks axis labels and a legend, making it difficult to determine which curves correspond to training versus testing and what quantity is plotted; the caption should describe the reported metric and class balance.","section":"Figure 6"},{"comment":"The phrase 'semantic similarity distance' is imprecise, since cosine similarity is a similarity measure rather than a distance; the paper should use consistent terminology (e.g., 'cosine similarity' or 'semantic dissimilarity').","section":"Section 5"},{"comment":"The manuscript contains multiple typographical and formatting errors, including 'o ffering', 'e fficiency', a duplicated affiliation '2Lilypad Network', and inconsistent author footnotes; these should be corrected before resubmission.","section":"General"},{"comment":"The FHE cost estimate is computed for CPU-based execution ('on a modern CPU... $5,000 per token'), but the paper's focus is GPU workloads; the authors should either provide GPU-relevant cost estimates or clarify that the estimate is only indicative of orders of magnitude.","section":"Section 2.2.1"}],"recommendation":"reject","confidential_remarks":"The paper's title and abstract promise validation of GPU computation, but the experimental content only evaluates semantic similarity of LLM text outputs. This is not a minor fix: the central verification claim would require new experiments with realistic incorrect-execution classes, GPU non-determinism, and an adversary model. I would encourage the authors to reframe the paper as a preliminary study of semantic similarity for LLM-output verification in decentralized networks, and to gather the missing GPU evidence before resubmitting to a journal. There is no indication of misconduct, and the held-out evaluation and protocol descriptions are useful starting points."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe punchline: this paper proposes semantic similarity as a cheap way to verify GPU computation in decentralized networks, but the experiments never touch GPU computation. The positive class is same-model LLM responses; the negative class is random text. So the 76.5% accuracy measures something real but narrow—discriminating coherent model output from noise—and not the claimed verification property.\n\nWhat is genuinely useful: the survey of FHE and TEE limitations is solid, with a concrete per-token cost estimate. The fingerprinting pilot is honestly reported (15% exact, 25% partial match) and the authors correctly drop that direction. The binary and ternary protocol descriptions are clear enough to implement, and the threshold search is proper train/test split methodology.\n\nThe soft spots are load-bearing. First, the negative class is trivial. Random Arena text is semantically far away; the failure modes that matter—a node running a quantized or smaller model, a swapped model, or an adversarial topic-consistent answer—are absent. The paper's own numbers show same-model similarity around 0.55 and cross-model around 0.557, with threshold 0.5. The margin is thin and no distributional analysis is given. Second, no GPU experiments at all. No repeated runs, no non-determinism measurements, no profiling. The title and abstract claim GPU validation; the data are text embeddings. Third, the ternary consensus protocol just asserts that two verifier nodes eliminate trust, with no adversary model or collusion analysis.\n\nThe citations are appropriate and prior work is credited. No artifacts, which is a minor further weakness.\n\nVerdict: the central claim is unsupported by the experiments. This could become a decent proof-of-concept for semantic verification of LLM outputs if reframed, with an adversary model and GPU-based tests. As written, it overclaims. I think it deserves a serious referee—the problem is real and the protocols are concrete—but I would expect a reject or major revision. It is a good reading-group example of verification claims outrunning test sets.\n\nRecommendation: send to peer review, but brace for heavy revision.","headline":"Semantic verification is a real idea, but the paper tests it only against random text, not GPU failures, so the central claim is unsupported.","tokens_in":98,"tokens_out":3095,"would_cite":false,"duration_ms":59045,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that semantic similarity between outputs can probabilistically verify correct GPU execution in decentralized networks, replacing bitwise comparison.","keywords":["semantic similarity","GPU non-determinism","decentralized verification","trustless networks","LLM inference","consensus protocol","semantic threshold","computational integrity"],"falsifier":"Run the same prompt through the same LLM on two different GPU architectures and measure cosine similarity between their outputs across many prompts; if a sizable share of these correct-but-non-deterministic pairs falls below the 0.5 threshold, the semantic proxy cannot distinguish correct execution from error. Conversely, show that an adversarial node can pass verification by copying a valid response or by emitting text whose embedding lands above 0.5.","tokens_in":8823,"feed_emoji":"🤖","tokens_out":5026,"duration_ms":47659,"temperature":0.7,"pith_summary":"Because GPU execution is inherently non-deterministic, bitwise recomputation cannot verify decentralized GPU work, and the paper argues that cryptographic alternatives are impractical. It proposes to verify correctness by comparing the semantic similarity of outputs: an untrusted node's answer is accepted when its meaning is close to a trusted reference or to enough independent peer answers. Using a cosine-similarity threshold of 0.5 tuned on the Chatbot Arena dataset, the paper reports 76.5% held-out accuracy, precision 66.9%, recall 81.8%, and F1 0.736. The paper's ternary consensus variant claims to remove the need for a trusted node, replacing it with two independent verifiers and a two-tier consensus rule.","feed_headline":"Semantic similarity can verify GPU compute at 76.5 percent","feed_subtitle":"A 0.5 cosine threshold separates valid LLM outputs from random garbage, enabling trustless network verification.","key_machinery":"The load-bearing mechanism is semantic-similarity thresholding on neural embeddings: each output is transformed into a vector by a sentence-transformer embedding model, and correctness is decided by whether the cosine similarity between two outputs reaches an offline-optimized threshold $t^* = 0.5$. The ternary protocol adds a two-tier rule: first both verifier nodes must agree on the pairwise similarity pattern, and then a response set is accepted only when at least two of the three responses are mutually similar. This machinery converts the intractable problem of checking bitwise equality into a tractable problem of checking meaning preservation.","core_discovery":"The central discovery is that neural text embeddings can serve as a probabilistic substitute for bitwise comparison when verifying non-deterministic GPU workloads. The paper establishes that outputs from the same language model embed close together (intra-model cosine similarity around 0.55), while unrelated responses embed far apart (around 0.05), so a threshold of 0.5 can separate legitimate outputs from arbitrary ones. It packages this into two protocols: a binary reference model where a trusted node compares outputs against its own reference, and a ternary consensus model where three nodes produce responses and two verifier nodes judge pairwise similarities. The paper reports training accuracy of 76.1% and held-out accuracy of 76.5%, and it asserts that the ternary protocol eliminates the requirement of trust in the verification process.","pith_inferences":["Editorial extension: the experiments compare LLM outputs with random unrelated text, so the claim that this verifies GPU computation assumes that any real execution error will shift the output's embedding far enough to cross the 0.5 threshold; that remains unmeasured.","A natural test is to run the same model on different GPU architectures and measure the actual similarity distribution of correct-but-bitwise-different outputs; if that distribution dips below 0.5, the threshold would need recalibration by hardware class.","The ternary protocol's trustlessness depends on the two verifiers not colluding, so an adversarial deployment would likely need staking or sybil-resistance to make collusion costly.","The semantic approach could extend to other stochastic output domains where correctness is defined by meaning, such as image captions or summaries, but it would not suit numerical computations where correctness is exact."],"forward_implications":["A decentralized network can validate non-deterministic LLM inference without deterministic recomputation or specialized hardware, so long as semantic equivalence is accepted as the standard of correctness.","The semantic threshold is reusable across queries: once tuned offline, the same 0.5 threshold can be applied at inference time to new outputs.","The ternary protocol enables verification without a trusted reference node, provided the two verifier nodes are independent and non-colluding.","The reported 76.5% accuracy means the guarantee is explicitly probabilistic: some invalid outputs will pass and some valid outputs will be rejected.","GPU profiling is left for future work, so the verified class of computations in this paper is currently text-generating models rather than arbitrary GPU kernels."],"supporting_citations":[{"why":"Supplies the Chatbot Arena conversations dataset used to tune the 0.5 threshold and to build the held-out test set.","marker":"[20]"},{"why":"Provides the SemScore semantic-textual-similarity method and the sentence-transformer architecture that computes the cosine similarities.","marker":"[16]"},{"why":"Offers the general semantic-similarity background the paper adapts, defining how concept distances can be measured.","marker":"[14]"},{"why":"Documents GPU non-determinism, the reason bitwise recomputation cannot serve as a verification baseline.","marker":"[2]"},{"why":"Supplies the instructional-fingerprinting method and codebase tested in the preliminary analysis that the paper then sets aside.","marker":"[12]"},{"why":"Establishes the FHE cost baseline the paper uses to argue that cryptographic verification is impractical.","marker":"[4]"}],"fun_headline_variants":["Semantic similarity verifies GPU compute at 76.5%","Embedding-based check: trustless GPU verification at 76.5%","Cosine similarity above 0.5 = valid GPU output (76.5%)","Decentralized GPU trust via neural embeddings: 76.5% accurate","AI embeddings replace bitwise checks for GPU trust (76.5%)"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that semantic similarity between text outputs is a reliable proxy for correct GPU computation, meaning a node that computes incorrectly will produce an output whose embedding is far from correct outputs, and a node that computes correctly will stay close regardless of hardware-induced non-determinism.","fun_headline_variants_meta":{"raw":{"variants":["Semantic similarity verifies GPU compute at 76.5%","Embedding-based check: trustless GPU verification at 76.5%","Cosine similarity above 0.5 = valid GPU output (76.5%)","Decentralized GPU trust via neural embeddings: 76.5% accurate","AI embeddings replace bitwise checks for GPU trust (76.5%)"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000571,"raw_usage":{"total_tokens":2647,"prompt_tokens":838,"completion_tokens":1809,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":1710}},"tokens_in":454,"tokens_out":1809,"duration_ms":11851,"temperature":1.0,"reasoning_tokens":1710,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:12:34.696555+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same prompt through the same LLM on two different GPU architectures and measure cosine similarity between their outputs across many prompts; if a sizable share of these correct-but-non-deterministic pairs falls below the 0.5 threshold, the semantic proxy cannot distinguish correct execution from error. Conversely, show that an adversarial node can pass verification by copying a valid response or by emitting text whose embedding lands above 0.5.","supporting_citations":[{"cited_title":"Chatbot Arena conversation dataset re- lease, July 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the Chatbot Arena conversations dataset used to tune the 0.5 threshold and to build the held-out test set."},{"cited_title":"Description and Evaluation of Semantic Similarity Measures Approaches","cited_arxiv_id":"1310.8059","evidence_quote":"Offers the general semantic-similarity background the paper adapts, defining how concept distances can be measured."},{"cited_title":"Determinism in deep learning","cited_arxiv_id":null,"evidence_quote":"Documents GPU non-determinism, the reason bitwise recomputation cannot serve as a verification baseline."},{"cited_title":"A guide to fully homomorphic encryption","cited_arxiv_id":null,"evidence_quote":"Establishes the FHE cost baseline the paper uses to argue that cryptographic verification is impractical."}],"review_version":1}