{"id":"c896f33a-5d6c-4a1d-8013-03f293596f08","arxiv_id":"2412.02263","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"C-LLM is a framework for feeding LLM answers into blockchain smart contracts via oracles, and SenteTruth aggregates textual answers using SBERT semantic similarity plus truth discovery to resist up to 40% malicious nodes.","lead":"A team proposes a framework, C-LLM, that connects blockchain smart contracts to large language models through oracle nodes, and introduces SenteTruth, an aggregation method that uses semantic similarity and truth discovery to filter out tampered answers. It matters because it offers a practical path for making LLM-generated data more trustworthy on-chain, which could support decentralized finance and other smart contract applications.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 17.74% accuracy claim is built on a circular metric: 'accuracy' counts tamper-resistance (unaltered data), not factual correctness, so the headline overstates what SenteTruth validates.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern, and I agree with it. The paper is otherwise reasonably executed: code and data are released, the adversary model is explicit, and the SBERT-plus-truth-discovery combination is plausible. But the headline claim's load-bearing term is 'accuracy,' and the paper's own definition makes that term a tamper-detection rate. This is not a matter of disagreement with field consensus; it is an internal mismatch between the abstract/conclusion and the metric in Section IV-B. The proposed ground-truth re-scoring check would settle whether SenteTruth improves factual answer quality or only tamper-resilience. A secondary technical instability also exists: Eq. (4) can make credibility change sign if a node's total cosine similarity is negative (which SBERT does not preclude), and Eq. (3) can then select a maximally dissimilar outlier. This reinforces conditional acceptance with re-evaluation, but it is secondary to the metric concern.","tokens_in":24950,"tokens_out":11386,"duration_ms":139916,"concrete_test":"Build a ground-truth answer key for the BASE, MIX, and PRO datasets (using known answers for fact/consistency and physics questions, plus independent human judgments for open questions), then re-run SenteTruth and every baseline under the three 40% attack scenarios, scoring the selected answer by semantic equality to the ground truth rather than by whether it is an unaltered node response. If SenteTruth's factual accuracy gain over SBERT is not significantly positive, the 17.74% claim should be restated as tamper-resilience only.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SenteTruth 'significantly enhances the accuracy and trustworthiness of data obtained from large models.' The evaluation cannot support that claim because the metric is defined to measure something else. Section IV-B defines Accuracy as 'the proportion of unaltered data in the final aggregated result,' and Section III-C defines truth as 'the data that reflects the majority consensus of the nodes and is free from tampering, even if the response is not entirely accurate in content.' Under this definition, an answer is 'accurate' iff it is an unchanged honest-node response, regardless of whether it is factually right. A unanimous hallucination scores 1.0; a correct answer that is paraphrased and therefore does not match the majority phrasing can score 0.0. Consequently, the reported 17.74% average improvement over the best baseline measures robustness against node-level tampering (random response, model substitution, prompt-induced incorrectness), not the accuracy or trustworthiness of the LLM-generated content promised in the abstract and conclusion. The method may be a plausible tamper-resilience mechanism, but the central quantitative claim about answer quality is unsupported without human/verified ground-truth labels.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes C-LLM, a framework for routing smart-contract oracle requests to LLM APIs and returning an aggregated answer, and SenteTruth, a method that combines SBERT sentence embeddings with a truth-discovery-style credibility update to select the most central answer from a set of node responses. The system is implemented on a local Ethereum/Ganache testbed with Solidity contracts and evaluated on a custom dataset built from 10 oracle nodes querying 5 LLMs under three attacks (random response, model substitution, incorrect response). The headline result is a 17.74% average accuracy improvement over the best baseline at 40% malicious nodes.","tokens_in":25170,"tokens_out":5808,"duration_ms":61982,"significance":"The practical scenario is plausible and the artifact is valuable: the authors provide code and data, deploy on a real local blockchain, include gas measurements, and cover five commercial/open LLMs. If the reported robustness were about factual answer quality, this would be a solid contribution to LLM oracle aggregation. As written, the evaluation metric is tamper-resilience (selecting an unaltered node response), not factual correctness, so the central quantitative claim needs reframing or a new evaluation. SenteTruth is a simple combination of two existing ideas; its novelty lies mainly in the application, the selection rule, and the open artifact.","major_comments":[{"comment":"The paper defines Accuracy as \"the proportion of unaltered data in the final aggregated result\" and defines truth as data that is \"free from tampering, even if the response is not entirely accurate in content.\" Under this metric, Tables I-III and Fig. 7, together with the 17.74% claim in the abstract, measure whether the final answer is an unchanged honest-node response, not whether it is factually correct. A unanimous hallucination would score 1.0, and a correct answer that is paraphrased relative to the majority wording could score 0.0. Because the abstract and conclusion claim improved \"accuracy and trustworthiness of data generated by LLMs,\" this is a load-bearing mismatch. Please either add a ground-truth-based factual accuracy evaluation with human or verified labels, or change the claims throughout to \"tamper-resilience\" or \"data-integrity selection accuracy.\"","section":"Section IV-B, RQ2; Section III-C"},{"comment":"Equation (2) defines phi(v_i) as a sum of cosine similarities over all j != i; with 10 nodes and identical answers this value is about 9, not less than 1. The sentence \"Since phi(v_i) < 1\" immediately after Eq. (4) is therefore false. The credibility update in Eq. (4) is a normalized multiplicative update only if phi is an average similarity or is otherwise bounded; as written, the stated reason for the update and its stability are not established. Please clarify the definition (e.g., use average similarity) and provide a convergence or stability argument for the iterative update.","section":"Section III-C, Eq. (2) and Eq. (4)"},{"comment":"Equation (3) selects one node's answer D_i that maximizes C_i * phi(v_i); the algorithm does not synthesize or aggregate text. The text says the goal is to \"synthesize an aggregated result\" and repeatedly calls SenteTruth an aggregation method, but the output is always one of the received responses. This means the method cannot fix a factual error shared by the majority of honest nodes; it only selects the most centrally credible answer among the received responses. The claims should be stated as robust selection rather than aggregation, or a true text-synthesis/aggregation step should be added.","section":"Section III-C, Eq. (3)"}],"minor_comments":[{"comment":"The captions and axis labels of Figures 2, 5, and 6 contain \"/uni...\" placeholder tokens, making the visual results unreadable in the submitted PDF; these figures need to be regenerated with proper text.","section":"Figures 2, 5, and 6"},{"comment":"Equation (4) uses i both for the node being updated and as the summation index; please use a separate index (e.g., k) in the numerator and denominator to avoid ambiguity.","section":"Section III-C, Eq. (4)"},{"comment":"The paper gives no stopping criterion for the credibility update. Figure 8 shows 20 epochs, but the algorithmic description does not state how many iterations are used in general or how convergence is detected.","section":"Section III-C, Section IV-E"},{"comment":"The \"TF-IDF Similarity + TD\" baseline reports a 0.0 accuracy for Llama under model substitution, meaning that adding truth discovery always selects a malicious response on that condition. Given that truth discovery should not normally make a method worse than chance, the authors should explain or investigate this pathological value.","section":"Table III and Table VII"},{"comment":"Reference [20] is a blog URL with no year or archival identifier; please replace it with a citable and preferably archival source.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a systems/applications contribution and the open artifact is a genuine strength. The main risk is that the headline result is currently supported by a circular metric: \"accuracy\" is defined as selecting unaltered honest-node data, not as factual correctness. I would send this back for major revision rather than reject, because the underlying selection algorithm could be presented honestly as a tamper-resilience mechanism and the artifact would then be a useful contribution. The paper may fit a blockchain/oracle systems venue better than a general AI venue, given the absence of factual-accuracy evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Plainly: the paper builds a workable mechanism for a genuine problem, and the headline claim is overstated.\n\nWhat's new: SenteTruth combines SBERT semantic relatedness with truth discovery to aggregate textual answers from multiple oracle nodes in an LLM-blockchain oracle. Prior truth-discovery oracle work handled numerical data, so the text extension is a real gap filled. The authors also ship code and data, and the RQ1 consistency analysis (question type, temperature, prompt effects) is a useful empirical note.\n\nWhat the paper does well: the three attack types (random response, model substitution, incorrect response via crafted prompt) are sensible, and the adversarial assumption of <50% malicious nodes is standard. The algorithm is simple: cosine similarity weighting, then credibility updates. The reported tables are internally consistent with their stated accuracy definition.\n\nThe soft spot is load-bearing. Section IV-B defines accuracy as 'proportion of unaltered data in the final aggregated result.' That is tamper-resilience, not factual correctness. A node's unchanged-but-hallucinated answer counts as accurate; a correct answer that happens to be paraphrased and sits in the minority can lose. So the abstract's claim that SenteTruth 'significantly enhances the accuracy and trustworthiness of data generated by LLMs' goes beyond what is measured. The 17.74% figure is credible as evidence of resistance to injected tampering, and that is a legitimate result for an oracle system, but it is not evidence about answer quality.\n\nMinor gaps: no human-labeled ground truth, no error bars, no sensitivity analysis of the truth-discovery epoch count, and the gains over plain SBERT similarity are sometimes marginal. Adding an ablation that isolates the contribution of the credibility-update loop would help.\n\nNone of this sinks the paper. The mechanism is plausible and the evaluation, once reframed, supports a useful claim. The fix is to either rename the metric and soften the claims, or add a human-evaluated correctness subset.\n\nRecommendation: yes, send it to peer review. It deserves a serious referee, with a request to align claims with the metric and add basic robustness reporting. I wouldn't cite it myself in the next year, but readers working on oracle design or LLM-blockchain integration should read it.","headline":"SenteTruth is a plausible tamper-resilient oracle aggregation method for LLM text, but the headline accuracy claim measures unaltered-data selection, not factual correctness.","tokens_in":25681,"tokens_out":3689,"would_cite":false,"duration_ms":39953,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that blockchain smart contracts can safely use large language models by routing questions through a decentralized oracle network and aggregating the LLM text answers with a method the authors call SenteTruth, and it…","keywords":["blockchain","smart contracts","oracles","large language models","truth discovery","semantic relatedness","SBERT","data aggregation"],"falsifier":"Build a dataset where 60 percent of nodes give correct but semantically varied answers to an open question and 40 percent collude on a single plausible but wrong answer; SenteTruth would pick the colluding answer because it maximizes semantic relatedness. Observing that outcome would show the central claim holds for tamper resistance but fails for factual answer quality.","tokens_in":24762,"feed_emoji":"🤖","tokens_out":5616,"duration_ms":54326,"temperature":0.7,"pith_summary":"This paper claims that blockchain smart contracts can safely use large language models by routing questions through a decentralized oracle network and aggregating the LLM text answers with a method the authors call SenteTruth. SenteTruth combines sentence-level semantic similarity, computed with SBERT embeddings and cosine distance, with iterative truth-discovery credibility weights, so that nodes whose answers consistently deviate from the majority are down-weighted. On a dataset of 10 oracle nodes querying 5 LLMs, the paper reports that with 40 percent malicious nodes SenteTruth improves data accuracy by an average of 17.74 percent over the best baseline. The claim matters because it suggests a practical path for adding open-ended reasoning to smart contracts without trusting any single oracle or LLM provider.","feed_headline":"Semantic voting lets smart contracts trust LLM answers","feed_subtitle":"A new aggregation method, SenteTruth, shrugs off 40% malicious oracle nodes and beats all baselines.","key_machinery":"The load-bearing mechanism is SenteTruth, a two-stage aggregation algorithm for text answers: an SBERT sentence encoder plus cosine similarity gives a semantic relatedness score for every answer, and a truth-discovery loop repeatedly multiplies relatedness by node credibility to select the winner and then updates credibility. The iterative weighting is what carries the argument: a single similarity error by SBERT is corrected by the accumulated credibility of nodes that have historically agreed with the majority, which is why the paper reports high accuracy even when malicious nodes are 40 percent of the network. The C-LLM framework around it supplies the commit-reveal data exchange that prevents nodes from copying each other's answers before aggregation. In mathematical terms the aggregation objective is $\\bar D = \\arg\\max_i C_i \\varphi(v_i)$ with credibility update $C_i \\leftarrow \\left(\\frac{\\sum_{i} C_i}{\\sum_{i} C_i \\varphi(v_i)}\\right) C_i \\varphi(v_i)$.","core_discovery":"The paper's central discovery is that truth-discovery weighting, previously limited to numerical data, can be carried over to free-form LLM text if semantic relatedness replaces numeric distance. SenteTruth maps each node's answer $D_i$ to an embedding $v_i = \\text{SBERT}(D_i)$, computes each answer's relatedness $\\varphi(v_i)$ as the sum of cosine similarities to all other answers, and selects the aggregate $\\bar D = \\arg\\max_i C_i \\varphi(v_i)$ using each node's learned credibility $C_i$. Credibility is then updated so nodes whose answers resemble the aggregate gain weight and those that deviate lose weight. The paper reports that this scheme achieves near-perfect accuracy under three text-specific attacks—random responses, model substitution, and prompt-engineered incorrect answers—and that at 40 percent malicious nodes the average accuracy gain over the best baseline is 17.74 percent. The authors define accuracy as the proportion of unaltered data in the final aggregated result, so the gain is specifically tamper resilience rather than factual correctness.","pith_inferences":["The paper's accuracy metric counts whether the final selected answer was an unaltered node response, not whether it is factually correct; the 17.74 percent improvement therefore demonstrates tamper resistance, and a reader should not infer that the selected answer is more truthful than the honest nodes' answers.","If malicious nodes collude and emit semantically similar wrong answers while honest nodes produce correct but diverse phrasings, SenteTruth's majority-similarity assumption would favor the attackers; the paper only assumes fewer than half malicious nodes and no collusion.","A natural test is to run SenteTruth on open-ended questions with a fact-checked gold standard, comparing selected answers against the gold standard rather than against the set of unaltered responses; this would separate tamper resistance from answer quality."],"forward_implications":["Smart-contract applications can request open-ended reasoning from LLMs through an oracle contract and receive a single consensus answer, with gas costs on the order of a few hundred thousand per request.","An oracle network with fewer than half malicious nodes can tolerate random responses, cheaper-model substitution, and prompt-injected incorrect answers, keeping data accuracy near 1.0 across the BASE, MIX, and PRO datasets.","Because SenteTruth treats node credibility as a learned quantity, honest nodes accumulate weight over time while persistently deviating nodes are suppressed, so the system improves as it operates.","The paper explicitly argues that replacing SBERT with a stronger text encoder should further improve SenteTruth's aggregation accuracy.","The framework makes blockchain contracts capable of handling ambiguity and dynamic scenarios, moving from pure automation toward contract intelligence."],"supporting_citations":[{"why":"Supplies the SBERT sentence-embedding model that maps node answers to vectors for semantic relatedness.","marker":"[35]"},{"why":"Supplies the truth-discovery framework of weighted aggregation plus credibility update that SenteTruth adapts from numeric data to text.","marker":"[33]"},{"why":"Provides the DAON oracle baseline whose majority-voting aggregation SenteTruth is compared against.","marker":"[21]"},{"why":"Provides the Chainlink 2.0 oracle network context, including the commit-reveal exchange that C-LLM uses to prevent freeloading.","marker":"[22]"},{"why":"Shows truth discovery applied to blockchain oracle problems on numeric data, the approach SenteTruth extends to textual LLM responses.","marker":"[26]"},{"why":"Introduces the initial SmartLLM concept of calling LLMs through oracles, the line of work this paper builds on.","marker":"[10]"}],"fun_headline_variants":["SenteTruth: semantic voting beats 40% malicious nodes in LLM oracle","Smart contracts get LLM smarts with tamper-proof aggregation","Blockchain LLM integration: semantic aggregation boosts trust by 17.74%","LLM-powered smart contracts: new method thwarts data sabotage"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the 'truth' is whatever answer is semantically closest to the majority of nodes, so if honest nodes give diverse answers or share the same misconception, the aggregated answer can still be wrong.","fun_headline_variants_meta":{"raw":{"variants":["SenteTruth: semantic voting beats 40% malicious nodes in LLM oracle","Smart contracts get LLM smarts with tamper-proof aggregation","Blockchain LLM integration: semantic aggregation boosts trust by 17.74%","LLM-powered smart contracts: new method thwarts data sabotage"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000527,"raw_usage":{"total_tokens":2566,"prompt_tokens":991,"completion_tokens":1575,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":607,"completion_tokens_details":{"reasoning_tokens":1496}},"tokens_in":607,"tokens_out":1575,"duration_ms":13049,"temperature":1.0,"reasoning_tokens":1496,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:38:48.090252+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a dataset where 60 percent of nodes give correct but semantically varied answers to an open question and 40 percent collude on a single plausible but wrong answer; SenteTruth would pick the colluding answer because it maximizes semantic relatedness. Observing that outcome would show the central claim holds for tamper resistance but fails for factual answer quality.","supporting_citations":[{"cited_title":"A survey on truth discovery,","cited_arxiv_id":null,"evidence_quote":"Supplies the truth-discovery framework of weighted aggregation plus credibility update that SenteTruth adapts from numeric data to text."},{"cited_title":"Daon: A decentralized autonomous oracle network to provide secure data for smart contracts,","cited_arxiv_id":null,"evidence_quote":"Provides the DAON oracle baseline whose majority-voting aggregation SenteTruth is compared against."},{"cited_title":"Chainlink 2.0: Next steps in the evolution of decentralized oracle networks,","cited_arxiv_id":null,"evidence_quote":"Provides the Chainlink 2.0 oracle network context, including the commit-reveal exchange that C-LLM uses to prevent freeloading."},{"cited_title":"A decentralized truth dis- covery approach to the blockchain oracle problem,","cited_arxiv_id":null,"evidence_quote":"Shows truth discovery applied to blockchain oracle problems on numeric data, the approach SenteTruth extends to textual LLM responses."},{"cited_title":"Smartllm: A new oracle system for smart contracts calling large language models,","cited_arxiv_id":null,"evidence_quote":"Introduces the initial SmartLLM concept of calling LLMs through oracles, the line of work this paper builds on."}],"review_version":1}