{"id":"f41e2cc0-df3c-459e-8126-81b684abd429","arxiv_id":"2506.21033","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"BLOCKS combines a Cosmos-based blockchain, a reputation mechanism, and a priority cache to let LLMs retrieve prompts from untrusted knowledge silos.","lead":"The paper presents BLOCKS, a blockchain-based framework that lets separate knowledge repositories share distilled prompts with large language models while tracking contributor trust through a reputation system. It is an early engineering attempt to make cross-organization knowledge sharing practical for LLM retrieval tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The security guarantee hinges on Assumption 1, which is never validated, and Eq. (10) does not follow from Eqs. (2)-(5); the trustworthiness claim is not established.","rationale":"The paper's central claim is that BLOCKS achieves trustworthy, low-latency knowledge sharing in a blockchain environment. The reader correctly identifies Assumption 1 as the weakest point: the entire reputation and PoI security argument depends on honest validators being able to distinguish malicious from honest prompts. I agree, and I would sharpen the concern in two ways. First, the system's validator model is described in Section III-A as a low-cost model plus a threshold, but no experiment measures how well this model actually separates adversarial prompts from honest ones. The reputation plots in Section V only show that simulated malicious nodes lose reputation when the simulation already labels them as malicious; they do not establish that the detection mechanism works on unseen attacks. Second, even granting perfect discrimination, the proof of convergence to zero reputation is not supported by the stated equations. Equations (2) and (5) place the updated reputation on both sides of the equality; if they are intended as EMA updates, the limiting reputation is 1−Mean(CS), which is zero only if Mean(CS)=1. Given the CS distance formula in Eq. (1), a malicious validator disagreeing with the honest majority can have CS below 1 and therefore a positive asymptotic reputation. Eq. (10) thus appears to be asserted rather than derived, and Eqs. (11)–(13) inherit this gap. This is not a disagreement with the research direction; the system has a real code release and a plausible integration of caching, reputation, and consensus. But the security property that makes the framework trustworthy is currently neither empirically validated nor formally established. The right verdict is still conditional: the paper could be acceptable if the authors provide either a concrete validator-discrimination evaluation or a rigorous convergence proof under a precise model of validator scores. Since the reader already reached CONDITIONAL, my stress-test does not change that verdict, which is why I recommend UNCHANGED.","tokens_in":8746,"tokens_out":5926,"duration_ms":73940,"concrete_test":"Run a held-out discrimination test with the Section V validator stack: take 100 honest prompts and 100 adversarial prompts (prompt-injection and false-knowledge variants), have the low-cost model with threshold score them, and compute ROC AUC for malicious-vs-honest classification. Independently simulate Eqs. (2)–(5) with time subscripts using the observed score distributions under f=3, n=11; record whether malicious-node reputation converges to 0 over 500 iterations. If AUC is materially below 1 or the simulated limiting reputation is positive, Assumption 1/Eq. (10) fails and the security claim needs revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that BLOCKS can provide trustworthy knowledge from untrusted silos under f<n/3 Byzantine failures—depends entirely on Assumption 1 (Section IV-B), that honest validators can recognize malicious prompts and score them lower. Section III-A delegates this to a low-cost model plus a threshold, but the paper reports no discrimination accuracy for that model; Section V only simulates predetermined malicious strategies and plots reputation, which does not test out-of-sample detectability. There is also an internal gap: Eqs. (2) and (5) write R_l and R_v on both sides. If interpreted as update rules, the fixed point is 1−Mean(CS), not 0; under Eq. (1), a minority malicious validator can have CS ≈ (n−f)/(n−1), which for f=3, n=11 is 0.8, yielding a positive limiting reputation of ≈0.2. Eq. (10)'s claim that malicious reputations tend to 0 is therefore not derived from the stated equations, and Eq. (12) for PoI inherits this problem. A reader cannot verify the central security property from the text.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BLOCKS, a Cosmos/Tendermint-based framework for cross-silo knowledge sharing tailored to LLM services. Knowledge is distilled into prompts on-chain; a smart-contract reputation mechanism with cross-validation, a Proof-of-Impact (PoI) consensus, a PROCache priority cache, and a hash-bucket key-value store are introduced to handle incentives, quality of service, and security. The experiments use 11 nodes and 253 questions from WikiQA, TruthfulQA, and MathQA, reporting reputation decay of malicious nodes, cache delay/reputation comparisons against LFU and LRU-k, and an approximately 80% reduction in blockchain storage.","tokens_in":9026,"tokens_out":3545,"duration_ms":39920,"significance":"If the security and incentive properties were rigorously established, BLOCKS would address a real problem: coordinating untrusted knowledge silos for LLM retrieval. The paper ships a concrete implementation on Cosmos SDK with a public code link, and the measured storage reduction and PROCache comparison are concrete empirical contributions. However, the central trustworthiness claim is not established: the security analysis rests on an unvalidated assumption and on convergence claims that do not follow from the equations as written. The evaluation is too small and too scripted to substitute for that missing analysis.","major_comments":[{"comment":"The claim that malicious reputations converge to zero is not derivable from the stated update rules. Eqs. (2) and (5) write R_l and R_v on both sides; if these are interpreted as fixed-point equations, their nontrivial fixed point is 1 - Mean(CS), which can be positive under the stated Byzantine bound (for example, with n=11 and f=3, a minority malicious validator can have CS around 0.8, yielding a limiting reputation of about 0.2). Eq. (10) needs a time-indexed derivation that explicitly shows how Mean(CS) for malicious nodes and the threshold mechanism drive reputation to zero; otherwise the PoI reward exclusion in Eq. (12) is unsupported.","section":"Section IV-B, Eqs. (2)-(5), (10)"},{"comment":"Assumption 1 is load-bearing but never validated. Section III-A delegates attack detection to a low-cost model plus a threshold, but the paper reports no discrimination accuracy for that model. Section V-B simulates predetermined malicious strategies and plots the reputation trajectories; because those trajectories are produced by the same reputation equations being tested, they do not demonstrate that honest validators can reliably distinguish malicious from honest prompts out-of-sample. Please report the validation model's detection accuracy (e.g., precision/recall or ROC) on held-out malicious prompts, or soften the security claims accordingly.","section":"Section III-A and Section V-B, Assumption 1"},{"comment":"Several equations cannot be checked because symbols are undefined or inconsistently used. In Eq. (1), m is defined as the number of validations but does not appear, while n appears both as the number of LLM services and as the denominator size. In Eq. (7), R_b is introduced as a 'reputation threshold' but its value and update rule are not given, and the priority expression has no clear range when R_t < R_b. Please define every symbol where it first appears and align the notation across Eqs. (1)-(8).","section":"Section III-A, Eq. (1), and Eq. (7)"},{"comment":"The reward formula U_p = R_b × R_p / Σ R_p reuses R_b, which was previously a cache reputation threshold, and does not define its value or relationship to provider reputation. This appears to conflate two different quantities. Please clarify whether R_b is a minimum reputation eligibility threshold or a typo for another variable, and state how the constant is chosen.","section":"Section IV-A, Eq. (8)"},{"comment":"The empirical evaluation is too small to support the strength of the claims. The setup uses 11 nodes and 253 questions, and Figs. 4-5 report a single simulation trajectory with confidence intervals across agents rather than across repeated runs with different random seeds. The security experiments would be more convincing with multiple seeds, statistical error bars, and a direct comparison of the validation model's detection performance against the simulated attack strategies.","section":"Section V-B"}],"minor_comments":[{"comment":"The heading 'PUTEVERYTHINGTOGETHER' appears to be a typo; it should read 'PUT EVERYTHING TOGETHER.'","section":"Section IV heading"},{"comment":"The URL for MathQA is misspelled as 'aclantholoy.org'; it should be 'aclanthology.org.'","section":"Reference [27]"},{"comment":"Figures 6 and 7 would benefit from explicit axis labels and units; in Fig. 7, 'Number of index' should be 'Number of stored prompt indices.'","section":"Fig. 7"},{"comment":"The notation in Eq. (3) uses V_i, R_i, Acc, and R_l without defining the index ranges or the exact meaning of 'Acc' (presumably accuracy); please define these terms.","section":"Section III-A, Eq. (3)"},{"comment":"The abstract describes 'extensive experiments,' but the evaluation uses subsets of three datasets and 253 questions; please calibrate the wording to the actual scale.","section":"Abstract and Section V"}],"recommendation":"major_revision","confidential_remarks":"The main issue is not novelty but rigor: the security proof in Section IV-B is a sketch and the convergence claims do not follow from the stated equations. The storage and cache measurements are independent of the security analysis and could be salvageable, so a major revision rather than rejection seems appropriate if the authors can supply a corrected derivation, validate Assumption 1, and run a more robust evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Zhaojiacheng and colleagues have built a real system—BLOCKS runs on Cosmos, the code is public, and the evaluation covers three QA datasets. The PROCache priority formula (Eq. 7) is a thoughtful combination of frequency, cost, size, and reputation, and the measured 80% reduction in on-chain storage is concrete. The motivation—siloed knowledge for LLM augmentation—is genuine, and the four-role design is clear.\n\nThe problem is the security analysis. The stress-test note is right: Eq. (2) and Eq. (5) are self-referential update rules, so their fixed point is 1−Mean(CS), not zero. Eq. (10) claims malicious reputations tend to 0, but that does not follow from the equations as written. The paper's own experiments show decay, but that is under a simulated scenario where malicious behavior is predefined. The real question is whether honest validators can actually score malicious prompts lower, which is Assumption 1. That assumption is never validated—there is no discrimination accuracy, no out-of-sample test. So the central claim that BLOCKS provides trustworthy knowledge from untrusted silos is not established.\n\nThere are other soft spots. Some variables are undefined (m, n in Eq. 1; R_b in Eq. 7). The experiments are small-scale: 11 nodes, 253 questions. The novelty is modest—Cosmos, RAG, reputation contracts, and caching are known; the new pieces are incremental. The 'first blockchain-based external knowledge framework for LLMs' claim is a bit strong given prior blockchain knowledge-sharing work, even if not LLM-specific.\n\nThe paper is a serious engineering effort with honest code. The security proof needs a proper fixed-point analysis and a validation of Assumption 1. With those, the framework could be credible. As it stands, the trustworthiness guarantee is unsupported. I'd send it to review—a good referee could force the authors to close this gap—but I wouldn't cite the security results until they do.","headline":"Real implementation and a sensible cache, but the security proof does not hold: the central trustworthiness claim rests on an unvalidated assumption.","tokens_in":9558,"tokens_out":3282,"would_cite":false,"duration_ms":35837,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BLOCKS claims that blockchain can turn isolated private knowledge into trustworthy LLM context by sharing distilled prompts instead of raw data.","keywords":["blockchain","large language models","knowledge sharing","reputation mechanism","retrieval-augmented generation","caching","proof of impact","cross-silo knowledge"],"falsifier":"Run BLOCKS against an adversarial benchmark of indirect prompt injection attacks specifically crafted to pass low-cost validator checks; if a malicious prompt's reputation stays high, or a provider whose prompts are later shown harmful continues to receive rewards, then Assumption 1 fails and the security guarantees in equations (10) and (12) collapse.","tokens_in":8571,"feed_emoji":"🔗","tokens_out":3944,"duration_ms":42677,"temperature":0.7,"pith_summary":"BLOCKS is a proposed blockchain-based platform whose central claim is that isolated, privacy-sensitive knowledge sources can be safely pooled into a shared external-knowledge service for LLMs without exposing raw data. Instead of moving raw data, each silo distills local knowledge into prompts, and the blockchain records, validates, and rewards those prompts. The paper argues that a smart-contract reputation mechanism combined with a Proof of Impact consensus makes honest contribution the only profitable strategy, while a priority-aware cache (PROCache) keeps retrieval fast enough for latency-sensitive LLM service. If this works, users of an LLM could query many private data silos and receive trustworthy, domain-specific context to reduce hallucination, while silo owners retain control of their underlying data and gain an incentive to share.","feed_headline":"Siloed data feeds LLMs via blockchain-shared prompts","feed_subtitle":"A reputation mechanism punishes bad actors and a cache cuts ledger storage by about 80 percent.","key_machinery":"The load-bearing mechanism has three parts. First, a smart-contract reputation system defined by equations (2)–(5) tracks interdependent reputations of LLM services, prompts, knowledge providers, and validators, using consistency (CS) and confidence (CF) metrics, with an exponential moving-average update. Second, Proof of Impact (PoI) consensus lets the next block be proposed by the knowledge provider with the highest impact $C = R(\\beta A_p + (1-\\beta) A_v)$, where $R$ is reputation and $A_p, A_v$ are access counts for prompt generation and validation; rewards are distributed proportionally to reputation, so equation (12) drives malicious providers' rewards to zero. Third, PROCache computes a priority score $\\text{Priority} = (\\text{Frequency} \\times \\text{Cost} / \\text{Size})^{R_t - R_b}$ that promotes prompts based on both access utility and reputation, defending the cache against adversarial pollution. Supporting components include the Query Generation Module (QGM) for turning queries into retrieval rationales and a hash-bucket key-value store layered over the Cosmos IA VL-Tree for compact on-chain storage.","core_discovery":"On the paper's own terms, the result is a complete four-role blockchain workflow—knowledge providers, LLM servers, validators, and external storage—that turns distributed private knowledge into retrievable, verifiable prompts. The discovery claim is that the combination of reputation formulas, threshold validation against prompt injection, and Proof of Impact consensus aligns incentives such that malicious providers' reputations and rewards converge to zero while honest providers' reputations stabilize at the true quality level of their contributions. The paper reports that this design, implemented on a Cosmos-based chain, achieves efficient LLM knowledge sharing in a blockchain environment, with PROCache reducing ledger storage by approximately 80% and maintaining higher in-cache reputation than LFU or LRU-2 baselines. To the best of the authors' knowledge, this is the first blockchain-based external knowledge framework tailored specifically for LLMs.","pith_inferences":["If Assumption 1 holds, the same reputation-plus-PoI machinery could be reused for other markets of untrusted machine-generated content, such as shared datasets, summaries, or code snippets, not only LLM prompts.","A natural extension the paper leaves implicit is measuring end-to-end LLM answer quality, for example hallucination rates with and without BLOCKS on a fixed multi-silo benchmark, rather than only reputation and cache metrics.","The PROCache priority formula could be extracted as a standalone defense against adversarial cache pollution in any retrieval system, independent of its blockchain setting.","The paper's security analysis assumes honest validators can reliably distinguish malicious prompts; an empirical stress test against state-of-the-art indirect prompt injection attacks would show how far this assumption extends in practice."],"forward_implications":["Knowledge silos can participate without publishing raw data, because only distilled prompts, hashes, and reputation records go on-chain.","Under the Byzantine bound $f < n/3$, malicious actors using self-promotion, collusion, or slandering see their reputations and rewards converge to zero, making honest contribution the stable strategy.","PROCache reduces blockchain ledger storage by about 80% in the reported setup while keeping in-cache reputation higher than LFU and LRU-2 baselines, addressing a central QoS bottleneck.","LLM services gain a direct API path through QGM plus PROCache retrieval, allowing external knowledge to be incorporated without requiring silos to expose their underlying datasets."],"supporting_citations":[{"why":"Supplies the Retrieval-Augmented Generation background that motivates external knowledge integration for LLMs.","marker":"[2]"},{"why":"Provides the Chain-of-Knowledge approach and the Query Generation Module that BLOCKS adapts for query rationales.","marker":"[3]"},{"why":"Defines the indirect prompt injection attack benchmark that the reputation and validation mechanism must defend against.","marker":"[6]"},{"why":"Provides the Cosmos blockchain infrastructure, including the IA VL-Tree storage and bank module that BLOCKS builds on.","marker":"[15]"},{"why":"Supplies the BERTScore metric used in the paper's motivation experiment comparing single-source versus multi-source knowledge.","marker":"[18]"},{"why":"Contributes the TruthfulQA dataset used in the preliminary multi-source knowledge experiment.","marker":"[19]"},{"why":"Contributes the WikiQA dataset used in both the motivation experiment and the main evaluation.","marker":"[20]"},{"why":"Underlies the PROCache Retrieve function for prompt embedding, indexing, and retrieval.","marker":"[23]"},{"why":"Supplies the threat model and adversarial strategies (self-promotion, collusion, slandering) that the security analysis examines.","marker":"[24]"}],"fun_headline_variants":["Blockchain rewards honest silos, punishes bad LLM knowledge","Blockchain shares cross-silo LLM knowledge with 80% less storage","Blockchain is first to feed LLMs from private silos","Reputation-based blockchain shares siloed knowledge for LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole reward and security story depends on honest validators actually being able to tell malicious prompts from honest ones, so that malicious content reliably receives lower scores.","fun_headline_variants_meta":{"raw":{"variants":["Blockchain rewards honest silos, punishes bad LLM knowledge","Blockchain shares cross-silo LLM knowledge with 80% less storage","Blockchain is first to feed LLMs from private silos","Reputation-based blockchain shares siloed knowledge for LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001658,"raw_usage":{"total_tokens":6551,"prompt_tokens":883,"completion_tokens":5668,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":499,"completion_tokens_details":{"reasoning_tokens":5594}},"tokens_in":499,"tokens_out":5668,"duration_ms":41738,"temperature":1.0,"reasoning_tokens":5594,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:34:44.379706+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run BLOCKS against an adversarial benchmark of indirect prompt injection attacks specifically crafted to pass low-cost validator checks; if a malicious prompt's reputation stays high, or a provider whose prompts are later shown harmful continues to receive rewards, then Assumption 1 fails and the security guarantees in equations (10) and (12) collapse.","supporting_citations":[{"cited_title":"Chain-of-knowledge: Grounding large language models via dynamic knowledge adapting over heterogeneous sources,","cited_arxiv_id":null,"evidence_quote":"Provides the Chain-of-Knowledge approach and the Query Generation Module that BLOCKS adapts for query rationales."},{"cited_title":"Wikiqa: A challenge dataset for open- domain question answering,","cited_arxiv_id":null,"evidence_quote":"Contributes the WikiQA dataset used in both the motivation experiment and the main evaluation."},{"cited_title":"Reputation-based distributed knowledge sharing system in blockchain,","cited_arxiv_id":null,"evidence_quote":"Supplies the threat model and adversarial strategies (self-promotion, collusion, slandering) that the security analysis examines."}],"review_version":1}