{"id":"b70c9d75-1ce4-4ec3-b1ae-730f699f191d","arxiv_id":"2506.11298","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Jelly is a Protobuf-based binary RDF serialization format supporting streaming and claiming higher speed and compression than existing formats.","lead":"Jelly is a new binary format for storing and streaming RDF knowledge graph data, built on Protocol Buffers, with open-source Java and Python tools. It promises faster processing and smaller files than common text formats like Turtle or N-Quads, though the paper asks readers to check an external website for its benchmark numbers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim of 'most compressed and fastest RDF format' depends entirely on external self-published benchmarks; the manuscript provides no dataset list, methodology, or comparison protocol, making the claim unverifiable from the paper.","rationale":"The paper is an artifact/tool paper; its scientific contribution is the claim that Jelly is both smaller and faster than existing RDF serialization formats in the two main Java RDF libraries. That claim is made in Section 2 with numbers sourced exclusively from a website. The reader's conditional verdict rests on this same observation. I agree that this is the load-bearing concern: if the external benchmarks are not reproducible or are configured favorably, the headline result collapses because there is no in-paper fallback. The absence of dataset names, hardware details, comparison commands, and an aggregation definition for the 16.2% average prevents a reviewer from checking the claim. I do not see an internal inconsistency in the format design; the streaming compression approach is plausible and the open-source implementation is a real artifact. However, external benchmarks are not part of the peer-reviewed record. A concrete reproduction of the published numbers, or inclusion of a benchmark section in a revised version, would resolve the concern. Since the reader already marked the paper conditional, my stress test does not change the verdict.","tokens_in":6461,"tokens_out":3436,"duration_ms":35235,"concrete_test":"Run the RiverBench datasets listed on the Jelly performance page with Jelly-JVM 2.7.0 using the exact configuration documented there (table sizes, replacement policy, frame size) on a publicly specified machine, and independently measure compression ratio, serialization MT/s, and deserialization MT/s against RDF4J Binary and Jena's binary formats. If the measured averages do not reproduce the reported 16.2% and 7.28/15.16 MT/s, or if the comparison protocol excludes a relevant format, the central claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 2 (Performance), the paper states that Jelly-JVM 2.7.0 achieves an average compression ratio of 16.2% relative to N-Triples, serialization of 7.28 MT/s, and deserialization of 15.16 MT/s, and concludes it is the most compressed and fastest RDF format in Apache Jena or RDF4J. All these numbers come from a project website (w3id.org/jelly/dev/performance) with no in-paper dataset list, machine specification, benchmark methodology, or aggregation method. The claim is a comparative superlative: it requires a controlled comparison against every relevant format in both ecosystems (Jena Thrift/Protobuf, RDF4J Binary, possibly others). The paper only names RDF4J Binary as the closest contender with speed ratios, again citing the website. Furthermore, the compression scheme depends on parameters (lookup table sizes, replacement policy, frame size) that are not fixed in the paper; if the benchmark used a tuned configuration or datasets favorable to Jelly, the 'most compressed and fastest' claim would not generalize. Because the entire value proposition of the paper is performance, this missing evidence is the load-bearing weakness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Jelly, a binary RDF serialization format based on Protocol Buffers, designed for high-throughput streaming and batch serialization. Jelly uses streaming compression with fixed-size lookup tables for IRI prefixes/suffixes and datatypes, delta encoding, and per-term repetition skipping. The paper reports implementations for Java (Jelly-JVM) and Python (pyjelly), integrations with Jena, RDF4J, Titanium, and RDFLib, a command-line tool, and several use cases. The central quantitative claim is that Jelly-JVM 2.7.0 achieves an average compression ratio of 16.2% relative to N-Triples, serialization at 7.28 MT/s, and deserialization at 15.16 MT/s, making it 'the most compressed and the fastest RDF format implemented in either Apache Jena or RDF4J.' These numbers are cited from an external project website rather than from data presented in the paper.","tokens_in":6779,"tokens_out":2972,"duration_ms":33746,"significance":"If the performance claims are reproducible, Jelly is a practically useful addition to the RDF tooling ecosystem: it provides an open, language-neutral binary format with multiple library integrations and a streaming model that existing W3C formats lack. The availability of open-source implementations and a CLI tool lowers adoption barriers, and the Nanopublication Network use case (Section 4) suggests real deployment value. The paper's significance is, however, currently contingent on external, self-published benchmarks; the manuscript itself does not supply the evidence needed to verify the superlative performance claims. The protocol description and tooling contributions are credible and potentially citable, but the performance-centric framing of the paper cannot be assessed from the submitted text alone.","major_comments":[{"comment":"The central claims (compression ratio of 16.2%, serialization of 7.28 MT/s, deserialization of 15.16 MT/s, and 'the most compressed and the fastest RDF format implemented in either Apache Jena or RDF4J') rest entirely on an external website (w3id.org/jelly/dev/performance). The paper provides no dataset list, no machine specification, no benchmark methodology, no comparison protocol, and no aggregation method (e.g., arithmetic vs. geometric mean across datasets). Because performance is the paper's main selling point, this missing evidence is load-bearing. Please include a reproducible benchmark description in the paper (or an appendix) with the exact dataset versions, hardware, JVM settings, and aggregation procedure, or alternatively reformulate the claims to reflect only what is demonstrated in the paper.","section":"Section 2, Performance paragraph"},{"comment":"The compression behavior depends on parameters that are not fixed in the paper: the sizes of the three lookup tables, the replacement policy (e.g., LRU is only given as an example), and the frame size. The benchmark configuration for Jelly-JVM 2.7.0 is not reported. If the published numbers were obtained with a tuned configuration or a dataset mix favorable to Jelly, the claim that these results generalize to 'the most compressed and fastest' format would not be supported. Please specify the exact parameter values used in the benchmark, or state the range of configurations over which the performance claim holds.","section":"Section 2, Compression paragraph"},{"comment":"The superlative performance claim is scoped to formats implemented in Apache Jena or RDF4J, which is stated, but the paper does not provide a direct comparison table even for those systems; it only names RDF4J Binary as the closest contender and gives speed ratios, again citing the external website. For a claim of being the fastest and most compressed, the paper should show per-format numbers for at least Jena Thrift/Protobuf, Jena's standard N-Triples, RDF4J Binary, and RDF4J's N-Triples, in a table in the paper, so that a reader can verify the relative performance without navigating to external resources.","section":"Section 5, Related Work"}],"minor_comments":[{"comment":"The corresponding author's email address appears corrupted as 'envel⌢pe-⌢penpiotr@neverblink.eu'; please replace it with a plain-text email address.","section":"Section 1, author affiliation block"},{"comment":"The Maven artifact version is shown as 3.4.0 while the performance numbers refer to Jelly-JVM 2.7.0; please clarify which version the benchmarks use and whether the API in the code snippet (e.g., JellyLanguage.JELLY) is current.","section":"Section 3.1, code snippet"},{"comment":"The text says 'the Jelly-RDF protocol described in Section 2', but Section 2 describes the Jelly serialization protocol and does not describe a distinct Jelly-RDF protocol; please either add a description or rephrase to avoid the mismatch.","section":"Section 4, database replication item"},{"comment":"The full compression algorithm is deferred to an external specification; while this is acceptable for a workshop paper, including a concise algorithmic description in an appendix would make the paper more self-contained and would help readers assess the streaming and constant-memory claims.","section":"Section 2, Compression paragraph"}],"recommendation":"major_revision","confidential_remarks":"The paper is a workshop-style systems/tool paper, and the protocol/tooling contributions are plausible. The main concern for the editor is that the headline performance claims are self-published on the authors' own website and use the authors' own benchmark suite (RiverBench) as the basis, with no in-paper methodology. I recommend requiring either a complete benchmark appendix or a significant toning-down of the superlative phrasing. A comparative evaluation by an independent party, or at least a fully specified reproducibility package, would materially increase confidence in the central claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look. Jelly is a real artifact: an open, Protobuf-based RDF serialization format with streaming dictionary compression, delta encoding, and support for grouped streams of graphs/datasets. The Java and Python implementations are public, integrate with Jena, RDF4J, RDFLib, and Titanium, and the CLI is genuinely useful for debugging and transcoding. The protocol specification is open, the code is out there, and the Nanopublication Network use case gives a concrete sense of what the format can do. As a tool description, this is solid.\n\nThe soft spot is exactly where the stress-test puts it. Section 2's headline numbers—16.2% compression ratio, 7.28 MT/s serialization, 15.16 MT/s deserialization—come from a project website, not the manuscript. There is no dataset list, no machine spec, no methodology, no comparison protocol. The claim 'most compressed and fastest' is a comparative superlative that requires controlled evaluation, and the paper doesn't provide enough for a reader to judge it. The benchmarks are also self-published by the same team. That doesn't mean the numbers are wrong, but it means the central value proposition currently rests on an external link. The paper even cites RDF4J Binary speed ratios from the same website. For a workshop paper, this is a notable gap; for a journal it would be disqualifying as-is.\n\nOne small technical ambiguity worth flagging: the compression description says lookup-table entries 'can be replaced by new ones using any policy (e.g., least recently used).' If the policy is truly arbitrary, a decoder can't reconstruct the table state unless the spec fixes it or communicates it. I suspect the actual spec pins this down, but the paper's wording invites the question and it should be clarified.\n\nThe related-work coverage is fair, the AI-use declaration is honest, and the paper doesn't oversell: it carefully scopes the superlative to Jena and RDF4J and says 'as far as we are aware.' Self-citation isn't a real problem here because the cited work (RDF Stream Taxonomy, RiverBench) is genuinely the basis for the design.\n\nBottom line: this is a paper for Semantic Web practitioners who care about binary RDF exchange, and it deserves a serious referee. I'd accept it with a request to move the benchmark methodology into the paper—at least a summary of datasets, hardware, and protocol—and to clarify the replacement policy. The artifact itself is reproducible, which is a meaningful point in its favor.","headline":"A genuinely useful and honestly described binary RDF format, but its central performance claim is currently unverifiable from the paper itself.","tokens_in":7211,"tokens_out":2943,"would_cite":false,"duration_ms":32841,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Jelly is a binary RDF serialization format that targets both batch and streaming use, claiming compression to about 16% of N-Triples size with multi-million-triples-per-second throughput.","keywords":["RDF","serialization format","RDF stream processing","knowledge graphs","Semantic Web data formats","Protocol Buffers","binary serialization","streaming compression"],"falsifier":"Re-run the exact benchmark suite behind the published performance page on the same machine and dataset mix used by the authors, comparing Jelly-JVM 2.7.0 with the closest competing binary RDF format; if the average compression ratio, serialization speed, or deserialization speed falls materially short of 16.2%, 7.28 million triples per second, or 15.16 million triples per second, or if the competitor wins, the central claim collapses.","tokens_in":6243,"feed_emoji":"🗜️","tokens_out":8601,"duration_ms":89491,"temperature":0.7,"pith_summary":"Jelly is a binary RDF serialization format aimed at a gap left by text formats such as Turtle, N-Quads, and JSON-LD: none of them parses quickly, compresses tightly, and natively represents RDF streams at the same time. The paper claims Jelly fills that gap with one format that works for both batch files and streaming pipelines, and it supports the claim with an open protocol specification, Java and Python implementations, a command-line tool, and two deployed use cases. The headline figures, drawn from benchmarks the authors publish on the project website, are an average compression ratio of 16.2% relative to N-Triples, serialization at 7.28 million triples per second, and deserialization at 15.16 million triples per second. If those figures hold, Jelly would be a practical drop-in replacement for common RDF formats in large dumps, client-server communication, and stream ingestion.","feed_headline":"Jelly packs RDF to 16% size at millions of triples per second","feed_subtitle":"The Protocol-Buffers format targets streaming, constant-memory RDF processing with ready-made Java and Python tooling.","key_machinery":"The load-bearing mechanism is Jelly's streaming compression scheme, specified in the Jelly protocol. The encoder maintains three fixed-sized string lookup tables, one for IRI prefixes, one for suffixes, and one for datatypes, indexed by integers; as the stream moves, old entries can be evicted and replaced, which keeps memory bounded for indefinite inputs. IRIs are written as pairs of variable-length integer table references, delta coding uses the value zero to mean 'repeat the last ID or increment it by one,' and a term identical to the corresponding term in the previous statement is omitted altogether. Frames group statements into batches or whole graphs/datasets, so the format is equally a stream of triples and a stream of graphs. Because Protobuf supplies the wire encoding, the binary layer is generated by standard compilers and implementers only write the translation between Protobuf messages and their RDF library's data types.","core_discovery":"The paper's central claim is that a single binary format, Jelly, can simultaneously give RDF practitioners high-throughput serialization, small files, and native stream support without sacrificing ease of integration. The format encodes RDF as framed Protobuf messages, and its compression comes from three fixed-sized lookup tables for IRI prefixes, IRI suffixes, and datatypes, together with variable-length integer identifiers, delta coding, and omission of terms that repeat in consecutive statements. The authors report that on their published benchmarks, Jelly-JVM 2.7.0 reaches an average compression ratio of 16.2% relative to N-Triples, 7.28 million triples per second serialization, and 15.16 million triples per second deserialization, which they state makes it the most compressed and fastest RDF format available in either of the two major Java RDF ecosystems. They also document practical deployments, including a nanopublication service where switching to Jelly streams over HTTP reduced the time to retrieve 60,000 nanopublications from over an hour to less than four seconds.","pith_inferences":["The durable contribution is likely the compression scheme rather than the headline numbers, since the benchmark ranking is tied to a specific version and will shift as competing formats improve.","The repeated-term and delta coding should make compression sensitive to triple ordering, so sorting triples by subject could yield even smaller files, a property the paper does not quantify.","Jelly files are not random-access: an application that needs to query a single triple without scanning the stream will still need an indexed format, so Jelly is complementary to stores and indexes."],"forward_implications":["RDF files and database dumps can be stored at roughly one-sixth the size of N-Triples and read back faster, shortening maintenance and transfer times.","Streaming RDF over message brokers can run in constant memory, because each frame is independent and the encoder processes one statement at a time.","Existing applications using mainstream Java RDF libraries can add Jelly as a plugin and use it wherever text formats are accepted.","Service-to-service RDF exchange gains a natively streaming option, with the reported nanopublication network upgrade cutting retrieval from over an hour to under four seconds.","Because the binary format is generated from a Protobuf schema, new language bindings can reuse the same specification instead of reimplementing by hand."],"supporting_citations":[{"why":"Introduces the original Jelly streaming format for edge-cloud RDF, which this paper presents as a formalized open protocol with broader tooling.","marker":"[1]"},{"why":"Defines the flat and grouped RDF stream taxonomy that motivates Jelly's frame-based design.","marker":"[2]"},{"why":"States the streaming-processing criteria (bounded memory, one triple at a time) that Jelly's design is measured against.","marker":"[3]"},{"why":"Provides the benchmark datasets behind the published compression and throughput figures.","marker":"[4]"},{"why":"Documents the nanopublication network use case that motivates the stream-retrieval performance claim.","marker":"[5]"},{"why":"Describes the binary RDF formats in one major Java RDF library that Jelly must outperform in the comparison.","marker":"[8]"},{"why":"Describes the compressed binary RDF format in the other major Java RDF library, the closest competitor in the benchmarks.","marker":"[9]"}],"fun_headline_variants":["Jelly binary RDF: 16% size, 15M triples/s","RDF at 15M reads/s with 84% smaller files via Jelly","Stream RDF at millions of triples/s with Jelly's Protobuf","Jelly: fastest RDF serialization, smallest files, native streaming","Jelly RDF: compression to 16%, speed to 15M triples/s"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central performance claim rests entirely on benchmarks published on the project website rather than on measurements in this paper, so if those external benchmarks are not reproducible or were obtained on unrepresentative hardware or datasets, the \"most compressed and fastest\" assertion has no in-paper support.","fun_headline_variants_meta":{"raw":{"variants":["Jelly binary RDF: 16% size, 15M triples/s","RDF at 15M reads/s with 84% smaller files via Jelly","Stream RDF at millions of triples/s with Jelly's Protobuf","Jelly: fastest RDF serialization, smallest files, native streaming","Jelly RDF: compression to 16%, speed to 15M triples/s"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00024,"raw_usage":{"total_tokens":1524,"prompt_tokens":954,"completion_tokens":570,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":462}},"tokens_in":570,"tokens_out":570,"duration_ms":5764,"temperature":1.0,"reasoning_tokens":462,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:10:35.632981+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the exact benchmark suite behind the published performance page on the same machine and dataset mix used by the authors, comparing Jelly-JVM 2.7.0 with the closest competing binary RDF format; if the average compression ratio, serialization speed, or deserialization speed falls materially short of 16.2%, 7.28 million triples per second, or 15.16 million triples per second, or if the competitor wins, the central claim collapses.","supporting_citations":[{"cited_title":"Sowiński, P","cited_arxiv_id":null,"evidence_quote":"Defines the flat and grouped RDF stream taxonomy that motivates Jelly's frame-based design."},{"cited_title":"Bifet, R","cited_arxiv_id":null,"evidence_quote":"States the streaming-processing criteria (bounded memory, one triple at a time) that Jelly's design is measured against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the nanopublication network use case that motivates the stream-retrieval performance claim."},{"cited_title":"https://rdf4j.org/documentation/ reference/rdf4j-binary/, accessed on 12 June 2025","cited_arxiv_id":null,"evidence_quote":"Describes the compressed binary RDF format in the other major Java RDF library, the closest competitor in the benchmarks."}],"review_version":1}