{"id":"f51166a5-dbda-4e2a-ae5f-e6c52b068065","arxiv_id":"2501.07666","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"HiPO, a data format from Jefferson Lab, reports columnar read and histogram fill speeds that beat ROOT and Parquet in a synthetic 50M-row benchmark.","lead":"This paper describes HiPO, a data format developed at Jefferson Lab for nuclear physics experiment data, and reports benchmark tests comparing its columnar reading speed with ROOT and Parquet. The reported tests show HiPO reading four columns and filling histograms several times faster than ROOT and Parquet on one laptop.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark supports a narrow speed advantage, but the broad generalization in the Abstract and Section 6 rests on one synthetic, warm-cache, single-machine test with the HiPO columnar C++ code not yet public; a cold-cache, real-data comparison is needed.","rationale":"The reader's weakest-assumption is correct that the Section 5.2 synthetic benchmark on one M1 MacBook with a warm page cache is a narrow basis for the wide claim in the Abstract and Section 6. I agree that this is the main load-bearing weakness. I add two related technical details that make the concern sharper: the Parquet comparison is not apples-to-apples with the native HiPO C++ loop, and the HiPO columnar C++ code is not public even though C++ timings are reported. The paper is still a useful design document: the Java implementation is available, the format is in production for CLAS12, the benchmark numbers are internally consistent, and the paper itself acknowledges that more comprehensive tests are future work. Nothing in the text contradicts the reported results. Therefore the appropriate verdict remains CONDITIONAL: the format and design can be accepted, but the general performance superiority claim should be accepted only after an independent, reproducible benchmark on realistic data and hardware. My concern does not move the verdict because the reader already conditioned acceptance on exactly this kind of evidence.","tokens_in":8797,"tokens_out":6465,"duration_ms":71346,"concrete_test":"Re-run the Section 5.2 workload using a real CLAS12 (or other experiment) columnar analysis file with mixed column types and variable row counts, on a Linux HPC node with a parallel filesystem, dropping OS page caches before each run. Compare the released HiPO Java and, once available, HiPO C++ implementation against ROOT RNTuple 6.34 and a native C++ Parquet reader using the same simplified histogram-fill code (Listing 8). If HiPO's speedup over RNTuple drops below about 2x, or if Parquet becomes competitive, the abstract's general performance claim should be narrowed to the specific benchmark conditions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, stated in the Abstract and Section 6, is that HiPO 'surpasses established data formats in performance' and can be applied to other scientific fields. The evidence for this is Section 5.2: 50 million rows of uniformly random floats in 24 columns, LZ4 compression, read on a single M1 MacBook with a local 1 TB SSD, with ten consecutive reads and the last four averaged (so the file is in the OS page cache). This exercises only full sequential scans of four float columns with histogram fills; it does not test predicates, varying column selectivity, integer/string/variable-length data, write paths, or cluster/parallel filesystems. The Parquet comparison uses a DataFrame path (likely Python-level), not a native C++ loop, while the HiPO C++ numbers in Table 1 are reported even though Section 5.2 states the C++ source 'will be made public after thorough debugging.' These limitations do not make the reported numbers false, but they do make the broad 'surpasses established formats' and 'other scientific fields' claims unsupported beyond the specific benchmark. The load-bearing assumption is that this one synthetic, warm-cache, laptop benchmark represents real experimental analysis workloads; that assumption is not justified in the paper.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents HiPO, a data format developed at Jefferson Laboratory for the CLAS12 experiment, designed to cover the full data lifecycle from raw detector data to columnar analysis. It describes the file structure (header, records, footer), event and bank/tuple APIs, an event-tagging mechanism for selective access to records, and an experimental columnar storage mode. The principal quantitative claim is in Section 5.2: a benchmark on 50M rows of synthetic data reports that HiPO reads four columns and fills histograms roughly three times faster than ROOT RNTuple in a simplified test, and faster than TTree and Parquet in other comparisons. This leads to the Abstract's claim that HiPO \"surpasses established data formats in performance\" in columnar data analysis and can be applied to other scientific fields.","tokens_in":9020,"tokens_out":4987,"duration_ms":45989,"significance":"If the performance claims were rigorously established, the paper would be relevant to data-intensive nuclear and particle physics analyses, where ROOT and Parquet dominate. The HiPO format has the strength of having been in production for CLAS12 for several years, with bindings to C++, Java, Python, Julia, and FORTRAN, and the paper gives a clear description of the file layout and APIs. The benchmark, despite its limitations, is a concrete attempt at quantitative comparison and the authors have been transparent about some of the setup. However, the evidence for the headline claim is currently a single warm-cache, single-machine, synthetic-data benchmark with no error bars, no public C++ implementation, and no released dataset, which is insufficient for the broad \"surpasses established data formats\" and \"other scientific fields\" conclusions.","major_comments":[{"comment":"The benchmark is not reproducible: the C++ source for HiPO is not yet public (the paper states it \"will be made public after thorough debugging\"), the synthetic dataset is not released, and the tests were run on a single M1 MacBook with the file in the OS page cache (ten consecutive reads, last four averaged). No error bars or standard deviations are reported. This narrow setup cannot support the general claim in the Abstract and Section 6 that HiPO \"surpasses established data formats\" for columnar data analysis; the paper should either provide a full reproducibility package or restrict the claim to the specific tested configuration.","section":"§5.2, Table 1"},{"comment":"The comparison is inconsistent across formats: Parquet is tested only via a DataFrame path, while HiPO is also tested via a native C++ loop; the simplified histogram test (Listing 8) that yields the \"three times faster\" figure compares only HiPO with RNTuple, not with TTree or Parquet. As a result, the data do not establish the broad claim that HiPO outperforms all established formats in columnar analysis. A consistent set of comparisons (same API level, same operations, same file sizes) across all formats is needed, or the conclusions must be limited to the specific implementations compared.","section":"§5.2, Table 1, Listing 8"},{"comment":"The 3x speedup claim depends on the choice of ROOT 6.34 as the comparison baseline, but the paper acknowledges that ROOT 6.32 gave RNTuple times of 1.87 seconds versus 2.62 seconds in 6.34, and justifies 6.34 only by stating that \"version 6.34 incorporates format changes that will serve as the foundation for future developments.\" This is not a neutral basis for selecting the comparison version. The authors need to justify why ROOT 6.34 is the fair baseline (e.g., by describing the relevant format changes and their impact) or report results for both versions; otherwise the reported speedup may overstate the improvement.","section":"§5.2, ROOT version choice"},{"comment":"The benchmark exercises only full sequential scans of four float columns with histogram filling. It does not test predicate selections, varying column selectivity, integer/string/variable-length data, write performance, or parallel/cluster file systems. The paper itself states, in Section 5.2, that \"more comprehensive performance tests are beyond the scope of this article,\" which is an explicit acknowledgment of limitation. Given this, the Abstract's assertion that HiPO \"can be effectively applied to data analysis in other scientific fields\" is not supported by the evidence presented; the claims should be tempered or the tests expanded.","section":"§5.2 and Section 6"}],"minor_comments":[{"comment":"\"Parquete\" should be \"Parquet\" in the Abstract, Section 5.2 heading, and Table 1.","section":"Abstract and Table 1"},{"comment":"The comment \"Writing arrays into an Event\" appears above both Listing 5 and Listing 6, but the code in these listings demonstrates tagged file reading and writing, not writing arrays into an event.","section":"Listings 5 and 6"},{"comment":"The printf statement in Listing 4 uses the variable `row`, but `row` is not declared in the snippet; this makes the example incomplete and potentially confusing.","section":"Listing 4"},{"comment":"HiPO files are given the extension \".h5\" in Listing 1 and elsewhere, which may be confused with HDF5 files; a distinct extension would avoid ambiguity.","section":"Listing 1"},{"comment":"The row \"Tuple HiPO Julia DataFrame\" is described in the text as a \"preliminary benchmark\" and a \"port,\" but the table gives no such qualification; the table should note the status of the Julia implementation.","section":"Table 1"},{"comment":"The paper should include a data-availability statement for the synthetic dataset and the benchmark scripts, particularly since the C++ code is not yet public.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"This is a format-description paper whose main scientific contribution is the performance comparison. The benchmark methodology is the central weakness. The authors should be asked to provide a full reproducibility package (code, data, benchmark scripts) and to either broaden the evaluation or explicitly restrict the claims. The paper may also need a clearer statement about the relationship between the Java/C++ HiPO implementations and the published benchmark numbers. The topic is appropriate for physics.data-an, but the acceptance should hinge on the credibility of the benchmark."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this is a useful format description with a benchmark that is more preliminary than the abstract's claims suggest. If you want to know how HiPO works, the paper is good; if you want to trust the 'surpasses ROOT and Parquet' claim, wait for the public C++ code and broader tests.\n\nThe genuinely new content is the experimental columnar storage mode and the performance comparison against ROOT and Parquet. The format itself has been in production at CLAS12 for years, so the design sections are concrete and grounded. I credit the paper for describing the tagging mechanism, record layout, and bindings clearly. The author also does something rare: he reports the simplified histogram test that removes TH1D overhead, reports the ROOT version discrepancy, and explicitly states the C++ code is not yet public. Those are honest disclosures and they help the reader calibrate.\n\nThe soft spots are where you'd expect. The central performance claim rests on one synthetic dataset (50M rows of floats, 24 columns, four read), on a single M1 MacBook with a warm page cache, using the last four of ten reads. No error bars, no cold-cache test, no predicate filters, no writes, no clustered file system, no integer/string data. The Parquet comparison goes through a DataFrame path (likely Python-level overhead), which isn't apples-to-apples with the native C++ loops used for HiPO. The 'surpasses established formats' language in the Abstract and Section 6 is too strong for this evidence. What the data supports is: 'on this one workload, HiPO is faster than the compared versions of ROOT and Parquet.' That is still worth knowing.\n\nThe paper is not circular and doesn't invent anything. It's a sincere engineering report. The main load-bearing assumption is representativeness, and that assumption isn't justified yet.\n\nBottom line: useful for people working with CLAS12 data or thinking about columnar formats in NP/HEP. It deserves a serious referee, but the referee should insist on the public C++ code, a cold-cache run, real-data workloads, and an apples-to-apples Parquet comparison before the broad claims are published. I'd send it back for a major revision.","headline":"Useful format description, but the headline performance claim outruns a single warm-cache laptop benchmark.","tokens_in":9538,"tokens_out":2231,"would_cite":false,"duration_ms":22416,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that HiPO, a data format built for the CLAS12 experiment, reads and fills histograms from a subset of columns about three times faster than ROOT RNTuple and Parquet on the same benchmark.","keywords":["HiPO","columnar data storage","event tagging","data format benchmarking","nuclear physics analysis","ROOT RNTuple","Apache Parquet","histogram filling"],"falsifier":"Run the same four-column read-and-fill benchmark on a multi-user cluster filesystem using real reconstructed physics data with non-uniform column distributions and many small files, and check whether HiPO's recorded lead over ROOT RNTuple and Parquet persists; a finding that ROOT or Parquet matches or beats HiPO under those conditions would falsify the paper's general performance claim.","tokens_in":8553,"feed_emoji":"📊","tokens_out":4823,"duration_ms":44329,"temperature":0.7,"pith_summary":"This paper presents the HiPO data format, developed for the CLAS12 nuclear physics experiment, and argues that the same format can serve the full data lifecycle, from raw detector output to columnar physics analysis. The paper claims that HiPO's columnar mode, which stores each table column in separately tagged records, lets analysis programs read only the columns they need and fill histograms without deserializing the rest of the file. On a synthetic 24-column, 50-million-row file, the reported read-and-fill times are about 0.5 seconds for HiPO, about 1.6 seconds for ROOT RNTuple, and about 1.8 seconds for Parquet when only four columns are read. A sympathetic reader would take this as evidence that a format originally built for one experiment's event data can compete with and outperform general-purpose columnar formats in common analysis workloads.","feed_headline":"HiPO reads columns 3x faster than ROOT RNTuple","feed_subtitle":"Tagged per-column records let the format skip unneeded data in a four-column histogram test.","key_machinery":"The load-bearing mechanism is the record-tagging scheme. In columnar mode, each column is assigned a unique tag and serialized into its own record, or bucket; the file footer maps every record's position, size, and tag. When a user declares branches to read, the reader opens only records whose tags match, deserializes those columns, and synchronizes rows across columns. A second piece of machinery is a purpose-built histogramming routine used in the benchmark, which the paper says avoids overhead in ROOT's TH1D.Fill() and contributes to the measured gap.","core_discovery":"The central claim is that HiPO's columnar storage design is faster than established formats for selective column reads. The mechanism is record tagging: each column is written into its own record with a unique tag, and the file footer records each record's position and tag, so a reader can jump directly to requested columns. The paper reports that in a loop-based histogram fill over four of 24 columns, HiPO takes 0.523 seconds versus 1.589 seconds for ROOT RNTuple after both use the same simplified histogramming code, and that HiPO Java DataFrame bulk filling reaches 0.452 seconds, compared with 1.812 seconds for Parquet. The paper also states that interactive use in JShell does not degrade HiPO performance, while RNTuple slows by a factor of about two in interactive mode.","pith_inferences":["If the record-tagging scheme generalizes, it could be adopted by other event-based formats as a lightweight columnar-read layer without restructuring their event model.","The benchmark's synthetic uniform random columns are unusually compressible and carry no selection selectivity; real physics columns with sparse values and correlated branches may shift the relative performance, so the threefold claim should be tested on real experimental data.","The reported advantage is for reading from a local SSD on a single laptop; on shared cluster file systems with many concurrent readers, metadata lookups and random seeks could change the ranking.","A testable extension is whether the tagging scheme helps when reading only a small fraction of rows by tag (event selection) as well as a subset of columns, since the format already supports tag-filtered event reads."],"forward_implications":["A single HiPO file can carry raw, reconstructed, and analysis-ready columnar data, removing format-conversion steps across the data lifecycle.","Analysis programs that read a few columns from wide tables can expect substantially lower I/O and CPU time with HiPO than with ROOT TTree, RNTuple, or Parquet in similar single-machine benchmarks.","The DataFrame-style bulk histogram fill is reported to be about 50% faster than the loop-based HiPO read, suggesting a cheap speedup for histogram-heavy workflows.","Interactive exploratory environments such as JShell can keep HiPO's read speed, while ROOT RNTuple's interactive read is reported to be roughly twice as slow as its batch read.","Because the columnar implementation is still labeled experimental, the performance numbers are presented as preliminary rather than as a final product."],"supporting_citations":[{"why":"Describes the CLAS experiment whose data lifecycle motivated the format's requirements.","marker":"[1]"},{"why":"Describes CLAS12, the experiment for which HiPO was built and used for petabytes of stored data.","marker":"[2]"},{"why":"Defines ROOT, the framework whose TTree and RNTuple are the main baselines in the benchmark.","marker":"[3]"},{"why":"Defines Apache Parquet, the columnar format used as the second comparison baseline.","marker":"[6]"},{"why":"Points to the HiPO source repository where the Java implementation and examples are published.","marker":"[7]"}],"fun_headline_variants":["HiPO column tags cut read time to a third of ROOT RNTuple","Tagged records let HiPO skip columns, run 3x faster than ROOT","HiPO jumps to needed data, beats ROOT by 3x in histogram fill","Selective column reads: HiPO speeds past ROOT and Parquet in tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The performance claim rests on the assumption that a synthetic 50-million-row file of uniform random numbers read from a laptop SSD represents the I/O patterns and data characteristics of real experimental analysis on cluster file systems.","fun_headline_variants_meta":{"raw":{"variants":["HiPO column tags cut read time to a third of ROOT RNTuple","Tagged records let HiPO skip columns, run 3x faster than ROOT","HiPO jumps to needed data, beats ROOT by 3x in histogram fill","Selective column reads: HiPO speeds past ROOT and Parquet in tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001073,"raw_usage":{"total_tokens":4459,"prompt_tokens":880,"completion_tokens":3579,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":3490}},"tokens_in":496,"tokens_out":3579,"duration_ms":24597,"temperature":1.0,"reasoning_tokens":3490,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:36:28.728584+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same four-column read-and-fill benchmark on a multi-user cluster filesystem using real reconstructed physics data with non-uniform column distributions and many small files, and check whether HiPO's recorded lead over ROOT RNTuple and Parquet persists; a finding that ROOT or Parquet matches or beats HiPO under those conditions would falsify the paper's general performance claim.","supporting_citations":[{"cited_title":"and others, The CLAS12 Spectrometer at Jefferson Laboratory, Nucl","cited_arxiv_id":null,"evidence_quote":"Describes CLAS12, the experiment for which HiPO was built and used for petabytes of stored data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Apache Parquet, the columnar format used as the second comparison baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Points to the HiPO source repository where the Java implementation and examples are published."}],"review_version":1}