{"id":"788aea9a-f1a9-4f19-bb6b-4949b8ad50fd","arxiv_id":"1908.02118","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A real, anonymized one-week network trace from a campus HVAC management system is made public and characterized for use in flow-based intrusion detection research.","lead":"The authors collected and publicly released a one-week network packet trace from the HVAC management system of a university campus, the first such trace from a real building automation system. The paper characterizes the traffic to help researchers build and test intrusion detection systems for industrial control networks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Completeness of the two-vantage-point capture is the load-bearing assumption; it is directly checkable from TCP headers in the published pcaps.","rationale":"I read the paper as a dataset-contribution paper: its value is a public trace from a real BMS, and the correctness condition is that the trace faithfully represents the network it describes. The reader's weakest_assumption names faithfulness broadly; I focus on the least-secured component, the two-vantage capture. The clock-correction issue is acknowledged and quantified (8500 to 85 out-of-order packets per hour), so it is a known limitation rather than a hidden risk. The filtering step is described in enough detail to be judged. The 'first' claim is hedged with 'to our knowledge' and is less load-bearing than data integrity. The weakest point is Section III.A's assertion that mirroring two routers captures everything, with no topology argument and no loss statistics. Unlike privacy or novelty concerns, this can be falsified from the artifact itself by checking TCP sequence-number contiguity. If the check passes, the central claim stands; if it fails, the trace is incomplete and downstream flow-based evaluations are compromised. This supports the same conditional verdict the reader reached, so I recommend no change.","tokens_in":7370,"tokens_out":14910,"duration_ms":164366,"concrete_test":"Download all one-hour pcap files from the repository and merge them exactly as described in Section III. For every TCP connection with payload, reconstruct the byte stream from (IP total length - IP header length - TCP header length) and sequence numbers, in the timestamp-corrected packet order. Check that each connection's sequence space is contiguous, allowing only exact repeats caused by TCP retransmissions. An unexplained gap is direct evidence that a packet was missed by the two mirrored vantage points; if all long-lived connections across the seven days are gap-free, the completeness concern is settled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III.A aims to collect 'all network communication' by mirroring two routers, because paths are asymmetric. The paper gives no evidence that the union of the two mirrored ports covers every path between the control server, HMI stations, and gateways. If a third router, a direct gateway link, or a load-balanced route is not mirrored, whole TCP sessions or one direction of a conversation are missing. That would make the Section IV flow statistics (Table II, flow durations, per-gateway traffic) describe an incomplete artifact and would weaken the central promise of a realistic public trace. This is a data-integrity condition, not a modeling choice. It can be settled from the published files: each retained TCP packet contains the IP total length and TCP sequence number needed to reconstruct per-connection byte coverage.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a public dataset of packet headers collected from the HVAC management system of a university campus over seven days. Traffic was captured by port mirroring at two routers, merged and clock-corrected, filtered to HVAC-related hosts, anonymized with CryptoPan, and published as hourly pcap files. The paper reports basic statistics (47 IPs, ~23k flows, 11.2 GB) and characterizes flow sizes/durations, application protocols (DCE/RPC, NetBIOS, proprietary port 2499, S7), diurnal/weekend patterns, packet size distributions for two gateways, and flow stability. It argues the dataset is the first public trace from a real BMS and can support flow-based IDS evaluation and synthetic trace generation.","tokens_in":7475,"tokens_out":5453,"duration_ms":58431,"significance":"If the data are authentic, complete, and accessible, the paper fills a real gap: most public SCADA/BMS datasets come from testbeds or synthetic environments, while this is a real production system trace of 11.2 GB. The descriptive statistics are consistent with known automation-system behavior (long-lived TCP connections, small payloads, periodic polling), and the authors are transparent that the trace is benign-only and payload-free. These features make the dataset a plausible benchmark for flow-based IDS false-positive evaluation and traffic modeling. However, the significance is conditional on two things the manuscript does not yet fully establish: that the two mirror points capture all relevant traffic, and that the filtering/anonymization steps are precisely documented. Those conditions are testable from the published pcaps, so the paper can be strengthened without changing its scope.","major_comments":[{"comment":"The paper states that capturing at two routers is necessary because paths are asymmetric, but it does not provide evidence that the union of those two mirror points covers every path between the control server, HMI stations, and gateways. If a path traverses a third router or a direct switch link that is not mirrored, entire conversations or one direction of a conversation would be missing, which would change the flow durations and per-gateway statistics in Section IV and the validity of Table II. Please add a topology diagram showing the mirror points and their coverage, or report a per-connection integrity check (e.g., TCP sequence/acknowledgment continuity) that demonstrates no packets are missing from the published pcaps; otherwise, the claims should be restricted to 'traffic observed at the two mirror points.'","section":"III.A"},{"comment":"The filtering and anonymization step is not specified precisely enough to reproduce the dataset. The rule 'we only keep packets that either originate from or are destined to an host being part of the HVAC system' requires the exact list of IP addresses or subnets and how the boundary of the HVAC system was determined; the removal of ARP, STP, VRRP, LLMR, and SNMP should state the exact criteria (EtherType, protocol, ports) and what fraction of captured traffic was discarded. Because the dataset is the main contribution, vague filtering rules make it impossible for downstream users to know what portion of the original network activity the pcap files represent.","section":"III.B"},{"comment":"The paper does not provide file-level metadata for the published dataset, such as a file manifest, checksums, and a persistent identifier, nor does it state whether the pcap files contain raw mirror captures or the filtered/merged output. As the central claim is that the dataset is publicly available and reusable, a stable archival link and integrity documentation should be provided; otherwise the reproducibility of the paper's statistics and the long-term availability of the data cannot be verified.","section":"IV.A / Table II"}],"minor_comments":[{"comment":"The abbreviation 'LLMR' should be 'LLMNR' (Link-Local Multicast Name Resolution).","section":"III.B"},{"comment":"The text contains 'HV AC' instead of 'HVAC' in several places, apparently a LaTeX line-break artifact; this should be fixed in the final version.","section":"Abstract / II"},{"comment":"The phrase 'apart form' should be 'apart from'.","section":"IV.E"},{"comment":"Please state explicitly that the clock-offset formula assumes equal forward and reverse delays, and comment on the residual 85 out-of-order packets per hour; if the residual is negligible for flow-level statistics, say so in the text.","section":"III.A"},{"comment":"Axis labels should be self-contained, with units and time windows clearly specified, so that a reader does not have to infer them from the body text.","section":"Figures 2 and 5"},{"comment":"The related-work comparison would be strengthened by a small table listing existing public SCADA/BMS datasets and their provenance, which would also make the 'first public trace' claim easier to evaluate.","section":"II.B / IV.B"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The main risk is not the analysis but the artifact. If possible, have an editor or a reviewer download the GitHub repository and run a quick sanity check: extract TCP connections from the hourly pcap files and verify that for each connection the byte counts are contiguous, or at least that no retransmissions are missing in a way that suggests a dropped capture leg. This would directly address the major concern in Section III.A. Also, the novelty claim 'first public trace from a real BMS' is difficult to verify; the authors should be asked to provide a short audit of existing public datasets."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a dataset paper, and the dataset is the contribution. A real HVAC/BMS trace from a university campus, released as pcaps, with a careful description of collection, merging, and anonymization. That fills a real gap: most ICS/SCADA traces are synthetic or from testbeds, and the authors are right that no public real BMS trace is available, to my knowledge. The traffic characterization is basic—flow sizes, durations, packet distributions, gateway differences—but that's fine for a data paper. They also deserve credit for the ethical process: they got consent from the university CISO and Honeywell, and they anonymized IPs with CryptoPan and truncated payloads.\n\nThe soft spot is the completeness claim. The paper says they captured 'all network communication' by mirroring two routers, because paths are asymmetric, but they provide no evidence that the union of those two mirrored ports covers every path between the server, HMIs, and gateways. If a third router or a direct link exists, whole TCP sessions or one direction of a conversation are missing. That would skew the flow statistics in Section IV. This is not a modeling choice; it is a data-integrity issue, and it is directly checkable from the published pcap files: TCP sequence numbers and IP total lengths let you reconstruct per-connection byte coverage and see whether any connection has missing bytes. The authors should either provide that validation or temper the 'all communication' language. The clock-sync method assumes symmetric RTT, which is a reasonable approximation given the 8500-to-85 out-of-order improvement, so I'd call that minor. The 'first public BMS trace' claim is plausible but not backed by a systematic survey; it would be safer to say 'to our knowledge' and invite counterexamples.\n\nOverall, the resource is valuable for ICS/SCADA researchers, especially people building flow-based IDS. The paper deserves a serious referee, but the reviewers should require the completeness check or a qualification. I'd accept it conditionally. Would I cite it? Yes, if I worked on ICS traffic. Worth a reading group slot to see if the dataset is actually downloadable and intact.","headline":"A genuinely useful real-world BMS trace with a completeness claim that needs to be validated before the dataset is used as a ground truth for flow-based IDS research.","tokens_in":7943,"tokens_out":2733,"would_cite":true,"duration_ms":28612,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper releases the first publicly available network trace from a real building management system: a one-week, benign, header-only capture from the HVAC network of a university campus, aimed at flow-based intrusion detection research.","keywords":["building management system","HVAC","SCADA","network trace","packet capture","flow-based intrusion detection","anonymization","traffic characterization"],"falsifier":"Check the published trace for internal timing consistency: if TCP replies appear timestamped earlier than their requests by more than the clock error, or if the per-gateway packet counts diverge from the BMS's own recorded activity for the same week, then the capture or the correction procedure is flawed and the claim of a faithful public trace fails.","tokens_in":7193,"feed_emoji":"🏢","tokens_out":10026,"duration_ms":94022,"temperature":0.7,"pith_summary":"This paper presents what it claims is the first public network trace collected from a real building management system (BMS), specifically the heating, ventilation, and air-conditioning (HVAC) network of a university campus. It argues that researchers studying network-based intrusion detection for automation systems have so far been limited to synthetic traces or small testbeds, because real installations are not normally opened for experimentation. To close that gap, the authors captured seven days of packet headers from the production system, processed them with clock-correction and anonymisation, and published them. If the dataset is genuine and representative, it gives flow-based intrusion detection researchers a reproducible way to test their solutions against realistic, benign automation traffic. The paper characterises the trace, showing long-lived connections, small packet sizes, diurnal patterns, and some deliberate irregularities such as software-update traffic.","feed_headline":"First public trace from a real building automation system","feed_subtitle":"A one-week benign packet-header trace from a campus HVAC network, released for flow-based intrusion detection research.","key_machinery":"The object that carries the contribution is the published trace itself, together with the capture and post-processing procedure that makes it credible as a record of real BMS traffic. The capture uses two routers with port mirroring because the paths between the control server and gateways are asymmetric; the two devices record different views of the same packets. To merge the views into a correctly ordered stream, the clock offset between the two capture devices is estimated every second with ICMP echo exchanges, using the formula $s = t_B - t_A - RTT/2$ and assuming the network delay is symmetric in both directions. This correction reduces out-of-order packets from about 8500 to 85 per hour. The merged trace is then filtered to hosts belonging to the HVAC system, certain management protocols are removed, and the remaining packets are anonymised with a prefix-preserving cryptographic anonymisation scheme, with payloads truncated to headers and MAC OUIs zeroed before publication.","core_discovery":"The central claim is that a trustworthy public trace of a real automation system now exists and is described in sufficient detail to be used by other researchers. The dataset consists of one week of header-only traffic from an HVAC management system on a university campus, spanning seven HMI workstations, one control server, and eight gateways that connect 58 non-IP field devices. The authors argue that the traffic exhibits the expected properties of automation networks—permanent TCP connections that exchange small control packets, gateway-specific behaviour, and diurnal and weekly activity patterns—while also containing realistic irregularities. Because the trace is intentionally benign, it is not an attack-labelled dataset, but it is offered as a foundation for flow-level characterisation, for parametrising synthetic traffic generators, and for replay experiments in which attack traffic is mixed into the background.","pith_inferences":["The ICMP-based clock-correction method, which assumes symmetric delay, is reusable in any multi-vantage-point capture; the residual 85 out-of-order packets per hour sets a floor on timing accuracy that may matter for microsecond-level analyses.","The observation that DCE/RPC traffic uses changing ports suggests a concrete robustness test for anomaly detectors: a protocol shifting across ports could easily be misclassified as an attack, and the trace provides a natural case study.","Because the trace contains headers only, researchers who need payload-level protocol details (for example, function codes in industrial protocols) cannot use it directly; a payload-inclusive companion capture would be a natural extension if privacy constraints can be resolved.","Comparing this trace against previously reported SCADA traffic characteristics would quantify how much BMS behaviour differs across automation sectors, a comparison the paper does not fully develop."],"forward_implications":["Flow-based intrusion detection researchers can evaluate false-positive rates against benign traffic from a production automation system rather than synthetic or testbed traffic.","The trace can be replayed and combined with synthetic attack traffic, providing reproducible attack-injection experiments without placing any real installation at risk.","The reported flow statistics (long-lived connections, small packet sizes, gateway-specific volumes) can be used to parametrise traffic models for synthetic trace generation.","The declared 'anomalies', such as software updates and changing DCE/RPC ports, give anomaly-based detectors a realistic mix of regular and irregular activity to process.","Other researchers can use the dataset as a reference point for comparing future BMS or SCADA traces."],"supporting_citations":[{"why":"Prior characterisation of SCADA network traffic that the paper uses as a comparison point for its BMS observations, particularly the point that not all automation systems show diurnal patterns.","marker":"[2]"},{"why":"Earlier measurement study of a BACnet-based building automation network whose dataset is not public; establishes the gap the paper fills.","marker":"[8]"},{"why":"Another analysis of building automation network traffic used to show that existing real-system studies did not publish their traces.","marker":"[22]"},{"why":"Public capture files from a testbed environment; represents the kind of synthetic or testbed data that the paper argues is insufficient.","marker":"[1]"},{"why":"Supplies the prefix-preserving IP anonymisation algorithm used to make the trace safe to publish.","marker":"[13]"},{"why":"Defines the IPFIX flow-record semantics that the paper adopts for its flow-based characterisation.","marker":"[7]"},{"why":"Ethical guidance that shapes the consultation with the campus information security officer and the system vendor before publication.","marker":"[5]"}],"fun_headline_variants":["Public HVAC automation trace for realistic IDS evaluation","Real control system network traffic now available to researchers","Campus HVAC network header trace made public for testing","One-week benign trace of a live automation system released"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The dataset faithfully represents the traffic of the real HVAC system: the two capture points recorded every relevant packet, the clock-offset correction based on symmetric network delay produces correct ordering, and the filtering removed only non-HVAC traffic.","fun_headline_variants_meta":{"raw":{"variants":["Public HVAC automation trace for realistic IDS evaluation","Real control system network traffic now available to researchers","Campus HVAC network header trace made public for testing","One-week benign trace of a live automation system released"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000476,"raw_usage":{"total_tokens":2337,"prompt_tokens":901,"completion_tokens":1436,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":1375}},"tokens_in":517,"tokens_out":1436,"duration_ms":16136,"temperature":1.0,"reasoning_tokens":1375,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:52:07.873808+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check the published trace for internal timing consistency: if TCP replies appear timestamped earlier than their requests by more than the clock error, or if the per-gateway packet counts diverge from the BMS's own recorded activity for the same week, then the capture or the correction procedure is flawed and the claim of a faithful public trace fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior characterisation of SCADA network traffic that the paper uses as a comparison point for its BMS observations, particularly the point that not all automation systems show diurnal patterns."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Earlier measurement study of a BACnet-based building automation network whose dataset is not public; establishes the gap the paper fills."},{"cited_title":"Zheng and A","cited_arxiv_id":null,"evidence_quote":"Another analysis of building automation network traffic used to show that existing real-system studies did not publish their traces."},{"cited_title":"https://www.netresec.com/ index.ashx?page=PCAP4SICS","cited_arxiv_id":null,"evidence_quote":"Public capture files from a testbed environment; represents the kind of synthetic or testbed data that the paper argues is insufficient."},{"cited_title":"Ammar, and Sue B","cited_arxiv_id":null,"evidence_quote":"Supplies the prefix-preserving IP anonymisation algorithm used to make the trace safe to publish."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the IPFIX flow-record semantics that the paper adopts for its flow-based characterisation."},{"cited_title":"Dittrich, M","cited_arxiv_id":null,"evidence_quote":"Ethical guidance that shapes the consultation with the campus information security officer and the system vendor before publication."}],"review_version":1}