{"id":"df18f510-b81a-451b-9f6e-23a7b802e58b","arxiv_id":"2412.21162","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"In a real SDR-based 5G standalone testbed, Open5GS shows the lowest control-plane latency, OpenAirInterface the highest data rates, and Free5GC the lowest resource consumption.","lead":"This paper compares four open-source 5G core network platforms and benchmarks three of them on a real radio testbed built from commercial off-the-shelf hardware and software-defined radios. It finds Open5GS fastest at control-plane signalling, OpenAirInterface fastest for data throughput, and Free5GC most efficient with server resources.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Most load-bearing: the testbed never specifies whether optional UPF optimizations (DPDK/VPP) were enabled; since Free5GC lacks DPDK, asymmetric configuration could fully explain the throughput and CPU rankings, and the CP latency claim lacks reported confidence intervals.","rationale":"The paper has real strengths: a live SDR testbed with a physical UE is more convincing than the simulated RANs used in related work, and the qualitative comparison is useful. The reader's weakest_assumption (OAI RAN fairness) is a reasonable concern, but I see an even more actionable issue: the testbed description omits platform-level data-plane configuration, which is exactly where OAI and Open5GS advertise optional DPDK/VPP acceleration (Section III) that Free5GC does not offer. Since the central claim is a comparative ranking, hidden configuration asymmetry is a direct threat to validity. This is not an allegation of misconduct; it is an omitted-controls problem. The paper also promises 95% confidence intervals but does not report them for the control-plane figures, and the Table IV latency standard deviations are larger than the mean differences, so even the data-plane latency sub-claim is fragile. A request for configuration files and a configuration-equivalent rerun would settle the issue. Therefore the reader's CONDITIONAL verdict remains appropriate; no verdict change is needed.","tokens_in":7804,"tokens_out":8726,"duration_ms":87064,"concrete_test":"Request the exact Docker Compose files, image tags or commits, and any DPDK/VPP flags used for each platform, then rerun the UPF throughput, latency, and resource-consumption tests with DPDK/VPP explicitly disabled for all three platforms (or enabled wherever supported), while recomputing the 95% confidence intervals for Delta-Tr and Delta-Ts from the 30 samples. If OAI's data-rate lead and Free5GC's CPU lead disappear or reverse under configuration-equivalent settings, the central ranking is an artifact of setup rather than platform capability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the observed rankings reflect the 5GC platforms themselves rather than the experimental setup. This is least secure at the point where the paper chooses platform configurations. Section III states that OAI and Open5GS have optional DPDK/VPP configurations for the UPF, while Free5GC does not support DPDK (Table I). Section V's testbed description lists only hardware and RAN details; it never states whether these optional optimizations were enabled, nor which software versions or commits were used. If OAI's UPF ran with VPP/DPDK and Free5GC did not, the 'OAI highest data rates' and 'Free5GC lowest resource consumption' results are explained by an asymmetric configuration, not by the core's architecture. The same asymmetry also affects CPU consumption, since DPDK/VPP busy-polling inflates CPU usage. Additionally, the CP latency ranking rests on 30 samples per platform but no confidence intervals or standard deviations are shown for Delta-Tr and Delta-Ts (Section VI.A), so an 11-17% gap may be within noise; indeed, Table IV's latency means differ by less than their standard deviations. Without configuration transparency and statistical support, the ranking cannot be attributed to the platforms.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reviews four open-source 5G core platforms (Free5GC, OAI, Open5GS, SD-Core), compares their features qualitatively, and experimentally evaluates three of them (SD-Core excluded) in a 5G Standalone testbed built from an OAI gNodeB over an Ettus B210 SDR and a COTS server. The measured metrics are registration and PDU-session establishment delays, end-to-end throughput, latency, video-streaming performance, and CPU/RAM consumption. The central claim is that Open5GS has the best control-plane latencies, OAI achieves the highest data rates, and Free5GC has the lowest resource consumption, so no single core is best for all low-cost private deployments.","tokens_in":8002,"tokens_out":5177,"duration_ms":50895,"significance":"The paper addresses a useful and timely gap: most prior quantitative work uses simulated UEs or RAN, whereas here the comparison is carried out on a real SDR-based SA testbed with a commercial handset. The qualitative feature comparison in Table I is a convenient resource, and the experimental setup is plausible and clearly described at the hardware level. If the performance ranking were fully substantiated, the paper would be a valuable practical guide for selecting an open-source 5GC in low-cost deployments. However, the ranking is currently supported only by point estimates and by a testbed description that does not specify key software configuration choices, so the quantitative conclusions are preliminary rather than definitive.","major_comments":[{"comment":"The paper states in Section VI that 30 samples were collected to establish a 95% confidence interval, but no confidence intervals, standard errors, or significance tests are reported for any control-plane metric; Figures 5 and 6 show only point estimates. Moreover, Table IV's latency values are all within one standard deviation of each other (OAI 12.7±3.02 ms, Open5GS 13.6±3.32 ms, Free5GC 14.14±3.77 ms), so the claimed 10% latency advantage may be noise. The abstract's control-plane latency claim rests on means without dispersion information, and the reported differences are 11-17%, which is a load-bearing statistical gap. Report the confidence intervals or provide error bars/boxplots, and if the intervals overlap, soften the claims accordingly.","section":"Section VI.A, Figures 5 and 6, Table IV"},{"comment":"Section III states that OAI and Open5GS have optional DPDK/VPP data-plane configurations while Free5GC does not (Table I), but Section V's testbed description never says whether these optimizations were enabled for the UPFs, nor does it give the 5GC release or commit identifiers for the three platforms. If the OAI and Open5GS UPFs ran with DPDK/VPP and Free5GC did not, the observed throughput and CPU-consumption gaps (50% download throughput difference and 97% CPU difference) are explained by an asymmetric configuration rather than by the platforms' architectures. The paper must state the exact default or tuned configuration used for each core, including whether acceleration was enabled, and should add a sensitivity discussion.","section":"Sections III and V"},{"comment":"All three cores are tested behind the same OAI-based gNodeB, which is part of the same software project as OAI's core. This creates a potential fairness confound: OAI's RAN may be tuned or informally better integrated with OAI's 5GC, while Free5GC and Open5GS have to interoperate through the same RAN stack. The paper does not discuss this possibility or provide RAN-side validation (e.g., signal quality, retransmission counts, or scheduler behavior) across the three runs. At least a discussion of this limitation is needed to defend attributing 'OAI highest data rates' to the core platform rather than to the RAN-core pairing.","section":"Section V"},{"comment":"Resource consumption is a central part of the paper's claim, but Figure 9 reports only average CPU/RAM values; although Section VI says 50 samples were collected for resource consumption, no standard deviations, confidence intervals, or statistical tests appear. The 85%, 45%, and 97% CPU-reduction statements in Section VI.C are therefore not substantiated, particularly given the DPDK/VPP confound noted above. Add dispersion information and state whether the CP and UP measurements were taken under identical signaling and data-plane loads.","section":"Section VI.C"}],"minor_comments":[{"comment":"The text contains ungrammatical phrases such as 'the between NFs' and 'exchanged one the 5GC NFs'; these should read 'the communication between NFs' and 'exchanged among the 5GC NFs'.","section":"Section II-A"},{"comment":"The description of communication models C and D appears garbled relative to 3GPP TS 23.501; the sentence about the consumer communicating directly with the NRF or via the SCP conflates discovery with communication, so the model definitions should be clarified.","section":"Section II-A"},{"comment":"The related-work table marks 'This Paper' as covering SD-Core, but SD-Core is excluded from the quantitative evaluation; this scope should be stated explicitly in the table footnote or in Section V.","section":"Table II and Section V"},{"comment":"The captions and axes should indicate the sample size and the type of error bar or dispersion display used, especially once confidence intervals are added.","section":"Figures 5 and 6"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the journal's scope, and the real SDR-based testbed is a genuine contribution when compared with prior simulation-only studies. The main risk is that the central rankings are over-claimed relative to the evidence presented; I would ask the authors to report full statistics and exact platform configurations before the quantitative conclusions are accepted. The paper should not be rejected, but the abstract and conclusion claims need to be made conditional on these clarifications."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, straightforward benchmark paper. It gives the 5G community the first real-SDR, real-UE comparison of three open-source cores, and that alone earns it a read. The qualitative feature table is also handy.\n\nWhat's new: prior work used simulated RANs; this one uses a B210 SDR and a Motorola phone. They measure registration and session setup latency, throughput, video loading, and CPU/RAM. The setup is described well enough to reproduce: hardware, OS, Docker versions, band n78, antenna, SIM. That's more than most papers in this area.\n\nWhat's weak: first, they say they collected 30 samples to establish a 95% confidence interval, but no interval ever appears. We get means and standard deviations for the UP metrics, and the CP latency differences (11-40%) look small relative to the spread. Without CIs or a significance test, I can't tell if Open5GS's CP advantage is real. Second, they never state whether the optional DPDK/VPP fast path was disabled. Table I lists DPDK support for OAI and Open5GS but not Free5GC. If DPDK was on for the first two, the throughput and CPU rankings are explained by that, not by the cores. My guess is they used defaults, but 'my guess' isn't a methods section. Third, the abstract claims 'Open5GS provides the best latencies...' without mentioning SD-Core was never measured. The body is careful about this in the conclusion, so the abstract should match. Using OAI's gNodeB for all cores is a smaller concern; a different RAN might shift relative results, but it's a reasonable choice for a low-cost testbed.\n\nBottom line: the rankings are plausible and useful, but the missing statistical backing and configuration transparency keep it from being definitive. A competent referee could push for those fixes in a revision. This deserves peer review, not desk rejection, because the community needs real-hardware numbers and these are the first decent set.","headline":"Useful real-SDR benchmark of three open-source 5G cores, but missing confidence intervals and DPDK configuration transparency keep the rankings from being definitive.","tokens_in":8558,"tokens_out":2215,"would_cite":true,"duration_ms":24795,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that among open-source 5G cores, Open5GS gives the best control-plane latency, OpenAirInterface the highest data rates, and Free5GC the lowest resource consumption.","keywords":["5G core network","open-source 5G","Open5GS","Free5GC","OpenAirInterface","software-defined radio","performance evaluation","COTS hardware"],"falsifier":"Replace the OAI-based gNodeB with an independent or commercial gNodeB and repeat the registration, session-establishment, throughput, latency, and resource tests; if OAI no longer has the highest data rates or Open5GS no longer has the fastest control plane, the reported ranking is an artifact of the chosen RAN.","tokens_in":7592,"feed_emoji":"📶","tokens_out":7118,"duration_ms":66209,"temperature":0.7,"pith_summary":"This paper argues that among open-source 5G core platforms, no single implementation dominates a low-cost standalone deployment. Running three open-source 5G cores against the same software-defined-radio access network and a commodity server, it finds a split result: Open5GS has the fastest registration and session-establishment signalling, OpenAirInterface delivers the highest data rates and lowest user-plane latency, and Free5GC consumes the least CPU and RAM. The finding matters because private and research 5G networks on commercial off-the-shelf hardware are becoming feasible, and operators can choose the core software according to their workload rather than settling for one overall winner.","feed_headline":"Open5GS wins signaling, OAI wins data, Free5GC sips resources","feed_subtitle":"Low-cost 5G standalone testbed shows the choice of core depends on whether you need speed, throughput, or efficiency.","key_machinery":"The central instrument is the testbed itself. It uses a single OAI-based gNodeB implemented with an Ettus B210 SDR and a log-periodic antenna, connected over a Docker network to whichever core is under test, with the same commodity server hosting all cores. Keeping the RAN and host fixed makes the measured differences in $\\Delta T_r$, $\\Delta T_s$, throughput, latency, video loading time, CPU, and RAM attributable to the core platforms. The registration and PDU session establishment signalling flows define the control-plane metrics, while Iperf3 and a Flask-based video server define the data-plane metrics.","core_discovery":"The central claim is a comparative verdict reached through a real end-to-end 5G Standalone testbed built from a software-defined radio running an OAI-based gNodeB, a commercial smartphone, and each candidate core deployed as Docker containers on the same Intel Xeon server. For the control plane, Open5GS posted the lowest registration time ($\\Delta T_r$) and session establishment time ($\\Delta T_s$); Free5GC and OAI were roughly 13% and 40% slower on registration and 11% and 17% slower on session establishment. For the data plane, OAI reached about 100 Mbps download in a congested Iperf3 path, roughly 50% higher than Free5GC, with about 10% lower latency; Open5GS sat in between. For resources, Free5GC used about 97% less CPU than OAI in both planes, while Open5GS used substantially more RAM than the other two. The paper also qualitatively compares four platforms, adding SD-Core, but excludes SD-Core from performance tests because it requires Kubernetes instead of Docker.","pith_inferences":["The paper leaves implicit that the rankings may shift under multi-UE load, since all measurements use a single phone; a natural extension is scaling the UE count and watching whether Open5GS's control-plane lead survives.","Because the RAN is OAI's own gNodeB, the comparison may be slightly favorable to OAI; swapping in an independent RAN implementation would test whether OAI's data-plane lead is intrinsic or a product of tight integration.","The split results suggest a hybrid architecture—Open5GS control plane with an OAI or DPDK-accelerated user plane—could combine low signalling latency with high throughput, which no single platform in the study achieves.","Free5GC's dramatically lower CPU use may make it the default choice for resource-constrained edge nodes, but only at traffic levels where its lower throughput is acceptable."],"forward_implications":["A low-cost private 5G standalone network on commodity hardware is workable; the paper demonstrates it with SDRs and a single server.","Operators prioritizing fast attach and session setup should choose Open5GS, at least at the tested scale.","Operators prioritizing user throughput and low data latency should choose OAI, accepting higher CPU consumption.","Resource-constrained edge deployments could choose Free5GC, accepting lower data rates.","The choice of core software is workload-dependent, so performance comparisons should report control-plane, data-plane, and resource metrics together."],"supporting_citations":[{"why":"Earlier qualitative comparison of Magma, Free5GC, and Open5GS that this paper extends by adding feature-level detail and real measurements.","marker":"[8]"},{"why":"Prior qualitative and quantitative simulation analysis of Free5GC, OAI, and Open5GS; gives baseline rankings this paper challenges with a real testbed.","marker":"[9]"},{"why":"Compliance testing of the same open-source cores against 3GPP Release 16 procedures; establishes maturity differences the paper builds on.","marker":"[10]"},{"why":"Prior quantitative comparison of Free5GC and Open5GS in simulation; the paper's real-SDR results are positioned against this baseline.","marker":"[11]"}],"fun_headline_variants":["5G core shootout: Open5GS for latency, OAI for speed, Free5GC for frugality","Pick your 5G core: Open5GS fastest signaling, OAI fastest data, Free5GC lightest","Open5GS, OAI, Free5GC: Which 5G core wins? Depends on your priority","5G core face-off: Low-cost testbed reveals trade-offs in latency, throughput, footprint","Open-source 5G cores compared: Open5GS for control, OAI for data, Free5GC for resources"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The measured differences are treated as properties of the three core platforms, which assumes that using the same OAI-based gNodeB and server gives each core a fair and equivalent radio access network; if OAI's RAN is better tuned to its own core or the other cores are not optimally configured, the ranking could change.","fun_headline_variants_meta":{"raw":{"variants":["5G core shootout: Open5GS for latency, OAI for speed, Free5GC for frugality","Pick your 5G core: Open5GS fastest signaling, OAI fastest data, Free5GC lightest","Open5GS, OAI, Free5GC: Which 5G core wins? Depends on your priority","5G core face-off: Low-cost testbed reveals trade-offs in latency, throughput, footprint","Open-source 5G cores compared: Open5GS for control, OAI for data, Free5GC for resources"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000289,"raw_usage":{"total_tokens":1689,"prompt_tokens":938,"completion_tokens":751,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":613}},"tokens_in":554,"tokens_out":751,"duration_ms":5805,"temperature":1.0,"reasoning_tokens":613,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:59:50.093564+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the OAI-based gNodeB with an independent or commercial gNodeB and repeat the registration, session-establishment, throughput, latency, and resource tests; if OAI no longer has the highest data rates or Open5GS no longer has the fastest control plane, the reported ranking is an artifact of the chosen RAN.","supporting_citations":[{"cited_title":"Analysis for Comparison of Framework for 5G Core Implementation","cited_arxiv_id":null,"evidence_quote":"Earlier qualitative comparison of Magma, Free5GC, and Open5GS that this paper extends by adding feature-level detail and real measurements."},{"cited_title":"Open Source 5G Core Network Implementations: A Qualitative and Quantitative Analysis","cited_arxiv_id":null,"evidence_quote":"Prior qualitative and quantitative simulation analysis of Free5GC, OAI, and Open5GS; gives baseline rankings this paper challenges with a real testbed."},{"cited_title":"Tutorial on communication between access networks and the 5G core","cited_arxiv_id":null,"evidence_quote":"Compliance testing of the same open-source cores against 3GPP Release 16 procedures; establishes maturity differences the paper builds on."},{"cited_title":"Evaluating the performance of open source software implementations of the 5G network core","cited_arxiv_id":null,"evidence_quote":"Prior quantitative comparison of Free5GC and Open5GS in simulation; the paper's real-SDR results are positioned against this baseline."}],"review_version":1}