{"id":"dc009c12-6768-431b-b038-65bb99ed57e8","arxiv_id":"2505.14501","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"open5Gcube is a publicly released framework that combines open-source 2G/4G/5G network components with COTS smartphones and modems, with documented compatibility results.","lead":"This paper presents open5Gcube, a container-based framework that connects open-source mobile network software to real phones and modems in a portable lab rack. It reports which of seven commercial devices could attach to 2G, 4G, and 5G test networks built from tools like srsRAN, Open5GS, and OpenAirInterface.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that open5Gcube supports different project versions is not exercised by the evaluation; without a version-switch test, this central flexibility feature is unverified.","rationale":"The reader's conditional verdict is appropriate: the paper describes a useful, released framework and presents a real UE compatibility evaluation, but the central claims about usability, reproducibility, and version flexibility are not fully supported by the evidence. My stress-test focuses on one specific aspect that is even more concrete than the reader's weakest assumption: the paper explicitly advertises support for 'different versions' of open-source projects, yet Section IV only tests a single fixed combination of versions per stack. No experiment demonstrates that a version can be swapped without manual changes, so the central flexibility claim is unverified. I also agree with the reader that reproducibility is asserted rather than demonstrated; the repository link is a positive step, but the paper lacks commit-hash-level pinning or an independent deployment. These concerns do not overturn the paper's core technical contribution, but they do justify keeping the verdict conditional on additional evidence, especially a version-switch demonstration and a reproducibility check.","tokens_in":9004,"tokens_out":4119,"duration_ms":56253,"concrete_test":"Check out the public repository, then run the srsRAN+Open5GS 5G SA stack exactly as documented, first at the released commit corresponding to Table IV and then at a second, different released version of srsRAN (or Open5GS) by changing only the version reference in the compose file. If the second deployment does not come up with the same UE (e.g., Samsung Galaxy S25) attaching and passing the two-minute flood-ping, the 'different versions' claim is not supported. This directly tests the framework's advertised version flexibility under controlled conditions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The framework's headline claim is that it allows combining open-source implementations, including different versions, and simplifies reproducible setups. What is actually demonstrated in Section IV is a fixed set of version combinations (Table IV) tested for UE compatibility (Table V). No experiment switches the version of any RAN or CN project, and no baseline compares setup effort with and without the framework. The paper's conclusion that the results 'underline versatility and reproducibility' therefore rests on an inference, not on data: versatility is shown only for fixed versions, and reproducibility is asserted from the existence of a repository without commit-hash pinning, a clean-build check, or an independent deployment. The seven tested UEs are also non-random and mostly high-end devices, so the 'compatibility is adequate' conclusion is a judgment for the specific sample. None of this invalidates the framework's existence or the reported UE connections, but the central usability, reproducibility, and version-flexibility claims are load-bearing and currently unsupported. A single failed version switch would undercut a core advertised feature.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents open5Gcube, a container-based framework for integrating open-source mobile network components (RAN, CN, and UE-side tools) into a unified laboratory environment. The framework uses Docker-Compose stacks, a script-based orchestration layer, centralized configuration templates, and a Web-UI to manage stacks across multiple hosts. The authors evaluate the framework by building 2G (Osmocom), 4G VoLTE (srsRAN/Open5GS/Kamailio), and 5G SA (srsRAN/OAI-RAN with Open5GS/OAI-Core/Free5GC) stacks and testing them with seven COTS UEs (five smartphones, one modem, and one software-based UE not actually used in the tests). The paper claims that the framework allows flexible combination of different open-source implementations, including different versions, and that the compatibility results demonstrate its versatility and reproducibility.","tokens_in":9093,"tokens_out":4476,"duration_ms":44368,"significance":"If the claimed version-flexibility and reproducibility properties hold, open5Gcube would address a real gap in the literature: most prior testbed papers provide neither unified management nor publicly available configurations. The paper provides a useful architecture description, and the UE compatibility observations are direct empirical data that other groups could reproduce from the released repository. The framework also deliberately avoids heavyweight orchestration tools like Kubernetes, which is a reasonable design choice for small-scale labs. However, the significance is currently limited by the absence of evidence for the two central differentiators claimed in the abstract: version switching and simplified reproducible setup. The evaluation demonstrates only fixed version combinations and a small, non-representative UE sample, so the broader claims about versatility and reproducibility are not yet supported.","major_comments":[{"comment":"The central claim that open5Gcube 'allows researchers to flexibly combine different open-source implementations, including different versions' is not exercised anywhere in the evaluation. Section IV uses exactly one version per project per stack (Table IV), and no experiment changes the version of a RAN or CN component. The configuration-template mechanism described in Section III-B is precisely where version flexibility would be demonstrated, so the absence of a version-switch test leaves the primary differentiator unvalidated. A single failed version change would undercut this advertised feature; this should either be demonstrated or removed from the abstract.","section":"Abstract; Section IV, Table IV"},{"comment":"The conclusion that the results 'underline the versatility and reproducibility' is an overstatement. Versatility is shown only for the fixed combinations in Table IV, not for version flexibility. Reproducibility is asserted from the existence of the repository in [34], but the paper provides no clean-build check, no pinned commit hashes or release tags for the framework or the integrated projects, no explicit base-image versions, and no independent replication. Given that Section III-A names a specific Ubuntu version and kernel (5.15.0-133), a reproducible setup requires more than a GitHub link. The reproducibility claim should be either supported with precise version pinning and a build verification, or explicitly scoped to 'the provided configuration templates and documented hardware'.","section":"Section IV, final paragraph; Section V"},{"comment":"The statement that 'the compatibility of the tested UEs is adequate for mobile network laboratories' is a sample-specific judgment, not a general result. The sample consists of seven devices, five of them 2024 flagship smartphones with Qualcomm Snapdragon modems (Table II), and the authors themselves note known problem cases: the Motorola Edge 50 Pro disconnects due to missing IMS, the Google Pixel 9 fails to detect srsRAN, and the iPhone requires SUCI concealment settings. No older, low-end, or non-flagship devices are included, and no justification is given that this sample represents the space of COTS UEs. This conclusion should be restricted to the tested devices or supported by a broader, more diverse sample.","section":"Section IV, Table V and conclusion"},{"comment":"The framework's usability claim — that it 'simplifies experimental setups through containerization and lightweight orchestration' — is not quantitatively evaluated. There is no measurement of setup time, number of commands, or comparison with a manual deployment of the same stacks without open5Gcube. The orchestration design in Section III-B is plausible, but the paper provides no empirical evidence that the framework reduces effort. Since 'usable' appears in the title, a baseline comparison or at least a qualitative expert assessment of effort reduction would be needed to substantiate this core claim.","section":"Section IV (overall); Section I"}],"minor_comments":[{"comment":"The symbols used in Table I (#, G, blank) are not explained. Please add a legend describing what each symbol and each column means, and complete the final row labeled 'This 2025', which currently contains only the label and empty entries.","section":"Table I"},{"comment":"Tables III and V use checkmarks, circles, dashes, 'G', and blank cells without a legend. The reader cannot unambiguously determine whether 'G' means 'works with restrictions' or something else; please add explicit legends.","section":"Tables III and V"},{"comment":"The text says 'We were able to connect all smartphones shown in Table II', but Table II also includes a Quectel modem (QRM500), which is not a smartphone. Please clarify that QRM500 is a modem and does not affect the smartphone count.","section":"Section IV-A"},{"comment":"The hardware description gives qualitative statements about mobility and portability ('fitting into a standard car', 'easily transported') without any quantitative details such as total weight or dimensions. Adding these would strengthen the reproducibility of the hardware setup.","section":"Section III-A"},{"comment":"The stress test is described as 'a flood-ping was performed for two minutes without any connection loss', but no parameters (packet size, interval, number of parallel pings, or definition of connection loss) are given. Please specify the exact test procedure.","section":"Section IV-C"},{"comment":"Reference [34] points to a GitHub repository without a commit hash, branch, or release tag. If the paper appears in a camera-ready venue, please cite a specific release or commit to make the artifact auditable.","section":"References"},{"comment":"There are minor typographical issues: 'V oLTE' appears with a spurious space in several places, 'User Equipments' should be lowercase, and the index terms are overly generic ('5G, Testbed, Open-Source'). These should be cleaned up.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is best seen as a systems/architecture contribution with an empirical compatibility survey. I see no integrity issues: the self-citation [1] is peripheral, there are no fitted parameters or circular derivations, and the reported UE connection results are direct empirical observations. The main risk is that the abstract and conclusion claim version flexibility and reproducibility that are not demonstrated. A version-switch experiment would be the most convincing addition; at minimum, the claims must be scoped to fixed versions and to the tested devices. The lack of a quantitative usability comparison is hard to add retroactively, but the paper could still stand if the claims are calibrated accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth knowing: this is a genuine lab-infrastructure contribution, not a research result. The authors built a public, containerized framework (open5Gcube) that orchestrates srsRAN, OAI, Open5GS, Free5GC, and Osmocom stacks across a controller host and RAN hosts, with a web UI and centralized config templates. They make it available [34], which is more than most testbed papers do, and their Table I correctly shows that prior work mostly lacks unified management and public configs.\n\nWhat is solid: the architecture is described in enough detail to reproduce (hardware, networking, orchestrator design, configuration substitution). The compatibility tests are real hands-on observations across 2G, 4G, and 5G, with a per-UE breakdown in Tables III and V. The pattern they find—broad data connectivity, selective VoLTE and SUCI support—matches what srsRAN and OAI users report. No fitting, no invented entities, no circularity. The single self-citation is peripheral.\n\nSoft spots, in proportion. The central advertised feature—switching between project versions—is never exercised. Table IV fixes one version per project; no experiment changes a version and shows the framework handles it. The conclusion's claim that it was \"particularly helpful to ... simply change the versions\" is anecdotal. So the stress-test note is right on that point, though I would call it a moderate gap rather than a load-bearing flaw for the paper's core value, which is the working stacks plus the compatibility matrix. Reproducibility is also asserted from the repo's existence; the paper gives no commit hash and no independent deployment test. That is worth a revision, not a rejection. The UE sample is seven devices, mostly high-end; the \"compatibility is adequate\" conclusion should be scoped to the tested devices. The authors mostly stay near that scope, though the abstract's \"versatility and reproducibility\" is broad.\n\nWho this is for: anyone building an open-source mobile network lab. It will save setup time and gives a concrete starting point. As a paper, it is a systems/engineering contribution rather than a scientific finding, and it should be reviewed as such.\n\nRecommendation: send it out. It deserves referee time. Ask for a version-switch test and a scoped conclusion; those are cheap fixes. I would not desk-reject.","headline":"open5Gcube is a useful, honest systems paper: the repository is a real contribution, but the version-flexibility and reproducibility claims are asserted rather than demonstrated, and the UE sample is small.","tokens_in":9660,"tokens_out":1822,"would_cite":true,"duration_ms":17668,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"open5Gcube packages open-source 2G, 4G, and 5G network components into container stacks that a single controller host can build, tear down, and rebuild, and reports which commercial phones and modems actually connect.","keywords":["open5Gcube","mobile network laboratory","open-source network integration","COTS UE compatibility","containerized network stacks","experimental reproducibility","2G/4G/5G testbed","unified orchestration"],"falsifier":"Run the published stacks against a deliberately diverse device set—older phones, mid-range Android devices with MediaTek or older Exynos basebands, and iPhones with SUPI concealment disabled—and record basic data-connectivity success for 2G, 4G, and each 5G stack; a substantial failure rate where the seven tested devices succeeded would refute the 'adequate compatibility' conclusion.","tokens_in":8773,"feed_emoji":"📡","tokens_out":9554,"duration_ms":87750,"temperature":0.7,"pith_summary":"The paper introduces open5Gcube, a framework for building small mobile-network laboratories from open-source components. The paper claims that containerization plus a lightweight orchestration layer makes it practical for a research group to combine different radio, core, and subscriber-management implementations, including different software versions, and to switch between complete network scenarios quickly and reproducibly. The authors demonstrate the framework by testing seven commercial off-the-shelf devices (six smartphones and one modem) against a 2G stack, a 4G stack with VoLTE, and several 5G standalone stacks built from different core and radio projects. Basic data connectivity worked broadly, while VoLTE registration and SUCI-based authentication worked only for selected devices. If the claim is right, other laboratories can reproduce these setups from the public repository rather than re-integrating the software by hand.","feed_headline":"One container framework runs 2G, 4G, and 5G labs","feed_subtitle":"Researchers can mix and match open-source stacks and see which real phones connect to each.","key_machinery":"The stack abstraction is the object that carries the argument: a complete network scenario condensed into declarative container-composition files, centrally managed environment variables, and configuration templates. Each stack pins fixed IP addresses on dedicated virtual networks so configurable services can rely on static addresses, and a lightweight script-based orchestrator on the controller host starts remote radio containers by having an already-launched container detect the wrong host, disconnect from the stack network, and re-launch the service on the correct host via SSH. Subscriber databases are repopulated from scratch at every start, which makes experiments deterministic. Together these pieces make the framework's central promise—quick switching between network implementations and versions with reproducible state—concrete.","core_discovery":"The paper's central claim is that the main obstacle to using open-source mobile networks in research is orchestration, not the network software itself, and that a stack-based container abstraction removes that obstacle. In open5Gcube, every complete scenario—core network, radio access, subscriber database, and fixed IP assignments—is packaged as a stack that can be started and stopped from one controller host; radio containers that start on the wrong host detect this, release their network assignments, and relaunch themselves on the designated radio host over SSH. The authors report that all seven devices achieved basic 2G and 4G data connectivity, three registered to the VoLTE/IMS network, and 5G standalone success varied by combination: most devices worked with all core networks, one phone failed to detect the cell broadcast by one of the two radio projects, one dropped its connection without IMS, and one required SUPI concealment that one core does not support. This device-dependent behavior is presented as the kind of experimentally relevant result the framework is built to make easy to obtain.","pith_inferences":["A natural next step would be to turn the seven-device compatibility table into a community-maintained, machine-readable matrix, since phone models and Android versions change quickly and the current snapshot will age.","The framework's ability to hold the radio and device fixed while swapping core networks also enables controlled A/B tests of software-version changes, a use the paper mentions but does not exercise in its evaluation.","The pattern of failures—one phone not seeing a radio's cell, one disconnecting without IMS, one blocked by an unsupported authentication mode—suggests device-side behavior, not the open-source core, is often the binding constraint, which could be tested by varying carrier settings and SIM configuration while keeping the stack fixed.","If other groups adopt the stacks, the same repository could grow into a standardized benchmark for COTS device compatibility across open-source mobile networks, giving the field a shared reference point instead of private testbeds."],"forward_implications":["A laboratory with the documented hardware should be able to deploy the published 2G, 4G, and 5G stacks from the public repository and observe similar basic-data connectivity across commercial devices.","Switching between core-network implementations or radio versions becomes a stack-selection operation instead of a re-integration project, so comparative experiments need much less setup time.","Because subscriber databases are rebuilt from scratch on startup and IP assignments are fixed, repeated experiments start from deterministic state rather than from whatever the previous run left behind.","Advanced-feature experiments now come with a documented device guide: VoLTE registration worked on only three of seven devices, and one core network lacks SUCI-based authentication, so security and IMS studies can pick equipment accordingly."],"supporting_citations":[{"why":"Documents that smartphone compatibility with open-source radio networks is not universal and depends on OS, baseband, and configuration; motivates the device tests.","marker":"[15]"},{"why":"Supplies the radio access network software used in the 4G and 5G stacks.","marker":"[17]"},{"why":"Supplies the alternative radio and core implementations used in the 5G stacks.","marker":"[18]"},{"why":"Provides the emulated user equipment used as a reference alongside the physical devices.","marker":"[19]"},{"why":"Supplies the 2G base station software on which the 2G stack is built.","marker":"[20]"},{"why":"Provides one of the 5G core networks against which every device is tested.","marker":"[22]"},{"why":"Provides the core network used in the 4G stack and one 5G stack.","marker":"[23]"},{"why":"Supplies the VoLTE and IMS configuration the 4G stack builds on.","marker":"[29]"},{"why":"The public repository that carries the framework's reproducibility claim.","marker":"[34]"}],"fun_headline_variants":["One controller runs 2G, 4G, 5G lab stacks","open5Gcube orchestrates open-source mobile networks for real phones","Stack-based container framework simplifies mobile network labs","From 2G to 5G: a modular framework for phone testing","Framework lets researchers mix cores and see device quirks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion that UE compatibility is adequate for mobile network laboratories rests on seven specific commercial devices, most of them recent high-end phones; if other common devices with different chipsets or Android versions connect far less reliably, the general claim weakens.","fun_headline_variants_meta":{"raw":{"variants":["One controller runs 2G, 4G, 5G lab stacks","open5Gcube orchestrates open-source mobile networks for real phones","Stack-based container framework simplifies mobile network labs","From 2G to 5G: a modular framework for phone testing","Framework lets researchers mix cores and see device quirks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000207,"raw_usage":{"total_tokens":1372,"prompt_tokens":887,"completion_tokens":485,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":397}},"tokens_in":503,"tokens_out":485,"duration_ms":4690,"temperature":1.0,"reasoning_tokens":397,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:32:10.768563+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the published stacks against a deliberately diverse device set—older phones, mid-range Android devices with MediaTek or older Exynos basebands, and iPhones with SUPI concealment disabled—and record basic data-connectivity success for 2G, 4G, and each 5G stack; a substantial failure rate where the seven tested devices succeeded would refute the 'adequate compatibility' conclusion.","supporting_citations":[{"cited_title":"COTS UEs - srsRAN Project documentation,","cited_arxiv_id":null,"evidence_quote":"Documents that smartphone compatibility with open-source radio networks is not universal and depends on OS, baseband, and configuration; motivates the device tests."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the radio access network software used in the 4G and 5G stacks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the alternative radio and core implementations used in the 5G stacks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the emulated user equipment used as a reference alongside the physical devices."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 2G base station software on which the 2G stack is built."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides one of the 5G core networks against which every device is tested."},{"cited_title":"Lee, https://open5gs.org/","cited_arxiv_id":null,"evidence_quote":"Provides the core network used in the 4G stack and one 5G stack."},{"cited_title":"Herle, https://github.com/herlesupreeth/docker open5gs","cited_arxiv_id":null,"evidence_quote":"Supplies the VoLTE and IMS configuration the 4G stack builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The public repository that carries the framework's reproducibility claim."}],"review_version":1}