A Systematic Literature Review on Continuous Integration and Deployment (CI/CD) for Secure Cloud Computing

John Steinbacher; Nazim Madhavji; Sabbir M. Saleh

REVIEW 5 major objections 6 minor 6 references

A Systematic Literature Review on Continuous Integration and Deployment (CI/CD) for Secure Cloud Computing

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This systematic literature review claims that 66 selected papers provide a working map of tools, proposed solutions, and challenges for securing CI/CD pipelines in cloud environments.

desk verdict A useful first-stop map of CI/CD security tools and challenges, but the 'systematic' claim is not yet supported because the corpus is unauditable. read the letter →

arxiv 2506.08055 v1 pith:RDGFIYKG submitted 2025-06-09 cs.SE cs.CR

Sabbir M. Saleh , Nazim Madhavji , John Steinbacher This is my paper

classification cs.SEcs.CR

keywords ContinuousIntegrationDeploymentCI/CDCloudSecuritySystematicLiteratureReviewDevSecOpsSoftwareSupplyChainContainer

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a systematic review of 66 papers can map the current state of secure continuous integration and continuous delivery/deployment (CI/CD) in the cloud. Starting from 4,889 search results and filtering to 573 then 66 studies, it catalogues 62 tools and eight approaches used in practice, plus five proposed tools and twelve proposed frameworks. The review identifies recurring security problems, including image manipulation, unauthorised access, weak authentication, third-party and open-source dependency risks, and supply-chain attacks such as Log4j and SolarWinds, and argues these reveal research gaps. A reader should care because the map gives practitioners a concrete starting point for tooling choices and gives researchers a gap list to target.

What carries the argument

The carrying mechanism is the systematic literature review protocol: research questions framed with PICOC (Population, Intervention, Comparison, Outcomes, Context), a search string run across six digital libraries, snowballing that added 91 candidate papers, and a three-step selection funnel from 4,889 initial articles to 573 screened studies to 66 included papers. This funnel defines the evidence base, and every catalog count, challenge category, and gap statement in the paper is an aggregation of what those 66 papers report.

What would settle it

A replication of the reported selection funnel, from 4,889 initial articles to 573 screened studies to 66 included papers, should reproduce the same 66 papers; if the included-paper list, which the paper does not publish, differs materially, or if the counts of 62 tools and eight approaches cannot be traced to the included papers, the map and gap claims are not stable.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central finding is that the literature on CI/CD security in the cloud is rich in tools but thin on verified integration: the review compiled 62 tools and eight approaches or frameworks for existing practice, plus five proposed tools and twelve proposed approaches or frameworks. Challenges such as image manipulation, unauthorised access, weak authentication, disconnection between security tools and IDEs, third-party and open-source dependency risks, and supply-chain attacks such as Log4j, SolarWinds, and CodeCov recur across the selected studies. These recurring challenges, the paper argues, reveal research gaps in how tools and practices address security in CI/CD pipelines, justifying further study.

Load-bearing premise

The 66 papers that survived the selection process are a complete and unbiased representation of the literature on CI/CD security in the cloud, since no exclusion log or quality assessment is reported.

Editorial extensions

If this is right

A practitioner can use the catalog of 62 tools and eight approaches as a starting checklist for container scanning, static and dynamic analysis, monitoring, and DevSecOps in cloud CI/CD.
The challenge categories of installation and updating, practitioner and developer issues, organisational issues, and third-party and open-source tool difficulties show where current tooling is least reliable.
GitHub Actions and low-code platforms emerge as specific weak points that need targeted hardening and further study.
The paper's planned next steps, topic modeling of fragmented security text and a blockchain-based solution for container and deployment security, follow directly from the identified gaps.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because 40 of the 66 included papers were published in 2021 to 2023, the field is young, and the tool and challenge map may need to be re-run within a few years as GitHub Actions and supply-chain security mature.
The review's tool list is heavily Docker- and container-centric; a follow-up review targeting serverless, Kubernetes-native, or multi-cloud deployment models could find a different challenge profile.
The paper counts tools but does not compare their effectiveness; a natural testable extension would be to run a fixed vulnerability suite through the most-cited scanners to see which reported gaps actually close.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This one is useful as a map, but I'd hold the 'systematic' label until the authors open their box.

What the paper does well: it picks a real niche—cloud-specific CI/CD security—that sits between Shahin et al. 2017 and Rajapakse et al. 2022, and it lists a practical range of tools (the 62-tool table includes Harbor, SonarQube, GitHub Actions, Trivy, etc.) and challenge categories (authorization, vulnerability assessment, third-party/OSS, supply chain). Practitioners starting in this area will find the tables genuinely useful, and the demographic data showing 60% of the 66 papers published 2021–2023 confirms this is a live topic. The process is recognizable: search strategy, digital libraries, snowballing, inclusion/exclusion, data extraction.

The soft spot is not the selection criteria; it's that the execution doesn't match the claim. There is no list of the 66 included studies, no PRISMA-style flow, no exclusion log, and no quality assessment, so nobody can audit the corpus. Worse, the results tables contain citations that are not in the reference list—'Jyothsna et al., 2011; Kumar and Sangwan, 2012' for signature/anomaly-based approaches don't appear anywhere in the references. Other rows cite conceptual works like STRIDE and Asylo even though the inclusion criteria require empirical research. That tells me the tables were partially filled from background knowledge or reference chasing, not strictly from the 66 vetted papers. If so, RQ1 and RQ2 are not a systematic synthesis; they're an informed newsletter. The same holds for Section 2.2's research gaps, which feel assembled from prior reviews rather than derived from the corpus.

None of this is fatal to the paper's usefulness as a first stop for someone entering the field. But it is fatal to its current claim to be a systematic literature review. The fix is straightforward: supply an appendix with the full list of 66 papers, a flow diagram, a mapping from each table row to an included study, and a quality assessment. If the authors can't do that, they should reframe the paper as a narrative survey.

I'd send this to peer review—the topic matters, and a revised version could meet SLR standards. But the reviewer should demand the supplementary material before any acceptance. Don't cite it in its current form.

Referee Report

5 major / 6 minor

Summary. This manuscript presents a systematic literature review (SLR) on the security of CI/CD pipelines in cloud computing. The authors define three research questions: existing tools and methods (RQ1), proposed solutions (RQ2), and challenges (RQ3). They report a search across six digital libraries, 4,889 initial records, 573 after screening, and 66 included primary studies. The results comprise tables of existing and proposed tools/approaches and a narrative of challenges, followed by discussion, validity threats, conclusions, and future work. The core claim is that the 66 selected studies provide a comprehensive map of the field and reveal research gaps in the security of cloud-based CI/CD pipelines.

Significance. If the synthesis were fully traceable, this SLR would fill a useful niche by focusing specifically on security of CI/CD in the cloud, complementing broader prior surveys such as Shahin et al. and Rajapakse et al. The paper follows an established SLR structure: PICOC-framed RQs, automated search over six libraries, snowballing, and demographic analysis. The demographic breakdown (Section 4) and the identification of specific challenges such as image manipulation, unauthorized access, and weak authentication are potentially useful. However, the contribution is currently unverifiable because the included-study list, exact search string, and quality assessment are not reported. The stated mapping of tools and challenges therefore cannot be audited, which limits the practical value of the review as a systematic secondary study.

major comments (5)

[Section 3.5, Step 3] The paper states 'After thoroughly reviewing the full articles, 66 were included in our final selection' but never lists these 66 papers or provides a PRISMA-style flow diagram with counts at each stage. This is load-bearing because RQ1-RQ3 results are aggregates over this sample; without the list, the reader cannot verify that the sample matches the inclusion criteria or that the tables and narrative in Section 4 indeed derive from the sample. Please add a supplementary list of included studies with unique identifiers and use those identifiers consistently in all results tables and in Section 4.3 challenges.
[Tables 2 and 3] Several citations in the results tables are not traceable to the declared corpus or the reference list. For example, Table 2's 'Signature-based, Anomaly-based' row cites 'Jyothsna et al., 2011; Kumar and Sangwan, 2012,' neither of which appears in the References. The 'STRIDE' row cites Davis et al. (2022), which per the reference list is a paper on a first offering of a software engineering course, not an empirical study of secure CI/CD. These entries indicate that some table rows were not sourced from the 66 vetted primary studies, undermining the claim that the tool map is a systematic synthesis. Provide a row-by-row mapping to included studies; rows without such a mapping should be removed or moved to a clearly labeled background section.
[Section 3.2 and Figure 1] The search string is only shown in an image (Figure 1) and is not provided as text. Reproducibility is a core requirement of an SLR; the reader cannot see the exact terms, Boolean operators, field restrictions, or date filters. Include the full search string for at least one digital library, the date of the search, and the number of hits per library.
[Section 6] No quality assessment of the 66 primary studies is reported, even though the method section cites Kitchenham et al. (2022a). Section 6 mentions variability in study design and quality but does not describe an assessment instrument, how quality was scored, or how it influenced the synthesis. Without this, the challenge synthesis in Section 4.3 gives equal weight to all sources. Add a quality checklist, report scores or a summary, and state how quality was used in the synthesis.
[Section 2.2] The 'Research Gaps' subsection is framed as an outcome of this review, but the paragraphs cite prior literature (e.g., Garg and Stavik 2019; Rafi et al. 2022; Decan et al. 2022) rather than being demonstrably derived from the 66 included studies. It is unclear whether these gaps emerged from the included studies or from the authors' broader reading. Clarify the provenance of these gaps; if they come from related work, label them accordingly and show how the current SLR extends them.

minor comments (6)

[Section 1] The sentence '66 met our selection criteria (see Section 3.3)' points to the wrong section; the selection criteria are defined in Section 3.4, not Section 3.3.
[Section 3.5, Step 2] The numbers are confusing: the text says 573 papers remained after screening, of which '482 directly met our criteria, and an additional 91 were found using the Snowballing method.' Since 482 + 91 = 573, clarify whether the 573 already includes the snowballed papers, and provide exclusion counts for each reason.
[Table 2] The 'GitHub Actions (GHA)' row cites 'Tu et al., 2021,' but the reference list entry for Tu et al. (2021) is about Open vSwitch dataplane performance, which appears unrelated to GitHub Actions. Verify or replace this citation.
[References] There are inconsistent author names between text and references: 'Garg and Stavik, 2019' vs. 'Garg and Garg, 2019'; 'Brandy et al., 2020' vs. 'Brady et al., 2020'; and the reference list contains duplicate entries for Shahin et al. (2023). Standardize these.
[Section 4.2] The bullet 'Trust developers' cites Shahin et al. (2017b), a prior survey; if this is a finding from the included studies, provide the corresponding included-study ID, otherwise clarify that it is background advice.
[Figure 2] The study-selection figure is labeled 'Steps of Study Selection for SLR' but contains no numbers at each step. Replace or augment it with a flow diagram showing the counts for retrieval, screening, eligibility, and inclusion, as is standard in SLR reporting.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the SLR's descriptive synthesis draws on primary studies and does not reduce to its own inputs or to self-citations.

full rationale

This paper is a systematic literature review, not a derivation or prediction exercise. Its central outputs—the tool inventories in Tables 2 and 3, the challenge summaries in Section 4.3, and the research gaps in Section 2.2—are descriptive syntheses of cited primary studies. The review does not fit parameters to data and then rename them as predictions; it does not define its key terms in terms of its conclusions; and it does not invoke an author-specific uniqueness theorem or ansatz. The authors' own prior work does not appear as load-bearing evidence; the methodology follows external guidelines (Kitchenham, Wohlin, Brereton, Zhang), and the identified tools and challenges are attributed to the surveyed literature. The skeptical concern that the 66-paper corpus is not listed and that some table citations may come from outside the corpus is a legitimate threat to traceability, completeness, and validity, but it is not circularity: even if the tables were populated from background knowledge, that would mean the review is unsystematic or unverifiable, not that its claims are equivalent to its inputs by construction. No circular step can be quoted and reduced to an equation or fitted parameter, so the appropriate finding is no significant circularity with score 0.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The review's conclusions depend on the completeness and representativeness of the included primary studies and the adequacy of the search sources. No free parameters or invented entities are introduced. The exact search string and the list of included studies are not provided, so the search protocol itself is an unverified assumption.

assumptions (2)

domain assumption The 66 selected papers form a representative and complete sample of the literature on CI/CD security in the cloud.
All conclusions in Sections 4 and 5 generalize from this sample. Section 3.5 reports 66 final papers but provides no list or exclusion log to verify representativeness.
domain assumption The six chosen digital libraries plus snowballing cover the relevant literature.
Section 3.3 excludes CiteSeerX, AIS eLibrary, Kluwer, Inspec, and Google Scholar for search-function or overlap reasons, but the completeness of this coverage is not empirically validated.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Systematic Literature Review on Continuous Integration and Deployment (CI/CD) for Secure Cloud Computing." pith.science (2026). https://pith.science/paper/RDGFIYKG

@misc{pith2026250608055,
  author       = {Pith},
  title        = {Pith review of: A Systematic Literature Review on Continuous Integration and Deployment (CI/CD) for Secure Cloud Computing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RDGFIYKG}},
  note         = {Machine review of arXiv:2506.08055}
}

read the original abstract

As cloud environments become widespread, cybersecurity has emerged as a top priority across areas such as networks, communication, data privacy, response times, and availability. Various sectors, including industries, healthcare, and government, have recently faced cyberattacks targeting their computing systems. Ensuring secure app deployment in cloud environments requires substantial effort. With the growing interest in cloud security, conducting a systematic literature review (SLR) is critical to identifying research gaps. Continuous Software Engineering, which includes continuous integration (CI), delivery (CDE), and deployment (CD), is essential for software development and deployment. In our SLR, we reviewed 66 papers, summarising tools, approaches, and challenges related to the security of CI/CD in the cloud. We addressed key aspects of cloud security and CI/CD and reported on tools such as Harbor, SonarQube, and GitHub Actions. Challenges such as image manipulation, unauthorised access, and weak authentication were highlighted. The review also uncovered research gaps in how tools and practices address these security issues in CI/CD pipelines, revealing a need for further study to improve cloud-based security solutions.

Figures

Figures reproduced from arXiv: 2506.08055 by the authors.

**Figure 1.** Search String of the initial search for SLR. 3.3 Data Collection Sources The automatic search was carried out across six digital libraries: Scopus, ACM, IEEE Xplore, Wiley, Springer Link (SL), and ScienceDirect (SD) (Chen et al., 2010). CiteSeerX and AIS eLibrary have complex search functions and lack post-query refinements (Li & Rainer, 2022; Brereton et al., 2007). Kluwer has merged with and is indexed by Springer… view at source ↗

**Figure 2.** Steps of the Study Selection for SLR. We passed the subsequent steps for this SLR: Step 1: We started with 4,889 articles based on the search criteria. Step 2: We screened the titles, keywords, and abstracts to narrow it down to 573 papers. Of these, 482 directly met our criteria, and an additional 91 were found using the Snowballing method. Step 3: We reviewed the introductions and conclusions of the 573 papers, se… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

6 extracted references · 6 canonical work pages

[12]

(2020, January)

Brady, K., Moon, S., Nguyen, T., & Coff man, J. (2020, January). Docker container security in cloud computing. In 2020 10th Annual Computing and Communication Workshop and Conference (CCWC) (pp. 0975-0980). IEEE. Brereton, P., Kitchenham, B. A., Budgen, D., Turner, M., & Khalil, M. (2007). Lesso ns from applying the systematic literature review process wi...

work page 2007
[22]

2747-2763)

(pp. 2747-2763). Kumar, R., & Goyal, R. (2020). Modeling continuous security: A conceptual model for automated DevSecOps using open -source software over cloud (ADOC). Computers & Security, 97, 101967. Lacoste, F. J. (2009, August). Killing the gatekeeper: Introducing a continuous integration system. In 2009 agile conference (pp. 387-392). IEEE. Le, M. V....

work page 2020
[44]

A., & Tell, P

Zhang, H., Babar, M. A., & Tell, P. (2011). Identifying relevant studies in software engineering. Information and Software Technology, 53(6), 625-637. Zhang, Y., Vasilescu, B., Wang, H., & Filkov, V. (2018, October). One size does not fit all : an empirical study of containerized continuous deployment workflows. In Proceedings of the 2018 26th ACM Joint M...

work page 2011
[164]

Security Support in Continuous Deployment Pipeline

Saboor, A., Hassan, M. F., Akbar, R., Susanto, E., Shah, S. N. M., Siddiqui, M. A., & Magsi, S. A. (2022). Root - Of-Trust for Continuous Integration and Continuous Deployment Pipeline in Cloud Computing. Computers, Materials and Continua, 73(2), 2223-2239. Saroar, S. G., & Nayebi, M. (2023, June). Developers’ perception of GitHub Actions: A survey analys...

work page Pith review arXiv 2022
[2006]

Proceedings 7 (pp. 3-3). Springer Berlin Heidelberg. Kitchenham, B. A., Dyba, T., & Jorgensen, M. (2004, May). Evidence -based software engineering. In Proceedings. 26th International Conference on Software Engineering (pp. 273-281). IEEE. Kitchenham, B., Madeyski, L., & Budgen, D. (2022). How should software engineering secondary studies include grey mat...

work page 2022
[2017]

14(1)-14(8)

SQAMIA’17, 2017, pp. 14(1)-14(8). Abhishek, M. K., & Rao, D. R. (2021, July). Framework to secure docker containers. In 2021 Fifth Wo rld Conference on Smart Trends in Systems Security and Sustainability (WorldS4) (pp. 152-156). IEEE. Akbar, M. A., Mahmood, S., & Siemon, D. (2022, June). Toward effective and efficient DevOps using blockchain. In Proceedin...

work page 2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [12]

(2020, January)

Brady, K., Moon, S., Nguyen, T., & Coff man, J. (2020, January). Docker container security in cloud computing. In 2020 10th Annual Computing and Communication Workshop and Conference (CCWC) (pp. 0975-0980). IEEE. Brereton, P., Kitchenham, B. A., Budgen, D., Turner, M., & Khalil, M. (2007). Lesso ns from applying the systematic literature review process wi...

work page 2007

[2] [22]

2747-2763)

(pp. 2747-2763). Kumar, R., & Goyal, R. (2020). Modeling continuous security: A conceptual model for automated DevSecOps using open -source software over cloud (ADOC). Computers & Security, 97, 101967. Lacoste, F. J. (2009, August). Killing the gatekeeper: Introducing a continuous integration system. In 2009 agile conference (pp. 387-392). IEEE. Le, M. V....

work page 2020

[3] [44]

A., & Tell, P

Zhang, H., Babar, M. A., & Tell, P. (2011). Identifying relevant studies in software engineering. Information and Software Technology, 53(6), 625-637. Zhang, Y., Vasilescu, B., Wang, H., & Filkov, V. (2018, October). One size does not fit all : an empirical study of containerized continuous deployment workflows. In Proceedings of the 2018 26th ACM Joint M...

work page 2011

[4] [164]

Security Support in Continuous Deployment Pipeline

Saboor, A., Hassan, M. F., Akbar, R., Susanto, E., Shah, S. N. M., Siddiqui, M. A., & Magsi, S. A. (2022). Root - Of-Trust for Continuous Integration and Continuous Deployment Pipeline in Cloud Computing. Computers, Materials and Continua, 73(2), 2223-2239. Saroar, S. G., & Nayebi, M. (2023, June). Developers’ perception of GitHub Actions: A survey analys...

work page Pith review arXiv 2022

[5] [2006]

Proceedings 7 (pp. 3-3). Springer Berlin Heidelberg. Kitchenham, B. A., Dyba, T., & Jorgensen, M. (2004, May). Evidence -based software engineering. In Proceedings. 26th International Conference on Software Engineering (pp. 273-281). IEEE. Kitchenham, B., Madeyski, L., & Budgen, D. (2022). How should software engineering secondary studies include grey mat...

work page 2022

[6] [2017]

14(1)-14(8)

SQAMIA’17, 2017, pp. 14(1)-14(8). Abhishek, M. K., & Rao, D. R. (2021, July). Framework to secure docker containers. In 2021 Fifth Wo rld Conference on Smart Trends in Systems Security and Sustainability (WorldS4) (pp. 152-156). IEEE. Akbar, M. A., Mahmood, S., & Siemon, D. (2022, June). Toward effective and efficient DevOps using blockchain. In Proceedin...

work page 2023