CI/CD: Webshell deployed into production environment

The Vulnerability:

We have found a critical vulnerability that is important for every developer to look out for: the ability to deploy malicious code directly into the production environment. This was not merely hypothetical; our assessment was successfully executed.

Details:

Misconfigured Deployment Workflow: The heart of the issue lies in the CI/CD (Continuous Integration/Continuous Deployment) pipeline. This crucial process lacked the safeguards necessary to ensure every piece of code deployed was thoroughly reviewed and approved. The absence of these safeguards is an invitation for attackers.

Exploitation of Vulnerabilities:

Leveraging the gaps in the CI/CD pipeline, we were able to deploy a NodeJS webshell—a type of malicious code that provides an attacker with a remote command line interface to the web server—into the production environment. This was not just any production environment, but one residing within an OCP4 (OpenShift Container Platform 4) cluster, which is typically known for its robust security features.

Why did this happen?

There are several factors that contributed to this vulnerability:

• Oversights during CI/CD configuration: During the setup phase, crucial security steps were either missed or misconfigured.
• Underestimation of approval processes: The processes designed to ensure each deployment was vetted were not given the weight they required.
• Knowledge gaps: There may have been a lack of understanding or awareness of the grave threats posed by unchecked deployments within the team responsible for these systems.

How to Mitigate:

To close this gaping security hole, we recommend the following steps:

• Mandatory Review and Approval: Introduce an ironclad rule that no code reaches production without passing through a rigorous review and approval process.
• Pipeline Audits: Make it a routine to comb through the CI/CD pipeline, searching for misconfigurations or security steps that might have been skipped.
• Education and Awareness: Foster a culture of security by educating the development and operations teams about the risks of unchecked deployments.
• Automated Security Tools: Employ advanced tools that automatically detect and block any unauthorized or abnormal changes during the development and deployment process.

This incident underscores the necessity of vigilance and stringent security protocols in today’s digital landscape. It’s a call to action for organizations to reassess and reinforce their cybersecurity strategies.

The Hidden Dangers of Found Unscanned Secrets in GitHub Repositories

In the realm of source code management, GitHub stands as a titan, hosting millions of code repositories. However, it is crucial to know how to store data in repositories. We have found and highlighted a significant security misconfiguration: the lack of properly configured secret scanning. This oversight underlines the importance of adhering to best practices in repository management to ensure data integrity and security.

The Vulnerability:

Our assessment has revealed that secret scanning, an automated feature in GitHub (https://docs.github.com/en/code-security/secret-scanning/about-secret-scanning) designed to detect sensitive information like passwords and private keys, was not enabled. Additionally, no protection nor detection against saving sensitive data in repositories was applied. This leaves repositories vulnerable to accidental exposure of sensitive data.

Details:

Secrets Exposed: Without secret scanning, sensitive data can be inadvertently committed to public or private repositories, posing a substantial risk to security.

Potential Impacts:

• Data Breach: Exposed secrets are a goldmine for attackers, potentially leading to data breaches and unauthorized access to other services.
• Infrastructure Compromise: Access tokens, SSH keys, and other credentials could be exploited, compromising not just the repositories but also the infrastructure and services they interact with.

Why did this happen?

The lack of configuration of secret scanning can stem from:

• Lack of Awareness: Developers and repository administrators might be unaware of the feature or its importance.
• Misconfiguration: In the process of repository setup or during migration between systems, configurations such as secret scanning may have been overlooked.
• Expenses and Expertise: Secret scanning feature license from Github must be purchased which can be costly resulting in a less secure environment. Expertise is important in regards to CI/CD and security, therefore experts are responsible for controlling and upholding the rules to assure the highest level of security.

How to Mitigate:

To mitigate this vulnerability, the following steps should be taken immediately:

• Enable Secret Scanning: Turn on secret scanning for all GitHub repositories to detect committed secrets in real-time, markets offer different solutions that can fit your preferences.
• Educate and Train: Provide training for developers on secure coding practices and the importance of keeping secrets out of code repositories.
• Automate Secret Removal: Implement tools that automatically revoke and reissue secrets that have been exposed.
• Policy Enforcement: Establish policies that prohibit the inclusion of secrets in code and enforce them with automated checks.

It is essential to recognize that securing code repositories is just as crucial as securing the infrastructure they deploy to.

Github’s Branch Protection Bypass: A High-Risk Vulnerability Uncovered

Recently, we used a known bypass to be able to exploit GitHub Actions Branch Protection Rules, a feature that is crucial for maintaining the integrity of code in multiple development environments.

The Vulnerability:

In essence, this vulnerability centers around the unauthorized bypass of branch protection mechanisms in Github. Branch protection rules are pivotal in ensuring that changes to important branches require proper reviews and approvals. A known bypass that we used in GitHub’s system enables the exploitation of GitHub Actions to circumvent the mandatory review process. This breach permits the unauthorized insertion of unreviewed code into a secure branch. Such a loophole could lead to the unintended use of harmful code by other users or its inadvertent integration into the production pipeline.

Details:

Mechanism of Bypass: The exploitation process involved leveraging the permissions of Github Actions. By utilizing write permissions on the pull requests API endpoint, we could create a pull request to merge malicious code into a protected branch. Although such a pull request typically cannot be merged without approval, the Github Actions bot, associated with the GITHUB_TOKEN, could approve it. Since the bot is not an organization member but still counts for approval purposes, it allows an attacker to self-approve the pull request, effectively bypassing branch protection mechanisms. This fact permits the unauthorized insertion of unreviewed code into a secure branch. Such a loophole leads us into unauthorized code integration into the production environment.

Risks and Threats:

• Unauthorized code changes to protected branches.
• Deployment of malicious code to production and other environments.
• Compromised application integrity and reliability.
• Damaging trust in the deployment process and overall infrastructure.

Why did this happen?

This issue arises from a gap in Github’s settings which allowed GitHub Actions to approve pull requests, a default that was not secure in existing organizations.

How to Mitigate:

To address this significant risk, the following solutions are proposed:

• Update Github Settings: Use the Github setting that disallows Github Actions from approving pull requests. This needs to be set organization-wide to be effective.

• Review Approvals: Enforce pull request reviews from Code Owners to ensure that any changes have been vetted by authorized personnel.
• Increase Approval Requirements: Amend the number of required approvals to two or more, which adds an extra layer of security by necessitating multiple reviewers.

This vulnerability underscores the critical importance of constant vigilance and regular updates to security configurations in all aspects of development operations. By implementing the recommended changes, organizations can safeguard their code from such exploits and reinforce the trust in their deployment processes.

Artifactory Service Configuration - Unintended Open Data Access

The integrity of any development pipeline hinges on the security and controlled access to its artifacts. In a recent security assessment we did, we managed to use a gaping hole in the Artifactory service to log in without explicit permissions.

The Vulnerability:

Artifactory allows any domain user to log in without requiring explicit permissions. Once logged in, the user can view, download artifacts, and check vulnerability reports from Xray scans related to the repositories.

Details:

Overly Permissive Access Controls: The default configuration permitted every user within the domain to access the Artifactory service. This kind of configuration neglects the principle of least privilege, which is essential in securing sensitive data.

This way, a lot of sensitive information is made available to each domain user. The ability to identify which applications contain critical vulnerabilities can be used by malicious actors to exploit these vulnerabilities. In addition, artifacts may contain sensitive data because there is no mandated scanning of such data.

Risks and Consequences:

• Unauthorized access to sensitive artifacts.
• Potential data leakage.
• Erosion of data integrity.
• Increased risk of insider threats.

Why did this happen?

The reasons for this misconfiguration might include:

• Lack of Role-Based Access Controls (RBAC): Insufficiently defined user roles and permissions could lead to such wide-open access.

How to Mitigate:

Addressing this vulnerability involves a strategic overhaul of access permissions:

• Implement RBAC: Define user roles with precision and assign permissions that align with the principle of least privilege.
• Audit Access Logs: Regularly review access logs to monitor for any unauthorized attempts or accesses to the service.
• Continuous Monitoring: Implement real-time monitoring tools to detect and alert unusual access patterns.
• Regular Access Reviews: Periodically review user access rights to ensure they remain appropriate to each user’s role and responsibilities.

The Artifactory data exposure risk serves as a stark reminder that even the most trusted services within the development pipeline must be scrutinized for security gaps. By taking decisive action to correct these misconfigurations, organizations can reinforce their defenses against both external and internal threats.

OpenShift Misconfiguration: Unrestricted Host Network Access from POD

The agility and scalability provided by containerized environments can also introduce significant security risks if not properly managed. A recent assessment has shed light on a serious misconfiguration within an OpenShift cluster, highlighting a vulnerability that could have severe implications for network security.

The Vulnerability:

In the spotlight is a configuration mishap in OpenShift, a popular container orchestration platform. This misconfiguration allowed for pods (the smallest deployable units of computing that can be created and managed in OpenShift) to have the ability to reach the IP address of the host machine and even the entire host subnet. In layman’s terms, if a pod were to be compromised, the attacker would have unfettered access to potentially sensitive internal network resources.

Details:

Wide-Ranging Network Visibility: This misconfiguration was not trivial; it meant that a deployed pod in OpenShift could see and potentially interact with a vast array of internal systems. This kind of access is akin to giving someone the keys to the kingdom and could be exploited to devastating effect.

Risks and Consequences:

• Potential for widespread internal network reconnaissance by a compromised pod.
• Increased risk of lateral movement within the network, leading to more systems being compromised.
• Data breaches due to access to sensitive internal systems.
• Compromised network integrity and security posture.

Why did this happen?

The root cause of this misconfiguration could be:

Insufficient Network Policies: lack of stringent network policies to restrict the PODs’ network access to only necessary resources and services within their own subnet Default Configuration Pitfalls: Default settings that favor ease of deployment over security, which are not revisited and properly secured post-deployment.

How to Mitigate:

To correct this dangerous oversight, organizations should take the following actions:

• Network Segmentation: Implement network policies which specifically define which network connections are allowed for each pod, based on the principle of least privilege.
• Regular Security Audits: Conduct periodic reviews of the network configurations within the OpenShift environment to ensure compliance with security best practices.
• Pod Security Standards: Adopt and enforce pod security standards that limit the capabilities and access of pods, especially those running in the production environment.
• Security Training: Provide ongoing training for staff on the potential risks of misconfigurations and the importance of secure deployment practices.

This example of the OpenShift misconfiguration serves as a cautionary tale about the critical need for rigorous network security practices in containerized environments. Through diligent management and constant vigilance, organizations can safeguard their networks from such vulnerabilities.

GitHub Configuration Flaws: A Gateway to Unauthorized Deployments

The Vulnerability:

Recently, we have highlighted a glaring issue within GitHub’s repository configuration where source codes of applications were saved within a GitHub organization, which opens the door to a vulnerability that could lead to unauthorized and potentially harmful deployments.

Details:

The core of this problem lies in the permissiveness of default repository configurations and the lack of stringent flow control mechanisms. As there are no branch protection rules, no environments with protection rules and secrets, this situation allows anyone with write access to the repository to perform potentially malicious activities such as reading all secrets, creating any artifacts (including malicious ones), removing static source code analysis checks in the pipeline, and potentially exploiting self-hosted runners.

Risks and Consequences:

• Access to any secret available to the CI job, such as secrets injected as environment variables or additional secrets stored in the CI. Being responsible for building code and deploying artifacts, CI/CD systems typically contain dozens of high-value credentials and tokens – such as to a cloud provider, to artifact registries, and to the SCM itself.
• Access to external assets the runner has permissions to, such as files stored in the node’s file system, or credentials to a cloud environment accessible through the underlying host.
• Ability to ship code and artifacts further down the pipeline, in the guise of legitimate code built by the build process.
• Ability to access additional hosts and assets in the network/environment of the job node.

Why did this happen?

This problem often arises from:

• Inadequate Repository Settings: Default settings that are geared towards convenience over security can leave repositories vulnerable.
• Flow Control Neglect: Insufficient measures to ensure that changes go through a rigorous approval process before being merged or deployed.

How to Mitigate:

To prevent such unauthorized deployments and to bolster security, the following steps are recommended:

• Configure Branch Protection Rules: Enforce rules that require reviews before pull requests are merged, particularly into critical branches.
• Protect Environments: Set up protection rules for environments to control deployment, such as allowing deployment only from protected branches.
• Secure Secrets in Environments: Store all secrets within environments to limit access only to jobs that have been approved.
• Usage of GitHub Repository Rulesets: provide organization administrators with enhanced control and features like unified configuration and branch targeting. (https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-rulesets/about-rulesets)
• Mandatory Code Analysis Workflow: Implement required workflows for static code analysis on pull requests across all branches within the organization.

Artifactory - Deployment of Vulnerable Artifacts

Next up was shedding light on a vulnerability in the way artifacts are managed and deployed. This issue, if left unchecked, could pave the way for significant security threats in production environments.

The Vulnerability:

During the deployment process of artifacts stored in Artifactory, we have discovered that artifacts, even those with critical or high-level vulnerabilities, could be deployed to production environments without any pre-deployment verification. This lapse in the verification process leaves the production environment open to potential security breaches.

Artifactory X-ray: node. Js webshell docker image containing critical vulnerability. Later deployed into production

We recognize that blocking all vulnerable artifacts could disrupt the build/deployment process and could have business implications. Nevertheless, these vulnerabilities should at least be reviewed and approved prior to deployment to reduce the risk of introducing security weaknesses into the production environment.

Impact:

The potential impacts of deploying vulnerable artifacts are far-reaching and alarming:

• Increased Cyber-Attack Risk: Deploying artifacts with vulnerabilities directly into production increases the likelihood of cyber-attacks.
• Business Operations Compromise: The exploitation of critical vulnerabilities could disrupt or even compromise key business operations.
• Regulatory Non-Compliance: Such practices might contravene industry standards or regulations that mandate effective vulnerability and patch management.

The Solution:

To mitigate these risks, the following measures are strongly recommended:

• Vulnerability Scanning Process: Implement a process that scans all artifacts for high or critical vulnerabilities before deployment into the production environment. This process should include a mandatory approval for deploying artifacts with identified vulnerabilities.
• Use of Xray: In Xray, configure policies to identify issues with high/critical vulnerabilities. Initially, avoid actions like failing builds or preventing downloads but instead focus on sorting through each violation.
• Comprehensive Patch Management: Establish a robust patch management system to ensure the timely detection, assessment, and remediation of vulnerabilities in artifacts.

Conclusion:

Our goal was to bypass all security measures and deploy malicious code to the production environment with knowledge and access to the OCP4 configuration and the entire CI/CD pipeline and assigned namespace in the cluster.

We found that the scenario is feasible by exploiting a chain of misconfigurations and vulnerabilities, which are specifically described in the following sections:

• Github - Unauthorized Bypass of Branch Protection Mechanisms
• Github - Permissive Default Repository Configuration and Insufficient Flow Control Mechanisms
• Artifactory - Deployment of Vulnerable Artifacts

In the end, we were successful in deploying the malicious application - NodeJS webshell to a production environment within an OCP4 cluster. We want to highlight the importance of developers’ responsibility and how a combination of small vulnerabilities can lead into a critical situation where all the hard work and security of companies can be in danger.

Are you not sure about the security of your pipeline? Test it with us.