Transcript
Emma Yuan Fang: I’m going to talk about two important topics. The first one is Zero Trust and how do we apply that into securing our supply chain, basically. The second one is the supply chain, and what are the attacks that we would expect in our supply chain and CI/CD pipeline, and how do we mitigate them?
My name is Emma. I also run by the name of Yuan. I’m a security architect at EPAM. I’m also leading the UK and Ireland security practice for my security team. I’m leading a team of security consultants, security architects, and engineers. I have been involved in a few DEI initiatives such as the Women Techmaker Ambassador. I’m also part of the leadership team helping them to build the WiCyS UK and Ireland affiliate. I’m also a passionate mentor. I have been involved in a few mentorship programs. I have spoken at a number of conferences in the UK and the USA, the States. I’m going to talk about Zero Trust.
Roadmap
I’m going to run through the threat landscape of software supply chain. Then I’m going to explain what are the Zero Trust principles. How do we utilize them as embedded into developers’ mindsets to apply to securing the software supply chain. Then, I’m going to talk about how do we effectively manage and select dependencies and all the security controls that we can implement within the CI/CD pipeline. Finally, instead of key takeaways I’m going to give away a checklist. It’s going to be a high-level checklist but it’s for you to take away to your company and do further research and apply to your process.
Threat Landscape: Software Supply Chain
First of all, I’m going to talk about the threat landscape of the software supply chain. We need to understand what are the threats coming out from this supply chain in terms of producing the code, the dependencies, and coming out. How do we as developers and companies consume those packages, those dependencies. Many of you might have already heard about the XZ Utils’ backdoor. This is an attack that does have a human indication in it because a malicious developer planted backdoors into libraries and packages of open-source data. How does it affect the end user? It has been around for three years but nobody knows about it because it looks like it’s a legitimate update to the packages.
The next one I’m going to talk about is the famous SolarWinds attack. This campaign is quite big, it affects tens of thousands of downstream customers. The beauty of this attack is that the attacker injected malicious backdoor into the build processes by compromising a vulnerability in that build process. The impact has been enormous. The reason for that is Orion is a software built by SolarWinds, so it’s used by tens of thousands of customers to do their IT monitoring system, to have the IT monitoring capability. This update has been affecting all the downstream systems and environments.
Dependency confusion attack is the next big thing in terms of software supply chain. You’ve probably heard about it. This is an example, PyTorch. Basically, it’s exploiting the name confusion in the software component. It can be used to trick systems into downloading and running malicious code instead of the legitimate one. Another one that I want to talk about is typosquatting. Typosquatting is to have a name that’s similar to the original package name, but instead of the legitimate name it’s using something that’s misspelled. It’s really hard sometimes to check over those names when you have hundreds of thousands of packages in your systems. That is exactly why attackers like to use this technique. It’s a social engineering technique, so it can be used to trick the users into downloading malicious packages. That’s all it takes.
Another one that I want to mention here that’s affecting the packages and the build process, is the artifact. If you think about supply chain as all about dependency, that’s wrong. Why is that? That’s because the supply chain attack can also happen in the build process because our artifact, our container images, can also be compromised even in the runtime. This example, leaked credentials have been used to upload a malicious artifact into the build process.
What are the lessons learned from open-source security? I talk about open-source security because most of our code is actually coming from open source. Open-source vulnerabilities can have widespread impact like the SolarWinds example. It impacted all the customers downstream. Mostly, this kind of attack often finds its way into the internal system. If you think that it will stop at the dependency level, that’s wrong because the victims can download those packages containing the malicious code, and backdoors and everything else can be triggered as part of that action. Then an attacker can launch a command-and-control channel. Have a look at this diagram for me. It looks really complicated, but what I want to highlight here is an attack can also happen in the CI/CD pipeline, in your build process.
If you think that the supply chain attack is just at the beginning, that’s wrong because it can affect all the way to the runtime. I want to highlight a few key points that I haven’t discussed in the previous attack. There is an instance where a researcher was able to trick developers into downloading an IDE that’s actually malicious, it’s not legitimate, by just using typosquatting techniques. They placed a dev extension that’s just similar to the legitimate one in the marketplace. If you fully trust the marketplace, trust everything on there, then there is a possibility that you could download something that’s malicious. Another thing that I would like to mention is the CI Runner.
If you think, I’m going to just have one CI Runner that is going to be used for every single build that I have, that’s wrong because you might have a project that has different sensitive information that is going to be processed and hosted. If you’re using a low-prioritized or low-sensitive project, if you’re using the CI Runner for that one to run a high-prioritized project, what could happen? If anything is happening with that low-priority runner, then your higher-priority build system could be affected. Other things about the CI/CD pipeline would be, for example, the secret. I want to talk about the secret. If you don’t manage the secret properly, that could affect pretty much all stages of the CI/CD pipeline. Let me tell you why.
First of all, you have application secrets. Second, you have secrets that you use in the build pipelines. That’s why we talk about secrets that affect all the stages of CI/CD. What are the impacts coming out from the malicious packages? Data breach and data exfiltration. Packages can be used to gather sensitive information. A lot of incidents have told us that further system compromise could happen because the malicious code can be used in the lateral movement by the attacker. As a result, financial and reputation impacts. I don’t need to mention that. If you have to, compliance with DORA. If you don’t manage your supply chain, you could potentially face a fine.
Zero Trust Principles, and Zero Trust for the Software Supply Chain
Why do we do Zero Trust? Zero Trust, by the look of it, is about technology and a framework. However, it’s not just that. It’s more about a mindset, in this case, for our supply chain. The three principles of Zero Trust that I summarized here are least privilege access, verify everything, and assume breach. These are the questions that I would ask when you are thinking about Zero Trust principle. What are the key pillars of Zero Trust? How does it apply to supply chain? If you look at Zero Trust, first of all, if you want to verify everything, identity is something that’s really important. You need to verify every single access, whether that’s for the system interaction or for your developers to access your build pipeline. You need to secure your DevOps environment and your workload security.
Thirdly, when applying to the supply chain, is to ensure your code and artifact security and integrity. If you have the mindset of Zero Trust, that is mapping to those three different approaches. I’m talking about the methodology here. We want to be able to continue verifying every step in your CI/CD pipeline, whether that is code commit, build, deployment. In terms of ensuring our package and artifact integrity, we can use signatures. You can apply signatures and hashes. To enhance our least privilege principle, we can restrict access to CI/CD pipeline tools and apply just-in-time access. If you think about, ok, we have all those tools, then a breach wouldn’t happen. That’s wrong. We need to also apply assume breach mindset. The reason for that is that we assume all the packages and all the dependencies are untrusted. When you have that mindset, you will stay vigilant.
Zero Trust for Dependency Management
How do we manage the dependencies and library? Everything seems to be obvious when you have a piece of libraries or packages. My advice here is before you actually pull down a package or install a library, you need to check whether this library is legitimate. I would say, use checksum and digital signature. I’m going to talk about that. Look for the signs of legitimacy. For example, whether this package has verified publisher badges, and have community feedback and reviews. Look for signs of typosquatting. When you’ve found a nice library that you want to adopt, take a pause and check for everything before you actually download that package into your code. Another sign is the high version of jumps. The high version of jumps could be a sign of malicious code or malicious version being updated by someone. Look for those signs. In terms of package integrity, how do we verify packages? There are two ways. They are being used differently.
The first one is to use the hash. Hash is used to verify your package integrity. The second one is to use digital signatures. Digital signatures use a public key infrastructure. It has a public key and a private key pair. The public key is used to verify that your code is legitimate. There are a lot of ways that you can enforce integrity. Package managers, for example, you can enforce signature in npm. For Python, it also has different ways of enforcing that. Another thing I would like to talk about is version pinning. As developers, you might already have heard about version pinning. Why is it important? Because we want to prevent unexpected breaking changes in our software. Use lockfiles. Lockfile is something that you can lock down the versions. Semantic versioning is an approach where you can specify and have a more controlled update.
For example, patches are for security updates. You can apply that more frequently. Minor updates is something that allows the backwards compatible features. When you have major updates, audit that. Don’t just apply that major update because a new vulnerability can show up. Again, dependency checks. We have different ways of doing dependency checks. For example, a package manager. You have a lot of audit features within the package managers that you can run to enforce automation of the dependency check. SCA tools, software composition tools. OWASP Dependency-Check is an open source one.
These are all the open-source tools, that’s why I’m referencing them here. For image and artifact, you can use Clair. That’s for the container image. Also, a lot of SCA have the ability to generate SBOM. For the OWASP version, CycloneDX, I’m going to provide an example of that. The malware scanner. Why do we need malware scanner in the CI/CD pipeline? That’s because it could stop the malware before it comes into your CI/CD pipeline. There are tools that also combine with a malware scanner.
What is a software bill of materials? SBOM is a tool that can be used to list all the dependencies, including all the transitive dependencies. The type of thing that can be red flagged in the SBOM to look for is unexpected executables in your list of dependencies and vulnerability. SBOM is normally used in combination with the vulnerability exchange program. The VEX is a tool that you can combine with SBOM to provide more details about your vulnerabilities. There are some best practices about SBOM. Why do I mention this? Because SBOM needs to be enforced in your CI/CD pipeline. It’s not just something that you run on a demand basis, because it doesn’t do anything. You need to enforce that in your CI/CD pipeline. Every time before your build is to be run, you need to run a version of the SBOM to pull up all the details of your packages. You need to reject the deployment into the environment if the SBOM highlights any high‑risk CVEs, high‑risk vulnerabilities, and anything that’s unexpected in the component. This is an example of CycloneDX.
On the left‑hand side, figure 1, I have a format of CycloneDX. It contains the metadata of your packages, components, and dependencies, and vulnerabilities details. On the second image here, it basically shows all the outputs of the dependencies, where you can see everything that’s related to that software component and the packages and the libraries. This is an example of the VEX. The VEX actually tells you details about the description of the vulnerability that has been detected in the system, and also the recommendation of how you’re going to mitigate them. Additionally, some of the VEX program can also tell you whether this vulnerability is exploitable. That is very powerful when it comes to prioritizing vulnerabilities.
Zero Trust for Builds and Artifacts
I’ve talked enough about dependency management. How about builds? First of all, I want to talk about the artifact. Why do we sign artifacts? It seems like an additional step that could create a lot of workload. If you don’t sign your artifact, you wouldn’t be able to prove that your code has not been tampered with.
The best practice is to attach a digital signature to your build artifact. Put a timestamp on that build that could be verified. The date and the time of the signing could be verified, so you know that is the signature that’s legitimate. Store that signature in a location that would be accessible to the system that needs this artifact, but it would not be somewhere that’s compromisable. In order to sign your artifact, you can use built-in plugins in your artifact tools. For example, Docker images. There are tools that you can use for enforcing the policies as code. You can enforce signing of the artifact as part of your build process by integrating it into your policy as code.
Then the second step is to enforce it in the CI/CD pipeline. If you only signed it, but not enforced in the CI/CD pipeline, then you wouldn’t be able to ensure that all the artifacts in the build artifact are being signed. You want to ensure and build that into the code. The CI/CD gating will be able to block any unsigned artifacts, so you can ensure that all the signatures are going to be taken and verified before your code is deployed into the higher environment. That’s artifact signing.
Then you need to look into your CI/CD pipelines. There is a thing called branch protection policy, that is quite famously used for approving the workflow of the CI/CD, such as the code changes and the code reviews performed by your developers. Always do manual code reviews. The reason for that is that you always want to have someone who can audit that code before it is changing the software, before it’s rolling into the environment. Another thing is, enforce signed Git commit. This way, you can have all the pull requests commit will be Cosigned, and you need to verify every commit before it’s making changes and merged into the main branch. Of course, audit and monitor.
One of the things that I would like to highlight here is to audit and monitor your CI/CD runners with tools like eBPF. This way, you can detect anomalies in your CI/CD pipelines and generate an audit log for the pipeline. Of course, perform a vendor risk assessment for all the extensions. If you think that, as a developer, I have the power to download any single plugin to improve my productivity, then that’s wrong. It has to be integrated within your company’s third-party risk assessment process to ensure that all the tools that you are using are actually being security assessed. Enforce access to your DevOps and cloud environments. The identity and access management can be enforced in different ways. For example, implementing conditional access just in time to access where the developers can only access that built environment at a particular time or a particular length of time. After that, they shouldn’t be able to access that built environment.
Another tool that I would like to recommend, not a particular tool, but it’s called Cloud Infrastructure Entitlement Management. Microsoft has one, and you also have something that’s on the market. These ones are used to manage your cloud infrastructure identities. Of course, secure the Kubernetes environment control plane, because a lot of supply chain attacks can happen from there if your API server, for example, is not being secured. Secrets management, don’t hardcode your secrets. Your security team probably already told you this, if you’re a developer. That is the number one don’t. As I already mentioned, the secret is required for your applications. For example, if you’re using token-based authentication, that token needs to be signed. That needs to be signed with a public key infrastructure. You can use secret managers, KMS, and Azure Key Vault that comes with the cloud environment to secure your secrets.
Another secret that you need to secure is all the secrets that is being used in your CI/CD pipeline. You can automate those secrets, supply the secrets to the build time by using your CI/CD pipeline tools, but configure them correctly and properly. Don’t forget to secure the APIs that you use to get and inject your secrets. Secrets need to be stored as environment variables or as a volume and shouldn’t be in the code. There is a link that I attached here. I want to also emphasize on the fact that you need to have strategies towards your secrets lifecycle management. Rotating, revoking, and expiring, and storing your secrets. For every project, we sit with our developers. We discuss about how we’re going to manage the secrets within this application. What are the secrets that are coming out from this project?
Finally, is about runtime security. Why do we want to enforce an immutable deployment? Immutable deployment means that once that container image, for example, is deployed into the runtime, it shouldn’t be repatched, it shouldn’t be redeployed if there are any security patches happening. It needs to be destroyed and rebuilt. This can be used to prevent runtime tampering. We need to implement tools to block the runtime tampering.
For example, if an attacker exploits a vulnerability to modify files in a running container, in most of the cases for the container attack, for example, you need to deploy the immutable containers to reject all the writes. The container image should be read only. Next one is the documentation and measuring. I will present some of the top metrics for you later in the slides. Here I wanted to summarize why we need this. Because of the DORA and the EU Cyber Resilience Act, I don’t know whether your security team have already told you this, but this is a compliance regulatory requirement that we need to address. In those acts, it has stated we need to have the ability to list all the software components.
The security framework, such as the NIST Secure Software Development Framework and the supply chain levels for software artifacts can be applied to assess that, to assess the security posture of the supply chain. I’ve also mentioned about the SBOM and the combination of SBOM and the Vulnerability Exploitability Exchange Program. This is an example of supply chain level for software artifacts. The key thing about this framework is that it has different levels of enforcement. At the very bare minimum, you want to ensure that you have visibility of all the packages. In this case, you can adopt tools like SBOM. In the next stage, you can implement something that can verify that your code or artifact is not being tampered with, that is with the signatures. The third level is to harden your build systems. What I mentioned about hardening your build systems is ensuring that your CI Runners are isolated for each of the projects. That’s just one of the examples.
Top Metrics for Software Supply Chain Security
Top metrics for software supply chain security. Of course, you want to know what are the critical vulnerabilities, not all the vulnerabilities because you can’t track everything. If you have a limited budget, track the critical and the high-risk vulnerability being patched within a particular time. For example, seven days. If you are implementing a security gating program, you need to ensure that you sit down with your security team, if you’re a developer, or if you’re a security person, sit down with your developers to identify what are the critical codes or areas of code that needs to be enforced, that needs to have all the security gates being implemented. Another piece of advice I would like to give you is that when you are looking into security gating, it needs to be done in terms of different components. For example, if you have a native frontend, which is mobile app, if you have a web frontend, an API, each of those different types requires a single CI/CD pipeline, basically a separate CI build.
Checklist for Developers
I will advise that if you want to start looking at it today, look at the framework like the SLSA and SSDF. That can be used as a first step to assess your supply chain security posture. Of course, look into the dependency management with version pinning and start scanning your dependency within your package manager. That’s like the first step. Then enforce artifact signing and your Cosigning, and enforce that within your workflow. Harden your CI/CD pipeline by implementing strong access control and policy as code, and implement security gating to block and sign artifacts. Monitor and measure. If you want to monitor your software components, generate the SBOM and integrate that into CI/CD pipeline. Of course, if you want to present this to your management, using metrics is something that can help you to identify the areas where you can improve, and also for reporting to your management. Finally, is the runtime protection.
Questions and Answers
Moisset: Let’s imagine that we have a pinned dependency that has a critical severity, how would you deal with that?
Emma Yuan Fang: If there is a critical vulnerability in the pinned version of your dependency, then you need to look into the vulnerability and identify whether there’s a critical vulnerability that would affect your components or other components. Then if there is, then you need to patch that particular version, patch those particular dependencies and pin the new version of those dependencies. Then enforce it in the CI/CD pipeline.
Participant 1: Some of the example policies that you listed that lay out checklists that you need to go through, do those evolve themselves over time or do companies need to seek out newer policies for newer threats that are coming up?
Emma Yuan Fang: First of all, you need to have baseline policies that are implemented as code. What I mentioned about policy is something that is put into the code that is enforced as part of your CI code. Your CI/CD pipeline is basically code-based. What you want to do is to enforce your policy about signing in that code. Is your question about if there is something new coming into the policy, do I need to change that?
Participant 1: Yes. These organizations that publish these policies, do they make a new set or are they just adapting the steps that you just need to follow and keep up with the same?
Participant 2: My question is regarding the audit capability. The verification of integrity of the packages that we are using is a very tedious process and more prone to errors. Is there any automated way to verify those integrities, especially for the third-party packages? How to do an audit for those third-party packages.
Emma Yuan Fang: Your question is about how to audit?
Participant 2: Yes, how to do an audit for the third-party packages that are downloaded as dependencies.
Emma Yuan Fang: What I meant about audit is to scan. In your package manager like npm, it comes with the capability to scan your dependencies. That is what I meant about audit.
Moisset: Let’s imagine in the CI/CD pipeline, we do have malicious code injection. SQL injection is the first one, the top one, if you’re familiar with OWASP. How would you go from there? How would you prevent that? Do you have any best practices?
Emma Yuan Fang: How do I prevent that? You need to block the malware thing coming into your CI build. I mentioned about the malware detection tools or scanning tools. That’s one of the ways to prevent this. Another thing is, how do we prevent the malicious code going into your pipeline? I’ve already mentioned the different entry points where the malicious code can come into your build pipelines. Monitoring, I think that’s one of the ways, if you’re talking about runtime. Use eBPF, which is Berkeley Packet Filter. That is something you can use to monitor if there are any malicious network connections with your runtime. That’s something that can be done in the runtime.
Participant 3: I’m thinking for CI/CD attacks, because normally CI/CD is a pipeline that is internal to the organization. Would you say that it’s true if there’s a CI/CD attack, it would normally involve insider knowledge or insider attack?
Emma Yuan Fang: One of the mistakes that a lot of teams are making is, CI is something that we build internally so we can trust it. You just don’t know how much human mistakes can be made just using that CI Runner. It could be that one of your people, I’m not saying that is malicious, but maybe unintentionally leaks a secret into the public repo, for example. Then that secret can be used by someone to get into your CI/CD pipeline and maybe launch a job that can push malicious code into your build process, into your application. That’s something that actually happens in an instant. It’s not something that is just made up, but this is a possible path that your CI/CD pipeline can be compromised.
Participant 4: I’ve got a question about runtime versus build time dependencies, because that is something that hit us some years ago when we first started doing dependency checks on some of our projects. When we first run it, everything went red and everything was terrible. We had a lot of vulnerable dependencies. Then when we were talking to the teams, they were saying, that is no problem because that is just build time. If you use a modern web framework like Angular or whatever it comes with, a zillion transient dependencies that have dependencies again, and then sooner or later, you will meet a vulnerable dependency, but it is not actually used. It doesn’t end in the product that you build. How do you deal with that differentiation between build time and runtime? Is that something that you feel that should be addressed by Zero Trust as well or how do you deal with that?
When you build a project. You have a lot of dependencies that are just there. If you have Angular, for example, it comes with a zillion dependencies, but in your product, you don’t use everything. You don’t use all those dependencies that come with Angular in your final product.
Emma Yuan Fang: Your question is about if you don’t use those dependencies in your final product, but it has been highlighted in your SBOM process, so how do you deal with this situation? My question to you is, does it actually affect your product? Does it actually affect your product in the higher environment? For example, your production environment. I know that some of the developers also tell me that, ok, this dependency vulnerability is only related to something that’s in the dev environment.
My question is, does it actually affect our application in the production environment? It wouldn’t be propagated into the higher environment, that’s the key thing. If it only affects the lower environment, my question to you is, does that lower environment also host sensitive information? Does it also host live production data? If it does, then you need to fix this, even if it’s not going to propagate into your higher environment. My baseline is to fix everything. I know that’s not always achievable for the project. You probably need to prioritize based on what the dependency is and based on what the environment is.
See more presentations with transcripts
