Transcript
Andra Lezza: My name’s Andra. We’ll talk about some of the technical aspects of securing AI assistants, tools that as we obviously see are becoming rapidly the nerve center of all operations within our companies. I’ll focus on how to protect the data that powers these systems. I’ll address everything from the initial ingestion of data and transformation, all the way to the deployment and continuous monitoring and security controls. I am a Principal AppSec Specialist at Sage. This has nothing to do with Sage’s official stance on AI, this is mostly based on my work with OWASP and my own observations.
We begin to encounter AI assistants everywhere, in more and more development workflows, chatbots, and so on, so securing the sensitive data throughout the pipeline becomes mission critical. You can think of it as a very simple solution. You have a copilot or an assistant, and it’s simply able to answer questions based on our data and whatever account we’re using. In reality, it’s way more complicated than that, and it is very difficult to keep things simple. Most of these copilots will end up reaching in backend systems or other products in order to be efficient and useful enough, and retrieve the information a user is asking for.
This is what we’ll talk about, we’ll start with a brief introduction and go through the importance of AI assistants, obviously, and the criticality of data security, so why do we care about this? We’ll then review the OWASP Top 10 LLMs versus web apps, see what the differences are and how they compare. We’ll go through a bit of the OWASP AI Exchange threat model. We’ll then go through a couple of copilot implementations, and for each of them, we’ll list threats and controls. Finally, we’ll have some key takeaways.
I do application security at Sage, but I’m also one of the three London chapter leads for OWASP. Those are two of my interests. I’m here to learn more about AI security. This is something that no one has cracked yet, so we’re all still learning. I’m also interested in threat modeling and how to make it as fast and efficient as possible. If you’ve ever talked to your security teams about threat modeling, you’ll know how difficult it is to get it right.
Introduction to Assistants
AI copilots are conversational AI assistants that augment human productivity in a given domain. They use natural language processing to understand queries and automate responses. They are very much reactive in nature. They’re very different between chatbots, so text-only interaction, and agents, so things that have the capability to perform actions on their own, autonomous actions, which are more proactive flows.
Then there’s also the traditional ML systems. Today we only focus on copilots. Getting an assistant right is very difficult, and some of the giants that you probably know of, like Microsoft and GitHub, which are actually innovators in this space, are also struggling. Their copilots have been in the media over the last few months because of a few problems. In the Microsoft Copilot, you have extensive permissions, and unless you actually configure it to be secure, it will broadcast your internal enterprise data to anyone on the internet. Then, on the GitHub side, there have been security concerns with regards to legal risks associated with copyright issues, privacy concerns, and of course, insecure code suggestions, for example, Copilot telling you to hardcode secrets in code.
Access to Critical Data
Let’s move on to critical data, and why do we actually care about this? AI copilots operate at the intersection of automation and decision support by processing highly sensitive data, such as intellectual property, financial systems and transactions, personal identifiable information, or PII, as you might hear about it. This data is not only the engine behind the systems, but it’s also major liability if it gets leaked. The core argument today is that securing every stage of this pipeline, so data ingestion, transformation and pre-processing, model training and validation, model deployment and integration, and in the end, monitoring and logging, is critical. The main risk factor here is actual persistent access to this data. Cross-domain knowledge and credentials. It’s essential to maintain trust, meet compliance requirements, as we usually do in most of our industries, and ultimately, protect the organization’s strategic assets.
OWASP Top 10 and The AI Exchange
Now let’s look at OWASP Top 10, and start with the LLMs versus web apps. Many of the traditional web app vulnerabilities, such as broken access control, misconfiguration, and data disclosure, still show up in an LLM environment. They’re nothing new. They do require a slight adaptation to new interfaces like prompts, plugin integrations, or model-specific data. Hence, many of the previously known principles. We had a panel and we spoke about doing the basics right, like least privilege, thorough input sanitization and validation, strict separation of duties, and auditing, also explicitly adapt to LLM pipelines, so that the same fundamental mistakes that we’ve seen in the last 30, 40 years before, are not repeated. All of this means that 85% of an AI system security is just traditional security done well.
Now let’s go through how they compare. First, LLM risk, prompt injection. It’s definitely not something new. It happens when a user prompt alters the way the LLM behaves, or its output. Then we have sensitive information that can affect both the LLM and its application context. It’s something we have seen before in the shape of either a sensitive data exposure, if you’re familiar with the previous version, the OWASP Top 10, and now it’s called cryptographic failures in web apps. Data is used during the entire supply chain, as we’ve seen, so data poisoning can happen when pre-training or fine-tuning, or when embeddings are manipulated to introduce vulnerabilities, backdoors, or biases. They’re both connected to vulnerable or outdated components, and obviously, software and data integrity failures.
Then we have improper output handling. This is all about insufficient validation, sanitization of the outputs generated by an LLM, before they are actually passed downstream to other systems or components. Successful exploitation here can result in cross-site scripting, CSRF, SSRF, all of these things that we’ve seen before. Privilege escalation or remote code execution on backend systems. Now excessive agency, which sounds new, but it is not really that new. It’s when an LLM-based app has the ability to call other functions or interface with other systems via extensions. We can think of these extensions as tools, skills, plugins, in order to take actions in response to a prompt. This one relates to multiple traditional top 10 issues, and it has a high impact on CIA: confidentiality, integrity, and availability for a system.
Now, actually, if we look at it, there are only three completely new types of risks associated with LLMs. First one is system prompt leakage. This is unique to LLMs, which is the risk of exposing internal instructions that guide a model’s behavior. System prompts are designed to guide the model’s output based on the requirements of the application it’s part of, and may contain secrets. Exposing these secrets can then be used to go on to other attacks. The next completely new risk is vector and embedding weaknesses. As the name says, it’s specific to LLM systems using RAG, or Retrieval-Augmented Generation, and embedding techniques. Then, lastly, misinformation. This new risk is intrinsic to generative models, as it relates to the potential for generating and propagating misleading information.
Has anyone seen the OWASP AI Exchange before? Here’s the OWASP Exchange AI threat model. If you want to read more about it, the link is up there, owaspai.org. I’m only going to focus on the three types of threats as they distinguish them. Threats during development time, so when data is obtained and prepared, and the model is trained. Through using the model, so providing the input, reading the output. Getting the job of the copilot done. By attacking the system during runtime, so in production. Based on this general overview, so you don’t have to go too much into details, we will go through a few more in-depth points related to specific copilot implementations that I’ll introduce.
Overview of Copilot Architecture Models
Let’s look at the two copilots that I’ve been mentioning. One of them, I’ve called it the independent copilot, is designed for a single domain. Think of it as an accounting application, for example, where it’s embedded directly in the user interface. It has deep custom logic tailored to specific actions in that application, like ledger management, transaction processing. Its data pipeline is internally managed and optimized for that one specific use case. Think of it as only one team, for example, one AI team, implementing copilot chats and services for one end product, one-to-one relation. This copilot has initial access to data through API calls for the rest of the product functionality, but this can turn out to be not enough.
In order to make the assistant more useful and more efficient, the AI team decides to introduce additional services like that AI-friendly layer based on the product data, so the local copilot, for example. However, as copilot, should not be given direct access to an entire data warehouse. There is the additional layer that’s built, like a mini custom API, that can access a controlled specific subset of data fed in by the product into copilot, so unidirectional. The chat function, as we know, has to connect to an LLM provider. Then there’s local storage for the chat history, which is also important to have a look at, and other product actionable intelligence. This type of copilot has a very tight integration with the end product, with a single domain application. It has custom logic for the extra copilot services layer. Its security implications are around simpler isolation, but deeper access to domain data.
The next one, in contrast, called an integrated copilot, is built as a multi-tenant system. It’s part of a suite of shared services that support multiple domain applications by providing a set of generic skills or job definitions. Here, the shared services generate instructions for data retrieval and processing. Each domain application, in turn, has its own user permissions. This creates a huge matrix of permissions and cross-domain supply chain that requires very tight isolation and access controls.
In this case, we have multiple teams that need a copilot, and they’re creating their own jobs and skills based on the use cases and the data that they understand in their own domain. Job definitions delegate tasks to individual applications, meaning that these skills and jobs are actually executed within the end product and not by the larger platform. Security implications here are all about complex isolation and standardized security controls. Slightly more work to do for the security team. I’m also going to talk about a slight distinction between the jobs and skills. A job is obviously a unit of work, and it uses skills in a sequence in order to achieve a desired outcome. For example, an entire conversation with a user, schedule jobs, background, and so on. A skill is the actual capability granted to copilot based on that one user’s permissions within the application. You can have API calls, content generation or export, or other actions triggered by events.
There are a few common risks. As one might expect, we have the authentication and authorization. They have to be done in both versions of copilot. For authentication, when we think of all API interactions, we can have challenges at each integration point. For authorization, we have things called On-Behalf-Of Scoped Tokens, so based on the user in that application, meaning the copilot can impersonate the user. Then there are different issues based on the type of architecture. In terms of data, we have one single source of data on the independent copilot, so domain-specific injection risks, more possible. We have a bit of an overreliance on domain-specific data, whereas for the integrated model, we have cross-tenant data leakage possible. Exploitation of shared resources, which means that a very complex permission matrix is required.
Then in terms of threat landscapes, so more general threats. Some of the threats around the independent model could be, it’s more predictable, but more deeply exploitable. It has a lower complexity, but more targeted attacks are possible. Whereas for the integrated copilot, the threats could be more complex and multi-vector attack possibilities. Please note that there is no inherent more secure architecture. Security is all about the actual implementation of these controls, and not just the architectural choice. There’s a need for continuous monitoring and adaptive controls.
Threats and Controls
Now let’s go into the specifics. For threats and controls, we will go through the two copilot threat models, so the diagrams you’ve seen, but slightly changed. We’ll cover controls, and for each identified threat, best practices. In order to showcase defense in depth and what it means for each threat, we might go through multiple controls. For the independent copilot, here’s the reorganized diagram. If you’ve ever done threat modeling slightly within your roles, you might understand the concept of trust zones or trust boundaries. This is where the level of trust changes for data or code as it passes through the system. A few attack goals for this copilot would be to get customer information from the end product, get access to critical infrastructure for the entire system, maybe cause a denial of service or gain access to the services hosting LLMs, think OpenAI.
If we remember the OWASP AI Exchange, that very complicated diagram that we saw from OWASP, here we’re mostly concerned with development time threats given the functionality of this particular copilot, and runtime threats, so using it in production. The first threat that I thought of would be information disclosure. This is going to be common for the two architectures. An attacker here might try to retrieve financial information stored in a user’s message history, so chat histories, or obtain through the skills APIs that the copilot can access. The first flow, when it comes to information disclosure, is here. One vulnerability that can lead to info disclosure is an issue with how caching is done.
Obviously, when a user wants actionable intelligence based on their data, they would want that really quickly as a copilot response, rather than wait for a batch job to complete every 12 hours. Caching is done for copilot services looking into the subsets of data when a user needs quick understanding of their own information. This is a flow of how data was cached for retrieval of data-driven observations. Multiple data points were actually cached because of a human mistake, using the same token, which allowed certain users to view cached analytical findings containing sensitive information from other customers or tenants.
As you can see, in an assistant like this one, it’s difficult to enforce proper access control. One tiny mistake can lead to data disclosure. This is the second flow that we can have a look at, chat history. This is also vulnerable. Another information disclosure possibility is data access through calls to other product endpoints, through creating those findings based on the database supporting the product, storing the data findings, and finally, log exposure. We are looking at continuous logging and monitoring with a system like this to make sure that the answers it gives are accurate, actionable, and make sense to the user. We could have sensitive data mistakenly recorded in various applications for logging within the Copilot environment. Here they all are put together under the umbrella of information disclosure at each point.
If we think of some controls that can be applied for this particular threat, we can think of authorization and permissioning, tenant segregation, and third-party APIs. For example, Amazon offers tooling to scrub sensitive data from logs. For authorization, there are these things called On-Behalf-Of Scoped Tokens, which are based on the user credentials or on the user token. You can have session context maintenance and validation. Only keep the chat history for that one session.
Then another layer with explicit consent tracking for sensitive operations. If the user wants to access data that they shouldn’t have access to, then you log that. The payload is very simplistic code here, but it basically grants the user 123 with a finance manager role privileges, with the ability to read and write financial data and impersonate the user. Here’s another way to think of authorization when it comes to your copilot. A least privilege best practice for configuring authorization would be derived from a user’s token, so token and permissions for the user, what the AI copilot can do. Third, what’s the scope for each job and skill?
Then from all of these three, you take the common denominator in terms of least privilege. That’s the permission that your copilot should have. We can also speak about ACLs or RBAC. If you’ve ever configured any kind of infrastructure code, then you might have seen them before. There are slight differences between ACLs and RBAC or ABAC, role-based access control or attribute-based access control. ACLs, so access control lists are the simplest form of access control. They directly associate permissions with specific users or groups for individual resources. Permissions are managed at the resource level. Versus something like RBAC, which introduces a layer of abstraction through roles. Users are assigned roles and then permissions are assigned to roles rather than directly to users. Here’s an example of an ACL.
Again, very simple type of code. It could look like this. This is for the independent copilot in an accounting domain. In this scenario, the ACL enforces that only the users with the appropriate roles and permissions can access the financial sensitive information. Then, here’s the RBAC example, very similar, where a finance manager might have both read and write permissions while the accountant might only be able to see the data.
Let’s move on to supply chain attacks. There are different types of supply chain that we can think of. Some target tools in the CI/CD pipeline. Some target the actual data used in an LLM-based app. Others are around the tools that an AI copilot can access, and can potentially use maliciously if it’s not configured, a Microsoft Copilot. Here we can think of databases, RAG, IDs, and so on. The last point, the tooling for copilot, could be more apparent in things like agents that can perform autonomous tasks. As mentioned, they are not really the focus for today.
That part of the motivation there is because in industries subject to very strict oversight and stringent compliance controls, such as finance, health, defense, the mapping of traditional controls to AI security best practices is very much playing catch-up, hence the picture of the dragon. An OWASP friend referred to this area as there be dragons, because we still don’t know. The appetite for GenAI apps that can do whatever they want without human oversight is extremely low in frameworks like SOCs or PCI. Here’s the enriched diagram of the threat model. I’ve added the connection to a model registry such as Hugging Face.
For this one supply chain attack flow, an attacker can upload malicious models to Hugging Face, for example, and could gain arbitrary code execution in the final product environments if the product preventative measures aren’t in place or are just not working properly. A notable mitigation here, which you might have seen if you’re working with AI and ML systems, is something called safetensors. It’s one of the primary formats designed to store large arrays of numbers, tensors that are then used in machine learning. It’s designed with very strict limitations on the file content and behavior. It prevents remote code execution from maliciously serialized files.
Then another example here, so the second flow is data poisoning. An attacker gains access to a production database and deliberately injects malicious or misleading data to the system’s data sources, because the independent copilot might draw on various inputs, so it can have unauthenticated external resources or dynamic internal data. If an attacker introduces compromised information into these sources, obviously an assistant may generate inaccurate, unexpected, or even harmful outputs. In terms of controls, if an application sources AI or ML models from external repositories, such as Hugging Face, a set of pipeline controls can be implemented, so you have all the traditional best practices in place. They check performance, ensuring the models function as they should. They verify the model hashes to make sure that the model hasn’t been altered.
Then another thing that you can do is model behavior testing. This is also called AI red teaming. It’s not just looking for security issues in an AI system, but it also looks to probe for responsible AI harms. It tries to identify potential risks and harmful outcomes for the user. These are two examples of the independent copilot being tested. One of them is all about inference. You give it two halves of the question: one of them makes sense for the application you’re in, and the other one tries to do something malicious. The second example is all about different answer, from Romanian to English. It shouldn’t be that since it’s based on the same data. Here’s a couple more examples. One of them is obviously a hallucination, and the other one is a pirate talk. One solution to not end up in situations like this is to try to lock the assistant down to a very small set of skills or things that it has knowledge of and can answer questions on.
Another very complicated diagram. This is a very high-level overview of a secure AI pipeline with the five key stages that we saw before, from data ingestion all the way to monitoring and logging. At each stage, we can have specific security measures that we can apply. Let’s start with data ingestion. We can have encryption, chain of custody. Digital signatures and automated compliance checks during data transformation. Adversarial training and differential privacy during the model training itself. Secure CI/CD pipelines with SBOM generation, Emma talked about SBOM quite a lot, and data provenance checks during the deployment of the model. Finally, real-time auditing with continuous monitoring during the monitoring phase.
Then, just to put all of this together, there’s a small feedback loop going from monitoring back to training, which ensures continuous improvement and reinforcing end-to-end integrity for the entire supply chain. Then, in this corner here, we have the DevSecOps image. This is all part of the CI/CD pipeline that we have working in our teams anyway, and all the best practices still apply, like pre-merging, we have SAST, DAST, all the scans with tools like CodeQL and Semgrep and Snyk. We have post-merge checks, like generating the SBOMs, signing containers with Sigstore, for example. Then, during the deployment phase, best practices include immutable artifacts and ephemeral training environments.
Next threat is prompt injection. It has to be there, because it hasn’t been solved, and it’s very difficult to solve. Copilot’s dynamic chat functionality can be vulnerable to prompt injection, as all copilots and assistants can, even with data segregation for each user. A malicious user could discover details about the prompt and how the data is being sent back to the LLM in order to be processed, and can potentially gain access to additional resources to run further prompts. I’m going to talk about some controls, but as I said, prompt injection is not fixed. I think everybody is trying to figure it out. Here are some examples. We’ll talk about the first two, so prompt engineering and guardrails.
Then, for templates, we’ll talk about templates more during the integrated model. Prompt engineering works hand-in-hand with guardrails. As part of prompt engineering, we can have tailored prompts for domain-specific tasks. For example, when generating a financial report or a ledger summary, the prompt must include specific context, such as accounting rules, data formats, and KPIs as the user needs, which would ensure that the model generates outputs which are accurate and relevant. You can then incorporate safety guardrails, so system may validate the prompts before processing, to use another LLM to make sure that the prompt is all about the application or not.
Then enforce additional constraints on the outputs. That’s what I meant with guardrails on both input and output, ensuring that both inputs and generated responses adhere to strict security controls. Then, on top of this, we can continuously improve. That’s where you have the feedback loop. You analyze outputs and refine the prompt so you can enhance the performance of copilot, make it faster, make it more efficient. This might include adjusting the level of detail, modifying instructions, or specifying examples within the prompt to guide the model’s behavior.
Here’s a very tiny snippet of Python code which looks at implementing guardrails for Azure OpenAPI. Both Azure OpenAPI and AWS Bedrock have their own sets of guardrails that can be used out of the box. We can do the same for the output, just like we’re doing here for the input, just sending it to OpenAI, and checking that the data returned from the other API calls doesn’t contain any toxic language. There are actually different types of guardrails that you can think of. You can have one that checks output for hallucinations. You can have one against prompt leakage, one for checking the sizes of files, if file upload is a functionality that your copilot has, and the one that we all think of around filtering toxic language. If you want to learn more about prompt injection, that’s why the link is there. You should go to Lakera’s Gandalf, it’s pretty cool. It’s an application that’s intended to be like a prompt injection game. They’re using various types of guardrails, and you have to get Gandalf to give you the password. It’s quite fun.
That was the independent copilot. Now let’s move on to the integrated one. Here’s the reorganized diagram, same with the other one. This one has its own trust boundaries as well. We’ll go through a couple of flows and their associated threats and controls. When we think of attacker goals for this case, very similar, obtaining sensitive information from the product customers, or pivoting to other environments or products within that shared services layer, as well as trying to misuse LLM services on behalf of the application.
The first one, information disclosure. This shows up no matter what with copilots because it’s all about the data they use. A threat actor with access to one product may try to exploit the suite of shared services’ internal infrastructure to obtain sensitive information from other users in different interconnected products, as well as try to exploit the product’s own authorization guardrails to gain access to the underlying database.
If we think of controls, the thing that saves us quite a lot here is all about templates, so templates for how this copilot works and answers questions. Ensure that each user is only able to share a specific subset of their own information with the backend LLM, and that responses from the shared services suites are then executed in the user’s own application with their own session. Also, the LLM’s visibility into the requesting user’s data is limited as well, and it remains unaware of other users or other tenants. We have other standard security measures as well. We can think of ACLs, RBAC, and authorization in general.
Here’s the image of authorization yet again, trying to figure out what kind of permissions to give to this copilot. Next threat is all about pivoting or lateral movement. This copilot is based on a platform of products. An attacker might try to use prompt injection or any other type of technique to expand their reach into other functionalities and environments, taking advantage of the LLM’s permissions across the rest of the service suite. Controls, very similar as before. Templates strictly confine the available actions.
Then we have these templates for jobs and skills. These are the interactions that operate exclusively within the context of the requesting end product user. Each product uses the template to define jobs, and then these skills to provide the required information. The LLM itself will only receive a prompt that’s generated by these templates rather than a direct prompt from the user. Think of it as, prompt is coming in, you’re checking it with the guardrails. You’re sending it to a skill to understand if it’s something that it can do as an action.
Then it creates the definition for the action, gets the data from all the other services that it has access to. Then sends the job to be run within the end product itself. There is no direct human interaction with the LLM. A malicious or a compromised user might also attempt to do a prompt injection on this copilot as well, and gain access to backend LLMs. Then, controls, very similar. Templates, once again, save us by limiting what data can be shared with the LLM. I think another thing that I wanted to mention here was a threat around the attacker trying to deplete the tokens allocated for the LLM usage, so trying to cause financial damage for this particular application, by flooding the shared services with excessive requests. Obviously, one mitigation here would be rate limiting within both the product and the larger platform itself.
Key Takeaways
Then some key takeaways. End-to-end security is critical. We’ve known about most of these principles and best practices for quite a while. Securing AI copilots is not all new. It requires a comprehensive end-to-end approach for our supply chain, and each stage of the actual data pipeline, since all of these apps are based on our data. It’s essential to maintain data integrity and protect our highly sensitive information, so our crown jewels. Next one is all about architectural tradeoffs, and these leading tailored security controls. You still have to apply all the security controls that you would in a normal environment. Independent copilots offer deep, domain-specific integration with simpler pipelines, while integrated copilots can involve multi-tenant, cross-domain challenges. Each architecture requires customized security strategies, so understand the use case in order to mitigate unique vulnerabilities when they arise.
The last one, granular access control and least privilege. I think before, I wanted to say zero trust, but there was an entire talk about zero trust. Basically, look at your authorization model and make it as granular as possible for a copilot. Even with all this, security is actually never done. Apologies for any developers out there. AI evolves, new threats emerge. We’re all learning about this. Adversarial techniques, deeper multi-tenant complexities, as well as regulatory changes require ongoing ways to adapt. My call to action to you is embrace defense in depth. Please don’t run away from your security team. Do continuous monitoring, and have a proactive incident response.
Questions and Answers
Participant 1: This is a specific problem to us that we’re thinking about, but how worried should I be when I’m using an LLM system in which I may have a templated prompt, which is fixed, even, apart from one thing. That thing that I’m ingesting that the LLM is querying, you say, for something like a user-supplied PDF, because I think this is quite a common use case. We get some fuzzy documents or blob of data that has been supplied by end user. We’re feeding it into our LLMs to try and make sense of what’s in that data. What risks and approaches might you see there, in terms of, can it escape from the prompt? Has anyone done any research on that?
Andra Lezza: I would still just try to have a look at the authorization. You’re using a templated copilot. Technically, it’s only supposed to do one thing, but you’re still sending data. Then you have to look on the back of the copilot itself, what kind of systems are getting that prompt? Where’s your data going, that blob of data? Is there any way to split it? Is there any way to sanitize it and not just broadcast it to all the other services that have access to this?
Moisset: What is your favorite from the new OWASP Top 10 LLMs?
Andra Lezza: I like the excessive agency one. I think it used to be called overreliance. It’s more about the human nature of, we’re talking to these chatbots and these copilots and they can do stuff for us. It’s so cool now, we just ask them to do everything. There was a talk about white coding. Yes, they can code for us. They can do so many things, so why not rely on what they do? We have to stop being, I’m not going to say lazy, because it’s not laziness, it’s just the hype around it. We just have to check what they do.
Moisset: If you don’t visualize what is excessive agency, let’s say that you do have an agent plugged into your inbox in your email. The task is to summarize all the emails you’re getting from a day. That’s cool, because it’s getting you a little bit more productive. Let’s imagine that actually this agent gets compromised, so now it has access to all of your inbox and sensitive info. This is excessive agency. This would be because it’s not only read access, it’s also write access potentially. That’s just to give you an overview of it.
Participant 2: Do you foresee security becoming an agent, so security done by AI agents? Like you have AI agents, you develop AI agents, and then you’ve got on the other side, security actually done by the AI agents.
Andra Lezza: What could possibly go wrong? No, probably not, not just yet.
Moisset: We do have some security tools that are also backed up by AI models. Somehow there’s a little bit of AI just in it, but not fully automated.
Participant 3: Just thinking about assessing and measuring risk presented by vulnerabilities in AI, is CVSS up to that? Or if not, are there better alternatives for doing that sort of assessment and prioritization?
Andra Lezza: I don’t think the whole landscape is so mature yet to have these kinds of mappings. I think people, especially in OWASP, are working on it. Typical security answer, you found a problem, now fix it.
Participant 4: Have you faced any challenges in your organization with implementing this or gaining traction on this? Or, in other words, how mature are you in your organization with threat modeling AI or having an AI framework, or just governing this? Where are you on the journey when it comes to AI security, basically?
Andra Lezza: Generally, all the companies are now implementing something that has to do with AI, because it’s high visibility, everyone wants it, everyone wants their tools to include it. It’s such a visible project within any company that people want to get it right. In my experience, we can do all of these checks and we handle all of the scans, because these risks go all the way up to the C level of the board. Yes, I’m working on threat modeling with a lot of my teams that build AI products.
Participant 5: You mentioned that one of the issues with AI agents is gaining insight into what they’re doing, so what they actually access now. Like this issue with caching you mentioned earlier, how would you know that this has actually happened? How do you know that that’s actually occurred? One of the things I noticed is that many companies will now, instead of developing their own agents or LLMs, they just now buy something off the shelf. Those things tend to be black boxes. There’s very little insight into what exactly they access, how they operate, and so on. Have you had any experience with gaining knowledge of what those off-the-shelf things actually do at your company? You just bought this Excel copilot, and now it’s got access to all your spreadsheets that you have at your company. Is it safe?
Andra Lezza: I actually haven’t worked with something like that. Most of the teams I work with build everything from scratch. It depends what kind of company you’re in. If it’s an industry where you have to do that and you have a more stringent oversight of the tools you’re using off the shelf. That would probably just take extensive testing, a lot of testing, to understand if they do anything dodgy that you don’t expect.
See more presentations with transcripts
