GenAI Security: Defending Against Deepfakes and Automated Social Engineering

Transcript

Wes Reisz: On today’s episode of the InfoQ Podcast, we’re diving into one of the more urgent challenges faced on the internet today: trust. Specifically, we’ll discuss the rise of deepfakes and disinformation, and the erosion of digital trust in the age of GenAI.

Hi, my name is Wes Reisz. I’m the creator of the InfoQ Podcast, though these days I play more of a guest-host role on the show. In my day job, I work as a technical principal at Thoughtworks, building engineering teams, platforms, and delivering solutions built with and using AI. I’m also the chair for the upcoming software conference, QCon AI, which is a QCon focused specifically on helping software engineers and technical leaders build and ship with AI responsibly. Today’s guest, Shuman Ghosemajumder, is one of the three keynote speakers at QCon AI this December. Shuman is the co-founder and CEO of Reken, a stealth AI company building products to protect against generative AI threats.

Shuman has spent his career at the intersection of trust, safety, and scale. He was part of launching Gmail and founding Google’s trust and safety product group. He served as a CTO at Shape Security, which was acquired by F5 where he went on to be the GM of AI. Today, Shuman leads efforts to defend online integrity against AI threats. In today’s conversation, we’ll talk to Shuman about QCon AI’s keynotes, deepfakes, distant information, and AI content taking over the internet. We’ll explore how GenAI is creating both opportunity and risk, how attackers are exploiting GenAI, and most importantly, for us at least, what engineers can do to build systems that don’t just trust by default. Shuman, welcome to the show.

Shuman Ghosemajumder: Great to be here, Wes. Thanks for having me.

Wes Reisz: As I mentioned in the intro, you’ve spent your career at that intersection of trust, safety, and scale, from launching Gmail and Google’s trust and safety product group to leading AI security at F5 and now at Reken. What first drew you to this field of security and defending trust in online systems?

The evolution from physical crime to exponential cyber attacks [02:38]

Shuman Ghosemajumder: Well, early on at Google, when I was working on advertising products, I was one of the early product managers for AdSense, which was our contextual advertising product. I realized that we had so much influence on how people consumed content online and how, frankly, societies would even behave because of the importance of Google, that we had a real duty to try and do everything that we could to protect society. So that meant certain things from a security perspective, from a fraud perspective, from a privacy perspective. So, I got involved in all of those areas, started the trust and safety product group, and that included initiatives around advertising, fraud protection, as well as privacy and data policy across the entire company. That was really the beginning of my career in this area at the intersection of both AI and cybersecurity. So, cybersecurity became a really big deal about 10, 15 years ago for the industry as a whole.

Back in the early 2000s, there wasn’t as much attention that people paid to cybersecurity. It was one of those things that was important to certain groups, but it wasn’t yet a societal-level issue. It wasn’t something that CEOs and boards would talk about. All of that changed about 10, 15 years ago when we started to see a bunch of very large data breaches that affected well-known companies. So at Shape Security, we were developing basically the next iteration of some of the things that we were focused on at Google, and protecting against cybercriminals who themselves were simulating human behavior. So in the case of Google, what we saw was cybercriminals who were engaged in their own form of AI, and that they were simulating human behavior to create false clicks. If they could make their clicks look like they were coming from real humans, then they could create click fraud successfully in their minds.

Of course, we developed some of the largest machine learning models in the world to try, and detect and protect against that. At Shape, we took that into a new domain where cybercriminals were simulating human behavior to take stolen usernames and passwords, and put them into login forms on the big banks, airlines, and federal agencies, and basically all of the big companies that serve millions of users around the world. We were once again using machine learning to be able to analyze all of those patterns, look for anomalies, and figure out ways to be able to stay ahead of what the cybercriminals were doing. So, this is a problem that is never going to go away. There’s always going to be crime.

Crime today is basically cybercrime. Cybercrime is a much bigger deal from a dollar-figure perspective and an impact perspective than physical robberies or burglaries are. So as technologies evolve, as the attack surface, as we refer to it, and cybersecurity evolves, there are always going to be new challenges to solve, and AI is now on both sides of it.

Wes Reisz: When you talk about protecting at scale, and you’re talking about F5, and you’re talking about Google and the amount of bots that we’re hitting, how does someone who’s listening to this get their mind around the type of scale that you were talking about?

Cyber threats at scale: moving beyond physical intuition [06:13]

Shuman Ghosemajumder: People have intuition for the physical world, and this is one of the things that Nassim Taleb talks about in Black Swan: that you have a sense of how big or small certain things can be. So, there’s no such thing as a 3,000-pound human or a 20-foot-tall human, but there is such a thing as a cybercriminal being able to attack a million people at once in a way that a human would never be able to attack people with a bow and arrow or with a knife or something like that. Now, the scale of those attacks can change when we have more powerful physical weapons like nuclear weapons, for instance. But when we’re talking about cyber weapons, there’s really no limit to the amount of scale that a cybercriminal can achieve.

You can actually have a cybercriminal attacking billions of people simultaneously, and that’s just not something that we have intuition for. So, it’s really hard for folks outside of cybersecurity, and especially outside of the technology industry, to even comprehend how the attack functions, let alone how the defense for that would function. It really all comes down to being able to scale whatever your approach is based on large scale analysis, which in the modern era absolutely requires machine learning and other forms of AI.

Wes Reisz: We’ll get deeper into that here in a bit. It’s like, “How do you even think about the type of scale that you’re trying to deal with when you’re at both Google and F5?” So, I think that’s an interesting analogy to be able to… I can think about protecting my home from one person or a group of people, but how do you protect a billion homes? The scale is just so huge. It’s an interesting way of thinking about it.

Shuman Ghosemajumder: It also changes the incentives for an attack.

Wes Reisz: Oh, yes.

Shuman Ghosemajumder: So if you’re someone that wants to go and mug someone, or go and rob someone, you have a choice that you have to make first, which is, “Who am I going to rob?” So, what’s the risk and reward that’s associated with picking my particular victim? Because of that, people think about defense in a similar kind of way. They think, “What can I do to prevent myself from being victimized? What’s so special about me that a certain type of criminal would want to target me, and how can I diminish that in some way?” But that’s not the way that the incentives work when you’re dealing at exponential scale. Now, the cybercriminals can basically attack simultaneously, and there doesn’t have to be anything special about them, and there doesn’t have to be any choice that the cybercriminals have to make.

Wes Reisz: If you’re defending, you got to be right every time. If you’re attacking, you just have to occasionally get lucky.

Shuman Ghosemajumder: That’s an interesting metaphor, and I think that the traditional trope in cybersecurity has been exactly that when you’re talking about breaches and when you’re talking about exploiting vulnerabilities, that you just, as a cybercriminal, need to find one way in, one mistake that the defender has made, and now all of a sudden, you’ve been successful, and the defender has failed. But what’s interesting is that when we’re talking about other forms of attacks, especially things in the fraud space, that can sometimes get inverted. Because if you are a cybercriminal and you’re trying to make a million login attempts look like they’re organic, that they’re coming from a million real users, then you need to simulate a million real users in such a way that you don’t get caught. Because as defenders, if we can figure out what that pattern or anomaly is, suddenly, we can discover a million fraudulent transactions because the cybercriminal slipped up in just one dimension.

Wes Reisz: That’s interesting. So, I mentioned at the very onset when I was introducing you that you’re doing a keynote at QCon AI. That keynote is deepfakes, disinformation, and AI content are taking over the internet. What’s the big idea? What’s the big takeaway that you’re hoping folks will walk away from that talk with?

The proliferation of deepfakes and AI-generated content [10:14]

Shuman Ghosemajumder: I think that when people get content through the internet, they look for content that’s interesting. They look for content that’s high quality. They look for content that’s relevant, and there’s so much content that’s on the internet. It’s difficult to be able to find precisely what you’re looking for. This is a problem that has existed for decades now, which is why search engines have been so important and successful, and why LLMs and chatbots are now being so successful, because users want to be able to ask questions and then find relevant answers. There’s typically thousands of different pieces of content that could potentially be relevant, and they don’t have time to wade through all of that.

The problem is that people that create content and who want to monetize content have been trying for decades to fool others into coming to their content, and believing their content, and thinking that their content is high quality because they can literally make money off of that if they can drive traffic that way, so search engine manipulation, and now we’re seeing chatbot manipulation as well, people asking, “How do I do essentially search engine optimization or SEO for chatbots?” Part of this problem is that there’s a great deal of fake and fraudulent content that’s also being created. There’s some research that was done a few years ago at MIT by a professor named Sinan Aral where he found that lies spread six times faster on social media than the truth does. That makes sense when you think about it, because if you’re the person who’s trying to propagate a lie, then of course you are going to be trying to spread it as far and wide as you possibly can.

But if you are the recipient of one of those lies, and you recognize it as a lie, your reaction is often to be incensed by it, and then share it with a bunch of people saying, “Look at this outrageous lie”, but then you are part of the problem as well. You’re helping that lie spread. So, this is one of the reasons that social media has turned into a rage circus, where it’s the things that make people angriest that end up getting the most traffic. When we’re talking about fake content, that’s actually some of the content that can be the most interesting and create the greatest emotional response, because it’s going to be more outrageous. It’s going to be more inherently interesting than real content, because it doesn’t have to stick to the truth.

If you look at TikTok feeds now, or you look at something like YouTube shorts and the default feeds that come up, we’ve done some analysis that’s shown that somewhere between 10 to as high as 30% of those feeds may be AI generated already. So, this isn’t even a far off problem where we’re worried about consuming tons of AI-generated content and fake information. We’re consuming it already.

Wes Reisz: Just to prep for this, I wanted to actually experiment with some of the things that are out there today on creating deepfakes. So, I took some videos of me uploaded into a tool. Did not use my own voice, because I thought that was giving too much of an easy model, but used a different voice, and literally just fed a script and in space of, I don’t know, 35 minutes, maybe less, created a deepfake of me from just a video of some examples with my mannerisms that look pretty accurate. So, I work in the industry, and I’m floored by how easy it is to make a deepfake these days.

Shuman Ghosemajumder: I’m going to be showing a couple of clips in the keynote of deepfakes that we made in 60 seconds. So, there’s enough public content out there, and you have given talks in so many different public contexts that your voice is already on the internet. It is tied to your name. So, all of this has been pulled into those models. If they’re not using your voice, that’s actually extra effort for them. The reason that they might not show your actual voice is because they don’t want to give the game away and show how much copyrighted content is actually used in the model, but rest assured, they have your voice tagged to your face, tagged to your name.

Wes Reisz: Yes, I was floored about what I was able to do as a total amateur doing the things. I can’t imagine what the bad actors are doing these days.

Shuman Ghosemajumder: Well, the amazing thing is that even amateurs have powerful tools now that you can go to Grok for instance, and upload a single frame or a single image, and it’ll turn that into a video with the correct voice. That’s remarkable, that now you don’t even need to do any kind of training or upload sample videos. All you need is a single image, and it can interpolate and extrapolate the rest.

Wes Reisz: That’s terrifying. So, it’s going to be an interesting keynote with a lot of good examples. I can’t wait to see it.

Shuman Ghosemajumder: I hope so.

Wes Reisz: Let’s start looking at this explosion of AI-generated content, and start talking about, I guess, this risk that we just mentioned and the opportunity that exists out there. So, one of the things that you’ve said in the past is that GenAI is the first tech that pretends to be AGI. What do you mean by it pretends to be AGI?

The “Gell-Mann Amnesia” effect and the illusion of AGI [15:42]

Shuman Ghosemajumder: I’m specifically talking about LLMs in this context. So when we have the structure of prompting a large language model to give us the answer to a question, the way that LLMs work is they’re predictive text models, and they’re trying to figure out what is the most statistically likely answer to the particular prompt that you’ve entered. As a result of that, they tend to answer with supreme confidence whatever your question is. So, a lot of the time, they get the answer right. Almost all of the time, they produce an answer that sounds very articulate, that sounds justified, and it’s presented in a very confident way. So, when the answer actually contains a hallucination or an error, that’s really difficult to identify unless you yourself are the subject matter expert on that particular prompt.

Now, think about the way that people use search engines or LLMs. Why would you ask a question where you already know every single detail of the answer? You generally ask questions where you’re trying to learn something, where you’re not the expert, and you’d like to know what other people on the internet or what expert resources would say the answer actually is. So, there’s this phenomenon called the Gell-Mann Amnesia effect. It’s named after Nobel laureate Murray Gell-Mann. The idea is when he would read an article in the newspaper about physics in which he was an expert, then he would rant and rave about all of the mistakes that the reporter would make, and say that the quality of journalism here is extremely low and everything is wrong. Then he would turn the page, and read something in finance or the sports section or in politics, and assume that that reporter had done all of their research and had all of the details right, because now he’s reading about something where he wasn’t an expert.

That’s the way we all consume information. So when we hear something or see something where we are an expert, we can pick it apart, and we can say, “Here are all the little things that they got wrong. Actually, this is the right way that that question should have been answered”. But as soon as we step outside of that area of expertise, if we talk to anyone that knows just 5% more about a subject than we do, then they seem like they’re very well-informed about that subject, and we naturally defer to them. So, that’s the problem when LLMs hallucinate. They’re essentially giving us answers as though they always know the answers, which is what we would expect AGI to be able to do. Except the problem is that the LLM doesn’t know when it’s hallucinating.

Wes Reisz: Exactly. It’s not… I don’t even like the term hallucination because hallucination, it uses vocabulary that’s imagining. It’s creating this thing that didn’t exist, but the LLM is a mathematical system that is producing a mathematical result. It’s giving you an answer that it’s trained to do. It makes total sense for what it is trained to do. I’ve got some teams right now that are using supervised approach to building software. What I’m finding is that with this approach, it requires more senior people so that as you build AI-generated code, you’re able to actually see where that mathematical formula has just gone off rails and needs to be pulled back to what an experienced developer or a senior developer has. So, what you say makes total total sense then.

Managing code quality and the risks of AI-generated code [19:23]

Shuman Ghosemajumder: I’ve seen that senior developers can also be split on this. So, there are some senior developers that are extremely excited by the idea of using AI agents and having autonomous coding happening behind the scenes. There are real limits to what you can build using autonomous techniques. Basically, what it comes down to is there is a rate at which you yourself can inspect code in order to make sure that it’s correct. So if you are producing 10 times as much code because of the fact that you’ve got these autonomous agents that are working for you, you are going to give each bit of code one-tenth the amount of attention that you might’ve otherwise. So, that has implications in terms of code quality, in terms of what kind of defects we can expect to see, and there’s no stopping the fact that AI is getting integrated into so many different development processes.

So, my prediction is that we are going to see a lot of organizations learn very quickly how to get good at this, how to be able to put the right kinds of controls in place so that they can produce code, and then verify it in a fashion that allows them to be able to rely on it, and that’ll improve their productivity, but we’ll also have other organizations that never produced good code to begin with. There’s lots of organizations producing lots of very bad code, and those are going to be the organizations that are going to most drink the Kool-Aid of this approach, and they’re going to produce 10 times as much code that’s going to be 100 times as bad.

Wes Reisz: I think that you bring up a really valid point. It’s why my teams in particular right now, we take a supervised approach, and we use spectrum of developments. We are taking specs, building those down as individual tasks, using our traditional sensible defaults, pairing with the things that are produced so that we can observe and understand what it’s doing. Because like you described, the cognitive load of just reviewing the amount of code that can be produced can be pretty high. So, we take this approach so that we’re still putting these guardrails in place. The traditional software engineering, there are things that are more autonomous, but for what we’re delivering for customers, we’re putting these in place to try to really be intentional about what we’re doing because of exactly those type of risks.

Shuman Ghosemajumder: It’s a skill set. We’ve never had a tool like prompting an LLM before. So, one of the things that is a challenge for anyone that’s trying to get high quality output out of an LLM for any kind of purpose is the fact that you can’t have complete control over what you get back. So, you enter one prompt, and you get something that looks like it’s in the right ballpark, but then you discover that there are a few mistakes that it’s made, or maybe there are some assumptions that it’s made that are incorrect. So then you refine the prompt. What you would hope is that as a result of refining the prompt, it keeps all of the things that were right, and fixes the things that were wrong, but in fact, it generates something brand new that might have discarded some of the things that were right, and introduce some new things that are wrong.

Wes Reisz: Exactly.

Shuman Ghosemajumder: So, you have to go back and forth with it. That doesn’t mean that you can’t improve productivity, but it does mean that you have to figure out this new mode of operating with it.

Wes Reisz: Absolutely. It’s that doom loop you can so easily get into. That’s why I like that spec-driven development, because you go back to the spec, and then you build from that. You’re not just iterating down this doom loop. When we were talking before, you mentioned that all of our expectations are shaped by science fiction about how we name things, how things are used. It’s like science fiction is one of the biggest problems with how we’re thinking about LLMS. Explain a little bit more about that. What does that mean that our expectations are shaped by science fiction?

Shuman Ghosemajumder: So, we’ve been talking about AI in fiction for many decades. So, one of the things that’s often surprising to folks is that the term robot is barely 100 years old. People didn’t talk about robots 150 years ago. We have a century approximately now of fiction that talks about robots in various forms and artificial intelligence after the 1950s, when that term was coined. People have an intuitive sense of what the science fiction is warning us about after watching so many TV shows and movies and reading books, and now all of a sudden, we have the first time that the industry as a whole from a technology perspective has embraced the term AI. Prior to the launch of ChatGPT, those of us who are working in AI, we’re very specific about the terms that we use. When are we using a supervised learning model? When are we using a deep learning model and so on?

If you just called something AI, the assumption was that you didn’t have that level of precision or technical depth to be able to really discuss what you’re doing. Yet, now, everyone uses AI as the term for every single technical mechanism that they’re building. That’s resulted in a collision between society’s expectations and what the technology industry is actually building, because now, you’ve got a whole bunch of companies that call themselves AI companies, and that means different things to different people. You really see this illustrated when you get into conversations around AGI, which by the way is not a term that people use outside the technology industry. Outside tech, they just call it AI. But as soon as you step into tech, people say, “Oh, well, we’re an AI company, but we’re not necessarily an AGI company”, or you talk to OpenAI, or you talk to Anthropic or Google and so on.

They might say that, “Yes, we have a number of different AI products that we’ve created, and AI initiatives, but AGI is a separate category”. But for the regular public, they’re the same thing. In fact, by not using the term AGI, what they think AI is is AGI. So, when you talk to ChatGPT, or you talk to Gemini, or you talk to Claude, it understands the nuances of your language. It speaks back with very articulate thoughts, and that really seems like AGI or what the rest of the world calls AI. So, people believe that we have essentially achieved something that we haven’t achieved from a technology perspective. It’s all really kind of a magic trick in terms of getting people to be fooled by how powerful the technology is. So, there’s a lot of misinformation and a lot of false assumptions that have come about as a result of this trick essentially.

Wes Reisz: Let’s talk about the new attack surface. We already just talked a bit about the scale from the things that you were working with before with Shape Security, with F5, with Google, what it means to be operating at that scale. But at this new attack surface, what is this new attack surface that presents with us with GenAI, particularly at scale? For example, you in your keynote are going to be talking about fraudsters are the biggest users of GenAI now, because hallucinations that we just talked about doesn’t matter at all to them. How are you seeing GenAI being weaponized?

GenAI solves the “last mile” problem for fraudsters [26:58]

Shuman Ghosemajumder: So, when you are engaged in fraud, then everything that you’re producing is essentially a hallucination. So, you’re not trying to produce content that is accurate. You’re trying to fool someone with something that sounds or looks plausible, and so that’s exactly what GenAI is designed to do. The test for that is actually whether or not it convinces humans. So when we look at the way that frontier models have been trained in the last several years, and you look at RLHF, reinforcement learning from human feedback, that’s basically taking the output of a model, and then having large groups of humans go through that output, and say whether or not that looks right to them. So, that’s not an objective metric of accuracy. That’s humans saying, “That looks right to me”.

So as a result of that, models have an uncanny ability to produce content that looks right to humans, regardless of whether or not it is right. This is something that is incredibly attractive to cybercriminals, because now what they can do is produce something in any language, in any context, at whatever level of automation they want, and solve their last mile problem simultaneously. The last mile problem that cybercriminals have always had is that at a certain level of social engineering or at a certain level of a scam, you need to have a human in the loop. So, cybercriminals have their own call centers, and people in those call centers get on the phone or get on the keyboard to socially engineer someone manually, because that’s the only way to be responsive to human language, and to be contextual and to be persuasive.

But, LLMs can not only replace that function now. They can do it even better than the humans can, and so now extend that to other media. When we’re talking about video, when we’re talking about audio, they can not only produce a real-time deepfake that can look like there’s somebody else on a Zoom call, but they can do it in multiple languages, in multiple accents. This is an example of the type of attack that we don’t really have intuition for. There’s no such thing as a human being that speaks every human language. There’s no such thing as a human being that can instantly change their form into another human being, and imitate them perfectly outside, once again, of science fiction, which is why science fiction I think is such a great way to be able to think about these problems because when you’re only limited by your imagination, we’ve seen what are the scariest things that people can do in movies with technology that doesn’t exist today? Almost all of those technologies have some kind of analog in the digital world of 2025.

Wes Reisz: So even before IA, social engineering was one of the biggest attack vectors in security that you had to protect for. In this space of GenAI, it just seems like… I don’t know. It’s just massive. It’s unapproachable. It’s so large. How are you reasoning about real-time social engineering today in the face of GenAI?

Applying game theory to defense strategies [30:26]

Shuman Ghosemajumder: At Reken, we’ve been building on the experience that we gained at Shape Security. My co-founder and I, the work that we did at Google, really spent a lot of time thinking about the attack surface, how the technology is evolving, and the game theory that’s involved in terms of, “How do you deal with this problem in the long run?” Like I said, you’re never going to solve the problem of crime. As long as there are humans, there are going to be other humans that want to take advantage of them, and we’ll always have crime, and it’s going to be technology-based even more going forward. There are specific things that we can do to make a huge difference, and I’m a big believer in technology being able to make a big difference in certain types of threat scenarios.

One of my favorite examples is automobile theft dropped by 95% in New York City as a result of the introduction of a single technology in the 1990s, and that was engine immobilizer units. If you needed a coded key to be able to start a car, that completely changed the economics of a car thief. That wasn’t going to completely eliminate automobile theft. It wouldn’t prevent somebody like Nicolas Cage’s character in Gone in 60 Seconds, and his high-end crew, from figuring out how to be able to steal a Ferrari that they have their eye on. But it was effective in being able to protect the vast majority of consumers from the vast majority of threats.

Wes Reisz: As you were describing that in that answer, you mentioned game theory. How does game theory come into play with how GenAI can be, I don’t know, detected and used for real-time social engineering?

Shuman Ghosemajumder: So whenever you’re trying to identify attacks, and implement countermeasures, or create mechanisms that are designed to dissuade or prevent attacks, you always have to deal with the response from the other side. So, cybercriminals look at those defenses, or look at those detection mechanisms, and they try and figure out, “How do I get around that?” A big part of what their response is going to be and you being able to predict their response is you being able to understand their incentives and their overall ecosystem. That’s the game theory that I’m referring to, because there are a lot of attacks that are theoretically possible, but practically, we don’t really see.

So an example of this is a lot of folks are concerned about, “How do I protect my internally-hosted LLM from being attacked by an adversary who is hoping to try and change the responses that I’m going to get when I ask it questions?” So, that’s absolutely a theoretically valid concern. But from a practical standpoint, cybercriminals are primarily motivated by money. So, if they’re going to go to all of that effort, they don’t want to have to wait two to three years to see if somebody enters just the right prompt at the right moment to be misled in a certain way that they make a certain type of decision that results in them somehow being able to extract money from the system. That’s too complicated.

So, what they want instead is the opportunity to directly or at least one or two steps from being direct, have an opportunity to be able to steal money. That means that they’re going to put a lot more energy into certain types of attacks like social engineering as opposed to trying to pollute models.

Wes Reisz: You’re not saying don’t worry about one thing. It’s more about a mental model to help you prioritize the things that you should be focused on. Just because there’s not a financial incentive doesn’t mean ignore it. It just means these are maybe the larger opportunities, the larger threats that you might need to consider.

Shuman Ghosemajumder: That is a great way of putting it. I think that this is a really difficult thing for companies especially to accept from a cybersecurity perspective, that there are going to be certain areas where you have to say, “You know what? I don’t think we’re going to invest very much in that area”, even though it’s theoretically possible that they could be victimized on that attack surface. You have to decide with your finite security budget, how do you want to spend that? Do you want to try and figure out where you’ve got the greatest risk of harm and prevent against that as effectively as possible, or do you want to try and identify all of the different theoretical areas where you could be harmed, and put a little into each of those areas from a compliance perspective so that you can tell people that, “Yes, we’ve invested in all of these areas”, but then not invest enough in the area that you’re really insecure?

Wes Reisz: Yes, makes sense. Let’s talk a bit about trusting by default and talk about some of the things that we may want to consider with the Zero Trust surfaces. From everything from voice to video that can now be spoofed when we start talking about the idea of Zero Trust, what does that mean in a GenAI era?

Zero Trust requires behavioral telemetry and continuous observation [35:31]

Shuman Ghosemajumder: Zero Trust has become popularized in the last 15 years or so, especially in the InfoSec world. There used to be this idea of once you’ve authenticated someone successfully or you’ve passed certain gates, now, that identity has been validated as being trustworthy. So, they’re granted certain privileges, and you don’t have to worry about it once they’ve been granted those privileges. But what we’ve discovered is that almost no trust mechanism is foolproof, and as a result of that, you can never really fully trust any entity. One of the things that’s interesting about this is that this is in direct opposition to the way that the fraud world has always functioned. The fraud world has always tried to analyze behavior associated with every single account, every single product, every single entity that you’re tracking in your system, because you’re looking for signs of abuse, and you know you’re never going to be able to get 100% of it.

You’re trying to improve the problem from a percentage standpoint. So, Zero Trust is basically the InfoSec world implementing a fraud mindset, where you know that just about anything can be faked. Anything can be abused, and so you have to use all the data that’s available to you to be able to identify where there are potential problem areas where you need to put additional resources, and then investigate what’s going on, or put in additional products or mechanisms to try and disrupt or prevent those types of attacks.

Wes Reisz: Well, when we talked about this before, you talked about this behavioral telemetry. So, that’s what you’re talking about, so access to when you log on, language patterns. So, all of that becomes telemetry that you’re using to help identify threats.

Shuman Ghosemajumder: Yes. If you have the usage history of a particular user on your locally-hosted LLM within your company, and all of a sudden, there’s a pattern of usage that deviates from that historical use, so suddenly, they start asking questions in a different language. They start entering prompts from a new geographic location that you’ve never seen before, or from a time zone, or a time of day that you’ve never seen before. All of those are examples of things that could be indicative of that account being compromised, or there being some other breach in your system that allows the LLM to be accessed that way. We’re using LLMs as an example, but this applies to any resource within your enterprise.

It doesn’t necessarily mean when you see a behavioral anomaly like this that there’s an attack, but what it does mean is that you need to have an explanation for what’s going on that allows you to be able to take a specific action. So in some cases, what you might create is a set of rules that say that we have certain users who work in multiple time zones. So if we suddenly see that a user is accessing this resource from a new time zone, let’s ask ourselves whether or not that’s a time zone that they might actually work in. If so, we don’t take any action. We don’t worry about it. But if not, maybe we escalate it to the next level, and we say, “Okay, here’s another system that’s going to analyze that pattern of behavior. Look for other signs that it could potentially be an attack”.

By the way, if you’ve got an organization that has 100,000 employees, then it’s not a trivial thing to look at all of those cases from a manual perspective. So, you want to try and automate that as much as possible, but the end of that entire chain is there are certain cases that do have to go in front of an individual. So with more automation and more use of AI, hopefully you can leverage those individuals, the humans that are in the loop as much as possible. But, you’re trying to look at as much data as you possibly can because everything is potentially a clue.

Wes Reisz: I actually was talking to a colleague of mine this morning about something very similar, Ben O’Mahony based out of London. He’s capturing open telemetry data off of things that are happening in a model, classifying things that are good, things that are bad, and then using that to create a small language model from frontier models to create a domain-specific vocabulary of the things that people are doing using their LLM. I was just like, “That’s awesome”. So, I’m trying to get them to speak at QCon AI. Maybe he’ll talk specifically about that. So, I totally follow what you’re saying. Telemetry with LLMs, particularly on user behavior, is a great way to know.

I guess people at scale or at large groups, they move in common patterns. So if you start capturing telemetry, you can understand the patterns for individuals, and look for those bad actors. It makes total sense to me.

Shuman Ghosemajumder: There was a lot of inspiration that we would get in this type of work from great fictional detectives, especially Sherlock Holmes. The reason that Holmes was capable of solving crimes that nobody else could was because of three things that he talked about, or Sir Arthur Conan Doyle talked about Holmes possessing attributes. One was his powers of observation. He was able to see things that nobody else could see. So, that’s basically telemetry, being able to observe things that are going on in the environment that other systems don’t necessarily take in from a data perspective. There were his powers of deduction, which is what is the rule system or the set of models that we put that telemetry through in terms of being able to analyze it, and then discover patterns and anomalies and how smart can those systems be.

The third thing was his knowledge. So, he just had knowledge about so many different things. So when you’re looking for fraud for instance, it’s so important for you to have knowledge of the business, for you to have knowledge of what real users look like, for you to have knowledge of what cybercriminals look like and how they’re motivated, because that lets you explain away certain things, and identify other areas that require more investigation and effort.

Wes Reisz: Absolutely. One of the things that I’ve been hearing a lot lately, and I wanted to ask you about is this notion of negative day attacks. First, is that even a thing? Second, in today’s world, how do you prepare for a negative day attack?

Shuman Ghosemajumder: It’s a marketing term in some ways, because zero day attacks are the first time that you ever spot an attack, so you can’t really talk about a zero day until you discover it. It’s just something that’s latently out there. That’s how people refer to negative day attacks as well. It’s an attack that may not be discovered in the wild, but you’ve discovered it through some other mechanism, but that’s really a zero day when we’re talking about it.

Wes Reisz: With that said, though, it’s the window from when it’s discovered to when it is decreasing to the point that it makes.

Shuman Ghosemajumder: Exactly. Exactly. So like we were talking about attacks going exponential before, there’s a difference between society and the industry having a manageable number of zero days that are discovered every single week versus having 100 times as many or 1,000 times as many. Those windows are rapidly getting smaller, and so we have to have new mechanisms to be able to automate a lot of those manual processes.

Wes Reisz: We have architects. We’ve got VPs. We’ve got CTOs. We’ve got developers all listening to this podcast, all fully recognizing that GenAI does an amazing job at being able to tell lies if it wants to. As someone who has been in the trenches of establishing trust online and catching fraudulent actors, what advice do you give to software practitioners today on how to deal with this incredible threat that is facing us?

Aligning threat models with business incentives [43:45]

Shuman Ghosemajumder: I think that it’s critically important that every organization think about their own business model first. There are many different threats that are posed by Generative AI. There’s the attack surface that’s associated with using an LLM at all internally. There’s the attack surface that is created by sending your own data and your prompts to an externally hosted LLM or a chatbot service, and there are probably about a dozen different attacks that people are thinking about. What really matters to your organization is going to be very dependent on your business model and the way that your technology functions today, and how you’re trying grow that technology stack.

So, I would start with that context and your own business model first. Now, that being said, when we’re talking about something like social engineering, that affects every company to some extent, and some companies have social engineering as an existential threat. So if you are an organization that is constantly working with confidential information or is interacting with the outside world in such a way that you’re moving money around like banks do, then none of those organizations are learning about these types of attacks for the first time. They’ve been thinking about this for years now, because it’s such an existential threat.

What I would say is starting with that exercise of thinking about your business model and thinking about your own risks, now map out all of these different threats and figure out what you’re going to invest in, because you will have a finite budget. Everyone does. Doing those war gaming exercises, figuring out how to anticipate what problems you may not have encountered yet but may encounter in the very new future, those are the types of things that really save organizations. It’s up to the security folks and the IT folks within every organization to take the lead on that.

Wes Reisz: So, establish a threat model, identify the things that GenAI can do, and make sure that your business is investing in the right areas to protect for those things.

Shuman Ghosemajumder: Absolutely. It’s fairly straightforward advice that many, if not most, organizations don’t quite follow for various reasons.

Wes Reisz: Well, Shuman, thank you for speaking to me today. If you like what you heard and you want to hear more, Shuman will be keynoting in New York, December 16th and 17th, deepfakes, disinformation, and AI content are taking over the internet. Please join us. Shuman, once again, thank you for being a guest on the podcast today.

Shuman Ghosemajumder: Thanks so much, Wes.

Mentioned: