Allie Howe: Welcome to today's episode of the Insecure Agents podcast. I'm your host, Allie Howe, and today I'm super excited to be here in San Francisco at AI Council with Diana Kelley- Yay... CISO of Noma. Diana has spent many years in security across different enterprise companies like Microsoft and Symantec. So super excited to have Diana here today with her wealth of knowledge. She actually already spoke here at the conference today. So super grateful for her time here with us. Diana, would you like to introduce yourself?

Diana Kelley: Sure. I mean, I think you did a great job. Yeah, Diana Kelley, and I'm right now CISO at Noma, and I've been interested in AI and AI security for... Thank you for not saying many decades, and 'cause-... it's, 'cause well over three decades now. But I've actually been interested in AI since I was at IBM. Because that was around the time of Watson, and a lot of people don't, you know, think of Watson as AI quite so much anymore, but Watson absolutely was a, a, a, a different kind, not a post-Transformers AI, but was an AI that actually had gotten so well trained it, it won Jeopardy!

Allie Howe: Yes, that's amazing. So you've definitely seen AI evolve and transform over the years, and have to deal with securing it Through the...

Diana Kelley: Yeah, and thinking about, yeah, the threat model and, and what we need to do to secure it. Yeah.

Allie Howe: Yeah, and I'm super excited to touch on some of the things, like, you've seen over your career and what you're seeing now with AI, and like, so what is your take on that? Specifically, like, the AI shared security model. Yeah. Because we've gone through these, like, revolutions in the past where there was, like, cloud security and developers are wondering, "Okay, is it on me to secure the system? Is it on the cloud provider?" Like, where do the lines blur? Yeah. So super excited to dive into that as well as like, of course, how can we secure agents and the latest on that today. But to kick us off, I really wanted to start with the Pocket OS incident where- Yeah... a developer had a Railway API key and gave it to the agent, and the agent more or less, long story short, deleted a database within- Yeah... nine seconds.

If you read the post on, on X from the individual that was the developer for this system, it seems like to me, they're sort of, like, blaming the vendor, like Railway of like- oh, you didn't have enough controls in place. And like maybe that's true, maybe that's not. But like where does the line blur for AI in terms of the shared security model between vendors and developers?

Diana Kelley: I mean, that, that's exactly... You know, you're making a, a great point, and that's exactly it is that you... Who's responsible? In this case, there was a deletion, right? There was a, a model, at some point a model went through inference and came out with, "Delete this database," which was then fed into the agent. The agent took the action and it deleted the database. So, was it... And there was... And the backup was on the same volume, so it wasn't even able... So, so who's responsible for that? Is it the person who gave the API key to the agent? Is it the vendor that trained the AI that came out with the inference to do this destructive task?

Is it the platform owner for not controlling the... and putting controls in place that, oh, before there's a deletion you always have to have a human say yes. And it's, it's, it's very complex, and we had talked about the cloud. And in the cloud it felt a little bit easier. It was... There were sort of two responsible parties, you and the cloud provider, and your data was always your data, which is why when I was at Microsoft, we could use the m- the, the bank analogy of, well, the bank's responsible to make sure that the bad guys don't come in and stick 'em up. Willie Sutton doesn't, you know like stick 'em up and get your money out of the bank.

But if you give your, your ATM card and your PIN to somebody, or if you overdraft, you know, the money's yours, and how you use it, it's your responsibility. And you could obviously... You know, if you gamble it all away on Like snail racing or something like that, right? Yeah. Again, the bank's not gonna step in. So the... When we looked at the cloud, the data sort of became the money. And at the IaaS level, they were respon- the infrastructure level, they were responsible to make sure that there was always power, that the data center itself was locked up. But then what you loaded on top of those systems and how you managed them was your own responsibility, but...

And the data was yours up to SaaS, where the data's still yours, but a lot more of the controls of the software were owned by the provider themselves. But you always ultimately had the decision about your data. And AI's different because the data that goes into the model, the frontier model, you don't have control over that. What comes out of the frontier model, you don't have control over it. It, it starts to create this much more, you know, interconnected set of at least three different sets of folks that are, or entities that are responsible, and it, it's getting... It's a lot more nuanced than what we had with, with cloud, I think.

Allie Howe: Yes, definitely. For sure. And I think you mentioned in the post I read about this that you wrote about, like, there's three, like, different parts of this, like, trifecta or of, of like there's the model and then there's, like, the, the developer's, like, platform themselves. And what was the third one from that? Or-

Diana Kelley: Well, there's, there's the developer. There's- Yeah... So there's the folks that do the, the software-... around the model or are deploying and how they're using it. There's the user themselves-... and how they interact with it. Like, what you're promp- what you prompt is gonna impact... We've all seen that just when we use models ourselves. What we put in as a prompt impacts what we get out. And then there's obviously the, the holders and the creators of the model, because they picked the data that trained the model. They have the, the, you know, a, human reinforcement learning human feedback. Like, it's their humans that are giving it the feedback, and that feedback could be biasing it in one direction or another.

We don't, we don't know. They don't tell us what they're doing. So that's, those are, you know, at the very least, and as we look at, at more complex deployments, I think we could have even other, you know, responsible parties within this very complex... Like, what happens when we start doing multi-swarm agents- and it's multiple models used by different agents? The responsibility starts to get very, you know... It gets very complicated 'cause we're creating workflows that are tying together a lot of different systems now.

Allie Howe: Yes, that's very true. And I feel like in some cases we don't even know, like, what data the model was, like, trained on or what the model weights are. 'cause those could easily get tweaked and then, like, one thing could be preferred or given a lot higher significance than something else, and you're not in control of that, so. And if it's operating over your data and serving, you know, subsets of your data to customers, like, you know, that definitely affects your business.

Diana Kelley: Yeah. Yeah. And I mean, like, is... Again, the, the, the data that it was trained on, we don't know that, and that directly impacts what it's gonna put out. And if it has a bias that's been either trained in or just developed over time to recommend you should do this versus that, you should always buy this kind of a vendor versus that kind, these If they're, we're using the outputs for decisions, this is critical decisions. This is gonna be really it's gonna be really interesting to see how this, this plays out. And I think there's gonna be some interesting legal precedents too that are, you know, some case law is gonna come, come along and start to It's gonna be interesting.

Allie Howe: Yes. Oh, for sure. Absolutely. If you're evaluating, like, an AI vendor today, whether it's a model provider or it's a tool that has AI built into it, like, what's the one question you recommend as a CISO asking those vendors to assess their security posture and whether or not you can trust them, given all this complexity?

Diana Kelley: Yeah. I mean, I, I would, I would ask them what they're doing to... You know, what, what models they're using, what they're doing to manage all of the trust boundaries around the user. I would a- ask them what their security approach... Are they, have they threat modeled their system? And what does that threat model, threat model look like, and can they sit down with you and explain it? Because, you know, we've got... My friend so he's... I'm from Newton, Massachusetts originally, and so is Adam Shostack, so we've known each other a long time. So when I say friend, I really mean it- Yeah... with Adam.

But you know, he, he did a beautiful job of, of bringing threat modeling down to, you know, four questions. What are we doing? But what's embedded in what are we doing is y- the, the topology. So understand- asking your AI vendor, "What went into this? Can you explain to me what went into it, what all the pieces are? If they're agentic, then what's in the, the agent, the agent harness? How did they test it?" Really getting that topology. So what are we doing is not just what you as the company are doing and what they as the, the vendor are doing, but also understanding the components and the pieces that are, are part of that.

And if they can't sit down and start to have those kinds of, of conversations with you and explain to you what they did to threat model and to be able to decompose the system so you can start to get an idea of were they able to threat model it in a robust way- Then I, I think you probably maybe if it's, if it's a low impact use case, okay. But, you know... Or if it's a use case where you can, can look at the outcome very easily so that if it does make mistakes, it's not gonna impact your business significantly. But if it's a high impact use case and you wanna have high trust, then you really do wanna understand what that vendor has put in, the rigor they put into the, the designing and, and testing that system.

Allie Howe: That makes sense, and that's good advice. In terms of threat modeling, I feel like we've done better at that as an industry over the last, like, six months or a year just because we've really needed to, needed to for AI. But one of the things I saw that you're on the working group, you're a chair, co-chair of, is the ARM. Yeah. The, a new specification that describes what a runtime security solution must do. Yeah. And- It was really interesting to me about how they handle authorization, and it's not just like allow and deny for these decisions around like what action an AI system is allowed to do.

But they also include actions such as modify step-up or defer. Defer, yeah. Which is like probably my favorite one. Which is super interesting of like, okay, like let's just admit that there's not enough context to make a decision about if this AI is allowed to do this or not, and so let's defer it to like a later point when, I don't know, a human could get involved or we have more context. But how does... Like, you know, describe like if you want like your involvement in Arm and like what that's all about and how that could help us like threat model or feel more confident about AI systems.

Diana Kelley: Yeah. Yeah. So Arm first it was created by Herman Narico. So he, he came up with all of this himself. He created the, the standard with the intent of exactly what we're talking about, giving people some normalized way to understand we have to control at runtime-... what, what the systems need to do to be able to do that so that we could start to have a standard to con- have a conversation about. And he started the work himself independently, and then a lot of people heard about what he was doing and the CSA now, so that there's this open standards, you know, body. Cloud Security Alliance is now the owner of Arm, and the working group is made up of a lot of people from different, you know, stripes and different areas, different organizations.

You know, there are competitors to the company I work at on that, which is great. I mean, that's what we want. What we really want is a lot of people coming together and having this conversation so we can normalize it. And, and then, yeah, it's, it basically it's a, a conformance to if you're, if you as a vendor are stating, "I can stop agentic badness at runtime"- what would need to happen? And yeah, there's obviously-... the, you know, you can allow or deny, right? Those are gonna be the, the baselines. And then, yeah, th- other things like, you know, can you, you, can you stop and can you do a check on it?

Do you bring a human in the loop? Do you defer it to a later time? And being able to bring in these other components, which nicely echoes some of the things that we saw traditionally in security because, you know, way, way back in when I was doing like firewalls and installs in the '90s, you know, it was like that was pretty much it. It was like allow or del- deny, and as you know, just left it on the floor. Yeah. But as we saw with things like email or as we were looking at zero trust with step-up authorization and things, we realized that we need a much more nuanced approach, and we can't just say yes or no.

There do, does have to be sometimes it would be quarantine or there's a double check. So, so that's the nice thing about Arm is that it takes into account that it's not just a yes or a no. Sometimes you, you need to have more information or more context. And then organizations can test themselves against conformant- conformance. And obviously, 'cause there's a big working group going on right now within Cloud Security Alliance, a very active working group, I would expect that over time it's going to change and mature so that it can really keep up with what organizations need to do. And hopefully will be a baseline so that if you're buying something you can get a sense of- Is this really doing what it's supposed to?

Because when we don't have standards or we don't have baselines for everybody to sh- and shared context, very often each vendor goes out, you know... Love, love the each vendor, but each vendor's gonna go out and sort of define it their own way. So this is an opportunity for the, the the market to come together, have a shared definition so that as companies look at, is this gonna help me actually control and govern runtime of agentic action and badness, you know, is it something that is, that I can really kind of test against in this, like, normalized, you know, canonical fashion?

Allie Howe: Yeah, that's super helpful. And I, I saw Noma as one of the conformant companies to ARM- We are... so congrats on that. Yeah, yeah. That's huge. I, I, I talked to Herman yesterday. He was on the podcast. Oh. He came in here yesterday. Okay. Oh, wonderful. Yeah. Yeah. Yeah. No, that was a really interesting conversation. And one of my biggest takeaways was the need for, like, dynamic policy that's able to change, like, on the fly as you learn more about what an agent what its mission is or what- Yeah... The context around that decision-making point is. And that got me thinking about if we're making decisions, like, based on context, how much context is really trustworthy?

How do we decide that? Yeah. And, like, what, what are your thoughts on that?

Diana Kelley: It's such a difficult problem. I think that this is a big... Y- yeah, and it's, and it's interesting 'cause there's two kinds of context. There's the context of what is that agent doing within your organization, who built it, who deployed it, what was, what was the original goal, what workflow is it part of, what's its full blast radius, all of that, and then there's also the context that gets into the context window for inference and then impacts what's gonna come out of the LLM, which will drive additional agent actions. I think that this stuff is, is pretty complex, so we're in a, a bit of a, an early phase.

I'm glad that there are really smart people working on it. Not necessarily me being smart, but, like, that there's like Herman, you know, like, very smart people, you know, looking at, at these problems and trying to address what we need to do because I do think that this is a moment that the industry does really need to come together so that we can look at what this new challenge is, what it really means to our, our... 'Cause our, our old tools were not built for th- this future. So what it really means, and then making sure that... 'Cause something like a dynamic policy, yeah, look, i- if it can...

If not, if not deployed properly, it could be as easy to game that as it is to game an agent itself.

Allie Howe: Yes. And it's super interesting to just, like, watch the industry at a high level evolve around this problem. So not just necessarily, like, solutions like Arm, which are fantastic, but there's also, like, new auth protocols being created. Spiffi... yeah, yeah, yeah Yeah And, like, Aauth that will l- let you, like, cryptographically attest to, like, who this agent is. Yeah. So, like, over time we'll get better and better context. Hopefully. Hopefully. Yeah. Yeah. Yeah So we'll be able to make, like, better decisions.

Diana Kelley: Yeah. Yeah. Yeah. It's, it's a, it's a, it's an interesting... This is, I think, one of the most dynamic times in technology. I mean, cloud was, was a very interesting time, but this is really, there's, there's so much interesting work. A lot of people have been fearful, "Oh, what's gonna happen to my job?". But if you really look, we need so many people leaning in and working on how we govern and manage these systems going forward, how we use them responsibly and ethically, and how we make sure that AI is, is democratized and available to all and not used in a way that, that creates more problems rather than addresses.

There's a lot of work to be done, so yeah, there's a lot of work for people.

Allie Howe: Absolutely. And there's such an urgency around it too, because I know we'll talk about this on the panel here at AI Council later today, but with the rise of better model technology that maybe are better at finding vulnerabilities in some of the best security researchers in the world it's super imperative to now have agentic solutions to defend against these attacks. But these agents are going to need to be able to rotate credentials, shut down VMs, and be able to take, like, very authoritative actions in productions because human response time to these AI attacks is just gonna be too slow. But we can't ship these kinds of agents into production if we can't trust them to operate autonomously with high amounts of capability.

In order to do that, they're going to need, like, high levels of security. So to your point, yes, like, we need all hands on deck, and there's plenty of work to go around.

Diana Kelley: Yeah. Yeah, well, 'cause even what you're describing, right? It's like, well, they have to shut down a VM. Well, you know, d- do they? Well, maybe we stop the VM from being spun up if it's not supposed to... I mean, this is gonna be some really interesting yeah, it, there's gonna be a lot of work to figure out what it is that needs to get done. And traditionally in security, it's always been we didn't do the automated block because we started breaking systems, and that was problematic, and people were upset. So it was, we ended up going back more manual. But now when we've got agents moving so fast, we really...

I- if, if the attacker's going at, quote, "machine speed," yeah, our defenses have to go at that same speed.

Allie Howe: Yes, absolutely. And I know there's not only a very, like, small subset of people that have access to things like Glasswing and then Mythos at the moment. For, for those of us that don't, which is most of us, if we're concerned about new AI capabilities and attacks from better more powerful models what's the best action to take to start, like, scanning your code base for vulnerabilities or finding out, like, where the scary, like, zero-day vulnerabilities- Yeah we've never found before lie? Like me personally, I've seen things like teams like Sentry build their own, like, harnesses around- Yeah... Claude, like Warden. Yeah... which is, they, David Cramer reported, like, publicly on his blog that they found, like, 60 different vulnerabilities that they would've paid bug bounties for, and it only cost them, like, 100 or $1,000- Yeah in credits.

Diana Kelley: Yeah Yeah, yeah. No, I mean, that's it. It's like, Mythos obviously it's in preview with just the, the Glasswing folks, but there are a lot of powerful models out there that can be used already right now to start as, as, you know, you, you've got with Sentry and Warden, you know, that show that... You know, there's also Daybreak at OpenAI, and then just this morning Microsoft announced something called M-Dash where they've got over 100 agents. They're using both Frontier and open source models, and they're, and that's a, an orchestrated multi-swarm system for finding vulnerabilities and addressing them. So right now, there are a lot of options and, that organizations can take to start to use autonomous AI-driven vulnerability discovery systems so that they can get a view into what's there.

So you don't, you don't need to wait for Mythos or wait for your, your access to it. You can really start taking action now, and I haven't looked at or played around with the M-Dash from Microsoft, but it sounded really it sounded pretty promising just from the, the blog post that I read. So that could be another, you know, option for organizations to start taking action. Get their own orchestration in order to start doing their own testing, and then addressing their vulnerabilities. The other big thing, and it's, it kind of, it, it's below the headlines, but it's really, really critical-... which is that a lot of this is about security hygiene-... and there's been a lot of legacy security debt that organizations have just kind of been like, "Eh, it's hard to find the vault.

It's hard to find the attack path," 'cause it's not just always, you know, this vuln or that vuln. It's like chaining through a series of vulnerabilities or a vulnerability that gets you to an exposed area in the organization and, and chaining through an attack path. Most of those paths are there because something wasn't patched, something wasn't closed, a control isn't in place, and a lot of companies have been... Because look, they got, they got priorities, and you gotta, you know, the business has to get done. So I'm not trying to, you know, say anybody did anything wrong or, or throw anybody under the bus. But there is a lot of legacy security debt in most companies, and this is a time to really go back and assess can you afford to keep that legacy debt the way that you have in the past?

Because now you've got a, an, you know, an, an adversary that's got some pretty low-cost tools that move very, very quickly and, and, or accurately. So they may find some of those vulnerabilities that you felt were okay to leave open, they may no longer be okay. So definitely look at the hygiene and the debt.

Allie Howe: No, that's great advice for sure to look at the hygiene and start taking the first steps there to action on that. And then circling back to the shared AI security model, while we're waiting on AI vendors to maybe adopt an arm security solution or to create API keys that have very fine-grained scopes so you don't end up deleting like a database and improve those patterns there, like-

Diana Kelley: Take all the access.

Allie Howe: Yeah, exactly. It's just like that's just the easiest thing to do- Yeah... give us all the access. It was interesting talking to David Creamer from Sentry yesterday. I had him on the podcast also, and he was saying- Yeah... in their MCP server they don't even allow you to delete stuff in Sentry. Like, it's- it's not even an option. Like, you can go- Yeah... to the UI console for that. Like, that to me, that's like, that's a good pattern. It just removes a lot of like, like, like, ultimately it's probably like the developer's responsibility to like have an arm system or to have agentic security in place such that that never occurs.

But like why even give someone's agent the option to shoot themself in the foot when you can like, as a human, go to take that destructive action yourself? This is another- Yeah... layer of like defense. So- I think that's a good, like, exhibition of, like- Yeah... a shared security model probably could look like.

Diana Kelley: Well, I think certainly looking, being granular and really thinking about what does this system need to do? And something that, you- even since way back when I was working on ML SecOps years ago- you know, like, the quest- the first question is, do you need AI for this? Yes. Because you don't always need... And by AI, obviously we're, we're sort of short-handing into Gen AI, post Transformers, LLM. But you know, it's not, just because you can, doesn't mean that you should. When there was the, the code leak from Claude a few weeks ago, and people were shocked that it, that for sentiment analysis, Claude was using regular expressions to assess if people were mad or not, you know?

And, and a lot of people were like, "That's like using the Pony Express when you've got, you know, like a su- a supersonic jet. Why would you do that?" It's like, well, 'cause regular expressions are very accurate, they're very cheap. You know, like, so why not use them? That, that was a really good use. Anthropic knows. I mean, Anthropic software developers know it, and I think a lot of, of maybe folks that, that read headlines and, and get very excited about, oh, AI's gonna, you know, solve every problem and, you know, do all the jobs better than any human ever could. That's not true. The truth is that AI is very good in some use cases.

It's not great in other use cases, either because it's too expensive or it's not accurate, and sometimes it's just not the right fit. Yeah.

Allie Howe: So what you're talking about is like, you know, if you're saying y- do you get an API key with all the access or, like, limited access, what you're saying is really let's be thoughtful and, and th- and look at what we want to do, what's the best way to do it? And maybe that could be an LLM-driven agent, but with a scoped access.

Diana Kelley: It could be, or like the CenturyLink MCP where it's like you can never delete something. You know, or it could be that you give it more access, or maybe that you don't use AI at all. All of those are okay. It's just a matter of companies you know, the builders sitting down and thinking about what they want to do, and then getting the best fit for it. I think where people have been kind of caught out was really some of those headlines that kind of made it out like, wow, everything's gonna be perfect and take care of itself going forward and, and that's not quite how, how it works.

Allie Howe: Yes. Absolutely. Yeah, for sure. Any other things you find interesting right now or looking forward to, or any other advice for our audience?

Diana Kelley: Yeah. I, just s- s- s- keep reading. I mean, I think that that's really the, the biggest one. Keep reading for, especially for the security teams. You know, keep, stay on top of this stuff. Take deep breaths and, and have the conversations with your executives. There have been a lot of, of- belief that, you know, you're gonna be able to save 50% of your workforce and, you know, you, you start a whole company with, you know, just agents. And, and there's been a lot of, of, you know, hopeful thinking that AI d- works in a way it doesn't. So, you know, as a security person, you know, take a, take a deep breath.

Make sure you understand how it works. Be okay with having to read and reread and go back you know, 'cause sometimes some of these, these concepts I've noticed that even with my own learning journey, you know, I'll think I got a concept. Yeah. And then I'll see some other thing and I'll realize, oh, there was a really critical component of that. I think prompt injection is a great example where we, we kind of all know what it is, but then when you really get a deeper dive into it and you truly understand what's happening in the context window and the- flattening of the, the tokens, you know, it's, your, the, your understanding can deepen.

So I think just stay educated, stay humble. None of us knows everything that's happening. Know that nobody has all the answers yet. And, you know, keep being collaborative, working with each other. I think us, all of us speaking to each other, we're, we're helping each other. It's a collective effort. And try and educate your, your executives on the fact that there's no magic here, and there's gonna still be a lot of security that we need.

Allie Howe: Yes, absolutely. That's a really good point. I, I imagine that push and pull between execs and, and CISOs right now is very, very not contentious, but just like, you know, it's, it's a dance for sure.

Diana Kelley: It's, it's frustrating. I mean, there's, if you're, you're seeing headlines that say, "Here's this incredible tool, adopt it, and your, your company's gonna be supercharged. You'll have, you know, 10X your, your profitability." And, you know, and then the CISO's like, "Well, no, actually..." You know, so I mean, yeah, it is. And, and I think that the CISOs that do say, "Hey, I love this. I'm a huge fan," it's just this is, the more you know about what the business is trying to do- Yeah... the better. Because then instead of just saying, "No, AI's scary," you can really have a nuanced conversation about, this is what we could do for the business that would be better without AI, this is the risk to the business with AI and this is what we can do moving forward.

So you're not seeming like, you know, the Dr. No, but you're coming in as you're the innovative enabler. But protecting the company from, you know, Pocket OS and having your data... There was another one that was in the news just this week too where it was something that was, they, it wasn't even, it was worse for the company- and it took them, like, almost a full day to, to recover. So there's, th- I think we're just gonna keep seeing more of these coming down. And, and actually that's a little bit helpful too because executives see that and they're like, "Oh. O- okay." Yeah. "The CISO wasn't just trying to say, 'Don't do it 'cause I'm, like, work to standing your way.' The CISO had a legitimate concern about the, the safety of the business.".

Allie Howe: That's true. Yes. Yes, absolutely. Well, it'll be interesting to continue to follow along there and see how the space evolves, but I'm super excited about some of the new protocols like, and then ARM, that specification, and all of the conversations we're having together as a community- Yeah that's gonna rise the tide for all ships. So- Yes... super excited about that. Yeah. Thank you so much for coming on the podcast.

Diana Kelley: Oh, thank you. No, I'm excited. And so we've been trying to do this for a while, so it's like, yeah, it finally worked out, so.

Allie Howe: I know, absolutely. Yes. Finally worked out. Yeah. And super excited to do a, a bunch of different talks and panels, like, here at AI Council. Yes. Awesome. Yeah. Well, thank you so much, Diana.

Diana Kelley: Thank you, Ally.