Allie Howe I'm super excited to be here today with Michael Davis. He's the global security architect for JPMorgan Chase. Super excited to get all of his insights here today about what's going on in his world, what he's seeing at Black Hat and how he's seeing AI security progress. Michael, you go by Davis, you told me?

Michael Davis Correct. Yes.

Allie Howe Yeah. Davis, would you like to introduce yourself?

Michael Davis Sure. Yeah, so Mike Davis, go by Davis the global chief security architect for JPMorgan. Most of my work is focusing on how do we bring innovative new technologies, capabilities, both from a defensive and, you know, also from a technology for our customers and the, you know, actual bank itself into the firm, and then scale those out across the whole company. And so I'm intimately involved in our AI issues and AI security items, quantum computing a variety of other different areas that we're doing. So excited to talk to you today.

Allie Howe Awesome. Yeah. Thank you so much for coming on. Thanks for making the time. I know I was chatting with a few people today already about just, like, how fast things are moving 'cause of AI and how we have to react at machine speed to all these different attacks and how you're able to chain together like low vulnerabilities to get- Yeah to the next point. If you think of like the, like Hugging Face incident of an a model escaping a sandbox and using like finding a zero-day in it to be able to do so. How do you think about, like, just, like, the threat ecosystem that maybe not, not only, like, JPMorgan Chase is faced with but maybe smaller companies and smaller supply chains that maybe don't have the resources that you do when it comes to securing against these threats?

Michael Davis Yeah. So I actually think that's gonna be probably the biggest problem we'll have to solve over the next few years. May not be familiar, but there's a term called the cybersecurity poverty line-... which is the idea that, you know, cybersecurity enterprises that sell cybersecurity software generally it's very expensive. And so companies that want to do the right thing, they buy the, you know, greatest type of detection capability or whatnot, they just can't afford it. On top of that, you know, salaries are of course high, right? It's hard to actually find people, right? So you put those two things together, and many companies that are incentivized to do the right thing just can't.

And so those companies end up becoming suppliers to large organizations like ourselves, right? Many of those end up using SaaS products and other things where they don't have to run and keep the infrastructure going, right? 'Cause it's easier for them. Those two things together cause situations in which many large corporations have problem where their suppliers are constantly being breached that includes the SaaS vendors themselves, you know, causing issues. Identities being taken, et cetera. And the side effect of all that is that ultimately those suppliers will cause issues potentially for us, which then therefore we have to kinda worry about what's gonna happen to our customers.

And so some of the things that we're seeing right now I think AI's gonna help a lot of these companies in many ways. It'll help kinda augment the kinda limited resources they have. But I think we're gonna have to rethink the whole way we're doing software fundamentally to deal with the amount of vulnerabilities we're getting. But more importantly, to have these companies that don't have the resources figure out how they're gonna solve and mitigate the problems, right? It's not always as simple as just installing a patch, right? Many of these companies, they'd love to just go install a patch, but they've got 85 other things to do with their limited resources.

They may not even have the right tools to deploy the patch in an easy way. So the timeline between a patch being available to actually being deployable by some of these companies is much higher than I think people realize. And we as a kind of industry have to do something about that.

Allie Howe Yes, absolutely. And, and I think you're right that AI's gonna help these companies in some way. One thing I'm really interested in seeing is how AI's gonna speed up the loop between, like, finding a vulnerability- Yeah... and fixing it. 'Cause I feel like, you know, traditionally before you had the security team go and find a v- vulnerability, report it, hand it off to engineering- Yep... and wait for them to fix it, convince them to care about it. But now that'll all be part of the same loop.

Michael Davis Kind of... yes, although that's not what our data is showing, to be honest with you.

Allie Howe Oh, really?

Michael Davis Yeah. Okay. So, give you some insights and I don't have maybe the exact percentages in front of me, but you kinda get the point of it. We're seeing that patching is a lot harder for the models to do than finding the vulnerabilities. Right? Kinda what, what I've found is that they're really good at the long-running tasks. They're good at following, if you will, a graph, right? So looking at source code, it can see all the different ways that potentially a vulnerability or data could be going through a set of function calls, exploit something. But when you ask that same model to take that whole source code repository and then put in a fix, a lot of times you'll see it add another function that it didn't need to add, that introduces a new bug.

You'll see cases where instead of it modifying it in all the spots it needs to do it, it only modifies it in one spot, right? So we're starting to see what I would call partial vulnerability, you know, coverage. And then two, we're also just seeing flat out introduction of operational issues. So we trade off a security incident, yeah, the vulnerability is no longer exploitable, but now that same software has a new crash in it that we don't know of, which causes an operations ins- incident, right? And so as we've been doing our work in both, you know, the DeepSeek and the kind of open model type stuff as well as the fr- frontier models it's still a hard problem.

And some of the benchmarks that have come out, you know, DeepSWEE there's stuff that Endor Labs has done as well with their benchmarks, I think are starting to point at this is not as easy as we think, right? And then a line on top of that, that's just to get the patch So the AI really right now, that loop you're talking about, that loop usually ends when the patch is available. But that's really where most of the teams, the many thousands of application teams we have at J.P. Morgan, that's where their work starts. Because now that they have a patch, they have to go deploy it, right?

And so the AI agents that we have that we've seen kind of in the market are still nascent in the areas to actually help with production-related items. So whether you wanna call it DevOps, DevSecOps, those agents are still maturing, right? And it's mostly not AI's fault, to be really honest with you. It's the fact that we have very diverse heterogeneous application teams, right? One of the benefits generally of being a developer is you get to pick and choose what libraries you're gonna use, et cetera. And I think that that is now causing for larger companies a problem because you may have that same library that's vulnerable used in 100 applications, but all 100 of those are actually doing it and deploying in different ways on different cloud platforms with different user interfaces and APIs.

And so the reachability for each of those is different, but the impact of actually installing the patch, dramatically different. For one team, it could be easy. Let's just go ahead and do it on our next, you know, deployment and we're good to go. For others, it could be a major effort to be able to do a lot of retesting. And so I'd love to see AI kinda start helping more in that side of the world, right? Where we're actually helping the decision-making that the developers are having to do on how they can actually deploy, maintain their system better. And you know, probably a little bit of a longer conversation for many, but Yeah there's a concept by Dan Greer that he released way back, I think it was in 2004 or 5, about the software monoculture.

And, where I see the real issue being is like, why do we accept that there's a patch Just one, right? Yeah. Like we get one binary from a vendor. Why do we all, every company in the world, get the same binary? We needed that back when we had CDs and floppy drives, right? Like the distribution problem was hard to get you the actual program. Now, though, distribution's easy, right? You can download it, you can easily distribute it to your team. So why are we accepting, you know, software to be built and distributed in a way in which the attack surface of the software is the same for everybody?

Instead, I'm only using three out of those 100 features. Like maybe only those three features, only the binary, the functions needed for that capability should be in the stuff I'm running, right? And, a lot of open source software, even though we can download the open source package, compile it ourselves, most companies are using the releases from the actual open source maintainer. They're not compiling it themselves or whatnot. So, this monoculture issue, I think is affecting more than just the commercial vendors. It affects open source, it affects anyone using, you know, software today. And I think that there's ways that AI can dramatically help with that, you know, help manage configuration, et cetera.

But we're not having that conversation. You know, we should be talking about how do we change the way we actually build the software.

Allie Howe And I feel like that conversation's like overdue for so long too. It's not just- Oh, yes... an AI problem.

Michael Davis Yeah. Yeah. Yeah. It's a-- I think, you know, until we had AI that had the ability to do, you know, reasoning in the loops that we see now, et cetera, no human could do that. Like you'd have a massive team trying to do all this type of stuff. I mean, imagine being a large software vendor and you have 7,000 configurations you have to ship. That'd be crazy. Yes. But nowadays that's, you know, AI agents can handle different configuration settings. That's compute. You scale your compute rather than having to actually scale humans. So I think as we start talking about first principles and how AI is changing how we think about things, the way we distribute software, the way we build it needs to be looked at.

Allie Howe Yes, that makes sense. And when a piece of software is vulnerable, it takes so long to actually patch all of it across the supply chain. Oh, yes. Yeah. Like I think when Log4j came out, someone-- some article said it, "It's gonna take 10 years- Yeah... to patch every single instance." Yeah. And that was just for that. Now we have Shai Hulud and all these- Yeah... worms, and it's just like, oh my gosh. Yes. Like there's a new supply chain attack every single day. So- If we have this problem already, like it's just gotten even worse.

Michael Davis Yeah. And you add in like skills, which also need to be versioned, skills are code, right? You know, so now you've got all the dependencies of skills with your AI agents. You've got... So I think you're seeing an exponential increase in the amount of dependencies that a team has to manage, and it's not really feasible for those teams, you know, to deal with that. And so I think we're starting to see and I know we at J.P. Morgan are looking at how do we embrace and use open source in the future? What does it mean for us, right? What does software and how we interface between teams mean?

Because if, for example, a team is providing a API, if that API is changing a lot, it's gonna be hard for those teams that are consuming it to adapt. Yeah. Well, we kind of get that when we talk about APIs, but yet we have no problem with packages being released every single day from different open source maintainers, and you've got, you know, no way to really keep up with that pace. That actually creates a different type of risk because now that I can't keep up, I'm not maybe gonna get those security fixes or other things. And so I have to start making feature for security trade-offs.

And those are not conversations we wanna have with the business, right? Right. They want the features but they also wanna be secure. And we're starting to see, you know, some different open source packages and maintainers, they decide that, you know what? You're only gonna get that security feature if you go to the latest, newest version with this new capability and forcing you to change. Yeah. Right? And I think that's kind of gonna be a interesting tipping point for many teams to realize maybe we're not gonna use that open source package anymore then. We're gonna have to rewrite it ourselves.

Allie Howe Y- yes, which is, this is an interesting point because I've heard some people say like, "Oh, like open source is dead now that code is so cheap to write- Yeah... we can just write our own packages." Like, what are your thoughts on that?

Michael Davis I don't think so. I mean, I, I, I've been open source, so I've got many open source things I've done. You know, way back in the day, I did a lot of work with, you know, Nmap and all these different tools. So I was a part of that movement. I think though it's a little bit different now, though, is, you know, a lot of that was learning and, kind of publishing and showing things to show different capabilities, and it was really a community that grew around those things. And many open source projects start like that. But now you have open source projects that, you know, are millions of dollars, you know, that actually are using...

They have huge teams. They're funded by corporations, et cetera. And I think that there might be a time where we have to determine, okay, well, what is the liability, and w- where does liability sit with these type of packages, et cetera? And the old idea of like, well, the source code's there, you can do what you want is somewhat true, but the build systems, the ability to actually get the thing to function the way it does when it's being released is not a trivial process. Right? It could be back, you know, when we were all doing C applications or whatnot, but as you mentioned, the supply chain, you know, now you're pulling in 1,000 transitive dependencies.

You know, and this one package that you need you pulled it in because it was quick and easy to use, but in reality, you just pulled in another 500 things that you don't want. And so thinking about our decisions at the design time is gonna become, I think, more important because, you know, maybe you want that package that actually is a little bit more obtuse to work with- Maybe not as easy to integrate, but it has no transitive dependencies. Yes. Right? And so I think that's why you're starting to see different languages and, and in many actually open source community revisiting just how we're building that software so that it's easier to distribute the actual entire end-to-end software.

You know, that open source library is not just that code anymore. It's all the things that the developer decided. So I think open source is gonna change to be more collaborative, but you're gonna have a lot more optionality in like the same functionality. Yeah. Does that make sense? Yeah, yeah, yeah. So I'm anticipating forks are gonna become more common than we realize. There's gonna be lots of deviations that are, you know, very similar functionality-wise, but maybe there's one or two critical decisions that have been made by the person who forked it-... That you agree with. Yes. And the old adage of like, well, if it has a lot of GitHub stars, then it's a good package, it's definitely gonna go away.

I think you're really gonna be able to start leveraging the models that are available to assess the quality of that code or, you know, what its core capabilities are for you, and then make that decision of maybe I'll duplicate it, you know, and write it myself or maybe I'll be able to actually pull it in and use it. So these are decisions that I don't think we're really occurring in most enterprises. They were being pushed down to the engineers, and the engineers are gonna generally do what's fastest, right? That isn't always what's best from a long-term maintainability.

Allie Howe Yes, for sure, and it feels like we need a lot of, like, collaboration between security and also engineers to be able to- Yeah like fix this problem. And that definitely makes sense, and we'll have more, like, nuanced versions of packages to solve different pieces, and I, I think that's super smart too of, like, this realization of, you know, why do we just get one patch? Why don't we get- Yeah... like multiple different patches for different functions, different functionalities, different uses of it which is a really interesting thing about, and I've seen some supply chain vendors kind of approach it of like, "Oh, here's a certified patch."

Yeah. "Like we'll just patch it in place, you know, if something isn't available." Yeah. So that's kind of, I feel like, similar, like-

Michael Davis Yeah, I think what, what we need is something like we're familiar with VEX and like open o- OSV and whatnot, these... We've built taxonomies, we've built schemas around ways to describe vulnerabilities and mitigations. Why don't we have that for patches? Yeah. Or mitigations, right? Like, you know what the best mitigation is for a vulnerability? Deleting the code. Yes. We forget that. Yeah. Right? Yeah. And so if the code is just gone, right? I didn't patch it, it's gone. Right? That's a mitigation that's actually the best. I never need it. But patching is one mitigation, but you have others. And so I think we have to look at both open source and, and commercial software.

Why are they just doing a release note with a new patch that comes out that says, "Oh, we fixed 1,000, you know, vulnerabilities"? Give me mitigation options. Yeah. And give it to me in a taxonomy and in a schema that I can machine consume. So now what I-- what that allay- allows, like a large enterprise like us, instead of it being a, the, a large enterprise having to do it for its individual teams, the developers can start making those decisions again. Yeah. Right? 'Cause we've-- with everything that's happened with Patch McGannon and the Vuln Pocalypse- like they're talking about, right? Really what I've seen happen is we've actually removed some of the autonomy of engineers and developers.

Because we've now said it's very complex, all this risk we'll make the decision for you. And I think that's gonna end up causing velocity issues with delivery of software, right? 'Cause we usually wanna put the decision closest to where the problem is, which is the engineer. And if we had something that was, you know, a schema, an ability to really communicate consistently that was embraced, we'd probably have them making better decisions. Right? When you only give them one option, patch or else, that's not a fun spot to be in, right?

Allie Howe No, not at all.

Michael Davis I mean, you know, I know a lot of companies and a lot of engineers around the world that as new vulnerabilities have come out and zero-days release and, you know, big patch releases, their whole weekends are gone- Yeah... because they don't have an option. They have to patch or else. Right? And I think that's that's not good. You have to have empathy for those folks.

Allie Howe Yes, absolutely. For sure. For sure. I know JPMorgan Chase is one of the most innovative organizations in the world which is amazing, like, given, you know, your, your size and also in being in, like, in finance as well. Yeah. How do you manage the trade-off there of like, okay, a-adopting, like, the latest technology, but also remaining, like, safe and secure?

Michael Davis Oh, that is a, a big question. -... it, it, it isn't just one thing, but, you know, I'll, I'll start with actually the people, okay? It really is, even though we're talking about AI and machines and all type of stuff, you know, people, having the right people that are willing to actually work with you, collaborate. I mean, we have some of the best people in the world on the team. And, you know, in my mind, that's, it starts there. So the way we approach the problem is the first thing, right? But I think that we've actually started embedding the way we innovate into our actual process.

So I don't know if you're familiar with the concept of Wardley mapping. Have you heard of that term before?

Allie Howe No.

Michael Davis So Simon Wardley is a gentleman, he came up with this, kind of, way of explaining innovation over time, which was you have pioneers, you have settlers, you have town planners. And the concept is very easy. Town planners is what you normally think of for a large bank, right? We got, you know, we wanna make sure that your money's always there every time perfectly. And so generally you equate that with moving slow, right? And town planners, you know, usually they move slow, right? Pioneers going crazy, right? Moving fast, breaking things. What people forget about is that middle piece, the settlers. How do you take the great parts of the pioneers and have the town planners be able to actually adopt it, right?

So a lot of great ideas for innovation have too many rough edges to be consumable. So one of the things that we've done, especially within cybersecurity at Jabil Morgan, is we have an emphasis on building settlers. So we have teams built inside the company that are there to adopt the pioneering capabilities, right? And kind of smooth out those rough edges and make them adoptable by the rest of the organization, right? By doing that, it actually allows the innovation to just be, you know, much rapid velocity of adoption, and then we get the ability for teams that do have maybe more maturity or capability that can adopt earlier, they can choose if they want to, right?

So I think we look at it more like a life cycle and an ongoing piece, but being cognizant of how we do that I think it's something that many companies don't do. They just-. Maybe they put hope as a strategy. Yeah. Right? Yeah. Like, "Oh, hope. We'll, we'll, we'll put some cool stuff in the kitchen table, right? You know, at the break room and see if any developers adopt it." No. Like, we have found that you actually have to show them, you know, here's what this new technology could do, here's use cases, here's examples. And then help smooth those adoption curves out. Not all technology is easy to adopt.

Even though it may be downloadable from GitHub, doesn't mean you can actually start using it.

Allie Howe Yes. Yeah. Exactly. And I think about the different, like, MCP servers that are out there at the moment. Yeah. Different, you know, the agent supply chain is, is growing so, so rapidly. And like you can have the same MCP server, but it's not created equal if like, you know, one developer's using it with a PAT. Yep. Like in a PAT, but one of them's using with a GitHub app and you've got OAuth behind it. Oh, yes. So it's like how do you keep eyes on everything your developers- Yeah... are doing and provide them with like a paved road, which is also like easy and not blocking for them.

Michael Davis Yeah. No, I think it's a, it's actually a great analogy you always hear in enterprise, like paved path, right? But we don't talk about the rocky road. We don't talk about the dirt road, right? Yeah. We don't talk about the expressway. So like we, we actually use some of those terminologies at Team Forger, and like the idea here is you wanna create a paved path, but the paved path isn't always the right path for everyone. And so sometimes actually it is better to spend a little bit more time to build that expressway that's gonna work for everybody-... than to give five different paved paths to different teams, right?

So being, I think, very explicit in when you make a decision of paving a path is important. So, there's an kinda old adage, I'm not really sure it came from, where I heard it from, but it's the idea that two is greater than zero, one is greater than two. So the idea here is if we have two teams that have two totally different paved paths, but allows them to deliver the, you know, value quicker to our customer, then let's have the duplication. Let's have multiple paved paths, right? But ultimately cost will probably be an issue over time if it goes well, so we probably wanna merge those paved paths into one maybe express path, right?

And so we've kind of been looking at when do we make the paved paths for teams? And two, how do we actually get down to the nuts and bolts with those teams of like, what's gonna work for you, and can we make a paved path just for you right now 'cause it's gonna be worth it, right? And so I think we're gonna see the, the ability for teams to choose their own routes with AI is gonna become I think we all know, like easier. I can generate the code, right? Why do I need to be told what to do? I actually think we're gonna end up seeing less diversity of choice because of that.

Because as the engineers start focusing more on the outcomes, they're not gonna care about like whether I have to use, you know, MySQL or Postgres, right? They just need a database. And so a lot of, I think the issues that come up around paved paths or not is opinionation by humans because they're trying to opinionate, right? Yeah. Meaning like, maybe 'cause they're trying to get a promotion, right? They wanna do this new technology or whatnot, right? And so when we kind of remove some of that because we've moved what we want their goal to be, which is the outcome, not the actual how- Yeah... what are you trying to achieve?

We'll be able to actually inform the AI agents, right? Say, "You know what? Just always choose Postgres." Right? Like, it just will not, it will be less of a discussion and except in cases where it actually matters. If that makes sense.

Allie Howe That makes sense. That makes sense. It's like a lot of reduced complexity of like... 'Cause managing differences- Yes... like, that gets expensive over time in terms of- Oh, yeah... just like bandwidth and yeah, I mean-

Michael Davis And well, and vulnerabilities, that's what we were talking about, right? Like, I mean, if you have... It's, it's interesting. There's a two-sided, you know, to this problem. One is if you, everybody was using Postgres, then all teams are vulnerable when a Postgres vulnerability comes out, right? And we don't talk about that. So like there is times where you want diversity, right? But being explicit is the key of when you want that and when you don't, right? But we don't want 100 different databases. So like there's a, there's a medium that has to be there. It has to be managed. Yes. Right? And many companies don't.

They, they're just making the decisions as they go, and then they go, "Oh, shoot, we need to refactor this whole thing and start over." and that works for a while, but for critical applications and things that have, you know, in my mind, like a payment, right? You, you s- you have to think a little bit ahead of those items before you make the decision.

Allie Howe Interesting. Yeah, that makes sense. And since you mentioned payment, I wonder if you've are following AAuth at all?

Michael Davis Oh, yes.

Allie Howe That new protocol?

Michael Davis Oh, yes.

Allie Howe Okay. Yeah. 'Cause you know Descartes?

Michael Davis Yes.

Allie Howe Yeah. Okay. Amazing.

Michael Davis Oh, yeah.

Allie Howe I've been talking to Ian as well about this

Michael Davis Oh, nice...

Allie Howe and I saw him yesterday, so

Michael Davis oh yeah, definitely been following it. And, we actually, we at JPMorgan, so we are involved in some of the IETF working groups. Okay. We do a lot with confidential computing, for example, too. And so we've been working and following, not just agent identity, but all the things around proof of possession and binding to actually, whether it's hardware or cryptographic binding. Because in m- our mind, those are the type of capabilities that are gonna be needed in the future.

Allie Howe Amazing. Yes. Yeah, and AAuth was so interesting to me when I learned about it, 'cause it was just like, okay, here's this technology, this new protocol for agents that could, could replace OAuth, that will allow agents to run without an API key- Yeah but also allows you to track the agent's mission over time. Which is so interesting, especially if you get to like, when you talk about like payment, you can be like, "Hey, I understand now why this agent or why this person was trying to buy this thing." So it's like, you know, it's additional data for the business as well as just like a cybersecurity outcome.

Michael Davis Oh, 100% agree with you. Matter of fact, like my opinion is that not just AAuth, but the idea of getting more closer to a transaction-like capability, right? So, you know, AAuths and transaction tokens are not mutually exclusive, right? Like you can... But the, the concept of having a capability or a mission that you're understanding what the, what the real explicit intent was- that allows you to actually monitor drift, right? Yes. And I think there's a point that I haven't heard discussed about AAuth yet, but like, in general, I think it'll allow us to stop talking about that part of the problem-... to be honest with you.

Which, 'cause in my mind, that's actually not the problem we should be talking about. The problem we should be talking about is how hard is it for a developer, like especially within a company or integrating with another, you know, service, to pick the right permission to have I mean, like, you know, there's many a situation where we've probably all seen it, like if you, especially if some of you listeners work at a company, you go to like request access to something, and how do you know if that read-only role is really read-only, right? What does, you know, database\_rw mean, right? Does that mean I can do something with stored procedures?

Like, and so we have such this opaqueness to the definition of what we're authorizing, right? And I think actually as we get some more consistency in the authorization, like AAuth does, it'll allow us to actually start talking about the developers and how they make decisions on how they design their permissions and their entitlements, how we actually allow others to have better consent UXs and stuff like that. Because that's where I actually see the real friction is, you know, there's definitely problems with all kinds of threats with replay tokens and, and that type of thing. But if you've granted five thousand people in a company an excessive privilege because you named it wrong- That's real world.

That's like really what happens in most enterprises is the, the developer named it the wrong thing, and people just were clicking it 'cause they assumed that was the read-only one. It actually has been read-write from day one. Oh my gosh. Right? Yes. So I think us starting to, you know, talk about the how, becoming a protocol like AOS, right, where it can just be a little bit more ubiquitous, but we have the data traversing within the context of the mission, will allow us to be able to check, was this actually our intention- or not? And I think we'll start seeing systems starting to look at that more semantic area of identity rather than just like the actual, you know, nuts and bolts of protocols.

Allie Howe Yes. That's so true 'cause I've, I've seen like OAuth scopes come back and it's like, this is the right scope, but it also does delete. Yes. So it's like, oh- Yeah... that's not like- Yeah... transparent- Yep... like right off the bat. And it's interesting, like I've seen, I've talked to different people developing their own MCP servers for their businesses, and the different design choices the vendors are making with their MCPs are very interesting. Like, I talked to David Cramer, CPO of Century, on this podcast, and he was like, "Yeah, we didn't even put the devel- ability to delete something- Right. Yes... In our MCP 'cause we just didn't want like agents to be able to do that."

Michael Davis That's exactly what we've done. We've done very similar. Matter of fact, we, I was just talking to some folks earlier today at Black Hat about like we actually removed the ability within sessions to be able to read an email and send it at the same time. For certain cases, because like we actually, those co- toxic combinations for this type of user using that agent, we don't want that. Yeah. Right? But what's interesting is why are we having to do that at the tool level, right? And it goes back to it's really hard to like design your permission kind of system, if you will, so that it's intuitive to a human, machine readable for, you know, the machines, but then auditable- Yeah and verifiable, right?

Yes. And you know, majority of the work that actually I think most security teams do in regulated companies like J.P. Morgan is about verification, right? Audibility is one part of that, but verification of did the thing that we intended to get done, get done. Yes. Right? And that's a lot harder than people realize. -... and AI's making it worse because it's, it's generating more and more it's generating easier paths for people to find those little itty bitty gaps, right? And to realize that, oh, no, actually, you know, there's 5,000 options here. AI can iterate through all 5,000. A human won't- generally, and they'll find that one that's just, you know, the wrong permission, right?

Allie Howe Yeah, this concept of verification is something I s- see coming up in the concept of, like, software factories. Yes, right. We're like, "Oh, we have to have the verification of software to be able to do that." And we talked about it in, like, the loops debate- Yeah and on stage at AI World's Fair, which was, like, very fun. But it's really hard to say, "Hey, if I have a software factory, did it actually build the right thing?"

Michael Davis Yeah. The way people go about building their verification is generally fundamentally flawed. They leak information into the actual system that they're testing. And, you know, I used to have a background in machine learning and things like that at startups that I had. And so, you know, my brain always goes back to, you know, I have a trained test and I have a validation set. You have three. If you look at how even most cybersecurity products are tested, benchmarks, you know, et cetera, you have a train and you have a test, right? And we've seen over and over again with just the AI benchmarks and how they've been wrong.

There's been errors in the benchmark, right? And, over time, the agents have gotten intelligent enough to exploit those errors to guess the right answer, even though they're not really doing the right task. They're usually asking the wrong questions. We're asking questions around did it get built right? Did the unit test complete? Did it make the API call right? You actually need to control the thing it's interfacing with too. Because it may actually have got the outcome, meaning it called the API. But you need to make sure it called the API in the right order with the right parameters at the right time, and you need to make sure it does that when you, you know, change the data.

So there's a concept called mutation testing, which many teams don't do. Most software teams don't do because it's very time intensive. But-... mutation testing is where you change one piece of the function. So you maybe change an if statement from, you know, if five is greater than zero to, you know, if zero is greater than five, and nothing else changes. And you see, do your tests fail? 'Cause they should. Right? But if they don't, you probably have poorly written tests. 'Cause when it was being built, I guarantee that AI agent had some assumptions built in of like, "When I query the Jira ticket, I'm gonna get these five fields back.".

Okay. So while it's doing its test case, only did two fields What happens? Right? And most teams I see building software factories, they're all about automating the process, not actually doing a factory-like approach to validating what's being built. And, I think if teams did that, what they'd find is that you have to build a lot smaller things. People wanna build a software factory and just do like a one-shot prompt of like, you know, go build me Microsoft Outlook. Like it'll be magic. Yeah. No. You have to actually have it build many, many smaller things. And then you have another factory, could be the same one, put those things together, right?

So it's more of a cumulative progressive process-... in creating software, which is amazingly how we do it as humans, right? Like we have sprints and sprints equal to epics and epics equal to features. Well, your software factory should be working in that way, right? It's not about one-shotting. So, but I do think, yeah, validation is gonna be key to all of this and I just think key to, you know, really the future of software engineers in general.

Allie Howe Yes, absolutely. Those are great points for sure. And I mean, when we talk about modeling software factories off the ways that our humans developers work today it's, it's very collaborative where like- Yeah... you know, one person wrote the code I have it reviewed by someone else. And if you now model that in the agentic world, well now you're handing off like delegating work from one agent to another. Yeah. So like what does that permission set look like?

Michael Davis Oh, yes. Yeah, what, what data do you hand off? And how-- 'Cause going back to what I just said, the information leakage, if you hand off and give too much information, that new agent may be actually cheap, right? So you have not only the interaction agent to agent, like what should I share, what should I not share? So a g- a simple example that we saw in real world was testing an MCP. The way the developer built the MCP tool was it was querying a database, but they built a tool in which you basically just send in the request you want, it comes up with a SQL query.

Actually, it's a SQL query. Well, the end flow that the user was trying to build in the AI agent was to deal with a single customer. Right? But really the agent could have just done a query saying, "Give me the 5,000 rows," and it would get all 5,000 rows. Well, that goes into context, right? So now if that was part of the test set or whatnot, you now have five or, you know, 100 different customers' data sitting in there. When you go to do a handoff, you now have just actually leaked data. Even if you authorize and authenticated that agent to talk to each other- Yes... you're not looking at the data.

Right? And so I think some of the newer kind of, authorization and identity related conversations, you know, getting down to the data element type of area is gonna be a great change for us because in general it'll allow us to actually know that we yes, we're allowed to talk, but I gave you too much information. Right? Yes. So handoffs are hard and I don't know if we have a good solution for it right now.

Allie Howe Yes. No, that is hard for sure. Like that, that handoff and also making sure that you're not handing off like a longstanding credential or a credential more access to that agent so that- Yeah... now we've got, you know, the combined permission set- Yes versus just like what's actually needed. And I also feel like this is hard too, when you talk about like at a large organization, really any organization turning one person's win into- Yeah... something the whole company can benefit off. Yes. You've got like shared memory, but it's like if everyone's writing the shared memory store- Yeah then you really got to monitor the rates.

Michael Davis Oh, yeah. Memory is one of the things so right now I, I don't believe memory is a thing. Okay. So what I mean by that is in almost all the cases we've seen with real systems, and we have real agents deployed, right? It's public, a lot of the stuff that we've done with our AI journey, right? It's pretty significant investment all the way down, and what we've seen over and over again is if you think you need memory, you're probably not thinking of your problem the right way, right? Because, you know, what's the old joke of, you know, if you think you need regex, you now have two problems.

Right? Like it's- Yeah... it's so complex of a thing to do that, you know, reaching for that tool adds more complexity. When you add memory, you now have to have a life cycle of that memory. You have to figure out when do you remove it, when do you inject it-... when do I delete it so that it's no longer applicable.

Allie Howe You had to remove something from it-

Michael Davis Exactly...

Allie Howe if it gets poisoned.

Michael Davis Yeah. You got it. Yeah. And so like all those things, if you go into a regulated environment like us, now I have to have provenance of who changed what, why, when, and how. And you then have to be like a scalpel to go change that one data note in the memory from two years ago, so the agent now works properly. What we've generally seen is actually the real reality is ignore the memory. Just go change your actual prompt, the skill or whatever it was that the agent was using, whatever the start of the workflow was. Like go put it into its actual context in the start of the session.

And if you maintain that, it becomes way easier to deal with, right? And so a lot of the stuff that I'm seeing that is using memory, I see a lot in personal AI assistant items. And they're remembering facts about you as your or your family or something like that. That, that's static over time. You're just adding to it generally, and then you, you know, reuse it every time. I think when we add things like when do we inject it or using, you know, these fancy algorithms for decaying things and whatnot-... it just overcomplicates the situation. And so I think what we're gonna end up getting is just, you know, we'll get better at putting the information needed and maintaining that upfront.

Allie Howe That makes sense. And it's like how do we make use of existing technology like databases, Redis caches? Like why do we need this new concept of memory in the first place?

Michael Davis Right. Well, e- and even if we do use, you'd use those, like let's say you use Redis, you now have a new component in your architecture. So now you gotta patch that. Now you gotta maintain it. Now you gotta update it. Why? You know, and countless things that we've done internally, like we're still seeing when it comes to RAG, like Markdown wins still. Right? Like it just... There is cases where it doesn't, right? But in general, if you're doing something that's, you know, a general AI agent, you know, something that's simple for a small type of user, like a single persona or two, you know, using Markdown related approach, you know, the Carpathia, LM, Wiki, it wins.

Grape is an amazing tool. Right? When you start adding in these other components, you're just adding in complexity that you now have to manage. Right? Yeah. And usually simplicity is what I've seen work at scale. Okay. Right? So like put some numbers into, you know, a concept like y- having a, a company where you have a couple skills and you're sharing them is fine. When you get to a larger organization at scale, like three skills times 45,000 developers times whatever, all of a sudden you've got hundreds of thousands to millions-... of things that you're tracking and managing. And that just becomes really, really, really hard.

Right? And it, I just don't think that's gonna, you know, work long sc- long term. It may... Right now we're iterating, we're learning as an industry. But I think y-your earlier comment that you made about the access tokens of some of these things, right? Just-in-time access. If you add memory, you add a whole nother complexity, right? Because just in time now goes to be impossible because isn't that memory from two months ago? It's no longer just in time. I'm giving you access to that data now today. Whereas if I did a just-in-time access, you would've only had access to it back then. And so you get this accumulative situation where you inadvertently are actually adding a lot of data, and whatever has access to that, even if it's a just-in-time token for a one-time access, you're giving it data from the past.

So you're adding more and more and more and more data over time.

Allie Howe So you're essentially over-permissioning it.

Michael Davis Yes.

Allie Howe Yeah, that's true. And, and you can, like, poison it so quickly. Like I- Right... I was reading this, like, Google DeepMind paper- Yeah... where they talk about AI agent traps. Yeah. Where it's like, okay, now you've got this... And, like, we talk about, like, chaining vulnerabilities together, like CVEs, like multiple, and then you've got a, you know, zero day or exploit or whatever. Yeah. But then there's this concept of like, okay, here's my agent, it's injecting context from all these different sources, and maybe all of the sources say, like, "This is the leading industry standard. This is the thing to do," and then all of a sudden you have this cognitive bias in the model- Right... that believes that.

Yep. But, like, that, that one single, like, sentence, like that's not a threat. Like, how would you identify that and put that together? Like, that seems so difficult.

Michael Davis I I think we don't have the tools for it right now. I'll take maybe a step back and, and ask a question of why are we still accepting the architecture that we have today for the models? So meaning we have a single control plane and data plane merged together with our models, right? And we've known for years within computer science that that's a bad idea, right? And there are other architectures that have been talked about in academic literature, et cetera, ways of separating, you know, the data plane and the control plane so that an injection or, you know, poisoning doesn't have an effect, right? But we've chosen not to force the frontier labs, the other providers, to give us those assurances.

And in my mind, we're we're basically buying velocity right now for new features, and we're just paying debt-... right? For this risk that's gonna come, you know, due, right? It is very difficult for teams to understand semantics, right? Like, what I mean by that is abstraction is a difficult concept for many engineers and, and just general folks. You start talking about things like meta languages you have to have to, like, analyze prompts and what not like that, it gets really abstract really quick. That's just hard to reason about. And, you know, the, the classic people will respond back to me saying, "We'll have AI agents solve that," so it'll be AI for AI.

Maybe but you have to be able to diagnose and debug those systems.

Allie Howe Yes, like when it breaks at 3:00 a.m. in the morning- Right... like you're gonna- Yep... get paged, so you have to know how to fix it, yes.

Michael Davis Correct. Yeah, it's like we've been talking about a lot of these poisoning attacks. It's not, it's... We actually look at it as poisoning is just a class of inaccuracy. Right? So in reality, if your team is building something and it has good evals-... good, you know, test data sets around things that should cause it to fail, right? It's not just the happy path, right? The- actually purposely make bad paths, right? They will be much less susceptible to poisoning issues because they've actually built a much more resilient agent, meaning it handles error gracefully. So when the prompt does occur and it's going off the rails, the system will know that it's going off the rails.

So you may not have prevented the actual poisoning, but you may have prevented the actual blast radius from becoming much larger, right? So for example, maybe the first command that it's executed on the bash tool, right, gets executed, and then the agent's able to realize, "Whoa, why did I do that? Like, I need to go back to this workflow process that I'm doing." And it will steer itself back. We've seen steering be another way that you can help with poisoning, et cetera. And so the old adage of you can never trust any input is still true. Right? But the problem is AI agents, everything is an input that you can't control, right?

Yeah. Markdown files, emails. I, I forget the name of the attack the-- but we just saw it was a few weeks ago where they were using Outlook calendar requests, text field data.

Allie Howe Ooh, I don't know if I heard about that one.

Michael Davis Yeah. So very simply, like generating a whole bunch of calendar invites on, you know, using Office 365 and putting the exfiltrated data across all those invites. And then that gets sent as an email to the user, puts the data in it So it becomes a nice, you know, command and control channel, right? Yeah. And AI agents, because of how fast they work, can do that very quickly. You know?

Allie Howe Yes. Yes. Oh, that's so scary. I was wondering, my last question was, like, I was really interested to hear your take on obviously I've talked to a lot of people, and they think that defense in depth for AI is key. Yeah. And when you think of something like, okay, like just we'll take a software factory 'cause that's the hot topic- Sure... against the autonomous system that most people are, like, looking to build at the moment. What's the architecture look like for that or the security model? It's like, do we need like a sandbox around the agent? You know, just-in-time credentials that are- Yeah... injected into the sandbox at runtime.

Do we need some sort of behavioral monitoring to make sure the agent is has like a happy path, and it's not like thrashing, and it's- Yeah... gets course-corrected? Like, what is the pref-- Like, what does our stack look like for AI security at the moment, and then, like, what's maybe missing?

Michael Davis So, I think our stacks right now are incomplete. Or actually, I'll be maybe more blunt. W-where could you point me to a good reference architecture? That's a really, you know, legitimate question is like, what does good look like? Right? And how do we get people in the industry to build good? And I don't mean build it as in I'm gonna go build a, a, a cybersecurity product and get VC and sell it. Yeah. I mean, like, the way we used to do it back, you know, years ago, where we were all collaborating together, open source, to develop either a protocol or something that we could all learn from, right?

So I think we don't have that, first. Two, we don't actually have, in my mind, any like example kind of systems that we can all use as like the canonical reference of examples, right? So like, what's the, the default AI agent example we should all be using when we draw on the whiteboard, so to speak? Is it personal AI assistant? Is it a workflow that sends an email? Like, what is that? Now, at J.P. Morgan, we built a couple of these-... so that we can actually communicate between teams much easier. Right? Because it's, it's a way to have that conversation around, okay, what type of security architecture do you need?

That starts with what are you trying to do, right? The kind of classic threat modeling. What are we trying to do? What are you building? Those type of questions. But if you don't have some grounding around just scenarios to get like, okay, how are you like this or not like this? Right? So I think that, that's the second thing. But when it comes to actual, like, technical architecture, what to put in, I think we're only solving part of the problem right now. The, the problem that we're solving right now is mostly on, like, what I'll call model safety. We talk about poisoning and whatnot, which is still kind of about attacking the model ultimately.

Harnesses are more important than the model, right? I think we kind of agree to that. But we're not looking at it as a complete system. So, like, the example that I use internally is look at how crash safety in cars evolved, right? We didn't just add seat belts and then we walked away. Right? That's adding a sandbox. Yes. And walking away. Yes. Right? What we had to do to actually, you know, start saving lives was we had to have seat belts, we had to have crumple zones. We had to start actually then looking at, well, hold on, are the roads safe? What's the speed limit?

How do we look at how we do this? How do we look at who's driving and what their licensing is? So you end up having, like, this, these multiple dimensions you have to look at. I think right now the velocity is so much on, like, building things, we're not looking at all parts of the problem, right? And, I actually think where we should be looking at is what robotics has done. So, actually just this weekend was the Berkeley Agentic AI Summit, right? And so I thought, you know, wish I could've been there, but they live streamed it, which was great. But there were some awesome new things that were talked about on there about how robotics is looking at uncertainty.

Right? And ultimately, what, what your question was about in my mind is, we've got all this stuff, we're uncertain of, like, what we should have to protect it, right? Yes. And so to address that uncertainty, how does a robot determine if it's safe to move its arm up or down? Right? And the way, there's just been a bunch of research on this, whatnot, but they look at what are called four areas, right? Look at the, the state of the actual system itself. So this would be the equivalent of us saying, "Okay, what is the state of my AI agent?" Right? Is it in an okay state that I know of health-wise, right?

They also, a robot will look at the environment itself. Like, did the environment around me change, right? An example of that could be like in our world would be do I now have more tools than I had available before? Do I now have more network connections open? Do I now have, like, other things that we can observe, right? But you also look at the state of the actual decision-making. Is the AI agent still making good decisions? Yeah. Maybe it's made 30 tool calls already. You can evaluate that with a classifier and say-... well, did it make the right tool calls? Is it still moving semantically towards its goal, right?

If it's starting to deviate, then you know what? Maybe I stop the thing. Yes. Right? And then the last thing they look at is the actual uncertainty of the action itself. Right? So I think we need to start looking at cybersecurity, especially within AI security, about the actions, not just like the permissions or what I am entitled to do, but actually what is the action that it's done, what actions has it already done? And then be able to leverage some of the more forecasting-like capabilities, right? 'Cause when a robot's going to move its arm, it's generally predicting what are all the outcomes that potentially could happen next.

We very rarely do that in cyber. Sure. Most of our detective stuff is binary. Right? Like, is this tool call allowed? Yes or no? True. It's never that simple. No. Right?

Allie Howe That's such a good point.

Michael Davis And so if we start thinking like that, I think the ultimate tech stack that we build would be something where we're coupling the agent with the sandbox, but we're actually coupling the tools, the skills, et cetera, in cases where we can, right? And we're putting those things together, but then we're making sure that if the things around that sandbox change, we make a change in the sandbox. If the sandbox itself, meaning maybe the rate of tool calls, the, you know, number of files being written per second, those type of things, those metrics you can observe, if that's changing, do we then change the controls around the sandbox?

Yes. Like, you know what? I see the thing writing a whole bunch of Python files, I'm gonna turn off web access. Right? Worst case that happens is it does a web fetch call, it fails. Okay, well, if I determine that that was actually an okay web fetch call, turn it back on and let it retry. And so I think ultimately when we're designing systems, we need to be looking at it more from a resiliency perspective, right? Yeah. Like retries are not a bad thing. We think so, I think because we're humans. Like if I do an action and it doesn't work, oh my gosh, that's a bad thing.

I maybe it could take me another hour to redo that work. But if an agent's doing that, you know, reasoning in a, a second or two and it makes a bad decision, you can steer it, it can try again. It can do 500 tries. Yeah. It's still faster than a human just doing it once. So if we build that system to be able to handle retries and things like that, I think it gives us more capabilities of what we can do that'll change the reference architectures that we'll build. So if anybody's interested in building some reference architecture, I'd love to work with that and, you know, release something.

Allie Howe Yeah. Yes. Absolutely. Well, unfortunately, that's all the time we have. Yeah. But I really enjoyed this conversation, and where can people connect with you if they'd like to find your continuing conversation?

Michael Davis So there's a research page on jpmorgan.com. Okay. You can go through there. But also LinkedIn is a way to get to me or on X.

Allie Howe Amazing. Yeah. Awesome. Well, thank you so much for your time.

Michael Davis Great. Thanks, Allie. Appreciate it.