Allie Howe: Hey, my name is Allie Howe. I'm the host of the Insecure Agents podcast. Welcome to another episode. Today, I'm super excited to have on Max Pollard. He's the CEO and founder of Cotool. Cotool is a company that's helping defenders with AI-enabled agents that can help with automated defense and response. So super excited to talk to Max today and how he's seeing that market evolve and how Cotool is helping his customers. Max, would you like to give us an introduction?

Max Pollard: Yeah, sure thing. And Allie, thanks for having me. S- long time listener, first time caller, sort of spiel, so really happy to be here. And yeah, like, like you said we're, we're helping teams scale primarily detection and response. And, we work with some of the most technical teams looking to gain leverage and, and scale their teams beyond headcount. And so yeah, really privileged to work with with some really sharp teams on lots of different use cases. But, but yeah, there's, there's tons we can talk about on that front, so excited to dig in.

Allie Howe: Amazing. Yeah, well just to set the stage, one thing that caught my eye recently that I thought was interesting was this article from a company called Irregular, which I guess has been helping the foundation labs with their sandbox testing or their testing of like, you know, Exploit Bench and all these different evaluations that they're putting their models on where they're like going into keeping like sandboxes and everything. But they put out this whole report about like the end state fallacy is, was what they're calling it, around like, okay, like maybe attackers are more enabled with AI today, but in the end, like we might equilibrium or equal out in terms of like attackers and defenders will be both equally enabled like with AI.

But in the meantime, that article called out, okay, there's a huge need for new technology that differentially enables defenders over attackers 'cause it sounds like attackers have the edge. Is that something that you agree with today?

Max Pollard: Yeah, I think broadly speaking, and you know, I, I wouldn't be starting a company and, and working on on something on the defensive side if, if I didn't believe that that, that we need to equip defenders with newer better technology. And it's kind of a boring answer, but I think it's the right answer, which is, you know, there, there's a reason anytime new technology, you know, comes out that like attackers can adopt it first, right? Like they don't have to go through a 12 months legal privacy compliance review to bring n- new technology in, but some enterprises do, right? You know, you can think of like large health systems, for example.

And so, you know, d-defenders are always going to be hampered by, you know, due process and, and things that are in place for good reason. So I think that's probably the reason why, why adoption is a little bit slower. Now we can speed some of that up, right? Like we can, we can, you know, cut corners to, to, to, to make sure that we get the right technology at the right time. But, but I think that, yeah, that's the, that's the boring but also, you know, potentially correct answer is it's just gonna take us a while to get this in place, rolled out working with other tools.

Whereas an attacker can say, "Yesterday I was using Burp Suite, now I'm using an agent," right? Like I have no boss that tells me what I can do and not do, right? I just, I just hot swap the tools.

Allie Howe: Yes, absolutely. And I remember talking to Alex Stamos on the podcast during RSA that March, which seems like forever ago. And one of the things that he told me which really stuck with me was, you know, back in the day, just like Fortune 500s used to have to worry about a zero day. Now it's literally everyone in- including like the local school down the street or like the grocery store or whatever, like teams that don't have these like very large, expensive, empowered defensive security teams. And so for someone like that or even like critical like, like infrastructure for like the city or whatever all of these different like institutions and not necessarily big tech, like have to worry about defensive technology against like AI-enabled attacks.

So like, yes, I totally agree that it's gonna take time and like education to roll o- this out. And that article that I mentioned from Maria like called that out as well as like that's kinda like one of the reasons it just, it's lagging is that there's education and tooling and like upskilling lag here. But anything that helps make this new technology less abstract and like more usable and, you know, it's really interesting seeing like, you know, agents roll out. The very first kinda like way for agents was really like coding agents, you know, targeted like at very technical software developers. And it's like, that's fantastic, but like for security, we have to, you know, not only enable that group of very technical users, but maybe folks that are like less technical as well.

And I'm wondering like at Cotool, how do you like manage that between like making sure like you're serving customers of all levels of technical ability?

Max Pollard: Yeah, it's a really great, great question, and I'd be lying if I said we had it totally figured out. I mean, this is something the co- the co-founders and I like like grapple with every day is, is, you know, we've got some very, very technical customers that want all the bells and whistles and knobs here and there, you know, the, the traditional power user all the way to y-you brought up a great example, like everyone has to care. Now we work with, you know, a nine-person, you know, finance startup, right? And so they, they're like, you know, rightly so concerned that, hey, you know, we're, we're, we're now like getting, getting fuzz, getting attacked, like we need to worry about this stuff.

And no one on their team is like, they don't have a security hire, right? And so the, the primitives need to make sense for, for them as well. And so, y-you know, I think there's like kind of a combination of, of things you can do. You know, previously y-y-you would say like, "Hey, we're for one of those camps of users," right? Like, you would just pick a lane and stay in it. Nowadays, I think you have to say, "Here are the rails, here are the paved roads, and then here's the off-ramp," right? Like, if you want to, if you want to take a detour, like there's controls here for you as well.

And so two things for us are very, very important. One is it's easy to ship the feature. It's hard to make sure that when the feature lands in your product, it is explainable, usable. It's, it's very clear that what it's for, and not just to a user logging in, right? Like maybe to an agent that is setting up your product on behalf of some user, right? So that's very, very hard to get right but, but is super important. The second is just, you know, like on the documentation side. So having the right altitude in the right places of here's how things, you know, work at the nitty-gritty level, but then also at a higher altitude, like, you know, here's the product's primitive and why it exists and what it means for security teams.

And again, like I, I-- we're starting to move into this world where docs are for agents right? But like that agent is turning around and summarizing stuff to a user. And so, you know, if they need the primitive level explanation, I think, think that's important too. And so I rambled a bit there, but like the, the, the thing that we always pull back to is, you know, rails for the first time users, paved roads for the first time users, and then like, you know, like off-ramps for for power users, right? Is, is kind of the, the philosophy that we adopt.

Allie Howe: Yes, for sure. Yeah, that makes, like, total sense. I think that's a great, like, product experience is, like, starting so all like the paved road and then, like, if you wanna add the customization automation like later that makes sense. When you talk about, like, creating a paved road is that something that you derived after, like, a bunch of, like, different customer conversations and, like, use cases of Cotool? Or was that something that, like, after maybe, like, spending time, like, on security teams, your own, like, experience in security, you're like, you know, these are, like, the five workflows or the five things, like, people need to have as part of, like, their defensive or automated response strategy?

Max Pollard: Yeah. That's a-- I mean, that's a really interesting question. I, I think it, like, you know, it's really hard to distill it to, like, a key insight or, you know, th-these two environments that did it really, really well. I mean, in a, in a past life, I spent my time, like, embedded with lots of different security teams from, you know, the Coinbases and the Stripes of the world to, like, the Chubb Insurance and the MassMutual. So, like wow, I just randomly picked four companies in FinServ which is, which is really odd. But anyways, like, all different shapes and sizes. And so you'd notice different approaches that worked for different shapes of organizations.

And so there's, there's kind of like a wealth of ex-experience that we draw on there. But, but I think, like, there's also kind of a trailblazing aspect to this where, you know, you don't start a company if you think, like, the way that all of those companies are doing things is A-okay, and, like, there's no problems here, right? And it's not a knock on the people or the process, it's a knock on the tooling, right? Like, there needs to be a better way. And so I would say the, the problems that we are trying to solve come from experience. The solutions kind of come from novel thinking, right?

And so one example of that is, you know, for a process that we help folks a lot with, you know, triage and detection tuning, right? The, the, the past experience was like, okay, you go out, you gather a bunch of analyst feedback on, on, you know, the data involved, and then you use that to then tune the detection, right? And so, like, a very, very manual process. You know, and, and, and now, you know, you could, you could do the same thing today and say, like, "We'll just use AI for the, you know, detection tuning part." But, like, you kind of are like, okay, the paved road is a road in which you can use AI both to help label the data and also do the detection tuning and just have a human approve, right?

And, and so, like, short circuit all of the work involved minus the review, right? And so in that sense, it's like, okay, you have to use novel thinking to imagine an approach by which you can do that accurately and well. And so, you know, we have lots of stuff in Cotool that, that helps us achieve that. But, but I, I would say kind of it's a, a mixture of, of both experience and then novel thinking.

Allie Howe: That makes sense. Do you have any agents that, like, are able to actually, like, take actions, like, on behalf of, like, security teams, like, maybe, like, shut down services or, like, actively, like, respond to incidents?

Max Pollard: Yeah, we do. There's we-- A-a-again, like, one of our core tenets is is flexibility and control for these teams. And so, you know, I'll start by saying we do, and then caveat it with, like, here's exactly how. For an example of, like, you know, revoke Max's session, right? What we see teams do oftentimes is start with a read-only approach. And, you know, th-this is probably near and dear to you, but, like, we go very, very granular beyond the integration as to what specific API calls get attached to an agent. So we're, like, very governed from that perspective. A user can say, "Hey, this is the, you know, agent that reads from logs and can also read files in GitHub."

That's it. It's deterministic, right? But let's say, you know, you gave that agent a, a tool to say, "You investigate, you know, suspicious logins. You have the ability to revoke a session." What we most often see with teams is, like, that happens seldom enough that that's something they still want hands-on keyboard for every time it happens. And so we have this concept of, like, hooks in, in Cotool where the agent will either in our app, in, you know, the messaging protocol of choice, like a Slack or Teams, go find the right Cotool admin and say, "Hey, are you... You know, an agent's about to take this action, like, need you to approve," right?

So that's kind of like the, the, you know, the walk phase of the crawl, walk, run. And then we have folks, you know, measure the number of times. And this, y-you can really only do this at scale, right? Because you need a sample set that, that holds out. We have folks that measure the number of times those right actions were correct. And you know, if they're like, "Hey, ten times out of ten it's correct," you know, ideally you wanna see fifty, a hundred, you know, more instances of this, then you just say, "Cool. We don't need a human approver anymore," right? So that's the run.

And, and when folks are, are... There's a lot of marketing noise to say, like, "Hey, take the human out of the loop," right? And it's, like, really enticing because we're like, "Cool, no humans for anything. Great." like, you know, it's like a, a, a an interesting vision to, to, to live in. But, but I think in reality, right, like, we're not really that close to that. And so, you know, we've seen very few people hit that, like, run, run phase. Yeah, the trade-off's just not quite there.

Allie Howe: Yeah, so I'm pretty sure I think a lot of people are struggling to give their agents both the security, autonomy, and capability to actually let them get to the run phase and deploy like production like agents at, at scale, which I think we definitely need to get there, though, just because, yeah, the response and machine speed is just like critical with some of these like incidents, especially with like, you know, agents escaping sandboxes and doing like, you know, a ton of damage. Like being able to revoke something or respond to like an intrusive, like an attacker, like AI attack, enabled attack is huge. So I feel like like I don't know how we get folks to that run phase.

Do you have any ideas? Or like what's their common blocker? Like they just can't trust the agent. They're not sure if it's gonna use the right permissions. It's just like the LLM's judgment call. Like what is the blocker to get to the run phase?

Max Pollard: Yeah. I mean, if I were fl-flipping the situation on its head and saying like, "I at Cotool are, are, am making this decision," I think the things that I would care about is, you know, ultimately it's a business decision, it's an economic decision. Like do we take this action? What's the, what's the economic risk of taking this action? And so, you know, for me, it's like y-yes, the issue is trust, but like to get a little bit more specific than that, like the issue is trust that the agent can determine at runtime like the server that I'm about to, about to kill, right, or network contain or whatever powers a million dollars a day in transactions, for example, right?

Like you, you know, you're thinking about like this is, you know, app.stripe.com, right, that you're about to take down, right? And so I think that's like, that's a trust problem, but also I think that's a data annotation problem, right? It, you know, it's, it's like the agent can sometimes infer like here's what this server does and like intuit like here's maybe the economic value of this being on hour by hour. But really it's kinda up to humans to, to, to, to guide and give the context there. And so there's a lot of work to be done on the, on the setup side to say, "Look, agents are great at crawling the environment and building intuition about certain things, but ultimately I have to review that and say, you know, 'You got this wrong.

This is a dev server. It doesn't matter,' right? 'This is a prod server.'" Like yeah, I think there's just a lot of work to be done there to be confident that like you can just, you know, take the leash off, right, and, and, and let these things run wild.

Allie Howe: Is that like simulations and like testing? It kind of sounds like evals and as like Cotool grows, is that something you'll get like data on as more and more people try to like, you know, adopt this technology and do that themselves within their security programs?

Max Pollard: Yeah, I think, so I think the data annotation part is, is maybe like in its own bucket, right? Like I, I, you know, organizations just need to know... And agents can help with this, right? Like agents are very, very good at like annotating and saying like, you know, "Here's a customer financial transaction database," and you can go, "Yep, c-confirmed. I can, I can confirm this. I see what's in it," you know, and, and, and kind of move on. But I think I think, you know, when you, when you get into like once that data is annotated, that's like the very, very helpful part. You can, you can then kind of start to bootstrap like okay, did the agent do the right thing under these circumstances?

And I think you would, you would get pretty tight like and, and solid evals if you had that data annotation up front. If you don't you could still run evals, right? But, but like you'd kind of be like post-labeling and saying "Okay, you know, this thing did this action and took down a broad server, like it cost us X amount," right? And like de-emphasize this behavior. But I, I think it's just like you're not gonna be able to, to as a business gather that much data because it will be so impactful to your business. Like it's taking things down left and right, that you will never, never be able to build a set large enough to say, "Here's what to do and what not to do."

Like someone at the, at the company is gonna say, "Kill, kill this experiment," right? Like, "You've taken down this broad server three times." and so I think the annotation is kind of the bigger part. Now on the eval side, generally, you know, I think there's tons and tons of work to be done on the everything leading up to the decision point, and nothing blocks us there because you don't have to, you know, have the, the right action in that set. You can just say, "Hey, did everything leading up to the decision that the agent said, 'I want to go,' you know, network contain Allie's host was everything sound and correct there?"

And, and at Cotool, we pour a ton of effort into that, right? It's based on the instructions of the agent, the entire trajectory, you know, we evaluate on several different things, right? Like, you know, was it cost efficient and accurate, right? Are we just-- are we burning through a crazy amount of tokens to get... Are we, are we, you know, making a toothpick out of a tree, right? And then, and then, you know, the second thing is like how well did it adhere to my instructions, right? The, the instructions that I give. Again, this is somewhat unique to Cotool. We, we let our customers build agents and customize agents, and so, you know, we're less so saying like, "Here's an out-of-box thing that we keep a static prompt for." and so this is very important on the Cotool side as well, but also for folks building their own agents, you know, that, that will be important as well.

And then lastly, like yeah, the, the binary end result, like do I agree with how the agent dispositioned this? And that can be is it a false positive, true positive? It can be, you know, do I agree with the risk score for this maybe insider threat use case? You know, it can be a number of number of different things, but we pour a ton of, ton of effort into that so that, you know, our users can see when eval scores start to dip, right? And they need to maybe take a look at an agent. And this could be happening because of a, a plethora of different things, but it's really important when you have 20, 30, even 40 agents in, in production on the defensive security side to know when agent number 17 is not, you know, performing the way we thought it should be.

So, yeah.

Allie Howe: Yeah, super interesting. And I feel like that connects back to what we were saying earlier about like, hey, like, attackers are more enabled today, but defenders are sort of waiting on this, like, education and tooling and, you know, n- maybe this, like, new data, all these, like eval sets that need to be run over time to help us be confident that we're actually gonna shut down the right thing when we actually get to that run phase and turn that on. What does that timeline look like in your opinion of like how much data do we actually need to gather? Like if that's like the key blocker to getting into that run phase.

Max Pollard: Yeah. Oof, I, I wish I had a like... You know, my error bars on this one are, are massive. So I'm hesitant to make a a static prediction as to how much time we need. Yeah, I think the tricky thing is, so like we do evals internally for our customers. Now we also do public evals at Cotool, which we basically at-- use a mixture of synthetic datasets and then also real datasets. But, you know, to create these real datasets, it's expensive. You know, you need to spin up an environment. You, you know, and, and like they try as you might, they may not be one-to-one with, you know, each and every enterprise that you support.

And so you have this problem of like, you know, you're at-- like in order to build the right size of dataset, you need customers to like share their crown jewels, which is their data. And, and like that's, that's not gonna happen overnight. And so, y- you know, from our perspective, it's like if you can do, you know, a great job with enough of the plumbing and say like, "Hey, this is incredibly anonymized," which is like very, very hard work, you know, you could potentially get to it more quickly. But yeah, I, I couldn't tell you whether it's four months, a year but hopefully, hopefully it's closer to that, that, that month, month figure.

It's just hard. It's hard and we're working on it, but it's hard to say when, when it'll be solved.

Allie Howe: Absolutely. Yeah, and I'm so glad that you're working o-on it, and like, I, yeah, I feel like that's, like, a question that no one really would know the answer to. And it's very like, you know, vague and out there. But yeah, I feel like there's that, like, problem of like, okay, like what's the new data that we need, the new education piece for this new technology? But then, like, the classic thing that we're seeing with AI as well is, you know, people are trying to like, you know, go and drop AI into their existing, like, code base or security program or whatever, but they haven't gotten the best practices right to be able to like e-effectively enable that, like, transition.

So, sometimes like, you know, like you said, like, you know, data is not labeled correctly or infrastructure is not set up in the most clean and way that makes, like, that makes sense. You can't just like drop AI on top of it. And it's like what I'm getting at here is like, you know, maybe it would be easier to like, have AI, like, make a mistake or like drop not drop a database, but just like shut down like a server, like you were saying. If it's like, you know, app.stripe.com, like if I drop this server, there's another one that's gonna be spun back up because I've got the infrastructure and my cloud security program is set up in such a way that we're able to fall over and feel very confident that we can shut things down and spin things back up.

But if you're not even there yet as like an infrastructure and security program at your company, then of course you're not gonna feel confident like taking a risk and like deploying this like defensive AI agent that's like gonna be able to like go to that run phase like we were talking about and shut down a VM. And so like I'm not sure if you're seeing any of that in the wild while you're talking to customers where it's like, "Hey, like, this would be so great if you had just like done this like tech or this work that you're supposed to do like months ago. But like here I am like trying to s-sell you this thing that's like gonna change your whole security program and I'd love to help you like with this new technology, but because you didn't do the prerequisites, like we're gonna have to like, you know, circle back and go all the way back there before we even get to where we want to be like to start with."

Max Pollard: Yeah. I mean, I, I think like that's, that's a really interesting take. Like the, the almost like the, you know, taking the Ne- Netflix chaos monkey like approach to not just like SRE prod outages, sort of stuff, but like security, right? Like, hey, assume, assume like an agent's gonna roll through town and like knock some buildings down. And like we have to be ready for that, right? Yeah, I think so, so there's, there's a couple things here. I, I think a, a good way to like think about that is like to cleanly separate maybe like AppSec, ProdSec from corporate security. On the, on the, you know, AppSec, ProdSec side, it's, it's just a clear like cost reward like function, right?

Okay, what's the cost of having... You know, if, if, if I don't have a backup app.stripe.com, like, you know, it's, it's probably bad news, right? Regardless of, of the scenario. But, but also then like, okay, you know, if I like I, I now like maybe I, I don't have like a... I don't know what Stripe's like product portfolio looks like. But y- you know, like maybe something s- less important, like what's the value of keeping, you know, hot, hot like infrastructure running for this sub-service versus what's the like cost if it goes down for 30 minutes, right? And so like it's a lot of... If you do that for all the products you, you, you run or own or, or infrastructure you have, like, you know, it, it's like a pretty complex m- m- modeling thing.

But I, but I do agree, like we have to try, right? We have to try and say like, "Cool, you know, we're gonna measure twice, cut once, and like see what happens." Now, on the corporate security side and by the way, I, I totally was like long-winded. I agree with you on that first part and didn't really offer up any solution other than like measure it, you know. But, but on the CorpSec side, I think that the-- there's a little bit of trickiness here in that, like, you, you don't own the infrastructure, right? Like, I'm using CrowdStrike on the endpoint, right? And so if I isolate a host and like, you know, network contain a user, you know, I'm-- it's up to them really how long it takes for me to get spun back up.

And there's stuff I can do to make that a lot easier, right? Like make sure I'm not having to log into the UI and do it. But, this actually happened, you know, I think like when we were getting started as a company, it was maybe a year ago. Like I network contained myself on a demo. And so like I dropped from the demo, my co-founders are like, "Oh, that's funny." Like, you know, he accidentally hit approve. He meant to hit reject. But it was a good example of like, okay, like it's gonna take me thirty-- back then, it's like gonna take me thirty minutes, forty-five minutes to like get back online.

And, and that's not through like anything that, you know, we were doing. It's just like that's the limitation of the tools we had at our disposal. And so, yeah, I think it's like it's a brave new world. We have to be ready to accept that that is the cost of doing business. But, but yeah, I think for the value you get out of like, you know, being able to react quickly there's some scenarios where it's just so clear that like you should allow the agent to act and accept the consequences.

Allie Howe: Yeah, that makes sense. That's a great answer. And I was also wondering, as you're deploying agents like into production, do detection and response for customers do those agents typically work on behalf of those individuals, the users, the humans, or do they run in the background? Do they have their own identity so that they can just like act autonomously when they need to, to react to an event where like I said, they might not be present?

Max Pollard: Yeah, it's a great question. So within Cotool each agent, you know, has, has kind of like its own identity. And then like for, for certain-- Maybe I'll, I'll back way up and say like, here's how we do RBAC in, in Cotool because I think that's maybe the, the clearest way to explain it. So unfortunately, the-- at the integration level the connector level, we are limited by what the application we are integrating with supports, right? So this is a mixture of, you know, typically OAuth or API key. And some platforms I used to work at, at Okta on the customer identity side, so this is like near and dear to me.

I, I used to help pe-people build, you know, authorization in their web apps. And so, you know, some, some apps are very good, and they give you fine-grained controls, and that's great. Other apps, they're kind of like, "You want read or read write?" and it's like, you know, so be it. And so how we do this in Cotool is we say, okay, just because you gave us a connection into a tool doesn't mean like all agents should be able to use everything in that tool. So the first thing we do is we say everything is at the almost API endpoint level, like explicitly granted to an agent at config time, right?

So I integrate with GitHub but just because I integrate with GitHub, it doesn't mean agent number one can, you know, issue pull requests, right? Like maybe it can only read files in GitHub. And so there's kind of like that first layer, right? The second layer is like users within Cotool can also be granted sub tools of an integration. They don't have to get like all of the functionality, and they also can be granted the ability to, you know, only have access to a subset of agents in Cotool. And so this keeps like the data boundaries fairly sane, right? And then, you know, the, the other thing that we do is, although the agents use the creds in order to like call these tools in Cotool, we have audit logging that says like, "Agent acted on behalf of," you know, a user or some automation, right?

Like this is an agent that is not invoked by a user. It is invoked every time you get a web hook from a different system, right? And so we back out like very granular audit logging on our side so that, you know, folks can, can transparently see here are all the agent-- actions an agent took and whether a user prompted them to do that, or they're just kind of occurring in an automated fashion.

Allie Howe: Yeah, that makes sense. Yeah, that audit log is, is key. And so is like keeping that log out of reach from the agent. I know in like the Hugging Face incident, they discovered that the agents tried to massage their transcripts such that an observer wouldn't be able to tell that they were like cheating to obtain the flag for exploit gem or whatever cyber evaluation it was. Which, yeah, is, is kind of alarming, and I feel like well, like the saving grace they said was like, okay, well, the chain of thought reasoning was intact, so you could see. But it's like, okay, well, what if an agent next time goes out into the web and does like a web fetch and finds like an article on this incident and goes, "Oh, well, the way I got caught last time was I left the chain of thought reasoning intact, so maybe I should go obfuscate that too."

And you can see how this like improves over time, where it's like maybe like agents like develop their own like language to obfuscate what they're doing. So like eventually, like I feel like that's just like not something we're gonna be able to, to trust. We basically can't even trust it today. So having that audit log be somewhere where the agent itself can't like modify it, like, like super key. And I think that's something that like in these like, especially like AI, like agent related security incidents, having that log be quickly, readily available and be data that you can trust is super key. Is that data that your agents are looking at too during like investigations and response?

Max Pollard: Sorry, you, you mean like the, the data being like their own audit log or their own traces or, or what data specifically?

Allie Howe: Like, I guess if you, if your like agents were deployed and monitoring some sort of maybe like AI like enabled attack or like, just like take the Hugging Face incident for example-

Max Pollard: Yeah...

Allie Howe: where like that agent was in your production infrastructure and you see all of these like actions like taking place. Or maybe it's your own like internal agent that's like gone rogue and you've got access to your internal like, you know, company logs of like, here's all like what the agents are doing. Like is that in like increasingly like a data source, I guess, that teams should be looking at?

Max Pollard: Yeah, 100%. I think, you know, there's a reason there are several new endpoint companies that all raised a boatload of cash and are, are pursuing this. The process level telemetry the, you know, like the traditional endpoint approach just doesn't work quite as well for, for agent behavior. I mean, like, it is quite fascinating to me that it's like what-- like, what's the best way to find out why the agent did what it did on a user's laptop? Well, there's this just like this JSONL file that contains like all of the chat trajectory. And like for-- in most cases, you know, CrowdStrike, SentinelOne are, are looking at that, right?

And so it's, yeah, it's imperative, right, to, to understand that that, that data source. It's just unfortunate in that it turns out like it's a huge, huge volume. I mean, I think like, you know, we're a very, very small company, and we're doing like a couple terabytes a month in, in that alone, maybe more. I, I like, kind of... I remember looking at this a while back. But, but yeah, so it's like, it's not something that for like, you know, a, a super, super large organization, like you can necessarily inf-- afford to ingest all of. And so yeah, it's, it's it's scary to think that like things could be modifying this if I need to go hydrate or look at past ones, and I hadn't stored it somewhere.

But yeah, it is really, really important that, that you're able to look at these. But, you know, at the same time, like you can kind of still, for the most part today, piece together like the story of, you know, what a lot of these agent trajectories are trying to accomplish. You can't get 100% coverage, but just from, you know, the existing telemetry. Like, a lot of the stuff that's interesting at Cotool that our detection agents will find, you know, six months ago, a year ago, it was like user did X, right? But nowadays, it's like, you know, user on sales team I'll try not to be too specific here, like is, has like unintentionally exfilled a bunch of creds to like a largely open AWS account, right?

And so and you're like, okay, this is clearly like a user telling its agent to like go deploy something or like, you know, its agent saying like, "I'm gonna be helpful and create a website for you where you're, you know, pulling in prospect data and things like that." And it's just like, okay, you know, I-- we can pretty clearly present like, here's what happens. The user is conversing with the agent. The agent is going to de-go deploy something. So, yeah, I think this stuff gets much, much better if you're able to see that telemetry and something that teams should focus on smartly looking at i-in terms of like a number of, of, of, of different things on the investigation side, on the detection side on the threat hunting side.

Allie Howe: Yeah. For sure. And to circle back to the fine-grained access problem that you were talking about before. So something that I've heard from multiple people that I've had on the podcast and just like in day-to-day conversation about people that are building agents, where it's like, you know, I'd love to limit access to this, like this tool or this service, but I can't because their API is completely coarse-grained. I'm just gonna get an API token that has the ability to basically do everything. And that's kinda what happened with like the PocketOS Railway incident, where they had this like god mode credential and ended up like dropping a database accidentally.

And so having these like static long-lived god mode credentials really just like isn't an option with agents. I'm curious like how you're solving that. Something that I've been doing on my end to like solve this is like I ran this with the Luma API, where I had like just a static API token from, from Luma. Like they have a read-only MCP, but if you actually wanna like do like updates and, you know, creates and stuff with Luma then you're stuck with the API, which gives you kind of like a god mode credential. You can't scope it. So what I did was like use like Keycards like SDKs to create my own like MCP server and roll the Luma API up in it.

And then with Keycard, I can apply like policy on the MCP tools themselves which is like super helpful because now I'm actually able to get like you're only allowed to like read from Luma, but then like the one update feature from Luma that I did wanna have, and I didn't want you to be able to delete an event, which like the god mode, you know, credential would allow. But I did want you to be able to like update an event, for example. And like that one tool was allowed. And maybe I've got like, you know, a policy that says like only certain people on the team can do it under certain, certain circumstances or whatever.

But, that's how like I've been solving it, but like wondering like how you are solving it or you're seeing other people solve that problem.

Max Pollard: Yeah, it's a great question. And by the way, big fan of like everything that Keycard's doing in the space. I think there's the, there's the there's the cop-out answer on our side, which is like, you know, we serve security teams and, you know, luckily that means that the subset of connectors, integrations, et cetera, are usually pretty good about at least giving us like enough to say, "Hey, I'm confident that like this, you know, hopefully ephemeral but sometimes long-lived cred is like finally scoped enough," right? So that's, that's kind of the cop-out side, right? The in between side is we, we roll it internally. And so if there's certain scenarios where someone's like, "Hey, I really, you know, like for this app, like here's a good reason why, you know, we need to scope in a, in a finer way," we build the policy engine right into, into Cotool.

We try not to do this, you know, too much and like if it ends up we having-- we're having to do it for hundreds of apps, then like we'd probably reach for something like Keycard, right? Like we don't want to build that layer, you know, internally. But, but you know, as of now it's, it's, it's, it's worked totally fine. And then I think like, you know, going back to like the original, you know, cop-out answer and then flipping it on its head is, is I think like when you're dealing with, you know, the, the applications that, you know, developers internally will wanna use or like the sales teams will wanna use I think you do need to reach for something like Keycard because it's like, you know, Luma's roadmap does not include fine-grained policy controls on their API.

They're kind of like ship the MCP, we've unblocked the productivity use case, like job done, move on, right? And so, you know, when you start to get into the land of productivity and, you know, it gets, it gets a lot dicier from the security side because that's just not what these companies are built to prioritize. And so you, you kinda need some, some middleware or something in between to, to kinda help out.

Allie Howe: Absolutely, yeah. And this is something that I think we're, yeah, definitely seeing teams that run into this problem like, "Oh, yeah, maybe it's, like, not that big a deal." But then like as you start to scale and, like, add more resources and tools and have more users interacting with your agents, it becomes, like, definitely, like, more of a, a problem, especially as, like, agent workflows become, like, more complex, and they start to span, you know, multiple agents and multiple applications and resources, and even, like, enterprise, like, legacy systems that are sprawled across, like, different cr- clouds and different environments. Making sure that, like, policy is applied and your delegation chain remains, like, enforced and audited all the way through definitely, like, a huge, a huge challenge.

But having that, like, that audit and having that record that system of record is so, so important when you talk about, like, these security incidents that we're talking about.

Max Pollard: Yeah. Yeah, 100%.

Allie Howe: Amazing. I guess just, like, to circle back and, like, kind of put, like, a bow on it it, it sounds like the, kind of the theme of this episode to-- in my opinion is, like, the future, like, is here in terms of, like, AI-assisted, like, security technology, whether that's, like, offensive or defensive. It's just not d- evenly distributed or, like, you know, maybe the attackers have the advantage today just because it's easier for them to adopt quickly and then try things versus defenders need some sort of, like, education, tooling, and reassurance that what they roll out is going to behave correctly and, and not negatively impact their business, especially if they have only, like, one, you know, app.stripe.com server and, you know, taking that down is, like, a huge business impact.

What is next for Cotool, and how are you seeing the future when it comes to evenly distributing the defensive tooling to teams?

Max Pollard: Yeah. I think, I think there's a, a level of, kind of like education and provability. And so one of the things that we're working on quite, you know, quite a lot at Cotool is basically the idea to say, "I'm starting from zero and I just want to, you know, click go and both learn how my environment's being secured and also, you know, even if I'm an expert have it assist me to get to, get from, you know, from good maturity to great maturity." And unfortunately, that matur-maturity curve is just in a spot where, like it's shifting every day, right? And so that means probably a couple things.

One, everything I do has to be on its own auto research loop. Detection, response, like it cannot be static. It cannot be an opaque black box system where, you know, data in, result out, and I have no control over, over how that's, that's, that's working. And so one is making sure that auto research loop is super, super tight for folks and there's a high level of transparency by which they can see, here are the things being detected, here's how we're responding to them. If I need to make a change or intervene, I can, right? So that s- serves two purposes. It serves the f- the person who's learning about how their environment's being protected and the person who's trying to, you know, go super deep and like apply their expertise and tweak it in positive ways.

The second thing is basically moving into this, this very like brave new world of simulation. Some folks are saying digital twin, right? Where you can say, "Hey, we're going to, you know, stand up a, you know, a very similar environment to yours and start knocking the blocks down and seeing, you know, how we can respond to those." And that's a much larger ordeal, especially as you get into these, these, you know, larger enterprises. But it's something that, you know, we're starting to help folks with. And then three is just listen to the customers, right? Like a, a lot of our great ideas come, come to our customers.

And so like besides those first two that we're h- hyper-focused on, I'm sure you talk to me in a week, there'll be a third, a fourth. But, but yeah, we're excited. We're, we're building really, really quickly over here. And yeah, it's, it's exciting times for sure.

Allie Howe: Definitely. Well, yeah, super excited to continue to follow along. If folks want to connect with you and continue the conversation, where can they find you?

Max Pollard: Yeah. I'm on, I'm on LinkedIn. I'm on Twitter as well. DMs are open. And, yeah, those are probably the two best places. Let's see. I'm gonna break character for a sec. Do folks typically just drop like their handle or their name? Or like what's, what's common here?

Allie Howe: Yeah, I usually just say like what platforms they're on, and then like if you want to give like your handle, you can. If not... If it's like easy to find you, like say I know some people have like a very like like hard to find handle or sometimes, then up to you.

Max Pollard: Okay. Then I'm good with the response I, that I just gave, I think.

Allie Howe: Yeah, I think that's, that's all good. Perfect. All right. That sounds good. Well, thank you so much, Max, for coming on today. Thanks so much for your time and expertise, and we'll catch up with you again sometime.

Max Pollard: Thank you. Yeah. Cheers, Allie. Really appreciate it and have a good rest of your day.

Allie Howe: Thank you.