Allie Howe Allie Howe (00:00) So tonight's focus is securing your coding agent and what that means for the road to the software factory, why that's important. And we're also going to touch on a new vendor-neutral reference architecture called Agent Baseline that these three companies collaborated and created. But it was not done in private. We also involved some CISOs from our networks, and now it's completely open source, open for public comment. it's completely vendor neutral. we would love your thoughts and feedback on that, and we'll touch on how you can use that reference architecture to help secure your coding agents tonight. So to set this stage, I'm sure we all are aware if we're using coding agents in our day-to-day lives, you really can't trust them. so I've got some quotes on screen from some folks that I have had on my podcast and who I've talked to about this issue. and namely we've got Malte Umul, the CTO of Russell. who came on and told us that it's really important for you know sandboxes to not get actual credentials and for those to be injected into the sandbox. we also had David Creamer, CPO, co-founder of Sentry, come on and tell us that verification of code is really important.

So we're thinking talking about software factories. You know, how do verify the act the agent and the factory actually created the right thing? and we also have Derek Meegan came on the podcast, he's from Browser Base, he's responsible for building the internal agent BB at Browser Base. Which you all may have heard of. and he he called out just as a developer, you know, you never you never know when an agent might decide I need to install some like random package and potentially you know compromise the security of my of my system. So these are all like very technical important people living this problem day to day in their lives. so you know, as security practitioners and leaders, how do we solve this problem for our developers, for our companies? that's what we're here tonight to talk to about.

I also wanted to touch on some incidents that have been in the news recently, which you all I'm sure. I've heard of the Hugging Face incident where OpenAI went and escaped a sandbox and went and found its way into OpenAI, or not OpenAI, but Hugging Faces production infrastructure. We also had the story of the railway incident with Pocket OS, where an AI agent, a coding agent, decided to delete a database as part of a fix that it was implementing for something else unrelated. And then there was also this example. Which I thought was like very interesting from a distinguished engineer at Geico, so obviously a very technical individual knows exactly what they're doing.

Was using a coding agent and told the coding agent, you know, let's and let's land number 118. And then what it did was decided that meant to push a production deploy when that you know that coaching was not meant for production deploy. so these incidents are happening all of the time. people are using coding agents. clearly, we need solutions to secure them, use them properly. If we don't solve These issues, we're not going to see a software factory anytime soon. And then today we have with us Ellie from Docker. We also have Ian Livingstone, the CEO of KeyCard, with us today, and Ezra Tanzer from Sneak. they're all here that are solving these issues. and these are the folks that also helped champion the reference architecture agent baseline that was released just last Thursday.

And so I'm super excited to have them on a page. Shortly to discuss their thoughts on how we solve some and prevent some of these incidents. and with that, I'm gonna turn it over to Ellie to introduce the reference architecture agent baseline.

Eli Aleyner So we got together with Ski Card, with Sneak, and sat and st started thinking what are the types of things that would be interesting for everyone to know about to solve those kind of problems. And early on it became obvious to us that there is this tension between security, capability, and autonomy. Those are three things that one needs to ensure in order to have a software factory. or safe agents running inside of an organization. The way we thought through this is, and this came actually very naturally to us, we start when we talk to customers, is first they want to secure whatever agent is running.

Hey, can we make sure that it it that it either runs in some kind of secure environment or that a human needs to approve every single action. Second is capabilities. How do we make sure that it actually can do things? How do we make sure that it's not just completely isolated from everything and can't do anything? And also, how do you do it? You need th those two in order to enable it to have some level of autonomy. You can't have autonomy unless you ensure those two. In the old days we felt it w the saying used to be, you know, timeline, cost or quality.

Nowadays it's oftentimes Security, capability, and autonomy. So when we thought about this, we thought about three main questions that one needs to be able to answer with agents running. Number one, what is operating and in what capacities? And in order to do that, you need to do two outcomes that we talked about. Discovery. How do you discover what is running and what capabilities does it have? How do you constrain it? How do you make sure that it ha that you know that it is, it can and cannot do certain elements? Number two, what is operating and with what capabilities? How do you authorize it? How do you have identity around the agent that's running?

How do you validate that it's actually Doing what it's supposed to be doing and only what it's supposed to be doing. Finally, what's operating in this what capabilities? How do you observe? When a change happens, an agent did something, which agent did it? And how do you roll it back if you have to? So the last stage is how do you observe and how do you respond? When we talk to customers, it you know, a kill switch comes up. So hey, how do we How do we yank this agent out? But also how do we respond? And how are we able to remediate the action that has happened?

So those are six outcomes that we've identified and wrote a white paper around that anyone needs to think about when thinking about deploying agents in scale in production. With that, the white paper is open sourced. It's available for anyone to contribute and anyone to help with. It has 35 controls in the six outcomes that we discussed. It's available at agentbaseline.org and we're taking contributions from everyone. Thank you.

Allie Howe to answer kind of the questions on everyone's minds of like I really want to progress towards a software factory. I'm running coding agents today. How do I creep up the autonomy curve towards a software factory? And that all starts with securing my coding agent. And I know software factories were a really big topic at AI Engineer World's Fair back in July. It was actually the topic on the main stage track on day one. And I think there's such a buzz around it because you know this capability is here now where model vision has improved. And models can now verify work that they weren't able to before.

Context windows have garden larger. So you can keep track of state now and see what work has been done before to keep things more cost efficient. There's far richer tool ecosystems and data that agents can now access. So, you know, the dream is here. It's become a practical reality, but people can't actually harness it and make this real for themselves at their at their companies because of the security issue. And that really circles back to what Ellie was talking about earlier. Around security, autonomy, and capability. Unfortunately, most teams believe they can only pick two, but a coding agent that wants to work its way up to a software factory needs all three.

So my beginning question for the panel, kick it off tonight, is why do most teams only have two today, and how do they get all three?

Ian Livingstone Look, let's let's take a step back and kind of think about where we're at in this journey. first and foremost, the security models have changed, right? Like the way we design How we secure our systems was really based on this idea that if a human was involved and we knew who the human was, everything was gonna be okay. and agents kind of break that model. So most people can't have like true autonomy because the systems upon which they use and operate and the way they think were designed based on how we thought about computing, and we're now in this new world where agents are a native component of how computing works, and we're trying to discover how that works.

So I think that's where we take a step back and think about it from that way. And then and then fundamentally the question is, okay, well, if we're we're moving up, if we're moving to a world where computers are reasoning on our behalf, and we're also moving to a world where computers are deciding for us on our behalf, and they're actually acting on us on behalf. And like the purpose of a human now is to provide intent or some goal for these agents to achieve, that finally changes the way we think about computing, right? And so I think first and foremost, How does it what are the different parts of that that have changed?

And I I was having a really great conversation last night with a great member of the community over over drinks, and he really came back and said, you know, look, I think there's like three or maybe four different layers to this problem, right? There's a supply chain problem, there's a runtime isolation problem, which is unique and new and different, and I'd love to different people to have chime in that. There's a network problem, there's an identity and access problem, which you can kind of like maybe say that's an identity and access problem, depending upon what your viewpoint. But I think that's that's why you can't have all three, is because like the computing and our security model and the infrastructure just wasn't designed for this type of actor that we are now all interacting with and these new interaction patterns.

And so the real question is, okay, cool, that's great. what is it what do we actually need to enable that? So I think there's

Eli Aleyner the way I think through this i i is slightly different angle as well. One, what is the destination? Is the destination complete autonomy? And the reality, you know, I look at all three of us sitting on stage, all three of us are hiring engineers. Right? And we're hiring engineers because we believe that they are needed to go and build the software that we're gonna do, that we're gonna have. The practitioner can do more. And yes, the systems that we are operating today, though, you know, we've learned a lot from the world of cloud native and the world of security. But most of them have never been built for, you know, taking a persistent threat, installing it on a developer's machine, asking it nicely to go and do something on your behalf, and then hoping it does it.

And it's very good at it. It's very good at going and doing whatever you ask it to do. Sometimes I think about it. Is taking a laptop, bringing it to a coffee shop, SSHing into production, putting a sticky, asking someone for help, and leaving. In a lot of ways, when we are installing agents on our machine, this is what we're doing. And they're great and they will not stop. To achieve a particular task. The question is then how do you how do you constrain them? How do you inline how do you put some certain of policy? How do you apply all of the existing practices that one had that would in that particular environment?

And how do you make sure that those are followed when you do not necessarily have a person to blame for whatever happened? Right. So those are all things that we need to think through, things that are required in order for us to move in the autonomy direction.

Ezra Tanzer Yeah, I think a lot of it comes down to being intentional around what capabilities you want the agent to do. If you just say open-ended, I want you to solve all of these problems, and you give it full autonomy to do that, it might try to do that, and you've got a big security risk there. and most people, most companies that are purchasing these tools, they want the 10x or 20x productivity gains that are being promised to them. And so if you can be really selective and say, this is the thing that I want to solve, this is where I want to remove the human from the loop, this is where I want to grant it as much autonomy, you can do that. more securely when you are intentional about it.

Allie Howe Absolutely. And I think that's mirrored pretty well in the six different outcomes that agent baseline talks about. So for more context on agent and baseline, it doesn't start from you know vendor product categories, maybe like some other things do. It's it starts from six different outcomes that your security program, your agent, should have, and then also provides 35 different controls to tell you how to get there. and so those six different outcomes are discovered. constrain, authorize, observe, validate, and respond. And so what Ezra was talking about of making sure you're aware of the different capabilities your agent has and being intentional about that. That's definitely mirrored in agent baseline and some of the controls that we have.

I'm also super interested to explore the fact that, like what Ellie was saying, around okay, you know, we're all hiring engineers, we're iteratively working our way up towards software factories, and what's happening today is we're going to end up modeling software factories and long running coding agents off of developers and workloads that already exist today. And so today a developer might write a piece of code and another developer might go and review it. But what that what does that look like in an agentics? System? Well, it looks like a multi-agent system where one agent is writing the code, one agent is reviewing it. So you end up with these agent-to-agent delegation patterns where one agent is telling another agent to do work on behalf of that agent, which might have been which may have been authorized to do something on behalf of a user originally.

So now you've got these different chains of action that are modeled off of what might happen in a real-day like software engineering team today. And this mirrors all Authorize one and two, some of the controls in agent baseline, which I encourage you all to go and read. And authorize control one says agents must have distinct identity and authorization. And authorize number three says that there must be delegation attenuation. So essentially that means that an agent should not have any more permissions than the caller that is calling that agent, whether it was another agent or a user. So I'd like to open up the question to the panel of how do we prepare or model these software factories and coding agents such that we can actually achieve a reliable secure system where these guarantees are in place.

Eli Aleyner It requires us to have like when we were living in a cloud native world, we built systems that allowed us to go and build and deploy software at scale, right? Kubernetes, one of those systems, right? That had enforced certain capabilities and enforced certain enforced certain contracts one needed to have. If we think about this, you know, the We we now are living in a world where most organizations are running multiple agents or like in today's world, I'm assuming most of you are operating in environments where you're buying you have Cloud Code running, you have Codex running, you have lot lots of different vendors who are selling you agents, and also open source ones, Hermes and Open Code and and and others.

And need to have a system in place to A be able to know what's going on and B to enforce certain set certain types of control. Lots of different solutions out there. Docker has our server, which is Docker Sandbox, which is a micro VM-based solution that allows you to run agents. That are constrained, but lots of other other ones that also exist potentially in the market. and those allow you to have identity as core part of that, and this is the collaboration viz key card. yeah, that's that's at least our position on this. Ian, you want?

Ian Livingstone I mean, I think if we were to summate like where we're at, sort of this energetic journey, we're and it's interesting. Ellie brought up good point. Like we think about cloud native, that was sort of the journey of like how do we take a bunch of pet rack and stack servers and turn Into cattle. Like that's if we if we look back at cloud native, it was like, how do we take something that like I created and made it like hyper repeatable and programmable? And that's like the last, you know, that's what the cloud really was about. If we think about where we're at in our agentic journey, and I think about my own company's use of agents, but also think about like what do we talk about is like, hey, it's my artisanal open claw setup, or hey, it's my weirdo harness setup upon which I build a top of pie, maybe using cloud code.

We're really in the pets phase of agents, right? So the question is as you go to build an enterprise and you think about okay, I would need to automate my enterprise. I want to like take the advantage of the value that these agents will provide today and will provide in future. The question is how do I actually, the curve we're all going along is how do we go from pets, agents as pets, to agents as cattle, right? Which is the idea that we can spin these things up, they're hyper programmable, we can shoot them if we need to, because you know, maybe they are disease and we just don't have the medicine.

And it's also okay if that happens. They can't, you know, they were In a constrained box. And so I think there's multi-different facets of that problem statement, but that's really the journey we're on. And then of course, like as a vendor that has like lots of identity and access problems, which is why we started the company. But I think like the challenge for buyers and people trying to think about how do I evolve my security program, how do I evolve my business is okay, well, what in the world do I actually need to enable us to go through this transformation? Because we are in this, we're in it, we're in a transformation, right?

Which is we're going from a world where we were racking and stacking servers. To where we're running stuff in the cloud and it's hyper-reactive and programmable. And now we're going in agents, it's like we're going from, hey, it's Ian's agent or LI's agent to our enterprises run by agents, and humans sit there and they do some higher order function that they couldn't do before because they're busy doing things that were not necessarily like as valuable as they are could be doing, which is ultimately the reason we adopt agents because they can do work that humans no longer need to do, and now humans can do higher value function, higher value work they couldn't do before.

Which by the way was the whole point of the cloud in the first place, is instead of racking and stacking servers that someone else did it for me, right? So there's just a nut journey along the same abstraction curve.

Ezra Tanzer I like the the pets versus cattle analogy. And I I don't know if this is against the rules here, but I'm curious kind of with the group that we have here within your companies, do you guys lean more towards today kind of having your agents as pets? or is it more towards kind of cattle where you can say we're gonna spin these up, we're gonna kill them if they're not doing what we're doing? And so maybe by show of hands, do people generally within your organizations lean more towards kind of pets that you guys are working with than kind of coaching through? Any show of hands for for that?

I don't know if these lights will And how about more on the cattle side where you have a bit more of an established factory? Okay. Fewer hands there. So yeah, I mean I think I think that's indicative of where we are right now. We're trying to help many of us as vendors kind of shepherd organizations who want to transform from the pre-agentic world to kind of adopting agents slowly to then saying this is just a core part of how we build software.

Ian Livingstone And and one thing I want to mention, you know, while you do have three vendors here, like the antithesis, like the reason we we spent time together to do agent baseline was we spend a lot of time individually talking to customers. Customers, people like yourselves. And inevitably, we ended up in a position where all of our customers, and we actually got together around a round of drinks, being hey, look, I have this problem. All of my customers are confused, right? They just don't know what to do. And I don't really know what to do. So we got around the table and said, can we at least like do something together that, like, based on the different value, different things we're hearing, put together something that we could put out in the ecosystem and say, hey, look, based on what we're learning and where we see the world as going, this What we think is important to enable this.

And one of the most important things about it is that it's completely open source. Anyone can go and disagree with it. You can go write your comments about it. You can we'll improve on it. We'll open up a collaboration of it. The point we're at right now is if we all want to be in the in the cattle world, we're missing a bunch of functionality, and no one vendor is going to be the right answer to solving all of the problems. But more importantly, it's to cut through the bullshit that most vendors will like push down our throats, including you. Know our own companies to a certain degree to say, hey, we think like the world's we're gonna solve AI for you.

And the reality is there's no one answer that solves AI, it's a collaboration, it's a bunch of different pieces that'll come together to do it for you, and no one vendor has the full answer, and that's what we're trying to help everyone navigate.

Eli Aleyner Maybe just to build on what you just said. when we when Ezra asked for a sort of show of hands, right, not all hands have end up. And I think that's the reality of the world we're in, right? While we are here talking about software factories, the reality of most organizations is most organizations are making the move from, you know, the original copilot, not the copilot CLI, to, I don't know, codex, clawed code, you know, copilot CLI? Not to forget our friends at Microsoft. But you know, this is just a transition that people are in. And the intent of this white paper and this collaboration is to try and start a conversation of how you help organizations move towards this journey of the cloud native world, which was pets versus cattle, and many of us were part of that journey as well.

And how do we apply all of the same lessons that we've learned along this journey, including software supply? chain including a lot of the smart things that one needs to be able to do in order to ensure just basic good quality of things that you produce.

Ezra Tanzer Well one thing I'll add is I I met a bunch of people before we were talking tonight, and in the introductions, people would say, what does your company do? And I'd say half the responses were, I work in agent security. I secure agents. And I think that's what most of us are up against when as customers talking to potential vendors vendors saying, What problem do you solve for me? It's not super clear right now. And as a vendor, we we all don't we don't do everything, right? None of us do everything in this space. And in many cases, a vendor is not Going to be able to replicate some of the things that need to live in-house.

And so hopefully, this framework provides a mechanism for vendors to say, here's where I fit in, here's what I think I'm contributing, and I can really kind of help convey that. And for practitioners that are trying to build this up, they can say, Here's my pain point, here's my priority, here's where I'm making progress, and then this is another area that I need to continue to invest in next month, next quarter.

Allie Howe Yeah, I agree. It's really important to continue to build a really big tent around this conversation. And it's gonna need all of us, vendors, practitioners, from all sizes and shapes, to go collaborate and solve this problem. So definitely check out agentbaseline.org and tell us what you think, give us your feedback and add to it. but I wanted to circle back to something that Ian had said around you know, companies and and users wanna take their agent and put you know this protective box around it. That got me curious because I've I've thought I mean about the Hugging Face incident recently, which I'm sure a lot of you have, where What happened was OpenAI's model was running some sort of cyber evaluation, found a zero day in its sandbox, escaped, and then found its way into Hugging Face production infrastructure.

And it took a pretty long time for them to realize that it was in their production infrastructure. And that agent had done something like 17,000 attacker actions that the post-borden had said. Agent baseline outcome of constrain. If one of the outcomes is we must constrain our agents, but we have agents that are escaping sandboxes, how do we fulfill some of the different controls agent baseline outlines, such as constraint number three, isolated, and confined execution, and also use case-scoped capability profiles, such as like making sure that runtime is scoped, we have certain network destinations how do we fulfill those outcomes? How do we keep our agents in a box if they're escaping sandboxes?

And I'd love to hear from you, Ellie, first on this.

Eli Aleyner All right. Cool. Awesome. So maybe you know, I'll start with, you know, zero day vulnerabilities are a thing. You know, they happen, right? If someone is gonna be here and gonna give you insurance against zero days vulnerabilities. Probably not, I'm not that person. So the question is really how do you have security in depths? How do you create a system in such a single single zero-day vulnerability does not lead to an agent escape? When we think about this in our mind, number one, you want at least two zero days vulnerabilities to be chained. In our solution, we use containers in micro in in micro VMs, right?

So almost as a hardware level of of isolation. Number two is egress of traffic going through a reverse proxy which you control, right? So making sure that even if so Anything that the agent is allowed to do is controlled specifically on the actions. Number three, making sure that the agent itself does not have access to secrets. Right? If you go and run cloud code today on any developer's machine and you ask it nicely, tell me what secrets you have access to, you'll be surprised that by what cloud code will tell you. typically GitHub credentials, access to your AWS account. you'll be lucky if if if it doesn't have access to other things.

But you know, when you think about sandboxing, how do you make sure that you only give it the credentials when it is needed, and those credentials are given on a specif in specific needs only? And finally, how do you in how do you only give the agent the data and code that it needs to be able to act on. Right. So how do you only give it the directory that it that it has access to and nothing else on the system? Because those agents are going to try to, you know, go up authorization levels, go and try and do anything possible to go and try and do whatever it needs to do.

So you want to be able to only give it access to the particular thing. And finally, how do you apply the the the same lessons we've always had around Building secure software, building from source, building software with known supply chain, such that the vulnerabilities inside of the sandbox itself do not persist. It is all of those things together are non-trivial, right? We've seen people try and build those themselves. This is why we built a product around it, which we released a few months ago called Docker Sandbox. In our mind, it's not a simple it's not a single solution. It's a it's a security in depth that one needs to have when one thinks about preventing escapes

Ian Livingstone I mean I think look there's I think there's the component diagram and then there's the reality of the fact that like the the interesting thing about the hugging face stuff is It used was actually protected by a squid proxy inside a container. They found an exploit in the squid proxy that then allowed it to get to get outside. So it self-discovered a zero day that no one had seen before. So I think first and foremost, I think we have to accept like the unfortunate fact that software is just full of vulnerabilities. And I had this great conversation recently with like a reporter at the NPR who was trying to understand like why anyone cared about the fact that like Fable 5 like got shut off for a period of time and what that meant from a security perspective.

The reality is that, like, in the same way that we need to have agents be cattle, we have to treat our security component tree as cattle as well. And so we're moving from a world where you know, both on the attacker side, in this case, you have a malaligned agent doing something you didn't intend it to do, and you can kind of look at that and be like, okay, it's an attacker. That attacker is highly advanced. The defensive side also has to be highly advanced. So the defensive side has to be agent driven as well. So I know Docker in this case is a vendor.

One of the questions you should be able to ask them is like, so what are Doing to ensure that you're updating your supply chain and finding all these things at the same time, right? Like, what are you doing from a model intelligence perspective to ensure that you're closing these zero days all the time? Like these are these are good questions you need to ask, right? There's a shift in in things, and are your vendors staying up to date? Is a fun mo question. I think the next question is Now that we now you kind of expect it is like there's because the attack surface has changed, what's changing on the defender side?

Well the defender side has to be intelligence driven by intelligence as well and has to be driven by LLMs. And so just in the same way that you're gonna have lots of agents, you're gonna have and agents are gonna formulate, you know, you're gonna have agents that are runtime agents. Like Docker will look and t tell you, I'm sure in a a year or two, or maybe tomorrow, that they're they're like a runtime agent of some formulation, right? Like they're gonna describe themselves interestingly, the same way that like a key card or a sync will describe themselves And agents. But the transition that I'm talking about is not only that, like the pattern, the malignant or that the attacker is this agent, this LOM-driven thing, is that your core software inside your core stack also has to be driven.

And ultimately, at the end of the day, for the defenders to win, defenders have to have the same level of access to intelligence that the attackers do, but more importantly, they have to have this have even greater levels of access to intelligence than the attackers do. And the fun little question ultimately comes back to security, is the cost-benefit analysis of. Of we've always lived in this world where it was I could do 80% and that would close out most of the pro risk area and it would be okay. And now you have to close out so much more of the risk area than you ever had before.

And so that's really the one of the primary changes from a security posture that's happened. And so I think I think broadly speaking, stepping back to solve something like this open AI hugging phase escape is well, ultimately at the end of the day, there are undiscovered zero days that no one knew about it. It somehow found it through trial and error, it got out. and what does that mean? Well, you also need to defend To be able to see, hey, this is a thing and be able to react immediately to the fact that that happened. So it can close that vulnerability quit very very fast in a way that you wouldn't need before.

Ezra Tanzer Yeah, I think we could look at the hug and face incident and we could decide to throw our hands up and just say, you can't defend every against everything, so why defend against anything? and that that we we don't believe that that's the right approach. you think about any zero day, or you think about log4j, if we go back to to 2021, Once this was exploited, there was a mass rush to remediate this. And it wasn't just because we said, this zero day happened, now We we're exposed, like our our data got out. no, we need to protect ourselves for the future. And it took some companies months or even years in some cases to remediate that entirely. and so when there are vulnerabilities that have been identified in your agentic solutions, as the new zero days get identified, we need to be continuously applying those patches.

And of course, we're not gonna be able to solve for every single future-facing zero day that might emerge. but as long as we understand and have visibility into where we are exposed, we can be continuously closing those. Gaps.

Allie Howe Yeah, I definitely agree with the defense in depth approach that that Ellie was talking about. and you'll definitely see that throughout agent baseline as well, particularly in the runtime isolation section, which talks about you know taking you know a sandbox and making sure that credentials are not inside the sandbox, so the agent doesn't have a hold of those like permanently. and this is a design pattern that I was also made aware of by a couple of the guests I've had on my podcast, multi-Uble. Who built Vercel's DeepSec harness, which finds vulnerabilities? And the way that harness works is it does not have credentials stored inside the sandbox.

A credential is injected at runtime. Derek Megan at Browserbase, who built BB, the internal AI agent that all of BrowserBase uses, follows the exact same pattern where a short-lived credential is injected into the sandbox. So that pattern is delivering two of the outcomes that baseline talks about: the authorize and the constrain. outcomes both working together simultaneously to create new security guarantees. what other places are you seeing security guarantees be delivered by different outcomes in baseline working together or expand more upon how runtime isolation is so important for agents

Eli Aleyner so number one why is this design pattern super important is because like if you look at the very basic kind of Setup many developers have today, right? The amount of access those agents have is very significant. And when you can argue that the developer or an engineer working in your organization has morals, it is very difficult to argue that an agent does have them. The agent will do what you ask it to do, and it's very and the nothing will stop the agent from doing what you asked it to do, even if it misses. understood what you've asked it to do and it will use all the tools to its that that are within its the its access.

So you must constrain what the agent has access to and give them ability to use those tools only for specific tasks and specific needs that the agent has. When we think about, like when we often talk to customers and we often talk to about this, it often comes to like this hierarchy of needs that companies have, right? So, first it's around how do we constrain things, right? That's the first typical thing. Like, how do we how do we have a constraint. constraint around an agent. Second, not surprisingly, especially I guess August 2026, is cost. How do we make sure that you know costs don't go completely out the window as as more and more people adopt agents.

And finally, okay, you start talking about cost and finally what comes on top of it is intelligence. How do you apply an intelligence layer on top of those agents? and the intelligence layer can be in terms of in terms of what credentials it can have access to, what models it uses and so on and so forth that so it's as you s as you achieve one, the next layer of questions usually comes through and you must be able to help organizations move down, move up. the the needs.

Allie Howe Another really critical important part of a software factory is the ability to validate the work that's been done, the software that's produced. So, how do you answer the question of, you know, did the software factory go and build the right thing? another part of Agent Baseline that caught my attention was the validate outcome, and the control where that says you have to make sure you can like validate agentic artifacts. Ezra, I was wondering how are you all thinking about that at sneak or how do your developers approach that? Like we Get to software, you factoring that becomes a reality. How do we make sure we're able to validate what the agents are doing?

Ezra Tanzer Yeah, so people who are maybe unfamiliar with kind of SNEC's origin, but we start out in SCA security, supply chain security for packages that then may be integrated into code. and where we kind of pioneered the space was to say we're going to help you shift left and make sure that you can prevent these issues from making it to production. This was all kind of pre-agents. so integrating to CICD pipelines, integrating into PR checks. And we got this demand from customers to say, how can we shift ship shift even further left? How can we kind of solve these issues, prevent issues from even making it to a bottleneck that happens later on? and that had that was, I'd say, mildly effective to be able to shift even even further left, kind of in the pre-agentic world, but it required developers to kind of step in and adopt and say, I'm willing to use these tools locally to test and to fix issues. now with agents though that that's really changed.

We can actually we we think that we can solve prevention from an application code security perspective pretty completely without a human in the loop. If you as a developer are prompting an agent, or if you've got a factory and you give it a spec and you ask it to to generate some code, we have autonomous processes using hooks or different different levers that might exist that can execute tests of that code. If there are issues, pass that back to the agent and ultimately have that code fixed. And that can iterate. In a loop where needed, but we kind of inject intelligence there to try to get that fixed done the first time.

And that has really not only like agents and hooks have kind of been a big vehicle for that, but we found a way to distribute this at scale. So you don't actually need developers to choose to opt into this. If you actually want to say, I want to push something out onto the workstation that developers are working in, great, I can do that. But I think where this has kind of evolved is if the application code for the most part part is somewhat of a solved problem, now how do I protect the workstation itself? In the future, I think we're gonna see much more cloud agents kind of working in sandboxes, micro VMs.

But we know the reality today is that many, many folks are running cloud code or cursor or codecs on their own machines. and so the big question is now how do I actually secure the behavior that these agents are taking? How do I make sure that I'm not letting Take destructive or exfiltrative actions. And so that's been a big area that our customers have kind of said, hey, we we need help solving this, that we've we've been investing a lot over the last six months.

Allie Howe Awesome. And then another part of a software factory I think is really important is the ability to actually observe it. And also very important for any coding agent or as you're working your way up to a software factory, being able to have a coding agent that runs in the background. Maybe it's a long-running agent. As you work your way up towards a software factory, that might be like the first step is this like long-running agent. But once you kick that off, how do you have eyes on it to make sure it's doing the right thing, to make sure it's taking the right actions? when I think about the hugging face incident where that agent took you know 17,000 attacker actions, like the report said, before anyone noticed, where is the observability there?

Or how does that become super important in the forensic log? one thing from that report that caught my attention was because that agent ran in a sandbox, they were able to go back and find the logs and observe what it had to do. done. I also think about you know maybe potential like audit logs that this agent might have. If it was using sort of sort of like you know credential service, we're able to have some sort of an audit log could see exactly what credentials that agent held at what time, who authorized the agent to do something. That all becomes critically important for incidents. and I think it's a very important part of a software factory, something to get right when you're building it.

It's also one of the outcomes of agent baseline. so I'd love to hear from the panel what does observability mean to and why is that critical for a software factory?

Eli Aleyner So maybe I'll kick it off if that's okay. The interesting part about that that report that you just quoted is they could. Tell you the exact number of actions the agent took after it escaped. And I venture to ask the room over here: how many of you would be able to know? And specify, like, hey, if an agent goes rogue, would any one of you be able to know the number of agents and the number of actions it did wrong? Most of the people I usually talk to, the answer is a no, because most of the agents today are running on developer laptops. Most of the agents are not reporting to a central location.

Most of the agent like you don't even know what what happened and what caused it to happen. In a world where we have developers running agents and not just developers, any citizen and and engineer inside of your organization could be running an agent, how do you know what caused the particular action? That becomes an incredibly difficult question to answer and a lot of headache for a lot of our security professionals here in the room.

Ezra Tanzer Yeah, I mean credit to Hug and Face to be be able to do that. I think that that would be challenging for for I think any company out there. Maybe a handful that could do that. and so observability is just critically important, even if it's just logging right now. My question earlier to the audience was kind of where do you guys stand on kind of moving towards the software factory? And I think the the reality is that many of us work at companies where that's just that that's not the case yet. It's engineers that are working in clawed code on their machines or whatever tool they might choose. and they're trying to do things.

Like nobody is, I shouldn't say nobody, very few engineers within companies are trying to take malicious actions. Most of the risk that's happening is due to negligence or it's due to an over-eager or over-provisioned agent that's got access to do something, and it's try it's trying to do the right thing. All the examples that were talked about, it was the result of kind of accidents there. I I see. as if we can understand what are our developers trying to do today when they give a prompt, when they're leveraging certain skills, and we can see here's what they want to do, and here's the insecure or dangerous or risky action that an agent took, you can now build paved paths in the new world for that to say, hey, we are gonna have an explicit MCP server that does this in a secure way, that we build internally for our own purposes. or we're gonna make sure that we can dynamically create a credential that can access only this that accomplishes this task. we've seen a million times, that otherwise, you know, with an over provision credential could be really, really risky.

So even if you're not taking the action today to create policies to prevent all of that, if you start to gather the visibility now, you'll know how to prevent that in the future as you do start to build out policies.

Ian Livingstone I don't know how you get to autonomous software if you don't have observability, right? Because like what we're really talking about is just more software, less human, more software, more software. so I think the question for for security folks is really like where does the attestation and and how do you how do you actually create like you're gonna have all these different streams of data from all your different sources and how do you test that back to security data lake? And which vendors provide you those data sources and what how does that provide what controls? And then I think the more important question is you know, as your vendors and as pieces of software and your infrastructure evolve, where does that automated EDI response loop come in and how d what controls does that automated ED response loop have to prevent that agent from continuing to perform malicious action, right?

Or mal aligned action. And so I think that's a really you know, we're we're moving to a world where like i auto remediation is is the is the future. ideally we get to a world where you remediation is never required because the software's just that good. The reality is that software self evolves, it evolves today. Independent of agents. So it's likely that you need both like strong verification, both in terms of the generation of those software artifacts, but also then automated remediation that actually scales. And with agents you can do both, but then the more important component is where's the attested evidence that your controls are actually working?

And more importantly, that those controls no longer are like a static compliance thing that you do once a year or once every 24 months, but have to be something that are continuous and evolving at all times, and you need that evidence for it to work. Because that's going to be the thing that lets you know whether or not, hey, we're good or shit, that thing went way off the rails and we gotta do something about it. And the we have to do something about it is like the actions happen quick. You must be able to do it in a d in in an not an autonomous but but but automated way right so how do you do that and when you think about controls in a lot of ways, you know, in the bet you know, in the cloud native world it was all about supply chain controls and how do you know the software came from the right place.

With this world it extends also to the autonomy that's given to the agent. How do you know that the agent is doing what it's supposed to do because the agent now has is can do things that that you d never you never programmed it to do?

Allie Howe Absolutely. And my last question for the panel here tonight is something that Ellie mentioned at the beginning, this concept of a kill switch. If agents do take action so quickly, how do we stop an agent immediately? That's actually modeled after one of the outcomes in agent baseline called Respond. And in Respond, it talks about the ability to revoke an agent's access or be able to provide some sort of you know, not maybe if it's complete killt switch. I don't I don't know the specifics there, but if it's you need the concept of being able to revoke access or maybe it's an egress rule or maybe it's a package. what does revocation and respond look like to each of you?

Ezra Tanzer I I'll probably want to hear from you, Ian, but I just I one of the reasons why I think that this is so important is that right now, even if we've got agents doing a lot of the work and if we're aspiring towards agents being the remediation actor, when there's an incident at two o'clock in the morning, it's a human that gets paged. And so everybody who is a practitioner here or who works with practitioners, like that that's what we think about. There's still the human side of how do we s how do we stop this? How do we stop this quickly? So if they are up at two and a two AM, they're able to address this, they can go back to bed at three, right, and deal with it. kind of in in morning time, but I think a lot of it probably comes down to the identity itself.

Ian Livingstone Look I I mean putting my vendor hat on it for a second. the reality is is that most IDP's identity access programs are built around role based static access controls that work totally fine for humans. It also revolves around static API keys that are living around, long lived on disk. If you think about the light alarm attack that happened earlier this year, which ended up in Merkur, you know, relatively popular Silicon Valley company getting popped, you know at the end of the day as it was a supply chain attack that then turned into reaping keys off disk, right? And that those keys were long lived, broad based and associated with a human and that was totally Fine in a previous world.

Today you need an ability to actually like revoke an agent's access to a session, all things, whatever it is. That's a very different paradigm than what previously came before. And it requires like a different formulation of what that means. I know all the downstream systems. That gives you like a central source of truth for hey, what's this? The ac agent capable of doing? What has it done potentially? What things does it ask permission for? And so then if a result of that, if I have a central system, I can click a button or you know, send an API call or have an automated response loop, do this thing that makes sure that all that access gets revoked.

And that's that's what we're working on at Keycard. but I think it's a fun independent of like how you get that outcome, that ability to immediately revoke an agent's access or kill its ability to perform an action is fundamental.

Eli Aleyner So for us, I mean the kill switch is Simplest and the crudest thing that one can do to an agent, just like let's stop it from happening. But in order to be able to apply a more refined policy or more refined action on an agent, an agent needs to be constrained and its capabilities need to be governed in some way. So you Using systems like Keycard or using other identity providers, and being able to know what policy to apply, and then being able to apply that policy is becomes critically important. That's how we think see that particular element.

Allie Howe Absolutely. Thank you all so much for your time and your expertise. Thank you all for coming out tonight and listening to our amazing panel. Round of applause for all of our panelists.

Ian Livingstone Also, round of applause to Ali. That was a great panel. Yes. Thank you, Ali. Thank you.

Allie Howe And then I'll put agent baseline. There's the QR code. Please go check it out. Give us your feedback. Like we said, it's open source, vendor neutral. We want to hear from the community, practitioners, security leaders, other vendors. let's help get to a reality where software factories are actually secure. Thank you. Thank you. Thank you.