Insecure Agents — Episode 31: Daytona Compute, Sandboxes & the infrastructure underneath
Podcast: https://insecureagents.com
Listen: https://podcasters.spotify.com/pod/show/insecure-agents/episodes/Ep--31-Sandboxes--the-Infrastructure-Underneath--and-What-that-Means-for-Your-Security-Posture--Daytona-Compute-e3hkl8c
Transcript (auto-generated, lightly edited for clarity).

Allie Howe: Welcome to today's episode of the Insecure Agents Podcast. I'm your host, Allie Howe, and today we are at Daytona Compute learning about sandboxes and Daytona. And today I have Sherwood Callaway with us. He's the founder and CEO of Sazabi. He recently just came out of stealth with this new company. Super excited to hear his experience with Sazabi and his use of sandboxes here today. It's an AI native observability platform, but I will let Sherwood tell you more about himself and that platform.

Sherwood Callaway: Yeah, Thank you. Thanks for having me, Allie. Super excited to be here at Daytona Compute to be on the Insecure Agents podcast. This is my first appearance. I've been waiting for a long time for this. I guess for the audience who may not know me, my name is Sherwood. I'm a software developer, a second time founder now, and I'm building a company called Sazabi. Sazabi is an AI native observability platform. We're specifically designed for fast moving engineering teams. So we'll work exceptionally well at high growth venture back tech startups between the sort of Series A and Series C range. And it's born out of, you know, 10 plus years of engineering experience in in my career.

And mostly most of that was focused on infrastructure, DevOps, production engineering and observability. So I basically care a lot about making engineers happier and more productive. And I felt like observability was a part of my life that had really not been revolutionized by AI. And so I wanted to create a new AI native observability platform.

Allie Howe: Amazing, I too started off in software engineering in my career and I've used a few different observability platforms and none of them are very good and or fun to use. They're they're not loved.

Sherwood Callaway: They're not loved, no.

Allie Howe: And everyone's familiar with the monitors going off here and there and trying to figure out like what's actually happening. I think one of the advantages that we spoke about earlier about maybe an AI native approach to an observability platform is the fact that agents can dynamically understand what's going on. And if there's like a certain problem versus like a static metric or monitor might just have a certain threshold and say every time that this goes off under this condition, then this would be problematic. But an agent can take context across multiple different nodes and understand, you know, maybe this metric isn't the best way to understand this.

Or maybe given this extra context, like this monitor shouldn't go off essentially. So I'm really excited about your approach. To this.

Sherwood Callaway: You're totally right. I mean, I, I think you're, you're getting at we have these three pillars, I guess I should say 3 core principles. Sometimes we refer to them as our hot takes, which these are the three things that makes Sazabi really different from a traditional observability platform. The 1st is that you can only chat with your telemetry. You can't see it directly. And that the reason we care about that is we, we think existing observability tools are really bloated with all of these modules and features that developers don't really use and that only tend to kind of create a a steep learning curve and make things confusing and disorienting in the event of an incident.

So we get rid of all of that and then we only have a chat interface and then you can ask the agent things like why is production down, which customers are affected, which commits responsible? How long has this been happening? What, what remediation actions should I take? That's number one. The second one is that we don't allow you to send us metrics or traces. You can only send us log data. And this is pretty controversial in the observability space. Like it's definitely pissed off a lot of people. We have got David Cramer from Sentry Retweeting with some. The. Rebuttal What really makes this possible is that agents can interpret large amounts of unstructured data in a way that humans could never do efficiently before.

And logs are unstructured data. That type of telemetry is now much more valuable than it was ever before. The last thing that is that sets us apart or makes Sazabi really different than a traditional observability platform is what you were talking about just now, which is the our unique approach to alerting. And what contains is essentially we get rid of traditional static monitors. So your team doesn't need to go into the platform and configure individual monitors and set up thresholds. Like, you know, if this host has CPU that goes above 80%, like sending send this cryptic message to the Slack channel that no one that everyone has muted, one junior person on the team's going to have to get page have no idea what to do with that information.

We get rid of monitors completely. And instead, we have something called agentic anomaly detection, which means that basically we have agents that are reading your telemetry 24/7, looking for anomalies, anything that could indicate some kind of problem in production, and then investigating them, deciding whether this is worth your attention and bringing it to you with all of the context that you need in order to action on it or make a decision. This is a very long response to what you were just saying, but the through line here is that all three of those core principles or hot takes are predicated on AI and agents. They're only things that we can do because of agents and allows us to create a a pretty different type of observability product.

Allie Howe: Amazing. And I imagine just even single users tenant, how much data and how many logs there could be at any given time that agents are going to have to go and sift through constantly to do that agentic anomaly detection 24/7. Is that how you're using sandboxes mainly today?

Sherwood Callaway: Yeah, I guess to be as explicit as possible, the main primitives within our platform are like a thread, a run, a message. And every time you have a conversation with an agent in our platform or an agent does a background investigation that creates a thread. Then for every thread we have a sandbox. So you can imagine like our users are doing 10s of conversations a day. The conversations are designed to be ephemeral throwaway things like the way you treat a Claude Code conversation. And then in the background, we're running these these anomaly detection agents all the time. And each one of those has an associated thread too.

So that results in a lot of sandboxes being created. The main use case that we have for sandboxes is 2 things. One is that we we represent our memory as files, markdown files like honestly it's quite similar to open open clause approach to to memory. It's a very unstructured and flexible and it's just, we basically gave the agenda a simple file structure and say like, how about it? You can store whatever you want here. The other thing that we use sandboxes for is is CLI tool access. So obviously sandbox gives us a bash tool that allows us to execute arbitrary commands in that. In that environment, we let users install additional CLI tools into their sandbox environment.

For example, if you want Sazabi to be able to introspect your AWS configuration and look for like hosts that are, you know, that might have some issue or Terraform drift or misconfigured ACLs or security groups, you can do that by adding this the AWS CLI to our sandbox environment. And basically the sandbox allows us to do that in times for any relevant CLI tool that that you might want us or Sazabi to have access to. So it's a pretty essential part of our architecture.

Allie Howe: Yes, Oh my gosh, that's amazing. I'm sure there's Security benefits in there too as well. And risks. And once it like sharing context across all of those different agents, like I don't know how you're approaching memory or because I know one of the main use cases of sandbox is they're ephemeral, but if you want to share context.

Sherwood Callaway: Yes, I actually think this is like a really hard problem and I'm happy you're mostly happy with our solution. I think it's, it's quite clever, but I also think there's opportunity for something better. And I know a couple of companies that might be working on something that could make this much easier. And it represents a big opportunity in my opinion, to be less vague about it. I mentioned that our memory is represented as files in the sandbox. We want that memory obviously to persist across threads and to be shared across threads. So like if I talk to an agent today and I say my name is Sherwood and I talked to an agent again tomorrow in a different thread, I would love for the agent to still know that my name is Sherwood or more relevant to this is obvious like an incident happens today.

Next time I talk to the agent tomorrow, the agent should remember that there was an incident and that it involved these services and it was caused by this commit. And this is what the error message looked like. We store all that information in files. We need to persist those files beyond the lifespan of the sandbox. The way that we do that is with Git. So we have self hosted Git server. Internally. Every sandbox is a branch on a git repository or receives a a branch on a git repository. That branch is forked from the main which contains some like universal state that all of the agents share.

And when we create the sandbox, we basically do a git clone right when the sandbox is created that populates the file system within the within the sandbox environment. And the agent actually is like none the wiser. The agent doesn't even know that's really working with with Git or that Git's being used under the hood every time the agent calls the bash tool. There's no easy way for us to know whether the agent has edited the memory or not. I guess one potential approach is we could like check some the entire file system of the sandbox and see if anything's changed. What we do is we literally just git push to that branch at the end of every sandbox execution.

And that ensures that if there were any changes to the file system, they are persisted to the remote Git server. And then the next time an agent runs, it pulls down whatever the latest state is for its sandbox. And this is how we share share memory across the agent runs. One interesting case here is that in some cases, certain agents will modify the same file like at the same time, or let's say, like I said, you know, I said my name is Sherwood, you said your name is Allie. And there's like a the agent file system currently has like some file which is like names.

And both of us modified them. If those changes, they both need to get persisted, but we've modified the same file. So the way that we can do that is like, or it creates basically a conflict. So it needs to be some kind of conflict resolution process. Fortunately, like git is, that's exactly what it is built for, for version control and conflict, conflict resolution. And so there's a merge conflict created in the back end. We have a, a back end agentic workflow that resolves it. So it's, yeah, I think it's a really clever and elegant solution to this problem, but it's it's also I think was not intuitive or easy for or it wasn't something that was supported well out-of-the-box.

So this could be an opportunity for sandbox providers like Daytona to have better first class support for that or other companies that are thinking about virtual file systems and which I think are a super important primitive within sandboxes.

Allie Howe: Yeah, definitely. I know there's many sandbox providers out there today for sure. So I'm interesting to see like how they differentiate and what different systems they support better like file systems.

Sherwood Callaway: Yeah, that raises questions too about like if do I need a third party file system provider so that I can mix and match the sandboxes that I use or I can migrate from one sandbox provider to another without actually having my memory, my memory or my file system kind of locked up in that in that particular vendor.

Allie Howe: That's true, but if it's all in git, at least you have it there.

Sherwood Callaway: Yeah, if it's in yourself as a git server, you're you're totally fine. But if if Daytona builds their own like virtual file system thing then that's a way of them creating lock in. It does create more value, but it also I think creates lock in.

Allie Howe: Interesting. OK, Yeah. And then one last question, How are you thinking about taking advantage of the fact that you're storing all of this context from all these certain agents across all these different nodes and someone's and across their entire logging and observability suite at their company? Like, could you predict when things are going to go down? Or like, hey, like we've solved this problem before. Like we solved this, like monitor essentially went off when this happened and we're seeing the same signs of it maybe happening again. Like could you preemptively warn people?

Sherwood Callaway: You absolutely can. I think that one useful analogy, it's useful in some ways, but not in others, but is open claw. And I think open claw actually is the reason that it got so popular is because it surprises you in really pleasant ways, Like it's, it's observant, it's thoughtful, it's proactive, and it, it, it draws connections that we're not used to AI agents making. So Sazabi is very similar where we give Sazabi, like we give it memory, we give it access to your telemetry, we give it access to your code base, and then we give it tools to contact you. And what we're finding increasingly like, especially as we improve the prompts and we start to do more evals and, and RL and post training is that the Sazabi is reaching out about things before they become problems.

We're really long AGI in that sense. Like we want to defer to the agent and believe that the agents going to do a really good job, a better job of detecting and root causing issues than humans would with traditional tools. And so we actually want to remove constraints where possible.

Allie Howe: Amazing. Yeah, that's what I feel like. I read some article on Twitter by Anamush Cortana earlier this year about context crafts and essentially the same idea where the full truly realized power of AI agents are agents that are able to take this context and all behaviors, perfect outcomes and help us make better decisions. So super excited about new AI native solutions like Sazabi that have the potential to do this, so thank you so much for coming. On

Sherwood Callaway: Thank you for having me.

Allie Howe: Yeah, this is amazing. And we'll have to do it again.

Sherwood Callaway: Thanks.

Allie Howe: Hey, my name's Allie Howe. Welcome to another episode of the Insecure Agents podcast. Today we are at Daytona Compute and I've got Anthony Shew. And with me coming to you remotely. Anthony is the core maintainer of the Turbo repo and then is using sandboxes to dramatically improve the performance of that repo. And I'm really interested to capture his sandbox story today to be able to share it with all of you. Anthony, would you like to introduce yourself?

Anthony Shew: Yeah, Hi. Thanks for having me on Allie. I as mentioned, work on Turbo repo. Turbo repo is tool to make your mono repo work faster, make your dev workflow workflows a little more seamless, and ultimately you can end up being saving 8090% of your CICD time with turbo repo. It's been particularly interesting to see folks lately leaning into Turbo Repo a lot with AI and coding agent stories, just because AI seems to love mono repos. They get all that context just right, all in one place, don't have to go reach around for different things. And so yeah, Turbo Repo has been been doing well lately, thankfully.

Allie Howe: Yes, that's amazing. And Turbo Repo, something that's owned by Vercel. Can you give us like a high level overview of what Turbo Repo does or why you'd use it?

Anthony Shew: Yeah, I think kind of the so yes, we, I, we work out of Vercel. But yeah, you'd normally use Turbo repo, particularly most of our users are in mono repos working on large enterprise settings. But also I see a lot of particularly open source libraries that like to use us for baking together and string together all these workflows that you need to build, test, lint, format, run, whatever it is, any of those things that you need done in your repository. So you can string together a lot of really simple and complex things that you might, you know, I don't want to run 4 different terminals just to have my development environment going.

I don't want to have all of these strange dependencies to try to wire together all this stuff. You can just have turbo run dev and then all of a sudden you have your entire dev environment up. So that helps a lot. Also, when it comes to CI CD story, mono repos can be, you know, a series of string together bash scripts and all of these crazy things. And Turbo repo, you just get to write 1 file and it's as simple as that and you're done. You have everything you need baked into that one small piece of JSON. And we will read a lot of what is necessary for your mono repo to act the way you would want it to.

So you don't get to with very little configuration, you get a lot of speed. So that kind of ends up being our story is simple, but really, really effective, hopefully.

Allie Howe: Amazing. Yeah, that makes sense. I know mono repos are very popular and also very large. Semester performance definitely comes into play there when you're trying to manage all of those different files and like the large contact size that gets associated with those. Is that something that you ran into when you started looking at turbo repos performance?

Anthony Shew: Yeah, I noticed one day. So I guess maybe a little bit of back story and history is in Turbo repo. We've always had, you know, just like anybody does, plenty of feature requests coming through and we have always looked at it as like folks need the, you know, are asking for these things, we need to build them. And so a lot of our growth sent for the past couple years really was built on adding features, you know, not a unique story, but for us, it was really important to be able to cover all the things that people wanted us to do. That continues to be true.

But for whatever reason, couple weeks ago, I opened up one of our internal mono repos, which is a turbo repo. And I typed turbo run build or whatever it was. And it took just a lot longer than I wanted it to for the task to start spinning up. And me working on turbo repo, I know that that's Turbo repo's job right there. This is the part where turbo Repo's doing all its work. And it took, I don't know what it was like 30 seconds. It's a very large repository and I had made some specific changes. It was a little bit of a pathological like performance kink that you that it could be ran into.

But at the same time, it really bothered me. I don't like slow things, right? The things named turbo, it should be fast. So yeah, it really bothered me. And I just kind of started poking around in the repo and I just was like, you know, I need a time box a week here to make sure that we are fast in every given situation. And so one thing led to another and I just kept going and going and going and in that large mono repo and ended up improving our performance by like 96%. And so, yeah, it was just one thing LED do. I just kind of got a little bit obsessed, to be honest.

Allie Howe: Yeah, that definitely came through in the Twitter thread where you captured all of the work that you did. You were very persistent. You're like, not going to give up. And it looked like when you got to the last, like, 80 to 90% of the performance improvement you wanted to make last, you know, always the last, like 10% of whatever is like, always the hardest. And it seems like sandboxes were the key to unlocking that 10%. Is that true?

Anthony Shew: Yeah, absolutely. So I was, as the story kept playing itself out, I had started with kind of large items that I knew were holding us back. There were things that we knew about and things that we didn't know about as I kept going. But we knew there were a lot of just large hold UPS that we were running into and that we could make go away as long as we invested the time. So I was like it's time to invest. So I started knocking out a lot of those big things. I was scripting out a lot of benchmarking tooling that I could have an age, just have my coding agent and you know, just hey, take a profile, create a performance profile for this turbo repo, tell me about where we're slow and then we'll work on it, right?

One of the key things that I did early on was we have always, as most tools do, spit out profiles in JSON format. And I kind of watched my agent stumble around in this JSON file like I probably would. I would, you know, try to connect different pieces of the JSON that are connected together and try to like read a line. Then suddenly, OK, I realize that one doesn't matter. I was watching my agent kind of do the same thing. And I said, well, what if this was just markdown? This is way easier to read, right, in theory. And so that was actually something that a couple weeks before I was doing this, Bun had done.

So I kind of made a markdown profile and my agent got me like 85% of the way there as far as their performance gains that could be gotten. And then I started noticing the agent struggle with just kind of too much noise on my laptop. Effectively reading from disk is potentially it's nondeterministic, the timings that you're gonna get back. CPU tends to be a little bit tighter, but as I was getting faster and faster, the variances in the readings that I was getting, there's just too much noise in there. I couldn't make anything out of, you know, it. Did that change actually make things faster?

The agent can really figure it out. There was no, you know, the standard deviations were just too weird. I started realizing, OK, like this is my laptop. It has, you know, all of these various programs and daemons that are running behind. It's making network requests I don't even know about. Probably I need something quieter. And for whatever reason, I had a revelation. Maybe I can just spin up a Vercel sandbox and go start working in there, right? Because that's going to only have the various programs and utilities and things that I put in there. It's just a Linux container. And so I spun up a Vercel sandbox.

I stuffed kind of a few of our repositories that I wanted to see, even the Turbo repository itself, and started just running stuff against that. And I noticed pretty immediately the amount of noise that was in my readings went down by a ton. And so, yeah, I built up this nice big scripts that if you go on X and read my thread, you'll find it in there. And effectively, I take my test fixtures, those were the repositories I was working on, loop over them, run a bunch of hyper fines, run a bunch of profiles, and then take those results back out back to my laptop where I can kind of mess with them for myself a little bit more.

And yeah, I just kept looping through that. And that sandbox proved to be very, very useful for getting things from feeling like, oh, OK, this is like, kind of fast all the way out to wow. This I, you know, don't blink. You're gonna miss it.

Allie Howe: Yes, that's amazing. Yeah, that's a huge success story for sure. And really cool that Vercel provides their own sandboxes too, because I know there's like a lot of options on the market right now in terms of like what people are using for sandboxing. When you chose to use the sandbox, do you think about other benefits as well such as like observability or security? Was mostly just like hey I want to use this to see if it helps with the performance increase.

Anthony Shew: I would say the first thing that went through my mind is I know how just from being familiar with our product and talking to my and other engineering friends around the company that I knew I was logged into Vercel CLI have all of our stuff set up right. We're constantly dog fooding and making sure that our own products work for us. And so Part 1 just using my friend's stuff, of course, but Part 2 was, I'm already logged in. I know that everything that I use in the Vercel stack already is going to play well. Vercel sandbox. One of the things that you can do with a Vercel sandbox is that you're, you're already going to be connected to AI Gateway.

For instance, if you're building an agent, you want, you know, some AI workload to go into a sandbox. We have the ability to put OIDC tokens into that sandbox. Those will work with Vercel AI Gateway right out-of-the-box so you can just write your code. You don't have to really worry too much about like authing to an external service for anything that you need done inside of that sandbox for that agent. So yeah, I just knew that like everything that I was already using was going to pair really well with this and then on top of that I could give feedback to my friends and I even like made a couple of PRS with TomTom Leonard who works on the sandbox SDK.

Me and him went back and forth making a bunch of PRS so that was that was useful too.

Allie Howe: Amazing. Yeah, that's really cool that you can put like the OIDC tokens within the sandbox also So I know I'm here at Daytona Compute today and and this morning Writer CTO got on stage and was talking about agent identity problems are like the hardest problem to solve or one of them, he said. And when it comes to deploying agents, especially deploying them securely and the fact that you can use sandboxes to limit your blast radius in terms of secret exposure is helpful. But also like if you've got an OIDC token, that's short lived. And so even if it does, you know, get out or get exposed, hopefully it doesn't, that's not the goal.

But if it did, it's short lived. So maybe it wouldn't work or work for very long. And these are the kind of things that I've sort of been talking about for a while, like, hey, we need to build like defense and depth. It's not just one single thing like one LM guardrail that's going to secure your agent. You do need layers in terms of like sandboxing and making sure you've got the right like auth solutions in place and short lived credentials. That's really amazing that you all are thinking about that. What is next for you? Or Vercel sandboxes or turbo repo?

Anthony Shew: Gosh, I think some things to look forward to for Vercel Sandbox. I know the team is working on a 2. 0 beta, I think it's just for us for salience at the moment, but I'm sure by the time this airs you'll have access to it. Going to make sandboxes a lot easier to work with, particularly at the persistence layer. Something that we were noticing was that a lot of our customers were having to juggle the life cycle of a sandbox. And you would use that sandbox and then if you wanted a snapshot, you would like call snapshot. But sometimes you forget and you know, OK, now you got to go grab that snapshot.

If you did remember, you want to make sure you use it in this specific workflow and not that one. And we were noticing a lot of just having to manage a lot of those things. And at the end of the day, you just want the sandbox to go up. You want to go down when you're not using it. You want to come back when you don't want it, right? Like or when you do want it, I should say. And so they've given them a very just super, super simple way in this sandbox, SDKV 2, just start that sandbox, close it down, start it up, start it down, start it up, start it down.

It just feels really natural. It just feels like actually what you wanted to begin with. So it feels like a nice evolution of like not having to deal with a lot of stuff that doesn't matter to your product. It just works the way you would hope that would.

Allie Howe: Amazing. Yeah. And that ability to kill a sandbox is super important too, in terms of like, limiting costs and also like if you do have an agent that becomes like a security threat, you could shut it down pretty quickly. So that's definitely a key feature too. So really cool to see all that. Yeah. Well, thank you so much for taking the time to come on the podcast today and share your sandbox story, and Turbo Repo story with us. If people want to try Turbo Repo or learn more about you and join the conversation, what's the best way for them to do that?

Anthony Shew: Yeah, visit us at Turbo Repo dot Dev. I'm Anthony Shew on X if I'm interesting, but Turbo repo's probably way more interesting. And yeah, you can get started with Turbo repo really quick and have your first mono repo or your next mono repo, I should say, with NPX. Create Turbo at latest and you'll have a turbo repo that you're gonna love, hopefully.

Allie Howe: Awesome. Thank you so much, Anthony, for coming on the podcast today. And we'll be sure to check out Turbo Repo.

Anthony Shew: Thank you.

Dex Horthy: What's up? I literally just got here so like how's how's the vibe been? How's the other conversations been like? How do you feel about the show so far?

Allie Howe: Yeah, I know. It's been going really well. I'm super excited to be here at Daytona Compute running about sandboxes. We got to hear from a lot of interesting speakers. I heard from the CTO of Writer this morning. He was one of the speakers and he was talking about lots of the hard problems today in AI and how sandboxes can help them. Things that I thought was notable because I'm always understood, of course, in like the security side of things is he was saying that being able to kill an agent is something that a sandbox can provide. You can just shut down the inference loop immediately, basically.

Dex Horthy: Yeah. And remember, like you did the OWASP Agentic Top Ten podcast with me in December. Remember the CSO that we had on an episode of Stanley with us?

Allie Howe: Oh yeah, the DBT guy, right?

Dex Horthy: Yeah, yeah, yeah, I think it was, I think it might been him on that episode that was saying like Rogue Agents is one of the OWASP top ten that he was like the most concerned about. Something off there running and no one knows what it's actually doing and no one's even maybe aware that it's doing stuff kind of thing.

Allie Howe: Yes, like misaligned, it's got a different goal now you can't trust it anymore. So essentially you just want to be able to kill it and kill it fast. Sandboxes provide that, which is, I don't know why I never like really kind of put that together, but like total sense to me because it's like so obvious. But I think one of the things that sandboxes provide is you can not only kill it, but you can prove that you have killed it and the data is gone. It's ephemeral. Like if you're in any sort of like enterprise, you know, security setting and having that actual reporting feature and able to prove that, like that's key.

Dex Horthy: And I can see that it like, it feel like CSO's probably like that model because it maps very much onto like the traditional, like something's in our infrastructure and it's moving around laterally and all this stuff. It's like, OK, I can guarantee that that thing is dead and every trace of it is gone. Then I wonder, like, what are the, what are the differences? We always run in challenge when people try to map like old ways of thinking about security on to new technology and that's where all the gaps are. But have you thought about this? How do agents change the change the game specifically?

This is like infrastructure sandboxes, right? Like how are they a part of that? And they're just like rematching security, like new primitives or like how they help with security essentially. Yeah. Like does the new generation of sandbox technology do a good job of helping adapt the mental model or helping like plug the holes that might exist if you were to just take a traditional cybersecurity like mostly network based security model and try to apply it to AI agents basically?

Allie Howe: That makes sense. Yeah. I feel like yes, before the AI agent everything mostly you could, you could have a lot of security around like the network layer and around like your cloud infrastructure setup. But now with agents, everything's moving to the application layer.

Dex Horthy: Right. And this was the, it was the Google thing, like BeyondCorp or whatever where they were just like, yeah, everyone else is just like, OK, if it's inside the network, it can do whatever it wants. And Google's like, no, every single service has to authenticate like mutual TLS to every other service. And I'm wondering like, we're probably not even doing that. Most people aren't even doing that. They're just trusting the firewall as they're like security boundary and everything inside is safe. I wonder if we need like a BeyondCorp style thing or even like what is the next layer of that? I know you talked to Ian a lot.

Maybe there's some alpha there that I haven't heard yet. But like, yeah, what is the like cryptographically secure way to secure like inter service communications when some of those services could be agents that have been goal poisoned or otherwise compromised?

Allie Howe: Totally, absolutely. And I think agent identity is a really interesting hot space and a hard problem to solve. Actually, the Writer CEO said that this morning that agent identity is one of the hardest problems today to solve. But if we're able to do that, we can give our agents not only the security, but the autonomy and the capability that they need to actually go and do meaningful work within an enterprise. And then we also don't have to give our agents God level privileges to access Salesforce or access whatever. So that's what's happening mostly today as people don't know how to solve this problem, but they want agents that can do meaningful work.

So they're over permissioned. Everything is like sitting at the application layer. So you want as much security as you can around the LLM, but you also don't want to induce latency. And all these cartels take so much time, so it's a fairly difficult problem to solve.

Dex Horthy: And have most of the conversations today been? I know you have questions. The last question, I promise. Yeah. Most of the conversations today, I feel like in my mind there's like 3 tiers of sandboxing. There's like infra level like spin me up a box, I'm gonna run my thing over there. There's like the operating system level sandboxing, like the stuff that we can run this Mac OS thing in a sandbox and it's like chrooted. So it only sees part of the file system and there's certain rules that are enforced without having to use all the way into a VM or something like Docker. And then there's like business level sandboxing like because like, I don't know, just opening a network hole so that I can like punch out to Google because I want to read Gmails is like very much a blunt instrument compared to like some sort of fine grained auth system or human approval system.

And I'm wondering like is the conversation today, I know Daytona right now focuses a lot on infrastructure and remote compute and computer use and all these are kind of just like give it a thing that's completely like infrastructure isolated. But has there been action around the like, more fine grained, more like precise instrument sandbox technologies today?

Allie Howe: Yeah, it's a great question. I think different providers have different approaches to this. For example, like Vercel's creating sandboxes, Stickhonas creating sandboxes, Docker has their own sandboxes. It'll be interesting to see how these providers approach fine grained auth because like you said, like having just like a network, an egress rule that says, hey, you can go talk to this other agent or you can go out to this other service. I don't think that really provides as much protection as people like might think. Just because if you're in a sandbox and agents can write code, they're really good at that. They could write their own code to sort of like almost escape the sandbox.

So as long as that exists like, they can do whatever.

Dex Horthy: Yeah, and they're very good at like even if you accidentally like you can forget an attacker is now like prompting your Claude Code running in your infrastructure somewhere as a human have accidentally prompted Claude where it's like okay, go do this thing. And it's like couldn't figure out how to do a git push. And I came back to my computer and it was like mucking around in my OSX system, system keychain with no malicious prompt just like it trying to solve my problem. And it was like doing a bunch of stuff where I was like, Nope, control C, control C, like you have to stop that immediately.

Allie Howe: Yes, absolutely. No, for sure. That's that's why we need solutions like fine grained auth and like I feel like at the end of the day, like that's how we're just going to that's the final layer. And the most like the hardest boundary you can have for agent security, yeah is agent auth solution. And that's like the hardest thing there is.

Dex Horthy: Like it seems like the only way to do that today. The only way I could see if I was going to like go build a company in agent auth I feel like it would involve brokering a protocol with every single service provider. Like I need the service providers to like take a public key and follow some rules about how to validate like signed request to them. And then I need a way that like integrates with the users device so they can approve actions with like a Yubikey or a Face ID or some kind of passkey thing. And then also two sided integration into the app layer where the app knows how to get permission to do a thing, whether it's like long term permission or just like one time.

And then also how to correctly transmit those. It's just like it's a big complicated thing and I don't, I don't see a clean path to solving it other than just like doing a ton of work and designing the best thing and then like trying to build kind of like what happened with MCP. It was like they just picked a thing and it happened to everyone jumped on. But most protocols, even in AI, which is moving fast, don't seem to get adopted that fast. And most of them just don't.

Allie Howe: No. And they and they get a lot of pushback too. And then any new protocol is going to be slow in terms of like improvement. Takes a lot of work to get to the place it needs to be or people expect it to be.

Dex Horthy: OK, cool. Sorry what did you have for today? I know I said last question but.

Allie Howe: No, it's, it's so easy to talk forever about this kind of stuff. Yeah. I really was curious to hear about your experience at the sandbox symposium that you hosted over the AIT girls on Saturday. What was that like?

Dex Horthy: Oh, that was, yeah. So that was really fun. Good crew. So I had this moment where I was like, OK, I am going to need some kind of remote compute sandbox environment that scales nicely and has a nice API for a bunch of stuff that I'm building. And in my head, I'm like, all right, I just got to sit down on a Saturday and like try all the things and feel out the developer experience and the agent experience and which of these are the best to work with. And then I was like, you know, it'd be fun. I bet every other founder that I know in my crew is like probably having similar like thoughts in their head of like, man, there's just a new one every single week.

And I just got to go sit down and try them all. And it's the kind of thing where it's like once you do it, you're pretty, you're pretty locked into whatever provider you choose. And so like you actually don't just want to pick one and see if it works and then keep running. You kind of want to try them and compare them and see which one you like the best. So we got like 50 or 60 people down to a place in downtown SF and we just sat for like 7 or 8 hours and just tried all the tech. We had a couple companies there.

Daytona was one of the sponsors, one of the sponsors. We had the folks who want to come out and it was a really good time. We just had a ton of good builders they're building. And so a bunch of people came up with, these are all kind of like amateur benchmarks, right? You can't do like a bulletproof benchmark in seven hours. But it was really interesting to see the types of dimensions that people cared about. So there was some people did the standard like speed, you know, security, someone, someone launched A benchmark that was like, I think they probably had Claude or Codex basically come up with like 17 breakout scenarios and then just run this breakout script in every provider and then just count the number of ones that weren't handled.

So some of them were very niche. It was like, Oh yeah, this one can look up an arbitrary DNS, which means an attacker could control it over DNS or something like this. And like none of them handled that that one. You could just do any DNS look up you want from other. So there's a lot of examples. This is just like the depth to which people went that I was really excited and impressed by.

Allie Howe: Super interesting. So the for the security piece, how did you evaluate like what does that mean? Like for how you could you benchmark that?

Dex Horthy: So again, it was like not everybody evaluated security and I, I tend, I happen not to do that. And they do this like most hackathons are like get together and a bunch of sponsors give you a bunch of credits and you're incentivized to like build things with the sponsor tech. And then like you do prizes at the end and everyone's just trying to win all the prizes. And I was like, I think I have enough people we can get in one room that just want to like learn things together and learn from each other. So it was, it was less about like go build something on this tech.

It was more like, just try all of them. Look at the things that are interesting to you. And then we did like 3 minute demos at the end of people sharing a bunch of it is like public on GitHub. One thing that was cool, a lot of people did like interactive like HTML reports of like, hey, here's all the comparisons side by side of all the things they tried in terms of like startup speed and how easy it was from like an SDK or ergonomics perspective to like create and snapshot and resume and like fork these things. Yeah, hope. Hopefully that gives you an idea of like the kinds of things we're looking at.

Allie Howe: Oh for sure, yes. And I love that style of hackathon. It's so much better than the traditional one necessarily where you can like try a different tech without the vendor like push there. What surprised you about the results?

Dex Horthy: So I was doing, I didn't get to do a full 7 hours because people just kept. The problem is you get a bunch of really smart people in a room. There's a lot of yapping that ends up happening and a lot, not quite so much like heads down building. But I did get a couple hours in, and I think the thing that was most interesting to me, it was how important it was to me to 1, like not have to do all the implementation of myself, right? We use coding agents to do this stuff. But I found that like, and so people talk about the difference between like, you know, they're like DX versus AX basically, yeah.

Developer experience versus. Like agent experience of like if you have a dev tool, like yeah, human can read the docs, but how easy is it for an agent to go like learn about this tech and implement it on stuff? And so like the the pitch that some people will give you, is that OK? Well, now that everything's been going to be ripped by coding agents, Dev X doesn't matter. And the only thing that matters is like, do you have a skill that an agent can pull? And is the skill well written and is it like context efficient? And does it lead to an agent being able to put good stuff on top of your tech?

I find myself feeling like I'm not ready, especially with new tech. I actually really wanted to understand the ergonomics and the architecture because I mean, I think we always say is like do not outsource the thinking. And so I wanted to be in the loop doing the thinking. And so there's this weird like overlap between like developer experience and agent experience that is like, and I get my coding agents at Claude Code to go explore these different options and then come back to me with enough information and findings that gives me confidence as a architect, as a builder to make the right decision. Not just in terms of like which one did Claude like the most, but also like which one do I?

At the end of the day, if something breaks at 3:00 in the morning, you can wire that up to a Sentry alert that runs a Claude Code that writes a PR that tries to fix the issue. But a certain point you're going to hit an issue that Claude can't solve, we've all been there and you have to dig in and see it. It's like.

Allie Howe: Yeah, I mean, just at the slot machine, like piece by piece, like, so you're like, OK, I guess it's like time to turn on my brain at like 3:00 AM when everything's broken.

Dex Horthy: Yeah, you're all like dopamine drained out from just like rolling the dice on the coding agent for the last two hours. I was like, OK, I need to be at least enough in the loop that if that happens, I have made a decision that's going to like optimize my chance for being able to solve it when the coding agents can't solve it. So I don't know if that's AX or DX or something in between, but like it's kind of a weird merger of parts of each of them.

Allie Howe: Yes, like you need to understand what the agents are doing or like what they're looking at, why they're failing, what you need to look at instead.

Dex Horthy: Yeah. And it's like when it's upfront like a research project like this, and I'm just like, should I be using black solar data, OK. It's not like most of them, to be fair. Like so there's no. There's different categories of coding like when I say coding agent workflow. I guess I I should be more specific. It's like, I want to like launch a box, kick off Claude Code in the box, have it like go make some changes and make a pull request. And then I want the box to die. I don't want it to be, I don't want to pay for it while it's not doing anything.

And then if I come back and I have feedback, I want to be able to resume that same Claude session, have all the same state on disk, have all the same get state, everything is the same. And some of them were like clearly designed for that exactly of like launch, let it spin down and then like spin that same one back up and be able to do it quickly. Whereas I think some other providers were more like, hey, you get a compute box and when it's done and it's done. So you either leave it up and running and you hope it doesn't crash or get garbage collected or whatever.

Or you have to write some code to basically launch your thing. And then when it's done, you got to go snapshot all the stuff you care about and save it yourself or save it in their API somehow. Just keeping the like glue it all together. And I'm not sure if that's like just because they're focused on other features or like they think that certain classes of users actually want to control and own that stuff versus having a kind of just be all black box like batteries included. So jury's still out on that one.

Allie Howe: Gotcha. OK. Yeah, no, really cool. Yeah. Thank you so much for taking the time to like talk to us about this symposium. I'm really interested to hear the results. They're glad you all had a good time and it was worthwhile. Else you'd like to share with us about what you're doing at Human Layer in terms of like sandboxes or anything else you.

Dex Horthy: I mean, the reason why I wanted to do this is like, you know, we build a system that runs on your desktop and it launches a daemon on your desktop that manages a bunch of Claude Code session. And the thing that we get asked for a bunch of users and that we really want to build just for ourselves internally, we use the thing to build. The thing is like, I hate that if I close my laptop and it goes to sleep, like Claude stops working. And so I wanted the ability to like launch a session in remote compute. And so it's, it's architected so that you could do that today.

Like the daemon and the client are separate, but we're working on a fun way to like run that daemon outside and have it connect up through a control plane where you can manage Claudes running everywhere. That's the closest I'll get to a pitch. I'm very anti self promotion, but those are the kinds of fun problems that we're working on and we are hiring. So if those sound like the kind of problems you want to hack on founders at humanlayer dot dev.

Allie Howe: Amazing. Yes, we'll have to hit you up for sure.

Dex Horthy: Super fun.

Allie Howe: Thanks, Dexter. Yes, thanks for coming. All right, we are at the Daytona Compute Conference. This time we have Renee Brandel, CEO of Casco with us today. Super excited to talk to you about how you are seeing sandboxes today, the Security benefits, especially because you know you're a security professional yourself. You spent a lot of time in this space. Casco is a security product. Would you like to introduce yourself and tell us more about you and Casco?

Renee Brandel: Yeah, absolutely. Thank you for having me here. I'm Renee. I'm your Co founder, CEO of Casco. Casco performs autonomous security testing. That means we hack software and tell those companies how it's hackable. And we obviously use a swarm of AI agents to perform this work that otherwise would be humanly impossible to do.

Allie Howe: Amazing. I know today there's so many different options for sandboxes, but sandboxes are not created equally and there's different ways to create sandboxes under the hood in terms of the infrastructure they use to actually isolate different software pieces. Say there's options like using docker containers. Or. Self hosted VMS or shared VMS are you? Can you tell us more about the options that exist today?

Renee Brandel: Yeah, absolutely. I think there's a adoption curve that almost every single developer is kind of going through. Like you start building your first AI agent, you're like, oh, it has to run some code now to perform really important work. Well, I'm, I'm already hosting the agent on a server. Let me just reuse the same server. So that's like typically like starting off, people typically do that. The problem with that is we basically now have combined the data plane and the control plane of your code sandbox together. That means if there's a single vulnerability that you can make the agent execute on, you're now affecting all of your customers, right?

So that's, that's really bad. Now then people are realizing, OK, there's needs to be another approach then running everything inside the same virtual machine, people start adopting Docker containers for isolation. But it's actually a very common misconception that Docker containers provide perfect isolation. Like most people think, Oh yeah, just because I run in Docker, it's going to be good. But the problem with Docker containers is that they actually work with a shared Linux kernel. So the moment there's a zero day vulnerability inside the kernel 1, customer can basically run some nefarious code, point that vulnerability and start start escaping effectively a container and affecting other users.

So ultimately that approach is another natural progression where developers just realized, yeah, I used to do VM container seems like the most logical thing, but it also has its vulnerabilities. And ultimately what we see now, and also similar to how Daytona is built, is this concept of micro VMs. So the way it works is effectively instead of using a shared Linux kernel, you run something like Firecracker. It's like a micro VM setup effectively that does isolation not only just on the software level, but actually deep down into the hardware level as well. Yeah. So this way, when even my most vulnerable sandbox code is running, it will never be able to affect other code is running other sandboxes.

So in fact it would be a lot of tiny little isolations based on a hardware level.

Allie Howe: Amazing. OK. I feel like yes, we need to probably put more attention on this as like the year progresses on prioritizing making sure that your sandbox is isolated from other users and their vulnerabilities. And the reason I bring this up is because I was actually just at a snooze snack event last week. It was like Reddit security event that they put on monthly or something like that. And Alex Stamos presents at Reddit and he was telling us that in the next like 6 months or so, it's going to become such that every company is going to have to worry about 0 days. Like it used to be just like the largest companies or a handful had to actually worry about 0 days because the models are getting so good, they'll be able to find vulnerabilities that humans haven't been able to find before.

So now they're finding that some of these models are so good that they're finding vulnerabilities in Linux all the way back from 2003. And a human was never able to find that this threat of like, oh, OK, a zero day might happen and that might affect your container. And if your agent, like, escapes or executes nefarious code, it's going to affect somebody else. That reality is a lot more probable now, given that.

Renee Brandel: Absolutely. I mean that reality is already here. Like we built Casco where we effectively run a swarm of agents that actively hack on software and tries to really find exploitable issues. And the things we find are just like humanly impossible to find and it's humanly impossible to do 2 dimensions. The first one is when you're starting to do security research, you need to gather a lot of data points. So for example, just a simple web application might have like 46,000 individual data points that you have to combine together and look at semantically. What is the reasoning behind all these data points? That's just not humanly possible.

The second thing is, even if you take all of them together, you don't have the time to actually test every single hypothesis you can actually you have built up, right? So really you're building a lot on human intuition. And while that is good for people to have lots of experience, the reality is it's missing like 99% of all possible attack paths, right? And so with agents coming down to play, being able to reason about all these data points, being able to develop hypothesis why things might be vulnerable, and then also executing it on this global scale, if you will. Right, by sending a swarm of them after these different vulnerabilities and trying them out with it to just test things in a matter of few hours that would have probably otherwise taken like 6-7 weeks to test progress.

Allie Howe: Amazing. And yeah, one of the common questions I get from people is like, OK, if AI is enabling attackers to perform so much better, how do I use AI as a defender to make myself so much better? And I don't think, I mean, I'm sure there's ways to use AI, you know better in in defense, but you should really use the AI offensive things yourself. So basically whoever gets there first in terms of having access to technology like Casco's or technology like the latest LLMs, whoever gets there first buys themselves the time to find all of those critical vulnerabilities that humans couldn't find for. And then you could decide what you want to do with that knowledge.

Do you want to repair and patch your own system? Probably. Do you want to use this Newfoundival really to go hack everybody else? That's that's your own discretion depending on like who you are in the world. But yeah, I I think the way that you do defense is just by stronger offense. I think that's that's why it's great that we have Casco today.

Renee Brandel: Yeah, absolutely. And I think we should really also think about like the the impact of the human attention we have for vulnerabilities because like, it is true that there's going to be more and more vulnerabilities discovered, but not every single one might be affecting your system, even though they're part of your dependency chain. So in some ways you have to have this army of offensive security agents trying different approaches to give you the guidance on OK, these are really the ones that are truly exploitable that you should focus on. Even though they might have a low score due to like due to CVSS, it could be very exploitable in your system that cause a bigger impact, whereas something else that might have a higher score might have less impact.

Allie Howe: Where can people find you in Casco if they want to try Casco or join them on the conversation?

Renee Brandel: Yeah, absolutely. You can go to casco. com, casco. com and you can follow me on LinkedIn. Renee Brandel.

Allie Howe: Awesome. Thank you so much for joining us all. Right.

Renee Brandel: Thank you so much for having me.