Allie Howe: Hey, welcome to another episode of the Insecure Agents podcast. I'm your host, Allie Howe, and today I'm super excited to be here at AI Engineer World's Fair, day three. I'm here with Diptanu Choudhury, founder of Tensor Lake. He's got 20 years of experience in infrastructure, and I'm super excited to hear his takes today on AI infra, and also talk about Tensor Lake, which provides scalable sandbox and orchestration infrastructure for agents. Diptanu, would you like to introduce yourself?

Diptanu Choudhury: Yeah. Hi I'm Diptanu founder of Tensor Lake. I've been working on infrastructure for a long time. Back in the day, I used to be a committer of Apache Mesos which was a cluster scheduler which was super early. I-i-it was, it was super early in the days of when cloud infrastructure was starting. I was at Netflix back then. I built a cluster scheduler called Titan on top of Apache Mesos, and for a long time, Netflix was using it to run production infrastructure on it. Then I was at HashiCorp. I worked on another open source project called I worked on another open source project called Nomad. And Nomad is also a cluster scheduler that is being used pretty widely even today.

And, took a break for some time was working on AI infrastructure for a bit at Facebook. I led the training infrastructure at Facebook around the time twenty seventeen, eighteen. Spent some time working on research for doing speech models at Facebook for pre-training of speech model, language ID, voice activity detector, and then got back into infrastructure with TensorLake couple of years back when I thought we had to like reimagine like how infrastructure has to be done in the age of AI. And, you know, I s- I... That, that is the mission of TensorLake.

Allie Howe: Amazing. Yeah, it's really cool that you've had all of this experience over twenty years at many different companies. When you see this AI wave and what a- agents need to run at production scales what is different about agents? Like, what new infrastructure do we need specifically because of agents now today?

Diptanu Choudhury: Yeah, I think it's interesting like how infrastructure has generally evolved in our industry, right? Back in the day, circa twen- two thousand and five when, you know, like Face- Facebook and Google were coming up we were trying to build... We were trying to go from a server running on a bare metal machine, which is stateful, with a database somewhere, which is also stateful, to running a cluster of servers on, on a lot of servers, on a lot of commodity server. Google and Facebook's infrastructure was built on that principle. And we did this at Extreme in the two thousand tens, like with, with AWS and, and cloud infrastructure, like where you are doing this at scale, and everyone started doing it.

That's where like we started working on cluster schedulers, Mesos, Nomad, Kubernetes, and, and today, like that is like the de facto standard. I think what is interesting about agents is that agents bring us back into like the stateful nature of compute. Whenever like you're running an agent, like it's doing some work it's stateful in the sense that a model is driving its state, the harness, and the harness is acquiring state and accumulating state over the run, over the duration of a, of a session. And so the local state where the agent is running is really important because the agent can resume from where it, it, it finished previously.

Yeah. And so now what we are seeing is that stateful infrastructure is coming back again, but we are also doing this at scale. Right? Like you are, you are running like hundreds of thousands of agents in parallel for some of the very successful agent native companies, where you're not running a single agent, but you're running like thousands of them together and they're stateful. So in a way, like we are running with agents, like the infrastructure feels like, you know, we are running hundreds of thousands of stateful databases.

Allie Howe: Amazing. Yeah, I feel like I've caught onto that when I was listening to the Software Factories track here at AI Engineer, where they said, "Okay, now that we've got increased context windows, we can have better memory. Now you can have an agent start to track what work has been done before." So if you're in a loop, you can, you know, not spend 1,000 tokens or tons of money trying to figure out, okay, re-retrying the same thing. You keep a record of what's been done before. But I also know ephemerality is really important for agents also to have, like, very short-lived credentials- Yeah To not be spun up for very long.

Like, you know, get a sandbox up, do something, and then be spun back down- Yeah... wait for the next task. What's the balance there? How do those two... Like, when you're designing AI infrastructure for agents, what's sort of the balance between, like, designing for the ephemerality, but also designing for that statefulness?

Diptanu Choudhury: Yeah. So I think if you think about it, like why do people want the sandbox to go away? Because people don't want to like spend compute when compute is not necessary, right? But you don't need the sandbox to be like have ephemeral state because, you know, you can suspend and resume sandboxes today. So once a coding agent is done working, you can suspend the snapshot, retain like the memory and file system in a file, and store it in a blob store, which costs nothing or very little. Then you can resuming it back. And so you still don't pay for infrastructure when you don't use it, but you can still get state.

Now, on the s- on the side of credentials, I think that's like another area where agent infrastructure needs net new infrastructure. For a long time, like we were putting secrets and so on, like in environment variables or, you know, we were giving certificates to authenticate a system, and then the system would use a certificate and do mutual TLS and so on and, you know, access another system like a database and so on. I think agents cannot be trusted because, you know, agents can do... Agents are more autonomous than a software that a human has written for the most part. And so, so, so what I see people trying to do is that people are trying to externalize the, the secrets out of like the sandbox where an agent is running.

And so you see like many different ways of doing it. I don't think there is like a way, like where the industry has settled in. The thing that I see people do most often than not is like putting a man in the middle proxy And kind of like intercepting all the calls that an agent is making to the external world, and then auditing what requests are going through tracking like what the agent is doing, like at the network level, and also like inserting secrets. There are other areas, there are other ways of like solving the problem that is going on. But I would say, I would say like it's like not yet settled on what the long-term direction for this is going to be.

I think the man in the middle proxy is like what I see getting done today most widely.

Allie Howe: I agree. That's what I see getting done today as well, and that resonates with a previous episode I did on the Integrations Podcast with Malte Ubl, the CTO of Vercel where he was telling me about the deep set harness that he built and how Vercel's AI gateway injects- Yeah... a credential like for GitHub into the sandbox, and then it clones what it needs from GitHub. Then it's done with the credential. It dies. It's no longer needed. It was highly ephemeral to begin with, and so it's just, it's done. And I've seen that across different, a couple different companies. But I totally agree with you that agents cannot be trusted with static credentials, and it's like a really big problem across the industry.

And when I talk to people that are building agents, it seems like everyone's rolling their own solutions for these. Yeah. And like the infrastructure solution isn't-- like if there is a single solution, it's not like been widely a-adopted. But it's top of mind for me 'cause I just came off of AAuth Night last night where we had Dick Hardt, the founder of OAuth, come and introduce a new protocol called AAuth, which he's building, that is a AI native, AI first protocol for agents and allows you to run your agent without an API key Which I think is super exciting. And we'll have to watch for different solutions out there- Yeah... that are building infrastructure not only for OAuth today, and optimizing for, like, the advances we have in OAuth, like ID Jag, that makes, like, cross app access and logging into an IPP one time versus logging into every MCP that you have access to every single time.

Yeah. Like that makes that better, right? But it's not the end all. We could do it much better. And so there are solutions coming down the pipeline, but we're waiting on these AI infrastructure vendors and providers to make use of what we have today, but also build for the future. And there's just a lot to keep track of, and I think that's hard for devs and people that are using sandboxes or even building sandboxes or agents across the whole ecosystem to track like, okay, like what are the best practices here? Yeah. Like what's new and, like, how can I make use of them today?

Diptanu Choudhury: Yeah. I feel like every, every new protocol takes like some time- Yeah... to, to, to get adopted. I mean, if you, if you have followed OAuth, OAuth has been around for a long time. Yes. And, it was not adopted, well adopted in the first few years of its life. And I think the same will happen now. I think the adoption, the time that it takes to adopt something new has shrunk quite a bit. Like it went from years to like couple of months now. But I think we'll have to like see like all... We'll have to try out all of these protocols and see like what sticks.

I think today, like what you see happening with AI infrastructure companies is that we're trying to do the gateway thing because that's the most simple approach. It works with what we have today. And then it kind of like solves the problem for the most part, like while we wait and see like which is the, which is like a more longer term solution here.

Allie Howe: When you... I think you meant a longer term solution, and what do you think is important for that solution to incorporate into itself?

Diptanu Choudhury: I think the thing that, like, as an, as an AI infrastructure provider, like what I feel is when you are provisioning an environment, you have to have like enough identity-... of the environment known to the external system that software inside that environment is interacting with. What I mean by that is when you're creating a sandbox, you should be able to You, you should be able to, like, pr- tell the infrastructure layer what kind of credentials it should have access to. And then somehow at the network layer, when we interact with the services, like we pass enough information about who is requesting access, and then the authentication layer needs to negotiate and tell the infrastructure whether a request should be should be allowed to go through or not.

I, I would say like keep making it like as transparent as we can where we don't have to provision anything inside the sandbox- I think would make lives a lot of easy and adoption faster.

Allie Howe: I agree. Yeah, and then people don't have to roll their own AI gateway to it. Just like an extra... Relative to how hard it is, it's just like extra infrastructure to maintain.

Diptanu Choudhury: Yeah, exactly. And I think what you see today is like, as all software is getting written by agents, the less agents are writing software and less, less agents are like managing the, the environment, the better it is like in terms of observability and reliability. Like, y-you want to be able to see, like, what is happening to the environment or what the agent is doing from the outside- Yes... without i- without understanding, like, every line of code that is getting generated, because we're writing a lot more code with agents, so a lot of code is getting checked in and a lot of code is getting merged.

So I think it's important to, like, just take it out of there, and then, like, have, like, the control plane somehow, like, manage the credentials and so on.

Allie Howe: Yes, absolutely. I feel like there's a lot to put together when in terms of like, you know, security for agents in general. It's, like, not a one solution fits all thing. I think it's a lot of things coming together to make that work, and then sandboxes are for sure one of those parts. Aside from security, what are you seeing in terms of, like, what people are expecting now from sandboxes in terms of, like, number of nines and the performance?

Diptanu Choudhury: Yeah. So I think with sandbox infrastructure there are two axes that we have to optimize on. One is speed at which, like, we spin up sandboxes, and a lot of times people tell me, like, you know, it doesn't matter whether it's, like, two hundred milliseconds or two seconds or three seconds. I feel like people who say that hasn't built an agent, like, to the point where a lot of users are using them, because you don't want a user to see, like, a spinner while your sandbox is work- or spinning up. When, when someone, when someone is interacting with an agent, you want the agent to start, like, immediately and do its work.

So that is, like, one aspect, like, where we are optimizing. The other op- area, like, where we are optimizing and people really care about is reliability. Yes. As, as people are building agents now which are in production, these are not science experiments anymore. People really want the underlying infrastructure to be very, very stable. You almost want your sandbox infrastructure to have as much reliability as a hyperscaler has, which is like five nines of availability, at least four nines of avail-availability. And what we are doing is much more complex than what EC2 does, for example, in terms of the amount of resources that we have to manage.

EC2 is not, like, stateful for the most part, unless, like, you use EBS or whatever. But from sandboxes, you expect statefulness, right? You expect statefulness, you expect the, the sandbox to be alive for as long as the agent is running. You don't, you don't... You can't lose a node and lose the agent that was also running in it. Or if the agent is resuming, you can't lose the snapshot. And you need to have enough capacity and so on. So there is a lot of tension between s- the speed at which, like, people want sandboxes and and also, like, the amount of statefulness the, the sandboxes have.

So building infrastructure, a reliable infrastructure for this is, like, extremely difficult.

Allie Howe: Yeah, no, that makes sense for sure. And it's really interesting to think about like, what agents need, and, like, why would they actually need all of these nines to begin with. And you think about, like, a healthcare agent or something that has to spin up to make a decision super fast or something in, like, financial services. I mean, also in voice agents particular- Yeah... I mean, latency is very noticeable, so having that reliability is, like, super important. And of course, reliability and security go hand in hand. Yeah. Having both is, is definitely essential for agents. One other thing I saw recently that was interesting as people seem to now be calling for file systems to be decoupled from sandboxes.

Yeah. Yeah. Why is that?

Diptanu Choudhury: Yeah, I think it's interesting. You know, There, there are like some really good use cases, like where you need that, and also like there is also s-some aspects of this like which is which I didn't think, you know, was going to be the case. So, so let's like dissect this. So one is, in some cases, you need a file system that you can attach to sandboxes because you want the same session that... or the same skills and so on that you want to give to an agent and all the agents that you run. So you want like a file system like where you put your skills, you put like everything else that the agents needs, like files and so on, inject it from outside, and then like have the same file system be loaded in all the sandboxes, which lets you keep your skills refreshed, which lets you like update files and keep them up to date and so on.

And so that like every sandbox is seeing like a as up-to-date version of the files and skills that it can. So that is like one use case that I see. The other interesting use case that people are using file systems for now is that they don't really trust a sandbox provider to be, have the same level of uptime that, let's say, like a mature hyperscaler has. So what people are trying to do is like they're trying to decouple the state of the sandbox from the sandbox itself. Their whole idea is that if the sandbox is not available for some reason, I have not lost my state.

I can still like mount the same file system on another sandbox, and I can resume from where I had left. You don't hear this that much, but if you talk to people privately-... in a, in a room like you, they, they would tell you that, that, oh, like, you know, the reliability of sandboxes are not that high, so I'm going to like do whatever it takes for my product to be highly available.

Allie Howe: Interesting. If you have all of those like file systems and contacts scattered across different sandboxes and maybe in version control, all these different places, how do you make sure that agents that may be touching it o-only have the access that they should? Because if multiple agents are touching the same like shared memory layer, that gets complicated fast.

Diptanu Choudhury: Yeah. So the direction that we are going, I, I, I forgot to mention like another use case for file systems. Yeah. Like file systems are really useful for coding agents. Coding agents are producing a lot of code, and so those code needs to go somewhere, right? So, so what I'm seeing, the trend that I'm seeing and the direction where TensorLake is going is that our file system, which is distributed, is going to be automatically versioned. So what that means is that the file system is out of the box Git compatible. But also, like when you write something on the file system, it gets versioned automatically.

And when you snapshot, it creates a commit in the same way like a code repository today works. So that's the direction where we're going. Now, whenever like a commit happens or like a write happens, we also can capture who is touching it, what is the identity of the agent, like who is the-- which sandbox it is coming from, what credentials it had, like what kind of stable identity it had, and then create a log of operations that have happened. So I think with a log of operations, you get the au-auditability, and then a Git-like semantics on the file system, you can go back and forth in time and do forking, cloning revert changes that a s- an agent might have made, and make it, like very transparent and auditable.

Allie Howe: Yeah, that makes sense. And it's, it's, it's very fascinating because it's like almost like two different disciplines are coming together.

Diptanu Choudhury: Yes. Building a file system is like really hard. Like, companies like NetApp has been pouring like billions and billions of dollars over the last twenty years on it. Whereas like we are trying to build the same infrastructure for agents now with different, slightly different semantics. Because when you, when you optimize for raw throughput, like you, you are not optimizing too much for snapshots, forking, and so on. So you're almost like bringing Git and file systems together now. And, we're reinventing, I think, like for, for the most part like the stack, like for agents available.

Allie Howe: Super interesting. I feel like over the last, like year or so, there's been so many, like sandbox companies like spin up. It-- From a high level they might seem like commoditized, but I think when you start digging into- Yeah... the right liability, the security, the identity, the auditability, and what all of these different providers like offer. Also the the way that they actually do the sandboxing like at the like lower level, like is it a micro VM? Yeah. Or is it like a shared like Docker- Yeah... Platform where you can like one person's compromise is on the same like Docker virtual machine as someone else, then both users could be affected.

And so I think what I've heard previously on the podcast that like I did-- I was at Daytona Compute, and I did a couple podcasts there, and people were telling me like, oh it's super important to have like a micro VM for this. I thought that that was super interesting also. So I guess like if you were someone shopping for a sandbox today, like how do you go about that?

Diptanu Choudhury: Yeah, I, I, I want to like, I, I want to like pin on one s- something that you said here. Yeah. You know, sandbox, there is a lot of sandboxes companies out there, and when you l- dig a little deeper, you understand like how they are differentiated. What I think is that, and I tell this to people, is that today you can write enough code in an hour with a coding agent which will let you like boot up a Firecracker VM in 30 minutes. But it's an endless game of optimization at the networking layer, at the scheduler layer at the layer like where you are taking a snapshot, storing them, restoring them.

It, it's just like an endless game of optimization. Like, like for example, we did this 100K benchmark couple of weeks back where the benchmark Tensor Lake to see like whether we can start 100K sandboxes in one shot, like concurrently. Yeah, yeah. The amount of work that has gone in in there we put in a lot of work, let's, let's, let's put it that way. When you are scheduling like a lot of sandboxes together, like if you're a platform, you're scheduling a lot of sandboxes together, you have to make sure that the control plane can absorb all the requests at one shot, find from a cluster of like hundreds of machines, like where sandboxes can run enough resources so that they can boot up, and send the data that they need to boot up a sandbox in less than two, three milliseconds.

And then when that message arrives on that machine, like where the sandbox has to run, you have about like 20 milliseconds-. -to start the process of booting up a, a VM where the sandbox is going to run. And if you have to restore a sandbox, then you probably have like 100 or 200 milliseconds or two seconds at most to get a snapshot from durable storage. So there is a lot of optimization that goes in, and I believe that once you start digging deeper, then you see like where, to your point, like where people are differentiating. Yes, absolutely. Like stateless sandboxes is easy. Like that is not a category anymore, and I don't think it's very useful.

At scale, reliable and stateful sandbox is like where the engineering challenges are, and I don't think people will want to build it themselves.

Allie Howe: Totally. Absolutely. And I think people will start getting into more of this. They're actually gonna try to build their own software factories. The reliability is key, and definitely that state is key also to track the work that's been done in the loop before. And if you can imagine across like different parallelized sessions, like how much context and how much memory that would actually be- Yeah And be able to recall it super fast, like if something goes down, you need to like patch something super fast or react to maybe like a security vulnerability that came in, and I need this patch, like it's critical, like ASAP.

Yeah, the reliability is gonna be huge.

Diptanu Choudhury: Yeah. Like if there is a background agent that needs to patch a security bug, you don't-- You want that to Started work immediately.

Allie Howe: Yes. Well, and it was super interesting, at RSA I talked to Alex Stamos, past CISO at Facebook, about, you know, when, you know, those MIT those was kind of like first online there, which that was on the horizon. I'm like, okay, like, you know, models are getting so much better and the offensive security attacks from models enabled AI-enabled attacks are gonna be so much better. And all these vulnerabilities are gonna be found. Like, you know, what do we, what's the counterpoint or how do we defend against that? He really advocated for defensive security agents that can react at machine speed. Yeah. Be able to take very high authoritative action, such as shut down a VM, rotate a credential basically, like make changes to your infrastructure, like in real time.

But in able to do that, they're gonna need, you know, to be spun up so fast. Yeah. Be potentially like, you know, always available, always like be... In order to have that machine reaction time are you s- like what's your customer-

Diptanu Choudhury: Well, we, we do this ourselves actually. Okay. So we are SOC 2 and HIPAA compliant. And one of the things that we have to do is like, whenever like a new vulnerability comes in through Vanta, you know, we have to rec- we have to like fix it. Yeah. Which could be like updating a package, updating a CVE and so on. Updating a package to like fix a CVE And so for a long time we were doing all of this manually. Everybody in the company hated it. Like, we had Mondays for it to, like, work on this, like, every Monday and have, like, a clean slate.

And now we have automated it. We have an agent that gets a web hook from Vanta, gets a report that, you know, something is vulnerable, and then it checks out the code and then it starts working. Because our sandboxes are stateful, checkout is, like, really fast because it already had the previous checked out version of the code, so now it's, like, just rebasing. It had all the dependencies, all the build caches are there. So then, like, it can put up the fix, do its testing, and then, like, create a PR.

Allie Howe: Amazing. Yeah, I, I feel like now when you're, like, s- doing shopping from either a security vendor or other vendor, you're just looking for the vendor that has the best, most reliable data, and then you can, like, build your own agents around them to react to it how you want to.

Diptanu Choudhury: I, I, I, I think that's the future. I feel like most of the SaaS software that we are using today is going to be, like, data APIs. I, I mean, it's not a hot take. I feel like it's- Yeah... pretty well accepted right now.

Allie Howe: I, I agree. Yeah.

Diptanu Choudhury: You know, it's like, you know, yeah, you have, you have a SaaS, which is really a data API, and the data API is going to be accessed by agents and, you know, it will react to, like, anything that the, the data system is saying it needs to do.

Allie Howe: Amazing. What's one thing on the horizon that you're excited about? What's next for, like, TensorLake? Yeah. What are you excited about here at AI Engineer? Where do you think the industry's going?

Diptanu Choudhury: Yeah. So, I, I, I never thought that, you know, we are just a sandbox company. Sandboxes are, like, a feature, like, of a infrastructure platform like we are. Like, we are building a composable compute infrastructure API for people to build agents. And if you think about it, sandboxes are a very important part of it. The thing that we are working towards is giving sandboxes a private network. So today, like, people are installing everything in a single sandbox, and how they do it is that they actually run Docker inside sandboxes. Wow. So they're running a database, they're running an application server, and so on. And so they're replicating a very complex environment inside a sandbox.

Where I think the industry is going towards is that sandboxes won't be like a snowflake like this, but you will want to have an environment built with sandboxes similar to like your production infrastructure. And in your production infrastructure, you have a private network, like a VPC, and a bunch of servers which work together to provide a service to your end user. So we want to enable users to build very complex environments using sandboxes and private networks. There would be like interconnectivity between s- sandboxes. We are working on it. We should be able to release it, like, fairly quickly. Oh. We are introducing GPUs, and for GPUs, we had to invent a new file system.

GPU root file systems are pretty heavy because they have, like, CUDA drivers and so on. Usually those root file systems are, like, 10 to 20 GB. We created a new file system which is content addressable, deduplicated. So what it means is that you can start a sandbox with GPUs really fast, and then it can... It will most likely have all the files that it already needs because most of the files are shared. Whereas if you see how root file systems and Docker images work today, they don't, they're not very granular to the sense where they're sharing files, they're sharing layers. So we have a new file system that we are going to announce with GPUs.

We are working towards a distributed file system, like, which is versioned-... so that you can, when you start a sandbox, like, you attach a, a Git repository as a file system so coding agents can code and write to it, and it's automatically versioned. So we are- That makes sense... we are going towards, like, building all the infrastructure primitives that we think will make it easy for people to build those things.

Allie Howe: Very cool. Yeah. Okay, amazing. Super excited to continue to follow all the amazing work that you're doing, and thank you so much for explaining, like, what's different about infrastructure now because of AI, what do we need, why does it need to be reliable, secure, all of that. Where can people connect with you if they wanna continue the conversation?

Diptanu Choudhury: I'm on Twitter on X @diptanu. I'm on LinkedIn. So yeah connect me there. Connect with me there.

Allie Howe: Amazing. Thank you so much for coming on.

Diptanu Choudhury: Of course. Thanks, Ali.