Allie Howe Hey, my name's Allie Howe. Welcome to another episode of the Insecure Agents podcast. I'm super excited to be here at AI Engineer World's Fair. This is day three. I'm here with Andrew and Cornelia from Temporal. I'm super excited to chat with them about what they're seeing here at AI Engineer and also get into more about what Temporal's up to here at AI Engineer and talk about some of maybe the new things with MCP that we're seeing. Andrew and Cornelia, would you like to introduce yourselves?

Andrew Baker Yeah, yeah. Thanks so much for having us, Allie. So yeah, I'm Andrew Baker. I lead developer relations here at Temporal. Before that was working at Twilio for a long time. Did like a little founder stint in between. Also organize the SF AI Tinkerer's meetup alongside a couple other very talented AI engineers here in SF. And so this is like my favorite event of the year. It's been so exciting to be here this week.

Allie Howe Amazing. Super.

Cornelia Davis Yeah, hi everyone. My name is Cornelia Davis. I work at Temporal as well as a principal technologist and have been doing this... We'll talk a little bit about Temporal, but Temporal is about, all about distributed systems, making them resilient and durable. I have been doing distributed systems for a long time. Probably part of the best time of my life was working at Pivotal and working on Cloud Foundry, and then eventually Kubernetes and a number of other things around the whole cloud native ecosystem. Wrote a book on that called Cloud Native Patterns, and so it's interesting because a lot of those patterns are coming up again here now in the AI space, so...

Allie Howe So interesting. Can you tell us more about what those patterns are that are coming up again?

Cornelia Davis Yeah. So I'll give you one example. It's related to the talk that I gave here at AI Engineer in that my talk was on a- MCP Async, or otherwise known as MCP Tasks. And it turns out that in the beginning, MCP was this really overly simplified protocol. It was just a request response protocol. It was just like in the early days of the microservice architectures, we did everything with HTTP requests where we expected to get back a response, and then we had to put all sorts of patterns around that to make those things resilient. Like we had to put in automatic retries, and then when you do lots of retries, you have to put circuit breakers in s- in, in place so you don't DDoS your services, and all of those types of things.

And then eventually we moved to like event-driven patterns, event sourcing patterns. Well, that's exactly what's happening over the MCP side. So the initial version of this spess- spec was just request response. In November, they added an async protocol- Yeah... which they're actually now revving pretty significantly. It's coming out in July. They've already given a preview of that, and so I talked about how you can manage these async... Because MCP is essentially APIs for agents, so now the... Sure, the agents are doing the calls, but we still have long-running asynchronous things that need to happen in the background, and even more so every day.

Allie Howe Yeah, that's so true. I I did an MCP debate at AI Engineer Code right around November, and I remember, like, we did that and we talked about all the problems with MCP, and then literally like three days later, a new version of the spec came out. That was in November, yep. And it, it improved so many of the things that we talked about. I was like, "Wow," like that was one of the pieces of the content of like, you know, sometimes I film content and it lasts a while, but sometimes it just, it's, it's dead on arrival. And that was kind of one of those pieces I was like, "Oh, like it really improved."

Cornelia Davis Yeah, I mean, that literally happened to me as well because I was putting together my talk for this, which had been accepted, and so I did everything around V1 of the Tasks protocol, which came out with the November spec. And literally I finished that work, and then there was a blog post from Angie Jones of the AAIF, which is where MCP lives, saying, "Hey, we're V2-ing it." And so one of the other things that I would love to mention is that the other thing that they did was that the V1 of the spec had a lot of statefulness in it, and statefulness does not scale well in a distributed environment and in a large scale environment.

And so they've now gone stateless, and then they're also growing up. They have an extension framework where OAuth, for example, is an extension. So they, the, the title of her blog post was MCP is Growing Up, and it really is.

Allie Howe Oh, yeah, that's amazing. Yeah. 'Cause MCP has so much value. It's almost like a reverse integration proxy into all the tools you already have. So it's like no one has to write their own like Jira extension again, which is like great.

Cornelia Davis Yep, yep, and it, and it's a surface area for all of the security stuff that you need, you know? It, th- again, it's another one of those things that we knew from these earlier microservice architectures is that we, we had APIs, and we built API gateways where you could do everything from security to throttling, all of those things. Well, MCP gives us that same... It's the API that we can put A, you know, API or AI gateways in front of, and so it's super valuable there as well.

Andrew Baker Yeah, yeah. I, I totally agree, and I think like part of the debate on it this past year, right? It's like, of course, like MCP crashed onto the scene, like, like so many technologies, right? Maybe it got a little bit ahead of its skis in terms of the hype wave. But, I think it really is the right tool for the job for so many types of agents that folks are building. And I've even noticed this, like Cornelia and I have been talking about it here at the show. I think because AI coding agents are the most, you know, successful and widely adopted type of agent so far in the industry, right?

I think a lot of engineers and maybe even product leaders sort of have some tunnel vision on that type of architecture being the best way to like build an agent. And so a lot of the, you know, discourse around like sandboxes and like giving your agents the ability to like write any kind of arbitrary code, pull in any kind of library, you know, that works really, really great for these like long-running agents using the most intelligent models, working on like very, very complex tasks. But I don't think that's necessarily the agent architecture that's gonna be the architecture for every agent that we build from here on out.

Allie Howe No, that's a good point. And to your point about MCP growing up, I think one of the other announcements that caught my eye was the recent extension that got added to MCP around the same thing that Pr- Programs or powers Anthropics' enterprise manage auth for Claude. It's this new extension that takes the IE JAG grant and extends OAuth to make it so that you only have to log into your IDP one time not multiple times. Like ev- so em- now you don't have to log into every single MCP server in your organization. Because you're already logged into your IDP, you're now logged into all of your MCP servers.

So it makes using MCP a lot better, and it also makes-- it's a win for security too, because now security engineers and your security team has, can have eyes on where all those grants exist versus like different OAuth islands of like, oh, like I think they're connected to these MCP servers. Like, but I don't know where all these OAuth tokens live, so I can't revoke them or have any eyes onto them. So it is really interesting to see the protocol evolve and grow up over time. And so I, I th- I think MCP is like for sure, like here to stay. I don't think there's any debate about that anymore.

Yeah.

Cornelia Davis I totally agree. I totally agree, and I think going back to your use case of the coding agents, I think in a coding agent's world, there's so much bias almost against MCP at this point. There's the like, "Well, I'll just use Bash and run CLIs no problem," but that's just one use case. And when we are out talking to organizations, large enterprises that are in regulated industries, they're, they're all in on MCP. They're like, "CLI l- running locally?" I mean, even if you can wrap those things, with MCP, you've already got the wrapper, you've already got the surface area. So, so I'm, I'm definitely an MCP Yeah, for the win.

Yep

Allie Howe Yeah, no, I totally agree. And like, also like what's tough with CLIs is it's a set of credentials made for one person to use locally. It's not meant to serve- Exactly... multiple reusable, so MCB solves that too.

Cornelia Davis Yep, it sure does. It sure does. Yeah.

Andrew Baker It's funny to see, like, some engineers kind of solving this with the, like, Dev Box infrastructure and, like, architecture where it's like, okay, well I gotta get, like, some shared volume with some, like, file system credentials in, like, the right place at the right time. And, you know, like obsessing over, like sandbox, like cold start times and things like that. And, it almost feels like a little bit of like a faster horse type of mentality when it's really just the wrong tool for the job in some cases. Yep.

Allie Howe Yeah, that makes sense. One of the topics here at A&J has been software factories and, and loops. And people have said, "Oh, if you can have, like, multiple loops running in parallel or multiple agents in a software factory, if they're able to communicate and understand maybe, like, what one agent's tried before, maybe they solved part of the problem, another agent can, like, go and pick that up." And in order to make that actually work across parallelized sessions, you have to have this concept of shared memory which is great in theory, but in practice there's a lot of access control problems, and then the identity and Auth for agents really comes up there in that space.

But, you were telling me right before this was really interesting. It's like- Yeah... we've done a lot of work in the industry to keep this separate. Now we're putting it together. Yeah. Why?

Cornelia Davis It's another one of those things that carries over from this microservices world. I mean, in the microservices world we did so much to eliminate shared memory. And it's not to say that there were microservices that never shared some state in at the back end, but when it comes to, you know, multiple things executing in parallel, you've got this concurrency problem. I get the access control. For sure, there's an access control problem. But what about the concurrency problem? You've got these agents that are running in parallel, and what if one of them is overriding the other? Like transactional semantics, how does that work? And so what we really need is some kind of a, you know, even a protocol that sits around how do we govern that shared memory from a protocols perspective.

So there has to be like a, a, an either optimistic locking or pessimistic locking, and I mean that metaphorically. I think we as an industry are still figuring out what those patterns are. I almost can guarantee that we're gonna see some of the same patterns that we've done before resurface. But yeah, I think that- It's an interesting thing

Andrew Baker it, it's funny, when you think about, like, what could be the, like, atomic unit of a memory, like, entry, right? Like, it probably does start to feel a lot like a row in a database. Yep. And like, you know, all the structure and, like, transactional guarantees you'd want around that.

Cornelia Davis Yeah, that's true. Yeah. But one of the things that's interesting is if you are tr- if you are, like, keeping a, like, if you will, an event history f- that each of the agents are doing, like, they're, they're doing ticks, they're doing turns. And if we hang on to that, then we can probably layer some semantics over the top of that which says, "Oh, okay, well, let's be optimistic. Let's let two agents do things at once." But then when it comes to the, the actual memory itself, then maybe we have these queued-up attempts to do actions that we can serialize. I don't know, I'm spitballing it here.

Yeah. Yeah.

Andrew Baker I, I feel like I've talked to, like, five founders specifically here this week who are, like, trying to, like, sink their teeth into that problem.

Cornelia Davis Yeah. Yeah, yeah. It's very much unsolved.

Allie Howe Yeah. Yeah. It's, it's amazing there's all these unsolved, like, AI infer related problems right now where developers are just looking for, like, the best practice around, like, you know, maybe we've used something before from the past that should be, you know, applied now, but in, in a way that makes sense for AI agents. How do you keep track of, like, what's going on, like, in the future, what's going on in the pipeline and also take advantage of, like, what's available now, and do your best with, like, what you're building? I know you said that you were building your talk with, like, the one of a spec, and then all of a sudden, like, V2 came out.

Like, what's it like as a developer, like, managing all of this, like, shifting that's going around the ecosystem right now?

Cornelia Davis Yeah. I mean, I, I'm personally a change junkie, so I embrace every one of those. Yeah. But I, I miss stuff. Like, one of, one of the talks this morning one of the keynotes this morning was something that I felt arguably that I should have known about, and it was the... I think it's DS Pi. Oh, yeah, yeah. I had no idea about this, this notion of because I think it's so interesting that today with this, the world we live in, we're programming with natural language, and their whole thesis is actually programming models, like programming languages are more precise, and you can do your programming instead of pro-programming prompts, pro- programming code, you know, program your agents in code.

I haven't looked into it, but back to your, like, how do you stay up on everything? Well, I listen to a whole ton of podcasts. Yeah. Do my core in the morning. I've got the podcast going in the car. Yeah. Got the podcast going. Yeah. But, like, coming to events like this and seeing that this morning was, was pretty interesting.

Andrew Baker Yeah, yeah. For, for me, like, and this is my main advice for, like, anyone, like someone who's, like, working at any size as a company, like students like you really just have to get hands-on and be a builder, like every week- Yep... these days, and it's hard to carve out the time for it. Like I know especially, like, if you've been, like, working as part of, like, a big engineering team for the last few years, like, it's very easy to kind of like let a lot of this stuff maybe just scroll by on Twitter if you're even seeing it at all, right? Yeah... but, for me, like, before I joined Temporal, I basically took a year off to just get my hands back on the keyboard, like re-skill as an engineer, do my own, like, founder exploration, probably build, like, a dozen different product concepts along the way.

Some of them, you know, with, like, a founder hat on, some of them with just a, like, "Hey, let's, like, push the boundaries and, like, see what's possible." And, yeah, I feel like in this moment, you know, there's a lot of different aspects of this new chapter of the industry which have a lot of hype attached. Like, the only way to really find out which of these new ideas are gonna be able to go the distance is to try them out- Yeah... yourself.

Allie Howe Yeah, that makes sense. Yeah, I think I'm seeing this in the auth space right now. Like, for us, whereas, okay, like OAuth has gotten a lot, like, better for now using like new advances like IE Jag. But we're looking also to, like, the future of, like, AAuth, which is a new protocol coming down the line that has this concept of the agent's mission that allows you to track that over time to see if an agent's actions is gonna diverge from what it was originally intended to do, and that's, like, baked into the protocol. Oh, very interesting. Yeah. Yeah. Which I thought, like, that's so helpful.

Like, that's really gonna help, like, agent runtime solutions, and also AAuth allows you to run your agents without an API key, so it helps solve the static credential problem. And, like, what's interesting to me is, like- How do we take advancements like that and, like, bake them into what are, what are primitives are we're gonna build into our products, and, like, be able to share that with, like, users? And I'm not sure if you run into, like, similar things with it. You work on at Temporal where it's like, oh, like, you know, I think this is, like, something that I don't want to maybe, like, re-roll every time or every single developer needs to go build the same thing.

Like, I think this is really gonna stick. I wanna add this into Temporal, my product. Like, how do you make those decisions?

Cornelia Davis Yeah. Also, you know, always a very good question. Experimental. I mean, I, I'll echo what, what Andrew said, which is, you know, build and try things out. Temporal is an interesting company in that it's been around since before the AI era, and it's about durability for distributed systems. I studied programming languages in graduate school for a while, so I was really into compilers and those types of things. And then in my career, which has spanned way longer than I'm even g- willing to admit I've rarely worked with a technology that is quite as solid as this. And it's because in order to make... We do a whole bunch of magic for distributed systems.

We take a lot of that heavy lifting off of the developer, and we take that on their behalf, but it has to work. And I sometimes say it's maybe not quite as complete and correct as most compilers are, but it's pretty darn close. And so i- in order to, to create a product that, that's, that's that solid, you don't do it. You don't... You're not throwing stuff over the wall. You're not like, "Eh, if it doesn't work, somebody'll give us feedback," because there's real systems that are running on this. There's financial systems that are running on this. Stripe, for example, runs on Temporal. Snapchat, you wouldn't wanna lose a Snap, right?

Right. Yeah, yeah, yeah. God forbid. God forbid. I'm sure Snapchat does believe that's as, as, as important as financial transactions. Yeah, yeah. And so it's been, for us at Temporal, a really interesting shift to move from this like, "Everything has to be perfect before we let it out the door," to, "Okay, now we're gonna experiment." But there are ways to do that, and so I'll give you one example. You were you were just talking... There was something else that you were mentioning that sparked a thought. So Temporal has these, these abstractions that are very specific to distributed systems. Okay. And it's also, and by the way, I mentioned that we've been around for a long time.

The reason that we've blown up in this AI era is because AI systems are distributed, and they're long-running, and that's what we're really good at. And so to give you a concrete example of where it makes sense, we've got these lower, lower level primitives that you can use for long-running workflows, asynchronous operations, and things like that. We call them signals and updates and queries. So you can have a long-running agent, and then you can inject into that agent, you can inject signals, or you can query out of that agent. What we've done now is we've created... Now, those are low-level primitives, but what we wanna do is we wanna project them out into this space that we're here in the AI space.

What does that look like in an AI context? Well, what it looks like is a tool So you can take these low level primitives and you can wrap them up in a tool, so you can hand your agent a tool that is a human being. You can literally hand your agent that's a human being. Yes. And so you can use those low level primitives. The agent can say, "Oh, I need some interaction with a, a human," and it's projected to the agent as a tool, and under the covers it's implemented using these primitives. And so those types of things we wanna be able to take out, and those w- we're trying to figure out how to do these things in an experimental way, and that's a big part of what we do.

Andrew Baker Yeah, yeah. Cornelia and I talk about it a lot because I would say part of why we're both here right now is I think, you know, a experience we have with a lot of engineers we talk to these days is that they tried one of the other kind of more like batteries included frameworks for building their agent, and they felt, "Hey, like, this thing did help me get started, but like it then it kind of ended up feeling like it was getting in my way. And so I wanna sort of bump down to like one lower layer of abstraction, but still something that I can depend on to solve this big category of distributed systems problems, which I don't wanna waste my time- Yeah...

Trying to figure out." and so it's interesting seeing like some of this new generation of engineers that are finding Temporal because they're trying to build basically their own harness or like their own distributed agent platform.

Allie Howe What do you think as a distributed systems expert, what do you think about I heard someone say this morning that they think just the file system should be separated from a sandbox. So if a sandbox goes down, and you still have access to your file system, you could like pull it up in a different sandbox to solve like some of the reliability concerns there. Is that something that you're seeing at Temporal also, or...?

Cornelia Davis Yeah. There's definitely a bit of that. It's interesting because part of the whole reason of doing sandboxes is so that you can bash on the file system and not actually make long lasting, you know, like oopsie and, and kill your real system. Yeah. But one of the things that we do at Temporal, so OpenAI has the, the agents SDK, which is their open source framework that allows people... That it's a harness basically. It's a pretty low level harness. It's not quite at the same level of as the Codex harness, but it's a, a harness, and one of the things that they did earlier this year, maybe two months ago, is they released the second version of that, and they did a number of interesting things.

The first thing that they did was originally it was designed so that you could create your own sandbox and run the harness inside of the sandbox. What they did now in the next version is they pulled the harness out of the sandbox. So now you've got the sandbox sitting at the side, the harness is running on the outside, and it's instrumenting things into the sandbox. What's interesting about that is that it does allow them a surface area to be able to take snapshots of the sandbox. And so th- that's one of the advantages of them being on the outside. And so that is maybe along the lines of what you're talking about there, where now they can do these things, and we integrate as temporal, as a durable execution framework, we integrate with that.

So we can also see where those, those hook points are and say, "Oh, a big body of work just finished." We can see that because we have an event log for all those things. This is a perfect time to snapshot the, the thing. And so then, and then we, you can preserve all of these snapshots, and we preserve the entire history on how you got there. So if something does go awry you can always back up and say, "Oh, something went off the rails over here. I can back up and I can find my old snapshots."

Allie Howe Yeah. So, like, if an agent drops a database, you can, like, go back? Yeah. Actually?

Cornelia Davis Yeah. I- if you were running, if you were running that database in the sandbox, you could, yeah.

Andrew Baker I, I spent some time in, like, the sandbox's talk track yesterday, and one of my favorite talks on this topic was Adam Asam's talk from Modal. He had, like, a really great metaphor for this around basically, like, taking the distributed systems concepts of separating your data plane from your control plane, sort of into this, like, new world of sandboxes. He has, like, one really good slide. I was, like, bugging him afterwards. I was like, "Hey, like, post the slide so that I can tweet this slide." Basically a slide showing, like, hey, the sandbox can catch on fire, but if you put the right pieces outside of it, like- Yeah... you're not gonna have a problem.

The harness is outside.

Allie Howe Makes sense. Yeah. Yeah, and the credentials are outside too nowadays. Yep, that's right. So all of them have their own, like gateway or proxy that's, like, putting them in the sandbox just for the task that is at hand, nothing more than that. Yeah. But yeah, that's another piece I feel like I see people, like, re-rolling on their own. They're all creating the same patterns. So it's like, can some of this just be, like, AI infrastructure that we all use, and, like, coming together on a, as an industry around... Like, it's interesting to see, like- How people are like, what part of harnesses that people are creating are very similar and what's customized.

Cornelia Davis Yeah. Yeah. Yep. Yep. I actually think that harnesses should have kind of layers to them, like an inner harness that does this stuff that is not going to change. You're always gonna have a, you know, some kind of a loop. You're always gonna de- be to- doing tool calling. But if you're in a highly regulated industry, then maybe your outer harness has a lot of security stuff in it. If you're not, I mean, like OpenClou is arguably a harness, right? It doesn't have any of those security concerns by design, and so different verticals are gonna layer other things over the top of... And, and that's to some extent what we're seeing with these, these loops on loops, but that, that, that's actually on a, most people think of it on a different axis.

But I definitely think that harnesses should have different layers to them.

Allie Howe Yeah, that makes sense. I agree with that. So you said you spoke at Angie or just you spoke?

Cornelia Davis I spoke.

Allie Howe Oh, you spoke. Yeah.

Cornelia Davis Yeah, yeah, yeah. Yeah, so- That's amazing... yeah, and, and kind of circling back on this, you know, we're reinventing 'cause you just came up with another example of like we're reinventing and everybody's inventing these patterns again. Literally, one of the f- one of the people that was at my talk yesterday, he walked out and he's like, "Oh, thank God for MCP tasks." And then he looked at me and he said, "Why are we reinventing this again?" I'm like, "Well, that's what we do." That's true. It is a different context. You know, there's a lot of similarity, but the shape of the applications are different. The fact that they're non-deterministic means that we need an eval framework that we didn't need before.

Yes. What used to be unit tests and integration tests now is constantly running evals because these things aren't locked down when they go out to prod. So there are plenty of things that are changing in the AI space where it does kind of make sense that we have to reinvent some stuff because we're reinventing stuff in a different context.

Allie Howe That makes sense. Yeah. Is there anything you would like to share with us before we have to wrap? 'Cause it's like close to the end of the day, and the podcast studio has to go away, and-.

Andrew Baker Unfortunately, I know it's like I've, I've been... It feels, it feels like the day's like flown by. Like, I've been at AI Engineer the whole day, but it's like, oh, I feel like I almost just got started.

Allie Howe It's always like such great energy here.

Andrew Baker Totally. Totally. Yeah, I mean, for me, I'd just say, like anybody watching this at home this is my favorite AI engineering show of the year. I think it's a really, really unique community that's here at this show because so many of the other big tent events in like the AI engineering world are usually run by like one of the big labs, right? This is one of the few that I think really brings together a large community that's not under the banner of any one particular vendor. And so to me, the hallway track is like such meaningful, deep technical conversations. I learn so much when I come to the show.

If I wasn't working at Temporal, I would be buying my own ticket to come. And so yeah, like highly recommend folks going out of their way to join this community next year.

Cornelia Davis Yeah, it's a pretty great community. I was also at Code Summit in November, and I, I am somebody who... Everybody's got a different personality. I find this a little overwhelming. I'm a small town kind of a gal. But I do love it. I love being here. I also loved Code Summit because it was so intimate, and it was single track, and you really had a chance to, you know... And yes, I've, I've met more people here. It's a larger audience, but I also love the intimate things. So that's the cool thing is that AI Engineer is doing both of those things.

Allie Howe I fully agree. Yeah, I like both for both reasons. Yeah. And yeah, the, the conferences they've done in New York have felt so special 'cause the single track, and that like- Yeah... the Time Center's beautiful, and like it's a whole other vibe. Both are great, yeah. Yeah, and in November, we had Anthropic, who that was like almost their rollout of Agent Skills.

Andrew Baker Yeah, I think it had sh- it had dropped like two weeks before, but I think it was probably the first event where they talked about it in a big way.

Allie Howe Yeah. So it's cool when these things happen. It is cool. Yeah, yeah, yeah. Well, we'll have to catch up again at the next AI Engineer, but thank you so much for coming on. I really appreciate it.

Andrew Baker Oh, so much fun, Allie. Thanks for having us. Thank you.