Derek Meegan: The idea for BB was rooted in wanting to, spend less time debugging complex customer workflows or debug them just more efficiently. Browsers emit a significant amount of data Quickly context became an issue. CLIs attempt to kind of attack the context problem of MCPs, but then you want to talk about permissioning, and that becomes a whole rabbit hole with CLIs. It's a bit of a puzzle.

Allie Howe: Code Mode is better because it keeps a lot of secrets or extra information outside of the context window

Allie Howe: All right. Welcome to another episode of the Insecure Agents podcast. I'm your host, Allie Howe, and today I'm super excited to be here at BrowserBase HQ with Derek, the engineer that built BB. I'm super excited to chat with Derek about the engineering challenges, some of the security challenges, and what it was like building a very successful example of internal AI. Derek, do you wanna introduce yourself?

Derek Meegan: Yeah. Hi. I'm Derek. I'm a software engineer at BrowserBase. And yeah, really happy to be here and talk about BB.

Allie Howe: Awesome When did you all first get the idea for BB and start working on it?

Derek Meegan: Yeah. Great question. Necessity is like the mother of invention, and so there wasn't really like an idea around like, "Hey, let's kick off this initiative to build an internal agent." Although I think a lot of teams are doing that today. I actually started out at BrowserBase as a customer engineer, and so the idea for BB was rooted in wanting to spend less time debugging complex customer workflows or debug them just more efficiently. And so I invested, you know, some time in a local open code setup to autonomously debug our customer issues. They're actually quite complex. There's a lot of systems kind of interfacing with each other.

There's the browser, our infrastructure, the target website the customer's code and together that creates a, you know, dynamic debugging environment. And so yeah, it's a bit of a puzzle. So I wanted to make my debugging time a bit higher leverage and invest in a coding agent setup that would do that, and that ultimately led me to be able to deploy that to Slack. And then from there, there was just a bunch of demand for different, more general features, and it kind of became this internal agent that we all use.

Allie Howe: Amazing. So how long did it take to, like, build BB essentially?

Derek Meegan: The initial... Just to go from, like, local open code to Slack, actually quite a, quite a bit of time because I iterated on, like, several different like, local agent setups, right? I mean, obviously the first one is just, like, having one agent, right? And then quickly context became an issue. browsers emit a significant amount of data. So, like, a bit about implementation, you know, Chrome developer protocol in particular, it's very chatty, right? And we need to look over customer CDP logs to understand what exactly they're doing in the browser. So having, like, one agent y- the entire context window could basically get cooked with a single, like, CDP log and then along with having it generate scripts, like run those scripts on a browser-based browser, collect data from those browsers itself and, and very quickly you could saturate the context window- of these models.

And so then the next, you know, iteration was like, all right, well, how do I do this a little bit more savvier with subagents, right? And that exposed a whole, like, slew of issues with, like, agent handoff, right? Inference is, like, an inherent, inherently like like lossy function, right? You lose some intent between handoff from agent to agent. And then also I, I just think at the time, the UX for interacting with open code subagents wasn't super great, so I felt like, you know, once this main orchestrator agent handed off a task to, like, a session investigator agent or a developer agent it was sort of like lost in the ether, and I didn't have, like, good visibility over how to steer those sub-agents in the right direction.

And then it was, like, really difficult to kind of iterate in that instance, right? And then, you know, there's also the problem of adding, like, many different tools and how that impacts the context as well. And eventually, actually, I read a blog post from Cloudflare about-... Code mode, and I was quickly code mode pilled. Yeah, yeah. Like, I dropped the link and s- and it, it's funny 'cause I don't think code mode is broadly adopted a- as much as I as I think it should be. But you know, I think skills solve the issue of, you know, routing your agent to, like, many different types of workflows, right?

You don't have to have this, like, giant system prompt. You know, CLIs attempt to kind of, attack the context problem of MCPs, but then you want to talk about, permissioning and, that becomes a whole rabbit hole with CLIs. So code mode seemed to be, like, this sorta interesting in between, where you have the dynamic call- tool calling capabilities of CLIs. Yeah. But then also the authentication controls of MCPs. And then you add on the, the added benefit of, you know, promise.all ergonomics in TypeScript. And it was, like, super nice. And actually it was, like, a random, like, Saturday night. You know, I'd been thinking about implementing code mode.

I, I wanted... I was visiting home in Southern California. I wanted to, like, hang out with my friends, and none of my friends were available. Yeah. And so I was like, "All right. Well, you know, Saturday night, guess we're gonna code." And that was the first time I started experimenting with code mode. And I was like, wow, you know, even when we think about digesting these CDP logs, you can give the agent sort of helper functions inside of its code mode tool call to automatically pipe, let's say, CDP logs, which are, like, prohibitively large, directly to a file on its box. And then it can use, like, traditional grep or, like, jq or, or other CLI tools to reason about, like, this larger document and only expose, like, the data that's necessary for its decision making.

So yeah, big fan of, like, Cloudflare and, and code mode generally.

Allie Howe: Yeah, I think I remember reading that article when I, like... around the time I did, like, the MCP debate. 'cause a lot of people were like, "Oh, like, MCP is just, like, kinda, like, too slow," or, like, the contacts issue. Yeah. And even, like, from the security angle, like, code mode is better because it keeps a lot of, like, secrets or extra, information outside of the context window like potentially. So no, I definitely have heard that, like, code mode is sort of like an enabler for MCP, which is great because like you said, like, once you start getting into OAuth and some of the authorization s- MCP is better for that if you're building an agent that multiple people are gonna use because it provides that Totally like, granular authorization versus the CLI.

You get a set of credentials that's good for just one person, essentially. That's what it's meant for.

Derek Meegan: Right. Yeah, I'm, I'm actually really surprised the degree to which code mode has not been adopted. And honestly attribute it to, like, I don't really think it's easy for people to understand Like what, okay, well what is the benefit of this? Well, like can't models like just write a code to a file and like execute it and... But I think there's like a ton, like you mentioned, there's like a ton of interesting both like capability and security things that you can do So for example, in BB, right, all of the TypeScript functions that are exposed in the, like, code mode execution runtime are evaluated outside of the sandbox.

In the sandbox, the agent has a short-term, temporary token, and there's also, a secondary, Vercel automation bypass token credential brokered between the sandbox and the, integration proxy. And this integration proxy holds all of the real, authentication keys, right? Like, to, our CRM or our support tool, et cetera, right? And so the agent, has actually nothing important in its box. And when it writes this code, each function call, so, exact.searchSlack or exact.searchHubSpot or Snowflake, et cetera, those individual calls are evaluated outside of the sandbox even though the, code mode script is, written in this one contiguous script. so I find that just, super interesting. Being able to, pre-transform output before you return to context, super interesting.

I find the promise.all, semantics more powerful than traditional parallel tool calling. I think it's easier for the model to, write promise.all, tool one, tool two, three, four, five, six, as opposed to saying "Hey, let me, write, eight tool calls." And that streams out faster as well.

Allie Howe: Yeah. Okay. Yeah, that makes sense. Super interesting. And so code mode is, like, difficult to use, do you think- I-... for people? Or, like, what?

Derek Meegan: Well, one, like, I think it's actually much more difficult to standardize around. Because there's, like, there's another just, like, set of infrastructure, right? Like, you're already saying, like, okay, you know, you need some runtime for the agent, but now you also need, like, this runtime for the code that it runs, right? I think that's much more natural in the Cloudflare world where it's like, you know, I'm also, like, very workers pilled. I'm a big fan of Cloudflare. I, I- Yeah, yeah... I definitely like Cloudflare. They, they have, like, a really interesting take on I think just, like, computing, like, generally. But yeah. So I, I think it's... it, it can be somewhat difficult to use or, like, the... it, it, it doesn't come off quite natural.

Like, oh, you know, maybe if you don't even wanna use, like, a completely separate infrastructure for evaluating that code, you know, you can just kind of say, like, "Hey, this is gonna be an arbitrary async function," and run that on, on the sandbox. So I, I think it just doesn't automatically click for people, but every person that I've talked to that actually uses code mode is, like, just completely, like, code mode. They're like, "Yeah, this is, like, the best thing-"... "you know, since sliced bread." So I, I definitely agree. I see, like... You know, one of the benefits of agents is agents solve problems in, like, very novel ways.

And I think this is actually, like, one of the fundamental insights of OpenClaw. And you know, I'm not necessarily the, the biggest OpenClaw... We were talking about this before- Yeah... the podcast. Yeah, like, I'm not the biggest OpenClaw user because of its security model. But I think it exposed some very interesting genius around, like, two particular things with agents, right? Yeah. Number one, just like having broad access to stuff, right? Like just being able to access like a broad, like wide set of tools. Because agents don't solve problems the same way as humans, and so you can get some like very interesting or like novel ways that the agent will reason about solving a problem.

And so you unlock these like unforeseen capabilities because you're not even thinking in advance like, oh, the agent's able to like string this together and that together-... and this together. I think the creator of OpenClaw even talked about how it's... his OpenClaw like on the fly created... like was able to ingest a voi- voice note because it, it could access the internet, and then it used the internet to get some like speech-to-text, and then used that speech-to-text to understand like what the direction was and then like execute on that, right? And I think the other thing OpenClaw exposed was even- how important event-driven executions were- to that agentic feel, right?

Like, oh, this is AGI, right? Because it's not just waiting for me to tell it to do something. It's seeing that something happened in my life, like an email came in or a calendar event is coming up or, my grandma texted me, and then it's proactively either prompting me or performing some action on my behalf. and I've seen event-driven executions be powerful, internally at BrowserBase as well.

Allie Howe: Yes. No, it seems like from the article that you all published that that an event-driven capability has led to a lot of like increased like value at BrowserBase in terms of- Yeah I think the stats that you all shared were that the feature request pipeline is now at like 100% coverage. You've got support first time to res- respond to support inquiries down to like less than like 24 hours for everything. So it's qui- like clearly been like a huge like unlock in terms of like productivity, like internally at Bro- BrowserBase. Is that just because of its ability to react to these like events like au- autonomously or anything else?

Derek Meegan: Yeah, I mean, I, I, I think it's a couple things, right? You, you know, I, I see the, the adoption curve that we saw at BrowserBase, and I think, you know, we're, we're seeing it, like, more broadly as well, is you have, like, one person who's, like, experimenting or several people, right? Like, I, I wasn't even the first person to build an agent that's, like, plugged into our stuff at BrowserBase. Shout out to Shubhankar for building the first one. It was called Magic, right? Okay. And so it got some inspiration from that, of course. But so you have, like, some, some key, like, experimenters, right?

Like saying like, "All right. Well, what is this stuff? Like, can we, can we make it useful," right? Then you have something that's, like, actually useful, right? And, in, in the case of BB, it was, like, so useful. I was able to start, like, triage... I was, like, five or six, you know, support requests at one time, you know, send them off. And, there was a couple instances that were illuminating, right? Where it's like, oh, like, I don't even need BB to do the hard thing. I just need it to reliably do easy things, right? And so there was one investigation where the customer was like, "Oh, well," you know, like BrowserBase, we record browser sessions for you.

So for observability. You can come back and you say, "Hey, like, what," you know, "What did my agent do on this website? Like, why did it fail, or why did it succeed?" Et cetera. And they were like, "Yeah, I'm not seeing the recording.". And I would've, you know, gone down into, like, okay, well, you know, the recording pipeline and, like, what... You know, looking at the dashboard logs and, like, trying to piece things together. And you know, I expected BB to go on this, like, very, like, long trajectory to kind of search all of that information, and it came back rather quickly and was like, "Hey the customer denoted that they didn't want the session to be recorded in the API."

They said, "Record session," like, "false." And it was like, boom. Like, that just saved me so much time so anyway, so you see this adoption path where you have someone innovating, like, on a local machine. That becomes powerful to the point where that gives them enough time to invest in deploying that to others. People are using that synchronously, and then people start to say, "Hey, you know, like, I'm always just, like, a, a proxy for this event," right? "This event happens, I take the information, I give it to the agent. Like, why don't I just cut myself, like, out of the loop and when that event happens, just immediately have BB triage it?"

Yes. And so there's this adoption curve, and what's interesting about that is there's so many more events in an organization than there are synchronous conversations, right? Like, a person has to make a choice that when something happens to reach for the agent and do something, whereas, like, an event is just an event, and it's just an event. Like, it's gonna happen, and when it happens, it will trigger that trajectory and something will happen. And that's been powerful for productivity, in my opinion, and, like, exposing the ability for users to say like, "Hey, you know, when this GitHub issue is open, or when this PR is closed, or when this customer support issue is closed, I want to attach a trajectory to this event with a special directive and a scope set of tools, and that thing to just always fire off in the background or just on, like, an arbitrary schedule."

And that is where you're, like, eliminating tire... entire classes of, of work for people and allowing them to focus on work that's, like, distinctly requires a human.

Allie Howe: Is a trajectory like a mission or a trace? Like, what's that primitive?

Derek Meegan: Yeah, I mean, a trajectory is the, the execution of an agent. Right? And so you can in- invoke one, like, synchronously, right? And, and sort of, you know, message back and forth like a chat, right? A chat is just a, a, a window into a trajectory, right? And so yeah. So same thing for an event. An event happens, that kicks off the agent, and it does something. And of course, we record, like, those trajectories and the tool calls and the token usage, et cetera.

Allie Howe: And then what was trying to get, like, adoption across, like, all of Razorbase to use this tool? Did it work right away for everyone? Do you roll out to certain segments first? Like, what surprised you about it?

Derek Meegan: Yeah. I, I think I just, like, broke it several times- Yeah... Like, over the course of a, a few months. It was definitely, like, not, like, super pretty or super clean. I don't really come from a traditional engineering background, although, like, I've definitely like, self-taught myself a lot of, like, sel... Software engineering stuff or, or, you know, worked in a... and I now am, like, a full-time software engineer. But yeah. I was really, like, focusing on creating capability, and it was, like, a huge, I think, like, research and, you know, research and development effort and, like, really understand, like, okay, you know, I have basically a functionally unlimited token usage.

Paul, don't... Though it's a functionally unlimited token usage, right? And then I have this, like, tranche of labor that I would like to complete, and I would like to use these tokens to complete that labor more quickly. Yes. So for me, the adoption was fast, and for many others, the adoption was fast. But there was, of course, like, some w- For some, some skepticism, and then others, it was more friction, right? You know, it's, it's actually rather like a tall order to like walk across the office and go to someone and say like, "Hey, like I know you're doing these things, like I know you have your way of working, but now I want you to work differently." and that's actually like a, a much taller order than people think.

Allie Howe: Yeah, it's a people problem- Yeah... not just a technical problem.

Derek Meegan: Yeah. Yeah. There's a huge like people hurdle that you have to overcome in order to get like broad adoption. And then, you know, on the other side of that was the security piece, right? Yes. We have a lot of like very security focused engineers at BrowserBase. You know, especially in the context of all of these like Shai Huluds and like- Oh, yeah... You know, companies getting hacked and their GitHub repositories getting yanked. Like I don't want any of that to happen, you know, not only because y- you know, we want our company to be secure and our customers to be secure, but also like, you know, I'd like to, you know, keep developing on BB, and if there's a serious like security incident, that could threaten that whole development effort or project 'cause you create like existential risk to the business.

So there was a lot of like upfront conversations with our engineering org around like what are some hard boundaries around like what we can plug the agent into and what we cannot plug it into. What does like permissioning look like? Like is the agent even allowed to install arbitrary packages on its box? Which the answer is no, right? Yes. You know, or there's no like direct connection to our production DB, right? Because even if it only had read access, you can write bad read queries that can create latencies on our production database and therefore impact our customers. And then even for like more sensitive tasks, you know, we are BrowserBase, right?

We wanna give our agent a browser, right? We wanna give our agent access to the web, right? There's a very like real two-way door problem where it's like, okay, yeah, I, I want my agent access to browser, but I also want it to like read data from Snowflake, right? Well, how to prevent agent to... from reading data from Snowflake and posting it like online if it's like prompt injected or it like arrives at a malicious website, and somehow that malicious website is able to get BB to invoke a tool and expose that tool output. And that's like a really difficult problem, right? That's like not immediately obvious.

And, and there's like two ways to attack it, right? One in the synchronous workflows. Just don't let the agent access like arbitrary websites. And so we were able to, of course, you know, having rich visibility into how browsers work, instrument the browser tool in a way that it required specific approvals for each domain. And so even, you know, in the face of like... You know, for me, my opinion is that harness security... harnesses should be verifiably secure. And the security of a harness is the root of its scalability. If a harness is not secure, we're not gonna be able to scale it to, to different use cases 'cause there's, there lacks trust.

Yes. Right?

Allie Howe: And then you're not gonna be autonomy and the capability out of your agent that you want.

Derek Meegan: Exactly. And so like one of my success metrics is like maximize agent executions like per day.

Allie Howe: So security is definitely something, you know, you're thinking about from the beginning, right?

Derek Meegan: So for synchronous workflows, you know, even if we do... Since we do wanna permit people to use a browser, I, I at least need some level of accountability over like, hey, you approved this domain. Like you are the one that like created this, right? Fundamentally, LLM, you cannot hold an LLM accountable. Like you can't hold a calculator accountable. It's like the same thing for an agent. Now for the background trajectories, that's like a whole other conundrum because it could be consuming information from like outside of the org, right? So you think customer support tickets.

Allie Howe: So untrusted data essentially.

Derek Meegan: Untrusted data. Yeah. Malware- Yeah... essentially, right? Prompt injections. Prompt injection.

Allie Howe: Love it. Yeah. So how do you make that like verifiably safe?

Derek Meegan: And every event traje- every event-driven trajectory has least privileged tools, right? So a lot of times we think about this, you know, your open code.json or your Claude SDK configuration, or your AI SDK configuration as this like static like code file, like static artifact. And, you know, we version control it in Git, and then based on the new version, you deploy that. And so I kind of, you know, brought that into question in the BB harness and said, "Okay, well, what if the tools that were at the agent's disposal, the skills that were at the agent's disposal like software that's available in its environment, right?

So that's like certain CLIs or, or, or other tool, other infrastructure and even like, you know, the system prompt, what if these things are not something that's like static and you're beholden to-... but rather the result of some function, right? These are actually like dynamic values. And that function taking in the inputs of like, well, first of all, who are you, right? Who's the user that this event i- or that this trajectory is occurring for? Where is this coming from, right? Like, is this coming from some type of event, right, which is a background trajectory or is this coming from like a synchronous like workflow?

And then c- cascading down a specific agent configuration when you invoke the agent, that reflects the necessary tooling and permissioning according to that workflow. And so I think that's like a huge unlock, especially, you know, as many people are thinking about building internal agents. I think it's easy to reach for... It's like sort of, sort of what we did with sub-agents at the very beginning, right? Where it's like, oh, I have my developer sub-agent and I have my CEO sub-agent and my marketer sub-agent, and it's like all slop, right? Yeah. Basically, those are all skills, right? So we wanted to create like many agents at first.

That was natural. But then you have this like- You know, basically, you know, you have to upkeep all this different software. It's, like, much more difficult to, like, orchestrate between them and then like-

Allie Howe: It's like a monolith versus, like, microservices?

Derek Meegan: Yeah, exactly. So, like, we wanna have, like... Do we wanna have micro agents or we wanna have, like, one harness? And so I like monorepos, right? Like, I also like one agent harness, right? And th- this allows you to focus on, like, observability, security tooling in a single harness but make things that are dangerous, right, like tools like external tools, skills or other, like, software that's available on the inside the agent's environment configurable based on the intent of the workflow, right? And so you learn something about the intent from the user, and you learn something int- about intent from the source. And then I think there's this, there's this notion of mode as well that can get kind of thrown in the mix where I think people, you know, most, most popularly know this as, like, deep research mode.

Yes. And so it's the same harness, right, approximately, but you flip on deep research mode, and it changes how the agent performs. And it also gives it, like, different types of tools. I think it's another... It's, it's sort of like sidestepping some of the concerns around, like the issue with the browser tool, for example, where it's like, oh, I need to approve each domain. You know, we want people to be able to use, like, a deep research mode agent on our BB harness because the output of the deep research agent is sort of ava- available in this agentic environment that we've constructed. So you can s- you know, ask your user to say, like, "Hey, you know, you need to give me a little bit more information about your intent."

"And in return I'll give you a little bit more privileges." Maybe now in deep research mode, you know, I shut off access to internal data or internal systems, but I increase the autonomy of the browser tool or of the agent on the web, right? And that trade-off can be valuable for, for certain use cases.

Allie Howe: Yeah. It's really hard to decide, like, what context is, like, trustworthy, but if you have the right context and you have, like, the intention of, like, the user and even, like, the identity of the user and, like- Right... the credentials, like, they have access to, then you can start to put together this, like, picture and story around, okay, like, what's the mission here? Right. That this user wants to accomplish, and does this align with, like, the tool calls and what I'm seeing the agent want to do? And at runtime for every action decide- Yeah... dynamically. Like, should this be allowed or not? Yeah. That's a dumb model that I'm seeing a lot of people move to, is, like, this, like, runtime solution that's got all of these different, like, context- Yeah. pieces in play.

But it's, it's hard to gather, and it's hard to be able to dynamically make those decisions without the right information.

Derek Meegan: Right. Yeah. And, and- There, there's a substrate beneath the, the agent that you have to build. I mean, I think observability is the, the... Observability and iteration is the primary activity of developing agents. You're basically just, like, you're watching peop- like, there's four things, right? There's the model, there's what's around the model. There's the h- there's the human, and then there's what's around the human, right? And so you're basically just, like, watching agent trajectories or, like, hearing people's complaints, and then, you know, you can't really do anything about the model. You have what's around the model. You have what's around the human, and you have the human, and you can kind of modify each of those to, like, make the, the system better.

And then, you know, you basically, like, wait a month or two, and Mythos or GPT-6 comes out, and then, you know, you rebuild the entire harness to be aligned with this higher capability. But yeah, I, I, I think that we're, I, I think we're learning a lot still-... which is, like, really surprising because we came out with LLMs, like, several years ago. I think the form factor of how we interact with them is, is changing drastically as we, as we kind of mess with every piece of the stack, right? There's, like, new infrastructure or there's, like, new model capabilities or even this whole notion of, like, a persona around the model.

I think actually this is one of, like, the key differentiators of Anthropic is they created this, like, Claude persona. And how that persona mediates interactions between the human and the underlying mechanics of the model is what makes, like, the, the experience, like, so delightful. But again, that's, like, only one part of it, right? And so then now you see a lot of innovation in the UI layer. So we went into UI land for a bit. We were like, "Oh, all UIs are dead," like blah, blah, blah. Yeah. And then the Codex desktop app came out, and it's, like, incredibly pleasant to work with. Or, like, Conductor is another example of, of a sort of an innovative tool that's, that's a, a different take than, like, purely a chat interface where sort of the foundational primitive is, like, a project or an environment, and you can have multiple agents operating, like, on that one project or, or those, like, sets of artifacts.

Allie Howe: And what about MCP apps?

Derek Meegan: Yeah, and MCP apps as well. I mean, that's, like, a whole, like- I don't know. And maybe that's like over-indexing on the MCP primitive itself, right? Like, what is MCP really? It's just like tool calling. Yes. But I, I think that there's like a really, like, actually long horizon and much more discovery to do on what agentic like UI primitives like look like. And, and I think these apps can get really deep, and we'll start to see like interface providers. Of course, like every lab or, or like Perplexity is another example, m- that will highly optimize around like specific types of queries, right? So I think the first thing that comes to mind is like Perplexity's focus on like financial bound queries and requests, and they're completely molding like...

It's almost like this like evolving like application, right? And like MCP apps are similar. This application evolves as you kind of go back and forth with the agent. I think it just creates like a much more dynamic experience, and there's so many different things that you can do with an agent that you'll have like this like base like UI that's like very low cognitive load, and then the opportunity to build, to go like deeper and deeper and deeper and build something like more rich or, or representative of like the task that you're trying to complete.

Allie Howe: Yeah, absolutely. When you said that agents are not allowed to download their own software packages 'cause out of fear- Yeah... of like supply chain compromises, which is like totally, like a rational, like thing that's been like in the news recently, like Axios, LLM, like Tan Stack, like all of these different like supply chain compromises that end up like stealing your like on-disk credentials- Yeah essentially. These attacks I feel like are less scary if there's just like less credentials, like on this. If there's nothing for the, those attacks to go and steal then... I'm not saying we don't have to like worry about them entirely, but they certainly become like less like of a high priority or less of a high risk.

If their agents are running in sandboxes and you have the credentials like being like injected like in, in real time, which I feel like I saw that like in the article you were- Yeah... like you had some sort of like short-term like credential solution. Yep, yep. Like given that, like if your agents don't have like access to like static stored credentials, like does that change the like risk level there in your opinion on like- No sovereign identities?

Derek Meegan: No, because it's so tough Yeah, yeah. Like with, it's, it's really tough with agents because they just create this like almost inauditable security vulnerability surface area. So it's like even if you imagine a perfect world, like no credentials... I mean, number one, like even in the sandbox, it has access to our internal repos, right? So that's sensitive. But even imagine a world if, if we didn't, right? And we're al- we're using this for internal stuff, so it, it's gotta touch some internal, internal information, right?. All of a sudden that internal information's in its context window. Yes. That context window's in the box. Or it creates an artifact, and it's operating on that artifact.

It's operating on that artifact in the box that it generated, right? So it's like you never know at what point in the trajectory that you're at that you will be like shy hallooed. Exactly. Right? So if your- Yeah... you could, your agent could have nothing in its sandbox that's like worth anything, but it read from your s- it read from Snowflake, it read from h- you know, the CRM, right? It, it, you, you gave it its, you gave it some information like who's out here giving their agent API keys? Pretty much everyone, probably you. Yeah. So like- Yeah... You know, you don't know what the human is going to give the agent and you never know at what, at what turn that the agent may decide like, "Oh, I need to NPM install like this random packages, that ran- this random package that has like four GitHub stars."

You know what I mean?

Allie Howe: Yeah, and it's like all of a sudden like why are we curling out to this random like server and sending this credential there? Like if we were able to like actually look at what the agent's doing on every single tool call, every single action, and shut it down like on a runtime security like model- Yeah Then we'd be able to catch some of these things. And also like if we had observability in place, we'd be able to say- Right... okay, like look, like- This action was denied multiple times. Looks like something like nefarious might be happening. Like, should we go down and, like, shut down this agent?

And then- Right be great if you, like, had the, like, agent identity on hand to go ahead and, like, revoke that agent.

Derek Meegan: 100... Well, we do have a BB kill switch.

Allie Howe: Oh, you do?

Derek Meegan: I will say there is a big red button in the, the- There's, like, we have a admin dashboard. We monitor the chats, and, like, the spend, and tokens, and all that kind of stuff, and there is, like, a kill BB if we see some, like, weird unauthorized acts. We have, like, a lot of different layers of security, of course. But, yeah. Like, li- like you mentioned, the security's very important, and, and like I said, I, I don't think, like, you know, asking your agent, you know, like, "Please don't do these things," or like- To not let it work. Yeah. "Please don't, like, exfiltrate these credentials," or, like, "Please do-"

Allie Howe: We showed that in the livestream we did at Browser Based- Really? with OpenClaw.

Derek Meegan: Really? Yeah. You did?

Allie Howe: Yeah. Yeah, yeah, yeah. Yeah. Where... And it just, like, downloaded it anyway after, like, you told it multiple times- Yeah... like, "Just do it." Just do it. Just do it. Yeah. Yeah.

Derek Meegan: We actually... Yeah. In an earlier version of, of BB, it w- it wasn't as secure. There was, like like, one or two credentials, like, still in the box, and a few of our members of our identity team took the liberty of penetration testing our bot. And they were able to exfiltrate it, even, even though, like, earlier in the conversation, the agent said, like, "No," like, "I'm not gonna give this to you," like, blah, blah, blah, blah. You know, we, we anthropomorphize, like, agents, like, too much, you know what I mean? The output of an LLM is a probability distribution. Totally. And you just push that probability distribution towards your desired outcome.

That works for both, you know, prompt injecting for malicious, like, intent, but, you know, also for, like, eliciting agent performance, so.

Allie Howe: What's next for Beebe? What are you gonna add to it?

Derek Meegan: Yeah. I think, you know, quickly the real problem in Beebe became like maintainability, right? And, I think we're, we're still in the infancy of internal agents, and a lot of the big players i- in the internal agent space are, like, larger companies. And so I think maintainability is a little bit easier, right? 'Cause they have full teams behind this. You know, it's me and, you know, it's a few other engineers contributing and, and, and that easy- easily becomes like unmaintainable spaghetti, like, very quick. Not to mention the security vulnerabilities that potentially introduces, right? So the question becomes like how do you make Beebe, like, more extensible and maintainable-...

And also let operators of Beebe, who may be non-technical build, like, bespoke workflows that help, like, augment their time, right? It's easy for me to, like, code something up on Beebe to add a new capability. I have the code base in my head, right? Yeah. But if we have, you know, AEs or ops members like on the other side of the house, that may not be trivial, right? So a couple things. Like number one, really formalizing this identity gateway that sits between every invocation source and like, invoking the agent, right? And, and right now this is, like, very bespoke. But creating a proper service that's, that is that function of who are you?

Where is this coming from? And/or like are you running in this particular mode? And then cascading down the necessary, like, tools, permissions and skills for the agent for that particular execution. So, like, that's-... sort of in the front there, right? And then also, you know, there are people saying like, "Oh, hey, there's this external in MCP I wanna integrate. There are these external skills." And again, like, that's just like downloading an arbitrary package, right? Like, okay, are those chill? Are those not chill? Not sure. Right? So creating internal registries where people can bring an MCP from the internet or bring a skill from the internet, and those can be audited and approved and loaded into the Beebe harness without making specific code changes.

And though... So even though we're using, like, the code mode execute tool proxy, still aligning on the MCP format as the ingestion layer and then, you know, consuming that and then exposing that as a callable TypeScript method in that execute proxy. So you have, like, the identity gateway and then the registries, and then being able to in a GUI, like self-serves, you know, select different events that may occur in the organization- Sentry error, PR open, support issue close, a message is sent to like a particular channel or just like a pure cron, pure schedule. And then attaching some like directive, right? Like when this happens, do this thing.

S- having those tools kinda like pre-scoped based on the prompt, submitting that for approval where, you know, those tools are, are more heavily interrogated and then approving that. And so you really just make something that's more maintainable from an engineering perspective, and then extensible from non-technical operators. You know, one of the things that Paul requested for BB is that like, hey, you know, ideally this is not this like evergreen thing that we're always hacking on. And for me too, you know, I would like that. I'd like to, you know, build productively with this. So yeah, that's the goal. Maintainability and extensibility.

Allie Howe: That makes sense. Yeah, I'm excited to continue to follow along and it sounds like a lot of interesting problems there with like the identity- Kinda excited... gateway and some of the other ones.

Derek Meegan: Yeah, definitely. Yeah. I, and I think it's Yeah, it's really interesting just, like, seeing it work, right? Like, seeing it not work. I think broadly, like, we're still in this, like, exploration phase with agents, right? We're still like, you know, anthrophi- you know, mistaking an agent for a person. You know what I mean? Or like, trying to say like, "Oh, you know, I wanna put this agent in a human role," right? I actually just dropped a Twitter article today about, kind of about this topic. Like-... is fully autonomous, like, the real ult- Is that the right goal, right? And a lot of times co- and a lot of, like, AI companies and AI products, they're like, "We're gonna make this fully autonomous," blah, blah, blah, you know?

And, i- that makes me question like, you know, what sort of... You know, that's your solution, but what problems are you solving with that? And are you thinking about creating this solution from the framing of I want to build something autonomous, or I want to satiate, like, these specific business goals? Yes. And so, you know, one perspective I've brought to BB and, like, kind of espoused internally is that this is a tool to augment the capability of humans, right? This is a tool to increase the leverage and the value of time of people, right? This is not like, "Oh, I wanna create this separate entity and see if this entity can be as productive as a person."

And that's because the purpose of BB has always been rooted in tangible business goals, right? Can we drive the time that it takes us to respond to customers down, or the time to resolve customer issues down, right? Can we reach, like, full feature request coverage so we know across the board what all our customers wanna do? And then now, you know, working internally to understand, hey, how can we catalyze our development flow by curating a, an environment in which the agent's more likely to produce code that's not only functionally correct, but organizationally aligned with how... Like, you know, our styles or our idioms, or even our, like, unique business goals that may be transitory in the broader development process, but expedient to, like, get a PR merged.

So I think, you know... I don't know. It's easy to get, like, lost in the agent sauce, right? Totally. AI psychosis, blah, blah, blah, blah. Token maxing, token leaderboards, like, blah, blah, blah. I, I think at a certain point, though, y- y- the industry broadly will reground itself and say like, "Okay, maybe spending, like, a million dollars on tokens a month is like..." Like, what are we getting back from that, right? Like-

Allie Howe: Yes, exactly. People are starting to ask this question. Yeah, yeah. Yeah. Like, we're like, this is cool, but, like, where's the results?

Derek Meegan: Right, and that- Yeah... was sort of, like, the end of the article was, like, it, it, it, it... We're maybe only a few months away before CFOs and, like, operators start being like, "Oh," like, "What's going on?" Like, like, "What are we getting?" Right?

Allie Howe: Yeah. Fun's over, guys.

Derek Meegan: Yeah, yeah. Yeah. Exactly. A- a- and I think it'll be really interesting to see, like, innovation. I, you know, I don't necessarily know how incentivized the model providers are, are, you know, beyond kind of undercutting each other on cost. I think Google does a really good job of that. You know, they're very well verticalized, right? They can, you know, they not, not really compress their margins, but they can be more competitive on price than, than their competitors. I think there'll be some interesting, you know, model agnostic solutions specifically focused on curating environments in which desirable trajectories are more likely. And therefore, you know, you kind of can a modify the ratio of productive output versus just like total generative output.

Allie Howe: Yes. Yeah, absolutely. Yeah. It'll be interesting to see like where that goes from here. But yeah, definitely would love to like, hear how, like you end up solving those challenges with like the gateway and some of the other ones and... But I think you all are doing a great job, like congrats. Thank you. It's one of the best, like internal AI stories I feel like I've seen- Thank you. Yeah... out there in the public. Yeah. So it's been... It's amazing that it only took like a few engineers to build it. Yeah. So congrats. Super cool.

Derek Meegan: Thank you so much.

Allie Howe: Thanks for coming on.

Derek Meegan: Thank you.