David Cramer Usually the people that are arguing about, say, MCP versus CLI fundamentally don't understand all of the complexity that goes into any of these systems. It'sthe first time in history you can build the integration in a way that's great for you, and you don't have to go through the painful, painful hoops of like, "I'm gonna build another Jira extension, and I'm gonna love every minute of it."

David Cramer So I just built this thing, published it to GitHub, and I ran it a few times on our code base, and I'm like, "Look, I found a bunch of bugs." Like, a bunch of like legitimate vulnerabilities

David Cramer The 60 vulnerabilities we diag- These are all verified. 60 vulnerabilities in one section of our code base. The amount of money would've been, like, 30 to 50K for bug bounties if we paid that out, and it was, like, $1,000 in, in compute

Allie Howe That's really the value of Sentry and like Seer, that it's able to see all this extra context to help you as a developer find bugs that maybe you wouldn't even be able to find.

Allie Howe Welcome to another episode of the Insecure Agents podcast. My name is Allie Howe, and I'm super excited to be here today at AI Council that's happening here in San Francisco. And today I'm joined by David Cramer, CPO at Sentry and also co-founder of Sentry. Super excited to talk to David about his new SEER agent that Sentry built, and also Warden which is their harness that they built for finding security vulnerabilities in code bases and trying to chat with him too about, like, the difference he's seeing between that and maybe, like, Mythos and anything else that's going on in the ecosystem right now that seems interesting.

David, would you like to introduce yourself?

David Cramer Yeah. Thanks for having me. David Cramer, started Sentry a long, long time ago. I'm a software engineer. I have the CPO title because it was available-... And so I gave it to myself. But I, I write code, actually probably half of my... half of the hours in a good week. And so I'm still very in the weeds. I build a little bit here and there for Sentry, but, like, Warden I built, and the MCP server I built, but I unfortunately do not get to spend a lot of time on our core product these days.

Allie Howe Yeah. Amazing. One of the things I saw recently that caught my attention was the Railway incident where a database was dropped by a coding agent, and the programmer sort of seemed to blame the vendors for not having as fine-grained permissions on their, like, API keys. I think there's a lot of confusion in the industry generally around the, like, what's the sh- shared security model for AI. And then there's also a lot of, like, I think, of tweets I've seen on Twitter where it's like people seem to say they've got all these, like, agents and, like, dark software factories doing all of this, like, work for them.

But in reality it's like, it seems like we can't trust agents to do basic tasks for us. Like, where's the disconnect there? Where's the line?

David Cramer I think the disconnect-... is the internet. Yeah... if you look at people actually at, at the edge, the frontier, whatever you wanna call it, this, this week Everybody looks the same. You're like, you have a coding agent in front of you, you're synchronously prompting it to do stuff, and maybe you have two or three more running And it's outputting kind of junk. You're constantly trying to refine it, get it to be good, and that's it. That's the state of the art, despite, despite whatever is popular on the internet. I think, I think a lot of what you see is people... People are like some of it's grift for sure.

But there's a lot of like imagination and, and interest and curiosity. Whenever there's anything that's like future-leaning, people just jump to the future, even if it doesn't exist. And so I, I think a lot of this is like people imagining what could be, even if it's like technically not possible with the technology. And so I think that's one of the interesting things is like, what are LLMs capable of, like, at peak? What... They might only be capable of what we see today. And then a lot of these imaginary things just don't exist. And I, I think you see this in, like, railway, and I think there's kind of an opportunity here, but the, the challenge is like people They're trying to fast-forward.

And some people will take that risk. And I, I, I know of the incident, but I didn't read their, their total postmortem, so I don't know. What I do know is, like, an agent will kind of do what you tell it, and sometimes it'll be somebody else told it to do the thing, but it won't just It generally is not gonna, like, just make up things, right? It is just pattern matching at the end of the day from the model. I think you have to give people safety and controls, and security's like, nobody cares about it right now, but it is like the most worrying thing, I think, in this domain, and there's gonna be a lot more problems coming.

And I think you should be cautious depending on the industry you're in, like what you expose to agents or expose your customers to. Because at the end of the day, it's like, well, whose fault? In that scenario, whose fault is it? Yeah. It's definitely the customer's fault. But it's like you can say it's the customer's fault, which I, actually truly believe it is in this scenario. But if I'm the customer and that happens to me, even if it's my fault and you didn't protect me from myself, I'm going to blame you. And I'm like, that's also fair. You should. and so, we do stuff Like, we have a Slack bot that I built for work, and I spent an inordinate amount of time on, like, how do I not expose anything that shouldn't be exposed, whether it's in shared Slack channels, thus it needs to only be able to query data that either everybody in the channel can already see because it's public information in the company, or that the user has specifically asked to see that they have access to.

But that should be, like, a rare anomaly. it also needs to not have access to any keys. And if it does do anything on behalf of the user, that needs to be sort of almost, like, revocable so it can't then do more things that it wasn't asked to do. And so you end up with all these, patterns that frankly you can't really work past. And I just think, I think people should slow down. And I don't think this is in the way where Anthropic says "Safety, safety, safety." I don't-- I think their version is a little bit more religious than it is practical. but I, do think people need to slow down.

And, you kind of have to come back to, the reality that software is still hard. Valuable things are hard to accomplish. It's not just you can, you know, generate some new software and solve every problem that's in your head tomorrow. It takes a long time to build things still. And so I just think it's people being a little bit too casual, I guess, with things- that I don't think they should be casual about. And they're probably well-meaning, but they, don't fit my pattern or my sort of constraints around how to operate. And so, you know, we're very careful about this at Sentry. And, And we obviously have a lot of data with customers, but, you know, you can't really do anything destructive with anything we expose, which is somewhat intentional.

And people have actually asked, like we have an MCP server. It's like, "Oh, it would be nice to be able to delete projects from this." I'm like, "No, it's fine. You don't need to be able to do that. You can go to the UI, we don't care." It's like such an outlier case, and I think that's the problem is that people just have to recognize things versus just pretending the problems will solve themselves, And I... But again, I think we'll see a lot more of this, and especially with like less technical users adopting things, and, and even the technical folks, just people less experienced and potentially not gaining experience as fast as they might once have.

And so.

Allie Howe That makes sense. Yeah. It, it's so hard to make AI like truly useful for multiple people. Yeah. In the case where you built this, like the Slackbot that's in all these different channels, and having to prevent certain users from accessing memories they shouldn't have access to. and we see that today, all over this concept of the shared memory store- whether it's the multi-agent thing in a, company, or if it's like a context graph. It's like that's a really great idea, but in practice, very difficult. Like how are you handling authorization?

David Cramer And I think this is OpenClaw too. It's imagination. It's what could be, but practically what can't exist right now yet today- in a safe way. And so it's fine, I've got a little robot at home or whatever and it has access to see some things, and like the risk to me is low or something. It's not fine in an enterprise environment, again, in my world. And I think memory is probably one of the most interesting, domains in the- field right now, and I think it is probably one of the most difficult. And it's probably... You can probably do a good job of implementing it for a single isolated session kind of thing, but if you try to do shared memory, good luck.

Like it's so unbelievably hard to secure that. And, I, spent a few weeks trying to build one for a, personal home chatbot with my wife, and it was just a disaster of constantly exposing the wrong information to the wrong people. and it's just, very hard. Like all these problems are very, very hard. And again, it goes back to I think hard problems are hard. They haven't really gotten any easier, and that's also usually what makes them valuable if you can solve them at all. But you can't just, you just can't pretend the problems go away or you fast-forward past them or whatever. And, and so, but yeah, I think we're still early on this journey for a lot of these things.

Allie Howe Yeah. And I feel like I've seen a lot of people that are maybe approaching the shared memory thing do it their own way. Like, there's no like, you know, accepted like third-party solution for this yet.

David Cramer Yeah. Maybe there won't be, I don't know.

Allie Howe But, yeah, definitely interesting to see. In terms of like the practicality of like agents today and what they're able to do how have you seen that with the Seer agent that you built? 'Cause I, I watched one of the videos of you talking about this. I think it might be at Mastra's one of their events. But, it was really interesting. You said it's like an AI debugger that is able to gather all this context within Sentry, and that's really the value of Sentry and like Seer, that it's able to see all this extra context to help you as a developer find bugs that maybe you wouldn't even be able to find.

David Cramer Yeah. I think the value... So we, I mean, Sentry is a company that's been around a long time. Yeah. We do... We're an infrastructure company, which is actually a good domain to be in. We're much safer than a lot of companies from disruption, but it doesn't mean we're just by default safe. And we think a lot about, well, the data remains valuable no matter. It's not like all of a sudden bugs go away or any of this, right? So we're not worried about that, but we think a lot about what value can you now unlock that you couldn't before, and how will you consume like any of the value?

And so Seer is a little bit of both. It's like, well, one, how you consume it is easier. It's like, well, I have coding agents. I like don't need the UI as much, and that was part MCP. It's not even Seer in this regard. MCP, CLI, whatever your flavor is, it's just like bring the data to the right location. And for us it was like bring bugs to coding harnesses 'cause that's where you fix bugs, right? So that was already one huge unlock. And it's not be all, end all. But like quickly what you then find in that is it, there's like two problems you hit.

One is it goes off the rails. Like it has the information. It kind of does the thing, but it does it like, I don't know, 50% right. Like any coding harness does for any problem right now, basically. And then the other problem is it does completely the wrong thing. And- I don't know, as a human you can, like, kind of fix that and stuff, but, like, it's, it's worse 'cause now you're wasting time. You're wasting a lot of expensive compute at the very least. And so what we did is, like, we looked at it and, like, we have a lot of data. That data's hard to use.

And it's all kind of interconnected by various attributes and stuff. And one common example I always talk about is a lot of people these days, they'll use something like a Next.js technology or something, and you've got code that's running on the server and the client that's kind of the same application, if you will. It's, like, in the same code base and it's a little bit harder to reason about. And that's just connected by tracing it across the wire. As a human, you can kind of build things that, like, make this easier to consume, but it's still confusing. You're like, "What's the actual bug? What's going on?"

As a machine, it's very easy. You can process lots of information very fast. But historically it's been hard to generalize that kind of stuff. And I think one of the cool things about LLMs is just, like, arbitrary information you can throw at it, and it can actually do a really good job of pulling information out of it and pulling signal out, right? I, I have this a company I invested in, got acquired by one of the big frontier labs, and it was, like, the most absurd paperwork you can imagine. Wow. So I immediately upload it to another frontier lab-... and I'm like, "Tell me what happened here."

Like, I have no idea what my sort of recourse or, like, what happened to me in this, I guess. And it does it in, like, seconds. And I'm like, "This is cool." And it was, I'm pretty sure, accurate. But so w- with us we're like, we have a lot of this data. We can make it more consumable now, which is one big thing. Which is like, can we sort of truly identify, like, a root cause of problems? The value of that is not super high in, in abstract unfortunately. And it's like once a, once in a while that is, like, extraordinarily high value and saves you lots and lots and lots of human hours.

Most of the time you don't need that much depth. So okay, so that's fine. So we saw that. We recognized that. The other thing we recognized as part of that is, like, well, we can actually make things more consumable, which is actually really important with the market expansion we're seeing happening where there's just, like, new people trying to build software. Whether they should be or they will be successful kind of irrelevant, but if you wanna capture that you, like, have to think about, well, how do you make it even friendlier or even easier without sort of, like, dumbing it down? Because then you risk your core audience.

And so what we realized is, like, you can give them all this information that was already pretty curated before. We did a good job of this. Now we can make it even easier to reason about some of these, like, cryptic errors and stuff, and so that's really valuable. But what we think is the opportunity for us, and this is sort of reverse engineering who we are and what we think is plausibly doable in a time period, is you take the long tail of trivial problems in your code base like typos, small little errors that just, like, they're not that important but, like, they all take time You can throw 'em in a bunch of coding agents today, and you'll probably get like an okay outcome.

But like if we could automate a world where, a- again, even if it's the most trivial of problems that just don't matter that much, but you would still rather they not exist, and we could build you like a queue of like all these things are actually like pretty well verified that you would just mash, merge, merge, merge, merge, merge- that to me is very high value. Because everything is a productivity game. It's like, how do I take a thing that either I wanted to do and couldn't do, or that was taking me longer than it needs to now, and how do I just do better at that?

And so we just repeat that game over and over. And so that's kind of our big bet and call it self-healing, which is now I guess a trendy topic. Which again, is not possible really today for a lot of things, let alone like full self-healing. But like we think we can get to a, at least what I said, a queue of trivial bugs. We have, we actually have like an internal thing right now where like in the next, I don't know, 60 to 90 days, we need to ship this where a measurable percentage of customers could experience that reality. Again, not for hard things, for easy things.

Maybe one day we'll get to hard things, but there's only so much that's possible anyways today. And that's really what we see for our value add there. But I actually think a l- a l- that's, that's like R&D greenfield. I think a lot of the stuff for businesses is like how do you do something that was hard before, do it better now, more efficiently now, or do, or just do more of it, or something in that shape or frame. Versus like let's pretend we're gonna reinvent everything as a chatbot or something like that. Nobody, nobody really wants a chatbot for all things, so.

Allie Howe No. No, I know for sure. I, and I think some part of the value of like MCP is the fact that it's kind of like a reverse integration protocol, how they say.

David Cramer It's, yeah, it's great. Yeah. I it's like the first time in history you can build the integration in a way that's great for you, and you don't have to go through the, the, the painful, painful hoops of like, "I'm gonna build another Jira extension, and I'm gonna love every minute of it." Oh, yeah. "And the customers are gonna have all these gripes. And then I gotta go build now a GitHub integration and everything else under the sun," which was always like a low ROI thing. Now you build this. It's a little bit complicated how you build it, but yeah, all of a sudden like you can give your customer something that genuinely is useful, and you don't have to integrate it everywhere.

It's, it's so good from that regard.

Allie Howe Yeah, absolutely. And it's better for security, or it's like at a, you know, it's a point in time where you can actually physically add like a, in a network boundary or security boundary- Yeah... onto it. Versus like a CLI, which generates maybe like a static credential for like- Yeah one person that gets used by all of these agents.

David Cramer People, people... There's always hot conversation about these things. People do not... Usually the people that are arguing about, say, MCP versus CLI fundamentally don't understand all of the complexity that goes into any of these systems. Like our Slackbot, for example- has MCP built into it. It also has the ability to call like CLIs or anything else. How do you secure the CLI? Do you intercept all shell commands? Do you build a custom shell because you need to like intercept network egress and all this stuff? MCP, it's all just baked into a protocol. You've got authentication baked in. It's HTP. Say what you will, the, the protocol's a little complicated.

It doesn't need to be that complicated. But, but- Yeah, yeah... if you take that away, there is at least a tried and true mechanism to make that function because it's just how networks and the internet works, with a little bit of excessive complexity on top. Whereas like CLIs and all these other systems, they're just so more freeform that you can't rein it in. And, and again, I just think people misunderstand why people value a lot of this technology because of like, they just don't have the experience. They haven't lived in some of these worlds or something. Or they're just proclaiming stuff that they read on the internet, one of the two.

But, but we, we get a lot of value out of both. We, we co-develop both a CLI and an MCP at this point because we're like, there's a lot of situational stuff where one makes sense, the other doesn't, and there's different things you can accomplish in each of them. So...

Allie Howe Yeah. There's definitely security benefits, I agree, to like MCP as well. But also I'd love to hear from you about Warden that you built. Why did you choose to build Warden instead of just like use a thorough coding model just like we have it?

David Cramer Yeah. So Warden is... I would describe it as like a code review harness. Yeah. It's actually just, right now it's a shim on top of Cloud SDK, and then it takes a skill as a container and turns that effectively into like a- Kind of an, a coding agent, but not really. But it just tries to make Claude be like a code reviewer, basically. And then it tries to coerce the output into something useful. And I forget why I started this project, but we have... Technically it competes with a paid product we have-... And which is a little contentious internally, but I, I, I think what it was is like we had, we continue to find security vulnerabilities.

We... It's just, it's more active now, right? And I'm like, "LLMs can definitely do a good job here, but you need it to be shaped in a certain way." And I'm like, "We're not gonna build this as a product. It makes no sense." So I just built this thing, published it to GitHub, and I ran it a few times on our code base, and I'm like, "Look, I found a bunch of bugs." Like, a bunch of like legitimate vulnerabilities. And the first time it actually was not... There were no like, like high severity ones. There was nothing like very, very critical, but they were clear like, "Yeah, we should fix this." but I focused it on security mostly to play nice internally.

It does anything. It doesn't actually matter. It does anything that looks like code review of any shape or form. But I did that, and then I was experimenting. I'm like, "Okay, there's gotta be a way to get more depth out of this," 'cause the... I, I basically have this my thesis was you can do a form of reinforcement learning, which is take a bunch of prior art, synthesize that into like a skill, which is just a container for us and a bunch of references, and it'll, it'll create more accuracy because effectively, LLMs don't know anything unless you tell them the thing. Like, they, they, they can't give you the right answer if you don't give them the right answer, which is why they need search and all these other tools, right?

And so I'm like, "Well, if I give it more of the right answers, it'll be more likely to find the right problems." And it does, 100% does. But it was like kinda tedious to do that, and I was experimenting. I'm like, "Well, you have limitations of the complexity." Like, the more complicated the, the ask you give it, the less likely it's gonna fulfill it. And you see this in any of the tools, right? Yeah. And so I'm like, "Well, what if we just give it lots of different asks that equally have their own depth-... Like in an, an umbrella of this kind of thing?" And, and security was my goal, so I'm like, "We'll have a whole thing that just tries to find auth bypass, and then we'll have a whole thing that tries to find like remote injection or something like this." it turns out that wa- did not work at all, or rather it did not perf- it performed better than what we were doing, but it performed- It was not necessary, I guess is what I'd say.

And what we found was, like, I did the same thing, and I was... And this was sort of our second iteration of, of tackling security things with Warden. I did another one where I just, like, I thinned it out and built, like, a more rich context and a skill with a bunch of references that were sort of like, "Well, here's the JavaScript stuff," or, "Here's the Python stuff." Domain specific, right? Or maybe, "Here's, like, what authentication bypasses look like in this framework." And I did that, and the accuracy improved even more. And it turns out I was also accidentally using Sonnet, which was kind of interesting which is drastically cheaper than Opus, mind you.

And we found a ton more vulnerabilities. And that, that was, like, the last post I made where it was like I found two critical auth bypasses in Sentry, for example, that just had gone undiscovered for five plus years. And they were not, like... They were not the place where everybody will go to test when you have pen testers and stuff, because we do this non-stop. Yeah. And so all these things slip through, and I'm like, "But how do I get people to be interested?" Because we unfortunately are not... We are a well-known company, but we are never cool, and it's, it's hard to build hype around something.

And for me, I'm like, this is very cool what you can accomplish with this, and I think people are sleeping on it and it's a really important problem. And so I'm like, I'm gonna go find other active vulnerabilities in open source projects. And I'm also someone who does not wanna spend a lot of time doing something. I'm like, "How do I get this done and move on?" And so I'm like, I'll spend one day doing this. And so I went to Home Assistant and I went to Discourse, like the old forum software, and I, I did my sort of routine on them, ran it in both.

I found, like, I think it was, like, 50 findings in Home Assistant. They were not... Because of what Home Assistant is, they're not super critical, but, but they went and fixed that, like, a, a huge chunk of them really quickly. Discourse, I don't know if they've shared it publicly yet, but they were definitely issuing some CVEs for the findings. And that was, that was literally me spending like 30 minutes on the problem for them. And a little bit of money, but not that much. And I'm like, and I'm like, "Oh, this will get hype because now people will see it's not just a Sentry thing."

And then still mostly nobody cared. And so I think we're still... And this is where I go back to, like, I think we're still early in this industry where people just have not gotten past the dopamine hit of being able to generate some new app. But there's so many cool things you can do that the way you can approach them is different or more scalable. And, and, and you might say, "Well, tokens are really expensive." The 60 vulnerabilities we diag- These are all verified. 60 vulnerabilities in one section of our code base. The amount of money would've been, like, 30 to 50K for bug bounties if we paid that out, and it was, like, $1,000 in, in compute.

And so one, it's not cheap, but it's pretty cheap in, in aggregate. And so but yeah. And so it just takes these skills, synthesizes it into this thing, and it... And you can run it locally or you can run it on GitHub on, like, pull requests, and it's kinda like a linter is the way I think about it. That's my analogy. But it looks like CodeRabbit and BugBot and Sentry's thing, and it's actually more reliable than a lot of those for the same amount of compute spent. And you control the whole thing, so.

Allie Howe Yeah. That's really cool. I the part about controllable thing is really interesting to me 'cause now I feel like with all of this like potential to customize your own software, there's a lot of like build and buy conversations going on-... right now, where I feel like if you're buying something, you really just want like, you care about the data more than anything because you can build whatever you want on top of it.

David Cramer Yeah, and this was kind of the thing. When I told the team I'd build this, they're like, "Wait, isn't this competing with something?" I'm like, "Look, if I can build this in two days, anybody can." Then, and it's like not the easiest thing in the world, and to be fair, it was two days to get it running and then many weeks to polish it. But the reality is like this is just traditional dev tooling that will be public open source. Ours is licensed under fair source, but like, but it's just stuff that will exist no matter what. And so I'm like, "It's fine. We'll put it out there.

People will or won't use it, it doesn't matter." and it provides a lot of value to us at the very least, so, so it's, it's been a good win.

Allie Howe Yeah. Totally. And I think more people will care. I think I heard from a lot of different people that I've talked to that are very concerned about Mythos and the fact the models are getting better just because they feel like there's like some bug apocalypse that's coming. And so in order to get ahead of that, you should probably use something like Warden or- Yeah anything to start testing your code for race or vulnerabilities.

David Cramer I, I agree. And I think- And you can argue a lot of this, the necessity is because we're using agents to generate more code and, and definitely more stuff slips through. But this isn't just new problems, it's old problems too, and it makes it more efficient and stuff. And, and I, I think the verification part of code, including security, is the challenge. That's the only hurdle, genuinely speaking. Like, the coding agents, maybe they'll get better. They're not great, they're fine. But the verification has been bas- like, largely it's still unsolved. And so yeah, we have some code review stuff, but it's still non-deterministic and, and how much compute does it take?

And so there's all these, like, problems in there and I, I do think it's really, really, really important. But every company kind of needs it and, and so it was like a no-brainer to me. And, but again, I think there's lots of things like this that are really cool unlocks. Like the Slackbot is just another one of those unlocks, and it's like, yeah, you give it access to data, people can ask the Slackbot questions. They probably weren't gonna search the docs in the first place, but they might, they might, like, ping the Slackbot instead. And so it's just, it's another one of those, like, efficiency plays where it's like, yeah, I could spend a lot more time on security review and, and pen testers and all this stuff, and, and there's certainly a price to that.

But, it's all like, well, how do you do this more and ideally more cheaply? But...

Allie Howe Yes, for sure. Yeah, you said it'd take a couple of weeks to polish that. What do you think about, like, the current, like, developer agent, like, feedback loop in terms of, like, self-improving agents' capabilities? Like, I know a lot of agents will be able to, like, dynamically modify their tools on the fly. I think that creates a lot of, like, security -... concerns though- Yeah... about, like, sandbox es- escapes or evasion around, like, different, like, firewalls and stuff. What's the best... Like, should that be allowed, like dynamically, like, editing your own tools as an agent or no?

David Cramer Would you allow a-... Sentry to dynamically inject fixes into your production app? Probably not, for good reason. Yeah. Same thing. I think- Yeah... this is, this is all tech demo territory for me, which is, like, OpenClaus is a little bit like that, but if you talk to anybody that does it, they're like, "Yeah, it just breaks itself all the time." And I'm like, "Of course it does. That's obviously what would happen," because, like, verification is really, really hard. And so I think it's cool what you can accomplish, and some of these tech demos are very cool and we should pay attention to them. But everybody needs to, like, stay grounded in the reality of, like, what requirements are of real world software.

And like, it has to be online, it has to be secure. Like, you can't actually be like, "Well, if we leak something, it's gonna be okay." It's like, no, no, no, no, you gotta operate like it's never okay, and then you gotta prevent every class of problem you can. And, and so I, I don't know. I'm, I'm skeptical of that. I think the, the more interesting version is not Agents modifying themselves because, like, we're so far from that being a practical reality. But can agents continue to sort of enri- learn, if you will. Learn is not a great word, but some version of memory, whether that's like it'll just update its own knowledge base based on, like, past prompts or any version that can kinda take something and improve context over time.

I think that's really interesting, but I'm also a little suspect because my experiences are often the model controls almost everything. Like, you, you can't overcome most of what the model is. And, and my analogy here for folks if they're suspect, 'cause everybody has their own crazy belief system, is like try to build an agent using exclusive... I haven't written code, like, by hand since, like, this time of last year. And so I'm, I'm pretty-- I've got a lot of knowledge here. I'm not just talking. Try to build an agent using an agent and watch everything that goes wrong and how you can't keep it on the rails, and what it decides to test.

Or my favorite one, 'cause we use Sentry aggressively, and I'll mo- I'll instrument a lot of stuff. And no matter what, it goes and writes tests for my log statements or something like that. It's like, "Let me assert that the log statement lo- like, was called," or something like this. And I'm like, "I don't want that in my test." It's just not a way I, I write code. Yeah. And no matter what, no matter how I apply the context of that in a sort of implicit way, so agent set MD, any pattern, it doesn't matter, it's gonna do it anyways. And you cannot overcome the training biases.

This is kind of what a truth I've come to believe in based on a lot of experiences like that thing. And, and if that's true, well, then how does it shape what, how you think about things? And, and so I think a lot more people would be better off really just, like, dive in, learn, figure out what does and doesn't work, and then form some conclusions versus, like, a lot of what happens in the industry right now is, like, I don't know, people read a 100-character tweet and they're like, "Oh, that's the truth." And I'm like, usually it's, like, zero truth. And there's a lot of placebo even more so, where you're like, you convince yourself something's working, and then you're like, "Oh, I don't think that did anything at all."

Like, you just, it's like this grounding moment because it is so random. Like, it's so non-predictable. But yeah.

Allie Howe Yeah, like the latest one of people are like, "Oh, you need to switch to HTML files instead of- "... get them marked on and watch your context window explode."

David Cramer I know. Yeah. And then there's, like, some things that are pure usability, which I think are interesting thought experiments. But again, it's like the, the reality moment over and over, and I don't know, you constantly see this, like the, the Ralph Wiggum thing that was, like, popular forever. It's like, it died so fast. Yeah. And before that was an even, like, more crazy version of the same thing, and, and it's because they just don't actually get you to the true goal, which is not just to produce some junk thing. It's like, "No, I need to solve an actual problem," and unfortunately producing software is one of the harder problems that we can go after, you know, versus search.

Search is great. It's phenomenal at search and synthesis, so. But, but yeah, I think there's a lot there that is just still... I don't know. People are sort of figuring out what they can and can't do. There's just not enough that's been Sort of, there's a little bit, there's too much optimism, I guess, is what I would say. We need a little bit more skepticism or, you know, maybe a little pessimism in there to just, like, ground it a little bit more, so.

Allie Howe Yeah, definitely. Yeah, I mean, I feel like there's definitely, like, whole categories of developer tooling that could be, like, wiped out now that we've got agents. But the principles of, like, verification of software is still very much, like, important and something that people are struggling to get right today, and I think always kind of have. Besides something like Warden for the security skills, what other, like, verification mechanisms are good for, like, agent development workflows?

David Cramer So what I think is interesting is, like, so what we do internally is just traditional. It's like we do tests, which I actually think TDD is a really good practice now with agents, except they often write bad tests, so you, you gotta still reel them. But, like, we mostly just do normal CI stuff, and then we've layered in some of these agentic scanning things, which is mostly Warden. But, like, we use our own stuff, we use BugBot from Cursor, and they're great for picking up a bunch of things. Like, it just makes it, you know, a little bit better of what's happening. Beyond that, I honestly don't know, 'cause the, the, the pipe dream is like, well, what if you could actually verify the code?

Like, like, see it's working in, like, a, a canary version of production and prove it that way. But the reality is you can't. For any system that's worth money in the world, you can't do that because they have data and they have databases and all this stuff that's stateful, and you can't really reproduce that, that scenario, right? And so you gotta take that off the table, and then you're like, "Well, what's the next best thing?" I'm like, well, you could do a lot of synthetic testing and stuff, and that's good, but it's hard still. Like, nobody's made that any easier and, and unfortunately, LLMs also don't make that easier because you need to run all the infrastructure to do the testing and all this stuff.

And so I don't kn- I don't know. I think one problem in the industry right now is we're over-rotating on LLMs. We're saying, "Oh, I, how do I do this with an LLM?" Versus, "How can I augment the thing I was trying to do with an LLM, not just replace it with an LLM?" And there's... And I think that's especially true in verification. And so I don't, I don't really have any profound ideas in this space. I just think, like, the things that were true before are still true, and we still have to solve those problems, and they were largely unsolved historically. They're not magically gonna solve themselves overnight, and they're even more sort of in the roadblock path now going forward.

Like, if your, your build took an hour before, that's way more of a problem now than it was a year ago because there's a lot more sort of push o- It's like GitHub. The constant outages are mostly, like, load-driven, right? Yeah. And so I, I was telling somebody the other day, I think I've always worked... I, I just like solving my own problems. Yeah. And so I've always effectively worked in sort of infrastructure, but I'd often lean towards, like, internal infrastructure. And this is, like, also why Sentry is a thing for me, 'cause I'm like, "Oh, how do I make my own job easier and more, like, optimize myself," basically.

And now that's actually pretty much the entire industry's job. Yeah. It's how do you optimize your own job? How do you... Everybody works in, like, developer infrastructure now. Even if it's as simple as like, how do I refine the prompts I give or the context I surround it with? Like, literally, it's, like, the most important thing everybody can do, and that's kind of interesting. Like, that's why I'm like, I don't know what happens from that. But I was thinking about this the other day. I'm like, actually, like, our whole company should be contributing to sort of our developer infrastructure goals or our productivity goals or whatever that is, you know?

And you kind of have to, 'cause you can't just outsource, like- Like, like even just knowing how to prompt things in a way and knowing that's unique to you how your brain maps to the thing is like you can't just outsource any of that to anybody else. And so it becomes like a really interesting like thought experiment I guess at this point. But yeah, I don't, I don't know what the solutions are gonna look like. I just think there's like, there's a lot of technology we need that is not just the model gets better, so.

Allie Howe Absolutely, yeah. I feel like that's why a lot of people are excited about internal AI, building that. I know you're on our panel for that tomorrow night, which I'm super excited about. But I think, yeah, a lot of people are trying to figure out how to make one person's breakthrough and one learning the entire company's- Yeah... breakthrough.

David Cramer Yeah. And it's hard when... And most of us do this now with skills and stuff, and we do this, but then you'll find somebody like, "Well, for me, it's annoying because it does this." And but you, you gotta look at it as like it's all a tool chain. How do you share the tool chain and evolve the tool chain? But the, the interesting thing, I was, I was telling somebody the other day, I'm like, "I wonder if there's gonna be like something that we realize sometime in the near future where it's like, oh, this person, because the way they th- speak or, but like speak with their fingers or whatever- Yeah just somehow inherently maps better to the goals within the language model that they're just able to produce significantly better outputs or something like this, you know?

Because fundamentally what the, the technology is versus other people that maybe produce worse. And you might simplify it and be like, "Oh yeah, people that can communicate better will be more successful." I don't actually think that's true. I think it's the way you formulate your thoughts and, and sort of describe something, not necessarily how thoroughly you communicate something necessarily, right? But it's like, is that a teaching thing? Is that luck of the draw? Like what are the actual controls and the things you can put in place that actually do sort of unilaterally just improve things? And I think that's really, really hard, and ours is mostly, it's like workflow stuff.

It's like how do you just do the thing a little bit easier, a little bit more turnkey than you... Like the repetitive task, right? Yeah. And the repetitive task is not generally build the new feature. Like yeah, that's a repeat task, but it's a very, each one's a unique snowflake, so.

Allie Howe Yeah, definitely. Yeah, it's interesting, like maybe like different models respond differently to different ways people describe their thoughts. Like you would think maybe if I was communicating to Groq and I described my thought in a, in a kind of a tweet format-... it would do better than Claude.

David Cramer Yeah. I, I do wonder, you know. 'cause I do wonder, I'm like sometimes am I just like, am I just saying things the way like I think in my head, and I'm, I'm just not contextualizing enough? And, and that's definitely true. But I'm like, is there just somebody that's like naturally gifted at this-... just because of, of how they like, how they would speak or translate their thoughts into anything, you know? And I'm, I'm, I'm genuinely sure there are, but maybe it ends up being like a, a rounding error of a, of a delta, so.

Allie Howe Yeah. It's interesting too, like I feel like for some tasks, like more context is needed than for others. Like whenever I use like an LLM to help me with my writing or, you know, podcast outline drafts, like I have to provide so much context- to make that viable, to make it sound like me, something that I wrote.

David Cramer Yeah. I, I tried writing a blog post with just an LLM one time, and it was, it was so brutal. And I didn't publish it until I was good with it. So it just meant I had to like intervene over and over and over, and, and basically force it to rewrite so many things with so much sort of, precision in my prompts. And I'm like, "This is awful." but I do have like a thing that's just like, look at all my blog posts and come up with some writing guidelines or something, and I basically use this like a s- almost like a spell check. And I'm like, just run this over it because it works better than spell check. 'Cause sometimes I have this like habit, it's almost like dyslexia or something, where I'll just insert the wrong words.

And I, I've never been able to fix it, and I'll just like publish things 'cause I'm also like very impatient. I'll just publish things-... and they have the dumbest typos in them. But it's not like a misspelling. Yeah. It's just the wrong word for the thing. Yeah. And the LLM will definitely fix that, but if you don't give it some curation, it'll just, it'll just change your voice entirely. Where I'm like, "No, I just want you to fix the word and put the right word in" kind of thing, so.

Allie Howe Yeah, I do that too, like fine versus fine with like a D or an E or something. Yeah. Yeah. I always do like simply versus simple or just something- Oh... like silly like that, and it, it's so- Yeah... frustrating.

David Cramer It is, but it's authentic when you read it. You're like, "Oh-... human wrote that."

Allie Howe Yeah. Like it's a good sign, yeah. Yeah. And you can still very clearly tell when something is AI written and, and not, but yeah. Very cool. Well, what's next for Century and Cerer?

David Cramer We're very much more of the same. We are like very consistent in like this is the thing we're trying to build, which is general application monitoring with a bias towards people that are product facing versus systems. And, and Cerer's a small component of that, but, but if we can make the self-healing stuff, I think it'll be good. Like that, that's really the big thing for us, is like can we do something that we couldn't do before with the data while in parallel continuing to invest in sort of, you know, the whole stack of telemetry, which is like logs and metrics and everything else. And I think we'll be able to, and hopefully we'll have something to share soon.

But, but still TBD how, how good it is and, and what that ends up actually looking like. But that is probably like the next- Cer will see because that's like the constantly changing name, but the core is like the sort of far horizon we'll continue to like work on that problem of like just the best telemetry for- with a focus on the software side, so.

Allie Howe Awesome. Yeah, that makes sense, and I'm super excited to continue to follow that, and excited for the panel tomorrow.

David Cramer Yeah. Should be fun.

Allie Howe Awesome. Well, thank you so much for coming on. It was great to have you, and well, yeah, we'll definitely continue to follow Cer and Sentry.

David Cramer Yeah, awesome. Thanks for having me. This was fun.

Allie Howe Awesome. Thank you.