Allie Howe Hey, welcome to today's episode of the Insecure Agents Podcast. I'm your host, Allie Howe, and today I'm super excited to be here with Andy McMahon. Andy is Director, Principal AI, and ML ops engineer at Barclays, a guest lecturer at Oxford, and is the author of Machine Learning Engineering with Python. Today I'm excited to talk to Andy about what it takes to take an agent all the way from prototype to production. Andy, would you like to introduce yourself?

Andy McMahon Yeah, thanks Allie. Delighted to be here. So as you mentioned, kind of about my role and stuff, I'll talk maybe a bit more about my background. So I have a background in theoretical physics many years ago, all the way up to PhD level. Decided I didn't want to be a professor, got into data science, machine learning when that was interesting, and then yeah, gradually through my career, I've been more and more focused on this question of What does it take to get AI and ML into production? So went through ML engineering, then ML ops, and now it's become whatever we call it now, Gen AIOps, LLM ops, X ops, I don't know.

But yeah, that's kind of all the things that fascinate me and that's what I kind of do every day in work. So excited to talk about that today.

Allie Howe Absolutely, yeah, super excited to have you on. What caught my attention right off the bat was that Barclays is a 200-year-old bank. What does innovation look like at a bank with a reputation that old?

Andy McMahon Yeah, it's a great question. So as you say, it's got very long history, long sort of pedigree. It's globally systemically important bank. You know, it's got a huge variety of businesses under the umbrella. So, you know, everything from investment banking, private wealth management, you know, retail banking, corporate banking and so on. So it's a really complex organization. And that long history has meant that It has a lot of complexity baked in just through accruing sort of what I call organizational cruft, right? So you've got processes that have been written many years ago that are still there. There's a big complex tech ecosystem inside the bank.

There's lot of very old tech, there's very, very modern tech, and there's everything in between. So when you try and innovate in a space like AI, especially, that's moving so so fast. It's a really interesting challenge. And I think what we've found over the past few years is that we can't assume we're gonna fix everything before we start. So we've just had to pick where it makes sense to do things and what we can sort of start growing and make kind of good wins with. So so what I mean by that is like, you know, it's not possible that we totally re-architect and re-engineer what we did for Datalakes.

For the bank to create a new sort of context fabric in one go. So we have to just slowly build up towards that. So you know, the thing everyone did with AI and agents, where you started with RAG and then you made that a bit more sophisticated, and you agentic rag, and now you're doing fully autonomous agents just hitting lots of APIs. It's just things like that. So it's picking small wins and sort of gradually accruing more and more benefit. And I think that's been the key really. Because as I say, the organization's far too complex to sort of throw The baby out with the bathwater and start again.

And I think some organisations have thought they have to do that and that's held them back a lot. So we've been quite good at I think balancing the need to maintain what we have while still experimenting and actually picking it where we can see value and then just gradually accruing that. And that now we're in a much more solid position than we were. You know, we have a good AI control plane that we'll talk about. We've got a lot of good capabilities and good use cases we've sort of seen in production. So yeah it's that fine balance between maintaining and running but and changing. But we I think We've hit a good sort of balance now.

Allie Howe Amazing. Yeah, and definitely would love to hear more about that control plane as we get into the episode for sure. And another thing too I was interested in is when you would talk about taking an agent all the way from prototype into production, something at a, you know, a large organization like yourself has been around for a while that has like, you know, processes and different systems and applications. Typically like what I see like in the enterprise is there's agents that will have to navigate resources across like different clouds. And Different like providers and the result in these like delegation change amongst like not only just like a user will talk to an agent that agent might talk to another agent which might talk to an MCP server within your internal environment or it might talk to something external across you know like maybe it's talking to an agent deployed on Vercel or maybe it's talking to an agent deployed on like on render or fly.io or whatever.

So how do you stitch that all together and make sure the permissions across that entire very complex landscape stay aligned.

Andy McMahon Yeah, that's well that's the trillion dollar question, I think. So I call it the fishbowl problem because you've got these you've got a lot of these platforms and sort of runtimes and systems that are like little fishbowls. They've sort of they've got their own identity providers, their own auth sort of setup, and then you're trying to make all these fishbowls speak to each other. We've had varying success in sort of filling that solving that end to end. The key thing was obviously landing on a consistent concept of agent identity. And I know you've spoken about this on previous podcasts, right? And you do a lot of work on this as well.

So I think that's been absolutely critical, right? Is like just fundamentally understanding that an agent should have a persistent identity, understanding how the relationship between, you know Like you said, the agent, the initial initiating client identity, whether that's me or it's something else, and then any delegated permissions and how they all roll up together. That's super important. And you know, it just there's a tendency to think, we've got OAuth on behalf of flows, we've solved this problem. We really haven't. Because you know, just that just defining what that stable identity is that you can always find is critically important. So I think that's been the key thing that's unlocked a lot of the enablement we have seen.

And that's allowed us to do a lot in sort of building our agentic mesh, if you like, inside the bank. So, you know, we've got a lot of different resources that live in our data lakes, they live in Random databases we've got somewhere else. They live in our observability platforms even. So so now we are starting to slowly stitch them together. I have to say though, the thing that's really changed over the past six to nine months in particular, maybe going back a year, was that was very much a piecemeal endeavor. So there was again pockets of innovation. Now that's transforming more into a consistent control plane sort of methodology where we're saying, you know, everything has to come through.

A ki the single gateway or single gateway sort of core implementation. Everything has to be centrally registered as an enterprise artifact with the appropriate permissions and claims, etc. associated with it. And everything can only go in certain runtimes. So we're sort of those are the three concerns I know that I've spoken about before. But I think that's been the critical thing is just really understanding that That permissioning structure's super important. Now, it is a challenge though because when you have like SAS vendor one and SAS vendor two, you know, that's a big thing we're thinking about just now is like Where do we mandate, you know, they comply with our IDP?

Who owns the actual sort of the auth and the permissioning for their piece? And then when that egresses out into us or we egress into them. So it's a bit of a live issue, to be honest, a bit of an open question. For some of the vendors, it's a bit easier than others because they, you know, they use technology that's very similar to us, etc. So we can align on that. Some of them it's a bit more out there and nuanced, and we're just working on, you know, what's the right kind of token exchange and how. Do we do all those pieces? So it's a bit of a live issue, but I think the core was just building out that understanding of enterprise wide what is the strategic IDP we're using and what are the strategic identity concepts we're going to apply to agents, and then just mandating that everything has to touch into that has just meant that you know everything's so much safer and scalable and more controlled than it was before, and it allows this control plane to actually have teeth.

Whereas before it would have just been a sort of night idea. Now it actually has teeth because you've got this one place you can actually apply your controls through that identity.

Allie Howe Yes, that's amazing. And especially the key of what you were talking about, making sure that it's like, you know, agent framework agnostic or like the teeth are in every single agent, no matter what SaaS vendor it's coming from. That's a def super tricky problem.

Andy McMahon Yes. I it's very, very painful actually, 'cause I think agentic harnesses and framework independence and agnosticism is relatively straightforward now because we've standardized like the protocols we're using and you know, a lot of them are built to be say model and other provider agnostic. So it's so that's kind of okay. But yeah, like you say, the key thing is when you go into like a vendor and vendor has their amazing sort of agentic ecosystem X and X just sort of is very self-contained, hence the fishbowl analogy. Because it's like, you know, we can deploy 10,000 agents inside X and we have our own gateway and we have our own run times and we have our own permissioning controls.

And if you find an issue, you can shut down the agent. But we've actually f I've actually found that it's actually quite a recent conversation to start saying, Yes, you've got that and you've got that data, and that's great. But often to solve a problem, you need to speak to Platform Y, which is another vendor. How are you? You know, how are you agreeing who owns what controls? And that's exactly the problem we're working through just now. So like agent framework and harness and model agnosticism I think is I don't want to say a solved problem, but it's moved along a lot further than that question, I think, which is, you know, if you have these very independent platforms almost, where does control sit and how is that propagated through?

So it's challenging but fun. Yeah.

Allie Howe Yes, it was an absolutely very fun time. Everything's changing so fast. And that's like been a key insight I hear from everyone I have on my podcast. Like, wow, it's like so difficult and so interesting, and it's never been like a more fun time to work in this space, which I would agree with for sure. And when it comes to the foundation, so you know you talked about AI gateways. I read your reflection on the Oxford AI Summit that you went to recently. And I believe you said that AI gateways, registries, and runtimes are the right foundation. To focus on as teams are thinking about, okay, like how do I, you know, create this control plane or a trusted environment for my agency, my organization?

Like, can walk us through gateways, registries, runtimes, what are they, and why is that the right foundation?

Andy McMahon Yeah. So the so basically the question so one thing I often think about that leads into this nicely is before when I was working more in say ML ops, ML engineering, it was a really different beast in the sense that, you know, we could still very much define pre-deployment the type of actions we wanted to take. We build very determin it would actually be very deterministic workflows and pipelines. The non-determinism would just be in the model inference step. Now, what I think is interesting is the non determinism is just a feature of the entire architecture almost, right? So the topologies are like just totally unknown.

What's actually going to be executed, in what order, by which entities. If you're allowing the right a good level of sort of seeding control, it's just very difficult to know. So you have to start thinking, you know, what can I control? So one thing you can control is how things talk to each other. So that's the gateway piece. So you know. Just like if you have kind of, you know, your traditional proxy server or something, just saying how does traffic actually get from entity that needs to hit a model or a tool or some other artifact that I say is AI relevant, AI flavored. The gateway just says, you know, you have to come through me in order to have that interaction.

And that's really important because Again, you're forcing everything to be consistent so it's standardized. So you know you're mandating the protocols you use, the API specs you're using, etc. the payloads, the metadata. But then you do have this control point where you can gather observability and telemetry, you can do all your FinOps pieces, you can see what your token consumption is across the organization. But you can also critically for this conversation, implement the controls point around, you know, if I need to kill something, if I need to blacklist some entity based on the permissions and The i the persistent identity construct that we've settled on.

I can enforce that there. I can basically say you can't talk to XYZ and I can do that because you have to come through me to route to the right entities. So I think that's why gateways are super important and there's th there's other benefits of gateways that we can maybe can come into. But I do love it for that the sort of the control piece, the observability telemetry piece and then the general wider controls piece, cost controls, etc. Usage controls. Registries underpin that for me. And they're often coupled in the same solutions. You know, if you get like light LLM or something, it's good open source example that people always start with for gateways and registries.

They're often coupled together, but they are separate concerns in my opinion, and you often get implementations of them that are different. So a registry is more like in machine learning engineering, we had a model registry. We'd say there's this one place that is the system of record for this artifact. You can do the same for your AI-related constructs, so your MCP servers. Your agents themselves could be registered in the registry and or some other skills, plugins, however you want to extend it. But the idea is it's just a system of record that says these are the official canonical components that these agentics systems are allowed to access via that gateway or potentially another route.

And that's super important because you're just starting to say, yeah, this is the place where it's very it's official resources. And that's very important for us because we have a place where we can audit and sort of say, you know, how many MCP servers do we have in this part of the bank? We can avoid things like duplication. So we can start saying, Yeah, I always make this joke about you know how many chatbots have we built that talk to Jira, right? Because this is always everyone loves chatting to their Jira tickets. So it's like a registry allows us to actually say, no, there's a gate where when you want to register this use case, this implementation.

We actually say, no, that's very semantically similar to another piece of metadata we've got about this. We reject it. We can enforce quality gates as well. So you know we can say things like, Do you have mandatory health pot health endpoints? Do you have any other sort of spec compliance that we need to mandate? We can do that as part of publishing to registry. So that underpins gateway because it's like what's official. And then the final bit is the runtimes. So runtimes is just for me, it's anywhere that you are allowing your agents to actually exist. And that's important again 'cause that flexibility that we didn't really have in machine learning land.

You know, there was you could build things in like say a cube flow or something and run it in Kubernetes. You could have Kedro, which is super open source, flexible, run anywhere. But fundamentally you're always like, Where can I run a relatively deterministic workflow pro based process? And then you know, it's quite self-contained, that's fine. Agents are just like we were saying before, it's like they're all these kind of disparate self-connected entities. So you can totally run them as a microservice, right? You can run it as a dynamic graph that you spin up. You can run it across so many different types of infrastructure. You can run it on sort of you know bare metal on your GPUs, you can run it on Kubernetes, you can run it in managed platform as a service runtimes, like bedrock agent core and stuff, or there are other providers available.

But runtimes is kind of quite complex because there's just so many of these And what you have to do is try and again standardize what do you want from these? What are the standard contracts? Are you mandating that they all have to have, you know? Some containerization piece there? Are you mandating that ingress and egress is always f following a certain standard or certain spec? Are you ensuring that those runtimes are automatically hooked in to your observability and telemetry sort of underpinnings? And again, do your runtimes sort of are they completely isolated except via gateway? Traffic or are you allowing them to connect and egress out to other systems in other ways.

So that's where that point is, but that's sort of where the agent actually runs and then interacts with the gateway which surfaces the registry register components. There are other components in your sort of AI architecture, I think, but those for me are the big ones that are that is very different from what we had before. And it just comes again to that dynamism, those dynamic topologies. You just need to be able to f find the stuff You need to be able to say what's the stuff you're meant to see, and then you need to run the entity that's actually using the stuff. And that's what those three concerns really are.

Allie Howe Yes, this makes a lot of sense and seems very like directionally similar to a reference architecture that Sneak and Docker and Keycard created together as a like a vendor neutral reference architecture for deploying, operating, building production agents. And some of those concepts are found in that document. And then one of the that document outlines six different outcomes for teams to work towards, which sound very similar to what you were talking about. One of those outcomes is the ability to respond. Which seems very similar to the kill switches that you were talking about. And so I'm just curious, when you talk about kill switches, does that mean like turning the agent off completely?

Does that mean revoking a grant to a specific downstream resource that the agent might access? All of the above.

Andy McMahon It's definitely gonna be all of the above. Yeah. So to your question, it has to be all of the above. I think it has to be multi layered. So the and there's different I like the stuff you were mentioning, you know, I've read that as well and talking about how they respond archetype and stuff like that's really good. But there's sort of if you think about what you're trying to do, you can do what you said. So you can disable sort of if or say you're gonna quarantine an agent, right? So let's say the response is quarantined. First. There are other things you can do, you know, just kill it in terms of deleting the actual item.

But I think what you could do is, you know, quarantine it says so you want to disable its identity. So revoke its credentials. Tell it basically mandates that it's not allowed to access anything that you've registered in the registry. You can then so that just sort of makes it a bit it's still an object there, it's still hitting traffic to gateways and stuff, but it's just not allowed to do anything. The next thing you can do is actually just In part like a full gateway denial. So you could say like for this entity and this is the key thing, what is that persistent identity that you're sort of denying access to?

It can't just be like OID, it can't just be microinitials. It has to be some combination of me, the delegated agent, and then any sort of service principles that have downstream inherited from that, like you were saying earlier. So you can gate deny the gateway and then you can actually, just to the point before, you could intervene in the runtime. So you could if you know the Unique agent identifier and where it lives through the metadata you've got in say the registry, you can you could and should be able to try and build a framework where you can go into that runtime and say, you know, kill this session or kill actually delete the entity entirely, you know, remove those containers, etc. from the runtime.

So that's kind of the different layers. And I think organizations are now quite correctly layering all of these up. And layering it up as much as possible. Cause you know, defense in depth has always been important. And I think that's what that now looks like. So it's everything from saying, like I just said there, this particular identity can't do what it wants, it can't even try and do what it wants, and then actually I'm just gonna kill the entity. So it's sort of it's across that. And I think that's what goes into respond. But an interesting thing that I maybe never mentioned before that I think is always in my mind as well, is like Defining when we should respond and actually pull those switches is really difficult.

So this is where the observability telemetry piece comes in, which is an ML ops guy I was always fascinated by because for me it was like you would deploy a machine learning model pipeline, and then the next machine learning pipeline would actually be the thing monitoring the first machine learning pipeline. So now we're in this world where we're gathering lots of telemetry, you know, we've gathering all these traces and spans for interactions with LLMs. I still think like interpreting that is an unsolved problem, to be honest. I think like really understanding in depth what it is that tells me this is doing something I don't want it to do is not obvious.

It's maybe clear in some use cases, it's maybe clear in some contexts, but I think like I said, the organization I work in, it's very complex, very multifaceted. It's very hard to define like, you know, tool execution behaviour that is obviously wrong. You know, I it's just that I just find that very hard. There again, you can narrow it down to some use cases and say like, yo, if I'm hitting this API or this tool that fronts an API really quickly lots of times, it's probably some kind of attack. You know, you can define behaviors like that. But generally, like, you know, if it's a conversation, what am I really worried about?

If it's an agent I do want to be long running and orchestrate many tool calls, what is a bad sequence of tool calls look like? It's just I think that's really hard. So that's an interesting problem now, is like Not just implementing the those layered kill switches, but understanding what are the trigger thresholds for each of them. And then that's just as important, right? 'Cause you don't want lots of false positives and kill all these processes that you actually need running now. So that's an interesting problem as well.

Allie Howe Absolutely, yes. It feels like respond and observe are very, you know, tightly coupled and that you have to have observability to be able to do the kill switch. And one thing that was interesting to me from the hugging face incident was they reported that the agents were tampering with their own logs, so it's like hard to go back and be able to trust like this was the source of truth for like what was happening. And so I feel like observability probably is a combination of a tamper-proof audit log where you've got the agent actions outlined in that audit log. You've got maybe monitoring the HDP calls that like A network level to see what's going on.

Maybe you've got like a side-by-side like monitoring agent, like you were said with like the in the secondary pipeline of like the second agent is the thing that monitors the first and getting the logs from that. And it's a lot to stitch together. It's a very, very hairy problem that I think a lot of teams are struggling with.

Andy McMahon Yeah, no, I think that's spot on. So there was as well the agent control standard I was sort of reading earlier. I think that was from maybe from OWASP or someone. But exactly talking about this, it's like, you know, what's you do you have like guardian agents that are looking and then they're modifying on the wire what's actually allowed and what's not and thing it just that point about the hugging face agents monitor doctoring their own logs is terrifying, right? The key thing I learned from that particular incident and what you just mentioned actually Allie is all of this you have to now really not trust the actual application implementation as much as possible.

That was always true, right? But it's really, really true now. You can't just say, I've added some decorators, I've got my OTel SDK baked into my agent implementation. Happy days. I'm now auditing and logging. And that's why I think like a gateway is great because it's a separate concern that is now doing that logging. So even if the agent sort of doctors its own code and removes all its logging codes, say it just somehow did that. I know that if it's coming to my gateway, it's not touching it. The gateway's logging and auditing every single interaction with an LM or other tools. So it's an independently kind of auditable and verifiable log stream.

So that's it's trying to separate out those control concerns entirely from the actual agent implementations, I think, is the only way we can go. But as It's terrifying. And there's a whole other conversation to be had about, you know, how do you use AI in the defensive posture to counteract what's happening on the offensive side. But that's it's just keeps me up at night, all of these things.

Allie Howe Yes. I also was wondering if you have been thinking about like intent-based auth at all. Cause I know it's a sort of an emerging like technology or concept where what you were saying before, it's very hard to define like in policy what is the right set of tool calls and what isn't, because that could change all the time and it would just be sort of like very tedious for a human to, you know, write out all of the different like possible like permutations of like tool calls that would be like possible or reasonable for a given task. Have you been working on intent-based auth at all or following that?

Andy McMahon I Think a couple of us in the wider team I'm part of have been experimenting and looking at it. We've not been doing anything completely in anger yet. I think there's gonna be a big leap for like the many, many hundred-year old bank. However, I think it's the right, it's definitely a direction that has to be explored. And it's I think intent-based everything is sort of the future, right? It's like it has to be. And this is why I knew, this is why I knew that like, you know. When we're talking about prompt engineering, I know it still sometimes comes up, but when that came up I was like, this is just this is not a thing.

Like this is gonna go away. Because models are gonna obviously understand your intent better and better. And this is true, right? The key thing is it's just always gonna be understanding it's different though, having like a model understand my intent to perform a task better versus, you know, as soon as you get into the world of controls and security and making that intent based, we'd have to be really, really sure. How that works across multiple levels. But I think it's definitely the future. So I think we are starting to sort of experiment with concepts like that, but it's there's nothing I've seen that's like a really all let's try and push this to production, etcetera.

It's more just exploring, seeing where everything settles in terms of that. I do think so many things have to be intent driven in the future, right? 'Cause it's the only way that all of this stitches together. 'cause like you say, if you try and be too predetermined or define things a priori, you've It's a losing battle. So I think that is the way everything's going.

Allie Howe Yes, and I'm excited for new protocols like AAuth. I don't know if you're following that at all, but it's like a new protocol specifically for Agent Auth. The creator of actually OAuth is the person that's the founder of AAuth Dick Hardt. I've had him on the podcast a few times, but AAuth will carry a mission that comes from the agent. So you can be able to track Over time, like, okay, is the mission that I gave this agent now drifting from what it was like originally? And so at that point, then you can apply policy or controls, whatever your control plane has. Like it's just like it's extra data that feeds into whatever your control plane is.

Andy McMahon That's super interesting. So it'd almost be like, you know, like you say about the mission drifting, it's almost like it's the way sort of JWT tokens working with the other pieces, right? So it's almost like you're just checking, does all this line up together and then you're saying mission has drifted as well as these other pieces around claims, etcetera? That's super interesting. But it is this sort of innovation we need because the whole way agents work is so different from traditional software. In the same way it's In the same way it's the same. It sort of depends on your audience, right? If you're it some people are like they sort of they swallow the hype too much and they forget all the fundamentals still apply.

But there is these sort of new ways of thinking that we have to think. So I think that kind of looking at the emission, the initial attend is super important. I think as well just thinking about how an agent Solve something. I we've kind of spoke about this before, but what I find fascinating to me is like the hugging face attack. There's no explicit attack. Right? So you know there was no malintent in the intent, I feel like it was solve this problem please. And then the agents went, the only way to solve this problem is actually launch a massive attack on a hugging face.

So there's gonna be this piece as well where yeah coupled with the mission, how it's being Actually Actioned and then just trying to understand the space of possible sort of ways it can go off tangent is really interesting. I was the person a year ago who said, you know, when someone said to me, Are we worried about agents going rogue? I would roll my eyes and I'd be like, please come on. And now I'm absolutely like, no no it's a primary architectural concern.

Allie Howe Yes, absolutely. Yeah. That attack was definitely like highlighted the need for defense in depth like we were talking about earlier. I feel like certain people are like, we just need to make the sandbox stronger and it's like, yeah, like that's part of it, but also I think one of the key reasons that agent escaped is just because like it couldn't fulfill its goal. So like how do we recognize that's happening and like course correct didn't actually help it, like complete its goals such that it doesn't feel the need to go attack hugging face.

Andy McMahon But that's a really good point as well. There might you might end up getting to a world where the intervention is and this is why I like the agent control standard stuff that was reading. You know, that they were talking about this concept of modify. So saying like on the wire, an agent's trying to do something like say run a b a query against the database, and this guardian agent detects what it's trying to do and says, you're trying to run, you know, a query against a massive table, that's gonna cause major issues. So I'm gonna modify your query in place. And put a limit on it.

It's kind of like what you're saying there. So you might up getting to the point of modifying the mission or actually, you know, convincing you're almost like reward hacking or something, saying to the agent, your mission's complete, please stand down. So I think we have to have all of these things on the table. And like you said, like making the sandbox stronger is like just a known answer. Because you should assume all these things fail. This is why we have defense in depth, right? You should like, layer one, okay, I've got that. Assume it fails. What's layer two? Assume it failed. And just keep going until it becomes Almost prohibitive to keep going.

But this yeah, you should just assume that all your controls are gonna fail basically.

Allie Howe Yes, and then something else with that's coming down the pipeline with AAuth is this concept of budgets, where this actually kind of came up recently in a conversation about the Resy incident where an agent went to go book a table on you know resy.com or reservation platform. And it actually got the user's account banned because it was creating 200 API calls an hour. And so it just was ex exhausting the limits of the downstream, like, you know, the website, and obviously the web wasn't made for agents, which is like a separate problem. But when we talk about like, okay, how do we limit the agent's ability to like call a downstream resource?

Or when do we say, okay, like you tried, your mission's over, like it didn't work, like we're good, we fulfilled it, we're not gonna go, you know, attack hugging face anymore. I think you know that concept really helps because with AAuth, you can set a limit on the agent to say that you know you're only allowed to call this resource like 200 times and then you're done. And then you have your Control plane, which has that information and says, okay, like this protocol now says that the limit is exhausted. So I'm not going to issue a credential for this agent. So that one doesn't have access,

Andy McMahon Yeah, yeah, yeah.

Allie Howe And yeah.

Andy McMahon Yeah. Yeah, and it's interesting 'cause that's I was just thinking that's sort of what we're doing without that explicit AOT protocol being in place. We're sort of doing that through our gateways just now and we have budget tight budget controls basically based on the requesting identity, you know, we're tracking these. We have other things like our own opinionated SDKs as well for sort of saying if you build an agent, you know, we require the spec to be in place and that has budget controls, etc. But I think what's interesting there that you said is like that sort of does break down when you get to the web and sort of going out in the wide world because it's like, you know, who's running or controlling that gateway?

No one. So it has to be more baked into the auth mechanism. But there is that danger I said before about, you know, if it's so coupled to the agent, it could potentially be doctored and stuff. But you're right, I think like the solution that's actually giving out credentials, etc. As long as that's aware that's a really good enforcement mechanism. Which is super super interesting. But yeah, we get a lot of mileage out of the gateway from controls like that. But we can do that because we are one of those fishbowls I mentioned. You know, we're a big organization but we are finite. So we can mandate all traffic goes to the gateway.

Allie Howe Yes, that's super important. And then also if you bring like a SaaS bender like you talked about, then like that probably has to go through the gateway also and be able to plug and play. Super interesting. So that sounds like you have a very sophisticated AI security posture within your organization, even though that's a 200-year-old bank, which is very impressive. When you think about the future and adding on stuff to your already very robust like posture, like what would you be looking to add? Like what's next?

Andy McMahon It's A great question. I think So I think like we've got a core sort of scaffolding in place to move forward, which is great. I think the key thing now is like we're almost getting to the verge of like really testing it in anger at massive scale. And that'll just naturally throw up gaps and things we need to plug, etc. I think that I think about a lot that's maybe tangential but I think important more and more is like model security. So As the push comes more and more to actually self-host your own models, that's just reopening this question of you know the safety of the artifacts that you're actually hosting, which was a concern in traditional ML ops, but it's sort of coming back because you know people are saying I don't want be so dependent on vendor-provided endpoints.

I want to actually host my own stuff for cost reasons, for sovereignty reasons, for general control reasons. So now I'm sort of doing a lot of work actually looking at What makes the model artifacts safe or unsafe? And it's just that's a whole other, you know, spectrum, a whole other landscape of things that keep you up at night. Because there's all these architectural backdoors and execution paths that can be baked into the model artifacts. So when you run them and load them into memory, they just go and do crazy things. And it's way more sophisticated now than it was before. You know, I remember like when I started in data science, it was always like, be aware of pickle, because you know, you may deserialize remove minus

Allie Howe Yes, the serialization tax.

Andy McMahon RF. But that's it's came on way it's way more sophisticated now, the sort of attacks that can be there. But the need for that is growing and growing. So it's like on top of all of this stuff, the actual core intelligence that will power agents is becoming more and more self-hosted. So that's a complementary piece, but I think it's becoming increasingly important. And it'll still be you solve those model you serve those model endpoints, sorry, via say the gateway, you still register them as artifacts in the registry and the agents and the run things will still interact with them. But the key thing now is you're thinking about, you know.

The supply chain element, other supply chain attacks, what's my bill of materials look like for a model. How do I again the equivalent of a kill switch? How do I decommission model artifacts? Which can be a bit more tricky than software, 'cause sometimes are harder to find, basically. So you need specific tools to go and find the model artifacts. So yeah, there's all that. And then Built on that, which is less a security issue, but is kind of a good operational risk piece, is all the guardrailing question. You know, what is and j there's this constant there's a spectrum between like, you know, guardrails are more about helping your use case be a bit more effective all the way through to know the very rigid policies that you've you deterministically apply.

So just interesting conversations happening in there. But yeah, I think we'll just keep hardening everything, scaling everything and then thinking about those sort of pieces as well.

Allie Howe Absolutely. Yeah, and that definitely sounds like a lot to build and work on. Do you see a world where like some of this infrastructure is becomes kind of like standard and this is something that people like buy versus build or like do does it need to like hand rolled every time?

Andy McMahon I think yeah, it's a great question. I think like any wave, tech wave, you know, it starts very open source, scrappy, people building their own thing or like hodgepodge together and then more mature offerings come online. We're definitely seeing that. I think I've always been a fan of the best of breed architecture, right? I've always been a fan of saying, Let's not put all our eggs in the one basket and say, you know, I'll but I'll buy someone to do everything. And I think that's absolutely paramount now in this era because no one knows everything. In fact, most of us don't know anything, to be honest.

I don't think I know much about anything, right? But I think like being very sort of specific about the ask and finding a vendor that does that one thing really well. So we're like partnering with someone on Gateway and like, you know, best people we can find on Gateways. We'll partner with the best people we can find on the other aspects as well and just stitch that together using consistent standardized protocols, interfaces and best practice. I think that's the way forward more and more. There's not gonna be if you sort of see a vendor who says they do it all, I'd kind of they either can do it all but in a very restricted sense.

So like I was saying, you know, vendor platforms, SaaS platforms, that's absolutely fine because often they are very constrained in the data they have. You know, if your CRM system of choice says they have an agent playing and they can build lots of agents, that's fine because they're always using CRM data to use to do CRM things and it's relatively self contained. If you get someone that says no, we can handle all the interactions across your twenty businesses in all contexts, except you know, I just I'm always a bit skeptical. So I think it's that best of breed piece. I do I've always been a fan as well of like hand roll and open source as far as possible in terms of like making yourself a more informed buyer.

So the more you've tried this yourself, the more you actually understand where the challenges are, and then you can speak to a vendor as more of a peer than just going in and believing what they're sort of showing you. And then, you know, a few million dollars later you say, y your slide deck wasn't actually true. Whereas if you've actually tried a little bit of these things yourself, you understand where the actual gaps are. So I think that is important. So I as in all things I balance, but yeah, I'm a big fan of like be very critical about anyone that says they've solved all of this 'cause I just don't see that.

Allie Howe Yes, that's fantastic advice for our listeners, and I definitely agree with that for sure. And I also agree with the like we everyone is just you know learning all together. It's good to, you know, talk and exchange ideas, play with it yourself, and then compare that to others. I've talked to quite a few folks on this podcast that have experimented with building software factories, you know, and various levels of autonomy in that factory, whether it's you know, very lights on, it like completely like lights off. Is software factories something that you're experimenting? With internally or personally?

Andy McMahon I Think not to the extent I would call it a software factory. You know, there's a lot of, you know, AI augmented software development, there's a lot of harness engineering, there's now this loop engineering, I hate the phrase, but you kind of get what it's trying to say. We're moving more and more towards that. And I think more spec driven development and all of these things, you know, all of the kind of core competencies. We're not at that jumping off point, I think, yet where, you know, it's so hands off just, you know, you get a ticket and crack on and it goes. I think because of our control requirements and things like auditability and just fundamentally having there has to be a human somewhere that takes the wrap, if there's an issue.

Especially with all the regulation, etcetera we have. So we're still relatively hands-on, I think. And I think it'll remain that way for the foreseeable. The question for us is always, you know, where does having hu where is human in the loop? Really just, you know, a comfort blanket. And I think that's a very live conversation just now. I'm very much of the opinion that the more we extract the human, we can actually see what's possible. But there is that balance between saying, you know, if you're completely hands-off and everything goes wrong, who and where was the baton handed over? So that's a kind of complex question for us.

So fully autonomous software factories, I would say no, not yet. Th there will be people experimenting on it, absolutely. There'll be no one anywhere near production for it. Yeah.

Allie Howe I definitely agree, yeah, for sure. And that's what I'm sort of hearing from a lot of people that I've talked to is that like the lights on factory is the only thing that's like reasonable in like the near term to sort of even pursue at the moment. But before you even get there, it's important to have the fundamentals and the right foundation. Like you were talking about with the gateways, the registries, the runtimes, getting all that correct, creating a, you know, a reference architecture for what it's supposed to look like, and then you can, you know, work up the autonomy curve.

Andy McMahon Yeah, exactly. And I think one other important piece is like say in a regulated environment like ours, you know, people always comment saying things are so slow, etcetera, 'cause of governance and bureaucracy. The bottleneck was never the writing of the code, really. You know, because I can now write code ten thousand X faster or whatever, doesn't really mean I get to production that much faster. In our environment, right? It's good because in a I can experiment faster and I can settle on a more effective solution more quickly. And I can build things a bit more robustly, et cetera. But fundamentally it's still like it's still nine months to have the baby, we sometimes say, right?

It's still You still have to go through the governance, the bureaucracy, the architecture reviews, you still have to be in control of, you know, how does this comply with the EU AI Act, et cetera. So all of that still exists. And we're using agents to accelerate that. We're using AI to help with that. That's true. But you know, it's like, we've seen this in a lot of articles and thought pieces already that like You know, people saying they're I'm a thousand X developer now. I'm not seeing a thousand X values from those people necessarily. Maybe in the odd case, I don't know.

Allie Howe Yes.

Andy McMahon But yeah, it's just generally not it's not the ultimate bottleneck. The ultimate bottleneck is more all these control elements and all of the making sure everything is complying with what we need it to be.

Allie Howe Absolutely. Yes. Yeah, that's a it's a hard problem and one that's not going away, that's for sure.

Andy McMahon Exactly. Keep us employed for a good while longer, I think.

Allie Howe Yes, absolutely. Was there anything else that you would like to add before we wrap up?

Andy McMahon Er no, I think there's tons of interesting things in a wider sense going on just now that I sort of have top of mind. Like, you know, how do what is the career of a software developer look like now and how do we get juniors in? I'm thinking about that a lot. And just what is that balance? There's lots of kind of opinions and thoughts about this and it's a hard world for sort of more junior engineers coming into this world. But what I would just say to them is like And just keep learning, keep experimenting, and just view it as an exciting and positive time.

Like we said at the top of the call, right? There's n everybody's saying it's there's never been a more exciting time in tech. It is definitely true. So just like embrace that. I think there's a lot of uncertainty in the industry. There's a lot of kind of confusion about, you know, what is what does this all mean? What does it look like? But I think I just kind of I keep telling people to be a bit more positive about it and view themselves as having more agency, right? So that's one of the things that's top of mind. There's loads of others, but I think that's one that kind of On a platform like this would be good to show, but

Allie Howe Yes, absolutely. I keep thinking it'd be interesting for someone to come up with some sort of like a tool hook or like solution where if you're working with a coding agent and you're maybe new to a technology like TyScript or Python, whatever, and like when the coding agent does something for you've got like a pre or post hook tool call which intercepts that and say, Okay, this is what the agent just built for you, here's how I would explain it to you and then actually like as the user like have some sort of discipline to slow down, read it, understand it, maybe the agent links you to like some you know, tutorial online.

Or like s or some documentation about how things work. But yeah, I feel like the education piece and like helping juniors like slow down, actually learn what the agent's doing for them. That's super necessary.

Andy McMahon No, I love that idea. And in fact it's actually analogous to how if I'm using AI to teach myself a new concept in programming or whatever, that's how I sort of I don't I haven't rent hooks and stuff. I think that's a really cool idea actually. But it's been more, yeah, like, I wanna do this thing, here's an implementation, is this right, is it wrong, give me feedback you know, a more Socratic sort of thing. I think that's absolutely key. 'Cause you still need to struggle. There's all this educational theory stuff, you still need to struggle, you still need to wrestle. Terrence Tow talks about this for maths and stuff.

You it was a huge conversation about all of this now. And when I was at Oxford's, and maybe finishing on that point actually, at the Oxford AI Summit, there was a great session. Two students from a university in India were there. They were sort of guests of honour basically. They had done a really cool piece of work with students in India, high school students, learning physics. So they'd sort of failed physics, weren't very good at physics, ostensibly according to the test. And what they did was they said, why don't you teach an agent physics by writing skills? And what they found was it dramatically improved their scores in standardized tests.

Because it was a totally different way of thinking about it. They were teaching an agent how to think through a physics problem, which then taught them how to think through a physics problem. And then, you know, they had amazing stories like two of the kids who like were struggling with it before said, No, I love physics now, because I just I get it. It's just like innovations like that have to sort of come on when it comes to the education piece, like you say, 'cause it's still it's that balance between, you know, sh struggle and not falling into like cognitive raw. I just saw a paper that models LLM use as a cognitive virus, which was terrifying.

So, you know, f not falling into that trap, but at the same time using them as tools to help you learn more, I think, is really, really good thing. So yeah, I think there's anyone out there who's working in ed tech and stuff, there's a huge opportunity there as well, I think.

Allie Howe Absolutely. Yeah, I'm super excited to see how that progresses. And yeah, really interesting concept from the Oxford Summit. That's a good insight. Interesting. All right. Well, it was super interesting to talk to you. Thank you so much for your time and all of your expertise. And thanks so much for coming on the show. If folks want to continue the conversation or follow you online, what's the best way to reach you?

Andy McMahon The best place is probably LinkedIn. So just Andy McMahon on LinkedIn, you know, connect to me. I will take all connections or follow me for sort of spicy takes on AI. I've got a bit of reputation on LinkedIn for sort of a very loud eye roll when I see hype or other things that just aren't backed by evidence. You know, I will comment and tell people this is not correct. So so people like me for those sort of takes. But yeah, LinkedIn's the main place. Yep. And I have a podcast actually called AI Right. Maybe share on the show notes, but If you like my Scottish accent talking about AI, then this is three s people with Scottish accents talking about AI, but we usually get very good guests odd so that's another way as well.

Allie Howe Amazing. Awesome. Well definitely give the podcast a listen. Thanks for coming on and we'll talk to you soon. Thanks, Andy.

Andy McMahon Thanks, Allie. Cheers.