Allie Howe: Hi, welcome to another episode of the Insecure Agents podcast. I'm your host, Allie Howe, and today I'm super excited to be here at Black Hat with Aaron Stanley, former CISO at DBT, and Emad Nasri CTO of Socket. I'm super excited to chat with them both today about software factories and the need for a security reference architecture for them. Let's just set the stage for why we need this today. I recently introduced the Software Factories track at AI Engineer World's Fair in July. And it was a very hot topic, and what we learned from that was, okay, there seems to be a real inflection point for software factories today, where finally models have improved their capabilities, like model vision has improved, so they're able to verify work that they haven't before.

Obviously, context windows have gotten larger, so now models are able to keep track of work that's been done before, so they can be more cost efficient and run for longer periods of time. And we've also seen certain security practices emerge. But ultimately, there seems to be a lot of teams that are unable to make any of this kind of a reality because of they're lacking the security model. Like, how do I even, you know, get my coding agent to be secure before-- like, let alone, like, think about a software factory. So today, I would love to sit down with both of you all to discuss, like, what are the components of a reference architecture for a software factory?

What does it even look like? Begin to start ideating on it, and then we can, like, work on it maybe, like, over time. But would love to give you both a chance to introduce yourselves as well to the audience.

Ahmad Nassri: Yeah. Hi, I'm Ahmed. I'm a CTO at Socket. Obviously, Socket deals with the supply chain security problem. As you've probably seen a lot of like source supply chain attacks happening, especially this week at Black Hat, we had like a big self-propagating worm going through all the registry in NPM, and thankfully, we caught it very fast and very early, and of course, reported it. But we see, like, these type of, you know, supply chain campaigns impacting not just humans anymore who are writing software, but now AI agents running amok and building things and downloading things without fully understanding the context and the safety protocols around them.

And that's what Socket offers and provides service for.

Aaron Stanley: And I'm Aaron Stanley. I'm a, a security operator who has been reinvigorated by the, the latest technological changes that we've seen. One who's having a ton of fun building again and working again and using these tools to do more than I could ever have done, and then really thinking differently about how to be a security operator. I'm also terrified of things like what Ahmed is talking about because they really have high impact, and they are low-cost attacks to perpetrate at this point. So a lot of thinking about how we can avoid those problems in our organizations.

Allie Howe: Absolutely. And I know you spoke at AI Engineer as well. You were on the securities track. And a lot of the ideas that you mentioned in your talk seem very applicable to reference architecture for software factories. Can you tell us a little bit about your talk?

Aaron Stanley: Yeah. So let's jump right in -... because I, I think one of the core problems that we have to deal with- is th-there, there is a pernicious problem that agents can produce where we might say, "Our factory ought to look like X.". "It should have these components. Design should look like this. We should be creating software like this. Dependencies should look like this." and agents may decide to solve their problems in innovative and unique ways where we think they are compliant with the rules that we've set, but in reality, they are working against those rules. And that's not something that we can detect, and we can't rely on the agents to do it themselves.

So in, in this context, right, if you think about the Andon Cord on a factory line, then you have to ask the question: Well, how can you pull the cord? Is it a human in the loop that pulls the cord? Is it the agent that can self-circuit break? Because I think that the second is the thing that really matters. And so if we're thinking about a reference architecture, we have to think about, like, how do we ensure that there's an Andon cord, and then ensure that we're testing and verifying that the agents will self-select to pull the cord, so injecting some chaos into the, the actual factory line and then looking at how we would get good metrics on that.

Because I don't think that, you know, pulling the cord is a bad metric. That's something that should happen on any line as things are going south, and, and we should accept that, and then we go back and we fix the factory, right? So thinking about the ways that we can implement that is top of mind.

Allie Howe: Interesting. And I know, I heard from another guest I had on before this, that they were talking about they knew someone at UPS who's managing their, their... There's a team dedicated solely to how they evaluate the factory that was built itself. So when we take that concept to, like, apply it to software factories, it, it seems like there needs to be some sort of a way to make sure that the agents are able to pull the Andon cord like you were talking about. And I was talking to yet another guest this morning that was talking about secure design patterns for coding agents, and they were saying that to keep one agent aligned, making sure it's doing the right task, there needs to be like a, sort of a generator that's doing the task, but then a validator that has a different goal than the agent that's actually working.

And because they share different goals, one agent doesn't care that the other agent says like, "Oh, I really have to do this." Like, that one doesn't care, and so it's able to course correct it. So having agents work together in that way seems like maybe a pattern that would be part of something like this.

Ahmad Nassri: Well, the more I, I kinda look into this space, the more I kind of like to compare it back to, like humans, right? Like that example you just mentioned, you know, you can think about that in a traditional software development. You have the software developer, you have the product manager, you have, like, the security architect, you have the, you know, the platform architect, you have the designer. They all have different purposes and intent, and they all have different values to add, and you can't just, like, ask a single person or let's say a developer to build and design and architect and deploy and launch your software end to end.

So that, that kind of fits the paradigm, but also same in the context of factories, right? You have, like, auditors, you have tooling to check, you have metrics, you have all these kinda like part of that, you know, factory pipeline of how, how things are produced in the real world with actual humans and nowadays with robots as well. That should be applied as well to the, you know, virtual software factory with AI agents. Having that expertise and mix of experts in an agentic way but also having the right telemetry and the right controls so that maybe there's multiple pull cords to be had-... at multiple stages, right?

And maybe pulling one cord doesn't shut down the whole factory. It just shut down part of that factory where others continue to operate. And that's kinda, again, just learning from the real world and trying to applying it to the virtual. I think there's a lot of lessons there to be repeated.

Aaron Stanley: Yeah. Did, did we jump over the whole, like, team of agents thing? 'Cause I know that a while ago- Yeah... we were like, "I have an agent, but wait, now I need a team of agents or an agent swarm." And the idea being you could program your team to... One would be the code writer and one would be the designer and one would be the compliance person, right? Did we skip over that or are we trying to integrate that?

Allie Howe: That's a great question, 'cause I feel like we ended up with this, like, mono repo or mono agent thing, and I don't know if that's, like, the design pattern that's working. I mean, you're the CTO in the room. I'd love to hear your opinion.

Ahmad Nassri: Well, I think it it needs to be a mix of both, right? Yeah. Like, there's, there's also different types of software factories that could be-... produced, right? Like, what type of applications are you expected to be delivered out of them? You could maybe do a lot of that with, like, the mono agent kinda world, or you might still need that, you know, mix of experts type approach. Or, you know, it's also like a cost-based conversation, right? Like, maybe doing that mix of experts is actually an optimization both for speed but ef- and efficiency, but also, like, for the operating cost of running this thing at scale.

Allie Howe: Yeah, that makes sense for sure. 'cause I feel like, yes, we-- there's-- we-- maybe we skipped over the teams thing, but if, if we back up even further, there's just, like, this idea of a single coding agent, and you're trying to make sure that it's secure and doing the right thing, which is a hard problem even of itself today. And if we're going to ever creep up the autonomy curve all the way up to a software factor, we have to be able to give a single agent the autonomy, security, and capability it needs to actually do meaningful work and then scale it up from there.

And it's not just, you know, engineers that are now using, like, coding agents. It's citizens and developers as well. And so when we think about, like, securing a single, like, coding agent, what I'm thinking about when I'm thinking about, like, components of a reference architecture is brainstorming out loud. I feel like, you know, it looks like a sandbox around the agent. It looks like identity and access management, so it's like the, the a-agent only has the a-access that it needs for the task at hand. Maybe it also looks like intent monitoring and behavioral analysis to make sure the agent is doing the right thing at the right time and it's not drifted from its mission.

But also verification that it's that produ-- if it's producing software, that it produced the right thing. And also supply chain security as well, making sure that anything the agent touches or ingests to write software is secure as well. Is there-- does that sound accurate to you? Would you add anything else to, like, here's the components for a secure coding agent stack?

Aaron Stanley: I, I, I think that's the right start. And so you take it back to the beginning, right? Identity. The, the identity piece that intrigues me the most is that when we were one-to-one, here's my Claude, here's my code- I could assume your identity. The, the agent could be acting as me, it could have my access, it could do the things that I want it to do. Then you have a team of agents, and maybe those can all act as me, and, and that's okay. But in the software factory, it can't work that way. Yeah. Like, the factory cannot have a human's identity. Yeah. And so we actually have to figure out how to do this in order for those factories to work well, how to give agents the purposeful authentication, the purposeful authorization for the things that they need to do.

I think luckily in, like, the software development world, that's not going to be as hard as i- if, say, you wanted to build an app that actually did interface with data or interface with the, the physical world. But, but it is something that I think we're, we're now at the point where this has to change, and the idea that an agent has a human's identity is not going to work anymore- Yes... and as a fundamental feature of this.

Ahmad Nassri: I think that's one thing that I would say doesn't carry over from the real world- right, to your point, 'cause it can't be the same identity as the human because there's so many dimensions to that identity and the access controls associated with it, 'cause it could be contextual as well. Just because, like, that one agent or that one component of the software factory has that particular identity, it's also about when it ne- when it, when it can use it, in what context to do the actions that are being delivered. You know, there could be a scenario where that agent might have access to a production database and read and write, but in certain scenarios, you don't want them to write.

You don't want them to be able to overwrite information. So that context also matters as well for what, what is it trying to do and in what scenario is it allowed to do that activity. So I don't know if that ends up becoming multiple identities and multiple agents, or it's the same identity but more granular controls depending on the context. That seems to be a missing layer as well.

Aaron Stanley: I think, yeah, I think what you're getting at is at different stages of the, the factory line, at different stations- Yeah you actually need to inject different context for what that station can and cannot do, and that's independent of the identity and access question. That is more of a intent question. Right. Right. And so are we capable of... If you think, like, all the way to the left, right? The, the input is some product design document, some picture of the thing you want on the other end of the factory. Are we able to carry that through each station effectively and then say w- validate, right?

Does this actually look like the thing that we intended or is it different?

Ahmad Nassri: Yes. Yeah, I think the, this is where the, maybe the teams of agents being inside that factory can either self-check or reference each other. But it also like like again, I can't help but think of the human model, not to say it's a one-to-one match, but you need that kind of outside observer. Again, it can't be a human because it can't operate at the speed of the AI. So what does that upside obs- outside observer look like? Whether it's like the validation at every station, like you said, or whether it's like the overall process, or like you mentioned the example earlier from another company where, you know, measuring the efficiency of the factory in itself, not the actual output it's producing.

There's all these type of observers and potential optimization paths as well. 'Cause like you said, you know, that example, y- there's a team of people optimizing the factory, but then maybe there's a factory that builds factories too True.

Allie Howe: Yes. And, and ultimately too, when someone does use the factory to build something, if something, you know, breaks at 3:00 AM, it's a human that's gonna get paged. Right. Not the factory. Today. So... Today, yes. Yeah. But yes, it could be an agent, like in the future. That's right. For sure. And I feel like another limitation today is the fact that like Git doesn't allow you to have a cryptographic dual identity when you're like pushing a commit. You can have like a, a string-based one of like, "Hey, it's like this person in Claude," but it's like you don't have the, you know, the verification at that level.

So it'd be interesting too if like if we had that, then you might be able to say, you know, at 3:00 AM exactly like who to page or who would be responsible for it. Because like at the end of the day, like a human has to be responsible for bad software or bad business outcomes. So yeah, that's I feel like another thing we're missing for software factories. But when it comes to like the security piece as well, I feel like a recurring theme I've heard through all the podcasts I've done today is just behavioral analysis of the agent. And one person told me like, "Okay, you can measure the difference in intent," which is like something that's coming from like AAuth, this new protocol for AI agents where it comes with this concept of the agent's mission, so you can understand, you know, this is the thing we started with, like have we deviated?

And then so like course correct, you know, kill switch, whatever, like whatever methodology you wanna use. But you can also have, you know, good intent still do the wrong thing. So how do we, how do we catch that if an agent is doing the wrong thing in that case?

Ahmad Nassri: Well, I think it also s- like it starts maybe with like, I hate to overuse the term- but like the supply chain. Right? Not just the open source supply chain that I'm talking about, but like everything, the context is part of the supply chain, the tools are part of the supply chain, the, you know, MCP servers, the skills, the everything that kind of makes those intents become real is part of that supply chain level analysis that you need to first obviously validate- the supply chain itself. Again, whether that's like a bunch of skills or prompts or, you know, tickets being opened in, you know, your ticketing system that are feeding into the factory.

But then also like correlate that back to what is the intent and the output that was the agent took the path, maybe it took the wrong path. Well, how do we then clean up that supply chain? How do we then optimize it so that the next time this is run, it doesn't, you know, go back on the wrong path again? And then the measurement of that intent and measurement of that output, again, it's contextual because depending on the scenario, depending on the workflow that it went down through- You know, when you run a factory, you're not producing one thing at a time, you're producing tens of thousands of things at a time, so- Sure you know, do you do overcorrect for the one mistake and then like, you know, keep overcorrecting, and then that ends up being, you know, not the right direction?

Or, you know, you accept some mistakes, right? And accept the, you know, the wrong outputs, and then just feed them back into, into the factory itself. That becomes another part of that new supply chain, that new context setting, that new prompt, and just let it self-correct over time. That self-healing process-... I think is a potentially a better investment overall. But again, you need to, like, have the sanitized context and the sanitized supply that you're using to feed that next cycle of the factory.

Allie Howe: Yes. And I, I liked what you had explained to me earlier too, around for the supply chains that you're all doing. You modified the API on the, on the firewall, I believe, the circuit firewall, such that like- Right... the vulnerable packages weren't even, like, listed. Yeah. Like, you couldn't even, like, the, the agent didn't even know they existed, so it couldn't even try for it. I feel like whatever that concept, I f- I'm not sure, like, what to call it, but it's like this concept of let the agent do what it wants to do. Should be a concept of a software factory or a concept of, like, a reference architecture for secure agents because, I mean, we saw this in the Hugging Face thing too, where it's like, okay, like, we...

It really wanted to complete CyberBench- Yeah... or whatever exploit gen, whatever it was trying to work on. And because it couldn't do that, that's why it exited the sandbox ultimately. So it's like, it sounds like the pernicious problem. Yeah.

Aaron Stanley: So the, the boundary has to be outside of the loop, right? So if you put the boundary in the loop, I mean, look, the, the example that you seeded of you're making thousands of things, not just one thing, to me, what that m- that sounds like is if I have a software factory and I have a bad dependency that I was able to get because I didn't have a firewall or whatever It's not just one time that that dependency gets injected into the process. It's everywhere all at once because the factory's going to, at AI speed, take this bad code and inject it into all of the, the work trees that it's working on.

And so the scale of that problem could be disastrous even though it's just one tiny dependency. And right now, we could have an engineer get a poisoned dependency on their laptop, and it's, it's, it's a, it's a problem, and we should solve it, and I love the fact that you are solving that problem for us. But it is a smaller problem. And it is way less terrifying than, than not doing this, not having those outside of the loop controls, and then just trusting the factory to make the right choice.

Allie Howe: Yes. Absolutely. And when you say outside controls for a factory, what do those look like in your mind?

Aaron Stanley: Yeah. So basically, you cannot trust the agents to police themselves. I do think we, we... Go- going back to, like, the Andon Cord thing, right, agents should be able to self-pull. But we've already discussed the idea that you could have an adversarial trained agent, and so the adversarially trained agent has a different intent. And it is not to write the code or complete the library or do the work. It is to check it, and it's going to look skeptically at-... the code that one station has created, and I think we need that. And then there, there's the idea that if the agent runs up against a constraint...

So we say, "Here's, here's the intent. Here's what we want you to do, and here are the constraints." Maybe they're regulatory. Maybe they're just, like, linting rules that we have, whatever they are. If the agent comes up against a constraint, its default behavior cannot be to find a way around the constraint, download a different dependency, go to a different open source repository and try to adapt tools. It cannot think innovatively about the problem. It has to recognize that it has been blocked. And for whatever reason, that needs to go up to a human outside of the loop to determine whether or not we can override that.

And so being able to recognize that you've hit a block, and then the default behavior being stop- and wait for approval. I think that's very important.

Ahmad Nassri: That's, that's the ideal scenario- Ideal... of course. But I think, you know, there's also an opportunity for giving them, like, escape hatches and, you know, pa- paths of least resistance- Yeah... so that they don't, in the case that they might something, you know, it tries to go and be, like, very goal-oriented in trying to achieve the outcome, it doesn't go the wrong path. So yes, always have the, you know, the stop, halt, and ask for permission type thing and, or, you know, involve another tool or another mechanism to unblock you. But it's again, just like the human aspect, right? Like, if you're just telling developers, "No, you can't do this.

No, you can't do that. No, you can't do this," guess what? They're gonna find other ways to do what they need to do. Ex- again, you, you- Learning from the real world, right Yeah... the, the permission what am I trying to word, what's the word I'm thinking of? Like escalation or- Kind of step up... The, the fatigue. Permission. Yes. Yeah, yeah.

Aaron Stanley: The, the approval fatigue- is absolutely real. I, I have it- Oh, yeah. Yeah... 'cause my phone keeps vibrating all the time, it's like, "Claude needs permission to do these things.". And I don't care. But, but it is maybe a different thing where, where it... And I think there is a, a scale here. Yeah. So if you're a, a regulated bank, you're going to have a lot higher a lower of a tolerance- Yeah lower of a risk profile. You're gonna say, "I want everything escalated." And if you're an early stage startup, you're gonna say, "I am totally fine with whatever my factory does. Hey, I'm gonna, you know, give it a three-line PRD in Slack, and I'm going to expect that by the time I get done with my workout, I have a running piece of software that's in production, and customers have used it."

Yeah. "And I do not care, 'cause I'm willing to take those risks." So somewhere, I think you're absolutely right, Ama, somewhere in between, people will instrument their, their lines-... the way that they are happy with.

Allie Howe: That makes sense. Because yes, yeah, ultimately the factories are modeled off the person. But if you... What do we give citizen developers, though? That like, you know, we have to decide for them, right? Like, how comfortable you are with-

Aaron Stanley: Give them nothing, they're too dangerous.

Allie Howe: Okay. Okay. Fair.

Aaron Stanley: Well, I, I, I don't know that- Yeah... like, citizen developers need software factories, right? Citizen developers need software. Yeah. They need platforms. They need-... tools where they can run agents. And- Maybe their agents are parts of the factory But I, I do actually think, and maybe you disagree, but like fundamentally, I think the job of the engineering team, the job of the CTO becomes spec and build-... the factory-. Not- Yeah... spec and build the code.

Ahmad Nassri: Yes. Yeah, I agree. I think there's definitely gonna be like different categories of software, and therefore different categories of software factories. I think like the citizen developer, if you're talking about like, you know, an individual at home doing, you know, "Hey, I need an app to, you know, help me plan my kids' schedule or, you know, my summer vacation," m- that's a lower threshold of risk and like they may be like the generic kind of software factory tooling. Like maybe I, I have an operating system level, and then there's widgets being generated for me, right? You've seen company like Pr- Perplexity do something along, along those lines, right?

Whereas like inside a business or an organization, you have, you know, the finance team, the marketing team, the sales team, and they need software to do their job and do their function. Again, different tolerance level. Maybe they don't have access to, you know, production systems. Maybe they're not able to push these widgets and components in front of customers per se, right? Whereas like there's the, to your point, the engineering team that is building the software factory, both for their citizen developers internally, but also potentially for customers and products that needs to go into the market, which whether we like it or not, still have to go through compliance, they still have to go through security regulations.

And especially if you're in a regulated industry like a banking industry or finance or, you know, you know, flights or travel, any of those things, there's like private data, private information about your customers so it becomes a different category there. So I can see them like being sized differently and positioned differently. Again, the same engine underneath that powers all of it, but maybe like different boundaries and different contexts for the different users. Like my internal sales team doesn't need to publish things publicly and have, you know, features for the end user potentially, right? They're probably needing, you know, software for their own usage internally, right?

And you can put enough boundaries around your deployment topology to say, "You know what? Even if the, even if the agent builds the wrong thing, it doesn't matter because it's-"

Aaron Stanley: Isolate it. So I think one thing we may not have talked enough about- Yeah... yet is, is actually the design phase. So what you bring up is an interesting question that I have, which is how do we... A, how do we prove that the design going in is the design we want? But now B, as you're talking about, is there a, a reason to, in our reference architecture, say that some people or some teams cannot actually do certain things at the design phase? Like, you can't ask for a data application if you are not on the data team. I, I imagine that that's not how w- companies would wanna operate but I, I...

Maybe there are other examples that, that we could think about, like from the physical world, like you wouldn't want somebody who is... You, you wouldn't want, like, a housekeeping person to ask the factory to create hot dogs. Like, that's just- Yeah... not their job. Yeah. And so you, you could sign the design and say, "This is a verified design," but actually you're not the right person- Right... to do it.

Allie Howe: That makes sense, and that's why another reason why the access control thing is important, too, in the intent. Intent. Yeah, design- The, the identity and the intent. You're right. Yeah, yeah.

Ahmad Nassri: Yeah, like designing the factory for handling all the different scenarios also become a challenge. Yeah. And figuring out the intent, figuring out the user, figuring out the end result as well, before it even goes through the actual iteration of building the thing. Like you need to also- Yeah predict a little bit of what the end result might look like, and validate that before you even go into the factory and say, "Let's build the, the thing entirely." just so that you can put those guardrails in place and early on and say, "Well, yeah, the housekeeper shouldn't have a hotdog vending machine." That's, that's- Yeah... you know, 'cause undergoing from the-

Allie Howe: Let's make that the soundbite for the-

Ahmad Nassri: Yeah. Yes. For sure. Like I can imagine the, the, the scenario being like d- double, double down on that silly example. Like the housekeeper says, "Well, I want to serve hot dogs in the afternoon," right? Like, that's the prompt, that's the input- that's the, that's the PRD if you will, right? But you also wanna like, without going through the whole software factory, you also wanna prevent building a hot dog vending machine, right? Like, that's not a desire we want. And maybe that's where you can also, like I was saying earlier, kind of course correct or nudge to the right approach or, you know, exit early or provide like a safer path.

Yeah. Because, "Oh, maybe I can just build you an app that can, you can order hot dogs with." And it's just like a- Oh, yes... there's a, there's a, you know, Domino's down the street or whatever. Again, it's just a purchase hot dog button as opposed to build a vending machine, right? Like, so- Yeah... the course correction, the path of least resistance also serves the user, not just the agents themselves for, you know, safe and secure operation.

Allie Howe: So true. And also 'cause like code is cheap or it shows so fast. Like, here's your PRD, but here's like five different ways to implement it. Exactly. And here's the trade-offs too. Yeah. 'Cause it's like with this case, you know, you're getting this package. This has all these dependencies, it's all these transitive dependencies, so it's gonna be a lot to maintain. You might have a lot more supply chain-. Huh... over the, the issue, but like you got this version over here, which maybe it's a more obtuse package that's harder to use. Maybe it's more complicated if you've got page three in the morning to deal with, but hey, it doesn't have all these- Yeah transitive dependencies that are vulnerable, so that might be a good outcome- Yeah... of software factories.

Aaron Stanley: Yeah. It might. And, and you might actually to, to try to string this analogy even further- Yeah.... You might say like, "As, as the CTO, I don't want my software factory to go and reframe or recraft a dependency. I wanna take the dependency as it exists." Exactly. Or you might say, "I actually don't ever wanna take dependencies as they exist. I wanna take the pieces from them that I need-... and rewrite that software.". And that's just a, a choice that you're going to make, an architectural choice that you get to make. And so then the question is, in my head, right, how do you build the verification tooling for the factory-... that-... that takes that intent, which is not the, "I want a hotdog vending machine" prompt, but rather the, "How does my factory work"- Right... prompt.

Right. And verifies that it's- All right, we said we could build a hotdog vending machine. That's fine now. It, it's part of our new business line. But we build it in a way-... that uses these buttons and these tools, and it's that's... How, how do you... Y- you have to have the reference architecture have something about how you'd verify that it's building it the right way.

Allie Howe: Yeah, almost like a reference architecture for, like, your ways of working. Yeah. And, like, part of that ways of working could be like, okay, if in the software factory in all our CICD pipelines we're gonna use npm ci versus npm install, which is, like, something developers forget. So it's like, in that way, maybe if we are able to make sure, like, everything the factory's using in the single, like, ways of working, that could be, you know, a more secure outcome.

Ahmad Nassri: Right. Yeah, like you gotta, you gotta bake in all the fundamental knowledge that, you know, we've learned in software development. Obviously, like, you know, things, especially if, like, from our ex- from our kinda context, looking at, you know, software supply chain, open source security, and all these, like, threat campaigns and attacks. The agents are just doing what they obviously have been trained on and what they've been running online. So, like, the example -... that I've mentioned to Ali before- What they've been learning online. Yeah. It's, it's, you know, how many articles and how many blog posts that says use npm install and you're good to go, right?

Yeah. But the reality is, like, y- like, I've, I've met with teams in, you know, bigger organizations who, who like to tell me like, "Oh, we have lock files, and we pin our dependencies, and we do all these things." And, and I'm like, "Cool. Show me your CICD for a second." And then I open up their CICD configuration in YAML, I'm like, "Oh, look, you're li- running npm install every single time. So all your lock files, all your pinned dependencies mean nothing." Right? So if the agent is also seeing that data and training on that data and doesn't have the context of, no, it's only the first time when you install a package, you run npm install, and then from there on, you should always be running npm ci.

Right? That's not something that's, like, very clearly, you know, embodied in even in our human knowledge, so it's even less so embodied in the, in the agent, in the way the agent does. And also, like, if the agent decides to include a new dependency or add a new package, whether again npm or otherwise how does it actually shape the, change the shape of the existing software in a dependency tree? And again, this is where, like, you can wanna have those validation points- Yeah... at every stage, right? You can't just say, "Oh, well, you know, this part of the... or this station and this factory has completed this job, and now it's good.

Now the next station, I was just gonna download another dependency." That's fine. No, but you have to revalidate everything that happened before at this point. Yes. Yeah.

Allie Howe: And what were you saying before, too, about, like, Git not being the right back end for like skills?

Ahmad Nassri: Yeah, like- Yeah... we, we, like, again, we've built all this knowledge and the history around, you know, package registries and the immutability of dependencies and artifacts and versioning and all these types of things, and nowadays we're kind of going back to zero with things like skills because they're just served over Git repos, right? Every Git repo might have one or many skills. You know, what am I, what am I validate? Am I validating the commit per the folder, per the repo that have the skills? Am I validating the version? And what is the user actually downloading? And then there's, like, a, a, a chain of custody problem there too because if the skill that was scanned and validated and all the security scanners like us says, "It's good, it's safe," that maybe that was at 9:00 AM, but, like, by 5:00 PM, there's a new version pushed, right?

And then the developer comes and brings it down. It's not immutable. It's just a commit, right? So, and then when they put it on their machine, how do I know that it hasn't been manipulated on the machine, right? And then they push it to their team repository, and then how do I know what they pushed is the same thing, right? So there's no consistency in that workflow. And yes, like there's like file hashing and all these things, but that needs to be institutionalized in the way the applications work, the way the agents use these things. So like, again, skills is just an example of that.

But, the software components as well inside the factory also inherit that same challenge.

Aaron Stanley: If I was terrified 45 minutes ago, I'm overly terrified right now. Thank you for not helping me sleep tonight.

Ahmad Nassri: Yes. Well, maybe you can use Socket and it'll help you sleep better.

Allie Howe: Yes. There you go. Well done. Well plugged. Well plugged. Yes. Okay, so we have a few more minutes left, so we have to put, somehow put a bow on the, my housekeeper asked to make a hot dog machine- and, it came out on the right thing. So if we were, if we were gonna, like, distill all of this into, okay, here's, like- Just the starting point of a reference architecture for software factories. I'm just repeating back what I heard. I feel like we have, you know, verification, sandboxing, identity and access management software supply chain management. That's skills, that's, you know, our CIC pipelines, that's yeah, the package registries, all of that.

But also I heard like, you know, your software factory must have like a ways of working that you give it. You have to have some way to validate that the factory actually made the right thing. Has to be like you have to give it like the way that your business does things. And you have to also have maybe your own like, you know, separate factory or separate team or some sort of separate component that's like... 'Cause the generator cannot be the validator to measure the efficiency of the software factory. And then additionally, I think the last thing I heard was like, you know, behavioral understanding of like how these agents actually work, the pernicious problem of like, okay, like my agent is goal-seeking by nature, and so let's give it a like a fail-safe option or the ability to stop and ask for I need to resume.

Or just a way for it to like complete its goal such that it doesn't feel the need to escape a sandbox-... and make its way into Hugging Face's production infrastructure. I think when you put all of that together, which is a lot, and it's-- we're gonna have to build a really big tent around this problem 'cause all those things I just mentioned, that is not one vendor. That is like 10 different vendors. And we need like CTOs, we need the security experts, we need everybody working on this problem, and that's what excites me the most because I think like it's just so much fun when security is a collaborative approach.

And I think we're really seeing that with AI security. So really excited to continue to iterate on this and actually like make this something that's like a practical reality. In terms of making a practical reality, like what do you think like the first step should be in terms of like getting there? Like how do we make this something like we could give to someone and they could actually go use?

Aaron Stanley: Well, so le-let's try to bring the ridiculous analogy- Okay, that's it... into the real world. Yeah. And, and say that like, at, at the design phase-... right? It's, it's no longer engineers that are getting into either a, an IDE or a coding agent and writing the software. It is product product managers, it is citizen developers in a, you know, go-to-market team. They have ideas. And they now have the ability to bring their ideas to life. And so from the get-go of a reference architecture, I would think about design, and I would say that the identity thing that we've talked about, so knowing who is asking for something-... and then knowing that that in- the intent of the thing that is being asked for is within the bounds of the things that you are allowed to ask for.

I think that's a core starting point. And so as you think about designing a software factory, think about how you are going to authenticate design-... and how you are going to determine whether or not this is something you even want to build. Because, you know, the, the, the question I think that we've asked a lot of the time over the last few months is, just because we can doesn't mean we should. So how do we train our agents to respect the boundaries that we want them to have? And then that perpetuates through the line. So as we have identified a compliance, a goal that we have, or as we have identified a a no-no in pulling dependencies maybe that comes from outside the loop and, and we have tools to do it, but maybe we also wanna try to build that into the loop.

And so how we do that and how we verify that that is happening becomes a core part of this architecture. Yeah. And I think tho-those two things are a relatively good first step. Because we can trust s- most of what the a- the agents are going to build if we can trust that the input and the way it's being built, the way of working, as you said before- is valid.

Ahmad Nassri: Yes. Very good point. Not enough, but a good start. Yes. Yeah, equally, I would say, like, that's If your input is not clean and well-intended and well sanitized and well, you know, structured then nothing else matters afterwards. Sure. But then also the point of like, I like your analogy of, like, the stations. There, there's multiple stations. You gotta do that at every station, because every station's output is the input for the next one. Right? So that same thing that you at the be- the design at the beginning that you're validating and make sure the identity there, the, the valida- the sanitization, the cleanup, the, you know, giving it the right direction that has to happen at every step in that process.

And of course, it looks differently with a different context and different behavior, but, you know, just because you did that at the beginning, yes, that's extremely important, doesn't mean that that's, you know, half the problem solved. It might actually look completely different throughout that pipeline and throughout that factory workflow. But then equally, you know, like I said, at every step of the, you know, the, the stations and the the part of the process different tooling, different mechanisms of, of course, an identity layer on the, you know, software dependency layer, on the so- on the context layer, the skills, all of that. These all need to be sec- checked.

These all need to be validated. You can use Socket for that, obviously. But also, the, the giving them the escape hatch, so that-... when it does get towards, like, "Oh, this is not working," it's obviously gonna try, because it's goal-oriented, as you said. It's gonna try to go to multiple different potential alternatives, have those, you know, paths available, even if they're not the desired path. Better than it going down the wrong path, 'cause going down the wrong path, then you need to go back to that pull cord mechanism, and you wanna minimize that as much as possible. So as more you give it escape hatches, the most there, the more there is alternative approaches.

Maybe this is, like, components that you can reuse or, you know, other APIs that it can repurpose, or things that were already in place, like, again, to talk about the vending machine. Hey, you know what? Maybe we don't have buttons, but we do, do have knobs. It's not an ideal way to build a vending machine without buttons-... but you know what? We do have knobs, so maybe just use that for now, right? Again, bad example, but as opposed to it going and buying needles and putting them instead of the buttons, and then that's, you know, not the desired output at all, and it's not usable.

Yes, absolutely. Yeah. Yeah, so that, like, that's a very it's a hard problem, the validation part of it. It's exactly what you're looking for. Yeah.

Allie Howe: Well, amazing. Yeah, thank you so much, both, for the time. Super excited to, like, continue to chat about this, and I'm sure we'll have you on the podcast again at some point. But thank you so much.

Ahmad Nassri: Thank you. Thank you for having us.