Yeah, so for me, Twitter was a pretty transformative period in my career because I actually started at Twitter in what they called, at that point, the Relevance team. And we were doing stuff called natural language processing and machine learning and things that like, you know, nowadays are a hot topic but at that point were kind of a struggle to productize.
I mean, lots of fun challenges, but actually getting those things to the point where you could do anything that was really product oriented was really hard. And I was like, "Well, this stuff is never going to work. You know, like this whole area sucks. I'm actually going to go move to the..." You can see how, how good I am at predicting the future.
But at that point I was like, you know what Twitter is really having trouble with and that is making a huge impact is infrastructure. So I kind of moved over to the infrastructure side of the house. And that's where, as you kind of alluded to, at that point, we were on this Ruby on Rails monolith.
We knew it wasn't going to scale. It wasn't going to scale for like a bunch of reasons. Some of them good, some of them bad, frankly, some of them were just the way Twitter did it and the specifics of where we were hosting it were just forcing this choice to be made. Which I initially was against because I was really pro Ruby at that point.
But eventually we moved on to what we called then a service-oriented architecture - we would now call microservices. And that transition was quite a dramatic one. And there are many analogies.
This was before Kubernetes, before... maybe Docker had just started to rear its head. We didn't have any of that. We weren't using any of that, but the system that we built at the end was what we would now call, you know, a cloud native system. It was containerized in the sense that it was on the JVM and we use cgroups for isolation. It was distributed and it was orchestrated.
It was running on this thing called Mesos, which was where I cut my teeth in terms of orchestration, platforms. And so basically the idea for Buoyant was, "Hey, we've just seen this, like, tremendous transformation. We had to build all this technology to make it happen. The rest of the world is probably going to need to do something similar to that. So let's take what was one of the most critical components of that, which is... it actually was a library for kind of managing the communication between microservices and let's turn that into a product that anyone can use."
And we kind of did that right at the time when Docker and Kubernetes were really taking off. And our genius insight was like, no one wants a library. Well, it was a Scala library. Like, no one wants a Scala. It was even worse. It was a functional programming Scala library, you know, for doing RPC calls. So, like, you really have to be at this really niche intersection.
Yeah, yeah. And you know, in the modern world like, you know, a lot of the basics have been solved by Kubernetes.
But what Kubernetes doesn't do, and I think this is a positive thing, is like it really doesn't do anything at the networking layer besides, "I will do Layer four load balancing." And so a lot of what Linkerd gets used for today is basically it's kind of like the missing piece to running Kubernetes at scale.
So for example, if you're running gRPC, Kubernetes default load balancing actually doesn't really work for you because gRPC runs on HTTP/2. HTTP/2, you make a single connection and you start multiplexing connections. And like, you know, layer four load balancing doesn't really do it. You know, you get pinned to one pod and it's like, okay, all is well.
So Linkerd will like sit in there, it'll intercept those calls, it'll establish a connection to each pod and it'll start doing request level load balancing.
So you know, if you even... something as simple as like, "Well, I'm going to run gRPC and I'm going to run on Kubernetes," suddenly you find yourself needing something like Linkerd. Or any kind of cross cluster communication. Kubernetes doesn't really provide any affordances for that.
So like if I'm service A and I'm talking to service B, but I actually want B to like live on this other cluster where I want to migrate between clusters, where I want to balance traffic dynamically. You know, Kubernetes kind of leaves you on your own. Well, that's exactly where Linkerd comes in.
Yeah, yeah, that's right. And you know, sometimes the problems are surprising. Like we have a tremendous number of customers who the number one reason why they use Linkerd is because they're running on AWS.
And even in the single cluster case, the best practice on AWS is to run your cluster so it spans Multiple Availability Zones (AZs) within the same region. Because you're like, okay, if one AZ goes down, at least a cluster, you know, still survives. I've got nodes running on another.
So that sounds great, except AWS will charge you lots of money for traffic between availability zones. And so if you have a low cluster traffic, like you're getting started, it's pennies and you don't care. And then you know, as soon as you start sending lots of traffic, suddenly you're paying like millions of dollars. And you know, Linkerd, because of where it sits, it can make these really advanced load balancing decisions.
It can take into account the fact that the destination is on a different zone and it can actually like, not just cut off that traffic so that by default when the system's in a happy state not sending any cross zone traffic, but if the system is under stress, it can actually start opening the gate and allowing you to send cross zone traffic. So then you kind of get the best of both worlds.
In the happy case you don't have any of that charge. But then when you actually need the cross zone thing for reliability reasons, Linkerd will help you. So that's a feature we call HAZL - High Availability Zonal Load Balancing.
And this was not in the plan, in the beginning, it was something that we stumbled into because we had customers who were like, "Holy crap, we're spending so much money on this one thing. And like, what do we do? Like, we don't want to give up the multi AZ deployment, but we also don't want to spend all this money on it."
So you know, sometimes you like learn these things from your customers, which it's been fun for me.
I think there's a couple things. You know, obviously Linkerd was a first, like we're the ones who created the term. So everyone thought like what a great idea, let's go copy them.
So you know, via imitation. Well, no, I don't. You know, I'd like to think that's true. I'm not sure how true it is. I think part of it is like, they seem fun to build, right?
Like you get to build a control plane and a data plane and like, you know, you can add a proxy and everyone likes, you know, every infrastructure nerd on the planet is like, oh cool, like, let's add a proxy here. Like, let's have a talk to the control plane. And so there was this explosion early on when Linkerd first came out of people doing their own service.
Like every company would build their own because the infra team was like, oh yeah, this looks really cool. So I think part of it is that. And then part of it, I don't know. I think it just, that area got.
That field got really hyped up for reasons that I don't think have really held water in the long run. But for a while everyone was like, oh my gosh, this is the.
There were at least three or four Kubecons in a row where it was like, this is a year of the service mesh. That's all everyone was talking about.
And you know, the cloud native community, they love new things and they love getting excited about technology and like, you know, sometimes making decisions because of the love of the technology more than like, because they're trying to solve an actual problem. So I think that all got kind of bundled up into, into this. Yeah, there might be some other factors in there too.
But you know, from my perspective, I was just like, "Wow, you know, yeah, sure, like take a swing at it. It's really hard."
Yeah, I have a pretty strong opinion on this one, which is I don't think the developers should see the service mesh at all. I think that's. That is like a part of the platform that doesn't really need to be exposed to them.
And it doesn't even speak a language that developers, I think understand. Right. Like the way you, you configure linkerd. We try and be as Kubernetes.
One of our design principles is like to be as Kubernetes-centric as so look like Kubernetes, smell like Kubernetes, feel like Kubernetes. So you know, you like, how do you configure it?
Like you write a bunch of CRDs and the CRDs these days are often like Gateway API CRDs, which means it's a mapping of like six CRDs that all refer to each other in like some complicated scheme that's a platform owner concern. And like, you don't want the developers to spend their precious little brain cells on that. Like, you want them off building business logic.
So I think they should see the output of the service mesh.
So for example, one of the things that linkerd does, because we're sitting in between, you know, every call that's happening between services in your system, we emit metrics.
And so, you know, we can give you this totally uniform view of what are the, you know, HTTP success rates and what is the latency distribution and how much traffic is going and actually who is talking to whom in here. Like, is A calling B yes or no? That was a big problem for us at Twitter. Like, we literally had no idea.
Well, we can provide all that information to you and so the developer should see that, right?
I think a big part of the self service model is, yes, like I, you know, push a button and deploy my app, but also, you know, you as a dev are kind of on the hook for if that, you know, what is the app doing? Like, you know, like, is it, is it behaving, you know, is it running out of memory? Is it talking to things?
I think it should be talking to what is that spike in traffic coming from? Like, those are all things the service mesh can help you answer.
And also when you get into policy and things like that, like, actually I don't want A to be able to talk to B because A is in the non pci. You know, it's a non PCI service and this one is pci.
Like you can enforce all that with linkerd, but again, that configuration I wouldn't expose to the devs. I think the devs need to operate at a level that's kind of independent of like the mechanics of how you're, you know, configuring stuff.
Not everyone agrees with that, but that's my opinion. Save their brain cells for, for something more important.
Yeah, it seems like, I mean, it's obviously tied so much into what I'm doing as a developer. Right. I want those retries. I want to not think about mutual tls, right.
I want to not think about having a God token for talking from service A to service B all the time. Right. It's just like it feels so tied to what we're doing, but at the same time it does feel very commodity. Right?
And it's like for, for me to have to learn that to use it, and that's that I've seen it surface that way in a handful of platforms that I've been on where it's just like, oh, I have to, I have to reckon with this thing instead of it just kind of like soaking into, into the background for me. I definitely agree. I think it's something I like to see hidden a little bit more. And I think also it's.
It's one of those things that, you know, as you're saying with like the.
Being able to just like profile the traffic and the metrics, like that is a part that I do love the ability to see because that's one of those things that I feel like is so hard, especially when you're breaking down an app, is to figure out what is even talking to what you'll see something like, maybe you have a client in your code and it's like, oh, this is a client for some service. And it's like, are we calling that anymore?
I know the code is there, I can see the code is there, but I don't think anyone's used that in a year or whatever. And then all of a sudden you'll look at traffic in between. You're like, oh shit, that thing gets called all the time. Why?
And it's one of these things it's hard to see unless you go through and literally instrument every single I/O call that you're making.
But to be able to have that just natively appear there and be able to see these services are communicated or even seeing things like, geez, just seeing that cron job or that background job that got scheduled 50,000 times for some reason. And it's just like, yeah, it's, it works, it's doing the work.
But then all of a sudden it's like, huh, that's 50,000 HTTP calls, like while that thing is running.
It's like, it's one of those places where it can be difficult to see that level of metrics and information many times at the application layer, especially if it's not your application that's receiving the calls. I think that's one of the places that I really enjoy it.
Yeah.
And that's kind of our value prop to the platform owner, you know, which is kind of the target audience for Linkerd anyways, is we're going to do this across your entire cluster, across your entire platform. We're going to do it in a way that's uniform.
It doesn't matter what the devs are using for their language or framework or whatever, if they're speaking HTTP or grpc, then we'll give you this uniform layer, not just of visibility, but also of control. Now there is to be fair to the platform owners who have exposed the service mesh config to the devs or whatever, there is an intersection there.
And this is something that I don't think the industry has really like has a great answer for. And I think it's an interesting problem. There is an overlap in control there.
So for example, who should be responsible for the retry or let's say the rate limit of a particular method on a service? Does a platform team own that? Does a developer own that? Does a developer own it but the platform team gets to put some guardrails in place? And if so, like whatever the answer is there, how does a developer get that config into the hands of the platform team? That's not a solved problem.
I've seen people address that in a variety of ways. You know, you can like have annotations on like, what's the open API spec used to be swagger.
So like you could say, "Hey, Devs, you need to like use a spec and the spec you need to annotate it with. And from there we'll like build the Linkerd configs." That's probably the most sophisticated version that I've seen.
But that is an unsolved problem for a lot of people and I have some ideas, but I don't have any great code for solving that yet.
I mean the surface level answer is like, yeah, it applies. They're all speaking HTTP anyway. So yes, to a first approximation, any inference workload is.
It's just a workload that makes these extremely long multi second or more HTTP calls. I think there's a... you know, you can get a little deeper than that and it depends what that workload is.
So for example, there's kind of two areas we've explored. One is, are these actually like agentic workloads that are making MCP calls? If so, yes. MCP happens over HTTP.
But there's a lot of details that you want to dig into there.
If you really want to do observability on, on agentic workloads making MCP calls, or I should say, if you really want to do observability on MCP traffic, you kind of have to speak that protocol. So we gave a little prototype demo of that Kubecon last November, I think in Atlanta.
The other thing we've been diving into a lot is the actual inference serving itself. And this is something that I think is becoming increasingly interesting, especially as people's anthropic and OpenAI bills start skyrocketing.
And the fact that a lot of the open weight models are extremely good, extremely good, you know, even compared to the frontier models. You can run those things locally and you can use VLM or SGLANG or TRT LLM. And a lot of that is happening on Kubernetes.
And now you have these workloads that actually are a little weird from the Kubernetes perspective because they're very large, they consume a whole GPU or if you have multiple GPUs on the machine, they like consume all of them. So they have to kind of be, you know, pinned to that machine. They're very, very slow.
They take a long time to start up, you know, and like takes 17 minutes to load the models into the memory and compile your CUDA kernel and whatever the hell else they're doing. But you know, once they're running, they are these extremely high concurrency, extremely powerful workloads and then what do you do?
You know, like, how do you manage that? Right. And that's a problem. Not just for Linkerd.
I think that's a bit of a challenge for, for Kubernetes itself because they're so outsized and they're so extreme in the spectrum of workloads. And Kubernetes was designed much like Linkerd. It was designed for these really fast microservices where every millisecond is important.
We spent all this time tuning Linkerd to make sure we wrote the data plane in rust and we're measuring things at the microsecond level. And now when we're putting it in front of this workload, that's like, well, it's going to take me 17 and a half seconds to respond anyways. Okay.
But yeah, it's super, super interesting, super interesting area.
And I have a belief, which I'm kind of trying to implement the corresponding code here, that local inference or people running inference on their Kubernetes clusters, whether it's in the cloud or on a VM, I think that's going to be a huge part of what the future looks like for AI.
Dude, I have a much stronger state version of your statement which is, I think it is, it is astonishing when you get the hang of Claude.
If you've used Claude, especially in the past couple months and you have tried to do something serious with it, it is astonishing at how powerful it is now. It's not perfect. You have to learn the tools. You have to know like what it's good for and what's not good for.
You have to know when to insert yourself and that's a process that you like any tool you have to get through. But when you have that thing humming, it's astonishing. I cannot imagine going back.
So I think you're exactly right that we're going to be in this really interesting situation where you know, the devs at least are going to be, you know, and everyone who like takes the output are going to be, you know, they've seen nirvana, right? They've seen just how powerful things are. Can you go back to handwriting code? Boy, I don't know.
You know, that seems like a big regression but at the same time, yeah, if costs are skyrocketing then what are you going to do?
And especially, you know, I think the other thing, the other hammer that's waiting to drop is these companies are going to go public and when they go public now, they're going to have this like S1 filing and you know, they're going to be under the gun. So all the subsidies that are happening right now with Anthropic and OpenAI were like, oh yeah, pay us 200 bucks a month and we'll give you all this.
Like that's going to skyrocket for other reasons and now you're going to be in a really tough situation.
So I think that is all kind of pointing, pointing to, I was pointing to a couple of things, but I think one of the things it's pointing to is a lot more reliance on, on local inference because the big advantage there is you're renting a machine from AWS. Let's say you're in the cloud, right? On AWS, you're renting a machine from AWS, a machine with GPUs. You're not paying per token, right?
You're like, you're using it.
This is an economic model that like you already have a whole company built around like, okay, it's going to cost us this much in AWS spend and you know, maybe we can negotiate with them for a discount and oh, if AWS is too expensive then we'll use another provider over there.
Like companies already know how to manage that and so you can transform all of your token costs into like just good old fashioned cloud compute costs, you know, or, you know, or, and get right back into this very well understood calculus. So it's going to be interesting.
But I think it's going to be a really interesting time for Kubernetes.
I mean I think, I think we're probably going to end up in, I think we're going to have a Google Doc with about a thousand different ways of running inference on kubernetes. But like, but I just, I can't see it. I can't see it either.
Like for the people that have been successful with it to think about going back, it is a thing to behold.
Like, you know, I think we've done almost 100% of our development over the past year has been, you know, AI assisted and it's, you know, our problem domain. Like we think about different problems now. It's, it's exciting. Like it's, it's, it feels, it feels honestly much more rewarding.
But the weird part, that I'd be curious what your take on this is, is I've felt personally that like while it seems more rewarding, it's like, "Oh, I got that feature out that this customer's been waiting on for a month and I got it out in a day." And it's just like that would have been months of work for me to sit down and like knock out. The one thing that does feel a little weird is, while it's rewarding to see the output, the output feels less yours. Do you feel that? Like, it's just like you don't feel as connected to the code.
It's just like, oh, it does the thing that was kind of magical and now I'm off to the next magical thing.
What do you think about open source in this new era? Like, that's one of the things that I feel like that the supply chain attacks have been absolutely bonkers, right?
And I followed Mitchell Hashimoto on what I call Twitter. Still, as long as the domain name resolves, that's what I'm calling it. I'm sorry, ditch the domain name and I'll call it something else.
But someone... Mitchell talked about this over the past few days about how HashiCorp's approach to libraries was fork and maintain, not always taking an update.
And it seems like something that we've. It's funny, to your point, the code was never the value, right? Open source kind of also shows this and doesn't show it at the same time, right?
It's like, hey, it's my OAuth library. I don't give a shit, I didn't write it. I'm just going to grab one, right? It's like I don't care.
I mean like I want the person to off, but like, do I care enough to write it? No. Right? I'm just going to take something, run it, right?
But now, you know, especially in AI assisted development, like for me to take a spec, something like the SCIM spec or something like that, and it's like, hey, generate this library and we just own it and if it works, it works, and that's great. And I'm not going to think about it anymore.
Like, I'm curious, like what happens to open source in the AI era when you get to the point where you can't trust the left pad level of libraries anymore, but they're scattered throughout your code base. Your dependency that you do care about is depending on some dependency you don't care about.
I'm really curious, what do you think happens with supply chain long term and just the proliferation of dependencies in our code base when, you know, we're seeing just the ability to sneak attacks in just all over the stack.
I'm hesitant now. I'm just like, oh, shit. Like, is that. Is this a good thing or a bad thing? I don't know. Like, now I got to go. Look at this.
In the past, I was like, yeah, fucking green upgrade. Yeah, that's good to go.
But, like, I think, like, I don't know, I feel like it may be asymmetrical just from, like, the amount of shitheads out there. I mean, there's a bunch of shitheads, right? Like, and I think we're going to see it fall apart. And like, the. And this is.
This is software adjacent is like, I don't think we're long from people using Open Call to do horrible things to other people. Like, to just, like, single somebody out, you know, and just be like, you know what? This guy's an asshole.
I'm going to use my open claw to just attack his network as long as possible until I get in, right? Like, I don't know.
So it's like, it feels a bit asymmetrical because it's so there's just so many libraries and there's so many, like, I mean, I have libraries that are used in other libraries. I'm just like, "Oh my God, Like, I can't believe people are using my shit for this."
Like, and it just, it feels like it's getting to the point where like, you have to take in, you have to consider your dependencies now, which was just.
It felt like for a very long time, and this may be so naive to say, but it felt like for a long time you could just trust open source for the most part. And now it's starting to feel like a bit like I have to second guess, like a dependency I reach for.
Right? Like, you should have been doing this the whole time, like this guy. You know, I think there's a couple areas.
I mean, gosh, this is such a broad topic. You know, we could, we could talk about this for like three hours.
I think there's a lot of areas where what AI has done, if you step back, it's bas. It's amplified the dynamics. So those dynamics were always there, but now like, oh, you can't ignore it anymore.
So, for example, one that comes up a lot with Linkerd is, is zero trust. Right. Like, one of the things that Linkerd does is it gives you the ability to basically put a little firewall in every pod.
And that pod makes its own decisions about like, hey, this client wants to talk to us and it's calling method A, you know, and its identity is foo. Right. And I have to decide whether I want that or not.
People have been talking about zero trust for a long time, but very few companies have actually gone to the trouble of like implementing that.
Because most of the time the code that's running in your environment, well, it's written by developers and it's been code reviewed and like, you know, it's, you know, kind of like you don't have to worry about that much trust on it. Right, Right. But now, now you're running agentic workloads and they're powered by an LLM and like, what's the LLM going to do?
Well, at some point it's going to find the call that deletes the database and it's going to call it. Like, there's no question, it's not it's not a maybe, it's a question of when. Right. At some point it is going to call it. So now you have to do this.
You have to take this zero trust approach. You cannot allow the LLM to do whatever it wants. Okay?
That dynamic was already there, but now we've amplified it, you know, and it seems like the same thing, you know, with like supply chain attacks and supply chain defenses and stuff like yeah, those dynamics were there. You never could really trust. I mean look at the XC attack. Oh my gosh. The fact that that thing was caught feels like a miracle.
Which suggests to me that there's probably 99 other XZ attacks just as nefarious that were never caught.
Because what, you know, like it just happened that this guy in Microsoft was like profiling something and he just happened to wonder why did take more, you know, 200 milliseconds more in this situation. What a happenstance that was. So there's got to be 99 of those things, you know, buried around little sleeper.
Sleeper agents, you know, agent sleeper agents, you know, so AI is just making it all, making it all obvious for us. Yeah, but I don't know, it's, you know, it's a landscape like it's easy to say, hey, you know, it's just amplify the successing problem.
But the consequence is there's a big shift in the landscape. And the way you have to think about all of these dynamics now is different. You have to be pretty first principled in your thinking. Good luck.
Yeah, yeah, for sure. Last question for you.
So I would love to know because I feel like, you know, just based on the conversation we have, like you're somebody who's leaning into AI. You're doing a little bit of the AI development.
I know many people today that have started leaning into it, like, don't feel like they're doing any less work, right. Like, it can feel exhausting. And I'd be curious, like at the end of your day, like, how different do you feel from a job?
And I know you're CEO now, but writing as an IC writing software all day versus assessing the output of LLMs all day is a very different mental task. And I'd be curious, how are your teams adjusting to that and how does Buoyant look at the value extraction from LLMs?
So it's like if I sit down and I'm the only person on my team using AI code assisted development, I'm not a boon to my team, I'm a hindrance. If I'm the only one doing it, I'm producing this 7,000 PRs a day and just creating work for everybody else.
Now if the whole company is using LLMs, and 100% of the day, we're all just doing LLM output. I've done 60 years of work in a week. The company gets that full benefit.
How are you guys thinking about the value abstraction and how engineers use LLMs without like necessarily giving 100% of the value to the individual engineer, 100% of the value to the company? How are you thinking about that? And like, how is it changing like the way that you work?
Is it changing like how you think about like work hours and days or. I'm just really curious, like, what, where you guys are at with that?
Yeah, there's your problem. You don't review the PRs.
So for us, you know, it starts with, and I kind of alluded to this with my little story about nginx, different parts of our code base are dramatically different in terms of the risk profile, in terms of the necessity to get things right and the spectrum spans. I would say on one extreme side is the core of what Linkerd does. That thing has to work, has to be performant, has to be predictable.
When people insert Linkerd into their platform, we are a pacemaker, we're not an Apple Watch that you put it on like, okay, you don't like it, then you take it off. We go into your body and like you need surgery to put us in and you need surgery to take us out and we have a big dependency.
So that part of it, our usage to date has been on things like analysis, find bugs like things like that.
Maybe limited generation, but everything gets extremely peer reviewed and like a lot of hands on effort like, no line of code gets into the core of Linkerd without a bunch of extremely smart systems engineers who are familiar with all the intricacies there, you know, are okay with it and probably have like tweaked it a bit on that. So that's one extreme. On the other extreme, which is where I operate, you know, is on like the product process.
And there it's been really transformative because now instead of writing a Google Doc and like we have these like, you know, long conversations and like, you know, flesh out the whatever. And that takes place over several weeks. I just ship a prototype. I'm like, hey, here's a prototype, it works.
Here's the five design choices that I had to make in practice. Now let's talk about those things. And like, if we have a question about this one, well, let me refactor it so that it does this other thing.
And like we answer product questions through prototypes and that, you know, that code is basically throwaway and then there's a middle area and that's where there's a gray area. Okay. We run a SaaS service, it's called Buoyant Cloud. You know, it has to be up, right? It has to like serve our customer.
It provides this like kind of management layer and visibility layer for your Linkerd installation which is running in your cluster. But if that thing has a failing component, the worst case scenario there is like, oh, the app is down for some number of minutes. Well, that sucks.
We have SLAs we have to meet. But that's a way better thing than like, oh, a customer's application got taken down because we made a mistake.
So like our tolerance for LLM generated code is higher there. And so if you can map out, I mean, this would be my advice in general.
If you can map out kind of the risk, risk levels for the different parts of code and recognize that they're different, then I think that from there you can naturally draw the type of usage and the way that you interact. That makes sense. I know there are some very AI forward companies where what they do is.
And we're not quite at this point, and I'll leave it as an exercise to the reader about whether this is a good idea or not. But Claude has a plan mode and they do peer review, code review, but they do plan review.
They don't look at the actual code, they review the plan, they go back and forth on the plan and once everyone's okay, then Claw generates a code. I'm sure there's other protections in place but that's how they've evolved.
Like I said, we're not quite at that point and I would say that probably would make more sense for our SaaS product where the risk service area is lower than for the core of linkerd.
But those are the sorts of things that I think also evolve and the benefits and kind of the practicalities evolve as the models themselves get better. So yeah, that's our approach today. I think we're kind of, we do some unique stuff.
The fact that Linkerd is such a critical piece of infrastructure and the requirements on it are so tight, it means that we have to treat it in a very specific way. And I think the majority of the world that are building SaaS apps probably are operating like, you know, in a different way.
But yeah, I'd say the most transformational part of it to date has been in the product process and in the move to prototyping versus writing PRDs product.
And prototyping has been amazing.
I think the other place that I found that it's very, very beneficial without like hitting you with that like the cognitive load of getting the 4000 line PR 16 times in a row. That's heavy, right? But one of the places I found that it's very beneficial is like going in and attacking debt.
Like just like the stuff that like nobody's ever had time for that like hinders everybody. But like you also like, eh, it's like oh, this we know that I've got, we got a flaky test that like 10% of runs failed.
But it's like, ah, it was a hairy thing to figure out. Nobody wanted to spend time on it. Couldn't get a, you know, a PM to give us those T shirt sizes to go in there and do that work.
It's just like, go fix this thing so it never bugs an engineer again. And like that's one of those places where it's like you can, it doesn't matter like what code comes out.
You could, you could throw that PR up three, three or four times until it's right. But as soon as it is, it's like, dude, if that just, if, if 10% of our builds were fail, I wouldn't. If it was 10%, I'd shoot myself, honestly.
But if it was 10% of your builds were failing. No, I know that team's out there. Somebody's listening right now is like 10% I'd kill for 10% of my builds failing.
But like that's a place that I find that it's very Useful to use where it's not necessarily like mentally taxing to like deal with the output.
