Using Feature Flags to Tame Complexity with Mike Zorn

Welcome back to the Platform Engineering Podcast. I'm your host, Cory O'Daniel, and today I have with me Mike Zorn, senior staff engineer at Rippling.

Mike and I met, geez, maybe a month or two ago at a Honeycomb conference. You were working at LaunchDarkly at the time. A little bit about Mike's journey - he started out in middle school doing TI-83 calculator programming, that's right?

That's right.

We've got some stuff to talk about there because I used that calculator to do a ton of cheating in college. He's carried the love of problem-solving in different roles in different SaaS companies, from LaunchDarkling to Rippling. Now he's helping teams build platforms. And so happy to have Mike on the show today.

Mike, thanks for coming on and sorry about the reschedule.

Oh, no worries. I'm here now, so it works out great.

Awesome. So today I think it's going to be a fun conversation. We're going to talk about platforms, feature flags, I might try to bug you a little bit about the AWS outage recently. But before we hop into it, I just want to take a moment to say thanks for Kelsey Hightower hosting the last three episodes. Really appreciated that. Had some really awesome sponsors, SigNoz and Signadot, working on those episodes. So thanks again, Kelsey, and apologies to everybody who had to hear me talk about what I do during my day job for three episodes. But it was a fun combo with Kelsey.

So, Mike, tell us a little bit about your background. Like, how did you get into software? Maybe talk about these TI-83 shenanigans and what led you to Rippling?

The first time I wrote a computer program was definitely on the TI-83. That's where it started. I was initially using... it was like, "Oh, let me see if I can make a blackjack game." And then I was able to do that. That worked great. But then there was one later, I tried to make like a poker game.

Yeah.

And, you know, I was in middle school, I didn't know anything about programming, so I didn't know that, like, arrays existed. So I was like, "Oh, I need to, like, put all the cards in the deck into variables. But I've only got 26 variables from, like, the letters of the alphabet." And I was like, "Shoot, I guess you can't make this program." So that was, you know, my first of many programming failures that I Learned from later.

I kind of got more serious into software in college. I studied math. I was doing, you know, some computer programs to solve some math problems and was like, "Oh, this is like a pretty enjoyable thing - to write a computer program that, like, does this specific task for me." So when I finished that degree program, I was like, "Oh, you know, there's actually a lot more jobs in software than there are in math, so maybe I should try the software thing out." And it's been a fun ride for the last, I don't know, like, more than 10 years. I don't really like thinking about how big that number is now.

It's getting big. It's getting big.

Yeah, I've got a lot of gray hairs now.

But in any case, yeah, I got maybe more into platform types stuff as I was working at LaunchDarkly... or I mean, the job right before LaunchDarkly where I took ownership of our internal feature flagging system and then replaced it with LaunchDarkly. And saw how much more efficient that made us by having a really good tool for releasing software. Really made that team, I think, more efficient.

That's fun. I didn't realize that you worked on a feature flagging tool before you went to LaunchDarkly. That's extra fun for me because I feel like when you talk to somebody about the product that they work on day in, day out, there's obviously passion for the idea... but you've actually experienced both sides of it. Trying to build one in-house, which I've also experienced that and was like, "Wow, I'm very glad things like LaunchDarkly exist so I don't have to do that anymore." It is a bit more than just a side project somebody could work on at a company, right? There's a lot to feature flagging, more than just is this thing true or false, yes?

Yeah, yeah. It starts out as is this thing true or false? But then you get Customer Success people that are like, "Hey, can I turn this on just for one customer? They want to try it out." And then it's like, "Oh, okay, well, now it's not just true or false. It's true or false for some people, but not for others."

And then you build that for them and then you realize, like, you've created a nightmare for yourself. Because now you have to turn things on and off for different customers because you don't want to give Customer Success people - as great as they are - direct access to one of your prod databases to make these changes. And you're like, "Okay, now I need like a UI or something to enable that." And then you're like, "Okay, now I'm personally maintaining like a minor product." At that point it's like, "Well, maybe I just buy this thing from somebody else." So that was my journey to that company.

That's an interesting one too, because I feel like, you know, looking at like the... we have a lot of classes of different tools in the software engineering space and I think obviously, or maybe not obviously... but the earlier you adopt some of these tools, like the better patterns you get in. But there's obviously weight and time and effort to adopting tools like CI/CD, ephemeral environments, feature flags.

There's so many ways to build and deliver and deploy good quality software and I feel like, honestly, one of the places where you can yield the most out of good quality software early without a ton of friction is adopting feature flagging.

I feel like I've seen so many teams that struggle around staging swim lanes and all sorts of stuff like that. And there's so much pain and there's just so much effort and delivery of software that can be resolved by starting to lean into feature flags early. And I feel like it's one of those things you don't see enough teams lean into until they start hitting that later stage, big teams. And it's like the stuff you can get out of day one, just like MVP testing, being able to get something out there and test it just a little bit in like a real production-like environment is something that you can really yield benefits from as an early stage company.

At LaunchDarkly, what was the customer base like? Were you getting a lot more mature customers? Or like, were you seeing earlier stage companies start to realize like the benefits that they could get out of feature flagging and start to lean into it? Is it still something that is not really realized until companies are at like much later stages?

So I think it's pretty variable.

So, you know, LaunchDarkly has a ton of really big customers and working as an engineer there, those are kind of your focus because you bring in the most money. But there also are a ton of, you know, smaller customers.

This company I was at before LaunchDarkly was kind of one of those customers. I was there from like Series A through B. And this was back when that was not a lot of... you know, series A was like a few million dollars, not like $30 million.

Yeah.

And that company... the CTO, made a pretty good decision early to, like, have feature flags be a core concept. The way we developed software there was, like, just very easy because we didn't have staging environments or anything like this. It was like you just shipped your new thing behind a flag. It was in production within, like, a minute after you merged, and then you turn it on for yourself to test it out. If it didn't work, you'd like, you know, make some changes and, like, that'd be your feedback. You'd still have a really tight feedback loop like that.

You know, that company didn't turn out for reasons that were not really related to our engineering efficiency, but, you know, we had a really efficient engineering team when it came to, like, our feedback loop and how we developed software.

I'm a little bit removed from LaunchDarkly now, so I don't know exactly what the customer makeup, the real customer makeup is, versus the one I perceive from the support requests that got escalated.

Yeah, no, that's fine. But I mean, you were seeing, like, earlier stage companies start to adopt feature flags though, there. Yeah?

Yeah, yeah, yeah. And I think in the early days of LaunchDarkly... like, you know, it's a company where you put the code, like, really deep in your own code base, so it's really hard to take out. So churn has always been pretty low at LaunchDarkly. There was a while where the leading source of churn were the small companies that adopted us that like had just ran out of runway and didn't take off.

Oh, yeah, yeah. That happens with startups. Been there, done that.

Yeah, yeah, yeah.

They're hopefully not doing that right now.

Yeah.

It's funny, I feel like it's one of those places that is so easy to get it in. Right? I mean, like, when you have an MVP out, I think this is one of those, like, technologies also has, like, staying power. Right.

Like, you know, you don't meet a project manager at Series C who's like, man, I really wish we didn't have feature flags. Right.

And I feel like it's one of those things that's so easy to get in early, and it just helps you ship better quality product and also, like, maintain those cloud costs. Like you're saying, it's like not having a staging environment. It's amazing.

Right now there's a bit more rigor to like using feature flags, like responsibly. I would love to like, you know, hear some of your thoughts on that.

But like that same rigor is used to set up and maintain that staging environment and that thing costs you money all day long. Right. So like being able to get stuff out in the production I think is really, really valuable. That being said, it is deeper in your code.

What are some of the ways that you've seen platform teams start to embrace feature flagging?

Maybe what you're doing now at rippling and how do you I guess instill good practices in a team for where to use them and how to use them to cut out a feature and how to clean them up efficiently?

One of the things I've seen is some teams just tend to use them very much on the front end and they say, hey, we built the feature out, you can actually hit it if you hit our API, but we hide it from you using a feature flag. And I've seen other teams where it's like they feature flag down the whole stack.

You can't see it in the UI and you can't access the endpoint unless you're in the flag. Right. And it's like kind of different approaches to like how you want to scope this stuff.

But like how are you seeing platform teams use this internally? Are they using it, are they surfacing it to their end user customers?

And what are some of the practices you're seeing that work well with feature flags?

Yeah, okay, so there's a lot, sorry, those questions.

Okay, I like it.

So I'm probably going to miss a couple things and we'll maybe revisit them, we'll dig in. But I think yeah, there's a lot of different ways to use flags and I think you touched on a few of them and all of them are pretty reasonable.

It just kind of depends on like what type of product you have.

Like, I think that a really simple way to use flags is to just use them in the front end and like ship APIs that aren't used unless you are calling them from the front end.

And that works great if you know, you don't have any concerns about like your customers spelunking into your APIs and trying just like to do random curl requests and you have like a really UI driven product. Like that's a really good approach.

I think that one place where the, you know, when you start thinking about like full stack flagging and like using flags on the back end, a place where this comes in is a lot of value.

Is like whenever you're doing some kind of like performance optimization, you can now be really rigorous about it because like if you're like this query is really important, that the performance is like uniformly better. Yeah. And I'm like spending a bunch of time optimizing this query with a flag.

You can like really do like an actual like a B test type experiment and see how the query latency is changed like with live traffic. And if it doesn't have the result you expected, like you can turn it off really quickly.

So those type of things are I think super powerful for back end flagging.

And if you're on a team that is like kind of, you know, responsible for the performance of some kind of like shared database or something like this, it's a really, really effective tool because you not only get really kind of like high fidelity measurement of live traffic, but you also have the ability to react super quickly if something isn't as you expected. You don't have to do a redeploy or anything like this.

So yeah, I think that query optimization is like a good, it's kind of a canonical example of kind of like a backend flag.

You know what's funny about that? I've used feature flagging for years. I've never thought to use it around like optimizing a query that you're not.

It's funny because like my background is in databases. I've got a ton of postgres experience. I've just never even thought about it along like the read level.

Right.

Like, besides like showing like a new thing but like having an alternate version of like a SQL query that's like, hey, we're trying to optimize this, we don't know how this is going to work in producing and like being able to flip between that is actually pretty cool.

And I've got multiple things I'm working on right now in my day job that I'm like, we're going to push this to prod and hope it performs well, that I might have to take a gander.

One of the things that we've actually seen that works that we've used is like we've had a couple of times where we've done a lot of like consolidation as a startup. Like we have a lot of changes happen to our software and we've had a couple of like subsystem migrations.

For example, we originally built our registry from scratch. We're a registry that stores like, you know, NPM modules, stuff like that. We built ours from scratch. We later rebuilt it on top of like the OCI standard. And this is one of those places where we got to use feature flags.

When we rolled out our OCI stuff, there was a subset of the system that like, went to OCI versus our old registry, and we had issues with it, and we were able to just turn that flag off and everybody just went back to the old system. We were kind of like dual writing, so we're putting everything in both places. But it was one of those places where it was like doing a migration. Right? It's one thing to turn a feature on and off, but like, being able to like, migrate a system live is also a very harrowing thing to do. And to do that nowadays without a feature flag, like, I don't know how I wouldn't go bald from the stress.

Oh, yeah, yeah, yeah, yeah. Even with the flag, you can still get a little stressed.

Several years ago, you know, we needed to kind of like change the entire data ingest layer of LaunchDarkly. And you know, we used a feature flag to do the migration.

It was like, there's this sort of daily ritual I had where I was like, okay, let's expose like 5 more percent of traffic into the new data ingest system that like has some scale issues that we're working through. And it would be like, okay, turn on 5% and then like, stare at these graphs for like 30 minutes and see if anything goes up and to the right really quick, like abruptly, and if so, then like, roll it back. But that was just like a ritual I had for a while of that was, you know... it was a little stressful, but it would have been way, way worse if it weren't that.

I actually wrote a talk about that and then I compared it to like... in this Bay Area, there's like the San Francisco-Oakland Bay Bridge, and they replaced one of the spans like, like 10 years ago now. And when they did that, they had to like, they built like a whole second bridge next to the bridge and then they'd close the bridge for a long weekend and reroute all the lanes between it. And if anything was wrong, it was just like, "Oh, this crucial artery is just out forever," or not forever, but for a significant amount of time. So the amount of testing and stuff they had to do to get the confidence that that would work, I mean, it's a bridge, so they were going to do it anyways, I guess.

But it's Just interesting to compare how things operate in real world infrastructure and software infrastructure and how much more control you get and how much faster you're able to move as a result.

And so for platform teams, I think this is another place where like I said earlier, I feel like feature flags are something that is beneficial to any team that has the time to adopt them. And I feel like there doesn't get to a point where you're like, "Man, we should get rid of this thing."

Thinking of teams as they kind of mature and scale like up the DevOps maturity model and they start moving towards platform engineering. I feel like this is one of those places... like I feel like a platform team in and of itself can be a force multiplier, but this is one of those technologies I feel like can yield a ton of values for engineering velocity, not just for getting stuff to prod, but I mean like (A) surfacing feature flags as a feature of your platform, but (B) just using feature flags in the platform in and of itself, whether it's like releasing new features or pathways to your engineers or even doing things like load shedding some of their services.

Like what are some of the interesting ways that you've seen platform teams kind of adopt feature flags in ways that aren't like the traditional sense of like just hiding a UI element or enabling a feature in a backend?

Rate limiting and logging are kind of, I think, two places.

Logging is may be a little bit more universal where like it's pretty often like if you've got a system with some throughput, like you're not logging at like info level in your whole application. And if you've got some customer that's doing something weird, you often need like, "Oh, I just need to see those info and debug logs for this one person because they're having a weird experience and I don't have enough information to figure it out." A really interesting use of feature flags is to basically change your log levels for a given user kind of in real time.

So like, you know, this person's having an issue, you're talking to them on support, you can increase your log level for them and then now you get all the stuff that you needed to, to see what the problem was.

That's pretty cool.

We do something today, we lean pretty heavy into OTel and so we actually have an attribute in our OTel that like what we can essentially mark that like something has happened in like the, just the full OTel process that's gone wrong. And we actually change our error level if we see that, so we can actually log some more information. But it's interesting. There's been times where we've run into issues where it's like, there is no error - there's no error, but it's like something weird is still going on. No exception has happened, but something weird is going on. And we're like, "Damn, we need some extra information, but how do we signal to the system that doesn't think anything is going wrong that we need additional information?" And how we do that is we redeploy and change our log level - which sucks, it's so primitive.

That is tight. That is sweet. Okay, sorry, continue, I'm bookmarking that one.

After working at LaunchDarkly for so long, I've come to believe that feature flags are not feature flags, but they're really just a tool for changing your application behavior at runtime without doing a deploy.

Yeah.

And when you think of them in that general way, there's a lot of weird stuff you could do. Some of it good, some of it bad. I think the logging one is pretty just fine.

If you have that philosophy around feature flags, another thing that we used at LaunchDarkly at least was rate limiting. So sometimes you have a rate limit you want to apply across everybody, but every now and then you have some important customer that's like, "Oh, I'll give you a little bit of money extra or something to get a higher rate limit." And being able to change that through feature flags is pretty nice because if you already are rolling out features for a customer, you can just kind of use the same way to reconfigure rate limits or something like this for them.

Yeah. And for people that haven't used a more advanced feature flagging system like LaunchDarkly, a lot of the feature flagging systems nowadays, it's not just like a, "Hey, this email address..." or "Hey, is this thing true or false?" You can actually push data into them as well. So you can say like, "Hey, this is the email address and they happen to be of this org that has this SLA." So you can actually make the flags a bit more dynamic because you can actually push data into them. Sorry, just a little note there.

Oh, yeah. That's a good comment because I've been in the game too long to remember that that isn't like the normal baseline.

I mean, they weren't like that five or six years ago... seven or eight years ago, right? I think I saw that feature... I can't remember the first platform I saw that and I was like, "Oh, this makes so many things so much more easier." Because it's like, "Oh, I can give you some context to make the decision versus, like, have it being so Boolean."

Yeah.

So now you're at Rippling and you're actually working on a platform team. Are you already starting to use feature flags? Is it something that was already like a pattern there, or is that something you're introducing in your new role?

Oh, yeah. No, no, no. So Rippling actually has been a LaunchDarkly customer for a while.

Ooh, spicy.

So, yeah, they've had feature flags adopted for a while there. Rippling actually, like, has... I'm still kind of undecided about what I like to think about this pattern but... they have an interesting way for managing having too many flags. Because there's a lot of engineering teams at Rippling, a lot of people are working in kind of like this giant monolith code base, and there's a ton of flags in that code base. That leads to sometimes unpredictable and surprising behavior when there are flags left kind of in a partially rolled out state and somebody forgets to fully roll out a feature or something like this. So they're pretty mature in their adoption of flags to the point where they have so many flags it's causing some problems.

Oh, wow.

It's like thousands and thousands of flags. I think, on average, each developer has like four flags or something like this... and that's after implementing this process that's like driven it down pretty significantly. The process is that each team gets a cap on how many partially rolled out flags they can have.

Okay.

So, like, if you have too many flags that are partially rolled out, you can't create a new flag to do a new rollout. Which is the way we've sold it. We've tried to mitigate the issue at Rippling, and it's been pretty effective. If you look at the charts of, like, how many partially rolled out flags we have, the cap is being respected. But yeah, they definitely are using flags a lot. We're using flags a lot there.

Oh, man, you got me. We had... geez, this was like two years ago. We had redone a major, like, replatforming, and we were kind of doing the thing where... I think people use feature flags a lot for this... where you want to opt into the new UI, right? And it's like, "Oh, I don't like it. I want to go back to the old one."

Well, we hadn't got to that level of maturity yet where we're ready for customers to pick whether or not they wanted the UI. And internally, we're using it. We're like, "This is great. This is awesome." And we put a few of, like, our bigger customers into it, and then we just forgot that there was a feature flag, like, gating off this whole new great experience of the old experience. And we just had a while where people were signing up for the platform and they were like, "This thing's a real piece of shit." And we're like, "What are you talking about? We overhauled the shit out of this thing for, like, months." And we just forgot, like... all of our paying customers were in the flag, everybody internally was in the flag, but no new customers were in the flag.

I think that is like one of the two big things that I'm always watching for with flags is (A) we have 100% flags that are rolled out that we forgot to clean up, and they're like, littered across the code base. And (B), this exact thing of, like, you have a flag, you've rolled it out to, like, a high percent people, but not to everybody, and you've just forgotten about it. Right?

So you said you kind of have a pattern now where it's like, max four flags per person. How have you seen people solve this in the past when you're at LaunchDarkly? And how are you thinking about it now with the team that has such a high number of flags in flight?

Yeah. So I think that, and this is maybe a place where platform teams can come in... is integrating flags with observability is pretty important. What I mean by that is like, if you have spans where like flags are evaluated, you probably want to know what the value of that flag was. And then you can like create monitors to let you know, like, are, you know, people of this segment, you know, seeing the experience that you expect they are. And the thing is like, getting that type of observability to be standardized across your stack is something that is really, really nice.

That was something that we had set up really well at LaunchDarkly, where you know, like pretty much any, any flag being evaluated, we would record kind of like the context that we were evaluating flags with and like the flag value served like, and it would just be like annotated on, on the trace.

If you have an observability tool that lets you, you know, run queries across traces and spans, then you can figure out pretty easily like what fraction of people are seeing this feature.

And then if you're rolling out something new, you can kind of keep yourself honest by looking at such a dashboard or something when you're doing the rollout and realizing like, "Oh, wait a second, there's like 20% of people that are still seeing the old experience. I probably need to go finish this."

One thing that I've seen that's kind of an interesting way to use feature flags along with OTel that we do internally that we've really grown to like. We're an early stage startup, so our API is rapidly changing, adding functionality as we're kind of stabilizing the business.

And at the same time we don't want to release a million versions of our API. We also don't want to break stuff for customers. And so we kind of have this sort of fluid live API.

And so what we do is currently we mark essentially like fields on our API. We use GraphQL, so fields, queries, mutations, we'll mark them as deprecated, like when we're ready to start deprecating.

And that'll remove it from our docs, which is cool. So like nobody's going to learn about it and start using it.

And then what we do is we actually put those deprecated accesses on our OTel traces so we can actually see in OTel we have a dashboard of like who's still using deprecated features. So this is obviously only people that are actively using it.

But like our, our activity is pretty high so it's like people are using it for deploying their software every day. So it's like we're not seeing customers leave for a month and then come back to use something. It's like it's integrated constantly, right?

So we can see people kind of using these deprecated fields. We'll email people, say, hey, we got this better experience, like, start using this or upgrade your CLI, et cetera.

But one of the things that we do like, as a part of our deprecation process is use feature flags.

So when we actually get to the point where we're ready to fully deprecate something, we actually feature flag people out of it so that if we do find somebody, for whatever reason was using a feature that we deprecated and they're still dependent on it, we can actually bring that back because we see there start to be some essentially SLO that's being violated. Sorry, SLO.

Gosh, I mix up my SLA's, SLI's and SLO's, I know that makes me a bad person, but dyslexia, we'll start to see that there's errors there and it's around the deprecated field and we'll actually bring it back in for that person so that we're not breaking their software. And then we'll be like, hey, like, you know, we had a deprecation window for this. Like, we enabled it back for you, right?

And so it allows us to kind of like get people off of that and make sure people aren't using something deprecated and kind of like slowly like wean off the customer base. And then we'll just have like a very small portion of people that are able to use it.

And then we, as we turn that off and like work through like the deprecation, we're like, ah, we were able to fully remove it, but like, if we ever get to that point where it's like a week later, it's this weird feature and somebody's using it again, it'll fail, but then like flip back on. So it's been a really interesting way to like, kind of control not just the introduction of a feature, but like the retirement of a feature.

Oh, yeah, definitely.

And that was, yeah, we employed a pretty similar pattern at LaunchDarkly when it was like, whenever we were making like an API change that would have to be breaking in some way.

You know, you can use flags for this and then identify customers where like, you think there might be an issue and kind of turn it off for everyone that isn't in that group and then work with the people in that group as it's practical to deprecate it further for them and kind of winnow it down until you can actually turn off, delete the code. Yeah.

So at LaunchDarkly, so you said earlier you had a feature flag on that ingest system. Does LaunchDarkly dog food its own live system? Is that how that works? Or do you have a separate LaunchDarkly for LaunchDarkly?

We basically have a separate. Or LaunchDarkly basically has a separate LaunchDarkly for Launchdarkly.

And the reason for that is like, it's like an exact copy of the code and everything. It's just like a second environment that LaunchDarkly deploys to after the normal production environment.

Okay.

Because, you know, if there is like, you know, you turn on a feature flag that brings down LaunchDarkly's UI or something and you can't turn it off, that's pretty bad. That's why there's the second one. Yeah, that's cool.

I love seeing teams dog food their own things. But there is certain classes of software where it gets like.

It's one thing to be like, hey, we're an accounting company, we use our own software to account. But it's like, it's the thing that manages releasing it. It always gets tricky.

So this is a full second clone sitting there that you all are actually using internally. You got to use the full suite of functionality then, which is pretty rad.

Well, it was rad and not rad because I was able to use all the functionality, but we also had to keep healthy. A whole second instance of LaunchDarkly. Yeah, a whole nother set of computers are gonna have problems.

So as being an on call person periodically, that would be sometimes annoying. But it was, yeah, it was definitely nice to have every... all the functionality.

That's not a trivial system to support either. Like LaunchDarkly is like you're in the hot path of a lot of very, very big companies. So even a feature flagging system operating a feature flagging system is still, I'd assume, handling a ton of traffic.

Yeah, it's always like, kind of amazing to just like when I was working there, see the numbers of like how many flags are being evaluated and how many like concurrent clients are connected to the service. But then it would also be like, "Okay, you're on call for this." And you know, the impact of a LaunchDarkly outage is pretty, pretty high. So it was always like, you know, on call, I took it pretty seriously.

Yeah. So how are you seeing teams so like, you know, I know.

I think a lot of ops and platform teams have different takes on the extent in which AI should be involved in our work. But like, how, how are you seeing people start to use AI in and around feature flags? And like, what is, I guess the like, state of the art, as it were? If there can be a state of the art, I feel like the state of the art is changing so quick.

But yeah, so I think there's like a couple of different directions here. Like, one is like, how are teams that are building kind of like AI products using flags? I think that there's some variance there.

And then there's also the question of like, you know, teams are just building normal software. How are they using flags like as they're using, like having AI help them write code?

Yeah.

You know, for the, the second one's maybe more generally applicable. So we start there.

But a pattern that I have seen is where people will just kind of like put a flag around, you know, the code you're writing with AI and you know, just develop the feature kind of like, like normal, but then you, you know, you have, you know, you have that, that flag around it. So if you. Something doesn't work out, you're all good.

And similarly in that additional level of control lets you kind of delegate more to an AI coding agent or something like this to have them produce a feature, like the agent produce a feature and put it wrapped in a flag is definitely something I've seen.

And then on the other side of people that are building AI products, like, I think observability for AI products has kind of been an interesting place where it's hard to always know, like there's not like a, an error that's going to be logged if you get like a not great response from your like LLM call.

So there's, you know, a lot of products that are made to help you figure out like based, like offline, like, you know, you had all these evaluations and then there's a lot of products that help you figure out were these good or bad.

And one tool that it seems like more mature teams in that space use is experimentation to, as they make a change to how they're calling the model or which model they're using, they'll use just normal experimentation, A/B testing as kind of like the final step of a release because that's kind of like the best signal you'll get. Is did this change have the effect on user behavior that I anticipated? And experimentation is a really good tool for that.

So those are a couple of ways that people use it.

It's interesting too because when the new models come out and it's like everybody kind of recoils because it's like got so much dumber, lost all my whatever.

But yeah, I feel like when you've spent so much effort and I feel like it is a lot of effort to get a good prompt, it's like everybody's like, oh, why don't you just AI it? It's like, well the prompt is so hard.

Yeah, but like once you, once you've got a good prompt and like a good set of agents like configured and like working together and then a model changes like oof, that's disruptive. Right. And it's hard. Kind of going back to what I was saying earlier, like how we use it, like migrate subsystems.

I think that's a really great place to be able to swap out like subsets of users. Say like, hey, what is, what does it look like for us with these prompts and agents on the new model? Right?

Just being able to see like how much it has changed and like slowly rolling your customers onto that new model before you just kind of like whole shebang, like hey, let me just change the model name in this config and deploy to production and see how different this is for everybody. Right? Like we do have a big bag of non-determinism and this is one of those one places where it's like you can exhibit some control over it.

I think that in the age of AI, these feature flags become even more important.

Another place I've seen it used a little bit is internally is around the actual prompts, controlling the prompts themselves through the feature flags. And somebody might be saying why. It's like it's very easy to get your rigging into prod, but this one of those things that also kind of sucks a bit.

It's like your data that you have locally, like while you're testing a prompt or whatever is very different than the data that you have in production. Right?

And like that whole like test in Prod, it's like it's very hard to get a good prompt working with like pseudo fake factory eyes, like local data versus like the stuff you have in prod. And when you have super sensitive stuff, it's hard to take super sensitive stuff from PROD to local to test on it.

And like some of the things that we found is pretty interesting is like, hey, push up our rigging for, like, agents and whatnot, but we're actually going to control the prompts until we're happy through feature flags and essentially feed them in.

Yeah, it's just like. It's just kind of like the.

The idea of going through your whole deployment process to change, like, some English sentences is just, on the face of it, seems a little crazy.

Yes.

So, yeah, that's. That's definitely the. The way to. Way to go is, you know, speed up your feedback loop for production by, you know, using flags or a similar tool to.

To like, manage your prompts. Because, yeah, like, there's no reason to build a new binary and all this just to change some strings. Right?

So I know we're coming up on time here, and I know you got a busy, busy new job to get back to, but, like, what are you most excited about in the space right now? Like, what is some technology you've seen or something that you're really excited to start working with?

Okay, so I've been, I don't know, maybe doing more, like, data stuff lately, and I think that the data space is like, I don't know, very rapidly moving towards like, like these iceberg tables and all this, like, a lot. So much more. So much more stuff is interoperating at, like, that kind of catalog level. So I'm just excited to see like data processing systems go from these crazy, incoherent, complicated things that are impossible to interop with and have big egress cost implications to, "Oh, maybe we actually can share big amounts of data between companies in a pretty easy way." So that's something that I'm really excited about.

What are these called? Did you say iceberg?

Yeah, iceberg - Apache Iceberg.

Oh, okay.

Yeah, it's like a data catalog over, like, kind of like object storage.

Interesting.

But it seems to become sort of. It's becoming more and more like the lingua franca of data catalog. So it's exciting to see.

Yeah.

Because that has been such a mess. Like, the amount of time I've spent building products that just, like, ferry data from one system to another is more than I'd like to consider.

Is this for... is it designed for essentially moving data between organizations?

Like, is it more internally... like internal organizations or external organizations? Like, if I'm a big company like Macy's and I've got a bunch of, like, catalog stuff that I want to run through an AI system. Is it like a technology for how do I manage giving my data to another company to build a model and have governance over my data? Or is it like more internal organization organizations?

So kind of both. Like a company could very reasonably build kind of like a data lakehouse thing using Apache Iceberg for just their internal data. They'll have data and other systems that they want to query as well. And today that looks like, you know, some combination of like use Fivetran, build some of your own ingest stuff and other like ETLs.

And it seems like there's a chance that the industry is moving to something that is more like just have your query layer also talk to this like other thing that's being exposed through Iceberg and a different object storage system and use that which, you know, makes these things, these problems much easier because you don't have to like slurp up all the data into your own object storage from somewhere else.

That is pretty cool. Is this something that you're just tinkering with or is this like an initiative at work?

It's something that's like very closely related to my work.

One of the things, the thing I'm working on right now at work is like this ETL system and it would be really nice if in the medium term the industry moves to some kind of agreed upon standard for how we can exchange data that doesn't involve like ETL from rest APIs.

Yeah, yeah. Is there a talk in your future on this or...

Maybe. We'll see. Maybe in the next year or two. It depends on if it really becomes more of a trend or not.

Okay, I can wait. Awesome, dude. Well, thanks for coming on the show. It was great to finally catch back up with you after meeting a few months back.

Again, sorry about having to reschedule, but really appreciate the time today.

Yeah, no, it's all good. Yeah, my pleasure. Thanks for having me on. It's been fun. Yeah.

Where can people find you online?

Shoot. I don't know if I've really got a good online. I can talk about LinkedIn, Mike Zorn. You can also send me an email, I'm Mike at Zorn.Zone.

Those are probably the main two places. I'm not really big on social media right now.

Dude, I envy you. I've peeled so much social media out of my life and it's like the only social media I do is for the Pod and for my day job.

And I'm so happy to be off, like, personally, but, like, oh, man, I still log in and I just see, like, the nonsense and I'm like, "I'm getting pulled back into this."

Yeah. And it's like, I don't know, every now and then I'll go onto, like, Instagram or something. Like, oh, my gosh, it just like sucks you in. Yeah.

It is like, oh, I wanted to do this one task... like, look up if this beer garden has a food truck today. And then it's like I've been on Instagram for 30 minutes doing completely unrelated things.

Yep.

Yep. Like, they've. They've really optimized it to. To get your attention.

Yeah. Awesome, man. Well, thanks again for coming on the show. I really appreciate it. And I'm going to be up in San Francisco soon, so I'm going to track you down.

All right, sounds good. Looking forward to it.

Awesome. Thanks so much. Have a good one.