Democratizing Kubernetes: The Kubefirst Journey with John Dietz

You're listening to the Platform Engineering Podcast, your expert guide to the fascinating world of platform engineering.

Each episode brings you in depth interviews with industry experts and professionals who break down the intricacies of platform architecture, cloud operations and DevOps practices.

From tool reviews to valuable lessons from real world projects, to insights about the best approaches and strategies, you can count on this show to provide you with expert knowledge that will truly elevate your own journey in the world of platform engineering.

Hey everybody, welcome to the platform engineering podcast. I'm your host, Cory O'Daniel, freshly back from KubeCon North America in Utah. And today I've got with me John Dietz, CEO of Konstruct.

John, thanks for coming on the show today.

Hey, I'm glad to be here. Thanks so much for having me on today.

So John and I met during our interview prep a few weeks back, and calendars happened, and so we had to reschedule a few times. So we actually had the real chance to meet at KubeCon last week, which is pretty fun. Actually, before we just dive into KubeCon, let's talk a little bit about what Kubefirst is and what you're doing at Kubefirst and Konstruct, and then maybe kind of segue into KubeCon just a bit.

Thanks so much. Yeah, I would love to give you just a quick overview of who we are and what we're doing. For the last five and a half years, we used to be called Kubefirst. We just recently rebranded. Now our company name is Konstruct. That's because we just introduced our second product called Colony. So now the Konstruct company has really two product lines.

One of them is called Kubefirst. Kubefirst is an instant GitOps platform, completely MIT open source and free. We believe aggressively that Cloud Native has gotten quite complicated and that there's a better way to start. So basically we look at the surveys to find all the most open source and free stuff that people are using - we look at the popularity of the products. We figure out the pieces that fit together really well so that you can start out with a production-ready environment in a couple of minutes instead of 18 months.

That was kind of the origin of our company. Our mission was just enabling people to adopt Kubernetes and Cloud Native faster. Over the last year, we have been working through the commercial line for Kubefirst, which is an optional user interface called Kubefirst Pro that sits on top of the Kubefirst platform.

You can use it. It's got a great free tier and you can just delete it to get rid of it and keep the platform if you want. So it's a pretty risk-free implementation, but the user interface has a nice sexy way to create clusters, to add applications to your clusters, stuff like that.

Our new product that we're equally excited about, but is very new to the cloud native space, is a product called Colony that we've built to do a couple things. We needed a way to get into data centers. The Kubefirst platform works in nine or so different clouds, but we've never had a data center oriented way to do bare metal provisioning, to do hybrid cloud platform building.

So Colony is very selfish to Kubefirst - it's allowing us an opportunity to repurpose hardware and turn into Kubernetes clusters so that you can have a Kubernetes cluster platform in your data center as well.

With that implementation, we're able to do a lot of really cool tricks. We have the ability to host private clouds. We just recently added a deep integration with the Civo Cloud so that you can privately host their public cloud in your data center, which is pretty cool.

As we deepen the integrations between Kubefirst and Colony, we're going to start expanding into the hybrid data center, multi-cloud ecosystem so that you can start in your data center and just extend it into all of the clouds that you want to - however many of them that we support that you're interested in.

It's a ton of work. It's kind of the infrastructure underbelly of Kubernetes. We're very GitOps forward, but that's really the agenda that we're up against. Our mission is to enable organizations with faster ways to adopt Cloud Native.

I definitely want to get like way, way into the weeds on Kubefirst, but I do want to talk about Kubecon first.

But I do have one thing I want to say about what I saw about Kubefirst when I first saw it. I feel like in our space, when you haven't started using something like Kubernetes yet it, in and of itself, is intimidating. One of the things that was really exciting to me about your project and why I reached out to you was that Kubefirst really does give you an amazing set of tools to get started.

Everybody who's familiar with Kubernetes, rewind back to where you weren't. Just getting to that place where you've got the best tools connected together, ready to go - that's a monumental amount of effort. We do have opinions as ops folk, right? But like, it's hard to get those opinions until you've started using some of the tooling.

So one of the things I really saw that was exciting about Kubefirst was the ability to actually get this up and running quick, start getting familiar with the tools and how they work in your cluster. And then if you decide a tool's not the one that your team needs, not a right fit, you at least get 98% of the work done for you, which is pretty exciting.

Yeah, thank you. You hit the nail on the head in terms of how the whole thing started. My co-founder and I, we started out Kubernetes just like everybody did. And I'm sure we all share that exact same journey.

That journey takes about a year or so to start feeling like you know your way around the block. You go through some tech that you wish you didn't waste your time and you go through some tech that is just like critically valuable to everything that you do.

It's not that that lesson isn't valuable - to walk through the slow way - but it's also really valuable to a lot of organizations to have a faster way to pull business value out of Kubernetes. So when we started the project, we were like, we have to be able to do both. We have to be able to put an opinion forward that helps businesses immediately get value out of Kubernetes, but we also have to let the end users hate our ideas. If there's a tool that we use that they don't want, we have to let them hate it and get rid of it.

That was an architectural challenge for us. How do you organize basically an operating system where you can swap out the parts of the operating system after you get the computer? Which is like a fundamental challenge.

It's been a great project. We have some of the most brilliant minds in the scene and we're excited about our new legs. We've kind of had a long journey. We started grassroots with just the two of us working nights and weekends. For years, we went through a year-long pilot with an enterprise while we were stealth. Again, working on this thing nights and weekends. We finally got a little bit of funding to put a tiny team together. And then just recently we got acquired by this guy Mark Boost.

Now we have suddenly tripled the size of our team. We feel like we're not underwater for the first time in a half decade journey. And we're feeling really inspired about things. So more than anything, we're just now trying to get the word out that, we made it, we did it. We've got our new lease on life and we're ready to get punching with folks who might think that cloud native is a little bit complicated.

Yeah, I don't know how many Ops people there are that are listening that aren't a little underwater. I feel like we've all been for about 20 years, haven't we?

Of course. Yeah, it's madness. The amount of stuff that you have to do to just keep a platform up and operational and current. And then you start to sprinkle in nuances about what the different app teams need. And then you sprinkle in nuances of what the security team needs and what your posture needs to be and how multi-cloud you need to be. Then maybe you need some stuff in a data center and compliance requirements come in and blah, blah, blah. And it's just like… there's a lot to keep your finger on the pulse of, for sure.

You guys were recently acquired, but this KubeCon I had a mild disappointment. One of my favorite things to do is meet all the products that I've discovered throughout the year and go and hunt down swag. And I was not able to find a Kubefirst or a Konstruct booth to get swag this year, so I left Utah extremely disappointed. So why no booth this year?

The sting. We both left SLC a little bit disappointed. We weren't sure how this was going to go. This was a little bit of an experiment for us. We've been honored to be at KubeCon a number of times before. I think we all know that those booths can be a little bit pricey and we had never really taken an opportunity to measure the ROI on a booth.

What is the experience when you're not handcuffed to the booth? What advantages are there to be had by being untethered and just sort of walking the floor and able to go to the talks and able to explore the booths on your own? If I can be honest, I missed having the booth.

Cory 09.11

You missed the booth!

I didn't think I was going to. So running the booth is the most exhausting day you could ever have.

It is.

It's just this relentless stream of people who are all potential clients. You've got this undeniable fresh opportunity to pitch and experiment with your pitch and see what works and what doesn't with your pitch. Figure out what people like and don't like about your product and just this whole matrix of things that as a founder are just like - it's the juice of everything that you try so hard to pull out of the internet.

At KubeCon when you have a booth, you get a billion of them and it's exhausting. You can't do almost anything else it seems like. But if you don't have that, it does introduce new experience.

I got to walk around and actually hang out in a relaxed way with peers that I respect, that I value their opinion. And to talk in a future forward way about the future of Kubernetes, stuff that you want to get out of KubeCon and you could never get with the booth, if you're tied to the booth. Obviously, if you're a bigger team and money's no object and you can just do whatever you want, maybe it's a little bit easier. But when you're kind of small like we are, it's a little bit of give and take whether you do the booth or not.

What's your opinion on the booth? How do you guys do it?

I enjoyed our booth. And I think honestly, like you're right, they are pricey and calculating the ROI is hard as a startup. I'd say to anybody who's listening, it's hard as a startup. Once you hit that Datadog, HashiCorp scale I'm sure it's a straight ROI calculator.

We have a customer from a conference last year and we're almost to nine months into negotiations with them, right? And it's just like some of the bigger companies just take forever to go through. And it's hard to try to hold on to that, to put it into like your ROI calculations: Do we get them? Do we not? When do you abandon a sale? Those are all hard questions and it makes it very hard to calculate - I put a hundred grand into this or 30 grand (whatever it is), did I make money? That conference I'm talking about, we wrote that off as a wash, and now it turns out that we may make multiple times the amount that that conference cost us. So it's super duper hard.

For people that are watching on the YouTube replay, I was pointing at my hairline when he was talking about all the stuff you have to plan for. It's exhausting. It is so exhausting.

We had a great time at our booth this year. I think one the hard things for people is like you see it as sales. Everybody that's walking up to you as a potential sale, and it's not.

I think that's one of the hard things that we did not learn at our first couple of booths. They are conversations and those conversations, if they go well, may turn into a lead. Not everybody that scans that badge for a t-shirt wants to receive your emails, right? And I think it's just hard. You have got to figure out very quickly, is this a person that's genuinely got a problem that we solve? And then do they like our solution? That's your potential lead that may lead to a sale.

A lot of people come up, they just want a shirt or whatever, and it is what it is. And a lot of people come up and they don't like your idea and they want to talk about it.

But the hallway track, I mean, honestly, every time we staff a booth, we staff it with our full sales team and full founder team every conference we go to. And the reason why is so each of the founders and even sales team can get out and spend a few hours on that hallway track. There is so much value in those conversations - seeing the other stuff that's there, seeing what people's responses are.

I hang out at my competitors' booths and listen to the way they talk. I listen to the questions that their competitors ask. I want to know that stuff. It's very hard to get that from SEM rush or whatever, right?

Yeah, that's very true. I actually spent a ton of time listening to pitches that were sort of adjacent to the sphere that Konstruct works in. And that was exceptionally eye-opening! To be able to get that sort of understanding of the angles that they're taking and the direction that they're headed, and what feels good to their users.

The talks as well, you can kind of drop into an Open Tofu talk and kind of measure the pulse of the community on IaC considerations that keep you awake at night or whatever.It is really good to deeply ingrain. Certainly walking the floor is undeniably valuable, but having a home base is nice too - “Meet me at my booth at two o'clock.” is a great thing to say.

It's a mixed bag. It's exhausting no matter what you do. It's always give and take.

Did you see anything there that you were like, “Whoa!”? Was there anything that just blew your mind that was like the coolest takeaway, whether it was at a talk or just a booth?

There were a of couple pieces of tech that I ended up pretty excited about. I like groundcover. I think that they have a neat architecture for observability that could be a little bit disruptive. In fact, I only walked over to groundcover to talk smack a little bit because when we were rebranding Kubefirst, I had a ground cover written down on my notebook as a potential candidate and I ended up bailing on it because it was taken. I had to walk over there to congratulate them on their great branding.

I ended up falling head over heels for their product. I think their product is just exceptional.

I found a vault alternative also by the name of...

Infisical?

Infisical, thank you so much. God, I was going to struggle for that!

Infisical, I think is legit. If I can be transparent, I think I was probably about 95% certain that if we were to switch Vault, we would switch it for OpenVal to just lean into the support of the community that wants an open source product - like what Vault used to be for us. But after seeing some of the nuances about how they differentiate from OpenVal, I think that there is perhaps a play where we don't have to wait for something better anymore. Maybe there's something that's better right here.

Ooh, ooh.

We'll see how it goes when we get through some spike work. But I really liked both of those products.

In the Kubernetes space, I really liked Flatcar. They have a sort of Talos-like immutable Kubernetes distro that I think we're probably going to toy with in Colony and provide a third distribution option beyond K3s and Talos.

Talos is a word I just learned very recently by accident, by the way. But what is Flatcar?

I don't want to give the full pitch because I won't do it well. It's basically designed a lot like Talos is, but it has, in my opinion, potentially a little bit less risk of a license change down the road. The Talos implementation is at Equinix Metal - not to say that they would, but they could down the road do some kind of a license change. Flatcar is a CNCF donated project that sort of covers the same type of initiative.

So I think we're going to add that as a tertiary option for us.

Awesome, that's cool.

That's a great little segue, a license change is always a nice casual segue into some open source combo. Have I made it uncasual?

Not one bit. It's always nice and casual with me.

Have I ruined the amazing segue I took advantage of?

So Kubefirst is open source, Kube Pro is not - that's the pro version on top of it, it does have a free layer. And then Colony, is Colony open source as well?

There are parts of Kubefirst and parts of Colony that are open source. So let me break that down a little bit.

We started the whole company on the Kubefirst open source platform. And the concept there is, “Listen, there's all this great open source in the world. Can we give it to everyone for free in a couple minutes instead of making everyone work 18 months for it?” So that's the Kubefirst open source platform. That's the free stuff. That's the free lunch. That's the one that's going to save your company half a million dollars by lunchtime.

What that does is it basically puts Terraform and Atlantis and Argo CD and Argo Workflows and GitLab and GitHub and Reloader and Ingress NGINX and CertManager and ExternalDNS and blah, blah, blah… all those tools that are in every single one of your clusters, it puts them all together, configures them to work with each other, gives you single sign-on on minute one…

Ooh.

And gives you the keys and says, time to onboard your team to your new platform.

The way that it works is that we have this upstream GitOps template repository that we pull down and we have to ask you three questions in order to give you this platform:

- We have to know what cloud you want to put it in and you have to give us your keys to that cloud.

- We have to know what Git provider you want to use and you have to give me the keys to the Git provider.

- Optionally, if you want to use a separate DNS provider like Cloudflare, you would have to give us those details as well - what domain you want to hook it into.

Then you hit this magic go button and we pull down this GitOps template repository, hydrate it with the details about where you want this thing placed. Then we'll run some infrastructure as code that we manage to create in your own cloud that you self-host and self-manage.

We'll create for you your network, your VPC, eventually we'll work our way to a management Kubernetes cluster that we provision in your cloud. And then we bootstrap that Kubernetes cluster with Argo CD. Argo CD then hydrates using the app of apps pattern, all of the usual suspects that you need in that management control plane. So that's how you get Crossplane. That's how you get Atlantis. That's how you get GitHub and GitLab self-hosted runners. That's how you get all the stuff that makes sort of that management control plane work.

Now you don't need to go beyond the open source platform if you don't want. But at the very end of it, we also dropped this nice user interface on it called Kubefirst Pro. So that's kind of like… imagine the platform was just YAML, Kubefirst Pro is a UI that understands that YAML and knows how to write more YAML. So if you want to use Kubefirst Pro, which has a nice user interface experience, you can just start clicking and saying, “Hey, I want a dev cluster, a stage cluster, a prod cluster. And I want them to be managed with a fleet of these 8 or 9 or 10 applications so that every time I create a cluster, it always has the same usual suspects.”

We have all of that mechanic built into it all the way down to like external secrets operator. When it installs in your workload cluster, it automatically checks back with your management cluster and ties into your secrets manager and all that kind of nice stuff. So it's a really good user interface for a good cluster management user experience in Kubefirst Pro.

On the free tier of Kubefirst Pro, we give you three clusters for free. And then once you need that fourth cluster, we're like, all right, that's where the paywall comes in. And even then it's just 33 cents an hour per cluster to manage clusters beyond that. So it's a very reasonable approach.

That was my extremely reasonable face, if anybody saw it. I thought the pricing was going to be much higher than what you said. It was like, “Oh shit, 33 cents is pretty savvy.”

So it's funny, because three clusters, I think, is a lot of clusters. That's a lot of clusters. I mean, now somebody's like, “I work at SuperCorp, we got eight million clusters.” Yes, I know you do. But for the average startup trying to get started on Kubernetes for the first time in a cloud, three clusters would actually last them…

A good long time, yeah.

A good long time. Even if you have just like a staging cluster, like you’ve got your US prod, you could probably throw an EU prod in there if you need to do some isolation. And that's just on the UI, right? If they're using Kubefirst, just the open source, they can make as many as they want.

Yep, you can make 100 million clusters, no cost to you to construct. Obviously, if you're creating clusters in your cloud, you're going to have cloud costs and that's kind of in between you and Uncle Bezos.

Uncle Bezos is like, bring me those 100 million clusters! Okay, that's an extremely permissive free tool. That's pretty awesome.

Yeah, thank you.

With the ability to create these additional clusters… so I run Kubefirst, I’ve got Kubefirst Pro, I can make some additional clusters… if I'm making those clusters through the UI, is that also backed by Git? So that's actually opening up Gits on a repo that I can reproduce from the full GitOps strategy.

I'm so glad you asked because I didn't explain it well, but you're exactly right.

It'll feel a lot like you're in a cloud console. You'll be like, “I want a new Kubernetes cluster. I want it to be this size. I want it to go to this region.” But when you hit create, you might think that it's making an API call to your cloud, but it's not. We never touch your cloud. We instead are just committing Git commits to your GitOps repository that's managing your ecosystem.

We have Argo CD GitOps managing the state of that GitOps repository and knowledgeable of where your clusters are being managed. So when you say I want to create a new development cluster, boom, a new directory shows up in your clusters directory of your GitOps repo. And then if you wait 30 or 60 seconds, Argo CD is going to be like, “Hey, wait a second, there's something new in this Git repository. I've got to make this real.” And so it's Argo CD and Crossplane that's responsible for driving that new cluster creation into your cloud.

Similarly, when it's time to delete, we don't go to your cloud and delete it either. We go to your GitOps repository and we remove the registration of that cluster from your registry. And then GitOps, through the magic of GitOps… and it's such a beautiful thing if you've never seen it before… it'll actually go backward through the sync waves of what got installed into that cluster and remove them in reverse order. So it's a perfectly clean reversal of everything that got installed to the cluster. It gets torn down in the opposite order. All thanks to the brilliance of Argo CD, but it was put together really well for a good user experience through KubeFirst.

So I know you said you’ve got some parts of it that are open source, some parts that aren't. I mean, this is something I love. I’m obviously kind of involved in this Open Tofu kerfuffle… I don't know if anybody else loves licensing talk, but I love licensing talk.

I love licensing talk.

All right, this is about to become the most boring hour of your life if you're listening.

Everybody settle into your comfy chairs. Let's go.

Yeah, I know. Unless you're a lawyer or have recently been burned... we've all been recently burned though. Sometime in the last five years you've been burned, and if you haven't you will be in the next five.

What goes into your thinking as you're working on a project as big as Kubefirst? It’s pretty big, touches a lot of different licenses, I'm sure. How do you decide what is and isn't a part of that open source license on Kubefirst?

Yeah, it's a really good question. It's unfortunately terribly nuanced and intricate. The general stance that we've taken with Kubefirst is that the platform, because it's a GitOps platform and because it's a GitOps platform that delivers other open source tools, we want that ecosystem to just be 100% ejectable from anything that is closed source.

We know that there are huge enterprise organizations that just have hard requirements about what their license models are, about what their policy is to build into their platform and to rely on tools. It’s just not okay for it to not be open source in a lot of environments.

That's just in terms of compliance, but it gets worse than because as a platform engineer, if I'm going to buy into the Kubefirst platform, I have to trust this company Konstruct. Who is this company Konstruct? How long have they been around? Five and a half years, still not good enough for me. What are their credentials, right?

When you're talking about the infra of your org, it scales depending on how mission-critical your operations need to be from a compliance and security standpoint, but there's just no reason to trust another company. You should not have to trust another company for how your platform got put together. And to make any part of that closed source is putting that end user at risk that they're buying into something and that entity is going to make a bad decision. It could be any bad decision and they're not going to be able to do anything about it. And that's unacceptable to us.

We think that if you're going to rely on us to do your cloud native platform construction and engineering, and use it as a foundational tool for how you put pieces of cloud native tech together, then it's just as much yours as it is ours. That's just the stance of what we wanted to build.

Now the billion dollar question is, how the heck do you commercialize that? And that was a tough thing to get to the end of the road on. First of all, it's a GitOps repository that we literally hand you, right? So it's yours, it lives in your org, we don't have access to it. You're the only person on God's green earth that has access to it after an installation. And it's literal YAML source code that put all that stuff together. So any of that is just like impossible to close source because we give it to you the second you do an installation.

So there's really only an area of the platform that's valuable to monetize on. And we found that to be in the user interface and the user experience of cluster lifecycle management and application management of what goes into those clusters. So we edged out this nice little user interface section where we could say, “Hey, we're going to give you the sun and the moon for free, but we think we can convince you through the user experience and through the information that we're going to be able to present to you in that user interface in a single pane of glass… we think we're going to be able to convince you to become a paying customer and help support that open source initiative.”

We'll see how it works out. We've literally just turned on our paywall last month. And we're just starting to get our first users in the dashboard ecosystem. We’re still waiting on that very, very, very first celebrated swipe. To be determined on how much tuning is in our future at finding that exact right spot.

Basically, the things that we need to protect need isolation from the things that we want to be ubiquitous. And everything that we want to be ubiquitous and everything that we want companies to need to rely on, we believe must be open source. So that if we make a bad decision, you're able to continue on with the initiative of what you thought you were buying into.

Very much like the HashiCorp ecosystem. We bought into HashiCorp years and years ago. To this day, we think that it makes a compelling argument to be still the best IaC tool in the industry. Despite the fact that a year or so ago they said, “Hey, we're not Apache anymore. We're going with the Boozle license.” That made a lot of people upset [Cory laughs] and companies like mine that looked at HashiCorp with sort of arms in the air saying, “Listen, we depended on you. You said that you were open source and we depended on you. And we don't like that decision.”

The fact that it was open source allows a really healthy thing to happen, which is OpenTofu. If there's enough interest to say, “Hey, we can pick these pieces up and run with it.”, then there is an avenue for a whole new thing to blossom that remains open source despite the business direction of a particular company. That's extremely valuable, particularly in the infrastructure space because you're putting so much of your company's discipline on the decision-making of the companies that you depend upon.

That's mostly what guides our decision-making - if we were to cease to exist, how do we set it up so that our commercial users are least negatively impacted and so that, with or without us, the companies that are running this software can continue to run and operate on all those open source software tools that are put together by our platforms? That's kind of our guiding light.

How do you do it though? You probably have to consider very, very many of the same types of scenarios, probably in all kinds of different weird ways.

It's interesting, right? I think we had this very similar problem when we were kind of getting started at my day job. One of the hardest things to sell operations engineers is trust.

Yeah.

Like earned in drops, lost in buckets. And one of the things that people are always concerned about… it's really funny, I feel like when you talk about the cloud, people are always concerned about lock-in. Like lock-in this, lock-in that, am I going to have lock-in? And I feel like we never really talk about lock-in around the things that will actually lock you in.

If I put my data in RDS, will I be locked in? No, the interface is Postgres or MySQL, I can get that anywhere else on the planet super easy. It's a lot of data. I'm going to be sitting around, it's got gravity. I know why it's a pain to move, but I can move it and I don’t have to change my applications. To me, that's not lock-in.

Now, I start building a system really, really dependent on how IAM works… I've always said IAM is AWS's real lock-in. Once you've gotten in there and tightened up that security, that thing is hard to replicate elsewhere.

It really is.

But the other thing is, using tools like ours and like yours and some of the others in the space… if you are operating through us, you have to have a vast amount of trust that we've done enough for security, that we're not going to change a license and put you in a poor place.

So in my daytime job, most of our product is closed source. We are starting to open source parts of it. But the one benefit that we've always had is we've had a feature that we call anti-lock-in from the get-go. Everything runs in your cloud and it's all built on open source tools. So under the hood, we say you're already ejected - you can leave at any point in time. Now we are starting to open source other parts of our platform to make the DevEx not as miserable when you do.

That's always been our big thing, “Hey, it just works when you're gone. You've lost the developer experience, which we think is what you came for.” But now we're trying to open source like smaller tools so that when you leave and if you leave, it's not a shell shock.

Now the wild thing is… I'm not sure if you've been in a similar place… this is one of those features that I feel like if you build it, you just jump the line in building that trust.

We talk about this with people all the time. We built this thing, we designed it so you can leave when you're ready to, or if you want to, because I've been in your shoes where I couldn't walk away from something before.

Right.

And the funny thing is people go, “Oh, that's so cool.” Nobody's ever used it in three years. Like not a single person. We still test it, we still make sure it works and that you can eject from the platform fine. But it's one of those things - If we didn't have it, we'd have less sales.

It’s one of those features that nobody ever uses, but I think it’s important to show that I am trying to put a good foot forward as a CEO to make sure that I can have a good relationship with you. And I'm going to do that by extending you the opportunity to not have one with me, if that's what's right for your business.

The way that you guys are going about that, it sounds like it's very similar. That is trust that not a lot of tools in our space are extending to operations engineers, right?

Right. You know, it's funny… the options that you make available to your end users, you have to pay such an expensive cost for. And some of those are just critical.

I think you're nailing it with the ejection pattern. To be able to fast track somebody into the platform space, it does require a little bit of a leap of faith. It does require a little bit of like, “You're going to set these pieces up for me,” but the rest of that sentence has to be, “and I need to be able to change them back if it's time for me to change them back.”

Building those into the architecture of what we're both up against is, I think, paramount to getting it right.

Yeah. I think there are a lot of things that we're selling, you're selling, many other people in this space are selling… if you're not giving people the ability to walk away from you, it's going to be hard to sell in the first place. Internally, me and my co-founders (both also operations engineers for 5, 10, 15 years or something like that)... like this idea of an ejection pattern or a way to cleanly leave the ecosystem if you want to.

In my opinion, when people are looking at that enterprise grid, when you're looking at all the features that your site offers and there's the basic and then the pro and then the enterprise. It's like SSO, RBAC, we know those are already there. SSO shouldn't be IMO, but whatever, that's where we are. The other one should be, can I leave the platform easily? And I think if you start to think about that when you're buying products, it starts to get real uncomfortable quick for a lot of stuff out there.

So I was very excited to see that as one of the things that's kind of built into your guys' product. Because it's like we need that flexibility for operations engineers to feel comfortable. If we're selling developer efficiency, we can't grind your developer efficiency to a halt if you decide our product's not for you anymore.

Yeah, exactly.

You've got to have that efficiency on the way in and the way out.

Yeah, the lock-in thing is an interesting problem. I agree that it's overstated, but I also agree that it's like underhandedly in the shadows all over the place - it's crazy.

You'll read some cloud’s docs and they'll tell you in a very helpful way, “Hey, here's how you create a Kubernetes platform” and there's no lies in it. But your Ingress controller is suddenly invented by that cloud. And the place that you're storing your secrets is suddenly invented by that cloud. And the place that you're storing your images is suddenly inside that cloud. By the time you're at step five of a seven-step “How to create your first Kubernetes cluster”, you've already bought into five cloud services that have better alternatives in cloud native, that are more portable in cloud native, but all you did was follow the guide.

That's not their fault for doing it, I get it. But there are hooks to be avoided, right?

Yeah.

You don't even know them until you've been in Kubernetes for a year, a year and a half.

So it's a delicate little dance that we're all playing in the hook versus portability versus over-engineering versus keep it simple stupid. There's a lot to weigh, for sure.

Oh man, the hidden lock-in, I love that. I think besides IAM, the other one is those DNS zones. Those records don't quite all work the same from provider for provider. We found that out the hard way with migrating some very weird DNS servers to Route 53, but whatever.

Yup.

So Colony is a bit of a shift in what you guys have been doing, right? Kubefirst was all about getting people into the different hyperscalers, but Colony is a real shift towards on-prem.

Yeah, Colony was a huge shift for us. We probably should have had our heads checked before we said yes.

So here's how it broke down. We ended up getting acquired by this guy, Mark Boost. Mark Boost is the CEO of the Civo Cloud. Civo Cloud has a on-prem version of their cloud that you can self-host if you just buy their appliance. So you can call up Civo and be like, “Hey, you know, I want to host my own region of the Civo cloud.” They'll ship it to you. You plug it in, it phones home. Now you have a private cloud - it works just like your public cloud does, it's great.

That works for almost all their use cases, but it doesn't work in a scenario where people already have their own hardware. And you look at the virtualization space right now and it's a mess. There's a lot of commodity hardware that's being managed all kinds of different ways, but a lot of those users aren't very happy with what's going on in the virtualization space.

Mark was like, “I’ve got to have a way to take advantage of hardware for Civo but, if you guys have that same need, maybe we could just build it up under Konstruct and do integrations instead. Because if we're able to host Civo’s cloud privately, then we could do the same thing - rinse and repeat - with Azure's variation, with EKS Anywhere, with you name it.

If it runs on hardware, once you solve the problem of auto discovery and auto registration, and you have bare metal provisioning and orchestration ability, it doesn't matter what you're doing with that hardware. It could be to host public clouds. It could be to host Kubernetes clusters.

Which is what we selfishly wanted Colony more for - it was to be able to build our stronghold, not in AWS or Google or Azure, but to build that stronghold in your private data center in a way that allows you to extend into those clouds. Because we already have all these different clouds. We're paying that cost anyway.

We natively support AWS, Civo, Vultr, GCP, DigitalOcean, Akamai, you name it. There's a ton of work that goes into all of that. And there's an environment that would love to burst into that. But that environment is a data center oriented environment, generally speaking. And we didn't have any foothold there. So we were like, “All right, let's do it.”

We staffed up, we got into the bare metal space - which is a dark, ugly space. Think Kubernetes maybe seven years ago - that is how bare metal feels. It's just like this gross, poorly blogged…

[Cory laughs]

…dark section of the internet that everybody knows is valuable, but nobody wants to touch because it's ugly and it doesn't work the way they say it works on the internet.

So we just had to roll up our sleeves and learn a whole bunch of stuff that we didn't know. It took way longer than we thought it was going to take. We came in a little arrogant and we thought that it was just going to be like, you know, three months max. It ended up taking us a lot more like six to get it to the point that we have it right now.

Right now we have this cool new product called Colony that you can brew install or grab our binary - however you want to grab it. And if you're on a network and on that same network you have machines that can PXE boot, then all you have to do is run the single Colony init command to register your data center with our Colony cloud. And then you reboot machines on that network and we automatically can drop an operating system onto the machine, figure out everything that it's got inside it (all the disk devices, everything), push it all up, make it available for you to transform that hardware that you already have and turn it into K3s clusters or Talos clusters. I think soon we're going to do Flatcar clusters as well.

Once we have a cluster, you can do anything on it. You can run the Civo Enterprise stack on it and have a private cloud. You could host Kubefirst on it and have a Kubefirst bare metal platform that doesn't touch any clouds - It could just be a privately hosted ecosystem for something that's more sensitive. Or, we can also like flash operating systems onto it like Ubuntu or whatever. And there may be a day when we try to do some hypervisor virtualization stuff as well. We started messing around with Proxmox just a little bit ago for home labs scenarios. It seems to play really nicely with the way we organize the tech.

It's got a really bright future. I think the virtualization bare metal hybrid ecosystem is just ripe for a renaissance. I don't think it's ever going to get that renaissance without it being a cloud native renaissance.

We're leaning on this really neat product that's a CNCF project called Tinkerbell.

Oh, yeah.

They do bare metal automation against your hardwares. And we natively integrate with their ecosystem so that you can just run native Tinkerbell templates and orchestrate it through our Colony user interface.

So a nice easy way if you're in that Tinkerbell space to be able to take any automations that you're building and extend it to something more comprehensive - like organizing fleets of hardware or clusters or stuff like that.

Yeah, it's interesting. Not to offer you business ideas… I'm sure you've probably already thought about this… but like one of things I'm hearing as you talk through…

I’ll take them all.

I don't know if any of them are good, let me just preface it with that.

I've helped a lot of companies with their on-prem to cloud migrations over the years. It's funny, hearing about Colony and then the ability for Colony to be able to boot something up that can run Kubefirst… one of the things that I've seen with a lot of organizations that don't have that cloud footprint yet is you typically start to see a lot more Ops Engineers and SysAdmins who may know Linux administration but may not be familiar with writing Terraform or CloudFormation or Crossplane. They might not be “software developers”...

Yeah, yep.

But it's funny because I meet a lot of these teams migrating to the cloud and they're learning all this stuff as they go. They're migrating a thing. They're learning how the tools work. And they're learning how the services work. And it's always been interesting to me that… especially some of these companies are trying to move on to Kubernetes in the clouds so they can move away from their data centers... we'll do that by building net new stuff in AWS. And then they're literally just playing with all the variables at the same time. They're outside of their network, they’re outside of their space.

It seems like Colony and Kubefirst could actually serve as a pretty decent test bed before a migration.

I agree with what you're saying. I hope that ends up being true for a whole section of the industry. It's hard work. The amount that goes into repurposing hardware in a data center is hard enough on its own. And then needing to build up the rig that's able to provision stuff on that hardware - that's a whole other section and that's also hard enough.

And then Kubernetes itself… man, that is a bear of bears. And it's not that Kubernetes itself is hard, but it is so unorthodox from everything that led up to it. Everything that led up to it led you to believe that everything was going to continue to work a certain way. There was always going to be a binary, there was always going to be some kind of job that runs through some kind of script through some kind of Git thing that happened. And none of that is fully wrong, but Kubernetes changed the paradigm to a data-driven paradigm to describe the contract between what you want and the API you're talking to.

That just blew everything out of the water, right? So as soon as you adopt Kubernetes for the first time, as a SysAdmin or whatever, what I found over and over again is there's this tendency to look back at where you've been and keep trying to lean on that stuff. And it's all incorrect for what you're trying to do. That script is wrong. The security paradigm is wrong. The delivery pattern is wrong. The IaC is wrong. The groups are wrong. The users are wrong. Everything is wrong about what you were doing when you're trying to do what's next, which is Kubernetes, which is cloud native.

That doesn't mean that you can't now learn some new tricks. You can. But my watch says that it's going to take somewhere between 12 and 18 months. And that's a huge cost to organizations. Maybe some organizations, they shrug - millions of dollars, who cares? But for most of us, we have a budget that matters. We have to defend the decisions about our evolutions of our platforms and our infrastructure against the business value that you're going to get out of it.

That's what drives me crazy about Kubernetes - it's so powerful that literally me on my project, I had at one point three… and my co-founder and I, so we should count ourselves too… we had five total engineers. We were supporting eight clouds with five engineers.

Name any other technology where you can run your infra in eight clouds in a way that makes perfect sense to everyone, with only the people that you can count on one hand. It's very unheard of with older practices. But in order to get that, you have to buy in. You have to take that evolutionary leap in the way that things are structured and organized.

I think that's where the whole platform engineering products that are trying to organize this stuff, like Konstruct and so many others, are just trying to level the playing field a little bit in various different ways.

Our way is just like, “Hey, we know you won't agree with all our opinions, but let us give you these anyway so that you can be running in production today. And we'll let you change them.” Have those debates about what the best Ingress controller on Earth is. It's an important debate. You may not land where we did. You might say, you know what, Ingress NGINX, it's great, but there's something newer and hotter and I just need a little bit of that juice.

That is a fine thing to discuss and debate, and to even decide that we were wrong and you were right. But while you're spending that two months, really just doing all the hard science behind which one's going to beat NGINX, let's be shipping a prod and get the business to take the value out of Kubernetes. Instead of trying to get it all right up front and take forever.

I think if we can change that game a little bit, it'll only further embolden how deep Kubernetes roots go into the cloud computing industry. And it's only going to make everybody start to see things in the same ways.

You could drop in to our Slack community right now and you would find 500 engineers that have all in one way or another decided, “Hey, if I was using these set of tools, I would have all these people in here that were all also using these same set of tools.” And if you're a person on a team of four, that might be really valuable to you.

That's what Cloud Native was supposed to give us - that community, that ability to tap into each other and to take advantage of the skills that we're all bringing together. Because we're all basically up against the same challenge. It's not one product versus the other. It's us against the machines. And we're all just trying to survive the swell of tech coming in our direction.

I've said for a long time that the two most valuable features of Kubernetes are portable knowledge and HR. Like if somebody goes, if somebody leaves a team… at the end of the day, it's business continuity.

Let's say that bus factor analogy we always use actually turns into a real bus factor. You know, old Chauncey gets slammed by a bus and you're like, “Agh, I’ve got to replace Chauncey.” If all of our stuff's running in Kubernetes, I can bring somebody in. They're not going to how our stuff works, but they're going to understand how the system works, versus trying to figure out this bespoke thing.

There are certainly places for bespoke ways of designing things if you have some operational complexity that's outside the norm. But like starting to have something that feels a bit POSIXy. Where it's like, “Hey, this is the standard of how we run things on the cloud.”

I don't know. I mean, especially with things like Crossplane, I often wonder how long until the APIs of the hyperscalers themselves isn't just the Kubernetes API. And that might sound bonkers to some folks, but like, we're seeing more and more of the services that they offer covered by the default set of resources that are offered. Let alone the custom ones that you can put in there, right?

Yup.

That value to me as an operator… It's not just the business. I want to be clear about that… there are plenty of operations teams that are sitting here, probably listening to this, where you're outnumbered one to ten, one to twenty. Your time is probably one of the most valuable slices of time in a business. Every minute that I can get out of you doing more stuff that adds value to the business, versus doing something that's necessary but not really valuable, it makes you more valuable. It makes you appear more valuable to the business.

People often kind of glaze over the Kubernetes like it's too complicated. Yes, but it's consistent complexity from cloud to cloud, business to business that you're not building on your own.

That's exactly right.

That's me on a bike shed. Not a bike shed, what is it? Soapbox. [Cory chuckles]

No, I think you're exactly right. Sometimes I look at my own platform and I'm like, “How valuable is a portable platform?” And you should stop and question if it’s valuable to be able to leave AWS and go somewhere else. Because I've worked a lot of places and for one reason or another, it was always a monumental undertaking to abandon a cloud for another cloud.

As a founder of a product like mine, you have to wonder if that message is even important - that it's portable. But then I got acquired by an owner of the Civo Cloud, right? [John and Cory laugh] Which is fun. Meanwhile, all my infra is running in AWS. Kubefirst started in AWS. I'm an old AWS head.

So how do I put my money where my mouth is here?

We were like, all right, we'll move production to Civo. No big deal, obviously. There were nuances and threads about it that lasted like a couple weeks because this domain and blah, blah, and this email and blah… But the actual migration of our management ecosystem… through a stealth operation, mind you. We're doing all this stuff in secret. We're not telling anybody that we're Konstruct.

According to everyone in our open source world, we're still Kubefirst. And meanwhile, we have this whole other Kubefirst on the other side of GitHub that's called Konstruct, and we're building it up in Civo. And we're replacing our AWS ecosystem with a Civo version of that exact same platform.

We don't have that many apps, like we're Konstruct. What apps do we have? We have some charts, we have some CLIs, we have some APIs. In terms of like production facing stuff, we have maybe like 20 things. It's not that huge a footprint in terms of what's public and forward facing.

Still, we were able to take prod and move it out of AWS and into Civo. We didn't even tell half of our team that we were doing it. We just did it in the middle of the afternoon, and we were in Civo late that afternoon.

The only reason we could do that was because the platform was built to avoid the hooks, right?

Yeah.

So all of a sudden, ECR concerns aren't ECR concerns to us. All of a sudden, it doesn't matter that we were using the wrong Ingress controller. All of a sudden, doesn't matter that our secrets is a nightmare and IAM is wrapped it all around 17 times over. We just get to be elsewhere.

It was a one-time pill that will take until the owner of the next cloud buys us again, and then we change production to their cloud. It's nice to know that when that day comes, we'll just boop our way over into the next cloud and there we go.

It was a little trickier than that, but it truly only took us like two hours to shift over. That's an encouraging state for what Kubernetes has done, right? Our platform is Kubernetes and we can take our 100 microservice/20 forward production facing signatures and just ship them across into some other section in some other cloud in some other region in some other Git org in some other domain, and then say, “Hey, everyone, we're done.” Switch the DNSs over and, all of a sudden, we're Konstruct.

That's neat. It's a neat success story. We kind of dog fooded our own success story out of that one. But it's encouraging all the statefulness and the GitOps and defining your contract with APIs as YAML. It provides things that no other architecture can.

It's funny, I have a slightly similar anecdote. We had a customer at my day job that is small, they're a startup. And when they were getting started with us, they were like, “Oh, we're going to roll out on AWS.” They were on a platform as a service and they were trying to get into the cloud. And they're like, “How should we run our apps?” I think they had them in Docker containers or build packs, I can't remember which. But they already had some packaging format. And we're like, you should run on Kubernetes.

And they're like, “Oh, we're a startup. We've never used Kubernetes before. Should we try ECS or something else first?” And we're like, “Look, our platform already takes away a decent amount of the pain. The EKS control plane takes off some more of the pain. And then the IaC - we have like a template library - the IaC that you get started with for your cluster gives you a pretty good place. And we've already got several hundred people running Kubernetes clusters on these modules. You're going to be fine.” And so they sign up, they get on Kubernetes, they're having a great time.

We got a little customer (not a huge customer, not massive), but a few weeks later they canceled their account. And I was like, “What the f**k?” We just did this whole migration, we helped them, they canceled their account. Like, how dare you guys?

So I emailed the guy and I'm like, “Hey, like what happened? What's up? I noticed you cancelled your account. I hadn't heard anything from you.” I was like, shit, what do we do? Right? It was one of those panic founder moments.

So Massdriver (my day job), we're the official infrastructure partner for startups that are a part of Azure's startup program. They do this big credit package thing and as a part of that, you get Massdriver for free for a year.

What happened was he got into that program and was like in their dashboard and it was like, you get Massdriver for free if you go through this. And so what he did was he just built all this stuff on Azure and just literally moved his apps over from his EKS cluster to his Azure and just he migrated off.

The reason he canceled his account was like, I get you guys for free now. And I was just like, “Kubernetes, it's so easy. It's stealing customers from us.” I was like, “Should we cancel this? Should we like tell Azure that we're out, we don't want to help them anymore?”

It wasn't a big enough contract to freak us out, but it was one of those things like, “What is going on?” And when he told me why, I was kind of proud of that. Like I enabled you guys to not pay us. Cool.

That’s right.

I want to be respectful of your time and know we're over an hour. I really appreciate you coming on the show. I do have a couple of questions that I wanted to ask you at KubeCon, but I couldn't track you down on that last day.

There's a new segment that we're adding to the show called TrashOps. It is a bunch of random questions about just the trashiest things you can do in DevOps. And I want to ask you a few and see if we get anywhere interesting with it.

That sounds so fun. Let's definitely do it. Do not worry about my time. My time is yours, buddy.

Okay, first question in TrashOps. Have you ever blamed an outage you caused on a former employee?

Ooh, I can't imagine that I would. I have trouble sleeping enough as it is. I would never do it, I would never do it. Is that true? Hang on, let me think real hard.

How about not them but their work? You're like, “Chauncey wrote this, I don't know.”

I'm so old, there's no way it hasn't happened. I'm not going to say it hasn't happened, but man, the new me is frowning on the old me if it ever did.

Okay, I'll take it. When was the last time you forced push to main?

I am not a force pusher.

No?!

I will do anything before I force push, I will delete my Git repository from my localhost, reclose, and write every line back over. I will not force push to main. No way. Zero chance.

You can't be the first person in this segment and set a good baseline. Like we gotta get some trashy stuff in here. Okay, I gotta dig. I gotta dig. Okay, hold on a second.

Maybe I'm the wrong guy for this segment. You brought an angel onto the show. What do you want from me?

You can't be a cut above the rest. Okay, hold on, here we go, hold on. Have you silenced a pager duty alarm without investigating it?

Oooh, god no! No! Eh... So, absolutely maybe. If I felt like I had coverage and I was part of a group alert, I would hit mute and let it go and allow somebody else to pick it up - If I was in a bind in a soccer game or something like that, I would absolutely do that. But if it rang back, I'm gone.

Okay, have you ever fixed a server issue by rebooting it and hoping no one noticed?

Yeah. I mean, isn't that how Kubernetes works?

I should have said pre-Cattle. Sorry. Have you ever rebooted a Pet server?

Oh man, in the Pet days there was nothing you could do right. Yeah, absolutely! In the Pet days I did everything wrong. [Cory laughs] If you could do it wrong, I did it wrong 65 wrong ways before I figured out one sort of right way and then tried to turn that sort of right way into a career. Before Kubernetes... we were all fighting terrible monsters back then.

Last question, have you ever copied a command directly from Stack Overflow or from ChatGPT and pasted it directly into a production system?

Oooh, directly into a production system… Man, I can't say that I have.

Dang!

I don't think I have. I've always had a good pre-prod and I was born as a QA tester. Spent my first five years in QA. I just don't have it in me. That's my problem.

That's what it is. QA people have zero time for bullshit. That's what it is. First person on TrashOps, 100 % score. I gotta get some trashier guests.

You gotta dock me one point for something. I lied about one of them. Let's just pretend I lied about one of them. Give me an 80%. I can't come out of here with a hundred.

Okay, do you pronounce Kubectl as cube cuddle?

I do, cube cuddle for sure.

Okay, there you go. There you go, there it is.

Is that… Oh, no! Whose camp are you in?

I'm in the Cube Cartel Camp. Cause it's killer.

Cube Cartel? You are not cube cartelling! Like in your day to day, you're like, “Cube cartel apply that?”

Oh yeah, cube cartel apply that.

Oh my goodness, I can't believe I finally found one of you.

I'm just kidding, I'm cube cuddle too! I'm the trash... This list is mostly just stuff I did in the last week that I'm seeing if anybody else is better than me.

That was a fantastic quiz, fantastic quiz.

I'm kidding. This is when I was a consultant for Google. These are all the things I did. I'm just kidding Google.

Okay. Awesome. John, thanks so much for coming on the show. Where can people find you on LinkedIn, X, Blue Sky, etc.?

Yeah, thank you for asking. We're most social on LinkedIn. That's where I do all my work stuff and all I do is work, so that's where most of it is. I do have an X account - you can find me at Vitamin Dietz. We're easy to find, it's Konstruct with a K.

And the Kubefirst project, is that under the Konstruct GitHub org or is under its own GitHub org?

Yeah, thanks for asking. So our GitHub org is Konstructio, all one word.

Awesome, cool man. Well, thanks so much for coming on the show. It was awesome to catch up with you again.

Thanks everybody for tuning in!