Yeah, I understand that. It's great on this end.
Well, I mean, I'm in the business for many, many years, as you already pointed out. And I grew up coding. I mean, I started coding when I was like, 14. It was all assembly back in the day and then at some point, I moved on to C, writing assembly code, and then I moved on to other stuff, still writing assembly code. So this is all in the past.
I was with all kinds of companies because I'm naturally curious and I don't want to stick with one single business domain for too long. That's why I kept hopping... well, not really hopping, but staying for a few years here and there, going into insurances, you know, all this fintech - before it was called fintech - banks, retail, all these kind of things just because I wanted to know how it works.
Then naturally, with a growing experience, you get more visibility and more responsibility. They start calling you more like a lead, or then they start calling you architect, et cetera, et cetera. I'm naturally curious, as I said, this means that I actually never wanted not to do things. Like the worst thing for me to happen is to be an ivory tower architect and not... you know, just to do stuff with my brain.
So I actually grew up coding. I never gave up coding. And that's just. For me, there's no difference between, well, developing and then running stuff. I mean, I'm one of those who are actually willingly going on calls. Wherever I go, I always was very close to the Ops. We're helping them with doing their stuff - even not being part of them - because that's... I think this belongs to an overall profile. But maybe I'm just weird enough to do that.
Well, how did Instana happen? I was with a consulting company back then. I was traveling everywhere, also traveling the world a lot, being at many conferences, giving talks at QCons, and was at some point involved with the Erlang community - that I really like, but I don't have enough time anymore. Well, and I needed to evolve.
So I was coding in the hotel room at night. I was very curious about all this big data stuff. I wrote books about stuff like that, like big data, Erlang, etc. Authored four books, have to admit, very far back. Yeah, I still get paid for them, but...
That's a different part of life.
At some point I started digging deep into how to actually process telemetry data on streams. And that was just a hobby, like figuring out how all this stuff works, what technology do you need. Because back then I was a massive supporter of the NoSQL movement. So I dug into all kinds of alternative technologies, data stores, ideas, distributed systems, all of these things that really satisfied my curiosity.
And then I started building something that the CEO of the company then saw at a conference and said, "Oh yeah, we are APM experts by the way. How about we apply that onto APM because it feels like that's the next step in the evolution?" And then, well, we firmed out and started Instana, got some initial funding and I hired a few people and developed this over a course of not even six years, I think. Well, plus the time that I was hacking alone in the hotel room.
Yeah, we made it into a company with up to like... I don't know, I don't remember really... 200 people maybe. And then, well, IBM asked us to join because they liked the technology, they liked the vision, they liked the IP. And yeah, we decided to exit.
And then I stayed with IBM because, well, I was in a strategic space. The automation space at IBM is very diverse and complex. As I said, I'm naturally curious. I'm not sticking with names or whatsoever. The domain was exciting. I learned a lot. I met a few people whom I actually really admired. I stuck my nose into things that I would never have access to otherwise, like real hardware and stuff. That's, wow, exciting.
Well, at some point I just said, "Hey, it's time to start something different."
Right. Yeah. You can cut together half of what I just said.
So formae, that's an interesting story by itself. So my co-founder, Zachary Schneider, he was my first hire in the US when I was Instana CTO. I basically was doing pretty much all of the Ops and at some point I needed sleep. So I was looking through my old contacts and I said, okay, who is... because we started doing international business and you can imagine, this is a thing that was running twenty-four seven, like, it doesn't matter what time zone on. And then suddenly you have American users while you are still having the team in Europe. I live in Munich, Germany, by the way.
I was looking at the map and said, "Okay, the middle of the country seems to be Austin, Texas. It seems to be a tech hub." I was talking to everybody I knew and then I found a guy who knew a guy who was just... they exited with Boundary to BMC and he was not happy with BMC at this moment already. So I chatted him up and he said, "Oh yeah, I can take this over, I'll do the Ops side." Yeah, we ended up hiring him and he stayed with us almost up until the end.
I mean, we're really friends and we kept talking all the time. I mean, multiple petabyte in a short period of time ingress that was installed, that Instana was processing, that is quite a challenge. It's technically definitely not easy. So we had to make choices that we throw overboard, then made different choices, et cetera. We kept rolling and adjusting, et cetera, et cetera. Now through all of that time, we kept discussing.
By the way, when we went apart and he went into a few startups himself, he ultimately was part of the core iCloud team at Apple. His last station before we started this company here was Apple.
Now we kept talking about, "Why is this that we have to work with low level tools that don't help us?" Like literally everything in the Dev space is so shiny and innovative and makes unnecessary work magically disappear, go away. Like optimization over optimization and everything in the Ops space is just so freaking low-level that you have to basically build Rube Goldberg machines over and over to solve a very simple problem.
So we were talking, we were like, "How can we change that? Like, what do we need to do to make it go away? We're tired. We don't want to do any of that."
I mean, everybody in this field you talk with, when you really get closer to them, they will tell you, "Man, I want to check it all in and go garden. I'm fed up, I'm tired. This is just too much."
On one side, there's Ops pressure... we will talk about that, I guess... but then on the other side, the tools are just like basic. Right? Well, and then at some point we've heard that IBM acquired Hashicorp.
This is an area I'm very interested in, obviously. I think we all are, right? When you're saying, like, how it really feels like Ops never really got the shiniest of tools. I think our shiniest tool is Kubernetes, right? And it's a lovely tool, but it is not shiny.
But it is funny, right? Because our jobs are just so fundamentally different than software developers. And people outside of our space, they see the keyboard and they're like, "Ah, you all just write software and deal with computers." But like, our job is so like, "hey, handle arbitrary requests for things", people go and create till kingdom come, and when things break sometime in the future, that's us again. We're just this sandwich... like what's inside of it? Who knows?
You know, it's like, lay down the bread, some software developers do some stuff for a while, something breaks and it's, I guess, our problem. Should be theirs, but, you know, it depends on your DevOps maturity model, right? So like, the job's just so different and it does feel hard to get good tooling in the space.
And so, you know, as you're thinking about just the gamut... because I feel like there's just so many places in the Ops, DevOps and Platform world where you can go in and make the tooling better.
What was it... where you like, "It's the IaC, It's Terraform?" Because I agree, like Terraform, all IaC has not overwhelmed people with successful adoption stories, right? We're still stuck, like low 30ish percent IaC adoption.
And it's crazy, everybody listening to the podcast is like, "Wait a second, we do it at work." It's like, yeah, you might use the tool, but have you fully adopted it? Is your organization running on it? Or do you have a project on it, or some projects, or some percent of your infrastructure?
Most organizations are not as far ahead as the organizations that have platform engineering teams. There's still so many teams that just have an Ops team and they're still in a data center where they've built some stuff. And it's just like getting to the cloud and getting to IaC is just not even on their agenda.
So now everything has been just rushing to the cloud for a decade and now, seemingly faster with AI, this becomes even more important. Right? Especially when we start talking about how do we arbitrate what me and the AI are even talking about. Are we even talking about the same thing? I need a network. What does that mean to AI?
And so I think it's really important that we nail IaC. But what was it about the IaC tool in particular that you're like, "It's Terraform. It's the way that we think about Infrastructure as Code and provisioning and reconciliation, et cetera. That's the problem. And these are the main areas that formae is going to attack to fix it"?
First of all, the reason why we called the company, not formae, but Platform Engineering Labs is actually already showing our broader vision. Platform engineering from my point of view is... you can call it whatever, it doesn't matter... the point is that after a whole decade almost of DevOps cultural discussions, I personally think that it's a tooling problem we're fighting. We're not fighting misalignment between people. We can't call them lazy just because they have to slow down in order to do their job.
I mean, that cultural thing, I don't buy it. Been there, saw a lot. But the thing is, when you look at how Terraform was created, it followed the same standard of kind of... it feels like it is a UNIX tool from the early seventies. That's the thing. Everything happens on your machine. It doesn't have a backhand. You have to babysit that. You have to exactly tell this thing what to do.
And this whole that is happening with AI is actually already going completely in a different direction - you don't need to babysit anything anymore. You need to establish guardrails. Depending on your job, you're letting it go more and more. But also completely without AI, there's no point in inventing a term called drift if you can solve it. Why would I always be dogmatic and say, "Oh, my infrastructure drifted because my code is not the same as my infrastructure."
We think it's completely wrong because it needs to be use case oriented. And that's what the early UNIX tools actually did not do. I mean, okay, I'm a big fan, but there's a limit, there's a line. I think that right now we need to think everything in use cases, because that's how actually successful solutions win. If they understand what the user actually needs. It doesn't matter that you have 350,000 command line flags, you need to nail that one.
Speaking of use cases, the use case clearly is sometimes it is okay to roll the whole thing, sometimes it is okay to enforce from your code into reality, but sometimes it's not. And sometimes you just want a little tiny patch and no matter how hard you try to avoid ClickOps, it's going to happen.
Now who are you to decide that what has been done at night to fix a problem with duct tape is not a valid change? Probably it is. So what people kept doing now is manually catching up on this freaking drift all the time, maintaining this illusion of ownership of the reality, while the very same tool does not support you because it doesn't have anything active running that can catch up on any changes.
So that was the first thing that we said, we need to change that. This whole manual state management, all of this stuff that you keep yourself busy with, that doesn't matter at all.
I mean, how can you even call that a Infrastructure as Code solution if you end up working in the freaking state JSON all the time instead of working in your HCL? How is this even possible? It's not code. That's a technical data exchange format, right?
So long story short, we said, "Yeah, we'll go tackle this one." Because we think that the Infrastructure as Code approach is absolutely important, but we see that, you know, when you try to sell it like Pulumi does with this whole let's go after developers and give them the full blown Python typescript... that's not how I know Ops either. Ops people don't code. I mean, they don't really do full blown programming. The script is probably the maximum they want to go with. So yeah, we saw the whole landscape and we said, "Okay, the last word in IaC is not spoken." Are we the last word? We don't know, but we need to change a few things fundamentally to just redefine it for now (for twenty twenty-five back then) and the future.
I agree with something you said and I feel like it's one of these things that people get so testy on when you talk about it on the LinkedIn. But when you go back to that OG deploying 10 times a day Flickr video... the one that everybody hails, like, this was the beginning of DevOps, this conversation. It was great, it's a great talk, we'll link it in the show notes if you haven't seen it, very old, like two thousand six or something, maybe even before that, two thousand three, something like that... but it's funny, when they're going through this and they're talking about, like, hey, we got to do this, like, this DevOps thing that's going to get coined, the first thing they talk about is the tooling. It's the first thing they talk about. They don't talk about the culture and this and that.
They talk about, "Hey, software engineers, there's this crazy thing called Git, and there's this crazy thing called CI/CD." And people are like, "What are you talking about?" You know, in two thousand three, we didn't know any of this stuff. Two thousand three, it was just shell scripts for names, right? And they show up with these ideas of technology that are, like, growing at this point in time. And, like, this stuff is key to introducing change effectively, efficiently, and making sure it's not broken.
That was the idea, was we can actually take some tooling and treat it as the API between teams - starting to sound a little like platform engineering - and let teams have access to stuff. And what's funny is that was such a grand vision.
It's just like, "Hey, we should use some technology to create a barrier between these teams so people can get what they need and do what they need without breaking and stuff." And then we went. And we're like, "Well, it's all... it's actually... it's cultural. That's why your company is not succeeding it's because you're not culturing right." And the reality is you can't walk into a boardroom with a banner and be like, "Guys, we did it. We're Ron Paul DevOps emojied, we're DevOps now."
That whole pitch of like, "Well, you guys just aren't doing the culture part right." It's like, yeah, or maybe my company's not spending the time and effort to buy some tools and let us experiment with tools. And you might say, like, "Ah, that's the cultural bit. You're not doing it." It's like, yeah, but the cultural bit there is getting to the tools that lets you figure out how to embrace the culture.
And so I've always just thought that that was such a wild... "Well, you got to start with the culture." It's like, "No, man, you got to get some shit in there that works and create some bandwidth for yourself. That's what you gotta to do."
The culture does not fucking matter until like a year out. Like, you see some progress, you're like, "Hey, how do we get people excited about the bandwidth that we've created here?"
Now you can go make yourself a culture, but until then, if you just sit in culture land all day long, you're not going to get anywhere. And like, that was the wild part, is like, our tooling hasn't changed much, right?
And like, I'm seeing conversations now on LinkedIn and Reddit where people are like, hey, like, you know, whether or not you buy into the AI generation, whoever's listening, like, there are people that are generating just tens of thousands of lines of code a day and they're reaching the limits of CI/CD, they're reaching the limits of Git, and people are starting to think, like, are there other tools that we should be considering instead of these things that we've just been using for 20 years? Right?
And so the tooling landscape has been so slow in our space and so far behind, and we're finally at a point where I feel like the world's changing enough, where we might be able to get some breathing room for Ops to consider their tooling, considering how this goes.
Looking back at old tools, we're taking decisions and debt and problems from 20 years ago, and we're saying, "Things still work the same" and it's like they don't. We had servers then, we went to VMs, we went to Cloud, we went to Docker, we went to Serverless, and now we're just horking stuff into LLMs. Like, the world changes. And our tooling has been pretty much band aids that we've patched around for the last 20 years as Ops folk.
You already mentioned this initial DevOps discussion that already came up with a few tools that you need. But the tool sprawl is massive.
You want to do the right job, you need to use something for your templating, then you need to understand the storage details so your state doesn't disappear overnight, you need to add all of these fat layers on top of the bones that are so minimalistic that it almost hurts.
So that was also the thinking that Zach and I, we had by the time when we started it. We said, "Hey, how can I reduce this massive amount of tools to a minimum to get the job done?"
Okay, it's going to hurt because you probably need to let go of a few of them. And in some cases we probably, in the early days, we will not be able to replace some of them for that particular use case. This is fine. But even if I manage to get rid of two or three of them, that's already great.
And this is how we ended up also, you know, making the choice for the language that is in formae, which is Apple's Pkl. Which is, you know, good enough to do some programming, but it doesn't feel like you're doing full blown typescript. And on the other hand, it's completely schema safe, right? So you don't make stupid mistakes.
So yeah, this is the number one. I mean there's so many, so many other things on the list, but I would say this drifting and the tools problem is definitely something that need to be addressed.
Yeah, the tool sprawl is interesting because, you know, I feel like in business and the markets, like you have this concept of bundling and unbundling, right?
Over the years there's like, "Hey, I got cable and now I've got the option of buying 7,000 channels" and then it all kind of coalesces back into a Netflix, right? And like we've seen this happen over and over and over again.
But it just feels like... and I don't know if it is just how different organizations are, if it is how just different people's multi cloud strategies are or whatever it is... but there is something that I feel like just drives Operations, DevOps and Platform Engineers into the anti-bundling world.
They're just like, "There is no way in hell..." Like everybody wants a pitch like, "Oh, my thing is a single pane of glass." But like people are just like, "I don't want a single pane of glass... I mean I do want a single pane of glass. I just want 8,000 paintings behind it that are all kind of collaged together." And it's just like, "Well, if you want a single pane of glass, you want 8,000 things." Right? And so you see this idea of like the portals and whatnot trying to bring all these tools back together so you can see it all in one place. And it's one of those things where it's just like, "Why don't some of these tools just coalesce? Like, why don't they come together? Like, why don't..."
It drives me nuts. I feel like every resume, every job description I see, it's just a smattering on the Ops side of just like, here's... not just buzzwords... but here's a hundred buzzword technologies and buzzword-like products.
The job is so hard because you can do the exact same job, go to another company and they've picked 45 different tools.
And it's like if I'm a React developer or I'm a Node.js developer and I go from one company to the other and they use Vue here and they use React here. Like, sure, like a framework that I use for rendering changes, but like the event loop doesn't change, building my software doesn't change, package and NPM doesn't change. Like, I've got this ecosystem I can behave and exist and work in and I've got one thing and it's great.
And then you get in OPS and it's just like, "Well, we got YAML over here, we got HCL over here, and then we've got this like weird thing in the middle for like our CI/CD over here, and then we've got the CI/CD system, the GIT system, we got this..." It's just like there's so many moving parts that it just feels like the job.
Well, but this is the original idea behind Unix, right? I mean, you make little tools that do one job and then you marry them together. I mean, this is fine. I don't think that this needs to go away, but there's a natural... I mean, well, you can write off a few of these things to the personal ego. We have all egos in the world. I mean, I'm a freaking diva myself.
The point here is that, you know, the operations job... and I'm not excluding platform engineering or, you know, DevOps or SRE... I mean, there is a part of the job that is Ops. I mean, you need to make sure that your shit works. And this is a special job and it has nothing to do with software development per se, although you can use the methods to improve or you know, to sleep better to some extent, but at the end of the fucking day, it's all duct tape.
I mean, that's what it is. Now, the feeling that you constantly need to have as an Ops guy is that you are in the freaking full control. And that's why the single pane of glass does not really resonate well until you can really use this freaking thing as a database. Because you know, in the worst of all cases, you will have to bypass the shiny tool that you're using and solve the problem by hand. If you can't, you're fucked. That is the nature of the job. And that is why people actually go to, well, as low level as possible, because everything that comes on top comes with shiny UIs and all this stuff that was like, "Man, I'm looking at that and it's like, okay, in the worst of all cases when this thing is not available, what do I do?" What is it that I can do to fix this thing if that SaaS solution is not available to me? Or decides to misbehave? Or whatever. Which brings us back to AI that has much more potential of doing stuff like that than anything that is hard coded if you wish.
Yeah, we go with our mentality, with our philosophy as far as saying, "Okay, yeah, it's not necessarily better tooling, but it understands what you need." This is our goal. We want to implement a few things that follow the use case rather than just giving them a freaking roll of duct tape to just do whatever they want.
Because I think that more shit is landing on the desk of the Ops people and that's definitely happening more and more and more and more with all this sprawl of infrastructure that is happening. You kind of need to start thinking more in systems that you kind of combine rather than in low level hacks.
Yeah, it's funny too because like thinking about AI, I would love to talk a bit more about like how you're using it, not only how you're using it, but like, like how you're thinking about it, you know, as a co founder and leader of a company.
But it's funny like we know that things work better with good context, right? And like at the end of the day looking at a YAML file, not a ton of context there, right? I love YAML, but we all joke about like how crappy of representation that is. The no problem, right? All that jazz. But it's funny, like just the more constraint and refinement and just concreteness you can give around your prompt, the better the results are, right?
And like, you know, looking at Terraform, we have a type system in Terraform now. One of the things you mentioned was how types work in formae. And I feel like we had this like big swing over like the past twenty years where it was like, you look back in the nineties and it's all about type systems. And then like early two thousands, like Rubies and Nodes and the Pythons and like types were just gone for a while. And software. It was just pure chaos, right? I mean, yeah, you had objects, but like we didn't really have a type system. Y
Now you're seeing languages start to embrace type systems again. We have TypeScript, we have Ruby 3 with Sorbet and all this other stuff. And type systems are coming back because it turns out that they're very helpful in getting our C tags to work, and autocomplete to work, and everything else.
And so in this age where we're going to be working with machines... and I feel like a few weeks ago I wouldn't have said that. I feel like I'm there, I've seen it. Machines, they're going to be writing code with us and we got to figure out how to deal with that. I'm not somebody who's like, "Yeah, I don't have to write code anymore." We have to figure out how to deal with that. This is a huge fucking problem. It's great, but it's a huge fucking problem at the same time.
But one of the things I think is going to be key is embracing of types is important. And looking at our IaC tools, that is something that can stand apart, is if you can be able to give use cases and use types to be able to guide and almost converse and have a way of talking to these machines of like, "Hey, this is what I want from the cloud." And it just gives you a sentence. You're like, "I don't know if that's actually the interpretation that you got. I want a public network. What does that mean? Right? Like, ah, now if you show me the config, I know what that means."
And so to be able to start talking in like types and things that it can understand and not mix up the idea of a string and a string that comes from a set - very different things. Strings that can come from anywhere versus it has to be one of these threes. Right?
And that is just where I see this being very difficult. And I see people struggling with this. It's like, "Oh, well, I'll do this to make myself some YAML." And it's like there's not a ton of structure there. You might have JSON schema to open API or whatever, like behind your kubernetes manifest, but like most YAML doesn't have a schema that is visibly aware. It's just like it's in a ReadMe someplace. And that's just going to be hard.
So like, how does the type system... and you said it's based off of Pkl by Apple... so how does the type system differ from something like a Terraform type system? And I'd also be curious, what's that learning curve like for an Ops person jumping from an HCL or YAML to a Pkl?
Yeah, I'm with you there. You asked me for a whole list, I didn't want to give you that, but this is number three, the level of detail and the lack of abstraction that is going on.
This is why I'm hoping that platform engineering is going to take off. Because one core idea is that a few people who are more on the low level detail side of things are preparing abstractions for the ones who don't care.
I'm a developer, if I'm curious enough, I can go into the platform engineering team and do whatever the heck I want. But most of the developers, let's agree, they just need to do their job. The job is to write applications that are doing some business stuff. That is the job. The other job is to prepare services for them.
I can tell you, you gave me this example, I don't even agree that the developer needs to know they need a Postgres 16 because who gives a fuck?
The point is your job as a platform engineering team is to give them a service called database that will be something like T-shirt size, have a T-shirt size parameter, team name... I don't know, like green or whatever... from which they will extract enough information and derive low level detail that they can keep changing behind the scenes. Updating and nobody else sees them. That is the whole point.
That is the beauty of... well, what I think, I mean of course I'm biased, but that is the beauty of what we're building. We allow you to abstract that for the others. Like they can abstract that with Pkl, but there's even no obligation to do so because our CLI effectively turns parameters defined in your forma into command line flags. And you have to provide them when you apply. You have to do it and you need to give them values that are constrained. How freaking awesome is that? Because you don't care about all these levels of details.
But that is number three because that is kind of a problem that needs understanding, right?The other two that I mentioned, they're already there. We already actually hate them.
Another thing. Like looking at databases, instead of making people think about availability zones and all this shit, it's like, "What's the SLA of your service?" And do it as an Eno - ninety, ninety five, ninety nine plus.
As the person who's making the Postgres module or whatever, it's like I can say, "Hey, you know what? If you have a service that's 90% uptime, I'm gonna throw that bitch in one zone. We're gonna keep it cheap."
And what's crazy is you can start to unlock the self service that goes fast. And what I see people hang on to all the time is this... this thing that is like rot, it's rot and I feel like it's early, solid principles rot of like DRY. Like DRY is the principle that just needs to fuck off because it just ruins so much stuff. People will be like, "Well, I have a Postgres module, so I don't want to make another one." And it's like, "It's fine to make another one."
If you have one that 80% of your team is just like, "This thing's fucking great." that's 80% of requests that aren't bothering you for anything. You have all the time in the world to find this one teammate who's like, "Hey, that one doesn't do what I need. I need this one parameter that's at odds with what you've done over here." It's like, great, we have a special use case. That right there, that's the DevOps happening. That's the collaboration and the shared responsibility. Like figuring out how to work through something together and make an offering. And you might even find that that thing this developer's asking for, maybe it does eventually make sense in that 80% module. And guess what? You can open a PR and add it when you make that decision.
But I feel like so many people just stop and grind to a halt. They're like, "How do I make this RDS module (database engine aside) handle every case possible to make everyone happy?" And it's like, you're going to spend months working on that module. No one's gonna give a shit about 97% of the inputs to it.
Start with something dumb and simple. Codify your practices into it. And like, give yourself some time to deal with the dumb bullshit later.
Oh yeah, so if people want to check it out then it's very easy, the URL is Platform.Engineering.
That's great. This is awesome. We love it. Yeah, we have a lot of material out there.
So I'm going to the US very soon, actually next week already and I'm going to stay on the West Coast for quite a while. There's a lot of people to meet and discussions to do.
Conferences... don't look at the conferences at the moment, because we kind of found this flow with the team... we're a small team and we kind of found this flow where we can use a lot of AI to help us develop real quick. What a great time to live in, man. I mean when you're running a company now, Oh my God, this is amazing, really. So we're really focused on our roadmap. There's a roadmap that we think that we want to give the community as innovation. But there's also official, you know, expanding support to that and this and this. So it's turning already into constant cases of caring about people who are using it.
We also recently launched the... well, our idea of how builders should build around Infrastructure as Code. So it's not only your code, but extensibility. So basically, writing a plugin with us is already absolutely easy without AI. But when you use AI assistants, it's quick and cheap and completely safe - that's important.
So we see that, like, a few people are already building a Proxmox plugin. I mean, okay, we didn't deliver the marketplace yet. So this means they don't see each other yet. Yeah, that's a pain in the ass right now a little bit. Working... we got to improve that. I mean, small steps. It's a lot of work.
So conferences. We'll look around. We're not going... I don't even know, did it already happen in Amsterdam...the KubeCon?
