Chef is a configuration management system. It has a lovely history, starting with CFEngine 2, which is the primary branch of ancestry for Chef. Puppet was another entrant into that market. CFEngine 2 was the first open-source configuration management system that felt modern and flexible. Many of us built large automated infrastructures on top of CFEngine.
Puppet brought in a different perspective; CFEngine was constrained in terms of how you thought about when things happened. The big moment for CFEngine 2 was the idea of idempotent and convergent operations, where the order of operations didn’t really matter. What mattered was that the operations were idempotent, and the system would eventually self-heal. It thought about the world in terms of types of things you needed to do, like install packages or configure files.
Puppet started to think of the world in terms of resources that needed to be in a particular state, constructing a big graph of all the resources to be managed on a given operating system. For CFEngine, Puppet, Chef, and even Ansible, you are mostly talking about managing operating system instances. Similar techniques and technology apply to tools like Terraform, Pulumi, and the CDK. It’s all related, but what you manage changes. Puppet introduced resource management, and Chef brought the idea that you should write configuration as software. So, you know, when things were straightforward. The DSLs were fantastic. But Chef would let you do crazy shit.
Chef used a programming language instead of a DSL (domain-specific language) like CFEngine or Puppet did. Instead of learning a specific DSL syntax, you used Ruby and wrote an internal DSL in Ruby to express your configuration. This approach was particularly useful when things got complicated. While DSLs were fantastic for straightforward tasks, Chef allowed for more complex configurations.
A great example is a big media company with a popular TV show where people had to vote once a week. They had a huge amount of infrastructure running all the time, just waiting for the one day a week when the votes came in. This team automated that infrastructure, allowing them to turn it on and off the day before and after voting. They did it with Chef, and they were excited to show me. I went to the meeting, and they had written a bunch of Ruby code, using Chef primarily to execute their code. Chef was the runner. My reaction to that moment was fucking fantastic. I was like, "Amazing. You crushed it," because they did. They fucking killed it. They completely solved their problem. Then I was like, "Let me show you how to solve it with Chef. We can make it a little leaner, a little more elegant." But the point of view was that this is ultimately a pragmatic choice where you're creating a powerful tool designed for experts to solve really powerful problems. That's really what Chef was about. It came from my experience using other tools while trying to have big ambitions, like automating everything. The idea was, "What if everything was completely automated?" If the language is constraining, the answer is, "Well, mostly everything, but it only works 200 times, not 250 times in a row." And you're like, "No, I need that shit to work 250 times in a row."
I mean, it hurts. As someone who spent most of my professional life as a person, no one cared about, like, you know, I was a Morlock, I wasn't an Eloi. And I think the, um. So I'm allowed to say it because it was true. It was my people, you know, some of those cultural transitions were just getting people to have recognized that the operations piece mattered and it was as important as the engineering piece and that there was a valid career path that existed. Becoming great at operations was the same as becoming great at engineering, and that as you became good in both paths, you eventually reached the top of the pyramid and sort of indistinguishable from one another.
I think another was the idea that for a long time developers were allowed to write code and operations people didn't. They could write shell scripts, do some Python, or whatever, but they weren't real programmers. You still see this a little now. And so just getting people to the place where they recognize that, no, actually, if you are a skilled operations person who understands how to put together a complex infrastructure at scale, configure and manage it — all of that — you’re a programmer. The language you're writing happens to be the language of this infrastructure, how the switches are configured or how the cloud resources are instantiated or whatever. But your programming and what we need to do is start thinking about it and treating you with the same kind of respect that software developers had.
Now, due to some of the issues with how we've structured our systems and taught people to use them, we still automate infrastructure and the full stack exactly as I designed in my consulting company almost 20 years ago. I'm so proud of that because it's very rare to be a part of something that has that kind of longevity and works as well as it does, but it hasn't really changed. And so the outcomes that we're seeing, especially in big companies, tend to be mediocre. People tend to fall into the middle where they're like, “Well, we had these big aspirations, but mostly we got ground down because the way that we work didn't change enough.”
I think now the challenge is that people are ascribing some of the positive cultural change that came from the DevOps movement as the problem. They're like, “Well, now the problem is that my developers have to write all this operational code”, which was never the point of DevOps. But exactly. Now we start to see silos reemerge. You start to see people's marketing material say things like, “Engineers should write code, and operations people, all the operations people stuff should be an API that people hit through a thing.” There's some risk in that messaging of going back to how it was in 2001.
And, you know, most people don't remember what it was like to do this work in 2001 because most of them weren't doing it. And so they will get to rediscover for themselves the joy of what happens when you build those kinds of walls in between the teams.
Each of those things brings about both a technological capability that you didn't have before. People forget, but in the era before EC2, that was the era of Facebook apps. Facebook apps were the first time on the Internet that you could have a thing that you launched yesterday, and today you have 10 million users. I have friends who I had worked with for years who were working for companies that launched Facebook apps, and suddenly that happened to them. They were literally calling everyone they knew, just begging for gear. They were like, “Do you have rack-mount servers in your closet?” Because they literally couldn't rack systems fast enough for the immediate demand.
We made choices about how to solve those problems in terms of Amazon building EC2 and others building similar stuff. Eventually, the good user experience pieces will be out. A good example is Docker. For as much good as a lot of other containerization stuff has done, the true magic of Docker is the Dockerfile. It's just the ability that once I express what I want in this thing, which I can do pretty much by writing a shell script, what I get out is this repeatable artifact that then boots really quickly. Everybody who ran Docker, ran whatever the service was they wanted, hit enter, and it just downloaded an image from the Internet and ran that shit. You would crawl over broken glass for that. It changed what it meant to think about experiencing this kind of automation. Forget about what it did technically. The user experience was such a leap forward that you could never go backwards. Once you experienced it, you were like, “Oh, if it's not like that, then it's garbage.”
Eventually, those sorts of capabilities find their niches. They settle into the places where they make sense. Here's the place where they don't—things like containers. It turns out they make sense in a lot of places. Things like serverless may make sense in fewer places than we thought, at least in its current form. How it changes your approach to automation and tooling is mostly by thinking about how you want the interaction model to change, less a question of how do I want to change the idea of automation. Given these new capabilities, what new interactions can I enable for people at different parts of the process?
If you're an operations person who used to have to build automated provisioning systems by making the right layer two networks, DHCP servers, BOOTP, and this whole list, then you could get to a place where you could rack servers, and they would automatically install themselves. Now you have to figure out how to build an AMI. But it's roughly the same thing as figuring out what the BOOTP image was that you were going to boot when you built a data center. It's exactly the same. What's changed is the interaction, what changed is the API. Focusing on that layer is where, to me, all the interesting stuff lives because you can't control which technology is going to come into the world, what people like, or what they don't like.
I've never really loved Kubernetes, not because I think it's bad technology or that people who like it are bad. Most of the things I have built seem to not need it. This isn't a condemnation in any way of people who do need it or who love it; it's great. But it doesn't matter if I like it or not. It's a thing that I need to automate because people do like it and want to use it. If your argument as someone who automates stuff for a living begins with, “Well, you should only use technology I like in a way that I like it,” it tends to be a losing argument. You have to think about how to build automation that can automate things you don't understand, don't like, don't expect, or don't agree with. Frequently, the winners in this space are people who embrace that kind of pragmatism. There's always a lane for the people who don't, who are like, “Hey, this is a very opinionated, one-way street kind of move.” Over time, I think those tools tend to do less well.
I think the biggest challenge is the user experience. Whether you simply sit with the people who do the work, or if you're a person who's listening to this podcast and you do some of this work, either as an application developer, as an operations person, as a DevOps person, as a platform engineer, whatever your role is, and you just ask yourself, “How much do I like the experience of how all these things come together and what's it feel like to do my day to day?” And, “Where are the moments in my career where I felt true joy in the way that the work I needed to do and the tools that I was using to do it aligned to give me that moment of just pure, perfect zen.” Rails is a great example. Every single person who touched Rails in that initial glorious era of Rails forever transformed because of what it meant to write applications like that. Washington, it was so much better. I mean, it wasn't a little better. It was crazy better.
Once you felt it, everything from there forward had to be at least that good. And if it wasn't, you were a Luddite. And it's because it was worse, it's because no one was going to take that joy from you. The same thing is true for Docker, right? No one is going to give up the joy of running Docker build and deploying. It's so good. I think that user experience challenge for us is that we haven't really rethought the way that the entirety of the system connects together. We still essentially automate our workflows, the cloud, our data centers, and all of those components, exactly the way that we automated stuff when I started a consulting company 20 years ago. What's different are the tools. You don't use the provisioning scripts anymore, now you use Terraform. But if I was building a checklist of the things you had to cover off on, it's the same slot. You do it at the same time, the way it relates to the other tools.
And right now, for a lot of the industry, the best answer that they have is, “Well, we'll abstract those things from you. I'll give you a portal, I'll build a different kind of API, I'll do whatever.” But they're not really changing the fundamental shape of how we're doing the work. And my belief is that the trouble is we've done enough innovation in the last era to know that simply adding better tooling at the same slots in the stack just isn't going to change the outcomes.
And more abstraction on top of that same tooling, I don't think we'll change it either, because while it can change the user experience for some parts of the work, for others, it's going to make it materially worse. Your ability to understand what's happening in the low level will go way, way down, because your only interactions are these really high level platforms, where once those platforms don't work, your ability to pierce that veil and solve the deeper problems will go down, which means the platform will be adopted only in places where it fits and not in places where it doesn't. Which then hurts the learning value of that abstraction sort of over time, which then kind of leads you to the Dora report where you're like, “And in the end, on average, we deploy once a month”. And so I think our challenge as an industry is we have to fall in love again with the fact that we have control over these systems. These systems are not bigger than we are. They're not more complex than we can understand. We can know the details. And so that enables us to envision a different shape where the outcomes are different. But we don't have to lose what we learned over the last 20 years of doing this work. We can evolve in a way that really leaps us forward. I think that's the challenge for the industry, it's great and convenient to sort of stand on the shoulders of giants. But those giants didn't come from cautious leaps. CFEngine2 wasn't a minor leap from shell scripts. It was an incredible leap from shell scripts. It was a massive transition. And, you know, Puppet and Chef, I would argue, were smaller transitions than CFEngine2 was. Puppet was a pretty big transition, but Chef to Puppet, it wasn't that huge of a leap.
There are some fundamental differences. Using a programming language is different, but we're in an era now where what we need are new giant leaps forward, and those will be riskier, they'll be messier, they'll fail harder. But we need more of it, because I think we now know that without them, we kind of know what the results will be. And so, I don't think that's more of a problem for us as engineers to each other to just be like, “No, Cory, dream bigger, get weirder.” We're just not weird enough anymore because it's become the standard. And people take for granted that the way we built those configuration management systems is now the way that your infrastructure code tools are also built. Their architecture is the same. There's no difference, really, between the internal engine of how Terraform reconciles state. The only difference is that they don't have an Operating System they can interrogate. So they have to store it in data somewhere, and they slap it into JSON. But is that the best we can do? Was that the right design? It's a fundamentally different part of the stack. Maybe it wasn't the best design to do that. Maybe there was a different way we could have gone. And I don't think we're doing enough exploration of those alternative paths. Instead, we're sort of resting in the middle.
Yeah, we can talk about Wing too. I think you're totally right. That Wing is another perfect example of what I'm talking about. And for System Initiative, it starts with the conversation we just had. So that was the motivational piece for me, but also from my co-founders. And that belief that it was fundamentally a user experience problem led us to have… We've done four-plus years of R&D to get to a place where in a minute you're going to be able to use it in production. And because that's how hard the user experience problem was. If you wanted to deliver a fundamentally better user experience and sacrifice none of the power of your existing automation, it took four years of really deep R&D to get to a place where I believe that that's possible to do. It was not hard to build a toy that showed it'd be Neato. It was really hard to build a power tool that could hold up under really complex conditions, never before seen in a production environment like a bank.
How system initiative thought that through was by shifting the perspective of saying, “Well, one of the choices we made early on here was that we were going to automate these systems through writing code and that we would treat them roughly like application artifacts.” So the things that made building applications stable, continuous delivery, continuous integration, source control, that those things, we could just extend them naturally into the way we relate to infrastructure, and that through that extension we would gain all the benefits and a degree of user experience that we enjoyed because we all wanted it to be programming. It turns out that the environments that we're managing, to your point from earlier, are significantly more dynamic than that. They change from all kinds of different points of view. They change underneath you. The cloud providers go up and down, services crash. There's a million things that happen that make those things less good.
We wound up building systems that sort of start from their fundamental position of saying the source of truth is software, which is static, and it's supposed to be a reflection of the truth. But the truth is this incredibly dynamic environment that we don't even really understand and can't interrogate. And so, the System Initiative took that stuff, turned it into data. Then we took that data and we put it on this reactive hypergraph because what you need now is the ability to program it. You have to say, “Ok, I have some representation of what the truth is in the real world now I want to be able to really quickly make a change and see if it would work.” Well, if I have to apply that change to the real world, it'll take 20-minutes or longer, and maybe it's production infrastructure, and I don't want to take it down just to experiment.
Now you have to be able to fork that thing, fork the data, and then we stick it on a graph where each piece of data, like a string, is the result of a function that is itself reactive to its arguments. And the side effect is that you wind up programming this big graph of reactive functions. And once we have that, I can now build a user interface that's more like unity, or more like the things people use to make really complex movies than it is writing just an editor, because I can visualize the data in interesting ways. So it turns out it's much faster to compose infrastructure, if the infrastructure can tell, if different components of infrastructure can inform the configuration of each other. If I have a Docker image that exposes a port number, I should be able to say that the load balancer that load balances that container needs that port number as an input. And I can do it by showing it to you on a canvas and drawing a line. And then if I change the port number in the container, it'll update the load balancer and then tell me in real time if that change would or wouldn't work. It's kind of magic, but it took a very long time to build. Super complicated, and it's a big swing, but it's almost ready, and I'm stoked about later this year you'll be able to use it to run real production infrastructure. And it's sick.
Yeah, and I think Wing... Sorry, I'm going to jump into Wing because I really like it. Wing, and I love Elad, who is the CEO of Wing. Wing's perspective is different. It asserts that cloud systems and tools like SQS queues need the programming language itself, the way we express application and business logic, to seamlessly integrate with infrastructure as part of the code we write. By doing so, we could build simulators in the same way we build mocks for tests, providing incredibly rapid feedback loops for locally building and testing complex cloud infrastructures.
Using programming-level abstractions, developers can understand that messages coming in and out at runtime are mapped to different pieces of infrastructure. What's intriguing is that Wing bets heavily on user experience. It believes detailed infrastructure levels don't matter; getting programming primitives right allows programmers to focus on their core tasks. Operations teams focus on integrating new infrastructure only when necessary.
I'm interested to see if Wing's approach succeeds. Though I'm uncertain, I say this warmly. We recently had dinner together, and I'm a fan. What a bold bet! Even if it doesn't prevail, it pushes the boundaries of what's possible. Undoubtedly, if you're building systems using modern serverless technology and have to choose between Wing or traditional serverless, Wing offers a significantly better experience all day, every day.
Do I think Wing will become the primary mechanism for automation industry-wide? No. But System Initiative might, and that's why I'm building it.
