It all came out of my being a physicist, right? Because I like to understand things. My goal in life is to understand how stuff works, rather than necessarily to build stuff. But as I was diving into the computers and figuring out how they worked, it was taking up too much of my time. I decided that maybe I could try to automate some of it. Having seen some of the scripting stuff that people had done at the university to automate things like monthly backups and purge file systems, and all kinds of clever things, I thought, "This is quite impressive." At the time, I was reading a lot about artificial life, and I had always had an interest in artificial intelligence. So I thought, what if you could make a kind of system that would enable me to think about computers as I think about physics? I could study the problem, see what computers do—not just what we ask them to do, but actually what they do. Some of it's good, some of it's bad. If we could identify the bad parts, filter those out somehow, and steer it towards the good, maybe that would be a way to regulate computers, as you might regulate a physical system.
In physics, we think about thermodynamic systems, boilers, engines, heat systems, air conditioning, and stuff like that. But computers don't have those kinds of regulation principles built in, and I thought they should. If we built a system that was stable to begin with, it could just fly like a plane on autopilot and correct its minor mistakes. We would set the course that we wanted, the policy that we wanted it to adopt, and somehow it would be smart enough to analyze its sensory data and figure out how to steer towards that desired end state.
I started to look into how that could be done, and that's what got me on the path of writing CFEngine. I think people got their expectations for configuration management reset when it became more popular around the cloud, and several other systems like Puppet, Chef, and Ansible came out. But my idea goes back to 1993, right? Early days, when system administration was fly-by-the-seat-of-your-pants stuff done by bearded wizards with special arcane knowledge and spells and hocus pocus. Trying to turn that domain knowledge into something that could be automated, formalized, and made reproducible was a tall order.
The other side of that, though, was that at the university, things were very different from your average data center environment today, like the cloud. We somehow imagine everything should just be the same because it’s easy it’s kind of the stem cell approach to computing—churn out a lot of stem cells, and eventually they'll differentiate into specialized things by uploading some software. Back then, at the university, every environment was a special kid. They wanted their own software, their own configurations. Any attempt to standardize would be met with fierce resistance by the academic community in particular.
So, not only did this software have to adapt to everybody's special needs, it then had to convert those needs into a stable target state, a desired end state that it could converge towards and maintain—again, like the autopilot idea. But everybody had their own course setting, right? Some people wanted to go to Tokyo, some to New York, and you had to be able to fly their system in exactly the direction they wanted, while simultaneously protecting them from themselves against all kinds of issues, like garbage collection.
Even today, people rarely think about garbage collection in systems. They will run their systems until the disks fill up, the memory leaks to fullness, and then the system crashes. You restart it—Ctrl-Alt-Delete—you run the system by rebooting. As we move towards the cloud, this kind of biological approach to systems—just let things die and then reboot and restart—becomes more common. But back then, computers were much more valuable and expensive resources. They couldn't go down. It was a matter of pride to keep systems running for years at a time. If you managed to keep your system running for a year without rebooting, it was a celebration day.
Times were very different then, and the attitudes towards configurations were very different from the kind of disposable computing we have today. CFEngine emerged in that environment where failure was not an option. Systems could never go down; they had to continuously be stabilizing, bolstering, securing, and recovering in real-time against their worst enemies—the users who were constantly trying to crash them in a sense.
I may have had something to do with that kind of shift in the end, because as CFEngine developed, I ended up taking it to the conference circuit in America. The USINEX LISA conference was kind of the mecca for system administrators back in the day. After an initial attempt to present CFEngine, which amazed me because I didn't realize how many people were using it in 1997 when I first went there. I gave a talk, an academic talk about some of the theory and programming behind it. But then in the evening, I gave a birds-of-a-feather session for interested people. I said, "Come along, I'm going to talk about this thing I made called CFEngine." I rented a small room in the hotel for about 100 people, thinking that if I had a few attendees, that would be great. The room was full when I got there, and people were crowded outside because they couldn't get through the door. In the end, I was standing at the front, planning to show a few slides, but people were literally sitting around my feet. I could barely stand in the room. I just couldn't believe how many people were already using it, even by 1997. It actually threw me a bit.
On the plane going home, I got sick, as I often do on long flights, but I was thinking about how I could explain CFEngine to the academics because they didn't really get it - the users could see the utility of it - but they didn't understand the concept of desired state and stabilization and the different ways of thinking about managing and regulating systems. So I thought, "Okay, how can I describe this?" My first thought, because I was sick, was that it's a bit like an immune system. I went back to Oslo and wrote a paper called "Computer Immunology," which I then presented in 1998 at USINIX. It is still online and widely read.
I made the point that in biology, our strategy for survival has a lot in common with disposable technology. We've got billions and billions of cells, and if you scratch your arm, a few million cells roll off onto the floor, but it doesn't matter. We just make new ones and keep going. There's sufficient redundancy to make systems robust. Back then, we couldn't afford to do that because we only had a limited number of systems. The really big institutions maybe had a thousand, but that was rare.
At that conference, it was interesting. After my talk, some folks from the Sandia government laboratory came up to me and said, "Hey, do you know these guys at Los Alamos? They also have a project called Computer Immunology." One of the other guys at the conference was Steve Traugott, my friend who now runs infrastructures.org. He had similar notions of how to scale systems in production. He'd worked at NASA and other places.
Anyway, long story short, a discussion came out about how to balance the notion of throwing systems away and rebuilding them versus stabilizing this valuable thing that you can't possibly throw away or do without. These competing methodologies —converging towards a perfect, stable state and living forever versus the idea of throwing everything away and letting the next generation take over—were in play for a while.
When virtual machines came along in the early 2000s, suddenly there was a way to do both. You could have this kind of cellular reproductive approach to throwing away systems, rebuilding them, even freezing the state and transporting them elsewhere, and re-energizing them, recovering them from suspended animation, so to speak. That kind of approach took over as it became cheaper and cheaper to do that, and those technologies had to scale quickly. People put more effort into that side than into the desired end-state approach. By the time the end of the 2000s came along, when Puppet and Chef entered the picture, the cloud had definitely got a foothold, and the "throw it all away and rebuild" approach was starting to take over, eroding the focus on the desired state.
So back then, around the early millennium, say 2000-ish, I was very active in the research community around system administration. I was the chair of the mega-conference in 2001. I was involved in a lot of these processes. We had many workshops around CFEngine, which started out as the CFEngine workshop and gradually evolved into the Configuration Management workshop, where tools like Puppet emerged. So there's a whole history there. That meant that we were all talking to each other and adapting to the changes.
My PhD student at the time, Kyrre Begnum, and his friend John Sechrest, whom I met at LISA, were very interested in virtual machines. Honestly, they didn't interest me that much at the time, but they were really keen on these virtual machines. John, in particular, saw the business potential of virtualization as a way to manage data centers. So we built an early model of cloud using CFEngine to instantiate machines and control ongoing processes on different machines.
Because maybe not everybody knows this: CFEngine is designed to make every individual computer be an autonomous entity. It's not a system where you deploy one image to every machine; it had to adapt to various special uses. Every machine had its integrity and autonomy to the point where it was actually impossible to send instructions or “destructions” to the machine from outside, by design decision, both for security and privacy reasons. Basically, people didn't want their autonomy overridden by some central entity.
That model made it less straightforward to deploy to, say, a million machines and decide what each one would do and orchestrate all that behavior. So that required some pretty fancy jitsu stuff to make that happen, which all became part of the theory behind CFEngine. At some point, I realized that the original model I had was missing some pieces.
And I kind of went back to the drawing board and came up with something that ended up being called Promise Theory in 2004; that was actually the first time I wrote it down. And that has been... I thought it was maybe something I'd work on for a couple of months while I figured it out. But it's actually taken over 20 years of my life, figuring out all the consequences of this model, how things cooperate and form systems in everything from biology to physics to computer science to AI and all that stuff, today. It's all embeddable within this framework of Promise Theory, which is essentially about how initially autonomous things cooperate. Turning into something that can be scaled has kind of taken over my life.
The whole thing came out of an argument that I had around 2003 with some guys doing network management in Europe. The classical model that everybody in computer science still uses to this day is that you have to force computers and anything around you to do what you want. So it goes back to our manual intervention kind of stuff. We like to use our hands, get in there. We imagine ourselves wrestling systems with our hands and turning them into whatever we want by our control. But actually, that's not the way things really work. We need their cooperation to be able to do that. People would make systems based on the notion of obligation. "We must make the system do this. We must force the system to do that. It must obey our commands." I turned it on its head and realized that what if you can't do that? What if you need its permission to make a change? And it will only cooperate on a voluntary basis with your instructions or your wishes. So if you say, "I would like you to obey this security guideline," and it says, "Screw you. I'm not going to do it," there's not much you can do about that. If you can get the cooperation of the system, you could say, "I will take the bits that I want, and I will leave the bits that I don't, and I will cooperate to some extent."
So instead of forcing, obliging, and making things mandatory, I wanted things to promise what they would comply with or what they would agree to do or behave as. And to build the scaling of systems from that idea that we respect one another's autonomy and react to one another's promises. So if you won't promise to be a good guy, I'm going to promise to protect myself against you, and that's the way it's going to be.
As a scaling paradigm, that works extremely well because it requires a lot less communication to get on with your own thing and occasionally listen to hints and rumors about this and that, than it does to actually send every single instruction over the Internet, character by character, and often repeated 20 times a day. So it's enormously wasteful to do that mandatory push-based engineering, and it's very scalable and economical to do pull-based voluntary cooperation.
Promise Theory is about that. It says, imagine a world in which every agent, every thing, person, machine, whatever, is an independent entity. You can't force it to do anything. It will only promise what it intends to do. It's a statement of its intent, that declaration of intent that we were talking about in CFEngine. That's the key to the desired state, its lifestyle, what it intends to do. And you have this lifestyle model of engineering in which everything declares its intent to behave in a certain way, and we manage that and corral it in ways, much as we do with humans.
When I started the CFEngine company, I didn't think we'd be able to sell configuration management as a thing. Because it was free, right? It was open source. Everyone was already using CFEngine 2, again literally moss under every stone in every data center. Then came Puppet and Chef the year after, and they were supported by these VCs. And I thought maybe there is a story, they had some customers, so people buying this stuff - it was amazing, why not? So we had a go, that's how the company started.
But my thought was, we already know how to configure systems. What we don't know is how to understand the monster we've created, especially as we got these enormous systems, thousands and tens of thousands and millions of machines, after a while. It's easy to set something up, but as it evolves and degrades and misbehaves or whatever, how do we understand that? How do we keep it on track? We can't possibly keep it in our heads, so the machine needs to do some of the work. Hence the AI connection. Hence the machine learning connection.
But again, the fierce resistance of engineers to adopt those automation methodologies has held us back. And what we've ended up doing is instead of mastering the scaling through technology, we've divided up the scaling into small pizza teams. Everybody dealing with their small system and everybody managing an interaction… symbiosis between a bunch of developers and a bunch of containers which never scales beyond a few dozen boxes. And that people can keep in their head. This is the opposite of what I anticipated back in 1993, where I thought the machine will take over and we'll just interact with it as we do other technologies today.
But people valued their own expertise and saw it as a game, right? I believe many IT workers are gamers, and for them solving problems is like killing dragons on the command line. They would much rather figure out this perfect jutsu move with all the backslashes and minus options to some shell command to make something happen by magic than they would ask some AI or some automatic system, please figure this out and solve it for me. Because they want to have the honor of having done that and prove their skill to their coworkers. And that persists. It shifted away from sysadmins now to developers. So developers are doing a lot of those things, whether they like it or not, many of them don't. That's been forced upon them by that method of working. And it's evolved out of that need to scale the human management task rather than the technological management task.
Interestingly, there are still some of the old users of CFEngine. Pure systems, like, I think I'm allowed to say, LinkedIn, because they never hid that from anyone. They only had like five engineers running their entire world operation and they did it using CFEngine. Five people could scale millions of machines. That was the idea back in the day, because there weren't that many skilled people. But now it's shifted more towards that pizza team model, and I see that definitely the way things are going.
And so the scalability challenges and the communication and corporation challenges also change. Now, you never go to an IT conference without 90% of the conference almost being about IT guys saying, how do we make friends? How do we talk to each other? How do we form communities? How do we hold each other's hands and be good to each other? Because they know how to do the tech. They can build things on a penny, but they don't know how to talk to each other as human beings and cooperate and solve those human challenges of working together and productizing things and serving one another and being a good service industry, in a way.
Absolutely. Going back to your original question, which I lost in my diatribe, apologies. A lot of my research has been about how to represent knowledge and how to write it down in a way that allows it to live in people's heads and not just in some database. Because the worst thing you can do is put it in a wiki or database and let it die. You need to be getting it out and looking at it.
Right now, I'm learning a new language. And you understand the importance of practicing, rehearsing, using, and thinking about how to apply the language when you try to speak something, when you're trying to make use of knowledge. We don't often think about that in response to IT or technical skills or when we go to school or learn physics or something. We don't try to apply it every day. We'll be happy to consult a book after a few years and see if we can pick something up. But really, active knowledge on a day-to-day basis is useful to us, that requires keeping it in your mind.
This actually led me to apply Promise Theory in a different way, something that became known as semantic spacetime. Where you can imagine a world in which every volume in space and time has some kind of function, has some kind of message or purpose. It's solving a problem. Like the table in front of me is holding my glass up. It has the purpose of holding things up, whereas the shelf is for storing books, obviously, and the computer is for doing this and that, and everything keeps a kind of promise. And we organize space and time around us in order to be functional in that way.
This is how everything from ants to human beings adapts the environment around them to create this kind of symbiotic, accelerative process around us. I've spent a lot of time talking about how this works. I wrote a book about it called "Smart Spacetime." I wanted to write it to bring more people along on this journey of understanding how we get from the very basic ideas about organizing stuff, all the way through understanding physics and biology, to AI at the end, with a very small number of basic principles that come out of Promise Theory.
Everything's an autonomous thing. You offer something, somebody accepts it or not, and you make that work, or you don't. You kind of evolve that point of view. It’s the basis of Darwin's evolution. Things offer their random mutations, and they're either selected or not by something else. It's offer and accept. It's this contract idea between autonomous entities. It's not under any grand control; it's an emergent thing. And yet it's the basis on which all of our complex systems have come about.
I do feel we've got a long way to go in understanding how to make these things work. We tend to run after the latest shiny object in tech because it's cool, but figuring out how to put all this to the benefit of humanity is something we still don't know or understand very well.
I was in New York, just as the cloud was coming out, and I had a fierce argument with a good friend of mine about the cloud. He thought the cloud's going to take over everything, it’s so cheap, there's no argument against it. And I said, it's not going to be cheap forever because it's so easy, so good; it's going to end up being expensive. Supply and demand. He said, no, it's going to take over. Well, eventually people will go back to their small systems again. And I think we see that now, things fragmenting, people walking away from the cloud, back to on-premises systems, because not only the cost but the mental cost of coping with that complexity is hard.
The reason we've built microservices and pizza teams is to protect humans from having too much to cope with. The machines can do it; it's we who can't do it. And that's why we scale things in small cells. So we have this cellular - again, the biological model - creating new cells, new organelles, new organs, which we recombine on different levels. And this is the way we need to understand smart spacetime as the technological basis of our next-level civilization.
Huge number-crunching things like the ultimate AI, are not going to happen because we don't want it to happen. We don't want them to replace us, and therefore they won't. We will scale them back until they help us and make us better life forms independent of one another, because we value our autonomy too much to let it go.
