Yeah, I mean, it's obviously really cool and it's also very abstract, even while it was happening at Docker. I spent 10 years on that company and then I left because I needed a break. But the first five years we were crossing the desert, like no one cared. We were doing containers and we were nobody. Nobody cared.
We didn't invent the technology of Linux containers, but we had a lot of original ideas on how that could be used and no one cared about that. We kept doing it anyway because we didn't have anything better to do. And then, almost overnight, this demo gets people's attention.
I guess the market was in the right place, the conditions were right. Then it becomes ridiculous. It's like too much interest all at once. And then it was five more years of just keeping up with that.
As that happened, just keeping up with insane interest in adoption and also competition and like friction and just trying to figure out, “Okay, now this is serious. This impacts people's lives. How do we make decisions, and keep improving it, and balance new features versus stability?” Like a million things to worry about that don't matter if no one's using your product.
But inside the building, it was weird because it doesn't really change that much your day-to-day. You're worrying about bugs. You're talking to users. You're debating which feature you're going to do next. Then you're bikeshedding over an API or something. And then you're doing all the same stuff. And you're super busy, but you were already busy before. So we had this weird disconnect.
And then we would go to DockerCon (the big Docker conference), where everyone went to talk about containers, and it was just a really special event. Then it would be almost like a spiritual experience, especially for new employees. Their first DockerCon was crazy because… you spend the whole year talking about problems and how to fix them because that's our job, right? We're always at the point where things don't work and then we're just trying to fix that. And we kind of forget that behind us there's a long trail of hopefully happy implementations of Docker so far. And then you get the reminder, like people just come and say, “Oh man, I'm using Docker for this, it made things so much easier.”
So that was really nice. But that's the most concrete signal you're going to get that you're having an impact. Like people come to you and say it's awesome. But you can only talk to a small number of them at a time.
And so if you go to KubeCon, or if you talk to most of the Kubernetes community, containers are primarily about infrastructure. The real revolutionary aspect is we used to do VMs and now we're doing containers. And I mean, that's 100 % true. It had a huge impact. But it doesn't actually capture all of what containers are.
The other aspect that's just as important is that developers, it gives them a tool for thinking about the architecture of their application. And that's really the dividing line. It's about separation of concerns between developers and infrastructure teams and ops.
I think on the infrastructure side, it's very fragmented. I feel like there's a million ways to run containers. You can run them on bare metal machines, on VMs… there's all sorts of really interesting patterns emerging, but it's mostly hidden from the application.
In particular, the trend that I find really interesting is the line between what is a VM and what is a container, at the infrastructure level, is blurring. Like now you have projects like Firecracker… I forget, I'm really bad at remembering all the project names. But there are ways to run a VM as if it were a container, basically. Or make running a container feel like you're running a VM. So you can get the container tooling and ecosystem, but then when you're running that workload in your data center or in your cloud it has the security, the isolation properties of a VM.
So the lines are blurring. In infrastructure, I guess container versus VM increasingly will become an irrelevant question. Like it'll be both kind of. So you have VM deployments, but why would you not benefit from the ecosystem of container tooling? Like there's no downside, right?
On the application level, it's different. It's a war of the platforms. It's a war between platforms on getting developers to target their platform. So there it's more about, am I targeting Linux as my platform? Because containers are really Linux containers. Most of the time, it's not Docker, it's Linux/Docker, you know?
Well, we went through several phases. Initially, it was just a completely new category. And so for the longest time, while I was there, it really was about explaining the container model. Explaining containers, what they are, why they're useful, how to compare them to what you have.
You kind of had to buy into a completely new ecosystem, right? Life was good. You had storage, compute, networking, logging, security tools, whatever, like a whole suite of products for infrastructure. And now all of a sudden you're supposed to revisit all of it.
Some of those products you were unhappy with so maybe problems will be solved, maybe things will be faster or more scalable or whatever. But other products, you weren't planning on revisiting. Re-platforming is disruptive.
Once this movement started, a lot of people got sort of forced to pay attention and learn. And you know, depending on where you are in your business priorities or in your career, you may not be happy about that.
So the transition from the early adopters that were just by definition excited about containers… they were the ones who understood, they experienced the pain of not having containers, and they understood the potential if they could just get everyone to switch to containers… that was the fun part for us.
We were making things happen. We were changing how applications get deployed. But all the stuff we take for granted now, by the way… like we call cloud native, et cetera… I think especially younger engineers, they've never known how it was before.
Like the concept of building a container around the application instead of starting from a server that you had to upload files into and kind of change from the inside. Like that's just sort of how things work now if you're out of school this year. Like, “Well, of course I build a container around the app. What are you talking about? Like a build a server, like a little server around the app. That's how like, what's the… I don't understand. What are you talking about? How would you do it?”
But we have the opposite problem. Like, “What do mean build a server around the app? Like it's supposed to be the other way around.” Or even little things like the Docker file that came a little bit later that in itself was like an epiphany.
First we had Docker, you could run, run, run, and then you could commit the states of each container after each change. You can look at it from the outside, it's this movable unit and it's lightweight enough that you can do that, unlike a VM. And then, you know, basically this idea emerged, “Oh wait, we could actually use this to build. It's so cheap and fast, plus we could do some caching.”
And then we experimented with this very basic Docker file, like you can see still how basic it is. You could see it came out of a prototype and it just never really… we never got a chance to finish it because as soon as people saw it they were like, I want this right now.
In the community, the enthusiasts said, “I want this right now. This is perfect.” And here we are stuck with Docker files everywhere. But their colleagues who were forced to look at this and be like, “Okay, now I have to learn this new thing. Great.” They really struggled to understand why.
It's just a really different model. You look at a Docker file, it gives you a list of things to do from the beginning and then you're rebuilding again and again. In the pre-Docker world, if you have a list of 10 instructions to follow, then if you rerun these, the assumption is, it's going to redo everything.
It's going to re-download the damn image and then it's going to re-execute apt-get install this. It's going to rerun everything. It's going to be take forever. This is such a dumb way to do this. The state of the art is you have a lot of custom caching logic. Like, okay, first you check if you've downloaded the image and if you haven't, then you download it. Like that was the state of the art.
But we just did that automatically. Right? So all you have to do is write the 10 things you want it to do in order. And then Docker would figure out which ones needed to be done again or not. And that required such a change in how you thought about things. That was a painful transition.
So imagine that multiplied by everything containers touch: storage, monitoring, you know…
“What do you mean there might be a thousand more of these little servers you call containers in 10 seconds? But it takes me 10 minutes to manually hook up monitoring for each server. I have 20 servers. That's the number of servers I have. You're telling me I don't know how many servers I'm going to have?”
The biggest challenge was educating on the paradigm shift. After that it was just missing features, competitors, you know…
One way to explain Dagger is it's sort of the continuation of what we were trying to do with Docker. So, a never-ending quest to help developers save time, help you ship faster, which is what good tools should do.
That was the focus at Docker. So much time wasted on things other than building the app and deploying it, and then learning from your users and building again. You should be building the product, talking to users. That's what you should be doing. But you're spending all this time on other tasks that are a waste of your time. So insufficient automation.
The analogy I like to use is like a software factory. You start a project, it's like an artisanal workshop - You're just building stuff and figuring out, you're just winging it and you're moving fast. Then as your project grows, if you're lucky enough to have that problem, then there's more people on the team, more users, more scale, more complexity. And so it becomes really hard to keep that workshop going in an artisanal way. You need to start streamlining and automating. So whether you like it or not, the workshop becomes a factory. It becomes more automated, more standardized.
How you define that factory kind of defines how fast you're going to be able to go. The cost right now when you automate is that you have to lock everything down, you have to kind of define a monolithic standard. So you have to say this is the build tool we're going to use, this is the language stack, this is the CI. And you just sort of say this is how we're going to keep shipping fast at scale. But then you lose agility because you can't easily adapt, right?
Then over time what happens is (if you keep growing) you're going to acquire a company, you're going to build a new feature that requires a completely different set of tools, like an AI feature. Things will happen where your monolithic standard doesn't work anymore for everything. So you're going to have competing standards in the factory. So you have this more messy kind of organically growing factory and so that's how you get agility but now things slow down, things are complicated again.
This is a universal problem. This process of the software factory appearing and then sort of becoming more complex and things gradually slowing down. Today, the state of the art for solving this is that you just throw money at the problem by hiring smart engineers to keep it running anyway. And also throwing bigger and bigger servers at it to run the pipelines as they slow down. So that's kind of the state of the art.
Dagger's goal is to help fix that by making all those pipelines that build and test and deploy your app just way simpler and more modular so that you can still have your standard, but it's a modular standard.
So it's like a factory made of Lego. You have a standard. There's one way to do things. But at any time you can swap components out. You can customize and you can keep the factory kind of evolving alongside the product.
It really all boils down to velocity. You want speed and agility even as you grow. And that's just impossibly hard to get right now. Especially now that capital is more scarce. Most companies can't afford to hire 20 or 50 or 100 of the world's best SREs and throw them at the problem. You know, build us a better platform. You just can't, it's not cost efficient.
So that's kind of the high-level, lofty story here. We make your software factories better by making them more modular, like Lego. In practice, the less hand-wavy engineering answer is Dagger is an open source engine that runs your pipelines - build, test, deployment, whatever, a pipeline's a pipeline - and it runs them in containers. And it runs them in an API first way. So there's an API for all of it. The API has SDKs in several languages. You can drive that API from the command line. And it all runs locally as well as in CI.
That part's key because now you have an engine that lets you program your pipelines, run them in containers, and run them locally, which means that you can iterate on the pipeline itself as quickly as you can iterate on the app that it's going to ship.
Our typical user is like a DevOps engineer, an SRE, whoever's in charge of CI. And their biggest problem is that they have to go through what we call “Push and Pray.”
I was in Ops, I was like I am going to be a software developer. I changed careers and then like six months into my first job, my boss… it was when AWS was first rolling out… he's like, you're going to work with the Ops team to figure out how to take all these developers and all these Ops people and do some cloud stuff. And I was like, “That's not what I signed up for.”
I feel like there was this moment where (and this moment I guess still exists)...
Some people are probably going to hate what I'm about to say, but I'm going to go for it anyway.
I love pre-commit, the tool pre-commit. I don't know if it was around at the time that GitHub Actions came out, but it really felt like people moved away from it. I remember the big appeal of it early on was, especially going back to pre-GitHub Actions, pre-having decent CI options - Sorry, CI companies, you're all decent.
It was like I can't do this one thing in CircleCI or whatever. We've got a pre-commit that does our formatting or linting or whatever. But then it just became like, why isn't all of that just happening in CI? And so that's like, yeah, just kind of have consecutive actions now. And now it's like, you'll push something a lot of times, then the build will fail and you're like, “Agh, it's the linter. I didn't go format. If I had pre-commit installed, I would.” But I feel like a lot of people have moved away from using pre-commit because they're like, “Well, I'm split-brained. Half my CI is here, half my CI is there.”
I feel like with a CI that can actually run locally and save us that ‘push and pray’ problem, we can also start to get to a point where… I mean, despite how much faster Dagger makes the builds, the speed impact of not doing that extra work because you push something that was silly… pre-commit would have caught it or Dagger would have caught it. Like it was a format, the code works, the tests pass. Everything's great. Hell, it even built the image. But it wasn't formatted properly so it bailed, right?
I feel like a lot of that has gone away. What's funny about it is, I feel like every time I see benchmarks on GitHub Actions it's like, all of my CI stuff actually runs much faster if you execute those commands locally because I have so many more cores.
Yeah, I think so. It was a learning experience for me for sure. You mentioned earlier that it was like every founder's dream what happened in Docker, and I said well in some ways and in other ways not so much because there was so much interest so quickly. We became so disruptive so quickly… including to some very large tech companies that became threatened by what we were doing… it was a mad scramble to compete and get a slice of that pie.
Disruption and impact and tech means, okay, someone's going to make money, someone's going to lose money - if there's a business involved. So if you're making money selling an operating system or a data center OS or a storage solution, like in some way you're impacted by this shift to containers. And so you're going to make damn sure you're impacted in the right way.
The reason this is relevant to your question in open source is we were very naive… arrogant in some ways, like, “Yeah, of course our design's better. Like, what do you think? Like containers are awesome, you'll see.”
You know, that arrogance, in some ways, served us because we dared to do something completely different and ignore advice to just stick to what existed. But we were also naive in that we assumed since so many people in the community clearly loved the product and were appreciative of the fact that it was all open source (we were just opening it all)... We assumed the love would continue and everyone would continue to love us no matter what.
Also it became really important for us that everyone loves us. So as engineers, we started caring too much about the opinion of every other engineer. The opinion they have of us, our work, the aesthetics of it, the ethics of it, the morality of it, whatever.
The problem is, if you grow enough and you have enough impact, it's impossible for everyone to love you. People have a range of opinions and experiences. You're just going to have haters, first of all. And then you're going to have a lot of disagreements on how open should it be? You know, should it be controlled by one company?
On top of that, because of the business competition, the competitive dynamics, that open discourse is going to be poisoned a little bit by bad-faith participants. Specifically, competitors speaking on behalf of the open source community to try and steer the perceived opinion of the community towards their business interests.
