New Blog Post! The Citizen Developer

Read here
Engineering
Push a Module, Upgrade the Fleet

Push a Module, Upgrade the Fleet

How release channels, versioned resource types, and version-aware links roll a module upgrade across every project and environment from one push.

Cory O'Daniel9 min
Share

You find a bug in your Postgres module. Backup retention is set to one day, and it's been that way since you wrote it. The fix is two lines of Terraform and it takes ten minutes.

Then you go look at who runs it. Two hundred databases. Every service has one, most have a copy in dev, staging, and production, and each was stamped out from your module at whatever version happened to be current the week that team onboarded. Some are on 1.0.4. Some are on 1.1.2. Three are on a fork somebody made in 2023 and never mentioned.

The fix took ten minutes. The rollout takes a quarter.

That gap isn't a Terraform problem. It's what happens when infrastructure lives in files. Files don't know about each other. A file can't tell you who consumes it, has no way to check whether the thing on the other end is compatible, and will never upgrade itself. So you get grep, a spreadsheet, and a quarter.

Git is good at history. It'll tell you what a file said last Tuesday and who changed it, but not which two hundred instances depend on that module or which of them can safely take the new version. Those are questions about how things relate to each other, and that's what a database is for.

We built Massdriver on a different premise: your infrastructure is data. The modules, the contracts between them, the instances running in every environment, and the wiring between those instances are rows you can query and constrain. That idea deserves its own post and it's getting one. This post is about what you can do once it's true.

Pin an instance to a channel, not a version

Every instance runs a bundle at a version, and you choose how tightly it tracks the registry.

ChannelResolves to
1.2.3that version, until you say otherwise
~1.2the newest 1.2.x
~1the newest 1.x
latestthe newest stable release
latest+devthe newest release including dev builds

Now push 1.2.7 to the registry. Massdriver finds every instance in the organization whose channel resolves to that new version, moves it, and deploys it. No PR. No ticket. The deployment lands with its message already written: Release channel ~1.2 upgrading to 1.2.7.

One push to the registry moving every database instance whose channel resolves to it, while an instance pinned to an exact version and an instance on an older line stay where they are

checkout asked for exactly 1.2.3, so nothing touches it. billing is on ~1.1, and 1.2.7 falls outside that line, so it stays where it is. Everything else moves, across three projects and three environments, from one mass bundle publish.

The channel is the control. Production can sit on ~1.2 while dev rides latest+dev, and the same push reaches each on the terms it asked for. Nobody has to remember which is which, because the pin lives next to the instance rather than in someone's head.

Ranges resolve at deploy rather than at publish, so a bundle that declares postgres-authentication@~1 picks up a compatible newer version of its dependency without republishing.

Your modules have an API, and it changes

Think about two GitHub Actions. One produces an output and the next consumes it, so you hand the first one's output to the second one's input:

- id: db
  uses: acme/postgres-action@v1
- uses: acme/api-action@v1
  with:
    database: ${{ steps.db.outputs.authentication }}

Now api-action@v2 renames that input from database to primary_database. There's nothing wrong with the rename. It's a better name. But every workflow file that wires those two together is now broken, and the repair is mechanical: find them, edit them, review them, merge them, chase the stragglers. Two hundred repos means two hundred edits, or a codemod and a bulk PR bot and a month of follow up.

That wiring is duplicated because it lives in files. One copy per file, and no copy knows the others exist.

In Massdriver the wiring is a row, and the row carries a version range at each end. The link from your Postgres component into your API component isn't "authentication goes to database". It's "on postgres ~1 into api ~1, authentication goes to database". When api@2 renames the input, you connect the new one once, and that draws a second route: "on postgres ~1 into api ~2, authentication goes to primary_database".

Two hundred workflow files each needing the same edit, against one route drawn once and read by every environment

Both routes live in the project's blueprint. The old one keeps serving everything still on api 1.x. The new one sits there waiting.

So the upgrade stops being a migration. An environment moves its own api version to 2.0.0, and the route for ~2 is already drawn, so the connection re-forms on the renamed input by itself. You don't edit that environment, and the one next to it stays untouched. Change the version, and the wiring follows.

Upgrading your first environment is how you teach the platform to upgrade the rest.

A stack you can hand out

Point the same machinery at a whole stack instead of one module. Build your monitoring stack once as a project: the collector, the dashboards, the alert routing, and the links between them. That project is now a template, and you have three ways to hand it out. You don't have to pick one.

Stamp it out. Clone it per team. Cloning copies every component and every link, version ranges included, into a project that team owns and can evolve on its own.

The clones stay current. They're independent shapes, but the modules inside them aren't. Every component in every copy still sits on a channel. Publish a new version of the collector and it lands in all of them, in whatever environments each team happens to run, without you opening a single one of those projects.

Or share one. Stand up the monitoring stack once and let team stacks reach across project lines to consume its resources, either per instance as a remote reference or for a whole environment as the default of its type. The same check applies at the boundary: the resource has to be the type the slot wants, at a version the slot accepts.

The first two together are the part that's hard to get any other way. A team can rearrange their copy of the stack, add a component, rewire two others, and still be running your current code. You distribute the module and its version. They own the shape.

The contracts are versioned too

This is what makes an automatic rollout safe.

Dependencies and resources are typed. A resource type is its own versioned artifact in the registry, and it's the contract between two provisioning workflows that know nothing about each other. Your Postgres module publishes postgres-authentication@1.0.0. Your API module says it accepts postgres-authentication@~1. Neither module imports the other. Neither team reads the other's code.

At deploy, we check the resource against the slot it's going into: same type, and a version that slot accepts. Choose the wrong one by hand and you hear about it at the moment you choose it.

the resource you selected is "postgres-authentication@2.0.0", but this connection needs a version matching "~1"

That's a type error, caught where you made it. It isn't a plan that dies on apply, and it isn't a null in a module output that surfaces as a 500 in staging on Thursday.

If you've ever chased an Error: Invalid index through a remote state output, you know what this is worth. In Terraform, the contract between two modules is whatever the last person to touch the outputs decided it was. Ours has a version number and something that checks it.

Self service shouldn't stop at day 1

Most self service ends the moment the thing exists. You get a portal that provisions a database, hands you the keys, and walks away, and every upgrade after that lands back on the platform team as toil. Day 2 is where the work actually is: the patches, the version bumps, the renamed inputs, the environments that fell behind. Everything above is our answer to that, and it's most of the reason teams on Massdriver get their operations time back.

What each side gets

Developers stop waiting. You pick a channel and stop thinking about it. Your database keeps up with the patches your platform team ships, and you find out because it happened rather than because a ticket landed in your sprint. Nobody makes you read the Postgres module, and a retention window isn't your problem. Your bundle says it needs a database, and a database shows up.

Operators stop chasing. You ship a module and it lands in every environment that asked for it, on the terms that environment chose. You rename an input and repair the wiring once instead of two hundred times. You stop maintaining the spreadsheet of who runs what, because the platform already knows. "Which instances are still on postgres 1.1" takes one query.

The campaign disappears. Nobody has to cut a branch across two hundred repos or babysit a bulk PR bot until the stragglers are done.

Infrastructure as data

All of it falls out of one decision: put the infrastructure, the contracts between the pieces, and the wiring between the instances into a database with types and versions. After that, the interesting operations are queries.

Find every instance whose channel resolves to the version I just pushed. Check whether this resource satisfies that slot. Give me the route between these two versions. Each of those is one question against data. Against a directory of .tf files and a pile of workflow YAML, each of them is a grep, a guess, and a quarter.

Infrastructure as data, baby.

Stay in the loop