Dramatically. The similarities and information I've gotten from teaching just really taught me so much about how to be both a good leader and like a good platform engineer.
I went out of college thinking like chemistry is the way it is. Don't go into computer science. You don't want to do the thing you love, you might just hate doing it as a job. So I was like, sweet, do chemistry. And then I was like, teaching… you know I've always been pretty good at teaching… so I was like, sweet, let's give this a shot. So I taught high school for a while and college and it was amazing.
Inviting people into a world and a domain that you love is really exciting and brings a joy to my world that I really… I kind of thirst for. And that's why, when you talk about how does this make a good manager or platform engineer, you know, it's either customers or the people that I'm trying to empower as my team, to say like, listen, I want to help you out. I'm here to set you up for success and get you to be very, very successful here. And what can I do to bring you into that pit of success?
Teaching, it's a lot about that. So the skills that I get about making sure we're building the right thing for the customer comes from teaching. Where you're like okay if someone doesn't want to learn a subject, like why? What do you not like? What is not interesting you here? And every person's a little different, so you have to really like figure it out and understand. You have like your main path, like the class itself. And you're like okay, well generally these are the reasons that people don't like chemistry. It's just kind of boring, dry– I can solve that.
And you have the edge cases, right? Well, there are some students who just don't want to learn even after that. And I really blew something up for them, and they still don't want to learn chemistry. Like, what? You're driving for the next “What's wrong?”, “Why?”
That translates into the platform engineering–whether it be a DevEx team, a tooling team, whatever you're calling your platform team–what your team's trying to do is it's serving your product engineers.
Some of them have some pretty easy ones: “Be really nice if I had a great GitHub action flow” or “I wish I could just get an easy way into Kubernetes”. Cool, I mean that makes sense. But then there are other people who have lots of other issues that are like not in the mainstream, and so you have to then search them out, find them where they are, and then like bring that back. And then say okay what are we going to do to like bring them into it.
It's a lot of the same skill sets about teaching that go into both the managing and, what this podcast is about, the platform engineering. It's a lot of that.
We could dig into this question a lot because there's been stops, starts and continues along the way. In Hacker News, it's always fun. It's seeing how people model our architecture in Hacker News or on LinkedIn. I'm like, you know, I don’t know. I saw something earlier today, it was like you're cacheless, you don't use Redis. I'm like, we leverage Redis actually pretty heavy here. But anyway, there are a ton of blog posts and it's something we're really proud of and I'm proud of. I wouldn't be here today if the people who started this company didn't start with the monolith.
I think that was an amazing decision. I think what they've done to it to optimize it is fantastic. And it's a really, really slim and cool and highly optimized model. That's huge! The fact that they were to do this on nine web servers is insane to me. I think it is just a testament to what you can do if you buckle down and do things and just try to heavily opinionate and optimize. And so that's awesome.
The thing is though, like, when we're thinking about going to the cloud, well why would we want to go to the cloud? Well, we already did that with our SaaS product, our Stack Overflow Teams product. It got lifted into Azure. There's some blog posts on that as well.
That was not easy. That was mainly for a compliance reason, like it was a good way to add value to the business. But for the public platform, like it's a harder sell. Well, the decision, you know, mostly came down to the idea that, well, we have a data center in New York and that data center is closing. So we could do a few options. We could find a new data center. We could probably just lease equipment from somebody (and say, hey just you take care of the equipment part, we don't want to be in that business anymore) and run it.
When we took a look at those options and then compared it to like a cloud migration, it definitely made more sense just to go to cloud. It made sense from like a money point of view for us. Like if we could just go to the cloud, if we look at the estimates, it's going to help us kind of lower those costs and what we're maintaining two data centers right now, because we run like an active passive kind of mindset here. If Stack Overflow goes down, there's usually a read-only mode that you go into, and you're in a different data center then.
That's a yeah, that's that was definitely a big pivotal moment. I would probably take a step back first though and say that like around about a year and a couple of months ago… last year was pretty rough for a lot of companies, and so we had also gone through a layoff and it was hard.
But during that time, both before and after, there was a partnership made between Reliability Engineering at our company and my team. I had become manager at that point, and we kind of got together and said we need to find a way to make this work. What we tried before didn't work.
So (I told you before about some of the services that we had), what we did is we said, okay, first we just need to know can we even do this. Like you said, Stack Overflow has been on metal for so long, like, how do we take a service and put it into the cloud, right? Most of us know how, but like here, here, how do we do that?
So we took two services. There's one called Traduciere, which is a translation system, a community driven translation system. And IDP, our identity provider… I wouldn't say old school, I don't want to offend anybody… the legacy one that works great, and we still use today. How do we get them into the cloud? So my team, like I said, we partnered with them and said, let's just try this. We did it, we got them up there, and we learned a lot.
And then it comes to what you're asking. We had that learning. So how do we take this learning and do something with it? Because we’d now seen the pattern here at Stack Overflow of how this works. We needed to know our patterns, not someone else's patterns, we need to know what we need to do.
And that's when a team that wanted to deploy a new service… they were people who were kind of influential a little bit, a team that had high output. People had a lot of regard for what they would do. They kind of were innovators, right? They were what I would say when we talk product, they were early, early adopters. They were seeking the newest solution. AI had just come out and their thing was like, okay, let's try something new, we have space, let's try something new.
That's when I said, all right, you want to try something new? I have a team member that we'd be happy to lend over to you and embed, and we'll have Reliability Engineering support us for any infrastructure needs to understand what does that take to do what you're doing. And that was a pivotal moment. Embedding into that team and getting that team enabled because they wanted to be, just changed everything.
We learned a lot, we got a service out for them, we brought learning back to us and everybody was happy. We got the product done. We got engineers on that team interested. And we got learnings. It was just like win, win, win everywhere. And that's when the ball started really rolling.
Yes. The answer is yes to all of that.
So yeah, if starting platforms is hard and when you get a little bit of like evangelism and people are happy with it, you take everything you can get and you try to do more. So that team was very, very proud of what they did. And they did a lot of talking. If I remember correctly, they even did a demo or two explaining how it worked. And they even talked to our product managers to say, look how fast this can be done and look what we can do. I mean, not fast originally, because that's part of the issue at the beginning is you're learning, and it takes a little longer. But, you know, look how iterations can go faster. And so they started talking a lot themselves.
And that gave us like, okay, we have the spark, don't lose it this time. So that then is when my brain started going with the team, okay folks, how are we going to do this? How are we going to not lose this momentum?
And that's when I'd say the biggest thing is going back to the standard kind of product style. Marketing has to go into your big thing. Shine a light on the team. Shine a light on everything that they're doing well. Have my team, have a light shine on there so its people are proud and excited and like, look we did this. That Marketing–never stop.
I mean people probably get tired of me marketing when I'm trying to do this. So, you know, we'll do things like office hours every other week. We have a steering committee that meets to discuss what we should be doing and what the problems are right now.
Side note–any good team that you're going to do marketing with needs a good name and a good logo. One of my team members, out of hand one day, said, “Hey this is like free refills at a soda machine, right? You want a service, just go up and just refill, and you get a new one.” And we're like I first were like… at first we were like hahaha. Is that the name? And we're like, sure, that's the name. So we have this big cup logo with eyes and stuff. It's great, everybody loves it. We have a Slack emoji. We react to things with the free refills.
I wish embedding was more during the time period from then and in between. We'd have people like liaison, I'd say would be kind of the max. But, like I was talking about earlier, we were really slim on people, right? So embedding comes with a high cost too, right? Remember, I told you about these other services we have to handle too. And it's a balance to handle those things and that, but we did a pretty good job of that.
So then it became like let's build what the base product is that we're trying to do. Because now we can't just keep embedding right off the bat, we need to find a way to self-serve. And that's when it led to, let's document what we did. Let's get a process system set out. And then once we have documentation, highlight the points that were really the pain and automate those.
And that's kind of what took the next six months or so to create a CLI. Test it out. We'd have a few other teams do some exploring because there was some word of mouth going out. And we were loud about what we were doing to some degree saying, “Hey, come do this. It's really fun.” And that time was a big innovation time for us.
Embedding is starting back up again, actually. We found some space to do this. The services I was talking about, we're finding ways to pragmatically chunk them down and have them take less of our time. We also are partnering with another team.
We have a team called Stacks. They do the Stack Overflow design system. You can actually do StackOverflow.design and you can download it, it’s open source. They're responsible for the Chrome look and feel, but they also do a lot of UI enablement too. They actually are the ones who enabled us to do Svelte and Islands in our monolithic code base. And so they've always been interested.
One of the problems you're going to run into is definitely UI, right? How do you get these you UIs in there? So we're partnering with them. Actually as of like… next week or technically it might even have started… by the time this comes out we will definitely have started. There's a team that wants to do something very unique with the UI, like embed a hot path UI. And so we are embedding over there to try that out and see what that looks like, and bring that information back as well.
So we see the value in it, for sure. It's definitely been the most valuable, but you have to place it with a smaller team. You have to place it appropriately so you're not spending that cash all the time, that money in that way all the time.
Please do not. They are amazing people.
I have four software engineers right now that have worked in some version of platform engineering somewhere or they've had to do things themselves somewhere else. And they have already some good soft skills and willingness to talk to engineers. And that right there is already coming over some of the stumbling block. Because you have to go out and talk to these other engineers and understand what their problems are.
So there'd be a lot of times where, in my one-on-ones, I would talk with them and say, I need someone to facilitate this meeting, can you facilitate this meeting. And if you have issues like what can we do, what is worrying you, like why being in front of people worries you. And doing a lot of coaching. Like, let's try it, see what happens, and then if you don't like it we'll come back and figure it out again.
We'll get those soft skills to a point of… you don't have to be amazing, you’ve just got to be able to do it okay. And doing okay is all we need. I would say they do it really well, a lot of them. And that's really good. And that is going to be someone's toughest part. I think you need individuals on platform engineering that are more than just the best coder, best reliability engineer, SRE. You need those, it's like any composition of a team, you need the right T shapes and then you want to make that comb shape for the team.
But you definitely have to have people that if they don't have the soft skills, they're willing to develop them. And it's almost like developing TPMs out of them, right? Like I need you to become a TPM and that's really hard because that, for some people, can be a very scary word. I don't want to be a TPM. I don't need you to be a TPM, but I need to have qualities and put a hat on that says I can take this here and do it.
To like maybe put a final thought in on it. Finding someone in your org that probably has some of those skills and is probably good at them is a great idea. I always try to empower my people and one of my people, who actually came from the Stacks team, he is great at taking outcomes, boiling them down to like what are the opportunities, and then finding the problems. He can do that through frequent contacts, frequent talking. And then he mentors people when he does all this. Taking that person and saying okay… again shining that light. Showing this person's doing amazing. And that will pull other people to say, “That person is doing good stuff. What can I do to be a part of that.” because people like to be part of good change.
Yeah, I think that one's pretty interesting because I feel like a few years ago in some orgs it was like this team manages infrastructure and you all do your apps. There are a lot of orgs still like this today. Like if I need a database or a queue, I have to go open a ticket, change management, GitHub or Jira or whatever, tap somebody, sneak a Slack message out there and be like, “Hey. Can I get an SQS queue?” Or GCP pub sub queue or something.
But the funny thing is like when we think about our apps… like where the cloud started… our apps ran on the cloud. It was like, I got a classic VPC from AWS and I got an EC2 instance and I put my app on it and it runs. But now like our apps are the cloud. Like my app needs cloud services for it to function, right? And that shift of like, I'm using cues, I'm using step functions, I'm using object stores… like these things are very specific to my app, not how we run applications. I need that control.
That grinds teams to a halt, if all infrastructure provisioning has to go through this infrastructure team. Like go tap one of those people if you need a queue. And like being able to extend the ability for a developer to get certain cloud services when they need it (with your compliance and all your practices in place), like that right there is, I feel, one of the Holy grails in this shift towards self-service and platform engineering. Letting people just grab that stuff because it's...
I feel like we've just gotten in this weird spot where it's like, but that's infrastructure. That's not theirs. And it's like, no, it is. I can brew install that stuff locally. Like if I can brew install it, I should be able to make it myself there. The catches is like, are my guardrails in place? Now that's the part that freaks all those Ops folk out. They didn't copy the OPA policies, right?
How do we get all that stuff in place and let them run because that's what they need. Our apps are getting to be less code as we're starting to buy into some of these commodity services. So let the developers get it, otherwise, if they don't, they're going to code around you or they're going to ClickOps it… beware Ops folk.
I 100% agree. One of the things we talk about a lot in enabling platform is ownership, and ownership of lock, stock, and barrel of everything. That to me goes all the way to the costs.
I know in my organization, and every organization I think I've been part of (I just think it should change), is that I'm sitting here working on my code, and then someone taps me on the shoulder and says you know that Kubernetes instance is costing us $100,000 a year? You're like, “Whoops. How was I supposed to know? I didn't know.” And I think if you can expose that, like a metric right up front, that has such a power to it. Because now it's not just about how do I deploy the coolest engineering thing that delivers my product value, but also like how can I do this by minimizing that number?
Come on, we all like gamifying, right? I got my instance down to bucks as opposed to, you know, hundreds of dollars.
And it doesn't matter what org you're in, right? If I'm in a startup and my bill can only be X thousands of dollars and maybe even smaller, costs matter a lot there. And then if you're a big FinTech company, costs matter there too because everything's a budget. And it matters, it needs to be streamlined and understood. And even for us, we have the same issues, right? We're always looking at, do we really need this integration? Do we really need to have these on as VMs? Can we do Kubernetes here?
Honestly, we haven't talked much about it because it's AI. Ppeople are going to tell me AI is coming to platform too. But as we deploy more AI resources, anybody does, those costs are just going to go out of this world. And if your system's like, I'll make 20 LLM calls just to do this one request, to give this person a good recipe. I mean you could do that, but you could use Elasticsearch and then just have a good re-ranker on it and it’ll cost you like 1/100th the cost.
