Yeah, that's such a good, that's such a good point.
The trade offs are pretty asymmetric when it comes to systems administration, SRE Ops, and any system that's really in charge of controlling like the operationalization of software. So you know, software goes through multiple stages. You know, you probably know this really well where you're... at some point it's just a little nascent thing on your computer where it's pretty contained and you have a pretty high tolerance for, you know, hallucinations or anything that's not quite working well.
And then as it goes through the stages and you know, as it goes through CI/CD and then gets merged and then it gets deployed when it's actually the same software that's now running on production hardware where it's like servicing, you know, customers, in our case it's maybe processing hundreds of terabytes of data every single day. Right. So the stakes are really high and you don't really want anything messing that up, right?
So AI can make you a little bit more productive, but then if it gets something wrong, there's going to be a pretty steep downside. Right.
So most of our use cases and most of the users that use or in these sort of roles tend to be optimizing for, "Hey, let's not drop data, let's not lose production data, let's not lose uptime," because there's nothing more important in that role. They're not necessarily optimizing for, "How productive can I be?"
It was like, well, if I drop transaction data, that's going to bring the business down. And it's a very different modality of applying AI to those sort of problems.
And we have, over the last three or four years, we have perfected a set of techniques that have enabled us to do that thing exactly right. I mean, there were so many things.
One thing is you don't, you try to minimize the stuff that you're using AI for and surround it with a lot of harness. So things that, you know, deterministically check the output.
We also just break it up into really small chunks as opposed to one shotting an entire file or you know, like an entire pipeline. Right. You just break it up into small functions and you know, let's do it one by one.
You take that to the LLM, you see what you get back from the LLM. You validate that you, you run it and then you insert it into the stack. Just giving an example, but really just breaking things up into small pieces.
Lots of guardrails, lots of harness that's written to validate the output, to make sure that it's performant and it doesn't have any security vulnerabilities, all those sort of things, and then stitching it all together in a deterministic way.
And that has actually been a pretty cool unlock in that we have been now able to use AI and agentic user interface paradigms to produce code or to produce configuration objects that are very close to being deterministic. So if you wanted to convert A to B, you could write it by hand or you could use one of these agentic experiences that we have within the product.
And it will do a very good job of it, not just once, but repeatedly. Like if you do it, if you, if you, if you try to do it multiple times, you're going to get the same output.
Yeah, I think it's a mix of both. We have certainly felt... I mean, I'm actively hiring right now, I'll even make a little plug for it right now. If you're interested, please reach out. We are always hiring top quality engineers and candidates.
But the job or the role or the set of tasks, if I'm talking about that a software engineer was responsible for, is evolving. A typical software engineer would spend a certain fraction of their time writing code and a lot more time thinking about the problem, thinking about how that problem fits in or how their solution fits in to other systems that already exist within an enterprise. What are the trade offs and all those sort of things. I feel like that's still around.
It's the code authoring part that has been accelerated with the existing set of agentic software that we have like Claude Code and Cursor and everything else. And a new task family has expanded on an engineer's set of things to do, which is verification and validation.
Right now you have AI generating so much code and there's so many things that are, you know, engineered. Everyone is like suddenly 10x more productive. So you have a lot more code being generated. Well, who's going to validate that?
Who's going to verify that, that it's going to work, you know, just the way you intend for it to work in a hardcore production setting, you know, where there are millions or billions at stake.
The way the approach that we have taken is we have started to develop tooling that helps human software engineers to do that a little bit better, do that a little bit faster, right? To make that verification, validation a little bit easier.
That's like most of the job now, right?
I was talking to one of my friends the other day and it's like... yeah, I feel like our job for the longest time was, you know, we get a task assigned to us by a product manager, from a Scrum Master, whoever... and we sit and we think long and hard about how all these little pieces plug into a library or a standard library or an API.
And now the job is like that stuff... which I don't know, personally, I like coding, but I didn't... I've never really found the rewarding part of coding, like figuring out all the things that I need. It's like, it's the elegance of like the solution and the experience afterwards, right? And like, that's the whole job now, right?
Like that's... your stack of code is not the thing that's going to be your moat anymore. It's this experience and joy of the user afterwards of what you've created.
And I feel like a lot of the job now is just like verifying and making sure that this is good quality functionality. It's something that I feel like we're gonna have a problem with this in the space in the near future, because I feel like that intuition comes from years of production experience... like being able to look at things and understand how the code all goes together.
I feel like we're gonna have a gap in juniors in the near future of like, people that don't know how to do that anymore because they haven't written production code.
But like, I feel like a lot of the job now is gonna be guardrail engineering, observability engineering, quality engineering, less so lines of code engineering. So I would absolutely love to hear what you guys are doing there.
That's a pretty good workflow, you know, just using speech. I should give that a shot.
But if you think about the early days of vibe coding, when it was... not that long ago, it feels like long ago or like some time has passed, but maybe it was like a year ago or something... but when it was first crystallizing, it would generate everything. Right?
I mean, it wouldn't use something like React, it wouldn't use something like even Express. It would just write raw JavaScript or it would just write raw language without using any libraries, and it would still work. You know, like, it would still look like a little bespoke application. And then, like, if you try any vibe coding platform... you know, there's so many of these platforms these days... or even if you just use Claude, what it does now is it just uses React, it uses Tailwind, it uses Next.js, right?
So it's almost injecting all of these building blocks into the vibe coding thing and creating an app that kind of looks pretty robust and pretty modern. And, you know, it doesn't have as many failure modes as an app that was perhaps vibe coded a year ago.
And those building blocks are off the shelf building blocks. But when I think about platform engineering, I've been encouraging my teams to start producing those building blocks.
Like what are unique building blocks that a vibe coding harness pull from and insert into applications as your product teams are now building these applications. Right. So maybe it's a bespoke component library. Maybe it's a set of skills, you know, that, that have just... It's like it's a markdown file. A lot of engineering now has just come down to being able to clearly articulate yourself in English, which goes into, you know, a markdown file.
And what are those skills that you can produce as a platform engineering organization that can be used by vibe coding harnesses and pull in those dependencies as building blocks?
I'd be curious to get your take on that.
I can acknowledge that is what I intended. I'll put a comment or something if I wanted to switch.
But the thing that's fundamentally changed about my development is I'm very good at documenting things now, which I have just never been good at. And I think a lot of engineers, like, they don't like... engineers like to write code.
You still see so many engineers that don't like writing tests, like, people are like, allergic to it. It's like, "Oh, I hate writing tests." That to me has always been like the joyous part. It's like, "Oh, I've expressed my intent." Yeah, it's like I've expressed my intent and it works. This is exactly what I'm looking for.
But I've never been great at docs. It's like, "Hey, it works. Get this fucker out the door." And now I spend so much time on architecture decision records. Why is this the way it is? What impact from customers or our production infrastructure has caused us to think about solving this problem this way? And then it's like locked down in code. Right?
And that's one of those things. It's like, I feel like many teams that really leaned into things like ADRs and deep detailed specs on how your V2 API works, the constraints and things you put into it. Those are things that really established teams with plenty of time to do so do. And then other teams don't. Right?
Now, that is again, part of my job, and what's cool about this is it helps the LLMs quite a bit. And the humans, turns out they love documentation that's accurate too. Right?
And it's just like, that has always been like such a hard part to do, especially in startups where you're just constantly running and running and running, like moving on to the next MVP that a customer wants. It's so hard to go back and document the last thing.
But, like, again, like, that's the job now. It's this, like, quality engineering is kind of what I see? It's just like the code itself is just... it's... I don't care, like, if it rewrote it in Python tomorrow, I do not care so long as my quality engineering on the outside can tell you this is how the API works and I can guarantee it works. Doesn't matter to me what's inside the box.
I'd love to get your take on this because I've heard a few people start to say this... I think it was the CEO of Chainguard recently wrote a blog post about like open source is dead, it just doesn't know it yet... and I feel like I don't necessarily believe that, but I've got this gut feeling that it might be true.
And so it's like for an example, I would never say, like, "Hey, let's spin up a new React." It's good, there's a really good ecosystem, but at the same time there's a lot of vulnerabilities that are getting snuck in through like a dependency of a dependency of a dependency. While at the same time, like in our own products, like we get to things where we're like, "Hey, we're adding skim integration" and it's just like, that's important to us, we will actually take the standard and we will generate our own library that we can just own. It's so critical to... it's external system managing permissions. I want to own all of that code. It's not something I ever would have wanted to own in a world two or three years ago because... I've got to focus on my product. But now it's like, this thing is really important to us and microsecurity, so I want to understand how it works and I want to make sure that we have full control over its stack of dependencies.
So yeah, so I don't know, man, it just feels like a lot of it is good engineering principles that we've always had that we've never had the time for are the work now. And that's something that will benefit the LLMs and it's definitely going to benefit the person that replaces me, right? Or the person that's joining my team that I don't have time to go through everything with.
It's like, "Hey, here's an entire markdown directory of decision records of how we came to the architecture that we have and the tools that we have, and which libraries we do and don't use, and how we decide on when or when not to use a library versus just generating and owning something ourselves."
Yeah, that's such a good point. Right?
I mean, let me first talk about TDD and you know how much I love TDD. My first job was ThoughtWorks and you know we were like... ThoughtWorks was a big proponent of test driven development. And you know, it was just like the greats like Martin Fowler and you know, like a whole... some industry stalwarts, I mean who were part of that movement. So that's really how I cut my teeth. And it's the same thing that we see today, right? It's the spec first, right?
It's being really thoughtful about what is it that you want to build and being able to express the constraints of the product in a tight deterministic way. I mean that's test, right? And you write the test, first it fails and then you write the code that fulfills that and just that and nothing more. And now that test is, you know, it's either a spoken record in your case or you know, it's a detailed spec markdown file that you know, describes exactly how the software is supposed to be... exactly how the system is supposed to behave.
Which can be used for two things, right? One is for using it to generate the code that implements that behavior and then more importantly the code that validates that behavior. Right? And you can also think about splitting your tests in the same way that you split training data sets when you're training models, which is you know, you have 80% and then you withhold 20% and use that for validation to make sure that your spec, you know, your model is not overfitting on the 80%. Right?
So you could also do... you could also split tests up in that way where you can have, you know, certain part of the spec which is hidden from the LLM when it's generating the code. And then it's only used for validation to make sure that it's validating the spec. Right? So there's a pretty interesting way to structure a smaller spec which is not handed to the implementation aspect of it, which is a pretty cool pack that we've used.
When it comes to supply chain attacks that we have seen, you know, in the last few months. I mean, you know, they've been rising at a pretty exponential rate. And so, yeah, I do get that point. You know, open source is unfortunately in a tricky, tricky spot. And my take is that it's going to evolve from open source implementations into open source standards.
So there's going to be a lot more standardization where you pull the standard from, you know, some sort of a GitHub or an open source repository. And the standard again has validations and tests and all those sort of things. And then the implementation, the actual production code that runs in the hot path in your system is generated by an AI that you have or your team of engineers.
And then the open source is literally just a spec and the set of tests that tell you that your implementation is watertight and that way you can defend your supply chain from any, you know... because unfortunately the reality is untrusted code execution is the biggest threat everywhere right now, especially with vibe coding.
Be that a supply chain infiltration or you know, be that a vibe coding surface in your product where you're allowing customers to extend the product by, you know, giving this like, "Oh, enter a prompt here, now generate something." It's all untrusted code. And I feel like it's... my take is that the way to reign in that untrusted code is by tests and it's by specs.
One interesting thing, you know, I was talking to somebody who works at a large defense tech company that we all know of and one of the things that he was talking about it... you know, he works at Frontier AI, in their world... and they have a really sophisticated real world simulation. So it's like a real world simulation that's built on like a game engine that, you know, allows them to, you know, really test how things work in a versatile setting without really having to engineer anything. I mean, like, that's a really cool way to validate software, you know, as well, like, which is not... and again, in defense tech, you're producing real world things. So your simulation is really a physics engine. And now the laws of physics are, you know, pretty well known and, you know, well written down for a long time now. So it's almost easier to build a physics engine.
But what does it mean to build a physics engine for a platform engineering team within their own domain? You know, let's say you're a logistics company or, you know, if you're an observability company. Like, if I had to like break down some laws of physics in the observability space, you know, when you have network flow logs that are going from one place to another, there's a source IP address, there's a destination IP address, and then there is the size of the packet, right?
Yeah.
One of the things, like kind of going back to you were saying earlier about like, just like the different, like smaller tools that you kind of put together. When I talk to my LLM to do my tests... and for anybody who doesn't like TDD, you're going to recoil when you hear what I'm about to say that I do, but I swear it's the thing that helps us move so quick... I speak to it in plain English because I speak to it in like BDD, like Gherkin, like, "Given this, win this."
And our tests... our test suite has skewed that direction. And the thing that's interesting about it is like we now have these very like, factual statements, right? It's like, "When there is a cube set of deployments," it's a very English phrase that we use around the org all the time, like it's a sentence that even the sales guy understands what I'm saying. And so the interesting thing is it's so easy to reason about now because it's like it's a plain English sentence and that ties to a tiny function that does that exact thing, right?
And so now when I'm talking to my LLM, it's like a lot of the tests, it doesn't even have to write because I already have stuff tied to each of the given wins thens, right? So I get to speak in plain English. And even our sales guy can write like, "Hey, you know, I was talking to the customer and they said they need this."
It's like, "Oh, yeah, well, you can talk to this thing in plain English and it will start building features for you." And the crazy part is like seventy, eighty, ninety percent of the test suite at any point in time is already written because we have all these common phrases.
So it's coming to the point where it's like, "Hey, let's talk about doing some flow log stuff." It's like, oh, we just have to add a rule in plain English and a little function and it'll write that function for us, and it's so easy to reason about. And I see these people were it's like, "Oh, I've got a 10,000 line PR."
It's like, I've got 10,000 three line PRs. And it's very easy to reason about where almost anybody in the org can look at it and go, "Oh, we've added a new rule that does this." I can see the code that it produces for the test. And like, now there's some logic that's going to fulfill that, right?
I don't know, like, that feels like a really good flow. Again, I know that people that don't like TDD are like, "Ugh, cucumber. Like, I don't want Cucumber and Gherkin back in here." But it's so easy to develop this way, I cannot imagine going back.
And like, folks might be hearing like, "Oh, you're not developing anymore though." It's like, "No, I'm doing more architecture and thought about software than I ever have."
Like the last part of the TDD, right? It's red, green, refactor. Most people don't get to the refactor part. Like, how do you get your PM to give you time to refactor your code? It's like, it's part of the job. Now it's the job, right? It's like, I see stuff come in and I'm like, "Oh, this all looks like shit."
Like, now I get to think about like, what does a grander, better architecture look like for customers on the API or even internal developers using stuff. Like, how do I build something that works well for them? And it's like I get to focus a lot more time getting rid of debt than I ever have in any product I've ever worked on. It's really, really interesting.
So, coming back to platform engineers, I'd love to get your thoughts on this because I think the most underappreciated part of platform engineering - it's often invisible, right? Even a good DevOps team, good Sysadmins, like when it's working, nobody notices you, right?
Nobody's like running around like high fiving this admin and they're like, "That auto scaler really does its job well." It's like when it doesn't, that's when people recognize you. Right?
And so like in this new era of, you know, a lot of code being generated by agents and vibe coded by end users, like, how does that change what we're responsible for as platform engineers? And stability? Like, we can have a very... and I know it's kind of two phases - is the platform engineering team vibe coding stuff and then is the engineers on top of the platform vibe coding stuff - but I would love to know how you see the world through the lens of a platform engineer, where we're just getting things that may be a lot less stable at first, but it's unstable and coming through at a faster rate... which we can now apply quality to and get it stable.
First off, I mean, with BDD and Cucumber, I mean, you're bringing back some fun memories. Ruby on Rails. Man, that stuff is so well rooted in that... it still is so well rooted in that ecosystem. TypeScript has taken over the world these days, but I do miss using that. I remember the syntax and those sort of things used to be a little bit unwieldy with Cucumber, but that stuff got figured out pretty quickly.
And now, I mean, it's just, you know, using natural language to express yourself and not even bothering with those, you know, those, those syntactical nuances and quirks. And really, I mean, that becomes your spec... the guardrail in which your software has to operate.
And I feel when it comes to platform engineers, you're 100% right. It's pretty asymmetrical. Like when the system is working well, when you're doing a fantastic job, that means nobody's noticing that the system is working well.
And with vibe coding and with AI, that's exactly the thing that platform engineering has to uphold, right? Like can it sustain a ten x or even hundred x or even a thousand x more software running on the platform? So just scaling it up in a reliable way and really supporting these new coding paradigms that are emerging now. Which is, you know... coding factories is like my favorite one, right? I mean, where you now have... or development teams are moving toward a world where instead of writing code...
So, okay, I mean, so if you had to like back up, right, like before twenty twenty-one, developers were writing code character by character, right? One keystroke at a time. And then came AI based ID autocomplete, right? Where you start typing something and the AI was like, "Oh cool, maybe these are the parameters in the function that you wanted." And then came prompt driven agentic decoding, right? Where tools like Cursor and Claude Code became popular, where instead of starting in the IDE, you now start on the right hand panel and you, you know, it's like, "Hey, this is what I wanted to do." And then, you know, it proposes a diff that you then accept or reject.
And now I see more and more teams going toward a world where they're codifying their practices and, you know, the thing they produce in a set of markdown files and then creating a system where you just insert the spec from one end and it produces a fully functional pull request from the other side, right?
I was just like, "I feel like today developing is so much more exhausting than it was a year ago." And I was like, "And it feels weird because I don't write software anymore." I was like, "I'm just reading and processing." And he's like, "That's the scope, dude." And I was like, "What do you mean, the scope?" And the scope is what they call like the little radar thing that they watch. He's like, "That's what the entire job of ATC is like, you do not get to use the flow state part of your brain whatsoever. You're constantly analyzing and like using the executive function side of your brain." And it's literally more exhausting because you don't tend to use that side of your brain a lot. You're using the creative side of your brain, you're thinking through things, right? You're having these deep flow states versus like, "A plane is landing, a plane is landing. Two planes are near each other."
And that's what we're doing with these PRs. It's like, "These two PRs conflict with main, like, what does that mean?"
I'm shifting context constantly and he's like, "The way they solve this in ATC is every two hours you have to take a break." Because your brain like after about two hours, like it just... the executive side just plummets, right? And with all of these changes... it's like the change rate is insane.
A ton of PR is going through and you know, to have people overlooking all of those is hard. It's hard on people, right? And so like, you know, as we think about, "How do we get value out of AI for a business?" Like the AI can't... like the business can't take all that value because it's going to exhaust humans. Right? Humans got to get some value out of it too. Like do our hours change? Do we work less? Do we start to see companies as benefits? Saying, "Hey, engineers work for two hours a day and then they have off until after lunch and they work for two more hours." Right. Like, I think the job's really going to change a lot.
But you know, at the same time with all those PRs coming through, it's like starting to... again building blocks and guardrails. Like maybe every single PR now that you have more time, maybe every single PR does go through k6 and do load testing on it. Maybe every single one does land in a preview environment that has a full suite of browser tests go across it that you've never had the time to do before because you were kind of slaving away over every single save to a database. Right? And that quality work is the thing that every PM wish you had time to do but you never did because they're focusing on revenue, right?
Like that's the job now. And so it will be interesting to see more of that start to land on platform engineers' plates.
That's such a good comparison with ATC. I mean, like that's exactly how I feel when you... it's the shift away from being able to do deep generative work into like more reactive work.
When we talk about, well, AI is going to, you know, ten x or hundred x a team's productivity, ultimately the team is accountable for that ten x output, which means it's going to... from first principles... is going to come at ten x more cognitive load on that team. Because I don't see us as an industry getting to a point where we are okay with shifting that accountability to AI, right?
The team is still... the human team is still going to be accountable for whatever the team produces, right? Whether they use AI or not. So there is a pretty deep cognitive burden I feel when it comes to platform engineering. I mean just inventing tools and techniques that simplify that burden... that reactive burden... and helping, you know, that review process easier.
One of the ideas that we have talked about at Cribl is not all code in your code base is equal. You know, there is certain code that's important and that there's certain code that's maybe not as important. So like really splitting it up or you know, drawing rings around your code like, "Oh, this is the ring 0, the kernel." Like in Cribl's case that would be the score stream processing engine that can process hundreds of terabytes of data or transform, you know, hundreds of terabytes of data in an hour. Like that's the kernel and then the ring around that is maybe like the control plane for the kernel, right?
So what sort of stuff are you putting in? Optimizations, those sort of things, then kind of like expanding that. And then at the very edge you have smaller things like some settings panel hidden in the UI, UX somewhere. So like splitting that between kernel land and user land, kind of how the Linux system does it and then treating different grades of PRs. So (a) like a PR should not span across multiple rings, it should be in its own ring. And then (b) treating those differently. Right?
I mean like if it's a PR that introduces a bug, but then you can... if the bug is caught, like you know, if you... it's easy to remedy that, you know, like you can push out a fix in ten minutes and there would be no impact versus messing something up in like the core. You'll probably have like more human scrutiny.
Hey, and we don't... you don't have to be the guinea pig.
This is the thing that's exciting to me about this that I was like, I feel like all these like AI CEOs, like I feel like they are fucking terrible. They're great at raising money, but they're terrible at marketing.
Like coming out and being like, "Fifty percent of you are going to lose your jobs in the next three years." It's like, "Hey, you know why people throw Molotov cocktails at you? Like, it's because you aren't making people happy with your words."
But it's... at the same time, it's like we've had a few of these... like, I don't want to say revolutions... but these, like, big changes in, like, software engineering over the past, you know, twenty, thirty years. And we're in the next one right now, right?
And I feel like this is the time where it's like, you can get a lot of free labor from these subsidized tokens to make your code bases really good. And then you can think a lot more about what does the future of engineering look like for our industry and what does it look like for our orgs, right? And that to me is exciting. It's like, it's still engineering work. It's headier stuff.
I think it's more interesting problems to solve than save user to database. Like, you've done that at your last seven jobs in different languages. We're at an exciting time.
And I think most people that are afraid of it are afraid of it because of the words that these CEOs are saying, right? And it's like, I think they're saying that because they want to raise money again.
I need more people because I'm making more code, right? It's like, I need more people, right? So I don't know. It's wild times. We can be guinea pigs, but I think a lot of us can also be computer scientists now. Don't be the guinea pig. Be that, right? Be the guinea pig scientist. I don't know. Like, let's figure out how this new world looks because there's a lot of stuff to figure out.
I think CI is going to look a lot different in the next few years. People are starting to talk about... like, I know a lot of people are pointing to GitHub and be like, "GitHub's falling over." It's like, "Dude, GitHub is getting way more commits and repos than they've ever had in their lives. Like, they have a lot of just like, stability to deal with from just like the onslaught of data being thrown their way." Like, you can, you can blame it on core AI all you want, but, like, they're handling more writes than most of us, right?
So it's like the world, our engineering stuff that we have kind of clung onto since the first DevOps talk from Flickr, like CI/CD and Git and all this stuff... It's like, this stuff is likely going to change and it's us who gets to make the decision as to what that looks like.
It's us that has to get familiar with how these systems work to be able to make those decisions. And I feel like if you're sitting around like afraid of LLMs and AI right now, like those decisions are going to be made for you, which sucks.
Yeah. So you know, if you follow me on LinkedIn. Follow me on Twitter. We're pretty active on LinkedIn where we post, you know, all the cool stuff that we are doing at Cribl.
Just this week we released a bunch of benchmarks on LinkedIn where we compared privacy sensitive data scanning models that we build in-house, that we trained in-house, and how they stack up with OpenAI's new privacy filter. So OpenAI released a 1.5-billion parameter open source model on Hugging Face. It's awesome, definitely go check it out.
It can scan and redact sensitive information from any data that you throw at it. And at Cribl, we have our own in-house built models that are way faster than OpenAI's models. Much more accurate as well. So our F1 scores are... which is a score that really tells you about precision and recall when it comes to AI systems, really how accurate a model is... we're able to benchmark our models at more than ninety percent whereas OpenAI's model was about fifty-seven something percent.
And then our model is also a hundred and thirty-three x faster than OpenAI's model because what we have been able to do is we have been able to compress much higher accuracy in a really small model. So you're talking about a hundred million parameter model that can run on a lot of logs, a lot of telemetry data, observability data.
Because guess what? With agentic AI, LLMs are everywhere. And people are sticking all kinds of things into LLMs these days, right? Personally added information, health records, Social Security numbers, phone numbers. And one of the biggest challenge that IT teams and security teams have to conquer is how do you scrub all of that information from access logs, from audit logs... a really pressing problem across the industry.
So I'm really happy that we are leading the charge on that and developing these models that enable you to scrub sensitive information so it doesn't land in places that it shouldn't. You don't get one of those weird letters in your mail saying that, "Oh, there's been a data breach." Or, "You're signing up for one year free of some monitoring..."
