This video, titled "The Twin Mandate" by LeadDev, explains how observability has evolved from basic monitoring to a crucial tool for understanding complex software systems. The speaker, Charity Majors, argues that poor observability is a major roadblock for engineering teams and even hinders AI adoption. She introduces several tests to identify weak observability and proposes a dual mandate: investing in observability to both improve core business differentiators and accelerate internal development processes, advocating for unified data storage over siloed monitoring tools.

Key Takeaways

Observability, celebrating 10 years next year, has evolved from simple monitoring (checking if a system is up, down, or slow) to a sophisticated sense-making apparatus for complex systems.

Companies are spending an obscene amount on observability tools, with some going from $50,000 to $24 million annually due to increasing system complexity.

Poor observability is a significant barrier to achieving business goals, including the effective adoption of AI, as AI amplifies existing system dysfunctions.

Observability acts as the furthest upstream feedback loop, making it the most impactful leverage point for initiating change and improving system efficiency.

You can identify poor observability by asking engineers how they know their code works in production, observing if they rely on specific people instead of tools for answers, and tracking how many bugs are found by customers instead of internally.

Organizations can get stuck in a "doom loop" of firefighting due to poor sense-making capabilities, leading to constant tech debt and hindering progress.

To break the doom loop and achieve exponential returns, observability should be treated as a platform engineering team, run like a product, and viewed as an investment, not a cost center.

The "twin mandate" of observability involves investing in both external aspects (core business differentiators and customer-facing experiences) and internal aspects (accelerating the build pipeline and shipping code faster).

Using siloed "three pillars" (metrics, logs, traces) for observability is becoming outdated; unified storage of structured data is the future, enabling faster feedback loops and better analysis.

Good observability fosters confidence in engineers, allowing them to move swiftly, iterate rapidly, and reconnect with the joy of seeing their code's real-world impact.

0:02 [music]

0:07 [applause]

0:09 All right. So, the material that I'm

0:11 going to be

0:13 trying not to spit into the mic. I do

0:15 get a little excited. The material that

0:17 I'm going to be uh talking to you about

0:18 today is coming directly from the new

0:21 book has not been seen by anyone and I'm

0:23 going to try to desperately to fit it

0:25 into 25 minutes. So, let's go. Um, for

0:30 those of you who don't know me, Charity

0:31 Majors, um, this is the first book. This

0:34 is the second book. And if you happen to

0:37 come for a book signing tomorrow, uh, I

0:40 brought a bunch of sticker sheets so you

0:42 can decorate your little wolfie.

0:47 [applause]

0:50 Can't get that anywhere else.

0:53 [laughter]

0:54 All right. before observabil

0:56 observability will be tw 10 years old uh

0:58 next year by the way. Uh Christine and I

1:01 started Honeycomb the 1st of January

1:04 2016 and we started trying to preach the

1:07 good word of observability a couple

1:09 months after that. And before that many

1:12 many of you may remember this but we

1:14 called it monitoring. We called it

1:16 logging right and monitoring was an

1:19 [snorts] operational tool for

1:21 operational outcomes. Is it up? Is it

1:24 down? Is it slow? Are there errors? That

1:27 was what monitoring was for.

1:30 And observability. In the last book, um,

1:33 we gave it a very technical definition.

1:35 It's about being able to understand any

1:38 state that your systems can get

1:39 themselves into with no prior knowledge,

1:41 without adding any special

1:42 instrumentation to understand the state

1:44 that it's in. Can you slice and dice?

1:46 Can you zoom in? Can you can you treat

1:48 it like business intelligence tools?

1:50 Right?

1:51 uh that is not the definition that we're

1:53 going with this time around. This time

1:55 around we're talking about observability

1:58 as the sensemaking apparatus of complex

2:02 sociotechnical systems.

2:06 Most companies now spend somewhere

2:08 between 15 and 25% of their entire

2:13 infrastructure budget on observability

2:15 tools. Does this blow your mind? Maybe

2:19 not because you're maybe used to signing

2:20 the checks. Um the low bar is 10 to 15%,

2:24 the high bar is 50%, but it is it is it

2:27 is obscene. Gartner did us a favor and

2:30 published a bunch of data earlier this

2:31 year about how much people are spending

2:32 on costs. And they they they talked

2:36 about one representative customer who uh

2:38 in 2009 monitoring they were spending

2:42 $50,000 a year. Same customer in 2024

2:47 was spending $24 million a year on

2:51 observability.

2:53 And what is this driven by? It's driven

2:55 by complexity. Uh you know when when we

2:59 had monitoring, good old monitoring, we

3:00 had the lampstack, right? Remember when

3:03 the hardest part of your day was

3:04 figuring out whether you should use

3:05 whether the P should stand for Python or

3:08 PHP,

3:10 right? And now we've got

3:13 well this might look familiar. Now we've

3:15 got this

3:16 looks a little more terrifying every

3:18 time I look at it. Uh every medi every

3:20 single interaction that you have with

3:22 production is likely mediated by

3:24 observability in some way because you

3:25 can't hold it in your head anymore,

3:27 right? Used to be able to hold it in

3:28 your head and reason about it. All of

3:30 the hard complicated stuff was in the

3:33 app or the database. We don't have the

3:37 app and the database anymore. We've got

3:39 we've got that.

3:42 Um and and a metaphor I like to use is

3:44 finances. Like if you're if you're a new

3:46 grad, your system is pretty simple.

3:49 You're new new grad. One bank account,

3:51 one rent check, you auto deposit your

3:54 paycheck, you set up auto pay for banks,

3:57 check your balance now and then. Divert

3:59 some money to savings or retirement

4:01 accounts.

4:03 That's it. Those are your operational

4:04 metrics, right? That's op that's that's

4:07 monitoring. That's great. Simple, basic,

4:09 cheap, works. Now, let's imagine you're

4:12 a really, really rich person. Ah, okay.

4:14 No, none of us are really, really rich.

4:16 Let's imagine you are a financial

4:17 services company and you're take, you're

4:20 having to keep track of hundreds,

4:22 thousands, millions of accounts with

4:24 different interest rates. There's money

4:26 coming in from a lot of places, going

4:28 out of a lot of places. You've got a lot

4:30 of different assets to manage with

4:32 different appreciation rates or

4:34 depreciation rates. Some of them are

4:36 taxable. Some of their tax write offs,

4:38 some of their tax havens. Oh, and you

4:40 could go to jail if you don't get this

4:41 right. Stakes are high. That's a job for

4:44 observability. You need to be able to

4:45 drill into any transaction at any time

4:48 and get it right. Is it expensive? Yes.

4:52 But it is ultimately a lot more

4:54 expensive not to have it if you need it.

5:00 Um, observability matters to leaders

5:03 because it costs a lot. Yes.

5:07 Um, but perhaps more relevant to the

5:10 people in this room, poor observability

5:13 is very likely one of the maybe the

5:16 biggest things standing between what

5:18 you, your leadership team, your CEO,

5:22 your board, if there anything actually

5:25 cares about, which is

5:29 AI. [laughter]

5:33 I suspected that this was true and then

5:36 God bless the folks at the Dora report

5:37 because they published an amazing 25

5:40 2025 DORA report and they said this like

5:44 five times. They were like AI's primary

5:46 role in software development is that of

5:48 an amplifier. It amplifies what's

5:51 already there. If you're dysfunctional,

5:53 it makes it worse. [laughter]

5:58 >> [gasps]

5:58 >> Ah, they said this again and again. The

6:00 greatest returns on on AI investments

6:03 come not from the tools themselves, but

6:05 from a strategic focus on the underlying

6:08 organizational system. And a lot of

6:11 engineers are rightly a little bit

6:12 cynical about this. Oh, you wouldn't fix

6:14 the system for the humans, but you'll

6:16 fix it for the bots. Take what you can

6:18 get, dude. I I'm not arguing with you.

6:20 There's there's a window here for us to

6:23 fix things, make it better for humans

6:24 and uh and bots and we should do that.

6:28 There's that that word again, systems.

6:31 Successful AI adoption is a systems

6:33 problem, a systems view.

6:37 I honestly people ask me all the time

6:40 for leadership advice and lately I've

6:42 been telling them, how many of you here

6:44 have read this book by Danella Meadows,

6:46 Thinking and Systems? Isn't it good?

6:49 changed the way I I thought I thought

6:51 everything. I I think that, you know, if

6:53 you want to get out, whether you're an

6:54 engineer or a manager, you spend your

6:56 life firefighting,

6:58 if you want to get out of the

6:59 firefighting, you have to start looking

7:01 at the city fire codes,

7:05 right? You have to start looking at what

7:07 is causing these. And yeah, can't praise

7:11 it enough. Anyway,

7:13 the job of any engineering leader is to

7:16 craft sociotechnical systems that

7:17 efficiently convert engineering labor

7:20 into business outcomes. And as we all

7:23 know, the way that we do this is by

7:26 crafting fast feedback loops. Fast

7:28 feedback loops. You know, the most

7:31 famous split brain in history was Devon

7:33 Ops.

7:35 Remember the bad old days when you had

7:37 some people writing the Oh, and QA too.

7:39 some people writing the code, some

7:41 people testing the code, and some people

7:43 operating the code. Didn't work very

7:46 well. Did it wasn't a feedback loop.

7:49 [laughter]

7:50 Any split brain and and you know, I

7:52 think it's really interesting to to

7:53 notice that the complexity of our

7:54 systems

7:57 when the DevOps movement came around and

7:59 started linking up DevOps into a

8:00 feedback loop, complexity started to go

8:02 like that. And so it's like the Devonops

8:05 split brain kept a hard cap on

8:07 complexity for a really long time. Uh

8:12 good bad I don't know interesting.

8:15 Anytime you're in a scenario when X

8:17 causes Y but Y also causes X that's a

8:20 feedback loop. And there are two main

8:23 kinds. Amplifying ones

8:26 and balancing ones. And the amplifying

8:28 ones are the ones we're really going to

8:30 be talking and thinking about today.

8:31 These are the ones where uh

8:35 nonlinear returns, right? The farther

8:38 upstream you get them, like the more

8:41 exponential the returns. And so if

8:43 you're in the state where you're

8:44 firefighting and you're trying to look

8:46 for the point of leverage to get out of

8:50 it,

8:52 watch the fire codes. Observability is

8:56 the feedback loop of feedback loops.

8:59 From a systems perspective,

9:01 observability is the sensemaking

9:03 function which makes it the furthest

9:05 upstream feedback loop of all. This is

9:08 the single highest leverage point from

9:10 which to try and make change.

9:12 Observability can't solve everything,

9:15 but most problems, most transformation

9:18 efforts can't actually be efficiently

9:20 solved without it. And the effectiveness

9:22 of your observability is very likely to

9:25 determine uh whether or not your

9:27 transformation takes months, years or

9:30 never ends.

9:34 It's very hard to see poor

9:36 observability.

9:38 Um you can see the consequences of poor

9:40 observability everywhere, but you can't

9:42 I think of it sometimes as like the dark

9:43 matter of software engineering. You know

9:46 it's there because everything's just

9:48 it's like swimming through molasses,

9:50 right?

9:51 um especially if you've never worked any

9:53 place with good observability, you can

9:55 see its effects everywhere. So I pulled

9:57 together a few tests that you can use.

10:01 Ask any engineer this. Ask a wide range

10:04 of engineer this question. After you

10:06 commit a piece of code, how do you know

10:09 whether or not it's working? If you stop

10:12 there, promise you better say when my

10:14 tests pass, which is why you have to add

10:17 whether or not it's working or how well

10:19 it's working in production.

10:22 If they don't even know how to find out,

10:24 you don't have a feedback loop. And this

10:26 is probably where you should start.

10:29 Uh this doesn't have to be punitive or

10:31 painful or involve waking software

10:33 engineers up in the middle of the night,

10:35 although I recommend it. Uh [laughter]

10:38 it has to be a feedback loop. Uh it has

10:40 to be foundational. This is foundational

10:41 to the rest rest rest rest rest rest

10:42 rest rest rest rest rest rest rest rest

10:42 rest rest rest of it um because the

10:44 question that this test really asks is

10:46 just how much energy prior knowledge and

10:49 creativity do you demand of every single

10:51 engineer who ships code just to

10:53 understand what they've done feedback

10:56 loops are so powerful in helping people

10:58 adjust their behavior when engineers can

11:00 see the impact of their changes both

11:02 positive and negative within minutes

11:05 they they learn rapidly they develop

11:07 their intuition their their gut feeling

11:09 about what works They catch mistakes

11:11 early. They see its consequences on

11:14 their users. Engineer, we're we're

11:16 curious. You have to beat curiosity out

11:19 of an engineer. And sadly, a lot of

11:21 engineers have had that curiosity beaten

11:23 out of them because they can never

11:24 figure out what's happening. Um, but

11:27 when feedback loops are broken or

11:29 non-existent, engineers are working in

11:32 isolation from consequences.

11:34 Um, they're disconnected from the impact

11:36 because the impact is invisible.

11:39 Another test is the two or three people

11:41 test. Um, if you want to know some

11:45 something about what's happening in your

11:46 system and you go to an engineer, do

11:49 they turn to their tools or do they say

11:52 you should ask soand so

11:56 in these orgs? Someone always knows how

11:59 to answer the question, right? But if

12:01 everyone is used to going to the same

12:04 two or three people instead of turning

12:06 to their tools to ask the question,

12:08 that's not good.

12:11 It's a it's a bottleneck risk. It's a

12:13 burnout risk. It's a knowledge

12:15 concentration risk. It's a scaling

12:17 impossibility. There's a whole another

12:19 route we could go down here about. Is it

12:21 a tools problem or is it a training

12:22 problem, but you're just going to have

12:23 to wait for the book for that.

12:28 The customer discovery test. How many of

12:30 your bugs are discovered by engineering

12:32 versus by customers?

12:34 If you don't track this, you probably

12:37 should. It's really motivating.

12:41 Um, research has showed I didn't put the

12:44 graph here. Darn it. Research has showed

12:46 that the cost of finding and fixing bugs

12:49 goes up exponentially from the moment

12:51 that you write them.

12:53 So, you write your bug, you find it in

12:56 your tests, awesome. You ship it to

12:57 Pride, find it, great. But then it

13:01 becomes the new normal.

13:03 It if it's if and when it's found, it

13:05 will probably be found by someone else

13:07 doing something else, likely someone

13:09 who's paying you. They will likely not

13:11 be happy.

13:13 Cost of finding fixing those bugs goes

13:14 up exponentially. Your best shot of

13:17 finding it is right after you ship it,

13:19 but only if you go look for it.

13:22 And the mystery test. What kind of

13:24 mysteries do your engineers accept as a

13:26 normal course of doing business?

13:30 I really hope you're enjoying this

13:31 video. [music] Do you know Lead Dev

13:33 offers way more than this?

13:35 >> I'm a fan of the Lead Dev newsletter.

13:39 [music] I find really practical um

13:41 knowledge about leadership.

13:43 >> Then I was like, "Oh, there is this big

13:45 community here doing all of these

13:47 content. [music] So, let me uh figure

13:48 out a little more. Let me dig in a

13:50 little. What do you have?"

13:52 >> So, make sure you check out leadv.com.

14:00 And I just want to point out this is a

14:02 very rational response to bad tooling. I

14:05 mean I've been that engineer who's like

14:07 what is that spike every Tuesday and

14:10 then you go and you spend a day, two

14:12 days, three days. You haven't found you

14:15 haven't figured anything out. The rest

14:17 of your work is piling up. And what do

14:19 you learn from that? You learn don't be

14:20 curious. It doesn't pay.

14:24 That's not good. Bad observability tools

14:27 take the natural, healthy, normal

14:30 curiosity that engineers are born with

14:32 and crush it out of them.

14:35 And lastly, do your SLOs's have teeth?

14:38 Test.

14:40 SLOs's are slowly becoming

14:43 the norm in our industry, which is

14:45 amazing.

14:46 But does it does it actually drive

14:48 behavior?

14:50 When your SLOs's are burned out, do you

14:53 actually change gears and start working

14:55 on reliability and resiliency and the

14:57 things that or do people just kind of

14:59 shrug? Does your product team know that

15:03 that that they're they're not going to

15:05 get any more feature work done? Like if

15:08 not, then your SLOs's are just theater.

15:10 You're getting there closer than you

15:12 were. Um job isn't done.

15:16 We talked about feedback loops and I've

15:18 seen so many software engineering

15:19 companies that are in just a doom loop

15:23 where and the doom loop just it

15:25 accelerates with every iteration.

15:27 Organizations get stuck in this pattern

15:29 for years. Everyone knows it's

15:31 dysfunctional. Everyone wants to fix it

15:33 but they're all busy firefighting and it

15:35 feels impossible. We need to slow down

15:37 to speed up. We need to stop shipping

15:39 features for six months and invest in

15:41 observability, reliability and technical

15:43 debt. We need to accept we need to

15:44 accept lower velocity short term to

15:46 build foundations for higher. This is a

15:48 really hard sell [laughter]

15:51 to business stakeholders who are already

15:52 frustrated with engineering velocity.

15:54 You want to move slower while our

15:56 competitors are beating us to market.

15:58 What? So team just kind of meddle along.

16:01 Muddle along.

16:03 Breaking the doom loop

16:06 requires recognizing that the

16:07 dysfunction has a root cause, which is

16:11 almost every time poor sense making

16:14 capabilities. You can't tell what you're

16:17 doing. So, you stop trying to find out

16:20 what you're just flying blind. You're

16:22 just flying blind all the time. Uh which

16:25 means that you're constantly just like

16:27 spewing tech debt [laughter] everywhere.

16:28 I don't have to go. You all know what

16:30 this is like. We don't need to dwell in

16:32 this sad place any longer.

16:35 High performing orgs are characterized

16:37 by fast feedback loops.

16:42 These tight feedback loops are what

16:44 enable rapid iteration, learning and

16:47 improvement. They make excellence

16:48 achievable through systematic refinement

16:53 instead of heroic effort.

16:59 The final kind of cruel irony is is how

17:01 many organizations with weak

17:03 observability don't actually realize

17:04 that their observability is a core

17:06 problem.

17:08 But what to do about it? I have seven

17:10 minutes and 30 seconds left. All right,

17:13 we're going to speed through this. Uh

17:16 the problem with the current approach is

17:17 that you're going to be stuck getting

17:19 small marginal improvements for a great

17:20 deal of money, time, and effort at every

17:22 point. If you want to turn your

17:24 observability team into a lever that can

17:26 move the world, if you want those

17:28 exponential returns on your investment,

17:31 you need to reconceptualize it.

17:35 Run observability like a platform

17:36 engineering team, not like

17:38 infrastructure.

17:41 Some of the best observability engineers

17:42 in the world that I've ever seen who've

17:44 had the most impact have no background

17:46 in SR or platform or anything. They come

17:48 from the front end. They come from close

17:51 to the customer.

17:53 Um

17:54 the platform principles

17:57 build like a product build in design

17:59 thinking have have user research design

18:03 principles make it simple and easy to do

18:06 the right thing without thinking on

18:07 autopilot. Make it hard to do the wrong

18:10 wrong thing. The scarcest resource in

18:13 systems

18:14 the resource that matters the most in

18:16 every system thinking is the scarcest

18:18 one. And the scarcest resource in your

18:20 system is developer cognitive bandwidth.

18:24 So they need to be able to instrument

18:26 and understand their code almost without

18:28 thinking about it so that you can hold

18:30 them accountable for actually going and

18:32 looking at their code after they've

18:33 deploying it. That's the prime

18:35 directive.

18:37 Limit the cognitive bandwidth. Number

18:39 two, manage your invers observability

18:43 like an investment not a not a cost

18:45 center. Um,

18:49 you can't it's like the definition of a

18:51 cost center is you can't make more money

18:53 by spending more on it, right? Like you

18:55 if you give your if your your developers

18:57 five laptops, they're not going to make

18:58 five times as much code, right?

19:00 Observability is not infrastructure

19:02 anymore. Spending more on your

19:04 observability team and tools, if they

19:06 are the right people and the right

19:07 tools, can break you out of this doom

19:10 loop and and really like turn things

19:13 around. And the key here is

19:15 understanding the dual mandate of

19:16 observability. You don't just have an

19:18 observability team who, you know, runs

19:20 your Prometheus and your graphana for

19:22 you. That's that's not going to get it's

19:24 not going to change anything for you.

19:26 Um, but the dual- mandated observability

19:28 is external and internal.

19:30 The external one maps to your core

19:32 differentiators as a business, the

19:34 things that make you money. If you are a

19:36 shopping, you know, if you're

19:38 Amazon.com, then it means, you know,

19:40 your your your your website should load

19:42 really fast, right? your your shopping

19:44 cart shouldn't be abandoned, right? And

19:45 the PE obser observability teams are not

19:48 the ones who do this work, but they are

19:50 adjacent to these teams, right? They are

19:52 the lever that tells how much of that

19:55 engineering labor is going to be

19:56 efficiently converted into business

19:58 impact. And so, you know, every every

20:01 almost every company out there is going

20:02 to build some tools, buy some tools, and

20:05 use some open source. And the tools that

20:07 you build are going to be right next to

20:09 your core differentiators, the libraries

20:11 to make it so that whatever it is that

20:14 makes you you as a business, you can do

20:15 it really [ __ ] really freaking well.

20:21 That's the external. And then the

20:22 internal half, it's pretty easy to

20:24 quantify the external part of your

20:26 observability mandate. Uh lots of people

20:29 have done it. You know, Walmart

20:31 published the stuff about how every

20:32 additional 100 milliseconds of load

20:36 caused like a whole percentage less

20:38 revenue. Like that can be done. The

20:41 internal one is harder to quantify, but

20:43 the opportunity is even larger. Oops,

20:46 that was the slide I should have been

20:47 on. All right, there's your external

20:50 and here's your internal.

20:53 The internal half of your mandate is all

20:56 about your ability to move fast. It's

20:59 about instrumenting your build pipeline.

21:01 It's about instrumenting your CI/CD,

21:04 instrumenting your test suites, getting

21:06 down that time between the interval

21:08 between when the developer writes the

21:10 code when it's in production as short as

21:13 possible. Any intercom folks here. Yay.

21:18 Uh intercom CTO just wrote me a little

21:20 insert for the book talking about you

21:22 guys ship what 200 times a day, eight

21:23 minutes per Yeah, something like that.

21:25 It's absurd. And that's a Ruby Rails

21:27 monolith. So like none of us have any

21:30 excuse, right? [laughter]

21:35 But like when you get that interval down

21:37 short then all of these to there are so

21:39 many great dev tools that have been

21:41 released in the past over the past

21:42 decade feature flags you know uh

21:45 progressive deployments canaries and if

21:48 you don't have good observability it's

21:51 just it's not really going to help

21:52 because it's really a one-two punch of

21:55 really good observability and feature

21:57 flags really good observability and

21:59 progressive deployments but like once

22:02 you can get your site cycle time up.

22:04 That's what it's all about.

22:07 How fast are your deploys? How long does

22:09 it take you to ship a single line of

22:10 code? How long to run tests? How long to

22:12 understand the impact of the code you

22:14 wrote on a single user?

22:17 Our friends at Intercom have a saying

22:19 that shipping is your company's

22:21 heartbeat. And I love that. Actually

22:23 have a sticker with it. It's a little

22:24 rainbow on it. Shipping is your

22:26 company's heartbeat. Because it should

22:27 be constant. It should be a nothing

22:30 burger. It should not be a heavy lift.

22:31 It should not be something that scares

22:33 people. It should not be something that

22:34 takes 20 people to get it out the door

22:36 that happens every other Tuesday. Should

22:38 happen constantly. And you can't do that

22:40 without good observability because it's,

22:42 you know, moving swiftly with

22:44 confidence. You can't move swiftly if

22:46 you don't have confidence. And the

22:48 confidence comes from the tooling to see

22:50 what you're doing. And here's where I'm

22:52 really going to zoom through these

22:53 slides like like like real fast because

22:57 one of the and and I almost feel a

22:59 little busy about a little bad about

23:00 saying this because I don't like to

23:01 sound like a shill but the reality is

23:03 that if you're using tools that are like

23:05 the three pillars metrics logs and

23:07 traces you're kind of doomed before you

23:08 even start. Um,

23:12 multiple pillars will will doom your

23:13 attempts to have fast feedback loops

23:16 because if you have to hop from metrics

23:19 over here, logs over there, traces over

23:21 there, exceptions and errors over there,

23:23 profiling over there, uh,

23:26 it's it's not a feedback loop. um the

23:30 unified.

23:32 So prior to 2019, every single

23:34 observability company use of multiple

23:36 pillars and since 2019 every single

23:39 company has been founded has used

23:41 unified storage where you just have

23:43 structured data and you just put it in

23:44 there and then you could slice and dice

23:46 and zoom in and zoom out and

23:50 and it's just a game. Got my little

23:52 clock is 52 51 50.

23:56 Anyway, I'll put the slides on the on

23:58 web. Um, the point is

24:02 structured data is not that hard.

24:04 Metrics, logs, and traces are very hard.

24:10 Every O'Reilly book ends with a chapter

24:12 where they ask you for your predictions

24:13 for the future. And the prediction I was

24:16 too chicken to make in 2022 was that uh

24:20 the multiple pillars, the three pillars

24:21 model was toast and that the

24:23 observability ali 2.0 you know, our

24:26 unified storage model is inevitable. Uh,

24:28 but it does seem to be playing out that

24:29 way. Um, I think the Gartner folks seem

24:32 to think that over the next five years

24:34 or so, software telemetry is really

24:36 going to move away from the siloed off

24:38 pillars and toward the data lakehouse

24:40 type model where instead of storing the

24:43 data in a different silo for every data

24:46 format, every signal type, instead you

24:48 store it once very high density, wide

24:52 packed storage with lots of context and

24:54 you push the work of interpreting that

24:56 that data out to the client's user

24:59 interfaces. is uh the telemetry

25:01 pipelines uh it just makes sense and I

25:04 do think that AI is going to accelerate

25:06 this transformation

25:08 um those in the consumer side

25:12 but I should also just say if all you

25:14 need is an ops tool for operational

25:15 outcomes you don't need to pay for

25:17 observability it is literally hundreds

25:21 to thousands of times more expensive and

25:23 so if all you care about is is it up is

25:26 it down is it slow are there errors

25:29 There are plenty of little kind of mom

25:31 and pop shop monitoring tools out there.

25:33 I forget the names, but you don't have

25:35 to pay an arm and leg,

25:37 but you don't want to be stuck in the

25:38 worst of both worlds. So, paying $24

25:40 million a year when all you're getting

25:42 is is it up, is it down, is it slow? Are

25:45 there errors? Right? Uh so figure out

25:49 which lane you need to pay for and then

25:51 find the right tool.

25:53 Uh and just to recap, I know that was a

25:56 lot. I ran through it really fast. Uh

25:58 the book will be out in a few months.

26:00 We're going to do pre-release chapters

26:01 soon. There's a lot more there. But if

26:04 you're going to take anything away from

26:05 this, it's that

26:08 our systems are getting so complex. So

26:10 complex. Even people who have spent

26:12 their entire careers priding themselves

26:14 on being able to hold a hold the system

26:17 diagram up here and reason about I was

26:19 one of those guys. Closest I've ever

26:22 come to feeling like God is when

26:24 somebody's like, "What's wrong?" And I'd

26:26 just be like, "It's Reddus." And they're

26:28 like, "It doesn't say Reddus anyway."

26:29 And I'm like, "I know.

26:33 Those days are ending. Those days are

26:35 ending." Uh, if you have complex

26:38 systems, you increasingly need complex

26:40 observability tools and you will pay for

26:41 them. And the way that you make that

26:44 payment pay off, treat it like a

26:47 platform team, design, product. Um, and

26:52 remember the twin investment. You invest

26:55 in your core differentiators. You invest

26:58 in your ability to ship code and

27:00 everything else. Like there are a lot of

27:02 services that are not, you know, facing

27:04 customers. They're not latency

27:05 sensitive. They're not revenue making.

27:07 You know, there's a lot of stuff that

27:08 you can get away with with just

27:09 monitoring up, down, slow errors. Um,

27:13 and finally,

27:16 spark joy. I I it's so easy to get

27:19 burned out as an engineer and so much of

27:22 the reason we get burned out is because

27:24 we're so separated from the consequences

27:26 of what we do. When you can not just

27:29 write code in a corner and be in your

27:31 IDE all day, but like follow it out to

27:34 the real world and watch as users play

27:37 with it, see what they really do, which

27:39 is never exactly what you thought they

27:40 would do. It's so motivating and it's so

27:44 fun and I just really think it kind of

27:46 reawakens the reasons we became

27:48 engineers in the first place.

27:51 [applause]

27:56 [music]

28:03 >> [music]

The twin mandate

Name: The twin mandate
Channel: LeadDev

LeadDev

Key Takeaways

Start a conversation

The twin mandate