A long time ago, I learned the hard way that config is code. I used to think config was safe and code was not safe. But I think the opposite is true. I have found that bad config can break things just as fast or faster than bad code.

That’s why I always prefer to find a way to test my config. At a minimum, load the config using the same code that loads it in production, parse it, poke at it, verify it is doing exactly as you think it is. While it is still possible for that config to break something, at least it helps if that thing that is broken is not the config itself.

Your results may vary, but I prefer to test my config before pushing it to prod.

Availability

We spent a fair amount of time in the past three posts on availability. The purpose of computing, displaying and monitoring availability is that it most easily defines what “good” is across any service.

(SUM of successful responses)
Availability = ———————————————————————————
(SUM of valid requests)

Drops in availability have measurable business value that can easily be computed to $ and ¢.

Total Revenue
————————-————— - Total Revenue ~= REVENUE IMPACT
Availability

Now let’s take a look at additional metrics that are important to monitor for any service.

Latency

Caveat: For the purposes of this article, I am not going to cover end user experience latency, which is a complex and highly specialized topic. For this article, assume latency of a micro service…

Latency is the total amount of time it takes to complete a single transaction, and is typically measured in milliseconds for consistency across services. While availability is more of a high level gauge, Latency is a window into the application that can say and reveal a lot about what’s happening under the hood. It’s like taking an XRay. Take for example, the following graph. The first question I would ask is what is causing the spikes at p90?

If I were to wager a guess, more than likely it’s contention for shared resources, such as a database running into I/O contention. Is this a big deal? Well, it depends. How much traffic? How much headroom is available in the fleet? How often does it happen?

Latency allows us to generally determine a number of different things about a service. Sudden drops in latency can often indicate an availability drop as errors sometimes cause abnormal termination of the request. Sudden spikes in latency are indications of exhaustion of a shared resource, whether it’s a consistency lock on a database table, availability of a new connection from a connection pool, or lack of available cpu.

Latency can also be used to compute concurrency using Little’s law:

Concurrency = Arrival rate * Wait time

Concurrency can be used as a computational aid to understand concurrent resources needed to satisfy traffic at a specific level. An example would be connection limits on a load balancer as a simplistic load shedding device.

One of the strongest signals latency can give is during stress testing it often can be used to signal the breaking point of an application on a specific resource configuration. The breaking point, as described by Amdahl’s law, is the point at which a specific application on a specific hardware configuration becomes overwhelmed. It is at this point point that latency spikes dramatically. Some sophisticated load shedding techniques use latency rather than concurrency to predict when load shedding should occur to enable fleet protection in traffic overload scenarios.

Errors

Using metrics to monitor errors can be extremely helpful. However there is one caveat, which is that high cardinality metrics be both expensive and noisy. It can be really helpful to summarize errors by type into a lower cardinality list. Timeouts are a really good example error metric. Any timeout of a service endpoint is an availability drop. Any other availability causing errors (or server faults) should have some way to be summarized on a dashboard at a low cardinality for monitoring purposes.

Physical Resources

All service are dependent on physical resources in order to run and respond to traffic. It is important to monitor critical resources that may reach physical limits due to increased traffic or other application issues. CPU and Memory are good examples, but it can sometimes be important to monitor other resources that are either heavily optimized (e.g. limited) or known to cause contention issues, such as thread pools, connection pools, and dependency rate limits. If caching is an important part of the service design, a good graph on cache-hit ratio is a must. Generally latency can in many cases act as a proxy to resource contention signals, but it is valuable to have important resource metrics on a dashboard as well to enable rapid diagnosis of where the resource contention is once it arises.

Dependencies

For each critical dependency, whether it’s a storage resource such as a database or search index or a 3rd party or internal service, a couple of quick graphs per dependency should be on the primary service dashboard. It makes sense to monitor Availability, Latency, and Timeouts per dependency. If the number of dependencies is large, it can sometimes make sense to put these on a secondary dashboard that can be used for deep dives for operational issues.

Monitoring Each Endpoint

When monitoring a service, doing an overall monitor on all endpoints is really only useful if all endpoints are doing similar work (such as multiple web pages on a web server). Generally services with multiple endpoints require individual latency, availability, throughput, and error metrics per endpoint.

In summary, understanding availability, latency, throughput, and errors should be a key design goal of a service dashboard. Monitoring by endpoint is important. Understanding utilization and issues with physical resources and dependencies can help to quickly diagnose and respond to issues.

And the more the system scales horizontally and vertically, the more likely failure is to happen. I’m going to do a shallow dive into one practice that is a bedrock, foundational practice to driving learnings and improvements from systems failure. That practice is Correction of Errors (COE). I’m going to attempt to try this with one post as short as it possibly can be and no shorter. But bear in mind this is a summary of a subset of the process.

For many years, at Amazon, the Correction of Errors process was not publicly talked about. This process and the vast internal repository of learnings which have been accumulated over the years is more than likely one of the most valuable operational assets owned by the company. Amazon has an internal system that stores, manages and runs their COE process, machine learning that links new findings to previous COEs, and automated ticketing systems that align engineering organizations around findings and driving completion of action items. In a data-driven culture, it is essential to have real data that can be pointed to as the basis for core operating and engineering principles. The data contained in the COE system is used by engineers across the company to back up engineering and tradeoff decisions around why this feature or that attribute of a system needs to be built in a certain way. The COE repository provides real, documented evidence of failures that happened, the reasons why, and what actions were taken to make the affected systems more resilient or to further limit the blast radius of failure.

The wrong way to do COEs: It’s important to understand that human nature and bad practices can cause us to introduce human bias into the COE process. For example, Post-accident attribution to a ‘root cause’ is fundamentally wrong. Complex systems often have multiple failures that occur together to contribute to an incident. It can be very tempting to point to one thing and completely over look how bad all of the other contributing factors were. Hindsight Bias is also very easy to fall into, and cause us to look at the problem as if we understood what it looked like before the incident happened. We miss a lot when we look at an incident that way. The danger of both of these tendencies is a false sense of security that if we “just fix that one thing” or “if we had just done this other thing” that the entire system will be safer as a result. I highly recommend reading Dr Cook’s simple treatise, How Complex Systems Fail for a very high level view of how hubris leads to improper practices towards dealing with failure.

Blast Radius: We can’t really talk about Correction of Error without talking about a more basic and fundamental concept that is assumed and is critical to engineering for failure, which is the concept of limiting blast radius. Since nearly all outages are caused by changes, blast radius is all about limiting the potential impact of any given change that causes failure. There are at least two (possibly more) dimensions to limiting blast radius: exposure and time. Limiting blast radius along the exposure dimension means that you have built the ability to gradually roll out a change such that if a defect is determined, the impact was a subset of the potential impact if it hadn’t been rolled out gradually. The second dimension, time, examines how quickly an incident is able to be detected and mitigated, and examines measures that are available and/or used to both detect and quickly rollback affected changes.

Given an understanding that finding root cause is not the most important outcome from a COE, a belief that failure in complex systems happens all the time, and key to making change safe is limiting blast radius, that leads us to the three most important questions asked in the COE process. In most COE reviews, these questions need answers first before any others. It can be very tempting to try to focus on root cause or the PR that caused or fixed the issue, but these questions are ultimately the most important, which leads to the intentional design and structure of a well-written COE document.

“What Happened?” COEs are about forensic analysis. This doesn’t happen without data. Data is critical to a good COE document. Good data eliminates hunches, discourages bias, and focuses on customer and business impact. The most important data are: (1) The one metric that clearly demonstrates the outage. Obviously, if this is unable to be produced, it likely points to a potential observability failure that needs correction. A good COE has a graph with a link to source data that clearly demonstrates the impact, based on some operational metric and even better if a financial metric is also able to be tallied. This data can be used to demonstrate the “exposure” dimension of the blast radius of the incident. (2) Timeline. Without a precise timeline, it is really hard to understand the “time” dimension and ask the right questions about all of the factors that contribute to the amount of time it took to mitigate the incident. The timeline is key to many of the best learnings from a COE, and it’s important to refer often to the timeline when asking the two most important sets of questions, which come next…

“How did you detect the incident? As a thought exercise, what could you have done to cut that time in half?” Incident detection is the key first step to restoring service. A sure sign of poor monitoring is the system breaking and it taking hours for a customer or some random employee to happen to discover it’s down. Meanwhile, customers likely have been unable to use the system for those hours with no ability to tell us its broken. This question often raises important questions about whether the right monitors are in place and whether they are tuned appropriately. For a primary system outage, ideally on-call engineers should be paged within 2-5 minutes. If the metrics are noisy, it’s a good time to ask whether we have the right data to create the right monitor that can go off at the right time.

“How did you mitigate the incident? As a thought exercise, what could you have done to cut that time in half?” Time to mitigate is key to understanding how well incident management is done. During the time from detection to mitigation, root cause analysis should not necessarily be the focus. If the team knows what is not working, hopefully they have a lever in place that can mitigate quickly before root cause analysis is done. This could include rollback of a deployment, turning a feature gate off, or pulling a lever in dynamic config. Good on-call practices emphasize, where possible, mitigate first, diagnose root cause later. This question generally probes in the areas around operational levers that have been built to stabilize the system or turn off problematic behaviors.

After incident response is covered, it’s a good time to look at the root cause and ask another important question. What was the blast radius of the change that caused the outage? Is there any way the blast radius of the change could have been cut in half or less? Tools and practices around reducing blast radius probably deserve a lot more discussion, but you know you are getting better at it when you never have scheduled downtime (the very definition of large blast radius) and teams are using feature gates or dials for any changes of significance or any change in an area of risk to the system.

Pro tip: Preparing for a big launch? Wondering what you need to do to prepare for that next huge release? Try this: Create a simulated COE that simulates a potential failure of the system, anything from someone pushing a bad commit to a database running out of memory. Even though you’ll never likely guess what actually will go wrong, It’s surprising what can be learned if you think through the process of incident response for a failure that you think may never happen, and what the timeline would look like if it actually did. This is a great team-level exercise and it helps greatly with those key areas of monitoring, incident response, team dependencies on other teams, and fallback strategies.

External References:

1. “Correction of Error” (AWS Well Architected Framework) -AWS Well Architected Framework - Correction of Error

2. “How Complex Systems Fail” (Richard I. Cook, MD) - https://how.complexsystems.fail/

It can be helpful to think of an application as a black box that has gauges on the outside that tell you what’s going on inside. We as developers are responsible for wiring those gauges to the things that we want to know about. And when the gauges start spiking, we would prefer if the gauge could actually tell us something about where things are going wrong, not just that something is going wrong. And if your black box is dependent on another black box, you want to know if it is your black box that is causing the error or if the problem is in another black box at the other end of a wire.

I generally believe that effective service operations involve being able (1) tell at a glance on a service dashboard if it is operating within expected parameters, and (2) have faith that if a new health condition affects my service, I can see it in the dashboard and classify it one of three domains: Server fault, client fault, or processing error. I like these domains because they attempt to completely separate the space into separate responsibility domains.

A Server Fault has the following characteristics: (1) It is 100% the responsibility of the service owner to fix, (2) counts against the service’s availability metric, (3) while not possible to drive to zero, driving to zero should always be a goal and (4) is always a reflection of the sum of the system design and engineering that have been applied to the service and the availability of its upstream dependencies. These errors count because they reduce the availability of the overall business and can have a direct impact on revenue for better or for worse in a big way.

In the worst possible case, if the availability of a service drops significantly, it has huge impact on all customers of our service, causing a direct drop in revenue. In the best possible, case a service with high availability can easily beat out any other competitor in an A/B test even with inferior CX, simply by being more available.

Driving to a high service availability posture can produce a sense of pride in the owning team and every order of magnitude increase in availability (adding a '9) requires a whole new set of engineering skills and advantages in availability can lead to overwhelmingly crushing defeats when comparing like competitors. When a server fault alarm goes off, the service owner is 100% responsible for fixing it.

A Client Fault has the following characteristics: (1) Is is 100% the responsibility of the client to fix (putting aside service-vended client scenarios). (2) Almost always indicates a bug or misconfiguration in the client code or lack of data validation. When a client fault alarm goes off, the the owner of the client is 100% responsible for fixing it.

A Processing Error has the following characteristics: (1) unlike server and client faults, processing errors are intentionally produced by the system and are usually expected and explainable based on some physical resource state or constraint, and (2) there will always be some sort of rate of these errors, which may or may not be predictable. Processing errors happen because as a normal fact of life, the application has to honor all of the rules of the system which usually map to resources in the physical world. Products cannot be sold that are not in inventory. Credit cards cannot be honored if they are expired or over limit. Addresses cannot be shipped to if they do not exist. Monitoring these errors is important from a business perspective to understand “how” the system is operating. Monitoring generally requires some understanding of what “normal” error rate is and may require investigation only when deviance from normal is observed.

Service Endpoint Error Handling

In order for a service endpoint to reliably emit observable metrics, here are some opinionated heuristics for service response behavior and metrics behavior related to errors:

1. The handler for a service endpoint should emit a reliable, observable metric stream that classifies errors into (a) Service Faults, (b) Client Faults, and (c) Processing Errors.

2. Error metrics are emitted with a separate (more useful) cardinality than HTTP status codes.

3. Separately from emitting metrics, in order to build predictable HTTP clients, services have a predictable response behavior at the HTTP layer and map

   1. Service Faults to HTTP status codes in the 5xx range

   2. Client Faults to HTTP status codes in the 4xx range

   3. Processing Errors to HTTP status codes in the 2xx range

4. Processing Errors are always serialized into and returned reliably in the application response, so (a) that the client application has a chance of observing them and (b) the application can in turn return them to downstream systems.

5. Processing errors from an upstream dependency should be (a) expected and (b) passed through back to the client of the service endpoint, either (a) unchanged or (b) translated. There should be no confusion in the code between an expected processing error and an unexpected client or server fault.

6. Any server or client faults from calling an upstream dependency should be wrapped into a new Server fault.

7. If a service cannot get a valid response from a critical dependency and fails the transaction it should directly count against the service’s availability.

8. The definition of 'critical dependency' is defined by the service owner and allows for degradation and fallback behaviors.

Counts require too much context: As a service owner, if I look at my dashboard every single day, I might have a gut feeling on whether “83” or “2.22k” is a good or not good number for a specific metric for my service, but usually I don’t. Because in addition to count, I also need to ensure I am looking at the correct timeframe. Is “83” a good number when I’m looking at a days worth of data? 5 minutes? 2 weeks? What should that look like over 90 days? Even if I know the answer to that question, it is highly unlikely everybody else on my team does. And what about anyone else who comes to look at my dashboard? Are they going to understand what these counts mean and whether they are good or bad? Or am I going to have to always be present to interpret the data for them? That doesn’t scale.

Counts don’t scale: As a business changes over time, and hopefully grows exponentially, counts will inevitably change. A count that looked good yesterday at 100tps, might not look good tomorrow at 200tps or in the middle of the night at 50 tps and and certainly will be quite different when traffic reaches 12,000tps.

Monitoring with “magic numbers”: Another problem with counts is when we need to build some sort of logic to do something when a count reaches a certain threshold. This can include monitors and can also include automations like blocklists. If I build a monitor that pages when the number of errors reaches “10” every 5 minutes, how do I know that’s the right threshold? When does it need to change? Is 10 errors every 5 minutes OK during peak traffic in the middle of the day? Is the same number OK when traffic drops significantly in the middle of the night.

Normalization of deviance: Using “magic numbers” for monitors is almost always the wrong approach because as humans we tend to base these numbers on how much pain we are willing to endure. If I set my monitor to page me in the middle of the night, you better believe I’m going to make sure that magic number is set high enough that I won’t get woken up very often. It’s very normal practice to tune monitor thresholds to try to keep monitors from being too noisy and also not noisy enough, so that part is normal. However, being as counts don’t scale, what ends up happening in practice is there is a lot more noise than signal to work with. Therefore, we normalize the deviance by setting the thresholds higher than they need to be to account for noise, and as traffic increases were are constantly fighting with thresholds rather than searching for signal and understanding if the problem we are dealing with is getting better or worse.

Rates Scale Better: Rates are a pretty simple concept. Let me start with a simple example.

Rate = Signal Count / Total Transactions * 100

Using this approach scales to any scale of transactions and any timeframe. If you’ve established what the normal error rate is, such as x%, you can now do a number of different things that you cannot do with counts.

1. Signal becomes constant over timescale: Viewing the dashboard for any timescale requires no interpretation. You can view rate over 5 minutes, 60 minutes, 2 weeks, or 90 days, and it becomes much more constant.

2. Signal over noise is much easier to visualize: If you look at a graph and the error rate spikes, you know that you have deviance over the norm without interpretation. If you are displaying counts, you may see spikes, but without context, you might not know of those spikes are caused by traffic spikes or actual deviation from the normal error rate.

3. Much easier to tune thresholds: Rates smooth out traffic spikes. So traffic based on time of day or day of year doesn’t matter so much. Your are monitoring the rate at which something happens. You still need to tune the monitor to make sure action happens at the proper threshold, but over time, it tends to be a much more stable number.

4. Deviance becomes less normal: If you have constant error rate that is expected and that changes with a given deployment, you know it. If error rate goes up, you know you might have a problem. It might be good, it might not be good, but at least you can see easily, react to it, determine what’s happening, and respond accordingly. It’s hard to see that with “magic number” counts that usually are set too high to see deviance.

5. Easier to convey context: Once you understand your rates much better, understand what “normal” looks like, and have set your monitor thresholds accordingly, you can now do something to even better convey context to those outside of your service who come and take a look at your dashboard. You can add indicators to your graphs to include thresholds for Sev1/Sev2 notifications and in some cases it is best practice to include green/orange/red bands or lines to indicate expected performance or desired targets. This allows for easy interpretation of how your service is operating for anyone coming to see how your service is operating and makes for much easier monitoring for oncall operations.

Overall “Goodness” Rate: The above formula makes sense for determining rate of occurrence of some signal in your system, such as a known error, authorization failure, etc. However, there is another approach that works even better for high-level “goodness” measurement. There are two primary “goodness” indicators every service owner should know, availability and success rate. These are binary. The service is either available or it is not. The transaction was either successful or it was not. For determining “goodness” of availability, my preferred approach (when I have control over the metrics being emitted) is to compute availability by emitting a metric from the service client called “Available” with only two possible states:

Available=1 // Request was successfully processed
Available=0 // Request failed

Turning these into a rate is much simpler using a formula like this:

Availability = AVG(Available)*100

Requests that I choose not to count towards availability do not emit a metric, so there is need to have context and understanding of all the possible states. There are only two states. It’s binary. Simple to dashboard, simple to monitor

How we define availability goes a long way towards defining how we monitor it. Availability for any given request generally means that we have been able to send the service a request and we have received a valid response. How this is actually computed may slightly vary by service and use case. I would like to start by describing a general method for computing availability, and then look at how it may be adapted for specific use cases.

A common heuristic that is used internally at Amazon for all tier 1 services is:

(SUM of successful responses)
Availability = ————————————————————————————
(SUM of valid requests)

This computation is done using time series data in order to be able to compute availability for any time frame. The keys to utilizing this formula for any given service is defining “successful response” and “valid request”. This is where the service team should be able to weigh in with a (hopefully concise) definition of how they have chosen to compute these data points through their service’s metrics. A “valid request” generally means that the request passed AuthN/AuthZ checks and input validation checks. A “successful response” generally means that a request was successfully processed by the service and it returned a successful response.

Let me start with a real world example of how I have seen these metrics computed for a Tier 1 web page service on a high traffic website. The service is available to the public internet, has dozens of upstream dependencies that are used to render the page, and is called through a client proxy. In this case we used metrics from the client proxy and computed availability using the following implementation of the above heuristic:

(SUM of Http 2xx and 3xx Responses)
Availability = ———————————————————————————————-
(All requests - Http 4xx responses)

For this formula, we had landed on the following definitions:

• “Successful Response” = We were able to successfully render a web page and return it with an HTTP 2xx or 3xx response.

• “Valid Request” = Any request that did not result in an HTTP 4xx response.

The reason why we eliminated HTTP 4xx responses from the calculation for this use case was primarily to eliminate spurious invalid requests caused by bot traffic from the computation. For a public web page on a high traffic website, there is potential for a non-trivial amount of bot traffic that can cause spurious errors. We wanted the availability of our service to be computed against known valid requests, that is real customers. We didn’t want the presence or absence of traffic from invalid requests to skew our availability metric. It’s important to note that we separately monitored HTTP 4xx errors by type in order to observe anomalies and find content errors. However, we considered that use case to be orthogonal to the availability of our service. If we included this traffic in the computation, it could potentially skew the statistic, and our goal was to maintain 5 9’s of availability against real customers and valid use cases not bots or content errors.

NOTE: Using HTTP status codes in the manner above to compute availability only works if all availability issues result in an HTTP 5xx error. This includes critical dependency calls and timeouts. I have found that in practice, it is possible that a developer might chose to obscure availability issues and return a creative HTTP code other than 5xx, which would make this method problematic to monitor at the HTTP layer.

It is a best practice, where possible, to collect service availability metrics from the downstream client of your service. This is generally possible in cases where you have control over the client and the environment. Internal services are good candidates for monitoring availability from a client that has been vended and instrumented by the service team. However, for external endpoints, client-side monitoring is usually not possible. In that case, the best you can do is emit metrics either from the service itself or from a load balancer or proxy layer downstream from the service. The caveat for this is that this this monitoring will be blind of infrastructure/network connectivity issues that clients may have in reaching the service. Service teams should have a means of detecting these issues, if possible through secondary client-side metrics, or synthetics/canary style monitoring.

Just one additional point that is worth calling out. In addition to monitoring the service itself, the same technique can be used to monitor the availability of a service’s upstream dependencies. A well-built dashboard and monitoring system for a service will cover availability of critical dependencies at the same incident response level as the service itself.

In today's world, some have twisted success into a competitive sport, where winners stand on the shoulders of those they've pushed down. But here's the truth: real success isn't measured by how high you climb, but by how many people you bring up with you.

Bringing people with you means prioritizing their needs and bringing them with you on your growth journey. A growth journey comes in three steps:

1. Commitment to constant learning. Every person you meet is a potential teacher, every challenge a hidden lesson.

2. Investment in yourself not for personal gain, but to better serve others. Think of it as filling your cup so you have more to share.

3. Focus on growing those around you. If you're a leader, this isn't just part of your job – it's your primary mission.

When someone on your team faces a challenge, don't see a problem – see potential waiting to be unlocked. When they struggle, don't see failure – see an opportunity for growth. Your role isn't to judge but to nurture, guide, and elevate. Here's the irony: when you make others' success your priority, your own success becomes inevitable. Not because you chased it, but because it's the natural byproduct of making your environment better.

True success isn't about reaching the summit alone – it's about how many people you've inspired to climb their own mountains. That's the kind of success that is long lasting and makes the world a better place to live in.

These facets don't exist in isolation; they connect. Excellence in one area often enables success in others, overemphasis on one area can cause tradeoffs with others. And most importantly, neglect in any single facet can undermine the entire organization. An over-emphasis on delivery without strong people development creates burnout and turnover. Excessive focus on operations without strategic thinking leads to reshuffling deck chairs on a ship headed to nowhere. Too much strategy without solidly delivering results is like building a map to the end of the rainbow but never leaving the port.

The facets in brief...

The People facet is the starting point for any tech lead growing into a people manager. At its core, it's about creating environments where technical talent thrives through meaningful growth opportunities and psychological safety. However, mastering this skill also involves dealing with poor performance and addressing issues quickly and with empathy. A great people manager pushes each of his team members to be a better version of themselves, points out where they are not taking agency over their own destiny, and takes away excuses that are holding them back. It's not always about keeping individuals happy in a corner, it's about helping them win on the main stage.

As scope expands, people management evolves into building leadership benches, eliminating single points of failure, emphasizing fungibility, developing organizational culture, and creating scalable systems for growth and development across multiple teams. More senior levels of engineering leadership require balancing psychological needs with a sociological view of building an organization that drives business impact. This requires backbone and willingness to do and say the hard things.

The Delivery facet is where the rubber meets that road and is the most externally visible aspect of engineering. Delivery is essential to an engineering leader's success. Delivery requires going beyond being a nice people manager to being the coach who can take his team and drive the ball towards the goal. One of the most important delivery skills a manager can learn is saying "no". No sets a standard, defines a boundary, and enables the ability to prioritize. Saying No is really just the start of having the right conversation on focus and impact. Delivery also includes keeping an eye on quality. Quality is part of what you deliver. It's quite often that quality and delivery time end up being tradeoffs within any delivery cycle, and it's important for engineering managers to continuously help their teams make the appropriate tradeoffs between them.

Operations excellence starts with system reliability and a focus on metrics. Metrics are a proxy to understanding the customer experience. New leaders learn to build and maintain reliable systems and read and pay daily attention to the metrics flowing out of the system in order to understand how system performance is impacting customers. Failure detection becomes measured in minutes, not hours. Change is viewed as constant, but not allowed to degrade performance. Operations is not about preventing failure, rather it is a mindset focused on taming failure. Good operational practices require you to think about failure as something you try to coax into a state of minimal impact, while maximizing the learnings.

The Strategy facet is an important muscle, but takes time to develop right. I would encourage any engineering manager to master the first three before worrying whether they are strategic. However, by the time a leader is managing managers, they should be able to build a view on where the tech needs to head to meet the ever-changing tech landscape, keeping an eye on key technological advances, and finding every opportunity to skate where the puck is moving.

Being able to drive Business Impact is ultimately the goal of every engineering team. Every member of an engineering organization should understand how their role directly relates to moving the needle for the business, which unlocks autonomy and decentralized decision-making for the benefit of all. Good engineering leadership is able to pass enough business context on to teams that individuals can become empowered to make the right choices on where they focus and how they solve problems. That doesn't mean the team will always get it right. There is no magic sauce to decentralization, but the closest ingredient is lots of communication of context. Be real, be hands-on, and help the team, but help them by getting out of their way until you need to get in their way.

The power of this framework is that it can be used for hiring and performance conversations, and it scales across leadership levels. Each facet provides clear paths for growth and development, whether you're leading a single team or an entire organization. The weight given to each facet naturally shifts based on leadership level. Engineering managers and senior engineering managers typically concentrate on people development, day-to-day execution, and operational excellence. Their focus remains closer to the ground—building high-performing teams, delivering projects efficiently, and ensuring system reliability. More senior managers and directors should be proficient and then elevate their perspective. Their people focus changes from psychological to sociological. Delivery starts becoming more focused on GTM. Operational concerns expand to include risk management, security, and scale. Most importantly, they must dedicate significant energy to strategy and business impact—understanding market dynamics, talent relevance, driving tech innovation, and ensuring engineering efforts align with business objectives.

If this works out, I am hoping that in a set of future posts, we can spend more time doing a deep dive into each facet, exploring specific practices, metrics, and tools that engineering leaders can use to develop their capabilities. We'll examine how to assess your current state in each area and create actionable plans for improvement. Whether you're just starting your leadership journey or seeking to elevate your impact at a higher level, these facets provide a clear framework for continuous growth and development.

Stay tuned for our first deep dive into the People facet, where we'll explore some thoughts and ideas on performance and growth. Because great engineering leadership isn't about being perfect in all areas, it's about always learning.

I'd love to learn from you all. What parts of this don't quite resonate with you? Any frameworks you are using that you like better? I'd love to hear other perspectives.

Enter LinkedIn, circa 2012. Suddenly, I'm obsessed. My LinkedIn profile became my digital shrine, a testament to every job, certification, and recommendation I'd ever received. I was like a digital hoarder, but instead of old newspapers and cats, I was collecting endorsements and connection requests. Hitting the magic 500 in those days was a feat, these days, not so much.

Fast forward to 2016. I've hit the jackpot - Vice President at a Fortune 500 company. My LinkedIn profile was so shiny you could see your reflection in it. I'd made it, baby!

Or so I thought.

That year was a rollercoaster ride through corporate hell. Restructuring, politics, and more buzzwords than a TED Talk convention. By the end of it, my boss was fired, my team was gone, and I was left wondering what the $%&* just happened.

Reality check time. I realized two things:

1. I wasn't the leadership god I thought I was.

2. I'd been so busy climbing the corporate ladder, I'd forgotten how to actually build it.

Eight years of management had left me about as technically relevant as a floppy disk at a blockchain conference. I'd never personally worked on highly scalable distributed systems or built anything truly reliable. The cloud had happened and I was just an observer. I was all title, no substance.

So, I did what any self-respecting VP would do. I quit.

I deleted my LinkedIn profile. I realized my profile had accomplished nothing for my career short of an anxiety inducing shrine of irrelevance.

I did something that I have done multiple times in my career. I pivoted. I took a massive step back and became an individual contributor at Amazon. For five years, I immersed myself in the world of cloud, highly reliable, scalable systems, and learned how a massively scaled company operated under the hood. I learned more in those five years than in the previous decade of management.

Coming out of Amazon, I had a revelation. Titles? They're just words. What really matters are two simple questions:

1. Am I going to learn?

2. Am I going to have an impact?

That's it. That's the secret sauce.

Now, here's where it gets juicy. I recreated my LinkedIn account, but this time with a completely different perspective. And you know what? I realized something profound:

In the industry, our hiring process is more broken than a chocolate teapot.

LinkedIn has become a proxy for who we are as professionals. But does it really show our ability to learn, teach, and make an impact?

So, here's my radical suggestion: Delete your LinkedIn account.

Yes, you heard me right. In this job market, where opportunities are scarcer than honest politicians, it might seem insane. But hear me out.

By deleting your LinkedIn, you're forcing yourself to redefine who you are as a professional. You're breaking free from the shackles of titles and buzzwords. You're giving yourself permission to focus on what really matters: your ability to learn, teach, and make a real impact.

Is this for everyone? Of course not. But it worked for me, and it might just work for you.

So, here's my challenge to you: Stop obsessing over your digital resume. Stop chasing titles like they're the last slice of pizza at a party. Instead, focus on becoming someone who can learn, adapt, and make a difference.

Because at the end of the day, that's what really matters. Not the fancy title on your LinkedIn profile, but the impact you can make in the real world.

Are you ready to take the plunge? Are you brave enough to define yourself by your abilities rather than your digital persona?

The delete button is waiting. The choice is yours.

(credit to my buddy Claude for helping me get this story on a page and CG for supplying artwork)

Bikeshedding refers to the phenomenon, which has some behavioral research behind it where teams spend too much time discussing trivial issues while ignoring more important, complex ones. The term originates from the idea that if you assign a committee to build a nuclear plant they will tend to spend a disproportionate amount of time debating how to build the bike shed than to tackle the design of the nuclear power plant. In essence, bike shedding occurs when people focus on small, simple decisions that need to be made because they are easier to reason about, than addressing the more complex designs that truly need attention. An example of possible bikeshedding is a long, protracted discussion on naming something or turf wars on using tech A vs tech B, when either are equivalent.

Bikeshedding is an anti-pattern. It can be overt, but usually not. It slows teams down and leads to suboptimal outcomes. While I am open to some diversity in approaches to engineering, there are things that I hold closely to and that are important. One of those things is that if teams are bikeshedding, it's a clear sign that we need a decision. There are only two possibilities for bikeshedding. (1) The issue isn’t trivial, it’s actually important that we get it right or (2) It’s truly not important and we need to make a decision and move on. So in either case, the best thing we can do is make the decision quickly and go.

In my role, I manage a series of meetings throughout the week that are designed to keep our team focused and aligned. (Full disclosure, I have enjoyed employing recommendations from Will Larson's writing). On Mondays, we have a staff leadership meeting where myself and my product leader meet with our leadership teams. On Tuesdays we have the Eng Ops Review meeting where we dive deep into metrics, initiatives and postmortems. On Wednesdays we have our Tech Spec Review, which allows us to take a close look at the design work being done. These are my favorite meetings all week. It allows me to dive deep with the teams across all aspects of engineering regularly.

Team members know that these meetings are where decisions can be made and are actually starting to use the process. So it’s highly likely that if they need help making a decision, they will raise the issue in one of these meetings. It’s hugely rewarding to see that happening. It makes the meeting valuable to me and everyone when we start using it for its intended purpose instead of it being a one-way meeting. Most decisions are either (1) Architectural, (2) Product oriented or (3) Bikeshedding. I highly encourage debate in these meetings to make sure every side is heard, but it’s also important to end the discussion with a promise on how the decision will be made.

Just last week during one of our tech spec reviews an engineer asked a simple question: "What should we name this feature?" It was a classic bikeshedding moment. Naming something may seem trivial, but it's the kind of question that developers could spend hours debating. My favorite approach to naming things like services is to try to not come up with a name that represents what the service does. Naming things can lock you or those that come later into a mental state that limits utility or expansion. However, in some cases, it makes sense to name things well, such as when it comes to schema objects and attributes. This could have been a bikeshedding moment, but it was easily solvable in this case.

There are really just two effective options for any bikeshedding moment. The first is to apply Occam's Razor, a principle that suggests the simplest solution is often the best one and just make the decision on move on. The second option, if it's not so clear, is to delegate the decision to someone who owns that aspect of the project to come up with tradeoffs to help make the decision quickly. Ideal timeframe for any decision is a small number of days. In this particular instance, I chose to delegate the decision to the product team. Although I could have easily used Occam's Razor to settle the debate, I recognized that naming the feature was something that the product team needed to own. It was important for the product, not engineering, to own this part of the project. By delegating, I placed the decision in the right hands and reinforced the importance of ownership within our organization. I also asked them to come back with an answer within a day.

Naming things is an easy example, but these techniques also work with architecture or tech stack decisions. These sorts of decisions can actually be more gnarly, especially if code has been written. My favorite quote on this is from Ben Horowitz's book, the Hard thing about Hard things:

Early in my career as an engineer, I'd learned that all decisions were objective until the first line of code was written. After that, all decisions were emotional.

Decisions made after code has been written almost always have an emotional element to them. Sometimes it helps to remind teams of the emotional part of the decision and ask them for objectivity. It can help to ask them to come up with a short list of the most important tradeoffs involved in the decision and which of those are most important to the business. Ultimately, decisions that involve code already written almost always have to be made by the most senior engineer or engineering leader.

Using Occam’s Razor or delegating to a reasonable owner and asking for quick turnaround are both effective tools for driving through and making decisions quickly. Some decisions can be deferred for later, which is another technique, but not making a decision that needs to be made is the worst possible outcome for a team that can leave lasting impacts. It’s better to come to a decision, even if it’s tempered with “for now”, and everyone can move on.

Bikeshedding is a common challenge in any team. So the next time a bikeshedding moment arises, remember to stay focused, delegate when necessary, help your team get to an answer quickly and move on.

4 things software needs before building features:

1. Trust (safety): This is so obvious that I at first didn't have it on the list. In order for customers to experience the delight of your new service and actually invest a part of themselves in using it, they need to trust it. They need to trust you. They need to trust that if they install your app, you won't do something bad to them or to their device. Whether they ask for it or not, a customer who stays with you wants an offering that prioritizes their safety over all else.

2. Scaling (+resiliency): This may often be overlooked early, and in some cases that's OK for early stage. However, before too long, economics + demand will require scaling and resiliency aspects. Customers expect you to be up when they want to use your service, at their convenience, not yours. This needs upfront design once demand emerges.

3. Engagement (+personalization): I can sense resistance to this requirement already. Why am I putting engagement and personalization in a list of otherwise nonfunctional features? It's intentional. In order to build features in the right way, we need to assume that we understand how our customers engage with our product, why do they keep coming back? And how is our service personalized to them and their needs. These are not features, they are critically core fundamentals that need to be understood and engineered as first class citizens before we consider adding anything else. If we start with these, features are just additive to the core.

4. Rendering (+globally): Understanding how you plan to render your experience to your customers across all of the devices your customer has in all regions of the world is often overlooked. I have seen so many systems that use HTML as some sort of content standard that I don't even blink anymore, but it shows a critical misunderstanding of how the internet works today, or an over-eagerness to jump on the latest javascript library before thinking about the customer experience. Customers expect to be able to access your service on any device that they have handy, at their preference, not yours, which more often than not is a mobile device. Building a service that scales globally across mobile (first) + web takes intention in the design and doing it well doesn't come for free.

3 things engineering teams need to build features:

1. Operations (minimize): Engineers need to own their code in production, but that shouldn't mean they are spending all of their time doing operations. Take time up front to build an automated set of operational capabilities, and always expect failure to happen and have a plan of action for when it does.

2. Safety (maximize): The number one thing, by far, that slows many engineering teams down is protocol that is intended to act as a safety net to keep failures from making it to production. It's the most frustrating thing in the world for everyone involved, including your customers to intentionally throw up safety protocols that require manual checks and sign-offs or manual testing before making a change. It's a defect, not a feature. Instead, make your deployments safe through automated testing, continuous integration and continuous deployments. The only safety nets you need (in most cases) are (1) instrumented observability that reliably tells you the availability of your experience (and wakes you up if it's down), (2) limiting your blast radius (rolling out change slowly), and (3) (most important) make rollbacks as fast as possible and automatic.

3. Tooling (stop coding stupid): Don't reinvent the wheel, embrace frameworks, use tools that auto generate the stupid code so that you can focus on features and functionality. Stop building snowflakes, and instead build what customers really need.

I'm 100% positive this is not a normative list, happy to hear other thoughts in the comments...

One of the skills that I think differentiates a software developer from an architect is being able to negotiate a technical decision path that achieves the company’s goals and delivers maximum business value. Business value is a complex topic, but in some respects I think that agile coach David Hussman describes it best in his “Dude’s Law”. He asserts that V = W / H, where V is value, W is why (intent) and H is how (mechanics). Go read his post (or better yet, have David come coach your team) for more details because he can describe it better than I. However, I think his simple formula has some profound implications for business.

I’m reading Josh Kaufman’s book, The Personal MBA. In his chapter, “Playing with Fire”, he discusses some of the pitfalls that many modern, especially large business fall into around using statistics, financial numbers, and algorithms to attempt to predict the future. Business executives have gotten used to using complex statistical models to attempt to divine the future, and in many respects are not all that better than reading tea leaves.

He espouses that creating and delivering value is essential for business success, but that many top executives come from a finance background and spend more time looking at and trusting their numbers than thinking about creating and delivering value. This can be exacerbated when the company doesn’t have enough people on the management team who have come through the ranks from the area where the value gets created. When the statistical models get trusted and attains higher weight than building and delivering value, I think the company’s future becomes suspect. It gets worse when a statistical model is used to drive “efficiencies” which in many cases end up lessening, not building value.

I also think that many companies today are far too short-sighted in their planning, especially those that are public. Planning, is based around annual financial statements (forced in some part by the SEC). Every decision is very heavily predicated on how the financial statements will look for the year, rather than building the overall value proposition.

So, it appears to me that learning how to quantify business value and use it in decision making is a skill not only important for software architects, but business leaders as well.

As time has passed, I’ve become more and more fond of the quote. I don’t know anything about Mr Bennett (he was before my time), or why he coined this phrase. However, I think it is fairly significant, and I’m still in the process of unwrapping all that it means. In fact, I’ve purposed to study and think on the topic over the next year to see where it leads. I’m convinced that integrity is key to success in life on many different fronts. Excellence is really more an accidental outcome, but more on that later.

In this post, I’m not going to be able to unwrap the meaning of integrity, how it applies to life on many fronts, and how one of its many outcomes is excellence. However, let me start with something simple, the definition of the word integrity.

Webster’s 1828 dictionary, which is one of my favorite dictionaries, defines integrity in this way:

1. Wholeness; entireness; unbroken state.

The constitution of the United States guaranties to each state the integrity of its territories.

The contracting parties guarantied the integrity of the empire.

2. The entire, unimpaired state of any thing, particularly of the mind; moral soundness or purity; incorruptness; uprightness; honesty.

Integrity comprehends the whole moral character, but has a special reference to uprightness in mutual dealings, transfers of property,and agencies for others.

The moral grandeur of independent integrity is the sublimest thing in nature, before which the pomp of eastern magnificence and the splendor of conquest are odious as well as perishable.

3. Purity; genuine, unadulterated, unimpaired state; as the integrity of language.

The more recent Miriam Webster Dictionary defines it this way

Definition of INTEGRITY

1. firm adherence to a code of especially moral or artistic values : incorruptibility

2. an unimpaired condition : soundness
3. the quality or state of being complete or undivided : completeness

Just a quick comparison of what nearly 200 years has done to this word in the Webster dictionary.

1) Both mention completeness, entireness. Integrity is the state of being complete without holes, without brokeness

2) Adherence to morality is mentioned in both, although the 1828 edition has much more to say about what that means. (Why this is true is surprising and worth a post in and of itself).

Just reading through these definitions of integrity gives me a lot of things to think about. The next thing that comes to mind is how these definitions drive me to believe that integrity is related to reality, which is the very definition of truth itself. Definitely much more to think on for a future post.

If you’re only willing to do what’s easy,life will be hard. If you’re willing to dowhat’s hard, life will be easy.” Doinghard things, going the extra mile, makingthings right in relationships, keepingcommitments even when no oneis looking, these are the things that

make us who we are…

Mark Bouman

Integrity is being true to who you are, and always doing the right thing, even if it’s not the easiest route. It’s providing the value your customers paid you for even if it costs you more than you originally thought to deliver. It’s tidying up your code before you commit it so that your coworkers don’t have to inherit a mess. It means being consistent to a methodology that you’ve committed to even when everyone around you is getting by with under-achieving.

Integrity… It’s a hard thing to commit to, but if you do, life becomes a whole lot easier.

Integrity makes you real.

God, give me grace to accept with serenity the things that cannot be changed, Courage to change the things which should be changed, and the Wisdom to distinguish the one from the other. Living one day at a time, Enjoying one moment at a time, Accepting hardship as a pathway to peace, Taking, as Jesus did, This sinful world as it is, Not as I would have it, Trusting that You will make all things right, If I surrender to Your will, So that I may be reasonably happy in this life, And supremely happy with You forever in the next.

Amen.

— Reinhold Niebuhr

What is important is seldom urgent and what is urgent is seldom important

I was looking at a long list of tasks that all seemed to urgently stare me in the face. Each of them was blinking like the red light screaming at Sandra Bullock in Gravity, calling to me to be worked on.

In times like this, I think the Time Management Matrix from Covey’s 7 Habits book is a good exercise to start with. (succinctly described here) It’s a good way to look at what’s screaming at you and think about separating the important from the non-important and critically analyzing yourself on whether you are taking the time to work on important things that are not urgent.

It’s really the latter (quadrant 2) that worries me the most sometimes. If we are not taking the time to strategically think through what are the most important things we need to get done this week, this month, this quarter, or this year, we tend to let those little blinking red lights tell us what to do and we end being automatons, following whatever fire-drill happens to have the most sirens.

It’s not all that bad to respond to fire-drills. Certainly any important things (quadrant 1) are important. But, for a lot of us, if the source of the tasks we choose to do is entirely driven by quadrant 1, then I think it’s a good time to stop. Think about what our strategy and goals are for the year, for the month, for the quarter, whatever, and then make sure that we are forcing what is important to the forefront whether or not it is the loudest voice.

Here are a few ideas I thought worth saving so far:

• There is a difference between invention and innovation. “An invention is simply a new idea. An innovation is the conversion of a new idea into revenues and profits. An idea that looks great in the lab and fails in the market is not an innovation. It is, at best, a curiosity.”

• Innovation enables you to be on the offensive. Not innovating puts you on the defensive. “Innovate or Die”.

• “Collaboration is essential; failure is a regular visitor. Innovation leaders are comfortable with uncertainty and have an open mind; they are receptive to ideas from very different disciplines. They have organized innovation into a disciplined process that is replicable.” They can manage risks.

• “Every company has a budgeting process that is repetitive, refined and ingrained … with every manager participating in the process…. But few corporations can say the same when it comes to innovation.”