Lessons from Charity Majors

Charity Majors is a prominent voice in the software industry, on topics ranging from observability and engineering management to team culture and career development.

On Observability

On the definition of observability: "Can you understand what is happening inside the system — can you understand ANY internal state the system may get itself into, simply by asking questions from the outside?” [1]
On the goal of observability: "Observability is not about finding the root cause. It's about understanding what's happening in your system so you can ask better questions." [2]
On data and context: "Data is made valuable by context. A data island is not valuable. The more connected it is to other things, the more valuable it is." [3]
On the shift from monitoring to observability: "People are over-paging themselves because their observability blows and they don't trust their tools to let them reliably debug and diagnose the problem." [4]
On the importance of instrumentation: "No pull request should ever be accepted unless the engineer can answer the question, 'How will I know if this breaks?'"
On the nature of modern systems: "Our systems are increasingly emergent, unpredictable... and if you don't invest in observability, you're going to have a really hard time." [5]
On the value of errors: "Errors are amazing because we learn something from every time something fails. Our goal is not to prevent them, it's to be resilient to as many errors as possible." [2]
On the limitation of dashboards: "When it comes to observability, dashboards often become a detriment, because they're simply snapshots of the past." [6]
On the right way to gather data: "We need to issue one single arbitrarily-wide event per service per request, and it must contain the full context of that request." [1]
On the future of development: "Observability 1.0 is very much about how you operate software, and 2.0 is about how you develop software."

On Engineering Management

On the nature of the management role: "Management is not a promotion, management is a change of profession. And you will be bad at it for a long time after you start doing it." [7]
On the motivation for management: "You want the people managing teams who, whose hearts are really in it and who really get joy out of helping others succeed."
On the responsibilities of a manager: "A manager's job is not just interfacing with your team. It's also managing up, making sure that the people... that you're reporting to have a clear understanding of what your team is doing and why." [8]
On the importance of technical skills for managers: "The best line managers... are never more than a few years removed from writing code and building system themselves, hands on." [9]
On what a team deserves from a manager: "A manager who wants to be managing people and developing that skill set... A manager who is genuinely interested in process, sociotechnical systems, and nurturing the careers of their teammates." [9]
On the manager as a "shit umbrella": "I think it's really unhealthy when engineering managers use that as currency... instead of thinking about who to give what information to... it's like what do they need and then you make sure they get it." [10]
On the value of management experience: "The skills that you learn as a manager... you'll take those skills with you throughout your life and and and not just to work." [10]
On becoming a manager: "Don't become a manager until you've done what you want to do as a senior engineer... until you feel solid and secure in your technical skills as an engineer." [11]
On the manager's role in career development: "Ask everyone about their career goals... if someone expresses you know that they'd like to be a manager someday... This is a learning opportunity." [11]
On the manager's impact: "A team with a good engineering manager is just you know run circles around a team without one or a team with a bad manager." [10]

On the Engineer/Manager Pendulum

On career identity: "Don't identify yourself as a manager or as an engineer. Think of yourself as a: Technologist who needs both skill sets to reach your fullest potential." [9]
On the benefits of switching roles: "The most powerful senior engineering leaders tend to be people who have done both, swinging back and forth between management and engineering over the course of their careers." [11]
On the necessity of returning to engineering: "It only takes about 3-5 years for your skills to begin to seriously decay. Go back to the well, while you are still hirable." [9]
On the value of management for engineers: "I think everyone should spend some time in management. It gives you so much more empathy for... that whole side of the house it it helps you understand much more viscerally how your actions are connected to the business impact." [12]
On the dual focus: "Being a good engineer involves blocking out interruptions, focusing on learning and solving hard problems. Being a good manager involves being available for your team and interruptible… even interrupt-driven. You can only do a good job at one at a time." [9]
On the path to Staff Engineer: "The fastest way to coin a a good staff engineer is often to take a good senior engineer and put them in management for two or three years." [10]
On career progression: "This is also how you preserve institutional memory and both leadership on the management side and leadership on the engineering side." [11]
On the minimum time in management: "I think that it takes about two years to put down your engineering skills and learn the basics of management and commit to that much the first time." [12]
On the value of a varied career: "A long and varied career, where you become more and more employable with time (not less and less)." [9]
On leadership vs. management: "Management is not synonymous with leadership. Technical leadership is just as important, it is not a loss." [13]

On Team Culture and High-Performing Teams

On the myth of the 10x engineer: "You don't need 10x engineers. You need a team that ships safely, learns constantly, and doesn't rely on heroics." [14][15]
On building great engineers: "Great teams make great engineers." [5]
On the nature of sociotechnical systems: "It is the people and the technology and the tools that we use to interact between the two." [16]
On the impact of the system: "You're going to write and ship code at the speed that the sociotechnical system around you allows you to write and ship code." [5]
On hiring: "Stop looking for the 'best engineer' and start looking for the right engineer... the one who's excited about your difficulties, communicates, lifts the people around them, and can learn faster than the job changes." [15]
On the importance of shipping: "Shipping is how engineers learn. Not architecture diagrams. Not months of planning. Not 400-comment design docs. Shipping." [15]
On team composition: "You have to be with people who like to work the way that you work." [5]
On the uniqueness of teams: "Every single company is a unique complex sociotechnical snowflake." [17]
On creating a healthy environment: "I've never actually seen a company where customers were super happy and engineers were miserable." [5]
On the goal of a high-performing team: "Build a world-class engineering organization that never needs heroes in the first place." [15]

On On-Call and Operations

On the responsibility of on-call: "On call is how we close the feedback loop. Someone needs to be responsible for your services in the off-hours." [18]
The manager's role in on-call: "If you're asking engineers to be on call for their code — and you should — you owe in return: – enough time to fix what's broken – hands to do the work – closely track how often they are interrupted/woken." [17]
On making on-call sustainable: "When an engineer is on call, they are not responsible for normal project work — period. That time is sacred and devoted to fixing things, building tooling, and creating guard-rails to protect people from themselves." [17]
On the potential of on-call: "I believe it is thoroughly possible to construct an on call rotation that is 100% opt-in, a badge of pride and accomplishment, something that brings meaning and mastery to people's engineering roles and ties them emotionally to their users." [17]
On the importance of an SLA budget: "If you haven't used your downtime, you're not taking enough risks. Use it, it's fun, try it." [19]
On alerting: "Never send an alert to someone who isn't fully equipped and empowered to fix it." [17]
On owning your code: "Promote an ownership mentality over the full software life cycle." [17]
On the human element of on-call: "On-call is a relentlessly human-scale process... everything is contextual, everything is contingent." [19]
On the first step to better on-call: "If there is no owner for a rotation... it's no one's job or everyone's job which means it's no one's job and that that just doesn't work." [19]
On the simplicity of building systems: "If you can get away with a monolith and a LAMP stack and a handful of monitoring checks, you should absolutely do that." [4]

Learn more:

Lessons from Charity Majors

On Observability

On Engineering Management

On the Engineer/Manager Pendulum

On Team Culture and High-Performing Teams

On On-Call and Operations

Written by Antoine Buteau

Lessons from Esther Crawford

Lessons from Byron Deeter