Lessons from Hillel Wayne

Lessons from Hillel Wayne

Hillel Wayne is a software consultant and author who translates formal methods like TLA+ for working programmers. He is best known for the Crossover Project, which demonstrated that software development is just as messy and constrained as traditional engineering. This profile covers his arguments on practical system design, tooling, and why correctness must be an explicit choice.

Part 1: Formal Methods and TLA+

1. On Debugging vs. Specification: "Weeks of debugging can save you hours of TLA+." — Source: [Hillel Wayne's Blog]
2. On the Purpose of TLA+: It allows developers to write precise blueprints of systems and find complex concurrency bugs in minutes rather than days. — Source: [Practical TLA+]
3. On Finding Unknown Bugs: TLA+ isn't just for mathematicians; it's a practical tool for finding the bugs you didn't even know your architecture had by exhaustively checking state spaces. — Source: [Learn TLA+]
4. On PlusCal: PlusCal acts as a gateway to formal methods because its C-like syntax feels significantly more familiar to traditional programmers than pure TLA+. — Source: [Learn TLA+]
5. On the Flossing Analogy: "Software correctness is a lot like flossing... everyone knows root canals are worse than buggy software, yet most people don't floss daily because it's fiddly and annoying." — Source: [Computer Things Newsletter]
6. On Pedagogical Barriers: Powerful formal verification tools don't have to be hard to learn; the primary barrier to industry adoption is pedagogical, not technical. — Source: [Hillel Wayne's Blog]
7. On Safety-Critical Systems: Formal methods are not exclusively for domains like space exploration; they are high-ROI tools for everyday distributed systems. — Source: [Practical TLA+]
8. On Alloy: The Alloy language is highly effective for domain modeling and verifying complex, graph-like structural properties in software systems. — Source: [Hillel Wayne's Blog]
9. On Tool Usability: The adoption gap for software correctness tools is primarily a usability problem, not an issue of developer apathy toward quality. — Source: [Computer Things Newsletter]
10. On Planning-Driven Development: Developers should spend more time specifying systems upfront rather than perpetually debugging implementation errors that stem from flawed initial designs. — Source: [Practical TLA+]

Part 2: The Crossover Project: Software vs. Traditional Engineering

1. On Software's Uniqueness: "To assume that software is uniquely unpredictable is a special kind of arrogance." — Source: [The Crossover Project]
2. On Real Engineering: Almost every traditional engineer who transitioned to software considers software engineering to be a legitimate, real engineering discipline. — Source: [The Crossover Project]
3. On Shared Messiness: Traditional engineering disciplines are often just as iterative and filled with unpredictable unknown unknowns as software development. — Source: [GOTO Conferences]
4. On Chemical Engineering: "If chemical engineering started now, people would refuse to call it engineering. And they'd be wrong to refuse." — Source: [The Crossover Project]
5. On the Electrician Distinction: There is a fundamental difference between simply writing code and engineering software, akin to the difference between an electrician and an electrical engineer. — Source: [Hillel Wayne's Blog]
6. On the Rigor Myth: Traditional engineering is not always inherently more rigorous; those fields also face constant pressures to cut corners and fix things later. — Source: [The Crossover Project]
7. On Upfront Planning: Software engineering can learn heavily from traditional engineering by mandating more upfront requirements gathering and thinking before executing. — Source: [GOTO Conferences]
8. On Version Control: Git and version control systems are the single greatest operational advantage software has that traditional engineering desperately lacks. — Source: [The Crossover Project]
9. On Community Sharing: Software's culture of open-source knowledge sharing and rapid collaboration is vastly superior to the siloed environments found in physical engineering firms. — Source: [The Crossover Project]
10. On Constraints: All engineering, whether physical or digital, is fundamentally about applying scientific principles to solve practical problems under strict constraints. — Source: [GOTO Conferences]

Part 3: Empirical Software Engineering and Research

1. On Software Beliefs: "Almost everything in software is a belief... it is something we have experience about, it is something we have opinions on, but it's not something we have hard data on." — Source: [GOTO Conferences]
2. On Scientific Rigor: "You don't study the research to convince others. You study the research because you'd rather be technically correct than happy." — Source: [Computer Things Newsletter]
3. On Charismatic-Driven Development: Much of what developers consider established best practices is actually driven by charisma, authority, and intuition rather than peer-reviewed data. — Source: [GOTO Conferences]
4. On the Parachute Problem: Some software practices are so obviously beneficial that there are few formal studies proving them, similar to the lack of double-blind studies on parachutes. — Source: [GOTO Conferences]
5. On Primary Sources: "Rule of thumb: the more steps you get away from the primary source the more corrupted everything gets." — Source: [Computer Things Newsletter]
6. On Language Wars: Claims that certain programming languages inherently produce more reliable code than others have largely been debunked by rigorous replication studies. — Source: [Hillel Wayne's Blog]
7. On Human Factors: Organizational structure and sleep deprivation have a demonstrably greater impact on software quality than specific technological choices. — Source: [GOTO Conferences]
8. On the Difficulty of Research: Empirical Software Engineering is exceptionally difficult because software development is a highly varied, human-centric activity rather than a hard science. — Source: [Computer Things Newsletter]
9. On Misquoting Knuth: People frequently misquote Donald Knuth on premature optimization by conveniently ignoring his critical qualifiers: small efficiencies and 97% of the time. — Source: [Computer Things Newsletter]
10. On the ESE Turf War: The academic community around software research is prone to intense disputes because the very definition of what constitutes a valid software study remains contested. — Source: [Hillel Wayne's Blog]

Part 4: Software Design and Architecture

1. On Expensive Mistakes: "Good design docs save time because they move the expensive mistakes earlier, when they are still cheap." — Source: [Hillel Wayne's Blog]
2. On Undoing Design: "The real waste is not spending an extra day writing a serious design doc. The real waste is spending eighteen months undoing a design that nobody challenged properly." — Source: [Hillel Wayne's Blog]
3. On Defenses of Design: Codebases desperately need documents that defend why specific, non-obvious choices were made, rather than just describing what the system does. — Source: [Hillel Wayne's Blog]
4. On Chesterton's Fence: Many strange design decisions in legacy code are rational responses to forgotten constraints; a Defense of Design prevents future developers from blindly breaking them. — Source: [Hillel Wayne's Blog]
5. On High-Fidelity Specs: Formal specifications function as high-fidelity design docs that allow engineers to test architectural integrity before implementation begins. — Source: [Practical TLA+]
6. On Correctness as a Choice: Correctness should be treated as a deliberate, upfront architectural design choice rather than an unattainable ideal chased during QA. — Source: [Hillel Wayne's Blog]
7. On State Machines: Composing system behavior through explicit state machines provides a clearer, logically sound foundation for complex application architectures. — Source: [Hillel Wayne's Blog]
8. On Vague Systems: If a system's architecture is difficult to explain in plain language, the underlying design model is likely fundamentally flawed. — Source: [Hillel Wayne's Blog]
9. On the Value of Modeling: Spending time modeling an architecture mathematically forces developers to confront edge cases they would otherwise ignore until production. — Source: [Learn TLA+]

Part 5: Logic for Programmers

1. On Mathematical Utility: Logic is arguably the most practically useful branch of mathematics for software engineers, yet it is rarely taught in a context connected to daily coding. — Source: [Logic for Programmers]
2. On Simplification: Applied logic allows developers to systematically simplify convoluted conditionals and impenetrable database queries. — Source: [Logic for Programmers]
3. On Property-Based Testing: Understanding formal logic is the fundamental key to writing effective property-based tests that can automatically generate hundreds of edge cases. — Source: [Logic for Programmers]
4. On Requirements Gathering: Formal logic can quickly expose hidden gaps, deep contradictions, and unspoken assumptions in basic product requirements. — Source: [Logic for Programmers]
5. On API Stability: Logical models and invariants help ensure that changes to an established API remain strictly and mathematically backwards-compatible. — Source: [Logic for Programmers]
6. On Decision Tables: Decision tables serve as an excellent, low-barrier entry point into applying strict formal logic to everyday programming tasks. — Source: [Hillel Wayne's Blog]
7. On Probabilistic Systems: Tools like PRISM are necessary for mathematically modeling the complex realities of retry logic and failure rates in distributed networks. — Source: [Hillel Wayne's Blog]
8. On the Core Disconnect: The gap between academic logic and industry coding is a massive failure of instruction, not a lack of practical applicability. — Source: [Logic for Programmers]
9. On Everyday Application: You do not need to be building spacecraft to benefit from applying basic propositional logic to your everyday function parameters. — Source: [Logic for Programmers]

Part 6: Friction, Testing, and Tooling

1. On Compounding Friction: "Friction compounds with itself: two setbacks are more than twice as bad as one setback." — Source: [Computer Things Newsletter]
2. On Anticipating Friction: "Friction is inevitable and impossible to fully remove. I don't think it's possible to even fully anticipate." — Source: [Computer Things Newsletter]
3. On Integration Tests: "Integration tests are just vibes." — Source: [Twitter/X]
4. On Testing Boundaries: When developers claim to be doing integration testing, they are often just testing arbitrary components together without rigorously defining the properties being verified. — Source: [Hillel Wayne's Blog]
5. On Tool Rejection: Developers do not reject better tools because they are apathetic about quality; they reject them because the tools have completely hostile user experiences. — Source: [Computer Things Newsletter]
6. On Misplaced Blame: The software industry frequently blames individual developer negligence for poor quality instead of addressing the terrible usability of its verification tooling. — Source: [Computer Things Newsletter]
7. On the True Cost of Bugs: The time spent manually chasing elusive concurrency bugs in production always eclipses the time it would take to formally model the system beforehand. — Source: [Practical TLA+]
8. On Tooling Mindset: We must stop treating advanced verification tools as esoteric academic curiosities and start treating them as standard, required developer utilities. — Source: [Hillel Wayne's Blog]
9. On Domain Falsehoods: Programmers believe many structural falsehoods about real-world domains because they fail to properly verify the constraints of their own data models. — Source: [Hillel Wayne's Blog]

Part 7: The Realities of AI and Industry Hype

1. On AI Hype: "LinkedIn needs a feature to filter out all AI hype written by people under 25." — Source: [Computer Things Newsletter]
2. On Junior Predictions: "If you graduated college three years ago you don't have enough software engineering experience to make sweeping predictions about how AI is going to destroy it." — Source: [Computer Things Newsletter]
3. On the Hard Parts: The true difficulty of software engineering lies in understanding opaque requirements and untangling legacy systems, which AI fundamentally underestimates. — Source: [Computer Things Newsletter]
4. On The Main Character: "Each day on twitter there is one main character. The goal is to never be it." — Source: [Twitter/X]
5. On Silver Bullets: There is no single language, framework, or methodology that inherently guarantees safer code in the absence of proper logical design. — Source: [Hillel Wayne's Blog]
6. On Industry Dogma: The tech industry is far too quick to follow charismatic figures blindly, leading to the widespread adoption of practices that actively ruin software architecture. — Source: [Hillel Wayne's Blog]
7. On the We Are Special Myth: The persistent belief that software is a completely unique endeavor prevents the industry from adopting deeply proven practices from older fields. — Source: [The Crossover Project]
8. On Maturing the Discipline: Despite the current messiness of the industry, the slow but steady adoption of better design practices points toward software becoming a mature engineering discipline. — Source: [The Crossover Project]
9. On Hype Cycles: Software development is plagued by cyclical hype because it lacks the empirical foundations necessary to cleanly reject bad ideas. — Source: [GOTO Conferences]

Part 8: Communication, Writing, and History

1. On Fuzzy Thinking: "The writing is vague because the thinking is vague. Bad writing is often a design smell." — Source: [Hillel Wayne's Blog]
2. On Recording Decisions: "If a decision matters enough to debate, it matters enough to record." — Source: [Hillel Wayne's Blog]
3. On the Value of History: Software engineering suffers from amnesia; studying the history of computing prevents us from constantly reinventing broken wheels. — Source: [Computer Things Newsletter]
4. On Clear Specifications: A specification is simply a way to force yourself to communicate your system's exact intent to a perfectly pedantic audience: the computer. — Source: [Learn TLA+]
5. On Architectural Narratives: The best technical writing does not just describe a system's current state; it tells the narrative of the constraints that shaped it. — Source: [Hillel Wayne's Blog]
6. On Nuance: Technical arguments online fail because they strip away the operational nuance required to understand why a specific tradeoff was made. — Source: [Computer Things Newsletter]
7. On the Cost of Ignorance: Teams waste immense amounts of time arguing over implementation details because they failed to document their core architectural assumptions in writing. — Source: [Hillel Wayne's Blog]
8. On Pedagogy: We write terrible technical documentation because we optimize for reference rather than optimizing for the psychological journey of the learner. — Source: [Learn TLA+]
9. On Legacy Code: Legacy code is often just historical writing that we have lost the contextual dictionary to read properly. — Source: [Hillel Wayne's Blog]