Avery Pennarun is the co-founder and CEO of Tailscale, an engineer who spent years at Google Fiber, and a prolific open-source contributor known for tools like sshuttle, bup, and redo. On his long-running blog, apenwarr, he writes extensively about systems design, management math, and the curse of over-engineering. This compilation organizes his best ideas on why simplicity scales, how organizations really make decisions, and why making the easy things easy is the hardest engineering challenge of all.

Part 1: The Magic of Simplicity

  1. On the primary directive: "Everybody wants to build the super fancy sounding stuff that scales to a billion, but it makes everything harder. Tailscale’s motivation is: let’s just build things to make the easy stuff easy." — Source: Stratechery Interview
  2. On the 90 percent rule: "If you look at most of the work you have to do to launch an internal dashboard, it’s not building the dashboard. It’s making sure people can log in, making sure attackers can’t reach it... Tailscale makes that 90% of the work go away." — Source: The Kubelist Podcast
  3. On saying no: "Today my job is keeping the product small and coherent, picking the right 'no’s,' and making sure we never break what customers already depend on." — Source: Web Summit 2025
  4. On making users feel something: "The value of a Net Promoter Score isn't the number itself, but the realization that people rate products based on emotions. A 'lukewarm' response is a failure to make the user feel anything." — Source: apenwarr.ca
  5. On bottom-up adoption: "We tackled security by staying bottom-up and adding the minimum enterprise surface area... without turning the product into a maze." — Source: Security Cryptography Whatever
  6. On magical technology: "Magic in technology is just things we know exist and work well, but which we can't quite explain the internal mechanics of yet." — Source: apenwarr.ca
  7. On invisible systems: "If we do our job right, the Internet feels like it used to — simple, safe, and personal — except it works for everyone this time." — Source: The Cloud Gambit
  8. On developer alignment: "Security buyers want power; developers want simplicity. Balancing those is hard. You have to make the secure thing the easiest thing to do." — Source: Screaming in the Cloud
  9. On the burden of legacy: "We glued modern identity to WireGuard for a Canadian bank because they needed 2FA for a legacy Windows app no one wanted to rewrite. We realized this 'one weird trick' could be repackaged for everyone." — Source: Tailscale Blog

Part 2: Systems Thinking and Architecture

  1. On defining systems design: "Systems design is boxes and arrows engineering. As you advance, your work shifts from fixing bugs to solving business problems by designing how components interact." — Source: apenwarr.ca
  2. On real engineering: "Much of what we call software development is not yet engineering. True engineering involves monitoring error rates, SLOs, and designing systems that assume and work around failure." — Source: apenwarr.ca
  3. On Layer 3 supremacy: "It’s surprising when you go all the way down to layer three. It works; it carries everything. You don’t need a special integration for every kind of app." — Source: Tech Talks Daily
  4. On batch processing: "If you want to process 40 billion log lines a day on a single machine, keep it simple, batch everything, and avoid unnecessary distributed complexity." — Source: apenwarr.ca
  5. On distributed systems: "Markets, governments, and the cloud are all distributed systems. Without some form of centralized regulation or coordination, they always go awry." — Source: apenwarr.ca
  6. On organizational drag: "Every layer of review makes you 10x slower. Organizational friction is often the primary bottleneck in software delivery." — Source: apenwarr.ca
  7. On review latency: "The only way to sustainably go faster in a complex system is to reduce the latency of reviews and coordination." — Source: apenwarr.ca
  8. On semantic communication: "LLMs represent a shift from technical standards with rigid protocols to semantic ones, where systems can communicate through meaning rather than just strict syntax." — Source: apenwarr.ca
  9. On interfaces: "Good systems design focuses heavily on interfaces and critical boundaries. That is where systems succeed or fail." — Source: apenwarr.ca
  10. On the cloud's tide: "The cloud infrastructure is a rising tide, increasingly absorbing traditional on-premise networking responsibilities." — Source: Nanog 91

Part 3: The Myth of "Google Scale"

  1. On copying giants: "Everyone was copying Google’s scale — but most people don’t need Google scale. If infrastructure makes it harder to prototype, you’ve done it wrong." — Source: Screaming in the Cloud
  2. On naming his company: "I wanted something that was the opposite of 'Internet-scale.' I’m more interested in solving the long tail of small problems that everyone has." — Source: Tailscale Blog
  3. On the DevOps tax: "Working at Google showed me the power of massive infrastructure, but also the massive DevOps tax required to maintain it." — Source: FLOSS Weekly
  4. On small hops: "Small, low-energy steps are good for stability but they can be the enemy of change when a system needs a major shift." — Source: apenwarr.ca
  5. On simulated annealing: "Annealing is the enemy of change. Some changes simply don't lend themselves to small steps; you have to break things to find the global optimum." — Source: apenwarr.ca
  6. On targeted audiences: "Build products bottom-up for individual engineers rather than top-down for executives. The engineers feel the pain of scale first." — Source: The Business of Open Source
  7. On over-engineering: "People apply big-data solutions to medium-data problems, guaranteeing high latency and massive server costs for no real benefit." — Source: apenwarr.ca
  8. On rack cabling: "You can use the simulated annealing algorithm to determine the most efficient way to route cables in a server rack to minimize total cable length and clutter." — Source: apenwarr.ca
  9. On the long tail: "The industry focuses on the head of the curve—the billions of users. But the long tail is where all the actual business problems live." — Source: Slush 2025

Part 4: Management, Estimation, and "Agile"

  1. On hoping: "Hope often looks exactly like a strategy—meetings, speeches, post-mortems—and for some, it actually works because they hope really hard. But it is nonsense." — Source: apenwarr.ca
  2. On forecasting failure: "Software estimates fail because they are fundamentally optimistic. No matter how much buffer you add, the math behind project scheduling always trends toward delays." — Source: SREcon18 Europe
  3. On story points: "Agile is often misunderstood as stupid management fluff, but the math behind small batches and visible priorities actually works." — Source: apenwarr.ca
  4. On Deming: "Management requires understanding the difference between special causes, like isolated bugs, and common causes, which are systemic performance issues." — Source: apenwarr.ca
  5. On management's true job: "Management is not about telling people what to do. It is about making people think the way you do so they can make the right decisions autonomously." — Source: apenwarr.ca
  6. On the decision gap: "You have to shrink the gap between the people experiencing the problems and the people making the decisions." — Source: SREcon18 Europe
  7. On executive function: "An executive's job isn't to set strategy in a vacuum. It is to define culture and values, and to ratify the decisions made by those closest to the facts." — Source: apenwarr.ca
  8. On reorganizations: "Frequent reorgs have deep social effects on a company. They disrupt the informal networks that actually get work done." — Source: apenwarr.ca
  9. On predicting outcomes: "Real leadership involves a complicated thought process that improves productivity and predicts outcomes, rather than just demanding arbitrary deadlines." — Source: apenwarr.ca
  10. On simulating teams: "If you build an ad-hoc engineer simulator, you quickly see that queuing theory explains why most software projects back up and fail." — Source: SREcon18 Europe

Part 5: Smart People, Ego, and Leadership

  1. On the trap of intellect: "Smart people have a unique problem: the ability to convincingly rationalize nearly anything, even terrible decisions." — Source: apenwarr.ca
  2. On impostor syndrome: "Impostor syndrome is valuable. It acts as a check against overconfidence. People with it realize that a logical proof of their own smartness isn't enough to guarantee they are right." — Source: apenwarr.ca
  3. On inconvenient facts: "Highly logical people often isolate themselves from the inconvenient facts of the real world because the real world doesn't always compile." — Source: apenwarr.ca
  4. On the engineering ladder: "Traditional career ladders often fail misfit engineers who are brilliant at building things but don't fit the pure business problem-solving mold of a Staff or Principal engineer." — Source: apenwarr.ca
  5. On people problems: "At some point in your startup, you start to realize that all of your problems are people problems because there is no other kind of problem anymore." — Source: Web Summit 2025
  6. On Wi-Fi and ego: "Making Wi-Fi reliable at scale requires immense hardware and software engineering. It humbles you quickly because physics doesn't care about your clean code." — Source: apenwarr.ca
  7. On company purpose: "As a company scales, leadership must actively nurture the original purpose, or the structure will slowly overwrite the mission." — Source: apenwarr.ca
  8. On vendor trust: "You cannot blindly trust vendor benchmarks or hope a third-party service will remain reliable without active verification and contingency planning." — Source: apenwarr.ca
  9. On the gift economy: "Open source operates on a gift economy. Understanding the social dynamics of that economy is just as important as understanding the code." — Source: FLOSS Weekly

Part 6: Startups, Markets, and "Enshittification"

  1. On market segments: "Avoid the 80/20 myth that 80% of people use 20% of features. Startups should nail just one use case and solve the specific needs of a single market segment first." — Source: apenwarr.ca
  2. On strange failures: "The nature of startup success and failure is strange. The common advice to 'just keep typing' doesn't always work if the system is misaligned." — Source: apenwarr.ca
  3. On enshittification: "Companies switch strategies out of fear when growth slows, often sacrificing the positive feedback loop of user love for short-term control." — Source: apenwarr.ca
  4. On billionaire math: "Startup exits often don't lead to the infinite money people imagine. Modest tastes are the actual requirement for true financial independence." — Source: apenwarr.ca
  5. On word of mouth: "Build trust and design onboarding that scales through word-of-mouth rather than relying on traditional enterprise sales motions." — Source: Slush 2025
  6. On traditional pitch decks: "We scaled Tailscale without a traditional pitch deck. If the product works and solves a real pain point, the usage graphs make the pitch for you." — Source: Web Summit 2025
  7. On TAM theater: "I don’t love spreadsheet TAM theater. Practically, our market is any device that should talk to any other device securely, which keeps expanding." — Source: Stratechery Interview
  8. On acquisitions: "I get asked a lot about whether Tailscale is looking to be acquired. Most acquisitions don’t make products better; they make roadmaps fuzzier." — Source: Tech Talks Daily
  9. On crossing the chasm: "You cross the chasm not by adding enterprise features early, but by making the core tool so indispensable that the enterprise has to adopt it." — Source: The Business of Open Source

Part 7: Open Source, Networks, and Building Blocks

  1. On legacy code survival: "The disappointingly ongoing success of WvDial, a modem dialer I co-authored in 1998, proves that simple tools survive long after they become obsolete." — Source: apenwarr.ca
  2. On backups: "bup takes the git packfile format and uses rolling checksums for deduplication. It works because it borrows an incredibly robust data structure." — Source: apenwarr.ca
  3. On build systems: "File timestamps (mtime) are a terrible and unreliable flag indicator for build systems. That is why tools like redo are necessary." — Source: apenwarr.ca
  4. On poor man's tools: "sshuttle is a poor man's VPN. It gained unexpected popularity because people just wanted to forward traffic without configuring a kernel module." — Source: apenwarr.ca
  5. On early networking: "I started writing Linux drivers in grade 10 so my sister and I could share a single dial-up modem. The core problem hasn't changed much since." — Source: Tailscale Blog
  6. On the WireGuard advantage: "WireGuard is faster and simpler than IPsec or OpenVPN. The data plane was solved; the missing piece was the control plane." — Source: Security Cryptography Whatever
  7. On NAT traversal: "Punching holes through firewalls using STUN and DERP allows devices to connect peer-to-peer. It is what makes the network feel magical." — Source: Wikipedia - Tailscale
  8. On identity as the network: "We made the network something you log into using SSO rather than something you configure with IP addresses." — Source: Pulse 2.0
  9. On incremental updates: "A good backup or transfer protocol should work efficiently over dumb HTTP servers, minimizing the required intelligence on the remote end." — Source: apenwarr.ca
  10. On protocol tunneling: "When you build a tunnel, you don't want TCP over TCP. You want a system that understands the layers and routing natively." — Source: apenwarr.ca

Part 8: Security, AI, and The Future of Work

  1. On accidental AI infrastructure: "We are now the network that powers AI, by accident. Developers took Tailscale with them to new AI companies so they wouldn't have to think about networking." — Source: Web Summit 2025
  2. On forcing simplicity: "LLMs might finally push us to make simpler, more secure software—not because it’s noble, but because it’s easier for the AI to handle." — Source: Stratechery Interview
  3. On zero trust: "You achieve Zero Trust by integrating security and connectivity into a single package so that users accidentally make their systems more secure." — Source: Screaming in the Cloud
  4. On enterprise readiness: "Secure, reliable connections are the boring but essential layer behind getting AI models ready for enterprise deployment." — Source: Web Summit AI Networking Talk
  5. On the real internet: "Tailscale makes the Internet work the way I thought it worked, before I learned how routing and firewalls actually broke everything." — Source: The Cloud Gambit
  6. On Proof of Concepts: "The uncomfortable truth behind enterprise POCs is that they are rarely about the technology; they are about proving to a committee that the vendor won't get them fired." — Source: Screaming in the Cloud
  7. On key management: "Manual key exchange is why older VPNs failed. Automatically rotating and distributing public keys based on user identity is the only way it scales securely." — Source: Security Cryptography Whatever
  8. On the future of protocols: "As machines talk to machines more frequently, we will need protocols that assume extreme latency and intermittent connectivity as the default state." — Source: apenwarr.ca
  9. On the singularity: "The technological singularity isn't just about faster computers. It is about removing the human coordination latency that prevents systems from evolving." — Source: apenwarr.ca