Visual summary of operating lessons from Jonathan Ross.

Lessons from Jonathan Ross

Jonathan Ross created Google's original Tensor Processing Unit (TPU) and is now the founder and CEO of Groq. He developed the Language Processing Unit (LPU) based on the premise that AI inference requires different hardware than training. This profile organizes his commentary on silicon design, deterministic computing, and AI hardware economics.

Part 1: The Google Years and TPU Origins

  1. On the TPU's beginnings: "The TPU started as a 20% project because I sat near the speech recognition team and saw how constrained they were by available compute." — Source: [All About Circuits]
  2. On early hardware iterations: "The first-generation TPU was a successful disaster. It was hard to build and program, but it proved that specialized silicon could drastically outperform general-purpose chips for specific AI tasks." — Source: [EE Times]
  3. On building at scale: "When you build hardware at Google, you are building for a scale that almost no one else on the planet has to worry about. That changes every engineering assumption you make." — Source: [Masters of Scale]
  4. On leaving Google: "I left Google to start Groq because I realized the architecture required to solve inference globally was fundamentally different from what hyperscalers were optimizing for internally." — Source: [Bloomberg Technology]
  5. On the inertia of software: "No one ever thinks of solving their problems with hardware, but sometimes hardware is the only way to physically bypass a software bottleneck." — Source: [All About Circuits]
  6. On early AI constraints: "The speech team at Google knew their models could be better, but they couldn't deploy them because the latency on existing hardware was simply too high for real-time voice interaction." — Source: [20VC]
  7. On career risk: "I dropped out of high school and NYU because I didn't want to cap my earning opportunity by graduating. The same logic applied to leaving a comfortable role at Google to build a new chip from scratch." — Source: [Business Insider]
  8. On hardware design culture: "Silicon design used to be incredibly slow and risk-averse. We approached the TPU with a software engineering mindset, focusing on rapid iteration and aggressive deadlines." — Source: [EE Times]
  9. On proving the concept: "Once the TPU was deployed, the difference was so stark that it shifted the entire company's perspective on how infrastructure should be built." — Source: [Masters of Scale]

Part 2: Hardware Philosophy and Silicon Realities

  1. On the tyranny of memory: "In AI hardware, compute is almost free. It is the movement of data between memory and the processor that costs you time and energy." — Source: [Groq Blog]
  2. On GPU complexity: "GPUs were designed to push pixels to a screen, which requires massive parallel processing. They were never originally intended for the sequential, token-by-token generation required by large language models." — Source: [20VC]
  3. On High Bandwidth Memory (HBM): "Relying on HBM forces you to constantly shuttle data back and forth. If you can keep the model weights entirely on-chip in SRAM, you eliminate the single biggest bottleneck in inference." — Source: [EE Times]
  4. On non-determinism: "GPUs suffer from jitter because they rely on dynamic scheduling and complex memory hierarchies. You never know exactly how long a specific operation will take." — Source: [All About Circuits]
  5. On deterministic execution: "If you know exactly when data will arrive at each stage of the chip, you don't need complex scheduling hardware. You can strip all that out and replace it with more raw compute." — Source: [All About Circuits]
  6. On the compiler's role: "Our philosophy was to move the complexity out of the silicon and into the software compiler. The compiler orchestrates the movement of every piece of data perfectly before the program even runs." — Source: [Groq Blog]
  7. On static scheduling: "Static scheduling means the chip operates like a perfectly choreographed dance. There are no traffic jams because every data packet knows its exact path and timing." — Source: [TheCUBE]
  8. On silicon efficiency: "We throw away the parts of the chip that manage unpredictable workloads. For AI inference, the workload is highly predictable, so you only need silicon dedicated to the math." — Source: [Business Insider]
  9. On architectural purity: "Sometimes the best hardware design is simply having the courage to delete features that legacy architectures force you to keep." — Source: [The Eric Ries Show]

Part 3: Founding Groq and the LPU Architecture

  1. On the founding vision: "We didn't set out to build a slightly better GPU. We set out to build a Language Processing Unit because language generation is a fundamentally different mathematical problem." — Source: [Bloomberg Technology]
  2. On the LPU moniker: "An LPU is designed specifically to handle the sequential nature of text generation. It focuses on pushing single batches of data through as fast as physically possible." — Source: [Groq Blog]
  3. On competing with giants: "You cannot beat NVIDIA by playing their game. You have to change the rules of the game to focus entirely on inference speed, where their architecture is naturally disadvantaged." — Source: [20VC]
  4. On SRAM advantages: "By using massive amounts of on-chip SRAM, an LPU can access data orders of magnitude faster than a GPU pulling from external memory." — Source: [All About Circuits]
  5. On scaling up: "Because our chips are deterministic, networking them together is incredibly simple. We don't need complex switches to handle unpredictable traffic; the compiler maps out the entire network." — Source: [EE Times]
  6. On the startup struggle: "Building a silicon startup is notoriously brutal because the capital requirements are massive and the feedback loops take years, not weeks." — Source: [Masters of Scale]
  7. On early customer skepticism: "At first, people didn't believe our latency numbers. We had to build physical demonstrations and let them type into the models themselves before they understood." — Source: [TheCUBE]
  8. On naming Groq: "The name Groq comes from Robert A. Heinlein's concept of 'grokking'—to understand something so intuitively that it becomes part of you. That is how we want AI to interact with human queries." — Source: [Groq Blog]
  9. On single-core design: "Unlike GPUs with thousands of tiny cores, an LPU acts as a single massive core. This eliminates the overhead of keeping thousands of cores synchronized." — Source: [All About Circuits]
  10. On focusing solely on inference: "We explicitly decided not to optimize for training. Trying to be good at both training and inference guarantees you will be suboptimal at both." — Source: [20VC]

Part 4: The Physics of Inference vs. Training

  1. On the truck vs. van analogy: "GPUs are 18-wheeler trucks. They are perfect for hauling massive amounts of data at once for training. LPUs are last-mile delivery vans, optimized for delivering single packages rapidly." — Source: [Bloomberg Technology]
  2. On batch sizes: "Training relies on massive batch sizes to achieve efficiency. Inference, especially real-time user interaction, often requires a batch size of one. GPUs struggle terribly with batch size one." — Source: [Groq Blog]
  3. On the prefill phase: "Processing the prompt—the prefill phase—is highly parallel and GPUs are good at it. It is the decode phase, generating the next word, that requires the sequential speed of an LPU." — Source: [20VC]
  4. On hybrid solutions: "The future of the data center will likely involve GPUs handling the initial prompt processing and immediately handing off the token generation to LPUs." — Source: [TheCUBE]
  5. On sequential bottlenecks: "You cannot generate the third word of a sentence until you have generated the second. It is a strictly sequential process that cannot be brute-forced with parallel cores." — Source: [EE Times]
  6. On utilization rates: "If you use a GPU for real-time inference, most of its compute cores sit idle while waiting for data from memory. It is a massive waste of energy." — Source: [All About Circuits]
  7. On the volume of inference: "The compute spent on training a model is a fixed cost. The compute spent on inference scales infinitely with user adoption and will eventually dwarf training costs." — Source: [Business Insider]
  8. On power consumption: "Because we don't move data back and forth to HBM, the power consumption per generated token on an LPU is a fraction of what it is on traditional hardware." — Source: [Groq Blog]
  9. On Mixture-of-Experts (MoE): "MoE models only activate a small portion of the network for any given token. This fragments memory access, which is a nightmare for GPUs but easily handled by the LPU's SRAM." — Source: [The Eric Ries Show]

Part 5: The Economics of Compute and the AI Bubble

  1. On AI demand: "We are not in an AI bubble. We are in the early stages of a fundamental shift in how the global economy consumes compute power." — Source: [Business Insider]
  2. On the Jevons Paradox: "As the cost of inference drops, demand will not shrink. It will explode. Cheaper compute makes entirely new categories of AI applications economically viable." — Source: [20VC]
  3. On hyperscaler strategies: "Every major tech company will eventually have to build their own silicon. Relying entirely on a single vendor for the core engine of the future economy is structurally untenable." — Source: [Bloomberg Technology]
  4. On capital expenditure: "The billions being poured into AI infrastructure today are just the down payment. The actual economic return will come when inference becomes ubiquitous and invisible." — Source: [TheCUBE]
  5. On hardware margins: "Hardware margins have historically been squeezed, but in the AI era, the company that provides the most efficient compute effectively taxes the entire software ecosystem." — Source: [20VC]
  6. On the cost of delay: "In capital markets, speed is everything. If your financial model runs ten times faster, you don't just save time—you make entirely different trades." — Source: [Business Insider]
  7. On market expansion: "The goal isn't to take market share from existing players; the goal is to unlock the 90% of AI applications that are currently too expensive to run." — Source: [The Eric Ries Show]
  8. On energy constraints: "The ultimate limit on AI growth won't be capital or silicon; it will be access to power grids capable of supporting gigawatt data centers." — Source: [Bloomberg Technology]
  9. On compute as a commodity: "Compute is transitioning from a specialized IT resource into a fundamental macroeconomic commodity, like oil or electricity." — Source: [Masters of Scale]
  10. On startup survival: "To survive in the silicon space against incumbents, your performance delta cannot be 20%. It has to be an order of magnitude, or customers won't bother rewriting their code." — Source: [20VC]

Part 6: Determinism, Latency, and Speed

  1. On the speed of thought: "For an AI to feel natural in conversation, the latency must drop below human perception. It needs to respond faster than you can realize you've stopped speaking." — Source: [TheCUBE]
  2. On engineering focus: "Our entire engineering philosophy is that every word matters. The moment the user has to wait for a token to appear, the magic of the interaction is broken." — Source: [Groq Blog]
  3. On voice interfaces: "Voice AI is the ultimate test of inference speed. You can get away with a three-second delay when generating an image, but a three-second pause in a voice conversation is agonizing." — Source: [20VC]
  4. On cognitive flow: "Fast inference isn't just a technical metric; it changes human behavior. When an AI responds instantly, users treat it as a collaborator rather than a search engine." — Source: [The Eric Ries Show]
  5. On predictable latency: "Because the LPU is deterministic, we can guarantee exactly how many milliseconds a request will take. You cannot build reliable enterprise software on top of hardware that has unpredictable latency spikes." — Source: [EE Times]
  6. On token generation rates: "We are pushing towards generating hundreds of tokens per second per user because that is the threshold required for AI to read ahead, verify facts, and correct itself before showing you the output." — Source: [Bloomberg Technology]
  7. On debugging determinism: "Software engineers love deterministic hardware because if a bug happens once, it will happen exactly the same way every single time. It makes profiling incredibly precise." — Source: [All About Circuits]
  8. On scaling speed: "You cannot easily buy lower latency by throwing more GPUs at a single prompt. The physics of moving data across a cluster puts a hard floor on how fast a GPU can generate a word." — Source: [Groq Blog]
  9. On instant translation: "True real-time language translation requires the AI to begin generating the translated sentence before the speaker has even finished their thought. That is only possible with ultra-low latency." — Source: [TheCUBE]

Part 7: Geopolitics and the Control of Compute

  1. On national security: "The countries that control compute will control AI. You cannot have compute without energy, and you cannot have sovereignty without compute." — Source: [20VC]
  2. On the new arms race: "We have moved from a nuclear arms race to a compute arms race. The strategic reserves of the 21st century are silicon manufacturing and energy infrastructure." — Source: [Bloomberg Technology]
  3. On sovereign AI: "Nations are waking up to the fact that routing all their citizens' data through servers in another country to be processed by a foreign AI model is a massive security vulnerability." — Source: [TheCUBE]
  4. On domestic supply chains: "Building domestic silicon capacity is no longer just an economic priority; it is the absolute baseline requirement for remaining an independent technological power." — Source: [Business Insider]
  5. On the export of silicon: "Regulating the export of advanced chips is the modern equivalent of regulating the flow of oil. It dictates who gets to participate in the future economy." — Source: [20VC]
  6. On energy as a bottleneck: "You can print money to buy more chips, but you cannot instantly print a nuclear power plant. The geopolitics of AI will eventually be dictated by the geopolitics of base-load energy." — Source: [Bloomberg Technology]
  7. On regulatory leverage: "Governments are realizing that regulating AI software is nearly impossible, but regulating the physical hardware where it runs is entirely feasible." — Source: [The Eric Ries Show]
  8. On global infrastructure: "The map of global power is being redrawn based on where the massive data centers are located and which power grids can sustain them." — Source: [EE Times]
  9. On intellectual property: "Hardware architectures are much harder to steal and replicate than model weights. A physical chip requires a sprawling, visible supply chain." — Source: [Masters of Scale]
  10. On defense applications: "In a defense context, AI latency is the difference between intercepting a threat and being hit. Fast inference is literally a matter of life and death." — Source: [TheCUBE]

Part 8: The Future of Agentic AI and Open Source

  1. On agentic workflows: "AI is moving from a chat interface to an agentic model. Instead of answering a question, the AI will spawn fifty sub-agents to research, verify, and execute a task in the background." — Source: [20VC]
  2. On the cost of agents: "Agentic AI requires the model to talk to itself hundreds of times before presenting an answer to the user. This makes fast, cheap inference the absolute prerequisite for the agentic economy." — Source: [Bloomberg Technology]
  3. On open-source models: "Open-source AI is catching up to proprietary models much faster than anyone anticipated. The differentiator won't be the model itself, but the hardware it runs on." — Source: [The Eric Ries Show]
  4. On model commoditization: "If model weights become open and commoditized, the value in the AI ecosystem shifts violently downward into the infrastructure layer." — Source: [Business Insider]
  5. On developers and open source: "Providing developers with instantaneous access to open-source models allows them to iterate at speeds that were previously impossible, accelerating the entire field." — Source: [Groq Blog]
  6. On AI speaking to AI: "AI likes to use AI. Soon, the vast majority of inference requests will not come from human keyboards, but from other AI agents coordinating tasks." — Source: [TheCUBE]
  7. On verifying truth: "When inference is fast enough, an AI can generate an answer, instantly query another model to fact-check it, and correct itself before the user ever sees a hallucination." — Source: [20VC]
  8. On democratizing compute: "If we only allow a few massive corporations to afford inference, we stifle innovation. Driving down the cost of compute is the only way to keep the AI ecosystem decentralized." — Source: [Masters of Scale]
  9. On the end goal: "The ultimate goal isn't just to build faster chips. The goal is to make compute so abundant and frictionless that developers stop thinking about hardware constraints entirely." — Source: [All About Circuits]