> ## Content Index
> Fetch the complete content index at: https://www.antoinebuteau.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Daily Digest - 2026-09-22
- URL: https://www.antoinebuteau.com/daily-digest-2026-09-22/
- Published: 2026-09-23T09:40:08.000Z
- Updated: 2026-09-23T09:40:08.000Z
- Description: Hamel Husain and Shreya Shankar explain why writing metrics too early causes AI product teams to fail, and share a practical workflow for finding and fixing hidden failures with coding agents.
- Author: Antoine Buteau
- Tags: Digest

## In this digest

1. [Advanced evals: How to find (and fix) hidden AI failures in your product](#digest-item-1)
2. [Inside “ChatGTM”: Cursor’s Internal Sales AI Used by their 400+ Sales Org](#digest-item-2)
3. [Eval rubrics that drive your AI strategy](#digest-item-3)
4. [Good Product Manager / Bad Product Manager in AI](#digest-item-4)
5. [How OpenAI Built GPT-Live](#digest-item-5)
6. [What it took to build an agent our CSMs actually trust](#digest-item-6)
7. [Jev introduces a new shape of LLM—System One, aka Decision Models](#digest-item-7)
8. [How XBOW’s team built one of the most sophisticated internal GTM systems I’ve seen](#digest-item-8)
9. [Your company needs a pricing constitution.](#digest-item-9)
10. [Software Factories: It’s All About the Data (Architecture)](#digest-item-10)
11. [Hear me out: Agentic TPMs (Technical Program Managers.) I had...](#digest-item-11)
12. [The current balance of power in open models](#digest-item-12)
13. [\[AINews\] Xiaomi MiMo-V2.6-Pro 1T-A42B: the new top Open Weights model, trained for $3M](#digest-item-13)
14. [Jev and AI Checkpoints: Using Decision Models to Wrangle Agent Work](#digest-item-14)
15. [I Think I’ve Been Using GPT-6 Astra Wrong](#digest-item-15)

## Themes from yesterday

- **The Transition from Generative Loops to Microsecond Decision Models**: The rollout of TypeSafe's Jev and similar decision models reflects a shift away from slow, token-by-token text generation for structured business logic. Instead of waiting for conversational output, teams are using these architectures to run instant classifications, ticket routing, and checkpoint validations at roughly an eightieth of the cost.
- **Hierarchical Multi-Model Orchestration Over Monolithic Prompting**: Rather than relying on a single prompt to handle every stage of development, engineering teams are organizing models into tiers. Frontier models act as executive planners that establish strict guardrails and coordinate work, delegating parallel coding tasks to cheaper worker models with automated testing and review.
- **The Industrialization of Internal AI Operations and Revenue Engines**: Companies like Cursor, Atlan, and XBOW are building unified internal operating platforms for their revenue and customer teams. Their deployments demonstrate that dependable agent systems require persistent long-term memory, direct queries to live company databases, and shifting human staff from routine tasks into supervisory roles.
- **Post-Training Reinforcement Learning and the Open Weights Frontier Shift**: Xiaomi's low-cost MiMo-V2.6-Pro release and recent congressional testimony show Chinese open-weight models pulling ahead in academic citations and developer adoption. These developments suggest that structured reinforcement learning environments and evaluation compute are replacing massive pretraining datasets as the primary driver of frontier capabilities.
- **Behavioral Eval Rubrics and Checkpoint Data as the Core AI Moat**: Across customer products and software factories, teams are finding that qualitative evaluation contracts across four areas (Outcome, Trajectory, Experience, Governance) and runtime telemetry matter more than simple metric dashboards. Capturing verified checkpoints and agent traces turns daily production history into a proprietary competitive moat.

## 1\. **Advanced evals: How to find (and fix) hidden AI failures in your product** — Lenny's Newsletter (Substack)

- Why read: Hamel Husain and Shreya Shankar explain why writing metrics too early causes AI product teams to fail, and share a practical workflow for finding and fixing hidden failures with coding agents.
- Summary: Based on advisory work with over 50 AI companies, the authors argue that finding errors is the evaluation equivalent of product discovery. When teams rush into quantitative metrics, they often hit criteria drift: once they look at actual interaction traces, their definition of good output inevitably changes. To evaluate traces at scale without losing nuance, teams can combine human judgment with an active learning workflow, using coding agents to group failures, pull diverse samples, and draft annotations for human review. Developers can also generate synthetic queries across set difficulty dimensions, like task clarity and user persona, to test unreleased features on edge cases. Doing structured error discovery before locking down dashboards helps engineers optimize for actual customer satisfaction rather than vanity benchmarks.
- [Read more](https://substack.com/app-link/post?post%5Fid=216168140&publication%5Fid=10845&ref=antoinebuteau.com)

## 2\. **Inside “ChatGTM”: Cursor’s Internal Sales AI Used by their 400+ Sales Org** — Brendan J Short (thesignal.club)

- Why read: Brendan Short explains how Cursor built ChatGTM, an internal sales tool that triples qualified outbound meetings and cuts onboarding time for sales reps in half.
- Summary: Cursor built ChatGTM to eliminate the 45 minutes sales reps spent before every meeting digging through Hex dashboards, Gong recordings, and Salesforce records. The system sends tailored morning digests, drafts personalized outreach messages with custom landing pages, and pulls real-time customer data to create live battlecards during meetings. Architecturally, ChatGTM skips static context stores and precomputed embeddings, querying raw APIs and data warehouses on demand using live tool calls. While Cursor built this internally thanks to their team's context-engineering background, leadership advises most enterprise revenue teams to buy commercial agent tools rather than spend months building custom software. The main operational takeaway is treating sales reps as product co-builders, giving non-technical domain experts direct control to design the prompts and workflows used across a 400-person sales team.
- [Read more](https://www.thesignal.club/p/chatgtm?ref=antoinebuteau.com)

## 3\. **Eval rubrics that drive your AI strategy**  — Sandhya (X)

- Why read: Sandhya explains why behavioral evaluation rubrics are the most critical yet neglected tool in AI product management, replacing static design reviews with scalable quality standards.
- Summary: Because generative agents produce unpredictable, non-deterministic user sessions, traditional visual design critiques cannot ensure product quality. A clear evaluation rubric acts as a contract covering four areas: Outcome, Trajectory, Experience, and Governance. Beyond measuring baseline quality, documented rubrics enable three strategic moves: automated self-improvement loops, safe routing to cheaper open-source models, and fine-tuning custom models on proprietary production data. New classification models like Jev now make it practical to run these behavioral checks across 100 percent of user traffic at low cost. Product managers need to write explicit rubrics into their initial specifications, because foundation models cannot align to business intentions without clear evaluation criteria.
- [Read more](https://twitter.com/sandhya/status/2102092474911207454/?rw%5Ftt%5Fthread=True&ref=antoinebuteau.com)

## 4\. **Good Product Manager / Bad Product Manager in AI** — Malika Aubakirova (X)

- Why read: Malika Aubakirova adapts Ben Horowitz's classic management memo for the AI era, detailing how product managers must navigate new bottlenecks around compute limits, capability jumps, and user trust.
- Summary: In traditional SaaS, engineering time was the primary bottleneck; AI flips this by making prototyping and code generation virtually free. A good AI product manager acts as the product's leader by clearly defining the capability target rather than getting distracted by the technical mechanics of routing layers and agent harnesses. Bad product managers chase public benchmark leaderboards, shipping experimental tweaks that score well with raters but perform poorly in actual production. Good product managers also avoid building unnecessary separate apps, choosing instead to deliver value directly inside existing channels like messaging threads and phone calls. By focusing on customer problems and tangible revenue rather than competitor feature matrices, product leaders ensure that automated coding speed translates into lasting business value.
- [Read more](https://twitter.com/MaikaThoughts/status/2102417119866769822/?rw%5Ftt%5Fthread=True&ref=antoinebuteau.com)

## 5\. **How OpenAI Built GPT-Live** — Alex Xu (X)

- Why read: Alex Xu breaks down the systems architecture behind OpenAI's full-duplex voice model, showing how separating speech generation from deeper reasoning eliminates conversational delay.
- Summary: Traditional voice assistants rely on multi-step pipelines or turn-based speech models that cut users off or pause awkwardly. OpenAI's GPT-Live-1 solves this with a full-duplex architecture that processes and streams audio tokens continuously on a fixed 80-millisecond clock. To maintain natural conversation during complex questions, the system splits speaking from thinking: a lightweight voice model handles small talk and conversational cadence, while a frontier model runs deeper reasoning and web searches in the background. The serving infrastructure routes audio over a low-latency WebRTC path with a custom single-roundtrip protocol, keeping persistent sessions directly on GPUs with smooth handoffs between instances. System architects building real-time voice applications need to design around high-percentile tail latency, because any processing hiccup causes immediate, audible glitches.
- [Read more](https://twitter.com/alexxubyte/status/2102421991240401113/?rw%5Ftt%5Fthread=True&ref=antoinebuteau.com)

## 6\. **What it took to build an agent our CSMs actually trust** — Dhruv Saharya (X)

- Why read: Dhruv Saharya describes how Atlan turned an early chatbot that kept failing into Relay, an autonomous customer success agent that team members trust to build presentation decks in seconds.
- Summary: After three earlier attempts failed due to context limits, high token costs, and rigid multi-agent orchestrators, Atlan successfully deployed Relay by onboarding the agent like a human hire with foundational documentation, company values, and organizational charts. The engineering team found that high-stakes account risks rarely surface during formal Zoom calls, so they set the agent to monitor unstructured signals across support tickets and informal late-night Slack messages. With persistent long-term memory instead of one-off prompts, Relay catches subtle organizational shifts, such as noticing when an executive promotion leaves a key renewal without an owner. The platform spread rapidly inside the company once customer success managers began building their own custom skills, cutting presentation prep time from two hours to thirty seconds. For operations teams, this rollout shows that employee trust comes from domain-specific memory architecture and human review rather than raw model capability.
- [Read more](https://twitter.com/dhruvsaharya/status/2102401563637711180/?rw%5Ftt%5Fthread=True&ref=antoinebuteau.com)

## 7\. **Jev introduces a new shape of LLM—System One, aka Decision Models** — Simon Willison (simonwillison.net)

- Why read: Simon Willison introduces TypeSafe AI's Jev, explaining how non-autoregressive decision models skip text generation to deliver sub-second classifications at a fraction of the cost.
- Summary: TypeSafe AI released Jev, establishing a category of fast "System One" decision models that take in unstructured text and output typed numbers, confidence scores, and probability distributions rather than generated text. The architecture charges only for input tokens at roughly four cents per million with free outputs, making it substantially cheaper than standard generative language models. Developers can evaluate multiple yes/no questions, multiple-choice distributions, and numerical rating scales in parallel against a single shared state. This setup fits high-throughput operational tasks like spam detection, data labeling, ticket triage, and search reranking. However, Willison notes that removing chain-of-thought text makes the model an opaque black box, requiring thorough evaluation suites to catch hidden algorithmic biases before using it in high-stakes workflows.
- [Read more](https://simonwillison.net/2026/Sep/21/jev/?ref=antoinebuteau.com)

## 8\. **How XBOW’s team built one of the most sophisticated internal GTM systems I’ve seen** — Brendan J Short (thesignal.club)

- Why read: Brendan Short explains how cybersecurity startup XBOW built a comprehensive sales platform with just three people, shifting sales development reps into workflow managers.
- Summary: Founded by the creator of GitHub Copilot, cybersecurity startup XBOW applied software engineering principles to revenue operations across their entire funnel. Following a rule to build proprietary intelligence while buying standard infrastructure, a three-person team used Claude Code to build an integrated web app and Slack assistant called Bolt. The system queues up prioritized daily tasks, translates technical security findings into sales materials for CISOs, tracks technical evaluation milestones, and runs eight specialized agents to coach reps on call recordings. Instead of relying on static vector embeddings, Bolt connects directly to live Slack, Gong, and Salesforce APIs to maintain accurate, real-time data. With routine prospecting and outreach automated, XBOW's business development reps now act as agent managers who monitor output quality and handle customer relationships.
- [Read more](https://www.thesignal.club/p/xbow?ref=antoinebuteau.com)

## 9\. **Your company needs a pricing constitution.** — Elena's Growth Scoop (Substack)

- Why read: Elena Verna explains why faster software shipping requires a clear pricing constitution and a cross-functional council to prevent continuous internal gridlock over monetization.
- Summary: Because AI coding assistants help teams ship software much faster, every new feature risks triggering prolonged debates over paywalls, tier placements, and credit metering. To resolve this friction, Elena Verna recommends establishing an official pricing constitution governed by a monetization council that acts like an internal supreme court. Her core rules are clear: monetize genuine customer value rather than artificial product friction, charge for finished outcomes rather than input usage, and keep collaboration and sharing free to support organic growth loops. Companies should keep packaging simple, offering four clear plans based on one or two primary value metrics to reduce buying friction. Growth teams should also set firm channel boundaries: self-serve handles initial customer acquisition, while direct sales focuses exclusively on managing complex enterprise expansions.
- [Read more](https://substack.com/app-link/post?post%5Fid=216880745&publication%5Fid=1435249&ref=antoinebuteau.com)

## 10\. **Software Factories: It’s All About the Data (Architecture)**  — Josh Rosen (X)

- Why read: Josh Rosen examines the data architectures behind software factories, showing how logging agent execution traces and runtime checkpoints builds an enduring competitive advantage.
- Summary: As code generation becomes commoditized, an engineering organization's main competitive advantage shifts to accumulating detailed telemetry on how its software is built. Modern software factories use OpenTelemetry traces to document every file inspection, tool call, test retry, and human edit across thousands of autonomous runs. By running LLM judges across batches of traces, platforms like Warp identify persistent migration failures and turn past development history into reusable evaluation benchmarks. Connecting agent runs to the broader software development lifecycle through semantic models links code generation directly to deployment stability and incident rates. In addition, runtime checkpoints require agents to produce verifiable artifacts and attested relationships throughout their execution loop, turning tacit model reasoning into structured data that continuously improves system performance.
- [Read more](https://twitter.com/JoshARosen/status/2102457068640711142/?rw%5Ftt%5Fthread=True&ref=antoinebuteau.com)

## 11\. **Hear me out: Agentic TPMs (Technical Program Managers.) I had...** — Steve Yegge (X)

- Why read: Steve Yegge argues that polite, persistent AI technical program managers are the lowest-risk, highest-impact way to introduce autonomous AI teammates into enterprise operations.
- Summary: While coding agents remain confined to the software development lifecycle and support agents handle narrow ticket queues, technical program manager agents can operate across an entire company. Running on the idea of making progress through polite persistence, these agents coordinate projects over email and Slack by tracking milestones, documenting blocker dependencies, and nudging stakeholders for status updates. Because they have no executive authority and make no code commits, their operational risk is minimal, allowing organizations to deploy them without major engineering overhauls. As they work, these agents break down communication silos by logging unwritten tribal knowledge, organizational dependencies, and duplicate work into a centralized company brain. This rollout gives enterprise teams a safe way to practice human-agent collaboration while providing leadership with immediate operational visibility.
- [Read more](https://twitter.com/Steve%5FYegge/status/2102268319919423606/?rw%5Ftt%5Fthread=True&ref=antoinebuteau.com)

## 12\. **The current balance of power in open models** — Nathan Lambert (interconnects.ai)

- Why read: Nathan Lambert's congressional briefing shows how Chinese open-weight models have taken a clear lead over American open models across downloads, benchmark scores, and enterprise adoption.
- Summary: In testimony before Congress, Nathan Lambert presented data showing that Chinese open-weight models have pulled ahead of American alternatives in both research and commercial environments. On Hugging Face, Chinese models account for over three billion downloads (double the total for American open models) and drive more than 80 percent of open-model token consumption on inference routing platforms. Benchmark tests show that leading releases from Moonshot AI and Z.ai trail closed American frontier models by only two to five months, whereas American open models lag frontier systems by up to nine months. In research, Alibaba's Qwen family is cited in 30 percent of machine learning arXiv preprints, passing Meta's Llama in citations. Because commercial tools like Cursor and Harvey increasingly rely on open Chinese weights, Lambert argued that managing AI safety requires funding domestic open model training rather than relying on impractical regulations.
- [Read more](https://www.interconnects.ai/p/the-current-balance-of-power-in-open?ref=antoinebuteau.com)

## 13\. **\[AINews\] Xiaomi MiMo-V2.6-Pro 1T-A42B: the new top Open Weights model, trained for $3M** — AINews (Substack)

- Why read: AINews details Xiaomi's release of MiMo-V2.6-Pro, an open omnimodal model that reached the top of major benchmark rankings through an efficient, three-million-dollar training run.
- Summary: Consumer electronics maker Xiaomi took the top spot on the Artificial Analysis Intelligence Index with MiMo-V2.6-Pro, an open-weight omnimodal Mixture-of-Experts model with over one trillion total parameters and 42 billion active parameters. Demonstrating high capital efficiency, the entire final reinforcement learning post-training phase finished in just 130 hours for roughly 2.6 million dollars on JAX and TPU infrastructure. Xiaomi also open-sourced its reinforcement learning tooling, training recipes, and multi-task evaluation environments covering software engineering, cyber vulnerability reproduction, and visual development. These results suggest that automated reinforcement learning setups and grader compute are replacing raw pretraining datasets as the primary lever for scaling model capabilities. The newsletter also notes growing adoption of TypeSafe's Jev decision models, which developers are using inside WebMCP tools and LangSmith judge pipelines for low-cost runtime supervision.
- [Read more](https://substack.com/app-link/post?post%5Fid=216854596&publication%5Fid=1084089&ref=antoinebuteau.com)

## 14\. **Jev and AI Checkpoints: Using Decision Models to Wrangle Agent Work** — Josh Rosen (X)

- Why read: Josh Rosen outlines a "System 1.5" architecture, using TypeSafe's Jev decision model to run sub-second semantic checks at key agent checkpoints.
- Summary: As reasoning models grow more capable, their inner reasoning chains become harder to audit, making passive observation inadequate for governing complex agent workflows. ThruWire addresses this by establishing verifiable checkpoints: agents explore freely between milestones, but must submit concrete, schema-validated artifacts with provable provenance before moving forward. Adding TypeSafe's Jev decision model introduces fast, inexpensive semantic checks at these checkpoints, evaluating subjective quality criteria without adding the latency of a full generative LLM. Beyond serving as pass-fail policy gates, Jev continuously labels passing artifacts with observations like user sentiment, confusion, and requirement drift. Testing this architecture on meeting workflows, the team converted transcripts into interactive applications that allow operators to track emotional friction and inspect the supporting dialogue.
- [Read more](https://twitter.com/JoshARosen/status/2102107649705734413/?rw%5Ftt%5Fthread=True&ref=antoinebuteau.com)

## 15\. **I Think I’ve Been Using GPT-6 Astra Wrong** — Brad Groux (X)

- Why read: Brad Groux describes a multi-model workflow that uses GPT-6 Astra as an executive planner while delegating coding tasks to parallel worker models.
- Summary: Relying on a frontier model like GPT-6 Astra as an all-in-one coder that plans, writes, and reviews complex code simultaneously leads to bottlenecks and inconsistent changes. Groux reorganized his workflow by treating Astra as an executive planner operating at high reasoning to review project documentation, prioritize backlogs, and establish behavioral guardrails. Astra then assigns implementation tasks concurrently to three to five isolated GPT-5.6 Sol threads running at medium reasoning, with each worker confined to a single branch and clear test requirements. To maintain progress without manual oversight, Astra reviews the completed diffs, runs local validation checks, automatically merges passing pull requests, and runs periodic background check-ins. This tiered structure helps software teams work through backlogs in parallel, maintain architectural discipline, and ship significantly more code without constant supervision.
- [Read more](https://twitter.com/BradGroux/status/2102455199583334625/?rw%5Ftt%5Fthread=True&ref=antoinebuteau.com)