
Lessons from Kevin Mandich
Kevin Mandich’s work spans machine learning and AI product engineering. After leading AI at Sprig, he became Principal AI Engineer at Fourier Health. This profile examines supported practices from his interviews, engineering writing and research. — Fourier Health: Kevin Mandich.
Part 1: The Evolution of Machine Learning
- On early model constraints: Earlier product ML often required specialized models and carefully prepared inputs; Mandich contrasts that work with today’s lower barriers. — Mixpanel: Building with ML and AI.
- On the pace of AI advancement: Sprig moved from expensive custom training and human annotation toward hosted models that reduced the work needed to build. — Mixpanel: Building with ML and AI.
- On transitioning from statistical ML: Mandich recalls using simpler statistical models before neural-network advances expanded the products his teams could build. — Mixpanel: Building with ML and AI.
- On evaluating new models: Test candidate models on actual customer tasks; switch when improvement holds up. — First Round: Building Product After LLMs.
- On legacy NLP techniques: Word counts and topic matching can miss whether similar language expresses a complaint, switching cost or feature request. — Sprig: Thematic Clustering.
- On building before ChatGPT: Sprig initially combined customized models with human review because qualitative interpretation is subjective. — First Round: Building Product After LLMs.
- On continuous learning: Mandich recommends following AI developments to understand what is newly possible, while acknowledging how hard staying current has become. — Sprig: AI Experts Q&A.
- On the democratization of AI: Hosted models lowered the technical expertise needed to prototype products; creativity and useful product design still matter. — Mixpanel: Building with ML and AI.
Part 2: Product Development with LLMs
- On post-LLM product strategy: Sprig CEO Ryan Glasgow puts customer value ahead of AI novelty. — First Round: Building Product After LLMs.
- On qualitative insights: Open-text analysis is a scaling problem: researchers must interpret varied responses, not merely count words or detect topics. — Sprig: Thematic Clustering.
- On conversational interfaces: Mandich describes a study-creation assistant that turns a product owner’s research goal into a proposed survey. — Mixpanel: Building with ML and AI.
- On prompt engineering: Mandich described employees sharing prompt ideas across departments. — First Round: Building Product After LLMs.
- On handling edge cases: In his later engineering work, potentially risky record merges go to human review before downstream processing. — Kevin Mandich: Structured AI Outputs.
- On data formatting: Mandich finds modern models more tolerant of imperfect inputs, reducing some of the preparation previously needed for product prototypes. — Mixpanel: Building with ML and AI.
- On context windows: His later pipeline uses overlapping context windows and page-level classification rather than repeatedly supplying every page with the whole document. — Kevin Mandich: Structured AI Outputs.
- On deterministic vs. probabilistic features: Qualitative analyses can differ without either being uniquely correct. — First Round: Building Product After LLMs.
Part 3: Structuring AI Teams
- On the AI Squad: Mandich’s AI squad integrated design, product and engineering to own features end to end. — First Round: Building Product After LLMs.
- On hiring ML engineers: Mandich values ML engineers who can also integrate backend systems. — First Round: Building Product After LLMs.
- On upskilling teams: Glasgow credits Mandich with training engineers on OpenAI APIs. — First Round: Building Product After LLMs.
- On rapid prototyping: Mandich describes early feasibility prototypes followed by customer testing. — First Round: Building Product After LLMs.
- On the role of early hires: Early Sprig ML required building substantial continuous-training infrastructure. — First Round: Building Product After LLMs.
- On managing AI expectations: Glasgow addressed skepticism through pilots using customers’ own data. — First Round: Building Product After LLMs.
Part 4: Data Security and Threat Intelligence
- On the volume of threats: His co-invented email-security design automatically builds sender models from previously observed messages, reducing reliance on manual sender assessment. — Agari: Message Authenticity and Risk Assessment.
- On phishing detection: The patent explains why known-malware and signature checks alone can miss new messages impersonating a trusted sender. — Agari: Message Authenticity and Risk Assessment.
- On adversarial environments: An apparently familiar sender can be deceptive: the design assesses whether the claimed identity matches the message’s actual origin. — Agari: Message Authenticity and Risk Assessment.
- On false positives: The disclosed design uses risk thresholds to decide which messages to deliver, flag or block; threshold choices shape handling. — Agari: Message Authenticity and Risk Assessment.
- On feature extraction in security: Sender IP addresses, sending infrastructure and timing patterns provide evidence for authenticity alongside other message attributes. — Agari: Message Authenticity and Risk Assessment.
- On identity deception: A sender’s observed history supplies a baseline for assessing whether a new message genuinely comes from that identity. — Agari: Message Authenticity and Risk Assessment.
Part 5: Extracting Signal from Product Data
- On user feedback: Manual spreadsheet coding makes rich open-text feedback expensive to interpret at scale; automation can increase researchers’ leverage. — Sprig: Thematic Clustering.
- On quantitative vs. qualitative: A response’s meaning depends on its question: the same short phrase can express praise or a request for improvement. — Sprig: Thematic Clustering.
- On real-time analysis: Mandich describes extending analysis beyond open text to recordings, session replays and heat maps across the research lifecycle. — Mixpanel: Building with ML and AI.
- On thematic clustering: Useful themes combine subject matter with intent; sentiment and urgency can help make the resulting groups actionable. — Sprig: Thematic Clustering.
- On actionable insights: Mandich’s clustering approach aims for decision-relevant takeaways, separating similar topics when users mean different things. — Sprig: Thematic Clustering.
- On synthesizing video: He identifies video recordings and session replays as additional material for analysis, not just open-text survey responses. — Mixpanel: Building with ML and AI.
- On longitudinal data: Mandich describes analyzing survey responses as they arrive. — First Round: Building Product After LLMs.
Part 6: From Aerospace to Algorithms
- On engineering fundamentals: His doctoral modeling makes assumptions explicit: small-perturbation approximations stop applying once disturbances grow too large. — UC San Diego: Stability of Gas-Fluidized Beds.
- On experimental design: His physical experiments compared measured pressure, bed height and fluidization velocity with theory to check the apparatus. — UC San Diego: Experimental Fluidized-Bed Study.
- On transitioning fields: Mandich applied ML across legal technology, cybersecurity and computer vision. — First Round: Building Product After LLMs.
- On handling noise: His experimental work calibrated the vessel’s mechanical response before interpreting wall-vibration signals as particle motion. — UC San Diego: Experimental Fluidized-Bed Study.
- On academic rigor: His dissertation compares model predictions with established empirical work and measurements, rather than treating equations alone as validation. — UC San Diego: Stability of Gas-Fluidized Beds.
Part 7: Overcoming AI Skepticism
- On user trust: Sprig’s AI features included thumbs-up/down and written feedback; Mandich cautions that low response volume complicates evaluation. — Mixpanel: Building with ML and AI.
- On the black box problem: His later pipeline maps retrieved claims back to precise source-document locations so reviewers can inspect the underlying evidence. — Kevin Mandich: Structured AI Outputs.
- On mitigating hallucinations: Retrieved evidence feeds structured outputs in his later system; grounding is part of the pipeline, not a guarantee against hallucinations. — Kevin Mandich: Structured AI Outputs.
- On gradual rollouts: Glasgow tested alternative AI features with customers before choosing. — First Round: Building Product After LLMs.
- On human-in-the-loop: Mandich frames automated analysis as leverage for researchers, with themes that people can use to guide product decisions. — Sprig: Thematic Clustering.
- On clear boundaries: Mandich warns that provider outages and model updates can disrupt workflows optimized around another company’s technology. — Sprig: AI Experts Q&A.
- On handling errors gracefully: His later QA process uses human scoring and reviews more new-customer, new-specialty and historically weaker output components. — Kevin Mandich: Structured AI Outputs.
- On data privacy: Sprig states it does not train on response data; for MCP connections, the receiving model provider’s own agreement governs retention and training. — Sprig: Data Security Questions.
Part 8: The Future of Applied AI
- On specialized agents: Mandich imagines agents connecting research collection, follow-up studies and proposed product changes into a broader workflow. — Mixpanel: Building with ML and AI.
- On the commodity of models: Mandich expects foundational models to commodify and sees greater control of that layer as a possible response to provider dependence. — Sprig: AI Experts Q&A.
- On continuous intelligence: Glasgow argues for owning proprietary data and the surrounding workflow. — First Round: Building Product After LLMs.
- On generative UI: Glasgow envisioned AI handing research into specifications, design and code. — First Round: Building Product After LLMs.
- On engineering productivity: Glasgow says Sprig encouraged coding-assistant use; no tenfold gain is established. — First Round: Building Product After LLMs.
- On the ultimate goal: Glasgow’s aim was shifting people toward higher-level work, not eliminating involvement. — First Round: Building Product After LLMs.