Hamel Husain is a machine-learning engineer and AI consultant who has worked at GitHub, DataRobot, and Airbnb. In a direct interview, he discusses how teams can evaluate and improve AI products. — Humanloop Interview.

On AI Product Development and Evaluation
- Define What Better Means: AI engineering requires a practical way to tell whether an application is improving on its users' tasks. Husain recommends measuring real failures and iterating, rather than merely changing models or prompts. — Field Guide to AI Products.
- Prioritize Process Over Tools: Husain warns that choosing new models, vector databases and frameworks without a process for inspecting outcomes leaves teams unable to tell whether changes help. — Field Guide to AI Products.
- Treat Evals as an Improvement System: Husain describes evaluation as a loop of tests, trace inspection, debugging and changes—not a single score on a dashboard. — Your AI Product Needs Evals.
- Start with Error Analysis: Husain recommends examining actual user interactions, taking open-ended notes about failures and grouping recurring problems before deciding what to test. — Field Guide to AI Products.
- Let Observed Failures Set the Eval Agenda: His field guide starts with real traces and bottom-up failure categories, then builds tests for the problems that matter most in the application. — Field Guide to AI Products.
- Fix Early Failures First: The coauthored eval guide recommends recording the first failure in a trace because upstream errors can cause downstream ones; the same heuristic helps prioritize large traces. — AI Evals Guide.
- Fix Obvious Gaps Before Building Evals: Husain and Shankar advise correcting plainly missing prompt instructions or product behavior first, then investing automated evaluators in failures that persist. — AI Evals Guide.
- Read Real Outputs Before Scoring Them: Husain argues that reading traces reveals bugs and user-specific failure modes that generic dashboards miss; the review interface should make this easy. — Field Guide to AI Products.
- A Spreadsheet Can Start the Analysis: For an initial error review, Husain describes writing notes beside conversations in a spreadsheet and using simple pivot tables to count recurring problems. — Field Guide to AI Products.
- Avoid Vanity Metrics: Husain warns that generic helpfulness dashboards can create false confidence while users still fail at their real tasks. Choose measures grounded in observed application problems. — Field Guide to AI Products.
- Use Application-Specific Evals: Husain advises against a generic evaluation framework as the measure of every AI product; tests should reflect the particular tasks and risks of the application. — Your AI Product Needs Evals.
- Expect Evaluation Criteria to Evolve: Husain describes criteria drift: reviewing outputs helps teams discover and sharpen what they actually want the system to do. Rubrics should be revised as understanding improves. — Field Guide to AI Products.
On the Mindset of an AI Engineer
- Run Experiments to Learn: Husain recommends planning a cadence of experiments and learning rather than promising specific AI features before the team knows what works. — Field Guide to AI Products.
- AI Engineering Is Already a Real Role: In a direct interview, Husain said the AI-engineer role already existed and argued that model specialists also benefit from application-development skills. — Humanloop Interview.
- Software Engineers Are Building AI Products: Husain observed that many people building LLM applications were full-stack software engineers rather than traditional machine-learning engineers, broadening who could create AI products. — Humanloop Interview.
- ML Specialists Benefit from App Skills: Husain argues that machine-learning practitioners can gain by learning how to build end-to-end applications, while acknowledging that application engineering itself requires craftsmanship. — Humanloop Interview.
- Practice Reading Logs: Husain says practitioners develop an intuition for AI failure modes by repeatedly examining logs and conducting error analysis; that skill is not replaced by dashboards. — Humanloop Interview.
- Observe Before Freezing Categories: Husain's bottom-up method begins with open-ended notes on real interactions; categories and counts are developed only after recurring failure patterns emerge. — Field Guide to AI Products.
- Understand the User Task: Husain argues that product evals must measure success on actual user tasks, rather than generic model benchmarks or abstract scores. — Field Guide to AI Products.
- Inspect the Actual Prompt: Husain's essay argues that the final prompts and API calls generated by LLM frameworks should be inspectable so teams can judge whether an abstraction adds value or accidental complexity. — Show Me the Prompt.
- Improve Retrieval Instead of Declaring RAG Dead: Husain and Ben Clavié criticize simplistic one-vector RAG but argue that retrieval remains essential; they point to coverage, diversity and relevance as useful goals. — Stop Saying RAG Is Dead.
On Building with LLMs
- Calibrate LLM Judges Against Humans: Husain's judge workflow begins with domain-expert pass/fail judgments and critiques, then tests the automated judge against those human labels. — LLM-as-a-Judge Guide.
- Use Critiques to Improve Judge Prompts: When a judge disagrees with an expert, Husain uses the expert's critiques and examples to revise its prompt, then rechecks agreement on labeled cases. — LLM-as-a-Judge Guide.
- Cover the Product's Edge Cases: Husain's Rechat case shows how a widening feature surface exposed failures that ad hoc prompt fixes could not systematically track; scoped tests and synthetic scenarios helped cover them. — Your AI Product Needs Evals.
- Break Evaluation into Scenarios: Husain recommends decomposing an application's features into concrete scenarios, then writing scoped assertions and examples for those scenarios. — Your AI Product Needs Evals.
- Question Complexity Without Measurement: Husain warns that elaborate RAG, routing or agent architecture can consume effort while a team lacks a way to tell whether the product works; begin with failure analysis. — Field Guide to AI Products.
- Choose Fine-Tuning for the Right Problem: Husain says fine-tuning remains useful for learning syntax, style and rules, whereas RAG is better for supplying current context; systematic evals should guide that choice. — Is Fine-Tuning Still Valuable?.
- Check Tokenization at Training and Inference: Husain shows that visually similar prompts can tokenize differently when training and inference construct them by different methods; decoding token IDs can expose the mismatch. — Tokenization Gotchas.
On Career and Learning
- Write From Direct Experience: Husain says technical writing has helped people find his work, and he reserves his own writing for topics he has experienced. He recommends contributing useful insights to others' work and showing up consistently. — Technical Writing Strategies.
- Build Useful Open-Source Tools: Husain says he led the team that created CodeSearchNet, an early project in code understanding. The example illustrates his practice of building tools that help developers work with machine learning. — Your AI Product Needs Evals.
- Combine ML and Application Engineering: Husain argues that machine-learning practitioners benefit from application-development skills so they can help build and evaluate complete AI products, not only models. — Humanloop Interview.
- Teach with Exercises and Feedback: Husain and Shreya Shankar's AI Evals course presents applied exercises, examples and office hours as part of teaching evaluation to engineers and product managers. — AI Evals Course.
- Learn from Repeated Implementation Mistakes: Husain's field guide distills recurring problems he encountered across more than 30 AI product implementations, especially teams choosing tools before understanding failures. — Field Guide to AI Products.
- Automate Data-Science Workflows with Actions: In a coauthored tutorial, Husain demonstrates using GitHub Actions for data-science workflows, including notebook publishing and reproducible automation. — GitHub Actions Tutorial.
- Learn Enough Kubernetes to Unblock ML Work: Husain argues that basic Kubernetes knowledge can help ML engineers debug infrastructure and communicate with platform teams, while warning that it is overkill for simple prototypes. — Kubernetes for ML Engineers.
- Bootstrap Testing Before User Data Arrives: Husain recommends using realistic synthetic inputs when an AI product lacks sufficient production traces, then testing them against actual system constraints and refining from what fails. — Field Guide to AI Products.
- Validate the Statistic Behind Judge Agreement: Husain warns that raw human–judge agreement can mislead when classes are imbalanced and recommends reporting true-positive and true-negative rates against human labels. — LLM-as-a-Judge Guide.
- Put Domain Expertise Into Evaluation: Husain argues that effective AI evaluation needs people who understand the real user task to make and critique judgments, rather than relying entirely on generic engineering metrics. — LLM-as-a-Judge Guide.
- Make Judgments Small and Clear: Husain recommends starting LLM-judge design with clear pass/fail decisions and detailed critiques instead of ambiguous multi-point scales. — LLM-as-a-Judge Guide.
- Turn Expert Critiques into Criteria: Husain's judge-building process asks domain experts to explain why an output passes or fails; those critiques become examples and guidance for the evaluator prompt. — LLM-as-a-Judge Guide.
Learn more:
- Why Your AI Product Needs Evals — Humanloop Interview
- Your AI Product Needs Evals — Hamel Husain
- A Field Guide to Rapidly Improving AI Products — Vanishing Gradients Interview
- A Field Guide to Rapidly Improving AI Products — Hamel Husain
- Using LLM-as-a-Judge for Evaluation — Hamel Husain
- Stop Saying RAG Is Dead — Hamel Husain and Ben Clavié
- AI Evals for Engineers & PMs — Hamel Husain and Shreya Shankar
- AI Evals: Everything You Need to Know — Hamel Husain and Shreya Shankar
- Is Fine-Tuning Still Valuable? — Hamel Husain