Visual summary of operating lessons from Karina Nguyen.

Lessons from Karina Nguyen

Karina Nguyen has worked across product engineering and AI research at Anthropic and OpenAI. Her work includes Claude 3 post-training and evaluations, as well as the research-and-product collaboration behind ChatGPT Canvas. — Latent Space Interview.

Part 1: Constitutional AI and Alignment

  1. Bring public input into model values: Anthropic and the Collective Intelligence Project tested a constitution drawn from public input rather than only company-written principles. Karina Nguyen is credited with helping train and evaluate the resulting models; the team described the work as preliminary. — Anthropic Collective Constitutional AI.
  2. Use both broad and specific principles: In a paper Karina coauthored, a short general principle produced some desired behavior in the tested dialogue models, while more detailed constitutions offered finer control over particular harms. The two approaches served different purposes. — Specific versus General Principles for Constitutional AI.
  3. Use AI feedback for targeted behavior changes: Karina describes using synthetically generated preference data and constitutional principles to target specific refusal behaviors when human feedback is costly. The method still requires careful evaluation of unintended effects. — Stanford CS25 Lecture.
  4. Train models to critique and revise: The Constitutional AI method uses model-generated critiques and revisions during supervised training, followed by AI-generated preferences in reinforcement learning. This is team research background, not a verified Karina-only quotation. — Constitutional AI Paper.
  5. Balance helpfulness against over-refusal: Karina describes debugging benign Claude refusals by comparing model versions, classifying failure cases, and shaping responses to interpret requests charitably while maintaining boundaries. — Stanford CS25 Lecture.
  6. Test factuality rather than presume it: Karina says the Claude 3 team made factuality and hallucination reduction explicit goals, and she worked on its post-training evaluations. That does not establish a universal constitutional rule against hallucinations. — Latent Space Interview.
  7. Represent uncertainty honestly: Karina describes honesty and factuality as model-behavior goals, while acknowledging that helpfulness, harmlessness and honesty can conflict in particular situations. — Latent Space Interview.
  8. Debug safety behavior with baselines: Karina describes using earlier model behavior as a baseline, grouping over-refusals by cause, and changing targeted training data rather than assuming every refusal shares one cause. — Stanford CS25 Lecture.

Part 2: Latent Space and Interpretability

  1. Find interpretable features in a small model: A research team including Karina as a support contributor used sparse autoencoders to extract interpretable features from a one-layer transformer. The result is progress toward interpretability, not a complete map of a frontier model’s concepts. — Towards Monosemanticity.
  2. Use dictionary learning to separate features: The monosemanticity team used dictionary learning to recover features that were more interpretable than individual neurons in its one-layer transformer, offering a way to study superposed activations. — Towards Monosemanticity.
  3. Use features to study internal representations: The monosemanticity team found that sparse features in a one-layer transformer were often easier to interpret than individual neurons, while leaving frontier-model representations far from fully mapped. — Towards Monosemanticity.
  4. Test whether feature interventions affect output: In the one-layer model studied by the monosemanticity team, activating selected learned features changed generated text in ways consistent with their interpretations, including base64 and Arabic-script examples. — Towards Monosemanticity.
  5. Use feature-level interpretability tools: The monosemanticity study shows how sparse autoencoders and feature visualizations can help researchers inspect a small model’s activations; the method is a research tool, not a complete safety solution. — Towards Monosemanticity.
  6. Study overlapping features, not just neurons: The monosemanticity study treats superposition as a reason individual neurons can mix concepts and tests whether sparse learned features offer a more interpretable decomposition in a small transformer. — Towards Monosemanticity.

Part 3: Model Personality and Behavior

  1. Design behavior for the context: Karina describes model behavior as a design problem: a collaborative writing surface may need a different tone and pattern of follow-up questions from an ordinary chat interface. — Latent Space Interview.
  2. Test behavior across contexts: Karina says a model acting as a collaborator in Canvas needs different context-sensitive behavior from a chat assistant, including decisions about tone and when to ask follow-up questions. — Latent Space Interview.
  3. Make refusals respectful: In her account of Claude over-refusal work, Karina emphasizes interpreting benign requests charitably and making necessary refusals nonjudgmental. — Stanford CS25 Lecture.
  4. Treat persona as behavioral design: Karina compares shaping a model persona to character design, then describes how helpfulness, harmlessness and honesty can pull against each other in a particular context. — Latent Space Interview.
  5. Adapt collaboration to the user’s task: Karina discusses tailoring the interface and the assistant’s behavior to whether someone is writing, coding or working in another mode, while presenting more personalized interfaces as a future direction. — Latent Space Interview.
  6. Shape behavior through targeted training: Karina describes constructing preference data for specific behaviors and evaluating whether the resulting model behaves appropriately in context, rather than assuming one broad training objective fixes personality. — Stanford CS25 Lecture.
  7. Earn trust through reliable collaboration: Karina suggests that people will delegate more to an agent as they experience it working reliably and adapting to their preferences through collaboration. — Latent Space Interview.
  8. Test behavior on difficult cases: Karina describes finding benign over-refusals by comparing model versions and grouping failures, then testing targeted changes for unwanted side effects. — Stanford CS25 Lecture.
  9. Ask useful follow-up questions: For a collaborative AI, Karina says a follow-up question can be useful when it advances the task, but too many questions can become annoying. — Stanford CS25 Lecture.

Part 4: AI Agents and Interaction Paradigms

  1. Ask before acting when intent is unclear: Karina says a computer-using agent should recognize when it needs a clarifying question before taking a long action that may not match the user’s intent. — Lenny’s Podcast.
  2. Design for verifiable reasoning outputs: Karina says complex reasoning outputs are harder for nonexperts to check, making trustworthy verification a central interface and alignment problem. — Latent Space Interview.
  3. Make Canvas a collaborative workspace: Karina describes Canvas as a writing and coding surface where the model must learn when to comment, make a targeted edit or rewrite a larger passage. — Latent Space Interview.
  4. Verify delegated actions: As agents move from answers toward delegated computer tasks, Karina says the interface and evaluation need to help people verify whether a complex task was actually completed. — Latent Space Interview.
  5. Combine interaction with model training: Canvas required interface design and model post-training together: Karina says the model had to learn context-appropriate collaborative behaviors, not merely sit behind a new window. — Latent Space Interview.
  6. Do not equate a reasoning trace with proof: Karina’s coauthored faithfulness research cautions that a model’s stated reasoning may not faithfully represent the factors that drove its answer; showing intermediate text alone is therefore not proof of correctness. — Measuring Faithfulness in Chain-of-Thought Reasoning.
  7. Treat proactive agents as a future possibility: Karina imagines Tasks evolving from explicit user instructions toward personalized proactive suggestions; in the interview she presents this as a future direction, not a deployed capability. — Latent Space Interview.
  8. Infer intent but confirm consequential actions: Karina describes teaching agents to understand user intent and ask a follow-up when the request is underspecified, rather than completing a lengthy action the user did not want. — Lenny’s Podcast.
  9. Meet people in existing workflows: Karina recounts an early Claude-in-Slack interface experiment, illustrating how model behavior changes when an assistant is placed inside a workplace conversation. — Latent Space Interview.

Part 5: Post-Training and Synthetic Data

  1. Inspect synthetic-data quality: Karina describes synthetic training data as useful but sensitive to diversity and quality; narrow examples can lose task-relevant variation, so teams must inspect examples as well as aggregate scores. — Stanford CS25 Lecture.
  2. Iterate through training and evaluations: Karina describes post-training as a loop of choosing a training intervention, evaluating task behavior, checking regressions and changing course when a cheaper or more targeted method works better. — Stanford CS25 Lecture.
  3. Use synthetic feedback for specific targets: Karina describes synthetic preference data as a way to target behavior where human feedback is costly, while stressing evaluation and debugging for unwanted correlations. — Stanford CS25 Lecture.
  4. Keep synthetic scenarios diverse: Karina cautions that synthetic examples need enough scenario diversity to cover the behavior a product will encounter, and that manual inspection helps catch missing modes. — Stanford CS25 Lecture.
  5. Curate behavior-specific data: When a model over-refuses benign requests, Karina describes grouping the failures, crafting targeted synthetic preferences and checking for spurious correlations in the new data. — Stanford CS25 Lecture.
  6. Evaluate real product behavior: Karina says new features need task-specific evaluations that reflect how people actually use them, plus regression checks when training changes model behavior. — Latent Space Interview.
  7. Use human feedback where it is most valuable: Karina says collecting targeted human preference data is expensive; synthetic preferences can help with specific behavior changes, but human inspection and evaluations remain important. — Stanford CS25 Lecture.
  8. Feed deployment lessons back into training: Karina says iterative deployment helps reveal unanticipated uses of Canvas and Tasks, which can then guide evaluations and later model or product changes. — Latent Space Interview.
  9. Watch for reward hacking: Karina warns that a model can earn a high score without doing the intended task, such as by causing code tests to be skipped, so harder agent tasks need more trustworthy output verification. — Stanford CS25 Lecture.

Part 6: Co-Design of Product and Research

  1. Build a tight research-product loop: Karina says Canvas development tied product design, model training and evaluations together because novel features reveal behaviors that are difficult to anticipate in advance. — Latent Space Interview.
  2. Find interfaces for new capabilities: Karina describes two directions of co-design: discover a model capability and make it usable through an interface, or start with a product need and train the model toward it. — Stanford CS25 Lecture.
  3. Prototype before treating an idea as solved: Karina describes the need to put a model capability inside an actual product flow, test where it fails, and use deployment feedback to refine both the interface and model. — Latent Space Interview.
  4. Measure the behavior users encounter: Karina says product evaluations for a feature such as Canvas must test the actual behaviors users encounter, including targeted edits and interactions with other tools. — Latent Space Interview.
  5. Pair long context with usable interaction: Karina uses Claude’s 100K context window as an example of a model capability that becomes useful through a familiar file-upload interaction, not merely a larger context number. — Stanford CS25 Lecture.
  6. Let user behavior inform evaluations: Karina says teams cannot fully predict how people will use a new feature such as Canvas, so iterative deployment and user feedback help refine product evaluations. — Latent Space Interview.
  7. Be explicit about speculative trust interfaces: Karina sketches confidence-highlighting as a possible future interface for communicating model uncertainty, while saying the idea was speculative and not deployed. — Stanford CS25 Lecture.
  8. Co-design the interface and model behavior: Karina argues that a new interface can change what behavior is appropriate from the model, so the product surface and post-training should be designed together. — Latent Space Interview.
  9. Work across disciplines: Karina describes the Canvas team as researchers, engineers, designers and product colleagues working together on model behavior and interface decisions. — Latent Space Interview.

Part 7: Building Frontier Models

  1. Balance Haiku training against compute: Karina says she worked on Claude 3 Haiku post-training, evaluations and model-card work, where limited compute and contradictory training datasets forced prioritization and regression checks. — Latent Space Interview.
  2. Prioritize small-model experiments: In discussing Claude 3 Haiku, Karina describes choosing which post-training experiments merit scarce compute and checking whether one dataset harms behavior elsewhere. — Latent Space Interview.
  3. Compare lab approaches cautiously: Karina contrasts aspects of product and research work at Anthropic and OpenAI from her own experience, while emphasizing that effective model behavior depends on the task and interface. — Latent Space Interview.
  4. Design for longer reasoning time: Karina distinguishes training-time work on reasoning from giving a model more time to think during an individual task, and asks how users should interact with slower, more deliberative responses. — Creating Agents That Co-Create.
  5. Check regressions across tasks: Karina describes using varied task-specific evaluations and regression checks when changing model behavior, because an improvement on one target can worsen another behavior. — Stanford CS25 Lecture.
  6. Prioritize experiments under compute constraints: Karina says limited compute required the Haiku post-training team to prioritize experiments and watch for regressions when datasets pulled model behavior in different directions. — Latent Space Interview.
  7. Reconsider interfaces for reasoning models: Karina says longer reasoning raises new product questions, including how to communicate progress while a model works and how to verify a complex answer. — Creating Agents That Co-Create.
  8. Learn after deployment: Karina describes iterative deployment as a way to discover unforeseen uses and failures of new AI features, then feed those observations back into evaluations and model improvements. — Latent Space Interview.

Part 8: Soft Skills and the Future of Work

  1. Expect more value from interpersonal judgment: Karina predicts that prioritization, communication, empathy, management and collaboration will grow in importance as model capabilities improve; she presents this as a view of the future, not a measured labor-market outcome. — Lenny’s Podcast.
  2. Build AI as a collaborator before full delegation: Karina describes collaborative surfaces such as Canvas as a step toward agents that people might later trust with longer autonomous tasks. — Latent Space Interview.
  3. Communicate across research and product: Karina highlights communication and collaboration as valuable skills in AI work and describes Canvas as a project shaped by researchers, engineers, designers and product colleagues. — Lenny’s Podcast.
  4. Use empathy to understand users and colleagues: Karina includes empathy and understanding people among the skills she expects to matter as AI improves, especially when teams must decide what to build and how to collaborate. — Lenny’s Podcast.
  5. Prioritize amid uncertainty: Karina describes choosing among projects and experiments under limited compute as a research-management challenge, requiring prioritization rather than trying every promising path. — Lenny’s Podcast.
  6. Ask questions that reveal useful work: Karina emphasizes understanding user intent and asking clarifying questions when an agent would otherwise pursue the wrong task; the skill is framing and checking the work, not assuming AI can answer anything. — Lenny’s Podcast.
  7. Build with multidisciplinary teams: Karina credits Canvas’s launch to collaboration among researchers, engineers and designers and explicitly values the people who made the project work. — Lenny’s Podcast.