Lilian Weng writes Lil’Log to organize and explain her learning. Her firsthand account of work at OpenAI spans robotics, applied AI and safety. These lessons draw on her technical essays, interviews and coauthored research. — The Power of Continuous Learning.

Part 1: The Anatomy of Autonomous Agents
- On the Brain Metaphor: The language model supplies reasoning; planning, memory and tools complete the agent system. — LLM Powered Autonomous Agents.
- On General Problem Solving: Language models can support problem-solving systems, not just text completion. — LLM Powered Autonomous Agents.
- On Task Decomposition: Breaking a difficult objective into subgoals makes planning more manageable. — LLM Powered Autonomous Agents.
- On Self-Reflection: Feedback on earlier attempts can help agents revise plans, but reliable self-correction remains difficult. — LLM Powered Autonomous Agents.
- On Short-Term Memory: An agent’s immediate working context is limited by the model’s context window. — LLM Powered Autonomous Agents.
- On Long-Term Memory: An external vector store can provide retrievable information beyond the immediate context. — LLM Powered Autonomous Agents.
- On Tool Use: External APIs give agents access to information and operations absent from their model weights. — LLM Powered Autonomous Agents.
- On ReAct Dynamics: ReAct alternates reasoning, actions and observations so subsequent steps can use environmental feedback. — LLM Powered Autonomous Agents.
- On Chain of Thought: Intermediate reasoning can allocate computation to harder questions; a longer chain is not automatically a better answer. — Why We Think.
- On Tree of Thoughts: Tree of Thoughts explores and evaluates alternative reasoning branches rather than following a single chain. — LLM Powered Autonomous Agents.
Part 2: Prompt Engineering & Model Steerability
- On Alignment over Tricks: Prompting aims to steer model behavior toward the intended task, not merely discover clever wording tricks. — Prompt Engineering.
- On Static Weights: Prompts guide behavior without updating model parameters. — Prompt Engineering.
- On Empirical Science: Useful prompting depends on experiments and heuristics; results can vary between models. — Prompt Engineering.
- On Academic Bloat: Weng favors reusable benchmarking infrastructure over lengthy papers built around a briefly explainable prompting trick. — Prompt Engineering.
- On In-Context Constraints: Retrieved evidence can conflict with a model’s prior knowledge; supplying newer information does not ensure the model will use it correctly. — Prompt Engineering.
- On Iterative Refinement: Revision can help when feedback identifies errors, but asking a model to reconsider without useful feedback can make answers worse. — Why We Think.
- On Test-Time Compute: Extra inference computation helps only when the system can use it productively, through approaches such as sampling or sequential reasoning. — Why We Think.
- On Structural Formatting: Output formatting needs validation: agents can violate instructions, and their integrations often require parsing model responses. — LLM Powered Autonomous Agents.
Part 3: Reinforcement Learning & Reward Hacking
- On Cumulative Rewards: Reinforcement learning evaluates actions through accumulated future rewards, not only the next immediate payoff. — A (Long) Peek into Reinforcement Learning.
- On Exploration vs. Exploitation: An agent must balance trying uncertain options with exploiting actions that have already paid off. — The Multi-Armed Bandit Problem.
- On Reward Hacking: A flawed reward can favor shortcuts that score well without accomplishing the intended task; specifying the objective is itself difficult. — Reward Hacking in Reinforcement Learning.
- On Value Functions: A value function estimates expected future return from a state or state-action pair under a policy. — A (Long) Peek into Reinforcement Learning.
- On Actor-Critic Architecture: Actor-critic methods learn a policy and a value estimator together, using the critic to guide policy updates. — Policy Gradient Algorithms.
- On the Bellman Equation: Bellman equations relate a state’s value to immediate rewards and the values of possible successor states. — A (Long) Peek into Reinforcement Learning.
- On Continuous Environments: Physical-world reinforcement learning faces costly, sample-inefficient data collection, motivating training in simulation. — Domain Randomization for Sim2Real Transfer.
- On RLHF Limitations: Human feedback can be inconsistent, and a learned reward model adds its own errors; optimizing its score may miss the real objective. — Reward Hacking in Reinforcement Learning.
Part 4: Data Quality & The Human Element
- On Data as Fuel: Training-data quality matters alongside improvements to models; collecting better labels deserves deliberate investment. — Thinking about High-Quality Human Data.
- On the Engineering Imbalance: Weng highlights Sambasivan and colleagues’ observation that model work attracts attention while essential data work is undervalued. — Thinking about High-Quality Human Data.
- On Rater Agreement: Agreement can reveal labeling problems, but disagreement on subjective tasks may reflect legitimate differences rather than poor annotation. — Thinking about High-Quality Human Data.
- On Few-Shot Learning: Meta-learning aims to make adaptation to new tasks possible with few examples, rather than retraining from scratch each time. — Meta-Learning.
- On Synthetic Data Limits: Generated training examples can contain incorrect labels; filtering noisy samples is one way to improve their usefulness. — Data Generation.
- On Active Learning: Active learning selects examples for labeling using criteria such as uncertainty or diversity, making limited annotation budgets more useful. — Active Learning.
- On Semi-Supervised Techniques: Semi-supervised learning combines labeled and unlabeled examples to reduce dependence on costly annotations. — Semi-Supervised Learning.
- On Human-in-the-Loop: In the coauthored moderation workflow, humans monitor and refine model judgments, concentrating attention on complex policy edge cases. — Using GPT-4 for Content Moderation.
- On Model Steerability via Data: The instructions and balance of feedback examples shape safety behavior; underspecified annotation policies can produce excessive refusals or unwanted response styles. — Rule Based Rewards for Language Model Safety.
Part 5: AI Safety, Alignment, & Adversarial Robustness
- On the Double-Edged Sword: AI offers benefits alongside risks, making human guidance and participation important to its development. — TMTPost: AI Safety and Cultivation.
- On Prioritizing Safety: In her 2022 interview, Weng stressed alignment and safety alongside scaling: powerful systems must also follow human intentions. — The Power of Continuous Learning.
- On Adversarial Vulnerabilities: Safety training does not eliminate adversarial failures; attackers can manipulate inputs to elicit unwanted model behavior. — Adversarial Attacks on LLMs.
- On Jailbreak Mechanics: Role-play is one jailbreak tactic; others exploit conflicting objectives or gaps between safety training and model capabilities. — Adversarial Attacks on LLMs.
- On the Safety Stack: Weng describes complementary safety work: evaluation tasks, unwanted-content classifiers and techniques that reduce unsafe outputs. — The Power of Continuous Learning.
- On Grounding against Hallucination: Retrieval can ground answers in external documents; evidence checking and revision help address claims the retrieved material does not support. — Extrinsic Hallucinations in LLMs.
- On Shared Responsibility: Weng calls for wider participation in AI safety through sharing insights, discussing problems and scrutinizing model behavior. — TMTPost: AI Safety and Cultivation.
- On High-Risk Evaluation: The GPT-4 report, coauthored by Weng, describes expert testing in high-risk domains to inform mitigations before release. — GPT-4 Technical Report.
- On Robustness as a Discipline: Robustness work needs concrete failure tests and careful evaluation of unsafe outputs, including human review where automated judgments are difficult. — Adversarial Attacks on LLMs.
- On Safety and Usefulness: Safety tuning must balance blocking harmful requests against refusing harmless ones; coauthored experiments show that training choices affect this trade-off. — Rule Based Rewards for Language Model Safety.
Part 6: Navigating Robotics & Sim-to-Real Transfer
- On the Reality Gap: A policy that succeeds in simulation may fail on real hardware when the simulator does not match real-world conditions. — Domain Randomization for Sim2Real Transfer.
- On Domain Randomization: Varying simulated visual and physical conditions can help policies transfer to real robots rather than overfit one simulator. — Domain Randomization for Sim2Real Transfer.
- On the Rubik's Cube Project: Weng’s team combined reinforcement learning, simulation and domain randomization to train a robotic hand to solve a Rubik’s Cube. — The Power of Continuous Learning.
- On the Emotional Toll of Robotics: Weng recalls the Rubik’s Cube project as emotionally demanding as well as technically challenging. — The Power of Continuous Learning.
- On Dexterous Manipulation: A dexterous hand must coordinate many controls while coping with noisy, partially hidden observations. — Girl Geek X: OpenAI Robotics Talk.
- On Hardware Brittleness: Fragile hardware and maintenance make large-scale real-robot training expensive; hardware changes can also invalidate collected experience. — Girl Geek X: OpenAI Robotics Talk.
- On Simulation Fidelity: Matching simulation to reality is difficult because calibration is costly and physical properties change with wear and operating conditions. — Domain Randomization for Sim2Real Transfer.
- On Physical Autonomy: General-purpose robots need to handle varied, unfamiliar situations rather than depend entirely on preprogrammed motions. — Girl Geek X: OpenAI Robotics Talk.
- On Continuous Action Spaces: In continuous action spaces, exhaustive action-value searches are expensive; directly parameterized policies offer an alternative. — Policy Gradient Algorithms.
Part 7: Model Mechanics: Diffusion, Memory & Scaling
- On the Forward Diffusion Process: Forward diffusion progressively adds Gaussian noise, moving data toward a simple noise distribution. — What Are Diffusion Models?.
- On the Reverse Diffusion Process: A learned model approximates reverse denoising steps, enabling generation from noise. — What Are Diffusion Models?.
- On Temporal Consistency: Video generation must preserve consistency between frames, adding a challenge absent from generating a single image. — Diffusion Models for Video Generation.
- On Hardware Optimization: Inference optimization must consider both memory use and latency, including communication costs when computation is distributed. — Large Transformer Model Inference Optimization.
- On Distributed Training: Large-model training splits work across devices; data, model and pipeline parallelism address different memory and throughput constraints. — How to Train Really Large Models on Many GPUs?.
- On RAG Architecture: Retrieval-augmented generation supplies relevant documents as context, supplementing information stored in model parameters. — Prompt Engineering.
- On Generalized Architectures: Transformer variants reuse attention-based building blocks across translation and language-modeling tasks, with different encoder and decoder designs. — The Transformer Family Version 2.0.
- On Multimodal Reasoning: Visual-language models extend language systems to consume visual information, supporting tasks such as image captioning and visual question answering. — Generalized Visual Language Models.
Part 8: The Philosophy of AI Development & Future of Work
- On Job Evolution: Weng hopes general-purpose robots can take on dangerous or repetitive work, leaving people more interesting and creative tasks. — Girl Geek X: OpenAI Robotics Talk.
- On the Genesis of Lil'Log: Lil’Log began as Weng’s personal notes for organizing papers and concepts while learning deep learning. — The Power of Continuous Learning.
- On Teaching to Learn: Explaining a concept clearly to others tests understanding; writing helps Weng develop that understanding. — The Power of Continuous Learning.
- On Research vs. Product: The moderation example turns policy into labeled tests, compares model judgments with expert judgments and iterates before deployment. — Using GPT-4 for Content Moderation.
- On Diversity of Perspective: Feedback design should recognize that cultural and personal perspectives can produce legitimate differences in subjective judgments. — Thinking about High-Quality Human Data.
- On the Long Game: Weng describes bias reduction as continuing work, with early interventions followed by incremental improvements. — The Power of Continuous Learning.