Visual summary of operating lessons from Denny Zhou.

Lessons from Denny Zhou

Denny Zhou is a research scientist at Google DeepMind and founder of its Reasoning Team. He coauthored the Chain-of-Thought prompting paper and led the Least-to-Most Prompting study. These lessons examine how intermediate steps, decomposition, decoding, and feedback can improve language-model performance on tested reasoning tasks, alongside the limits of those methods. — Denny Zhou research homepage.

Part 1: Defining Reasoning in AI

  1. Reasoning beyond examples: Zhou frames reasoning as a way to handle tasks with few demonstrations. His last-letter example shows why merely supplying input-output examples can fail, while an intermediate explanation lets the model apply a rule to a new input. — Teach Language Models to Reason (KDD slides).
  2. The parrot analogy: Zhou compares next-word training to teaching a parrot to mimic language, then uses a simple letter task to show how a model can fail to apply a rule from bare examples. — Teach Language Models to Reason (KDD slides).
  3. Labels are not enough: In Zhou’s last-letter example, a conventional supervised approach needs many labeled examples for a simple rule. His demonstration motivates prompting for a worked-out intermediate explanation. — Teach Language Models to Reason (KDD slides).
  4. Show intermediate work: Chain-of-thought examples put an intermediate explanation between the input and the answer. In the paper Zhou coauthored, this improves performance on tested arithmetic, commonsense, and symbolic tasks at sufficient model scale. — Chain-of-Thought Prompting paper.
  5. Generate useful analogies: The analogical-reasoning work Zhou coauthored asks a language model to generate relevant examples or knowledge for a new problem before solving it; experiments found gains over tested zero-shot and manually supplied few-shot baselines. — Analogical Reasoning paper.
  6. Prediction can support reasoning prompts: Zhou describes language models as next-word predictors and demonstrates that worked-out prompts can elicit more useful problem-solving behavior on his toy task than bare input-output examples. — Teach Language Models to Reason (KDD slides).
  7. Do not equate CoT with human thought: The chain-of-thought results show improved benchmark performance from intermediate written steps. They do not establish that the model thinks in the same way a person does. — Chain-of-Thought Prompting paper.

Part 2: The Chain-of-Thought Breakthrough

  1. Few-shot reasoning exemplars: In the Chain-of-Thought study Zhou coauthored, prompts include a few worked examples with intermediate reasoning steps. The paper reports gains on tested arithmetic, commonsense, and symbolic reasoning tasks. — Chain-of-Thought Prompting paper.
  2. A scale-dependent effect: The Chain-of-Thought paper reports that its strongest prompting gains emerge in sufficiently large models; the result is not a blanket property of every language model. — Chain-of-Thought Prompting paper.
  3. Math benchmark gains: Few-shot chain-of-thought prompting improved performance on tested math word-problem benchmarks, including GSM8K. The paper’s result concerns worked reasoning exemplars, not merely appending one zero-shot phrase. — Chain-of-Thought Prompting paper.
  4. Fewer labeled examples in tested settings: Zhou’s slides compare chain-of-thought and self-consistency results with supervised approaches and emphasize strong performance from a small set of demonstrations. The comparison is benchmark-specific, not a universal data-efficiency guarantee. — Teach Language Models to Reason (KDD slides).
  5. Visible steps aid inspection: Chain-of-thought prompting produces intermediate written steps that a reader can examine alongside the answer. These visible steps can expose a mistake in a response, but they are not proof of a faithful account of internal model computation. — Chain-of-Thought Prompting paper.
  6. No model retraining for the prompting method: The study elicits chain-of-thought by changing few-shot prompt examples rather than changing the model architecture or retraining it for each task. — Chain-of-Thought Prompting paper.

Part 3: Self-Consistency and Reliability

  1. Sample multiple paths: Self-consistency samples diverse chain-of-thought paths and selects an answer that remains consistent across them, rather than relying on one greedy-decoded response. — Self-Consistency paper.
  2. A flawed path need not decide the answer: The authors show that one sampled chain can contain a mistake while other paths reach the same correct answer; aggregating final answers can mitigate reliance on that single flawed path in tested tasks. — Self-Consistency paper.
  3. Aggregate answers, not wording: The method marginalizes over sampled reasoning paths and can implement answer aggregation with a majority vote. Zhou’s slides caution that the conceptual point is marginalization, not a superficial voting analogy. — Teach Language Models to Reason (KDD slides).
  4. Agreement is evidence, not calibrated confidence: The paper’s hypothesis is that independently sampled correct paths tend to converge on one final answer more often than wrong paths. It does not establish that answer frequency is a calibrated confidence score. — Self-Consistency paper.
  5. Sampling provides path diversity: The paper says self-consistency is compatible with temperature, top-k, and nucleus sampling as ways to generate diverse reasoning paths; it does not require a particular temperature setting. — Self-Consistency paper.
  6. Tested robustness to imperfect examples: In an experiment with deliberately corrupted reasoning exemplars, self-consistency improved accuracy relative to greedy decoding. This is narrower than immunity to all prompt-phrasing changes. — Self-Consistency paper.
  7. Accuracy requires more sampled outputs: Self-consistency evaluates multiple generations rather than one, adding inference work; the paper reports benchmark accuracy gains but does not establish a general reduction in hallucinations. — Self-Consistency paper.
  8. A self-ensemble from one model: The authors describe self-consistency as a self-ensemble: several paths from one model are aggregated by final answer, without training a committee of separate models. — Self-Consistency paper.

Part 4: Problem Decomposition and Least-to-Most Prompting

  1. Generalize from easy to hard: Least-to-most prompting decomposes a complex question into simpler subproblems and solves them sequentially. The authors report improved generalization to harder tested problems than the prompt exemplars. — Least-to-Most Prompting paper.
  2. Length generalization on SCAN: On SCAN, the authors report at least 99% accuracy across splits, including the length split, with 14 exemplars for one tested model; this is a benchmark-specific result. — Least-to-Most Prompting paper.
  3. Carry forward earlier answers: The sequential solving stage supplies answers to earlier subproblems as context for subsequent subproblems; it does not independently verify those earlier answers. — Least-to-Most Prompting paper.
  4. Decompose before solving: Unlike a single worked chain, least-to-most prompting has an explicit decomposition stage that produces easier subquestions before the sequential solving stage. — Least-to-Most Prompting paper.
  5. Build longer symbolic outputs incrementally: The paper’s last-letter concatenation examples solve short prefixes first and extend them step by step, illustrating the method’s length-generalization test rather than a general cure for lost state. — Least-to-Most Prompting paper.

Part 5: In-Context Learning and Model Optimization

  1. Zero-shot step-by-step prompting: A separate study by Kojima and colleagues, not Zhou, found that adding “Let’s think step by step” improved zero-shot performance on several tested reasoning benchmarks. Zhou later discussed this technique in his reasoning lecture. — Zero-Shot Reasoners paper.
  2. Mix instruction-tuning formats: The Flan Collection study, which Zhou coauthored, found that mixing zero-shot, few-shot, and chain-of-thought prompt settings during instruction tuning improved tested evaluation results. — The Flan Collection.
  3. A diverse tuning collection: Flan 2022 packages task datasets, templates, and tuning methods. Its ablations identify task balancing, enrichment, and mixed prompt settings as important in tested instruction-tuned models. — The Flan Collection.
  4. Optimize prompts through evaluation: OPRO uses a language model to propose new candidate instructions, evaluates them on a task, then gives previous candidates and scores back to the model for another step. — Large Language Models as Optimizers.
  5. Search natural-language instructions: In the authors’ tests, OPRO found task instructions that outperformed human-designed prompts on GSM8K and Big-Bench Hard. This is an empirical search result, not proof of a globally optimal prompt. — Large Language Models as Optimizers.
  6. Use a meta-prompt with scored history: OPRO’s prompt describes the optimization task and includes previously generated solutions with their values; the model uses that context to propose the next candidate. — Large Language Models as Optimizers.
  7. A gradient-free prompt search: OPRO treats candidate instructions as natural-language solutions and evaluates their task scores, providing a way to optimize prompts without computing gradients through those discrete instructions. — Large Language Models as Optimizers.
  8. Algorithmic ICL in a controlled setting: A study Zhou coauthored found that transformers can implement or approximate algorithms such as gradient descent and ridge regression in linear in-context learning tasks. It does not establish that all in-context learning is gradient descent. — In-Context Learning study.

Part 6: Decoding Strategies and Intrinsic Capabilities

  1. Reasoning paths without CoT prompts: Wang and Zhou found that alternative decoding choices can elicit chain-of-thought paths from pretrained models without an explicit chain-of-thought prompt in tested settings. — Chain-of-Thought Without Prompting.
  2. Greedy output may hide a path: The study reports chain-of-thought paths available among top-k alternatives that conventional greedy decoding can miss on evaluated reasoning tasks. — Chain-of-Thought Without Prompting.
  3. Inspect top-k alternatives: Instead of always selecting the top token, the study explores alternative high-ranked tokens and can recover reasoning paths without hand-crafted chain-of-thought examples. — Chain-of-Thought Without Prompting.
  4. Confidence is measured at the answer level: The paper reports that decoded paths containing a chain of thought correlate with higher confidence in the resulting answer. It does not say each token in a logical sequence is individually high-confidence. — Chain-of-Thought Without Prompting.

Part 7: The Limitations of Current Models

  1. Irrelevant context can distract: On a modified math-word-problem benchmark, adding irrelevant information reduced the tested language models’ accuracy. The paper also evaluates instructions to ignore irrelevant details as a partial mitigation. — Irrelevant Context study.
  2. Fluent revision is not reliable correction: The self-correction study found that models reviewing their own reasoning without external feedback sometimes worsened initially correct answers. Fluency of the revision is not evidence that the answer improved. — Self-Correction study.
  3. Premise order affects reasoning: A study Zhou coauthored found that reordering logically equivalent premises can materially reduce model accuracy on tested deductive and math problems. This supports a wording-sensitivity lesson, not the original negative-constraint example. — Premise Order study.
  4. Verify reasoning, not just revisions: Zhou’s Berkeley lecture reviews why unguided self-correction can turn a correct answer into an incorrect one and highlights external feedback for correction. This does not establish that all existing benchmarks are inadequate. — Berkeley reasoning slides.

Part 8: The Path Toward Artificial General Intelligence

  1. Reasoning as an AGI research aim: On his own research homepage, Zhou describes building language models that reason and generalize as part of an AGI ambition. That is a stated research goal, not evidence that a particular planning-and-verification architecture is necessary. — Denny Zhou research homepage.
  2. Reliable feedback matters for correction: Zhou’s Stanford slides call a reliable verifier crucial for reinforcement-learning-based reasoning training; his Berkeley slides also show why unguided self-correction can worsen an answer. — Stanford CS25 reasoning slides.
  3. Output length as a reasoning resource: A theoretical study Zhou coauthored shows that intermediate chain-of-thought steps can increase the expressive power of certain low-depth transformers on inherently serial problems. His lecture accordingly highlights scaling intermediate output length, without claiming training compute no longer matters. — Serial Problems paper.
  4. Reusable tools can extend a model: The Tool Makers paper tests models that create reusable tools for later problems. It supports a bounded role for tools in complex-task solving, not a claim that robust reasoning already enables reliable multi-day autonomous agents. — Large Language Models as Tool Makers.
  5. Generate solutions and use feedback: Zhou’s Stanford lecture describes reasoning finetuning that uses model-generated solutions and feedback about correctness. It is a training approach, not a demonstrated inevitable self-improvement loop. — Stanford CS25 reasoning slides.
  6. Execution feedback for self-debugging: A paper Zhou coauthored tests models that generate code, inspect execution results or explanations, and revise outputs. This supports using external feedback for some reasoning tasks, not an unrestricted claim about all calculators or search tools. — Self-Debugging paper.
  7. An autonomous reasoning goal: In his Berkeley lecture Zhou names a goal of models that autonomously learn reasoning techniques while overcoming observed limitations. That is a stated aspiration, not evidence that an automated scientist has been built. — Berkeley reasoning slides.