Visual summary of operating lessons from Percy Liang.

The items below are things Percy Liang said in interviews and panels about foundation models, benchmarking, open and closed AI, and academic research. — Percy Liang on No Priors (episode 7, April 2023), transcript.

On the No Priors podcast, Liang said: "I've been in the field of machine learning and natural language processing for over 20 years". — Percy Liang on No Priors (episode 7, April 2023), transcript.

On the No Priors podcast, Liang said: "we should be working on things that are, lean on academia's competitive advantage". — Percy Liang on No Priors (episode 7, April 2023), transcript.

On the No Priors podcast, Liang said: "If you think about predicting the next word, it's, it seems very simple, but you have to really internalize a lot of what is going on in this context". — Percy Liang on No Priors (episode 7, April 2023), transcript.

On the No Priors podcast, Liang said: "it will be much easier to control that model if it has a notion of what hallucination even is". — Percy Liang on No Priors (episode 7, April 2023), transcript.

On the No Priors podcast, Liang said: "we know that humans, when they solve a problem, you try different things, you backtrack". — Percy Liang on No Priors (episode 7, April 2023), transcript.

On a 2023 open-or-closed panel, Liang said: "if you're building applications downstream, it's important to build them in a bit of a future-proof way". — Percy Liang on the panel Navigating the AI Landscape: Open or Closed? (Gen AI Summit 2023, Fellows Fund), transcript.

On a 2023 open-or-closed panel, Liang said: "only through rigorous benchmarking can you make intelligent decisions whether to switch". — Percy Liang on the panel Navigating the AI Landscape: Open or Closed? (Gen AI Summit 2023, Fellows Fund), transcript.

On a 2023 open-or-closed panel, Liang said: "in an open community, the majority of people are good". — Percy Liang on the panel Navigating the AI Landscape: Open or Closed? (Gen AI Summit 2023, Fellows Fund), transcript.

On a 2023 open-or-closed panel, Liang said: "as long as the best practices are established, and it's easy for people to do the right thing, I think people more or less do the right thing". — Percy Liang on the panel Navigating the AI Landscape: Open or Closed? (Gen AI Summit 2023, Fellows Fund), transcript.

On a 2023 open-or-closed panel, Liang said: "this is just the nature of dual-use technology that we have to confront, whether it's generative AI models or a human doing it". — Percy Liang on the panel Navigating the AI Landscape: Open or Closed? (Gen AI Summit 2023, Fellows Fund), transcript.

On a 2023 open-or-closed panel, Liang said: "if you think open, close, you're choosing between two things, I think it's a false dichotomy". — Percy Liang on the panel Navigating the AI Landscape: Open or Closed? (Gen AI Summit 2023, Fellows Fund), transcript.

On a 2023 open-or-closed panel, Liang said: "One thing that would be nice is if there were more transparency on the data that's going into training these models". — Percy Liang on the panel Navigating the AI Landscape: Open or Closed? (Gen AI Summit 2023, Fellows Fund), transcript.

On a 2023 open-or-closed panel, Liang said: "I think there are benefits to openness, and there are obviously competitive dynamics and other safety reasons for closed models". — Percy Liang on the panel Navigating the AI Landscape: Open or Closed? (Gen AI Summit 2023, Fellows Fund), transcript.

In interviews about foundation models and benchmarking, Liang said: "The feeling was that language models were a different object, unlike classical machine learning methods because they were so broad". — Percy Liang on Gradient Dissent, the a16z podcast and the CS224U podcast (excerpts of his answers).

In interviews about foundation models and benchmarking, Liang said: "standardized the evaluations using the same prompts and same methodology across all the models and data sets so that it had apples to apples comparison". — Percy Liang on Gradient Dissent, the a16z podcast and the CS224U podcast (excerpts of his answers).

In interviews about foundation models and benchmarking, Liang said: "not just benchmarking, but also reproducibility so that you could have a full audit trail of what you did to get your benchmarking results". — Percy Liang on Gradient Dissent, the a16z podcast and the CS224U podcast (excerpts of his answers).

In interviews about foundation models and benchmarking, Liang said: "some of the regulation decisions are based on speculation and uncertainty, because we just don't have evidence". — Percy Liang on Gradient Dissent, the a16z podcast and the CS224U podcast (excerpts of his answers).

In interviews about foundation models and benchmarking, Liang said: "maybe regulation should be targeted more downstream as opposed to upstream on the actual raw model". — Percy Liang on Gradient Dissent, the a16z podcast and the CS224U podcast (excerpts of his answers).

In interviews about foundation models and benchmarking, Liang said: "these technologies can be misused. I think that's sort of a given, but how you respond to that is something that needs actually nuance and scrutiny". — Percy Liang on Gradient Dissent, the a16z podcast and the CS224U podcast (excerpts of his answers).

In interviews about foundation models and benchmarking, Liang said: "academia is full of deep interdisciplinary expertise across not just people in AI, but also from the social sciences, economics, political science, the Medical School". — Percy Liang on Gradient Dissent, the a16z podcast and the CS224U podcast (excerpts of his answers).

In interviews about foundation models and benchmarking, Liang said: "how these models could benefit society rather than being driven by commercial objectives". — Percy Liang on Gradient Dissent, the a16z podcast and the CS224U podcast (excerpts of his answers).

In interviews about foundation models and benchmarking, Liang said: "hopefully more modular, hopefully more trustworthy and understandable". — Percy Liang on Gradient Dissent, the a16z podcast and the CS224U podcast (excerpts of his answers).

In a talk on foundation models, Liang said: "everyone has been surprised by how far you can get with these simple principles". — Percy Liang, The paradigm shift and societal effects of foundation models (auto-generated captions).

In a talk on foundation models, Liang said: "I've sort of learned my lesson in some ways to not think oh yeah that's never going to work". — Percy Liang, The paradigm shift and societal effects of foundation models (auto-generated captions).

In a talk on foundation models, Liang said: "we've been trying evaluate humans for a millennia and we haven't really figured that one out". — Percy Liang, The paradigm shift and societal effects of foundation models (auto-generated captions).

In a talk on foundation models, Liang said: "you are not fooling yourselves and not gaming it then I think it could be very productive". — Percy Liang, The paradigm shift and societal effects of foundation models (auto-generated captions).

In a talk on foundation models, Liang said: "you have the source code of Linux in theory you can go read the millions of lines of code if you wanted to at least it's auditable". — Percy Liang, The paradigm shift and societal effects of foundation models (auto-generated captions).

In the talk Are Foundation Models Castles in the Air?, Liang said: "language modeling is actually pretty powerful because in order to predict the next word well, you actually have to at least capture enough information". — Percy Liang, Are Foundation Models Castles in the Air? (auto-generated captions).

In the talk Are Foundation Models Castles in the Air?, Liang said: "foundation models are these models which are trained on a broad data, which means you have to do this at scale and do it in a self-supervised way if you're truly going to be truly broad". — Percy Liang, Are Foundation Models Castles in the Air? (auto-generated captions).

In the talk Are Foundation Models Castles in the Air?, Liang said: "language models are not foundation models. Foundation models is a strict super set of language models". — Percy Liang, Are Foundation Models Castles in the Air? (auto-generated captions).