Visual summary of operating lessons from Riley Goodside.

Lessons from Riley Goodside

The items below are things Riley Goodside said in a podcast interview about early prompting experiments with large language models. — Riley Goodside on The TWIML AI Podcast (Mental Models for Advanced ChatGPT Prompting), auto-generated captions.

On The TWIML AI Podcast, Goodside said: "there wasn't a lot of material out there on how to prompt". — Riley Goodside on The TWIML AI Podcast (Mental Models for Advanced ChatGPT Prompting), auto-generated captions.

On The TWIML AI Podcast, Goodside said: "what capabilities might be lurking in here". — Riley Goodside on The TWIML AI Podcast (Mental Models for Advanced ChatGPT Prompting), auto-generated captions.

On The TWIML AI Podcast, Goodside said: "the scope of what might work is wide open". — Riley Goodside on The TWIML AI Podcast (Mental Models for Advanced ChatGPT Prompting), auto-generated captions.

On The TWIML AI Podcast, Goodside said: "it's really a waste of the weights". — Riley Goodside on The TWIML AI Podcast (Mental Models for Advanced ChatGPT Prompting), auto-generated captions.

On The TWIML AI Podcast, Goodside said: "there's a wealth of knowledge in there". — Riley Goodside on The TWIML AI Podcast (Mental Models for Advanced ChatGPT Prompting), auto-generated captions.

On The TWIML AI Podcast, Goodside said: "the diversity of text is just so great that we couldn't keep it from learning these things". — Riley Goodside on The TWIML AI Podcast (Mental Models for Advanced ChatGPT Prompting), auto-generated captions.

On The TWIML AI Podcast, Goodside said: "it was sort of a whale that I was chasing for a bit". — Riley Goodside on The TWIML AI Podcast (Mental Models for Advanced ChatGPT Prompting), auto-generated captions.

On The TWIML AI Podcast, Goodside said: "they wouldn't object to anything". — Riley Goodside on The TWIML AI Podcast (Mental Models for Advanced ChatGPT Prompting), auto-generated captions.

On The TWIML AI Podcast, Goodside said: "it acted much like an improv partner". — Riley Goodside on The TWIML AI Podcast (Mental Models for Advanced ChatGPT Prompting), auto-generated captions.

On the Interconnects podcast, Goodside said: "you can only do the things that are the most conservative and the things that you can justify and explain that you're not going to produce as good of a work as you might have otherwise". — Riley Goodside on Interconnects (Interviewing Riley Goodside on the science of prompting, September 30, 2024), transcript.

On the Interconnects podcast, Goodside said: "people need to be a bit more imaginative and like what's possible and just through autoregression". — Riley Goodside on Interconnects (Interviewing Riley Goodside on the science of prompting, September 30, 2024), transcript.

On the Interconnects podcast, Goodside said: "the models get mileage out of like thinking for long periods of time". — Riley Goodside on Interconnects (Interviewing Riley Goodside on the science of prompting, September 30, 2024), transcript.

On the Interconnects podcast, Goodside said: "it's pretty evident after a while that it's not making progress, that it's just gone like deeper and deeper into like some failed path of reasoning". — Riley Goodside on Interconnects (Interviewing Riley Goodside on the science of prompting, September 30, 2024), transcript.

On the Interconnects podcast, Goodside said: "you're not going to put a human on the task of figuring out what is the best prompt for each model". — Riley Goodside on Interconnects (Interviewing Riley Goodside on the science of prompting, September 30, 2024), transcript.

On the Interconnects podcast, Goodside said: "ideal and optimized prompts are very different than what's the default". — Riley Goodside on Interconnects (Interviewing Riley Goodside on the science of prompting, September 30, 2024), transcript.

On the Interconnects podcast, Goodside said: "maybe the fair way to do this is to have some secondary model on the end of it that puts everything into a consistent format". — Riley Goodside on Interconnects (Interviewing Riley Goodside on the science of prompting, September 30, 2024), transcript.

On the Interconnects podcast, Goodside said: "I've often like been frustrated by the ambiguity and a lot of academic publications over like how prompts are formatted". — Riley Goodside on Interconnects (Interviewing Riley Goodside on the science of prompting, September 30, 2024), transcript.

On the Interconnects podcast, Goodside said: "if you do self-consistency on 40 samples of GPT-3.5 turbo, it's on par with one sample from GPT-4". — Riley Goodside on Interconnects (Interviewing Riley Goodside on the science of prompting, September 30, 2024), transcript.

On the Interconnects podcast, Goodside said: "lets you search over the variation in your strategies that actually matters". — Riley Goodside on Interconnects (Interviewing Riley Goodside on the science of prompting, September 30, 2024), transcript.

On the Interconnects podcast, Goodside said: "things, even in the summarized reasoning traces that O1 puts out, that are eerily anthropomorphic". — Riley Goodside on Interconnects (Interviewing Riley Goodside on the science of prompting, September 30, 2024), transcript.

On the Interconnects podcast, Goodside said: "our ability to prompt these models sometimes exceeds our ability to explain how we're doing it and what the general strategies to apply are". — Riley Goodside on Interconnects (Interviewing Riley Goodside on the science of prompting, September 30, 2024), transcript.

On the Interconnects podcast, Goodside said: "people who have acclimated to models forget what it's like to be new". — Riley Goodside on Interconnects (Interviewing Riley Goodside on the science of prompting, September 30, 2024), transcript.

On the Interconnects podcast, Goodside said: "There's practical consequences of these models being in the hands of people that don't know about these issues". — Riley Goodside on Interconnects (Interviewing Riley Goodside on the science of prompting, September 30, 2024), transcript.

On the Interconnects podcast, Goodside said: "if you emphasize enough the need to validate things, it will just keep validating and failing and loop around". — Riley Goodside on Interconnects (Interviewing Riley Goodside on the science of prompting, September 30, 2024), transcript.

On The Cognitive Revolution podcast in 2024, Goodside said: "you start with a prompt, right, just a sort of instruction prompt or MVP of how you'd want the task done". — Riley Goodside on The Cognitive Revolution (From Poetry to Programming, July 24, 2024), transcript.

On The Cognitive Revolution podcast in 2024, Goodside said: "add more K Shot examples until no more will fit". — Riley Goodside on The Cognitive Revolution (From Poetry to Programming, July 24, 2024), transcript.

On The Cognitive Revolution podcast in 2024, Goodside said: "only once you've maxed out those options, I'd say you wanna dive into fine tuning". — Riley Goodside on The Cognitive Revolution (From Poetry to Programming, July 24, 2024), transcript.

On The Cognitive Revolution podcast in 2024, Goodside said: "it's not wasted effort". — Riley Goodside on The Cognitive Revolution (From Poetry to Programming, July 24, 2024), transcript.

On The Cognitive Revolution podcast in 2024, Goodside said: "the thing that people forget throughout this whole process of refining from, like, a prompt to a, like, a more advanced prompt engineered pipeline is decomposition". — Riley Goodside on The Cognitive Revolution (From Poetry to Programming, July 24, 2024), transcript.

On The Cognitive Revolution podcast in 2024, Goodside said: "is there any piece of this task that I could pull out and make it easier and make it simpler?". — Riley Goodside on The Cognitive Revolution (From Poetry to Programming, July 24, 2024), transcript.

On The Cognitive Revolution podcast in 2024, Goodside said: "how could you fix this one example?". — Riley Goodside on The Cognitive Revolution (From Poetry to Programming, July 24, 2024), transcript.

On The Cognitive Revolution podcast in 2024, Goodside said: "I think people really just neglect annotation, really, and labeling". — Riley Goodside on The Cognitive Revolution (From Poetry to Programming, July 24, 2024), transcript.

On The Cognitive Revolution podcast in 2024, Goodside said: "what is a realistic input to this task?". — Riley Goodside on The Cognitive Revolution (From Poetry to Programming, July 24, 2024), transcript.

On The Cognitive Revolution podcast in 2024, Goodside said: "what are the edge cases?". — Riley Goodside on The Cognitive Revolution (From Poetry to Programming, July 24, 2024), transcript.

On The Cognitive Revolution podcast in 2024, Goodside said: "you really wanna be considering what are the boundary conditions through which your high level task could break". — Riley Goodside on The Cognitive Revolution (From Poetry to Programming, July 24, 2024), transcript.

On The Cognitive Revolution podcast in 2024, Goodside said: "you wanna define those cases so that it can see every real world example as sort of an interpolation of the examples that you gave it". — Riley Goodside on The Cognitive Revolution (From Poetry to Programming, July 24, 2024), transcript.

On The Cognitive Revolution podcast in 2024, Goodside said: "The stuff that seemed weird to you in some deeply ingrained sense that you learned during pre training isn't necessarily wrong, so ignore that instinct sometimes". — Riley Goodside on The Cognitive Revolution (From Poetry to Programming, July 24, 2024), transcript.

On The Cognitive Revolution podcast in 2024, Goodside said: "you really want to evaluate all of the reasonable options because performance on different task domains is all over the map". — Riley Goodside on The Cognitive Revolution (From Poetry to Programming, July 24, 2024), transcript.

On The Cognitive Revolution podcast in 2024, Goodside said: "leaderboards are a guide". — Riley Goodside on The Cognitive Revolution (From Poetry to Programming, July 24, 2024), transcript.

On The Cognitive Revolution podcast in 2023, Goodside said: "A lot of the things that I was tweeting early on, a lot of my most successful things, a lot of my most successful tweets were demos, really". — Riley Goodside on The Cognitive Revolution (The Art of Prompting ChatGPT, April 18, 2023), transcript.

On The Cognitive Revolution podcast in 2023, Goodside said: "It's me being aware of what are the things that it's good at and the things that it's not good at, and then arranging an impressive task that's an assemblage of the things that I know that it's good at". — Riley Goodside on The Cognitive Revolution (The Art of Prompting ChatGPT, April 18, 2023), transcript.

On The Cognitive Revolution podcast in 2023, Goodside said: "It can't do letter-by-letter operations because it only sees the world through tokens". — Riley Goodside on The Cognitive Revolution (The Art of Prompting ChatGPT, April 18, 2023), transcript.

On The Cognitive Revolution podcast in 2023, Goodside said: "Wow, maybe I'm actually doing something new here". — Riley Goodside on The Cognitive Revolution (The Art of Prompting ChatGPT, April 18, 2023), transcript.

On The Cognitive Revolution podcast in 2023, Goodside said: "I was interested in capabilities that were just on the fringe of reliability that nobody had really thought to chase yet". — Riley Goodside on The Cognitive Revolution (The Art of Prompting ChatGPT, April 18, 2023), transcript.

On The Cognitive Revolution podcast in 2023, Goodside said: "models are going to improve". — Riley Goodside on The Cognitive Revolution (The Art of Prompting ChatGPT, April 18, 2023), transcript.

On the Interconnects podcast, Goodside said: "text is a more universal interface than we were giving you credit for". — Riley Goodside on Interconnects (Interviewing Riley Goodside on the science of prompting, September 30, 2024), transcript.

On The Cognitive Revolution podcast in 2023, Goodside said: "put in little angle bracket placeholders, the same way you would for a human if you were describing a format". — Riley Goodside on The Cognitive Revolution (The Art of Prompting ChatGPT, April 18, 2023), transcript.

On The Cognitive Revolution podcast in 2023, Goodside said: "I immediately started craving structured output". — Riley Goodside on The Cognitive Revolution (The Art of Prompting ChatGPT, April 18, 2023), transcript.

On The Cognitive Revolution podcast in 2023, Goodside said: "This era is the closest to what people mean when you hear the cliche that these models just predict text". — Riley Goodside on The Cognitive Revolution (The Art of Prompting ChatGPT, April 18, 2023), transcript.

On The Cognitive Revolution podcast in 2023, Goodside said: "It needed to have it be established that this is a fictional narrative where this is a good French translator and he's going to do it right". — Riley Goodside on The Cognitive Revolution (The Art of Prompting ChatGPT, April 18, 2023), transcript.

On The Cognitive Revolution podcast in 2023, Goodside said: "That style of modeling text runs into conflict with the ways that many people would want a chatbot to behave". — Riley Goodside on The Cognitive Revolution (The Art of Prompting ChatGPT, April 18, 2023), transcript.

On The Cognitive Revolution podcast in 2023, Goodside said: "in order to prevent this unintuitive behavior and to make the model more capable and more able to do as it's told, they fine-tuned the model". — Riley Goodside on The Cognitive Revolution (The Art of Prompting ChatGPT, April 18, 2023), transcript.