
Lessons from Victoria Krakovna
Victoria Krakovna is an AI safety researcher. These lessons are drawn from posts on their own blog. — Paradigms of AI alignment: components and enablers, Victoria Krakovna (June 2, 2022).
In a 2022 blog post, Krakovna wrote: "The problem of alignment is getting AI systems to do what we want them to do". — Paradigms of AI alignment: components and enablers, Victoria Krakovna (June 2, 2022).
In a 2022 blog post, Krakovna wrote: "An alignment problem arises when the revealed specification doesn’t match the ideal specification: the system is not doing what we want it to do". — Paradigms of AI alignment: components and enablers, Victoria Krakovna (June 2, 2022).
In a 2022 blog post, Krakovna wrote: "The system’s capabilities generalize but its goal does not, which means the system is competently doing the wrong thing". — Paradigms of AI alignment: components and enablers, Victoria Krakovna (June 2, 2022).
In a 2022 blog post, Krakovna wrote: "Goal misgeneralization is caused by underspecification". — Paradigms of AI alignment: components and enablers, Victoria Krakovna (June 2, 2022).
In a 2022 blog post, Krakovna wrote: "It’s helpful to use more diverse training data". — Paradigms of AI alignment: components and enablers, Victoria Krakovna (June 2, 2022).
In a 2022 blog post, Krakovna wrote: "This is a manifestation of Goodhart’s law: when a metric becomes a target, it ceases to be a good metric". — Paradigms of AI alignment: components and enablers, Victoria Krakovna (June 2, 2022).
In a 2022 blog post, Krakovna wrote: "We expect that the specification gaming problem is only going to get worse as our systems get smarter and better at optimizing for the wrong goal". — Paradigms of AI alignment: components and enablers, Victoria Krakovna (June 2, 2022).
In a 2022 blog post, Krakovna wrote: "it’s difficult to specify all the things the agent should not do while pursuing its goal". — Paradigms of AI alignment: components and enablers, Victoria Krakovna (June 2, 2022).
In a 2022 blog post, Krakovna wrote: "any channel for communicating the reward to the agent could in principle be corrupted by the agent". — Paradigms of AI alignment: components and enablers, Victoria Krakovna (June 2, 2022).
In a 2022 blog post, Krakovna wrote: "A key assumption here is that it’s easier to argue for truth than for falsehood". — Paradigms of AI alignment: components and enablers, Victoria Krakovna (June 2, 2022).
In a 2022 blog post, Krakovna wrote: "If you’re looking to get into alignment research, ELK is a great topic to get started on". — Paradigms of AI alignment: components and enablers, Victoria Krakovna (June 2, 2022).
In a 2022 blog post, Krakovna wrote: "try playing the builder breaker game and see if you can find unexplored parts of the solution space". — Paradigms of AI alignment: components and enablers, Victoria Krakovna (June 2, 2022).
In a 2022 blog post, Krakovna wrote: "I feel somewhat pessimistic about this but I’d love to be wrong on this point since foundational progress is pretty hard". — Paradigms of AI alignment: components and enablers, Victoria Krakovna (June 2, 2022).
In a 2022 blog post, Krakovna wrote: "Mechanistic interpretability aims to build a complete understanding of the systems we build". — Paradigms of AI alignment: components and enablers, Victoria Krakovna (June 2, 2022).
In a 2022 blog post, Krakovna wrote: "This makes me more optimistic about being able to understand advanced AI systems even if they have a somewhat different architecture from today’s systems". — Paradigms of AI alignment: components and enablers, Victoria Krakovna (June 2, 2022).
In a 2022 blog post, Krakovna wrote: "AI systems are also likely to have power-seeking incentives". — Paradigms of AI alignment: components and enablers, Victoria Krakovna (June 2, 2022).
In a 2019 blog post, Krakovna wrote: "safety outreach tools are more likely to be impactful if they are fun and easy for people to engage with". — Retrospective on the specification gaming examples list, Victoria Krakovna (December 20, 2019).
In a 2019 blog post, Krakovna wrote: "This list had a surprising amount of impact relative to how little work it took me to put it together and maintain it". — Retrospective on the specification gaming examples list, Victoria Krakovna (December 20, 2019).
In a 2019 blog post, Krakovna wrote: "This makes me wonder if there is other low hanging fruit in the safety resources space that we haven’t picked yet". — Retrospective on the specification gaming examples list, Victoria Krakovna (December 20, 2019).
In a 2019 blog post, Krakovna wrote: "Any given loophole can seem obvious in hindsight, but 50 loopholes are much less so". — Retrospective on the specification gaming examples list, Victoria Krakovna (December 20, 2019).
In a 2019 blog post, Krakovna wrote: "It’s easier to see that this approach does not scale when presented with 50 examples of gaming behaviors". — Retrospective on the specification gaming examples list, Victoria Krakovna (December 20, 2019).
In a 2019 blog post, Krakovna wrote: "Reward gaming happens when the reward function is designed incorrectly". — Retrospective on the specification gaming examples list, Victoria Krakovna (December 20, 2019).
In a 2019 blog post, Krakovna wrote: "the crowdsourcing aspect is also helpful". — Retrospective on the specification gaming examples list, Victoria Krakovna (December 20, 2019).
In a 2019 blog post, Krakovna wrote: "it’s fun, standardized, up-to-date, comprehensive, and collaborative". — Retrospective on the specification gaming examples list, Victoria Krakovna (December 20, 2019).
In a 2023 blog post, Krakovna wrote: "I remain uncertain about the likelihood of the deceptive alignment part of the threat model". — Retrospective on AI threat models, Victoria Krakovna (December 20, 2023).
In a 2023 blog post, Krakovna wrote: "I still endorse this categorization of threat models and the consensus threat model". — Retrospective on AI threat models, Victoria Krakovna (December 20, 2023).
In a 2018 blog post, Krakovna wrote: "it is particularly referring to behaviors that are clearly hacks, not just suboptimal solutions". — Specification gaming examples in AI, Victoria Krakovna (April 2, 2018).
In a 2018 blog post, Krakovna wrote: "generating a solution that literally satisfies the stated objective but fails to solve the problem according to the human designer’s intent". — Specification gaming examples in AI, Victoria Krakovna (April 2, 2018).