Lessons from Carl Shulman

Lessons from Carl Shulman

Carl Shulman is an advisor to the Open Philanthropy project and a research associate at the Future of Humanity Institute at Oxford. — Carl Shulman (Pt 1) - Intelligence Explosion, Primate Evolution, Robot Doublings, & Alignment.

Part 1: Intelligence Explosions and Takeoff Speeds

1. On AI and Shared Epistemology: carl-shulman: "My actual best guess is that the result of these technologies comes out hugely in favour of improved epistemology, and we get largely convergence on empirical truth wherever it exists." — Carl Shulman on government and society after AGI (Part 2).
2. On Why Imitation Is a Handy Way to Learn: carl-shulman: "The community knows a lot of things, which is the reason why we have evolved such strong drives to imitate, because it’s a very handy way to learn and get configured in society." — Carl Shulman on the common-sense case for existential risk work and its practical implications.
3. On Diminishing Returns in Catastrophe Work: carl-shulman: "Certainly if you move further along the diminishing returns curve for global catastrophic risks, eventually the expected lives saved would get worse than bed nets." — Carl Shulman on the common-sense case for existential risk work and its practical implications.
4. On Seeing AI Systems as Creatures: carl-shulman: "Once the AI can keep character, can engage on an extended, ongoing basis like a human, I think people will form intuitions that are more in the direction of, this is a creature and not just an object." — Carl Shulman on the economy and national security after AGI (Part 1).
5. On Systematic Disease Surveillance: carl-shulman: "I think we should think about it as like, it’s worth it to have systematic surveillance, it pays at the level of individual doctors’ offices and hospitals to be getting systematic disease information." — Carl Shulman on the common-sense case for existential risk work and its practical implications.
6. On How We Treat the Creatures in Our Society: carl-shulman: "I still think it’s an area that it’s worth some people doing research and developing capacity, because it really does matter how we treat most of the creatures in our society." — Carl Shulman on the economy and national security after AGI (Part 1).
7. On Building a Place for the Economy to Happen: carl-shulman: "The United Arab Emirates built up Dubai, Abu Dhabi and has been trying to expand this non-oil economy by just creating a place for it to happen and providing a favourable environment." — Carl Shulman on the economy and national security after AGI (Part 1).
8. On How Hard Extinction From a Pandemic Is: carl-shulman: "I think COVID-19 has showed a lot of damage can still be inflicted, but also they’re very hard to cause extinction because we can change our behavior a lot, we can restrict spread." — Carl Shulman on the common-sense case for existential risk work and its practical implications.
9. On Economists and Explosive Growth: carl-shulman: "Indeed, the vast majority of economists seem to assign extremely low probability to any scenario where growth even increases by as much as, say, it did during the Industrial Revolution." — Carl Shulman on the economy and national security after AGI (Part 1).
10. On Growth Accelerating or Decelerating: carl-shulman: "I think actually the case is actually better that growth will resume accelerating or continue decelerating rather than stay on a constant exponential for hundreds or thousands of years." — Carl Shulman on the common-sense case for existential risk work and its practical implications.

Part 2: The Economics of Artificial General Intelligence

1. On Exaggerated Fears of Automation: carl-shulman: "In other fields, there’s a history of people being concerned, say, that automation would soon cause mass unemployment or like huge reductions in hours worked per week that were exaggerated." — Carl Shulman on the economy and national security after AGI (Part 1).
2. On Civilizational Collapse and the Most Important Century: carl-shulman: "If 99% of people died from super COVID and then we don’t get back to our current technology level for another 500 years, then obviously then this is not the most important century." — Carl Shulman on the common-sense case for existential risk work and its practical implications.
3. On AI Workers Beating Humans on Nearly Every Feature: carl-shulman: "If we think about the actual physical and mental capacities of a worker, then the AI and robot provider is going to do better on almost every objective feature you can give, unless it’s basically like a pure taste-based discrimination." — Carl Shulman on the economy and national security after AGI (Part 1).
4. On Surviving Chip Boom and Bust: carl-shulman: "TSMC is a survivor of many semiconductor firms going bust, because when the boom and bust of chip production goes, during the bust, companies that have overinvested can then die." — Carl Shulman on the economy and national security after AGI (Part 1).
5. On Acting Wrongly During a Short Transition: carl-shulman: "I think because of the pace of AI progress and the self-catalysing nature of AI progress, that period will be short, and we should worry about acting wrongly in the course of that." — Carl Shulman on the economy and national security after AGI (Part 1).
6. On Where Politics and Policy Are Juiciest: carl-shulman: "Probably the places where it really seems the juiciest, perhaps, are in politics and policy, which is an area that most people engaging within it put relatively low effort in." — Carl Shulman on government and society after AGI (Part 2).
7. On the Costs and Benefits of Intelligence: carl-shulman: "If you actually look at it quantitatively that's not true and even in recent history it looks like a pretty close balance between the costs and the benefits of having more cognitive abilities." — Carl Shulman (Pt 1) - Intelligence Explosion, Primate Evolution, Robot Doublings, & Alignment.
8. On Taking Low-Hanging Fruit in Important Questions: carl-shulman: "Hopefully that's something that can change in the future but for my career it's been a matter of taking low-hanging fruit of important questions that sadly people haven't invested in doing the basic analysis on I’m glad to hear that." — Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future.
9. On Improving Reliability and Honesty: carl-shulman: "I think, on the whole, we should look forward to the incremental changes from really souping up and improving the reliability and honesty and robustness and sophistication of those processes." — Carl Shulman on government and society after AGI (Part 2).
10. On Resistance to Operation Warp Speed: carl-shulman: "One reason why there was resistance to Operation Warp Speed and similar efforts is you’re supporting the development of vaccines before they’ve been fully tested on the actual pandemic." — Carl Shulman on government and society after AGI (Part 2).

Part 3: AI Alignment and Takeover Mechanisms

1. On Enforcing Any Restriction: carl-shulman: "In avoiding it, just like the problem of banning nuclear weapons, if you’re going to restrict it, you have to set up a system such that any attempt to break that arrangement is itself stopped." — Carl Shulman on government and society after AGI (Part 2).
2. On Practical Discoveries Driving Influence: carl-shulman: "As it generated more practical industrial discoveries, answers that were checked and solid against other dimensions of knowledge acquisition, it became very popular, very powerful, very influential." — Carl Shulman on government and society after AGI (Part 2).
3. On the Timing of a Pause: carl-shulman: "Even worse, it’s possible that a pause, especially a voluntary pause, then is disproportionately giving up the opportunity to do pauses at that later stage when things are more important." — Carl Shulman on government and society after AGI (Part 2).
4. On Lab Escapes After the Epidemic: carl-shulman: "The bioweapons testing incident in the Soviet Union and similarly SARS-1 had several lab escapes that caused infections after the epidemic was over." — Carl Shulman on the common-sense case for existential risk work and its practical implications | 80,000 Hours.
5. On AI and War Fighters: carl-shulman: "Existing AI can incrementally increase the effectiveness of war fighters in various ways, but you won’t have automated tanks, planes, robots doing their infrastructure and maintenance, et cetera." — Carl Shulman on government and society after AGI (Part 2) | 80,000 Hours.
6. On Imagining a Changed World: carl-shulman: "I think it shows that people have a very hard time actually fully considering a hypothetical of a world that has changed from our current one in significant ways." — Carl Shulman on the economy and national security after AGI (Part 1) | 80,000 Hours.
7. On Catching One in a Million Crimes: carl-shulman: "If human law enforcement would catch one in a million crimes then that wouldn't influence the behavior of people committing crimes at all because they expect that this is very unlikely to affect them." — Carl Shulman (Pt 1) - Intelligence Explosion, Primate Evolution, Robot Doublings, & Alignment.
8. On Learning Most at the Beginning: carl-shulman: "If the situation is ill understood, then you learn the most about it at the very beginning because then you get to rule out the possibility of the situation being super unstable." — Carl Shulman on the common-sense case for existential risk work and its practical implications | 80,000 Hours.

Part 4: National Security and Government Coordination

1. On a United Front of Experts: carl-shulman: "We seem to have avoided a world where governments are making these decisions in the face of a united front from AI expert voices saying, don't worry about it, we've got it under control." — Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future.
2. On Lab Noises About Timelines: carl-shulman: "The people at OpenAI and DeepMind are making noises suggesting timelines in line with what we've discussed and saying there is serious risk of apocalyptic outcomes from them." — Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future.
3. On Assessing Nuclear Damage Early: carl-shulman: "Early on, people assessed the damage from nuclear weapons in terms of direct destruction, fire and killing, destroying cities, and then radioactive fallout." — Carl Shulman on the common-sense case for existential risk work and its practical implications | 80,000 Hours.
4. On Quick Challenge Trials: carl-shulman: "If we had intense, quick challenge trials in like the first months of 2020, and then shown that the vaccines work as we now know they do and as scientists commonly thought they would work earlier on." — Carl Shulman on the common-sense case for existential risk work and its practical implications | 80,000 Hours.
5. On Diplomacy With a Nuclear Power: carl-shulman: "Diplomatic relations with a power that has enough nuclear weapons to destroy your country is just different than negotiations with a random rogue citizen engaging in criminal activity or an employee." — Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future.
6. On the Difficulty of Improving Chips: carl-shulman: "We can look at the increasing difficulty of improving chips and sure, each time you double the performance of computers it’s harder and as we approach physical limits eventually it becomes impossible." — Carl Shulman (Pt 1) - Intelligence Explosion, Primate Evolution, Robot Doublings, & Alignment.
7. On Finding Hidden Submarines: carl-shulman: "The ability to find hidden nuclear subs, which is an important part in nuclear deterrence, AI interpretation of that sensor data may find where all those subs are allowing them to be struck first." — Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future.
8. On Masks and SARS-Affected Areas: carl-shulman: "Western medical authorities adopting the view already held in the previously SARS-affected areas that masks are a good idea given the state of the evidence, things like that." — Carl Shulman on the common-sense case for existential risk work and its practical implications | 80,000 Hours.
9. On Relying on Data in Forecasting: carl-shulman: "My approach compared to some other people in forecasting and assessing some of these things, I try to obtain and rely on any data that I can find that is relevant." — Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future.

Part 5: Cognitive Enhancement and Genetics

1. On Better Epistemology: carl-shulman: "My actual best guess is that the result of these technologies comes out hugely in favour of improved epistemology, and we get largely convergence on empirical truth wherever it exists." — Carl Shulman on government and society after AGI (Part 2) | 80,000 Hours.
2. On Mean Income Versus Skilled Wages: carl-shulman: "In the world at large right now, mean income is on the order of $10,000 a year but in rich countries, skilled workers earn more than a hundred thousand per year." — Carl Shulman (Pt 1) - Intelligence Explosion, Primate Evolution, Robot Doublings, & Alignment.

Part 6: Space Expansion and the Long-Term Future

1. On Compute-Intensive Industry Off Earth: carl-shulman: "If you’re going to have a compute or energy intensive industry that makes information goods that don’t need to be colocated with people on Earth, then sure, get them off Earth." — Carl Shulman on the economy and national security after AGI (Part 1) | 80,000 Hours.
2. On Solar Efficiency: carl-shulman: "We have solar panels with efficiencies of tens of percent, and it’s possible to make multi-junction cells that absorb multiple wavelengths of light, and the theoretical limit for those is very high." — Carl Shulman on the economy and national security after AGI (Part 1) | 80,000 Hours.
3. On Managing Information in Authoritarian States: carl-shulman: "When I think about, say, an application in North Korea or in the People’s Republic of China, it is already the official doctrine that information needs to be managed in many ways to manipulate the opinions and loyalties of the population." — Carl Shulman on government and society after AGI (Part 2) | 80,000 Hours.
4. On Different People Liking Different Things: carl-shulman: "Different people tend to like somewhat different things and some of that may persist over time rather than everyone coming to agree on one particular monoculture or a very repetitive thing being the best thing to fill all of the available space with." — Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future.
5. On Consciousness Arguments and Sapient Creatures: carl-shulman: "In the short term, appeals to hard problem of consciousness issues or dualism will be the basis for some people saying they can do whatever they like with these sapient creatures that seem to or behave as though they have various desires." — Carl Shulman on the economy and national security after AGI (Part 1) | 80,000 Hours.

Part 7: Animal Welfare and Moral Status

1. On Policing Animal Abuse: carl-shulman: "Even in cases where there’s intentional abuse, harm or torture of nonhuman animals, there’s very little investment of policing resources or investigation to make it actually happen." — Carl Shulman on the economy and national security after AGI (Part 1) | 80,000 Hours.

Part 8: Existential Risk Mitigation and Career Choice

1. On Saving Lives Like Highway Construction: carl-shulman: "The cost benefit works out just fine in terms of just saving American lives at the rates you would, say, arrange highway construction to reduce road accidents." — Carl Shulman on the common-sense case for existential risk work and its practical implications | 80,000 Hours.
2. On Being Better Placed to Make Change: carl-shulman: "If you happen to be fortunate enough to have more financial resources or be part of a more narrow scientific community, you might be 100, 1,000 times better placed than the average to make change." — Carl Shulman on the common-sense case for existential risk work and its practical implications | 80,000 Hours.
3. On Publicly Known Honest Advice: carl-shulman: "Having this publicly accessible thing, where everyone knows that HonestGPT is saying this, then you look much worse to the voters when you go against the advice that everyone knows is the best estimate, and then things go wrong. > > I think from that, you get vaccines being produced in quantity earlier and used." — Carl Shulman on government and society after AGI (Part 2) | 80,000 Hours.
4. On Preserving the Long-Term Future: carl-shulman: "I think I should first clarify that I’m more into efforts to preserve and improve the long-term future than probably 99% of people." — Carl Shulman on the common-sense case for existential risk work and its practical implications | 80,000 Hours.
5. On Assembling Lists of Catastrophic Risks: carl-shulman: "Trying to find comprehensive assemblies of major technologies people have discussed and the lists of possible disasters and catastrophic risks people have considered." — Carl Shulman on the common-sense case for existential risk work and its practical implications | 80,000 Hours.
6. On Only Having So Many Data Points: carl-shulman: "When we talk about learning from these long-term macro-scale things — things like predicting quantum mechanics before seeing any data about it — there’s a problem that you still only have so many data points, because a lot of the world is correlated." — Carl Shulman on government and society after AGI (Part 2) | 80,000 Hours.
7. On Countries That Weathered COVID: carl-shulman: "The fact that some countries have basically weathered COVID unscathed, that gives you a stronger argument against existential risk from anything that’s susceptible to these sorts of countermeasures." — Carl Shulman on the common-sense case for existential risk work and its practical implications | 80,000 Hours.
8. On a Continuous Tug Toward Truth: carl-shulman: "What you want is a continuous steady tug from things like regular scientific experiments, and seeing what worked, and what didn’t, things like mathematics that draw us towards the truth." — Carl Shulman on the common-sense case for existential risk work and its practical implications | 80,000 Hours.