Visual summary of operating lessons from Emma Strubell.

Lessons from Emma Strubell

Emma Strubell is a computer science researcher at Carnegie Mellon University who studies the environmental costs of natural language processing. Her 2019 paper on the carbon footprint of training large language models forced the machine learning field to confront its energy use. This profile collects her insights on computational efficiency, hardware infrastructure, and the policies required to make artificial intelligence sustainable.

Part 1: The Carbon Footprint of Deep Learning

  1. On the scale of emissions: "Training a single large NLP model can emit as much carbon as five cars over their entire lifetimes." — Source: [Energy and Policy Considerations for Deep Learning in NLP]
  2. On ignored costs: "For a long time, the NLP community focused almost entirely on improving accuracy metrics, ignoring the underlying computational cost." — Source: [The TWIML AI Podcast Episode 286]
  3. On financial and environmental dualities: "The computational resources required to achieve state-of-the-art results have both a staggering financial price and a massive carbon footprint." — Source: [Energy and Policy Considerations for Deep Learning in NLP]
  4. On Moore's Law: "The computational requirements of modern deep learning are increasing at a rate that far outpaces Moore's Law, making current trajectories unsustainable." — Source: [Energy and Policy Considerations for Modern Deep Learning Research]
  5. On cloud computing opacity: "Researchers often rely on cloud compute instances without knowing the underlying energy mix or carbon intensity of the data center." — Source: [Energy and Policy Considerations for Deep Learning in NLP]
  6. On lifecycle analysis: "We have to look past the catastrophic headlines about AI and evaluate the actual, measurable lifecycle emissions of these systems." — Source: [DAIR Institute Podcast Episode 19]
  7. On natural language processing: "NLP is particularly compute-intensive because language models require massive amounts of text data and millions of parameters to capture linguistic nuance." — Source: [The TWIML AI Podcast Episode 286]
  8. On neural architecture search: "Techniques like neural architecture search demand an extraordinary amount of compute for marginal gains in accuracy, exacerbating the carbon issue." — Source: [Energy and Policy Considerations for Deep Learning in NLP]
  9. On shifting baselines: "As models grow exponentially larger, what was considered a massive model a few years ago is now just a baseline." — Source: Environmentally Sustainable AI Lecture
  10. On quantifying the abstract: "It is difficult for researchers to conceptualize the environmental impact of their work when the computation happens in a distant, invisible data center." — Source: [The TWIML AI Podcast Episode 286]

Part 2: Hardware and Infrastructure

  1. On GPU manufacturing: "The environmental cost of AI is not just in the electricity used for training, but in the extraction and manufacturing required to build GPUs." — Source: [DAIR Institute Podcast Episode 19]
  2. On water consumption: "Data centers require millions of gallons of water for cooling, a hidden environmental cost that is rarely factored into AI research." — Source: [DAIR Institute Podcast Episode 19]
  3. On specialized hardware: "While TPUs and specialized accelerators are more efficient per operation, the sheer scale of their use offsets these efficiency gains." — Source: [Energy and Policy Considerations for Deep Learning in NLP]
  4. On e-waste: "The rapid obsolescence of AI hardware contributes to a growing e-waste problem that the tech industry is largely ignoring." — Source: Environmentally Sustainable AI Lecture
  5. On geographical location: "Training models in regions powered by renewable energy can significantly reduce the associated carbon emissions compared to regions reliant on coal." — Source: [Energy and Policy Considerations for Deep Learning in NLP]
  6. On data center efficiency: "PUE metrics for data centers are improving, but they don't capture the absolute growth in energy demand driven by AI." — Source: [Energy and Policy Considerations for Modern Deep Learning Research]
  7. On embodied carbon: "The embodied carbon of the hardware infrastructure often rivals or exceeds the operational carbon of running the models." — Source: [DAIR Institute Podcast Episode 19]
  8. On hardware utilization: "Many research clusters operate with low utilization rates, meaning energy is wasted keeping idle machines online." — Source: [The TWIML AI Podcast Episode 286]
  9. On supply chain transparency: "We need greater transparency from hardware vendors regarding the full environmental footprint of their manufacturing processes." — Source: [DAIR Institute Podcast Episode 19]

Part 3: Model Efficiency and Optimization

  1. On algorithmic efficiency: "There is an urgent need to prioritize algorithmic efficiency alongside accuracy when developing new NLP architectures." — Source: [Energy and Policy Considerations for Deep Learning in NLP]
  2. On the limits of scaling: "Simply scaling up model size is a brute-force approach; true innovation lies in achieving better results with fewer parameters." — Source: [The TWIML AI Podcast Episode 286]
  3. On hyperparameter tuning: "The hidden cost of machine learning is the thousands of training runs required for hyperparameter optimization before a final model is published." — Source: [Energy and Policy Considerations for Deep Learning in NLP]
  4. On knowledge distillation: "Techniques like knowledge distillation offer a path to deploy smaller, more efficient models without sacrificing too much performance." — Source: Environmentally Sustainable AI Lecture
  5. On sparse representations: "Moving away from dense representations toward sparse models can drastically reduce the compute needed for inference." — Source: [Large Language Models: Everything's Different and Nothing Has Changed]
  6. On pruning: "Network pruning is a valuable tool for removing redundant weights, though the energy spent on the initial large-scale training must still be accounted for." — Source: Environmentally Sustainable AI Lecture
  7. On early stopping: "Implementing aggressive early stopping criteria can save significant energy during the experimental phase of research." — Source: [Energy and Policy Considerations for Modern Deep Learning Research]
  8. On data quality: "Curating higher quality datasets often yields better results than simply training a massive model on noisy, uncurated web text." — Source: [Large Language Models: Everything's Different and Nothing Has Changed]
  9. On inference vs. training: "While training is incredibly resource-intensive, the aggregate energy cost of inference at scale often dominates over a model's lifetime." — Source: [Energy and Policy Considerations for Deep Learning in NLP]

Part 4: Measuring and Reporting Emissions

  1. On standardized reporting: "The research community must adopt standardized practices for reporting the energy consumption and carbon emissions of their experiments." — Source: [Energy and Policy Considerations for Deep Learning in NLP]
  2. On conference requirements: "Academic conferences should require authors to include an energy cost analysis alongside their traditional performance metrics." — Source: [Energy and Policy Considerations for Deep Learning in NLP]
  3. On estimation tools: "Software tools that estimate carbon footprints based on hardware specifications and runtime are essential for raising awareness among practitioners." — Source: [The TWIML AI Podcast Episode 286]
  4. On the difficulty of measurement: "It is surprisingly difficult to get an accurate reading of power consumption from cloud providers, hindering transparency." — Source: [DAIR Institute Podcast Episode 19]
  5. On carbon intensity data: "Researchers need access to real-time carbon intensity data from grid operators to make informed decisions about when and where to train." — Source: [Energy and Policy Considerations for Deep Learning in NLP]
  6. On full transparency: "A complete accounting of AI's environmental impact must include the failed experiments, not just the successful final run." — Source: [The TWIML AI Podcast Episode 286]
  7. On peer review: "Reviewers should penalize papers that achieve trivial gains in accuracy through massive, unjustified increases in computation." — Source: [Energy and Policy Considerations for Modern Deep Learning Research]
  8. On leaderboards: "Public leaderboards in NLP should track energy efficiency and model size alongside standard accuracy scores like F1." — Source: [Energy and Policy Considerations for Deep Learning in NLP]
  9. On open science: "Sharing pre-trained models openly is one of the most effective ways to prevent the redundant expenditure of training energy across the community." — Source: Environmentally Sustainable AI Lecture

Part 5: Accessibility and Democratization

  1. On academic resource constraints: "Academic institutions are increasingly priced out of competitive NLP research because they cannot afford the compute required to train large models." — Source: [Energy and Policy Considerations for Deep Learning in NLP]
  2. On industry dominance: "The reliance on massive compute infrastructure concentrates research power in the hands of a few large tech companies." — Source: [The TWIML AI Podcast Episode 286]
  3. On equitable AI: "Building equitable AI means developing models that can be run on consumer hardware, not just supercomputers." — Source: Environmentally Sustainable AI Lecture
  4. On low-resource languages: "The compute demands of current architectures make it difficult to develop effective models for low-resource languages that lack commercial backing." — Source: [Large Language Models: Everything's Different and Nothing Has Changed]
  5. On the researcher divide: "We are creating a two-tiered system where those with compute do empirical work, and those without are forced to focus strictly on theory." — Source: [Energy and Policy Considerations for Deep Learning in NLP]
  6. On open-source hardware: "The community would benefit immensely from subsidized, shared computing clusters dedicated to academic AI research." — Source: [Energy and Policy Considerations for Modern Deep Learning Research]
  7. On lightweight models: "We must incentivize the development of lightweight models that provide ninety percent of the performance for ten percent of the computational cost." — Source: [The TWIML AI Podcast Episode 286]
  8. On student accessibility: "It is challenging to teach modern NLP when students cannot run state-of-the-art models on their university-provided laptops." — Source: UW Data Science Seminar
  9. On edge computing: "Deploying models to edge devices requires a fundamental shift in how we approach model compression and efficiency." — Source: Environmentally Sustainable AI Lecture
  10. On reproducible research: "True reproducibility requires that a typical researcher can actually afford the compute necessary to replicate a published result." — Source: [Energy and Policy Considerations for Deep Learning in NLP]

Part 6: AI Policy and Regulation

  1. On government intervention: "Without policy interventions, the AI industry is unlikely to self-regulate its growing environmental footprint." — Source: [Energy and Policy Considerations for Deep Learning in NLP]
  2. On carbon taxes: "Applying a carbon tax to compute usage would quickly force companies to optimize their training pipelines." — Source: [DAIR Institute Podcast Episode 19]
  3. On funding agency roles: "Grant-making organizations should require applicants to detail the environmental impact of their proposed computational research." — Source: [Energy and Policy Considerations for Modern Deep Learning Research]
  4. On corporate responsibility: "Tech companies must be held accountable for the emissions generated by the AI models they deploy in commercial products." — Source: [The Environmental Imprint of AI]
  5. On auditing AI: "We need independent auditing bodies capable of verifying the energy claims made by organizations training massive models." — Source: [DAIR Institute Podcast Episode 19]
  6. On international standards: "The environmental impact of AI is a global issue that requires standardized, international frameworks for measurement and mitigation." — Source: [The Environmental Imprint of AI]
  7. On disclosing energy use: "Legislation may be required to force cloud providers to disclose the precise energy mix and carbon intensity of their facilities." — Source: [Energy and Policy Considerations for Deep Learning in NLP]
  8. On public awareness: "Policymakers generally lack a technical understanding of why AI requires so much energy, making effective regulation difficult." — Source: [The TWIML AI Podcast Episode 286]
  9. On the cost of inaction: "If we do not implement policies now, the energy demands of future AI systems will undermine broader climate change mitigation efforts." — Source: [DAIR Institute Podcast Episode 19]

Part 7: Sustainable NLP Practices

  1. On pragmatic research: "Researchers should ask whether the marginal gain in accuracy scores justifies the massive expenditure of electricity required to achieve it." — Source: [Energy and Policy Considerations for Deep Learning in NLP]
  2. On pre-training norms: "The practice of training a language model from scratch for every new project must be replaced by fine-tuning existing models." — Source: [The TWIML AI Podcast Episode 286]
  3. On targeted fine-tuning: "Parameter-efficient fine-tuning methods allow us to adapt large models to specific tasks with a fraction of the energy." — Source: [Large Language Models: Everything's Different and Nothing Has Changed]
  4. On evaluating models: "We must expand our definition of a capable model to include its speed, size, and energy efficiency, not just its accuracy." — Source: Environmentally Sustainable AI Lecture
  5. On hardware awareness: "NLP practitioners need a better understanding of how their code executes on GPUs to write more energy-efficient implementations." — Source: UW Data Science Seminar
  6. On dataset size: "More data does not always mean a better model; filtering out noise can lead to faster training and lower emissions." — Source: [Large Language Models: Everything's Different and Nothing Has Changed]
  7. On cross-disciplinary collaboration: "Solving the efficiency problem in NLP requires collaboration with hardware architects and systems engineers." — Source: [Energy and Policy Considerations for Modern Deep Learning Research]
  8. On shifting schedules: "Running computationally heavy jobs during times when the electrical grid is drawing from renewable sources is a simple, effective change." — Source: [The TWIML AI Podcast Episode 286]
  9. On reusing computation: "Storing and sharing intermediate checkpoints can prevent other teams from burning energy to recreate the same representations." — Source: [Energy and Policy Considerations for Deep Learning in NLP]

Part 8: The Future of Green AI

  1. On cultural shifts: "The most important change we need is a cultural shift in the AI community to value efficiency as highly as state-of-the-art performance." — Source: [The TWIML AI Podcast Episode 286]
  2. On the role of academia: "Universities have a responsibility to lead research into sustainable AI, rather than just competing with industry on scale." — Source: [The Environmental Imprint of AI]
  3. On long-term viability: "If AI is to be a net positive for society, its development process must become environmentally sustainable." — Source: [DAIR Institute Podcast Episode 19]
  4. On new architectures: "We are likely to see a move away from standard Transformers toward architectures specifically designed to minimize energy consumption." — Source: [Large Language Models: Everything's Different and Nothing Has Changed]
  5. On algorithmic breakthroughs: "The next major breakthrough in AI won't be a larger model, but a drastically more efficient way to learn representations." — Source: Environmentally Sustainable AI Lecture
  6. On community awareness: "It is encouraging to see how quickly the NLP community has started to acknowledge and discuss the climate impact of its work." — Source: [The TWIML AI Podcast Episode 286]
  7. On continuous learning: "Models that can learn continuously without requiring full retraining from scratch will be essential for reducing future energy use." — Source: Environmentally Sustainable AI Lecture
  8. On lifecycle thinking: "Researchers must adopt a lifecycle perspective, considering everything from data collection to inference when evaluating a model's cost." — Source: [Energy and Policy Considerations for Modern Deep Learning Research]
  9. On aligning incentives: "We must restructure academic and industry incentives so that researchers are rewarded for building smaller, faster models." — Source: [Energy and Policy Considerations for Deep Learning in NLP]
  10. On optimism: "Despite the scale of the challenge, the technical talent in the AI field makes me optimistic that we can engineer our way to sustainable machine learning." — Source: [DAIR Institute Podcast Episode 19]