> ## Content Index
> Fetch the complete content index at: https://www.antoinebuteau.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Lessons from Sergey Edunov
- URL: https://www.antoinebuteau.com/lessons-from-sergey-edunov/
- Published: 2026-07-29T02:35:09.000Z
- Updated: 2026-09-05T03:02:04.000Z
- Description: Sergey Edunov is a physicist turned software engineer who led large scale model training at Meta for Llama 2 and machine translation.
- Author: Antoine Buteau
- Tags: Profile, AI & Machine Learning Profiles

Sergey Edunov is a physicist turned software engineer who led large-scale model training at Meta for Llama 2 and machine translation. He worked extensively on distributed graph processing and scaling sequence-to-sequence models before moving into drug discovery as CTO of Genesis Molecular AI. This profile covers his work optimizing training infrastructure, using synthetic data, and applying generative AI to molecular biology.

![Visual summary of operating lessons from Sergey Edunov.](https://www.antoinebuteau.com/content/images/2026/07/lessons-from-sergey-edunov-profile-infographic.webp)

## Part 1: Large-Scale Graph Processing and Infrastructure

1. **On Distributed System Selection:** Apache Giraph was chosen for collaborative filtering at Facebook because of its ability to handle massive datasets and its extensibility. — *Reference:* [*Recommending items to more than a billion people - Engineering at Meta*](https://engineering.fb.com/2015/06/02/core-infra/recommending-items-to-more-than-a-billion-people/?ref=antoinebuteau.com)
2. **On Matrix Factorization Formulation:** Instead of finding an exact solution for matrix factorization, iterative approaches that start from random feature vectors and gradually improve the solution are used to handle extreme sparsity. — *Reference:* [*Recommending items to more than a billion people - Engineering at Meta*](https://engineering.fb.com/2015/06/02/core-infra/recommending-items-to-more-than-a-billion-people/?ref=antoinebuteau.com)
3. **On Network Bottlenecks in Matrix Factorization:** The primary limitation in distributed matrix factorization algorithms is the massive volume of data transmitted across the network during iterations. — *Reference:* [*Recommending items to more than a billion people - Engineering at Meta*](https://engineering.fb.com/2015/06/02/core-infra/recommending-items-to-more-than-a-billion-people/?ref=antoinebuteau.com)
4. **On Skewed Item Degrees:** Highly popular items in datasets cause memory constraints and processing bottlenecks, as some nodes receive disproportionately large amounts of data. — *Reference:* [*Recommending items to more than a billion people - Engineering at Meta*](https://engineering.fb.com/2015/06/02/core-infra/recommending-items-to-more-than-a-billion-people/?ref=antoinebuteau.com)
5. **On Mitigating Network Traffic:** To reduce data transfer, Facebook developed a rotational hybrid approach where items are partitioned among workers and passed along a ring during processing. — *Reference:* [*Recommending items to more than a billion people - Engineering at Meta*](https://engineering.fb.com/2015/06/02/core-infra/recommending-items-to-more-than-a-billion-people/?ref=antoinebuteau.com)
6. **On System Scalability Comparisons:** "Giraph can currently process at least 50x larger graphs than GraphX." — *Source:* [*engineering.fb.com*](https://engineering.fb.com/2016/10/19/core-infra/a-comparison-of-state-of-the-art-graph-processing-systems/?ref=antoinebuteau.com)
7. **On Memory Efficiency:** Giraph proved to be significantly more memory-efficient than GraphX, requiring far less total memory for a given graph size. — *Reference:* [*A comparison of state-of-the-art graph processing systems - Engineering at Meta*](https://engineering.fb.com/2016/10/19/core-infra/a-comparison-of-state-of-the-art-graph-processing-systems/?ref=antoinebuteau.com)
8. **On Resource Allocation:** When running graph processing frameworks, it is often more efficient to allocate all available memory to a single worker process per physical machine rather than running multiple workers. — *Reference:* [*A comparison of state-of-the-art graph processing systems - Engineering at Meta*](https://engineering.fb.com/2016/10/19/core-infra/a-comparison-of-state-of-the-art-graph-processing-systems/?ref=antoinebuteau.com)
9. **On Algorithmic Patterns:** The performance of graph processing systems varies because different algorithms have distinct computation and communication patterns that stress different system components. — *Reference:* [*A comparison of state-of-the-art graph processing systems - Engineering at Meta*](https://engineering.fb.com/2016/10/19/core-infra/a-comparison-of-state-of-the-art-graph-processing-systems/?ref=antoinebuteau.com)

## Part 2: Scaling Neural Machine Translation

1. **On Hardware Optimization:** Switching from 32-bit to 16-bit floating-point precision reduced GPU memory footprints and allowed the use of optimized Tensor Cores without compromising model quality. — *Reference:* [*Scaling neural machine translation to bigger data sets with faster training and inference - Engineering at Meta*](https://engineering.fb.com/2018/09/07/ai-research/scaling-neural-machine-translation-to-bigger-data-sets-with-faster-training-and-inference/?ref=antoinebuteau.com)
2. **On Delayed Gradient Synchronization:** By accumulating results from several mini-batches before communicating across workers, they minimized network overhead and mitigated the impact of slower workers. — *Reference:* [*Scaling neural machine translation to bigger data sets with faster training and inference - Engineering at Meta*](https://engineering.fb.com/2018/09/07/ai-research/scaling-neural-machine-translation-to-bigger-data-sets-with-faster-training-and-inference/?ref=antoinebuteau.com)
3. **On Increasing Batch Sizes:** Accumulating updates across batches effectively increased the overall batch size, which permitted a doubling of the learning rate and significantly reduced training time. — *Reference:* [*Scaling neural machine translation to bigger data sets with faster training and inference - Engineering at Meta*](https://engineering.fb.com/2018/09/07/ai-research/scaling-neural-machine-translation-to-bigger-data-sets-with-faster-training-and-inference/?ref=antoinebuteau.com)
4. **On Overlapping Operations:** Training time can be further reduced by initiating gradient synchronization for a portion of the neural network while the backward pass is still completing. — *Reference:* [*Scaling neural machine translation to bigger data sets with faster training and inference - Engineering at Meta*](https://engineering.fb.com/2018/09/07/ai-research/scaling-neural-machine-translation-to-bigger-data-sets-with-faster-training-and-inference/?ref=antoinebuteau.com)
5. **On Distributed Scaling Efficiency:** Optimizations originally designed for single-machine training translated well to distributed environments, accelerating training across 128 GPUs by over 38x. — *Reference:* [*Scaling neural machine translation to bigger data sets with faster training and inference - Engineering at Meta*](https://engineering.fb.com/2018/09/07/ai-research/scaling-neural-machine-translation-to-bigger-data-sets-with-faster-training-and-inference/?ref=antoinebuteau.com)
6. **On Translating Speeds:** Optimization strategies like caching finished sentences and batching by word count rather than sentence count can dramatically improve inference speeds for translation models. — *Reference:* [*Scaling neural machine translation to bigger data sets with faster training and inference - Engineering at Meta*](https://engineering.fb.com/2018/09/07/ai-research/scaling-neural-machine-translation-to-bigger-data-sets-with-faster-training-and-inference/?ref=antoinebuteau.com)
7. **On Scaling Multilingual Models:** Building a single translation model for hundreds of languages is difficult because performance typically degrades as more language directions are added to the architecture. — *Reference:* [*200 languages within a single AI model: A breakthrough in high-quality machine translation*](https://ai.meta.com/blog/nllb-200-high-quality-machine-translation/?ref=antoinebuteau.com)

## Part 3: Data Augmentation and Model Training Techniques

1. **On Utilizing Monolingual Data:** Back-translation is highly effective for improving neural machine translation models using texts available in only the target language. — *Reference:* [*Scaling neural machine translation to bigger data sets with faster training and inference - Engineering at Meta*](https://engineering.fb.com/2018/09/07/ai-research/scaling-neural-machine-translation-to-bigger-data-sets-with-faster-training-and-inference/?ref=antoinebuteau.com)
2. **On Synthetic Data Generation:** "We find that in all but resource poor settings back-translations obtained via sampling or noised beam outputs are most effective." — *Source:* [*arxiv.org*](https://arxiv.org/abs/1808.09381?ref=antoinebuteau.com)
3. **On Training Signals:** "Our analysis shows that sampling or noisy synthetic data gives a much stronger training signal than data generated by beam or greedy search." — *Source:* [*arxiv.org*](https://arxiv.org/abs/1808.09381?ref=antoinebuteau.com)
4. **On Structured Prediction Losses:** Classical objective functions historically used for linear models can be adapted to neural sequence-to-sequence models with strong results. — *Reference:* [*Classical Structured Prediction Losses for Sequence to Sequence Learning*](https://arxiv.org/abs/1711.04956?ref=antoinebuteau.com)
5. **On Outperforming Beam Search:** "Our experiments show that these losses can perform surprisingly well by slightly outperforming beam search optimization in a like for like setup." — *Source:* [*arxiv.org*](https://arxiv.org/abs/1711.04956?ref=antoinebuteau.com)
6. **On Data Filtering:** Scaling translation to hundreds of languages requires aggressive filtering steps, using identification models to remove noise and hallucinated toxicity from internet-scale corpora. — *Reference:* [*200 languages within a single AI model: A breakthrough in high-quality machine translation*](https://ai.meta.com/blog/nllb-200-high-quality-machine-translation/?ref=antoinebuteau.com)
7. **On Mining Translation Data:** Sourcing parallel sentences directly from the web often yields poor results due to spelling inconsistencies, missing diacritical marks, and disparate source texts. — *Reference:* [*200 languages within a single AI model: A breakthrough in high-quality machine translation*](https://ai.meta.com/blog/nllb-200-high-quality-machine-translation/?ref=antoinebuteau.com)

## Part 4: Open Source Foundation Models

1. **On the Power of Open Source:** "We believe that state-of-the-art AI technology is safer and better aligned when it’s open and accessible to everyone." — *Source:* [*ai.meta.com*](https://ai.meta.com/blog/llama-2-updates-connect-2023/?ref=antoinebuteau.com)
2. **On Community Innovation:** Releasing foundational models allows the broader research community to contribute novel techniques, performance enhancements, and evaluation methods faster than a single company could. — *Reference:* [*The Llama Ecosystem: Past, Present, and Future*](https://ai.meta.com/blog/llama-2-updates-connect-2023/?ref=antoinebuteau.com)
3. **On Enterprise Feedback Loops:** As more startups and enterprises build upon open-source technology, creators gain valuable insights into commercial use cases and safe deployment strategies. — *Reference:* [*The Llama Ecosystem: Past, Present, and Future*](https://ai.meta.com/blog/llama-2-updates-connect-2023/?ref=antoinebuteau.com)
4. **On Standardizing Ecosystems:** Providing a shared foundation model establishes a common language for the developer community, spurring the creation of new tools that eventually accelerate internal development as well. — *Reference:* [*The Llama Ecosystem: Past, Present, and Future*](https://ai.meta.com/blog/llama-2-updates-connect-2023/?ref=antoinebuteau.com)
5. **On Extending Capabilities:** The open-source community rapidly adapts released models to support larger context windows, additional languages, and edge-device deployments. — *Reference:* [*The Llama Ecosystem: Past, Present, and Future*](https://ai.meta.com/blog/llama-2-updates-connect-2023/?ref=antoinebuteau.com)

## Part 5: Physics, Machine Learning, and Molecular Biology

1. **On Interdisciplinary Overlap:** Many of the methodologies and mathematical foundations utilized in modern machine learning bear a strong resemblance to concepts found in physics. — *Reference:* [*🔬 The Coolest Diffusion Research Isn't in LLMs — Evan Feinberg & Sergey Edunov, Genesis Molecular AI Transcript — Latent Space: The AI Engineer Podcast | spoken.md*](https://spoken.md/episode/the-coolest-diffusion-research-isnt-in-llms-evan-1000775021934?ref=antoinebuteau.com)
2. **On the Scale of Drug Discovery:** "There are 10 to the 60 drug-like small molecules in the universe… it’s like finding a needle in a haystack, where everything except your needle is very, very dangerous." — *Source:* [*latent.space*](https://www.latent.space/p/the-coolest-diffusion-research-isnt?ref=antoinebuteau.com)
3. **On Multi-Parameter Optimization:** Developing small molecule drugs requires balancing conflicting properties, such as a molecule's binding affinity and its solubility in the bloodstream. — *Reference:* [*🔬 The Coolest Diffusion Research Isn't in LLMs — Evan Feinberg & Sergey Edunov, Genesis Molecular AI*](https://www.latent.space/p/the-coolest-diffusion-research-isnt?ref=antoinebuteau.com)
4. **On Industry Benchmarks:** Common structural prediction benchmarks, like the 2 Angstrom RMSD threshold, often accept inaccurate molecular poses that fail to model precise interactions like hydrogen bonds. — *Reference:* [*🔬 The Coolest Diffusion Research Isn't in LLMs — Evan Feinberg & Sergey Edunov, Genesis Molecular AI*](https://www.latent.space/p/the-coolest-diffusion-research-isnt?ref=antoinebuteau.com)
5. **On Diffusion in Biology:** Rather than large language models, the most innovative applications of diffusion research are currently occurring in 3D molecular structure prediction. — *Reference:* [*🔬 The Coolest Diffusion Research Isn't in LLMs — Evan Feinberg & Sergey Edunov, Genesis Molecular AI*](https://www.latent.space/p/the-coolest-diffusion-research-isnt?ref=antoinebuteau.com)
6. **On Modeling Protein Flexibility:** Genesis' PEARL model succeeded by accurately predicting how a protein structure moves and adjusts to accommodate a ligand, a process known as induced fit. — *Reference:* [*🔬 The Coolest Diffusion Research Isn't in LLMs — Evan Feinberg & Sergey Edunov, Genesis Molecular AI*](https://www.latent.space/p/the-coolest-diffusion-research-isnt?ref=antoinebuteau.com)
7. **On Model Precision:** "Where PEARL was exceptionally good is figuring out how to move this loop. We are basically correct for every single pose." — *Source:* [*latent.space*](https://www.latent.space/p/the-coolest-diffusion-research-isnt?ref=antoinebuteau.com)
8. **On Generative AI in Chemistry:** Advances in structural prediction models have reached a threshold where agentic drug discovery loops—systems that iterate, form hypotheses, and propose candidates autonomously—are becoming feasible. — *Reference:* [*🔬 The Coolest Diffusion Research Isn't in LLMs — Evan Feinberg & Sergey Edunov, Genesis Molecular AI*](https://www.latent.space/p/the-coolest-diffusion-research-isnt?ref=antoinebuteau.com)

## Part 6: Retrieval, Representations, and Open Research Tools

1. **On Where Pretraining Helps Most:** In sequence-to-sequence systems, adding pretrained language representations to the encoder produced the strongest gains while increasing inference time by only 14 percent. — *Reference:* [*Pre-trained Language Model Representations for Language Generation*](https://arxiv.org/abs/1903.09722?ref=antoinebuteau.com)
2. **On Pretraining Beyond Low-Resource Tasks:** Pretrained representations delivered gains of up to 5.3 BLEU in a simulated low-resource setting and still improved translation when millions of sentence pairs were available. — *Reference:* [*Pre-trained Language Model Representations for Language Generation*](https://arxiv.org/abs/1903.09722?ref=antoinebuteau.com)
3. **On Learned Retrieval:** Open-domain question answering can retrieve useful passages with dense representations alone, using a straightforward dual-encoder trained on a relatively small set of question-passage examples. — *Reference:* [*Dense Passage Retrieval for Open-Domain Question Answering*](https://arxiv.org/abs/2004.04906?ref=antoinebuteau.com)
4. **On Measuring Retrieval Progress:** A dense retriever improved top-20 passage retrieval accuracy by 9 to 19 percentage points over a strong Lucene-BM25 baseline across several open-domain question-answering datasets. — *Reference:* [*Dense Passage Retrieval for Open-Domain Question Answering*](https://arxiv.org/abs/2004.04906?ref=antoinebuteau.com)
5. **On Building Research Infrastructure:** A useful open-source research toolkit should support multiple sequence-modeling tasks rather than locking researchers into one application or architecture. — *Reference:* [*fairseq: A Fast, Extensible Toolkit for Sequence Modeling*](https://arxiv.org/abs/1904.01038?ref=antoinebuteau.com)
6. **On Making Scale Reusable:** fairseq combined multi-GPU and multi-machine distributed training with fast mixed-precision training and inference, turning performance techniques into shared research infrastructure. — *Reference:* [*fairseq: A Fast, Extensible Toolkit for Sequence Modeling*](https://arxiv.org/abs/1904.01038?ref=antoinebuteau.com)

## Part 7: Document Context and Long-Context Scaling

1. **On Simplicity in Document Translation:** Back-translating monolingual document-level data matched more elaborate document-translation architectures and decoding methods on both specialized metrics and human evaluation. — *Reference:* [*A Comparison of Approaches to Document-level Machine Translation*](https://arxiv.org/abs/2101.11040?ref=antoinebuteau.com)
2. **On Evaluating Coherence:** Document-level translation methods should be judged with tests designed for cross-sentence phenomena and with human evaluation, not only sentence-level quality scores. — *Reference:* [*A Comparison of Approaches to Document-level Machine Translation*](https://arxiv.org/abs/2101.11040?ref=antoinebuteau.com)
3. **On Extending Context Efficiently:** Foundation models can reach effective 32,768-token context windows by continually pretraining an existing Llama 2 model on longer sequences rather than starting over. — *Reference:* [*Effective Long-Context Scaling of Foundation Models*](https://arxiv.org/abs/2309.16039?ref=antoinebuteau.com)
4. **On Training from a Strong Base:** Long-context continual pretraining was more efficient and similarly effective compared with pretraining a long-context model from scratch. — *Reference:* [*Effective Long-Context Scaling of Foundation Models*](https://arxiv.org/abs/2309.16039?ref=antoinebuteau.com)
5. **On Long-Context Data Design:** Simply supplying abundant long documents is not enough; the pretraining data mix and the curriculum of sequence lengths materially affect long-context performance. — *Reference:* [*Effective Long-Context Scaling of Foundation Models*](https://arxiv.org/abs/2309.16039?ref=antoinebuteau.com)
6. **On Cost-Effective Long-Context Tuning:** Instruction tuning for long-context tasks can work without human-authored long-form instruction data, showing that careful procedure design can substitute for expensive annotation. — *Reference:* [*Effective Long-Context Scaling of Foundation Models*](https://arxiv.org/abs/2309.16039?ref=antoinebuteau.com)