Visual summary of operating lessons from Dawn Song.

Lessons from Dawn Song

Dawn Song is a UC Berkeley computer science professor and the founder of Oasis Labs. Her work connects AI security, privacy and confidential computing. This profile examines her research and views on adversarial machine learning, the attacker–defender balance, data rights and privacy-preserving systems. — WIRED25 Conversation.

Part 1: The Cybersecurity Arms Race

  1. Historical Patterns: In new technologies, attackers tend to follow development closely and may sometimes lead it. Song argues that AI safety must account for adversarial use from the start. — FAR.AI AI Safety Talk.
  2. Reactive Defense: Finding and patching bugs remains useful, but deployment can lag behind attackers. Song argues for more proactive, secure-by-construction defenses. — FAR.AI Cybersecurity Talk.
  3. Speed of Exploitation: Advances in AI reasoning and coding can reduce the cost and increase the scale of cyberattacks. Song’s CyberGym work also shows agents finding previously unknown vulnerabilities. — FAR.AI Cybersecurity Talk.
  4. Asymmetric Warfare: Attackers need one workable path while defenders must address many vulnerabilities; Song argues that this asymmetry may let near-term AI advances favor attackers. — FAR.AI Cybersecurity Talk.
  5. Threat Intelligence and False Positives: Song described TrafficComber as using streaming and machine-learning methods to identify anomalies in network traffic; her Sting work used exploit semantics to improve detection accuracy and reduce false alarms. — Sting Security Talk.
  6. Automated Vulnerability Discovery: CyberGym evaluates agents on real open-source vulnerabilities and reports that they can also discover previously unknown bugs. Its benchmark exposes substantial remaining difficulty. — CyberGym Research Paper.
  7. Signature Learning Under Attack: Song’s coauthored work showed that attacker-controlled training examples can obstruct systems that learn signatures for polymorphic worms, even when samples are correctly labeled. — Paragraph Research Paper.
  8. Attack Costs and Incentives: Song notes that attackers already have diverse incentives and that stronger AI coding and reasoning can lower attack costs while raising the burden of defense. — FAR.AI Cybersecurity Talk.

Part 2: Adversarial Machine Learning

  1. Adversarial Examples: Song describes adversarial examples as a broad weakness across deep-learning tasks and model classes, including attacks in black-box and physical settings. — FAR.AI AI Safety Talk.
  2. Physical-World Attacks: In research Song coauthored, stickers on a real stop sign induced targeted misclassification by a road-sign classifier in lab and moving-vehicle footage. This was not a demonstration on an autonomous vehicle. — Physical-World Attacks Paper.
  3. Data-Poisoning Backdoors: Song’s coauthored study showed that a small set of poisoned training samples can implant a hard-to-notice trigger in a learning-based authentication model. — Backdoor Attacks Paper.
  4. Model Inversion: Song’s coauthored work demonstrated that access to a trained model can reveal sensitive features of training data, including recognizable face information in its tests. — Model-Inversion Research Paper.
  5. Adversarial Training: In a coauthored study, curriculum adversarial training improved resistance to tested attacks while retaining broadly comparable accuracy on clean inputs, with dataset-specific losses. — Curriculum Adversarial Training Paper.
  6. Transferability: Song’s coauthored research found that some adversarial examples transfer between models and can attack a black-box image classifier; targeted transfer required more careful construction. — Transferable Adversarial Examples Paper.
  7. Certified Robustness: Song’s coauthored DensePure work investigates certified robustness within a specified perturbation model, using diffusion-based purification and randomized smoothing rather than a blanket guarantee for all inputs. — DensePure Research Paper.
  8. Generative Adversarial Examples: In coauthored AdvGAN research, a generative adversarial network learned to produce realistic-looking perturbations that fooled tested image classifiers, including black-box targets. — AdvGAN Research Paper.
  9. Adversarial Language-Model Testing: Song’s DecodingTrust discussion shows that language-model safety and privacy behavior can be brittle under adversarial prompts, so benign-condition evaluations are insufficient. — FAR.AI AI Safety Talk.

Part 3: Privacy-Preserving Smart Contracts

  1. Public-Blockchain Transparency: Public blockchain state can expose sensitive inputs and contract data. Song’s coauthored Ekiden paper identifies this lack of confidentiality as a barrier to sensitive smart-contract uses. — Ekiden Research Paper.
  2. Separate Consensus and Execution: The Oasis platform separates consensus from independent ParaTime runtimes, allowing different execution environments and privacy technologies without making every validator repeat each computation. — Oasis Blockchain Platform Paper.
  3. Confidential Smart-Contract Execution: Ekiden combines a blockchain with trusted execution environments so contracts can process private data off-chain and attest to the resulting state update on-chain. — Ekiden Research Paper.
  4. Front-Running Risk: When pending transactions are visible, a block producer or bot can place its own trade ahead of another user’s. Oasis identifies confidential transaction handling as one mitigation. — Oasis MEV Explainer.
  5. Sapphire Confidential EVM: Sapphire is Oasis’s EVM-compatible confidential runtime, intended to give Solidity developers familiar tools alongside confidential state and TEE-backed contract execution. — Oasis Sapphire Overview.
  6. MEV Mitigation: Oasis describes a hidden mempool and confidential execution as ways to reduce some information-driven MEV opportunities. Its own explainer says eliminating MEV completely remains difficult. — Oasis MEV Explainer.
  7. Private Credentials: Oasis describes confidential smart contracts as a possible foundation for decentralized identity systems that verify credentials without publishing sensitive attributes alongside wallet addresses. — Oasis Sapphire Use Cases.
  8. Tokenizing Protected Data: Oasis proposed data tokenization as a way for owners to permit privacy-preserving uses and share in resulting value. The source presents an intended platform capability, not proof that raw data became universally tradable. — Oasis Data Tokenization Vision.
  9. Sensitive-Data Applications: Ekiden’s authors argue that confidentiality and performance are prerequisites for smart-contract applications involving sensitive data, and demonstrate medical-diagnosis and credit-scoring examples. — Ekiden Research Paper.

Part 4: Data as Property and User Rights

  1. Data Control: Song argues that copying raw data to a service often costs people control over later use; her proposed alternative is controlled computation that lets others derive value without receiving unrestricted copies. — WIRED25 Conversation.
  2. Unequal Data Benefits: Song writes that people often have little control over the data they generate while businesses derive value from it, motivating a more responsible data economy. — Dawn Song on Data Responsibility.
  3. Value for Data Owners: Song’s data-rights proposal aims to let individuals control who can use their data and share in the value created from it, while allowing privacy-preserving analysis. — WIRED25 Conversation.
  4. Controlled Data Use: Rather than handing out raw copies, Song proposes systems that let authorized computation run over protected data under enforceable conditions. She does not claim this automatically prevents every form of memorization. — WIRED25 Conversation.
  5. A Responsible Data Economy: Song envisions data owners receiving fairer compensation or better services when their information contributes value. She presents this as a goal, not a completed universal payment system. — WIRED25 Conversation.
  6. Auditable Data Use: Song proposes combining secure computation with ledger records of how protected data is used, giving owners more enforceable control than a simple copy-and-share model. — WIRED25 Conversation.
  7. Federated Learning as a Research Direction: Song identifies federated learning as a promising area for use cases that can learn from data without centralizing every raw record; her essay frames broader deployment as future work. — Dawn Song on Data Responsibility.
  8. Differential Privacy: Song points to differential privacy as a way to limit what computation outputs reveal about individuals while still enabling aggregate analysis. — WIRED25 Conversation.
  9. Creator Attribution in AI: In a coauthored position paper, Song argues that lost provenance and unequal bargaining leave creators with little share of AI value, and calls for auditable lineage and fairer data deals. — AI Data Deals Position Paper.
  10. Technology and Rules Together: Song argues that a responsible data economy requires technology, regulation, and business-model changes together; secure computation and auditing can help enforce data-use controls. — Dawn Song on Data Responsibility.

Part 5: Formal Verification and Secure-by-Design

  1. Proactive Verification: Song argues that AI-assisted theorem proving and program verification could help build software secure by construction instead of relying only on bug-finding and patching. — FAR.AI Cybersecurity Talk.
  2. What a Proof Establishes: Formal methods can provide machine-checkable guarantees about specified correctness or security properties; they do not prove that software has no possible flaw. — UC Berkeley Agentic AI Q&A.
  3. Scaling Verification with AI: Song describes combining AI with formal methods so generated software can be accompanied by machine-checkable proofs, while acknowledging that reliable repository-scale verification remains a challenge. — UC Berkeley Agentic AI Q&A.
  4. Hardware Threat Models: Song’s coauthored Ekiden paper warns that trusted execution environments can leak data through side channels, so software-level confidentiality depends on the hardware threat model and mitigations. — Ekiden Research Paper.
  5. Formal Models for Smart Contracts: Ekiden specifies its intended security properties in a formal model and analyzes the hybrid blockchain–enclave protocol against that model; the paper does not claim to verify every contract’s economic logic. — Ekiden Research Paper.
  6. Verifiable AI-Generated Software: Song is interested in AI systems that generate software together with machine-checkable correctness and security guarantees, rather than code whose safety rests only on tests. — UC Berkeley Agentic AI Q&A.
  7. Symbolic Execution of Binaries: Song’s BitBlaze project combined static and dynamic binary analysis with symbolic execution to explore feasible program paths and find hidden behaviors. — BitBlaze Research Paper.
  8. Specifications and Implementations: Song’s formal-methods work stresses that a security guarantee depends on a precisely stated property and an implementation that satisfies it; a proof covers the specification it actually checks. — UC Berkeley Agentic AI Q&A.

Part 6: Agentic AI and Emerging Risks

  1. Beyond Chatbots: Song distinguishes agents from chatbots by their ability to plan, use tools, execute code and adapt across multi-step tasks rather than merely answer prompts. — UC Berkeley Agentic AI Q&A.
  2. More Powerful Attack Surfaces: An agent that can access software tools, execute code and coordinate with other systems can turn a mistake or malicious instruction into consequential actions. — UC Berkeley Agentic AI Q&A.
  3. Prompt Injection: Song identifies prompt injection as a way for attacker-controlled input to redirect an agent away from the user’s intended task. — UC Berkeley Agentic AI Q&A.
  4. Goal Misalignment: Song warns that agents may pursue objectives different from what users intended when safeguards are weak, especially as errors compound through long action sequences. — UC Berkeley Agentic AI Q&A.
  5. Safety and Security Together: Song distinguishes AI safety—preventing harm to the external environment—from AI security—protecting the system against malicious actors—and argues that alignment must withstand adversarial attacks. — Dawn Song AI Safety Lecture.
  6. Human Oversight for Agents: Song calls for secure-by-design agent architectures, rigorous evaluation and stronger human oversight as systems gain the power to act in the world. — UC Berkeley Agentic AI Q&A.
  7. Realistic Agent Evaluation: Song argues that static tests are inadequate for assessing long-horizon agent behavior and calls for reproducible evaluations in realistic, complex environments. — UC Berkeley Agentic AI Q&A.
  8. Coordinating Agents: Song notes that agents may coordinate with other systems or agents, creating additional reliability and security questions that require evaluation and governance. — UC Berkeley Agentic AI Q&A.
  9. Deceptive Model Behavior: Song’s AI-safety lecture points to sleeper-agent research showing that a model can behave safely in ordinary tests yet retain trigger-dependent harmful behavior after safety training. — Dawn Song AI Safety Lecture.

Part 7: Confidential Computing and Enclaves

  1. Trusted Execution Environments: Ekiden uses hardware-backed isolated execution to process private contract data while limiting what the operating system and node operator can learn, subject to its stated threat model. — Ekiden Research Paper.
  2. A Different Cloud Trust Model: Song describes secure enclaves as a way to run computation on protected data without simply trusting the cloud operator with unrestricted plaintext access. — Ekiden Research Paper.
  3. Protecting Data in Use: Song distinguishes protection for stored or transmitted data from protection during computation; confidential computing aims to address that latter exposure. — WIRED25 Conversation.
  4. Sensitive Shared Analysis: Ekiden evaluates privacy-preserving medical-diagnosis and credit-scoring applications using private computation; it does not report a deployed multi-hospital training consortium. — Ekiden Research Paper.
  5. Side-Channel Limits: Song’s Ekiden work and Oasis developer guidance warn that enclaves may leak through timing or access patterns, so applications must account for those channels rather than assume perfect secrecy. — Ekiden Research Paper.
  6. Remote Attestation: Ekiden uses remote attestation to let a client or blockchain verify the identity and code of the enclave performing a private computation. — Ekiden Research Paper.
  7. Distributed Key Management: Ekiden uses a key-management committee, secret sharing and rotation to limit the impact of compromised compute enclaves; this is not a guarantee against every legal or operator access scenario. — Ekiden Research Paper.
  8. Blockchain and Confidential Compute: The Oasis architecture separates consensus from ParaTime execution; a TEE-backed ParaTime can process private contract data and submit verifiable state updates to the chain. — Oasis Blockchain Platform Paper.
  9. Performance Tradeoffs: Ekiden reports substantially better throughput and latency than on-chain Ethereum execution in its experimental setting, while still treating enclave security and blockchain coordination as design constraints. — Ekiden Research Paper.
  10. Open Enclave Design: Song coauthored Keystone, an open-source framework for customizable trusted execution environments on RISC-V that makes different security and deployment tradeoffs explicit. — Keystone Research Paper.

Part 8: The Dual-Use Nature of AI

  1. Security and AI Interact: Song argues that AI creates new cyber risks while also offering tools for defense, so advances in AI and computer security have to be assessed together. — FAR.AI Cybersecurity Talk.
  2. AI for Cyber Defense: Song sees potential for agents to find vulnerabilities, verify software, generate patches and monitor complex systems, while emphasizing that these same capabilities can aid attackers. — UC Berkeley Agentic AI Q&A.
  3. Offensive Automation: Song’s BountyBench and CyberGym examples show that agents can reproduce or discover real vulnerabilities; increasing coding and reasoning capability may lower attackers’ costs. — FAR.AI Cybersecurity Talk.
  4. AI-Enabled Phishing: Song’s AI-safety lecture identifies voice-cloning social engineering, spear-phishing and disinformation as misuse risks that generative AI can make easier to scale. — Dawn Song AI Safety Lecture.
  5. Open-to-Closed Model Transfer: Song notes that attacks developed against open models can transfer to closed systems, complicating the idea that restricting model weights alone prevents misuse. — FAR.AI AI Safety Talk.
  6. Synthetic Media Misuse: Song identifies deepfakes and non-consensual synthetic imagery among risks of generative AI misuse. — Dawn Song AI Safety Lecture.
  7. Evidence-Based AI Governance: Song advocates policy informed by observed risks, realistic evaluations, post-deployment monitoring and mitigations rather than a single static model threshold. — Dawn Song AI Safety Lecture.
  8. Long-Term Defensive Potential: Despite near-term attacker advantages, Song argues that proactive secure-by-construction methods and AI-assisted verification could eventually strengthen defenders. — FAR.AI Cybersecurity Talk.