Lessons from Sander Schulhoff

Lessons from Sander Schulhoff

AI researcher Sander Schulhoff created Learn Prompting, an open-source prompt engineering platform released shortly before ChatGPT. He also founded the global red-teaming competition HackAPrompt, which gathers adversarial attacks to test the structural security limits of language models.

Part 1: The Prompt Report & Research

1. On the scale of prompt engineering literature: "Analyzing over 1,500 academic papers reveals that prompting is a complex taxonomy of over 200 distinct techniques." — Source: The Prompt Report
2. On standardizing AI communication: "The field of prompt engineering requires formal taxonomies to move past trial-and-error and toward reproducible science." — Source: The Prompt Report
3. On few-shot prompting: "Providing a model with structured examples remains one of the most reliable methods for aligning output format and reasoning." — Source: Learn Prompting
4. On chain-of-thought limitations: "While forcing a model to explain its reasoning improves logic, it also exposes the system to lengthier, more complex injection vectors." — Source: The Prompt Report
5. On ad-hoc prompting: "Relying on informal prompt structures without rigorous evaluation leads to fragile systems that break under slight model updates." — Source: Latent Space Podcast
6. On academic rigor in AI: "Evaluating generative models requires systematic reviews across institutions to prevent biased or overly narrow benchmarks." — Source: The Cognitive Revolution Podcast
7. On deciphering specialized text: "Large language models can quantify dissent and parse highly technical jargon like Fedspeak when given the correct domain-specific context." — Source: GPT Deciphering Fedspeak
8. On multi-institution collaboration: "Mapping the landscape of prompting techniques requires input from multiple AI labs to capture the full scope of model behaviors." — Source: The Prompt Report
9. On the evolution of prompts: "Techniques that worked on earlier iterations of GPT often degrade or change fundamentally on newer architectures." — Source: The Cognitive Revolution Podcast
10. On the foundation of AI interactions: "Prompting is the fundamental interface layer between human intent and machine execution." — Source: Learn Prompting

Part 2: AI Security Fundamentals

1. On the nature of AI vulnerabilities: "You can patch a bug, but you cannot patch a brain." — Source: Lenny's Podcast
2. On the limits of code fixes: "Because language models operate probabilistically, attempting to secure them with deterministic software patches frequently fails." — Source: Lenny's Podcast
3. On the AI security crisis: "The enterprise integration of AI is outpacing our ability to secure the fundamental architecture of these models." — Source: Lenny's Podcast
4. On treating AI securely: "Developers must treat language models as untrusted entities within their system architecture." — Source: Sander Schulhoff Substack
5. On classical cybersecurity: "The most effective defense for an AI system is applying traditional security principles like sandboxing and strict permission limits." — Source: Lenny's Podcast
6. On stateless security: "Security protocols that do not maintain state or context are easily bypassed by multi-turn interactions." — Source: MLSecOps Podcast
7. On structural defense: "Security must be built into the infrastructure surrounding the model, rather than relying on the model to police its own behavior." — Source: Lenny's Podcast
8. On zero-trust architecture: "Every output generated by an external LLM should be validated before it is allowed to interact with internal databases." — Source: Sander Schulhoff Substack
9. On the illusion of control: "Writing a system prompt that tells a model to act securely is not a substitute for actual software security." — Source: Learn Prompting
10. On permission limiting: "An AI agent should only have the absolute minimum system permissions required to execute its specific task." — Source: Lenny's Podcast

Part 3: Prompt Injection & Red Teaming

1. On jailbreaking vs prompt injection: "Jailbreaking breaks the model's internal safety guidelines, while prompt injection subverts the developer's specific application instructions." — Source: Lenny's Podcast
2. On adaptive attacks: "The attacker moves second, meaning they can always adjust their strategy to bypass whatever static defense is put in place." — Source: The Attacker Moves Second
3. On exposing systemic flaws: "Only by collecting adversarial attacks at scale can we begin to understand the true fragility of current language models." — Source: HackAPrompt
4. On taxonomies of hacking: "Prompt hacking represents a diverse ontology of methods ranging from semantic manipulation to token smuggling." — Source: HackAPrompt
5. On benchmarking security: "Fortune 500 companies cannot secure their deployments without datasets that accurately reflect real-world adversarial inputs." — Source: Sander Schulhoff Website
6. On the necessity of red teaming: "If you do not red team your AI application before deployment, your users will do it for you in production." — Source: Learn Prompting
7. On the arms race: "Every time a new defense mechanism is published, a corresponding bypass is developed shortly after." — Source: The Cognitive Revolution Podcast
8. On crowdsourcing attacks: "Opening red teaming to a global audience uncovers edge cases that closed research teams inevitably miss." — Source: HackAPrompt
9. On inherent vulnerabilities: "The ability to follow instructions makes an LLM useful, but it also makes it inherently susceptible to malicious instructions." — Source: MLSecOps Podcast
10. On attack persistence: "Adversarial inputs often transfer across different model architectures, proving that vulnerabilities are systemic rather than model-specific." — Source: The Attacker Moves Second

Part 4: The Failure of Guardrails

1. On ineffective guardrails: "Many commercial AI guardrails offer a false sense of security because they rely on the same probabilistic mechanisms as the models they protect." — Source: Sander Schulhoff Substack
2. On alignment filters: "Filters designed to keep a model polite do very little to prevent it from executing a harmful system command." — Source: The Attacker Moves Second
3. On prompt-based defenses: "Appending explicit warnings to a prompt is a fragile defense against a dedicated attacker." — Source: Lenny's Podcast
4. On system prompts: "The system prompt is just another input token; it holds no special architectural authority over the user's input." — Source: Learn Prompting
5. On bypassing filters: "Attackers can bypass semantic filters by encoding their malicious instructions or breaking them across multiple turns." — Source: HackAPrompt
6. On RLHF limitations: "Reinforcement Learning from Human Feedback aligns the model with general human preferences, but it does not patch structural security holes." — Source: MLSecOps Podcast
7. On architectural containment: "True security requires placing the model in a confined environment where its outputs cannot directly trigger execution without secondary validation." — Source: Lenny's Podcast
8. On superficial safety: "Relying entirely on safety classifiers creates a single point of failure that is relatively trivial to evade." — Source: Sander Schulhoff Substack
9. On defense in depth: "Because AI models will eventually be tricked, systems must be designed assuming that the guardrails have already failed." — Source: Lenny's Podcast

Part 5: Prompt Engineering Best Practices

1. On the definition of prompt engineering: "Prompt engineering is the systematic process of designing and refining inputs to maximize the performance of generative models." — Source: Learn Prompting
2. On modern methodology: "As models grow more sophisticated, brute-force prompting is being replaced by structured data formatting and dynamic context retrieval." — Source: Lenny's Podcast
3. On effective communication: "Interacting with an AI requires precision, clarity, and an understanding of how the model parses syntax." — Source: What's AI Podcast
4. On artificial social intelligence: "We must learn to communicate with models using a form of artificial social intelligence, recognizing their unique mechanical behaviors." — Source: What's AI Podcast
5. On skill evolution: "The technical requirements for prompting change rapidly, but the underlying skill of clear, logical instruction remains constant." — Source: Learn Prompting
6. On prompt templates: "Using standardized, version-controlled templates prevents drift and ensures consistent outputs in production applications." — Source: The Prompt Report
7. On complex instructions: "Breaking a complex task into multiple sequential prompts yields far better results than cramming everything into a single query." — Source: The Cognitive Revolution Podcast
8. On iterative refinement: "The first prompt is rarely the best; successful engineering requires continuous testing and refinement against a baseline." — Source: Learn Prompting
9. On context management: "Filling a massive context window with irrelevant information degrades the model's ability to focus on the actual task." — Source: Lenny's Podcast

Part 6: Agentic AI & Advanced Capabilities

1. On the shift to agents: "The industry is moving from conversational bots to agentic systems that can plan, execute tools, and operate autonomously." — Source: Lenny's Podcast
2. On securing AI agents: "An autonomous agent reading an untrusted webpage is highly susceptible to indirect prompt injections hidden in the HTML." — Source: Lenny's Podcast
3. On benchmarking fuzzy tasks: "Evaluating agents on open-ended tasks requires specialized datasets that account for multiple valid paths to success." — Source: BEDD
4. On autonomous execution: "Giving an AI agent read and write access to your database fundamentally changes your threat model." — Source: Lenny's Podcast
5. On API vulnerabilities: "When agents interact with external APIs, those interfaces become attack vectors if the model's output is not strictly validated." — Source: MLSecOps Podcast
6. On reproducibility: "Standardized environments are crucial for reinforcement learning to ensure that agent performance can be independently verified." — Source: Gymnasium
7. On open-ended evaluation: "Agents must be tested on their ability to navigate ambiguous, real-world scenarios rather than narrow, structured metrics." — Source: BEDD
8. On agent-based workflows: "Building an agentic workflow requires designing a system where multiple specialized models verify and correct each other's work." — Source: Lenny's Podcast
9. On NLP and Reinforcement Learning: "The intersection of natural language processing and reinforcement learning is what enables models to take meaningful actions." — Source: Sander Schulhoff Website

Part 7: HackAPrompt & Crowdsourced Security

1. On gamifying safety: "Turning AI red teaming into a global competition incentivizes the discovery of vulnerabilities before they are exploited in the wild." — Source: HackAPrompt
2. On the first injection competition: "Organizing the largest prompt hacking competition provided unprecedented data on how humans intuitively try to break AI." — Source: HackAPrompt
3. On discovering novel jailbreaks: "Participants often invent techniques that academic researchers miss simply by approaching the problem with different assumptions." — Source: HackAPrompt
4. On scale in data collection: "Securing enterprise models requires datasets containing tens of thousands of varied adversarial prompts." — Source: HackAPrompt
5. On analyzing attacker strategies: "By studying crowdsourced attacks, we can categorize the specific linguistic patterns that consistently bypass safety filters." — Source: HackAPrompt
6. On red teaming at scale: "A small, dedicated team cannot simulate the sheer volume and creativity of a global hacking community." — Source: HackAPrompt
7. On diverse hacking techniques: "Effective prompt hacking relies heavily on psychological manipulation, structural formatting, and exploiting tokenizer weaknesses." — Source: HackAPrompt
8. On building robust datasets: "Open-sourcing the results of red teaming competitions allows the entire industry to train more resilient models." — Source: Sander Schulhoff Website
9. On community-driven defense: "The future of AI safety relies on a collaborative, open community actively testing and reporting model failures." — Source: HackAPrompt

Part 8: Education & The Future of AI

1. On democratizing AI literacy: "Open-source educational platforms ensure that the tools to understand and direct AI remain accessible rather than restricted to private labs." — Source: Learn Prompting
2. On preempting the wave: "Establishing foundational best practices before a technology hits mass adoption prevents the spread of misinformation." — Source: Learn Prompting
3. On massive scale education: "Reaching millions of learners proves that there is a massive global demand for practical, rigorous AI education." — Source: Learn Prompting
4. On open-source materials: "Keeping foundational knowledge free and accessible accelerates the entire field of generative AI." — Source: Learn Prompting
5. On bridging the knowledge gap: "The gap between what AI researchers know and what the public understands is dangerous; education is the bridge." — Source: The Cognitive Revolution Podcast
6. On training engineers: "The next generation of software engineers must view language models as complex systems requiring careful operation instead of infallible black boxes." — Source: Latent Space Podcast
7. On the business of teaching: "The challenge of teaching AI is that the curriculum must be rewritten every few months to keep pace with new capabilities." — Source: The Cognitive Revolution Podcast
8. On adapting to updates: "A successful AI curriculum teaches the underlying principles of reasoning rather than memorizing syntax for a specific model version." — Source: Lenny's Podcast
9. On human oversight: "No matter how advanced models become, the ability of a human to critically evaluate and direct the output will remain essential." — Source: Learn Prompting