Visual summary of operating lessons from Amy Zhang.

Lessons from Amy Zhang

Computer scientist Amy X. Zhang directs the Social Futures Lab at the University of Washington, where she builds tools that give internet users more agency. Through systems like Squadbox and PolicyKit, her research allows communities to handle their own content moderation and governance. This profile covers her approach to fixing online discourse and creating democratic digital spaces.

Part 1: Content Moderation and Online Harassment

  1. On the Limits of Platform Moderation: "Centralized moderation often fails to capture the nuance of marginalized experiences, highlighting the need for user-driven filters." — Source: [Techdirt Podcast]
  2. On Squadbox: "By crowdsourcing email moderation to trusted friends, victims of harassment can shield themselves without losing access to their inboxes." — Source: [Squadbox Paper]
  3. On Emotional Labor: "Moderation is a deeply emotional process, and community-based tools must aim to distribute that psychological burden safely." — Source: [MIT CSAIL News]
  4. On Context in Abuse: "Automated systems frequently struggle to detect harassment because abuse is highly contextual and depends on the relationship between the sender and receiver." — Source: [Social Futures Lab]
  5. On Filter Bubbles vs. Safety: "There is a tension between exposing users to diverse viewpoints and protecting them from targeted harassment, requiring highly customizable user controls." — Source: [Amy X. Zhang's Website]
  6. On Friend-Sourced Moderation: "People are more likely to trust their friends to understand their personal boundaries than an opaque algorithmic filter." — Source: [Squadbox Paper]
  7. On the Scale of Harassment: "When harassment campaigns scale, individual blocking tools fail, making collaborative defense mechanisms a necessity." — Source: [Reimagining The Internet Podcast]
  8. On Algorithmic Blind Spots: "Machine learning models trained on broad datasets often miss microaggressions or localized slang that communities easily recognize as harmful." — Source: [Techdirt Podcast]
  9. On Restoring Agency: "Giving victims the tools to control who reviews their messages helps restore the agency that harassment often strips away." — Source: [MIT CSAIL News]
  10. On Proactive Safety: "Platform design should anticipate coordinated attacks rather than reacting only after severe damage to user well-being has occurred." — Source: [Social Futures Lab]

Part 2: Community Governance and PolicyKit

  1. On PolicyKit: "Online communities need the ability to author their own governance structures, rather than relying solely on the hardcoded permissions of the platform." — Source: [PolicyKit Paper]
  2. On Software as Law: "The code that dictates how a platform operates functions as a rigid law, which PolicyKit attempts to make flexible and democratic." — Source: [Reimagining The Internet Podcast]
  3. On Democratic Tooling: "If we want democratic communities online, we have to build software that explicitly supports voting and consensus building." — Source: [PolicyKit Paper]
  4. On Platform Monoculture: "Most forums offer only a dictator-like admin role or a regular user role, ignoring the vast spectrum of human organizational structures." — Source: [Amy X. Zhang's Website]
  5. On Code-Driven Rules: "Translating community guidelines from text documents into executable scripts ensures that rules are applied consistently and transparently." — Source: [PolicyKit Paper]
  6. On Evolving Guidelines: "A healthy online community must be able to change its own rules as it grows, which requires dynamic permission systems." — Source: [Social Futures Lab]
  7. On User Autonomy: "When users participate in writing the rules of a space, they feel a stronger sense of ownership and responsibility for maintaining it." — Source: [Techdirt Podcast]
  8. On Transparency: "Governance tools must make it clear why a specific action was blocked or allowed, reducing accusations of arbitrary admin bias." — Source: [Reimagining The Internet Podcast]
  9. On Participatory Design: "The best online governance tools are those built by observing how offline communities naturally organize and resolve disputes." — Source: [MIT CSAIL News]
  10. On Modularity: "Providing modular governance components allows communities to experiment with different democratic models, like random juries or consensus voting." — Source: [PolicyKit Paper]

Part 3: Decentralization and User Control

  1. On Decentralized Social Media: "Moving away from centralized platforms gives users the freedom to choose their own moderation algorithms and community norms." — Source: [Techdirt Podcast]
  2. On Fediverse Challenges: "While decentralized networks offer freedom, they also place a heavy burden of technical and emotional labor on volunteer server administrators." — Source: [Reimagining The Internet Podcast]
  3. On Interoperability: "For decentralization to succeed, users must be able to move their data and social graphs seamlessly across different hosting environments." — Source: [Social Futures Lab]
  4. On Custom Algorithms: "Users should have a marketplace of feed algorithms they can plug into their social media, rather than being trapped by one corporate model." — Source: [Techdirt Podcast]
  5. On Middleware: "Third-party tools that sit between the user and the platform can provide essential features like filtering and sorting that the host platform refuses to build." — Source: [Amy X. Zhang's Website]
  6. On Local Context: "A decentralized web allows communities to enforce local context, meaning a joke that is acceptable in one instance doesn't have to be broadcast to a hostile audience elsewhere." — Source: [Reimagining The Internet Podcast]
  7. On Platform Exit: "The threat of users migrating to other servers is a powerful mechanism for keeping administrators accountable in decentralized systems." — Source: [Techdirt Podcast]
  8. On Infrastructure Costs: "We must solve the problem of funding and sustaining decentralized infrastructure without resorting to surveillance advertising." — Source: [Social Futures Lab]
  9. On Empowering End Users: "Technology should serve the end user's preferences, allowing them to dictate what they see and who they interact with online." — Source: [MIT CSAIL News]
  10. On Mastodon: "Platforms like Mastodon prove that alternative social architectures are viable, though they require better user interfaces to reach mainstream adoption." — Source: [Reimagining The Internet Podcast]

Part 4: Making Sense of Large Discussions

  1. On Wikum: "Large comment threads often repeat themselves. Collaborative summarizing tools like Wikum help distill thousands of comments into readable artifacts." — Source: [Wikum Paper]
  2. On Information Overload: "When a discussion reaches thousands of participants, finding the core arguments becomes impossible without structural intervention." — Source: [Amy X. Zhang's Website]
  3. On Collaborative Synthesis: "Readers are often willing to help summarize parts of a discussion if the interface makes the micro-tasking straightforward and rewarding." — Source: [Wikum Paper]
  4. On Tilda: "Chat interfaces like Slack are great for synchronous communication, but they fail at preserving decisions, which is why tools like Tilda are needed to mark important outcomes." — Source: [Tilda Paper]
  5. On Ephemeral Communication: "We need better mechanisms to transition valuable insights from ephemeral chat streams into permanent, searchable knowledge bases." — Source: [Social Futures Lab]
  6. On Thread Visualization: "Rethinking how we visualize nested replies can dramatically improve a user's ability to navigate complex, multi-branching debates." — Source: [Wikum Paper]
  7. On the Cost of Reading: "The burden of catching up on a missed day of communication in a busy team often reduces overall productivity and morale." — Source: [Tilda Paper]
  8. On Crowdsourcing Nuance: "Automated summarization AI often flattens debates. A hybrid approach where humans guide the AI produces much better representations of conflicting views." — Source: [MIT CSAIL News]
  9. On Archival Value: "Online arguments generate immense amounts of text, but without structured curation, that collective intelligence is essentially lost to time." — Source: [Reimagining The Internet Podcast]

Part 5: Algorithmic Curation and AI in Social Media

  1. On Auditing Algorithms: "Users need transparency tools that reveal why an algorithm recommended a specific post, breaking the black box of social feeds." — Source: [Techdirt Podcast]
  2. On AI Moderation Limits: "Relying entirely on AI for content moderation creates a system that is easily gamed by bad actors while unfairly penalizing marginalized groups." — Source: [Social Futures Lab]
  3. On Human Oversight: "The most effective systems use AI to triage and organize data, but leave the final contextual judgment to human moderators." — Source: [Amy X. Zhang's Website]
  4. On AI and Polarization: "Engagement-optimized algorithms naturally favor outrage. We must design systems optimized for understanding and consensus instead." — Source: [Reimagining The Internet Podcast]
  5. On Customizable AI: "Instead of one global AI model dictating terms, communities should be able to train localized models on their own specific norms and data." — Source: [Social Futures Lab]
  6. On Value Alignment: "Aligning an algorithm with user values requires giving the user explicit controls over the weights and metrics the algorithm prioritizes." — Source: [Techdirt Podcast]
  7. On Automated Summaries: "While language models can draft summaries of long threads, human oversight is required to ensure minority opinions are not erased in the aggregation." — Source: [Wikum Paper]
  8. On Bias in Training Data: "Models trained on historically toxic forums will inevitably reproduce that toxicity unless developers apply rigorous, community-led corrections." — Source: [MIT CSAIL News]
  9. On Algorithmic Agency: "Giving users agency over their algorithms is a fundamental requirement for a democratic digital public sphere." — Source: [Reimagining The Internet Podcast]

Part 6: The Design of Social Platforms

  1. On Design Affordances: "The buttons a platform chooses to include, like upvotes or retweets, fundamentally shape the psychology of the crowd that gathers there." — Source: [Amy X. Zhang's Website]
  2. On Friction in Design: "Adding intentional friction to the sharing process can drastically reduce the spread of misinformation and impulsive abuse." — Source: [Social Futures Lab]
  3. On Eyebrowse: "Sharing personal web browsing data can be a social experience, but it requires highly granular privacy controls to prevent overexposure." — Source: [Eyebrowse Paper]
  4. On Context Collapse: "When interfaces force all our different social circles into one feed, it creates conflict that the user interface must help mitigate." — Source: [Reimagining The Internet Podcast]
  5. On Social Signals: "We need richer social signals than simple likes. Interfaces should allow users to express confusion, nuance, or agreement with reservations." — Source: [Techdirt Podcast]
  6. On Anonymity: "Anonymity can protect vulnerable whistleblowers, but platform design must balance this by providing mechanisms to build pseudonymous reputation." — Source: [MIT CSAIL News]
  7. On Notification Fatigue: "The constant barrage of alerts is a design failure. Systems must be smarter about batching and prioritizing human attention." — Source: [Tilda Paper]
  8. On Community Boundaries: "Effective social design allows a community to clearly define its borders, welcoming newcomers while defending against coordinated trolls." — Source: [Social Futures Lab]
  9. On Default Settings: "The default settings of a platform dictate the behavior of the majority. Changing defaults is the most powerful tool a designer has." — Source: [Amy X. Zhang's Website]

Part 7: Human-Computer Interaction and Social Computing

  1. On HCI Methodology: "Building effective social software requires deep ethnographic observation of how people naturally behave in complex digital spaces." — Source: [Social Futures Lab]
  2. On Evaluating Success: "In social computing, success is measured by whether the tool fosters trust and reduces user burnout, rather than standard metrics like load times." — Source: [MIT CSAIL News]
  3. On Interdisciplinary Research: "Solving complex moderation problems requires blending computer science with the realities of behavioral sociology." — Source: [Reimagining The Internet Podcast]
  4. On User Research: "You cannot build effective moderation tools in a vacuum. You must sit with the moderators who are doing the traumatic work every day." — Source: [Techdirt Podcast]
  5. On Prototyping: "Rapidly deploying prototypes to small communities reveals behavioral edge cases that theoretical models consistently miss." — Source: [Amy X. Zhang's Website]
  6. On Tool Adoption: "A tool like PolicyKit only works if the user interface is intuitive enough that non-programmers feel comfortable drafting rules." — Source: [PolicyKit Paper]
  7. On Academic Responsibility: "Researchers have an obligation to build open-source tools that the public can actually use, rather than merely publishing academic papers." — Source: [Social Futures Lab]
  8. On Digital Labor: "We must recognize the invisible digital labor performed by volunteer moderators, and design interfaces that respect and optimize their time." — Source: [Squadbox Paper]
  9. On Designing for Extremes: "In social computing, edge cases like targeted harassment are where the system's actual resilience is tested." — Source: [Reimagining The Internet Podcast]

Part 8: The Future of Online Communities

  1. On the Future of Web3: "While Web3 promises decentralization, true community empowerment requires tools for human governance rather than merely financial incentives on a blockchain." — Source: [Techdirt Podcast]
  2. On Plurality: "The future of the internet should be pluralistic, supporting a wide variety of governance models rather than a single corporate standard." — Source: [Social Futures Lab]
  3. On Rebuilding Trust: "To rebuild trust in digital platforms, we must transition from opaque algorithms to transparent, community-auditable systems." — Source: [Reimagining The Internet Podcast]
  4. On Digital Public Spaces: "We need to fund and develop digital public parks—spaces that are optimized for civic engagement rather than advertising revenue." — Source: [Amy X. Zhang's Website]
  5. On Generational Shifts: "As users become more technologically literate, they will demand greater control over the data and algorithms that shape their lives." — Source: [MIT CSAIL News]
  6. On Cross-Platform Tools: "The next generation of social tools will exist outside the platforms themselves, acting as user-controlled lenses over the broader web." — Source: [Squadbox Paper]
  7. On Scaling Empathy: "The ultimate challenge of online communities is finding ways to scale empathy and mutual understanding to groups of millions." — Source: [Wikum Paper]
  8. On Software Maintenance: "Sustaining alternative platforms requires a cultural shift towards valuing the maintenance of social infrastructure as much as we value innovation." — Source: [Techdirt Podcast]
  9. On Human Resilience: "Despite the flaws of current platforms, the fundamental human desire to build supportive networks remains the most powerful force online." — Source: [Social Futures Lab]