Why this open letter matters now

In mid-2024, a group of current and former workers from top AI labs, including OpenAI and Google DeepMind, released an open letter urging companies and policymakers to establish a “right to warn” the public about risks posed by frontier AI models. They argue that the people building these systems often see problems first but are constrained by non-disparagement clauses, confidentiality agreements, and fear of retaliation. The signatories say companies should commit to non-retaliation, enable anonymous reporting, and support independent oversight of the most powerful models (RightToWarn.ai).

The letter arrives amid turbulence inside leading labs. OpenAI’s safety leadership saw high-profile departures in May 2024, with former safety co-lead Jan Leike saying the company struggled to prioritize safety over product launches. OpenAI’s CEO Sam Altman responded that the company remains committed to safety and has since moved to remove controversial non-disparagement terms for departing employees (The Verge). Media coverage of the open letter underscored growing internal unease across the sector, not just at a single lab (Financial Times), (Washington Post).

What the letter asks for

The authors propose practical steps labs can adopt today to strengthen accountability and allow employees to raise concerns without jeopardizing their careers. Key requests include:

• Stop using non-disparagement or sweeping confidentiality agreements that prevent workers from discussing safety risks in the public interest.
• Commit to non-retaliation against employees who raise risk concerns in good faith, including via external channels when internal options fail.
• Provide clear, protected avenues for anonymous reporting to independent, trusted third parties.
• Support independent oversight for frontier models, including evaluations and red-teaming by outside experts.
• Be candid with the public and regulators about model capabilities and limitations, including risks of misuse.

In short, the signatories argue that a basic right to warn is essential to surface problems early and reduce the chance of catastrophic failures as models become more capable (RightToWarn.ai).

What risks are they worried about?

The letter echoes concerns many AI researchers have raised over the past year: advanced models are rapidly improving in reasoning, coding, persuasion, and autonomy, while guardrails and governance lag behind. Potential risks include:

• Scale and autonomy: More agentic systems could take actions that are hard to predict or reverse without strong oversight.
• Cybersecurity: Models can help discover exploits or automate phishing and social engineering at scale.
• Biosecurity and chemical risks: Model-assisted discovery could lower barriers for misuse if access is not properly controlled.
• Disinformation and manipulation: Generative tools can flood information ecosystems, eroding trust and amplifying manipulation.
• Concentration of power: A few companies controlling frontier models could shape economies and information flows with limited accountability.

These concerns align with broader warnings from the research community that serious AI risks, including low-probability but high-impact events, should be treated as a global priority alongside other societal-scale risks (Center for AI Safety).

Context: a fast-moving policy landscape

The open letter arrives as governments race to catch up:

• United States: The White House issued a sweeping AI Executive Order in Oct. 2023 requiring safety test reporting for the most capable models, encouraging red-teaming, and directing agencies to develop standards for AI safety, security, and trust (White House). The U.S. also launched a dedicated AI Safety Institute at NIST to advance testing and evaluations.
• European Union: The EU AI Act, formally adopted in 2024, introduces a risk-based framework and special obligations for general-purpose and high-risk AI systems, with phased enforcement starting in 2025 (European Parliament).
• United Kingdom and partners: The Bletchley Declaration, signed by 28 countries in Nov. 2023, recognized the need for international cooperation on frontier AI risks and launched government AI Safety Institutes in the UK and US to evaluate cutting-edge models (UK Government).

Even with these moves, accountability inside labs remains critical. The letter’s focus on whistleblower protections and transparency complements, rather than replaces, formal regulation.

Inside the labs: culture, NDAs, and safety teams

The signatories argue that lab culture can unintentionally suppress vital safety signals. In early 2024, reporting highlighted restrictive non-disparagement agreements at some AI companies. After criticism, OpenAI said it would remove such terms and clarified that departing employees would not lose their vested equity for refusing to sign them (The Verge).

The letter also follows changes to safety leadership at OpenAI. Jan Leike, who co-led OpenAI’s superalignment efforts, resigned in May 2024, writing that resources and attention were too focused on products over safety. OpenAI responded that it is increasing safety investment, publishing external safety reports, and engaging more with outside researchers. Independent reporting has corroborated the staff changes and the ensuing debate about how labs prioritize safety work (Financial Times).

What a practical “right to warn” could look like

Many sectors with complex risks – from aviation to biomedicine – rely on strong safety cultures and protected reporting channels. For frontier AI, a pragmatic version of the right to warn might include:

• Clear non-retaliation policies that explicitly cover risk disclosures to regulators, independent auditors, or trusted third parties.
• Anonymous, third-party-managed hotlines with legal protections and escalation paths to oversight bodies.
• Regular, independent evaluations of frontier models for dangerous capabilities and misuse potential, with public summaries.
• Red-teaming by external experts with relevant domain knowledge, plus incentives for coordinated disclosure of vulnerabilities.
• Transparent release policies that use capability thresholds and staged access for potentially hazardous features.
• Board-level safety committees with access to internal metrics, incident reports, and authority to pause risky deployments.

These steps align with guidance from public bodies such as NIST’s AI risk management work and the aims of the U.S. and UK AI Safety Institutes, which focus on standardized evaluations and safety reporting for high-capability models (NIST AI RMF), (U.S. AI Safety Institute), (UK AI Safety Institute).

How companies are responding

OpenAI has said it provides anonymous integrity hotlines, welcomes outside safety research, and does not retaliate against good-faith concerns. The company also participated in voluntary commitments at the White House in 2023 and says it supports more standardized safety testing for frontier models (White House). Google and Anthropic have made similar public commitments around evaluations and red-teaming. The open letter challenges labs to codify these promises into enforceable employee protections and independent oversight.

What to watch next

• Standardized evaluations: Expect more model testing protocols from the U.S. and UK AI Safety Institutes. Clearer thresholds for restricted release could follow.
• Legal protections: Lawmakers may explore targeted whistleblower protections for AI risk disclosures, similar to regimes in finance or healthcare.
• Corporate governance: Boards face pressure to formalize safety oversight and incident response plans for AI deployment.
• Global coordination: The EU AI Act, U.S. Executive Order, and Bletchley Declaration point toward closer international alignment on frontier model risks.

Bottom line

The open letter from AI insiders is not a call to halt progress. It is a call to build safety culture as a first-class priority – including the right to raise alarms when internal processes fall short. As models accelerate toward more general capabilities, investing in practical transparency, strong reporting channels, and independent evaluations is a low-cost way to reduce high-stakes risks.

FAQs

Who signed the open letter?

The letter was authored by a group of current and former workers from leading labs such as OpenAI and Google DeepMind, with some signatories choosing anonymity. Major outlets independently reported on the signatories and their affiliations (Financial Times), (Washington Post).

Is this about pausing AI development?

No. The letter focuses on employee protections, transparency, and oversight for frontier AI models. It does not call for a blanket pause, but for safer processes and the ability to warn when risks are not being addressed (RightToWarn.ai).

What specific risks do frontier models pose?

Risks include misuse in cybersecurity and biosecurity, large-scale disinformation, and potential loss of control over highly capable systems. Researchers have urged governments and labs to treat these as serious, systemic risks (Center for AI Safety).

How does this fit with new AI regulations?

Regulations like the EU AI Act and the U.S. AI Executive Order set broad rules and testing expectations. The right to warn complements these by ensuring experts inside labs can speak up when they see risks being overlooked (EU AI Act), (U.S. Executive Order).

What can companies do now?

Adopt non-retaliation policies, remove overbroad non-disparagement terms, set up independent hotlines, and publish summaries of third-party model evaluations. These are practical, near-term steps aligned with public guidance from NIST and national AI Safety Institutes (NIST AI RMF).

Sources

1. RightToWarn.ai – Open letter by current and former AI lab employees
2. Financial Times – AI insiders at OpenAI and DeepMind warn of risks in open letter
3. Washington Post – AI lab workers call for a right to warn about AI risks
4. The Verge – OpenAI drops non-disparagement terms for departing employees
5. White House – U.S. Executive Order on AI safety and security
6. European Parliament – EU AI Act adopted
7. UK Government – The Bletchley Declaration
8. NIST – AI Risk Management Framework