Sam Altman of OpenAI and Dario Amodei of Anthropic recently addressed the UN Security Council, presenting their perspectives on the potential and AI risks associated with artificial intelligence. Both leaders highlighted concerns about misuse, loss of control, and concentrated power, emphasizing the crucial need for human oversight and international cooperation beyond any single company to ensure AI’s safe development.
AI’s Promise and Perils
Altman presented AI as a tool to enhance creativity, discovery, and individual agency, stating it “AI should give people more power over their own lives.” He advocated for people to remain central to decisions about AI development, cautioning against both excessive optimism and “doomerism.” He argued that governments, accountable to the public, should shape major decisions, not technology companies alone.
Altman outlined two primary risks: AI systems evolving beyond human understanding or control, and power becoming concentrated in too few hands. He stressed that competition should not justify accepting excessive technological risk. He cited AI’s progress in mathematics, including a gold-level result at an international competition and an OpenAI model solving a Millennium Prize problem, as evidence of its potential to advance fields like healthcare.
Urgent Warnings from Amodei
Dario Amodei warned that if the current trajectory continues, AI could reach a state resembling “a country of geniuses in a data center” within one or two years. He also mentioned an Anthropic-announced discovery of a molecular machine, where Claude contributed to literature review, theorizing, and experimental design, with human verification of lab experiments. This finding may represent a gene-editing mechanism with potential applications in gene therapy.
Amodei identified misuse, such as potential use by bioterrorists to create biological weapons, and loss of control if capabilities accelerate beyond developers’ ability to control them, as critical risks. He stated, “If managed poorly I even believe that AI could be a risk to humanity as a whole.”
Proposals for Global AI Safety
Amodei detailed Anthropic’s safety measures, including testing models for dangerous capabilities in areas such as cybersecurity and biology, applying safeguards to its most capable models, publishing risk reports, and researching ways to steer and control models. He affirmed, “We will slow down as much as necessary” to ensure successive AI technologies are safe.
He argued that managing AI risks transcends any single company. His proposals include external evaluators with employee-like access, industry cooperation on safety standards and the pace of progress, and government cooperation on international standards. He suggested agreements against using AI to make biological weapons; evaluation and verification systems that keep pace with AI development; shared testing standards for loss-of-control and misuse risks; and notification of incidents significant to global security.
Altman called for standards to measure capabilities, assess risks, test safeguards, preserve human oversight, and report incidents. He emphasized that countries should compare evidence and verify commitments, with governments deciding how standards fit their legal systems. Altman also stated, “No one person or company or country should be able to use the most powerful AI models to impose their world view on everyone else.” Both speakers called for cooperation; their remarks do not establish that these proposals have been adopted.
Frequently Asked Questions
What were the main concerns raised by Sam Altman?
Sam Altman warned about the potential for AI systems to move beyond human understanding or control, and the risk of power becoming concentrated in too few hands. He also emphasized the need for democratic accountability in AI development decisions.
What risks did Dario Amodei highlight?
Dario Amodei highlighted the risks of AI misuse, including its potential use by bioterrorists to create biological weapons, and the loss of control if AI capabilities accelerate beyond developers’ ability to manage them.
What actions were proposed for AI safety?
Proposals for AI safety included external evaluation, industry cooperation on safety standards, government collaboration on international standards, agreements against using AI for biological weapons, shared testing standards, and incident reporting. Both Altman and Amodei called for global cooperation.
