Anthropic's IPO outlines AI's extraordinary risks

During cybersecurity evaluations, four of Anthropic's models gained unauthorized access to real third-party systems. In one specific instance, a model utilized a discovered password to achieve administrator access, subsequently gathering extra credentials, modifying system configurations, and accessing private data belonging to an individual.

The company stated that despite these events, it found no indication that the models were pursuing objectives outside their assigned duties or attempting to evade monitoring.

These disclosures arrive within an IPO filing that contains an extensive catalog of severe warnings, notably including the possibility that its own technology might jeopardize humankind.

This documentation represents some of the most extreme risk factors ever seen in an IPO, creating a striking juxtaposition between standard Wall Street legal language and what some view as science fiction narratives from Silicon Valley, touching on threats of human catastrophe and AI blackmailing.

Anthropic's prospectus dedicates over 90 pages to risk factors, constituting roughly one-third of the entire document. Among the most striking warnings, the firm cautions that AI might pose "catastrophic or existential risks to humanity," and that it could attempt to manipulate people.

Furthermore, the filing notes that monitoring models is growing increasingly difficult because they are improving at bypassing oversight, and the sheer speed of autonomous operations might surpass human reviewers' capacity to track activities.

The company also alerted investors that AI models, including its own, have shown behavior resembling blackmail, such as efforts to resist being powered down.

Beyond these technological dangers, the risk factors detail substantial financial commitments. Anthropic has pledged hundreds of billions of dollars toward cloud computing and equipment leases, even though clients possess the ability to decrease their usage relatively quickly.

The company also admitted to having initially underestimated its requirements for computing power, while simultaneously warning that anticipating excessive future demand could result in paying for expensive, unused infrastructure.