Anthropic Built Its Brand on AI Safety as Researchers Warn of Extinction

Anthropic built its name on saying the frightening part out loud.

Pretraining researcher Jacob Coxon resigned from the company on September 8 and posted a seven-part thread on X the next day to explain why.

"Neither company is acting responsibly," he wrote of Anthropic and OpenAI, where he spent three years on pretraining research.

"They are racing straight to self-improving superintelligence and gambling with our lives."

He added that "the people building AI earnestly" believe it could kill everyone by the end of the decade.

The X thread has already passed 164 million views, with 775,000 likes and 197,000 comments.

Coxon also told Axios that he forfeited his unvested Anthropic equity by leaving about two months before his one-year cliff.

Then came the part that made this more than a resignation.

Evan Hubinger, who leads alignment science at Anthropic, put his own estimate of AI causing human extinction within the next decade above 10%.

"I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to," he wrote.

Anthropic Scalable Oversight Lead Samuel Marks added that the more senior the employee, the more concerned they tend to be about the AI endgame.

"Why do AI developers continue despite the risk?

Due to a mixture of commercial incentives and a belief that they are in a race with other, less responsible AI developers that will abuse the technology or develop it less safely," Marks explained.

Both men still work at Anthropic.

Anthropic's brand positioning has rested on candor about catastrophic risk since its founding.

And this week, the people it pays to measure this risk put a number on it in public.

The Safety Position That Cost $200 Million

Anthropic was founded in 2021 by former OpenAI researchers, including CEO Dario Amodei.

Safety has been its differentiator ever since, and the company has paid for it.

In February, the Pentagon gave Anthropic a deadline to grant access to Claude for "all lawful purposes" or lose its $200 million contract.

Anthropic refused, saying the language could cover fully autonomous weapons and mass surveillance of Americans.

"We cannot in good conscience accede to their request," Amodei said at the time.

The Pentagon labeled the company a supply-chain risk, and President Donald Trump ordered federal agencies to stop using its technology.

More than 100 enterprise customers raised concerns, and Anthropic estimated the measures could cost it billions in 2026 revenue.

A federal judge struck the designation down on August 27, calling it illegal and baseless, though the government is expected to appeal.

Customers rewarded Anthropic's stand.

Claude hit No. 1 on the U.S. Apple App Store, free active users rose 60%, and daily signups quadrupled.

So the safety position has a real price tag, and Anthropic has been willing to pay it.

Coxon told CNN anchor Anderson Cooper in an interview that AI executives asking Congress for regulation mean it.

"They find themselves in a scenario where they're compelled to race towards building a deadly technology.

And they would love for some sort of international body to allow them to approach it at a reasonable pace," Coxon shared.

"They just don't trust that the people around them are going to get there in a safe way, so they feel like they have to race there, too."

Coxon's point is that the same understanding of the stakes is what keeps everyone moving.

An Anthropic spokesperson also told CNN the company has always been transparent about both the benefits and the unprecedented risks of AI.

The spokesperson pointed to its responsible scaling policy, the first public framework from an AI lab for mitigating catastrophic risk.

Anthropic's safety researchers are the proof behind its AI model trustworthiness claims.

And this credibility doesn't switch off when the same people start describing what worries them.

Two days after Coxon's resignation, Anthropic answered with more disclosure on how it counters the misuse of AI. 

The report details how hackers and spyware vendors abused Claude, with no mention of the resignation or the extinction estimate from current employees.

It simply didn't address the elephant in the room. 

The $965 Billion Stake Behind the Warning

Anthropic raised $65 billion on May 28 at a $965 billion post-money valuation.

Altimeter Capital, Dragoneer, Greenoaks, and Sequoia Capital led the round, which put Anthropic past OpenAI's $852 billion for the first time.

In February, the company was worth only $380 billion. 

Anthropic then filed confidentially for an IPO on June 1, so Coxon's thread arrived while the company is still assembling a public offering.

He is also not the first to leave this way. Mrinank Sharma resigned from Anthropic's safety team on February 9, writing that the world is in peril.

That was three days before the company closed a $30 billion round.

Four days after the IPO filing, Anthropic co-founder Jack Clark asked the industry to build itself a brake.

The industry "has a gas pedal, but it doesn't have a brake pedal," he said.

Clark and Anthropic Institute lead Marina Favaro proposed a coordinated way for frontier labs to slow down.

The Institute would then research how to verify that anyone actually had.

Claude now writes more than 80% of the code in Anthropic's own codebase, and Clark told the BBC that this figure could reach 100% within a couple of years.

Not everyone accepts that an AI-caused extinction is the right thing to worry about.

Andrew Ng, who co-founded Google Brain, has argued since 2023 that inflated extinction talk gives lobbyists a case for rules that would suppress open-source AI.

Anthropic complicates this reading by asking for a pause that it would have to observe itself.

Hubinger's above-10% extinction estimate has also found backing outside Anthropic.

Nobel laureate Geoffrey Hinton, often called the godfather of AI, was asked about Hubinger's estimate on BBC Newsnight that same day.

"A 10% chance seems not an unreasonable estimate to me," he said, while adding that nobody knows how to produce a sensible one.

Hinton backs the number, Ng thinks the conversation itself does damage, and Anthropic has to price a public listing either way.

Anthropic's position points to three things any values-led brand should plan for:

• Audit your own promise first. A company that leads with ethics invites customers, regulators, and staff to check every decision against the claim.
• Commit to candor on the bad days. Transparency that only shows up alongside good news stops counting as transparency.
• Give employees room to speak. Staff who accept a personal cost to say something hold more weight than any corporate statement.

Brand reputation built on a principle is a standing contract, and the people who signed on internally are the ones who enforce it.

Our Take: Can Anthropic Still Control Its Own Candor?

Anthropic hired people who take AI catastrophic risk seriously and then told them they can say their findings out loud.

We think the company is losing its grip on this honesty, and the brand it built is the reason why.

A company gets the employees that its positioning attracts, and Anthropic attracted researchers who would forfeit equity before staying quiet.

Private investors priced this positioning in without blinking, and the valuation went up anyway.

Public markets are a different audience, since an alignment lead putting extinction above 10% reads as a risk disclosure to shareholders.

After a listing, the cheapest thing to manage is what employees post or say.

We think that the first real test of this candor comes after the first quarter, when Anthropic has a stock price to protect.

Anthropic showed the Pentagon what its principles cost, and shareholders will name the next price for a product its staff says might end the world.

Building on AI means inheriting your vendor's reputation. Find your match among these top AI development companies in our directory.