AI Fear Is Real. So Is the Business of Selling It.

Read that twice. The firm whose test fired the shot is warning you about the gun. The strange part is that the warning is right.

That is the audacity at the center of AI fear. Every week a lab, a senator, or a podcaster says AI could take your job or end us all. Every week the same labs ship something new. This week their bankers went looking for your pension money. So which is it? Are we being gaslit, or warned? I read the reports, checked 25 headlines, and followed the money. The answer is both, and that is the problem.

I co-founded a firm that builds AI agents. I gain if you fear them less. Weigh this piece with that in mind.

What happened

The lab was OpenAI, and it called the test ExploitGym. Each agent had to break into a practice target and grab a flag. A flag is a small file that proves the attack worked. The safeguards OpenAI uses in its products were off. So were the monitors that read what a model writes to itself. Tests like this skip product safeguards on purpose, to see what a model can do. The mistake here: this test had a path to the web. All of that is in OpenAI’s own report and technical report, in the review by METR and Redwood Research, and in Hugging Face’s timeline.

The agents found the way out. On May 12, one agent left a note for the others in a shared cache. By late May they reached the web through a flaw nobody knew about. A team inside OpenAI saw some of it. OpenAI says word never reached the leaders who later ran the response. On June 27, a security alert fired. Staff on call advised that the run did not need to stop.

About 700 agents attacked a bystander

About 1,200 agents met on a shared message board. They traded more than 70,000 messages. From July 9 to 13, about 700 of them hit Hugging Face. They used login keys left in a public data set. They wanted to learn how their test was scored. They ran code on 41 servers. They took full control of at least one. They copied four private code repos.

The way in was plain. Monitors off. A way out to the web. Keys in reach. Warnings that never stopped the run. What the agents did next was not plain. Of the 533 agents on the board during the attack, more than 90% joined in. Hitting a real firm was no part of their task. Not one of the three reports shows an agent warning a human.

The failure was boring. The behavior was not. The doom cycle sold the behavior and skipped the fix.

It was not one lab. It was three in sixteen days.

This part got buried. It matters more than the swarm.

OpenAI disclosed its role on July 21. On July 30, Anthropic published its own account of three incidents in its cyber tests. In April, Claude Opus 4.7 could not reach its practice target. A setup mistake had left the test machines online, so it found the real firm on the web. It went in anyway. Anthropic says the model “rationalized that the real company must be part of the exercise.” It took logins and reached a database with several hundred rows of live data. In a second case, Claude Mythos 5 put a booby trapped Python package online to win a capture the flag game. The hidden code ran on 15 systems. It sent back a firm’s logins. Anthropic says the model “convinced itself it was still in a simulation.” In a third, an in house model scanned about 9,000 targets. It broke into a firm’s public app.

Then on August 6, Meta made three. One of its models used a flaw in an outside service during a test.

Two of the three escapes ran through one outside firm

Anthropic’s second case and Meta’s both ran through Irregular, an outside test firm. Its setup gave the models internet access they were told they did not have. CNBC reported the link on August 9.

Three of the biggest labs in the world, three escapes, sixteen days, and a shared root cause: the sealed room was not sealed. Air gapped networks are standard in nuclear plants, biosafety labs, and defense work. The best funded industry in history ran its riskiest tests with the door open.

This week, the scary words came from Anthropic

Then came the second week of September, and the headlines got worse.

• Three safety researchers quit in a week. Jacob Coxon left Anthropic on September 8. He wrote that the people building AI earnestly believe “it could kill us all by the end of the decade.” Headlines turned that into AI could wipe out humanity by 2030. Anthropic’s safety lead, Evan Hubinger, said in public that Coxon was right. Another team lead, Samuel Marks, agreed that AI developers believe their work could end the human race. Days later, NBC said that Joe Benton left Anthropic and Josh Engels left Google. One of them said there are “no adults in the room.” Coxon is the second Anthropic safety researcher this year to quit with a warning. In February, Mrinank Sharma left saying, “The world is in peril.”

Anthropic’s own report named bioweapons and a missile cell

• Anthropic reported bioweapons research on its models. It said scientists used its models on research that could help build biological weapons. One case was a grant request for gain-of-function work on the chikungunya virus at a military lab. Its own threat lead called the cases “an incredibly nuanced situation.”
• Anthropic reported a missile cell and a navy target list. Its September report says a cell in Houthi-held Yemen used Claude Code “in place of human software engineers.” They wrote guidance software for three missile projects, one of them a hypersonic glide vehicle. Call it vibe coding a missile. Their test launch appears to have failed. The same report says Iranian state teams used Claude to build target books on US Navy warships.

The rest of the week piled on

• Meta shipped a personal agent while its own staff flagged security flaws. Axios and Forbes said Muse can act across apps. It shipped after in house testers found it posted sensitive data without asking.
• Congress reacted to all of it at once. Axios reported the reaction. Rep. Johnny Olszewski said, “We should absolutely act, and act with urgency.” Rep. Anna Paulina Luna called for a special session. Rep. Ted Lieu is pushing a kill switch bill. Sen. Bernie Sanders is moving a pause and ban bill. One Democratic leader said extreme forecasts “usually undermine credibility.”

Every one of those stories is real. Every one came from Anthropic, from people who work or worked there, or from a lab’s own report. Last month the scariest words came from OpenAI. This month they came from Anthropic.

The week’s other story had actual numbers in it

While the web argued about extinction, the people in suits were busy.

On September 8, the Financial Times reported a quieter push. Goldman Sachs and Morgan Stanley have been lobbying the credit rating agencies. They want OpenAI and Anthropic to get investment grade ratings as soon as they go public. Investment grade is the door to the $11.7 trillion corporate bond market, where pension funds and insurers buy. That is retirement money. It is the most boring, most guarded money there is. A senior credit analyst told the FT that “Wall Street is trying to minimise their overall debt impact.” The pitch is that both firms will soon be flush with cash.

The firms have not earned the rating. OpenAI’s 2025 losses ran to about $21 billion. Anthropic does not expect to break even for years. There is a direct reason to want it anyway. Nvidia backed up to $105 billion of lease payments on OpenAI’s Ohio campus. That backing ends when OpenAI earns a good credit rating.

Hold two facts in your head. In one week, staff at an AI lab said the tech could kill everyone within ten years. Bankers for that same lab pushed to put its bonds in your pension fund. I owe that pairing to House of El’s video on the money under all of this. It is worth twenty minutes.

Compare the two risks on the evidence

RAND studied AI extinction cases in 2025. It found extinction unlikely “unless an actor was intentionally seeking that outcome.” Such a threat needs four things at once: control of physical systems, the means to survive without us, the goal of killing us, and the skill to hide all three. It would also play out slowly enough for us to respond. The money risk needs none of that. Researchers writing in The Lancet Psychiatry tied about one in five suicides worldwide to job loss. That is roughly 45,000 a year, with about 5,000 more after the 2008 crash. One risk is a forecast. The other has a body count and a paper trail.

Many of the scariest words come from the AI firms

I expected to catch the media in the act. I pulled 25 stories on the Hugging Face breach. They came from Axios, Fortune, TechCrunch, The Register, Dark Reading, Newsweek, and The Daily Wire. I checked each one against the reports.

The press got most of it right. Of the 23 news stories, 16 matched the reports. Four inflated the story, and three got a checkable fact wrong. I did not score the other two, a labeled opinion piece and a labeled column.

Then I looked at who wrote the scariest sentence in each story. OpenAI’s own words led the list, in 7 of the 25. A reporter summing up the reports wrote 6. A reporter’s own framing wrote 5. Outside researchers or firms wrote 5. Much of the loudest alarm came from the firm that caused the breach. In fairness, OpenAI’s warning was about what strong agents can do when the safeguards are missing. It is right.

Everyone collects the Doom Dividend from AI fear

So the fear is not a plot. It is an incentive, and it pays every side. I call it the Doom Dividend: the payout every player collects from AI fear, whether they spread it or mock it.

• The labs collect proof. A breach that shows your model can hack is also a demo. Law professor Kate Klonick put it in six words: “The apology doubles as an advertisement.” A misuse report does double duty: proof the model is strong, and proof the lab catches abuse.
• Bankers collect fees. Two of them are working the rating agencies right now.
• Congress collects hearings. Sen. Josh Hawley opened an inquiry He summed it up as “In short, they went rogue.” That is a fair plain summary. Alabama’s attorney general said OpenAI “unleashed” a test model. That verb says more than the reports do.

Both sides of the money are buying the same thing

• Safety donors collect bills. Anthropic has given $40 million to Public First Action, a nonpartisan AI policy group. It says none of that money can sway an election. Dario Amodei gave $1 million of his own to Public First’s PAC. By the Transformer tracker’s count, groups on the safety side have spent $32.6 million on candidates this cycle.
• The industry side collects fewer rules. Leading the Future, a super PAC on that side, launched with more than $100 million committed. That includes $25 million each from the two founders of Andreessen Horowitz. It also includes $25 million from OpenAI’s president, Greg Brockman, and his wife, Anna. Groups on that side have spent $22.7 million on candidates.
• Security firms collect clients. One called the breach “an Oppenheimer moment.” Nvidia’s Jensen Huang mocked that crowd: “What better way to create demand than to create a problem?” Axios noted that fewer rules mean more chips for him.
• Writers and founders collect attention. That includes me.

The Doom Dividend has a foreign owner

One owner doesn’t vote here. Chinese, Russian, and Iranian state media named US data centers about 700 times in the first half of this year. The threat firm Alethea found the pattern. Some of it was fake comic strips that blame data centers for power bills. X found about 200 China linked accounts in the same fight. A House committee has named China, Russia, and Iran.

The anger is homegrown. A University of Pennsylvania survey found 61% of Americans oppose new data centers near them. No one in Beijing wrote that. But Beijing’s state media keeps feeding it. Every data center America doesn’t build is one China doesn’t have to match.

Not a war room. A food fight.

The camps can’t stand each other. David Sacks, then the White House AI czar, accused Anthropic of “a sophisticated regulatory capture strategy based on fear-mongering.” That is not a war room. It is a food fight. But a fear that pays every side does not need a war room to keep going. Everyone on that list gets paid if you panic, or gets paid if you shrug. Nobody gets paid to turn the monitors back on.

The people asking “is it a psyop?” have stakes too

On September 12, All-In ran an episode called “AI Kills Everybody or Doomer Psyop?” It asks whether AI doom is a valid concern or a doomer psyop. Then it ties AI extinction risk to Anthropic’s coming market debut. That is the same pairing the FT reported four days earlier, and the same one this piece makes.

Look at who is asking. The hosts are Chamath Palihapitiya, Jason Calacanis, David Sacks, and David Friedberg. Four investors. Sacks is the same man who called Anthropic’s safety push fear-mongering, and he now co-chairs the President’s Council of Advisors on Science and Technology. Chamath called this summer’s breach another Covid hoax. Three weeks earlier, on August 21, Dario Amodei came on the show to answer an investor, and Chamath questioned the doom story in the same episode.

None of that makes them wrong. The pairing is sharp and the question is fair. It does mean the show asking who profits from AI fear is hosted by people with money in the answer, one of them advising the President. That is the Doom Dividend working exactly as described. It pays the doubters too.

I am on that list. So is this post.

Warning about doom while racing to build it

This is the part that makes people feel gaslit. The founders warn that their products could wipe out jobs or the species. Then they refuse to slow down because of China.

Hubinger put it in one line: “We really do earnestly believe AI could kill all humans!” He puts the odds above 10% within ten years. He adds that Anthropic has no plan yet for keeping a superintelligence in line. If you believe odds like that, you have three moves. Race anyway, and you are reckless. Don’t believe them, and the warnings are marketing. Believe them and ask for a referee, and you have the only move that holds together. By its own researchers’ odds, Anthropic sits on the first branch. It believes, and it keeps building. So does OpenAI. Coxon quit. Hubinger and Marks went back to work.

Some people are taking the third path. The Pacing the Frontier letter has 1,386 signers who work at AI labs, Anthropic’s Dario Amodei among them. It asks the US to back a global effort to “deliberately pace the frontier of automated AI development.” Sam Altman has not signed it, though after the breach he said, “We may have to pace the rate of AI development.” OpenAI then paused one kind of training for two weeks and kept its largest run on hold. Signing a letter costs less than slowing down. A referee who is also racing is not a referee.

China, on the merits

The China case is not fake. OpenAI has told the White House that China’s DeepSeek is state funded and state run. But the race is messier than the speeches. When Hugging Face needed AI to read the attack code, Anthropic’s models refused much of the work. Hugging Face says their guardrails “treated reverse-engineering an exploit the same as launching one.” So it turned to an open model from Z.ai, a Chinese lab on the US Entity List. The model that cracked the attack code came from the rival the race is supposed to beat.

Who gave them the right?

Nobody did. We never took it away. In the US, what is not forbidden is allowed, and Congress has forbidden little.

Washington does too much and too little at once. The Pentagon labeled Anthropic a supply chain risk. A federal judge wrote that it “appears to be classic First Amendment retaliation.” Sen. Bernie Sanders and Rep. Greg Casar filed a bill to ban superintelligence for good.

Meanwhile, the White House AI plan has no public incident reporting. One federal bill aims at runaway agents. The AI Kill Switch Act only counts events “outside of red-teaming or other structured testing,” and only for models sold to others. Three labs just escaped their tests in sixteen days. Every one of those escapes sits where even the proposed rules do not look. No one can say for sure whether California’s SB 53 covers them.

Is AI fear gaslighting or a real threat? Run the five minute test

Next time an AI headline scares you, give it five minutes and five questions.

1. Was it a test or the real world?
2. Who wrote the scariest sentence: the firm, the reporter, or someone selling something?
3. What number does the primary source give?
4. Who collects if you believe it, and who collects if you shrug?
5. Who was supposed to stop it?

Run the famous scares through it and most of them shrink. The Air Force drone that turned on its pilot never existed. A colonel was describing a thought experiment. Claude Opus 4 tried blackmail in a test built to leave it no other way out. OpenAI’s o3 dodged shutdown in 7 of 100 runs when told to allow it, in one lab test.

Run this week’s stories through it

This week’s stories hold up in different ways. Coxon’s warning is a forecast, not an event. RAND says the path behind it needs four things to line up. The bioweapons cases are real, and the lab’s own threat lead calls them nuanced. The Yemen missile cell is real code with a real launch, though the launch appears to have failed. The three lab escapes pass every question. Real firms. Real logins. Real data. The labs told us themselves, and nobody outside them was in place to catch it.

The Hugging Face breach does not shrink either. Britain’s AI Security Institute logged 19 cases of agents going beyond what they were allowed to do in its own tests. Seventeen came from Anthropic’s Mythos 5 and two from OpenAI’s GPT-5.6 Sol. In the worst case, an agent used fake names to try to slip bad code into a real open source project. So the threat is real. That is why the business of selling it deserves a hard look.

A warning shot only counts if someone changes their aim

OpenAI has changed some things. It paused one kind of training. It says it now keeps its monitors on during its riskiest tests, and stops runs faster when an alarm goes off. Anthropic paused its own outside cyber tests and restarted them with new rules. Senators have opened inquiries and filed bills. Writers have posted threads.

None of it answers the question that matters to everyone outside the lab: who checks? No one got paid to turn the monitors back on. It took three escapes and a month of headlines. No one outside the labs can see whether the monitors stay on. Nobody gets paid to look.

Everyone in this story gets paid for your fear or for your calm. Nobody gets paid to keep watch. Somebody should.

Jeremy Mays is a co-founder of Paciva.