To summarize the week
Dario Amodei A deep dive into the future of AI and its risks. open art Raise $30 million. of bay area its extension start Superiority. robotaxi company Wabi close figure nine roundWith the help of Uber. Elon Musk pull back tesla I can see the car SpaceX and xAI merger before the biggest one IPO In history. Technology industry leaders speak out against it icebut employee is noisier. of new TikTok Get off to a tough start. OpenAI See new heights $100 billion Fundraising.
Main item
Anthropic CEO based on his accomplishments as an entrepreneur Dario Amodei I’m on top of the world. With Claude Code, he has the most popular product in an industry destined to be the biggest and most powerful in history, and the five-year-old company is close to reaching its goal. Valuation of $350 billion. Despite the odds, Anthropic more than put up a fight against OpenAI and a formidable group of Big Tech companies aiming to claim the AI throne.
With that in mind, you can trust Amodei to have some insight into generative AI. And the most impressive lesson he learned from this was explained in the following book: A creative 19,000 word essay This week, we’re talking about his humility in the face of the incredible power of technology.
He does not pretend to have a clear outlook on what will happen in the future. Instead, he uses the scientific language of probability to outline the many potential risks to humanity posed by the steady advance of generative AI. His tone and approach stand in dramatic contrast to the simplistic certainty often heard from people on both sides of the AI regulation debate.
Amodei believes that, although this is not a certainty, perhaps within a few years we will have AI that can do almost everything better than humans. It poses many risks to humanity, which he divides into five categories. Bad people abusing AI. Bad governments and factions that abuse AI. Social turmoil such as mass unemployment. And then there are the weird things we never expect.
On the first point, the classic “Sorry, Dave.” We take a scenario where an AI develops an evil idea and acts on it, and carefully deconstruct the argument as to why this is impossible. At the heart of Anthropic’s security measures is Claude’s “Constitution,” which is designed to instill good values in our models to prevent malicious behavior. think It would work, but he’s not sure.
“A lot of very strange and unpredictable things can happen,” he says. First, if the AI knows it’s being tested, it also knows it’s working optimally.
AI falling into the wrong hands, privately or state-sanctioned, is probably a more immediate risk, and at least some aspects of that problem are easier to mitigate. Amodei hopes that Claude’s constitution will prevent him from cooperating with biological weapons. But just in case that’s not the case, Anthropic has implemented a “classifier” that specifically blocks bioweapons-related output.
Amodei says this isn’t cheap, adding about 5% to the cost of inference, and points out that not everyone would do such work unless legally required to do so. Also, many other risks can be costly and difficult to manage.
Of course, Amodei’s essay is more than just a thought exercise. He is advocating for regulations focused on model transparency, similar to those recently passed in California, which would require Frontier Labs to disclose their policies for testing and evaluating models and ensuring their safety, as Anthropic has already said. This is done voluntarily.
He takes a not-so-veiled swipe. Elon Musk It discusses why strictly voluntary guidelines for safety practices are ineffective.
“While it is extremely valuable for individual AI companies to engage in good practices, get better at working with AI models, and publicly share their results, the reality is that not all AI companies are doing this. Even if the best companies have good practices, the worst can still be dangerous to everyone,” Amodei wrote. “For example, some AI companies have shown an alarming neglect towards the sexualization of children in their models today, and it is questionable whether they will show the inclination or ability to address autonomy risks in their future models.”
Amodei is target White House AI Advisor david saxwhich has accused Amodei of trying to take advantage of Anthropic through “regulatory capture.” his ideas are consistent We’ve been thinking about the safety risks of AI since before Anthropic was founded. bag and others The Trump administration seems to believe that any dangers that AI may pose have been exaggerated, that as long as we stay “ahead of the curve,” this technology will be of unmitigated benefit to the United States, and that advocates for regulation and broader policy discussions are either disingenuous or misinformed, or both.
Perhaps they are right. But what if it’s wrong?
Amodei makes a convincing case that it would be foolhardy to wave away the idea that bad things can happen, given how much we still don’t know about how AI will work and how terrible the bad things will be. You don’t have to be a destroyer to understand that logic.
Whether you agree with his recommendations or not, Amodei deserves credit for driving dialogue and sparking a much-needed national debate about the opportunities and challenges of AI. Given the company’s current trajectory, customers and investors may even appreciate his streamlined approach. Let’s hope so.
Funding limited
From movies to billboard charts, AI art is creeping into our visual environment. Former Google PM Coco Mao and senior engineer John Chao co-founder of open art, We predict that creators will want access to top AI visual models all in one place.
The company shared: newcomer The only difference is that it raised $30 million in a Series A round led by Canaan Partners. Basis Set and DCM also participated in the round. basic set Lan Xuezhao He had previously led a seed round for Open Art, and after that meeting, Mao picked up Xuezhao from the San Francisco airport and pitched her on the way to the next meeting.
Founded in 2022, open art started as a Pinterest-like network for AI-generated images, then pivoted to making access to generative AI tools a core service. The desktop website allows creators to create images and videos using cutting-edge photo, video, and audio models from a variety of providers (Veo 3, Sora 2, Stable Diffusion, etc.) with one subscription.
Although OpenArt started with a consumer focus, many of its paid subscribers are marketers, influencers, or artists who fit into the “prosumer” category. Strong subscription growth drove ARR from $10 million to $70 million through 2025.
“I believe the future of content will be created by AI, but most content will be created by humans using AI tools,” Mao said.
OpenArt competes directly with fast-growing AI audiovisual tools startups such as Krea and PE-owned European giant Freepik. Krea raised even more money. $83 million To date, we have a much larger team of 83 employees, compared to 20 at OpenArt.
newcomer podcast
This week’s Newcomer Podcast features: Tony Fadellthe creator of Apple’s iPod and an all-around Silicon Valley hardware legend.
We talk about where the next major tech device will come from, whether it’s a pin, pen, headphones, or the device you already have in your pocket, and how Apple and other major tech companies are approaching the future of hardware.
We also discuss the rumors surrounding Fadel as a candidate to be Apple’s next CEO, what he would do if he took the role, and how leadership decisions at that level are actually made. Fadel shares his perspective on why OpenAI is pursuing a “too big to fail” strategy and what that suggests about the next phase of the industry.
