Written by Peter Weinberger
For nearly 30 years, the Internet’s business model has been built on simple transactions.
The publisher created the content. Search engines like Google copied it and indexed it. Instead, the search engine sent readers back.
The arrangement was never perfect, and the delivery person was never actually paid for the story. But it sent readers our way. Local newspapers like the Claremont Courier report stories, publish them online, and can trust that readers looking for information about Claremont will find us through Google, Safari, Apple News, Facebook, Instagram, newsletters, email, or any other digital path.
That model is now rapidly changing. It was faster than I expected. Faster than almost all publishers expected.
Website analysis shows a sharp decline in suspected bot/crawler traffic from China and Singapore after the blocking measures were added, while a new spike in traffic from Japan is believed to be another automated crawler and is under investigation. Please note that there are very few participants from the US
Courier has seen a sharp decline in referrals from Google and Safari in recent months. At the same time, our websites are being attacked by automated bots from all over the world. This includes AI-related crawlers that scan and collect information from AI models at a scale that is impossible for human readers. These bots can appear to crowd traffic while doing little or nothing to support the journalism that created the information in the first place.
This is not just a courier issue. It’s happening on millions of websites. The reason is artificial intelligence.
Access to information is changing rapidly
More and more readers are no longer searching the web as much as they used to. Instead of typing a question into Google and clicking on a list of links, you’re asking the AI tool to answer you directly, using the information the AI gets by scanning websites like Courier.
Users may find this useful. For publishers, everything changes.
The old search model directed readers to the original source of information. New AI models often summarize these sources. Readers may get answers, but the newspaper, website, photographer, reporter, editor, or public agency that created the original information may not receive visits, subscriptions, donations, ad views, or a direct relationship with that reader.
This is a serious problem because original reports are not freely available. You may have heard me mention it once or twice on this page.
AI is not meant to replace fact-based stories. It’s up to them. And it’s the same battle we’ve had with Google over the use of Courier content. They’re not paying us. They use us to provide information that they themselves cannot produce. AI takes it a step further.
That’s why this moment is so important. The web is moving from a referral economy to an extraction economy. In the old model, publishers received traffic in exchange for access. In the new model, AI systems collect, summarize, and repackage information while sending far less value back to the people who created it.
Blocking all AI crawlers may make it difficult for readers to find accurate Courier reports when they ask questions about Claremont. If you allow all, our work may be used without compensation or proper credit. Blocking the wrong bots can hurt your visibility in search. Allowing the wrong bots can overload your site and skew your analytics.
we will fight back
Therefore, delivery personnel must be careful. But we also intend to fight back.
You need a deeper understanding of what traffic is human, what traffic is useful, what traffic is automated, and what bots are just taking things without giving anything back. We need better tools to separate the actual reader from the machine. We need to block rogue crawlers while making our reports visible to people looking for trusted local information.
Our web developers are already helping make this happen, but more work needs to be done. Currently, bots from 12 countries attempt to access websites every day.
But the larger solution is not technical. It’s about relationships. That means delivery drivers have to work harder and smarter to build more direct connections with readers.
We need readers to come to us directly. Please bookmark our website. Sign up for our newsletter.
Become a subscriber in print or online. Share Courier’s story with your neighbors. Send us a news tip. Support nonprofits that support local journalism in Claremont. Hundreds of people are already doing this, but we need more support.
Courier will focus on promoting more individual stories to avoid AI. We use a form of geofencing around our website, deploy the latest Cloudflare technology to push back bad bots, use Google Ads grants to promote specific stories beyond Claremont’s boundaries, and add 10 social media posts per week.
And that’s just the beginning.
Almost bot proof
The Courier has something that many local news publishers don’t have. It’s a trusted newspaper.
Unlike many digital-only publications, we’re still supported by a print edition that has a deep community trust. Our print edition remains central to our mission and business. It is not directly affected by AI crawlers. It is much more difficult for AI systems to scrape at scale. And it comes to you as a finished product, created, edited and fact-checked by people who live in and care about this community.
In an age of misinformation, that’s important. And its importance is only increasing.
Print may seem old-fashioned to some, but its value is increasing, not decreasing. It’s finite. Polished. We have a responsibility. It has a beginning and an end. Reflect your judgment. It tells our readers that this is what we believe was important this week in Claremont.
That doesn’t mean we’re turning away from digital news. Far from it. The Courier will continue to publish online. But we’ll be honest with our readers about what’s going on. Online business models are changing rapidly, and local journalism can’t survive by pretending nothing is happening.
AI will become part of how people find information. Some of it will be useful. Some of them may be harmful. The challenge is to ensure that original, fact-based reporting is not undermined in the process. And that doesn’t happen with the couriers I monitor.
For more than a century, Courier has adapted to new technologies such as hot type, offset printing, photography, desktop publishing, the internet, social media, smartphones, newsletters, video, and now artificial intelligence. Tools and technology are constantly changing. The mission is not.
That’s why local journalism is more important than ever.
How can we help?
