Two weeks ago, many in tech and even outside of it were debating answers generated by Google's AI, which sometimes instructed people to eat rocks or make pizza out of glue.
This week, we've been seeing and hearing about discussions about Google's bad AI answers. (Thanks for chiming in, Defector.)
But I see and hear about it a lot less, and I haven’t seen a social media post about bad AI answers going viral in a while.
So has Google already fixed the AI answers they're calling “AI summaries”? Or has it started showing the AI answers less frequently, making it less likely that users will find a bad answer?
In a blog post published a week ago, Google explained why some of the bad AI answers were generated, claiming that there weren't many of them, and that it was limiting the use of such answers, such as “hard news topics where freshness and factuality are important.”
Google PR also released an updated statement: “AI Summaries are designed to provide value beyond existing features on search results pages and to surface for helpful queries, and they will continue to appear for many searches. We're continually improving when and how we show AI Summaries to make them as helpful as possible, including several technical updates we've made over the past few weeks to improve response quality.”
However, there are two more data points that suggest the following: something Happened.
First of all, people seem to have stopped complaining about it on social media.
According to data from social media monitoring firm Brandwatch, users of Company X (which I still refer to as Twitter) started paying attention to Google's AI brief the day after Google's May 14 I/O event. Then, a week later, things started to pick up steam, likely as people saw examples of the really bad answers Google had provided. (As Google points out, some of those bad answers were actually fake. Note the correction at the end of this New York Times article.)
Of course, it's possible that Google is generating just as many “bad answers” as before, and X/Twitter users may have migrated to other, more appealing options.
However, it's also very possible that you just aren't seeing as many ads. First, Google says it's already working to resolve some of the issues, including the “throttling.”[ing] This includes “including satirical or humorous content” in answers, or in some cases simply not using AI answers.
Another argument in support of the “there's less to see” claim comes from search optimization company BrightEdge, which says it's been tracking Google's AI Overviews since Google first began testing it last fall, primarily among users who signed up to try it out through the initially experimental Google Labs.
At one point, BrightEdge founder Jim Yu said, some keywords were generating AI answers 84% of the time. But by the time of Google I/O, when the company announced it was rolling out AI answers to most users, that number had dropped to around 30%. Then, within a week of the announcement, that number had dropped again, this time to around 11%. (A Google spokesperson disputes BrightEdge's methodology and claims its results are inaccurate; Google doesn't provide its own statistics.)
Bright Edge
None of this is conclusive, but for now, it looks like Google has weathered the worst storm of its own making.
