Understanding AI Overview Fails: Pitfalls and How to Avoid Them

I hope you enjoy reading this blog post. If you want my team to handle your marketing for you, click here.

Key Takeaways

• AI Overview fails occur when Google’s new AI feature produces incorrect, inaccurate, or irrelevant answers.

• These failures happen because Google’s AI tools can’t fully understand the sources they cite, often misinterpreting irony and satire as fact.

• AI failures can damage brand reputation, associating them with inappropriate or misleading content.

• You can prevent your content from appearing in AI Overviews, but it will also remove your site from Google’s index.

• Google has made several changes to its AI Overview feature, including reducing reliance on user-generated content and improving safeguards for health topics.

What AI Overview Fails Are Users Seeing?

Google designed AI Overviews to summarize the most relevant search results, saving users from wading through numerous links. However, the feature often fails to work as intended. Below, we explore some of the common types of bad answers from Google AI and findings from our own research involving 623 U.S. digital marketers.

Inaccurate Answers

Inaccurate answers are probably the most common type of AI Overview fail. The feature often draws from inaccurate or ironic sources, presenting false information as fact.

Example:

Google suggested that 13 US presidents have attended UW-Madison:

• Andrew Jackson graduating in 2005

• Andrew Johnson studying from 1947 to 2012, earning 14 degrees

This type of fail was encountered by 67.35% of respondents in our study.

Outdated Answers

AI Overviews don’t always use the most up-to-date sources, leading to outdated and inaccurate answers.

Example:

Google suggested that Barack Hussein Obama was a Muslim president, citing an outdated academic study.

This issue was encountered by 58.02% of respondents.

Inappropriate Content

Sometimes, AI Overviews produce not just inaccurate but also inappropriate content.

Examples:

• Health Benefits of Running with Scissors: Suggested by Google AI.

• Recommendation to Drink Urine to Flush Kidney Stones: Another inappropriate suggestion.

Inappropriate content was seen by 60.06% of respondents.

Irrelevant Content for the Query

Sometimes, AI Overviews don’t even provide a relevant answer.

Example:

Jake Kleinman, the executive editor of Inverse, searched for the biggest box office flops of 2024 and received a speculative list with made-up box office numbers instead.

This type of failure was experienced by 53.06% of respondents.

Why AI Overviews Are Having Problems

AI Overviews use a custom version of the Gemini large language model (LLM) that incorporates Google’s Search systems. This means they work differently from chatbots and other AI tools, which only use training data to generate answers.

Issues:

• Misinterpretation of Questions: AI Overviews tend to misinterpret questions and provide inaccurate answers.

• Problems with Humor and Irony: For example, the suggestion to add glue to pizza originated from an ironic comment in a 12-year-old Reddit thread.

• Satirical Sources: The inaccurate presidential example came from a satirical article.

Google’s head of search, Liz Reid, has acknowledged these issues, but also pointed out that some nonsensical queries are aimed at producing erroneous results.

Avoiding AI Overview Fails: What You Can Do And What Google Is Doing

Potential Issues for Brands:

• Associating with False or Misleading Content

• Incorrectly Summarizing Work

• Reducing SERP Clickthrough Rates

What Can You Do?

• Removing Content from Google’s Index: This can be done using disallow directives in robots.txt or the data-nosnippet HTML tag.

• Rethinking Google Ads Strategy: Given that Google is experimenting with ads inside AI Overviews, you may want to avoid these placements to prevent associating your brand with inaccurate content.

What’s Google Doing to Prevent These Issues?

Google has taken several steps to reduce and prevent AI Overview failures:

• Better Detection of Nonsensical Queries

• Less Reliance on User-Generated Sites like Reddit and Quora

• Restricting Instances when AI Overviews are Not Helpful

• Strengthening Guardrails on Health-Related Topics

According to a blog post by Liz Reid, Google has made over a dozen technical improvements to its systems.

FAQs

What steps has Google taken to improve AI Overviews after identifying these issues?

Google has taken several steps, including:

• Strengthening guardrails on health-related topics

• Improving the detection of nonsensical queries

• Placing less reliance on user-generated sites like Reddit and Quora

• Reducing instances of AI Overviews

Can I turn off Google AI?

Not really. AI Overviews are part of the search engine results page and can’t be switched off. The only way to avoid them is by using a browser extension that filters them out or by using a VPN to mimic browsing from another country without them.

What are some common failures associated with the Google AI Overview update?

Common failures include:

• Inappropriate content

• Inaccurate answers

• Irrelevant answers

• Outdated answers

Conclusion

Unless you work for Google or want to forgo search traffic entirely, you can’t do much to stop these AI fails. However, you can optimize for them to better control the narrative and improve the results.