What Google said about crawling & indexing

Only Google speaking for itself: posts from Google Search Central, the Search Status Dashboard, Google's Search product blog, the Search Central YouTube channel and the Search Off the Record podcast. When someone says "Google said", this is where to check.

October 2026

Google Search Central

Supporting a shared AI training control for websites

Today, we're announcing our plans to support a shared AI training control in robots.txt. This will help websites express their AI training preferences across multiple AI providers in a single step and is based on work from the AIPREF working group at the…

Google Search Central (YouTube)

Should we be scared of a 404 status code?

Do 404 "Not Found" status codes trigger site wide disasters or haunt your crawl budget? Should you redirect every dead URL straight to your homepage? 👻 ...And the most terrifying question of all: where is Chicken? 🐔 Welcome back to The Non-Canonical Cabinet…

Search Off the Record

PDFs in Search: Indexing, AI & Pitfalls

Should you publish your content as a PDF or stick strictly to standard HTML? Does Google treat PDF files differently when it comes to indexing, ranking, and AI search features? In this episode of Search Off the Record, Martin Splitt and John Mueller from the…

Search Off the Record

Do sitemaps still matter?

Sitemaps have been a core element of web development and search engine optimization for two decades, yet Search Console warnings like "Couldn't fetch" continue to generate concern among site owners. Are XML sitemaps still necessary with modern search…

July 2026

Search Off the Record

Should you block your Search result pages?

Is your website's internal search feature secretly acting as an open invitation for crawling lots and lots? In this episode of Search off the Record, Martin Splitt and John Mueller pull back the curtain on how internal search results pages can turn into…

Search Off the Record

How to read the Indexing Report

Should you panic when your Search Console indexing report is showing pages that aren't indexed? Is a 404 error code always a sign of a broken website? In this episode of Search Off the Record, Martin Splitt and John Mueller from the Google Search Relations…

April 2026

Search Off the Record

Analysing Robots.txt at scale with HTTP Archive and BigQuery

In this episode of Search Off the Record, Martin and Gary turn a simple robots.txt question into a data‑driven deep dive using HTTP Archive, WebPageTest, custom JavaScript metrics, and BigQuery. They explore how millions of real robots.txt files are actually…

March 2026

What counts as "Google said"

Posts on Google's own channels only. Statements Googlers make on social media or at conferences are reported by the trade press and appear in the main news stream, not here — a quote passed along is not the same as a post Google published. Summaries are the first lines of each post; follow the link for the full text.