Crawling & Indexing: a timeline

Latest crawling & indexing news · This month's crawling & indexing recap · RSS for this topic · What Google said about it

1. October 2026 42 headlines

   • Supporting a shared AI training control for websites Google Search Central · 30 Oct
   • New learning paths: curated YouTube playlists for Search fundamentals and Technical SEO Google Search Central · 6 Oct
   • Decodo Launches Web Data API To Give AI Agents Web Access Search Engine Journal · 28 Oct
   • Google Explains 503s, Robots.txt Limits & Bracket URLs – SEO Pulse Search Engine Journal · 24 Oct
   • Three Reasons Why You Should Care About Video Indexing Onely · 30 Oct
   • Why isn’t Google indexing my page? 14 reasons Onely · 30 Oct

2. September 2026 1 headline

   Spam update September 2026 spam update began 24 Sep

   • Screaming Frog Crawling Clinic Returns to brightonSEO San Diego ’26! Screaming Frog · 9 Sep

3. August 2026 4 headlines

   Spam update August 2026 spam update began 18 Aug

   • Google Search Having Indexing/Serving Issues August 28th Search Engine Roundtable · 28 Aug
   • How Search Engines Actually Crawl and Index Sites SEO-Serps Blog · 8 Aug
   • Do LLMs Respect robots.txt? Where robots.txt Can and Can't Block AI Seer Interactive · 7 Aug
   • I Asked a 15-Year SEO Veteran How to Rank in ChatGPT | Dan Petrovic (DEJAN AI) Ahrefs Podcast · 18 Aug

4. July 2026 7 headlines

   • How to read the Indexing Report Search Off the Record · 16 Jul
   • Should you block your Search result pages? Search Off the Record · 30 Jul
   • Should You Noindex Thin or Low-Value Pages? SEO-Serps Blog · 18 Jul
   • Crawl Budget: Why It Matters on Larger Sites SEO-Serps Blog · 18 Jul
   • Why your pages are stuck in crawled-currently not indexed. And what to do about it. Marie Haynes · 16 Jul
   • How to find and fix crawl budget waste with log files Oncrawl · 28 Jul

5. May 2026 5 headlines

   Core update May 2026 core update began 21 May

   • 10 Best Technical SEO Agencies To Improve Your Site in 2026 Onely · 27 May
   • Oncrawl MCP: Connect your crawl data to Claude Code to automate your SEO audits Oncrawl · 21 May
   • Rise in Crawled – currently not indexed, preparing websites for agents, and how I’m using Antigravity. Marie Haynes · 13 May
   • Evolving role of the index: From ranking pages to supporting answers Bing Search Blog · 6 May
   • Evolving role of the index: From ranking pages to supporting answers Bing Search Blog · 6 May

6. April 2026 4 headlines

   • The brightonSEO Crawling Clinic April/May ’26 Screaming Frog · 27 Apr
   • Analysing Robots.txt at scale with HTTP Archive and BigQuery - transcript Search Off the Record · 23 Apr
   • Analysing Robots.txt at scale with HTTP Archive and BigQuery Search Off the Record · 23 Apr
   • Green SEO: Leveraging Screaming Frog for a Sustainable Web Screaming Frog · 16 Apr

7. March 2026 1 headline

   Core update March 2026 core update began 27 Mar

   Spam update March 2026 spam update began 24 Mar

   • Are websites getting "fat"? Page weight, HTML size & Googlebot limits explained - transcript Search Off the Record · 30 Mar

8. January 2026 1 headline

   • Data-Driven Content Pruning at Scale Using APIs and the Screaming Frog SEO Spider Screaming Frog · 12 Jan

9. July 2025 2 headlines

   • Keeping Content Discoverable with Sitemaps in AI Powered Search Bing Webmaster Blog · 31 Jul
   • Keeping Content Discoverable with Sitemaps in AI Powered Search Bing Webmaster Blog · 31 Jul

10. June 2025 2 headlines

    • Start Using Bing Webmaster Tools to Improve Your Site Visibility Bing Webmaster Blog · 17 Jun
    • Start Using Bing Webmaster Tools to Improve Your Site Visibility Bing Webmaster Blog · 17 Jun