Log file analysis
Reading a server's access logs to see exactly which URLs search engine crawlers request, and how often.
Logs are the only first-hand record of crawler behaviour. They show wasted crawling on parameters or redirects, important pages crawled rarely, and errors served only to bots. Verify bot identity by reverse DNS, since many requests fake well-known crawler names.
Crawling & Indexing timeline · Latest crawling & indexing news · What Google said