Google explains the use cases for its different crawler types – Search Engine Land
Google has now added new details that explain the three categories its Google crawlers fall into, they include Googlebot, special-case crawlers and user-triggered fetchers.
In addition, Google now lists a JSON formatted file containing the list of IP addresses each of these different crawler types use.
Types of Google crawlers. At the top of this Googlebot page, Google listed these three crawler types:
- Googlebot – The main crawler for Google’s search products. Google says this crawler always respects robots.txt rules.
- Special-case crawlers – Crawlers that perform specific functions (such as AdsBot), which may or may not respect robots.txt rules.
- User-triggered fetchers – Tools and product functions where the end-user triggers a fetch. For example, Google Site Verifier acts on the request of a user or some Google Search Console tools will send Google to fetch the page based on an action a user takes.
IP addresses. Google also listed the IP address ranges and reverse DNS mask for…
The post Google explains the use cases for its different crawler types – Search Engine Land first appeared on SEO, Marketing and Social News | OneSEOCompany.com.
source: https://news.oneseocompany.com/2023/04/21/google-explains-the-use-cases-for-its-different-crawler-types-search-engine-land_2023042143837.html
Your content is great. However, if any of the content contained herein violates any rights of yours, including those of copyright, please contact us immediately by e-mail at media[@]kissrpr.com.