Googlebot

What is the Googlebot?

The Googlebot is a webCrawling-software search bot (also known as a spider or web crawler) that collects the web page information needed to provide Google search engine results pages (SERP) can be used.

The bot collects documents from the web to build Google's search index. By constantly collecting documents, the software discovers new pages and updates existing pages. Googlebot Uses a distributed design that spans many computers so it can grow with the web.

How does the Googlebot work?

The web crawler uses algorithms to determine which pages to crawl, at what frequencies to crawl the pages, and from how many pages to retrieve information. The Googlebot starts with a list generated from previous sessions. This list is then supplemented with the webmasters' sitemaps. The software scans all linked items on the web pages it crawls and notices new pages, updates to pages and dead links. The collected information is used to generate the Index from Google on the web.

How can I influence the Google crawler?

The bot creates a Index within the restrictions set by webmasters in their robots.txt files. For example, if a webmaster wants to keep pages hidden from Google search, he can block theGooglebot in a robots.txt file in the top folder of the website.

To prevent Googlebot follows links on a particular page of a website, he can add the nofollow meta tag; to prevent the bot from following individual links, the webmaster can add rel="nofollow" to the links themselves.

The webmaster of a website can detect every few seconds the visits of computers to google.com that use the user agent Googlebot display. In general, Google tries to index as much of a site as possible without overloading the site's bandwidth. If a webmaster finds that Googlebot consumes too much bandwidth, it can be switched on the Homepage the Search Console from Google set a rate that remains valid for 90 days.