What does the robots.txt file do? - Effects on SEO

The robots.txt file has certain effects on the Search engine optimization (SEO) or the indexing of one's websites on search engines such as Google, Bing, Yahoo, Ask, etc.

Impact of robots.txt on SEO: what does the file do?

The robots.txt file is a file which, if it exists, tells the search engine robots which websites they should not index on their way through the network. The pages mentioned in robot.txt therefore do not appear in the SERPS (Search Engine Result Pages).

There are two ways to find out which pages are filtered by search engine robots:

1. a look into the robots.txt file
2. Google Webmaster Tools Status/Blocked URLs section

robots.txt - Effects on SEO: When do I need the file?

You only need the robot.txt if you want to exclude certain pages of your website from the indexing of the search engines. As long as you keep all your pages in the Index of the search engines, you can do without the robots.txt or simply delete it, if it should be present.

Examples of pages in connection with the use of robots.txt, i.e. the exclusion of pages from the SERPS, are first and foremost all security-relevant pages of a website. Pages relating to the payment process can and should therefore be written in robots.txt and thus excluded from search engines.

Everyone should also think about which pages he or she does not want to see in the SERPS.

Where is the robots.txt file located in WordPress?

With WordPress, the robots.txt is usually located in the root, i.e. in the root directory of the domain, i.e. in the folder where the wp-config.php file is also located.

Effects on the SEO

The actual impact on the Search engine optimization are rather secondary, as the robots.txt only has an inverse character. However, if you want to suppress word families that occur in certain pages and are not directly related to the main topic of the website, the pages listed in robots.txt and their links have an influence on this.

Of course, the smaller the number of content pages on the website, the greater this influence. Put the other way around. The larger a website becomes, the less influence robots.txt will have on the overall ranking of the website in the SERPS.