The robots.txt file has certain effects on the Search engine optimization (SEO) or the indexing of one's websites on search engines such as Google, Bing, Yahoo, Ask, etc.
Impact of robots.txt on SEO: what does the file do?
Die Datei robots.txt ist eine Datei, die, wenn Sie vorliegt, den Suchmaschinenrobots auf ihrem Weg durch das Netz mitteilt, welche Websites sie nicht indexieren sollen. Die in der robot.txt erwähnten Seiten erscheinen also nicht in den SERPS (Search Engine Result Pages = Suchmaschinenergebnisseiten).
There are two ways to find out which pages are filtered by search engine robots:
1. a look into the robots.txt file
2. Google Webmaster Tools Status/Blocked URLs section
robots.txt - Effects on SEO: When do I need the file?
You only need the robot.txt if you want to exclude certain pages of your website from being indexed by search engines. As long as you have all your pages in the Index of the search engines, you can do without the robots.txt or simply delete it if it is present.
Examples of pages in connection with the use of robots.txt, i.e. the exclusion of pages from the SERPS, are first and foremost all security-relevant pages of a website. Pages relating to the payment process can and should therefore be written in robots.txt and thus excluded from search engines.
Everyone should also think about which pages he or she does not want to see in the SERPS.
Where is the robots.txt file located in WordPress?
With WordPress, the robots.txt is usually located in the root, i.e. in the root directory of the domain, i.e. in the folder where the wp-config.php file is also located.
Effects on the SEO
Die tatsächlichen Auswirkungen auf die Suchmaschinenoptimierung sind eher sekundär zu sehen, da die robots.txt nur einen inversen Charakter hat. Will man allerdings Wortfamilien, die in bestimmten Seiten vorkommen und keinen direkten Zusammenhang zum Hauptthema der Website darstellen, unterdrücken, haben die in der robots.txt aufgeführten Seiten und ihre Verlinkungen einen Einfluss darauf.
Of course, the smaller the number of content pages on the website, the greater this influence. Put the other way around. The larger a website becomes, the less influence robots.txt will have on the overall ranking of the website in the SERPS.
By 
German 
English 