Meta Robots Tag

« Back to Glossary Index

Have you ever wondered what the robots file on your website is for? Maybe you use WordPress and stumble across this particular, unknown tag, the name ="robots" content="index"> says.

What is a Meta Robots tag?

The Meta Robots Tag is a tag that tells search engines what to crawl should and should not do. It is a piece of code in the section of your website. It is a simple code that allows you to decide which pages you want to hide from search engine crawlers and which pages you want to be indexed and viewed.

Another function of the meta robot tag is that it tells search engine crawlers which links to follow and which links to ignore.

What happens if you do not create a Meta Robots tag?

But if you don't have a meta robot tag, don't panic. By default, search engine crawlers will try to index all your websites and follow the links. Let me clarify that search engine crawlers following your links are not bad at all.

Basically, the meta robot tag can be broken down to four main functions for the search engine crawlers:

FOLLOW - a command for the search engine crawler to follow the links on this web page.
INDEX - a command for the search engine crawler to index this web page.
NOFOLLOW - a command for the search engine crawler NOT to follow the links on this website.
NOINDEX - a command for the search engine crawler NOT to index this web page.

Pretty simple, isn't it? Well, there are a few more commands for the meta robot tag, but these four are the main functions. These four functions are what meta robot tags are most often used for.

An example of meta robot tag code would look like this:

<meta name ="robots" content="index">

This tag is used to index the web page it is on. It's like telling someone to get a glass of water to get a glass of water. Because again indexes the Search Engine by default already your website, even if you do not use this code.

And you can also combine the commands if you want:

<meta name="robots" content="noindex,nofollow">

For me, this code is a good thing, but rather a small thing to keep in mind - especially when trying to work intelligently with Link Juice to deal with by nofollow applies to some outbound links. Apart from that, it's not something you need to check again and again when you edit your SEO.

« Back to Glossary Index

FAQ

What are meta tag robots?

Meta tag robots are special HTML tags included in the header of a website that instruct web engines to index, track, or ignore certain pages. These tags consist of the "name" attribute, the name of the tag, and the "content" attribute, which must specify the value of the tag.

What are the types of meta tag robots?

There are four basic tags you can use in your website header: index, noindex, follow and nofollow.

What is the effect of the respective index tag ?

- Index: This tag tells web engines that they can index the page. - Noindex: This tag tells web engines not to index the page. - Follow: This tag tells web engines to follow the links on the page. - Nofollow: This tag tells web engines not to follow the links on the page.

How can I use meta tag robots for my website?

To use meta tag robots on your website, you need to add the tags to the header of your web page. To do this, you must first go to the page you want to index, track or ignore and edit the header. Then you can add the meta tag robots tags.

When should I use noindex tags?

Noindex tags should be used when you do not want certain pages to be indexed by search engines. For example, you may not want directories, error pages, or special pages like login or registration pages to be indexed.

When should I use follow tags?

Follow tags should be used if you want search engines to follow the links on your page. This can be useful if you want to increase link weights on your site link network.

Can I prevent search engines from indexing my site?

Yes, you can prevent search engines from indexing your page with the "noindex" tag. This tag tells search engines to ignore the page so that it does not appear in search results.

Can meta tag robots affect my site if I use them?

Yes, meta tag robots can affect the way search engines treat your page, but they cannot guarantee that your page will show up in search results.

Can I use meta tag robots for my entire website or just for individual pages?

You can use meta robots tags for individual pages, but also for entire websites. Example: With the tag {meta name="robots" content="noindex, follow"} you can specify that search engines should not index your entire website, but should follow links on your page.

Can I use more than one meta robots tag on a page?

No, you can use only one meta robots tag on a page. Example: You cannot use {meta name="robots" content="noindex, follow"} and {meta name="robots" content="index, nofollow"} on the same page.

With top positions to the new sales channel.

Let Google work for you, because visitors become customers.

Niels Stuck

Niels Stuck has 10 years of SEO experience and is the founder of the SEO agency "WOLF OF SEO". He gained practical experience by building 20+ affiliate sites alongside his marketing studies. Finally, he wrote his bachelor thesis about the influence of SEO on Google rankings, traffic and sales development in the form of a case study. Today, he specializes in e-commerce SEO and helps more than 80 companies build a sustainable organic revenue channel through SEO. Niels advises startups, established brands and corporations in search engine optimization of their online stores and primarily focuses on data-based content strategies and link building. He shares his knowledge about SEO and online marketing in this blog, as a speaker at conferences, in podcasts and as a guest author for OMT, Forbes, Starting Up and many more platforms.

All contributions

Social Media & Links: