Table of Contents
You can use a specific <META> tag to tell search engine robots to not index the content of a specific webpage. If you would like to find out how to stop bots from accessing your entire website please read our article on creating a Robots.txt file. You can also use the same tag to direct the bot to not scan the page for links to follow as well. This is a great tag to use if you are only trying to restrict one page on your website. Here’s a basic ‘no follow, no index’ tag in HTML code:
There are three important peices of information you need to keep in mind when using the <META> tag:
- Robots can ignore the tag if they wish. Typically, the bots that will ignore the <META> tag are rogue bots that are scanning the web for security vulnerablities to exploit or email harvesters used by spammers to grab email addresses.
- the NOFOLLOW directive only applies to links on the page. If the bot finds a link on another page without a NOFOLLOW it will still arrive on the page
- The NOFOLLOW in this tag is NOT the same as the rel=”nofollow” attribute you can set on an <a href> tag
Writing a ROBOTS META Tag
Like other <META> tags used in HTML they should be placed in the HEAD section of your HTML code just like in the example above.
<META NAME=”ROBOTS” CONTENT=”NOINDEX, FOLLOW”>
So lets break the above example down further.
- The “NAME” attribute will always be “ROBOTS“.
- The “CONTENT” attribute can have four different values: “INDEX”, “NOINDEX”, “FOLLOW”, “NOFOLLOW”
Multiple comma separated values are allowed but only certain combinations make sense. If there’s no <META> tag for ROBOTS present the default is “INDEX, FOLLOW” so there’s no need to specify that. You can have the following:
If you need further assistance with blocking robots from indexing your webpages, please read our article on How to stop Search Engines from crawling your Website. If you need furhter assistance you can always contact our support department 24 hours a day 7 days a week.
