Using a META tag to block bots from indexing a webpage

You can use a specific <META> tag to tell search engine robots to not index the content of a specific webpage. If you would like to find out how to stop bots from accessing your entire website please read our article on creating a Robots.txt file. You can also use the same tag to direct the bot to not scan the page for links to follow as well. This is a great tag to use if you are only trying to restrict one page on your website. Here’s a basic ‘no follow, no index’ tag in HTML code:

 <html> <head> <title>...</title> <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"> </head>

There are three important peices of information you need to keep in mind when using the <META> tag:

Robots can ignore the tag if they wish. Typically, the bots that will ignore the <META> tag are rogue bots that are scanning the web for security vulnerablities to exploit or email harvesters used by spammers to grab email addresses.
the NOFOLLOW directive only applies to links on the page. If the bot finds a link on another page without a NOFOLLOW it will still arrive on the page
The NOFOLLOW in this tag is NOT the same as the rel=”nofollow” attribute you can set on an <a href> tag

Writing a ROBOTS META Tag

Like other <META> tags used in HTML they should be placed in the HEAD section of your HTML code just like in the example above.

So lets break the above example down further.

The “NAME” attribute will always be “ROBOTS“.
The “CONTENT” attribute can have four different values: “INDEX”, “NOINDEX”, “FOLLOW”, “NOFOLLOW”

Multiple comma separated values are allowed but only certain combinations make sense. If there’s no <META> tag for ROBOTS present the default is “INDEX, FOLLOW” so there’s no need to specify that. You can have the following:

 <META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW"> <META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW"> <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

If you need further assistance with blocking robots from indexing your webpages, please read our article on How to stop Search Engines from crawling your Website. If you need furhter assistance you can always contact our support department 24 hours a day 7 days a week.

0 thoughts on “Using a META tag to block bots from indexing a webpage”

Hi [email protected],

Thanks for getting back to me. It looks like you are running Moodle version2.1.1 – and you need PHP 5.3.2. The shared server you are on has 5.2.17 running. Currently, we do not have any shared servers running php 5.3.2 – so you will need to install an older version of Moodle for it to operate correctly on the server. To install a compatible version of Moodle (Moodle 2.0.5+), which requires PHP 5.2.8 or higher and MySQL 5.0.25 or higher, it can be downloaded from https://download.moodle.org/ and installed manually.

I hope this helps clarify the situation. If you need further assistance please feel free to contact us.

Thank you!

Tim S

The domain is freetechclass.com/lmsys
the blank pages dont happen until after you log in.

thanks!

Was this article helpful? Join the conversation!

Need More Help?