Recently I read a question that asked how one can block a web page from being indexed by search engines, since I think it is a popular question and it will benefit to my readers so I decided to post my
answer here:
Generally there are 2 methods that you can block search engine accessing your web page, one is by adding a simple META exclusion tag on the web page and the other one is by using a robots.txt.
Now how exactly can you do this?
META robots exclusion
The second part “noindex” tells the search engine not to list, index or store your page in their database. The “nofollow” basically tells the search engine ignore any URLs that appear on your page.
When you add the above code, you have to add it between
and in the HTML code otherwise it won’t workRobots.txt exclusion
The other method requires you to create another file named robots.txt (you have to name it exactly as I showed in ALL small letters for it to work) and upload it to your web server ROOT, home directory or home folder. You can use a notepad or any programmer style text editor (NOT MS office) to create that file. The format will looks like this:
User-agent: *
Disallow: /your-disallow-directory/your-disallow-file.html
Disallow: /your-disallow-directory-2/
etc
The big advantage of using robots.txt is you can actually hide the block search engines to access the entire web directory. Let’s say you have some digital products and the download page that only allow your customers to download. You place all the downloadable files into a separate directory. So you can block the whole directory with this bit of code :
or you can just block a particular page from accessing using this line:
Note the asterisk of the first line “User-agent: * ” means disallow all search engine robots. You can specify which search engines robots that you want them not to access your directories or web pages.
For example, if you don’t want Google to access you pages, you can use
“User-agent: Googlebot”
for Yahoo you can use
and for MSN, use:
beware of the capital letters of the robots name and also the directories or webpages name, since they are case SENSITIVES!
when you want to apply?
Web pages like admin directory, private messaging on online discussion forums, digital products page, sign-up page and so on are the common examples on how you would apply the exclusion.
If you are seeking more tips on how you should prevent your digital products page, be sure to check out my latest blog post on 5 Simple Steps to Stop Your Digital Products From Being Stolen
