A Better Way To Keep Pages From Being Indexed

Today I went through some of the indexed pages of one of my older sites and found that their were pages still indexed that I blocked years ago via a robots.txt file.  I guess at the time I did this I had assumed once I block these pages in robots.txt, that was all I needed to do to get them removed from the index.  Rookie mistake.

Basically all the robots.txt does is give you the ability to block Google and other search engines from crawling certain pages on your site.  This doesn’t mean they’ll remove anything from their index if it’s already been indexed.  So if you have pages already indexed, using meta noindex is the way to get them removed.  Also, once the noindex meta tag is in place on the page you want removed, remove the page from the robots.txt file so Google can crawl it -> find the noindex tag -> and then remove.

Here’s a small script I put together for the WordPress header.php file.  I also added it to this blog so you can see it in action.  There’s probably a plugin or a better way of coding this as I’m no expert with PHP.  If you have a better way, please feel free to comment.

<?php
$url = ‘http://’ . $_SERVER['SERVER_NAME'] . $_SERVER['REQUEST_URI'];

// adds meta noindex to dynamic urls such as site.com?duplicatecontent

if (false !== strpos($url,’?')) {
echo ‘<meta name=”robots” content=”noindex”>’;
}

// adds meta noindex to my 2008 archives – > allinone seo plugin can do this also

elseif (false !== strpos($url,’/2008/’)) {
echo ‘<meta name=”robots” content=”noindex”>’;
}

?>

 

Robots.txt and SEO Discussed:

http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449
http://tools.seobook.com/robots-txt/
http://www.seomoz.org/q/timely-use-of-robots-txt-and-meta-noindex

About author:

One Response to “A Better Way To Keep Pages From Being Indexed”

  1. Janet says:

    Its a very nice post. I will certainly use these tips. Is it possible to re-index a website after being de-index by Google?

Leave a Reply

 

Slider by webdesign

Powered by Google Talk Widget

Sorry, you can not to browse this website.

Because you are using an outdated version of MS Internet Explorer. For a better experience using websites, please upgrade to a modern web browser.

Mozilla Firefox Microsoft Internet Explorer Apple Safari Google Chrome