Today Google threw one of my website homepages in the basement from #1 positioning on google for several terms. Once I experienced this I started thinking to myself some of the changes I’ve recently made. Nothing I did came to mind that could have caused this. Then I checked for php injections and found no problems there. Then I gave it some more thought and remembered how Google recently indexed that particular website’s homepage with https instead of http. After some research I found that this does in fact cause a duplicate content penalty.
This is what I did to hopefully fix this issue. I’ll follow up on the status within 24-48 hours.
Review updates at the bottom of the article.
First I submitted this into my htaccess file
RewriteEngine on
rewritecond %{SERVER_PORT} ^443$
RewriteRule ^robots\.txt$ robots_ssl.txt [L]
This little snippet I found online will serve up two versions of your
robots.txt file; an http and https version.
So the http version shows the robots.txt file you currently have.
Next you need to create a robots_ssl.txt file and upload it to your root file.
Contents of the robots_ssl.txt file:
User-agent: *
Disallow: /
What this will do is block all https: urls from indexing.
No more duplicate content penalty? Did it work? Update coming soon…
Updates
Same day 4 hours Later:
Appears my rank is back but https is still showing on search engine listing. However I just checked my google webmaster tools to see if my robots.txt file was recently crawled and it appears the last download was just a couple of hours ago. Ill update this post again once I see the https come off in the search results.
Next Day
Well I clicked online today and guess what, as expected the https is gone and the rank is where it was prior to the shake up. I even moved up 2 notches on a competitive phrase (which could of been for any other reason). A good start to a tuesday morning.
