An interesting post from Google Web Master Central blog about Duplicate Content. The main point of the message is that Google has the way of identifying different versions of the same content page.
Duplicate content page means
1. Two identical pages, one for printer friendly version and the other for users.
2. Reproducing articles from another website
3. Domains pointing with and without the file name extension such as: www.grrajeshkumar.com and www.grrajeshkumar.com/index.php
You can tell Google not to index the duplicate page by having a meta tag inside the HEAD element:
meta content=”NOINDEX, NOFOLLOW” name=”ROBOTS”
Alternatively, you can use robots.txt 
file to inform search engines NOT to index the page.
Now you need not worry about getting banned from Google for duplicate content unless and until your intention is to decieve the search engine and get better rankings.
As long as Google doesn’t smell anything fishy, it will index only one version of the pages. So, you can have a peaceful sleep - not really worrying about robots.txt files.
Leave a Reply