Optimize the robots.txt under wordpress

The robots.txt file is a file allowing the crawlers, i.e. to (ro) bots that scour the web to index your site to the search engines.

This file gives bots the necessary indications so that they properly through your website but it is necessary to also indicate what the robots should not index!

WordPress, the robots.txt file is virtual by default, so you can not modify it. For him add instructions, it's simple, just create the file, it will automatically replace the virtual file!

To create it, nothing simpler:

• you open your Notepad
• you insert the following:

User-agent: *.

Disallow: cgi - bin

Disallow: / wp-admin

Disallow: / wp-includes

Disallow: / wp-content/plugins

Disallow: / wp-content/cache

Disallow: / wp-content/themes

Disallow: / WP - login.php

Disallow: / * wp - login.php *.

Disallow: trackback

Disallow: / feed

Disallow: / comments

Disallow: / author

Disallow: /contact/

Disallow: * trackback

Disallow: * / feed

Disallow: * / comments

Disallow: / z/j /.

Disallow: / z/c /.

Disallow: /stats/

Disallow: /dh_

Disallow: /login/

Disallow: /wget/

Disallow: /httpd/

Disallow: vowel

Disallow: participated

Disallow: optimal

Disallow: / c /

Disallow: /j/

Disallow: / * .php$

Disallow: / *? *.

Disallow: *.js$

Disallow: formats$

Disallow: *.css$

Disallow: *.gz$

Disallow: *.wmv$

Disallow: *.cgi$

Disallow: *.xhtml$

Disallow: / *?

Allow: / wp-content/uploads

# alexa archive

User-agent: ia_archiver

Disallow: /.

# allow the robots google image search images


User-agent: Googlebot-Image

Disallow: / wp-includes /.

Allow: / *.

• you save the file under the name of robots with the .txt extension
• and send it to the root of your site
• to check that it is well implemented and he will give good instructions allowing the robots to well to search your site without so far are index pages useless, little relevant that could bring down your SEO, you simply add the address of your site robots.txt

Note: If you use Google XML Sitemap Generator, make sure add the URL of the sitemap to virtual file robots.txt as well = unchecked.