Optimize the robots.txt under wordpress


robots-wordpress

The robots.txt file is a file allowing the crawlers, i.e. to (ro) bots that scour the web to index your site to the search engines.

This file gives bots the necessary indications so that they properly through your website but it is necessary to also indicate what the robots should not index!

WordPress, the robots.txt file is virtual by default, so you can not modify it. For him add instructions, it's simple, just create the file, it will automatically replace the virtual file!

To create it, nothing simpler:

  • you open your Notepad
  • you insert the following:
User-agent: *.
Disallow: cgi - bin
Disallow: / wp-admin
Disallow: / wp-includes
Disallow: / wp-content/plugins
Disallow: / wp-content/cache
Disallow: / wp-content/themes
Disallow: / WP - login.php
Disallow: / * wp - login.php *.
Disallow: trackback
Disallow: / feed
Disallow: / comments
Disallow: / author
Disallow: /contact/
Disallow: * trackback
Disallow: * / feed
Disallow: * / comments
Disallow: / z/j /.
Disallow: / z/c /.
Disallow: /stats/
Disallow: /dh_
Disallow: /login/
Disallow: /wget/
Disallow: /httpd/
Disallow: vowel
Disallow: participated
Disallow: optimal
Disallow: / c /
Disallow: /j/
Disallow: / * .php$
Disallow: / *? *.
Disallow: *.js$
Disallow: formats$
Disallow: *.css$
Disallow: *.gz$
Disallow: *.wmv$
Disallow: *.cgi$
Disallow: *.xhtml$
Disallow: / *?
Allow: / wp-content/uploads
# alexa archive
User-agent: ia_archiver
Disallow: /.
# allow the robots google image search images
User-agent: Googlebot-Image
Disallow: / wp-includes /.
Allow: / *.
  • you save the file under the name of robots with the .txt extension
  • and send it to the root of your site
  • to check that it is well implemented and he will give good instructions allowing the robots to well to search your site without so far are index pages useless, little relevant that could bring down your SEO, you simply add the address of your site robots.txt

Note: If you use Google XML Sitemap Generator, make sure add the URL of the sitemap to virtual file robots.txt as well = unchecked.

If you have found a spelling error, please, notify us by selecting that text and pressing Ctrl+Enter.

Leave a comment

Your email address will not be published. Required fields are marked *