Many person does not know importance of robots.txt file and how to use robots.txt file in wordpress. robots.txt file is important as per search engine. The robots.txt file is very important as per search engine and SEO prospective.
how to use robots.txt file in wordpress
What is robots.txt file?
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.
In wordpress you should put robots.txt file. you can use following code in robots.txt file and put that file under your ROOT directory( wordpress main folder)
User-agent: * Allow: / Disallow: /*?s= Disallow: /wp-admin/* Disallow: /wp-content/* Disallow: /wp-includes/* Disallow: /wp-content/cache Disallow: /wp-content/themes/*<code></code> Disallow: /wp-content/plugins/*
Important Note: if you are using the QA or testing enviourment then put following lines in your theme header.php file. You can put the domain name condition also.
<?php
if($SERVER)['HTTP_HOST']=='Testing domain name') {
?>
<?php } ?>
You should not disallow the images and comments also because that is more important for SEO purpose. I recommend to use always good permalink structure for your wordpress installation.
