How to Optimize WordPress Robots.txt File for Search Engine Crawler (SEO)
Whenever we talk about WordPress SEO, then the WordPress Robots.txt File plays an important role in Search Engine Ranking. With its help, You can prevent index and crawl the private portion of your site by website crawler. However, If you misconfigure the WordPress robots txt file, it may harm your website ranking.
So in this article, I will tell you in details about Robots.txt file. So that you can properly configure robots.txt file for SEO.
By the way, search engine optimization strategies have many ways, but the Robots.txt file is an important part of SEO. The robots.txt file is a small text file found in the root folder of your website that helps to website optimization. But there are many Webmasters or Site owners who are afraid to edit Robots.txt file, but this is not that difficult. Anyone can create and edit their robots.txt file with basic SEO knowledge, and if you are a new blogger, then you are in the right place. In this article, you will know what is the WordPress robots.txt file and How to create robots.txt.
What is WordPress Robots.txt and why should it be used?
The robots.txt file helps to understand to search engine bots that which part of the site you want to index and which part Block When search engine spider or web robots come to your site, they follow the robots configuration file on your site and index the part you want to be indexed in search engines.
In WordPress site, the WordPress robots txt file found in the WordPress root folder of WordPress installation. You can save by editing or create new WordPress Robots.txt file via FTP.
Check your WordPress Robots.txt file
As I mentioned above, the Robots.txt file exists in the root folder of your website. You can also view Robots.txt file by using www.domain.com/robots.txt in the search engines or you can also check the robots txt file by going to Google webmaster tools.
How to create your Robot.txt File for WordPress blog or site?
As I mentioned earlier, this robots.txt file is a general text file. If your website has not a Robots.txt file, you can make a robot.txt file by opening Notepad in your pc or laptop.
Add your XML Sitemap to Robots.txt File
However, if it does not add automatically, then you need to manually add the plugin’s XML Sitemaps link in your robots.txt file.
Here is a WordPress Robots.txt file sample for any domain. In the Sitemap, replace the Sitemap URL with your blog URL.
# disallow all files in these directories
That is all……
You can comment on any questions or suggestions related to this WordPress Robots.txt article and If you like this article, please do not forget to share it on Twitter, Google+ and Facebook…..