• Skip to main content
  • Skip to secondary menu
  • Skip to primary sidebar

JustBrightMe

Be Smart and Don’t Stop Learning

  • Best Apps
  • How To
  • Banking
  • Make Money
  • Story
Home » SEO » How to Create a Perfect Robots.txt file for SEO

How to Create a Perfect Robots.txt file for SEO

March 17, 2019 by AMAN SINGH 31 Comments

The robots.txt file is a small text file that is found in the root folder of your site. It tells search engine bots about which part of the site to crawl and which part is not.

If you make a slight mistake while editing/customizing it, then search engine bots will stop crawling and indexing your site and your site will not appear in the search result.

In this article, I will tell you what is robots.txt file is and how to create a Perfect robots.txt file for SEO.

Brief Descriptions of The Content

  • Why do you need robots.txt
  • Where is the Robots.txt Found on a Site?
  • Robots.txt File’s Basic Format
  • Preventing All Web Crawlers from Indexing Website
  • Allow All Web Crawlers to Index All Content
  • Blocking a Specific Folder for Specific Web Crawlers
  • Preventing a Specific Page (Thank You Page) from Index
  • Adding a Sitemap to a Robots.txt File

Why do you need robots.txt

When search engine bots come to websites, they follow the robots file and crawl the content. But your site will not have a robots.txt file, then search engine bots or web crawler will start indexing and crawling all the contents of your website which you do not want to index.

Web crawler searches the robots file before crawling any website. When they do not get any instructions from robots.txt file, they start crawling all the contents of the website. But if there are any instructions, they crawl it by following it.

Therefore, robots.txt file is required for these reasons. If we do not give Instructions to search engine Bots through this file, then they index our entire site. Also, index some of the data that you did not want to index.

Benefits of Robots.txt File

  • Tells the search engine about which part of the site to crawl and which part is not.
  • Prevent any particular file, folder, image, pdf etc from being indexed in the search engine.
  • Sometimes search engine crawlers crawl your site like a hungry lion, which greatly influences your site performance. But you can get rid of this problem by adding crawl-delay to your robots file. However, Googlebot does not follow this command. But you can set Crawl rate in Google Search Console. This prevents your server from being overloaded.
  • You can make the entire section of any website private.
  • You can stop the Internal search results page from showing in SERPs.
  • You can improve your website SEO by blocking low qaulity pages.

Where is the Robots.txt Found on a Site?

If you are a WordPress user, then it remains in the root folder of your site. If this file is not available in this location, the search engine bot starts indexing your entire website. Because search engine bots do not search your entire website for a robots.txt file.

If you do not know whether your site has a robots.txt file or not? Then in the search engine address bar, you just have to type this – example.com/robots.txt

A text page will open in front of you as you can see in the screenshot.

What is Robots.txt File How to Create a Perfect Robots.txt file for SEO

This is Robots.txt file of InHindiHelp. If you do not see any such text page, then you have to create a robots.txt file for your site.

In addition, you can check it by going to Google Search Console tools.

How to Create the Perfect Robots.txt File for SEO

Here is a guide on how to create a Perfect Robots.txt file for SEO…

Robots.txt File’s Basic Format

The basic format of the robots.txt file is very simple and it looks like this,

User-agent: [user-agent name]
Disallow: [URL or page that you do not want to crawl]

These two commands are considered as a complete robots.txt file. However, a robots file can contain many commands of user agents and directives (disallows, allow, crawl-delay etc).

  • User-agent: are search engines’ crawlers/bots. If you want to give the same instruction to all search engine bots, use the * symbol after User-agent: like this User-agent: *
  • Disallow: This prevents files and directories from being indexed in search engines.
  • Allow: This allows search engines bots to crawl and index your content.
  • Crawl-delay: How many seconds should the bot wait before loading and crawling the page contents.

Preventing All Web Crawlers from Indexing Website

User-agent: *
Disallow: /

Using this command in the robots.txt file, you can prevent all web crawlers/bots from crawling the website.

Allow All Web Crawlers to Index All Content

User-agent: *
Disallow:

In the robots.txt file, this command allows all search engine bots to crawl all the pages of your site.

Blocking a Specific Folder for Specific Web Crawlers

User-agent: Googlebot
Disallow: /example-subfolder/

This command prevents only Google’s crawler to crawl example-subfolder. But if you want to block all Crawlers, then your robots file should be like this.

User-agent: *
Disallow: /example-subfolder/

Preventing a Specific Page (Thank You Page) from Index

User-agent: *
Disallow: /page URL (Thank You Page)

This will prevent all crawlers from crawling your page URL. But if you want to block specific Crawlers, then you write it like this.

User-agent: Bingbot
Disallow: /page URL

This command will only prevent Bingbot from crawling your page URL.

Adding a Sitemap to a Robots.txt File

Sitemap: https://www.example.com/sitemap.xml

You can add your sitemap anywhere in robots.txt – at the top or just below. Here is a guide –How to add Sitemap to Robots.txt File and Why is it important?

Find this article helpful? Don’t forget to share!

Related Posts

Filed Under: SEO Tagged With: Beginners Guide, Robots.txt, SEO

Reader Interactions

Comments

  1. rovin singh says

    March 27, 2021 at 3:57 pm

    really a very useful article for robots.txt

    Reply
  2. munesh says

    March 28, 2021 at 11:34 am

    Brother, you have written a very amazing post, your way of explaining is also very good, keep writing like this, thank you

    Reply
  3. Aditi says

    April 6, 2021 at 1:50 am

    Thanks for this really helpful information

    Reply
  4. Prakash Das says

    April 17, 2021 at 5:23 pm

    Yeah it great information.

    Reply
  5. kaushik kumar says

    July 11, 2021 at 2:58 pm

    Thanks for the great post.

    Reply
    • Aman says

      September 3, 2021 at 3:45 pm

      Thank you keep visiting

      Reply
  6. Blogsamachar says

    August 8, 2021 at 5:24 pm

    very usefull information.

    Reply
    • Aman says

      August 14, 2021 at 3:43 pm

      Thank you keep visiting

      Reply
  7. Adil Khan says

    August 26, 2021 at 8:59 am

    Best and best information thank you so much Aman Singh

    Reply
    • Aman says

      September 3, 2021 at 3:44 pm

      Thank you keep visiting

      Reply
  8. sahil says

    September 29, 2021 at 4:00 pm

    Thanks for sharing this helpful & wonderful post. i really appreciate your hard work. this is very useful & informative for me.
    thanks for sharing with us. thanks a lot.

    Reply
  9. Krishna says

    October 7, 2021 at 8:58 am

    Very helpful article sir. Thanks for wonderful article

    Reply
  10. Sonu Singh says

    October 21, 2021 at 12:55 am

    Thanks for sharing problem-solving valuable articles for Robots.txt. I have solved my long-time issue robots.txt error.

    Reply
  11. Sunny says

    October 23, 2021 at 10:33 pm

    I was searching for this content from longtime, and finally, I got it

    Reply
  12. Deepak says

    October 25, 2021 at 1:23 pm

    Thanks for sharing this helpful & wonderful post. i really appreciate your hard work. this is very useful & informative for me.
    thanks for sharing with us. thanks a lot.

    Reply
  13. axis m says

    December 21, 2021 at 5:08 pm

    very informative blog really nice

    Reply
    • Aman says

      December 23, 2021 at 7:27 pm

      Thank you keep visiting

      Reply
  14. Deepak says

    March 13, 2022 at 2:39 pm

    very informative blog really nice

    Reply
  15. Deepak says

    March 13, 2022 at 2:40 pm

    Very helpful article sir. Thanks for wonderful article

    Reply
  16. Ranjeet Singh says

    April 22, 2022 at 2:08 pm

    Good Job Sir G. All About Explain Robot.txt File.

    Reply
  17. Deepak says

    May 23, 2022 at 8:34 am

    Nice information
    https://www.inhindi.co.in

    Reply
  18. Gaurav Nathiyal says

    December 7, 2022 at 8:09 pm

    I implemented just as you have said in this post and it worked for me! Thanks alot for this.

    Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Primary Sidebar

Recent Published

How to Change Photo Background Online 2023

How to Change Instagram Password 2023

How to Reset Instagram Password If Forgotten 2023

How to Change Instagram Username 2023

How to Check Instagram Password 2023

How to Earn Money From Meesho App 2023

What is Dailyhunt and how to earn money from it 2023

What Is Google Task Mate and How to Earn Money From It

11 Best Scratch Earning Apps: Scratch and Earn Money 2023

23 Ways How to Earn Money in India as a Student 2023

© 2015–2023 · JustBrightMe | All Rights Reserved · About us · Privacy Policy · Sitemap