• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • Skip to footer

JUST BRIGHT ME

Makes WordPress and SEO Easier

  • Start Blog
  • Blogging
  • SEO
  • WordPress Guide
Home » SEO » How to Create a Perfect Robots.txt file for SEO

How to Create a Perfect Robots.txt file for SEO

Written by AMAN SINGH

The robots.txt file is a small text file that is found in the root folder of your site. It tells search engine bots about which part of the site to crawl and which part is not.

If you make a slight mistake while editing/customizing it, then search engine bots will stop crawling and indexing your site and your site will not appear in the search result.

In this article, I will tell you what is robots.txt file is and how to create a Perfect robots.txt file for SEO.

  • Why do you need robots.txt
  • Where is the Robots.txt Found on a Site?
  • Robots.txt File’s Basic Format
  • Preventing All Web Crawlers from Indexing Website
  • Allow All Web Crawlers to Index All Content
  • Blocking a Specific Folder for Specific Web Crawlers
  • Preventing a Specific Page (Thank You Page) from Index
  • Adding a Sitemap to a Robots.txt File

Why do you need robots.txt

When search engine bots come to websites, they follow the robots file and crawl the content. But your site will not have a robots.txt file, then search engine bots or web crawler will start indexing and crawling all the contents of your website which you do not want to index.

Web crawler searches the robots file before crawling any website. When they do not get any instructions from robots.txt file, they start crawling all the contents of the website. But if there are any instructions, they crawl it by following it.

Therefore, robots.txt file is required for these reasons. If we do not give Instructions to search engine Bots through this file, then they index our entire site. Also, index some of the data that you did not want to index.

Benefits of Robots.txt File

  • Tells the search engine about which part of the site to crawl and which part is not.
  • Prevent any particular file, folder, image, pdf etc from being indexed in the search engine.
  • Sometimes search engine crawlers crawl your site like a hungry lion, which greatly influences your site performance. But you can get rid of this problem by adding crawl-delay to your robots file. However, Googlebot does not follow this command. But you can set Crawl rate in Google Search Console. This prevents your server from being overloaded.
  • You can make the entire section of any website private.
  • You can stop the Internal search results page from showing in SERPs.
  • You can improve your website SEO by blocking low qaulity pages.

Where is the Robots.txt Found on a Site?

If you are a WordPress user, then it remains in the root folder of your site. If this file is not available in this location, the search engine bot starts indexing your entire website. Because search engine bots do not search your entire website for a robots.txt file.

If you do not know whether your site has a robots.txt file or not? Then in the search engine address bar, you just have to type this – example.com/robots.txt

A text page will open in front of you as you can see in the screenshot.

What is Robots.txt File How to Create a Perfect Robots.txt file for SEO

This is Robots.txt file of InHindiHelp. If you do not see any such text page, then you have to create a robots.txt file for your site.

In addition, you can check it by going to Google Search Console tools.

How to Create the Perfect Robots.txt File for SEO

Here is a guide on how to create a Perfect Robots.txt file for SEO…

Robots.txt File’s Basic Format

The basic format of the robots.txt file is very simple and it looks like this,

User-agent: [user-agent name]
Disallow: [URL or page that you do not want to crawl]

These two commands are considered as a complete robots.txt file. However, a robots file can contain many commands of user agents and directives (disallows, allow, crawl-delay etc).

  • User-agent: are search engines’ crawlers/bots. If you want to give the same instruction to all search engine bots, use the * symbol after User-agent: like this User-agent: *
  • Disallow: This prevents files and directories from being indexed in search engines.
  • Allow: This allows search engines bots to crawl and index your content.
  • Crawl-delay: How many seconds should the bot wait before loading and crawling the page contents.

Preventing All Web Crawlers from Indexing Website

User-agent: * 
Disallow: /

Using this command in the robots.txt file, you can prevent all web crawlers/bots from crawling the website.

Allow All Web Crawlers to Index All Content

User-agent: * 
Disallow:

In the robots.txt file, this command allows all search engine bots to crawl all the pages of your site.

Blocking a Specific Folder for Specific Web Crawlers

User-agent: Googlebot 
Disallow: /example-subfolder/

This command prevents only Google’s crawler to crawl example-subfolder. But if you want to block all Crawlers, then your robots file should be like this.

User-agent: *
Disallow: /example-subfolder/

Preventing a Specific Page (Thank You Page) from Index

User-agent: *
Disallow: /page URL (Thank You Page)

This will prevent all crawlers from crawling your page URL. But if you want to block specific Crawlers, then you write it like this.

User-agent: Bingbot
Disallow: /page URL

This command will only prevent Bingbot from crawling your page URL.

Adding a Sitemap to a Robots.txt File

Sitemap: https://www.example.com/sitemap.xml

You can add your sitemap anywhere in robots.txt – at the top or just below. Here is a guide –How to add Sitemap to Robots.txt File and Why is it important?

Find this article helpful? Don’t forget to share!

About AMAN SINGH

AMAN SINGH, who owns JustBrightMe, is a full-time blogger and loves sharing content on WordPress, SEO & Blogging Tips.

Reader Interactions

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Primary Sidebar

Blog’s Popular Post

How To Increase Website Traffic in 2021 (54 Working Methods)

The Ultimate On-Page SEO Techniques

Best Autoptimize Settings Updated

Fix Submitted URL marked ‘noindex’ in Google Search Console Tool

How to Protect Your Website from Negative SEO

76 Best Free Responsive Blogger Templates

Rank Math SEO VS Yoast SEO Comparision

650+ High Domain Authority Sites to Get Dofollow Backlinks

How to Transfer SBI Account from One Branch to Another Branch

16 Free Ways to Promote Your Website

Blog’s Recent Post

How to Increase Domain Authority 2021

How to Change Name in Aadhar Card Online

How To Get Free Domain For Your Website

20 Ways How to Speed Up WordPress Site

How to Change Mobile Number in SBI

How to Unlock a PUK Blocked Sim Card

How to Remove Image Background Online in One Click

How to Block SBI ATM Card

How to Disable Windows Defender in Windows 10

How to Make Windows 10 Faster

Footer

SEO Guides

  • Learn SEO: A Beginner’s Guide to SEO
  • 40 Google Ranking Factors
  • How to Create Quality Backlinks
  • How to Improve Google Ranking
  • 23 Best On-Page SEO Techniques
  • How to Keyword Research for SEO
  • 29 Most Common SEO Mistakes to Avoid
  • How to Increase Domain Authority
  • Image SEO Tips You Need to Know
  • Keyword Density in SEO
  • Internal Linking for SEO
  • How to Protect Website from Negative SEO

WordPress Guides

  • How to Create a Blog on WordPress
  • Top 25 Must-have WordPress Plugins on Every Blog
  • 34 Best SEO Friendly WordPress Themes
  • 42 Most Common WordPress Mistakes to Avoid
  • How to Speed Up WordPress Site
  • 19 Essential Settings After Installing WordPress
  • 57 Best Responsive WordPress Themes
  • Top 41 Fast Loading WordPress Themes
  • How to Scan WordPress Site for Malware

Useful Guide

  • How To Increase Website Traffic (54 Working Methods)
  • A Complete Guide to Google Search Console
  • 20 Ways How to Reduce Bounce Rate
  • 46 Killer Blogging Tips for Beginners
  • How to Become a Successful Blogger
  • 50 Best SEO Tools (Ultimate List)
  • 16 Free Ways to Promote Your Website
  • 11 Best Blogging Platforms to Launch a Blog in 2020
  • How to Update Old Blog Posts (and Why You Should)

© 2021 · JUST BRIGHT ME

  • About us
  • Privacy Policy
  • Sitemap