Want to control which page of your website should be crawled and which not on blogger? This is just possible with the robots.txt file of your website.
Robots.txt is a simple but very important file of the page which controls crawling access of the bots or crawlers.
In this Blogger tutorial, I will explain the robots.txt file in detail like what is robots.txt file, how to generate or create it, and then how to submit it. After this tutorial, you don’t need any robots.txt generator for generating your robots.txt file for your blogger website.
So First we need to know what is robots.txt file?
What is a Robots.txt file?
Robots.txt is a simple text file (.txt) that is located in the root directory of your website and it tells search engine crawlers or bots which page or file you want (or don’t want) to crawl.
You can check your website robots.txt file just by adding the /robots.txt slug after your website URL like this:
This is My website: https://www.netwem.com/
then my robots.txt file will be located at: https://www.netwem.com/robots.txt
You Can Allow or disallow different Crawling access for a particular bot like Googlebot, yahoo bot, etc. So you might be curious now about what happens if I don’t use the robots.txt file on my website?
But is it really a necessary file to use or what happens if we are not using the robots.txt file on our website?
Is the Robots.txt file necessary?
This Answer simply depends on your requirements and needs but robots.txt is not a must-have file for your website.
Confused about when we should use the robots.txt file and when we don’t need the robots.txt file especially if our website is hosted on the blogger platform.
let’s clear your confusion…
When we must have a robots.txt file?
- You want to block specific pages and files from the search engine.
- You want that only Particular bots can access or crawl your website.
When we don’t need a robots.txt file?
- When you want to index everything available on your website.
- You don’t want to block any page or file blocked from the search engine.
How to create a Robots.txt file for blogger?
Creating a robots.txt file is really easy but many bloggers use a robots.txt generator to generate robots.txt files for blogger websites.
But I will show you how you can create a robots.txt file for your website by yourself.
So First look at the format of the robots.txt file.
User-agent: *
Disallow: /
Allow: /
This is the default format of any robots.txt file and you need to add rules for crawlers in the same manner.
Here, User-agent refers to the specific bot, and disallow refers to the content that we want to block from crawling.
Asterisk (*) Refers to all of the bots and crawlers.
For Example;
If you want to block your about and contact page for Googlebot you will use:
User-agent: Googlebot
Disallow: /about
Disallow: /contact
Allow: /
Let’s have one more example:
Suppose You want to block label archives from indexing but you want to allow particular label archives to index so your robots.txt will be like this: (Suppose you want to index only 1 label named API)
User-agent: *
Disallow: /search/label
Allow: /search/label/API
I hope you got it.
Let’s move on next part “What is the perfect robots.txt file for a blogger website”
Perfect Robots.Txt for Blogger
In Blogger We don’t have a login page, user details, and dashboard pages like WP and others to block from crawlers.
but we should block search queries from crawlers.
like if someone searches the “Median” keyword in our website's search box so blogger will generate this type of URL: https://www.website.com/search?q=median
but we don’t want to index these types of links in search engine.
So after disallowing these search query links your Robots.txt will look like this:
User-agent: *
Disallow: /search?q=
Allow: /
Sitemap: https://www.yourwebsite.com/sitemap.xml
Sitemap: https://www.yourwebsite.com/sitemap-pages.xml
Sitemap: https://www.yourwebsite.com/atom.xml?redirect=false&start-index=1&max-results=500
Replace www.yourwebsite.com with your Website name.
Just Copy this Code and paste it into your robots.txt section.
To open the robots.txt section:-
- Go to Blogger Dashboard.
- Click on setting > scroll down.
- Find Custom Robots.Txt and enable them.
- Paste the above code in that section.
Important links and resources
Robots.txt Testing Tool by Google
Robots.txt Google Developer blog
Wrapping up
Today I shared a full guide for creating robots.txt to uploading it on your website for blogger. I hope you find this helpful.
Drop a like and share with your blogger friend because-
Great Satisfaction Comes from sharing with others.
Source:
www.hob.com
