Wednesday, July 8, 2015

How To Create a Custom Robots.txt file for blogger

1 comment
In the world of SEO robot.txt file is an important matter. At first search engine crawlers scan the robots.txt file before crawling any web page. It tells search engines crawlers that which page have to crawl and which not. Robots.txt file is the instructor of crawler. Today I teach you how to create a search engine friendly robots.txt file for your blogger blog. It is very easy to custom your blogger default robots.txt file. Every blogspot blog has its own default robots.txt file like this.
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://example.blogspot.com/feeds/posts/default?orderby=UPDATED
You have to do little customization to make it search engine friendly. Before going to this job some basic knowledge about txt code in needed. Now I am explaining you about the code:
User-agent: Mediapartners-Google
This code help Google Adsense to serve better ads on your blog. If you don’t use Google Adsense then you can delete the code and if you use Adsense just leave it as it is.
User-agent: *
Thisasterisk (*) sign mean as like mark all. By this sign all crawlersuch as Googlebot,Bingbot etc. can crawling and indexing your blog.
Disallow: /search
Disallow means stop to crawl the page. Disallow:/search indicate that having keyword search after the domain name will be ignored. If you want to disallow any other particular page just copy the page URL remove blog address from it  and place it after the disallow code.
Disallow:post-url.html
Sitemap: http://example.blogspot.com/feeds/posts/default?orderby=UPDATED
This is the default sitemap of any blogspot sites blog. Many SEO specialist says that default sitemap is good for SEO. But it has some limit. By the default sitemap search engine can crawl 25 blog post. If you want to increase the number of page link in your sitemap then change default sitemap with below one. It will crawl and index your first 500(five hundreds) recent posts.
Sitemap: http://example.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500
If your blog’s have more than 500 publish page link then you have to use tow sitemap like bellow:
Sitemap: http://example.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500
Sitemap: http://example.blogspot.com/atom.xml?redirect=false&start-index=500&max-results=1000
Your robots.txt file customization is complete.  Now it is the time to adding custom file in your site. To adding this file Just follow the instruction as given bellow:
1. Log in into your blogspot blog.
2. Click on settings.
3. Click on Search Preferences>Crawlers and indexing ›› Custom robots.txt ›› Edit ›› Yes
4. Past your custom robots.txt file into the box.
5. Now click on save change button.
Congratulation you did it.

1 comment: