Robots.txt

It is great when search engines normally visit your site and index your content but often there are cases when indexing parts of your online content is not what you want. For instance, if you have two versions of a page (one for viewing in the browser and one for printing), you'd rather have the printing version excluded from crawling, otherwise you risk being imposed a duplicate content penalty. Also, if you happen to have sensitive data on your site that you do not want the world to see, you will also prefer that search engines do not index these pages (although in this case the only sure way for not indexing sensitive data is to keep it offline on a separate machine).

What Is Robots.txt?

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.

Site Map

Site Map | What are the benefits of having a sitemap for my website?

1. No page would be left unturned
Going back to the purpose of sitemaps, having one would mean faster and easier tracking and crawling of spiders all over your site. As a result, search engines would surely get to the view all the pages of your site and not just the pages containing random keywords.

2. Easier navigation for site visitors
Once a web user has accessed your sitemap, they need not go back to the search engine page to look for what they need. If what they are looking for is in your site, then they would have an easier and faster way of locating it.

3. Potential advertising value
If it so happens that a relevant product or service company reaches your site, then it would be easier for them to see how best they can position themselves in the different pages of your site as a paid page advertisement.

4. Encourage greater traffic to your site
If your company website has a sitemap then potential buyers would have an easier time in accessing your latest products and services. Moreover, they would not miss out on any product that might be off future interest to them since the sitemap would display all information found the site.