Importance of Having Your Site Crawled by The Search Engines

Thu, Oct 8, 2009

Link Management, Search Engine Optimization, Work From Home, seo-tips


Crawling rate of your site is something very important for your site at all stages of its development.

When you start a fresh new website, getting your site crawled by the search engines is a priority for you because you don’t stand a chance to be indexed as long as the search engine spiders didn’t came to visit your site.

Once you are indexed, maintaining a good crawling rate is an important factor for your SEO work as it is not only a direct reflection of the importance the search engines give to your content, but also determine how quickly your fresh content will be updated and indexed in the search engines too.

Getting your site crawled for the first time

In order to optimize your chance of getting crawled quickly by the search engines, you have to work on two fronts:

  • On your site structure.
  • On your inbound links.

When it comes to your site structure, the very first thing you want to make sure of is that you don’t have any broken link on your site. The navigation should be very smooth and ideally anyone should be able to access any page of your site within 2 clicks of the mouse only.

Once your website is ready to be crawled by the search engine spiders, it’s now time to create inbound links (or ways for the spiders and bots) to find out about your site.

Search engines’ bots love to find new webpages on internet but hate to be told where they should go. Therefore try to avoid submitting your site for indexing as it’s a waste of time and could take you anywhere from a couple of weeks to a couple of month before you actually get indexed.

Instead focus on building multiple inbound links towards your site.

At this point, it’s important to state that inbound links that are create with the purpose of getting crawled by the search engines are of different nature than the one you potentially may use or be seeking to increase your PageRank score.

While in essence of the perfect link they would all be the same, reality is different and in our situation we are looking for some inbound links with specific characteristics:

  • Your backlink shouldn’t have any “nofollow” attribute or anything alike that would prevent your link to be visited by the search engines.
  • Your backlink should be on a page that is cached daily preferably.

The first point is obvious if the search engines can’t follow your link, your page or site can’t be crawled. As for the second point, the reason why you are seeking a page that is cached daily is simply because this means that the search engines like this page and know that it’s providing fresh quality content regularly.

As the page gets cached each and every day, all the new links on it get captured in the process to, and this is what you are looking for.

Having hundreds of backlinks to your site but having none of them on pages freshly cached is useless for your crawling rate (and linking strategy in general).

My site has been crawled and indexed what’s next?

Ideally, you’d be keeping up the good work and make sure that you are getting more and more of those good backlinks.

Noticed that I didn’t made mention of PageRank for those pages. When it comes to crawling this is not important, the caching rate of the page however is.

Having a backlink from a PR5 page that hasn’t been cached in over 2 weeks time is like not having backlink at all.

This apply to your global linking strategy, a backlink is only effective once it has been crawled.

So, beyond building more and more backlinks, you should also optimize your site structure even more.

Not only based on your topics and categories but also at the on page level. That’s right, you want your pages to be properly structured.

This will make your content friendlier to your visitors but also the search engines.

For instance, Google is now implementing Section Site links to provide quick shortcuts to specific sections and paragraphs of your content based on the structure you created.

There’s no need to say that without a proper structure, you’ll never get a chance to benefit from those Section Links…

Once you manage to have your site crawled and cached frequently, you’ll enjoy great reward for each of your new added content.

For example, my site is crawled 50-70 times daily in average by Google and cached daily or at least as soon as I publish fresh content. For me this translate in having any new post I publish on my blog to be indexed within minutes and reach the primary index in a few days only. Also any change I make on site is very quickly reflected in the search engines.

In Conclusion

Crawling of your site is all about structure and speed. Make sure that your site has multiple inbound links that are cached frequently and you’ll ensure that not only your content will be crawled and indexed, but also that none of your pages will be dropped from the search engines index.

Keep in mind that while getting indexed is crucial, being able to stay indexed is equally important.


Did you enjoy this post? If yes, why not subscribe to my RSS feed or get it via Email. That way you’ll never miss out on new blog posts.

Grab The Bookmarketer For Your Site

Did you miss any of these?:

» Post Ad Purchase:  Intext Link
Print This Post Print This Post

Spread the Word!


Twit This Post!



Internet Home Business(Home)

This post was written by:

Nicolas Prudhon - who has written 8 posts on Work From Home Business Blog. Nicolas Prudhon, PhD is a professional SEO strategist, as well as published author.He's dedicated to provide SEO help to people through his site. Join his latest free SEO training course "21 Days SEO Mastery" now!

Contact the author

  • Hi Nicolas, You have explained well about crawling strategy of Google and it's importance for our sites. My friend was admiring of your course. You also provide knowledge about SEO by guest posting like this. I wanna follow you continuously.
  • Thank you very much for your feedback. I believe that everybody can benefit from sharing information.

    This is why I also like to do guest posting like this. :)
  • Now I know how should I structure properly my links.

    I have decided to subscribe :)
  • Yeah... I am very much agree with your article over here.. The crawling rate of the site and that too of the Google is really very important as this will give a sure shot benefit ... since the more crawling and more visitors will popularise the site dramatically..
  • Fresh post and info really help to get traffic on a site/blog. It can raise our ranking. I thinks that we both are standing at same point.
  • Freshness of queries and information is becoming increasingly popular recently, so we all should leverage on this as far as possible.
  • usedtrucks
    Site crawling is very important thing. A site must be crawl with in 2 or 3 days and new content or information should be post every day b'coz search engine likes fresh content and new informations.
    http://www.oilstruck.com
  • Sir I have read your post and know new thing that is "chache", I did not think about it before when work for build backlink. Thank you for this new information.
  • I'm glad that this post has been able to help you in some ways. ^_^
  • I thinks that Search Engine likes fresh content and new info. So, we have to update site daily.
  • I agree that crawl rate and inbound links are important but if you optimize your anchor text at the same time you can hit two birds with one stone; not only will you continue to have your site indexed but overtime your site will start to rank well also.
  • Although not specified in this article as it's not the main subject, it is of course intended and expected for each inbound link to have a proper anchor text if possible.
  • yeah..very nice articles..search engines like also content rich articles so that the information is not useless.
  • For getting "dofollow" links pointing to your site would you suggest to mainly comment on dofollow blogs or something similar?
  • No, this kind of links as very low value and is of very little help.
  • What about this crawling thing, does google do it to our blogs?
  • Blogs are favored by search engines and thus are the most crawled type of site you can create, this is why it is always strongly recommended to include one in your site if it's not a blog type already.
  • Hi Nicolas, you seem to be a continuous contributor of quality content to this site. I've learned a lot from these posts. i also visit your blog and you have some really very interesting stuff there. This post also very well explains the importance of a site being crawled by SEs. Thanks as I have learnt quiet a bit about getting my site indexed fast and frequently.
blog comments powered by Disqus