Crawl frequency refers to how often a website is crawled by search engine spiders, otherwise known as web crawlers. One example of a search engine spider is Googlebot.

Basically, Googlebot is responsible for visiting websites and storing data from new pages such that these pages can be indexed. Google’s index is essentially a massive and expanding library of information, and it’s the reason the search engine can produce hundreds of millions of results for a single search query. By constantly updating its index, Google invests in its user experience (UX) and makes its search results as relevant as possible.

Why does crawl frequency matter?

A website owner asking why crawl frequency matters is akin to an unemployed person asking why job applications matter.

The only way to get your pages and content into the organic search results is to get them indexed, and the only way to get them indexed is to have your site crawled by Googlebot. In other words, if your site doesn’t get crawled, you have no chance of driving organic traffic. No matter how great your pay-per-click (PPC) ads are, a complete lack of organic traffic is devastating.

So, if your business depends on organic traffic to drive qualified leads, you want to make sure your site gets crawled as frequently as possible.

What factors determine crawl frequency?

First and foremost, the frequency with which you update your site—add new pages, add new content, optimize existing pages and content—affects how often its crawled. Googlebot crawls new stuff in order to keep the index as fresh as possible. So, as long as you’re regularly updating and optimizing your site, it will get crawled.

Popularity and domain authority matter, too. Google wants to incentivize greatness among the sites it indexes—otherwise, there would be no reason to use Google. Therefore, sites that drive a lot of traffic and create truly engaging content will get crawled more often than others.

Domain Authority is the extent to which a site is considered an authoritative (reputable) resource. A huge factor behind domain authority is backlinks—links from other sites to yours. Basically, every time an outside site links to one of your pages or to a piece of your content, that site is yelling at Google, “These folks are smart and helpful and awesome!” As we have discussed, Google rewards these authoritative websites.

What can I do?

It’s easy: be prolific, popular, and authoritative.

Just kidding. But, at the same time, we’re really not kidding. Let’s talk tactics.

The good news is that popularity and domain authority pretty much go hand in hand. If you produce content that’s worth reading or watching, you produce content that’s worth sharing and linking. A huge part of creating high-quality, link-earning content is making sure it’s not thin. Thin content is uninformative content—300-word blog posts, for example.

Can you imagine if our big, easy guide to keyword research for business was only a couple paragraphs long? There’s just no way that would be engaging or helpful enough to generate buzz.

To review: create awesome, informative content on a regular basis → earn backlinks from reputable outside sources → build popularity and domain authority at the same time → get crawled more frequently → remain SERP dominant until the end of times.