It’s normal to have 20% of your site unindexed, Google says

You may have noticed that Google search crawlers do not index all the pages on your website. Moreover, you will notice that these pages are being discovered but not being crawled.

You might think that there is something qualitatively wrong with your website or perhaps your server.

While those issues do affect how your site is being crawled, they aren’t always the only culprits. This is especially true for bigger websites with lots of web pages.

According to Google’s John Mueller:

“It’s completely normal that we don’t index everything off of the website.

So if you look at any larger website or any even midsize or smaller website, you’ll see fluctuations in indexing. It’ll go up and down and it’s never going to be the case that we index 100% of everything that’s on a website.

So if you have a hundred pages and (I don’t know) 80 of them are being indexed, then I wouldn’t see that as being a problem that you need to fix. That’s sometimes just how it is for the moment.

And over time, when you get to like 200 pages on your website and we index 180 of them, then that percentage gets a little bit smaller.

But it’s always going to be the case that we don’t index 100% of everything that we know about.”

Page quality vs. Website quality

In addition, John Mueller also shared another interesting insight. According to him, if certain pages are not being indexed, it does not necessarily mean that those pages are of lower quality.

Instead, the issue could be with the overall quality of the website — not just with those pages specifically.

“With regards to the quality, when it comes to understanding the quality of the website, that is something that we take into account quite strongly with regards to crawling and indexing of the rest of the website.

But that’s not something that’s necessarily related to the individual URL.

So if you have five pages that are not indexed at the moment, it’s not that those five pages are the ones we would consider low quality.

It’s more that overall, we consider this website may be to be a little bit lower quality. And, therefore, we won’t go off and index everything on this site.

Because if we don’t have that page indexed, then we’re not really going to know if that’s high quality or low quality.

High percentage of non-indexed pages on smaller websites

As we just learned, it’s okay for large websites to have up to 20% of pages unindexed. However, for smaller websites, you need to be careful as a high percentage of non-indexed pages may represent a qualitative website issue.

According to John:

“If you have a smaller site and you’re seeing a significant part of your pages are not being indexed, then I would take a step back and try to reconsider the overall quality of the website and not focus so much on technical issues for those [individual] pages.”