Google highlights ways it removes websites from the search index

Ranking high on the search engine results pages (SERPs) for competitive keywords is no easy task. It requires a lengthy, concentrated effort on the back of ample time and resources.

However, despite all the efforts and success, it is possible to lose all those rankings and have your website de-index from the search engine.

There are many reasons why Google may decide to de-index your website. But a recent conversation with Google’s John Mueller revealed that there are also two different ways Google deindex a website from the search index.

Let’s see what John said.

During the SEO Office-hours hangouts, one person asked John Mueller the following question:

“I own a site, and it was ranking good before the 23rd of March. I upgraded from Yoast SEO free to premium. After that, the site got deindexed from Google, and we lost all our keywords.”

To provide more context, the person also mentioned that they did gain back rankings for a few keywords, but they would lose the rankings again in a matter of hours.

They also mentioned that they had double-checked robots.txt to ensure it is set up as it should.  They have also ensured that there are no manual penalties.

John Mueller first clarified that the upgrade to the premium version of the Yoast SEO plugin has nothing to do with it.

“I don’t know… it sounds kind of tricky… I would say offhand it probably doesn’t have to do with the updating of your plugin.”

The next part was more interesting as John Mueller shared some insights into how the deindexing process works from Google’s perspective.

He specifically highlighted the slow and long deindexing process in which Google gradually deindexes parts of a website that it no longer considers relevant.

“ … it could very well be a technical issue somewhere.

Because usually, when we reduce the indexing of a site, when we say we don’t need to have as many URLs indexed from a website, we tend to keep the URLs that are more relevant for that site and that tends to be something that happens over, I don’t know, this longer period of time where it like slowly changes the indexing.”

This was a valuable bit of information from John Mueller but not very relevant to the person in their particular scenario, as they were experiencing a total website deindex (not a partial or gradual one).

To that, Mueller responded:

“So if you’re seeing something where like the whole site disappears from indexing, it almost sounds like something that might be related to a technical issue… something along those lines.”

Our suggestions

If you have been noticing something similar, you should first check your robots.txt to ensure it’s not blocking pages. 

Next, double-check if you still have an active sitemap and ensure there are no crawling issues.

Third, do a full SEO audit and shortlist any technical and website quality issues. Those issues will most likely be the culprit.

Note that there were multiple reports of unconfirmed Google algorithm updates in March. So if your loss in keyword rankings coincide with that timeline, those algorithm updates may also be responsible.