A Dropped Sites Checklist.

For anyone who has ever had a website that has dropped out of the search engine results it is a distressing time. Unfortunately the search engines won’t write to you with a personal email explaining why they no longer have a place for you in their index. You need to understand why your site no longer appears in order to get in back in the index.

Test 1. – Check if your url really has departed the search engine index

Go to the search engine where you suspect that your site has disappeared and type in site:www.yoursite.com. This will display every page of your site currently in the search engine index. So either a result will appear or it won’t, immediately letting you know if your website is still in the search engines index.

Understanding the Results.

The search engine will return one entry for every page of your website in it’s index.

If the search engine reports something like this “ Sorry, no information is available for the URL www. yoursite.com”, make sure that your URL is spelt correctly first. If you are still unsure, copy a paragraph from your website and paste it in the search form in this format “ your paragraph here “ and see if your website comes up in the results.

If it also does not appear then your site is not in the index, but no need to panic just yet.

If your website is new and was previously in the index, this could be a search engine data centre problem. This means your website has been cached in one data centre but not in another.

Search engines openly admit that sometimes they miss sites. From experience I can tell you that this has happened with many sites just recently cached for the first time. That is, it will appear first in one data centre and then appear in lots of others. This is why sometimes if you perform a search for a particular keyword one minute and then again a few minutes later, you get a different set of results.

If the result is just www.yoursite.com, with no title or any other descriptive information, then it could be that your domain has expired, the search engine is having trouble reading your pages, your server has crashed for a sustained period of time or you have no content on your index page.

Test 2. – Assessing why your url has been removed

Possible Reasons

Google has set aside some solid reasons of why your page or website may not be included in it’s index. Visit http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=35769 . It is always a good idea to review these and assess which is applicable to your site. You can request re-inclusion by writing to help @ google.com (just make sure if there was a problem you have taken care of it so you are eligible for inclusion).

Aside from these, The Missing Component has assembled the most common reasons why your site has been omitted from the search engines.

Server Crash

First of all go to your domain and check that it works. Once you have established that it does function normally, check your referral reports. Did you have visitors in the last seven days? If numbers are way down or non existent it might be a good indication that your server has been down for some time. A server crash at the wrong time can possibly get your site dropped. It may be time to switch hosting plans if your down time gets too excessive.

This is one of the reasons The Missing Component offers up-time monitoring around the clock for your website.

Internal Errors

Review your error reports thoroughly, some of your pages may be set as forbidden, you might have a whole lot of 404 errors (page not found) and that caused your site to be dropped. Delete the links to the pages that no longer exist and fix all other errors for your site.

Invisible Text

If your site employed invisible (or near invisible text) this is usually classified as spamming. Search engines don’t take kindly to this. Invisible text can also include words that are positioned behind images. Remove it and ask for re-inclusion.

Duplicate Content

It is a common misconception that having the same or similar content to other websites will get your site dropped. It is more likely that a filter or penalty has been applied to the page (not site) using the duplicate content. If duplicate content was a spam trigger then most news sites that syndicate content from other sites would be banned in an instant.

Cloaking

A basic definition for cloaking is showing the search engine one thing and the user another. A common way is to display text based on the IP address of the user and a users ability to accept cookies (which spiders don’t), thus being able to identify if a user is a search engine spider. If you have cloaked some pages of your webpage and your site has been dropped it may be time to pursue another strategy.

Interlinking

If you have multiple domains and you link them all together you run the risk of losing the lot. It is very easy to identify a block of sites that only link to each other. Most search engines see this as attempting to artificially inflate rankings and so this is a good way to get your sites kicked out of the search engines.

Doorway Pages

A doorway page is simply a page designed solely for the purpose of luring people in from the search engines without any useful content. If you are employing these it may be time to add some usefulness to the pages, or delete the existing pages and start again.

Keyword Stuffing

This is basically repeating the same keyword on a page and or similar words in order to improve rankings. I have never seen a page dropped for this reason but have seen offenders disappear into rankings obscurity where they can do no further harm. Fortunately Google (and a few other search engines) usually have a policy of forgiveness so if you remove this from your site you will see your pages come back from the dead (although sometimes you will have to talk to the search engine directly to try and get the penalty lifted).

Inaccurate Title and Descriptions

Inaccurate titles and descriptions for pages basically implies spam. This means that your page title might be about umbrellas but the page is about boots. This is deceptive and while it might be difficult for the search engine spiders to determine this, all it takes is one dissatisfied search engine visitor to report your site and you can say goodbye.

Incorrect use of robots.txt

In your robots.txt file, if you inadvertently block all robots (like in the below example) you will not allow any search engine spider to cache your site

User-agent: *

Disallow: /

If you have a few links to your website and you are disallowing spiders, then it is likely your URL will be included in the index, but with no content cached.

Sneaky Redirects

I have seen plenty of pages that while they do not cloak, they present a page that has been keyword optimized and unreadable for the user, but they employ a mouse over redirect. This means that as soon as your mouse moves over the content they redirect you to their sales letter page (which isn’t optimized for that particular keyword). Again the life span of such a page is very short in the search engines.

Afterthought

Where you link to says a lot about your website, if you link to mostly quality sites that are in a fashion, related to yours, this will speak highly of your website in the search engines perspective. However, if you swap links with other sites who are employing some or all of these tactics you may (or may not) get penalized or even removed from the index. After all, this is the easiest way to remove a bad neighborhood, penalize all of their friends.


If your website has been dropped or penalized, the missing component can help determine why and get you back in the index. Get in contact with us here