I was just checking out the Little Big Voice site again after my previous post and found that the site is not indexed in Google at all. Not one page!
Thinking that this was a little strange I had a look at their robots.txt file and found the following:
User-agent: *
Disallow: /*?
Disallow:
Disallow: /
Disallow: /*.php
Disallow: /?N=A
Disallow: /cgi/
Disallow: /stats/
Disallow: /templates/
Disallow: /?*
The line in bold above is the culprit! "Disallow: /" tells the search engine crawlers that they should not visit any pages on the site.
Now, what would the reason for this be? I don't think this was done on purpose. It looks to me like a costly error! The robots.txt file was last edited on 9th August 2008 at 17:59:10 so it looks like it has been like this for over a month. You would think that they would notice that their site is no longer indexed! To find out the date a static page was last edited, type the following into your browser's address bar while on the page you're interested in:
javascript:alert(document.lastModified)
Checking in Yahoo shows that the site is still indexed, this highlights the fact that Yahoo is a lot slower than Google at crawling and updating their index (or it may show that Yahoo do not follow the robots.txt protocol!).
Yahoo Site Explorer also shows a number of pages that have been indexed at some point in the past and have now been removed from the site but Yahoo are still showing them in their index:
http://www.littlebigvoice.com/hotel-movenpick-aqaba.htm
http://www.littlebigvoice.com/athens/olympic_palace_hotel.htm
http://www.littlebigvoice.com/travel/34.htm
http://www.littlebigvoice.com/athens/hotel_emmantina.htm
http://www.littlebigvoice.com/resour/sites4_8.htm
http://www.littlebigvoice.com/resour/partners3_31.htm
Most of these are showing a 404, but the real question is what are they? Why were they indexed in the first place. It looks like another site was indexed on the www.littlebigvoice.com domain.
All round some pretty Bad SEO from Little Big Voice once again!
Little Big Voice Have No Idea About Robots.txt Files
Subscribe to:
Post Comments (Atom)
2 comments:
This comment is in addition to my previous comment about your posting.
The robots text file was intentionally written this way as the business model and domain are changing. (The site has not been optimised) I no longer wanted this site indexed. If you look at the source code you will even see each page has: meta name='robots' content='noindex'. I believe this proves my point.
I would appreciate it if you removed the negative comments from your blog, as they are unfounded and incorrect.
Many thanks
Paul, I still cannot believe that you're claiming that the site has not been optimised! You are an SEO and LittleBigVoice are an SEO only company!
Your robots.txt file contains a lot of useless and redundant lines and looks to me like it was written by someone who doesn't know what they are doing.
So, you don't want the site indexed? So much so that you've included the robots meta noindex tag twice!
<meta name="keywords" content="search engine marketing, search engine optimisation, search engine optimization, seo, search marketing, london, little big voice" />
<meta name="description" content="Search Engine Marketing from 'Little Big Voice' – the highly experienced search engine marketing team that gets big results" />
<meta name='robots' content='noindex' />
<meta name="author" content="Little Big Voice" />
<meta name='robots' content='noindex' />
Post a Comment