Girl Geekette

   

SlurpConfirm404

Another of my daily bad habits (yes, amonsgt many) is the continious monitering of my logs. At any given time I am watching the logs of my websites. Beleive it or not, sometimes this pays off! And other times, it brings more questions to mind which.. yes, I am going to search, read and learn until my questions are satisfied.

This time, I happen to be watching my logs and noticed that Yahoo’s Slurp engine was searching for pages that did not exist on my site. I had to stop and think about it for a minute, but I realized that the pages it was searching for were not only ones that did not exist, but they never had existed!

Now, through the various versions of my sites, of course pages have moved. And when this happens, search engines still might be looking for the old and outdated ones. But in this case, /SlurpConfirm404/Schultz/HSchorus.htm (see the image above) brought a big HUH?? to my lips.

Well, what in the wolrd is SlurpConfirm404? I do not have anything like that for a directory. Then it crossed my mind that my site had been compromised, but that was not the case, It was Yahoo’s own engine looking for a SlurpConfirm404 directory.

Well, naturally the next thing to do was starting to search for this. And the first place to look - where the entry said to look for. 

Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp

After a bit of searching, I did find this article

http://help.yahoo.com/l/us/yahoo/search/webcrawler/slurp-10.html

which is approriately titled: Why is your crawler asking for strange URLs that have never existed on my site? 

Some web servers send a site navigation page or other response page with a “HTTP 200 OK” response instead of a “HTTP 404 Not Found” result for page-not-found conditions. To check on web server handling of page-not-found conditions, Yahoo! Slurp occasionally sends deliberately odd URLs built from random words to sites from which no 404 results have been seen. These URLs are built intentionally to not match any actual content at the site. We save information on the web server response to requests for non-existent pages so we can correctly recognize and remove obsolete URLs in our search database.

A Yahoo! Slurp check for 404 results from a web server consists of requests for up to 10 such URLs. The check for 404 behavior is not a normal part of Yahoo! Slurp site refresh, so such requests are rare.
 

So, long story short, yahoo purposely makes their search engine crawl and look for non existant sites (Around 10 urls at a time) To purposely see how a website will respond to a 404 not found. Accordingly, it might be rare, but I just happen to catch the instance. And, it definitely caught my eye. But, for now, mystery solved!

And, rumors from other websites have it that google and other search engines do this as well. Have you encountered anything like this ?

Related posts:

1. Psychology of a Search and Reasons for Higher Listings Plain and simple: Whether you are a business who has...
2. Google Chrome Downloadable [caption id="attachment_264" align="alignleft" width="150" caption="Chrome and MySpace Example"][/caption] Google now...
3. SEO Category After reading my logs and noticing I have had a...
4. Googlebot Meta Tags Besides the keyword and description meta tags being the most...
5. Spyware and Adware: Introduction Introduction...

Related posts brought to you by Yet Another Related Posts Plugin.