Bill is on target. These are bots essentially are non-intrusive data miners that surf the web looking at all kinds of information to update the web index. Ever do a search on say
And then you see all the hits? This is cause these data miners that crawl the web are looking for information to add to their databses etc., so that when a search is posed, answers can be automatically generated. It is a good thing as long as they are passive, unobtrusive and are not harmful. There are some bots that exploit data mining and those you do not want. Google, AdSense, Yahoo and a few others are pretty benign, so they are usually allowed.