how often do bots crawl?

[bpflyr]

I just made a whole bunch of changes to my website. Do I need to resumbit them to google, yahoo, etc? or will the site get crawled again? If so, how often?

Thanks!

[Vasili]

Re: how often do bots crawl?

Bots crawl all the time without rhyme or reason.

Search Engines perform regular updating according to a schedule: Google your site, and click on "cached pages".....in the top of the first cached page (in the info header) you will see the last date your site was spidered, and that should be the approximate time of month set for your site to be re-examined for dynamics and Content changes.

Patience....

[bpflyr]

Re: how often do bots crawl?

Thanks for the info. That is a really useful tip and exactly what I was looking for! Thanks again!

[koolspaces.com]

Re: how often do bots crawl?

In addition to what Vasili said, you can look in cPanelX, click Web/FTP Stats, then "awstats."

A little more than halfway down you'll see a cloumn for "Robots, Spiders, Visitors" and they last date and time they visited your site.

http://www.koolspaces.com

The Best myspaces on myspace.com

[Vasili]

Re: how often do bots crawl?

SE's and bots are completely different animals, my freind, and have very different appetities.

[Bethers]

Re: how often do bots crawl?

For purposes of the stats mentioned above - if you see the Goolge bot - or Msn, or any of the major se's bot - that is the se spidering your site. Just because you do a search and the cached response doesn't show the date you see the spider was there - doesn't mean it wasn't there - and doesn't mean that the se isn't still processing that information.

I've actually seen the cached response to my page show an earlier page - and have had the se be picking up the changes on my pages in web results.

Yes, the bot is different from the se - but most of the bot's that are found in the AWStats are from the se's - and those bots are the ones crawling the site and making the changes in your results.

As to how often the se's will have their bots crawl sites - it is determined by many factors including how often you update the pages, how many links they find to the pages (and how trustworthy they find the links to be), etc etc.