how often do bots crawl?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts
  • bpflyr
    First Sergeant

    • Jul 2007
    • 72

    how often do bots crawl?

    I just made a whole bunch of changes to my website. Do I need to resumbit them to google, yahoo, etc? or will the site get crawled again? If so, how often?

    Thanks!
  • Vasili
    Moderator

    • Mar 2006
    • 14683

    #2
    Re: how often do bots crawl?

    Bots crawl all the time without rhyme or reason.

    Search Engines perform regular updating according to a schedule: Google your site, and click on "cached pages".....in the top of the first cached page (in the info header) you will see the last date your site was spidered, and that should be the approximate time of month set for your site to be re-examined for dynamics and Content changes.

    Patience....
    . VodaWebs....Luxury Group
    * Success Is Potential Realized *

    Comment

    • bpflyr
      First Sergeant

      • Jul 2007
      • 72

      #3
      Re: how often do bots crawl?

      Thanks for the info. That is a really useful tip and exactly what I was looking for! Thanks again!

      Comment

      • koolspaces.com
        Second Lieutenant

        • Jul 2007
        • 136

        #4
        Re: how often do bots crawl?

        In addition to what Vasili said, you can look in cPanelX, click Web/FTP Stats, then "awstats."

        A little more than halfway down you'll see a cloumn for "Robots, Spiders, Visitors" and they last date and time they visited your site.



        The Best myspaces on myspace.com

        Comment

        • Vasili
          Moderator

          • Mar 2006
          • 14683

          #5
          Re: how often do bots crawl?

          SE's and bots are completely different animals, my freind, and have very different appetities.
          . VodaWebs....Luxury Group
          * Success Is Potential Realized *

          Comment

          • Bethers
            Major General & Forum Moderator

            • Feb 2006
            • 5224

            #6
            Re: how often do bots crawl?

            For purposes of the stats mentioned above - if you see the Goolge bot - or Msn, or any of the major se's bot - that is the se spidering your site. Just because you do a search and the cached response doesn't show the date you see the spider was there - doesn't mean it wasn't there - and doesn't mean that the se isn't still processing that information.

            I've actually seen the cached response to my page show an earlier page - and have had the se be picking up the changes on my pages in web results.

            Yes, the bot is different from the se - but most of the bot's that are found in the AWStats are from the se's - and those bots are the ones crawling the site and making the changes in your results.

            As to how often the se's will have their bots crawl sites - it is determined by many factors including how often you update the pages, how many links they find to the pages (and how trustworthy they find the links to be), etc etc.
            Beth
            A Child's Palace - Pinata Palace - Moxie Enterprises

            SEO and Marketing Tools
            SEO - The Basics

            Comment

            Working...
            X