robots.txt file

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts
  • Spiritsoul
    Staff Sergeant

    • Jun 2006
    • 42

    robots.txt file

    I've noticed in my Stats, that I keep getting an error msg because of a file called robots.txt
    So I created one in notepad, and placed in the root file of my web page.
    My question is: What should be in this file. Right now, it's blank.

    Behind The Truth
    wolfer@charter.net

    Also, if you've looked at this site before, you may see some major changes.
  • racefan20
    Major General

    • Jul 2005
    • 2335

    #2
    Re: robots.txt file

    The robots look for robots.txt to find if you've set any "rules" for what they can and cannot access on your server. In most instances, you don't really need to worry about any of this, but you can read more here:

    A Standard for Robot Exclusion
    My NASCAR Forum

    Comment

    • Spiritsoul
      Staff Sergeant

      • Jun 2006
      • 42

      #3
      Re: robots.txt file

      Thanks Racefan. That answered my question. Looked at your site. Nice...

      Comment

      • racefan20
        Major General

        • Jul 2005
        • 2335

        #4
        Re: robots.txt file

        Thank you, I have a lucky spurt now and then.
        My NASCAR Forum

        Comment

        Working...
        X