Robot Text File Information....

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts
  • Mook25
    Brigadier General

    • Oct 2005
    • 1427

    Robot Text File Information....

    I noticed a few of you asking about the robot text file so thought you might be interested in this article that i found.

    What is the Robot Text File?

    The robot text file is used to disallow specific or all search engine spider’s access to folders or pages that you don't want indexed.

    Why would you want to do this?

    You may have created a personnel page for company employees that you don't want listed. Some webmasters use it to exclude their guest book pages so to avoid people spamming. There are many different reasons to use the robots text file.

    How do I use it?

    You need to upload it to the root of your web site or it will not work - if you don't have access to the root then you will need to use a Meta tag to disallow access. You need to include both the user agent and a file or folder to disallow.

    What does it look like?

    It's really nothing more than a "Notepad" type .txt file named "robots.txt"
    The basic syntax is:
    User-agent: spiders name here
    Disallow:/ filename here
    If you use:
    User-agent: *
    The * acts as a wildcard and disallows all spiders. You may want to use this to stop search engines listing unfinished pages.

    To disallow an entire directory use:
    Disallow:/mydirectory/

    To disallow an individual file use:
    Disallow:/file.htm

    You have to use a separate line for each disallow. You cannot you for example use:
    Disallow:/file1.htm,file2.html

    Instead, you should use:
    Use-agent/*
    Disallow:/file1.htm
    Disallow:/file2.htm

    For a list of spider names visit http://www.searchengineworld.com/cgi-bin/robotcheck.cgi

    Make sure you use the right syntax if you don't it will not work. You can check you syntax here http://www.robotstxt.org/wc/active/html/index.html

    For help on creating robot text files there is a program call robogen. There is a free version and an advanced version, which costs $12.99 http://www.rietta.com/robogen/

    Publication Date: Tuesday 9th November, 2004
    Author: Alan Murray View profile
    Arcade Ninja - Free Flash Arcade
    FreeGadget4me.Com - Learn how to get free gadgets delivered direct to your door for free
  • Winntec
    Master Sergeant

    • Nov 2006
    • 66

    #2
    Re: Robot Text File Information....

    There's a recent update whereby you can link your sitemap file to your robots.txt such that the spiders etc will only crawl what's in your sitemap.
    If you're like us and just moved hosting companies to VH we had to completely rename all our pages and we had a long list of Disallow entries in the robots.txt file we now have 2 lines:

    User-agent: *
    Sitemap: http://www.domain.com/sitemap.xml

    Of course you have to have a sitemap! We use Vigos Gsitemap to generate ours, it's free and you can check it out here: http://www.vigos.com/products/gsitemap/
    CedarAtticVents.com Genuine Cedar Attic Vents, Gable Vents and Shutters in many Standard Designs and Sizes as well as Custom Made to Your Specifications
    HingeAndLatch.com Gate and Door Hardware with the look and feel of yesteryear. Traditional and Antique styles to enhance all your entrance-ways

    Comment

    • Karen Mac
      General

      • Apr 2006
      • 8332

      #3
      Re: Robot Text File Information....

      Axandra's newsletter also reports that a link in the robots text file will trigger the other search engines too. So does a meta tag telling the location.

      Karen

      VodaHost

      Your Website People!
      1-302-283-3777 North America / International
      02036089024 / United Kingdom
      291916438 / Australia

      ------------------------

      Top 3 Best Sellers

      Web Hosting - Unlimited disk space & bandwidth.

      Reseller Hosting - Start your own web hosting business.

      Search Engine & Directory Submission - 300 directories + (Google,Yahoo,Bing)


      Comment

      • aconstas1
        Lieutenant Colonel

        • Mar 2007
        • 582

        #4
        Re: Robot Text File Information....

        Thanks for the great info guys. This information is priceless.
        Best Regards,
        Alex
        Surveillance Systems
        Restaurant POS | Retail POS
        Free WoW Gold Guide

        Comment

        • motoxxx
          Major

          • Dec 2005
          • 343

          #5
          Re: Robot Text File Information....

          Very good info indeed.
          Affordable Medical Insurance
          Discount Dental Plan
          Washington State Health Insurance
          Temporary Health Insurance
          Homeowner Insurance Quote

          Comment

          • alas
            Captain

            • May 2006
            • 229

            #6
            Re: Robot Text File Information....

            Finally I understood what is all this robot story about, I have been thinking.... Robot ... or not Robot???
            Thank you very much for this information.
            Alas

            Comment

            Working...
            X