robot text

[j&p04]

Ok, now I have been reading all night about site maps, now I have come across Robot Text, headache coming on...... is this important to have in your index page too? I have generated the text as explained on another thread, where do I then put the text and how do I link it to my index page? So many questions, so little time.... ha ha.

Thanks in advance to anybody that may reply to me.

Jan

[steve_w]

Re: robot text

It's not necessary to have a robots.txt file unless there are pages you don't want a bot to spider, or if you want to ban a particular bot. If you want to add a blank robots.txt file for peace of mind, simply save a blank notepad page as 'robots.txt' and upload it to your root directory.

I have a robots.txt file which contains the following:

User-agent: Rufusbot
Disallow: /

What this means is that Rufusbot is banned from spidering my site. If you want to ban all bots (you don't) your file would read:

User-agent: *
Disallow: /

Or if you wanted to stop a bot crawling a particular area:

User-agent: googlebot
Disallow: /cgi-bin/

(replace cgi bin with the page or area you don't want indexed).

Hope this is of some use.