/robot.txt what is this??

[Machiavelli]

I was having a good look through all the awstats to check everything out. Then at the bottom it in HTTP Error codes for the '404 Document not found I had hundreds of hits at

/robots.txt

I don't have a link to this url as I haven't made a page by this name. What is this referring to. Should I have some file there? ? ?

Thought it might be something to do with search bots? ? ?

[Bethers]

Re: /robot.txt what is this??

It is - and all the bots look to see if you have any information for them in that file. Some people swear by the file - I only use them when I have information to give the bots that is different from what they already do - like if I want them to ignore a page. As long as I want them to spider everything, I don't bother.

It's also something that people use to tell the spider how often to come. However, the spiders are not going to pay attn to that either if there isn't a reason to. For example - you put in that you want them to return every 5 days. So, maybe they do - but they don't see anything new. Guess what - that will automatically change them to a longer time even tho it says 5 days. Now, for a site like this forum where there is new content all the time - they will be here ALL the time.

A recent blog by Matt Cutts (Google guru) was interesting when he talked about seeing things on someone's site - and he said basically - if I can find it in minutes using my eyes - our bot will have found it much quicker.

[Machiavelli]

Re: /robot.txt what is this??

Thanks Bethers

When I saw you had replied I guessed it was one for the search engine experts. From what you said, and my more limited understanding of SEO I think i'll leave that as it is and worry more about my content.

I've been reading up a lot on SEO, much of it from information you have directed me to.

The impression I have is that bots are getting more like humans in regards to what they deem to be useful and worthy to show on search engines, and this is a trend that will only strengthen. The 'perfect bot' would be a human that could look at thousands of websites per second, so the 'tricks' people use to try to increase their search ranking will become increasingly redundant as the bots become more sophisticated.

Content is king.

I aim to make my site as useful to my target audience as possible, and hope the bots like it to.

Thanks

[Bethers]

Re: /robot.txt what is this??

Exactly - that's what the aim of the se's is - and they dislike it when web designers/webmasters try to trick them.

But using things like h1 tags, etc - is not tricking them - it's using real tools to help them.

So do as you said - and you'll do fine.

[LadyEye]

Re: /robot.txt what is this??

I agree with Bethers in regards to the spiders paying attention to the number of days you slot in there for changes to your site. While it may be the initial guideline, they will do what they see. On one of my site, I had put every 7 days. But I was changing the content daily, and pretty soon there after they arrived every day. Like Bethers says, they have their rules and guidelines and if you are attempting to trick them, they will discover it. Amazing really.