/robots.txt ?

[gabor]

Hi Everybody,

In my awstats there seems to be quite a large number of 404 errors indicating " /robots.txt - not found ".

What does this mean and how can I fix it?

(I never created a robots.txt file and it's not on my cPanel either).

Thanks,

Gabor

[karenwms63]

Re: /robots.txt ?

You will find the answer in this post: http://www.vodahost.com/vodatalk/cpa...tml#post213987

Good Luck

[karenwms63]

Re: /robots.txt ?

P.S. For most sites, the robot.txt is just a blank page that is uploaded. If you want your site to be spidered and indexed, then just use a blank one.

A robot.txt file is just to let the spider bots know what your preferences are. And doesn't mean they will do what you want them to do...just lets them know what you prefer.

[gabor]

Re: /robots.txt ?

Thank you Karen,

But, I believe, this is a case of "*****s and oranges"...

The point of my question goes somewhere else; in my case, it's apparently the SEs robots encountering some sort of "missing files" (or Text?) and indicate it as an error.

I want to know why that is happening.

Right now I'm not interested in "no follow"s. I'm OK with the robots crawlings, etc, as is.

I'm interested in why the robots (/robot.txt) are hitting misses.

Thanks,

Gabor

[karenwms63]

Re: /robots.txt ?

I believe you missed the point of my second post...just open notepad, leave it blank, name it robot and save...then upload...then your robot.txt file won't be missing anymore...

[gabor]

Re: /robots.txt ?

Right on.
I can see clearly now...
Thanks.

[karenwms63]

Re: /robots.txt ?

Your Welcome!

[gabor]

Re: /robots.txt ?

Oh..
I believe I was looking at the darn thing inside-out...
lol
Gabor

[karenwms63]

Re: /robots.txt ?

LOL...it's one of those simple things, that we need to know, but takes longer to understand than to actually do...