help with robots

[pet121]

For quite sometime I have been getting this error message

[Wed Aug 31 10:43:58 2005] [error] [client 213.180.216.253] File does not exist: /home/onlineg/public_html/robots.txt

I do have robot text in my source but robots, crawls whatever you call it, are having a hard time finding my page. Does it have something to do with me uploading my files to public_html? please help I am begging. pet121

[Maddog]

Search engines will look in your root domain for a special file named "robots.txt" (http://www.mydomain.com/robots.txt). The file tells the robot (spider) which files it may spider (download). This system is called, The Robots Exclusion Standard.

[Maddog]

check out

http://www.webtoolcentral.com

[pet121]

thanks

I am getting there slowly. This is turnning out to be no hobby thats for sure. My site well its not even a quarter finished but put it on the net anyway. I keep on getting sidetracked on learnning how to promote your site. I appreciate your response and will look into it. Some robots grab some don't. I am doing very well on google and yahoo. 35000 hits last month not to shabby for a quarter of a site. Thanks again pet121

[djdiamonds]

what is robot text and how do you put it on your website? i want to optimise my searches on search engines for my site www.davidjoffediamonds.com and i dont know why everyone keeps mentioning this robot text!! PLEASE HELP

[Maddog]

what is robot text and how do you put it on your website? i want to optimise my searches on search engines for my site www.davidjoffediamonds.com and i dont know why everyone keeps mentioning this robot text!! PLEASE HELP

david

go here

http://www.webtoolcentral.com/webmaster/tools/

this site will help you make your "robot.txt" and explain all about it, also there some other great tools for helping to get your site listed/ranked

any problem just ask and I will try and help

[Steven.H]

david

go here

http://www.webtoolcentral.com/webmaster/tools/

this site will help you make your "robot.txt" and explain all about it, also there some other great tools for helping to get your site listed/ranked

any problem just ask and I will try and help

Thanks for the website.

[djdiamonds]

thanks very much for you help - i didnt quite understand but i think i did it right :~

[Maddog]

thanks very much for you help - i didnt quite understand but i think i did it right :~

Not much to do other than copy to note pad save as robot.txt and the upload to you web site using FTP,

Also have a look at your meta tags, get this forum on changes to make.

The other part to help search bots is to have a sitemap, go to

http://www.blocklayer.com/googlesitemap/Default.aspx

will help you make what you need

[djdiamonds]

thanks so much - you make it all seem so easy!! im almost certain i created a sitemap and i managed to create meta tags!! (hooray)

[paulb]

Hi Maddog,

Just used the free tool for robots file, this is what came back, it doesn't seem a lot, I expected a lot of code,

Paul

Here is what your robots.txt file should look like;

# Robots.txt file created by http://www.webtoolcentral.com
# For domain: http://www.superfastincome.com

# All robots will spider the domain
User-agent: *
Disallow:

[Maddog]

Know what you mean Paul not alot to it, looks ok to me.

[pet121]

maddog

wouldn't the below robot text disallow robot to enter?



# Robots.txt file created by http://www.webtoolcentral.com
# For domain: http://www.superfastincome.com

# All robots will spider the domain
User-agent: *
Disallow

I don't want to get it wrong, big penalties. My ranking are going well without it. I am pr3, but futher optimization is needed. thanks pet121

[Maddog]

the * is not a wild card to disallow

User-agent: *

here you would put the bots disallow, my understanding is blank means allow all

[pet121]

thanks maddog

Okay I understand but I am still getting these error messages. Out of 72 robots that visited my site only 17% actually found it. Its just weird. Thanks anyway pet121