Proper location for robots.txt

[ZipZap]

I've read other posts on the subject, and I understand that unless I have a need to block the bots from crawling certain specific pages that a "blank" robots.txt file should be uploaded via the FTP manager in BV.

Question 1) My question is that where should I upload it to on the server? To the public_html folder or should it be all the way in the top (root) folder. The one that is like this /.. ?

Question 2) If I leave this robots.txt file blank, will the bots crawl only my site pages, or will they also crawl my server e-mail folders and my .cpanel, my .htpasswds, and .mozila folders?

Thanks in advance for your help.

[Karen Mac]

Re: Proper location for robots.txt

Actually a blank robots text file will do NOTHING.

You have to say which pages you dont want crawled.

As a rule they will only travel your public html folder.

You need one for each domain and it should be located in each domains folder

if its the main domain then public_html/
if its any other it will be public_html/domain name or subdomain name

Bots do not have access to your cpanel as a rule as they cannot enter passwords. Thats why you have a public html folder.

You will have to code in the pages you dont want them to access. Google for robots.txt and im sure you will come up with some tutorials on how to do this.

Karen

[zuriatman]

Re: Proper location for robots.txt

Beware also zipzap, when a robot cannot index your pages and contents,

your pages cannot be archived for searching by the browsers.

Dont expect to be No1 when you make a search for your website. Your website may not even be found!

[ZipZap]

Re: Proper location for robots.txt

Thanks, Karen and Zuriatman,

I really didn't have anything to put in there to shield from the bots, but in the Google Webmaster Tools, it was coming up that the bots were getting a 404 on my robots.txt file (it didn't exist anywhere) folder. I just put a blank robots.txt file in there just so I wouldn't see those 404s coming up on the Google webmaster tool page.

A follow up question for Karen or anyone else - since she mentioned multiple domains. I want to set up another domain, but I don't understand how to structure my server file folders. Right now, everything is pretty much just dumped into my public_html folder, and that's OK for now, because I only have 1 domain. But what would my file folder structure look like if I wanted to run two separate domains on the server?

Thanks in advance!

[zuriatman]

Re: Proper location for robots.txt

You shoould organize your public_html for ease of mantanence.

Set up Folders and sub-folders so that you can store your files where it will be easy to delete and republish.

Dont mix everything up like a thousand island salad.

When you setup a New Domain [Add-on Domain] a new folder will be automatically created in your public_html with the add-on domain name.

This folder represent your new website and you will have to setup new folders and sub-folders for this website.

[ZipZap]

Re: Proper location for robots.txt

Zuriatman -
So am I understanding you correctly that when I add a new domain, a new folder with that domain name will be automatically added to my public_html file? Next question, should my index page for the new domain also be created (by me) in that new folder? I was of the understanding that index pages have to be in the public_html folder to operate properly.

Thanks in advance for your help,

Matt

[Karen Mac]

Re: Proper location for robots.txt

Matt..
Only the index for your main domain goes in the public html folder.. after that they go in your addon domain folders....Which.. ARE in the public html/addon domain/

Karen