How to properly construct and use robots.txt file?

[kermy]

Having a problem with search engines. Some say my site does not provide the robots.txt file. How do I put it on my web site: http://www.bgpropertiesonline.com ? All help appreciated.
Cheers, kermy.

[Karen Mac]

Re: How to properly construct and use robots.txt file?

A robots.txt file is just that .. a simple text file that in general tells search engine robots what NOT to read or list. You simply make it and upload it to the root of the site in question. If its an addon domain.. public_html/addon folder. You can even use cpanel file manager to upload it.

heres a free generator that gives you an idea HOW to use it or to make one.

http://www.searchenginepromotionhelp.com/m/robots-text-creator/simple-robots-creator.php

For more info just google robots.txt

Karen

[kermy]

Re: How to properly construct and use robots.txt file?

Sorry, I am not good at term yet. What is the root of the site, please?
I tried to follow the 34 TUTORIAL, but ended up with ERRORS in Google.
What may be the reason please? Can it be because of format?
If all my pages are to be seen, why do I need to give any tasks to robots.txt? Or I do not understand it correctly...

[Karen Mac]

Re: How to properly construct and use robots.txt file?

Root of the site means the outermost folder you can access.. yet inside the public html. So if its your main site you would upload to the public_html/

if its an addon you upload to public_/html/yoursitename

if all your pages are to be seen then you dont really NEED a robots txt file you can use a meta tag instead if you like.

Karen

[kermy]

Re: How to properly construct and use robots.txt file?

Thank you very much, Karen Mac! Sorry to be a pain, but what is the outermost folder in my case if I have only one site so far?

I also think I don't need any robots, but what to do with ERRORS? Why does Google want robots.txt? Is it not possible to do without?
To use meta tags means just to put them in page properties? Or any other moves are needed?

[Marincky]

Re: How to properly construct and use robots.txt file?

Thank you very much, Karen Mac! Sorry to be a pain, but what is the outermost folder in my case if I have only one site so far?

I also think I don't need any robots, but what to do with ERRORS? Why does Google want robots.txt? Is it not possible to do without?
To use meta tags means just to put them in page properties? Or any other moves are needed?

Kermy.

Open up a blank notepad. Save it as 'robots.txt' leaving it a blank file.

Go to your BV tool and click on 'Tools' then 'FTP Manager'

Connect using your log in details

On the LH side are the files on your PC
On the RH side is your server.
It usually opens directly to your public_html folder. Just to check this look on the RH side on the blue bar at the top to ensure it says 'public_html'

then find the file you just saved on your PC by browsing in the LH window.

Once found just drag the file over to the RH side.

Done!

[kermy]

Re: How to properly construct and use robots.txt file?

Thank you very-very much, Marincky! It is clear to me now!
But I am afraid, I have one more question. I am not sure I get it right - where and how to create and upload the HTML verification file? It cannot be blank. Or can it?

[Marincky]

Re: How to properly construct and use robots.txt file?

Thank you very-very much, Marincky! It is clear to me now!
But I am afraid, I have one more question. I am not sure I get it right - where and how to create and upload the HTML verification file? It cannot be blank. Or can it?

see above fopr how and where to... and YES it can be blank unless you have any specific instuctions for Google? I.E if you are happy for all your pages to be indexed then leave it blank.

[kermy]

Re: How to properly construct and use robots.txt file?

Sorted out already! Thanks a lot, Marincky, - you are so helpful!

[Marincky]

Re: How to properly construct and use robots.txt file?

Sorted out already! Thanks a lot, Marincky, - you are so helpful!

I try to be, thank you. That's what this forum is here for.

[viliobn]

Re: How to properly construct and use robots.txt file?

Hi

I just place my tread in the section Google but I decide to write and here as well. I have a real estate web site www.goldenchoiceproperties.com and since last week I notice in webmaster tools that I had 38 pages indexed and this week they are only 19. Now after reading all this I thing my problem is with that robots.txt I don’t have that file and I don’t know do I have do create one and upload it? That’s itan addon domain and I have 19 sub-dir. in it. If I create a robot.txt file what should I write in it so google can crawl and index all my pages including and sub-dir. and should I upload it in to the addon folder or needs to be also in the sub-dir.

Thanks