Robot.txt

[PennieNPence]

Sorry all, I'm back with another question.

I want google to spider certain pages (metatags, images, ect.) I have but not post the links to them in a search.

Do I use the Robot.txt for this, or does that block the spiders from looking at the page at all.

If not is there a way to do this?

Thanks in advance.

[Karen Mac]

Re: Robot.txt

No youd use a sitemap for this. Generally you use robots.txt files to tell it what NOT to spider.

Karen

[PennieNPence]

Re: Robot.txt

Ohhhhhhhhhh.......... Thanks........ But...........

how do you tell the SE's which pages not to show in a sitemap?

:)

[Karen Mac]

Re: Robot.txt

Dont put them in the sitemap!

Then use the robots file to disallow pages you dont want indexed.

Karen

[PennieNPence]

Re: Robot.txt

Oh so if I don't put them in the sitemap
the spiders will scan the pages but won't put the link in searches.
Thanks again you've been a big help. :)

[Karen Mac]

Re: Robot.txt

Oy Vey.. no you put links in a sitemap you do want spidered. If you use one of the generators you can go edit out the pages you dont want spidered, or in some cases you can tell it which pages to omit at the start. So.. If you dont have a sitemap and you dont have links to the pages, how do u want them found?... You do a combo of the site map and also do a robots.txt file

Karen

[Bethers]

Re: Robot.txt

What good does it do to have them spider the page if you don't want them to give the links to it? This doesn't make any sense. If you don't want them to give the links, they you shouldn't have them spider it.

I think you need to explain this better - cuz you're talking *****s and oranges.

[PennieNPence]

Re: Robot.txt

I'm sorry for being so obtuse.. this is all just very new to me.

I understand that the robot.txt file blocks the spiders from seeing the page alltogether..

and that the site map tells them what to search and index in searches...

my problem is that I wanted the spiders to see the elements (buttons,images,ect.) on a page...

but not put the url of that specific page in to their searches
so potental customers won't come in for the first time to the wrong page and leave the site without seeing what we really carry.

[Bethers]

Re: Robot.txt

You can't have it both ways - and EVERY page to your site should be being optimized as an entry page - and for different keywords - so that you'll be able to have so many different pages for people to enter. If a customer wants to find blue widgets - they are much better off entering on the exact page that sells them than having to search your site to find them - so optimize the pages correctly - and let the se's send people to them directly. Don't fight what is good for you.

[PennieNPence]

Re: Robot.txt

the main reason is that I have pages that only hold data for other pages

[PennieNPence]

Re: Robot.txt

I guess I'll just leave them then thanks for all of your help...

I know I didn't make it easy for you... sorry and thanks again

[Karen Mac]

Re: Robot.txt

my problem is that I wanted the spiders to see the elements (buttons,images,ect.) on a page...

Thats what you use alt tags for. They dont SEE images, they can read the file names and the alt tags tho.

Karen

[PennieNPence]

Re: Robot.txt

Thanks again karen. I think I got it now

[pristinelife]

Re: Robot.txt

Oy Vey.. no you put links in a sitemap you do want spidered. If you use one of the generators you can go edit out the pages you dont want spidered, or in some cases you can tell it which pages to omit at the start. So.. If you dont have a sitemap and you dont have links to the pages, how do u want them found?... You do a combo of the site map and also do a robots.txt file

Karen

Inquiring minds want to know............
What happens if someone has BOTH sitemap and robot.text?

[Karen Mac]

Re: Robot.txt

One would get gas which would implode internally and give one mad cow ******* if one could only read!

Originally Posted by Karen Mac
Oy Vey.. no you put links in a sitemap you do want spidered. If you use one of the generators you can go edit out the pages you dont want spidered, or in some cases you can tell it which pages to omit at the start. So.. If you dont have a sitemap and you dont have links to the pages, how do u want them found?... You do a combo of the site map and also do a robots.txt file

Karen