Tweet
Googles New Mediabot
Google currently indexes over 8 billion web pages. However, before these
pages were placed in the index, they were each crawled by a special spider
known as the GoogleBot. Unfortunately, many web masters do not know
about the internal workings of this virtual robot.
In fact, Google actually uses a number of spiders to crawl the Web. You
can catch these spiders by examining your log files.
This article will attempt to reveal some of the most important Google
spiders, their function, and how they affect you as a web master. We'll
start with the well-known GoogleBot.
Creating a Google sitemap allows you to communicate with Google, telling
them about your most important pages, new pages, and updated pages. In
return, Google will provide you with some valuable information as well.
Google Sitemaps will tell you about pages it was unable to crawl and links it
was unable to follow. This allows you to pinpoint problems and fix them so
that you can gain increased exposure in the search results.
The next Google bot in our lineup is known as the MediaBot.
MediaBot - used to analyze Adsense pages
useragent: Mediapartners-Google
MediaBot is the Google crawler for Adsense Publishers. Mediabot is used to
determine wich ads Google should display on Adsense pages.
Google recommends that webmasters specifically add a command in their
robots.txt file that grants Mediabot access to their entire site. To do this,
simply enter the following code into your robots.txt file:
User-agent: Mediapartners-Google*
Disallow:
This will ensure that the MediaBot is able to place relevant Adsense ads on
your site.
Creating a Google sitemap allows you to communicate with Google, telling
them about your most important pages, new pages, and updated pages. In
return, Google will provide you with some valuable information as well.
Google Sitemaps will tell you about pages it was unable to crawl and links it
was unable to follow. This allows you to pinpoint problems and fix them so
that you can gain increased exposure in the search results.
The next Google bot in our lineup is known as the MediaBot.
MediaBot - used to analyze Adsense pages
useragent: Mediapartners-Google
MediaBot is the Google crawler for Adsense Publishers. Mediabot is used to
determine wich ads Google should display on Adsense pages.
Google recommends that webmasters specifically add a command in their
robots.txt file that grants Mediabot access to their entire site. To do this,
simply enter the following code into your robots.txt file:
User-agent: Mediapartners-Google*
Disallow:
This will ensure that the MediaBot is able to place relevant Adsense ads on
your site.
Matt Cutts claims that your website does not get any advantage by using
Adsense. However, in my mind, simply getting your site updated in and of
itself is an advantage.
This is very similar to Google Analytics, which also promotes a slightly
higher degree of spider activity.
Those who run Google Analytics on their site can expect additional spider
activity.
However, you certainly shouldn't depend on any of these tools for getting
your site indexed. The key to frequent spidering is having quality inbound
links, quality content, and frequent updates.
Have images on your site? If so, you have likely been visited by our next
Google spider, the ImageBot.
ImageBot - used to crawl for the Image Search
useragent: GoogleBot-Image
The Imagebot prowls the Web for images to place in Google's image search.
Images are ranked based upon their filename, surrounding text, alt text,
and page title.
If you have a website that is primarily image based, then you would
definitely want to optimize your images to receive some extra Google
traffïc.
On the other hand, some web sites may not benefit from Google image
search. In most cases, the traffïc from the Image search engine is very low
quality and rarely converts into buyers. Many people are often just looking
for images that they can swipe. So, if you want to save some bandwidth,
use your robots.txt file to block ImageBot from accessing your image
directory.
One of the few exceptions I would make is if you have a site dedicated to
downloadable images.
Our final bot is completely dedicated to the Google Adwords program.
AdsBot - Checks Adwords landing pages for quality
useragent: AdsBot-Google
AdsBot is one of Google's newest spiders. This new crawler is being used
to analyze the content of advertising landing pages, which helps determine
the Quality score that Google assigns to your ads.
Google uses this Quality score in combination with the amount you are
willing to bid to determine the position of your ads. Therefore, ads with
a high quality score can rank higher even if other advertisers are paying
more than you.
This is one of Google's many efforts to ensure that they are delivering the
best results to their users.
Can you still block being spidered? Of course, but it will lower your overall
Adwords quality score, which could end up lowering the positioning of your
ads. If possible, it is best to give AdsBot complete access to your site.
Today's Google bots are becoming more advanced all the time. However,
nothing beats relevant, quality, updated content. Deliver that and the
search engines will eat it up.
Author : Kim Roachs
pages were placed in the index, they were each crawled by a special spider
known as the GoogleBot. Unfortunately, many web masters do not know
about the internal workings of this virtual robot.
In fact, Google actually uses a number of spiders to crawl the Web. You
can catch these spiders by examining your log files.
This article will attempt to reveal some of the most important Google
spiders, their function, and how they affect you as a web master. We'll
start with the well-known GoogleBot.
Creating a Google sitemap allows you to communicate with Google, telling
them about your most important pages, new pages, and updated pages. In
return, Google will provide you with some valuable information as well.
Google Sitemaps will tell you about pages it was unable to crawl and links it
was unable to follow. This allows you to pinpoint problems and fix them so
that you can gain increased exposure in the search results.
The next Google bot in our lineup is known as the MediaBot.
MediaBot - used to analyze Adsense pages
useragent: Mediapartners-Google
MediaBot is the Google crawler for Adsense Publishers. Mediabot is used to
determine wich ads Google should display on Adsense pages.
Google recommends that webmasters specifically add a command in their
robots.txt file that grants Mediabot access to their entire site. To do this,
simply enter the following code into your robots.txt file:
User-agent: Mediapartners-Google*
Disallow:
This will ensure that the MediaBot is able to place relevant Adsense ads on
your site.
Creating a Google sitemap allows you to communicate with Google, telling
them about your most important pages, new pages, and updated pages. In
return, Google will provide you with some valuable information as well.
Google Sitemaps will tell you about pages it was unable to crawl and links it
was unable to follow. This allows you to pinpoint problems and fix them so
that you can gain increased exposure in the search results.
The next Google bot in our lineup is known as the MediaBot.
MediaBot - used to analyze Adsense pages
useragent: Mediapartners-Google
MediaBot is the Google crawler for Adsense Publishers. Mediabot is used to
determine wich ads Google should display on Adsense pages.
Google recommends that webmasters specifically add a command in their
robots.txt file that grants Mediabot access to their entire site. To do this,
simply enter the following code into your robots.txt file:
User-agent: Mediapartners-Google*
Disallow:
This will ensure that the MediaBot is able to place relevant Adsense ads on
your site.
Matt Cutts claims that your website does not get any advantage by using
Adsense. However, in my mind, simply getting your site updated in and of
itself is an advantage.
This is very similar to Google Analytics, which also promotes a slightly
higher degree of spider activity.
Those who run Google Analytics on their site can expect additional spider
activity.
However, you certainly shouldn't depend on any of these tools for getting
your site indexed. The key to frequent spidering is having quality inbound
links, quality content, and frequent updates.
Have images on your site? If so, you have likely been visited by our next
Google spider, the ImageBot.
ImageBot - used to crawl for the Image Search
useragent: GoogleBot-Image
The Imagebot prowls the Web for images to place in Google's image search.
Images are ranked based upon their filename, surrounding text, alt text,
and page title.
If you have a website that is primarily image based, then you would
definitely want to optimize your images to receive some extra Google
traffïc.
On the other hand, some web sites may not benefit from Google image
search. In most cases, the traffïc from the Image search engine is very low
quality and rarely converts into buyers. Many people are often just looking
for images that they can swipe. So, if you want to save some bandwidth,
use your robots.txt file to block ImageBot from accessing your image
directory.
One of the few exceptions I would make is if you have a site dedicated to
downloadable images.
Our final bot is completely dedicated to the Google Adwords program.
AdsBot - Checks Adwords landing pages for quality
useragent: AdsBot-Google
AdsBot is one of Google's newest spiders. This new crawler is being used
to analyze the content of advertising landing pages, which helps determine
the Quality score that Google assigns to your ads.
Google uses this Quality score in combination with the amount you are
willing to bid to determine the position of your ads. Therefore, ads with
a high quality score can rank higher even if other advertisers are paying
more than you.
This is one of Google's many efforts to ensure that they are delivering the
best results to their users.
Can you still block being spidered? Of course, but it will lower your overall
Adwords quality score, which could end up lowering the positioning of your
ads. If possible, it is best to give AdsBot complete access to your site.
Today's Google bots are becoming more advanced all the time. However,
nothing beats relevant, quality, updated content. Deliver that and the
search engines will eat it up.
Author : Kim Roachs
Comment