Googleのクローラー

Googleの検索エンジンクローラーを紹介しよう。
当サイトで確認できているのを抜粋してみよう。
Agentでhttp://www.google.com/bot.html
で抜粋してみた。

1. DoCoMo/1.0/N505i/c20/TB/W20H10 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
2. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
3. Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
4. Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
5. SoftBank/1.0/911SH/SHJ001 Browser/NetFront/3.3 Profile/MIDP-2.0 Configuration/CLDC-1.1 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
6. KDDI-CA34 UP.Browser/6.2.0.10.2.2 (GUI) MMP/2.0 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)

最近
Mediapartners-Google
ってagentが来した。
おそらくgooglesitemapに登録したサイトに来るクローラーだと思われる。


Googleのモバイル用クローラーについてはこちらにまとめます

サイトで確認されたhost一覧

• "crawl-66-249-73-46.googlebot.com",266543,68.7%
• "crawl-66-249-70-138.googlebot.com",121086,31.2%
• "crawl-66-249-72-15.googlebot.com",21,0.0%
• "crawl-66-249-67-235.googlebot.com",11,0.0%
• "crawl-66-249-70-242.googlebot.com",8,0.0%
• "crawl-66-249-70-75.googlebot.com",6,0.0%
• "crawl-66-249-72-109.googlebot.com",5,0.0%
• "crawl-66-249-73-6.googlebot.com",4,0.0%
• "crawl-66-249-65-207.googlebot.com",4,0.0%
• "crawl-66-249-70-177.googlebot.com",4,0.0%
• "crawl-66-249-72-146.googlebot.com",4,0.0%
• "crawl-66-249-65-1.googlebot.com",3,0.0%
• "crawl-66-249-65-206.googlebot.com",3,0.0%
• "crawl-66-249-65-232.googlebot.com",3,0.0%
• "crawl-66-249-66-170.googlebot.com",3,0.0%
• "crawl-66-249-66-208.googlebot.com",3,0.0%
• "crawl-66-249-72-179.googlebot.com",3,0.0%
• "crawl-66-249-72-193.googlebot.com",3,0.0%
• "crawl-66-249-72-211.googlebot.com",3,0.0%
• "crawl-66-249-73-2.googlebot.com",2,0.0%
• "crawl-66-249-65-38.googlebot.com",2,0.0%
• "crawl-66-249-65-43.googlebot.com",2,0.0%
• "crawl-66-249-65-44.googlebot.com",2,0.0%
• "crawl-66-249-65-72.googlebot.com",2,0.0%
• "crawl-66-249-65-74.googlebot.com",2,0.0%
• "crawl-66-249-65-78.googlebot.com",2,0.0%
• "crawl-66-249-66-16.googlebot.com",2,0.0%
• "crawl-66-249-66-52.googlebot.com",2,0.0%
• "crawl-66-249-70-56.googlebot.com",2,0.0%
• "crawl-66-249-72-11.googlebot.com",2,0.0%
• "crawl-66-249-72-14.googlebot.com",2,0.0%
• "crawl-66-249-72-16.googlebot.com",2,0.0%
• "crawl-66-249-72-33.googlebot.com",2,0.0%
• "crawl-66-249-72-69.googlebot.com",2,0.0%
• "crawl-66-249-72-72.googlebot.com",2,0.0%
• "crawl-66-249-72-83.googlebot.com",2,0.0%
• "crawl-66-249-65-145.googlebot.com",2,0.0%
• "crawl-66-249-65-148.googlebot.com",2,0.0%
• "crawl-66-249-65-163.googlebot.com",2,0.0%
• "crawl-66-249-65-174.googlebot.com",2,0.0%
• "crawl-66-249-65-208.googlebot.com",2,0.0%
• "crawl-66-249-65-211.googlebot.com",2,0.0%
• "crawl-66-249-66-106.googlebot.com",2,0.0%
• "crawl-66-249-66-130.googlebot.com",2,0.0%
• "crawl-66-249-66-162.googlebot.com",2,0.0%
• "crawl-66-249-66-242.googlebot.com",2,0.0%
• "crawl-66-249-72-141.googlebot.com",2,0.0%
• "crawl-66-249-72-235.googlebot.com",2,0.0%
• "crawl-66-249-72-243.googlebot.com",2,0.0%
• "crawl-66-249-65-7.googlebot.com",1,0.0%
• "crawl-66-249-65-8.googlebot.com",1,0.0%
• "crawl-66-249-66-8.googlebot.com",1,0.0%
• "crawl-66-249-72-1.googlebot.com",1,0.0%
• "crawl-66-249-72-3.googlebot.com",1,0.0%
• "crawl-66-249-72-6.googlebot.com",1,0.0%
• "crawl-66-249-65-13.googlebot.com",1,0.0%
• "crawl-66-249-65-19.googlebot.com",1,0.0%
• "crawl-66-249-65-34.googlebot.com",1,0.0%
• "crawl-66-249-65-39.googlebot.com",1,0.0%
• "crawl-66-249-65-40.googlebot.com",1,0.0%
• "crawl-66-249-65-47.googlebot.com",1,0.0%
• "crawl-66-249-65-48.googlebot.com",1,0.0%
• "crawl-66-249-65-51.googlebot.com",1,0.0%
• "crawl-66-249-65-52.googlebot.com",1,0.0%
• "crawl-66-249-65-67.googlebot.com",1,0.0%
• "crawl-66-249-65-75.googlebot.com",1,0.0%
• "crawl-66-249-65-80.googlebot.com",1,0.0%
• "crawl-66-249-65-82.googlebot.com",1,0.0%
• "crawl-66-249-65-83.googlebot.com",1,0.0%
• "crawl-66-249-65-97.googlebot.com",1,0.0%
• "crawl-66-249-65-98.googlebot.com",1,0.0%
• "crawl-66-249-65-99.googlebot.com",1,0.0%
• "crawl-66-249-66-35.googlebot.com",1,0.0%
• "crawl-66-249-66-39.googlebot.com",1,0.0%
• "crawl-66-249-66-42.googlebot.com",1,0.0%
• "crawl-66-249-66-49.googlebot.com",1,0.0%
• "crawl-66-249-66-81.googlebot.com",1,0.0%
• "crawl-66-249-66-82.googlebot.com",1,0.0%
• "crawl-66-249-72-20.googlebot.com",1,0.0%
• "crawl-66-249-72-36.googlebot.com",1,0.0%
• "crawl-66-249-72-37.googlebot.com",1,0.0%
• "crawl-66-249-72-49.googlebot.com",1,0.0%
• "crawl-66-249-72-51.googlebot.com",1,0.0%
• "crawl-66-249-72-71.googlebot.com",1,0.0%
• "crawl-66-249-72-74.googlebot.com",1,0.0%
• "crawl-66-249-72-75.googlebot.com",1,0.0%
• "crawl-66-249-72-76.googlebot.com",1,0.0%
• "crawl-66-249-72-77.googlebot.com",1,0.0%
• "crawl-66-249-72-78.googlebot.com",1,0.0%
• "crawl-66-249-72-80.googlebot.com",1,0.0%
• "crawl-66-249-72-97.googlebot.com",1,0.0%
• "crawl-66-249-65-101.googlebot.com",1,0.0%
• "crawl-66-249-65-104.googlebot.com",1,0.0%
• "crawl-66-249-65-133.googlebot.com",1,0.0%
• "crawl-66-249-65-136.googlebot.com",1,0.0%
• "crawl-66-249-65-137.googlebot.com",1,0.0%
• "crawl-66-249-65-143.googlebot.com",1,0.0%
• "crawl-66-249-65-144.googlebot.com",1,0.0%
• "crawl-66-249-65-166.googlebot.com",1,0.0%
• "crawl-66-249-65-171.googlebot.com",1,0.0%
• "crawl-66-249-65-176.googlebot.com",1,0.0%
• "crawl-66-249-65-179.googlebot.com",1,0.0%
• "crawl-66-249-65-194.googlebot.com",1,0.0%
• "crawl-66-249-65-198.googlebot.com",1,0.0%
• "crawl-66-249-65-199.googlebot.com",1,0.0%
• "crawl-66-249-65-200.googlebot.com",1,0.0%
• "crawl-66-249-65-201.googlebot.com",1,0.0%
• "crawl-66-249-65-209.googlebot.com",1,0.0%
• "crawl-66-249-65-212.googlebot.com",1,0.0%
• "crawl-66-249-65-226.googlebot.com",1,0.0%
• "crawl-66-249-65-227.googlebot.com",1,0.0%
• "crawl-66-249-65-229.googlebot.com",1,0.0%
• "crawl-66-249-65-230.googlebot.com",1,0.0%
• "crawl-66-249-65-233.googlebot.com",1,0.0%
• "crawl-66-249-65-237.googlebot.com",1,0.0%
• "crawl-66-249-65-239.googlebot.com",1,0.0%
• "crawl-66-249-66-112.googlebot.com",1,0.0%
• "crawl-66-249-66-129.googlebot.com",1,0.0%
• "crawl-66-249-66-136.googlebot.com",1,0.0%
• "crawl-66-249-66-137.googlebot.com",1,0.0%
• "crawl-66-249-66-147.googlebot.com",1,0.0%
• "crawl-66-249-66-161.googlebot.com",1,0.0%
• "crawl-66-249-66-167.googlebot.com",1,0.0%
• "crawl-66-249-66-194.googlebot.com",1,0.0%
• "crawl-66-249-66-195.googlebot.com",1,0.0%
• "crawl-66-249-66-198.googlebot.com",1,0.0%
• "crawl-66-249-66-200.googlebot.com",1,0.0%
• "crawl-66-249-66-227.googlebot.com",1,0.0%
• "crawl-66-249-66-237.googlebot.com",1,0.0%
• "crawl-66-249-66-239.googlebot.com",1,0.0%
• "crawl-66-249-66-243.googlebot.com",1,0.0%
• "crawl-66-249-66-244.googlebot.com",1,0.0%
• "crawl-66-249-72-103.googlebot.com",1,0.0%
• "crawl-66-249-72-105.googlebot.com",1,0.0%
• "crawl-66-249-72-106.googlebot.com",1,0.0%
• "crawl-66-249-72-107.googlebot.com",1,0.0%
• "crawl-66-249-72-111.googlebot.com",1,0.0%
• "crawl-66-249-72-115.googlebot.com",1,0.0%
• "crawl-66-249-72-129.googlebot.com",1,0.0%
• "crawl-66-249-72-133.googlebot.com",1,0.0%
• "crawl-66-249-72-135.googlebot.com",1,0.0%
• "crawl-66-249-72-144.googlebot.com",1,0.0%
• "crawl-66-249-72-163.googlebot.com",1,0.0%
• "crawl-66-249-72-168.googlebot.com",1,0.0%
• "crawl-66-249-72-174.googlebot.com",1,0.0%
• "crawl-66-249-72-180.googlebot.com",1,0.0%
• "crawl-66-249-72-202.googlebot.com",1,0.0%
• "crawl-66-249-72-230.googlebot.com",1,0.0%
• "crawl-66-249-72-231.googlebot.com",1,0.0%
• "crawl-66-249-72-232.googlebot.com",1,0.0%
• "crawl-66-249-72-233.googlebot.com",1,0.0%
• "crawl-66-249-72-234.googlebot.com",1,0.0%
• "crawl-66-249-72-237.googlebot.com",1,0.0%
• "crawl-66-249-72-239.googlebot.com",1,0.0%
• "crawl-66-249-72-242.googlebot.com",1,0.0%