几个主要搜索引擎的user_agent及抓取频率
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
32032
Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters[……]
这是fengyqf的博客
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
32032
Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters[……]