1
2
3
4
5
6
7
8
# robots.txt
User-agent: Googlebot
Disallow: 
User-agent: baiduspider
Disallow: 
User-agent: *
Disallow: 
Crawl-delay: 60