My Go-To WordPress Robots.txt File

A robots.txt can be crucial for both SEO and even keeping web crawlers at bay. It’s important to get the settings just right for your site. I have created a robots.txt file that I’ve been pretty happy with – it’s my go-to file now.

User-agent: Googlebot
Allow: /
Allow: *.js
Allow: *.css

User-agent: Mediapartners-Google
Allow: /
 
User-agent: AdsBot-Google
Allow: /
 
User-agent: Googlebot-Image
Allow: /
 
User-agent: Googlebot-Mobile
Allow: /

User-agent: Googlebot-Video
Allow: /

User-agent: Slurp
Allow: /

User-agent: DuckDuckBot
Allow: /

User-agent: Baiduspider
Allow: /

User-agent: Baiduspider-image
Allow: /

User-agent: YandexBot
Allow: /

User-agent: facebot
Allow: /

User-agent: Applebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: MSNBot
Allow: /

User-agent: AMZNKAssocBot
Allow: /

User-agent: ia_archiver
Allow: /

User-agent: dotbot
Disallow: /

User-agent: PetalBot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-Agent: MJ12bot
Disallow: /

User-agent: BLEXBot
Disallow: /

User-agent: DataForSeoBot
Disallow: /

User-agent: barkrowler
Disallow: /

User-Agent: ImagesiftBot
Crawl-delay: 15

Sitemap: https://www.yoursite.com/sitemap.xml

I don’t use any specific blocking in here.. because let’s face it, the bad bots will crawl anything regardless. Sometimes I’ll disallow specifics for legit search engines, such as my ‘tags’.

The only thing I care about is limiting crawls. If you’ve ever monitored your site traffic, you may notice some crawlers have the tendency to bombard the server. Most crawlers are worthless to me, I only want search engines crawling. It’s not that I’m against all other web crawlers, I just think some are a waste of server resources. It’s important that you search each bot and determine whether or not you should block/limit them on your site.

Some web hosts and services like Cloudflare may already provide some form of bot blocking automatically. Also, if you’re going to be using a custom robots.txt file, make sure that your SEO plugin isn’t auto overwriting it.