WooCommerce stores attract a lot of strange crawling. Bots follow add to cart links, wishlist links, search results and every filter or sort option in your shop. Each of those URLs is a request your server has to build from scratch, so a busy crawler can slow the store down for real customers.
A well set up robots.txt file fixes a surprising amount of this. It takes load off your server and helps Google spend its time on the pages you actually want ranked. Below is the template we use, plus what each section does.
The Recommended WooCommerce robots.txt
Copy the file below into the robots.txt in the root folder of your site. Change the Sitemap line to your own sitemap address before you save it.
User-agent: *
#Added to slow down aggressive crawlers from causing a denial of service attack
Crawl-delay: 5
# Block crawling of logon pages
Disallow: /wp-admin/
Disallow: /*wp-login.php*
Disallow: /my-account/*
# Block search pages
Disallow: *s=*
Disallow: */search/*
# Block add to cart and wishlist links, some themes link directly to these and can cause high CPU usage
Disallow: *add-to-cart*
Disallow: *add_to_wishlist*
# Block common query strings, you may want to block other filter strings if you theme has a sidebar filter
Disallow: *currency=*
Disallow: /*?orderby*
# Block the WordPress feed URLs
Disallow: */feed/*
# Block Plesk and Cpanel smart update test site crawling
Disallow: *wp-toolkit*
# We need this allow link because we blocked wp-admin earlier
Allow: /wp-admin/admin-ajax.php
# Change this to your sitemap link
Sitemap: https://www.yourdomain.com/sitemap_index.xml
# Block aggressive SEO tools
# See more at https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/blob/master/robots.txt/robots.txt
User-agent: AhrefsBot
Disallow: /
User-agent: Semrush
Disallow:/
User-agent: SemrushBot
Disallow:/If an SEO plugin is managing a virtual robots.txt for you, paste the rules into its robots.txt editor instead so they don’t get overwritten.
What Each Section Does
1. Crawl delay
Aggressive bots can eat a lot of CPU. The Crawl-delay: 5 line asks crawlers to wait five seconds between requests. Not every crawler respects it (Google ignores it completely), but plenty of others do, so it’s still worth having.
If your store has thousands of products, a long delay can stop some crawlers from getting through the whole site. Lower the number to suit.
2. Login and account pages
Search engines have no reason to visit your WordPress login screen, your admin area or your customers’ account pages. The file blocks these three paths:
/wp-admin//wp-login.php/my-account/
3. Internal search results
Letting Google crawl and index your site search pages opens the door to negative SEO attacks, where someone links to junk search URLs on your domain. Search pages are dynamic and shouldn’t be indexed anyway, so the file blocks them.
4. Add to cart and add to wishlist links
This is the big one for speed. When a bot follows add to cart or add to wishlist links, your server processes each hit as though a real shopper clicked the button, sometimes three to five times a second. Those requests can’t be served from cache, so they cause heavy load spikes. Some themes link straight to these URLs, which makes the problem worse. Our guide to a slow WooCommerce add to cart covers this in more detail.
5. Filter and sort query strings
WooCommerce and its add-ons generate lots of URL variations, such as ?orderby= for sorting and currency switcher parameters. Crawling these wastes crawl budget and creates duplicate content. If your theme has a sidebar filter, you may want to add its query strings to the list as well.
Before you block them, check your site navigation doesn’t depend on these filtered URLs to reach products.
6. Feeds and smart update test sites
The file also blocks the WordPress feed URLs and the wp-toolkit paths that Plesk and cPanel smart update tools use for their test copies of your site.
7. Allow admin-ajax.php and list your sitemap
Because the file blocks /wp-admin/, it needs to allow /wp-admin/admin-ajax.php again. WooCommerce uses Ajax for things like product variations, and blocking it can break those features for visitors and search engines alike.
Always finish with a link to your sitemap_index.xml so crawlers can find the URLs you want indexed.
8. Aggressive SEO tools
Crawlers from tools like Ahrefs and Semrush can be heavy. If you don’t use their data for your own site, blocking them saves server resources and keeps your data away from competitors who use those tools.
Don’t Build Your Menus From Searches or Filters
We regularly see stores where menu items point to a search URL or directly to a filter. That’s bad for speed and SEO. Those pages usually aren’t cached, so every visit hits the server, and they aren’t real pages, so they don’t count for much in search.
Caching these query strings can help in some cases, but real pages are the better answer. The right setup depends on how many products you have and how you use filters, but product categories and product tags are usually the best way to group products. A store with lots of uncached pages is a common reason for a slow WooCommerce store.
Back It Up With Cloudflare Firewall Rules
Google and other legitimate crawlers follow robots.txt. Plenty of bots don’t, and they’ll keep hammering your add to cart URLs regardless. That’s why we also use Cloudflare firewall rules to filter this traffic before it reaches your server. Create them under Security > WAF > Custom rules and block requests containing add-to-cart= from aggressive crawlers. Never block Googlebot or Bingbot.
If bots are also placing junk orders on your store, see how to stop spam and fake WooCommerce orders.
Quick Checklist
- Add the robots.txt above and update the sitemap line
- Lower the crawl delay if you have thousands of products
- Add any extra filter query strings your theme uses
- Make sure admin-ajax.php is still allowed
- Move menu links off search and filter URLs onto categories or tags
- Add Cloudflare firewall rules for bots that ignore robots.txt
Bots Slowing Your Store Down? We Can Help
Heavy bot crawling of cart and filter URLs is exactly the kind of load WP Alpha hosting is built to handle. Our WooCommerce servers come with server caching, Redis object caching and Nginx static file serving, so real shoppers stay fast even when crawlers come calling.
Moving from another host? We’ll migrate your site for free.
View WP Alpha Plans →