---
title: "AI Crawler Server Load on WordPress: What It Costs and How to Cut It"
description: "AI crawler server load on WordPress, measured: about 210 ms per uncached hit, under 0.4 ms cached. A week of real traffic, Crawl-delay and rate limits."
url: "https://skybootstrap.com/ai-bots-server-load/"
canonical_url: "https://skybootstrap.com/ai-bots-server-load/"
date: "2026-11-10T06:11:00+06:00"
modified: "2026-11-05T12:26:34+06:00"
author: Skybootstrap Team
type: article
site: SkyBootstrap
language: en-US
---

# AI Crawler Server Load on WordPress: What It Costs and How to Cut It

AI crawler server load depends mostly on whether the AI crawlers get cached pages. On our WordPress test site, an uncached page took about 210 ms to come back with five requests running at once, while a cached copy took under half a millisecond. Cache your pages first; then slow down or block the crawlers that still cause trouble.

For scale: tlinky.com, one of our live WordPress sites, logged about 1,700 requests from crawler names in about one week. That’s around 243 a day, light for a cached site.

## AI crawler server load: what one request costs

On our WordPress test site, the server managed about 23 uncached page requests a second against about 14,000 cached ones, so caching decides most of the cost. Server load here means the work your server does to answer each request, mostly running PHP and the database.

We sent 200 requests with GPTBot’s user agent, five at a time, using the `ab` benchmark tool, three runs per page. With a page cache, the server keeps the finished HTML and hands it out again, skipping WordPress; ours was nginx’s built-in cache.

![AI crawler load test on a WordPress test site: uncached post about 23 requests per second at about 210 ms each, uncached 404 about 31 per second at 162 ms, cached post about 14,000 to 17,000 per second under 0.4 ms](https://skybootstrap.com/wp-content/uploads/2026/11/ai-bots-server-load-ab-results.webp "Lab, 28 Oct 2026: WordPress 7.1.3, nginx 1.24.0, PHP 8.3, ab -n 200 -c 5 (lab output, retyped as a table)")

| Page | Requests per second | Time per request |
| --- | --- | --- |
| Uncached post | 23.2 to 23.8 | about 210 to 216 ms |
| Uncached 404 page | about 30.8 | about 162 ms |
| Cached post | about 14,000 to 17,400 | 0.29 to 0.36 ms |

Two things stand out. The cached page was hundreds of times cheaper than the uncached one. And a 404 wasn’t free: WordPress still had to run to say the page didn’t exist, which matters when crawlers chase old or made-up links.

These are numbers from a small two-core test server running SQLite, with the default theme and sample content, not a typical MySQL host. The ratio is the useful part.

Bandwidth, the data your server sends out, scales the same way. Our test post was about 68 KB of uncompressed HTML, so 1,000 bot requests send about 68 MB before compression; gzip makes the real transfer smaller. Caching doesn’t shrink that, but it makes each copy cheap to send.

## How much AI crawler traffic a WordPress site gets

Our link-tools site logged about 1,700 crawler-named requests in a week, about 243 a day, which is light on a cached site. Our [Skyboot AI Markdown plugin](https://skybootstrap.com/skyboot-ai-markdown/) recorded them on tlinky.com during the week before 28 October 2026 (about seven days of data):

- 684 hits that passed an IP check against the crawler owner’s list, 403 of them from bingbot (222) and Googlebot (181), then OAI-SearchBot (164).
- 726 requests using Anthropic’s and Meta’s bot names, which the plugin lists without an IP check.
- 291 requests from fake crawlers, using famous bot names from addresses those companies don’t own.

The plugin lists these per bot and per page ([our documentation](https://skybootstrap.com/docs/skyboot-ai-markdown/ai-traffic-analytics/)). Requests served entirely by a CDN never reach it, and in our cache test a server page cache hid repeat hits from WordPress too, so both are missing. Our AI crawler tracking guide does the same count from server logs.

What would that week cost? As an illustration only, at our test server’s speed: if every one of the 1,701 requests missed the cache, it would take about 72 seconds of the server’s full output a week (1,701 at 23.5 a second). At about 68 KB of uncompressed HTML each, that’s about 116 MB of bandwidth a week.

Your numbers will differ, and they’re rising across the web: [Kinsta’s own network data](https://kinsta.com/blog/ai-crawlers-wordpress/) puts AI bots at roughly 1 in 31 web visits, up from about 1 in 200 a year earlier. Count your own before deciding anything.

## Fix caching before you block anything

Make sure crawlers get cached pages. In our test, five GPTBot requests to one post reached WordPress once: the first filled the cache and the next four were served from it. If you have no page cache yet, turn on your host’s cache or a free plugin such as WP Super Cache (LiteSpeed Cache if your host runs LiteSpeed).

- **Check that bots get cached copies.** Run this command twice from your computer’s terminal, with one of your post URLs: `curl -sI -A "GPTBot" https://example.com/my-post/ | grep -i cache`. The second run should show a hit. On our nginx test site it showed `X-Cache: MISS`, then `X-Cache: HIT`. Header names vary: Cloudflare sends `cf-cache-status`, LiteSpeed `x-litespeed-cache`, and some caches send none.
- **Fix the broken links bots keep requesting,** since each 404 runs WordPress.
- **Put blocks in front of the cache.** In our nginx test, a user-agent block still refused Bytespider for a page already in the cache, because the rule runs first.

## Crawl-delay: which AI bots honour it

Crawl-delay is a robots.txt line that tells a bot how many seconds to pause between fetches, but only some bots read it: Anthropic, Common Crawl and Bing do, Google and Apple don’t. It’s a non-standard robots.txt line, so each company decides for itself.

| Crawler owner | Crawl-delay | Source |
| --- | --- | --- |
| Anthropic (ClaudeBot) | Supported | [Anthropic’s crawler page](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) |
| Common Crawl (CCBot) | Obeyed | [Common Crawl FAQ](https://commoncrawl.org/faq) |
| Microsoft (bingbot) | Supported, as a relative throttle | [Bing Webmaster blog](https://blogs.bing.com/webmaster/2009/08/10/crawl-delay-and-the-bing-crawler-msnbot) (2009 post) |
| Google (Googlebot) | Not supported | [Google’s robots.txt rules](https://developers.google.com/search/reference/robots_txt) |
| Apple (Applebot) | Ignored | [Apple’s Applebot page](https://support.apple.com/en-euro/119829) |
| Amazon (Amazonbot) | Not supported | [Amazon’s Amazonbot page](https://developer.amazon.com/amazonbot) |
| OpenAI, Perplexity | Not mentioned on their crawler pages | Their crawler pages |

For the bots that read it, one line per group does the job. A named group replaces your `User-agent: *` rules for that bot, so copy in any rules you still need, as below. Run through the Protego parser (0.7.0), this group gave ClaudeBot a 10-second pause:

```
User-agent: ClaudeBot
Crawl-delay: 10
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
```

Where WordPress keeps robots.txt depends on your setup and SEO plugin; [the guide to AI bot rules in robots.txt](https://skybootstrap.com/robots-txt-ai-bots-wordpress/) shows each place. Fake crawlers ignore Crawl-delay, as they ignore everything else in robots.txt. For those, and for bots that skip Crawl-delay, you need a server-side limit.

## Rate-limit AI crawlers in nginx

An nginx `limit_req` rule keyed to AI crawler names caps each crawler IP address at 2 requests a second, with a burst of 5, and answers extra requests with 429, while people aren’t limited. Rate limiting means capping how many requests one client can make in a period; 429 Too Many Requests is the status for a client that sent too many.

```
# In the http {} block:
map $http_user_agent $ai_bot_limit_key {
default "";
~*(gptbot|claudebot|ccbot|bytespider|perplexitybot|meta-externalagent) $binary_remote_addr;
}
limit_req_zone $ai_bot_limit_key zone=aibots:10m rate=2r/s;
limit_req_status 429;

# In your site's server {} block:
limit_req zone=aibots burst=5 nodelay;
```

This needs access to your nginx config. On shared hosting, ask your host, or use a CDN instead: our guide to Cloudflare’s AI bot controls covers that route. We haven’t tested an Apache or CDN rate limit.

Test the config with `sudo nginx -t` before anything goes live. Only when it says the test is successful, apply it with `sudo systemctl reload nginx`; on an error, take the new lines out again. Requests with an empty key aren’t counted, so anyone whose user agent doesn’t match is never limited.

The list covers the training and search crawlers named in it, not user-triggered fetchers such as Claude-User or ChatGPT-User, which open pages when a person asks; add their names if you want those slowed too. Behind Cloudflare or another proxy, set nginx’s real-IP settings first, or the limit applies to the proxy’s addresses; we didn’t test that setup.

We tested it on our nginx 1.24.0 test site, from one address:

- 50 GPTBot requests, five at a time: 6 got 200, 44 got 429.
- The same burst as ClaudeBot: the same split.
- 50 browser requests, five at a time: all 200.
- GPTBot at one request a second for ten seconds: all 200.

Pick a rate that suits your server. In our test, one request a second from GPTBot was never limited, while a burst of 50 mostly got 429. Some hosting guides quote much higher crawler rates and much lower limits, such as [Scala Hosting’s](https://www.scalahosting.com/kb/?p=5973) suggestion of about 30 requests a minute, without saying where those numbers come from. Start gentle, watch your logs, then tighten.

## When blocking makes sense

Block a crawler outright when it ignores Crawl-delay and keeps hitting uncached pages, after checking that it’s the real bot. A name in the log may be a fake, and blocking a real search crawler costs you visibility. Our guide to blocking Bytespider and CCBot has tested `.htaccess` and nginx rules, and [whether to block AI bots at all](https://skybootstrap.com/should-i-block-ai-bots/) weighs the trade-offs. For the full list of AI bots and what each one does, see the [AI crawler guide](https://skybootstrap.com/ai-crawlers-wordpress/).

## Frequently Asked Questions

### Do AI crawlers slow down WordPress sites?

AI crawlers can slow a site, mostly when they hit uncached pages in volume. On our test site, an uncached page took about 210 ms to come back, against under half a millisecond cached. tlinky.com saw about 243 crawler-named requests a day, which a cached site handles easily. Check your cache before blaming the bots.

### Is 429 or 403 better for AI crawlers?

Use 429 when you want a crawler to slow down and 403 when you want it gone. A 429 from a rate limit tells the crawler it sent too many requests and should try later. A 403 from a block refuses every request, which suits bots you don’t want at all.

### Do fake crawlers add load too?

Yes. On tlinky.com, 291 requests in roughly a week used famous bot names from addresses those companies don’t own. They ignore Crawl-delay and robots.txt, so only caching, rate limits and blocks affect them. The most-copied names were Applebot (62 fakes) and bingbot (60).

## How we tested

- Load: ApacheBench 2.3, 200 requests five at a time with GPTBot’s user agent, three runs each, 28 October 2026.
- Lab install: WordPress 7.1.3, nginx 1.24.0, PHP 8.3, SQLite rather than MySQL, default theme, sample content, two CPU cores.
- Rate limit, cache and curl cache-header checks on the same lab install, 29 October 2026, all from one address, then removed. robots.txt read with Protego 0.7.0.
- Weekly volume: tlinky.com, a live site, counted by Skyboot AI Markdown 3.0.0; the cost figures apply our lab timings to it as an illustration.