No, llms.txt doesn’t work in the way most people hope: in 2026 data, AI crawlers rarely fetch it and it shows no effect on AI citations. Of the llms.txt files Ahrefs tracked for a month, 97% were never requested, and SE Ranking found no link between having one and being cited across about 300,000 domains. It still has a narrower use, mainly for documentation that coding agents read.
“Does it work” hides three separate questions: does anything fetch the file, does it get you cited, and does it change rankings. The answers differ, so here they are one at a time, followed by a way to check your own site. If you still want one, adding llms.txt to WordPress takes a few minutes.
Do AI crawlers fetch llms.txt?
AI crawlers rarely fetch llms.txt. 97% of the files in Ahrefs’ May 2026 sample were never fetched once. Two single-site log studies, one on WordPress, saw no AI crawler request for llms.txt in 12 to 14 days. The same crawlers did read robots.txt and pages.
Server logs are the record your web server keeps of every request, including the path and the name the visitor gives itself. They’re the main first-hand way to see who reads the file.
It helps to separate two kinds of AI bot. Training crawlers, such as OpenAI’s GPTBot, collect pages for building models. Search bots, such as OAI-SearchBot, gather the pages an AI search tool can show, and user-triggered fetchers, such as ChatGPT-User, load a page during a conversation. Those last two are the requests that can lead to a citation.
What the logs show:
- Ahrefs, 137,210 sites, May 2026: the roughly 22,000 requests that did happen went to about 1,100 sites, and 96% came from bots. SEO audit tools made 21.7% of them. In Ahrefs’ own categories, AI assistants made 2.5% and AI retrieval bots 1.1%; GPTBot was the busiest AI fetcher at 4.51%.
- Saaslinks, one site, 14 days: 13 requests for llms.txt, none from an AI crawler, while AI crawlers made 723 robots.txt requests and 151 requests to content pages. The site sits behind Cloudflare, so requests answered from its cache wouldn’t show in these logs.
- AWRSHIFT, one WordPress site, 12 days: 39 requests for llms.txt, all from the owner’s own tools. AI crawlers made about 1,000 requests, read robots.txt 181 times, and never asked for llms.txt. OpenAI’s three bots, GPTBot, OAI-SearchBot and ChatGPT-User, made 220 requests between them, none for llms.txt.
The crawlers that matter are busy on these sites. They just go to robots.txt and pages instead.
The requests that do arrive tell you something too. Ahrefs counted about 900 Googlebot fetches. And in May, no AI bot asked for llms.txt on sites that didn’t have one, a sign that AI crawlers aren’t even probing for the file. Core dna’s summary of OtterlyAI’s data puts llms.txt at about 0.1% of more than 60,000 AI bot visits over 90 days; we couldn’t see the original, so treat it as indicative.
Does llms.txt get you cited in AI answers?
No measurable effect so far: across about 300,000 domains, SE Ranking found that having llms.txt made no difference to AI citations. An AI citation is a page named as a source in an answer from a tool like ChatGPT or Perplexity. SE Ranking tested the link three ways (a rank correlation, a machine-learning model, and a feature-importance analysis), and its model got more accurate when the llms.txt variable was removed.
The study has limits: it doesn’t name the AI platforms it sampled or publish its coefficients, and it calls its result “context-dependent”. Still, it’s the largest citation test we found, and none of the other 2026 studies we reviewed points the other way.
The log studies explain why. A file can’t influence an answer if the AI system never reads it, and in those logs the AI bots that did visit went to pages, not llms.txt. Logs collected by GoodBarber showed ChatGPT, Claude and Perplexity fetching ordinary HTML pages when they browsed, while coding agents and some crawlers asked for Markdown.
What nobody outside these companies knows is whether any AI product reads llms.txt files indirectly, through a third-party crawl rather than from your server. No vendor has said so, and no study can measure it.
Does Google use llms.txt?
Google Search doesn’t use llms.txt, and Google says the file will neither help nor harm rankings; Lighthouse checking for it is a separate developer audit. Google’s guide to its AI features says no special AI files are needed, and that llms.txt files “will neither harm nor help your site’s visibility or rankings”. Googlebot does fetch the file sometimes (Ahrefs counted about 900 requests in May), but fetching a file isn’t the same as using it in ranking.
Lighthouse, the page-audit tool in Chrome, added an llms.txt check in April 2026. It checks only that a file it finds looks roughly right (a # line, a link, 50 characters), skips sites with no file, and passing it does nothing for your Google rankings. It also misses problems the spec’s own parser catches, which is why a full llms.txt check runs both.
What the 2026 studies found, side by side
The independent 2026 studies that measured requests or citations all point the same way; the others add adoption and vendor numbers.
| Study | Sample | What it measured | Finding | Limits |
|---|---|---|---|---|
| Ahrefs (June 2026) | 137,210 sites, May 2026 | Requests to llms.txt | 97% of files got none; AI retrieval bots made 1.1% of requests | One month; one company’s sample |
| SE Ranking (Oct 2026) | ~300,000 domains | AI citation frequency | No effect of llms.txt | Platforms not named; no coefficients |
| Saaslinks (Aug 2026) | 1 site, 14 days | Server logs | 0 AI crawler requests for llms.txt | One site; behind Cloudflare, so cached requests may be missing |
| AWRSHIFT (Jul 2026) | 1 WordPress site, 12 days | Server logs | 0 AI crawler requests; ~1,000 AI crawler hits elsewhere | One site |
| Originality.ai via PPC Land (Jul 2026) | 3M+ monitored sites | Adoption | llms.txt files rose from 4,088 (June 2025) to 36,120 (May 2026) | Counts files, not use |
| Mintlify (2025, vendor) | 25 docs sites, 7 days | Visits | Median 14 visits to llms.txt, 79 to llms-full.txt | Sells the feature; docs sites only |
Adoption kept climbing, but most files still went unread. One likely reason is SEO plugins that publish the file from one setting (AIOSEO’s documentation says its file is on by default). That makes it easy to publish a file nobody has checked. In Common Crawl’s July 2026 crawl, only 22.32% of attempted llms.txt addresses returned “200 OK”, and only 52.5% of those were text files at all. If your file might be one of the broken ones, the validator guide above shows how to check it.
Among these studies, the only numbers showing regular use come from Mintlify’s documentation customers. That fits what the llms.txt proposal itself says: the files are used most heavily for software documentation, by coding agents. The larger companion file gets its own comparison in our llms-full.txt explainer.
Check your own logs
Download two to four weeks of access logs from your host and count requests to /llms.txt by user agent. The script below does it for standard nginx and Apache logs.
A user agent is the name a client sends with each request, such as GPTBot or ClaudeBot. It’s self-reported, so treat matches as likely rather than proven.
- Get the logs. On cPanel hosts, open Raw Access in the Metrics section and download the archives. Managed WordPress hosts usually offer logs in their dashboard or on request. On a server you run yourself, nginx and Apache on Debian or Ubuntu keep them in
/var/log/nginx/and/var/log/apache2/. - Save the script as
llms_log_check.pynext to the log files. - Run
python3 llms_log_check.py *.gzfor cPanel downloads, orpython3 llms_log_check.py access.log*for server logs. You need Python 3; on Windows, install it from python.org and typepyinstead ofpython3, followed by each file name.
#!/usr/bin/env python3
"""Who asked for llms.txt? Reads nginx/Apache "combined" access logs (plain or .gz).
Usage: python3 llms_log_check.py /var/log/nginx/access.log*
"""
import gzip
import re
import sys
from collections import Counter, defaultdict
LINE = re.compile(r'"(?:GET|HEAD) (?P<path>\S+) [^"]*" (?P<status>\d{3}) \S+ "[^"]*" "(?P<ua>[^"]*)"')
BOTS = ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "Claude-User", "Claude-SearchBot",
"PerplexityBot", "Perplexity-User", "Google-CloudVertexBot", "Googlebot", "bingbot", "Applebot",
"Amazonbot", "meta-externalagent", "Bytespider", "CCBot", "DuckAssistBot", "MistralAI-User"]
def bot_name(ua):
for b in BOTS:
if b.lower() in ua.lower():
return b
return "other"
def kind(path):
p = path.split("?")[0].lower()
if p.endswith("/llms.txt"):
return "llms.txt"
if p.endswith("/llms-full.txt"):
return "llms-full.txt"
if p.endswith("/robots.txt"):
return "robots.txt"
if p.endswith(".md"):
return ".md"
return "pages"
if len(sys.argv) < 2:
sys.exit("Usage: python3 llms_log_check.py LOGFILE [LOGFILE ...]")
counts = defaultdict(Counter)
lines = matched = skipped = 0
for name in sys.argv[1:]:
opener = gzip.open if name.endswith(".gz") else open
with opener(name, "rt", errors="replace") as f:
for line in f:
lines += 1
m = LINE.search(line)
if not m:
continue
matched += 1
if m["status"] not in ("200", "304"):
skipped += 1 # 404s, redirects and errors: the file wasn't served
continue
counts[bot_name(m["ua"])][kind(m["path"])] += 1
cols = ["llms.txt", "llms-full.txt", "robots.txt", ".md", "pages"]
print(f"{'requester':24}" + "".join(f"{c:>14}" for c in cols))
for bot in sorted(counts, key=lambda b: -sum(counts[b].values())):
print(f"{bot:24}" + "".join(f"{counts[bot][c]:>14}" for c in cols))
print(f"\n{lines} lines read, {matched} GET/HEAD requests parsed, {skipped} skipped (not 200/304).")
if lines and not matched:
print("No lines matched: this log isn't in the combined format, so the table above is empty.")How to read your result:
- An llms.txt column of zeros is normal: it matches every independent study above, so it isn’t a sign your file is broken.
- Compare it with the robots.txt column: if the same bots read robots.txt and pages but skip llms.txt, they can reach you and simply don’t use the file.
- A fetch isn’t a citation: a GPTBot count shows the file was served to that bot, not that it changed any answer. The script only counts successful responses (200 or 304), and its last line tells you if your log format didn’t match.
- No AI bots in any column: check that robots.txt and any security plugin or firewall aren’t blocking them before drawing conclusions.
Here’s the output format, from a small test log we made up to check the script (these numbers are not real bot data):
requester llms.txt llms-full.txt robots.txt .md pages
ClaudeBot 0 0 1 0 1
GPTBot 1 0 0 0 0
OAI-SearchBot 0 0 0 1 0
PerplexityBot 0 1 0 0 0
other 1 0 0 0 0
8 lines read, 7 GET/HEAD requests parsed, 1 skipped (not 200/304).Two things can hide requests.
If your site sits behind Cloudflare or another CDN, requests it answers from its cache never reach your server’s log, so check the CDN’s own analytics too. A WordPress page-cache plugin serves from your own server, so those requests are still logged; a cache your host runs in front of the server may not be, so ask your host. Caches cause a second problem for AI tools too: a cache that ignores Vary: Accept can serve agents the wrong format.
To watch AI crawlers over time without downloading logs, our Skyboot AI Markdown plugin has an AI Analytics screen. It counts IP-verified AI crawler visits and visitors from AI assistants, page by page. It has the same blind spot for CDN-cached requests, and its analytics docs don’t say whether /llms.txt requests are listed, so the log check above stays the way to answer that question.
Frequently Asked Questions
Do AI crawlers actually read llms.txt?
Rarely. In Ahrefs’ May 2026 sample, 97% of llms.txt files were never fetched once, and two single-site log studies saw no AI crawler request for the file in 12 to 14 days. The same crawlers did read robots.txt and ordinary pages, so they reached the sites and simply skipped llms.txt.
Does Google use llms.txt for rankings?
No. Google Search doesn’t use llms.txt, and Google’s guide to its AI features says such files will neither help nor harm a site’s visibility or rankings. Lighthouse checking for llms.txt is a separate developer audit, not a ranking signal, so a failing Lighthouse llms.txt check has no effect on search.
Is llms.txt worth keeping at all?
Keep a small, accurate llms.txt if you publish documentation or want coding agents to find your key pages, since that is where the file sees most use. Don’t expect more citations: SE Ranking found no link between having one and being cited across about 300,000 domains. Setting it up takes minutes and does no harm.
Does llms.txt work for your site? When to keep it
Keep a small, accurate llms.txt if you publish documentation or want coding agents to find your key pages, but don’t expect it to raise citations or rankings. A practical split:
- Plugin, theme or API docs: keep it, and keep it current. Documentation is where the llms.txt proposal says the files are used most, by coding agents, and the agents that ask for Markdown can also read your pages directly in that format. Our walkthrough on giving agents Markdown versions of your pages covers that.
- Blogs, stores and local business sites: optional. It costs little if a plugin maintains it, whether that’s your SEO plugin or ours, but don’t spend time on it before the basics.
- Anywhere it’s wrong or stale: fix or remove it. A file listing old URLs or test pages is worse than none: any agent that reads it gets sent to the wrong pages.
To remove a plugin-generated file, switch the llms.txt option off in your SEO plugin; Rank Math, Yoast and AIOSEO each keep it in a different place, and the hub guide above shows where. Then save your permalinks under Settings → Permalinks. In our lab, a deactivated plugin’s rewrite rule kept answering /llms.txt with the homepage until we did.
The effort pays off elsewhere, in the work that already earns links and search traffic. That means pages that answer a question better than the alternatives, in crawlable HTML, with robots.txt letting AI search crawlers in.