SEO Guide

robots.txt Directives Explained: User-agent, Allow, Disallow, Crawl-delay and Sitemap

By ToolFlic Expert Team Updated 9 October 2026 1054 words

A robots.txt file is a plain-text file at https://yourdomain.com/robots.txt that tells well-behaved crawlers which paths they should not request. It uses a handful of directives grouped by user-agent. The single most important thing to understand is that robots.txt is a request to polite bots, not a lock - it controls crawling, not indexing, and it is the wrong tool for hiding anything private.

This page explains each directive, how path matching really works, and the traps that block CSS, open up admin folders, or leak URLs anyway. Build a correct file with the free robots.txt generator and cross-check your crawl budget against the broken link checker.

Where robots.txt lives and what it controls

The file must sit at the root of your domain, exactly one per host and protocol: https://example.com/robots.txt. Subdomains get their own. Crawlers fetch it before crawling anything on that host.

robots.txt governs crawling (whether a bot may fetch a URL). It does not control indexing: a page blocked in robots.txt can still appear in search results if other pages link to it, and because the bot cannot fetch the page it cannot see a noindex tag on it. To actually keep a page out of the index, let it be crawled and add a noindex meta tag or an X-Robots-Tag header - do not use robots.txt for that.

The directive anatomy: user-agent, allow, disallow

A robots.txt file is a set of groups. Each group starts with a User-agent: line naming which bots the rules apply to, followed by the rules for that bot. A blank line separates groups.

Directive	Meaning	Example
User-agent	Which bot the next rules apply to; * = all bots	User-agent: *
Disallow	Path the bot must not request	Disallow: /wp-admin/
Allow	Exception re-permitting a path (paired with a broad Disallow)	Allow: /wp-admin/admin-ajax.php
Sitemap	Absolute URL of an XML sitemap (may be on another host)	Sitemap: https://example.com/sitemap.xml
Crawl-delay	Seconds between requests; honoured by some bots, not Google	Crawl-delay: 1

The classic pair on a WordPress site is Disallow: /wp-admin/ with Allow: /wp-admin/admin-ajax.php - block the admin area but let the one endpoint that many themes rely on through.

Matching rules: prefixes, the $ anchor and * wildcards

Paths are matched as prefixes, not exact strings: Disallow: /private also blocks /private.html and /private/report.pdf, because they all start with "/private". Put the trailing slash deliberately - Disallow: /articles catches /articles, /articles-old and /articles/123, while Disallow: /articles/ only catches the folder and its contents.

Two special characters refine matching: * matches any run of characters (so Disallow: /*.pdf$-style patterns or /tmp* work), and $ anchors the end of the URL, so Disallow: /*.gif$ blocks URLs that end in .gif but not .gif?v=2. When both an Allow and a Disallow match a URL, the longest (most specific) path wins, regardless of order - that is how the admin-ajax exception survives the broad /wp-admin/ block.

Crawl-delay, sitemap lines and comments

Crawl-delay asks a bot to wait N seconds between hits. Google explicitly ignores it (it self-throttles), but Bing and several AI/general crawlers may honour it, so it is a reasonable hammering guard for those agents. Comments start with # and run to end of line.

The Sitemap: line is independent of the user-agent groups - it can appear anywhere and, unlike every other directive, it may point at a different host. List all your maps there so bots that read robots.txt still find them.

The mistakes that silently hurt you

Most robots.txt damage is accidental. The recurring ones:

Blocking your assets. A broad Disallow: / on * during development is easy to forget and ships live, hiding the whole site. Even more subtle: blocking /wp-content/, the CSS folder or the images folder stops Google rendering pages and can hurt ranking, because the crawler cannot load the page the way a user sees it.

Using it for privacy. robots.txt is public and Disallow: lines are a published list of what you wanted to hide; anything secret belongs behind authentication or a noindex, never only in robots.txt.

Spaces and case. Directive names are case-sensitive (Disallow, not disallow), and stray spaces around the colon break parsing on some crawlers. Also remember an empty Disallow: means allow everything, not block everything - the opposite of what many people expect.

Test before you trust it

Write the rules, then verify behaviour rather than assuming. Fetch https://yourdomain.com/robots.txt in a plain browser to confirm it exists and is served as text (not a 404 or an HTML error page). In Search Console the Robots testing report and URL Inspection show whether a specific URL is blocked by your file, and the broken link checker confirms you have not blocked paths that your own pages still link to.

Frequently Asked Questions

Can robots.txt keep a page out of Google?

Not reliably, and often counter-productively. Blocking a URL in robots.txt stops Google fetching it, which also stops it seeing a noindex tag - so the page can still get indexed from links pointing to it, without you ever learning it is there. To keep something out of the index, let it be crawled and return a noindex directive, or protect it with a password.

What does Disallow: / mean?

It blocks the whole site for that user-agent group. Combined with User-agent: * it tells every polite crawler to fetch nothing. That is fine for a staging site but catastrophic if it ships to production, and it is the most common robots.txt accident.

Does Google respect Crawl-delay?

No, Google ignores Crawl-delay and decides its own pace. Bing and a number of other and AI crawlers may honour it, so it can still be useful to protect a slow server from those bots.

Allow versus Disallow - which wins?

The most specific (longest) matching path wins, not the one that appears first. That is why Disallow: /wp-admin/ with Allow: /wp-admin/admin-ajax.php lets the one ajax endpoint through.

Where should the file live?

One file at the root of each host and protocol, at /robots.txt, served as text. Subdomains need their own. A missing robots.txt is perfectly fine - crawlers simply assume everything is allowed - so you do not need a file unless you want to restrict something.

Do I need robots.txt at all?

Only if you want to block something or advertise a Sitemap. An empty or absent robots.txt means allow-all, which is the right default for most sites. Add rules deliberately, not defensively.