Skip to content

Robots.txt Generator: 3 Levels for WordPress

A robots.txt written by hand is usually wrong in one of two directions. It blocks /wp-includes/ and breaks the rendering Google uses to read the page, or it blocks nothing and leaves the admin, the search results and the feeds open to a crawler that has no reason to be there.

Code: WP-robots-generator, MIT licensed.

The generator runs in the browser at robots.globus.studio, understands WordPress paths, and writes a file in a minute. No signup, no backend, nothing sent anywhere.

Level 1: the minimum that stops indexing junk

Low blocks /wp-admin/, /wp-login.php and /readme.html. That is the floor: the login form and the setup notes should never appear in search results.

admin-ajax.php stays allowed at every level, because half the themes and plugins on a real site depend on it. A rule set that blocks it produces a site where forms stop working and nobody connects the two events.

Level 2: the recommended set

Recommended adds /?s=, /search/, /feed/ and /trackback/. Search result pages are infinite, they duplicate the content, and a crawler that finds them spends a budget on nothing.

Feeds are a judgement call. A news site may want them crawlable; a shop rarely does. The level is a default, and the file is editable for a reason.

Level 3: the high level, and what it costs

High adds /wp-includes/, /wp-content/cache/, /wp-json/, /xmlrpc.php, /comments/, /author/ and ?replytocom=.

Two of those need care. Blocking /wp-json/ breaks the block editor on some setups and any REST integration, including plugins that talk to the site. Blocking /wp-includes/ prevents Google from fetching CSS and JS, which damages the rendering it uses to judge the page.

That is why the levels are named rather than numbered. A shop on High and a blog on Recommended are both defensible; a shop on High that also runs a REST integration is a bug report.

The AI crawler switch

GPTBot, ClaudeBot, CCBot, Bytespider, Google-Extended and PerplexityBot can be blocked with a click, per engine or with a wildcard.

The decision is editorial, not technical. Blocking a crawler costs visibility in an answer engine, and allowing it costs the traffic the site already gets from search. Either way the rule belongs in the file rather than in a vague feeling about AI.

Plugin presets

WooCommerce brings /cart/, /checkout/ and /my-account/; Easy Digital Downloads brings a purchase confirmation and its actions; WPML brings language parameter duplicates; WPForms brings handler URLs; BuddyPress brings /members/, /activity/ and /groups/.

Those paths are the ones a generic generator misses. Each preset is a list of paths the plugin actually produces, and that list is why the tool is WordPress-specific.

Check the file after you place it

Open the address in a browser, confirm a 200 and text/plain, then fetch it once with the user agent of a crawler you care about. A file that works for a browser and 403 for a crawler is a hosting rule, and it is invisible from a normal tab.

Then submit the sitemap again in Search Console. A changed robots file is a good moment to ask the search engine to re-read the map.

What the tool does not do

It does not serve the file. Copy the result into robots.txt at the site root, or generate the same rules in the theme and let WordPress answer. It does not check what is already deployed, and it does not monitor a crawler’s behaviour after the change.

Verification stays manual: open https://example.com/robots.txt, compare with what a crawler sees, and keep the sitemap line valid. A wrong sitemap path in a robots file is a silent way to lose a month of indexing.

Related: documentation for the rest of the crawl controls on the site, and botblocker addon for the part that sits before requests reach WordPress.