Edit robots.txt

robots.txt is the file at the root of your site that tells crawlers which paths they may fetch. WordPress generates it on the fly (a "virtual" robots.txt). RankKit SEO adds three things to it: your sitemap, a block for each AI crawler you switch off, and any extra rules you type.

Everything is managed on one settings tab, and the file updates as soon as you save.

Open the settings

Go to RankKit SEO → Settings → Robots & AI crawlers. The tab has two cards: AI crawlers and robots.txt. Click View on the robots.txt card to open your live file.

Allow or block AI crawlers

"Choose which AI companies may crawl your content. Blocking training bots does not remove you from AI search answers."

Each crawler has an Allowed / Blocked switch. All are allowed by default.

Bot	What it is
GPTBot	OpenAI training
OAI-SearchBot	ChatGPT search
ChatGPT-User	ChatGPT browsing
ClaudeBot	Anthropic training
Claude-SearchBot	Claude search
Claude-User	Claude browsing
PerplexityBot	Perplexity search
Perplexity-User	Perplexity browsing
Google-Extended	Gemini training
Applebot-Extended	Apple AI training
Bytespider	ByteDance
CCBot	Common Crawl
Meta-ExternalAgent	Meta AI
Amazonbot	Amazon
cohere-ai	Cohere

The three kinds of bot work differently:

• Training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and similar) collect pages to train AI models. Blocking them keeps your content out of future training sets.
• Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) index pages so AI search can cite you. Block these and you are less likely to appear in AI answers.
• Browsing (user) bots (ChatGPT-User, Claude-User, Perplexity-User) fetch a page live because someone asked an assistant about it.

Tip A common setup is to block training bots and keep search and browsing bots allowed. You stay visible in ChatGPT, Claude and Perplexity answers without contributing content to model training. Google-Extended only controls Gemini training: Googlebot and Google Search are not affected.

For each blocked bot, RankKit SEO writes:

# RankKit SEO: AI crawlers blocked by the site owner
User-agent: GPTBot
Disallow: /

To see which AI bots actually visit your site, use AI crawler tracking.

Add your own rules

Use Extra rules on the robots.txt card. "Appended to the virtual robots.txt. The sitemap line is added for you."

User-agent: *
Disallow: /checkout-test/
Disallow: /*?add-to-cart=

User-agent: SemrushBot
Crawl-delay: 10

Click Save changes, then View to check the result.

To replace the whole file instead, or to edit a real robots.txt file in your site's root folder, use the file editor in RankKit SEO → Tools.

What your robots.txt looks like

The final file is, in order:

1. WordPress's default rules: User-agent: * with Disallow: /wp-admin/ and Allow: /wp-admin/admin-ajax.php, plus the Sitemap: line WordPress adds for its core sitemaps.
2. A block per AI crawler you switched off.
3. Your extra rules.
4. Sitemap: https://example.com/wp-sitemap.xml, only if XML sitemaps are on and no Sitemap line is there yet.

With RankKit SEO Pro, the news and video sitemaps are listed too when they are on.

Troubleshooting

Problem	Cause	Fix
Your changes don't show up	A physical robots.txt file exists in the site's root folder. A real file always overrides the virtual one.	Edit the real file in Tools → File editor, or click Move the file aside there so RankKit SEO manages robots.txt again. See File editor.
robots.txt only says Disallow: /	Settings → Reading → Discourage search engines from indexing this site is ticked. RankKit SEO then leaves robots.txt as WordPress writes it.	Untick it when the site goes live.
An old version still shows	A page cache or CDN caches /robots.txt.	Purge the cache for that URL.

Important Don't block CSS, JavaScript or /wp-content/ in Extra rules. Google needs them to render your pages, and blocking them can hurt rankings.

FAQ

Does robots.txt remove pages from Google?

No. It stops crawling, not indexing. A blocked URL can still appear in results if other sites link to it. To keep a page out of search, use noindex instead. See Noindex, nofollow and canonical URLs.

Do all AI companies respect robots.txt?

robots.txt is a convention, not an enforcement tool. The major AI companies document their crawler names and say they follow it, but a crawler that ignores it isn't stopped by it. Crawlers that fetch a page because a user asked for it may be treated differently by their operators.

Will blocking GPTBot remove me from ChatGPT?

No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot and live browsing uses ChatGPT-User, which you control with their own switches.