Control and track AI crawlers

AI companies crawl the web for two reasons: to train models, and to answer questions in AI search and chat. RankKit SEO lets you decide which of their crawlers may read your site, and shows you which ones actually do.

Both parts are free. Control lives in your robots.txt settings; tracking lives on the AI Visibility page.

Allow or block AI crawlers

Go to RankKit SEO → Settings → Robots & AI crawlers and use the Allowed / Blocked switch for each bot. All 15 are allowed by default. RankKit SEO adds a Disallow: / rule to your robots.txt for each one you block.

The full list, what each bot does and a recommended setup are in Edit robots.txt. The short version: blocking training bots does not remove you from AI search answers; blocking search and browsing bots does.

See which AI crawlers visit

Go to RankKit SEO → AI Visibility and open the AI crawlers tab. This tab is free, even though the AI Visibility menu item shows a PRO badge for its other tabs.

Tracking is on by default. If you see "AI crawler tracking is off", go to RankKit SEO → Dashboard → Modules and turn on AI crawler tracking.

What the report shows

Section	What it tells you
AI crawler visits	Total visits from AI bots in the date range, compared with the period before.
Different bots	How many distinct AI crawlers visited.
Visits for user questions	Pages "fetched live to answer someone": a person asked an assistant something and it read your page to answer.
llms.txt / Markdown reads	"AI-ready files fetched": requests for your llms.txt and Markdown pages.
Chart	Visits per day.
By crawler	Each bot with its Purpose (Training, AI search or User request), Visits, Pages, Last seen, and a "blocked" label if you block it in robots.txt.
Most-crawled pages	The pages AI bots read most.

Use Allow or block AI crawlers at the bottom to jump to the robots.txt settings.

Bots RankKit SEO recognises

GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, PerplexityBot, Perplexity-User, Google-CloudVertexBot, Gemini-Deep-Research, Applebot-Extended, Applebot, meta-externalagent, meta-externalfetcher, Bytespider, Amazonbot, DuckAssistBot, MistralAI-User, cohere-ai, CCBot, YouBot, and Bingbot (counted as Microsoft Copilot).

History and data kept

Choose 7 days, 28 days (default) or 90 days at the top of the report. RankKit SEO keeps 90 days of visits, on Free and Pro, so you can spot trends, for example whether ChatGPT-User visits grow after you publish a new guide.

Visits are stored as one small summary row per bot, page and day (capped at 50,000 rows), so tracking stays light. Human visits are not recorded.

RankKit SEO Pro RankKit SEO Pro adds AI visibility tracking: whether AI answers mention and cite you. See plans.

Troubleshooting

Problem	Cause	Fix
"No AI crawler visits yet"	The site is new, or no bot has visited since tracking started. Most sites see GPTBot, ClaudeBot or PerplexityBot within days of being published.	Wait a few days.
Far fewer visits than your server logs show	Visits are counted when WordPress handles the request. Visits served entirely by a page cache or CDN are not seen.	Expect lower numbers on heavily cached sites. Server or CDN logs show the full count.
A blocked bot still appears	It visited before you blocked it, or it doesn't follow robots.txt.	Check Last seen. Crawlers can take a while to re-read robots.txt; regular visits long after blocking mean the bot ignores your rules.

FAQ

Is blocking AI crawlers good for SEO?

It has no effect on Google Search rankings. Googlebot is separate from Google-Extended, which only controls Gemini training. Blocking AI search and browsing bots does reduce your chances of being cited in ChatGPT, Claude and Perplexity answers.

Why does the AI Visibility page say PRO if the crawlers tab is free?

The page has several tabs. AI answers (AI visibility tracking) and AI revenue (AI referral revenue for WooCommerce) are Pro. The AI crawlers tab is free.

Does tracking slow my site down?

No. It adds one small database write for requests from known AI bots, and nothing for human visitors.