Skip to content

AgentifyKitBot, our crawler

AgentifyKitBot fetches public pages from a website when someone runs an Agent Readiness Scan or one of our free tools on it. It is not a general web crawler and does not index the web.

How to recognise it

User agent: Mozilla/5.0 (compatible; AgentifyKitBot/1.0; +https://agentifykit.com/bot)

What it fetches

The page that was entered, plus a small number of the site’s own pages linked from it.

Files that tell AI agents about a site: robots.txt, sitemap.xml, llms.txt, agents.md and skills.md.

Structured data and metadata already published in those pages.

It only reads public pages. It never logs in, submits forms or downloads private content.

How it behaves

It runs only when a person requests a scan, fetches pages one at a time and stops after a small page limit.

It respects robots.txt rules for “AgentifyKitBot”.

Blocking it

Add this to your robots.txt:

User-agent: AgentifyKitBot
Disallow: /

Contact

If AgentifyKitBot caused a problem on your site, email contact@agentifykit.com and we’ll look into it the same day.