AgentifyKitBot, our crawler
AgentifyKitBot fetches public pages from a website when someone runs an Agent Readiness Scan or one of our free tools on it. It is not a general web crawler and does not index the web.
How to recognise it
User agent: Mozilla/5.0 (compatible; AgentifyKitBot/1.0; +https://agentifykit.com/bot)
What it fetches
The page that was entered, plus a small number of the site’s own pages linked from it.
Files that tell AI agents about a site: robots.txt, sitemap.xml, llms.txt, agents.md and skills.md.
Structured data and metadata already published in those pages.
It only reads public pages. It never logs in, submits forms or downloads private content.
How it behaves
It runs only when a person requests a scan, fetches pages one at a time and stops after a small page limit.
It respects robots.txt rules for “AgentifyKitBot”.
Blocking it
Add this to your robots.txt:
User-agent: AgentifyKitBot Disallow: /
Contact
If AgentifyKitBot caused a problem on your site, email contact@agentifykit.com and we’ll look into it the same day.