AgentIndexBot Crawler Policy

AgentIndexBot crawls publicly accessible agent endpoints to build an open index of agentic AI services (MCP servers, A2A agents, x402-gated APIs, ARD catalogs).

What we fetch

• /.well-known/ai-catalog.json — ARD catalog (once daily)
• /.well-known/agent-card.json, /.well-known/agent.json — A2A agent cards (once daily)
• /.well-known/mcp.json — MCP manifest (once daily)
• /mcp — MCP JSON-RPC initialize handshake (once daily, only if robots.txt permits)
• Common API paths for x402 detection (only if robots.txt permits)

How to opt out

Add a Disallow rule for AgentIndexBot in your /robots.txt:

User-agent: AgentIndexBot
Disallow: /

We honour robots.txt and cache it for 24 hours. Well-known paths (/.well-known/*) may still be fetched once as part of ARD/A2A/MCP standard discovery — these are standard agent-ecosystem paths that the protocols themselves define as public.

You can also email [email protected] to request immediate removal from the index.

Our bot

User-Agent: AgentIndexBot/0.1 (+https://agents.traderszone.net)

Crawl frequency: once daily (03:30 UTC). We back off 6 hours on 403 or 429 responses.

← AgentIndex home