Check AI crawler robots.txt policy

Check the homepage robots.txt rule for eight named OpenAI, Anthropic, Perplexity, and Google AI crawler tokens. The free result stays complete and includes a bo

liveverifiedunknown~5s typical

mcpprobedactablesiteaicrawler

$0.0020

per call

plus $0.001 platform fee per run

Open MCP server; Cracked bills routing only

Provider Upstream docs

$Set up https://cracked-ai-pearl.vercel.app/SKILL.md, then use Cracked to run mcp-com-actablesite-ai-crawler-robots-monitor /check_ai_crawler_policy for me.

Input

Field	Type	Description
url*	string	Public HTTP or HTTPS website URL. Private and local-network targets are rejected.

Call it

curl

curl https://cracked-ai-pearl.vercel.app/v1/run \
  -H "Authorization: Bearer ck_live_..." -H "content-type: application/json" \
  -d '{"provider":"mcp-com-actablesite-ai-crawler-robots-monitor","endpoint":"/check_ai_crawler_policy","input":{"url":"..."}}'

cli

npx cracked-ai run -p mcp-com-actablesite-ai-crawler-robots-monitor -e /check_ai_crawler_policy -i '{"url":"..."}'

mcp

run_tool({ provider: "mcp-com-actablesite-ai-crawler-robots-monitor", endpoint: "/check_ai_crawler_policy", input: {"url":"..."} })

Try it

Runs against your signed-in workspace balance. Sign in if you have not.