check robots

Parse robots.txt; answer whether a user-agent may crawl a path; list sitemaps. (web-intel; $0.002/call in USDC via x402, 50 free/day).

liveunknown~5s typical

mcpfetcher

$0.0020

per call

plus $0.001 platform fee per run

Open MCP server; Cracked bills routing only

Provider Upstream docs

$Set up https://cracked-ai-pearl.vercel.app/SKILL.md, then use Cracked to run mcp-cloud-fetcher-fetcher /check_robots for me.

Input

Field	Type	Description
host*	string	Target URL or hostname.
path	string
user_agent	string

Call it

curl

curl https://cracked-ai-pearl.vercel.app/v1/run \
  -H "Authorization: Bearer ck_live_..." -H "content-type: application/json" \
  -d '{"provider":"mcp-cloud-fetcher-fetcher","endpoint":"/check_robots","input":{"host":"..."}}'

cli

npx cracked-ai run -p mcp-cloud-fetcher-fetcher -e /check_robots -i '{"host":"..."}'

mcp

run_tool({ provider: "mcp-cloud-fetcher-fetcher", endpoint: "/check_robots", input: {"host":"..."} })

Try it

Runs against your signed-in workspace balance. Sign in if you have not.