Robots.txt

Fetch + parse the target domain's robots.txt — sitemaps, per-User-agent allow/disallow rules, crawl-delay, Host directive. Use BEFORE crawling/scraping a target

liveverifiedunknown~5s typical

mcpprobedcontrastapi

$0.0020

per call

plus $0.001 platform fee per run

Open MCP server; Cracked bills routing only

Provider Upstream docs

$Set up https://cracked-ai-pearl.vercel.app/SKILL.md, then use Cracked to run mcp-com-contrastcyber-api /robots_txt for me.

Input

Field	Type	Description
domain*	string	Registrable domain to fetch robots.txt for (e.g. 'example.com', 'github.com'). No scheme,

Call it

curl

curl https://cracked-ai-pearl.vercel.app/v1/run \
  -H "Authorization: Bearer ck_live_..." -H "content-type: application/json" \
  -d '{"provider":"mcp-com-contrastcyber-api","endpoint":"/robots_txt","input":{"domain":"..."}}'

cli

npx cracked-ai run -p mcp-com-contrastcyber-api -e /robots_txt -i '{"domain":"..."}'

mcp

run_tool({ provider: "mcp-com-contrastcyber-api", endpoint: "/robots_txt", input: {"domain":"..."} })

Try it

Runs against your signed-in workspace balance. Sign in if you have not.