Read robots.txt

Fetch and parse robots.txt: user-agents, allow/disallow rules, sitemaps.

liveverifieddegraded · 177ms

robotsseocrawl

$0.0020

per call

plus $0.001 platform fee per run

Provider

$Set up https://cracked-ai-pearl.vercel.app/SKILL.md, then use Cracked to run web /robots for me.

Input

Field	Type	Description
url*	string	Site root URL

Call it

curl

curl https://cracked-ai-pearl.vercel.app/v1/run \
  -H "Authorization: Bearer ck_live_..." -H "content-type: application/json" \
  -d '{"provider":"web","endpoint":"/robots","input":{"url":"..."}}'

cli

npx cracked-ai run -p web -e /robots -i '{"url":"..."}'

mcp

run_tool({ provider: "web", endpoint: "/robots", input: {"url":"..."} })

Try it

Runs against your signed-in workspace balance. Sign in if you have not.