← SEO Tools
seo by fourzerotwo
robots.txt check
Whether a crawler (Googlebot, GPTBot, ClaudeBot…) may fetch a URL, the rule that decided it, and declared sitemaps.
robots.txt check
Whether a crawler (Googlebot, GPTBot, ClaudeBot…) may fetch a URL, the rule that decided it, and declared sitemaps.
- price
- $0.002 / call
- free tier
- 3 free / day
- MCP tool
- seo__robots-check
- method
- GET
GET https://fourzerotwo.si/v1/seo/robots-check
| Param | Type | Description |
|---|---|---|
| url | string | URL |
| user_agent? | string | Crawler user agent |
# unpaid: returns 402 with the price curl -i "https://fourzerotwo.si/v1/seo/robots-check?url=https%3A%2F%2Fwww.bbc.co.uk%2Fnews" # free tier: send a free key from https://fourzerotwo.si/free curl -H "X-API-Key: fzt_…" "https://fourzerotwo.si/v1/seo/robots-check?url=https%3A%2F%2Fwww.bbc.co.uk%2Fnews"
Use it from an AI agent (MCP)
Add the server once, then ask for this data in plain English. The agent finds the endpoint with search_apis and runs it with call_api.
{
"mcpServers": {
"fourzerotwo": { "type": "http", "url": "https://fourzerotwo.si/mcp" }
}
}
# the call the agent makes
call_api {"tool": "seo__robots-check", "arguments": {"url":"https://www.bbc.co.uk/news","user_agent":"GPTBot"}}
More from SEO Tools
- On-page SEO audit — Title, meta, H1, canonical, noindex, hreflang, structured data, OG tags, word count, links, alt text, issues a...
- Extract links — Internal and external links on a page with anchor text and nofollow flags.
- Keyword intent — Classify a search keyword as transactional, commercial, navigational or informational.
Analysis by fourzerotwo.