Glintbase bots
Glintbase sends requests under two names. Both say who they are; neither pretends to be a browser or another company's crawler.
GlintbaseScanner
The Agent Readiness Score scanner. Requests come from a scan someone asked for (on scan.glintbase.dev, in the @glintbase/cli command line tool or through the Glintbase MCP server) or from the weekly re-check of companies on the public leaderboard.
Mozilla/5.0 (compatible; GlintbaseScanner/3.5; +https://glintbase.dev/research/bot)
What a scan requests
- The homepage, robots.txt, sitemap.xml, llms.txt and llms-full.txt, and up to 8 pages that llms.txt links to.
- The homepage again with Accept: text/markdown, its .md version on the same host, and one random path that should return 404 (to check error handling).
- Well-known discovery files such as /.well-known/ai-catalog.json, api-catalog, ard.json, oauth-protected-resource and poppy.json, and OpenAPI documents at standard paths or wherever the site links to them.
- Possible MCP endpoints on the site and its docs., api. and mcp. subdomains. Where one answers: protocol metadata (server/discover or initialize, tools/list), one call to a tool that does not exist, and the server card.
- A few lookups about the brand on third-party registries (the MCP Registry, npm, Wikidata, skills.sh), not on your site.
Volume: about 90 to 140 requests per scan to the site and its subdomains, over roughly 15 to 30 seconds. The hosted scanner allows 5 scans per hour per IP address.
robots.txt
A scan is made on behalf of the person who asked for it, so the scanner reads robots.txt as part of the score but doesn't treat rules aimed at its own name as an opt-out. To have your site removed from the weekly leaderboard re-checks, email research@glintbase.dev.
GlintbaseResearch
Glintbase's research harness. We run small, pre-registered measurement studies of how public AI-agent infrastructure behaves for software agents.
GlintbaseResearch/1.0 (+https://glintbase.dev/research/bot; research@glintbase.dev)
What it requests
Protocol metadata from MCP endpoints listed in the official MCP Registry: server/discover or initialize, tools/list, one call to a tool that does not exist (to check error handling), and OAuth protected-resource metadata if the endpoint requires auth.
At most 8 requests per endpoint per round, one endpoint per host at a time, two rounds.
robots.txt and sitemap from 150 popular websites, then the homepage and up to 3 public pages listed in the sitemap, fetched three ways for comparison: as GlintbaseResearch (which obeys robots.txt for its own name), as a plain HTTP client with its library's default user agent, and as a headless Chromium browser with its default (HeadlessChrome) user agent. Page bodies are not stored.
About 12 page requests per site per round, at most 1 per second, with a 30-second pause between client types; two rounds.
Opting out of research
Email research@glintbase.dev with the host name(s). We add them to the study's opt-out list and stop within 24 hours. You can also block the GlintbaseResearch user agent in robots.txt.
What neither bot does
- Call your real tools or send data intended to change state. The scanner calls a tool only if the server marks it read-only and it takes no arguments.
- Send security payloads, attempt authentication, or try to bypass challenges, CAPTCHAs or rate limits.
- Pretend to be a person's browser or another company's crawler. (The research harness's comparison clients use their own default identifiers: a library user agent or HeadlessChrome.)
- Log in, fill in forms, or fetch pages that are not public.
What we publish
Scanner results appear as scores and check results on the public leaderboard and score pages. Research publishes aggregate statistics plus per-endpoint derived fields, such as “handshake succeeded”. We never publish raw responses, and research never publishes lists naming the worst servers.