See what a crawler finds on your page.
Check one public domain or URL. You get four clear answers and concrete next steps.
Fetching and checking the page safely …
The check could not be completed.
Crawlers can access it
Indexing allowed
Main content detectable
Observed crawler rules
Search, training and a user-triggered fetch are separate jobs. This table shows only what the observed robots.txt says for each product token.
| Crawler | Purpose | Access |
|---|
An allowance does not prove a visit. Blocking GPTBot does not automatically block OAI-SearchBot for ChatGPT Search. Review roles and sources →
What to do next
Technical details
Take the result with you or compare it later
Up to five results stay only in this browser and are discarded after 24 hours.
What changed since the previous check?
Compare the response for another crawler token
AnalyseSpider sends one second bounded request with the selected product token in the User-Agent. This simulates a header, not the identity or full behaviour of the real crawler.
Observed differences
This check examines only the server response for this one URL. No logged-in content, no JavaScript rendering, and no guarantee of Google indexing or AI citations.
Check the whole website
Use a full technical crawl when you need to find and prioritise problems across many pages.
Open Contextter Site AuditCheck questions and sources in AI search
See which follow-up questions and sources appear in real OpenAI and Gemini queries.
Open AI FanoutAnalyseSpider, Contextter and AI Fanout are operated by Matthias Ramahi and are not independent recommendations.
How the check works
- Check a URLEnter one public domain or URL and start the check.
- Read the crawler responseWe fetch the server response like a simple HTTP crawler and explain it.
- Act on the resultYou get a few concrete pointers instead of an invented overall score.
Crawlers in plain language
What is a crawler, and what can this check actually show?What do you want to check?
What does a crawler find on my URL?Check fetching, access, indexing and detectable main content.Where does this redirect lead?Assemble each observed redirect step into one clear chain.Can this bot access my page?Test one robots.txt rule for a specific crawler.What do my server logs show?Find status codes, paths and claimed bot names locally in your browser.
Understand crawlers
Googlebot, ChatGPT and Claude do different jobs.
Some bots build search indexes. Others fetch a page for a user or collect training data. Our comparison shows the job, the relevant robots.txt control and how to verify real requests.
- GooglebotGoogle Search
- BingbotBing Search
- OAI-SearchBotChatGPT search
- GPTBotModel training
- Claude-SearchBotClaude search
Knowledge for a real decision.
Each article answers one practical question step by step. No thin glossary pages and no invented SEO promises.
How can a page appear in AI search?What OpenAI, Google and Anthropic actually require — and why eligibility is never a guarantee.How do I find search bots in server logs?Build a candidate set, verify important identities and keep the evidence boundary visible.What does a 301 response not prove?Separate one observed redirect from the final destination, indexing and canonical decisions.
Need to inspect the whole website?
The free check here examines exactly one URL. Contextter Site Audit crawls many pages, connects the findings and helps you prioritise them.
AnalyseSpider and Contextter are operated by Matthias Ramahi. This is a related next step, not an independent recommendation.
An old domain, a new website.
AnalyseSpider was formerly used for log analysis and IP tools. Today's website is a new implementation under new ownership. Former software, copy, data and customers did not transfer.
Read about the new ownership