See what crawlers see on your website.

Check redirects, robots.txt, HTTP responses and server logs. Free, without an account and without uploading your input.

Choose a free tool Learn how crawling works

Example: check one URLNot a live fetch

Status: 200 OKThe page returned a successful response.
Robots: index, followThe page allows indexing.
Canonical: Self-referencingNo conflict in this example.
Redirect: No redirectThe URL stays the same.

What do you want to check?

Why is my page not indexed?Check the status, robots directives and canonical.Where does this redirect lead?Build the redirect steps one by one.Can this crawler access my page?Test one robots.txt rule for a bot.What do my server logs show?Find status codes, paths and claimed bots.

Two sections. One clear purpose.

Use a tool when you already have data. Read a guide when you first need to understand what a result means.

Tools

Five small tools for logs, HTTP responses, redirects, robots.txt and IP addresses.

Open tools

Learn

Plain answers to practical questions about crawling, indexing, redirects and server logs.

Open guides

Crawlers do different jobs.

Googlebot, Bingbot and AI crawlers are built for different purposes. We show what each operator publishes, how a bot can be checked and when the information was reviewed.

Compare crawlers

Why is the site called AnalyseSpider?

The domain was formerly used for log analysis and IP tools. Today's website is a new implementation under new ownership. Former software, copy, data and customers did not transfer.

Read about the new ownership