Inspect theevidence a crawlerleaves behind.
Parse logs. Read response signals. Separate observation from assumption.
192.0.2.10 - - [12/May/2025:10:15:02 +0000] "GET / HTTP/1.1" 200 5120 192.0.2.10 - - [12/May/2025:10:15:03 +0000] "GET /app.js HTTP/1.1" 200 10240 203.0.113.55 - - [12/May/2025:10:15:05 +0000] "HEAD / HTTP/1.1" 200 0 198.51.100.23 - - [12/May/2025:10:15:06 +0000] "GET /wp-login.php HTTP/1.1" 404 1534
- TOTAL REQUESTS
- 8
- UNIQUE IP ADDRESSES
- 3
- STATUS CODES
- 200 / 301 / 404
- PROCESSING
- LOCAL
- 200
- 5 requests
- 301
- 1 request
- 404
- 2 requests
Four instruments. One evidence trail.
Log File Inspector
Parse raw Apache or Nginx access logs locally. Surface crawlers, status codes, and request paths.
Open tool 02Response Inspector
Paste HTTP headers and HTML. Read status, redirects, canonical signals, robots directives, and unknowns.
Open tool 03Robots.txt Rule Tester
Test one product token and URL against RFC 9309 group and longest-match rules without fetching a site.
Open tool 04IP Address Check
Classify address syntax and range locally—without pretending an IP identifies a person or precise place.
Open tool// Method principles
Evidence before conclusions
Every result separates what was observed, what was inferred, and what remains unknown.
- 01
- LOCAL BY DEFAULT —
Logs and pasted responses stay in your browser.
- 02
- BOUNDED BY DESIGN —
No unrestricted crawling, port scanning, or arbitrary server fetching.
- 03
- TRACEABLE —
Methods, limits, timestamps, and source roles stay visible.
// Legacy path decisions
Old URLs, explicit decisions
Useful historical intents receive independently written successors. Unsafe downloads and unrelated paths are retired—not swept into the homepage.
Open legacy map| # | Historical path | Intent | Decision |
|---|---|---|---|
| 01 | /analysespider.html | log analysis | RESTORE |
| 02 | /ip2country/lookup.php | IP lookup | REDIRECT |
| 03 | /download/analysespider.exe | executable | 410 |
Use the whole evidence library
Choose a tool for the evidence in hand, a guide for the workflow, a note for a disputed assumption, or a reference for the underlying definition.
Tools
Four browser-local inspectors for logs, responses, robots rules, and IP properties.
Open index 02Guides
Repeatable workflows for crawler traffic, responses, redirects, and data limits.
Open index 03Lab Notes
Dated notes that challenge one diagnostic shortcut at a time.
Open index 04Reference
Primary-source definitions for status codes, crawler identity, and robots rules.
Open index