An Eight Mile Services product

Crawler
Studio

Every site has a list. This one is in order.

Give it a URL. It crawls the site the way a search engine would, runs fifty nine checks against what it finds, and hands back a scored report saying what to fix, why it matters and how many points you get back for doing it.

Open Crawler Studio

Checks

59

Categories

6, weighted

Fifty pages

About a minute

Pages kept

None

audit.pipeline

crawling

START URLone originROBOTS.TXTlongest match winsSITEMAP.XMLindex followed onceCRAWL FRONTIERbreadth first · 4 at a time · polite pausePARSERone page in, one PageFacts out59 RULESsix weighted categoriesREPORTscore · issues · charts

no javascript executed

working

How it works

Paste a URL, wait a minute, read the list.

There is no configuration to get right first and no account of yours to connect. A crawl of fifty pages takes about a minute, and you can close the tab while it runs.

01

Point it at a site

Paste the address, choose how many pages and how deep. There is nothing to install, no tag to add to your site and nothing left running afterwards. The defaults are the ones we would have picked for you.

The new audit form: a website address, page and depth limits, and four crawl options each with an explanation

02

Read the score, then the list

One number with a letter beside it, the six weighted categories underneath, and then the part that matters: every failing check, worst first, with what was found and what to do about it.

An audit report showing the overall score ring, the six category bars and the points available in each

03

Work through it and watch it move

Every issue says how many points it puts back on the score. Fix the top one, run it again and the charts tell you whether you were right, which is the only honest way anybody has measured this work.

Report charts showing status codes, indexability, crawl depth, response time and words per page

Where the number comes from

Six categories, weighted by what actually moves the needle.

Fifty nine checks, each with a weight and a reason. The report shows the same arithmetic back to you, so the score is something you can argue with rather than something you have to trust.

Technical

21 checks

26%

Whether a crawler can reach, read and index the pages at all.

Content

8 checks

22%

Whether there is enough on the page, structured well and not repeated elsewhere.

Metadata

10 checks

20%

The titles, descriptions and social cards that decide whether anyone clicks.

Links

8 checks

14%

How pages connect: depth, orphans and links that go nowhere.

Performance

7 checks

12%

Page weight and response time, measured from the crawl itself.

Accessibility

5 checks

6%

The overlap between what screen readers need and what crawlers read.

Nothing is scored equally

Technical leads at 26% because everything else is conditional on a crawler being able to read the page at all. A category score is the weighted mean of its checks, and the overall score is the weighted mean of the categories.

Partial credit, not pass or fail

Every check returns a ratio rather than a yes or no, so thirty nine of forty one pages having a description scores differently from none of them having one. Fix six pages of twelve and the number moves.

Nothing to look at means skipped

A check with no evidence is skipped rather than passed. A site with no images does not earn credit for its perfect alt text coverage, which is the quiet way most scores get flattered.

Grade bands

A from 90

B from 78

C from 62

D from 45

F below that

The thing most tools get wrong

A 200 does not mean the page is in Google.

A page can return a perfect 200, load in eighty milliseconds and be beautifully written, and still be completely invisible, because it is noindex, or blocked in robots.txt, or canonicalises somewhere else. Every report here keeps those two questions apart, and the pages table has a column for each.

The order matters as much as the list

Blocking a page in robots.txt does not remove it from the index. It stops the page being read, so the noindex inside is never seen. That is the most common way a page ends up in Google that nobody wanted there, and the report names it specifically rather than reporting noindex and leaving you to work it out.

indexable.check

four gates

REACHEDnot blocked in robots.txtblocked200 OKthe response is a page4xx or 5xxINDEXABLEno noindex on the pagenoindexCANONICALpoints at itselfpoints elsewhereCAN APPEAR IN A RESULT

The list itself

One issue, not forty copies of the same one.

Twelve pages missing a title is one entry with twelve examples, not twelve identical entries. The list stays short enough to read to the bottom of, which is the only length that matters.

The issues panel: severity counts, then each failing check with what was found, what to do, the caveat and the points available

High

Read these first

Medium

Worth a morning

Low

Tidy up later

Severity comes first, then points. Sorting purely by points would put forty pages missing a social card above a home page returning a 500, which is arithmetically defensible and completely wrong.

What was found

Written for somebody who has not seen the site

One of one indexable pages have fewer than 150 words of visible text. The count, the scope and the threshold, in a sentence you could forward to somebody else.

What to do

Always an instruction, never an observation

Expand these past 150 words, merge them into a fuller page, or set them to noindex if they exist for navigation rather than for search.

Worth knowing

The caveat, where the rule is a judgement call

There is no minimum word count and never has been. 150 words is simply the point below which a page has not said enough about anything to be the best result for it.

Points available

What the score gets back when it is fixed

Plus 6.5 points if fixed. That number is what sorts the list, which is why one missing heading on a busy page can outrank eleven images with no alt text.

The honest bits

What it cannot see, said first.

A report that hides its own limits is a report you have to check by hand. These three are stated on the page, in the documentation and inside the product, in the same words.

It does not run JavaScript

The parser reads the HTML your server sends. A crawl of two hundred pages through a headless browser is minutes of processing for what plain HTTP does in under a minute.

That is also a finding. A page that is empty without JavaScript is empty to link previews, to some crawlers and to every reader on a slow connection. The report says so out loud rather than quietly scoring the page as thin.

The thresholds are not Google rules

Google has never said a title must be under sixty characters. What is true is that results are cut off at around 580 pixels, which for typical text lands near sixty.

Every number in the product is that kind of claim: a practical consequence rather than a ranking factor. Each one is written down with where it came from, and the report marks which are facts and which are judgement calls.

The accessibility score is not an audit

It is weighted at 6% for exactly that reason. What is measured is the overlap: alt text, heading order, a declared language and links with no text.

Colour contrast, keyboard traversal and anything needing a rendered page are not covered. A site could score 100 here and still be unusable, and saying otherwise would be the easiest lie in this category.

How it behaves

Every crawl is real traffic to a real server.

An audit is a few hundred requests made on your behalf. That shapes how fast it goes, where it refuses to go, and how much of what it saw is kept afterwards.

It will not read a private network

Loopback, private ranges, link local addresses including the cloud metadata endpoint, and their IPv6 equivalents are all refused. The check runs on the resolved address rather than the hostname, so a DNS record pointing somewhere private does not get past it.

Every redirect hop is checked again

A public URL that redirects to a private one is the same attack with an extra step, so redirects are followed manually, one hop at a time, with the guard running on each.

It crawls politely

Four requests at a time with a pause between them, an honest user agent that says what it is, and robots.txt respected by default, including the longest match precedence that makes the block everything and open one directory pattern work.

Your pages are not kept

What is stored is the report: the measurements, the issues and the URLs. Page text is hashed for the duplicate content check and then discarded, and old audits are deleted whole rather than trimmed.

Crawl defaults

Concurrency

4 requests

Pages per crawl

Up to 200

Link depth

Up to 6

Page text kept

None

The limits exist because the rules have to see every page together. A duplicate title is only duplicate relative to the others, so the whole crawl is held at once. When a crawl stops early, the report says so rather than pretending it saw everything.

Questions people ask

Things worth knowing before you run one.

Including the two we would rather you asked before signing up than after.

Crawler Studio by Eight Mile

Find out what your site is actually doing.

Places are limited on purpose, because every audit is a few hundred requests to a real website. If they have all gone, or you want a tool that measures something specific to your business, write to us instead.