SEO Crawl Configuration Brief Template

Scope, Hosts & Limits

When to use: Decide exactly which hosts, folders and URL patterns are in the crawl, and how big it is allowed to get.

What question is this crawl answering?

[e.g. why new product pages are not indexed; pre-migration baseline; post-launch QA]

Scope

• Start URL(s): [https://www.example.com/]
• Hosts in scope: [www, shop, blog subdomain]
• Hosts out of scope: [staging, app, help centre]
• Folders in scope: [/products/, /guides/] · crawl outside the start folder? [yes / no]
• Include patterns: [regex, tested on 5 sample URLs]
• Exclude patterns: [regex for /cart, /account, ?sort=, ?session=]

Limits

• Maximum URLs: [number] · Maximum depth: [clicks from start]
• Parameters: [crawl all / strip these / exclude these]
• Expected size: [indexed pages in Search Console, sitemap URL count]

Tip: compare the expected size with the sitemap count. If the crawl finds ten times more URLs, that's usually parameters or faceted navigation, and it's a finding in itself.

Rendering, User Agent & Directives

When to use: Choose whether to render JavaScript, which user agent to send, and whether the crawler obeys robots.txt, noindex and canonicals.

Rendering

• Mode: [raw HTML only / JavaScript rendering]
• Why: [e.g. links in the main menu are added by JavaScript]
• Render timeout and viewport: [seconds, mobile]
• Store raw and rendered HTML: [yes / no] · needed to compare what changes after rendering

User agent

• User agent string: [crawler default / mobile Googlebot-like string / custom]
• Identifiable in logs: [add a suffix, e.g. -audit, so your crawl can be told apart from the real Googlebot]

Directives

• robots.txt: [respect / ignore, and why]
• Follow nofollow links: [yes / no]
• Crawl canonicalised and noindex URLs: [yes, to report them / no]

Rule of thumb: respect robots.txt when you want to see the site as search engines do. Ignore it only when the job is to find what robots.txt is hiding, and write that down here.

Speed, Schedule & Permissions

When to use: Agree crawl speed, the time window, firewall allowlisting, authentication and who to call if something goes wrong.

Load

• Threads / URLs per second: [e.g. 2 threads, 3 URLs per second]
• Server notes: [shared hosting, CDN, known slow templates]

Schedule

• Window: [date, start and end time, time zone]
• Avoid: [sales, launches, backups, peak hours]

Permissions

• Firewall / WAF allowlist: [IP or user agent, requested from whom, confirmed on]
• Authentication: [none / basic auth for staging] · credentials shared through the client's password manager, never written in this brief
• Forms-based login: [avoid unless essential] · a logged-in crawler clicks every link, including delete and logout

During the crawl

• Contact if errors spike: [name, phone]
• Stop condition: [e.g. more than 5% 5xx responses in any 5 minutes]

Data Sources & Outputs

When to use: List the sitemaps and API data to connect, and the exports the crawl must produce.

Connect

• XML sitemaps: [URLs of every sitemap or the sitemap index]
• Search Console: [property, date range] · finds pages with impressions that the crawl never reached
• Analytics: [GA4 property, date range] · finds orphan pages that still get visits
• Previous crawl to compare: [file name and date]

Outputs

• Exports: [status codes, indexability, canonicals, redirects, orphan URLs]
• File naming: [domain_purpose_YYYY-MM-DD]
• Stored at: [shared folder] · Kept until: [date]

Keep these settings for the next crawl. A crawl a month later with different settings shows different numbers, and nobody can tell whether the site changed or the crawl did.

Approval & Completion Check

When to use: Sign the brief off before the crawl and tick the completion checks before anyone reads the results.

Before the crawl

1. The question this crawl answers is written in one sentence.
2. Include and exclude patterns are tested on sample URLs.
3. Speed and window are agreed with the site owner.
4. Firewall allowlisting is confirmed, if needed.
5. Approved by [name, role] on [date].

After the crawl

1. Crawled URLs vs expected size: [number / number]. A big gap is explained.
2. Share of 5xx and timeouts: [%]. If high, the results describe the server, not the site.
3. Crawl finished, not stopped early: [yes / no]
4. Settings saved with the export, so the next crawl can repeat them.

Worked Example (fictional online store)

When to use: A completed brief for an invented store checking why new category pages aren't indexed, including what the settings caught.

Worked example: Harbor & Pine

Example only. The company, people, URLs and figures are invented to show what a usable, filled-in version looks like.

What the settings caught

The raw-HTML pass found 3 links to the new categories. The rendered pass found 140. The new pages were only linked from a JavaScript menu that failed to render within the timeout on slow templates. The fix was adding plain HTML links from the parent categories, not resubmitting the sitemap.

Next step

Run a website audit for a quick first check before you set up a full crawl. Use Get Source Code of Webpage to see whether the links exist in the raw HTML, and the Redirect Checker to trace any chains the crawl reports. Want an expert to configure and read the crawl? Our SEO & GEO services include technical audits with an implementation plan.