AI Crawler Access Review Template

AI Crawler Access Review (fill-in)

When to use: One row per AI agent with its documented purpose, site policy, live robots rule, log evidence and approval, plus owners and a completion check.

AI crawler access review

One row per AI agent. Each row records what the vendor says the agent does, what your site decided, the rule that is live, what your logs show and who approved it.

Site: [https://www.example.com] - Review owner: [name] - Review date: [YYYY-MM-DD] - Next review: [date]

Per-agent detail (fill for every row)

• Documented purpose: [what the vendor says the agent does, in its own words]
• Source: [vendor docs URL] checked on [date]
• Agent type: [training / search / user-initiated fetch / control token only]
• Vendor says robots.txt applies: [yes / may not / generally ignores]
• Business reason for the policy: [why this site allows or blocks it]
• Log window: [start and end date], source [CDN / origin / host panel]
• Verified against: [vendor IP list URL or reverse DNS result]
• Other controls: [CDN or firewall rule, if any, and who owns it]

Next action

• Review owner: [name] - task: re-check every vendor docs page and update the purpose column by [date]
• Log owner: [name] - task: export [N] days of logs for the listed agents and fill the evidence column by [date]
• Approver: [name + role] - task: sign off each policy or send it back with a reason by [date]
• Developer: [name] - task: ship approved robots.txt changes and save the old file for rollback by [date]

Completion check

• Every row names one agent token, and its purpose cites the vendor's own page with a check date.
• Every policy has a written business reason, not just allow or block.
• The robots rule column is copied from the live file, not from a draft.
• Log evidence covers a stated date window, and claimed bot hits were checked against the vendor's IP list or reverse DNS.
• Each row has a named approver and a date. Rows marked mismatch have an owner and a fix date.

Documented Agent Reference (vendor-checked)

When to use: Eleven AI agent tokens with the purpose each vendor documents, its robots.txt stance, how to verify hits and a link to the source.

Documented agent reference (checked 14 November 2026)

Fill the purpose column from these vendor pages, not from blog lists. Vendors add and rename agents, so open each link again on review day.

Read the table carefully

• Robots.txt is advisory. It is a request. A crawler that ignores it can still fetch the page. Enforcement lives at the server, CDN or firewall.
• User-initiated fetchers are different. OpenAI says robots.txt may not apply to ChatGPT-User. Perplexity says Perplexity-User generally ignores it.
• Two tokens never show in logs. Google-Extended has no user agent string and Applebot-Extended does not crawl. Look for Googlebot and Applebot hits instead.
• Apple fallback: Apple says that if robots.txt does not mention Applebot but mentions Googlebot, Applebot follows the Googlebot rules.

OpenAI also lists OAI-AdsBot, which only visits pages submitted as ChatGPT ads. Add a row if you run those ads.

Policy Decision Picker

When to use: Decide training, AI search and user-initiated fetches separately, with the questions an approver should answer first.

Policy decision picker

Decide by agent type first, then by token. Training, search and user fetches are separate choices with separate consequences.

Questions the approver should answer

• Do we want to be cited in AI answers? [yes / no / only some sections]
• Is any content licensed, paid or under contract terms? [paths + who owns the terms]
• Is crawl load a real problem in the logs? [evidence, not a feeling]
• Who signs off a change that affects AI visibility? [role]

Common starting policies

• Block training, stay citeable: block the training and dataset tokens, allow the search agents. Fits businesses that want to be cited but not used for training.
• Allow all: fits sites that want maximum reach and have nothing licensed to protect.
• Block all AI agents: fits paywalled or licensed publishers. AI search citations can drop too.

The Robots.txt Generator has a one-click AI starter, "Block training, stay citeable", that drafts the first policy. Record the decision here before you ship it.

Log Evidence Checklist

When to use: Pull logs, verify hits against vendor IP lists or reverse DNS, and read mismatches between policy and behaviour.

Log evidence checklist

A policy is only half the review. The log column shows whether agents actually behave the way the robots rule says.

Pull the evidence

• Export [N] days of CDN or origin logs with timestamp, IP, user agent, URL and status code.
• Filter by each token in the review sheet. Keep the raw export; the sheet holds the summary.
• Note how often each agent requested robots.txt itself, and when.

Verify before you trust a hit

• A user agent string is easy to fake. Match the IP against the vendor's published list from the reference table.
• For Applebot and CCBot, the vendors also describe reverse DNS checks. Record which method you used.
• Mark each hit verified, unverified or spoofed. Only verified hits count as evidence about the vendor.

After a policy change, look up your domain in the AI Visibility Checker to see recorded AI answers that name or link you, and keep that next to the log evidence.

Approval and Change Record

When to use: A sign-off record for each policy change: old and new rule, reason, evidence, approver, rollback and re-check date.

Approval and change record

Allowing or blocking an AI agent is a business decision. Record who made it, when, and how to undo it.

Approval rules

• The person who drafts the rule does not approve it.
• Changes to AI search agents need sign-off from whoever owns organic and AI visibility.
• Changes that touch licensed content need sign-off from whoever owns the licence terms.
• No approval, no ship. A rule without a record gets reviewed again next cycle.

Want the full robots.txt rule review, line by line? Use the Robots.txt Review Template for testing and syntax. This page covers the decision and the evidence.

Spreadsheet Header (CSV)

When to use: A paste-ready header row and allowed values to run the review in Google Sheets or Excel.

Paste-ready spreadsheet header

Paste this row into cell A1 of Google Sheets or Excel, then split text to columns by comma. One row per agent per review.

review_date,site,agent_token,vendor,agent_type,documented_purpose,docs_url,docs_checked_date,vendor_robots_stance,site_policy,business_reason,robots_rule_live,log_source,log_window,hits_total,hits_verified,status_codes,verify_method,evidence_result,approver,approval_date,change_id,owner,next_action,next_review,notes

Allowed values

• agent_type: training, dataset, search, user_fetch, control_token
• vendor_robots_stance: follows, may_not_apply, generally_ignores, token_only
• site_policy: allow, block, partial
• verify_method: ip_list, reverse_dns, not_verified
• evidence_result: matches, mismatch, no_hits, spoofed

Sheet tips

• Highlight rows where docs_checked_date is older than your review cycle.
• Highlight rows where approver is empty. Those policies are drafts.
• Filter evidence_result to mismatch to build the fix list.

Worked Example (fictional Wendlethorn knitting patterns)

When to use: A filled review for an invented pattern publisher, with one CDN mismatch and one spoofed bot caught by the logs.

Worked example: Wendlethorn knitting patterns

Example only. Wendlethorn is an invented knitting pattern publisher. The domain, people, log counts and dates are made up to show a filled-in review.

Site: wendlethorn.example - Review owner: SEO lead - Review date: 2026-11-12 - Log window: 2026-10-29 to 2026-11-11

What the review found

• OAI-SearchBot: the policy said allow, but a CDN bot rule blocked it. The free guides could not be surfaced in ChatGPT search. The CDN owner fixed the rule the same week.
• CCBot: the two hits came from IPs outside the published list. They were logged as spoofed and handled at the firewall.
• Paid patterns: they sit behind a login, so robots.txt was never the only control. The review noted that on purpose.

Why this works

• Every policy has a reason, so the next reviewer knows why paid and free paths differ.
• Log evidence caught a block nobody intended, which robots.txt alone would never show.
• Verifying IPs stopped the team from blaming a vendor for spoofed traffic.

Need help setting AI crawler policy for a whole site? Book a free consultation with Anatolii Ulitovskyi, or see our SEO and GEO services.