AI Crawler Rules
14 / 26
bots allowed in robots.txt (12 blocked)
OBSERVEDLLMS.TXT
4.1 KB
File present; effect on AI citation unproven
OBSERVEDSitemap
19
URLs listed · lastmod 2026-09-01 to 2026-10-24 (not a defect by itself)
OBSERVEDStructured Data
19 / 19
sitemap pages carry parseable JSON-LD (0 parse errors)
PARSED CLEANSearch Performance
—
First-party clicks & impressions via Search Console
PENDINGAI Citations
—
Not directly measurable; proxy checks planned
PENDING◉AI & Search Crawler Rules — robots.txt (observed 20 Nov)
ALLOWED (14)
GPTBot · OAI-SearchBot · ChatGPT-User ClaudeBot · Claude-SearchBot · Claude-User PerplexityBot · Perplexity-User Google-Extended · Googlebot · Bingbot Applebot · Applebot-ExtendedBLOCKED (12)
CCBot · Bytespider · Amazonbot PetalBot · Diffbot · ImagesiftBot Omgilibot · FacebookBot · cohere-ai MJ12bot · DotBot · DataForSeoBotDeclared rules only — they do not show whether AI engines actually crawl or cite the site. DataForSeoBot is blocked; relevant only if a paid SEO data vendor is added later; owner decision, no change made.
◆Priority Queue — from observed baseline
1
Connect Google Search Console
HIGHFirst-party Google data (clicks, impressions, indexing); needs one-time OAuth approval. GA4 measures on-site behavior separately.
2
Review bot-filter behavior
HIGHHomepage returns 403 to plain non-browser fetchers; verify this does not block legitimate crawlers or monitoring tools.
3
Verify key pages carry accurate modified dates
MEDIUMFreshness signals matter only when dates reflect real content changes.
4
Decide DataForSeoBot policy
LOWCurrently blocked in robots.txt; only relevant if a paid SEO data vendor is ever connected.
⚙Visibility Workers — reference-inspired roster (truthful status; nothing running as a live agent yet)
TATechnical Audit
Ran 20 Nov (scripted)
robots.txt, sitemap, llms.txt, meta, fetch behavior
CACrawler Access
Ran 20 Nov (scripted)
26 bot rules mapped; policy questions flagged
SESchema Engineer
Ran 20 Nov (scripted)
19/19 pages parse clean; 1 heuristic note (/jobs), unconfirmed; Google eligibility unchecked
KRKeyword Research
Pending
Needs data source; best option = GSC queries after connect
COContent Optimizer
Pending
E-E-A-T & freshness review, page by page
LSLocal SEO
Pending
Indore focus; Google Business Profile check not run
ARAI Readability
Heuristic only
Rubric checks possible now; not measured AI visibility
LALink Analyst
Pending
Free sources: Bing Webmaster link details; paid vendors not connected
CWCompetitive Watch
Pending
Sitemap & public-page monitoring, competitors TBD
ANAnalytics
Pending GSC
Report performance charts once Search Console connects
▶Functioning Now vs Planned
NOW
Local audit engine (audit.py)
Runs the read-only checks behind this page on demand and emits audit.json: robots matrix, llms.txt, sitemap, homepage meta, JSON-LD parse check. Ran for real 20 Nov.
NOW
This dashboard view
Static render of the latest audit output; refreshes when the engine re-runs.
PLANNED
Scheduled audit cadence
Daily/weekly re-runs with change detection between snapshots (diffing already implemented).
PLANNED
Search Console + Bing + GA4 data
Real clicks, impressions, queries, indexing and link stats; needs your one-time approvals first.
PLANNED
Keyword, competitor & report workers
Free data sources first; any paid data vendor is a separate decision.
✓Readiness Observations (heuristic — not rankings)
| Check | Observation |
|---|---|
| AI crawlers permitted | Observed — 14 of 26 listed bots allowed |
| llms.txt published | Observed — present, 4.1 KB |
| XML sitemap present | Observed — 19 URLs |
| Title & meta description | Observed — present on all 19 crawled pages |
| Homepage structured data | Parsed — 2 blocks clean |
| Site-wide schema coverage | Observed — 19/19 parse clean, 1 heuristic note(s) |
| Image alt attributes | Observed — 201/239 image instances lack alt across 19 pages (may include shared/decorative assets; unique-asset split pending next crawl) |
| Real ranking / traffic data | Pending GSC connect |
●Integrations
| Service | Purpose | Status |
|---|---|---|
| Google Search Console | Search performance, indexing, coverage | NEEDS YOUR APPROVAL |
| Google Analytics 4 | On-site behavior & engagement | PENDING |
| Snapshot diffing | Detect changes between audit snapshots | IMPLEMENTED |
| PageSpeed Insights | Page speed, Core Web Vitals (free API) | RATE LIMITED - CHECK DEFERRED |
| Bing Webmaster + IndexNow | Bing rank/link/crawl stats and URL submission (free API) | PENDING |
| WhatsApp / calling outreach | Lead/alert notifications | PROPOSED (OUT OF PHASE 1) |