Crawler info

SikwatiBot — our web crawler

You probably found this page because SikwatiBot showed up in your server logs. SikwatiBot is the crawler that powers Sikwati, a Generative Engine Optimization (GEO) tool used by site owners and agencies to track how AI engines cite their domains. We only crawl sites whose owner has added them to a Sikwati account and verified ownership via DNS TXT.

User agent

All SikwatiBot requests use this exact User-Agent header:

Mozilla/5.0 (compatible; SikwatiBot/1.0; +https://sikwati.com/bot)

The +https://sikwati.com/bot token is the convention for identifying friendly crawlers — this page is where it points.

What it fetches

/robots.txt — to honor your crawl preferences.
/sitemap.xml and common sitemap paths — to discover pages.
HTML pages on your verified domain (capped at 200 per site) — to score them for GEO readiness.

We follow at most 5 redirects per request, time out after 10 seconds, and never download more than 5 MB per response. We do not execute JavaScript, do not log in, and never POST.

Verifying it's actually us

SikwatiBot does not currently publish a fixed egress IP range, so User-Agent matching is the recommended way to identify us. The string contains the literal substring SikwatiBot; matching on that is stable across releases. We will publish reverse-DNS validation and a published IP range as we grow.

Allowlist us on Cloudflare

If your site is behind Cloudflare and page discovery fails with a network-layer error, the cause is almost always Bot Fight Mode rejecting our requests. Note that Cloudflare’s legacy User Agent Blocking tool (under WAF → Tools) only blocks — it has no Skip / Allow action. To allow SikwatiBot, use one of the two paths below.

Pro plan or above — Custom rule

Cloudflare dashboard → your domain → Security → WAF → Custom rules
Create rule
Field: User Agent · Operator: contains · Value: SikwatiBot
Then take action: Skip
Skip the following: tick Bot Fight Mode and All managed rules
Deploy. Effective immediately at the edge.

The equivalent expression for the Expression Editor: (http.user_agent contains "SikwatiBot")

Free plan — disable Bot Fight Mode

Free-tier Custom rules don’t support the Skip action against Bot Fight Mode, so the practical option is to disable BFM for the site:

Cloudflare → your domain → Security → Bots (or Security → Settings on older dashboards)
Turn off Bot Fight Mode (and Super Bot Fight Mode if shown)
Save

Or upgrade to Pro and use the Custom rule above — that scopes the allowlist to SikwatiBot only and leaves BFM protecting the rest.

Allowlist on other WAFs

For AWS WAF, Akamai, Fastly, and similar, the rule is identical in spirit: match the substring SikwatiBot in the User-Agent header and bypass bot-management rules. Consult your provider’s docs for the dashboard click-path; the underlying match is the same.

If you do not want SikwatiBot to crawl your site

We honor robots.txt. To block us entirely, add this to https://yoursite.com/robots.txt:

User-agent: SikwatiBot
Disallow: /

This is the standard mechanism — same as for Googlebot or any other well-behaved crawler. SikwatiBot will stop hitting your site within one crawl cycle.

Reporting issues or abuse

If SikwatiBot is misbehaving — not honoring robots.txt, hitting your site too aggressively, or you suspect a User-Agent impersonator — email abuse@sikwati.com with the date, IP, and a sample log line. We treat these reports as P1.