ArgrowBot Crawler Policy

Last updated: 23 July 2026 · Version 2026-07-23

The short version. ArgrowBot is the web crawler behind Argrow's SEO audits. It visited your site because someone asked us to audit that site. It reads public pages only, obeys robots.txt, and identifies itself on every request. You can slow it down or block it entirely with two lines of robots.txt, and we will honour that. To reach a human, email support@argrow.ai.

1. How to identify it

ArgrowBot sends this User-Agent on every request:

User-Agent	`Mozilla/5.0 (compatible; ArgrowBot/1.0; +https://argrow.ai/bot)`
Operated by	Resimplify Pty Ltd (ABN 67 629 229 143), trading as Argrow, Australia
Contact	support@argrow.ai

We do not currently publish a fixed IP range, so please identify ArgrowBot by its User-Agent string rather than by address. Anything claiming to be ArgrowBot from another User-Agent is not us, and we would like to hear about it.

2. Why it visited your site

ArgrowBot crawls a website when someone asks Argrow to audit that website - either a customer whose site we manage, or someone who requested a free audit and entered the address. We do not crawl the web at large, we do not follow links off the site we were asked to look at, and we do not build a general-purpose index.

If a site was submitted to us that the submitter had no business submitting, tell us and we will stop and remove what we hold. See section 6.

3. What it does, and what it does not do

It does:

Request public pages and read their HTML, headers and response codes.
Check technical SEO signals - titles, meta descriptions, headings, canonical tags, structured data, hreflang, image attributes, security headers, broken links and page speed.
Render some pages in a headless browser, so that JavaScript-built content is assessed the way a search engine would see it.
Read robots.txt and any sitemaps it points to.

It does not:

Submit forms, attempt to log in, or try to reach anything behind authentication.
Collect personal data from your pages, or harvest email addresses.
Use your content to train machine-learning models.
Sell, publish or share your content. Audit results go only to the person who requested the audit.
Probe for vulnerabilities, or test anything it was not asked to assess.

4. How it behaves

Behaviour	What we do
`robots.txt`	Fetched before anything else and obeyed. A `Disallow` that applies to us means we do not request the path.
`Crawl-delay`	Honoured if you set one. It overrides our own pacing.
Request rate	By default no more than about 2 requests per second to a single site, with at most 3 pages in flight at once.
Scope	One site per audit, capped at a set page count. We stay on the domain we were asked to audit.
Caching	We send `If-None-Match` and `If-Modified-Since`, so an unchanged page can answer `304` and cost you almost nothing.
Frequency	Typically weekly for a managed site, and once for a one-off audit.

5. Slowing it down

To pace ArgrowBot without blocking it, add this to your robots.txt:

User-agent: ArgrowBot
Crawl-delay: 10

That asks for one request every ten seconds. We will follow it.

6. Blocking it

To stop ArgrowBot from crawling your site entirely:

User-agent: ArgrowBot
Disallow: /

We check robots.txt at the start of every crawl, so this takes effect on our next visit with nothing else required from you. You can also block the User-Agent at your firewall or CDN if you prefer, and we will not attempt to work around it.

One thing worth knowing before you do: if you are an Argrow customer, or someone has requested an audit of your site, blocking us means that audit cannot be completed and the report will be empty or wrong. If the crawl is unwanted, emailing support@argrow.ai is the cleaner fix - we will stop the crawl at our end and delete what we hold for that site.

7. Reporting a problem

If ArgrowBot is requesting too fast, reaching somewhere it should not, ignoring a rule on this page, or you simply want it gone, email support@argrow.ai. Include your domain and, if you can, a few log lines with timestamps. We treat crawler complaints as urgent, and a bug in our crawler is our problem to fix, not yours to work around.

8. Contact

Resimplify Pty Ltd (ABN 67 629 229 143), Gordon, Australian Capital Territory, Australia - trading as Argrow. Email support@argrow.ai.