T2Bot — the T2 Industries web crawler

User-Agent: Mozilla/5.0 (compatible; T2Bot/1.0; +https://t2ind.com/bot)
robots.txt token: T2Bot
IP list: /t2bot-ips.json · /t2bot-ips.txt
Descriptor: /t2bot.json
Contact: bot@t2ind.com
Status: pre-launch

How to identify T2Bot

User-Agent: Mozilla/5.0 (compatible; T2Bot/1.0; +https://t2ind.com/bot)
Requests come only from the IP addresses listed at https://t2ind.com/t2bot-ips.json (also as plain text at /t2bot-ips.txt). The list is authoritative; a request claiming to be T2Bot from any other address is not us.
Every crawler address has matching forward and reverse DNS under t2ind.com (for example crawl-1.t2ind.com). To verify: reverse-resolve the IP, confirm the name ends in .t2ind.com, then forward-resolve that name and confirm it returns the same IP.

What T2Bot fetches and keeps

It fetches HTML pages and robots.txt, sitemap.xml and favicons. It does not fetch images, video, audio, fonts or scripts, and it does not execute JavaScript on most pages.
It reads at most the first 256 KB of a page.
From each page it stores: the URL and canonical URL, HTTP status and headers relevant to caching, the page title, meta description, headings, the extracted main text, outbound links, language, structured data (schema.org / Open Graph) and a content hash. It stores the fetch time.
It does not store or process any data about visitors to your site, and it has no way to see any.

How fast it crawls

Default: no more than one request every 5 seconds per host, and never more than two connections to a host at a time. Most sites see a few requests per day.
It honours Crawl-delay in robots.txt, and backs off on HTTP 429, 503 and connection errors.
It sends If-Modified-Since / If-None-Match on re-visits, so unchanged pages cost you a 304 and no body.
Re-visit frequency depends on how often a page has changed in the past: from daily for fast-changing pages to roughly monthly for stable ones.

How to control T2Bot

robots.txt is honoured exactly, including wildcards, Allow, Disallow and Crawl-delay:
```
User-agent: T2Bot
Disallow: /
```
blocks the whole site; narrower rules work as for any other crawler. Changes take effect on the next robots.txt fetch (within 24 hours).
<meta name="robots" content="noindex"> and X-Robots-Tag: noindex remove a page from the index; nofollow is honoured.
<meta name="T2Bot" content="noindex"> applies to T2Bot only.
To remove content already indexed, or to report a problem, email bot@t2ind.com with the URLs. Removal requests are actioned within 5 business days; abuse reports are read daily. Step-by-step instructions are on the opt-out page.

What T2Bot does not do

Does not log in, submit forms, or fetch anything behind authentication.
Does not fetch pages that robots.txt disallows, and does not use proxies or residential IPs.
Does not collect visitor data, and does not set or read cookies.
Does not feed AI model training. See the AI section below.

AI and machine-learning use

T2Bot's crawl is used for search indexing and for derived data products (site categorisation, company enrichment, change detection). T2 Industries does not provide crawled page text to third parties for training machine-learning models. If that ever changes, it will be done by a separately named crawler with its own user agent and its own page here, so that a rule for T2Bot never silently covers a different use.

What the index holds and how it is ranked is described on the index page.

Contact

bot@t2ind.com — a person reads this mailbox. Postal address: TBD to be published. Abuse and security: security@t2ind.com (see /.well-known/security.txt).

Change log

2026-10-21 — page published; crawler pre-launch; IP list empty until launch.