T2 Industries

For site owners

Opt out of T2Bot

Three ways to keep T2Bot out, one way to remove what is already indexed, and a recipe to check that a crawler claiming to be T2Bot really is.

Block the whole site with robots.txt

User-agent: T2Bot
Disallow: /

Narrower Allow, Disallow, wildcard and Crawl-delay rules work as for any other crawler. Changes take effect on the next robots.txt fetch, within 24 hours.

Remove single pages with noindex

  • <meta name="robots" content="noindex"> in the page, or the X-Robots-Tag: noindex response header, removes a page from the index for every crawler that honours it, including T2Bot. nofollow is honoured too.
  • <meta name="T2Bot" content="noindex"> applies to T2Bot only.

Request removal by email

To remove content that is already indexed, email bot@t2ind.com with the URLs (or the domain). Removal requests are actioned within 5 business days. Abuse reports to the same address are read daily.

What opting out removes

Blocking T2Bot also removes the domain from the derived products. T2 Enrich and T2 Categories keep only the domain name and an "opted out" flag; nothing else about the site is kept or sold.

Verify a request came from us

  1. Check the User-Agent is exactly Mozilla/5.0 (compatible; T2Bot/1.0; +https://t2ind.com/bot).
  2. Check the source IP is in /t2bot-ips.json (plain text: /t2bot-ips.txt). That list is authoritative.
  3. Reverse-resolve the IP. The name must end in .t2ind.com (for example crawl-1.t2ind.com).
  4. Forward-resolve that name. It must return the same IP.

A request that fails any step is not T2Bot. Please report it to security@t2ind.com.

Full crawler details are on the T2Bot page.