The index
How the T2 Index is built
Everything in the index came from a page that was publicly served. Here is what we keep, how often we go back, and how we rank.
What we collect
Per page: URL, canonical URL, final URL after redirects, HTTP status, fetch time, Last-Modified / ETag, language, title, meta description, H1–H3 headings, extracted main text (boilerplate removed), outbound links with anchor text, schema.org / Open Graph / Twitter-card fields, Content-Type, page size, content hash.
Per host: robots.txt and its policy for each known crawler token, sitemap URLs, detected technologies (server headers, framework fingerprints, scripts referenced), TLS certificate issuer and expiry, business contact details only if the site publishes them on a contact/about page, category labels, inbound-link counts from the rest of the index.
What we do not collect
Anything about a site's visitors; anything behind a login; images, video, audio or files; pages that robots.txt disallows; pages marked noindex.
Sources
Our own crawler (T2Bot) and the public Common Crawl corpus (commoncrawl.org), which we use to bootstrap coverage. Everything in the index came from a page that was publicly served.
Scale and refresh
Re-visit cadence is tied to how often a page has changed before, so the budget goes where the web is moving. Unchanged pages are confirmed with a conditional request and a 304, which is what makes frequent re-checking affordable.
| Tier | What | Re-visit |
|---|---|---|
| Head | ~1M most-referenced sites, homepage + key pages | weekly; daily for news/fast-changing pages |
| Body | ~100M pages across sites with real content | monthly |
| Tail | long tail discovered via links and sitemaps | quarterly, or on a change signal |
| robots.txt | every known host | every 24 h |
Every stored record carries its fetch time and a content hash, so freshness is visible per record in every product, and the T2 Changes feed publishes what moved.
How we rank
Text relevance (BM25 over title, headings and body), host quality (inbound links from the rest of the index, duplicate ratio, text-to-markup ratio, spam and machine-generated-text detectors), freshness, and sitemap lastmod. We do not use click data, browsing data or any user-behaviour signal — we have none.
Removal and corrections
See /opt-out.