HauntedLineage Search Web Crawler

This page identifies our crawler and explains how we crawl content for search indexing.

Crawl Behavior & Politeness

• We fully honor robots.txt Disallow rules and Crawl-delay directives.
• We respect <meta name="robots" content="noindex"> and nofollow meta tags — pages marked noindex are excluded from our index.
• We apply per-domain request throttling to avoid placing excessive load on any single server.
• We prioritize XML sitemaps when available; sitemap URLs are crawled first and refresh schedules are respected.
• We identify ourselves honestly with the User-Agent shown above and do not attempt to disguise crawl traffic.

What We Index

• Publicly accessible HTML pages, documents, and media that are relevant to our focus topics.
• Page titles, descriptions, headings, body text, and metadata for search indexing purposes.
• We do not collect, store, or index content behind authentication walls or pages that return non-2xx HTTP status codes.
• We do not submit forms, execute logins, or interact with dynamic UI elements.

Crawl Frequency & Recrawl

• New pages discovered via sitemaps or outbound links are typically crawled within a few days of discovery.
• Already-indexed pages are periodically recrawled to detect content changes — frequency depends on how often a page has historically changed.
• We respect Cache-Control and Last-Modified headers when evaluating whether to re-fetch a resource.

How to Opt Out

To block our crawler, add the following to your robots.txt file:

User-agent: HauntedLineageGhostCrawler
Disallow: /

To block all crawlers including ours:

User-agent: *
Disallow: /

Changes to robots.txt are typically picked up within 24–48 hours on our next crawl pass. For immediate removal, contact us directly (see below).

Content Removal & Opt-Out Requests

If you need a specific page or your entire domain removed from our index, contact us with the exact URL(s) and we will process the request promptly.

For complete domain exclusion we can add your domain to our permanent blocklist in addition to honoring your robots.txt.

Contact: crawler@hauntedlineage.com

Please include in your message:

• The domain or specific URLs you want excluded
• Whether you want the URL(s) de-indexed, blocked from future crawling, or both
• Any verification that you control the domain (e.g., a DNS TXT record or a reply from the domain's admin email)

Data & Privacy

• We do not sell, share, or publish raw crawled content to third parties.
• Indexed data is used to power the HauntedLineage Search search index for https://search.hauntedlineage.com.
• Policy pages: Privacy Policy and Terms of Service.