Check robots.txt, detect AI bots, and generate XML sitemaps.
Detection of rules for GPTBot, CCBot, Google-Extended, etc.
Crawling your website and discovering pages...