# robots.txt — Dhende Associates # https://www.dhendeassociates.com/ # Last updated: 2026-10-18 User-agent: * Allow: / Disallow: /cgi-bin/ Disallow: /private/ # Sitemap location Sitemap: https://www.dhendeassociates.com/sitemap.xml # AI search, retrieval, and training crawlers # Keep private and non-public paths excluded for these specific user-agent groups too. User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: ClaudeBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Google-Extended User-agent: GoogleOther User-agent: Applebot-Extended User-agent: Meta-ExternalAgent User-agent: Amazonbot User-agent: cohere-ai Disallow: /cgi-bin/ Disallow: /private/ Allow: / # Search crawlers used by AI assistants and other discovery systems User-agent: Googlebot User-agent: Bingbot User-agent: Applebot Disallow: /cgi-bin/ Disallow: /private/ Allow: / # Retain legacy Anthropic crawler name for compatibility User-agent: Claude-Web Disallow: /cgi-bin/ Disallow: /private/ Allow: / # AI-readable site overview and content index # https://www.dhendeassociates.com/llms.txt