# robots.txt for Numezis # # Allow all public pages, but disallow demo/design-system routes User-agent: * Allow: / # Disallow design system gallery (public but not indexed) Disallow: /en/design-system/ Disallow: /fr/design-system/ Disallow: /de/design-system/ Disallow: /it/design-system/ # Disallow demo routes (marketing screenshots) Disallow: /en/demo/ Disallow: /fr/demo/ Disallow: /de/demo/ Disallow: /it/demo/ # Disallow pitch deck (internal presentation) # Disallow private portal routes (authenticated app) Disallow: /en/t/ Disallow: /fr/t/ Disallow: /de/t/ Disallow: /it/t/ Disallow: /en/c/ Disallow: /fr/c/ Disallow: /de/c/ Disallow: /it/c/ Disallow: /en/s/ Disallow: /fr/s/ Disallow: /de/s/ Disallow: /it/s/ Disallow: /en/p/ Disallow: /fr/p/ Disallow: /de/p/ Disallow: /it/p/ Disallow: /en/i/ Disallow: /fr/i/ Disallow: /de/i/ Disallow: /it/i/ # Disallow auth routes (login, register, etc.) Disallow: /en/login Disallow: /fr/login Disallow: /de/login Disallow: /it/login Disallow: /en/register Disallow: /fr/register Disallow: /de/register Disallow: /it/register Disallow: /en/reset-password Disallow: /fr/reset-password Disallow: /de/reset-password Disallow: /it/reset-password # --------------------------------------------------------------------------- # EDGE ENFORCEMENT IS GONE — robots.txt is now the ONLY crawler control. # # Until 2026-08-13 the rules below were also enforced at the edge by the Cloud # Armor policy numezis-web-crawler-guard (rule 1100). That policy lived on the # GCP load balancer stack, which was retired with the move to the Infomaniak VM. # Caddy replaced it, but runs the standard caddy:2-alpine build, which ships no # rate_limit module — so nothing enforces a request budget at the edge today. # Reinstating one means an xcaddy build with caddyserver/rate-limit. # # GROUP SEMANTICS — the trap that makes this file dangerous to edit: # a crawler obeys EXACTLY ONE group, the most specific one matching its token, # and that group REPLACES "User-agent: *" rather than adding to it. So every # named group below must repeat the private-area Disallow rules; naming a # crawler just to give it a Crawl-delay would otherwise hand it /c/, /t/ and # /login. The wildcard forms (/*/c/) collapse the four locales into one line # and are understood by Googlebot, bingbot and the AI crawlers named here. # All named groups must stay AFTER the "User-agent: *" rules above: a new # "User-agent:" line ends the preceding group. # --------------------------------------------------------------------------- # AI crawlers: throttled, not blocked. They are how the product surfaces inside # ChatGPT/Claude/Perplexity answers, so the directory should stay reachable to # them — but on 2026-08-13 GPTBot alone fetched ~650 directory pages in 40 # minutes (~16/min, including ?_rsc= payload prefetches). Crawl-delay: 5 caps a # well-behaved crawler near 12 req/min, which still allows ~17k pages/day. # Note: Googlebot deliberately ignores Crawl-delay (use Search Console's crawl # rate setting instead) — that is why it is not listed here. User-agent: GPTBot Crawl-delay: 5 Disallow: /*/design-system/ Disallow: /*/demo/ Disallow: /*/t/ Disallow: /*/c/ Disallow: /*/s/ Disallow: /*/p/ Disallow: /*/i/ Disallow: /*/login Disallow: /*/register Disallow: /*/reset-password User-agent: OAI-SearchBot Crawl-delay: 5 Disallow: /*/design-system/ Disallow: /*/demo/ Disallow: /*/t/ Disallow: /*/c/ Disallow: /*/s/ Disallow: /*/p/ Disallow: /*/i/ Disallow: /*/login Disallow: /*/register Disallow: /*/reset-password User-agent: ClaudeBot Crawl-delay: 5 Disallow: /*/design-system/ Disallow: /*/demo/ Disallow: /*/t/ Disallow: /*/c/ Disallow: /*/s/ Disallow: /*/p/ Disallow: /*/i/ Disallow: /*/login Disallow: /*/register Disallow: /*/reset-password User-agent: PerplexityBot Crawl-delay: 5 Disallow: /*/design-system/ Disallow: /*/demo/ Disallow: /*/t/ Disallow: /*/c/ Disallow: /*/s/ Disallow: /*/p/ Disallow: /*/i/ Disallow: /*/login Disallow: /*/register Disallow: /*/reset-password User-agent: Amazonbot Crawl-delay: 5 Disallow: /*/design-system/ Disallow: /*/demo/ Disallow: /*/t/ Disallow: /*/c/ Disallow: /*/s/ Disallow: /*/p/ Disallow: /*/i/ Disallow: /*/login Disallow: /*/register Disallow: /*/reset-password User-agent: CCBot Crawl-delay: 10 Disallow: /*/design-system/ Disallow: /*/demo/ Disallow: /*/t/ Disallow: /*/c/ Disallow: /*/s/ Disallow: /*/p/ Disallow: /*/i/ Disallow: /*/login Disallow: /*/register Disallow: /*/reset-password # Commercial SEO-data crawlers: they resell crawl data and send no visitors, but # they were rendering the server-side directory (thousands of pages x 4 locales) # hard enough to drive Cloud Run scale-up and gateway timeouts. Search engines # that actually send traffic (Googlebot, bingbot) are NOT listed here on purpose. User-agent: DataForSeoBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: PetalBot Disallow: / User-agent: Bytespider Disallow: / User-agent: DotBot Disallow: / # Sitemap # # /sitemaps/all is a sitemap INDEX: it points at one sitemap per section # (marketing, blog, directory places, directory profiles, compliance). Search # Console reports coverage per submitted sitemap, so sections are what turn # "12 soft 404s somewhere in 9 604 URLs" into a diagnosis. # # /sitemap.xml is the marketing section (a Next.js app/sitemap.ts route can only # serialize a , never a ). It stays listed on its own # because it is the URL already submitted in Search Console. Sitemap: https://www.numezis.com/sitemaps/all Sitemap: https://www.numezis.com/sitemap.xml Sitemap: https://www.numezis.com/sitemaps/compliance