# See https://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file User-agent: * Allow: / # Authenticated app + private routes — no value in crawling/indexing these. Disallow: /dashboard Disallow: /admin Disallow: /jobs Disallow: /session Disallow: /users Disallow: /passwords Disallow: /email_verification Disallow: /api_tokens Disallow: /model_credentials Disallow: /consensus_groups Disallow: /criteria_groups Disallow: /criteria Disallow: /verdicts Disallow: /evaluations Disallow: /stripe