User-agent: * Disallow: /admin/ Disallow: /api/ Disallow: /images/ Disallow: /*.asp Allow: /*.html