repos/website
1
Fork 0
Code Issues 1 Pull requests 23 Activity

Remove explicit URLs from robots.txt

Browse source 
Rely instead on the `robots` metatag on the respective pages
This commit is contained in:
Jake Howard 2024-05-26 15:45:58 +01:00
parent 8d724277b0
commit 54aaca3087
Signed by: jake
GPG key ID: 57AFB45680EDD477
1 changed files with 1 additions and 4 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
website/common/templates/robots.txt
View file

@ -1,11 +1,8 @@
{% if SEO_INDEX %}
User-agent: * User-agent: *
{% if SEO_INDEX %}
Allow: / Allow: /
{% else %} {% else %}
User-agent: *
Disallow: / Disallow: /
{% endif %} {% endif %}
Disallow: {% url "wagtailadmin_home" %}
Disallow: {% url "api:index" %}
Sitemap: {{ sitemap }} Sitemap: {{ sitemap }}