repos/website
1
Fork 0
Code Issues 1 Pull Requests 30 Activity

Remove explicit URLs from robots.txt

Browse Source 
Rely instead on the `robots` metatag on the respective pages
This commit is contained in:
Jake Howard 2024-05-26 15:45:58 +01:00
parent 8d724277b0
commit 54aaca3087
Signed by: jake
GPG Key ID: 57AFB45680EDD477
1 changed files with 1 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
website/common/templates/robots.txt
View File

@ -1,11 +1,8 @@
{% if SEO_INDEX %}
User-agent: *
{% if SEO_INDEX %}
Allow: /
{% else %}
User-agent: *
Disallow: /
{% endif %}
Disallow: {% url "wagtailadmin_home" %}
Disallow: {% url "api:index" %}
Sitemap: {{ sitemap }}