How to Customize robots.txt in Webflow
Learn how to customize your Webflow site’s robots.txt file: Disallow, Allow, and sitemap directives.
In this tutorial, Sandro, co-founder of the Gemeos Webflow agency, shows you how to customize your Webflow site’s robots.txt file.
1. Open Webflow’s robots.txt
In Project Settings > SEO, scroll down to the Robots.txt section. Webflow shows the default robots.txt and lets you customize it.
2. Block specific paths
To block robots from accessing certain pages or sections, add Disallow directives. Example: block a confirmation page and the admin folder:
User-agent: *
Disallow: /confirmation
Disallow: /admin/
3. Add your sitemap
Add your sitemap reference at the bottom of robots.txt: Sitemap: https://votre-domaine.fr/sitemap.xml. This helps search engines find it automatically, even without manual submission.
4. Check after publishing
After publishing, check your robots.txt by going to votre-domaine.fr/robots.txt. Also test it in Google Search Console > robots.txt Tester.
Conclusion
Webflow’s robots.txt is customized directly in Project Settings. See also: disable page indexing.
Good to know
Heading
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique. Duis cursus, mi quis viverra ornare, eros dolor interdum nulla, ut commodo diam libero vitae erat. Aenean faucibus nibh et justo cursus id rutrum lorem imperdiet. Nunc ut sem vitae risus tristique posuere.
Lorem ipsum
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
