How to Optimize Your robots.txt File to Block Crawling of Unnecessary Pages in Webflow

The robots.txt file tells crawlers which pages on your site they can and can’t visit. Set it up properly, and it focuses Google’s crawl budget on your strategic pages while speeding up the indexing of important content. Sandro, cofounder of Gemeos Agency, a Webflow agency, walks you through the essentials you need to know.

Prerequisites

• A Webflow site published on a custom domain
• Access to Project Settings > SEO in Webflow
• Google Search Console to check the impact

1. Open robots.txt in Webflow

In Webflow, go to Project Settings > SEO > Robots.txt. Webflow generates a default robots.txt file that allows everything. You can edit it directly in this interface. Changes take effect the next time you publish the site.

2. Understand the basic syntax

# Rules for all robots
User-agent: *
Disallow: /admin/
Disallow: /thank-you
Disallow: /404

# Explicitly allow a blocked section
Allow: /admin/public/

# Point to your XML sitemap
Sitemap: https://votresite.com/sitemap.xml

3. Identify the pages to block

On a typical Webflow site, these pages are worth blocking to save crawl budget:

• Confirmation and thank-you pages (/thank-you, /thank-you, /confirmation)
• Utility pages with no SEO value (/404, /search)
• Preview or staging pages if they’re publicly accessible
• Privacy policy and terms pages if they don’t have SEO value

4. Optimized robots.txt template for Webflow

User-agent: *
# Utility pages with no SEO value
Disallow: /thank-you
Disallow: /thank-you
Disallow: /confirmation
Disallow: /404

# Block AI crawlers if needed
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

Sitemap: https://votresite.com/sitemap.xml

5. Check in Search Console

In Search Console, the "URL Inspection" tool shows whether a page is blocked by robots.txt. The "Coverage" section also lists pages excluded by robots.txt. Make sure no strategic page is blocked by mistake.

Conclusion

A well-configured robots.txt is a sign of technical quality sent to Google. It doesn’t replace other SEO optimizations, but it does support a controlled indexing strategy.

• Use case 1: an e-commerce site with hundreds of filter pages that don’t need to be crawled
• Use case 2: an agency that wants to block AI crawlers from its premium content
• Use case 3: a site with limited crawl budget that wants to prioritize its conversion pages