Build a robots.txt file that tells search engines what to crawl
A robots.txt file tells search engine crawlers which parts of your site they can visit and which to leave alone, and getting it wrong can accidentally block Google from indexing pages you want found. This tool builds the file for you from a few simple choices about what to allow or block, then shows you the exact text to upload. Check the result against your own site structure before you publish it, because a rule that looks right can still block more than you intended.
Set up your robots.txt file correctly, without blocking the pages you want found
A robots.txt file tells search engines like Google which parts of your website they're allowed to look at and which parts to leave alone. This guide is for anyone who has never set one up before and wants to avoid the common mistake of accidentally blocking the pages they most want found. By the end, you'll have a working file uploaded to your site and know how to check it's doing what you intended.
-
Understand what robots.txt actually controls
Learn what a robots.txt file is, why search engines look for it, and what happens if your site doesn't have one.
-
Generate and customise your rules
Use the generator to build rules for which pages or folders search engines can and can't crawl, then review the output line by line.
-
Upload it and confirm it works
Upload the finished file to your site's root folder and check it in Google Search Console to confirm search engines can read it.
Robots.txt Generator
Choose which bots to block, list any folders search engines should not crawl, add your sitemap, then generate a ready-to-upload robots.txt file. Once created, check it with our Robots.txt Tester.
Your robots.txt
Generate a working robots.txt file in minutes
This tool builds a robots.txt file from the rules you pick, covering which pages search engines like Google can crawl and which to leave alone. It suits anyone managing a website who wants control over indexing without editing raw text by hand. Treat the output as a starting point and check it against your own site structure before you upload it, because a wrong rule can block pages you actually want found.
No account needed · free to use · works on any device
Popular Tools
Promotion Channel Prioritiser
Answer a few questions about your site, budget, and time — get a ranked shortlist of the promotion channels most likely to work for your specific situation.
Prioritiser ToolContent Distribution Planner
An interactive Content Distribution Planner, free to run in the browser. The companion guide sets out the method behind it, so you can check the working.
Content Distribution Planner | Tools
Website Launch Promotion Checklist Scorer
An interactive Website Launch Promotion Checklist Scorer, free to run in the browser. The companion guide sets out the method behind it, so you can check the working.
Website Launch Promotion Checklist Scorer | ToolsLaunch Promotion Countdown Planner
An interactive Launch Promotion Countdown Planner, free to run in the browser. The companion guide sets out the method behind it, so you can check the working.
Launch Promotion Countdown Planner | Tools
Organic Traffic Growth Estimator
An interactive Organic Traffic Growth Estimator, free to run in the browser. The companion guide sets out the method behind it, so you can check the working.
Organic Traffic Growth Estimator | ToolsPromotion Tactics Reference Table
An interactive Promotion Tactics Reference Table, free to run in the browser. The companion guide sets out the method behind it, so you can check the working.
Promotion Tactics Reference Table | Tools