What a robots.txt file is for
A robots.txt file is a small text file that sits at the root of your website and tells search engine crawlers (the automated programs, sometimes called bots, that visit your pages and decide what to index) which parts of your site they are welcome to look at and which parts to leave alone. It lives at an address like yoursite.co.uk/robots.txt, and it is usually the first thing a crawler checks before it goes anywhere else on your site. Get it wrong and you can end up hiding pages you wanted found, or inviting crawlers into folders that never needed their attention in the first place.
Most site owners never look at this file, because most website builders and content management systems generate a basic version automatically. That default version is often too broad or too narrow for what you actually need, which is where writing your own comes in.
How search engines actually read it
When a crawler such as Googlebot arrives at your site, it requests robots.txt first and reads a set of instructions written as simple rules: which user-agent (the name a particular crawler identifies itself by) the rule applies to, and which folders or pages are allowed or disallowed for that crawler. If no file exists, most crawlers assume everything is open and carry on regardless.
It is worth being clear about what robots.txt cannot do. It asks crawlers not to visit certain pages, but it does not remove a page from Google's index if that page is already there, and it does not stop a page being visited by a crawler that chooses to ignore the file, which some do. For blocking a page from appearing in search results specifically, you need a different instruction called a noindex tag, placed in the page itself. The two tools solve different problems, and mixing them up is one of the more common ways site owners end up confused about why a page they thought was hidden still shows up in Google.
What this generator builds for you
The Robots.txt Generator puts together a working file from a short set of choices, so you do not need to remember the exact syntax or worry about a stray character breaking the whole thing. You tell it which folders to block, which crawlers the rules should apply to, and where your sitemap file lives, and it assembles the text in the format crawlers expect.
It builds the file from the same handful of directives explained below, so the output is something you can read and understand. Like any tool that generates code for you, it can produce a file that does not match what you actually intended if the inputs are wrong, so it is worth checking the result against your own site structure before you publish it.
Writing the rules yourself
If you would rather understand the file well enough to edit it directly, or you want to check what the generator has produced, the syntax is short enough to learn in one sitting.
Allow and disallow
Two directives do most of the work. Disallow tells a crawler not to visit a folder or page, and Allow tells it that a specific page within a disallowed folder is fine after all. A typical block looks like this:
- User-agent: * (applies to every crawler)
- Disallow: /admin/ (keeps crawlers out of your admin area)
- Disallow: /cart/ (keeps crawlers out of shopping cart pages, which have no reason to be indexed)
- Allow: /admin/help-page.html (makes an exception for one page inside a blocked folder)
The order and the trailing slash both matter. Disallow: /cart blocks anything starting with those letters, including a page called /cart-instructions.html that you never meant to touch, while Disallow: /cart/ blocks only the folder itself.
Targeting specific bots
The asterisk in User-agent: * means the rule applies to every crawler that reads the file. You can write a separate block naming a specific one, such as Googlebot, if you want different rules for different search engines, though most small business sites have no real need for this and a single shared ruleset is simpler to maintain.
Adding your sitemap
A line near the end of the file, such as Sitemap: https://yoursite.co.uk/sitemap.xml, points crawlers to your sitemap (the file listing every page you want indexed). It is a small addition but a useful one, because it gives crawlers a direct route to your important pages.
Mistakes that quietly block your own site
The most common problem is not a missing file but an over-broad one. A single line reading Disallow: / blocks the entire site from every crawler, and this line is sometimes left over from a staging or test version of a site that then gets copied straight to the live version without anyone checking it. A site can sit unindexed for weeks because of one forgotten slash.
Other mistakes worth watching for:
- Blocking CSS or JavaScript files that Google needs to load in order to understand how your pages actually look, which can affect how your site is assessed.
- Assuming robots.txt keeps a page out of Google's search results, when a noindex tag is the correct tool for that job.
- Writing rules for one crawler and forgetting that others, including ones used by SEO (search engine optimisation, the practice of improving how a site performs in search results) tools you rely on for research, may follow different rules or ignore the file altogether.
- Editing the file directly on a live site without keeping a copy of the version that worked, so there is nothing to revert to if the new one causes a problem.
Checking the file once it's live
Once you have uploaded a new robots.txt file, the file itself is easy to check. Visit yoursite.co.uk/robots.txt in a browser and read through it as if you were the crawler, checking that nothing you actually want indexed sits behind a Disallow line. Google Search Console, a free tool from Google that shows how your site appears in search, includes a way to test how Google reads your current robots.txt file and flags whether any of your important pages are being blocked by it.
It is worth checking again any time you move to a new website builder or hosting platform, because these changes sometimes reset the file to a generic default without warning.
When robots.txt isn't enough
Robots.txt is one part of how a site presents itself to search engines, alongside things like structured data, which is markup that helps search engines understand what a page is about. If you are working through both, the Understanding SEO Schema Generator guide covers the companion piece, and the SEO Schema Generator is the tool that builds that markup. If you are setting up a new site from scratch, the Website Launch Promotion Checklist covers where a correctly configured robots.txt file fits among the other things worth checking before you promote a site publicly.