Robots.txt Generator

How to generate a robots.txt file

Follow these steps to create a correctly formatted robots.txt file for your site.

1. Choose your user agent. Select All Bots (*) to apply rules to every crawler. Pick Googlebot or Bingbot to target one specific search engine.
2. Set a crawl delay (optional). Leave the field at 0 to skip the Crawl-delay directive. Enter a positive number in seconds only if your server needs to limit crawler request frequency.
3. Add your sitemap URL. Enter the full URL to your XML sitemap. Most WordPress sites use a URL from their SEO plugin such as https://yourdomain.com/sitemap.xml.
4. List disallowed paths. Enter one path per line for any folder or file you want to block. Each path must start with a forward slash such as /wp-admin/.
5. Click Generate Robots.txt. The formatted file appears in the output editor below the form.
6. Copy or download the output. Use Copy Code to grab the text or click Download .txt to save the file.
7. Upload to your WordPress site root and verify. Name the file robots.txt and place it in your domain root via FTP or your hosting control panel. Open https://yourdomain.com/robots.txt in a browser to confirm the file is live.

Common robots.txt user-agent strings

Use the table below to identify the correct user-agent string for the search engine you want to target.

Common robots.txt disallow paths for WordPress

Copy the paths you need from this table into the Disallowed Paths field, one per line.

What each robots.txt field does

User-agent

The User-agent directive names the crawler the rules apply to. An asterisk (*) targets every bot that reads the robots.txt file. Named values like Googlebot or Bingbot restrict the rules to that crawler only. For most sites setting User-agent to * covers all search engines in one block.

Disallow

The Disallow directive tells a crawler which URL paths not to request. Each value must start with a forward slash and the tool uses prefix matching. A Disallow for /private/ blocks every URL that begins with that path including any subpages and files within it. See Google’s robots.txt documentation for the full list of supported directives and edge cases.

Crawl-delay

The Crawl-delay directive asks crawlers to wait a set number of seconds between requests. Not all search engines support it. Googlebot ignores the Crawl-delay directive and manages crawl rate through Search Console settings instead. Use crawl delay only if your hosting plan has limited bandwidth and crawler traffic causes visible server load.

Sitemap

The Sitemap directive points crawlers to your XML sitemap URL so they discover your pages faster. Include the full absolute URL starting with https. Most WordPress SEO plugins generate a sitemap URL you can paste directly into this field.

Where to use robots.txt

WordPress and WooCommerce sites

WordPress installs include admin, login and core files that search engines have no reason to request. WooCommerce stores benefit from blocking cart, checkout and account paths to keep crawl budget on product pages. Add your sitemap URL in the Sitemap field so Googlebot finds every published product from its first visit.

New site launches

A robots.txt file with your sitemap URL should be in place before your domain goes live. Adding it early means Googlebot finds your pages from the first crawl rather than waiting for organic link discovery. Use the IP address checker to confirm your server is publicly accessible before submitting your site to Google Search Console.

Staging and private directories

Block staging paths or internal admin directories with Disallow rules before your site receives public traffic. Robots.txt is publicly readable, so never rely on it alone to protect sensitive content. Combine it with HTTP authentication on private directories for real access control.

Multilingual sites and parameterized URLs

Sites with language subdirectories or URL parameters can generate duplicate pages that consume crawl budget. Block internal search result URLs and session parameters using Disallow rules to keep budget on canonical pages. Use a canonical sitemap to confirm which pages are the primary versions.

Agency client handoffs

Agencies building client sites can generate a robots.txt file at the end of every project and include it in the handoff documentation. A standardized file with the client’s sitemap URL and standard WordPress disallow rules reduces SEO issues after launch. Generate the file here, download it and attach it to your deployment checklist.

Robots.txt generator vs writing the file by hand

Manual robots.txt files require exact syntax and one wrong character can produce unintended results. A missing colon after Disallow or a path without a leading slash silently fails to block the URL. This generator formats every directive correctly so you get a valid file on the first attempt without memorizing syntax rules. Validation errors that affect files written by hand do not occur when the tool assembles output from your form inputs.

Use robots.txt with Spexo Addons for Elementor

Set up robots.txt before importing Elementor templates

Browse Elementor templates for WordPress in the Spexo library and choose your layout before uploading your robots.txt file. Generating the file first means your sitemap URL is in place from day one. Googlebot indexes your new Elementor pages immediately after launch with no extra steps.

Block private Elementor pages during development

When you build a site with Elementor, add a Disallow rule for any pages you are still designing before the domain becomes publicly accessible. Crawlers that reach draft pages may index partial content that hurts your launch quality. Remove the Disallow rules and regenerate the file once every page is complete and ready to publish.

Update your sitemap URL when adding new Elementor pages

Each time you publish a new Elementor page on your Spexo site, regenerate your robots.txt to make sure the sitemap URL reflects the current state of your XML sitemap. Keeping the sitemap reference current helps Googlebot discover new pages within days. Return to this generator, paste your updated sitemap URL and download the new file.