Generator

Robots.txt Generator

Create a clean base for your robots.txt file. Define which bots can crawl your website, which routes to block, and where your XML sitemap is located.

Robots.txt Configuration

Default Rule

Crawl-delay

Sitemap XML

Rules for Specific Bots

Google

Google Image

Bing (MSN)

Yahoo

Yandex

Baidu

Ahrefs

SEMrush

Restricted Directories

Enter one path per line (e.g., /admin/)

Generated Robots.txt Code

Where to save this file?

You must download this file and upload it to the root of your domain (e.g., mydomain.com/robots.txt).

INFO

WHAT IS A ROBOTS.TXT FILE?

The robots.txt file tells crawlers which areas of a website they can access and which paths they should not visit. It does not remove pages from Google by itself, but it helps organize crawling behavior.

A clear configuration prevents accidental blocking of important content and allows you to declare the main sitemap so that search engines can better understand the site structure.

Crawl control:
Defines allowed and blocked paths for bots.
Visible sitemap:
Includes the location of the main XML sitemap.
Error prevention:
Reduces accidental blocking of important content.
AISEO foundation:
Helps maintain a clear and verifiable technical signal.

Questions about robots.txt

Common questions about crawling, blocking, and security.

Where is the robots.txt file placed?

It must be placed at the root of the domain. For example: https://yourdomain.com/robots.txt.

Does robots.txt remove a URL from Google?

No. robots.txt controls crawling, but it does not guarantee removal from the index. To prevent indexing, other signals such as noindex should be used when appropriate.

Can I block the admin panel?

Yes, you can declare paths such as /admin/ or /login/. However, real security should rely on authentication, not robots.txt.