Technical Term
Back
Robots.txt
CrawlingRobots.txt is the file that guides bots on which areas they may crawl and which areas should be avoided. It is useful, but a badly placed rule can hide important parts of the website.
What means Robots.txt
Robots.txt: Robots.txt is a file placed at the root of the domain that gives crawling instructions to bots. It is not a security tool, but a guide for organizing the journey and preventing crawlers from wasting resources in unnecessary areas.
AISEO Area: Crawling. This tag helps to place the term within the glossary and understand if it belongs to crawling, semantics, performance, security, AI, accessibility or certification.
It matters because a single line can open or block large parts of the site. Used well, it improves crawl order; used badly, it can keep valuable content from being discovered.
- Review which folders are allowed and which are blocked.
- Check that resources needed to interpret the page are not blocked.
- Include the sitemap when useful.
- Test the rules after important changes.
Common errors
Blocking the entire site by keeping a test rule.
Using it as if it protected private information.
Blocking access to necessary CSS or JavaScript.
Not updating it after changing the folder structure.
Frequently asked questions about Robots.txt
What is Robots.txt?
Robots.txt is a file placed at the root of the domain that gives crawling instructions to bots. It is not a security tool, but a guide for organizing the journey and preventing crawlers from wasting resources in unnecessary areas.
Why should Robots.txt be reviewed?
It matters because a single line can open or block large parts of the site. Used well, it improves crawl order; used badly, it can keep valuable content from being discovered.
How is Robots.txt handled in AISEO?
Review which folders are allowed and which are blocked. Check that resources needed to interpret the page are not blocked. Include the sitemap when useful.