Technical Term
Back
Meta robots
CrawlingMeta robots is a tag that tells bots what to do with a specific page: index it, not index it, follow its links or limit certain snippets. It is a small signal with major impact.
What means Meta robots
Meta robots: Meta robots refers to a tag that tells bots what to do with a specific page: index it, not index it, follow its links or limit certain snippets. In an AISEO review, it checks whether bots reach the right content, receive a clear response and avoid wasting time on useless paths.
AISEO Area: Crawling. This tag helps to place the term within the glossary and understand if it belongs to crawling, semantics, performance, security, AI, accessibility or certification.
It matters because a wrong directive can remove an important page from the index or allow a URL to appear when it should not be visible. It is one of those details where a misplaced comma becomes expensive.
- Check the real server response, not only the visual appearance.
- Review robots.txt, HTTP status codes, sitemap and indexation rules.
- Prioritize important URLs and reduce low-value paths.
- Validate changes after migrations, redirects or template updates.
Common errors
Assuming Meta robots works without reviewing the real server response.
Blocking or hiding important URLs because of an overly broad rule.
Leaving old paths generating crawl noise.
Not checking logs, HTTP status codes or crawling tools after changes.
Frequently asked questions about Meta robots
What is Meta robots?
Meta robots refers to a tag that tells bots what to do with a specific page: index it, not index it, follow its links or limit certain snippets. In an AISEO review, it checks whether bots reach the right content, receive a clear response and avoid wasting time on useless paths.
Why should Meta robots be reviewed?
It matters because a wrong directive can remove an important page from the index or allow a URL to appear when it should not be visible. It is one of those details where a misplaced comma becomes expensive.
How is Meta robots handled in AISEO?
Check the real server response, not only the visual appearance. Review robots.txt, HTTP status codes, sitemap and indexation rules. Prioritize important URLs and reduce low-value paths.