MIGUEL.REGO // SEO+GEO Get in touch ↗

Glossary

robots.txt

Definition

robots.txt is a file at a site's root that tells crawlers which paths they may or may not request.

It controls crawling, not indexing: a disallowed page can still be indexed if linked elsewhere. To keep a page out of results, use a noindex tag instead, and do not block it in robots.txt.

Most sites need only a short, careful file plus a link to the XML sitemap. A single wrong Disallow can hide an entire section, so handle it deliberately.