Glossary
robots.txt
Definition
robots.txt is a file at a site's root that tells crawlers which paths they may or may not request.
It controls crawling, not indexing: a disallowed page can still be indexed if linked elsewhere. To keep a page out of results, use a noindex tag instead, and do not block it in robots.txt.
Most sites need only a short, careful file plus a link to the XML sitemap. A single wrong Disallow can hide an entire section, so handle it deliberately.