Robots.txt
A file that tells search engine crawlers which pages or sections not to crawl—used for control and performance.
What is Robots.txt?
A file that guides search engines—blocks or allows crawl access to parts of your site.
Dive Deeper
Robots Meta Tag, Crawl Budget, Indexability, Noindex Tag, Search Engine Crawlers
