Why the robots.txt file is important ??
It is a text file that tells web robots that which pages on your site to crawl and not to crawl.
It indicates some parts of your site you don’t want accessed by search engine crawlers.
It is a text file that tells web robots that which pages on your site to crawl and not to crawl.
Robots.txt is a text file that lists webpages which contain instructions for search engines robots.