Hlo Friends,
What is robots.txt?
It's kind of text file containing the instruction for Crawlers to crawl webpage(s), domains, files.
It is a set of instructions to tell the crawler which areas of the website should not be crawl.
It is a special file which tells the crawler which part of the website should index or not
I completely agree with you guys.