+- Web Design Forum HTML CSS JavaScript PHP Graphic Design SEO forum (http://www.webdesignforum.com.au)
+-- Forum: Search Engine Marketing (/forumdisplay.php?fid=18)
+--- Forum: Search Engine Optimisation (/forumdisplay.php?fid=8)
+--- Thread: What is robots.txt? (/showthread.php?tid=32945)
What is robots.txt? - panditmannat - 09-27-2017 05:17 PM
Hlo Friends,
What is robots.txt?
RE: What is robots.txt? - saketbhai - 09-27-2017 06:06 PM
It's kind of text file containing the instruction for Crawlers to crawl webpage(s), domains, files.
RE: What is robots.txt? - MoulanaRafi - 09-27-2017 06:13 PM
It is a set of instructions to tell the crawler which areas of the website should not be crawl.
RE: What is robots.txt? - HafijAli - 09-27-2017 08:17 PM
It is a special file which tells the crawler which part of the website should index or not
RE: What is robots.txt? - DevinMarshal - 09-27-2017 10:11 PM
I completely agree with you guys.