Robots.txt can be a file that exists on the root Listing of every Site and can be used to instruct serps on which directories/files of the web site they're able to crawl and include inside their index. Search engines like yahoo have just one intention only. They intention to offer https://news.rafeeg.ae