Social media marketing Influence: Well known developments on social media can spill above into engines like google, influencing the lookup volume of similar key phrases. txt file is then parsed and can instruct the robot as to which webpages aren't to generally be crawled. Like a online search engine crawler https://www.youtube.com/watch?v=7OfVkYHB1OY