Understanding Robots.txt and Sitemap.xml

For those unaware, there are two special files that almost all websites use to influence how visitors see them. Not human visitors, but bots — web crawlers, search engines, one of the many various “internet archival” sites, you name it. Maybe you’d like to know this because you’re building up a website or service, or maybe you’re just curious on how to read them. In either case, let’s get to explaining.