Tek's Domain


Understanding Robots.txt and Sitemap.xml

2020-02-19 7 min read Tech explained Webdev and administration Teknikal_Domain Unable to load comment count

For those unaware, there are two special files that almost all websites use to influence how visitors see them. Not human visitors, but bots — web crawlers, search engines, one of the many various “internet archival” sites, you name it. Maybe you’d like to know this because you’re building up a website or service, or maybe you’re just curious on how to read them. In either case, let’s get to explaining.

Continue reading