Teknikal_Domain's Personal Blog

#<ENT:NTA:NnT:SSrgS:H66-198:W200:CBWg>

Understanding Robots.txt and Sitemap.xml

2020-02-19 7 min read Tech explained Webdev and administration Teknikal_Domain Unable to load comment count

For those unaware, there are two special files that almost all websites use to influence how visitors see them. Not human visitors, but bots - web crawlers, search engines, one of the many various “internet archival” sites, you name it. Maybe you’d like to know this because you’re building up a website or service, or maybe you’re just curious on how to read them. In either case, let’s get to explaining.

Continue reading