# Show the way to the site map file #Sitemap: http://www.example.co.nz/sitemap.xml # LinkWalker knows where to go. User-agent: LinkWalker Disallow: / # Archive.org was hammering site - Now knows where to get off User-agent: archive.org_bot Disallow: / User-agent: MJ12bot Disallow: / User-agent: Twiceler Disallow: / User Agent: * Disallow: /images/ Disallow: /page-not-found/