An interesting post about robots.txt and favicon.ico.
Honestly I agree to the concept that “web is baed on links” so there is no reason to fishing robots.txt and favicon.ico … but at the same time there is no logic to read a page to discover where robots.txt is and then see that you can’t index anything in the site.
I think that crawlers can fish just robots.txt, but favicon.ico should be linked from the inside of the pages and not just fished.
