User-agent: * # directed to all spiders Disallow: /onrshell/ Disallow: /search/ Disallow: /images/ Disallow: /shell/ Disallow: /onrscripts/ Disallow: /onrerrors/ Disallow: /databases/ Disallow: /reports/ User-Agent: vspider Disallow: / User-Agent: Bullseye Disallow: / User-Agent: EmailCollector Disallow: / User-Agent: EmailWolf Disallow: / User-Agent: ExtractorPro Disallow: / User-Agent: Crescent Disallow: / User-Agent: fastlwspider/1.0 Disallow: / User-Agent: SurfWalker Disallow: / User-Agent: GetWebPage Disallow: / User-Agent: lwp-trivial Disallow: / User-Agent: NICErsPRO Disallow: /