# http://www.robotstxt.org/wc/norobots.html # dirs w/trailing slash refer to dir contents # names wo/trailing slash refer to _anything_ # beginning with that name # The following allows all robots to visit all # User-agent: * # Disallow: User-agent: * Disallow: *.txt Disallow: *.log Disallow: CVS Disallow: artists #Disallow: brdiary Disallow: catalog Disallow: counter/ #Disallow: images #Disallow: misc Disallow: pantry/ Disallow: php_include/ Disallow: raudio/ Disallow: words/