We've got a new feature for testing: robots.txt for Manila sites. We're
planning on releasing the feature through root updates, probably
tomorrow, but we'd like your help testing first.

*** How to test:

1. Download this file and import it in Frontier using the Open command
in Frontier's File menu:

http://jake.userland.com/gems/robotsTxt01.fttb

2. In the imported table, there's a script named install. Run the
script to install the changes.

3. Open your site's robots.txt file in the web browser. If your site's
URL is http://www.mysite.com/ then your robots.txt file is at
http://www.mysite.com/robots.txt. The file should look like this:

User-agent: *
Disallow: /stats/referers

By default, the referers page is blocked to discourage referer spam.

4. Open your site's robots.txt editor at /admin/editRobotsTxt. If your
site is at http://www.mysite.com/ then the URL for the robots.txt
editor for your site will be at
http://www.mysite.com/admin/editRobotsTxt.

On this page, you can add or delete paths in your site's robots.txt
file.

*** robots.txt and sub-sites

The feature supports sub-sites. Crawlers will attempt to read
robots.txt from the top-level of a given domain. If your site is a
sub-site, then restricted paths from the sub-site will be added to the
robots.txt file in the top-level site. For example, if there are three
sites on the server at the following URLs:

http://www.myserver.com/
http://www.myserver.com/site2
http://www.myserver.com/site3

The robots.txt file for the top-level site at
http://www.myserver.com/robots.txt will contain the following by
default:

User-agent: *
Disallow: /stats/referers
Disallow: /site2/stats/referers
Disallow: /site3/stats/referers

*** Feedback

If you have questions, problems or bugs to report, please send them as
a response to this message.

Thanks!
-Jake