Does this idea inspire anyone? Do you have specific technical
suggestions on elements that could be used to pull this together.
What technical specifications/feature would you add? Anyone want to
start building this?

Cheers,

Steven Clift
Democracies Online

P.S. Join the Democracies Online Code e-mail list to discuss
technical options and ideas: do-code-subscribe@yahoogroups.com




Ultimate Archive
With Joint E-mail and Web Conferencing Notes

Draft 1.0 by Steven Clift, October 18, 2001
clift@...
Copyright 2001, Steven Clift

NOT FOR PUBLIC REDISTRIBUTION

Base

Using a mySQL database each message will be stored for optimal and
quick retrieval and presentation. Fields include: Sender Name,
Sender E-mail, To:, CC: e-mail, CC: list crosspost, Subject, Message
Body, perhaps others (http://www.faqs.org/rfcs/rfc2822.html ).


Output

Logical linkable address with permanence - Each message is given a
logical address - even if called from the database each time. Option
to generate permanent HTML file with server side includes (or
equivalent to ensure that that top, bottom, side navigation and look
and feel can be uniformly updated across -part- or -all- of the
archive.

Pre-Set Web Views -
- Messages Today (last 24 hours) - recent at top by subject line
- Messages this Week (last seven days) - recent at top
- Messages by Month (each month with subjects listed by date with
replies nested in earliest to latest
- Multiple post message threads can be called up as linear thread
based on message ID and time, or subject line similarity, replies
with Message ID and new subject become new thread in archive
(similar to Google's ( http://groups.google.com ) newsgroup archives -
linear provides quick and efficient archive reading and really only
applies to discussion lists, although request to the database for
linear text files of all messages from a certain time period, list,
search term request could be an option.)
- Current web archive systems of note include:
http://www.mhonarc.org
http://www.hypermail.org
http://sourceforge.net/projects/php-mail-arc/ (don't know anything
about this one)
http://www.mail-archive.com


Spambot protection

E-mail archives are susceptible to spambots
(http://www.turnstep.com/Spambot/ )which harvest e-mail addresses.
By default all e-mail addresses would become reply-to links with web
forms (additional security required to make sure archive isn't used
to relay spam) for comments to the author.


Additional Views

Templates promoting comparative views would be modular to allow
sharing of unique ways of presenting e-mail list posts on the web.
- Grid or Comparison View - A one screen table presenting e-mail
newsletters from multiple, but similar sources on one page (i.e. all
e-mail newsletters from state political parties on one day).
- Columns View - Presenting two to six? active e-mail lists side by
side with longer list of recent subject lines broken by date
- Today Column - Presentation of one or multiple lists in one column
through an RSS feed
(http://www.gotzespace.dk/links/XML/Syndication/RSS/ ) that calls up
the archive in real-time each time that remote page is accessed.
(i.e. the three most recent subject lines on all E-Democracy lists
presented in a flexibility determined order in a column on the E-
Democracy home page.) Third party sites should be able to insert
simple HTML code to pull up like Moreover allows (
http://w.moreover.com/webmaster/ ). Mail-Archive generates (
http://www.mail-archive.com ) RDF files which can be used with RSS -
for example ( http://www.mail-archive.com/do-
wire@.../maillist.rdf - try Netscape to get this text file, IE
doesn't like the extension). Here is an example of an output from
an older RDF file ( http://my.userland.com/viewChannel$3822 ).

List Information

Using elements from the prototype OpenGroups (
http://www.opengroups.org - moving servers right now) standard those
adding lists to an archive would have the option of including
information about the list and links back to the source. The links
would be automatically checked weekly and broken link information
sent to the administrator/list archiver. Selection of the
appropriate Netscape Open Directory ( http://www.dmoz.org )and Yahoo
subject trees would also be included and optionally be embedded into
the message archives.


Statistics and Trends

Graphical charts and statistical charts will be available from each
list information page. In addition, Views based on trends across an
archive could be presented including, most cross-posted message or
message element, lists with higher than average traffic, authors with
higher or notable posting behavior. From a civic perspective it
would be useful to be able to compare which lists have more or less
diverse or varied participation.


Monitors and Notification

Think MyArchive - The ability to be actively notified via e-mail
about new posts of interest based on various search features. The
default setting would e-mail the subject lines and archive links to
relevant posts. See http://www.spyonit.com for potential
notification options. I suggest daily or weekly options. Future
versions could incorporate more immediate notification via IM and
other tools.


Content Analysis

Tools that allow us to identify trends would be useful. Which forums
seem most deliberative, which use harsh language, etc..


Civic Groups - Optimized Joint E-mail and Web Conferencing

The archive is only half of what I would like to see. While
archiving third party e-mail lists and newsletters has value,
integration of web view/archive features with existing open source e-
mail list software like Mailman http://www.list.org or perhaps Sympa
http://listes.cru.fr/sympa/ would tremendous value. More mailing
list programs (
http://sourceforge.net/softwaremap/trove_list.php?form_cat=30 ).

To goal is to viably combine the audiences and formats of these
different formats such that the largest audience for any one current
discussion can be maximized while still retaining a web view with
most of the structure and features of basic web forums. This is not
for the advanced web-based forum seeker, but would do much more than
YahooGroups to allow viable and meaningful web-based participation
with select e-mailing of subject line digests. Because e-mail lists
are time sensitive, posting via the web can be time limited to ensure
conversational contitunity.

It makes sense to build additional features that would optimize use
over extremely low-bandwidth environments where subscribers can do
all list administrative functions over e-mail and in addition to full-
text digest features also have the option for daily or weekly
subject line digests with the ability to request specific posts and
estimated linear thread digests of posts on the same subject. A
setting that allows lists to default to posting of new subject posts
only (perhaps moderated) where each participant must take an e-mail
action to participate (post or lurk via e-mail) in the following
discussion. This would mediate the message volume on larger lists
and allow sub- groups to form organically. A late request to join a
subject thread could also result in e-mail delivery of missed
messages. All the threads would find a home on the web archive and
a process for permanently establishing/nominating new e-mail lists
from long standing subject thread sub-groups could be established.


Member Directory

Every e-mail address on a list will have an automatically generated
member directory page with a logical url (i.e. http:
www.mnforum.org/member/clift@...).

Each participant would have access to update and add information and
links to their member page (with e-mail verification to confirm
updates). Member pages would include links to recent forum posts
across the archive and include substance and style ratings provided
by other members. Each post to an e-mail list would insert the link
to the member page at the bottom of each post. More complicated
database examples include MIT's Junior Summit in 1998 which used the
Lyris mailing list software which allowed them to take every post and
machine translate what was e-mailed to each participant based on
their preferred language (the preferred language version was followed
by the post in its original language).


Ratings

A number of web based systems use multiple moderator ratings to help
other readers screen for substantial contributions (i.e. Slashdot
http://slashdot.org/faq/com-mod.shtml#cm600 ). With e-mail,
unmoderated posts are sent out before they are reviewed by anyone,
even in a moderated list with one or multiple moderators providing
ranking this may be a big bottle neck. In the Civic Groups system,
ratings of substance and style would be cumulative and done by
interested members.

The purpose of member ratings is to provide context about
participants to other participants and promote self-governance over
centralized control or labor intensive moderation.

Substance - Via clickable web links within a post, evaluative +1
could be given to posts of substance. In the E-Democracy
environment, evaluation based on political opnion needs to be avoided
or channeled appropriately. "This post provide substantive content"
may be the optional link to click in the X-header (i.e. "Click here
for list options" would be replaced with "Click to rate message or
for list options").
In the web archive posts reach X level of substance votes could be
colored differently in the subject line or elevated through specific
views.

Style - E-Democracy receives ten times more complaints about personal
attacks than about the content or political views shared in a
message. Allowing people to vote negatively on the style of posts
would help establish which participants in aggregate offend others
with their style of participation. This mark against such
participation provides two values - self-correction by offensive
participants, easier identification of uncivil participants by other
members to help them make evaluative judgments about that person
based on the feelings of the group. This is essential to keep the
quiet voices/readers that matter in the "real world" tuned into the
discussion and mediate the loudest voices that tend to get more
shrill and belligerent over time.

The marks for substance and mark against style would be presented in
the member directory and as an e-democracy volunteer suggested,
inserted into every post by that member.

What scale? What formula? What about abuse or attempts to influence
the ratings for illegitimate reasons? My early thought is that both
positive and negative ratings would migrate toward zero over time
(or based on posts) and that consistent positive or negative rating
by one person of another would be weighed less than infrequent
rating combinations. This would mediate negative ratings for
political or personal purposes by one individual. Rating
participation will likely be light and easy to influence by a few
active evaluators, therefore special outreach and/or volunteers will
be required to build a base of data.

More to come …

^ ^ ^ ^
Steven L. Clift - W: http://www.publicus.net
Minneapolis - - - E: clift@...
Minnesota - - - - - T: +1.612.822.8667
USA - - - - - - - ICQ: 13789183