Groupware around mailing lists with Sympa
Olivier Salaün / Serge Aumont
Historically, mailing lists have been a very important factor for the growing of internet usage.
We must not forget that ListservTM was the main bitnet service at the beginning of the 80s. Mailing lists have been the unique groupware tool for a long time !
Mailing lists have decisive trumps :
e-mail is a universal service of the Internet. You can reasonably consider that each group member has email access.
unlike forum, web or news, email is a push technology.
unlike chat or videoconferencing, email is an asynchronous service : there is no need to make an appointment.
email address is a reasonable person identifiant (not as good as a Distinguish Name:), so mailing lists subscription process is an effective definition of the set of members of the group. In particular, any good mailing lists software provides a control on subscription by a list administrator.
Since groupware can't ignore mailing lists why not build a groupware architecture around mailing lists ?
(In the following article ML will stand for mailing lists)
Majordomo-like MLs are not enough for educational groupware. Classical ML provides too poor features. Integrating ML and other applications becomes difficult.
Sympa is a free software, running on UNIX system. It has been developed by the C.R.U. (the French Universities Network Team) to solve various problems with classic ML software such as full MIME support, enhanced security, high performances for large services, dynamic web interface with additional groupware features, dynamically defined list members.
Sympa recognises incoming complex MIME structures. When receiving mail commands in Multipart/alternative format, it will decode each part and looks for commands in text/plain parts. Sympa's bounce analyser also recognises RFC1891 compliant message deliver status.
Sympa also provides a good MIME compliance with messages it sends, especially on the following points :
digest messages are enclosed in a multipart/digest MIME structure.
footers and headers are clean MIME parts
command reports use the correct Charset, conforming to current locale
Early developments of Sympa already focussed on internationalisation. Every translation is out of the code, thereby making the translation job easier. Command reports have yet been translated to French, Spanish, German, Italian, Finish and Chinese. The web interface creates more constraints for translators since HTML pages have a more complex structure than a message body. The web interface has currently been translated to French and Spanish.
Sympa provides a hierarchical internationalisation that aims at using the more adapted language when communicating with users. Sympa has a default language setup (English) ; you may define one for your site ; lists have an official language ; users may choose their preferred language. Sympa will initialise the user's preferred language with the official language of the first list he/she subscribes to.
A scenario is an ordered list of rules that determines if the requestor is allowed to perform the current command. Each rule consists of a condition part, a required authentication method (smtp, md5 or smime) and a resulting action. The condition part is evaluated with a set of environment variables ([listname],[sender],[header->From],?).
Example: controlling "Who can post a message to a list".
is_editor([sender],[list]) smtp -> do_it
the message comes from the editor
it is distributed anyway
!is_subscriber([sender],[listname]) smtp -> reject
Message from non-subscriber are rejected
match([header->Content-type],/multipart/) smtp -> editor_key
attachments => submit to editor
true() smtp -> request_auth
Other messages need to be authentified
true() md5 -> do_it
Other messages authentified are distributed
This scenario has the following behaviour :
reject messages from non-subscribers
submit multipart messages to list moderator
request confirmation before sending accepted messages
Every command in Sympa is controlled by scenarios. The listmaster can thereby change the behaviour of commands to anything that is expressible by a scenario.
Sympa includes a web interface to all ML services. It has the following properties :
Shared data : all data are shared with the mail robot itself, so no additional management is needed for the web interface.
Single CGI : A single CGI provides different views of the ML service including a searchable lists directory, a list of subscriptions and a complete view of each list depending on the users privilege.
Admin : it includes administration features for list owners, lists editors and listmaster (such as bouncing subscribers management, list creation and configuration, etc)
Easy to customize : the HTML of the web interface has been separated from the code.
Persistence : for performances purpose, the CGI was written to be persistent in memory using FastCGI.
Sympa's web interface proposes archives of the ML and can also restricts access to these archives. The control applied to the archives may be chosen for each list and is based on ML-specific notions. Typical politics are public or private (for subscribers only) archives but may
be easily extended by listmaster (using scenarios). This is an essential feature to provide to private workgroup.
Sympa associates a shared web space to each list. It proposes basic functions (create dir, upload file, rename). Why adding this feature in Sympa ?
To gain from the Sympa's authentication scheme. A directory/file of the shared web can thereby be restricted to a population (subscribers/owners of a list) for either read or write access. To define privileges in Sympa you select a scenario.
Because they are based on email, MLs rely on messages From: header fields to identify people and give them corresponding privileges. This is insufficient because From: header fields are easy to forge. Sympa (like many other ML software) uses confirmation keys for verifying the requestor identity. The drawback of this method is that it makes commands heavier to use. Sympa's scenarios are a way to define for the entire site or for a list who is allowed to perform each operation but also what is the acceptable authentication method.
For some lists usage, confirmation keys for verifying the requestor is not secure enough. A better way to identify people is to use digital signatures. Sympa is able to use S/MIME signature as an optional or mandatory authentication method for each operation.
Sympa can also distribute S/MIME encrypted messages. In this case, the list has its own X509 certificate. A nice configuration is to require S/MIME signed message for subscription, so Sympa can store each subscriber's X509 certificate. The welcome message is automatically signed by Sympa, so each subscriber gets a copy of the list's certificate. Then any subscriber can send an encrypted message to the list, Sympa decrypts the incoming message and crypt it for each recipient of the list.
Control access on web documents is commonly handled by the HTTP server. With Apache you define a user/password database (.htpasswd) and then access rules on documents. This was not adapted to Sympa that already manages a user data. Moreover it could not rely on platform-dependant systems.
Every web operation with Sympa is performed via a single CGI, providing the access control feature. This ensures a uniform and complete authentication scheme. User password are stored in the preference table of Sympa's database. Passwords may be reminded and changed from the web. Authentication persistency is performed using HTTP Cookies. Cookies only carry identification information (this HTTP client is firstname.lastname@example.org), whereas privileges (based on scenario) are evaluated for each requested operation.
If you have a HTTPS server running, then you can configure it to ask for optional user authentication based on X509 certificates. If the user does not have a personal certificate, he/she will have to authenticate using a password in an encrypted session. If the server receives a user certificate, no password is required. Thereby usage of HTTPS in Sympa makes it stronger and more user-friendly.
Subscribers are called this way because they subscribe to the list ; ML population evolving with subscriptions and unsubscriptions. Now imagine you know exactly who is supposed to be on your ML and you don't want strangers in it. A good example of this usage is an administrative ML with all university students.
Sympa will allow you to define a ML, with a list of subscribers dynamically extracted from your RDBMS or LDAP directory. For Sympa to use your database information, you need to define user data sources (mainly the SQL/LDAP query) in the ML configuration file. Various universities now creates systematically at least one ML for each educational unit. This implies thousand of ML, but of course all the student records are already in the scholarship directory so Sympa can extract it dynamically. No additional management when renewal of students happens.
Example of an LDAP inclusions :
Extracting email attribute for a selection
Extracting emails from a GroupOfMembers
suffix dc=cru, dc=fr
suffix ou=students, o= tempere.edu.se
Another way to provide a nice integration of Sympa services in an existing information system is to share the authentication process with other application. The goals can either be to share the authentication form or only the user's password.
Sympa's web interface uses a HTTP cookie that contains the user email after authentication. It can be recognized by some other application using a dedicated library. In this case, users can switch from Sympa to other services without further authentication process.
Nowadays the ideal way of managing user authentication is to store authentication user information in a central LDAP directory and all authentication operation should be able to use LDAP. We plan to include an auth_LDAP module in Sympa for April 2001 ; it will query LDAP directories for set Internet domains and manage passwords for others.
Sympa, distributed with pre-configured command scenarios and template files, is a scalable product. Its usage reflects this quality and we know Sympa is used by :
Almost all French universities and others in Italy, Germany, USA, Japan? They mainly benefit from its web interface (with both internet & intranet views), LDAP/SQL extraction features.
ISPs (Voila.fr, List Avenue) for its internal structure (RDBMS) and high delivery performances.
Companies because Sympa provides a secured extranet definition.
Non English-speaking users, finding in Sympa real internationalised product.
Sympa was first released in April 1997 and because ML are still up to date tool, this project is still under active development . We plan in future version several new features such as :
User format reception option will allow users to receive messages in there prefered text format (as far as the initial message content type is multipart/alternative).
Pluggin filters to scan incoming messages with antiviral applications.
New LDAP usages for authentication, dynamic list owner definition and queries in scenario conditions.
Current virtual robot definition will be improved with a new concept in Sympa : list policies. Because Sympa allows a lot of different list usage, the list configuration is complex and may be difficult to master for list owners. In addition listmaster may loose control over lists, being misappropriated regarding the initial project. The list policy will be a set of constraints on list configurations, defined by listmaster. This will ensure list type integrity and make owners configuration job easier. Typical list policies would be "newsletter", "workgroup", "discussion","webforum","hotline", "administrative".