Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
dev:custom_archiving [2009/02/18 11:51]
david.verdin@cru.fr
dev:custom_archiving [2011/03/15 10:02] (current)
david.verdin@cru.fr
Line 14: Line 14:
  
 ^Field name  ^Data type  ^Field description ​ ^ ^Field name  ^Data type  ^Field description ​ ^
-|message_key ​ |int(11) ​ |the SMTP message identifier ​ |+|message_key ​ |int(11) ​ |an autoincremental numerical key  |
 |message_id ​ |varchar(150) ​ |the SMTP message identifier ​ | |message_id ​ |varchar(150) ​ |the SMTP message identifier ​ |
 |group_id ​ |varchar(150) ​ |the name of the mailing list to which the message was sent  | |group_id ​ |varchar(150) ​ |the name of the mailing list to which the message was sent  |
 |timestamp ​ |int(11) ​ |the epoch date when the message was sent  | |timestamp ​ |int(11) ​ |the epoch date when the message was sent  |
-|subject ​ |varchar(100)  |the content of the SMTP Subject field in the orginal message ​ |+|subject ​ |varchar(300)  |the content of the SMTP Subject field in the orginal message ​ |
 |message ​ |text  |the full SMTP message, including all the headers ​ | |message ​ |text  |the full SMTP message, including all the headers ​ |
-|uid  | |?  ​|+|uid  | varchar(150) ​ ​| ​we have a header X-UID in the message that we want to save here | 
  
-The message id doesn'​t seem to be a good primary key. If a message is sent to two lists, we could have a primary key collision. For this reason, we add an arbitrary numerical primary key. 
  
 ===== Customization mechanism ===== ===== Customization mechanism =====
  
-A parameter in sympa.conf activates / desactivates this feature : ''​custom_archive''​.+A parameter in sympa.conf activates / desactivates this feature : ''​custom_archiver''​.
  
-We add a test to archived.pl. When archiving a message, if the ''​custom_archive''​ parameter is set to ''​on'',​ an external script is called -- instead of the code using MHonArc -- that will store the message into the chosen database.+We add a test to archived.pl. When archiving a message, if the ''​custom_archiver''​ parameter is set to ''​on'',​ an external script is called -- instead of the code using MHonArc -- that will store the message into the chosen database.
  
 This script can't be used in a similar manner as what we thought for hooks, as it //changes// Sympa behaviour and doesn'​t only add new operations to an existing sub. This script can't be used in a similar manner as what we thought for hooks, as it //changes// Sympa behaviour and doesn'​t only add new operations to an existing sub.
Line 35: Line 35:
  
   * **Content**:​ The script contains only the code specifically used for archive storage in an RDBMS. It uses the Sympa scripts and libraries to work. It then contains the path to the Sympa bin directory. However, details about the connection to the database will have to be final user defined by editing the script. We don't think it is relevant to include the connection parameters to the archives database in one of the Sympa configuration files. Consequently,​ the distributed script must be considered as an example script. It can remain in the sources or be installed in one of the Sympa directories. Either ways, it must be copied in a place that will not be overwritten during the next Sympa upgrade.   * **Content**:​ The script contains only the code specifically used for archive storage in an RDBMS. It uses the Sympa scripts and libraries to work. It then contains the path to the Sympa bin directory. However, details about the connection to the database will have to be final user defined by editing the script. We don't think it is relevant to include the connection parameters to the archives database in one of the Sympa configuration files. Consequently,​ the distributed script must be considered as an example script. It can remain in the sources or be installed in one of the Sympa directories. Either ways, it must be copied in a place that will not be overwritten during the next Sympa upgrade.
-  * **Distribution**. In the Sympa sources, you will find the script in the ''<​sympa_home>/​bin/​etc/​scripts/''​ directory under the name ''​custom_archiving.pl''​. +  * **Distribution**. In the Sympa sources, you will find the script in the ''<​sympa_home>/​bin/​etc/​scripts/''​ directory under the name ''​db_archiver.pl''​. 
-  * **Name and path**. We have three solutions for the name and path of the script +  * **Name and path**. We use a parameter named ''​archiver_path''​ that replaces ​the existing ''​[[https://​www.sympa.org/​manual/​web-interface#​mhonarc|mhonarc]]''​ parameter. This parameter can be defined at the list or virtual level. You can use any script ​name or path for the script. You can have as much versions ​of the script ​as you want. One for each list if you like. 
-    - We define ​it once and for allIt won'​t ​be changed without changing the code of archived.plIt could be ''​etc/​custom_bin'​'. + 
-    We let people customize ​the path only. It could be a parameter like ''​[[manual:conf-parameters:part1#static_content_path|static_content_path]]''​ which could be later used for hooks, and whose purpose would be to host custom codes for a Sympa installI propose ​the name ''​custom_code_path''​. +====== Here it is ! ====== 
-    - We let people customize the script ​name. In this case, it must be a robot parameter. I propose the name ''​custom_archiving_script_path'' ​for this parameter+ 
-  the two later options can be used separately from one another: ​you can force to use a specific path and customize the script name and the otheer alternative is possible.+You can download ​ it as a tar file which is version **5.4.6ca** or a patch that can be applied in versions **5.4** 
 +  * {{:​dev:​sympa-custom-archiver.tar.gz|}} 
 +  * {{:​dev:​custom_archiver_5.4.patch.txt|}} 
 + 
 +This modification is allready in Sympa 6.0 
 + 
 +You can use this version just now. As long as you don't modify wwsympa.conf,​ it won't change Sympa's behaviour. You will introduce the new archiver engine by changing //​custom_archiver//​ parameter in wwsympa.conf
 +  
 +The tar include //​custom_archiver.pl//​ a sample archiver that does what you ask for. CAUTION: Note that Sympa always guess that the custom archiver script accepts two variables : "--list" whose value is the full name of the list (including the domain part) and "​--file"​ that will contain ​the absolute ​path to the message file that will be archived. Each time the custom archiver is called, it is called with these two parameters with their values filled. 
 + 
 +To use our sample custom archiver, you need to: 
 + 
 +  - create the database in which you want to store, with table having the fields we defined here: [[:dev:custom_archiving#storage_structure]] 
 +  - modify the wwsympa.conf ​to add the following paragraph: ​ <hi #​98fb98>​custom_archiver ​ <​absolute_path_to_custom_archiver.pl></​hi>​ 
 +  - restart archived.pl (you may check everything using debug mode : "​archived -d" 
 + 
 +In archived.pl, ​the parameters related to the database are: 
 +<​code>​ 
 +host name'localhost' 
 +database type: 'mysql'; 
 +database ​name: '​custom_archive';​ 
 +table in which data are stored: '​archive';​ 
 +user name'toto'
 +password: ​'tutu'
 +</​code>​ 
 + 
 +Either you use this values to test or you change them in custom_archiver.pl.  
 + 
 +If you have any problem, please report first by mail with as many details as possible, ​you can also phone Serge : +33 2 2323 7147 ((often not in my office but ...)), we will pleased ​to help youLet us known if this does what you ask for. 
 + 
 + 
 + 
  • dev/custom_archiving.1234954300.txt.gz
  • Last modified: 2009/02/18 11:51
  • by david.verdin@cru.fr