The resource of a software come from these three components:

1. Memory usage
2. CPU usage
3. Disk usage

However there is coherence between this resources I try to handle these as separate parts of the project. But if a modification has a big impact in one component but this decrease the overall performance I have to doubt it.

• Perl memory management has a weird behaviour: it doesn't free memory even the process exited from the scope (subroutine etc.) . If big objects are created than they have to be undefed at the and of scope.

• I take same profiling with tool named Gladiator. It go through the writes out the number of SVs, hashes. I have put this probe in the beginning of the sympa.pl's infinit loop. After processing messages the number of SVs slightly increased. The reasons:
  • Global variables

• Static classes

• Memory leaks

With the first two I can't do anything. Maybe regularly should be freed in the loop if didn't consists valid information. The third one is a serious problem I have to go deeper and check around the subrutines...

• The bulk.pl is one of the daemons which is leaking in our production environment. I haven't found the source of these leaks. I studies the spamassassin daemon which has a similar structure (multiprocess, all process the same task). The difference is that the spamd master daemon doesn't handle any messages, it only checks the conditions and forks if new daemon is needed. The master bulk daemon should do the same.

• Bulk::messageasstring seems to be dead code.

• Terrible memory usage and leaking...supply

Remarks by David

Yes... Please, help us!

• 27th of April: After debugging a whole night, I found a memory leak in external Template module. Only the version 2.20 is affected, unfortunately Ubuntu 10.04 ships this version.:(

• Drop message priorities: prioritizing the message sounds good at the first time, but I don't see any advantages: a non-overloaded system every message get through the sympa in few seconds even at a large traffic server. On the other had classifying a message takes some resources and make the code more complex.

• Cleanspool functions should be moved to task_manager.pl. They can hang the message process.

• The queue checking algorithm is O(n*n*log(n)) because every time the daemon process a message look into the spool for new messages. It should process all founded message in the spool.

• The currently used filesystem queue model has two major disadvantages:

1. The “queue” C program is called for every recipient and it has to be written to the disk in n copy (where n is the number of recepients) and has to process every copy of the letter
2. sympa.pl doesn't know about the other recipients. It is hard to implement eg. a crosspost-filter

My suggestion: At the MTA→sympa communication implement LMTP. LMTP is an extension of ESMTP where the receiver can reject a letter on per recipient basis. So no need to write the message to the disk.

Disadvantages:
  * Limited queue: because of the sympa.pl is a single thread app, the queue is limited by the OS TCP queue
  * The priority system can't be implemented: the sympa.pl become a FCFS system.

Facts noted by Olivier Salaün.

sympa.pl processes see their memory concumption brutally rise.

Characteristics:

• only sympa.pl is concerned
• the memory is never freed after this rise
• the memory rises correspond to large messages processing
• they happen even if the message is moved to bad (for example if it is too large)

Methodology:

1. launche sympa.pl
2. ps auwx to know the size of the process (le figure to take into account is RSS, expressed in kB)
3. send a big message to a list
4. ps auwx efter the message was processed

Tools:

• /usr/local/sympa/src/tools/test-mem-leaks.pl: a small daemin which load a message at each loop. Usefull to understand which functions make the memory grow.
• /usr/local/sympa/src/patches/memwatch.patch: A patch to apply on Log.pm to add the momoery use at each log entry. It is possible to increase the log level and even add development traces to narrow on what part of the code is faulty.

A test reveals the problem while sending a 26 MB message, rejected by sympa.pl:

1. process startup : 47MB
2. end of message processing : 1 273MB

The parts of the code in which memmory consumption rise (base=message de 26Mo) :

• +20Mo : Message::new / MIME::Parser::read()

</box>

• +52Mo : Message::new / MIME::Entity::as_string()

• +26Mo : sympa::DoMessage / (parameter passing?)

• +100Mo : sympa::DoMessage / do_log using the argument $message→{'msg_as_string'}

• +26Mo : sympa::DoMessage / creating a variable with $msg→as_string

• +200Mo : sympa::DoMessage / call to tools::checkcommand()

• +80Mo : sympa::DoMessage / call to reject_report_msg()

• +80Mo : List::send_file / dup_var

• +260Mo : mail::mail_file / TT2 parsing
• +170Mo : mail::mail_file / reformat_message
• +26Mo : mail::mail_file / passing a parameter to sending()

• +50Mo : mail::sending / $msg→as_string

To sum up, the memory consumption rise when:

1. a data structure is duplicated (dup_var ou $msg→as_string)
2. large data structures are passed as subroutines arguments.

Possbile solutions:

• For (1) : increase the exploitation of the Message object; don't duplicate data, don't parse the same content several times.

• For (2) : This problem seems to come from Perl. Deeper digging is required to solve it.