1 Topic by magnumit

  • magnumit
  • Member
  • Offline
  • Registered: 2024-04-09
  • Posts: 1

Topic: How I got Amavis + ClamAV stable

==== REQUIRED BASIC INFO OF YOUR IREDMAIL SERVER ====
- iRedMail version (check /etc/iredmail-release): 1.6.8
- Deployed with iRedMail Easy or the downloadable installer? installer
- Linux/BSD distribution name and version: Debian 12
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): MYSQL
- Web server (Apache or Nginx):Nginx
- Manage mail accounts with iRedAdmin-Pro? nope
- [IMPORTANT] Related original log or error message is required if you're experiencing an issue.
====

Hi All,

Just posting this here in case another user encounters similar issues as myself. Bit of background, I run a small mailserver for my domain and have used iRedmail since about 2009 to do so. I rebuild my mail server instead of upgrading between major Linux releases. End result was a very nice new (and shiny) iRedmail 1.6.8 on Debian 12. I have also upgraded roundcube to the latest version 1.6.6. 4GB RAM and 7GB swap.

All was going well but amavis would sporadically stop/deactivate which required manual restarting and then a flushing the postfix queue as a few hours would pass before I would catch on. Things work work fine and then the same thing would happen.

As had been noted in the forums in the past, ClamAV appeared to be the culprit. However, as all the fixes noted have been included in the current version, I was left scratching my head a bit. I have the RAM requirement met and also added SWAP to cover any out of memory issues.

After many hours of tinkering, playing with swappiness settings and just general head scratching. I came across modoboa/issues/2475 post on github from 2022 that seemed to resonate with me and the issues I was seeing.

It would appear that the default amavis settings in Debian set the child processes to 1. For reasons noted in the post that I agreed with I went ahead and adjusted the config as follows:

/etc/amavis/conf.d/50-user

$max_servers = 2;

/etc/postfix/master.cf

smtp-amavis unix - - n - 2 smtp

These settings have resulted in amavis being stable for mutiple days now without deactivation/crashes.

The settings above are worth a shot if you have similar issues

Cheers
Mark

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,121

Re: How I got Amavis + ClamAV stable

FYI https://docs.iredmail.org/concurrent.processing.html

The cause is ClamAV (not Amavisd), it requires more and more memory these years.
So for a low traffic mail server, reduce the concurrently processed messages to just 1 is an ideal and acceptable solution.

3 Reply by evenmoreconfused (edited by evenmoreconfused 2024-05-10 08:00:49)

  • evenmoreconfused
  • Member
  • Offline
  • Registered: 2017-06-19
  • Posts: 77

Re: How I got Amavis + ClamAV stable

ZhangHuangbin wrote:

FYI https://docs.iredmail.org/concurrent.processing.html

The cause is ClamAV (not Amavisd), it requires more and more memory these years.
So for a low traffic mail server, reduce the concurrently processed messages to just 1 is an ideal and acceptable solution.

Yes, it certainly is getting worse and worse. My swap usage keeps hitting 100% and I'm going to have to upgrade my cloud server to the next higher model.

I seems to me like amavisd is the main culprit, but maybe that's because the two seem to have a weird symbiosis that I don't really understand.

CPU%   MEM%  VIRT  RES Command
>0.0   5.2   1.57G 189M   clamd -c /etc/clamd.d/amavisd.conf
 0.0   3.3   195M  120M   /usr/sbin/amavisd (ch19-avail)
 0.0   3.1   187M  112M   /usr/sbin/amavisd (ch18-avail)
 0.0   3.0   182M  111M   /usr/sbin/amavisd (ch1-avail)
 0.0   2.8   338M  104M    netdata -P /run/netdata/netdata.pid -D
 0.0   2.7   1.50G 99.2M   mariadbd --basedir=/usr
 0.0   1.7   174M  61.8M  /usr/sbin/amavisd (virgin child)
 0.0   1.6   174M  59.7M   /usr/sbin/amavisd (master)

So clamd and mariadb are each about 1.7GB, and the amavisd's together make about 1.4gb.

4 Reply by Cthulhu

  • Cthulhu
  • Member
  • Offline
  • Registered: 2019-06-04
  • Posts: 914

Re: How I got Amavis + ClamAV stable

that's the reason you dont run stuff which has requirements that only barely fit your specs

5 Reply by evenmoreconfused

  • evenmoreconfused
  • Member
  • Offline
  • Registered: 2017-06-19
  • Posts: 77

Re: How I got Amavis + ClamAV stable

Cthulhu wrote:

that's the reason you dont run stuff which has requirements that only barely fit your specs

True. But it used to run happily on a 4GB instance at my cloud provider. Now I have to move it to an 8GB one, which is an extra $330/year -- and it's only because of the architecture of ClamAV.

Still, I suppose it's worth $330/yr to reduce the count of infected user machines I have to deal with.

However, I also notice (in the nightly reports) that only a few messages are getting filtered anyway

 --------------------- Amavisd-new Begin ------------------------

      570   Total messages scanned ------------------  100.00%
   60.191M  Total bytes scanned                     63,115,137
 ========   ==================================================
 
      570   Passed ----------------------------------  100.00%
        2     Spam passed                                0.35%
        4     Spammy passed                              0.70%
      564     Clean passed                              98.95%
 ========   ==================================================
 
        6   Spam ------------------------------------    1.05%
        4     Spammy passed                              0.70%
        2     Spam passed                                0.35%
 
      564   Ham -------------------------------------   98.95%
      564     Clean passed                              98.95%
 ========   ==================================================

Is that normal?

6 Reply by Cthulhu

  • Cthulhu
  • Member
  • Offline
  • Registered: 2019-06-04
  • Posts: 914

Re: How I got Amavis + ClamAV stable

Amavis itself is a milter, it pipes mails trough a virus scanner and trough a spam scanner

awell it handles DKIM signing for outgoing mails, and checks for various other stuff (dkim, spf, ...)

the curlpit is clamav most of the time, since it gets invoked for each mail

Modify this file:
/etc/clamav/clamd.conf

Add this line:
ConcurrentDatabaseReload false

this results in way less memory usage and should prevent a OOM process kill which can break a mailserver till reboot

7 Reply by evenmoreconfused (edited by evenmoreconfused 2024-05-12 00:19:30)

  • evenmoreconfused
  • Member
  • Offline
  • Registered: 2017-06-19
  • Posts: 77

Re: How I got Amavis + ClamAV stable

Hmmm....

I am pretty confident I had done that ages ago when these memory issues first popped up. But I went to double-check and now can't find clamav.conf anywhere.

There is no /etc/clamav, but there is a /etc/clamd.d . In it are:

-rw-r--r-- 1 root root   574 Mar  3  2023 amavisd.conf
-rw-r--r-- 1 root root   538 Mar  3  2023 amavisd.conf.2023.03.03.01.48.51
-rw-r--r-- 1 root root 27527 Apr 26 02:13 scan.conf

find / -name clamav.conf shows only two files deep in /usr/share/logwatch which is about monitoring it, not configuring it.

Should I add that line to scan.conf? Or avamisd.conf?

--------------

PS: the command starting clamd is 

clamd -c /etc/clamd.d/amavisd.conf

so I will try putting the line in amavisd.conf and see what happens!

8 Reply by evenmoreconfused

  • evenmoreconfused
  • Member
  • Offline
  • Registered: 2017-06-19
  • Posts: 77

Re: How I got Amavis + ClamAV stable

Update: the line

ConcurrentDatabaseReload no

is, in my installation, included as a commented-out example in scan.conf (on line 210). Note it uses "no", not "false", but I believe both work in config files.

I have removed the line from amavisd.conf and uncommented it in scan.conf, since this is a more "by the book" way of  setting that option. That said, I expect that it will work in either location (or even both).

9 Reply by Cthulhu

  • Cthulhu
  • Member
  • Offline
  • Registered: 2019-06-04
  • Posts: 914

Re: How I got Amavis + ClamAV stable

it depends, clamav is a standalone scanner and can get invoded by a socket, and it is possible that it uses different config files for both stuff, but i did not dig that deep into it