1 (edited by Neutro 2019-06-07 14:45:41)

Topic: clamd wont start and use 100% cpu

==== REQUIRED BASIC INFO OF YOUR IREDMAIL SERVER ====
- iRedMail version (check /etc/iredmail-release): 0.9.9
- Deployed with iRedMail Easy or the downloadable installer? downloadable
- Linux/BSD distribution name and version: Centos 7.6
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): MariaDB
- Web server (Apache or Nginx): Nginx
- Manage mail accounts with iRedAdmin-Pro? no
- [IMPORTANT] Related original log or error message is required if you're experiencing an issue.
====

Hey folks,

I've reinstalled my mail server from scratch a few days ago.

After the install i noticed that a process "clamd" ran by user "amavid" was taking 100% cpu. So i disabled amavis alltogether in postfix.

But i noticed today that since amavis also manages DKIM i can't disable it big_smile

So i enabled amavis back in postfix and found out that the problem was caused by the clamd@amavisd service.

If I run systemctl start clamd@amavisd the ssh command hangs forever, using another ssh windows systemctl status clamd@amavisd shows:

juin 07 08:00:34 smtp.ah1z.com systemd[1]: Starting Generic clamav scanner daemon...
juin 07 08:00:34 smtp.ah1z.com clamd[22034]: WARNING: Ignoring deprecated option AllowSupplementaryGroups at /etc/clamd.d/amavisd.conf:21
juin 07 08:00:34 smtp.ah1z.com clamd[22034]: Received 0 file descriptor(s) from systemd.
juin 07 08:00:34 smtp.ah1z.com clamd[22034]: clamd daemon 0.101.2 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
juin 07 08:00:34 smtp.ah1z.com clamd[22034]: Running as user amavis (UID 990, GID 987)
juin 07 08:00:34 smtp.ah1z.com clamd[22034]: Log file size limited to 1048576 bytes.
juin 07 08:00:34 smtp.ah1z.com clamd[22034]: Reading databases from /var/lib/clamav
juin 07 08:00:34 smtp.ah1z.com clamd[22034]: Not loading PUA signatures.
juin 07 08:00:34 smtp.ah1z.com clamd[22034]: Bytecode: Security mode set to "TrustSigned".

While it's trying to start, it eats up all CPU as shown in "top":

  PID    USER      PR  NI    VIRT       RES       SHR   S    %CPU     %MEM     TIME+    COMMAND
22334  amavis    20   0  364672   254996   3240   R      96,4      13,5    0:24.45       clamd
Jun  7 08:00:34 smtp clamd[22034]: Received 0 file descriptor(s) from systemd.
Jun  7 08:00:34 smtp clamd[22034]: clamd daemon 0.101.2 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Jun  7 08:00:34 smtp clamd[22034]: Running as user amavis (UID 990, GID 987)
Jun  7 08:00:34 smtp clamd[22034]: Log file size limited to 1048576 bytes.
Jun  7 08:00:34 smtp clamd[22034]: Reading databases from /var/lib/clamav
Jun  7 08:00:34 smtp clamd[22034]: Not loading PUA signatures.
Jun  7 08:00:34 smtp clamd[22034]: Bytecode: Security mode set to "TrustSigned".
Jun  7 08:02:05 smtp clamd[22334]: Received 0 file descriptor(s) from systemd.
Jun  7 08:02:05 smtp clamd[22334]: clamd daemon 0.101.2 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Jun  7 08:02:05 smtp clamd[22334]: Running as user amavis (UID 990, GID 987)
Jun  7 08:02:05 smtp clamd[22334]: Log file size limited to 1048576 bytes.
Jun  7 08:02:05 smtp clamd[22334]: Reading databases from /var/lib/clamav
Jun  7 08:02:05 smtp clamd[22334]: Not loading PUA signatures.
Jun  7 08:02:05 smtp clamd[22334]: Bytecode: Security mode set to "TrustSigned".
Jun  7 08:03:35 smtp clamd[22606]: Received 0 file descriptor(s) from systemd.
Jun  7 08:03:35 smtp clamd[22606]: clamd daemon 0.101.2 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Jun  7 08:03:35 smtp clamd[22606]: Running as user amavis (UID 990, GID 987)
Jun  7 08:03:35 smtp clamd[22606]: Log file size limited to 1048576 bytes.
Jun  7 08:03:35 smtp clamd[22606]: Reading databases from /var/lib/clamav
Jun  7 08:03:35 smtp clamd[22606]: Not loading PUA signatures.
Jun  7 08:03:35 smtp clamd[22606]: Bytecode: Security mode set to "TrustSigned".
Jun  7 08:05:06 smtp clamd[23505]: Received 0 file descriptor(s) from systemd.
Jun  7 08:05:06 smtp clamd[23505]: clamd daemon 0.101.2 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Jun  7 08:05:06 smtp clamd[23505]: Running as user amavis (UID 990, GID 987)
Jun  7 08:05:06 smtp clamd[23505]: Log file size limited to 1048576 bytes.
Jun  7 08:05:06 smtp clamd[23505]: Reading databases from /var/lib/clamav
Jun  7 08:05:06 smtp clamd[23505]: Not loading PUA signatures.
Jun  7 08:05:06 smtp clamd[23505]: Bytecode: Security mode set to "TrustSigned".

I've noticed i had several lines of this kind in my maillog file:

Jun  7 06:36:44 smtp amavis[11355]: (11355-01) (!)connect to /var/run/clamd.amavisd/clamd.socket failed, attempt #1: Can't connect to a UNIX socket /var/run/clamd.amavisd/clamd.socket: Aucun fichier ou dossier de ce type
Jun  7 06:36:44 smtp amavis[11355]: (11355-01) (!)clamav-socket: All attempts (1) failed connecting to /var/run/clamd.amavisd/clamd.socket, retrying (2)
Jun  7 06:36:50 smtp amavis[11355]: (11355-01) (!)connect to /var/run/clamd.amavisd/clamd.socket failed, attempt #1: Can't connect to a UNIX socket /var/run/clamd.amavisd/clamd.socket: Aucun fichier ou dossier de ce type
Jun  7 06:36:50 smtp amavis[11355]: (11355-01) (!)clamav-socket av-scanner FAILED: run_av error: Too many retries to talk to /var/run/clamd.amavisd/clamd.socket (All attempts (1) failed connecting to /var/run/clamd.amavisd/clamd.socket) at (eval 134) line 659.\n

But i don't know where they come from since they dont show when i try to start the clamd@amavisd service

This is my amavisd.conf in /etc/clamd.d:

# Use system logger.
LogSyslog yes

# Specify the type of syslog messages - please refer to 'man syslog'
# for facility names.
LogFacility LOG_MAIL

# This option allows you to save a process identifier of the listening
# daemon (main thread).
PidFile /var/run/clamd.amavisd/clamd.pid

# Remove stale socket after unclean shutdown.
# Default: disabled
FixStaleSocket yes

# Run as a selected user (clamd must be started by root).
User amavis

# Path to a local socket file the daemon will listen on.
LocalSocket /var/run/clamd.amavisd/clamd.sock

These are the packages installed on my server:

clamav.x86_64                         0.101.2-1.el7                    @epel
clamav-filesystem.noarch              0.101.2-1.el7                    @epel
clamav-lib.x86_64                     0.101.2-1.el7                    @epel
clamav-scanner-systemd.x86_64         0.101.2-1.el7                    @epel
clamav-server-systemd.x86_64          0.101.2-1.el7                    @epel
clamav-update.x86_64                  0.101.2-1.el7                    @epel
clamd.x86_64                          0.101.2-1.el7                    @epel

I've searched on the forum and found out that some people had the same problem because they were running only 1GB RAM but my server is running 2GB ram (it's a vmware virtual machine running on ESXI 6.7).

So i've disabled the service clamd@amavisd for now and my mail server is running fine with dkim but without clamav scan.

Would appreciate if anyone got an idea of why this is happening?

Thanks!

2 (edited by Neutro 2019-06-09 22:05:36)

Re: clamd wont start and use 100% cpu

Got this in logwatch as well:

WARNING: Ignoring deprecated option AllowSupplementaryGroups at /etc/clamd.d/amavisd.conf:21
WARNING: Ignoring deprecated option AllowSupplementaryGroups at /etc/clamd.d/amavisd.conf:23
ERROR: Can't initialize the internal logger
ERROR: Can't open /etc/clamd.d/log in append mode (check permissions!).
ERROR: Please define server type (local and/or TCP).
WARNING: Ignoring deprecated option AllowSupplementaryGroups at /etc/clamd.d/amavisd.conf:23
ERROR: Can't initialize the internal logger
ERROR: Please edit the example config file /etc/clamd.d/scan.conf

3

Re: clamd wont start and use 100% cpu

Neutro wrote:

WARNING: Ignoring deprecated option AllowSupplementaryGroups at /etc/clamd.d/amavisd.conf:21

Remove this parameter and restart clamd@amvisd service.

Neutro wrote:

I've searched on the forum and found out that some people had the same problem because they were running only 1GB RAM but my server is running 2GB ram (it's a vmware virtual machine running on ESXI 6.7).

Please pay close attention to clamav log file under /var/log/clamav/. 2GB may be not enough for clamav, in this case clamav will report error in log file.

----

Does my reply help a little? How about buying me a cup of coffee ($5) as an encouragement?

buy me a cup of coffee

4 (edited by Neutro 2019-06-10 23:50:34)

Re: clamd wont start and use 100% cpu

Thank you for your answer!

I've removed AllowSupplementaryGroup from the config file but still have the same problem.

I don't have a log in /var/log for clamav

This is my /var/log directory:

[root@smtp log][root@smtp log]# ls
anaconda           dmesg               maillog-20190609   secure-20190609       vmware-network.6.log
audit              dmesg.old           maillog.save       sogo                  vmware-network.7.log
boot.log           dovecot             mariadb            spooler               vmware-network.8.log
boot.log-20190605  firewalld           messages           spooler-20190609      vmware-network.9.log
boot.log-20190606  freshclam.log       messages-20190609  tallylog              vmware-network.log
boot.log-20190607  grubby              mlmmjadmin         tuned                 vmware-vgauthsvc.log.0
boot.log-20190608  grubby_prune_debug  netdata            vmware-install.log    vmware-vmsvc.log
boot.log-20190610  httpd               nginx              vmware-network.1.log  wtmp
btmp               iredapd             php-fpm            vmware-network.2.log  yum.log
chrony             lastlog             rhsm               vmware-network.3.log
cron               letsencrypt         sa-update.log      vmware-network.4.log
cron-20190609      maillog             secure             vmware-network.5.log

freshclam.log is empty

When i do cat /var/log/maillog.log | grep clamd i get this:

Jun  4 01:47:05 smtp clamd[7915]: Received 0 file descriptor(s) from systemd.
Jun  4 01:47:05 smtp clamd[7915]: clamd daemon 0.101.2 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Jun  4 01:47:05 smtp clamd[7915]: Running as user amavis (UID 990, GID 987)
Jun  4 01:47:05 smtp clamd[7915]: Log file size limited to 1048576 bytes.
Jun  4 01:47:05 smtp clamd[7915]: Reading databases from /var/lib/clamav
Jun  4 01:47:05 smtp clamd[7915]: Not loading PUA signatures.
Jun  4 01:47:05 smtp clamd[7915]: Bytecode: Security mode set to "TrustSigned".
Jun  4 01:44:48 smtp amavis[8978]: Using primary internal av scanner code for clamav-socket
Jun  4 01:44:48 smtp amavis[8978]: Found secondary av scanner clamav-clamscan at /usr/bin/clamscan
Jun  4 01:45:54 smtp clamd[11963]: Received 0 file descriptor(s) from systemd.
Jun  4 01:45:54 smtp clamd[11963]: clamd daemon 0.101.2 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Jun  4 01:45:54 smtp clamd[11963]: Running as user amavis (UID 990, GID 987)
Jun  4 01:45:54 smtp clamd[11963]: Log file size limited to 1048576 bytes.
Jun  4 01:45:54 smtp clamd[11963]: Reading databases from /var/lib/clamav
Jun  4 01:45:54 smtp clamd[11963]: Not loading PUA signatures.
Jun  4 01:45:54 smtp clamd[11963]: Bytecode: Security mode set to "TrustSigned".
Jun  4 01:47:25 smtp clamd[17185]: Received 0 file descriptor(s) from systemd.


Jun  4 02:37:37 smtp clamd[8837]: clamd daemon 0.101.2 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Jun  4 02:37:37 smtp clamd[8837]: Running as user amavis (UID 990, GID 987)
Jun  4 02:37:37 smtp clamd[8837]: Log file size limited to 1048576 bytes.
Jun  4 02:37:37 smtp clamd[8837]: Reading databases from /var/lib/clamav
Jun  4 02:37:37 smtp clamd[8837]: Not loading PUA signatures.
Jun  4 02:37:37 smtp clamd[8837]: Bytecode: Security mode set to "TrustSigned".
Jun  4 02:38:01 smtp amavis[7358]: (07358-04) (!)terminating process [8723] running clamav-clamscan (reason: on reading: timed out)
Jun  4 02:38:02 smtp amavis[7358]: (07358-04) (!)process [8723] running clamav-clamscan is still alive, using a bigger hammer (SIGKILL)
Jun  4 02:38:02 smtp amavis[7358]: (07358-04) (!)run_av (clamav-clamscan): collect_results - reading aborted: timed out at /usr/sbin/amavisd line 5115.
Jun  4 02:38:02 smtp amavis[7358]: (07358-04) (!)clamav-clamscan av-scanner FAILED: run_av error: Exceeded allowed time\n
Jun  4 02:38:06 smtp amavis[7358]: (07358-05) (!)connect to /var/run/clamd.amavisd/clamd.socket failed, attempt #1: Can't connect to a UNIX socket /var/run/clamd.amavisd/clamd.socket: Aucun fichier ou dossier de ce type
Jun  4 02:38:07 smtp amavis[7358]: (07358-05) (!)connect to /var/run/clamd.amavisd/clamd.socket failed, attempt #1: Can't connect to a UNIX socket /var/run/clamd.amavisd/clamd.socket: Aucun fichier ou dossier de ce type
Jun  4 02:38:07 smtp amavis[7358]: (07358-05) (!)clamav-socket: All attempts (1) failed connecting to /var/run/clamd.amavisd/clamd.socket, retrying (2)
Jun  4 02:38:13 smtp amavis[7358]: (07358-05) (!)connect to /var/run/clamd.amavisd/clamd.socket failed, attempt #1: Can't connect to a UNIX socket /var/run/clamd.amavisd/clamd.socket: Aucun fichier ou dossier de ce type
Jun  4 02:38:13 smtp amavis[7358]: (07358-05) (!)clamav-socket av-scanner FAILED: run_av error: Too many retries to talk to /var/run/clamd.amavisd/clamd.socket (All attempts (1) failed connecting to /var/run/clamd.amavisd/clamd.socket) at (eval 134) line 659.\n
Jun  4 02:38:56 smtp amavis[7359]: (07359-05) (!)terminating process [8750] running clamav-clamscan (reason: on reading: timed out)
Jun  4 02:38:57 smtp amavis[7359]: (07359-05) (!)process [8750] running clamav-clamscan is still alive, using a bigger hammer (SIGKILL)
Jun  4 02:38:57 smtp amavis[7359]: (07359-05) (!)run_av (clamav-clamscan): collect_results - reading aborted: timed out at /usr/sbin/amavisd line 5115.
Jun  4 02:38:57 smtp amavis[7359]: (07359-05) (!)clamav-clamscan av-scanner FAILED: run_av error: Exceeded allowed time\n
Jun  4 02:39:07 smtp clamd[8884]: Received 0 file descriptor(s) from systemd.


Jun  4 04:33:20 smtp clamd[7882]: clamd daemon 0.101.2 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Jun  4 04:33:20 smtp clamd[7882]: Running as user amavis (UID 990, GID 987)
Jun  4 04:33:20 smtp clamd[7882]: Log file size limited to 1048576 bytes.
Jun  4 04:33:20 smtp clamd[7882]: Reading databases from /var/lib/clamav
Jun  4 04:33:20 smtp clamd[7882]: Not loading PUA signatures.
Jun  4 04:33:20 smtp clamd[7882]: Bytecode: Security mode set to "TrustSigned".
Jun  4 04:33:26 smtp amavis[7883]: Using primary internal av scanner code for clamav-socket
Jun  4 04:33:26 smtp amavis[7883]: Found secondary av scanner clamav-clamscan at /usr/bin/clamscan
Jun  4 04:34:05 smtp amavis[7915]: Using primary internal av scanner code for clamav-socket
Jun  4 04:34:05 smtp amavis[7915]: Found secondary av scanner clamav-clamscan at /usr/bin/clamscan
Jun  4 04:34:12 smtp amavis[7941]: (07941-01) (!)connect to /var/run/clamd.amavisd/clamd.socket failed, attempt #1: Can't connect to a UNIX socket /var/run/clamd.amavisd/clamd.socket: Aucun fichier ou dossier de ce type
Jun  4 04:34:13 smtp amavis[7941]: (07941-01) (!)connect to /var/run/clamd.amavisd/clamd.socket failed, attempt #1: Can't connect to a UNIX socket /var/run/clamd.amavisd/clamd.socket: Aucun fichier ou dossier de ce type
Jun  4 04:34:13 smtp amavis[7941]: (07941-01) (!)clamav-socket: All attempts (1) failed connecting to /var/run/clamd.amavisd/clamd.socket, retrying (2)
Jun  4 04:34:19 smtp amavis[7941]: (07941-01) (!)connect to /var/run/clamd.amavisd/clamd.socket failed, attempt #1: Can't connect to a UNIX socket /var/run/clamd.amavisd/clamd.socket: Aucun fichier ou dossier de ce type
Jun  4 04:34:19 smtp amavis[7941]: (07941-01) (!)clamav-socket av-scanner FAILED: run_av error: Too many retries to talk to /var/run/clamd.amavisd/clamd.socket (All attempts (1) failed connecting to /var/run/clamd.amavisd/clamd.socket) at (eval 134) line 659.\n
Jun  4 04:36:02 smtp amavis[7942]: (07942-01) (!)connect to /var/run/clamd.amavisd/clamd.socket failed, attempt #1: Can't connect to a UNIX socket /var/run/clamd.amavisd/clamd.socket: Aucun fichier ou dossier de ce type
Jun  4 04:36:03 smtp amavis[7942]: (07942-01) (!)connect to /var/run/clamd.amavisd/clamd.socket failed, attempt #1: Can't connect to a UNIX socket /var/run/clamd.amavisd/clamd.socket: Aucun fichier ou dossier de ce type
Jun  4 04:36:03 smtp amavis[7942]: (07942-01) (!)clamav-socket: All attempts (1) failed connecting to /var/run/clamd.amavisd/clamd.socket, retrying (2)
Jun  4 04:36:09 smtp amavis[7942]: (07942-01) (!)connect to /var/run/clamd.amavisd/clamd.socket failed, attempt #1: Can't connect to a UNIX socket /var/run/clamd.amavisd/clamd.socket: Aucun fichier ou dossier de ce type
Jun  4 04:36:09 smtp amavis[7942]: (07942-01) (!)clamav-socket av-scanner FAILED: run_av error: Too many retries to talk to /var/run/clamd.amavisd/clamd.socket (All attempts (1) failed connecting to /var/run/clamd.amavisd/clamd.socket) at (eval 134) line 659.\n
Jun  7 06:26:10 smtp clamd[11346]: Received 0 file descriptor(s) from systemd.

Jun  7 08:48:01 smtp clamd[7227]: clamd daemon 0.101.2 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Jun  7 08:48:01 smtp clamd[7227]: Running as user amavis (UID 990, GID 987)
Jun  7 08:48:01 smtp clamd[7227]: Log file size limited to 1048576 bytes.
Jun  7 08:48:01 smtp clamd[7227]: Reading databases from /var/lib/clamav
Jun  7 08:48:01 smtp clamd[7227]: Not loading PUA signatures.
Jun  7 08:48:01 smtp clamd[7227]: Bytecode: Security mode set to "TrustSigned".
Jun  7 08:48:12 smtp amavis[7362]: Using primary internal av scanner code for clamav-socket
Jun  7 08:48:12 smtp amavis[7362]: Found secondary av scanner clamav-clamscan at /usr/bin/clamscan
Jun  7 08:53:56 smtp amavis[7569]: (07569-02) (!)connect to /var/run/clamd.amavisd/clamd.socket failed, attempt #1: Can't connect to a UNIX socket /var/run/clamd.amavisd/clamd.socket: Aucun fichier ou dossier de ce type
Jun  7 08:53:57 smtp amavis[7569]: (07569-02) (!)connect to /var/run/clamd.amavisd/clamd.socket failed, attempt #1: Can't connect to a UNIX socket /var/run/clamd.amavisd/clamd.socket: Aucun fichier ou dossier de ce type
Jun  7 08:53:57 smtp amavis[7569]: (07569-02) (!)clamav-socket: All attempts (1) failed connecting to /var/run/clamd.amavisd/clamd.socket, retrying (2)
Jun  7 08:54:03 smtp amavis[7569]: (07569-02) (!)connect to /var/run/clamd.amavisd/clamd.socket failed, attempt #1: Can't connect to a UNIX socket /var/run/clamd.amavisd/clamd.socket: Aucun fichier ou dossier de ce type
Jun  7 08:54:03 smtp amavis[7569]: (07569-02) (!)clamav-socket av-scanner FAILED: run_av error: Too many retries to talk to /var/run/clamd.amavisd/clamd.socket (All attempts (1) failed connecting to /var/run/clamd.amavisd/clamd.socket) at (eval 134) line 659.\n
Jun  7 08:58:35 smtp clamd[6681]: Received 0 file descriptor(s) from systemd.
Jun  7 08:58:35 smtp clamd[6681]: clamd daemon 0.101.2 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)


Please pay close attention to clamav log file under /var/log/clamav/. 2GB may be not enough for clamav, in this case clamav will report error in log file.

I dont think the memory is the problem because when i look at the ESXI performance status i see this:

https://nextcloud.ah1z.com/s/iYDpiJwCw7Nmm4p/preview

As you can see the CPU usage is maxed out but the memory usage almost doesn't change.

What's even worse is that even if I run systemctl disable clamd@amavisd it restarts itself randomly so i had to add systemctl stop clamd@amavisd every 5 mins in my cron sad

I dont even need clamav scan to run on my server but i've followed https://docs.iredmail.org/completely.di … assin.html and the clamd@amavisd service is still starting.

Also tried to update the policy in amavis database to disable virus check but still same problem sad

I'm gonna wait to see if you have an idea, otherwise i'll just delete everything and reinstall to see if i get the same results.

5

Re: clamd wont start and use 100% cpu

- How much memory does this server have? 2GB? Please increase to at least 4GB and try again.
- Does ClamAV service creates /var/run/clamd.amavisd/clamd.socket after started?

----

Does my reply help a little? How about buying me a cup of coffee ($5) as an encouragement?

buy me a cup of coffee

6 (edited by Neutro 2019-06-12 13:32:07)

Re: clamd wont start and use 100% cpu

Thanks again for your answer Zhang smile

I've tried increasing to 4GB same problem, clamAV does not create the clamd.socket file.

I've tried reinstalling my server from scratch, with the cleanest install of centos 7.6.1810 3 times in a row but same problem.

You can find the log from the installation here (i've verified that there are no passwords inside the logs):

https://nextcloud.ah1z.com/s/zqgsosxwrL4cfxK

So i gave up on iredmail with centos for now, i'm installing debian at the moment and crossing my fingers wink

#edit: switching to debian fixed the problem wink After reboot clamav takes 100% of cpu for 2 mins but then it lowers back to normal, and this works with 2GB ram.

I'm gonna stay like this for now seeing how it runs after a few days. I hope everything will be smoother big_smile

Going to bed now, i spent my whole night on this big_smile