1 Topic by LainX84 (edited by LainX84 2018-11-12 21:36:02)

  • LainX84
  • Member
  • Offline
  • Registered: 2017-10-05
  • Posts: 23

Topic: Huge CPU usage

==== REQUIRED BASIC INFO OF YOUR IREDMAIL SERVER ====
- iRedMail version (check /etc/iredmail-release): 0.9.8 MARIADB edition
- Linux/BSD distribution name and version: CentOS Linux release 7.5.1804 (Core)
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): MySQL
- Web server (Apache or Nginx): Nginx
- Manage mail accounts with iRedAdmin-Pro? No
- [IMPORTANT] Related original log or error message is required if you're experiencing an issue.
====

Hi to all,

recently on my server I noticed a big slowdown of the web part as well as via SSH.
That server is a VPS over KVM OpenStack with 1 vCore at 2,4  GHz, 2 GB of RAM and 20 GB SS over RAID.
I checked today and it turns out that the CPU is always 100% with a load avarage of 7.76 9.91 10.14 at this time.
I've restarted the machine and done some checks and there seems to be some problem.

Here a screenshot from HTOP :

https://i.postimg.cc/t7Fr0wrK/Capture.jpg

I've check the process usage i see that : 

[root@srv-main ~]# ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%cpu | head
  PID  PPID CMD                         %MEM %CPU
 1536  1460 ps -eo pid,ppid,cmd,%mem,%c  0.0 17.0
  500     1 /usr/bin/python -Es /usr/sb  1.5  7.6
 1484   856 /usr/bin/python2 -s /usr/bi  0.5  7.3
 1462   943 /usr/libexec/mysqld --based  4.1  7.2
 1330     1 /usr/bin/perl -T /usr/sbin/  0.5  6.7
 1333     1 /usr/sbin/clamd -c /etc/cla  0.9  6.7
 1290  1266 gzip -cd -- /boot/initramfs  0.0  6.2
  856     1 /usr/bin/python2 -s /usr/bi  0.6  4.2
    1     0 /usr/lib/systemd/systemd --  0.2  3.1

[root@srv-main etc]# ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%mem | head
  PID  PPID CMD                         %MEM %CPU
 1478   963 /usr/libexec/mysqld --based  6.1  1.7
 4331     1 /usr/sbin/clamd -c /etc/cla  5.0 45.5
 1056   878 php-fpm: pool www            3.2  0.3
 1051   878 php-fpm: pool www            3.2  0.3
 1059   878 php-fpm: pool www            3.2  0.3
 1046   878 php-fpm: pool www            1.8  0.3
  504     1 /usr/bin/python -Es /usr/sb  1.5  0.2
 1441     1 /opt/netdata/bin/srv/netdat  1.4  0.8
  505   491 /usr/sbin/uwsgi --ini ireda  1.1  0.4

The version of amavisd-new is : 

[root@srv-main run]# rpm -qi amavisd-new
Name        : amavisd-new
Version     : 2.11.1
Release     : 1.el7
Architecture: noarch
Install Date: Mon 12 Nov 2018 12:00:10 PM CET
Group       : Applications/System
Size        : 3318497
License     : GPLv2+ and BSD and GFDL
Signature   : RSA/SHA256, Thu 18 Oct 2018 04:03:17 PM CEST, Key ID 6a2faea2352c64e5
Source RPM  : amavisd-new-2.11.1-1.el7.src.rpm
Build Date  : Thu 18 Oct 2018 03:03:47 PM CEST
Build Host  : buildvm-20.phx2.fedoraproject.org
Relocations : (not relocatable)
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : [url]https://amavis.org/[/url]
Bug URL     : [url]https://bugz.fedoraproject.org/amavisd-new[/url]
Summary     : Email filter with virus scanner and spamassassin support
Description :
amavisd-new is a high-performance and reliable interface between mailer
(MTA) and one or more content checkers: virus scanners, and/or
Mail::SpamAssassin Perl module. It is written in Perl, assuring high
reliability, portability and maintainability. It talks to MTA via (E)SMTP
or LMTP, or by using helper programs. No timing gaps exist in the design
which could cause a mail loss.

From /var/log/messages i see the following error message with taile -f messages :

Nov 12 13:38:05 srv-main systemd: amavisd.service start operation timed out. Terminating.
Nov 12 13:38:06 srv-main systemd: Failed to start Amavisd-new is an interface between MTA and content checkers..
Nov 12 13:38:06 srv-main systemd: Unit amavisd.service entered failed state.
Nov 12 13:38:06 srv-main systemd: amavisd.service failed.
Nov 12 13:38:07 srv-main systemd: amavisd.service holdoff time over, scheduling restart.
Nov 12 13:38:07 srv-main systemd: clamd@amavisd.service start operation timed out. Terminating.
Nov 12 13:38:08 srv-main systemd: Starting Amavisd-new is an interface between MTA and content checkers....
Nov 12 13:38:08 srv-main systemd: Failed to start clamd scanner (amavisd) daemon.
Nov 12 13:38:08 srv-main systemd: Unit clamd@amavisd.service entered failed state.
Nov 12 13:38:08 srv-main systemd: clamd@amavisd.service failed.
Nov 12 13:38:08 srv-main systemd: clamd@amavisd.service holdoff time over, scheduling restart.
Nov 12 13:38:09 srv-main systemd: Starting clamd scanner (amavisd) daemon...
Nov 12 13:38:10 srv-main clamd: WARNING: Ignoring deprecated option AllowSupplementaryGroups at line 21

I load amavisd in debug mode too, after stop the service with the following command : amavisd -u amavis -c /etc/amavisd/amavisd.conf debug and attach the output to this topic.

Can someone help me out of this issue ?

Thanks again for your assist and regards.

LainX84

Post's attachments

amavisd.conf debug 89.95 kb, file has never been downloaded. 

You don't have the permssions to download the attachments of this post.

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2 Reply by LainX84

  • LainX84
  • Member
  • Offline
  • Registered: 2017-10-05
  • Posts: 23

Re: Huge CPU usage

Checking the maillog log i found that :

cat /var/log/maillog | grep 'amavisd'

Nov 13 04:18:30 srv-main amavis[4787]: (04787-17) (!)SA TIMED OUT, backtrace: at (eval 136) line 791.\n\teval {...} called at (eval 136) line 791\n\teval {...} called at (eval 136) line 838\n\tAmavis::SpamControl::SpamAssassin::call_spamassassin('Amavis::SpamControl::SpamAssassin=HASH(0x4abff40)', 'Amavis::In::Message=HASH(0x6749618)', 'ARRAY(0x4b0d950)', undef) called at (eval 136) line 942\n\teval {...} called at (eval 136) line 951\n\tAmavis::SpamControl::SpamAssassin::check('Amavis::SpamControl::SpamAssassin=HASH(0x4abff40)', 'Amavis::In::Message=HASH(0x6749618)') called at (eval 135) line 150\n\teval {...} called at (eval 135) line 151\n\tAmavis::SpamControl::spam_scan('Amavis::SpamControl=HASH(0x4d109a0)', 'Amavis::In::Message=HASH(0x6749618)') called at /usr/sbin/amavis  line 14878\n\teval {...} called at /usr/sbin/amavisd line 16005\n\tAmavis::check_mail('Amavis::In::Message=HASH(0x6749618)', 0) called at (eval 128) line 1258\n\tAmavis::In::SMTP::p[...]
Nov 13 04:18:42 srv-main amavis[4787]: (04787-17) (!)PRESERVING EVIDENCE in /var/spool/amavisd/tmp/amavis-20181113T041020-04787-klSmZdr9
...
...
Nov 13 05:17:58 srv-main amavis[8339]: (08339-02) (!)connect to /var/run/clamd.amavisd/clamd.socket failed, attempt #1: Can't connect to a UNIX socket /var/run/clamd.amavisd/clamd.socket: No such file or directory
Nov 13 05:17:59 srv-main amavis[8339]: (08339-02) (!)connect to /var/run/clamd.amavisd/clamd.socket failed, attempt #1: Can't connect to a UNIX socket /var/run/clamd.amavisd/clamd.socket: No such file or directory
Nov 13 05:18:00 srv-main amavis[8339]: (08339-02) (!)clamav-socket: All attempts (1) failed connecting to /var/run/clamd.amavisd/clamd.socket, retrying (2)
Nov 13 05:18:06 srv-main amavis[8339]: (08339-02) (!)connect to /var/run/clamd.amavisd/clamd.socket failed, attempt #1: Can't connect to a UNIX socket /var/run/clamd.amavisd/clamd.socket: No such file or directory
Nov 13 05:18:06 srv-main amavis[8339]: (08339-02) (!)clamav-socket av-scanner FAILED: run_av error: Too many retries to talk to /var/run/clamd.amavisd/clamd.socket (All attempts (1) failed connecting to /var/run/clamd.amavisd/clamd.socket) at (eval 134) line 659.\n
Nov 13 05:22:45 srv-main amavis[8339]: (08339-02) (!)run_av (clamav-clamscan): collect_results - reading aborted: timed out at /usr/sbin/amavisd line 5115.

3 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,159

Re: Huge CPU usage

LainX84 wrote:

Nov 13 05:17:58 srv-main amavis[8339]: (08339-02) (!)connect to /var/run/clamd.amavisd/clamd.socket failed, attempt #1: Can't connect to a UNIX socket /var/run/clamd.amavisd/clamd.socket: No such file or directory

ClamAV service is not running?