1 Topic by SteveLuxe

  • SteveLuxe
  • Member
  • Offline
  • Registered: 2015-06-02
  • Posts: 115

Topic: Amavis Crashing / Hanging after new Debian updates

==== Required information ====
- iRedMail version (check /etc/iredmail-release): 0.9.2
- Linux/BSD distribution name and version: Debian 8.1
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): MySql
- Web server (Apache or Nginx): Apache
- Manage mail accounts with iRedAdmin-Pro? No
- Related log if you're reporting an issue: /var/log/mail.log
====

Hey All! It seems that I'm running into something that I've never dealt with before. Since the new Debian 8.1 kernel security release upgrade, I find that Amavis seems to be crashing on me. I'm not quite sure how I can go about restarting the service if it becomes hung, without manual intervention. Mails are getting deferred, and I really need to figure this one out...

Here's an excerpt of what I'm seeing in /var/log/mail.log

Aug  9 22:42:51 mail postfix/qmgr[1526]: 6A933413B3: removed
Aug  9 22:44:21 mail amavis[1613]: (01613-02) NOTICE: reconnecting in response to: err=2013, HY000, timed out
Aug  9 22:44:21 mail amavis[1613]: (01613-02) (!)lookup_sql: timed out at (eval 95) line 200., ,
Aug  9 22:44:21 mail amavis[1613]: (01613-02) (!)Requesting process rundown, task exceeded allowed time

Any ideas as to what's going on here? Thanks!

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,165

Re: Amavis Crashing / Hanging after new Debian updates

Restart mysql service first, then restart amavis, postfix, dovecot, monitor the log files.

3 Reply by SteveLuxe

  • SteveLuxe
  • Member
  • Offline
  • Registered: 2015-06-02
  • Posts: 115

Re: Amavis Crashing / Hanging after new Debian updates

Thanks, Zhang. I'll give that a try. Should I do that after it crashes, or before?

4 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,165

Re: Amavis Crashing / Hanging after new Debian updates

After.

5 Reply by SteveLuxe

  • SteveLuxe
  • Member
  • Offline
  • Registered: 2015-06-02
  • Posts: 115

Re: Amavis Crashing / Hanging after new Debian updates

ZhangHuangbin wrote:

After.

Okay Zhang! Thank you very much! I'll keep you posted.

6 Reply by SteveLuxe (edited by SteveLuxe 2015-08-10 23:48:21)

  • SteveLuxe
  • Member
  • Offline
  • Registered: 2015-06-02
  • Posts: 115

Re: Amavis Crashing / Hanging after new Debian updates

It appears that the problem is getting stranger. I ended up doing what you recommended, Zhang. After I did, I monitored the logs. When I restarted everything, I got this in /var/log/mysql/error.log... I'm not sure if this is meaningful, at all, but it did seem like an error of noteworthiness.

150810  2:38:33 [Note] Event Scheduler: Purging the queue. 0 events
150810  2:38:35 [Warning] /usr/sbin/mysqld: Forcing close of thread 1336  user: 'vmailadmin'
150810  2:38:35 [Warning] /usr/sbin/mysqld: Forcing close of thread 1335  user: 'vmailadmin'
150810  2:38:35 [Warning] /usr/sbin/mysqld: Forcing close of thread 1300  user: 'vmail'
150810  2:38:35 [Warning] /usr/sbin/mysqld: Forcing close of thread 1299  user: 'amavisd'
150810  2:38:35 [Warning] /usr/sbin/mysqld: Forcing close of thread 1206  user: 'vmail'
150810  2:38:35 [Warning] /usr/sbin/mysqld: Forcing close of thread 1205  user: 'vmailadmin'
150810  2:38:35 [Warning] /usr/sbin/mysqld: Forcing close of thread 1204  user: 'vmailadmin'
150810  2:38:35 [Warning] /usr/sbin/mysqld: Forcing close of thread 1011  user: 'amavisd'
150810  2:38:35 [Warning] /usr/sbin/mysqld: Forcing close of thread 960  user: 'amavisd'
150810  2:38:35 [Warning] /usr/sbin/mysqld: Forcing close of thread 881  user: 'amavisd'
150810  2:38:35 [Warning] /usr/sbin/mysqld: Forcing close of thread 873  user: 'amavisd'

I'm not sure if that helps. It appears that Postfix and Dovecot continue working as usual, but when it passes the email to Amavis / mysql, that's where it just stops, suddenly. Sometimes it dies quietly, other times, there's an error like I posted above. It seems to be happening during times of inactivity. Could it be related to the "MySql has gone away" error?

Is there a way to force MySQL to stay open during times of inactivity?

7 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,165

Re: Amavis Crashing / Hanging after new Debian updates

How much RAM do you have on this server? At least 2GB RAM is recommended for a low traffic mail server.

8 Reply by SteveLuxe (edited by SteveLuxe 2015-08-11 00:02:19)

  • SteveLuxe
  • Member
  • Offline
  • Registered: 2015-06-02
  • Posts: 115

Re: Amavis Crashing / Hanging after new Debian updates

ZhangHuangbin wrote:

How much RAM do you have on this server? At least 2GB RAM is recommended for a low traffic mail server.

It has 2gb of RAM. I still have RAM free, with around 50 - 100 megabytes of ram available, and only about 16 megabytes written to swap during peak activity.

9 Reply by SteveLuxe

  • SteveLuxe
  • Member
  • Offline
  • Registered: 2015-06-02
  • Posts: 115

Re: Amavis Crashing / Hanging after new Debian updates

I also got this in /var/log/mail.warn

Aug 10 11:49:51 mail amavis[10441]: (10441-02) (!)lookup_sql: timed out at (eval 95) line 200., ,
Aug 10 11:49:51 mail amavis[10441]: (10441-02) (!)Requesting process rundown, task exceeded allowed time

Hopefully that helps narrow it down?

10 Reply by SteveLuxe (edited by SteveLuxe 2015-08-11 02:05:21)

  • SteveLuxe
  • Member
  • Offline
  • Registered: 2015-06-02
  • Posts: 115

Re: Amavis Crashing / Hanging after new Debian updates

For what it's worth, I was doing some reading on the "Requesting process rundown" error. I noticed that other people who had the error seemed to have an installation of mysql-server or amavis that seemed to be screwed up. I did an "apt-get install --reinstall amavisd-new mysql-server", so that there were new copies of the software in place, and I also did a "service mysql restart && service amavis restart && service postfix restart && service dovecot restart". All started normally after the re-installation of the new packages. I'm hoping that it's as simple as that, however, I'm still interested in other possibilities, as well. As I mentioned earlier, it just seems to time out during periods of inactivity. I find that to get the mail moving again, I have to restart the services. I hope that it's as simple as a reinstallation of the software packages.

Any other ideas about what could possibly be contributing to this?

***EDIT***
Since re-installation of mysql-server, and amavisd-new, I seem to be running a bit more stable. I've been monitoring MySQL this afternoon using a combination of top, as well as reading the logs, and I haven't seen any more crashes, or disconnects. I'm keeping an eye on this very closely. Hopefully all that it was was a bad install package of MySQL, or Amavis. I know that MySQL Server was upgraded recently. This could have contributed to the problem if something went strangely with the upgrade process. I only really noticed the issue after the new kernel update that Debian pushed out about 3-4 days ago. As far as the MySQL service running, I see it regularly running in top, now. Fingers crossed that a simple package reinstall did the trick. There were also a couple of occasions where I had to cold-boot the server, remotely, via a remote power switch, if I issued a reboot command, and it didn't seem to want to come back up. Perhaps the cold booting had something to do with the issue as well. The system may have corrupted a file that it was in the middle of reading, or writing to. Whatever the case, I'm doing what I can to monitor the situation closely,  and take action if needed. So far so good, and it's been over an hour now, and I've had fairly low to no traffic to the server. I send a test email to myself once in a while to make sure that they're still flowing... If you have any other suggestions, I'd be happy to hear them.

11 Reply by SteveLuxe

  • SteveLuxe
  • Member
  • Offline
  • Registered: 2015-06-02
  • Posts: 115

Re: Amavis Crashing / Hanging after new Debian updates

I've been running it all afternoon, after reinstalling Amavis and MySQL Server, and things seem much more stable now. I'm still not totally convinced that all is fixed, but I'm getting the idea that my problems stemmed from a corrupt software upgrade. Doing the reinstall on both packages seems to have done the trick. Here's hoping! If anything changes, I'll post. If not, I'll probably consider this case closed. I'll post back either later on today, or tomorrow with an update if anything changes!

Thanks for your help Zhang! I'll let you know if this is case closed as soon as possible!

-Steve

12 Reply by SteveLuxe

  • SteveLuxe
  • Member
  • Offline
  • Registered: 2015-06-02
  • Posts: 115

Re: Amavis Crashing / Hanging after new Debian updates

So it appears that the problem was rooted in the fact that I had a bad installation of either Amavis, or MySQL Server. Doing an "apt-get install --reinstall amavisd-new mysql-server" seems to have done the trick. Sorry to have made an issue out of it. I don't remember having modified those packages, but I'm thinking that a power outage / reboot corrupted one of the configuration files for one of the packages. The reinstalled versions have been up and running ever since, with no incident. Thanks, Zhang, for your assistance!