1 Topic by kiran (edited by kiran 2013-07-08 19:12:04)

  • kiran
  • Member
  • Offline
  • Registered: 2013-04-30
  • Posts: 30

Topic: mails going to nrcpt=1 (queue active) around 15000 mails got stuck

==== Required information ====
- iRedMail version: 0.8.4
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): MySQL
- Linux/BSD distribution name and version: Centos 6.4
- Related log if you're reporting an issue:
====

Hi All,

I am currently facing postfix active queue issue.

all mails going in to active queue and message delivery is very slow. I have cheked CPU and RAM everything normal.

Please help me to tune or trobuleshoot the issue.

How to speed up postfix mailq.

postfix sending mails 1 or 2 per very very slowly .....

what are the performance tunning setps.

Thanks,
Kiran

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2 Reply by kiran

  • kiran
  • Member
  • Offline
  • Registered: 2013-04-30
  • Posts: 30

Re: mails going to nrcpt=1 (queue active) around 15000 mails got stuck

Hi any one please help me ....

3 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,091

Re: mails going to nrcpt=1 (queue active) around 15000 mails got stuck

Weird. Here's my questions:

*) Default iRedMail installation? Did you modify any config files?
*) Any related log in Postfix and Dovecot log files? /var/log/maillog, /var/log/dovecot.log, /var/log/sieve.log.

4 Reply by kiran

  • kiran
  • Member
  • Offline
  • Registered: 2013-04-30
  • Posts: 30

Re: mails going to nrcpt=1 (queue active) around 15000 mails got stuck

ZhangHuangbin wrote:

Weird. Here's my questions:

*) Default iRedMail installation? Did you modify any config files?
*) Any related log in Postfix and Dovecot log files? /var/log/maillog, /var/log/dovecot.log, /var/log/sieve.log.

Hi ZhangHuangbin,

I am using iredmail default installation and there is no errors in mailog, dovecot.log, sieve.log

when i try to sendmail from the mailserver comandline even one mail it self going to active queue and after 20 sec it is sending.

Thanks,
Kiran

5 Reply by kiran

  • kiran
  • Member
  • Offline
  • Registered: 2013-04-30
  • Posts: 30

Re: mails going to nrcpt=1 (queue active) around 15000 mails got stuck

kiran wrote:
ZhangHuangbin wrote:

Weird. Here's my questions:

*) Default iRedMail installation? Did you modify any config files?
*) Any related log in Postfix and Dovecot log files? /var/log/maillog, /var/log/dovecot.log, /var/log/sieve.log.

Hi ZhangHuangbin,

I am using iredmail default installation and there is no errors in mailog, dovecot.log, sieve.log

when i try to sendmail from the mailserver comandline even one mail it self going to active queue and after 20 sec it is sending.

Thanks,
Kiran

Hi ,

Can i use subversion of iredmail insted of 0.8.4 ??

6 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,091

Re: mails going to nrcpt=1 (queue active) around 15000 mails got stuck

*) We don't know what the root cause is, so installing a development edition may not help. But you can give it a try.
*) We don't provide upgrade tutorial for development edition, so it's hard for you to upgrade development edition to a stable release. Use it on your own.

I think we'd better check log files to try to figure it out.

7 Reply by kiran

  • kiran
  • Member
  • Offline
  • Registered: 2013-04-30
  • Posts: 30

Re: mails going to nrcpt=1 (queue active) around 15000 mails got stuck

ZhangHuangbin wrote:

*) We don't know what the root cause is, so installing a development edition may not help. But you can give it a try.
*) We don't provide upgrade tutorial for development edition, so it's hard for you to upgrade development edition to a stable release. Use it on your own.

I think we'd better check log files to try to figure it out.

Hi ZhangHuangbin,

I found this error in maillog {warning: network_biopair_interop: error writing 37 bytes to the network: broken pipe}

Then i have upgraded to postfix-2.9.6-1.el6.x86_64 i am not able see above error.

Issue : I have started getting bulk incoming mails and postfix putting every mail in active queue and that active queue not processing at all even if i flush postfix mailq.

Please suggest me what are the performance tunning steps to process incoming mails faster than usual. Now postfix processing active queue mails 2 or 3 mails every 30 seconds or 120 seconds which is very huge delay to send to inbox. My system will have 50000 messages incoming per hour. We don't send mails out which is very less presently.

I have 2xquadcore processors 24Gb of RAM.

Regards,
Kiran

8 Reply by kiran

  • kiran
  • Member
  • Offline
  • Registered: 2013-04-30
  • Posts: 30

Re: mails going to nrcpt=1 (queue active) around 15000 mails got stuck

kiran wrote:
ZhangHuangbin wrote:

*) We don't know what the root cause is, so installing a development edition may not help. But you can give it a try.
*) We don't provide upgrade tutorial for development edition, so it's hard for you to upgrade development edition to a stable release. Use it on your own.

I think we'd better check log files to try to figure it out.

Hi ZhangHuangbin,

I found this error in maillog {warning: network_biopair_interop: error writing 37 bytes to the network: broken pipe}

Then i have upgraded to postfix-2.9.6-1.el6.x86_64 i am not able see above error.

Issue : I have started getting bulk incoming mails and postfix putting every mail in active queue and that active queue not processing at all even if i flush postfix mailq.

Please suggest me what are the performance tunning steps to process incoming mails faster than usual. Now postfix processing active queue mails 2 or 3 mails every 30 seconds or 120 seconds which is very huge delay to send to inbox. My system will have 50000 messages incoming per hour. We don't send mails out which is very less presently.

I have 2xquadcore processors 24Gb of RAM.

Regards,
Kiran

Hi ZhangHuangbin,

I found the issue in postfix main.conf we are using contentfilter after acive queue. This is causing the issue for slow delivery once mail come to server contentfilter is scaning which is taking longer time, when i commented it messages started going very very faster.

Regards,
Kiran

9 Reply by kiran

  • kiran
  • Member
  • Offline
  • Registered: 2013-04-30
  • Posts: 30

Re: mails going to nrcpt=1 (queue active) around 15000 mails got stuck

kiran wrote:
kiran wrote:
ZhangHuangbin wrote:

*) We don't know what the root cause is, so installing a development edition may not help. But you can give it a try.
*) We don't provide upgrade tutorial for development edition, so it's hard for you to upgrade development edition to a stable release. Use it on your own.

I think we'd better check log files to try to figure it out.

Hi ZhangHuangbin,

I found this error in maillog {warning: network_biopair_interop: error writing 37 bytes to the network: broken pipe}

Then i have upgraded to postfix-2.9.6-1.el6.x86_64 i am not able see above error.

Issue : I have started getting bulk incoming mails and postfix putting every mail in active queue and that active queue not processing at all even if i flush postfix mailq.

Please suggest me what are the performance tunning steps to process incoming mails faster than usual. Now postfix processing active queue mails 2 or 3 mails every 30 seconds or 120 seconds which is very huge delay to send to inbox. My system will have 50000 messages incoming per hour. We don't send mails out which is very less presently.

I have 2xquadcore processors 24Gb of RAM.

Regards,
Kiran

Hi ZhangHuangbin,

I found the issue in postfix main.conf we are using contentfilter after acive queue. This is causing the issue for slow delivery once mail come to server contentfilter is scaning which is taking longer time, when i commented it messages started going very very faster.

Regards,
Kiran


Please suggest the best performance tuning for incoming messages. I want to use content_filter as well but messages should process faster.

I don't know how others using I seen may people using iRedMail for 20000 or more users but how they processing outgoing and incoming messages so faster.

Regards,
Kiran

10 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,091

Re: mails going to nrcpt=1 (queue active) around 15000 mails got stuck

kiran wrote:

I have 2xquadcore processors 24Gb of RAM.

What's the number of total CPUs shown in "top" command?

Quote from Amavisd document: http://www.ijs.si/software/amavisd/README.postfix

Regardless of the maximum number of concurrent processes, both sides -
   Postfix and amavisd - should be synchronized. To synchronize both sides
   edit, the $max_servers parameter for amavisd-new (see: amavisd.conf) and
   the number of processes in master.cf listed in the dedicated transports
   maxproc column for Postfix.

   Both values should be identical for two reasons: If amavisd-new offers
   more processes than Postfix will ever use, amavisd-new wastes resources.
   On the other hand, if Postfix starts more dedicated transports than
   amavisd can handle simultaneously, e-mail transport will be refused and
   logged as error.

   [58][Note] Controlling the maximum number of concurrent processes in
              main.cf
              Instead of controlling the maximum number of concurrent
              processes of Postfix' dedicated transport in master.cf it is
              also possible to keep the default setting - in master.cf and
              set the following parameter and option in main.cf:

              amavisfeed_destination_concurrency_limit = 2

              The name of the parameter starts with the service in master.cf
              (here: amavisfeed) that should be controlled and goes on with
              the suffix _destination_concurrency_limit. Here also 2 is set
              as initial (conservative) value.

So, let's assume you have 8 CPUs (as shown in "top" command), you should set "$max_servers = 8;" in Amavisd config file, and set "smtp-amavis_destination_concurrency_limit = 8" in Postfix main.cf too.

NOTE: We define Amavisd as transport 'smtp-amavis' in Postfix master.cf, so you have to use parameter "smtp-amavis_destination_concurrency_limit" in Postfix main.cf.

11 Reply by amora

  • amora
  • Member
  • Offline
  • Registered: 2013-04-09
  • Posts: 58

Re: mails going to nrcpt=1 (queue active) around 15000 mails got stuck

That is a lot of emails/hr how many users do you have?

We have 32,000 users, just recently did a qmail/vpop migration and we don't even do that many mails/hour.

Add RBLS to main.cf to stop blacklisted emails from even entering the queues.
Of the ~1,000,000 connections we receive about 10% is rejected by RBLs

This is a sample of just one of our load balanced standard processing nodes:

1502   Reject relay denied                        1.69%
    50831   Reject HELO/EHLO                          57.04%
    26354   Reject unknown user                       29.58%
      132   Reject recipient address                   0.15%
      583   Reject sender address                      0.65%
      489   Reject client host                         0.55%
     9217   Reject RBL                                10.34%
--------   ------------------------------------------------
    89108   Total Rejects                            100.00%

To speed up amavisd:
- Disable spamassassin outbound scanning
- Disable virus scanning all together
- Use local DNS or an in house DNS server <- this was a big one for us. Dropped our processing times from ~16000ms to less than 3500ms, usual under 1000ms now.
- Relax spamassassin scanning.
*** By the way, the whole mounting the amavisd $tempdir on a ram drive does nothing for performance, at least on our servers it doesn't.

Use more servers.
ie Purchase the iRed Pro edition and use it as your forward facing(client facing) front end for webmail/imap/pop3.

After that grab 4-6 standard/pro iRed installs and place those behind a load balancer.
Load balance your smtp ports only or all port if you like...although that's a pain to manage if something goes wrong.
Configure those servers to remotely connect to the main front end node's mysql database for authentication.
Lastly use a shared storage server mounted to /var/vmail/vmail1 on all iredmail servers so that they all write to the same storage array.

Be mindful that this setup is only redundant as far as SMTP processing goes. You lose that one front end IRed Pro node which hosts the master mysql database and the entire cluster goes with it.

It's best to run failover load balancers, dual master-master mysql backend, dual front end webmail/imap/pop3 iRedmail-Pro nodes, and several remote iRed-standard/Pro nodes connecting to the master-master's DB's for smtp processing. Lastly dual DRDB(difficult to setup, difficult to manage, unknown small file performance)/glusterfs(easy setup, very slow with small files) failover storage nodes....or just run regular NFS shared storage(easy setup, fast, no redundancy).

With the above config you'll have 4-6 servers all processing mail simultaneously. This will allow you to have both speed and filtering...

Keep in mind there is a lot of configuring to do.