1 (edited by matthewjamesorlando 2012-09-07 13:46:51)

Topic: email sends really slow

==== Required information ====
- iRedMail version:  iRedMail-0.8.1
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): MySQL
- Linux/BSD distribution name and version: Ubuntu 12.04 Server
- Related log if you're reporting an issue:
====

I have a subscriber list of roughly 23,000 people. I am sending them email twice to three times a day...marketing/sales tips and general news in the industry. I was using a mail sending program that was hosted locally on a computer inside the network but behind a router. The mail server is sitting directly behind the internet connection and before the router. I was previously able to send roughly 20,000 emails an hour which was sufficient (I still knocked it down to 10K an hour through my program) since I could get out the 2 or 3 of my emails in one work day, but the tracking was horrible and needed to switch.

I've since switched to an email marketing program that I installed on the web server that handles my website (paid server). I've run the software in test mode (doesn't send the email) to test the database speed and it moves at 90 emails a second. Way too much for what I need to accomplish, but the real delivery is much slower now. My delivery rate has slipped down to about 2,000 an hour which is causing me to have to send the email out at the wrong time or not even get out the third email. I'm using port 587 to send my email.

I'm at a loss as to what can be causing the issue. It doesn't seem to be a database issue as it's really fast. Does anyone have an idea of what could be causing it? I don't see any errors in the logs.

# See /usr/share/postfix/main.cf.dist for a commented, more complete version


# Debian specific:  Specifying a file name will cause the first
# line of that file to be used as the name.  The Debian default
# is /etc/mailname.
#myorigin = /etc/mailname

smtpd_banner = $myhostname ESMTP $mail_name (Ubuntu)
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no

# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h

readme_directory = no

# TLS parameters
smtpd_tls_cert_file = /etc/ssl/certs/iRedMail_CA.pem
smtpd_tls_key_file = /etc/ssl/private/iRedMail.key
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache

# See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for
# information on enabling SSL in the smtp client.

myhostname = mail.mydomain.com
alias_maps = hash:/etc/postfix/aliases
alias_database = hash:/etc/postfix/aliases
#myorigin = mail.mydomain.com
mydestination = $myhostname, localhost, localhost.localdomain, localhost.$myhostname
relayhost =
mynetworks = 127.0.0.0/8 xx.xxx.xxx.xx xxx.xx.xxx.xx
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
virtual_alias_domains =
mydomain = mydomain.com
inet_protocols = ipv4
mynetworks_style = subnet
smtpd_data_restrictions = reject_unauth_pipelining
smtpd_reject_unlisted_recipient = yes
smtpd_sender_restrictions = permit_mynetworks, permit_sasl_authenticated
delay_warning_time = 0h
maximal_queue_lifetime = 1d
bounce_queue_lifetime = 1d
proxy_read_maps = $canonical_maps $lmtp_generic_maps $local_recipient_maps $mydestination $mynetworks $recipient_bcc_maps $recipient_canonical_maps $relay_domains $relay_recipient_maps $relocated_maps $sender_bcc_maps $sender_canonical_maps $smtp_generic_maps $smtpd_sender_login_maps $transport_maps $virtual_alias_domains $virtual_alias_maps $virtual_mailbox_domains $virtual_mailbox_maps $smtpd_sender_restrictions
smtp_data_init_timeout = 240s
smtp_data_xfer_timeout = 600s
smtpd_helo_required = yes
smtpd_helo_restrictions = permit_mynetworks,permit_sasl_authenticated, check_helo_access pcre:/etc/postfix/helo_access.pcre
queue_run_delay = 300s
minimal_backoff_time = 300s
maximal_backoff_time = 4000s
enable_original_recipient = no
disable_vrfy_command = yes
home_mailbox = Maildir/
allow_min_user = no
message_size_limit = 15728640
virtual_minimum_uid = 1001
virtual_uid_maps = static:1001
virtual_gid_maps = static:1001
virtual_mailbox_base = /var/vmail
smtpd_reject_unlisted_sender = yes
transport_maps = proxy:mysql:/etc/postfix/mysql/transport_maps_user.cf, proxy:mysql:/etc/postfix/mysql/transport_maps_domain.cf
virtual_mailbox_domains = proxy:mysql:/etc/postfix/mysql/virtual_mailbox_domains.cf
virtual_mailbox_maps = proxy:mysql:/etc/postfix/mysql/virtual_mailbox_maps.cf
virtual_alias_maps = proxy:mysql:/etc/postfix/mysql/virtual_alias_maps.cf, proxy:mysql:/etc/postfix/mysql/domain_alias_maps.cf, proxy:mysql:/etc/postfix/mysql/catchall_maps.cf, proxy:mysql:/etc/postfix/mysql/domain_alias_catchall_maps.cf
sender_bcc_maps = proxy:mysql:/etc/postfix/mysql/sender_bcc_maps_user.cf, proxy:mysql:/etc/postfix/mysql/sender_bcc_maps_domain.cf
recipient_bcc_maps = proxy:mysql:/etc/postfix/mysql/recipient_bcc_maps_user.cf, proxy:mysql:/etc/postfix/mysql/recipient_bcc_maps_domain.cf
relay_domains = $mydestination, proxy:mysql:/etc/postfix/mysql/relay_domains.cf
smtpd_sender_login_maps = proxy:mysql:/etc/postfix/mysql/sender_login_maps.cf
smtpd_sasl_auth_enable = yes
smtpd_sasl_local_domain =
broken_sasl_auth_clients = yes
smtpd_sasl_security_options = noanonymous
smtpd_sasl_authenticated_header = no
smtpd_recipient_restrictions = permit_mynetworks, reject_unknown_sender_domain, reject_unknown_recipient_domain, reject_non_fqdn_sender, reject_non_fqdn_recipient, reject_unlisted_recipient, check_policy_service inet:127.0.0.1:7777, check_policy_service inet:127.0.0.1:10031, permit_sasl_authenticated, reject_unauth_destination, reject_non_fqdn_helo_hostname, reject_invalid_helo_hostname, check_policy_service unix:private/policy-spf
smtpd_end_of_data_restrictions = check_policy_service inet:127.0.0.1:10031
smtpd_tls_security_level = may
smtpd_enforce_tls = no
smtpd_tls_loglevel = 0
smtpd_tls_CAfile = /etc/ssl/certs/iRedMail_CA.pem
tls_random_source = dev:/dev/urandom
mailbox_command = /usr/lib/dovecot/deliver
virtual_transport = dovecot
dovecot_destination_recipient_limit = 1
smtpd_sasl_type = dovecot
smtpd_sasl_path = ./dovecot-auth
content_filter = smtp-amavis:[localhost]:10024
smtp-amavis_destination_recipient_limit = 1
policy-spf_time_limit = 3600s

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2

Re: email sends really slow

matthewjamesorlando wrote:

I'm using port 587 to send my email.

Try disabling spam/virus scanning in /etc/postfix/master.cf:

submission inet n       -       n       -       -       smtpd                                                                 
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_mynetworks,permit_sasl_authenticated,reject
  -o content_filter=

3 (edited by matthewjamesorlando 2012-09-07 15:29:49)

Re: email sends really slow

Thanks for getting back to me. That entire section is commented out and the -o content_filter is near the end of the page with = blank. Here is the master.cf file

Is it possible that the web server is sending it over slowly? I don't think that would be an issue given the speeds in other areas, but you never know...anyway

#
# Postfix master process configuration file.  For details on the format
# of the file, see the master(5) manual page (command: "man 5 master").
#
# Do not forget to execute "postfix reload" after editing this file.
#
# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (yes)   (never) (100)
# ==========================================================================
smtp       inet  n       -       n       -       -       smtpd
#smtp      inet  n       -       -       -       1       postscreen
#smtpd     pass  -       -       -       -       -       smtpd
#dnsblog   unix  -       -       -       -       0       dnsblog
#tlsproxy  unix  -       -       -       -       0       tlsproxy
#submission inet n       -       -       -       -       smtpd
#  -o syslog_name=postfix/submission
#  -o smtpd_tls_security_level=encrypt
#  -o smtpd_sasl_auth_enable=yes
#  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
#  -o milter_macro_daemon_name=ORIGINATING
#smtps     inet  n       -       -       -       -       smtpd
#  -o syslog_name=postfix/smtps
#  -o smtpd_tls_wrappermode=yes
#  -o smtpd_sasl_auth_enable=yes
#  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
#  -o milter_macro_daemon_name=ORIGINATING
#628       inet  n       -       -       -       -       qmqpd
pickup    fifo  n       -       -       60      1       pickup
cleanup   unix  n       -       -       -       0       cleanup
qmgr      fifo  n       -       n       300     1       qmgr
#qmgr     fifo  n       -       n       300     1       oqmgr
tlsmgr    unix  -       -       -       1000?   1       tlsmgr
rewrite   unix  -       -       -       -       -       trivial-rewrite
bounce    unix  -       -       -       -       0       bounce
defer     unix  -       -       -       -       0       bounce
trace     unix  -       -       -       -       0       bounce
verify    unix  -       -       -       -       1       verify
flush     unix  n       -       -       1000?   0       flush
proxymap  unix  -       -       n       -       -       proxymap
proxywrite unix -       -       n       -       1       proxymap
smtp      unix  -       -       -       -       -       smtp
relay     unix  -       -       -       -       -       smtp
#       -o smtp_helo_timeout=5 -o smtp_connect_timeout=5
showq     unix  n       -       -       -       -       showq
error     unix  -       -       -       -       -       error
retry     unix  -       -       -       -       -       error
discard   unix  -       -       -       -       -       discard
local     unix  -       n       n       -       -       local
lmtp      unix  -       -       -       -       -       lmtp
anvil     unix  -       -       -       -       1       anvil
scache    unix  -       -       -       -       1       scache
#
# ====================================================================
# Interfaces to non-Postfix software. Be sure to examine the manual
# pages of the non-Postfix software to find out what options it wants.
#
# Many of the following services use the Postfix pipe(8) delivery
# agent.  See the pipe(8) man page for information about ${recipient}
# and other message envelope options.
# ====================================================================
#
# maildrop. See the Postfix MAILDROP_README file for details.
# Also specify in main.cf: maildrop_destination_recipient_limit=1
#
maildrop  unix  -       n       n       -       -       pipe
  flags=DRhu user=vmail argv=/usr/bin/maildrop -d ${recipient}
#
# ====================================================================
#
# Recent Cyrus versions can use the existing "lmtp" master.cf entry.
#
# Specify in cyrus.conf:
#   lmtp    cmd="lmtpd -a" listen="localhost:lmtp" proto=tcp4
#
# Specify in main.cf one or more of the following:
#  mailbox_transport = lmtp:inet:localhost
#  virtual_transport = lmtp:inet:localhost
#
# ====================================================================
#
# Cyrus 2.1.5 (Amos Gouaux)
# Also specify in main.cf: cyrus_destination_recipient_limit=1
#
#cyrus     unix  -       n       n       -       -       pipe
#  user=cyrus argv=/cyrus/bin/deliver -e -r ${sender} -m ${extension} ${user}
#
# ====================================================================
# Old example of delivery via Cyrus.
#
#old-cyrus unix  -       n       n       -       -       pipe
#  flags=R user=cyrus argv=/cyrus/bin/deliver -e -m ${extension} ${user}
#
# ====================================================================
#
# See the Postfix UUCP_README file for configuration details.
#
uucp      unix  -       n       n       -       -       pipe
 flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail ($recipient)
#
# Other external delivery methods.
#
ifmail    unix  -       n       n       -       -       pipe
  flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient)
bsmtp     unix  -       n       n       -       -       pipe
  flags=Fq. user=bsmtp argv=/usr/lib/bsmtp/bsmtp -t$nexthop -f$sender $recipient
scalemail-backend unix  -       n       n       -       2       pipe
  flags=R user=scalemail argv=/usr/lib/scalemail/bin/scalemail-store ${nexthop} ${user} ${extension}
mailman   unix  -       n       n       -       -       pipe
  flags=FR user=list argv=/usr/lib/mailman/bin/postfix-to-mailman.py
  ${nexthop} ${user}

submission inet n       -       n       -       -       smtpd
  -o smtpd_enforce_tls=yes
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_mynetworks,permit_sasl_authenticated,reject
#  -o content_filter=smtp-amavis:[127.0.0.1]:10026

# Use dovecot deliver program as LDA.
dovecot unix    -       n       n       -       -      pipe
    flags=DRhu user=vmail:vmail argv=/usr/lib/dovecot/deliver -f ${sender} -d ${user}@${domain} -m ${extension}

smtp-amavis unix -  -   -   -   2  smtp
    -o smtp_data_done_timeout=1200
    -o smtp_send_xforward_command=yes
    -o disable_dns_lookups=yes
    -o max_use=20

127.0.0.1:10025 inet n  -   -   -   -  smtpd
    -o content_filter=
    -o local_recipient_maps=
    -o relay_recipient_maps=
    -o smtpd_restriction_classes=
    -o smtpd_delay_reject=no
    -o smtpd_tls_security_level=none
    -o smtpd_client_restrictions=permit_mynetworks,reject
    -o smtpd_helo_restrictions=
    -o smtpd_sender_restrictions=
    -o smtpd_recipient_restrictions=permit_mynetworks,reject
    -o smtpd_end_of_data_restrictions=
    -o mynetworks_style=host
    -o mynetworks=127.0.0.0/8
    -o strict_rfc821_envelopes=yes
    -o smtpd_error_sleep_time=0
    -o smtpd_soft_error_limit=1001
    -o smtpd_hard_error_limit=1000
    -o smtpd_client_connection_count_limit=0
    -o smtpd_client_connection_rate_limit=0
    -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks,no_address_mappings
# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (yes)   (never) (100)
# ==========================================================================
# Added using postfix-add-policy script:
policy-spf unix    -       n       n       -       0     spawn
      user=nobody argv=/usr/bin/policyd-spf

4

Re: email sends really slow

matthewjamesorlando wrote:

That entire section is commented out and the -o content_filter is near the end of the page with = blank. Here is the master.cf file

There's another section at the bottom, please use that one instead of this uncommented one.

matthewjamesorlando wrote:

Is it possible that the web server is sending it over slowly? I don't think that would be an issue given the speeds in other areas, but you never know...anyway

How do you send emails? With Outlook/Thunderbird or Roundcube webmail?

5

Re: email sends really slow

Oops, my mistake, i forgot to explain why use "-o content_filter=" for submission transport.

With "-o content_filter=", all emails submitted via submission transport (port 587) will bypass spam and virus scanning directly. Actually, bypass Amavisd + SpamAssassin + ClamAV.

6 (edited by matthewjamesorlando 2012-09-08 04:12:33)

Re: email sends really slow

Hi Zhang...

I set the  "-o content_filter=" to look like this: -o content_filter=

I checked the mail log on a test and it still shows Amavis in the log....Here is the entry

Sep  7 16:04:37 mail amavis[2427]: (02427-10) Passed CLEAN, MYUSERS LOCAL [2XX.XX.XXX.26] [2XX.XX.XXX.26] <arpreach@mydomain.com> -> <example@mydomain.com>, Message-ID: <c44ba0271ff4a9396b6611a8dcea281b@mydomain.com>, mail_id: ru8FWL4EuztX, Hits: -10, size: 822, queued_as: 288A425101B, 864 ms

I'm currently using a program called ArpReach to send the mail to my subscribers. For personal mail I'm using roundcube. They took a look into my system and said there is nothing in there program or in MySQL that is causing the slow processing.

7

Re: email sends really slow

About ArpReach:

- Does is perform SMTP SASL authentication?
- Which port does it use? 25 or 587?

A quick solution: Don't use Amavisd + SpamAssassin + ClamAV for outgoing emails. Please edit /etc/amavis/conf.d/50-user, uncomment 4 bypass_* settings:

$policy_bank{'MYUSERS'} = {
    ...
    # don't perform spam/virus/header check.
    bypass_spam_checks_maps => [1],
    bypass_virus_checks_maps => [1],
    bypass_header_checks_maps => [1],

    # allow sending any file names and types
    bypass_banned_checks_maps => [1],
};

8

Re: email sends really slow

all content filter must be coment ? or only submission smtp ??

9

Re: email sends really slow

All.