1 Topic by maxie_ro

  • maxie_ro
  • Member
  • Offline
  • From: Constanţa, România
  • Registered: 2009-10-09
  • Posts: 214

Topic: Too many connections

Hello.

I found the mail service down today with errors in maillog:

Oct 13 13:51:35 mx2 postfix/master[2975]: warning: process /usr/libexec/postfix/trivial-rewrite pid 16373 exit status 1
Oct 13 13:51:35 mx2 postfix/smtpd[14931]: warning: problem talking to service rewrite: Success
Oct 13 13:51:35 mx2 postfix/master[2975]: warning: process /usr/libexec/postfix/trivial-rewrite pid 16374 exit status 1
Oct 13 13:51:35 mx2 postfix/smtpd[14932]: warning: problem talking to service rewrite: Success
[........]
Oct 13 13:52:36 mx2 postfix/trivial-rewrite[16477]: warning: connect to mysql server 127.0.0.1: Too many connections
Oct 13 13:52:36 mx2 postfix/trivial-rewrite[16477]: warning: connect to mysql server 127.0.0.1: Too many connections
Oct 13 13:52:36 mx2 postfix/trivial-rewrite[16477]: fatal: mysql:/etc/postfix/mysql_virtual_alias_maps.cf(0,lock|fold_fix): table lookup problem
Oct 13 13:52:36 mx2 postfix/trivial-rewrite[16478]: warning: connect to mysql server 127.0.0.1: Too many connections
Oct 13 13:52:36 mx2 postfix/trivial-rewrite[16478]: warning: connect to mysql server 127.0.0.1: Too many connections
Oct 13 13:52:36 mx2 postfix/trivial-rewrite[16478]: fatal: mysql:/etc/postfix/mysql_virtual_alias_maps.cf(0,lock|fold_fix): table lookup problem
[..............]

I guess these are from max_connections in my.cnf right? I set the limit to 1000, but I'm wondering why it crashed so soon already. There are not many mailboxes right now, just about 200 (and around 1000 more to import from the old server...). Should I check something?

Also, googling around, I saw "proxymap" might be a solution to this. Would it work with iRedOS?

Thanks.

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2 Reply by maxie_ro

  • maxie_ro
  • Member
  • Offline
  • From: Constanţa, România
  • Registered: 2009-10-09
  • Posts: 214

Re: Too many connections

Still getting

Oct 13 15:29:32 mx2 postfix/cleanup[22911]: warning: 5AE142C1010C: virtual_alias_maps map lookup problem for postmaster@mx2.mydsl.ro
Oct 13 15:30:07 mx2 postfix/cleanup[22911]: warning: mysql query failed: MySQL server has gone away

ETC.

max_connections set at 1000, timeouts at 60 sec...

3 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,122

Re: Too many connections

Could you please check MySQL server status and try to find out why it was gone way?

4 Reply by maxie_ro

  • maxie_ro
  • Member
  • Offline
  • From: Constanţa, România
  • Registered: 2009-10-09
  • Posts: 214

Re: Too many connections

There is nothing in mysqld.log, besides the usual things. Mysqld was running at below 5% CPU usage, memory usage below 10%. But the maillog filled with 4 MB of log data in only 15 minutes, all with the same errors. All tables in vmail are ok (I checked integrity).

After setting the parameters like above, everything was fine again in 1-2 minutes, no more errors. But that "rush-hour thing" was realy strange.

Do you think proxymap could be a future solution if this happens again?
My.cnf is right now like this, do you think it's ok?:

mysqld]
bind-address            = 127.0.0.1
port                    = 3306
default-character-set   = utf8
datadir                 = /var/lib/mysql
socket                  = /var/lib/mysql/mysql.sock
user                    = mysql

# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
old_passwords=1

skip-locking
skip-bdb
#skip-networking

query-cache-type    = 1
query-cache-size    = 64M
query_cache_limit   = 10M

thread_cache_size   = 8

max_connections=1000
wait_timeout = 60
connect_timeout = 60
#set-variable = max_connect_errors=150

#ssl-ca =
ssl-cert = /etc/pki/tls/certs/iRedMail_CA.pem
ssl-key = /etc/pki/tls/private/iRedMail.key
ssl-cipher = ALL

key_buffer = 64M
max_allowed_packet = 4M
table_cache = 265
read_buffer_size = 8M
read_rnd_buffer_size = 4M
myisam_sort_buffer_size = 32M
thread_concurrency = 8

[client]
default-character-set=utf8

[mysql.server]
user        = mysql
basedir     = /var/lib

[mysqld_safe]
log-error   = /var/log/mysqld.log
pid-file    = /var/run/mysqld/mysqld.pid

5 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,122

Re: Too many connections

maxie_ro wrote:

Do you think proxymap could be a future solution if this happens again?

I think it can solve similar issue, refer to postfix maunal page: proxymap(8) (http://www.postfix.org/proxymap.8.html).

o      To consolidate the number of open lookup tables  by
              sharing  one  open  table among multiple processes.
              For example, making mysql  connections  from  every
              Postfix daemon process results in "too many connec-
              tions" errors. The solution:

              virtual_alias_maps =
                  proxy:mysql:/etc/postfix/virtual_alias.cf

              The total number of connections is limited  by  the
              number of proxymap server processes.

Maybe we should change all LDAP/MYSQL lookup into proxymap to avoid similar issues.

6 Reply by maxie_ro

  • maxie_ro
  • Member
  • Offline
  • From: Constanţa, România
  • Registered: 2009-10-09
  • Posts: 214

Re: Too many connections

I also think it would be a better idea to use PostgreSQL instead of MySQL. It's a lot more stable and resilient. I saw web servers with mysql locking with low load, while postgresql can withstand 1000 times more...

7 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,122

Re: Too many connections

Many high-load sites runs MySQL, and it works fine. So i think that's not the point.

I will make iRedMail use proxymap in 0.5.1 by default, so that we can reduce server load average in high-load server.

Thanks for your feedback. smile

8 Reply by maxie_ro

  • maxie_ro
  • Member
  • Offline
  • From: Constanţa, România
  • Registered: 2009-10-09
  • Posts: 214

Re: Too many connections

Hi, me again.

I saw this in postmaster's logwatch:

mysql query failed: MySQL server has gone away : 71 Time(s) 
problem talking to service rewrite: Connection reset by peer : 72 Time(s)
problem talking to service rewrite: Success : 5034 Time(s)
see http://www.postfix.org/STRESS_README.html for examples of stress-adapting configuration settings : 1 Time(s)
service "smtp" (25) has reached its process limit "100": new clients may experience noticeable delays : 1 Time(s)
to avoid this condition, increase the process count in master.cf or reduce the service time per client : 1 Time(s)

The link above (http://www.postfix.org/STRESS_README.html) is very interesting and is appliable to postfix 2.5.9. I'm talking especially about automatic stress-adaptive behavior. Quote:

smtpd_timeout = ${stress?10}${stress:300}s
smtpd_hard_error_limit = ${stress?1}${stress:20}
smtpd_junk_command_limit = ${stress?1}${stress:100}

Are these safe for current iRedOS? Did you test them to see if they work ok? And if they do, where should I put them, master.cf or main.cf? I'm asking that because the page above says they should be in main.cf, but I see some of them in master.cf in iRedOS.

Thanks!