Nicolas,

Allow me to share few tips with you just in case it helps. I'm sure others will have additional suggestions to help you address the problem.

First, if there is no connection at all to webmail, to me it sounds as if the server is either inaccessible at the time or the client IP has been banned. Getting an MTR, traceroute or similar could help at the time to determine what's going wrong.

If clients are not banned, they could be hitting a dovecot limitation 'mail_max_userip_connections = 10'... 10 being the default. That means that only 10 connections per user/per IP can be established at any given time. Users with multiple accounts and multiple devices sourcing from the same IP could be hitting the default limit. Obviously, dovecot should log this somewhere and you mention there is nothing in the logs. The other thing is that users with multiple accounts do not have the same username/password on each account, so I don't believe this could be affecting all users in that sense. Irrespective, it has been my experience that hitting the 10 connections per user/per IP is rather very common. A client with a Desktop application, a mobile device and a tablet, sitting at home and connecting to her/his own account can easily go over 10 connections in a heartbeat. 

Also, dovecot process_limit is 500 by default (the number of simultaneous IMAP connections)... If there are 150 users hitting your mailserver from different clients at the same time it doesn't look too far away to max the 500 process_limit. Assuming that a client can establish (unnecessarily in most cases) up to 5 connections per requests (150 x 5 = 750 connections). That's way over the process_limit = 500 by default.

What I've done myself to mitigate a similar experience (making sure it is not network or service attack related) was to increase these defaults to a more tolerant number (depending on the service load):

Here is what has worked for me so far (although you may have to adjust these to your own environment):

# /etc/dovecot/dovecot.conf

For your protocols, set a higher number of connections per user/per IP:

...
protocol imap {
...
 mail_max_userip_connections = 20
...
}
...

...
protocol pop3 {
...
 mail_max_userip_connections = 20
...
}
...

Then, I have adjusted the login processes. (Refer to Dovecot wiki for more details: http://wiki2.dovecot.org/LoginProcess)

...
service imap-login {
    service_count = 1

    # To avoid startup latency for new client connections, set process_min_avail
    # to higher than zero. That many idling processes are always kept around
    # waiting for new connections.
    process_min_avail = 16

    # number of simultaneous IMAP connections
    #process_limit = $default_process_limit
    process_limit = 800
...

Finally, the more users connected to your service, the higher the load. Hence I have proxied roundcube webmail (imapproxy) to alleviate the load in the IMAP server. Run a test and you will find 10-15 IMAP connections per webmail user at any given time for every click/interaction in webmail; it is indeed stressful for any IMAP server. Using imapproxy in your box will keep those 15-20 simultaneous connections down to a single open connection between webmail and the backend IMAP server. It also helps to speed up folder listing, etc.

Hope this help if it isn't indeed a network issue. Logs are your friend and something must have been logged somewhere. Good luck.

Is the IP address blocked in iptables when this issue happens?

The big difference after server reboot is iptables rules will be emptied.

It drops connection directly, and client will time out.

Maybe you can add some additional rules in iptables to log dropped connection/packet for debugging purpose.
References:

- http://www.cyberciti.biz/tips/force-ipt … -file.html
- http://www.thegeekstuff.com/2012/08/ipt … g-packets/