1 (edited by maretodoric 2019-09-24 21:54:12)

Topic: Roundcube/Dovecot timeout when searching for large inboxes in Roundcub

==== REQUIRED BASIC INFO OF YOUR IREDMAIL SERVER ====
- iRedMail version (check /etc/iredmail-release): 0.9.9
- Deployed with iRedMail Easy or the downloadable installer? Downloadable installer
- Linux/BSD distribution name and version: CentOS 7.6.1810
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): MySQL (MariaDB)
- Web server (Apache or Nginx): Nginx
- Manage mail accounts with iRedAdmin-Pro? No
- [IMPORTANT] Related original log or error message is required if you're experiencing an issue.
Log file:

Sep 24 15:42:03 orion-mail1 dovecot: imap(kolektor@sezamhosting.com): Connection closed (UID SORT running for 98.229 + waiting input for 0.002 secs, 0.001 in locks, 49 B in + 395 B out, state=wait-external) in=86 out=1212

Error message on roundcube page:

Server Error! (Gateway Time-out)

====

My current Inbox size is 28801, beside that i have couple of other folders archived by years, all messages in total of over 300k. When i try to use search in roundcube, after some time i get timed out with message displayed above and in /var/log/dovecot/imap.log file i have line also displayed above.

For now, the troubleshooting steps I've applied are increasing php and php-fpm timeouts and memory in dovecot
/etc/php.ini:

max_execution_time = 600
max_input_time = 600

/etc/php-fpm.d/www.conf:

request_terminate_timeout = 60s
request_slowlog_timeout = 60s

/etc/dovecot/conf.d/10-master.conf:

service imap {
  # Most of the memory goes to mmap()ing files. You may need to increase this
  # limit if you have huge mailboxes.
  #vsz_limit = $default_vsz_limit
  vsz_limit = 1024M

  # Max. number of IMAP processes (connections)
  #process_limit = 1024
}

But nothing prevails. All i have managed is to increase time took that the search is going on for, but there had to be another way to better figure this out.
Any help would be greatly appreciated!

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2

Re: Roundcube/Dovecot timeout when searching for large inboxes in Roundcub

That's known "issue" of Dovecot with large mailbox. While searching, it's expected that Dovecot takes longer time to finish the search. The solution is setting up Full-Text Search support with Apache Solr or similar tools.

FYI: https://wiki.dovecot.org/Plugins/FTS/Solr

There's a new project which uses ElasticSearch instead of Solr:
https://github.com/filiphanes/fts-elastic

3

Re: Roundcube/Dovecot timeout when searching for large inboxes in Roundcub

Thanks for the info. For that to work, I've had to recompile dovecot since unfortunately doesn't come with native FTS plugins support. I've compiled it to work with solr 7.7.2 and will test it out further on.

We also use elasticsearch for some other purposes, will investigate that too, maybe i could incorporate that.

4

Re: Roundcube/Dovecot timeout when searching for large inboxes in Roundcub

have there been any other fixes or solutions to this problem?
I have a pretty high end S2 machine and the gateway timeouts are so short it is ridicules. I don't understand why someone hasn't fixed the programming.

Also ZhangHuangbin is this something I can have you do? Maybe i could have you do this and also my latest iRedAdmin-Pro update?

5

Re: Roundcube/Dovecot timeout when searching for large inboxes in Roundcub

Hi @cjacobsen, you can try FTS plugins here: https://doc.dovecot.org/configuration_manual/fts/
iRedMail Easy uses https://github.com/grosjo/fts-xapian

6

Re: Roundcube/Dovecot timeout when searching for large inboxes in Roundcub

Currently I am running centos and am uncomfortable installing this myself. Is this something that I could hire you or your support staff to install for me?

7

Re: Roundcube/Dovecot timeout when searching for large inboxes in Roundcub

cjacobsen wrote:

Currently I am running centos and am uncomfortable installing this myself. Is this something that I could hire you or your support staff to install for me?

Sure. Purchase a support ticket here and contact us (you will receive an email right after payment completed):
https://www.iredmail.org/support.html