1 (edited by jozevolf 2020-08-19 15:46:40)

Topic: Dovecot FTS (Full Text Search) using Apache Solr on Ubuntu 18.04 LTS

==== REQUIRED BASIC INFO OF YOUR IREDMAIL SERVER ====
- iRedMail version (check /etc/iredmail-release): 1.3.1
- Deployed with iRedMail Easy or the downloadable installer? downloadable
- Linux/BSD distribution name and version: Ubuntu 18.04.4 LTS
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): MariaDB
- Web server (Apache or Nginx): Nginx
- Manage mail accounts with iRedAdmin-Pro? no
- [IMPORTANT] Related original log or error message is required if you're experiencing an issue.
====

I would like to share the steps I used to enable FTS on my iRedMail server. The cookbook is made as root, please use sudo as appropriate. I am no nix guru so please comment if I did something wrong or if there is a better way of doing it.

First let's install dovecot-solr plugin and openjdk packages:

apt update
apt install dovecot-solr
apt install openjdk-11-jre-headless

Then we have to download Solr package and extract/run the install script:

cd /opt/
wget https://www.apache.si/lucene/solr/8.6.1/solr-8.6.1.tgz
tar xzf solr-8.6.1.tgz solr-8.6.1/bin/install_solr_service.sh --strip-components=2
bash ./install_solr_service.sh solr-8.6.1.tgz

We get some system limits warnings, so we create /etc/security/limits.d/solr.conf file and insert the following into it:

solr             soft    nofile          65000
solr             hard    nofile          65000
solr             soft    nproc           65000
solr             hard    nproc           65000

We can restart the solr service to confirm that we don't get those warnings anymore:

/etc/init.d/solr restart

Now let's create dovecot solr core and configure it:

sudo -u solr /opt/solr/bin/solr create -c dovecot
cd /var/solr/data/dovecot/conf/
rm -f /var/solr/data/dovecot/conf/schema.xml
rm -f /var/solr/data/dovecot/conf/managed-schema
rm -f /var/solr/data/dovecot/conf/solrconfig.xml
wget https://raw.githubusercontent.com/dovecot/core/master/doc/solr-config-7.7.0.xml
mv solr-config-7.7.0.xml solrconfig.xml
wget https://raw.githubusercontent.com/dovecot/core/master/doc/solr-schema-7.7.0.xml
mv solr-schema-7.7.0.xml schema.xml
/etc/init.d/solr restart

It's time to configure dovecot by editing /etc/dovecot/dovecot.conf
We have to add the following into appropriate places:

...
mail_plugins = $mail_plugins fts fts_solr
...
plugin {
  ...
  fts = solr
  fts_autoindex = yes
  fts_solr = url=http://127.0.0.1:8983/solr/dovecot/
}

Let's just restart dovecot and we have a working FTS setup.

/etc/init.d/dovecot restart

Now we can force reindex a mailbox:

doveadm fts rescan -u user@domain
doveadm index -u user@domain -q '*'

Please see https://wiki.dovecot.org/Plugins/FTS/Solr and also add cron jobs to keep Solr indexes optimal:

# Optimize should be run somewhat rarely, e.g. once a day
curl https://<hostname/ip>:<port|default 8983>/solr/dovecot/update?optimize=true
# Commit should be run pretty often, e.g. every minute
curl https://<hostname/ip>:<port|default 8983>/solr/dovecot/update?commit=true

Let's just secure the Solr a little by limiting the access only from localhost by adding the following environment variable to /etc/environment

SOLR_IP_WHITELIST="127.0.0.1, [::1]"

And restart the Solr:

/etc/init.d/solr restart

This is it.

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2

Re: Dovecot FTS (Full Text Search) using Apache Solr on Ubuntu 18.04 LTS

Thanks for sharing. smile

3

Re: Dovecot FTS (Full Text Search) using Apache Solr on Ubuntu 18.04 LTS

i would not recommend doing # Commit should be run pretty often, e.g. every minute
curl /solr/dovecot/update?commit=true

its built into solr


i cant post links but if you google

lucidworks understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/. has a decent read on it