1 Topic by brix

  • brix
  • Member
  • Offline
  • Registered: 2014-01-22
  • Posts: 30

Topic: High availability in different datacenters

==== Required information ====
- iRedMail version (check /etc/iredmail-release): 0.9.3
- Linux/BSD distribution name and version: CentOS 7
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): LDAP
- Web server (Apache or Nginx): Nginx
- Manage mail accounts with iRedAdmin-Pro? Yes
- Related log if you're reporting an issue: no issue, just a question :)
====
Hello!
A very happy iRedMail and iRedAdmin-Pro user here :) Our company have been using this implementation of mail server for some time and it working great. Now we want to step a little bit further and implement a second mail server which will be is sync with the first. Ideally they will work as master-master. It would be in different data centers so all the synchronization needs to be asynchronous (or very very optimized :)

iRedAdmin-Pro will be installed on the first master, so we won't have ability to access admin panel through second master (we just need a live backup - master, so clients can work)

So the first thing we did is to sync the OpenLDAP with it's replication, so we have users on both sides. We still need to sync back, if the user changes the password when the first master is offline.
The second thing is to sync the user dirs. We did this with dovecot (2.2.x) replication through ssh and it's working very very good. It goes both ways, when the first master goes offline, the messages go to the second server and when the first gets up, they are synced.
DKIM keys will also be synced with a custom script.
The webmail - roundcube will be in different datacenter that has a high availability implementation. The mysql database will be in percona cluster. We will probably implement Haproxy for redirecting traffic to SMTP and IMAP.

We tried to use two A records for the IMAP and SMTP connections and all the newer mail clients switch to the working server. We will probably also implement Haproxy for redirecting traffic. Postfix has a proxy implementation so it knows the client ip with some configuration.

As for domains, we just put two MX records on the same priority 1, so incoming mail will be load balanced.

Some things already work on testing servers and other things are theoretically and practically possible to implement. This is all asynchronous synchronization so we can put the second server in the different datacenter and don't worry about waiting to write on the second server as it is with synchronous synchronization (like percona cluster)

Do we need to sync any mysql databases? The sync through WAN is very slow, because master waits for writing on all nodes. We could implement WAN deployed percona cluster, if there are not so many writes on the mysql.

What about the iredapd (replaced cluebringer) and amavis (DKIM synced, anything else?)

I am aware of the tutorial - "An Ultra-HA, full Mult-Master E-mail cluster with iRedMail, MariaDB, and IPVS" but it's only for local implementation in one datacenter.

Zhang Huangbin, can you please highlight things that we still need to sync between both servers? As far as I'm aware, the spam implementation also uses some mysql databases to learn spam from incoming messages, can we also sync those? And is there anything else that we need to be aware of and that we need to implement?

So if I recap, we already "synced":
-users - ldap replication, sync back needed*
-user dirs - dovecot replication
-DKIM
-Roundcube and sogo on different platform
-SMTP and IMAP through two A records or haproxy on high available IP address

What else we need to sync?
-iredapd (database for gray, black, white listing?)
-amavis (DKIM already, anything else?)
-do we need iredadmin database on the second server?
-mysql?
-?

When the sistem will be implemented and working well, I am willing to help write a tutorial or even help implement master-master in different datacenters as an option to iRedMail if anybody is interested and if Zhang Huangbin is willing to help. I kindly ask you to look a little bit further into this. I think that a project strong as this one really needs a high available solution.

Thank you very much!
Greetings from Slovenia
brix

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 30,770

Re: High availability in different datacenters

brix wrote:

What else we need to sync?
-iredapd (database for gray, black, white listing?)
-amavis (DKIM already, anything else?)
-do we need iredadmin database on the second server?
-mysql?

1) iRedAPD stores greylisting and throttling settings and tracking data. They should be synced.
2) Amavisd stores 2 different data:

*) Basic info of in/out emails (no full mail body and mail headers). Used in Amavisd setting '@storage_sql_dsn'.
*) Global, per-domain, per-user white/blacklists, spam policies. Used in Amavisd setting '@lookup_sql_dsn'.

The data in @storage_sql_dsn it's ok to NOT sync (of course it still depends on your demand), but the @lookup_sql_dsn should be synced - and it's used by iRedAPD plugin 'amavisd_wblist'.

3) iredadmin database. With OpenLDAP backend, iRedMail stores some info in iredadmin database:

*) real-time quota usage - user has how many emails stored, and the size of them in total.

It's updated by Dovecot, but iRedAdmin-Pro needs them to show domain admin the mailbox quota usage, to make programming easier, we decided to store it in iredadmin database instead of a separate database.

*) info of deleted mailboxes - it stores the full maildir path of removed mailbox, admin can actually delete the mailbox in a future day manually. (BTW, For SQL backends, it's stored in `vmail` database.)

*) iredadmin itself doesn't store its configurations in SQL database.

Any suggestions we can make it better in a cluster setup?

FYI if you're interested in why we store white/blacklists (wblist) in Amavisd database: in the early days, we store wblist in Policyd or Cluebringer, but Amavisd doesn't know it's whitelisted senders, so sometimes Amavisd still marks them as spam based on SpamAssassin scanning result. To overcome this, we use the Amavisd built-in wblist implementation - it works in after-queue phase, and develop an iRedAPD plugin (amavisd_wblist) to work in before-queue phase.

brix wrote:

When the sistem will be implemented and working well, I am willing to help write a tutorial or even help implement master-master in different datacenters as an option to iRedMail if anybody is interested and if Zhang Huangbin is willing to help. I kindly ask you to look a little bit further into this. I think that a project strong as this one really needs a high available solution.

I'm very interesting in your cluster implementation, this will bring iRedMail to another level.

You can contact me directly via email or Skype (id: michaelbibby) or BBM (BlackBerry Messenger): 2AAB1F7E (this pin might be changed if i switched to another phone in the future).