1 Topic by NuLL3rr0r

  • NuLL3rr0r
  • Member
  • Offline
  • Registered: 2022-12-22
  • Posts: 18

Topic: iRedMail messes up the SSL certificates on it's own

==== REQUIRED BASIC INFO OF YOUR IREDMAIL SERVER ====
- iRedMail version (check /etc/iredmail-release): 1.6.2 PGSQL edition.
- Deployed with iRedMail Easy or the downloadable installer? downloadable installer
- Linux/BSD distribution name and version: FreeBSD 13.1-RELEASE-p6 GENERIC amd64
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): PGSQL
- Web server (Apache or Nginx): Nginx
- Manage mail accounts with iRedAdmin-Pro? No
- [IMPORTANT] Related original log or error message is required if you're experiencing an issue.
====

My mailserver suddenly stopped working and I haven't been logged on this server for about a month. It seems when iRedMail tries to renew the certificates using Let's Encrypt it somehow fails. This is the second time that it's happened. Last time I was able to revive the old certificates using a ZFS snapshot. And, I had a backup. I copied back the certs and now everything works as expected. The issue is the certificates are going to expire on 26th of May and probably the mail server stops again.

Here is the exact problem:

Inside the /etc/ssl/private when I run the following command, I see:

$ ls -lah
total 19
drwxr-xr-x  2 root      wheel        5B Dec 21 02:17 .
drwxr-xr-x  5 root      wheel        9B Dec 21 00:18 ..
-rw-------  1 postgres  postgres   3.2K Mar 25 02:13 iRedMail_PostgreSQL.key
lrwxr-xr-x  1 root      wheel       57B Dec 21 02:17 iRedMail.key -> /usr/local/etc/letsencrypt/live/somedomain.com/privkey.pem
-rw-r--r--  1 root      wheel      3.2K Mar 25 02:13 iRedMail.key.bak

The original backups I've had is just a plain file not a symlink. I also copied back the iRedMail.crt in the certs folder (it was not a symlink). Rebooted and now works. But, I'm afraid this is a bug in iRedMail that somehow I don't understand what's going on.

Sample Nginx logs:

2023/04/30 20:58:45 [emerg] 37812#100284: SSL_CTX_use_PrivateKey("/etc/ssl/private/iRedMail.key") failed (SSL: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch)
2023/04/30 20:58:45 [emerg] 65596#100284: SSL_CTX_use_PrivateKey("/etc/ssl/private/iRedMail.key") failed (SSL: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch)
2023/05/01 10:26:28 [emerg] 1698#100424: SSL_CTX_use_PrivateKey("/etc/ssl/private/iRedMail.key") failed (SSL: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch)
2023/05/01 10:29:39 [emerg] 21233#100337: SSL_CTX_use_PrivateKey("/etc/ssl/private/iRedMail.key") failed (SSL: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch)
2023/05/01 10:30:16 [emerg] 45523#100388: SSL_CTX_use_PrivateKey("/etc/ssl/private/iRedMail.key") failed (SSL: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch)

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2 Reply by Cthulhu (edited by Cthulhu 2023-05-01 21:16:37)

  • Cthulhu
  • Member
  • Offline
  • Registered: 2019-06-04
  • Posts: 914

Re: iRedMail messes up the SSL certificates on it's own

that's nginx and not iredmail

and where is the chainfile?
you actually need both, the pkey and the chainfile to get it working


aswell, you need to restart nginx and postfix after you update the certs

3 Reply by NuLL3rr0r

  • NuLL3rr0r
  • Member
  • Offline
  • Registered: 2022-12-22
  • Posts: 18

Re: iRedMail messes up the SSL certificates on it's own

Well, it's just not Nginx. Something is wrong with the cert update process. Even some people told me they cannot send me emails anymore. So, I've checked the Postfix log and see the following as well now (again the same mismatch errors as Nginx):

May  2 01:16:31 mx postfix/smtps/smtpd[26551]: SSL_accept error from unknown[112.27.129.78]: lost connection
May  2 01:16:44 mx postfix/smtps/smtpd[26551]: SSL_accept error from unknown[61.19.228.102]: lost connection
May  2 01:16:46 mx postfix/smtps/smtpd[49220]: SSL_accept error from unknown[121.22.99.2]: lost connection
May  2 02:10:43 mx postfix/smtps/smtpd[14432]: SSL_accept error from unknown[205.210.31.97]: -1
May  2 02:10:43 mx postfix/smtps/smtpd[14432]: warning: TLS library problem: error:1408F09C:SSL routines:ssl3_get_record:http request:ssl/record/ssl3_record.c:353:
May  2 02:11:53 mx postfix/smtpd[62533]: SSL_accept error from unknown[114.239.88.87]: lost connection
May  2 03:07:00 mx postfix/smtpd[41625]: SSL_accept error from unknown[58.209.197.148]: lost connection
May  2 03:19:02 mx postfix/smtpd[71925]: SSL_accept error from unknown[180.107.177.4]: lost connection
May  2 03:19:33 mx postfix/smtpd[71925]: SSL_accept error from unknown[180.107.177.4]: lost connection
May  2 03:34:25 mx postfix/smtpd[2053]: SSL_accept error from unknown[117.37.131.177]: lost connection
May  2 03:34:57 mx postfix/smtpd[2053]: SSL_accept error from unknown[117.37.131.177]: lost connection
May  2 03:39:20 mx postfix/smtpd[2053]: SSL_accept error from unknown[218.67.13.220]: Connection reset by peer
May  2 03:53:01 mx postfix/smtpd[41013]: SSL_accept error from unknown[221.225.147.132]: lost connection
May  2 04:04:49 mx postfix/smtpd[77737]: SSL_accept error from unknown[121.234.177.182]: lost connection
May  2 04:39:18 mx postfix/smtpd[46148]: SSL_accept error from unknown[60.184.193.172]: Connection reset by peer
May  2 04:40:32 mx postfix/smtps/smtpd[14933]: SSL_accept error from unknown[192.241.217.10]: -1
May  2 04:40:32 mx postfix/smtps/smtpd[14933]: warning: TLS library problem: error:1408F10B:SSL routines:ssl3_get_record:wrong version number:ssl/record/ssl3_record.c:363:
May  2 08:36:14 mx postfix/smtpd[10032]: warning: TLS library problem: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch:crypto/x509/x509_cmp.c:303:
May  2 08:37:11 mx postfix/smtps/smtpd[30401]: warning: TLS library problem: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch:crypto/x509/x509_cmp.c:303:
May  2 10:03:25 mx postfix/smtps/smtpd[76002]: SSL_accept error from unknown[170.106.115.253]: Connection reset by peer
May  2 10:03:56 mx postfix/smtps/smtpd[76002]: SSL_accept error from unknown[43.131.37.227]: Connection reset by peer
May  2 10:03:58 mx postfix/smtps/smtpd[36840]: SSL_accept error from unknown[43.131.37.227]: Connection reset by peer
May  2 10:04:00 mx postfix/smtps/smtpd[76002]: SSL_accept error from unknown[43.131.37.227]: Connection reset by peer
May  2 10:04:02 mx postfix/smtps/smtpd[36840]: SSL_accept error from unknown[43.131.37.227]: Connection reset by peer
May  2 10:04:04 mx postfix/smtps/smtpd[76002]: SSL_accept error from unknown[43.131.37.227]: Connection reset by peer
May  2 10:04:06 mx postfix/smtps/smtpd[36840]: SSL_accept error from unknown[43.131.37.227]: Connection reset by peer
May  2 10:04:08 mx postfix/smtps/smtpd[76002]: SSL_accept error from unknown[43.131.37.227]: Connection reset by peer
May  2 10:04:10 mx postfix/smtps/smtpd[36840]: SSL_accept error from unknown[43.131.37.227]: Connection reset by peer
May  2 10:04:12 mx postfix/smtps/smtpd[76002]: SSL_accept error from unknown[43.131.37.227]: -1
May  2 10:04:12 mx postfix/smtps/smtpd[76002]: warning: TLS library problem: error:142090C1:SSL routines:tls_early_post_process_client_hello:no shared cipher:ssl/statem/statem_srvr.c:1781:
May  2 10:04:14 mx postfix/smtps/smtpd[36840]: SSL_accept error from unknown[43.131.37.227]: -1
May  2 10:04:14 mx postfix/smtps/smtpd[36840]: warning: TLS library problem: error:141CF06C:SSL routines:tls_parse_ctos_key_share:bad key share:ssl/statem/extensions_srvr.c:698:
May  2 10:09:31 mx postfix/smtps/smtpd[50931]: SSL_accept error from unknown[183.253.125.205]: lost connection

4 Reply by NuLL3rr0r

  • NuLL3rr0r
  • Member
  • Offline
  • Registered: 2022-12-22
  • Posts: 18

Re: iRedMail messes up the SSL certificates on it's own

Also here are the cert files:

ls -lah /usr/local/etc/letsencrypt/live/mx.somedomain.net/
total 16
drwxr-xr-x  2 root  wheel     7B Apr 29 04:31 .
drwxr-xr-x  3 root  wheel     4B Dec 21 02:08 ..
lrwxr-xr-x  1 root  wheel    37B Apr 29 04:31 cert.pem -> ../../archive/mx.somedomain.net/cert3.pem
lrwxr-xr-x  1 root  wheel    38B Apr 29 04:31 chain.pem -> ../../archive/mx.somedomain.net/chain3.pem
lrwxr-xr-x  1 root  wheel    42B Apr 29 04:31 fullchain.pem -> ../../archive/mx.somedomain.net/fullchain3.pem
lrwxr-xr-x  1 root  wheel    40B Apr 29 04:31 privkey.pem -> ../../archive/mx.somedomain.net/privkey3.pem
-rw-r--r--  1 root  wheel   692B Dec 21 02:08 README

5 Reply by NuLL3rr0r

  • NuLL3rr0r
  • Member
  • Offline
  • Registered: 2022-12-22
  • Posts: 18

Re: iRedMail messes up the SSL certificates on it's own

I've also tried the documentation here https://docs.iredmail.org/letsencrypt.h … omatically

mv /etc/ssl/certs/iRedMail.crt{,.bak}       # Backup. Rename iRedMail.crt to iRedMail.crt.bak
mv /etc/ssl/private/iRedMail.key{,.bak}     # Backup. Rename iRedMail.key to iRedMail.key.bak
ln -s /usr/local/etc/letsencrypt/live/mx.somedomain.net/fullchain.pem /etc/ssl/certs/iRedMail.crt
ln -s /usr/local/etc/letsencrypt/live/mx.somedomain.net/privkey.pem /etc/ssl/private/iRedMail.key

In order to symlink the certs. But, again Nginx for example refuses to start.

6 Reply by NuLL3rr0r (edited by NuLL3rr0r 2023-05-02 19:38:22)

  • NuLL3rr0r
  • Member
  • Offline
  • Registered: 2022-12-22
  • Posts: 18

Re: iRedMail messes up the SSL certificates on it's own

According to this guide https://www.ssl247.com/knowledge-base/d … 015hscaay/ also it seems whatever the certbot has generated won't match.

$ openssl rsa -check -noout -in privkey.pem
RSA key ok

openssl rsa -modulus -noout -in privkey.pem | openssl md5
(stdin)= bf7a5987646d3f08c9d61a1ce84e339c

openssl x509 -modulus -noout -in fullchain.pem | openssl md5
(stdin)= 0737110fd603398a8c479be18e3628c8

Running the same command on the original private/public key:

$ openssl rsa -modulus -noout -in /etc/ssl/private/iRedMail.key | openssl md5
(stdin)= cba9c79208e2a363ddb2e2affa22cf69

$ openssl x509 -modulus -noout -in /etc/ssl/certs/iRedMail.crt | openssl md5
(stdin)= cba9c79208e2a363ddb2e2affa22cf69

So, somehow certbot or something else messes up those certs.

7 Reply by NuLL3rr0r

  • NuLL3rr0r
  • Member
  • Offline
  • Registered: 2022-12-22
  • Posts: 18

Re: iRedMail messes up the SSL certificates on it's own

OK, whatever the reason was (because it's not the first time this appears, maybe there is a bug with the cron script or something), I force renewed the certs from the command line:

$ certbot -d mx.somedomain.net --force-renewal

And, it created a new directory inside /usr/local/etc/letsencrypt/live/ named mx.somedomain.net-0001, so:

$ cd /usr/local/etc/letsencrypt/live/mx.somedomain.net-0001/
$ cp -vr * ../mx.somedomain.net/
$ cd ..
$ rm -rf mx.somedomain.net-0001

Then I tried the following:

$ mv /etc/ssl/certs/iRedMail.crt{,.bak}
$ mv /etc/ssl/private/iRedMail.key{,.bak}
$ ln -s /usr/local/etc/letsencrypt/live/mx.somedomain.net/fullchain.pem /etc/ssl/certs/iRedMail.crt
$ ln -s /usr/local/etc/letsencrypt/live/mx.somedomain.net/privkey.pem /etc/ssl/private/iRedMail.key

rebooted the whole server and certificates are updated.

In order to see whether this happens again or not, probably I have to wait for the next renewal.

And, also not sure yet why since this incident happened some people are not able to email me.

8 Reply by NuLL3rr0r (edited by NuLL3rr0r 2023-05-02 19:55:52)

  • NuLL3rr0r
  • Member
  • Offline
  • Registered: 2022-12-22
  • Posts: 18

Re: iRedMail messes up the SSL certificates on it's own

And, also if it helps this is how I noticed the incident. I have a Gmail account forwarded to this iRedMail-hosted mailbox.

Edit: oops! It seems the forum won't allow me to upload an image.

9 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,131

Re: iRedMail messes up the SSL certificates on it's own

The problem is "certbot" didn't update symbol link of /etc/letsencrypt/live/<domain>/privkey.pem to the renewed one automatically. So you must add a command in --post-hook argument to re-create this symbol link, for example:

1 3 * * * certbot renew --post-hook 'ln -sf /etc/letsencrypt/live/<domain>/privkey.pem /opt/iredmail/ssl/key.pem; /usr/sbin/service postfix restart; /usr/sbin/service nginx restart; /usr/sbin/service dovecot restart'