1 Topic by maz (edited by maz 2021-10-29 09:27:51)

  • maz
  • Member
  • Offline
  • Registered: 2021-09-13
  • Posts: 11

Topic: debian upgrade 10 to 11 : mariadb issue [not solved - end of story]

==== REQUIRED BASIC INFO OF YOUR IREDMAIL SERVER ====
- iRedMail version (check /etc/iredmail-release):  1.4.2
- Deployed with iRedMail Easy or the downloadable installer? installer
- Linux/BSD distribution name and version: debian 10 - 11
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): mariadb
- Web server (Apache or Nginx): nginx
- Manage mail accounts with iRedAdmin-Pro? no
- [IMPORTANT] Related original log or error message is required if you're experiencing an issue.
====

I upgraded and then, immediately after upgrade, i followed the steps described here : https://docs.iredmail.org/upgrade.debian.10-11.html

But then I got many mariadb errors :
Oct 23 19:23:43 mail mariadbd[569]: 2021-10-23 19:23:43 10 [Warning] Aborted connection 10 to db: 'vmail' user: 'vmailadmin' host: 'localhost' (Got an error reading communication packets)
Oct 23 19:26:08 mail mariadbd[569]: 2021-10-23 19:26:08 9 [Warning] Aborted connection 9 to db: 'vmail' user: 'vmailadmin' host: 'localhost' (Got an error reading communication packets)
Oct 23 19:27:19 mail mariadbd[569]: 2021-10-23 19:27:19 54 [Warning] Aborted connection 54 to db: 'vmail' user: 'vmailadmin' host: 'localhost' (Got an error reading communication packets)

As I couldn't stay without my email server, i restored my last VM dump (wich i did just before upgrade smile )

So i'm back on my debian 10, but I would like to upgrade it.

Is there a way to put the server in a maintenance mode during debian upgrade ? Or should I do anything else (dump the db and restore them after upgrade...) ?

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2 Reply by maz

  • maz
  • Member
  • Offline
  • Registered: 2021-09-13
  • Posts: 11

Re: debian upgrade 10 to 11 : mariadb issue [not solved - end of story]

Hello,

So now i tried to upgrade another time.

First, i listed services :

# systemctl list-units --type=service --state=running
UNIT                     LOAD   ACTIVE SUB     DESCRIPTION                                             
amavis.service           loaded active running LSB: Starts amavisd-new mailfilter                      
clamav-daemon.service    loaded active running Clam AntiVirus userspace daemon                         
clamav-freshclam.service loaded active running ClamAV virus database updater                           
cron.service             loaded active running Regular background program processing daemon            
dbus.service             loaded active running D-Bus System Message Bus                                
dovecot.service          loaded active running Dovecot IMAP/POP3 email server                          
fail2ban.service         loaded active running Fail2Ban Service                                        
getty@tty1.service       loaded active running Getty on tty1                                           
iredadmin.service        loaded active running iRedAdmin daemon service                                
iredapd.service          loaded active running iRedAPD (A simple posfix policy server)                 
mariadb.service          loaded active running MariaDB 10.3.31 database server                         
mlmmjadmin.service       loaded active running RESTful API server used to manage mlmmj mailing list man
netdata.service          loaded active running Real time performance monitoring                        
nginx.service            loaded active running A high performance web server and a reverse proxy server
ntp.service              loaded active running Network Time Service                                    
php7.3-fpm.service       loaded active running The PHP 7.3 FastCGI Process Manager                     
postfix@-.service        loaded active running Postfix Mail Transport Agent (instance -)               
qemu-guest-agent.service loaded active running QEMU Guest Agent                                        
rsyslog.service          loaded active running System Logging Service                                  
ssh.service              loaded active running OpenBSD Secure Shell server                             
systemd-journald.service loaded active running Journal Service                                         
systemd-logind.service   loaded active running Login Service                                           
systemd-udevd.service    loaded active running udev Kernel Device Manager                              
user@0.service           loaded active running User Manager for UID 0                                  

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

24 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

I stopped services not needed during upgrade process and only keeped : 

# systemctl list-units --type=service --state=running
  UNIT                     LOAD   ACTIVE SUB     DESCRIPTION
  dbus.service             loaded active running D-Bus System Message Bus
  getty@tty1.service       loaded active running Getty on tty1
  ntp.service              loaded active running Network Time Service
  qemu-guest-agent.service loaded active running QEMU Guest Agent
  rsyslog.service          loaded active running System Logging Service
  ssh.service              loaded active running OpenBSD Secure Shell server
  systemd-journald.service loaded active running Journal Service
  systemd-logind.service   loaded active running User Login Management
  systemd-udevd.service    loaded active running Rule-based Manager for Device Events and Files
  user@0.service           loaded active running User Manager for UID 0

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
10 loaded units listed.

I upgraded to debian bullseye, followed the doc : https://docs.iredmail.org/upgrade.debian.10-11.html

I rebooted the server and then... same errors.

I feel like there is something wrong in debian mariadb upgrade process as i also had some mariadb upgrade problems on other servers on this release. So I feel like the only way is :

  • dump the databases

  • upgrade

  • (maybe uninstall/reinstall mariadb)

  • restore the databases

See on next episode...

3 Reply by maz (edited by maz 2021-10-26 18:54:12)

  • maz
  • Member
  • Offline
  • Registered: 2021-09-13
  • Posts: 11

Re: debian upgrade 10 to 11 : mariadb issue [not solved - end of story]

I feel like my problem is solved. The only thing I changed (following a friend advise) is the upgrade steps : do not do  full-upgrade, do it in two times. If someone experienced the same issue, here is how I proceeded.

Make a snapshot of your system, in case something goes wrong (this saved me 3 times before...).

Prepare the system :

apt update
apt dist-upgrade
for i in amavis.service clamav-daemon.service clamav-freshclam.service cron.service dovecot.service fail2ban.service iredadmin.service iredapd.service mlmmjadmin.service netdata.service nginx.service php7.3-fpm.service postfix@-.service ; do systemctl stop $i; done
nano /etc/apt/sources.list #set bullseye repositories

Save the mariadb databases, to be sure, and then stop mariadb :

mysqldump -u root --all-databases > ~/mariadb_dump_202110261048.sql
systemctl stop mariadb.service

Upgrade using the two steps, step 1 :

apt update
apt upgrade --without-new-pkgs #step 1: this does not install new packages, just upgrade the installed ones
for i in clamav-daemon.service clamav-freshclam.service cron.service fail2ban.service; do systemctl stop $i; done #kill services which have restarted during the process

At this point, before performing full upgrade, i did the dovecot part of https://docs.iredmail.org/upgrade.debian.10-11.html :

nano /etc/dovecot/dovecot.conf

Upgrade using the two steps, step 2 :

apt full-upgrade #make a pray to the databases god

Then I killed the restarted services to look into logs and verify if something went wrong :

for i in  amavis.service dovecot.service mariadb.service nginx.service php7.4-fpm.service postfix@-.service ; do systemctl stop $i; done
less /var/log/syslog

Then i verified the databases :

systemctl start mariadb.service
mysqldump -u root --all-databases > ~/mariadb_dump_after_upgrade.sql
diff ~/mariadb_dump_202110261048.sql ~/mariadb_dump_after_upgrade.sql

The diff only showed me versions and dates differences, so I was happy smile

I finished the "Fixes you need after upgrading Debian from 10 to 11" tutorial : https://docs.iredmail.org/upgrade.debian.10-11.html

I rebooted my system, checked the logs, checked emails, test sending and receiving. Everything seems to be OK by this time. I'm waiting a full week without any problem to be sure it's OK, but i feel it's following the good direction.

4 Reply by maz (edited by maz 2021-10-28 19:53:03)

  • maz
  • Member
  • Offline
  • Registered: 2021-09-13
  • Posts: 11

Re: debian upgrade 10 to 11 : mariadb issue [not solved - end of story]

I thought the problem was solved, but since this afternoon, I cannot receive nor send email... ang syslog says :

Oct 28 13:39:13 mail postfix/postscreen[83315]: CONNECT from [209.85.221.41]:40916 to [192.168.132.104]:25
Oct 28 13:39:19 mail postfix/postscreen[83315]: PASS NEW [209.85.221.41]:40916
Oct 28 13:39:19 mail postfix/smtpd[83318]: connect from mail-wr1-f41.google.com[209.85.221.41]
Oct 28 13:39:19 mail postfix/smtpd[83318]: Anonymous TLS connection established from mail-wr1-f41.google.com[209.85.221.41]: TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256
Oct 28 13:39:19 mail postfix/smtpd[83318]: 4Hg3Vv4KByzyY7: client=mail-wr1-f41.google.com[209.85.221.41]
Oct 28 13:39:19 mail postfix/cleanup[83323]: 4Hg3Vv4KByzyY7: message-id=<52216655-b507-3eb6-0d3b-d3fb39a572e0@gmail.com>
Oct 28 13:39:19 mail postfix/cleanup[83323]: 4Hg3Vv4KByzyY7: prepend: header Content-Language: fr from mail-wr1-f41.google.com[209.85.221.41]; from=<XXXXXX@gmail.com> to=<XXXX@XXXXX.XXX> proto=ESMTP helo=<mail-wr1-f41.google.com>: List-Unsubscribe: <unsubscribe@XXXXX.XXX>
Oct 28 13:39:19 mail postfix/qmgr[83244]: 4Hg3Vv4KByzyY7: from=<XXXXXX@gmail.com>, size=2856, nrcpt=1 (queue active)
Oct 28 13:39:19 mail postfix/amavis/smtp[83324]: connect to 127.0.0.1[127.0.0.1]:10024: Connection refused
Oct 28 13:39:19 mail postfix/smtpd[83318]: disconnect from mail-wr1-f41.google.com[209.85.221.41] ehlo=2 starttls=1 mail=1 rcpt=1 bdat=1 quit=1 commands=7
Oct 28 13:39:19 mail postfix/amavis/smtp[83324]: 4Hg3Vv4KByzyY7: to=<XXXX@XXXXX.XXX>, relay=none, delay=0.18, delays=0.17/0.01/0/0, dsn=4.4.1, status=deferred (connect to 127.0.0.1[127.0.0.1]:10024: Connection refused)
Oct 28 13:39:37 mail mariadbd[81679]: 2021-10-28 13:39:37 27 [Warning] Aborted connection 27 to db: 'vmail' user: 'vmailadmin' host: 'localhost' (Got an error reading communication packets)
Oct 28 13:40:01 mail CRON[83333]: (root) CMD (/bin/bash /usr/local/bin/fail2ban_banned_db unban_db)
Oct 28 13:40:17 mail mariadbd[81679]: 2021-10-28 13:40:17 46 [Warning] Aborted connection 46 to db: 'vmail' user: 'vmailadmin' host: 'localhost' (Got an error reading communication packets)
Oct 28 13:41:01 mail CRON[83360]: (root) CMD (/bin/bash /usr/local/bin/fail2ban_banned_db unban_db)

I am a little bit tired with this mariadb problem... Please help...

5 Reply by maz

  • maz
  • Member
  • Offline
  • Registered: 2021-09-13
  • Posts: 11

Re: debian upgrade 10 to 11 : mariadb issue [not solved - end of story]

End of story :
I didn't get any answer, didn't find any solution, so i installed a new fresh server and migrated domains, users and emails on the new one.

And all this time is the price not to be a part of the GAFAM world.

I may have understood samething that may explain why this happened to me, but now it's not important anymore.

6 Reply by camel1cz

  • camel1cz
  • camel1cz
  • Member
  • Offline
  • From: Czech Republic
  • Registered: 2013-04-20
  • Posts: 220

Re: debian upgrade 10 to 11 : mariadb issue [not solved - end of story]

Sad story...

Your issues seems connected to amavis being down... more info:

grep amavis /var/log/syslog

7 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: debian upgrade 10 to 11 : mariadb issue [not solved - end of story]

The db issue can be fixed by adding or updating parameter below in MariaDB config file, under section "[mysqld]"

log_warnings = 1

Restarting mariadb service is required.