1 (edited by broth 2020-07-27 04:26:26)

Topic: iRedMail migration to new server - SQL data only

==== REQUIRED BASIC INFO OF YOUR IREDMAIL SERVER ====
- iRedMail version (check /etc/iredmail-release): 1.3.1
- Deployed with iRedMail Easy or the downloadable installer? installer
- Linux/BSD distribution name and version: Debian 10
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): MariaDB
- Web server (Apache or Nginx): nginx - what else smile
- Manage mail accounts with iRedAdmin-Pro? yes
- [IMPORTANT] Related original log or error message is required if you're experiencing an issue.
====

Dear all,

I'm in the process of migrating an existing updated iRedMail 1.3.1 installation on Ubuntu 16.04 Server to a new deployment on Debian 10.

On the old server we use MySQL, the new server has MariaDB installed.

Now I would like to transfer the SQL data from A (old) to B (new).

A simple "mysqldump > dump.sql" and "mysql < dump.sql" is undesired as it takes over the complete old structure as well.

I tested the procedure and all works well BUT

The table structure, indizes and default values do differ substancially.

I would like to keep the proper new table layout from the fresh installation but insert all the data from the old server.

So far so good.

Backup data only on machine A:

mysqldump --default-character-set=utf8mb4 --no-create-info --skip-triggers --no-create-db --compact --databases amavisd fail2ban iredadmin iredapd roundcubemail vmail  >/tmp/dump.sql

Restore in two steps on B, first

mysql < clean.sql

clean.sql contains only "truncate" statements to empty all tables. This keeps the nice new layout for us.

then after copying the dump.sql vis SCP to B

mysql < /tmp/dump.sql

This should restore all data.

Basically this should work but for some reason I get an error:

ERROR 1366 (22007) at line 16: Incorrect string value: '\xF0\x9F\xA4\xAB N...' for column `amavisd`.`msgs`.`from_addr` at row 101

Looks like some emoji or other unicode characters were in an address.

This is where I'd like to seek help.

Why can I restore the complete database dump with create statements but it fails with only the data?
The charset setting on both tables looks like to be the same.

@iRedMail:

It would be great to see https://docs.iredmail.org/migrate.to.ne … erver.html updated with some notes about the SQL data migration.
IMHO restoring the old database with the old table structure is bad and might cause issues with e.g. predefined and expected defaults when missing.

Thanks!

Best regards,
Bernhard

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2

Re: iRedMail migration to new server - SQL data only

Update: I am about to write a script which fixes some issues for me.

Just a remark:

It's makes very much sense not to copy the database and structures but copy only data.
I had many old fields in the tables which are not used anymore.
Why accumulate and save stale data?

I will post the script once finished.

3

Re: iRedMail migration to new server - SQL data only

Here is what I use currently.
Please note: Script has to be adjusted to your needs.

USE AT YOUR OWN RISK - YOU NEED TO KNOW WHAT YOU ARE DOING!

#!/bin/bash

# iRedMail database migration script
# 07/2020 by Bernhard Roth sparc128(at)gmail(dot)com

MY_HOST=xxxx
MY_USER=xxxx
MY_PASS=xxxx


# clean mysql table from all data, keep only structure
mysql --init-command="SET SESSION FOREIGN_KEY_CHECKS=0;" < clean.sql

# databases to copy
DATABASES="amavisd fail2ban iredadmin iredapd roundcubemail vmail"
#DATABASES="amavisd"

# create temp file for SQL dump
TFILE=$(mktemp)

#echo $TFILE

# perform some fixes to allow import from older database structure
# the previous installation had some stale columns which are not used anymore
# in order to properly import the data, those columns need to be created.
# they will be deleted afterwards
echo "Preparing..."

# please adjust to your requirements!
echo "ALTER TABLE domain ADD COLUMN defaultpasswordscheme VARCHAR(10) NOT NULL DEFAULT '';" | mysql vmail
echo "ALTER TABLE mailbox ADD COLUMN bytes BIGINT(20) NOT NULL DEFAULT '0';" | mysql vmail
echo "ALTER TABLE mailbox ADD COLUMN messages BIGINT(20) NOT NULL DEFAULT '0';" | mysql vmail
echo "ALTER TABLE msgs CHANGE from_addr from_addr VARBINARY(255);" | mysql amavisd

# go though the list of databases
for db in $DATABASES; do

    echo -n "Processing $db: "
    echo -n "Exporting..."
    mysqldump -h $MY_HOST --user=$MY_USER --password=$MY_PASS --default-character-set=utf8mb4 --complete-insert --no-create-info --skip-triggers --no-create-db --compact $db > $TFILE
    echo -n "Importing..."
    mysql --default-character-set=utf8mb4 --init-command="SET SESSION FOREIGN_KEY_CHECKS=0;" $db < $TFILE
    echo "done"
done


echo "Fixing..."

# now remove temporary columns, adjust to your requirements
echo "ALTER TABLE domain DROP COLUMN defaultpasswordscheme;" | mysql vmail
echo "ALTER TABLE mailbox DROP COLUMN bytes;" | mysql vmail
echo "ALTER TABLE mailbox DROP COLUMN messages;" | mysql vmail

# this is due to an issue in iRedMail <= 1.3.1 where fields in the database are not utf8mb4
# required to properly save emojis etc. in the database while allowing to actually see the contents of the field
echo "ALTER TABLE msgs CHANGE from_addr from_addr VARCHAR(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT '';" | mysql amavisd


#delete temp file
rm $TFILE

echo "done"

clean.sql:

USE `amavisd`;
TRUNCATE TABLE  `maddr`;
TRUNCATE TABLE  `mailaddr`;
TRUNCATE TABLE  `msgrcpt`;
TRUNCATE TABLE  `msgs`;
TRUNCATE TABLE  `outbound_wblist`;
TRUNCATE TABLE  `policy`;
TRUNCATE TABLE  `quarantine`;
TRUNCATE TABLE  `users`;
TRUNCATE TABLE  `wblist`;
USE `fail2ban`;
TRUNCATE TABLE  `banned`;
USE `iredadmin`;
TRUNCATE TABLE  `deleted_mailboxes`;
TRUNCATE TABLE  `domain_ownership`;
TRUNCATE TABLE  `log`;
TRUNCATE TABLE  `newsletter_subunsub_confirms`;
TRUNCATE TABLE  `sessions`;
TRUNCATE TABLE  `settings`;
TRUNCATE TABLE  `tracking`;
TRUNCATE TABLE  `updatelog`;
USE `iredapd`;
TRUNCATE TABLE  `greylisting`;
TRUNCATE TABLE  `greylisting_tracking`;
TRUNCATE TABLE  `greylisting_whitelist_domain_spf`;
TRUNCATE TABLE  `greylisting_whitelist_domains`;
TRUNCATE TABLE  `greylisting_whitelists`;
TRUNCATE TABLE  `senderscore_cache`;
TRUNCATE TABLE  `smtp_sessions`;
TRUNCATE TABLE  `srs_exclude_domains`;
TRUNCATE TABLE  `throttle`;
TRUNCATE TABLE  `throttle_tracking`;
TRUNCATE TABLE  `wblist_rdns`;
USE `roundcubemail`;
TRUNCATE TABLE  `cache`;
TRUNCATE TABLE  `cache_index`;
TRUNCATE TABLE  `cache_messages`;
TRUNCATE TABLE  `cache_shared`;
TRUNCATE TABLE  `cache_thread`;
TRUNCATE TABLE  `contactgroupmembers`;
TRUNCATE TABLE  `contactgroups`;
TRUNCATE TABLE  `contacts`;
TRUNCATE TABLE  `dictionary`;
TRUNCATE TABLE  `filestore`;
TRUNCATE TABLE  `identities`;
TRUNCATE TABLE  `searches`;
TRUNCATE TABLE  `session`;
TRUNCATE TABLE  `system`;
TRUNCATE TABLE  `users`;
USE `vmail`;
TRUNCATE TABLE  `admin`;
TRUNCATE TABLE  `alias`;
TRUNCATE TABLE  `alias_domain`;
TRUNCATE TABLE  `anyone_shares`;
TRUNCATE TABLE  `deleted_mailboxes`;
TRUNCATE TABLE  `domain`;
TRUNCATE TABLE  `domain_admins`;
TRUNCATE TABLE  `forwardings`;
TRUNCATE TABLE  `last_login`;
TRUNCATE TABLE  `mailbox`;
TRUNCATE TABLE  `maillists`;
TRUNCATE TABLE  `moderators`;
TRUNCATE TABLE  `recipient_bcc_domain`;
TRUNCATE TABLE  `recipient_bcc_user`;
TRUNCATE TABLE  `sender_bcc_domain`;
TRUNCATE TABLE  `sender_bcc_user`;
TRUNCATE TABLE  `sender_relayhost`;
TRUNCATE TABLE  `share_folder`;
TRUNCATE TABLE  `used_quota`;