We installed package "mcrypt" on other (production) server but it didn't work. It was our first option to try to solve this problem.
This forced us to set up two test servers (on one Debian 10, and on the other Debian 11) and to go through the same iRedMail installation process on both servers in order to solve this problem.
We solved the problem in another way.
Inside roundcubemail/plugins/password/config.inc.php we changed:
-----------------------------------------------------------------------------------------------------------------
/////$config['password_algorithm'] = 'clear';
$config['password_algorithm'] = 'ssha512';
/////$config['password_algorithm_prefix'] = '';
$config['password_algorithm_prefix'] = '{SSHA512}';
// The SQL query used to change the password.
// The query can contain the following macros that will be expanded as follows:
// %p is replaced with the plaintext new password
// %P is replaced with the crypted/hashed new password
// according to configured password_method
// %o is replaced with the old (current) password
// %O is replaced with the crypted/hashed old (current) password
// according to configured password_method
// %h is replaced with the imap host (from the session info)
// %u is replaced with the username (from the session info)
// %l is replaced with the local part of the username
// (in case the username is an email address)
// %d is replaced with the domain part of the username
// (in case the username is an email address)
// Deprecated macros:
// %c is replaced with the crypt version of the new password, MD5 if available
// otherwise DES. More hash function can be enabled using the password_crypt_hash
// configuration parameter.
// %D is replaced with the dovecotpw-crypted version of the new password
// %n is replaced with the hashed version of the new password
// %q is replaced with the hashed password before the change
// Escaping of macros is handled by this module.
// Default: "SELECT update_passwd(%c, %u)"
/////$config['password_query'] = "UPDATE mailbox SET password=%D,passwordlastchange=NOW() WHERE username=%u";
$config['password_query'] = "UPDATE mailbox SET password=%P,passwordlastchange=NOW() WHERE username=%u";
-----------------------------------------------------------------------------------------------------------------
We have seen that there are differences between the versions of Dovecot:
2.3.4.1 (f79e8e7e4) - Debian 10
2.3.13 (89f716dc2) - Debian 11
We assume that this could be the source of this problem. (Maybe you should investigate this in more detail.)