1 Topic by remington (edited by remington 2013-04-11 17:18:07)

  • remington
  • Member
  • Offline
  • Registered: 2013-03-10
  • Posts: 9

Topic: [SOLVED] FreeBSD 9.1 + ZFS + Jail + iRedMail = shutdown deadlocks

I have been struggling with this problem for quite some time.  I have iRedMail running perfectly in FreeBSD's Jail but I have trouble shutting or rebooting the server as it will hang after "All buffers synched".  This is not good if the server will be at the data center which will require someone to physically reset the server.

1) Create/Delete mail Jail without installing iRedMail.  Reboot Success
2) Installed iRedMail and start services in mail Jail.  Reboot Fail
3) Restart mail Jail.  Reboot Fail
4) Disabled or kill all jobs/services/daemons in mail Jail.  Reboot Fail
5) Delete mail Jail and reboot.  Reboot Fail
6) Reboot server after server restarted manually after number 5 above.  Reboot Success
7) Reboot/Shutdown server several times after mail Jail was removed/deleted.  Reboot Success

My zfs list:
NAME                               USED  AVAIL  REFER  MOUNTPOINT
tank                                31.8G   882G   443M  legacy
tank/ezjail                      4.54G   882G   501M  /jails
tank/ezjail/basejail         1.70G   882G  1.70G  /jails/basejail
tank/ezjail/mysql             228M   882G   228M  /jails/mysql
tank/ezjail/newjail          3.59M   882G  3.59M  /jails/newjail
tank/ezjail/web                876M   882G   876M  /jails/web
tank/ezjail/mail                276M   882G   276M  /jails/mail

I suspect that iRedMail installation changed some of the system configurations in mail Jail that prevented it from having normal shutdown/reboot procedure as it hangs after "All buffers synched".

I occasionally get this error... after I press the power button or ALT-CTRL-DEL keys after it hangs and it will shutdown after couple of minutes.
All buffers synched
writing entrophy file:/etc/rc.shutdown: WARNING: write failed (read-only fs?)
Terminated
init: some processes would not die; ps axl advised

It will not drop into Single Mode for me to do the "ps axl" command to find out what's causing this.

I looked in the /var/logs/messages to find what could be causing the problem but I do not see anything since shutdown procedure wasn't able to write to the log file.

Anyone else having this problem or is there a solution?

On the side note, I did the same procedure on my other computer in VirtualBox without ZFS.  It all works very well.  I have to use ZFS for two drives mirroring and to control the disk space for VPS users.

==== Required information ====
- iRedMail version: 0.8.3
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): LDAP/MySQL
- Linux/BSD distribution name and version: FreeBSD 9.1
- Related log if you're reporting an issue:
====

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,090

Re: [SOLVED] FreeBSD 9.1 + ZFS + Jail + iRedMail = shutdown deadlocks

Hi remington,

iRedMail install tutorial for FreeBSD + Jail is available here: http://iredmail.org/wiki/index.php?titl … eeBSD.Jail
You can verify each step and each configuration we set. I have no idea which one will cause this issue, sorry.

3 Reply by remington (edited by remington 2013-03-29 16:18:29)

  • remington
  • Member
  • Offline
  • Registered: 2013-03-10
  • Posts: 9

Re: [SOLVED] FreeBSD 9.1 + ZFS + Jail + iRedMail = shutdown deadlocks

ZhangHuangbin wrote:

Hi remington,

iRedMail install tutorial for FreeBSD + Jail is available here: http://iredmail.org/wiki/index.php?titl … eeBSD.Jail
You can verify each step and each configuration we set. I have no idea which one will cause this issue, sorry.

I followed the instructions exactly as provided.  I was able to replicate the problem in VirtualBox.  It hangs at reboot/shutdown.

I used this script to create ZFS filesystem at installation time.
http://iceblood.googlecode.com/svn/Free … fs_inst.sh

This is known problem with ZFS in FreeBSD as I have learned today.

I can force the server to shutdown remotely using this command but not recommended:
shutdown -n -o -r now

4 Reply by remington

  • remington
  • Member
  • Offline
  • Registered: 2013-03-10
  • Posts: 9

Re: [SOLVED] FreeBSD 9.1 + ZFS + Jail + iRedMail = shutdown deadlocks

I will try two things:

1) Install iRedMail in FreeBSD without Jail. I suspect this will hang at reboot/shutdown anyway.
2) Fix the ZFS installation script to create swap partition outside of ZFS.  If create swap inside ZFS and it is known to cause deadlock with vnodes during massive data read/write.  FreeBSD team is working to find a solution to this problem but this is a temporary solution.

I will report back with my results.

5 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,090

Re: [SOLVED] FreeBSD 9.1 + ZFS + Jail + iRedMail = shutdown deadlocks

remington wrote:

I was able to replicate the problem in VirtualBox

Is it possible that this is a VirtualBox related issue? Did you test it with either Xen/KVM/VMware/OpenVZ/etc?

6 Reply by remington

  • remington
  • Member
  • Offline
  • Registered: 2013-03-10
  • Posts: 9

Re: [SOLVED] FreeBSD 9.1 + ZFS + Jail + iRedMail = shutdown deadlocks

ZhangHuangbin wrote:
remington wrote:

I was able to replicate the problem in VirtualBox

Is it possible that this is a VirtualBox related issue? Did you test it with either Xen/KVM/VMware/OpenVZ/etc?

It's not VirtualBox related issue as I was able to reproduce the problem on my other computer.

Unfortunately, I tried both above, installation without Jail and swap partition outside of ZFS, and it hangs at shutdown.

I rolled back to FreeBSD 9.0 with ZFS and it was able to shutdown normally.  I would not recommend using FreeBSD 9.1 and ZFS with iRedMail until FreeBSD issue a ZFS fix.

7 Reply by remington (edited by remington 2013-04-09 00:03:33)

  • remington
  • Member
  • Offline
  • Registered: 2013-03-10
  • Posts: 9

Re: [SOLVED] FreeBSD 9.1 + ZFS + Jail + iRedMail = shutdown deadlocks

Okay.  I can confirm that iRedMail works normal with FreeBSD 9.0 -RELEASE AND 9.1 -STABLE branch.

FreeBSD 9.1 -RELEASE and -RELENG continue to have problems so I'm going to use the -STABLE branch instead.