1 Topic by hukaris

  • hukaris
  • Member
  • Offline
  • Registered: 2019-02-09
  • Posts: 18

Topic: Please help my server crash after few days!!

==== REQUIRED BASIC INFO OF YOUR IREDMAIL SERVER ====
- iRedMail version (check /etc/iredmail-release): 0.9.9 MYSQL edition
- Deployed with iRedMail Easy or the downloadable installer? downloader
- Linux/BSD distribution name and version: Freebsd 12.00
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): MYSQL
- Web server (Apache or Nginx): Ngix
- Manage mail accounts with iRedAdmin-Pro? Yes
- [IMPORTANT] Related original log or error message is required if you're experiencing an issue.
====

Hi All,

Please help me with this issue. My fresh installed server crash after running fewdays. I can't login into /iredadmin and gives me internal server error.

Here are the log outputs

root@nws:/var/log # tail maillog
Feb 14 13:19:43 nws postfix/pickup[49673]: warning: 440Kpl0gpXz2kbG: message has been queued for 1 days
Feb 14 13:19:43 nws postfix/pickup[49673]: 440Kpl0gpXz2kbG: uid=0 from=<root>
Feb 14 13:19:43 nws postfix/cleanup[49922]: warning: proxy:mysql:/usr/local/etc/postfix/mysql/sender_bcc_maps_user.cf lookup error for "root@nws

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: Please help my server crash after few days!!

Is mysql service running?

3 Reply by hukaris

  • hukaris
  • Member
  • Offline
  • Registered: 2019-02-09
  • Posts: 18

Re: Please help my server crash after few days!!

Yes

root@nws:/var/log # sockstat | grep 3306
netdata  python2.7  2598  8  tcp4   127.0.0.1:41582       127.0.0.1:3306
vscan    perl       2286  13 tcp4   127.0.0.1:41513       127.0.0.1:3306
vscan    perl       2284  13 tcp4   127.0.0.1:41512       127.0.0.1:3306
vscan    perl       2283  13 tcp4   127.0.0.1:41507       127.0.0.1:3306
vscan    perl       2282  13 tcp4   127.0.0.1:41509       127.0.0.1:3306
vscan    perl       2281  13 tcp4   127.0.0.1:41506       127.0.0.1:3306
mysql    mysqld     2265  29 tcp4   127.0.0.1:3306        *:*
mysql    mysqld     2265  104 tcp4  127.0.0.1:3306        127.0.0.1:41582
?        ?          ?     ?  tcp4   127.0.0.1:3306        127.0.0.1:20893
?        ?          ?     ?  tcp4   127.0.0.1:27215       127.0.0.1:3306
?        ?          ?     ?  tcp4   127.0.0.1:3306        127.0.0.1:27215



/vag/log/messages
Feb 14 14:07:49 nws php-fpm[2431]: [NOTICE] [pool inet] child 50732 exited with code 0 after 14.997855 seconds from start
Feb 14 14:07:49 nws php-fpm[2431]: [NOTICE] [pool inet] child 50734 started
Feb 14 14:08:04 nws php-fpm[2431]: [NOTICE] [pool inet] child 50734 exited with code 0 after 15.001611 seconds from start
Feb 14 14:08:04 nws php-fpm[2431]: [NOTICE] [pool inet] child 50740 started
Feb 14 14:08:19 nws php-fpm[2431]: [NOTICE] [pool inet] child 50740 exited with code 0 after 14.987264 seconds from start
Feb 14 14:08:19 nws php-fpm[2431]: [NOTICE] [pool inet] child 50742 started
Feb 14 14:08:34 nws php-fpm[2431]: [NOTICE] [pool inet] child 50742 exited with code 0 after 15.002843 seconds from start
Feb 14 14:08:34 nws php-fpm[2431]: [NOTICE] [pool inet] child 50745 started
Feb 14 14:08:49 nws php-fpm[2431]: [NOTICE] [pool inet] child 50745 exited with code 0 after 15.000507 seconds from start
Feb 14 14:08:49 nws php-fpm[2431]: [NOTICE] [pool inet] child 50747 started
Feb 14 14:09:04 nws php-fpm[2431]: [NOTICE] [pool inet] child 50747 exited with code 0 after 15.004261 seconds from start
Feb 14 14:09:04 nws php-fpm[2431]: [NOTICE] [pool inet] child 50754 started
Feb 14 14:09:10 nws kernel: sonewconn: pcb 0xfffff80018310400: Listen queue overflow: 151 already in queue awaiting acceptance (1 occurrences)
Feb 14 14:09:19 nws php-fpm[2431]: [NOTICE] [pool inet] child 50754 exited with code 0 after 14.989878 seconds from start
Feb 14 14:09:19 nws php-fpm[2431]: [NOTICE] [pool inet] child 50757 started
Feb 14 14:09:34 nws php-fpm[2431]: [NOTICE] [pool inet] child 50757 exited with code 0 after 15.005448 seconds from start
Feb 14 14:09:34 nws php-fpm[2431]: [NOTICE] [pool inet] child 50760 started
Feb 14 14:09:49 nws php-fpm[2431]: [NOTICE] [pool inet] child 50760 exited with code 0 after 14.995174 seconds from start
Feb 14 14:09:49 nws php-fpm[2431]: [NOTICE] [pool inet] child 50762 started
Feb 14 14:10:04 nws php-fpm[2431]: [NOTICE] [pool inet] child 50762 exited with code 0 after 15.001520 seconds from start
Feb 14 14:10:04 nws php-fpm[2431]: [NOTICE] [pool inet] child 50774 started
Feb 14 14:10:19 nws php-fpm[2431]: [NOTICE] [pool inet] child 50774 exited with code 0 after 14.996178 seconds from start
Feb 14 14:10:19 nws php-fpm[2431]: [NOTICE] [pool inet] child 50776 started
Feb 14 14:10:34 nws php-fpm[2431]: [NOTICE] [pool inet] child 50776 exited with code 0 after 15.008500 seconds from start
Feb 14 14:10:34 nws php-fpm[2431]: [NOTICE] [pool inet] child 50780 started
Feb 14 14:10:49 nws php-fpm[2431]: [NOTICE] [pool inet] child 50780 exited with code 0 after 14.988851 seconds from start

4 Reply by hukaris

  • hukaris
  • Member
  • Offline
  • Registered: 2019-02-09
  • Posts: 18

Re: Please help my server crash after few days!!

When I tried to login to the Sogo when iredadmin crash with internal server errors. I get following errors when trying to login.

Feb 13 22:40:01 sogod [1366]: [ERROR] <0x0x805b94ec8[SOGoSQLUserProfile]> failed to acquire channel for URL: mysql://sogo:a5b20727cbaafdc1bf8a8e6d24bbeee4@127.0.0.1:3306/sogo/sogo_user_profile
Feb 13 22:40:01 sogod [1366]: [ERROR] <0x0x805b94ec8[SOGoSQLUserProfile]> failed to acquire channel for URL: mysql://sogo:a5b20727cbaafdc1bf8a8e6d24bbeee4@127.0.0.1:3306/sogo/sogo_user_profile
Feb 13 22:40:01 sogod [1366]: [ERROR] <0x0x805b94ec8[SOGoSQLUserProfile]> failed to acquire channel for URL: mysql://sogo:a5b20727cbaafdc1bf8a8e6d24bbeee4@127.0.0.1:3306/sogo/sogo_user_profile
Feb 13 22:40:01 sogod [1366]: 192.168.3.25 "GET /SOGo/ HTTP/1.0" 200 7340/0 0.039 27100 72% -
2019-02-13 22:40:18.568 sogod[1366:100464] ERROR: could not open MySQL4 connection to database 'sogo': SSL connection error: error:00000001:lib(0):func(0):reason(1)
Feb 13 22:40:18 sogod [1366]: [ERROR] <0x0x805fdef48[GCSChannelManager]> could not open channel <MySQL4Channel[0x0x806ba8348] connection=0x(null)> for mysql://127.0.0.1/sogo/users
Feb 13 22:40:18 sogod [1366]: [WARN] <0x0x805fdef48[GCSChannelManager]>   will prevent opening of this channel 5 seconds after 2019-02-13 22:40:18 -0500
Feb 13 22:40:18 sogod [1366]: [ERROR] <0x0x806b04268[SQLSource]> failed to acquire channel for URL: mysql://sogo:a5b20727cbaafdc1bf8a8e6d24bbeee4@127.0.0.1:3306/sogo/users
Feb 13 22:40:18 sogod [1366]: SOGoRootPage Login from '192.168.3.25' for user 'pxxx@ing.net' might not have worked - password policy: 65535  grace: -1

5 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: Please help my server crash after few days!!

hukaris wrote:

- Linux/BSD distribution name and version: Freebsd 12.00

iRedMail has one issue on FreeBSD 12.0 (but ok on 11.x): SOGo has problem to connect to MySQL server periodly, have to restart mysql service to fix it. And the error log in SOGo reflects this issue, and i believe your another issue is impacted by same cause: https://forum.iredmail.org/post67570.html#p67570

please try to restart mysql service and check again.

6 Reply by hukaris

  • hukaris
  • Member
  • Offline
  • Registered: 2019-02-09
  • Posts: 18

Re: Please help my server crash after few days!!

Yes it fixed the issue. How can we fix this issue as I need this to be in a production server. That's why i brought the paid version.

7 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: Please help my server crash after few days!!

Unfortunately we didn't know why MySQL/MariaDB has such issue on FreeBSD 12.0, no solution yet. sad
Another solution is running CentOS/Debian/Ubuntu/OpenBSD instead.

8 Reply by hukaris

  • hukaris
  • Member
  • Offline
  • Registered: 2019-02-09
  • Posts: 18

Re: Please help my server crash after few days!!

Can you please look into this issue ASAP. I can't change the operating system to another.

9 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: Please help my server crash after few days!!

I will take a look at this issue today, but cannot guarantee to offer a fix.
I tried before, but didn't figure it out.

10 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: Please help my server crash after few days!!

Could you try this tutorial?
https://www.percona.com/blog/2016/05/16 … et-errors/

11 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: Please help my server crash after few days!!

Please check output of MySQL commands below:

SHOW GLOBAL STATUS like '%Abort%';
show global variables like '%timeout%';

I think there's some issue with the SOGo SQL connector. other applications (Postfix/Dovecot/...) and SQL users don't have issue like this.

12 Reply by hukaris

  • hukaris
  • Member
  • Offline
  • Registered: 2019-02-09
  • Posts: 18

Re: Please help my server crash after few days!!

Here are the outputs

+------------------+-------+
| Variable_name    | Value |
+------------------+-------+
| Aborted_clients  | 1456  |
| Aborted_connects | 0     |
+------------------+-------+

root@localhost [(none)]> show global variables like '%timeout%';
+-----------------------------+----------+
| Variable_name               | Value    |
+-----------------------------+----------+
| connect_timeout             | 10       |
| delayed_insert_timeout      | 300      |
| have_statement_timeout      | YES      |
| innodb_flush_log_at_timeout | 1        |
| innodb_lock_wait_timeout    | 50       |
| innodb_rollback_on_timeout  | OFF      |
| interactive_timeout         | 28800    |
| lock_wait_timeout           | 31536000 |
| net_read_timeout            | 30       |
| net_write_timeout           | 60       |
| rpl_stop_slave_timeout      | 31536000 |
| slave_net_timeout           | 60       |
| wait_timeout                | 28800    |
+-----------------------------+----------+

13 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: Please help my server crash after few days!!

hukaris wrote:

| Aborted_clients  | 1456  |

Please take a look at this blog post:
https://www.percona.com/blog/2016/05/16 … et-errors/

14 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: Please help my server crash after few days!!

By the way, i solved this issue on one FreeBSD 12 server by enabling "skip-name-resolve" in MySQL, but it didn't work for other FreeBSD 12 servers. You can give it a try.

15 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: Please help my server crash after few days!!

What's the value of MySQL server setting "max_allowed_packet" on your server? Try to increase it to 64M.

16 Reply by hukaris

  • hukaris
  • Member
  • Offline
  • Registered: 2019-02-09
  • Posts: 18

Re: Please help my server crash after few days!!

Where should i enable this "skip-name-resolve" in MySQL ?
This is already setup in
max_allowed_packet              = 64M

17 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: Please help my server crash after few days!!

I figured out it's related to the cron jobs ran by sogo daemon user, if you disable them, no more mysql errors.
Try to disable it temporarily with command "crontab -e -u sogod" (run as root user).

18 Reply by hukaris

  • hukaris
  • Member
  • Offline
  • Registered: 2019-02-09
  • Posts: 18

Re: Please help my server crash after few days!!

*   *   *   *   *   /usr/local/GNUstep/Local/Tools/Admin/sogo-tool expire-sessions 30 >/dev/null 2>&1; /usr/local/GNUstep/Local/Tools/Admin/sogo-ealarms-notif
y >/dev/null 2>&1

# SOGo vacation messages expiration
# The credentials file should contain the sieve admin credentials (username:passwd)
0   0   *   *   *   /usr/local/GNUstep/Local/Tools/Admin/sogo-tool update-autoreply -p /usr/local/etc/sogo/sieve.cred >/dev/null 2>&1


Can't seem to find the variable you mentioned on the files related cronjob

19 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: Please help my server crash after few days!!

Just comment out these 2 jobs (temporarily), and check MySQL log file - you will see no more errors at all.
So there must be a bug in "sogo-tool" command (ran by cron), i will report this to SOGo team soon, but since FreeBSD offers old sogo release (v4.0.4, latest is 4.0.5), i'm not sure whether the first answer i got will be "try the latest version first" big_smile

20 Reply by hukaris

  • hukaris
  • Member
  • Offline
  • Registered: 2019-02-09
  • Posts: 18

Re: Please help my server crash after few days!!

here is the version sogo4-4.0.4_3                  Groupware server with a focus on scalability and open standards

21 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: Please help my server crash after few days!!

I reported this issue to SOGo team, you can monitor the issue here:
https://sogo.nu/bugs/view.php?id=4684

Help push SOGo team to fix it.

22 Reply by hukaris

  • hukaris
  • Member
  • Offline
  • Registered: 2019-02-09
  • Posts: 18

Re: Please help my server crash after few days!!

Thanks, any update on the Freebsd fix with the sql?

23 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: Please help my server crash after few days!!

hukaris wrote:

any update on the Freebsd fix with the sql?

What do you mean "the sql"?

sogo-tool triggers mysql error, when it reaches max mysql error count, mysql will become unreachable. I suppose you mean this issue?

24 Reply by hukaris

  • hukaris
  • Member
  • Offline
  • Registered: 2019-02-09
  • Posts: 18

Re: Please help my server crash after few days!!

I mean the issue with Iredadminpro stops working with  Freebsd.

25 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: Please help my server crash after few days!!

hukaris wrote:

I mean the issue with Iredadminpro stops working with  Freebsd.

Does restart "iredadmin" service fix it? If not, any relevant error in log files under /var/log/? (majorly /var/log/messages)

If cannot get "iredadmin" service running, possible to give me direct ssh access with root privilege to debug it?
Contact me: zhb _at_ iredmail.org