1 Topic by awol123

  • awol123
  • Member
  • Offline
  • Registered: 2019-03-21
  • Posts: 28

Topic: rsyslogd stopped after upgrading to iRedMail 1.3.2

==== REQUIRED BASIC INFO OF YOUR IREDMAIL SERVER ====
- iRedMail version (check /etc/iredmail-release): 1.3.2
- Deployed with iRedMail Easy or the downloadable installer? downloadable
- Linux/BSD distribution name and version: Debian 9.13
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): MySQL
- Web server (Apache or Nginx): Apache
- Manage mail accounts with iRedAdmin-Pro? No
- [IMPORTANT] Related original log or error message is required if you're experiencing an issue.
====

This server is dedicated to iRedMail.
On 19th December I upgraded to 1.3.2.

Today I went on to the server and noticed that all logging stopped on 20th December at 06:25. A restart of the server fixed the problem. The last entry in syslog was as below, apologies if this is something really simple but my Linux skills are fading the older I get! smile Any help appreciated, it all seems to be working fine now but will keep an eye on it.

Dec 20 06:25:10 iredmail clamd[704]: Sun Dec 20 06:25:10 2020 -> SelfCheck: Database status OK.
Dec 20 06:25:11 iredmail clamd[704]: Sun Dec 20 06:25:11 2020 -> SIGHUP caught: re-opening log file.
Dec 20 06:25:11 iredmail freshclam[616]: Sun Dec 20 06:25:11 2020 -> Received signal: re-opening log file
Dec 20 06:25:11 iredmail freshclam[616]: Sun Dec 20 06:25:11 2020 -> ClamAV update process started at Sun Dec 20 06:25:11 2020
Dec 20 06:25:11 iredmail freshclam[616]: Sun Dec 20 06:25:11 2020 -> daily.cld database is up to date (version: 26022, sigs: 4354149, f-level: 63, builder: raynman)
Dec 20 06:25:11 iredmail freshclam[616]: Sun Dec 20 06:25:11 2020 -> main.cld database is up to date (version: 59, sigs: 4564902, f-level: 60, builder: sigmgr)
Dec 20 06:25:11 iredmail freshclam[616]: Sun Dec 20 06:25:11 2020 -> bytecode.cld database is up to date (version: 331, sigs: 94, f-level: 63, builder: anvilleg)
Dec 20 06:25:11 iredmail systemd[1]: Stopping System Logging Service...
Dec 20 06:25:11 iredmail liblogging-stdlog:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="2976" x-info="http://www.rsyslog.com"] exiting on signal 15.
Dec 20 06:25:11 iredmail systemd[1]: Stopped System Logging Service.
Dec 20 06:25:11 iredmail systemd[1]: Starting System Logging Service...
Dec 20 06:25:11 iredmail liblogging-stdlog:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="10554" x-info="http://www.rsyslog.com"] start
Dec 20 06:25:11 iredmail systemd[1]: Started System Logging Service.
Dec 20 06:25:11 iredmail liblogging-stdlog:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="10554" x-info="http://www.rsyslog.com"] exiting on signal 15.
Dec 20 06:25:11 iredmail systemd[1]: Stopping System Logging Service...
Dec 20 06:25:11 iredmail systemd[1]: Stopped System Logging Service.
Dec 20 06:25:11 iredmail systemd[1]: Starting System Logging Service...
Dec 20 06:25:11 iredmail liblogging-stdlog:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="10566" x-info="http://www.rsyslog.com"] start
Dec 20 06:25:11 iredmail systemd[1]: Started System Logging Service.
Dec 20 06:25:11 iredmail liblogging-stdlog:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="10566" x-info="http://www.rsyslog.com"] exiting on signal 15.
Dec 20 06:25:11 iredmail systemd[1]: Stopping System Logging Service...
Dec 20 06:25:11 iredmail systemd[1]: Stopped System Logging Service.
Dec 20 06:25:11 iredmail systemd[1]: Starting System Logging Service...
Dec 20 06:25:11 iredmail liblogging-stdlog:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="10578" x-info="http://www.rsyslog.com"] start
Dec 20 06:25:11 iredmail systemd[1]: Started System Logging Service.
Dec 20 06:25:11 iredmail fail2ban.server[6299]: INFO flush performed on SYSLOG
Dec 20 06:25:11 iredmail liblogging-stdlog:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="10578" x-info="http://www.rsyslog.com"] exiting on signal 15.
Dec 20 06:25:11 iredmail systemd[1]: Stopping System Logging Service...
Dec 20 06:25:11 iredmail systemd[1]: Stopped System Logging Service.
Dec 20 06:25:11 iredmail systemd[1]: Starting System Logging Service...
Dec 20 06:25:11 iredmail liblogging-stdlog:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="10594" x-info="http://www.rsyslog.com"] start
Dec 20 06:25:11 iredmail systemd[1]: Started System Logging Service.
Dec 20 06:25:11 iredmail liblogging-stdlog:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="10594" x-info="http://www.rsyslog.com"] exiting on signal 15.
Dec 20 06:25:11 iredmail systemd[1]: Stopping System Logging Service...
Dec 20 06:25:11 iredmail systemd[1]: Stopped System Logging Service.
Dec 20 06:25:11 iredmail systemd[1]: Starting System Logging Service...
Dec 20 06:25:11 iredmail liblogging-stdlog:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="10605" x-info="http://www.rsyslog.com"] start
Dec 20 06:25:11 iredmail systemd[1]: Started System Logging Service.

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: rsyslogd stopped after upgrading to iRedMail 1.3.2

It's likely systemd-journald takes over the logging. A simple link should do the trick:

ln -sf /run/systemd/journal/dev-log /dev/log

3 Reply by awol123 (edited by awol123 2021-01-05 04:50:54)

  • awol123
  • Member
  • Offline
  • Registered: 2019-03-21
  • Posts: 28

Re: rsyslogd stopped after upgrading to iRedMail 1.3.2

ZhangHuangbin wrote:

It's likely systemd-journald takes over the logging. A simple link should do the trick:

ln -sf /run/systemd/journal/dev-log /dev/log

Thanks for the reply.
There is already a symlink in place for that

lrwxrwxrwx 1 root root          28 Dec 30 18:49 log -> /run/systemd/journal/dev-log

root@ired:/run/systemd/journal# ls -l
total 4
srw-rw-rw- 1 root root   0 Dec 30 18:49 dev-log

Sure enough, at 6:25am the day after I restarted the server and posted this, it stopped logging again. The last entries in syslog are below but the status output of rsyslog shows it has hit a restart limit:

● rsyslog.service - System Logging Service
   Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: failed (Result: start-limit-hit) since Thu 2020-12-31 06:25:10 GMT; 4 days ago
     Docs: man:rsyslogd(8)
           http://www.rsyslog.com/doc/
  Process: 8720 ExecStart=/usr/sbin/rsyslogd -n (code=exited, status=0/SUCCESS)
Main PID: 8720 (code=exited, status=0/SUCCESS)

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.


Running 'service rsyslog start' starts logging again, i'll see how long for! From the logs it looks like something is stopping and starting the rsyslog service multiple times in quick succession which kills it:


Dec 31 06:25:01 ired CRON[8145]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
Dec 31 06:25:01 ired CRON[8147]: (root) CMD (/bin/bash /usr/local/bin/fail2ban_banned_db unban_db)
Dec 31 06:25:07 ired postfix/pickup[30635]: 4D5ynH0WWvz3wbS: uid=0 from=<root>
Dec 31 06:25:07 ired postfix/cleanup[8599]: 4D5ynH0WWvz3wbS: message-id=<4D5ynH0WWvz3wbS@ired.mail.net>
Dec 31 06:25:07 ired postfix/qmgr[1759]: 4D5ynH0WWvz3wbS: from=<root@ired.mail.net>, size=50346, nrcpt=1 (queue active)
Dec 31 06:25:07 ired clamd[660]: Thu Dec 31 06:25:07 2020 -> SelfCheck: Database status OK.
Dec 31 06:25:07 ired liblogging-stdlog:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="622" x-info="http://www.rsyslog.com"] exiting on signal 15.
Dec 31 06:25:07 ired systemd[1]: Stopping System Logging Service...
Dec 31 06:25:07 ired systemd[1]: Stopped System Logging Service.
Dec 31 06:25:07 ired systemd[1]: Starting System Logging Service...
Dec 31 06:25:07 ired liblogging-stdlog:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="8675" x-info="http://www.rsyslog.com"] start
Dec 31 06:25:07 ired systemd[1]: Started System Logging Service.
Dec 31 06:25:08 ired liblogging-stdlog:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="8675" x-info="http://www.rsyslog.com"] exiting on signal 15.
Dec 31 06:25:08 ired systemd[1]: Stopping System Logging Service...
Dec 31 06:25:08 ired systemd[1]: Stopped System Logging Service.
Dec 31 06:25:08 ired systemd[1]: Starting System Logging Service...
Dec 31 06:25:08 ired liblogging-stdlog:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="8687" x-info="http://www.rsyslog.com"] start
Dec 31 06:25:08 ired systemd[1]: Started System Logging Service.
Dec 31 06:25:08 ired liblogging-stdlog:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="8687" x-info="http://www.rsyslog.com"] exiting on signal 15.
Dec 31 06:25:08 ired systemd[1]: Stopping System Logging Service...
Dec 31 06:25:08 ired systemd[1]: Stopped System Logging Service.
Dec 31 06:25:08 ired systemd[1]: Starting System Logging Service...
Dec 31 06:25:08 ired liblogging-stdlog:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="8698" x-info="http://www.rsyslog.com"] start
Dec 31 06:25:08 ired systemd[1]: Started System Logging Service.
Dec 31 06:25:08 ired liblogging-stdlog:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="8698" x-info="http://www.rsyslog.com"] exiting on signal 15.
Dec 31 06:25:08 ired systemd[1]: Stopping System Logging Service...
Dec 31 06:25:08 ired systemd[1]: Stopped System Logging Service.
Dec 31 06:25:08 ired systemd[1]: Starting System Logging Service...
Dec 31 06:25:08 ired liblogging-stdlog:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="8709" x-info="http://www.rsyslog.com"] start
Dec 31 06:25:08 ired systemd[1]: Started System Logging Service.
Dec 31 06:25:08 ired liblogging-stdlog:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="8709" x-info="http://www.rsyslog.com"] exiting on signal 15.
Dec 31 06:25:08 ired systemd[1]: Stopping System Logging Service...
Dec 31 06:25:08 ired systemd[1]: Stopped System Logging Service.
Dec 31 06:25:08 ired systemd[1]: Starting System Logging Service...
Dec 31 06:25:08 ired liblogging-stdlog:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="8720" x-info="http://www.rsyslog.com"] start
Dec 31 06:25:08 ired systemd[1]: Started System Logging Service.

4 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: rsyslogd stopped after upgrading to iRedMail 1.3.2

systemd may try to restart it if it (rsyslog) had some issue and aborted or failed to start.