1

Topic: internal server error 3.9 LDAP

==== REQUIRED BASIC INFO OF YOUR IREDMAIL SERVER ====
- iRedMail version (check /etc/iredmail-release): 0.9.9
- Deployed with iRedMail Easy or the downloadable installer? NO
- Linux/BSD distribution name and version: Centos 7.6
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): LDAP
- Web server (Apache or Nginx): Nginx
- Manage mail accounts with iRedAdmin-Pro? Yes
- [IMPORTANT] Related original log or error message is required if you're experiencing an issue.
====
Upgraded to 3.9 directly from 3.6 and now get only internal server error when accessing the admin.
Nothing in the logs that I can see, other than:

/var/log/nginx/access.log
GET /admin/ HTTP/1.1" 500 32 "https://server.domain.com/admin/dashboard" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0

no entries in /var/log/nginx/error_log other than a few roundcube errors

only entry in /var/log/messages when I restart service:
Aug 13 06:29:57 mail systemd[1]: Stopping iRedAdmin daemon service...
Aug 13 06:29:58 mail systemd[1]: Stopped iRedAdmin daemon service.
Aug 13 06:29:58 mail systemd[1]: Starting iRedAdmin daemon service...
Aug 13 06:29:58 mail systemd[1]: Started iRedAdmin daemon service.
Aug 13 06:29:58 mail uwsgi[6876]: [uWSGI] getting INI configuration from /var/www/iredadmin/rc_scripts/uwsgi/rhel.ini

If I revert to 3.6 everything works fine:
"GET /admin/dashboard HTTP/1.1" 200 4932 "https://server.domain.com/admin/dashboard" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2

Re: internal server error 3.9 LDAP

With "internal server error", i expect some error log in /var/log/message.
Could you try to reproduce this error again and check all log files under /var/log/?

3 (edited by jstewart 2019-08-13 19:10:22)

Re: internal server error 3.9 LDAP

ZhangHuangbin wrote:

With "internal server error", i expect some error log in /var/log/message.
Could you try to reproduce this error again and check all log files under /var/log/?

The only entries I see are the ones I posted. I checked every file in every directory in /var/log

All services restarted, as well as a server reboot after the upgrade. When I downgrade back to 3.6 restarting the iredadmin service brings it back

4 (edited by jstewart 2019-08-13 20:29:14)

Re: internal server error 3.9 LDAP

jstewart wrote:
ZhangHuangbin wrote:

With "internal server error", i expect some error log in /var/log/message.
Could you try to reproduce this error again and check all log files under /var/log/?

The only entries I see are the ones I posted. I checked every file in every directory in /var/log

All services restarted, as well as a server reboot after the upgrade. When I downgrade back to 3.6 restarting the iredadmin service brings it back

I went directly from 3.6 to 3.9. I'm going to try 3.7 and 3.8 to see if they work...

Upgrade to 3.7 works fine.
Upgrade to 3.8 gives internal server error, same as 3.9

I spoke too soon - 3.7 allows access to the interface, but internal server error when accessing a user.

Update - this may be my fault - I made some alterations to allow for new LDAP attributes, and I likely broke it myself. I'll update this message if that is the case.

It wasn't that - I redid the update to 3.7 and it is working fine, no more errors. 3.8 and 3.9 are not. I'm going to continue trying, and go through logs to see if I can figure it out - which log file would generally log the internal server error? I'm not seeing any errors related to this in any log files.

5

Re: internal server error 3.9 LDAP

jstewart wrote:

which log file would generally log the internal server error? I'm not seeing any errors related to this in any log files.

iRedAdmin-Pro-LDAP-3.9 logs to syslog by default, so if there's some "internal server error", you can check Nginx log file and default syslog log file (/var/log/messages, or /var/log/syslog).

Another solution is set "MAIL_ERROR_TO_WEBMASTER = True" in iRedAdmin-Pro config file, then restart "iredadmin" service. Each time an "internal server error" is triggered, webmaster (defined in parameter "webmaster =" in iRedAdmin-Pro config file) will receive an email with detailed info. WARNING: this email may contain sensitive info like SQL username/password, make sure it won't be delivered to some address you don't trust.

6 (edited by jstewart 2019-08-15 21:34:58)

Re: internal server error 3.9 LDAP

Thanks.
I set MAIL_ERROR_TO_WEBMASTER = True, and here are the results:

Error after upgrading to 3.8:
email subject line: bug: <class 'socket.error'>: [Errno 13] Permission denied (/)

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/web/application.py", line 239, in process
    return self.handle()
  File "/usr/lib/python2.7/site-packages/web/application.py", line 230, in handle
    return self._delegate(fn, self.fvars, args)
  File "/usr/lib/python2.7/site-packages/web/application.py", line 416, in _delegate
    mod = __import__(mod, None, None, [''])
  File "/var/www/iRedAdmin-Pro-LDAP-3.8/controllers/ldap/basic.py", line 9, in <module>
    from libs import iredutils, sysinfo, form_utils
  File "/var/www/iRedAdmin-Pro-LDAP-3.8/libs/sysinfo.py", line 13, in <module>
    from libs.logger import log_traceback
  File "/var/www/iRedAdmin-Pro-LDAP-3.8/libs/logger.py", line 30, in <module>
    _handler = SysLogHandler(address=_server, facility=settings.SYSLOG_FACILITY)
  File "/usr/lib64/python2.7/logging/handlers.py", line 761, in __init__
    self._connect_unixsocket(address)
  File "/usr/lib64/python2.7/logging/handlers.py", line 789, in _connect_unixsocket
    self.socket.connect(address)
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 13] Permission denied



Error after upgrading to 3.9:
email subject line: bug: <class 'socket.error'>: [Errno 13] Permission denied (/)

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/web/application.py", line 239, in process
    return self.handle()
  File "/usr/lib/python2.7/site-packages/web/application.py", line 230, in handle
    return self._delegate(fn, self.fvars, args)
  File "/usr/lib/python2.7/site-packages/web/application.py", line 416, in _delegate
    mod = __import__(mod, None, None, [''])
  File "/var/www/iRedAdmin-Pro-LDAP-3.9/controllers/ldap/basic.py", line 9, in <module>
    from libs import iredutils, sysinfo, form_utils
  File "/var/www/iRedAdmin-Pro-LDAP-3.9/libs/sysinfo.py", line 13, in <module>
    from libs.logger import log_traceback
  File "/var/www/iRedAdmin-Pro-LDAP-3.9/libs/logger.py", line 30, in <module>
    _handler = SysLogHandler(address=_server, facility=settings.SYSLOG_FACILITY)
  File "/usr/lib64/python2.7/logging/handlers.py", line 761, in __init__
    self._connect_unixsocket(address)
  File "/usr/lib64/python2.7/logging/handlers.py", line 789, in _connect_unixsocket
    self.socket.connect(address)
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 13] Permission denied

symlink for iredadmin set back to iRedAdmin-Pro-LDAP-3.7 and everything works properly

the only entries in the other logs are the same as before:
/var/log/messages:
Aug 15 05:26:37 mail systemd[1]: Stopping iRedAdmin daemon service...
Aug 15 05:26:38 mail systemd[1]: Stopped iRedAdmin daemon service.
Aug 15 05:26:38 mail systemd[1]: Starting iRedAdmin daemon service...
Aug 15 05:26:38 mail systemd[1]: Started iRedAdmin daemon service.
Aug 15 05:26:38 mail uwsgi[9797]: [uWSGI] getting INI configuration from /var/www/iredadmin/rc_scripts/uwsgi/rhel.ini

/var/log/nginx/access.log:
192.168.104.68 - - [15/Aug/2019:05:24:05 -0400] "GET /admin HTTP/1.1" 500 32 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0"
192.168.104.68 - - [15/Aug/2019:05:30:13 -0400] "GET /admin HTTP/1.1" 500 32 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0"


Does any of that make sense?

FYI, all updates have gone without a problem since the initial install of iRedAdmin-Pro-LDAP-3.1 up to 3.7. upgrades to 3.8 and 3.9 result in the internal server error.

7

Re: internal server error 3.9 LDAP

Is port 7791 used by other program?

8

Re: internal server error 3.9 LDAP

ZhangHuangbin wrote:

Is port 7791 used by other program?

No.
As I had indicated, I upgraded to 3.7 from 3.6 without any issues, but it's the 3.8 and 3.9 installs that are giving the error.

9

Re: internal server error 3.9 LDAP

- Could you please check whether port 7791 is used by iredadmin or other program with command below?

lsof -i :7791

If there's some program, please stop it. You can also use `kill -9 <pid>` command to kill its pid.

- Then restart "iredadmin" service and try to access it.

10 (edited by jstewart 2019-08-27 00:54:27)

Re: internal server error 3.9 LDAP

lsof -i :7791
COMMAND  PID      USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
uwsgi   2133 iredadmin    6u  IPv4  25910      0t0  TCP localhost:7791 (LISTEN)
uwsgi   2352 iredadmin    6u  IPv4  25910      0t0  TCP localhost:7791 (LISTEN)
uwsgi   2353 iredadmin    6u  IPv4  25910      0t0  TCP localhost:7791 (LISTEN)
uwsgi   2358 iredadmin    6u  IPv4  25910      0t0  TCP localhost:7791 (LISTEN)
uwsgi   2360 iredadmin    6u  IPv4  25910      0t0  TCP localhost:7791 (LISTEN)
uwsgi   2362 iredadmin    6u  IPv4  25910      0t0  TCP localhost:7791 (LISTEN)

3.7 working fine
then:
service iredadmin stop
- nothing listening on 7791:
change symlink for iredadmin to either 3.8 or 3.9 and stop then start iredadmin the internal server error is back. Rebooted server, same results.
Set symlink back to 3.7, everything is fine again.

GET https://server.example.net/admin/

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/web/application.py", line 239, in process
    return self.handle()
  File "/usr/lib/python2.7/site-packages/web/application.py", line 230, in handle
    return self._delegate(fn, self.fvars, args)
  File "/usr/lib/python2.7/site-packages/web/application.py", line 416, in _delegate
    mod = __import__(mod, None, None, [''])
  File "/var/www/iRedAdmin-Pro-LDAP-3.8/controllers/ldap/basic.py", line 9, in <module>
    from libs import iredutils, sysinfo, form_utils
  File "/var/www/iRedAdmin-Pro-LDAP-3.8/libs/sysinfo.py", line 13, in <module>
    from libs.logger import log_traceback
  File "/var/www/iRedAdmin-Pro-LDAP-3.8/libs/logger.py", line 30, in <module>
    _handler = SysLogHandler(address=_server, facility=settings.SYSLOG_FACILITY)
  File "/usr/lib64/python2.7/logging/handlers.py", line 761, in __init__
    self._connect_unixsocket(address)
  File "/usr/lib64/python2.7/logging/handlers.py", line 789, in _connect_unixsocket
    self.socket.connect(address)
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 13] Permission denied

11

Re: internal server error 3.9 LDAP

I figured it out: do you have /dev/log on this server?

iRedAdmin-Pro expects syslog creates socket /dev/log, but your server seems uses different path. Which one is it?

After found the correct syslog socket path, please add setting below with the socket path in /opt/iredapd/settings.py and restart iredapd service:

SYSLOG_SERVER = '/dev/log'

12 (edited by jstewart 2019-08-20 22:05:32)

Re: internal server error 3.9 LDAP

ZhangHuangbin wrote:

I figured it out: do you have /dev/log on this server?

iRedAdmin-Pro expects syslog creates socket /dev/log, but your server seems uses different path. Which one is it?

After found the correct syslog socket path, please add setting below with the socket path in /opt/iredapd/settings.py and restart iredapd service:

SYSLOG_SERVER = '/dev/log'

Same result.
added that line to settings.py in /var/www/iRedAdmin-Pro-LDAP-3.9
settings.py:SYSLOG_SERVER = '/dev/log'

I also tried: SYSLOG_SERVER = '/run/systemd/journal/syslog' as per the suggestion in libs/default_settings.py


ls -l /dev/log
srw-rw---- 1 root mysyslog 0 Aug 15 07:59 /dev/log


Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/web/application.py", line 239, in process
    return self.handle()
  File "/usr/lib/python2.7/site-packages/web/application.py", line 230, in handle
    return self._delegate(fn, self.fvars, args)
  File "/usr/lib/python2.7/site-packages/web/application.py", line 416, in _delegate
    mod = __import__(mod, None, None, [''])
  File "/var/www/iRedAdmin-Pro-LDAP-3.9/controllers/ldap/basic.py", line 9, in <module>
    from libs import iredutils, sysinfo, form_utils
  File "/var/www/iRedAdmin-Pro-LDAP-3.9/libs/sysinfo.py", line 13, in <module>
    from libs.logger import log_traceback
  File "/var/www/iRedAdmin-Pro-LDAP-3.9/libs/logger.py", line 30, in <module>
    _handler = SysLogHandler(address=_server, facility=settings.SYSLOG_FACILITY)
  File "/usr/lib64/python2.7/logging/handlers.py", line 761, in __init__
    self._connect_unixsocket(address)
  File "/usr/lib64/python2.7/logging/handlers.py", line 789, in _connect_unixsocket
    self.socket.connect(address)
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 13] Permission denied

13

Re: internal server error 3.9 LDAP

The syslog socket must be readable and writable by others, not just owner and group. Could you please try again?
Also, make sure other user can access parent directories of the socket.

14

Re: internal server error 3.9 LDAP

OK , I tried changing permission on the /dev/log socket, but it seems permissions can't be changed with chmod, so I re-created the socket (systemctl restart systemd-journald.socket) which gave rw permissions to all:
srw-rw-rw- 1 root root           0 Aug 27 07:48 log

set iredadmin to 3.9, got the internal server error, and the socket changed itself back to:
srw-rw---- 1 root mysyslog        0 Aug 27 07:51 log


I tried setting the /dev directory to 777 with the same result.

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/web/application.py", line 239, in process
    return self.handle()
  File "/usr/lib/python2.7/site-packages/web/application.py", line 230, in handle
    return self._delegate(fn, self.fvars, args)
  File "/usr/lib/python2.7/site-packages/web/application.py", line 416, in _delegate
    mod = __import__(mod, None, None, [''])
  File "/var/www/iRedAdmin-Pro-LDAP-3.9/controllers/ldap/basic.py", line 9, in <module>
    from libs import iredutils, sysinfo, form_utils
  File "/var/www/iRedAdmin-Pro-LDAP-3.9/libs/sysinfo.py", line 13, in <module>
    from libs.logger import log_traceback
  File "/var/www/iRedAdmin-Pro-LDAP-3.9/libs/logger.py", line 30, in <module>
    _handler = SysLogHandler(address=_server, facility=settings.SYSLOG_FACILITY)
  File "/usr/lib64/python2.7/logging/handlers.py", line 761, in __init__
    self._connect_unixsocket(address)
  File "/usr/lib64/python2.7/logging/handlers.py", line 789, in _connect_unixsocket
    self.socket.connect(address)
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 13] Permission denied

15

Re: internal server error 3.9 LDAP

Finally got it, I had to change the SYSLOG_SERVER location in settings.py

SYSLOG_SERVER = '/run/systemd/journal/socket'
instead of:
SYSLOG_SERVER = '/dev/log'
or
SYSLOG_SERVER = '/run/systemd/journal/syslog' which is the suggested alternative.

srw-rw-rw- 1 root root       0 Aug 27 07:51 socket
srw-rw---- 1 root mysyslog   0 Aug 25 03:26 syslog

So far so good.

Does this solution present any problems going forward?

16 (edited by jstewart 2019-08-27 21:04:25)

Re: internal server error 3.9 LDAP

Turns out this was all sort of self-inflicted.

A note for anyone using CSF with iredmail, if the recommended RESTRICT_SYSLOG is turned on in CSF, these errors will be the result.

I have turned it off for now, version 3.9 is running perfectly, with no errors in /var/log/messages

I'm going to look for a way to keep this security setting, and still have it work with iredmail

Thanks for your patience!