1

Topic: internal server error 3.9 LDAP

==== REQUIRED BASIC INFO OF YOUR IREDMAIL SERVER ====
- iRedMail version (check /etc/iredmail-release): 0.9.9
- Deployed with iRedMail Easy or the downloadable installer? NO
- Linux/BSD distribution name and version: Centos 7.6
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): LDAP
- Web server (Apache or Nginx): Nginx
- Manage mail accounts with iRedAdmin-Pro? Yes
- [IMPORTANT] Related original log or error message is required if you're experiencing an issue.
====
Upgraded to 3.9 directly from 3.6 and now get only internal server error when accessing the admin.
Nothing in the logs that I can see, other than:

/var/log/nginx/access.log
GET /admin/ HTTP/1.1" 500 32 "https://server.domain.com/admin/dashboard" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0

no entries in /var/log/nginx/error_log other than a few roundcube errors

only entry in /var/log/messages when I restart service:
Aug 13 06:29:57 mail systemd[1]: Stopping iRedAdmin daemon service...
Aug 13 06:29:58 mail systemd[1]: Stopped iRedAdmin daemon service.
Aug 13 06:29:58 mail systemd[1]: Starting iRedAdmin daemon service...
Aug 13 06:29:58 mail systemd[1]: Started iRedAdmin daemon service.
Aug 13 06:29:58 mail uwsgi[6876]: [uWSGI] getting INI configuration from /var/www/iredadmin/rc_scripts/uwsgi/rhel.ini

If I revert to 3.6 everything works fine:
"GET /admin/dashboard HTTP/1.1" 200 4932 "https://server.domain.com/admin/dashboard" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0

2

Re: internal server error 3.9 LDAP

With "internal server error", i expect some error log in /var/log/message.
Could you try to reproduce this error again and check all log files under /var/log/?

----

Does my reply help a little? How about buying me a cup of coffee ($5) as an encouragement?

buy me a cup of coffee

3 (edited by jstewart 2019-08-13 19:10:22)

Re: internal server error 3.9 LDAP

ZhangHuangbin wrote:

With "internal server error", i expect some error log in /var/log/message.
Could you try to reproduce this error again and check all log files under /var/log/?

The only entries I see are the ones I posted. I checked every file in every directory in /var/log

All services restarted, as well as a server reboot after the upgrade. When I downgrade back to 3.6 restarting the iredadmin service brings it back

4 (edited by jstewart 2019-08-13 20:29:14)

Re: internal server error 3.9 LDAP

jstewart wrote:
ZhangHuangbin wrote:

With "internal server error", i expect some error log in /var/log/message.
Could you try to reproduce this error again and check all log files under /var/log/?

The only entries I see are the ones I posted. I checked every file in every directory in /var/log

All services restarted, as well as a server reboot after the upgrade. When I downgrade back to 3.6 restarting the iredadmin service brings it back

I went directly from 3.6 to 3.9. I'm going to try 3.7 and 3.8 to see if they work...

Upgrade to 3.7 works fine.
Upgrade to 3.8 gives internal server error, same as 3.9

I spoke too soon - 3.7 allows access to the interface, but internal server error when accessing a user.

Update - this may be my fault - I made some alterations to allow for new LDAP attributes, and I likely broke it myself. I'll update this message if that is the case.

It wasn't that - I redid the update to 3.7 and it is working fine, no more errors. 3.8 and 3.9 are not. I'm going to continue trying, and go through logs to see if I can figure it out - which log file would generally log the internal server error? I'm not seeing any errors related to this in any log files.

5

Re: internal server error 3.9 LDAP

jstewart wrote:

which log file would generally log the internal server error? I'm not seeing any errors related to this in any log files.

iRedAdmin-Pro-LDAP-3.9 logs to syslog by default, so if there's some "internal server error", you can check Nginx log file and default syslog log file (/var/log/messages, or /var/log/syslog).

Another solution is set "MAIL_ERROR_TO_WEBMASTER = True" in iRedAdmin-Pro config file, then restart "iredadmin" service. Each time an "internal server error" is triggered, webmaster (defined in parameter "webmaster =" in iRedAdmin-Pro config file) will receive an email with detailed info. WARNING: this email may contain sensitive info like SQL username/password, make sure it won't be delivered to some address you don't trust.

----

Does my reply help a little? How about buying me a cup of coffee ($5) as an encouragement?

buy me a cup of coffee

6 (edited by jstewart 2019-08-15 21:34:58)

Re: internal server error 3.9 LDAP

Thanks.
I set MAIL_ERROR_TO_WEBMASTER = True, and here are the results:

Error after upgrading to 3.8:
email subject line: bug: <class 'socket.error'>: [Errno 13] Permission denied (/)

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/web/application.py", line 239, in process
    return self.handle()
  File "/usr/lib/python2.7/site-packages/web/application.py", line 230, in handle
    return self._delegate(fn, self.fvars, args)
  File "/usr/lib/python2.7/site-packages/web/application.py", line 416, in _delegate
    mod = __import__(mod, None, None, [''])
  File "/var/www/iRedAdmin-Pro-LDAP-3.8/controllers/ldap/basic.py", line 9, in <module>
    from libs import iredutils, sysinfo, form_utils
  File "/var/www/iRedAdmin-Pro-LDAP-3.8/libs/sysinfo.py", line 13, in <module>
    from libs.logger import log_traceback
  File "/var/www/iRedAdmin-Pro-LDAP-3.8/libs/logger.py", line 30, in <module>
    _handler = SysLogHandler(address=_server, facility=settings.SYSLOG_FACILITY)
  File "/usr/lib64/python2.7/logging/handlers.py", line 761, in __init__
    self._connect_unixsocket(address)
  File "/usr/lib64/python2.7/logging/handlers.py", line 789, in _connect_unixsocket
    self.socket.connect(address)
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 13] Permission denied



Error after upgrading to 3.9:
email subject line: bug: <class 'socket.error'>: [Errno 13] Permission denied (/)

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/web/application.py", line 239, in process
    return self.handle()
  File "/usr/lib/python2.7/site-packages/web/application.py", line 230, in handle
    return self._delegate(fn, self.fvars, args)
  File "/usr/lib/python2.7/site-packages/web/application.py", line 416, in _delegate
    mod = __import__(mod, None, None, [''])
  File "/var/www/iRedAdmin-Pro-LDAP-3.9/controllers/ldap/basic.py", line 9, in <module>
    from libs import iredutils, sysinfo, form_utils
  File "/var/www/iRedAdmin-Pro-LDAP-3.9/libs/sysinfo.py", line 13, in <module>
    from libs.logger import log_traceback
  File "/var/www/iRedAdmin-Pro-LDAP-3.9/libs/logger.py", line 30, in <module>
    _handler = SysLogHandler(address=_server, facility=settings.SYSLOG_FACILITY)
  File "/usr/lib64/python2.7/logging/handlers.py", line 761, in __init__
    self._connect_unixsocket(address)
  File "/usr/lib64/python2.7/logging/handlers.py", line 789, in _connect_unixsocket
    self.socket.connect(address)
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 13] Permission denied

symlink for iredadmin set back to iRedAdmin-Pro-LDAP-3.7 and everything works properly

the only entries in the other logs are the same as before:
/var/log/messages:
Aug 15 05:26:37 mail systemd[1]: Stopping iRedAdmin daemon service...
Aug 15 05:26:38 mail systemd[1]: Stopped iRedAdmin daemon service.
Aug 15 05:26:38 mail systemd[1]: Starting iRedAdmin daemon service...
Aug 15 05:26:38 mail systemd[1]: Started iRedAdmin daemon service.
Aug 15 05:26:38 mail uwsgi[9797]: [uWSGI] getting INI configuration from /var/www/iredadmin/rc_scripts/uwsgi/rhel.ini

/var/log/nginx/access.log:
192.168.104.68 - - [15/Aug/2019:05:24:05 -0400] "GET /admin HTTP/1.1" 500 32 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0"
192.168.104.68 - - [15/Aug/2019:05:30:13 -0400] "GET /admin HTTP/1.1" 500 32 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0"


Does any of that make sense?

FYI, all updates have gone without a problem since the initial install of iRedAdmin-Pro-LDAP-3.1 up to 3.7. upgrades to 3.8 and 3.9 result in the internal server error.

7

Re: internal server error 3.9 LDAP

Is port 7791 used by other program?

----

Does my reply help a little? How about buying me a cup of coffee ($5) as an encouragement?

buy me a cup of coffee

8

Re: internal server error 3.9 LDAP

ZhangHuangbin wrote:

Is port 7791 used by other program?

No.
As I had indicated, I upgraded to 3.7 from 3.6 without any issues, but it's the 3.8 and 3.9 installs that are giving the error.

9

Re: internal server error 3.9 LDAP

- Could you please check whether port 7791 is used by iredadmin or other program with command below?

lsof -i :7791

If there's some program, please stop it. You can also use `kill -9 <pid>` command to kill its pid.

- Then restart "iredadmin" service and try to access it.

----

Does my reply help a little? How about buying me a cup of coffee ($5) as an encouragement?

buy me a cup of coffee

10

Re: internal server error 3.9 LDAP

lsof -i :7791
COMMAND  PID      USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
uwsgi   2133 iredadmin    6u  IPv4  25910      0t0  TCP localhost:7791 (LISTEN)
uwsgi   2352 iredadmin    6u  IPv4  25910      0t0  TCP localhost:7791 (LISTEN)
uwsgi   2353 iredadmin    6u  IPv4  25910      0t0  TCP localhost:7791 (LISTEN)
uwsgi   2358 iredadmin    6u  IPv4  25910      0t0  TCP localhost:7791 (LISTEN)
uwsgi   2360 iredadmin    6u  IPv4  25910      0t0  TCP localhost:7791 (LISTEN)
uwsgi   2362 iredadmin    6u  IPv4  25910      0t0  TCP localhost:7791 (LISTEN)

3.7 working fine
then:
service iredadmin stop
- nothing listening on 7791:
change symlink for iredadmin to either 3.8 or 3.9 and stop then start iredadmin the internal server error is back. Rebooted server, same results.
Set symlink back to 3.7, everything is fine again.

GET https://cordelia.cyg.net/admin/

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/web/application.py", line 239, in process
    return self.handle()
  File "/usr/lib/python2.7/site-packages/web/application.py", line 230, in handle
    return self._delegate(fn, self.fvars, args)
  File "/usr/lib/python2.7/site-packages/web/application.py", line 416, in _delegate
    mod = __import__(mod, None, None, [''])
  File "/var/www/iRedAdmin-Pro-LDAP-3.8/controllers/ldap/basic.py", line 9, in <module>
    from libs import iredutils, sysinfo, form_utils
  File "/var/www/iRedAdmin-Pro-LDAP-3.8/libs/sysinfo.py", line 13, in <module>
    from libs.logger import log_traceback
  File "/var/www/iRedAdmin-Pro-LDAP-3.8/libs/logger.py", line 30, in <module>
    _handler = SysLogHandler(address=_server, facility=settings.SYSLOG_FACILITY)
  File "/usr/lib64/python2.7/logging/handlers.py", line 761, in __init__
    self._connect_unixsocket(address)
  File "/usr/lib64/python2.7/logging/handlers.py", line 789, in _connect_unixsocket
    self.socket.connect(address)
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 13] Permission denied

11

Re: internal server error 3.9 LDAP

I figured it out: do you have /dev/log on this server?

iRedAdmin-Pro expects syslog creates socket /dev/log, but your server seems uses different path. Which one is it?

After found the correct syslog socket path, please add setting below with the socket path in /opt/iredapd/settings.py and restart iredapd service:

SYSLOG_SERVER = '/dev/log'

----

Does my reply help a little? How about buying me a cup of coffee ($5) as an encouragement?

buy me a cup of coffee

12 (edited by jstewart 2019-08-20 22:05:32)

Re: internal server error 3.9 LDAP

ZhangHuangbin wrote:

I figured it out: do you have /dev/log on this server?

iRedAdmin-Pro expects syslog creates socket /dev/log, but your server seems uses different path. Which one is it?

After found the correct syslog socket path, please add setting below with the socket path in /opt/iredapd/settings.py and restart iredapd service:

SYSLOG_SERVER = '/dev/log'

Same result.
added that line to settings.py in /var/www/iRedAdmin-Pro-LDAP-3.9
settings.py:SYSLOG_SERVER = '/dev/log'

I also tried: SYSLOG_SERVER = '/run/systemd/journal/syslog' as per the suggestion in libs/default_settings.py


ls -l /dev/log
srw-rw---- 1 root mysyslog 0 Aug 15 07:59 /dev/log


Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/web/application.py", line 239, in process
    return self.handle()
  File "/usr/lib/python2.7/site-packages/web/application.py", line 230, in handle
    return self._delegate(fn, self.fvars, args)
  File "/usr/lib/python2.7/site-packages/web/application.py", line 416, in _delegate
    mod = __import__(mod, None, None, [''])
  File "/var/www/iRedAdmin-Pro-LDAP-3.9/controllers/ldap/basic.py", line 9, in <module>
    from libs import iredutils, sysinfo, form_utils
  File "/var/www/iRedAdmin-Pro-LDAP-3.9/libs/sysinfo.py", line 13, in <module>
    from libs.logger import log_traceback
  File "/var/www/iRedAdmin-Pro-LDAP-3.9/libs/logger.py", line 30, in <module>
    _handler = SysLogHandler(address=_server, facility=settings.SYSLOG_FACILITY)
  File "/usr/lib64/python2.7/logging/handlers.py", line 761, in __init__
    self._connect_unixsocket(address)
  File "/usr/lib64/python2.7/logging/handlers.py", line 789, in _connect_unixsocket
    self.socket.connect(address)
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 13] Permission denied

13

Re: internal server error 3.9 LDAP

The syslog socket must be readable and writable by others, not just owner and group. Could you please try again?
Also, make sure other user can access parent directories of the socket.

----

Does my reply help a little? How about buying me a cup of coffee ($5) as an encouragement?

buy me a cup of coffee