1 Topic by matteo.frakka

  • matteo.frakka
  • Member
  • Offline
  • Registered: 2012-04-15
  • Posts: 66

Topic: Internal server errors after upgrading to iRedAdmin-Pro v2.1.3 (MySQL)

==== Required information ====
- iRedMail version (check /etc/iredmail-release): 0.9.2
- Linux/BSD distribution name and version: CentOS Linux 6.6 64bit
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): MySQL
- Web server (Apache or Nginx): Apache
- Manage mail accounts with iRedAdmin-Pro?
- Related log if you're reporting an issue:
====

Hi!

After upgrading to iRedAdmin-Pro     v2.1.3 (MySQL) I got an "Internal server error" when try to see the mail body of quarantined emails.
The issue wasn't logged in apache error logs... Any suggestions?

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: Internal server errors after upgrading to iRedAdmin-Pro v2.1.3 (MySQL)

The error should be logged in /var/log/httpd/ssl_error_log, please check this file.

3 Reply by matteo.frakka

  • matteo.frakka
  • Member
  • Offline
  • Registered: 2012-04-15
  • Posts: 66

Re: Internal server errors after upgrading to iRedAdmin-Pro v2.1.3 (MySQL)

You're right... I'm sorry!!

That's what the server logs before return an "Internal server error":

Script timed out before returning headers: iredadmin.py, referer: https://serverfqdn/iredadmin/activities/quarantined/user/info@eurojack.org

Nothing else, except a warning related to CA:

[warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)


Quarantined email can be deleted or released but not displayed.

4 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: Internal server errors after upgrading to iRedAdmin-Pro v2.1.3 (MySQL)

matteo.frakka wrote:

That's what the server logs before return an "Internal server error":

Could you try steps below to get useful log?

1) Monitor Apache log file with command: 'tail -f /var/log/httpd/ssl_error_log'
2) Login to iRedAdmin-Pro and reproduce this internal server error again.
3) Paste output of 'tail' command in step 1).

5 Reply by matteo.frakka

  • matteo.frakka
  • Member
  • Offline
  • Registered: 2012-04-15
  • Posts: 66

Re: Internal server errors after upgrading to iRedAdmin-Pro v2.1.3 (MySQL)

Here:

[root@postfix httpd]# tail -f /var/log/httpd/ssl_error_log
[Mon Jul 13 14:19:42 2015] [error] [client 192.168.10.89] Premature end of script headers: iredadmin.py, referer: https://server/iredadmin/activities/quarantined
[Mon Jul 13 14:19:42 2015] [error] [client 192.168.10.89] Premature end of script headers: iredadmin.py, referer: https://server/iredadmin/activities/quarantined
[Tue Jul 14 11:03:24 2015] [error] [client 192.168.10.89] File does not exist: /var/www/roundcubemail/favicon.ico
[Tue Jul 14 11:03:24 2015] [error] [client 192.168.10.89] File does not exist: /var/www/roundcubemail/favicon.ico
[Tue Jul 14 11:04:55 2015] [error] [client 192.168.10.89] Script timed out before returning headers: iredadmin.py, referer: https://server/iredadmin/activities/quarantined
[Tue Jul 14 11:06:18 2015] [error] [client 192.168.10.89] Script timed out before returning headers: iredadmin.py, referer: https://server/iredadmin/activities/quarantined
[Tue Jul 14 11:08:55 2015] [error] [client 192.168.10.89] Premature end of script headers: iredadmin.py, referer: https://server/iredadmin/activities/quarantined
[Tue Jul 14 11:14:23 2015] [error] [client 192.168.10.89] Premature end of script headers: iredadmin.py, referer: https://server/iredadmin/activities/quarantined
[Tue Jul 14 11:16:53 2015] [error] [client 192.168.10.89] Script timed out before returning headers: iredadmin.py, referer: https://server/iredadmin/dashboard/checknew
[Tue Jul 14 11:20:55 2015] [error] [client 192.168.10.89] Premature end of script headers: iredadmin.py, referer: https://server/iredadmin/activities/quarantined
[Tue Jul 14 11:36:55 2015] [error] [client 192.168.10.89] Premature end of script headers: iredadmin.py, referer: https://server/iredadmin/activities/quarantined


If may help you, today we get something different from others day: second and third emails in the following screenshot are the same (same sender and same recipient) but I can view the body only of the second. The third email give me the internal server error.

130494

Post's attachments

Untitled.jpg
Untitled.jpg 69.33 kb, file has never been downloaded. 

You don't have the permssions to download the attachments of this post.

6 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: Internal server errors after upgrading to iRedAdmin-Pro v2.1.3 (MySQL)

Cannot figure it out with pasted log. Is it possible to let me login to your server (via ssh) for further debug? If yes, please contact me: http://www.iredmail.org/contact.html

7 Reply by matteo.frakka

  • matteo.frakka
  • Member
  • Offline
  • Registered: 2012-04-15
  • Posts: 66

Re: Internal server errors after upgrading to iRedAdmin-Pro v2.1.3 (MySQL)

I'm sorry but today the issue happened again:

[Fri Jul 17 10:04:48 2015] [error] [client 192.168.10.89] Script timed out before returning headers: iredadmin.py, referer: https://server_fqdn/iredadmin/activities/quarantined
[Fri Jul 17 10:07:02 2015] [error] [client 192.168.10.89] Premature end of script headers: iredadmin.py, referer: https://server_fqdn/iredadmin/activities/quarantined

cronjob to clean amavis db was run as expected this night.
Only 1 email quarantined but attempting to view the body returns an "Internal Server error".

8 Reply by matteo.frakka

  • matteo.frakka
  • Member
  • Offline
  • Registered: 2012-04-15
  • Posts: 66

Re: Internal server errors after upgrading to iRedAdmin-Pro v2.1.3 (MySQL)

I think that there is something wrong...

This is the log of this night of the cronjob which clean the amavisdb:

* [2015-07-17 03:20:01] Backend: mysql
* [2015-07-17 03:20:02] Delete quarantined mails which older than 7 days
* [2015-07-17 03:20:02] Delete incoming/outgoing emails which older than 3 days
* [2015-07-17 03:20:02] [-] Deleting 100 records
* [2015-07-17 03:20:02] [-] Deleting 200 records
* [2015-07-17 03:20:02] [-] Deleting 300 records
* [2015-07-17 03:20:02] [-] Deleting 307 records
* [2015-07-17 03:20:02] Delete unreferenced records from table `msgrcpt`.
* [2015-07-17 03:20:02] Delete unreferenced records from table `quarantine`.
* [2015-07-17 03:20:02] Delete unreferenced records from table `maddr`.
* [2015-07-17 03:20:02] Delete unreferenced records from table `mailaddr`.
* [2015-07-17 03:20:02] Log cleanup status.

I run the job manually some minutes ago and it deletes others records:

[root@postfix ~]# python /var/www/iredadmin/tools/cleanup_amavisd_db.py
* [2015-07-17 11:05:30] Backend: mysql
* [2015-07-17 11:05:30] Delete quarantined mails which older than 7 days
* [2015-07-17 11:05:30] Delete incoming/outgoing emails which older than 3 days
* [2015-07-17 11:05:30] [-] Deleting 24 records
* [2015-07-17 11:05:30] Delete unreferenced records from table `msgrcpt`.
* [2015-07-17 11:05:30] Delete unreferenced records from table `quarantine`.
* [2015-07-17 11:05:30] Delete unreferenced records from table `maddr`.
* [2015-07-17 11:05:30] Delete unreferenced records from table `mailaddr`.
* [2015-07-17 11:05:30] Log cleanup status.


Again, the same error in apache log:

[Fri Jul 17 10:58:34 2015] [error] [client 192.168.10.89] Script timed out before returning headers: iredadmin.py, referer: https://server/iredadmin/activities/quarantined
[Fri Jul 17 10:59:03 2015] [error] [client 192.168.10.89] Premature end of script headers: iredadmin.py, referer: https://server/iredadmin/activities/quarantined
[Fri Jul 17 10:59:03 2015] [error] [client 192.168.10.89] Premature end of script headers: iredadmin.py, referer: https://server/iredadmin/activities/quarantined
[Fri Jul 17 11:00:21 2015] [error] [client 192.168.10.89] Script timed out before returning headers: iredadmin.py, referer: https://server/iredadmin/activities/quarantined
[Fri Jul 17 11:00:53 2015] [error] [client 192.168.10.89] Script timed out before returning headers: iredadmin.py, referer: https://server/iredadmin/activities/quarantined
[Fri Jul 17 11:02:40 2015] [error] [client 192.168.10.89] Script timed out before returning headers: iredadmin.py
[Fri Jul 17 11:04:22 2015] [error] [client 192.168.10.89] Premature end of script headers: iredadmin.py, referer: https://server/iredadmin/activities/quarantined
[Fri Jul 17 11:04:22 2015] [error] [client 192.168.10.89] Premature end of script headers: iredadmin.py

9 Reply by matteo.frakka

  • matteo.frakka
  • Member
  • Offline
  • Registered: 2012-04-15
  • Posts: 66

Re: Internal server errors after upgrading to iRedAdmin-Pro v2.1.3 (MySQL)

Additional information:

If l ogin to iRedAdmin and click on the arrow to expand the email body of a quarantined email, it works and let me view the mail body. After that, any other attempt to do the same on another emails or on the same too,  will give an "Internal Server error" until apache restart.
After restarting apache, once again I can view the body of a quarantined email (the same or another one) but, again, any subsequent attempt will ends in an internal server error.

10 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: Internal server errors after upgrading to iRedAdmin-Pro v2.1.3 (MySQL)

I have no idea right now, it looks like accessing the url (/activities/quarantined) timed out.

Could you please download the latest 'cleanup_amavisd_db.py', override the one on your server, and run it again? It will show you how many records stored in SQL database.
https://bitbucket.org/zhb/iredadmin-ose … at=default

11 Reply by matteo.frakka

  • matteo.frakka
  • Member
  • Offline
  • Registered: 2012-04-15
  • Posts: 66

Re: Internal server errors after upgrading to iRedAdmin-Pro v2.1.3 (MySQL)

Here attached:

[root@server tools]# python /var/www/iredadmin/tools/cleanup_amavisd_db.py
* [2015-07-21 14:08:21] Backend: mysql
* [2015-07-21 14:08:21] Delete quarantined mails which older than 7 days
* [2015-07-21 14:08:21] Delete incoming/outgoing emails which older than 3 days
* [2015-07-21 14:08:21]         [-] Deleting 9 records
* [2015-07-21 14:08:21] Delete unreferenced records from table `msgrcpt`.
* [2015-07-21 14:08:21] Delete unreferenced records from table `quarantine`.
* [2015-07-21 14:08:21] Delete unreferenced records from table `maddr`.
* [2015-07-21 14:08:21]         [-] Deleting 8 records from table `maddr`
* [2015-07-21 14:08:21] Delete unreferenced records from table `mailaddr`.
* [2015-07-21 14:08:21]
* [2015-07-21 14:08:21] Remained records:
* [2015-07-21 14:08:21]
* [2015-07-21 14:08:21]       `msgs`: 429
* [2015-07-21 14:08:21] `quarantine`: 2
* [2015-07-21 14:08:21]      `maddr`: 88
* [2015-07-21 14:08:21]   `mailaddr`: 7
* [2015-07-21 14:08:21] Log cleanup status.
[root@postfix tools]#

The issue is quite strange... It occurs only starting from the second email body : The first time I'm always able to view the body then any other attempts will ends in an internal server error up to service restart.
After that I'm able again to view the body of only one quarantined email and any other attempt will give an error.

Reverting to iRedAdmin-Pro-SQL-2.1.2 gives the error also with the first attempt.

12 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: Internal server errors after upgrading to iRedAdmin-Pro v2.1.3 (MySQL)

Try this:

open file libs/amavisd/quarantine.py, find line 's.recv(1024)', comment out this line, restart Apache or uwsgi ( if you're running Nginx) service and try again.

13 Reply by matteo.frakka

  • matteo.frakka
  • Member
  • Offline
  • Registered: 2012-04-15
  • Posts: 66

Re: Internal server errors after upgrading to iRedAdmin-Pro v2.1.3 (MySQL)

Done, no improvement.

14 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: Internal server errors after upgrading to iRedAdmin-Pro v2.1.3 (MySQL)

No idea at all. sad

Is your Amavisd SQL server running on localhost? or on a remote server?

15 Reply by matteo.frakka (edited by matteo.frakka 2015-07-24 06:44:21)

  • matteo.frakka
  • Member
  • Offline
  • Registered: 2012-04-15
  • Posts: 66

Re: Internal server errors after upgrading to iRedAdmin-Pro v2.1.3 (MySQL)

On localhost.

16 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: Internal server errors after upgrading to iRedAdmin-Pro v2.1.3 (MySQL)

No clue yet.

is it possible to let me login to your server and dive into the iRedAdmin-Pro source code for further debug?

17 Reply by extremeshok

  • extremeshok
  • Member
  • Offline
  • From: https://eXtremeSHOK.com
  • Registered: 2014-03-19
  • Posts: 25

Re: Internal server errors after upgrading to iRedAdmin-Pro v2.1.3 (MySQL)

Same issue here, some messages in the quarantine crash the page *error 500* when their bodies are viewed.

[Mon Jul 27 09:28:34 2015] [error] [client x.x.x.x] Script timed out before returning headers: iredadmin.py
[Mon Jul 27 09:29:32 2015] [error] [client x.x.x.x] Script timed out before returning headers: iredadmin.py, referer: https://server.xx/iredadmin/activities/quarantined
[Mon Jul 27 09:32:29 2015] [error] [client x.x.x.x] Premature end of script headers: iredadmin.py, referer: https://server.xx/iredadmin/activities/quarantined

extremeshok's Website

18 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: Internal server errors after upgrading to iRedAdmin-Pro v2.1.3 (MySQL)

I cannot reproduce this issue on my server, still not sure how to reproduce it on my server.

is it possible to let me login to your server and dive into the iRedAdmin-Pro source code for further debug? @matteo.frakka and extremeshok?

19 Reply by extremeshok

  • extremeshok
  • Member
  • Offline
  • From: https://eXtremeSHOK.com
  • Registered: 2014-03-19
  • Posts: 25

Re: Internal server errors after upgrading to iRedAdmin-Pro v2.1.3 (MySQL)

Im currently running cleanup_amavisd_db.py from bitbucket.
Applying the comment out of  libs/amavisd/quarantine.py, find line 's.recv(1024) does nothing.


How does one enable debugging of the  libs/amavisd/quarantine.py

extremeshok's Website

20 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: Internal server errors after upgrading to iRedAdmin-Pro v2.1.3 (MySQL)

extremeshok wrote:

How does one enable debugging of the  libs/amavisd/quarantine.py

There's no such setting to enable debugging for certain Python source file in this case, we have to add some 'print' lines in source file to see which part might cause the problem.

21 Reply by extremeshok

  • extremeshok
  • Member
  • Offline
  • From: https://eXtremeSHOK.com
  • Registered: 2014-03-19
  • Posts: 25

Re: Internal server errors after upgrading to iRedAdmin-Pro v2.1.3 (MySQL)

Do you have a modified file that i can upload and get some debug info ?


Still deleting records with the script... at 780 600 now

extremeshok's Website

22 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: Internal server errors after upgrading to iRedAdmin-Pro v2.1.3 (MySQL)

extremeshok wrote:

Do you have a modified file that i can upload and get some debug info ?

Unfortunately, no.

23 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: Internal server errors after upgrading to iRedAdmin-Pro v2.1.3 (MySQL)

Dear matteo.frakka, extremeshok,

Another customer reported this issue and solved it: please install Python module BeautifulSoup4 manually, then restart Apache or uwsgi service, and try again.

For example:

# easy_install beautifulsoup4

24 Reply by extremeshok (edited by extremeshok 2015-08-20 23:17:57)

  • extremeshok
  • Member
  • Offline
  • From: https://eXtremeSHOK.com
  • Registered: 2014-03-19
  • Posts: 25

Re: Internal server errors after upgrading to iRedAdmin-Pro v2.1.3 (MySQL)

This makes sense, however I solved the issue by rewriting (full customisation) of the spamassassin config

extremeshok's Website

25 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: Internal server errors after upgrading to iRedAdmin-Pro v2.1.3 (MySQL)

extremeshok wrote:

however I solved the issue by rewriting (full customisation) of the spamassassin config

Would you mind sharing your solution?