1 Topic by 3aagapov

  • 3aagapov
  • Member
  • Offline
  • Registered: 2016-05-12
  • Posts: 3

Topic: spf_to_greylist_whitelists.py

==== Required information ====
- iRedMail version (check /etc/iredmail-release):  iRedMail-0.9.5-1
- Linux/BSD distribution name and version: Centos 7.2
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): PGSQL
- Web server (Apache or Nginx): Nginx
- Manage mail accounts with iRedAdmin-Pro? No
- Related log if you're reporting an issue:
====
what this could mean?:
Cron <root@xxx-srv2> python /opt/iredapd/tools/spf_to_greylist_whitelists.py >/dev/null

Traceback (most recent call last):
  File "/opt/iredapd/tools/spf_to_greylist_whitelists.py", line 401, in <module>
    logger.info('* <<< ERROR >>> Cannot insert new record for domain %s: %s' % (domain, error))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 0: ordinal not in range(128)
Thank you.

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,165

Re: spf_to_greylist_whitelists.py

Weird, SPF record should not contain non-ascii characters.
Could you please try this:

python /opt/iredapd/tools/spf_to_greylist_whitelists.py --debug

if it occurs again, please show us the console output.

Also, does it happen every time?

3 Reply by iredmailsixow (edited by iredmailsixow 2016-07-11 19:30:17)

  • iredmailsixow
  • Member
  • Offline
  • Registered: 2016-06-09
  • Posts: 2

Re: spf_to_greylist_whitelists.py

==== Required information ====
- iRedMail version (check /etc/iredmail-release):  iRedMail-0.9.5-1
- Linux/BSD distribution name and version:Debian Jessie
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): PGSQL
- Web server (Apache or Nginx): Apache
- Manage mail accounts with iRedAdmin-Pro? No
- Related log if you're reporting an issue:
====

Hi, sorry for my English, i have the same problem, i write you the last of the console output:
[_spf3.zendesk.com] include: -> _spf4.zendesk.com
        + [include: _spf4.zendesk.com] v=spf1 ip4:41.74.201.0/24 ip4:41.74.205.0/24 ip4:103.13.69.0/24 ip4:124.47.150.0/24 ip4:124.47.189.0/24 ip4:180.189.28.0/24 ip4:205.217.25.135 ip4:205.217.25.132 ip4:207.211.41.113 ~all
        + Result: set(['208.66.139.0/25', '173.194.0.0/16', '66.249.80.0/20', '96.46.150.192/27', '41.74.204.0/22', '213.167.75.0/24', '2001:4860:4000::/36', '207.211.31.0/25', '2c0f:fb50:4000::/36', '64.18.0.0/20', '188.172.128.0/20', '205.139.110.0/23', '2404:6800:4000::/36', '207.126.144.0/20', '41.74.196.0/22', '2a00:1450:4000::/36', '41.74.192.0/22', '205.217.25.135', '205.217.25.132', '180.189.28.0/24', '2800:3f0:4000::/36', '41.74.205.0/24', '41.74.200.0/22', '216.239.32.0/19', '174.137.46.0/24', '216.229.156.0/25', '91.220.42.0/24', '216.205.24.0/24', '108.177.8.0/21', '66.102.0.0/20', '207.82.80.0/24', '216.58.192.0/19', '63.128.21.0/24', '64.233.160.0/19', '185.12.80.0/22', '124.47.150.0/24', '172.217.0.0/19', '216.198.0.0/18', '72.14.192.0/18', '146.101.78.0/24', '41.74.201.0/24', '74.125.0.0/16', '207.211.30.0/24', '103.13.69.0/24', '207.211.41.113', '195.130.217.0/24', '2607:f8b0:4000::/36', '209.85.128.0/17', '124.47.189.0/24', '213.167.81.0/24', '192.161.144.0/20'])
    + [zoho.com]
        + SPF -> v=spf1 include:spf2.zoho.com include:spf.zoho.com  ~all
        + [zoho.com] include: -> spf.zoho.com, spf2.zoho.com
        + [include: spf.zoho.com] v=spf1 ip4:74.201.152.59/32 ip4:74.201.155.27/32 ip4:74.201.155.28/32 ip4:74.201.154.0/24 ip4:74.201.84.0/24 ~all
        + [include: spf2.zoho.com] v=spf1 ip4:165.254.168.66/31 ip4:165.254.168.68/31 ip4:165.254.168.70/31 ip4:165.254.168.72/31 ip4:74.201.155.79/32 ip4:74.201.155.25/32 ip4:74.201.155.26/32 ip4:72.5.230.111/32 ip4:165.254.167.152/30 ip4:165.254.167.156/31 ip4:165.254.167.162/31 ~all
        + Result: set(['74.201.84.0/24', '165.254.168.70/31', '165.254.168.66/31', '165.254.167.152/30', '74.201.152.59/32', '165.254.167.162/31', '72.5.230.111/32', '74.201.154.0/24', '74.201.155.79/32', '74.201.155.27/32', '165.254.168.68/31', '74.201.155.25/32', '165.254.168.72/31', '74.201.155.28/32', '74.201.155.26/32', '165.254.167.156/31'])
Traceback (most recent call last):
  File "/opt/iredapd/tools/spf_to_greylist_whitelists.py", line 401, in <module>
    logger.info('* <<< ERROR >>> Cannot insert new record for domain %s: %s' % (domain, error))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 27: ordinal not in range(128)

Thanks in advance.

4 Reply by frank.brehm

  • frank.brehm
  • Member
  • Offline
  • Registered: 2016-07-27
  • Posts: 2

Re: spf_to_greylist_whitelists.py

Found the reason:
Postgres is responding on a duplicate key with a localized error message (I'm a German):

FEHLER:  doppelter Schlüsselwert verletzt Unique-Constraint „idx_greylisting_whitelists_account_sender“
DETAIL:  Schlüssel „(account, sender)=(@., 208.75.120.0/22)“ existiert bereits.

So the test in line 398 is completely wrong, because the German error message does not contain the strings 'duplicate key' or 'duplicate entry'. Additionally the logging output is crashing, because in the modified error message (lower()) there are some strange characters.

What to do? Different possibilities.
One might be to enforce the database session to use another locale (not sure, how to do it).
Another possibility might be to check the class of the exception at this point - I modified the check at this point to check, that the class name of the exception at this point is 'IntegrityError'.
Additionally in the logging output should not be used the manipulated error message, but the raw error message instead.

Here a patch, which is working for me:
--- spf_to_greylist_whitelists.py    2016-07-27 12:49:13.695005512 +0200
+++ spf_to_greylist_whitelists.py.new    2016-07-27 12:54:47.089108506 +0200
@@ -394,11 +394,13 @@
                         sender=ip,
                         comment=comment)
         except Exception, e:
-            error = str(e).lower()
-            if 'duplicate key' in error or 'duplicate entry' in error:
+            logger.debug("Got a %s for domain %r: %s", e.__class__.__name__, domain, str(e).rstrip())
+            # error = str(e).lower()
+            # if 'duplicate key' in error or 'duplicate entry' in error:
+            if e.__class__.__name__ == 'IntegrityError':
                 pass
             else:
-                logger.info('* <<< ERROR >>> Cannot insert new record for domain %s: %s' % (domain, error))
+                logger.error('* <<< ERROR >>> Cannot insert new record for domain %s: %s', domain, str(e).rstrip())

if submit_to_sql_db:
     logger.info('* Store domain names in SQL database as greylisting whitelists.')

5 Reply by frank.brehm

  • frank.brehm
  • Member
  • Offline
  • Registered: 2016-07-27
  • Posts: 2

Re: spf_to_greylist_whitelists.py

For the record:
==== Required information ====
- iRedMail version (check /etc/iredmail-release):  iRedMail-0.9.5-1
- Linux/BSD distribution name and version:Debian Jessie
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): PGSQL
- Web server (Apache or Nginx): Nginx
- Manage mail accounts with iRedAdmin-Pro? No
====

And of course I don't know, what's the name of the exception class at this point, if the backend store is LDAP. For PGSQL and MySQL/MariaDB it is 'IntegrityError'.

6 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,165

Re: spf_to_greylist_whitelists.py

Your code to handle error exception is better, i will merge it. Thanks for the contribution. smile

7 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,165

Re: spf_to_greylist_whitelists.py

UPDATE: Change committed.
https://bitbucket.org/zhb/iredapd/commi … 4849c19896