1 Topic by itechpro

  • itechpro
  • Member
  • Offline
  • Registered: 2018-07-18
  • Posts: 5

Topic: tld based greylisting time ?

======== REQUIRED BASIC INFO OF YOUR IREDMAIL SERVER ====
- iRedMail version (check /etc/iredmail-release): 0.9.8 MARIADB edition.
- Linux/BSD distribution name and version: Ubuntu 18.04.1
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): MySQL
- Web server (Apache or Nginx): Nginx
- Manage mail accounts with iRedAdmin-Pro? No
====

I like the greylisting for all new/unknown incoming mail addresses, but I'd prefer different greylisting time frames for certain tlds (top-level domains), since I am still getting spam esp. from certain ones.
For example I'd like to keep greylisting of (un)named tlds like '.com' for the standard 5-15min (GREYLISTING_BLOCK_EXPIRE), but other ones like '.space' or '.review' for e.g. 1 hour).
To achive this I would probably need to extend '/opt/iredapd/plugins/greylisting.py' and can optionally create a sql table under 'iredapd' to list these tlds, but my knowledge of python is to sparse for that.
Could someone (maybe also interested in further reducing spam this way) give me some hints or code pieces to start with that approach ?

Thanks,

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 30,792

Re: tld based greylisting time ?

I don't think this is a good idea.

The retry / re-deliver interval depends on the sender server MTA (e.g. Postfix, Exim, Exchange), not based on the TLD. If you extend the expire time to an unreasonable time, the email may be not able to enter your queue at all.

3 Reply by itechpro

  • itechpro
  • Member
  • Offline
  • Registered: 2018-07-18
  • Posts: 5

Re: tld based greylisting time ?

Hi Zhang,

Thank's for your reply. But I don't believe my approach is SO wrong:

I understand that "the retry/re-deliver interval depends on the" sending MTA, but that’s not what I am talking about.
From my experience decent configured MTAs have retry rules for hours or days, when they get a 4xx reply code.
I believe that the original idea of greylisting included, that mass spam senders either don’t have a mail queue and give up immediately, or nowadays their mail queue would get filled quicker with greylisting, which requires investment in additional resources.
With a TLD based, extended greylisting period, I could increase the costs of sending spam, and narrow down the disadvantage of further delayed mail delivery to certain TLDs. Additionally I have the choice to either block or whitelist the domains from these TLDs during that extended period. Therefore I would just emphasise the idea and concept of greylisting.
Until now I didn't get a single legitimate email from certain TLDs like '.review', but don't want to either block these TLDs in general nor whole countries via geoip.

4 Reply by itechpro

  • itechpro
  • Member
  • Offline
  • Registered: 2018-07-18
  • Posts: 5

Re: tld based greylisting time ?

As intended before I've now extended the code to greylist certain top-level domains for a longer time, and have it running for a couple of days under Linux with success.
Following I share my approach with you, and would be thankful for any comments or useful changes:

1. create another table in (Maria)DB e.g. with: 

CREATE TABLE iredapd.greylisting_ext (id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY, tld VARCHAR(64) );

2. add one or more tlds with: 

INSERT INTO iredapd.greylisting_ext SET tld='loan';

    see also most used tlds by spammers: https://www.spamhaus.org/statistics/tlds/

3. add variables to /opt/iredapd/settings.py:

    # settings for extended greylisting for certain tlds
    # allow after 1 day (1440min), forget without retrying after 2 days
    GREYLISTING_BLOCK_EXPIRE_EXT = 1440
    GREYLISTING_UNAUTH_TRIPLET_EXPIRE_EXT = 2

4. change some code in /opt/iredapd/plugins/greylisting.py:

--- greylisting.py      2018-02-07 03:16:56.000000000 +0100
+++ greylisting.py.ext  2018-12-01 11:14:46.974074092 +0100
@@ -178,12 +178,30 @@
                                       recipient,
                                       recipient_domain,
                                       client_address):
+
+    # check if extented greylisting should be applied
+    sender_tld_domain = sender_domain.split('.')[-1]
+    sql = """SELECT id
+              FROM greylisting_ext
+              WHERE tld='%s'
+              LIMIT 1""" % (sender_tld_domain)
+    logger.debug('[SQL] query tlds: \n%s' % sql)
+    qr = conn.execute(sql)
+    sql_record = qr.fetchone()
+
     # Time of now.
     now = int(time.time())

-    # timeout in seconds
-    block_expired = now + int(settings.GREYLISTING_BLOCK_EXPIRE) * 60
-    unauth_triplet_expire = now + int(settings.GREYLISTING_UNAUTH_TRIPLET_EXPIRE) * 24 * 60 * 60
+    if sql_record:
+        logger.debug('Client tld (%s) will be greylisted extra long.')
+        # timeout in seconds
+        block_expired = now + int(settings.GREYLISTING_BLOCK_EXPIRE_EXT) * 60
+        unauth_triplet_expire = now + int(settings.GREYLISTING_UNAUTH_TRIPLET_EXPIRE_EXT) * 24 * 60 * 60
+    else:
+        # timeout in seconds
+        block_expired = now + int(settings.GREYLISTING_BLOCK_EXPIRE) * 60
+        unauth_triplet_expire = now + int(settings.GREYLISTING_UNAUTH_TRIPLET_EXPIRE) * 24 * 60 * 60
+
     auth_triplet_expire = now + int(settings.GREYLISTING_AUTH_TRIPLET_EXPIRE) * 24 * 60 * 60

     sender = sqlquote(sender)

5. restart iredappd deamon:

systemctl restart iredapd

For my personal usage I've increased the values of GREYLISTING_BLOCK_EXPIRE_EXT and GREYLISTING_UNAUTH_TRIPLET_EXPIRE_EXT wink

Have Fun

5 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 30,792

Re: tld based greylisting time ?

Did you every try this without modifying iredapd source code? big_smile

cd /opt/iredapd/tools/
python greylisting_admin.py --enable --from '@.loan'

6 Reply by itechpro

  • itechpro
  • Member
  • Offline
  • Registered: 2018-07-18
  • Posts: 5

Re: tld based greylisting time ?

Thank you for your reply and suggestion, I hadn't tried that one.
But if I understand it right, your approach it doesn't fulfil my wish for longer, extended and different greylisting times for these tlds ?

7 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 30,792

Re: tld based greylisting time ?

itechpro wrote:

Thank you for your reply and suggestion, I hadn't tried that one.
But if I understand it right, your approach it doesn't fulfil my wish for longer, extended and different greylisting times for these tlds ?

It doesn't.

A better implementation might be: Add new column in sql table "iredapd.greylisting" to store per-account greylisting time. Defaults to 0 or NULL, if null, use the setting in settings.py.