1 Topic by rafaelr

  • rafaelr
  • Member
  • Offline
  • Registered: 2015-03-26
  • Posts: 26

Topic: exclude specific bad header emails from quarentine

==== Required information ====
- iRedMail version (check /etc/iredmail-release): 0.9.2
- Linux/BSD distribution name and version:  Ubuntu 14.04 LTS
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): MySQL
- Web server (Apache or Nginx): both
- Manage mail accounts with iRedAdmin-Pro? Yes
- Related log if you're reporting an issue:
====

Hello,

I would like to know if there is a way (or workaround) to create a whitelist for specific bad header tagged emails not to be quarantined.

Currently, we are quarantining spam and bad header emails to SQL. Some users we communicate with on daily basis send us emails from time to time with bad headers and since these emails are quarantined they never make it to the end user when they should. We have added those senders to the whitelist but that hasn't had any effect in the way bad header emails are treated and therefore we have to manually go into iredmail and release them from the quarantine. It would be really useful if there was a way to completely whitelist specific email addresses not to be scanned at all or simply not to pass them through amavis antispam on delivery to workaround this problem.

Any help is really appreciated because this is becoming too time consuming.

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,094

Re: exclude specific bad header emails from quarentine

A possible solution: login to iRedAdmin-Pro, go to user profile page, under tab "Spam Policy", change any settings to trigger iRedAdmin-Pro to create a record in sql table `amavisd.policy`, then set `bad_header_lover='Y'` for this new record.

Let me know whether or not it works for you. i will make it manageable with iRedAdmin-Pro.

3 Reply by rafaelr

  • rafaelr
  • Member
  • Offline
  • Registered: 2015-03-26
  • Posts: 26

Re: exclude specific bad header emails from quarentine

Zhang,

I don't believe that would be a good approach based on the fact that a lot of spam messages are caught due to bad headers. For example, I have users that receive a lot of spam that score 0.0 (not checked) and those messages are quarantined because they have bad headers, e.g:

-------
Content-type    multipart/alternative; boundary="=Njg3YmNkYmNkZGQ3NjZmZmY3ZmYzOTRiNTdkODMwN2Q_"
Date    Thu, 27 Aug 2015 21:18:41 -0400
From    "Google Jobs" <info@ilkshard.com>
Message-ID    <17673679441280150104659ab9abbb544ddd5df394b57d8307d@alt3.ilkshard.com>
Received    from alt3.ilkshard.com (unknown [98.142.107.36]) by mailer.ny.domain.com (Postfix) with ESMTP id 1E9E86DE5C1F for <user@domain.com>; Thu, 27 Aug 2015 21:19:36 -0400 (EDT)
Received    from mailer.ny.domain.com ([127.0.0.1]) by localhost (mailer.ny.domain.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IbotmHI80Ktv for <user@domain.com>; Thu, 27 Aug 2015 21:19:38 -0400 (EDT)
Subject    Congrats - your resume was accepted by Google
To    <user@domain.com>
X-Amavis-Alert    BAD HEADER SECTION MIME error: error: part did not end with expected boundary
X-Envelope-To    <user@domain.com>
X-Envelope-To-Blocked    <user@domain.com>
X-Quarantine-ID    <IbotmHI80Ktv>
X-Spam-Flag    NO
X-Spam-Level   
X-Spam-Score    0
X-Spam-Status    No, score=x tag=x tag2=x kill=x tests=[] autolearn=unavailable
-----

As you can see, those messages are not scoring (for whatever reason - it has been the same for the time the mailserver has been running since day one) yet they are caught because of bad headers.

Case in point, I enabled bad headers quarantine because these messages were making through to prevent the deliveries completely. Unfortunately, as I mentioned before, some legit users send us emails every other day that are also quarantined because of bad header triggering. Having a whitelist to allow these specific users will be a better option all together.

Unless there is a problem in my installation with the spam filtering (amavis not parsing specific emails) I don't see `bad_header_lover='Y'` to be the solution in my case.

Is it there anything else I could do about it?

4 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,094

Re: exclude specific bad header emails from quarentine

The problem is, we do have whitelists, but looks like Amavisd doesn't lookup whitelist for bad-header checking. We need to make sure whether it's true (not use whitelist for bad-header checking). Would you like to ask this in Amavisd mailing list? you have more details/log than me, could you please do me a favor?

with `bad_header_lover='Y'` for one user, amavisd will bypass bad-header checking for this user.

5 Reply by drosalevan (edited by drosalevan 2015-08-31 15:12:24)

  • drosalevan
  • Member
  • Offline
  • Registered: 2015-06-11
  • Posts: 14

Re: exclude specific bad header emails from quarentine

Amavis bad header check start after white list. So bad_header_lover='Y' works, but.... for example

local domain is domain.com
user1@domain.com (BAD HEADER) -> user2@domain.net , you must add user1@domain.com as  bad_header_lover='Y'

if domain.com and domain.net - are local domains
then you need to add both user1@domain.com and user2@domain.net as  bad_header_lover='Y'

anyway need to find optimal solution for this situation

6 Reply by rafaelr (edited by rafaelr 2015-12-06 18:27:26)

  • rafaelr
  • Member
  • Offline
  • Registered: 2015-03-26
  • Posts: 26

Re: exclude specific bad header emails from quarentine

Zhang, I'm posting here to make sure this helps others and also for your consideration to include  it within amavis configuration by default whether commented out or active:

Since Amavis-new version 2.4.4, one can actually tweak the bad-header tests. This is definitely very helpful in this case instead of going with the more radical bad_header_lover='Y' option previously mentioned.

Furthermore, there is an $allowed_header_tests variable in amavis by which we can define what should be looked up during the bad-header checks.

The final solution in my case was to insert the appropriate lines in the /etc/amavis/conf.d/50-user file as follows:

...
# EXCLUDE HEADER TEST
# ----------------------------
# Duplicate or multiple occurrence of a header field
$allowed_header_tests{'multiple'} = 0;
# the infamous Date missing
$allowed_header_tests{'missing'} = 0;

@av_scanners = (
...

The above allowed_header_tests variables would exclude missing items (the infamous Date missing) and Duplicate header field tests form the header check process.

Obviously, admins should include or exclude the ones that works for them. Those two options (specially the Duplicate header field) is what has caused me the most headaches. Providers such as Microsoft and other major ones tend to send out legit messages (notifications, passwords reset and so on) with Duplicate header fields, which caused me lots of pain having to go release each message from the quarantine manually yet bad header messages from spammers are still being caught which was my goal. As long as the Duplicate header and date fields are ignored, the message is parsed and checked with spamassassin which in turn determines whether the message is spam or not based on usual scoring.

----------------END OF SUGGESTION--------------


So, I have finally been able to fine tune my setup with a more radical configuration (note that I use SQL  to backup-quarantine spam and also for bad header messages) so messages are really not lost by discarding them:

...

$final_virus_destiny      = D_DISCARD;
$final_banned_destiny     = D_DISCARD;
$final_spam_destiny       = D_DISCARD;
$final_bad_header_destiny = D_DISCARD;

# EXCLUDE HEADER TEST
# ----------------------------
# Duplicate or multiple occurrence of a header field
$allowed_header_tests{'multiple'} = 0;
# the infamous Date missing
$allowed_header_tests{'missing'} = 0;

$sa_tag2_level_deflt = 5.31; # add 'spam detected' headers at that level
$sa_kill_level_deflt = 6.00; # triggers spam evasive actions
$sa_dsn_cutoff_level = 9;   # spam level beyond which a DSN is not sent
$sa_quarantine_cutoff_level = 9; # level at which spam isn't even quarantined
$penpals_bonus_score = 8;    # (no effect without a @storage_sql_dsn database)
$penpals_threshold_high = $sa_kill_level_deflt;  # don't waste time on hi spam

@av_scanners = (
...

Hope this helps. Cheers.

7 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,094

Re: exclude specific bad header emails from quarentine

Suggestion accepted, this will be enabled in upcoming iRedMail release (0.9.3) by default.

To be more clear, below is quote from Amavisd document to help understand what this parameter is used for (https://www.ijs.si/software/amavisd/release-notes.txt):

- new configuration variable %allowed_header_tests, also member of policy
  banks, allows for selectively disabling some of the header checks,
  e.g. checks for non-encoded 8-bit characters. The %allowed_header_tests
  hash contains all available header test names as its keys by default
  (with a value of true);  removing a key, or setting its value to false,
  disables a test, e.g.:
    $allowed_header_tests{'8bit'} = 0;
    $allowed_header_tests{'missing'} = 0;
  Currently available keys (i.e. test names) are:
    other mime 8bit control empty long syntax missing multiple
  each corresponding to its own minor contents category of CC_BADH;

    ccat test
    min  name      description
    ---  -------   -----------
      0  other     (catchall for everything else, normally not used)
      1  mime      Bad MIME (sub)headers or bad MIME structure
      2  8bit      Invalid non-encoded 8-bit characters in header
      3  control   Invalid control characters in header (CR or NUL)
      4  empty     Folded header field made up entirely of whitespace
      5  long      Header line longer than RFC 2822 limit of 998 characters
      6  syntax    Header field syntax error
      7  missing   Missing required header field
      8  multiple  Duplicate or multiple occurrence of a header field
  legend:
    ccat min:  minor contents category under a major category CC_BADH,
               available in templates as a macro ccat_min;
    test name: corresponding test name - a key in %allowed_header_tests;
    descr.:    description of a header test or MIME subheaders/structure test;