1 (edited by bw1984 2014-08-05 17:36:31)

Topic: default throttling limits

==== Required information ====
- iRedMail version: latest
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): MySQL
- Linux/BSD distribution name and version: CentOS latest
- Related log if you're reporting an issue: n/a
====

We recently had a problem with a huge amount of spam being sent out from the server from 1 specific inbox. I dont think it was a server/iredmail compromise (at least i hope not), most probably just one of our clients having malware on their computer or something. When we checked our throttling limits for a lot of the inboxes on our installation and found that they were set to fairly high values. These seemed arbitrary at first but they were all identical.

some examples of the values;

incoming
=========
Enable sender throttling - ticked
Number of max outgoing emails - 512
Quota size of all outgoing emails - 250MB (the equivalent value in bytes in the input box)

outgoing
=========
Enable recipient throttling - ticked
Number of max incoming emails - 64

These values seemingly inserted themselves as they we're not the sort of values which would be entered by a human, so i just wondered where they are set and if anyone has experienced a similar problem at any point? In addition to this a lot of the "20 per domain per hour" rules we had inserted on our domains had been wiped.

I'm 50/50 yet on whether this is suspicious because im not sure if it could have happened during an upgrade and gone unnoticed.

As you can imagine, this created huge problems for us as our per-domain settings were set to be fairly conservative but were in the most part being completely ignored because the per-user settings were overriding them based on the priority system in ired. We have had our amazon SES account suspended twice now and it seems increasingly difficult to do things by the book so to speak.

For anyone who is interested, the throtling rules are stored in the throttle and throttle_rcpt tables respectively in the policyd mysql table. We have manipulated these tables manually to remove the offending rules and restore the desired ones.

If anyone can offer any explaination to the potential cause and/or permanent fix for this i would be extremely grateful.

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2

Re: default throttling limits

Policyd has default settings in its config file /etc/policyd*.conf, 512 is the default value.
Not sure why your settings were wiped, but upgrading iRedMail doesn't mention any SQL command used to wipe them. No idea at all.

3

Re: default throttling limits

okay thanks, i'll have a look into that.

on a related note then, is there any way that the defaults for the throttling tab can be set for all new domains and/or inboxes? so for example... all new domains have the 'Number of max outgoing emails' set to 100 per hour and all new inboxes have their 'Max size of single outgoing email' set to 10mb?

also i think a feature that might be useful to consider for the future would a checkbox which would allow you to auto-suspend inboxes or domains based on certain thresholds being reached. I dont know if this would be helpful to anyone else but we are quite paranoid about sending limits as we use amazon SES as our SMTP relay and we have had a few cases where throttling has failed and we have been suspended.

4

Re: default throttling limits

bw1984 wrote:

on a related note then, is there any way that the defaults for the throttling tab can be set for all new domains and/or inboxes? so for example... all new domains have the 'Number of max outgoing emails' set to 100 per hour and all new inboxes have their 'Max size of single outgoing email' set to 10mb?

Unfortunately, not available in iRedAdmin-Pro.

bw1984 wrote:

... a checkbox which would allow you to auto-suspend inboxes or domains based on certain thresholds being reached

Currently, if throttling is reached, Policyd/Cluebringer will reject further requests from this user. Is it ok for you?

5 (edited by bw1984 2014-08-12 23:32:19)

Re: default throttling limits

yes i understand.

i think we're a bit closer to understanding the problem but unfortunately we've had a massive compromise again this morning with 45,000 emails being sent from 1 account. Having spent some time last week writing the throttle database entries manually to only apply to domains, it seems that iredmail has once again added entries for some individual inboxes, thus overriding our per-domain settings.

Here is what we think is happening... It seems that when an email is *received* by the server an entry gets added to the throttle table automatically. I presume this is by design in order to throttle inbound email from specific sources. The problem is that it seems that emails sent locally (between 2 accounts on our mail server) are also causing an entry to be added. Because the priority for individual inboxes is higher, and the default settings for an inbox are really high (see screenshot) they make the per-domain rules useless.

If anyone can offer any suggestions, what i need to do is prevent local addresses from getting their throttle limits set automatically by the system. Currently when i wipe the throttle table so it only contains the @domain.com rules, it starts to get written to again almost instantly as emails start to arrive. (see screenshot 2 for default inbox limits being set by the system)

I know a quick fix would be to change the priority values so that the per domain values were ahead of per inbox, but we want something more long term because in some cases we do set certain inboxes to have higher hourly send limits

Post's attachments

igoo_compromise_2014-08-12.png 15 kb, file has never been downloaded. 

igoo_compromise_throttlelimits_2014-08-12.png
igoo_compromise_throttlelimits_2014-08-12.png 10.02 kb, file has never been downloaded. 

You don't have the permssions to download the attachments of this post.

6

Re: default throttling limits

FYI; there are 2 attachments there but the first one is showing only as a link directly above the first.

7 (edited by bw1984 2014-08-12 18:31:17)

Re: default throttling limits

in case anyone is interested, the SQL command im using to flush all non-domain rules from the throttle table is;
DELETE FROM `throttle` WHERE _from NOT LIKE '@%'

8

Re: default throttling limits

Could you please show us output of below commands:

# postconf smtpd_recipient_restrictions
# postconf smtpd_end_of_data_restrictions

9

Re: default throttling limits

# postconf smtpd_recipient_restrictions

smtpd_recipient_restrictions = reject_unknown_sender_domain, reject_unknown_recipient_domain, reject_non_fqdn_sender, reject_non_fqdn_recipient, reject_unlisted_recipient, check_policy_service inet:127.0.0.1:7777, permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination, reject_non_fqdn_helo_hostname, reject_invalid_helo_hostname, check_policy_service inet:127.0.0.1:10031

-------------------------------------------------------

# postconf smtpd_end_of_data_restrictions

smtpd_end_of_data_restrictions = check_policy_service inet:127.0.0.1:10032

10

Re: default throttling limits

bw1984 wrote:

smtpd_recipient_restrictions = ..., permit_sasl_authenticated, ... check_policy_service inet:127.0.0.1:10031

As you can see the specified order, if email was sent by a sasl authenticated user, throttling will be bypassed directly.
So i guess spams were sent by a sasl authenticated user, maybe password is too weak. Please try to find out the sasl username in Postfix log file, then reset its password and keep monitoring.

11

Re: default throttling limits

i know what the source is, thats not a problem. i believe they were sent by a sasl authenticated user (probably our clients machine has malware installed)

are you saying that with my current configuration throttling is essentially disabled for all sasl authenticated users? how do i address this?

my problem is that the throttle table is being populated automatically my iredmail whenever a user sends an email. I dont want to have to set throttle limits for every single inbox but if i dont they automatically receive the throttle settings displayed in my screenshot(s) and automatically take priority over the per-domain settings i have.

currently throttling is useless to me because if i only allow 20 per hour per domain and then ired automatically sets each inbox to 512 per hour, then 512 emails can be sent before we even notice.

12

Re: default throttling limits

so for clarification, basically i have 2 questions now...

- how should i reorder those properly so that throttling is checked for ALL users? (and should i take out the duplicate check_policy_service entries also?)

- how do i prevent the per-inbox throttling rules being written to the 'throttle' table and therefore overriding my per-domain settings?

13

Re: default throttling limits

bw1984 wrote:

- how should i reorder those properly so that throttling is checked for ALL users? (and should i take out the duplicate check_policy_service entries also?)

Please just use iRedMail default setting:

smtpd_recipient_restrictions  = reject_unknown_sender_domain, reject_unknown_recipient_domain, reject_non_fqdn_sender, reject_non_fqdn_recipient, reject_unlisted_recipient, check_policy_service inet:127.0.0.1:7777, check_policy_service inet:127.0.0.1:10031, permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination

bw1984 wrote:

- how do i prevent the per-inbox throttling rules being written to the 'throttle' table and therefore overriding my per-domain settings?

This is working as expected, Policyd/Cluebringer need to log it to track throttling.

14

Re: default throttling limits

what we will have to do then is switch the priorities of per-domain and per-inbox throttling around so that the per-domain setting has a higher priority. currently there is no incentive for anyone to set per-domain throttle values as the per-inbox settings will be created automatically and are much higher than the defaults we set on a domain name. i will update this thread later once i have found a practical solution..

is it okay for me to insert the following lines at the end?

reject_non_fqdn_helo_hostname
reject_invalid_helo_hostname
reject_unknown_helo_hostname

i added them in based on advice i read in this forum and im not sure if they are necessary or required since i cant fully remember  why i added them in the first place.

15

Re: default throttling limits

bw1984 wrote:

is it okay for me to insert the following lines at the end?
reject_non_fqdn_helo_hostname
reject_invalid_helo_hostname
reject_unknown_helo_hostname

It's ok. But in the latest iRedMail, we set them in 'smtpd_helo_restrictions' instead:

smtpd_helo_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_non_fqdn_helo_hostname, reject_invalid_helo_hostname, check_helo_access pcre:/etc/postfix/helo_access.pcre

Note: Not all HELO hostnames are resolvable in DNS, so please be careful with 'reject_unknown_helo_hostname'.

16

Re: default throttling limits

since i changed these settings and removed the lines as you recommended i am getting <> appearing the the top 10 senders list again which makes me nervous. Thinking back im sure that was the reason why i was told to add those extra lines in the first place. any suggestions?

17

Re: default throttling limits

bw1984 wrote:

i am getting <> appearing the the top 10 senders list

<> is called null sender, it's used by Postfix to send non-delivery notification and other notification emails.
It's hard to say whether this is used by spammer or not, you have to check mail log to figure it out.

18

Re: default throttling limits

okay no problem.

I've just noticed your comment about smtpd_helo_restrictions. my line looked like this;

smtpd_helo_restrictions = permit_mynetworks, reject_authenticated_sender_login_mismatch, permit_sasl_authenticated

for some reason, so i have replced it with the one you posted and then removed reject_unknown_helo_hostname from the smtpd_recipient_restrictions

hopefully that will help