It may be caused by weak password. That's why iRedMail forces users to use complex password while changing password with Roundcube webmail. A complex password will help avoid this issue.

Policyd or Cuebringer (a.k.a. Policyd v2) can do this kind of throttling, no need to develop a new one.
The latest iRedAdmin-Pro supports throttling management with Policyd-1.8, but not Cluebringer. Upcoming release supports Cluebringer.

May i know how you detect this kind of misuse? Your detailed thought and process.

@hferreira: thanks, I started to think, I'm the only one facing such situations Pls feel free to share your Ideas/experience.

@Zhang: I'm not sure, what is the best way to detect such situation, but I see on my server following symptoms:

1) many different IPs are using this account (botnet?)
2) a lot of bounces coming to this account (the distribution list is poor quality),
3) statistically, the account becomes the most active one on the server in short time,
4) load of server is increased, but nothing exceptional / on my server the queue length is not so huge (< 100)

Note, the sent rate of emails is not always huge (just few emails per minute to several per 10 minutes), average is around 1 email per minute. But it's probably bcouse I have set limits of emails sent per specified amount of time (smtpd_client_connection_*_limit
).

I think, the easiest is to detect 1) and definitely is something wrong if one account is used from many IPs.

To be sure, it would be nice to implement one more symptom (in case the attack will come from one or just few hosts) / the statistics seems to be the second in the queue. The rough algorithm can eg. mark accounts which sent more than double (?) of average emails sent per specified period of time.

I'm not sure, about the design of solution - maybe create new iRedApD plugin?

Just thinking aloud...

Thanks @Zhang... nice... going forward:

To the design:
I personally don't like fail2ban much (I have it turned off currently), but of course it's possible to code it as you suggest. I would prefer the iRedApD plugin OR turn on the existing sql_log_smtp_session_info and analyze generated log in SQL from cron job.
It's also possible to add simple SQL (just single SELECT distinct(ip) ...) in separate plugin and get number of distinct IPs in last eg. 10 minutes and act if the result is exceptional.
I'm not sure, if the overhead would't be too much however - My sever has quite low mail traffic and a lot of spare performance.

To the symptoms:
- what exactly do you need? I have logs available from last 2 incidents and am able to analyze it...

@hferreira good catch!

The cluebringer.session_tracking table should have all the data needed to detect such situation.
What's weird is, my system stopped logging saslusername on third of July. Have no idea about the reason...

@Zhang: did you changed something about the logging into the session_tracking table in cluebringer? Is it ment to not store sasl information?

Just the action taken should be configurable (send email, create ticket, temporarily block the account, whatever).

Creating the SELECT statement to get suspicious accounts is the piece of cake - seems best to put it in cron job. Nice!

Hi Zhang,

noprob, just relax a bit - you definitely deserve it!

Btw. I fixed the main.cf / I run customized one and I didn't check authenticated emails with cluebringer. Now I do in sender checks right after iRedApD... hope I didn't turn into openrelay :-D

Well, back to topic. I have some progress: as we have all the data about sent emails via registered accounts, below are my first checks I use on my server.

Any notes/questions/enhancements are more than welcome!
Note: I use PgSQL backend, thus all SQL is tested on postgresql 9.1 All statements are executed in cluebringer database.

1) number of different IPs from which is account used. Trigger is more than 5 IPs in last hour.

SELECT saslusername, COUNT(DISTINCT clientaddress)
FROM session_tracking
WHERE saslusername <> '' AND to_timestamp(timestamp) + INTERVAL '+1 hour' > NOW()
GROUP BY saslusername
HAVING COUNT(DISTINCT clientaddress) > 5;

2) total amount of emails sent. This one needs some more work! Now I use as trigger more than 100 mails in last hour.

SELECT saslusername, count(*)
FROM session_tracking
WHERE saslusername <> '' AND to_timestamp(timestamp) + INTERVAL '+1 hour' > NOW()
GROUP BY saslusername
HAVING COUNT(*) > 100

Hi Zhang,

yes, i run both queries on my system every 10 minutes... so far the action is only email notification. We'll see...

Btw. I found out, there is no cleanup of the cluebringer database. The queries are quite cheap, but anyway it can be potential problem.
This affects all *_tracking tables... the session_tracking is the most problematic (~1M rows on my system), greylisting and checkhelo around 10k both.

Do you think it's necessary and/or useful to keep all the historic data or should we create a cleanup procedure?

@camel1cz: How do you run the scripts? In a bash script and just redirect the output to an email or file?

We use a small Python program which queries the amavisd database, sends an email to our postmaster and uses ldapmodify to automatically disable the offending account.   We're looking for THRESHOLD messages sent over INTERVAL and call the script through cron.   The lookup is cheap as we keep the database cleaned out with other cron jobs.

It has worked very successfully for us.

The RBLs are not concerned if you occasionally have a compromised account.   They are far more interested in how quickly you respond and correct the problem.

One caveat.   We run a bit of a strict service and do not allow our clients to use their email for high volume broadcasts.  This is by published policy and we enforce it strictly.  We refer clients to MailChimp or ConstantContact for broadcast mail service.

In last days I'd faced with the same problem with compromised account. Is it possible with fail2ban to count connections per user and if there is huge amount of connections from different IP's for a small period of time to add it in iptables ?

I agree with riverco in that counting connections is probably not the desired approach.

Here is the system which we have used quite successfully.

Use Fail2ban to block FAILED connection attempts.  This will take care of brute force attacks against your mail/web surfaces.

In Fail2ban, manually white list customer IPs for their office where a large number of users access your mail system from a single location.  This will prevent one errant user from blocking everyone in their company when they get a new device and can't remember their email password.

Monitor the Sent message count over a specific relatively short time period in the Amavisd database.   For this we use a custom Python script run via cron.  A compromised account will invariably show a spike in outbound message traffic.   Spammers are all about volume and are very impatient.

Ensure that you are following RFCs by enabling and monitoring the ABUSE and POSTMASTER addresses.  We set these up as aliases for our own mail admin for every mail domain we host.   If everything else is configured properly, you will have near zero traffic to your admin addresses - except when you have a problem.  If an account were to be compromised and was sending a slow, steady stream of spam to avoid the amavisd monitor, you would receive reports back and can take manual action.

Automatically disable compromised accounts by running the monitor frequently via cron.

Ensure that the amavisd database is kept cleaned out to prevent the monitor from becoming an expensive test and slowing your entire system down.  Zhang Huangbin has posted the scripts to do this effectively.

Use qshape to especially monitor the Deferred queue.  A spike in the deferred queue is typically a sign of a compromised account.  Qshape is your best friend in monitoring the health of your mail system.   We have cron jobs which send regular reports to our mail admins and monitor it manually frequently when there appear to be performance or other mail issues.   As a side note, qshape is very cheap to run.   It would be nice if this were part of the iRedAdminPro dashboard stats.    Our cron jobs print out the current Active, Incoming, Hold and Deferrred Queue and then print the top senders in the deferred queue.

Manually inspect the top sender list at least daily in the iRedAdmin Pro panel so you can watch for unusual traffic patterns.

We try to automate as much of the monitoring process as we can, given that we get busy with other tasks and have better things to do with our work day than manually watch mail logs.   This model has worked very well for us thus far.