1

Topic: Compromised accounts - prevention

==== Required information ====
- iRedMail version: 0.8.5
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): PgSQL
- Linux/BSD distribution name and version: Debian 7
- Related log if you're reporting an issue:
====

Hi Zhang and everyone interested in this topic.

Recently several accounts on my server was compromised (dunno how, maybe troyan) and misused to send a lot of spam.

This is causing a lot of problems - you know, server gets blacklisted, a lot of emails not delivered, a lot of angry customers, solution is damn slow a lot of headache...

The question is - how to do some prevention? How do you solve such situations?

The only solution I see is to summarize number of emails sent per each account and if the number exceeds some threshold - change password and notify administrator.

What do you think? Zhang, what do you think about this topic? Would you agree to incorporate something like this in iRedMail? ...and if so, how would you design such a solution?
I'm ready to do some coding and would be happy to give the code back under BSD licence.

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2

Re: Compromised accounts - prevention

camel1cz wrote:

The question is - how to do some prevention? How do you solve such situations?

It may be caused by weak password. That's why iRedMail forces users to use complex password while changing password with Roundcube webmail. A complex password will help avoid this issue.

camel1cz wrote:

The only solution I see is to summarize number of emails sent per each account and if the number exceeds some threshold - change password and notify administrator.

What do you think? Zhang, what do you think about this topic? Would you agree to incorporate something like this in iRedMail? ...and if so, how would you design such a solution?
I'm ready to do some coding and would be happy to give the code back under BSD licence.

Policyd or Cuebringer (a.k.a. Policyd v2) can do this kind of throttling, no need to develop a new one.
The latest iRedAdmin-Pro supports throttling management with Policyd-1.8, but not Cluebringer. Upcoming release supports Cluebringer.

3

Re: Compromised accounts - prevention

ZhangHuangbin wrote:

It may be caused by weak password.

The password is usually stolen by a virus. All users have generated passwords with 6 to 8 random characters/numbers and usually don't change them.
Complex passwords are definitely good idea but are by itself not though enough prevention.

ZhangHuangbin wrote:

Policyd or Cuebringer (a.k.a. Policyd v2) can do this kind of throttling, no need to develop a new one.
The latest iRedAdmin-Pro supports throttling management with Policyd-1.8, but not Cluebringer. Upcoming release supports Cluebringer.

It's not exactly what I try to achive / I need to detect compromised account and force the legitimate user check his/her computer for viruses and give them new password.

I don't like the idea to allow spammer to send eg. 100 emails per hour - it would be even harder to detect this misuse and the possibility to get banned by a blacklist is still huge.

Zhang, do you like similar functionality or you think, it's useless for your project?

Speaking for myself, I definitely need such a prevention - this problem popups at least every month.

4

Re: Compromised accounts - prevention

May i know how you detect this kind of misuse? Your detailed thought and process.

5

Re: Compromised accounts - prevention

In my old server (not iRedMail) i would detect it with postfix mail queue filling up pretty fast. Nagios would warn me about it.

@Zhang: It can also be a computer infected with a trojan that uses whatever definitions available on Outllok. It happens a lot really.

6

Re: Compromised accounts - prevention

@hferreira: thanks, I started to think, I'm the only one facing such situations smile Pls feel free to share your Ideas/experience.

@Zhang: I'm not sure, what is the best way to detect such situation, but I see on my server following symptoms:

1) many different IPs are using this account (botnet?)
2) a lot of bounces coming to this account (the distribution list is poor quality),
3) statistically, the account becomes the most active one on the server in short time,
4) load of server is increased, but nothing exceptional / on my server the queue length is not so huge (< 100)

Note, the sent rate of emails is not always huge (just few emails per minute to several per 10 minutes), average is around 1 email per minute. But it's probably bcouse I have set limits of emails sent per specified amount of time (smtpd_client_connection_*_limit
).

I think, the easiest is to detect 1) and definitely is something wrong if one account is used from many IPs.

To be sure, it would be nice to implement one more symptom (in case the attack will come from one or just few hosts) / the statistics seems to be the second in the queue. The rough algorithm can eg. mark accounts which sent more than double (?) of average emails sent per specified period of time.

I'm not sure, about the design of solution - maybe create new iRedApD plugin?

Just thinking aloud...

7

Re: Compromised accounts - prevention

Thank you very much, @hferreira and @camel1cz. I agree we need to do something to prevent this issue.

The first thing we should do is clearly know what will happen if this issue occurs, like @camel1cz said, "symptoms".
Then we can try to monitor log files (or other operations if necessary) to track it.

Currently, i guess we can try to use Fail2ban to monitor log files, and create a new Fail2ban action to deal with caught log entries (of course not submit to iptables). For example, log in to SQL database, analyze logged data via cron job every 5 minutes, notify admin if we found some unexpected behaves.

iRedAPD just handles SMTP session data, since most symptoms appear in Postfix/Dovecot log files, i don't think it's useful to help solve this issue. But it's too early to say this, let's see what we got later.

8

Re: Compromised accounts - prevention

Thanks @Zhang... nice... going forward:

To the design:
I personally don't like fail2ban much (I have it turned off currently), but of course it's possible to code it as you suggest. I would prefer the iRedApD plugin OR turn on the existing sql_log_smtp_session_info and analyze generated log in SQL from cron job.
It's also possible to add simple SQL (just single SELECT distinct(ip) ...) in separate plugin and get number of distinct IPs in last eg. 10 minutes and act if the result is exceptional.
I'm not sure, if the overhead would't be too much however - My sever has quite low mail traffic and a lot of spare performance.

To the symptoms:
- what exactly do you need? I have logs available from last 2 incidents and am able to analyze it...

9

Re: Compromised accounts - prevention

It's all about reading the postfix logs and detecting patterns. If, for example, joe@domain.tld sends 50 emails in an 2 minutes span something is up. It may be just a newsletter or it could be an infected machine sending spam.

Because of the throttle feature, Policyd/cluebringer already logs in their database every email sent by every user. So, maybe, there is a way to write a plugin in iRedAPD to query the policyd DB, to check how many emails has a specific email address sent in the last 2 minutes (for example). If that value is above, let's say, 50 emails send an email to the postmaster of the server.

I think that the default behaviour of this or similar filter should just warn and not block, because it could happen that an user is just sending a newsletter. The server admin should check it and see what measures, if any, must be taken.

10

Re: Compromised accounts - prevention

@hferreira good catch!

The cluebringer.session_tracking table should have all the data needed to detect such situation.
What's weird is, my system stopped logging saslusername on third of July. Have no idea about the reason...

@Zhang: did you changed something about the logging into the session_tracking table in cluebringer? Is it ment to not store sasl information?

Just the action taken should be configurable (send email, create ticket, temporarily block the account, whatever).

Creating the SELECT statement to get suspicious accounts is the piece of cake - seems best to put it in cron job. Nice!

11

Re: Compromised accounts - prevention

Hi @camel1cz,

I didn't change Cluebringer settings, especially on your server which i don't have access privilege. smile

I'm so sorry that i have very limit internet access these days, should be fine after Nov 23 (i will be back home). I can reply forum topics if there's something i can help, will catch up later. Sorry about this.

I guess you guys are close to success, well done. smile

12

Re: Compromised accounts - prevention

Hi Zhang,

noprob, just relax a bit - you definitely deserve it!

Btw. I fixed the main.cf / I run customized one and I didn't check authenticated emails with cluebringer. Now I do in sender checks right after iRedApD... hope I didn't turn into openrelay :-D

Well, back to topic. I have some progress: as we have all the data about sent emails via registered accounts, below are my first checks I use on my server.

Any notes/questions/enhancements are more than welcome!
Note: I use PgSQL backend, thus all SQL is tested on postgresql 9.1 All statements are executed in cluebringer database.

1) number of different IPs from which is account used. Trigger is more than 5 IPs in last hour.

SELECT saslusername, COUNT(DISTINCT clientaddress)
FROM session_tracking
WHERE saslusername <> '' AND to_timestamp(timestamp) + INTERVAL '+1 hour' > NOW()
GROUP BY saslusername
HAVING COUNT(DISTINCT clientaddress) > 5;

2) total amount of emails sent. This one needs some more work! Now I use as trigger more than 100 mails in last hour.

SELECT saslusername, count(*)
FROM session_tracking
WHERE saslusername <> '' AND to_timestamp(timestamp) + INTERVAL '+1 hour' > NOW()
GROUP BY saslusername
HAVING COUNT(*) > 100

13

Re: Compromised accounts - prevention

Hi @camel1cz,

Is it possible for you to run these SQL queries everyday? Let's see whether it's enough to predict the spamming.

14

Re: Compromised accounts - prevention

Hi Zhang,

yes, i run both queries on my system every 10 minutes... so far the action is only email notification. We'll see...

Btw. I found out, there is no cleanup of the cluebringer database. The queries are quite cheap, but anyway it can be potential problem.
This affects all *_tracking tables... the session_tracking is the most problematic (~1M rows on my system), greylisting and checkhelo around 10k both.

Do you think it's necessary and/or useful to keep all the historic data or should we create a cleanup procedure?

15

Re: Compromised accounts - prevention

Missing cron job to cleanup cluebringer database is a bug in iRedMail, fixed in upcoming release, and will cover this in upgrade tutorial.

16

Re: Compromised accounts - prevention

@camel1cz: How do you run the scripts? In a bash script and just redirect the output to an email or file?

17

Re: Compromised accounts - prevention

I have a simple PHP CLI (run from bash) application which executes the queries and also sends the email if necessary...

If you're interested - I will share the code.

(I'm sorry for the language, but I have with PHP a lot of experience)

18

Re: Compromised accounts - prevention

We use a small Python program which queries the amavisd database, sends an email to our postmaster and uses ldapmodify to automatically disable the offending account.   We're looking for THRESHOLD messages sent over INTERVAL and call the script through cron.   The lookup is cheap as we keep the database cleaned out with other cron jobs.

It has worked very successfully for us.

The RBLs are not concerned if you occasionally have a compromised account.   They are far more interested in how quickly you respond and correct the problem.

One caveat.   We run a bit of a strict service and do not allow our clients to use their email for high volume broadcasts.  This is by published policy and we enforce it strictly.  We refer clients to MailChimp or ConstantContact for broadcast mail service.

19 (edited by camel1cz 2013-12-13 01:17:36)

Re: Compromised accounts - prevention

Thank you for your feedback and sharing your experience - I really appreciate it!

I have the very same ideas and user policy for sending emails - bulk emailing is another service and is prohibited on our servers.

Now I'm considering to left out the second check (limit of distinct IPs using the account over some interval) - to keep things simple.
The IP check reports the breach much faster - but still the difference seems negligible.

20

Re: Compromised accounts - prevention

In last days I'd faced with the same problem with compromised account. Is it possible with fail2ban to count connections per user and if there is huge amount of connections from different IP's for a small period of time to add it in iptables ?

21 (edited by riverco 2013-12-13 02:59:20)

Re: Compromised accounts - prevention

Hi,

just wanted to give my little input there is however a big issue with the logic.
Although it might work it will also in my case and some other people i know completely break functionality if all you do is check on connections and ip.
Example : i use 2 phones 2 tables 3 pc's on all of them i let the client run (i would really hate being forced suddenly to turn off or logout clients). this already increase amount of ips ...
Clients like k9 mail, k@ mail etc open  10-30 connections depending on the amount of imap folders you have (i have lots of folders), this also would be an issue restricting.
This would trigger alerts or even block me , while i would be a legit user, i do not think  using only ip count and connection count is good UNLESS you build in logic like allow all localnet but LIMITED extranet etc..
(i also strongly suggest proper use of proxies and firewalls and not only count on the mail-server to handle this properly..i use a very well known firewall and mail proxy which is free for noncommercial use but there are plenty).
Please if you enable any of such allow us to kill/disable this feature it would seriously break functionality if localnet ip's aren't unrestricted with unrestricted count.
In conclusion you need more then just blocking on ip or connections (i would highly suggest using vpn to emailserver emailserver on non routeable ip and leave security to the firewall and vpnserver , trying to add lots of restrictions to the emailserver usually leads to breaking allot on the emailserver, VPN's and firewalls are harder to hack then a plain open emailserver and have lots of security features including geoblocking) else please add more logic to just ip and connection count.....
my two little cents and thoughts.
@Zhang please allow us to also turn cluebringer off (in my case allot of security like this is handles by astaro who sits in between the world and mail server)
thank you

22 (edited by bmackay 2013-12-13 23:14:06)

Re: Compromised accounts - prevention

netizen wrote:

In last days I'd faced with the same problem with compromised account. Is it possible with fail2ban to count connections per user and if there is huge amount of connections from different IP's for a small period of time to add it in iptables ?

I agree with riverco in that counting connections is probably not the desired approach.

Here is the system which we have used quite successfully.

Use Fail2ban to block FAILED connection attempts.  This will take care of brute force attacks against your mail/web surfaces.

In Fail2ban, manually white list customer IPs for their office where a large number of users access your mail system from a single location.  This will prevent one errant user from blocking everyone in their company when they get a new device and can't remember their email password.

Monitor the Sent message count over a specific relatively short time period in the Amavisd database.   For this we use a custom Python script run via cron.  A compromised account will invariably show a spike in outbound message traffic.   Spammers are all about volume and are very impatient.

Ensure that you are following RFCs by enabling and monitoring the ABUSE and POSTMASTER addresses.  We set these up as aliases for our own mail admin for every mail domain we host.   If everything else is configured properly, you will have near zero traffic to your admin addresses - except when you have a problem.  If an account were to be compromised and was sending a slow, steady stream of spam to avoid the amavisd monitor, you would receive reports back and can take manual action.

Automatically disable compromised accounts by running the monitor frequently via cron.

Ensure that the amavisd database is kept cleaned out to prevent the monitor from becoming an expensive test and slowing your entire system down.  Zhang Huangbin has posted the scripts to do this effectively.

Use qshape to especially monitor the Deferred queue.  A spike in the deferred queue is typically a sign of a compromised account.  Qshape is your best friend in monitoring the health of your mail system.   We have cron jobs which send regular reports to our mail admins and monitor it manually frequently when there appear to be performance or other mail issues.   As a side note, qshape is very cheap to run.   It would be nice if this were part of the iRedAdminPro dashboard stats.    Our cron jobs print out the current Active, Incoming, Hold and Deferrred Queue and then print the top senders in the deferred queue.

Manually inspect the top sender list at least daily in the iRedAdmin Pro panel so you can watch for unusual traffic patterns.

We try to automate as much of the monitoring process as we can, given that we get busy with other tasks and have better things to do with our work day than manually watch mail logs.   This model has worked very well for us thus far.

23 (edited by camel1cz 2013-12-14 00:23:06)

Re: Compromised accounts - prevention

Agree with @bmackay but the fail2ban / I have troubles keeping the list of "trusted IPs of customer offices" up2date (people don't call right away, complaining to marketing people which don't know what's going on etc.).
Currently I don't use fail2ban on my server (luckily don't see any huge brute force password cracking).

I would love to use fail2ban, but it should use smarter logic to catch the real cracker. Maybe use some kind of AWL what's not job for fail2ban anymore :-)

Didn't know about the qshape, thanks!

24

Re: Compromised accounts - prevention

riverco wrote:

@Zhang please allow us to also turn cluebringer off (in my case allot of security like this is handles by astaro who sits in between the world and mail server)

You can remove Cluebringer/Policyd support in Postfix (remove check_policy_service inet:127.0.0.1:1003X in main.cf), then disable Cluebringer/Policyd service.