1 (edited by Quack66 2017-06-20 23:57:18)

Topic: [Tutorial] Increasing ClamAV effectiveness

======================== Required information ====
- iRedMail version (check /etc/iredmail-release): 0.9.6
- Linux/BSD distribution name and version: Ubuntu 16.04 LTS
- Store mail accounts in which backend (LDAP/MySQL/PGSQL):  PGSQL
- Web server (Apache or Nginx): Apache
- Manage mail accounts with iRedAdmin-Pro? Yes
- [IMPORTANT] Related original log or error message is required if you're experiencing an issue.
====

As you probably know, Clamav is fine as an AV but is not on-par with paid commercial product and sometime viruses can slip through.

There is a way to add custom signatures (Databases) that are maintained by 3rd parties to clam to increase detection rate on viruses and 0-days. This tutorial is how to configure this.

The folks over at SaneSecurity are kind enough to host a bunch of different signatures that can be used with clam. Please do not abuse this service and consider making a donation.

They do offer a script to automatically configure everything but I don't recommend it since it might break your installation.

All the signatures can be found here. I've created this tutorial using the main ones. Feel free to modify/remove or use different signatures.

First we need to use freshclam to automatically download the databases. Open the following file /etc/clamav/freshclam.conf and add this at the end:

 
# Sanesecurity + Foxhole
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/junk.ndb
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/jurlbl.ndb
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/phish.ndb
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/rogue.ndb
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/sanesecurity.ftm
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/sigwhitelist.ign2
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/scam.ndb
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/spamimg.hdb
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/spamattach.hdb
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/blurl.ndb
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/foxhole_generic.cdb
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/foxhole_filename.cdb
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/foxhole_js.cdb
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/foxhole_js.ndb
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/foxhole_all.cdb
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/foxhole_all.ndb
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/foxhole_mail.cdb
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/malwarehash.hsb
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/hackingteam.hsb
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/badmacro.ndb
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/shelter.ldb

# winnow
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/winnow_malware.hdb
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/winnow_malware_links.ndb
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/winnow_phish_complete_url.ndb
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/winnow_extended_malware.hdb
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/winnow.attachments.hdb
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/winnow_bad_cw.hdb

# Malware.expert
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/malware.expert.hdb

# bofhland
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/bofhland_cracked_URL.ndb
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/bofhland_malware_URL.ndb
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/bofhland_phishing_URL.ndb
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/bofhland_malware_attach.hdb

# Porcupine
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/porcupine.ndb
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/phishtank.ndb
DatabaseCustomURL http://ftp.swin.edu.au/sanesecurity/porcupine.hsb

You then need to restart freshclam

service clamav-freshclam restart

To make sure the databases were downloaded check your logs in /var/log/clamav/freshclam.log.
You should not have any errors before continuing.

We need to tell amavis to pass the whole message to clam so it could also scan things like the subject.
To do so, open the following file:
/etc/amavis/conf.d/20-debian_defaults
and find the following lines:

@keep_decoded_original_maps = (new_RE(
  #qr'^MAIL$',   # retain full original message for virus checking (can be slow)
  qr'^MAIL-UNDECIPHERABLE$', # recheck full mail if it contains undecipherables
  qr'^(ASCII(?! cpio)|text|uuencoded|xxencoded|binhex)'i,
# qr'^Zip archive data',     # don't trust Archive::Zip
));

You need to uncomment the following line:

#qr'^MAIL$',   # retain full original message for virus checking (can be slow)

Clam will now detect everything based on our custom signature. The issue is that right now it will not know what to do with it since some signatures are spam, malware or even whitelist. We need to edit the following file:
/etc/amavis/conf.d/50-user
and add the following before @av_scanners

# Mark Spam/Virus
@virus_name_to_spam_score_maps =
 (new_RE(  # the order matters, first match wins. Set to 'undef' to keep as infected

  # SaneSecurity + Foxhole
  [ qr'^Sanesecurity\.(Malware|Badmacro|Foxhole|Rogue|Trojan)\.' => undef ],
  [ qr'^Sanesecurity\.MalwareHash\.'                  => undef ],
  [ qr'^Sanesecurity.TestSig_'                           => undef ],
  [ qr'^Sanesecurity\.'                                  => 0.1 ],

  # winnow
  [ qr'^winnow\.(Exploit|Trojan|malware)\.'           => undef ],
  [ qr'^winnow\.(botnet|compromised|trojan)'          => undef ],
  [ qr'^winnow\.(exe|ms|JS)\.'                        => undef ],
  [ qr'^winnow\.phish\.'                              => 30 ],
  [ qr'^winnow\.'                                     => 0.1 ],

  # bofhland
  [ qr'^Bofhland\.Malware\.'                          => undef ],
  [ qr'^BofhlandMWFile'                               => undef ],
  [ qr'^Bofhland\.Phishing\.'                         => 30 ],
  [ qr'^Bofhland\.'                                   => 0.1 ],

  # porcupine.ndb
  [ qr'^Porcupine\.(Malware|Trojan)\.'                => undef ],
  [ qr'^Porcupine\.(Junk|Spammer)\.'                  => 30 ],
  [ qr'^Porcupine\.Phishing\.'                        => 30 ],
  [ qr'^Porcupine\.'                                  => 0.01 ],

  # phishtank.ndb
  [ qr'^PhishTank\.Phishing\.'                        => 30 ],

  # Others
  [ qr'^Structured\.(SSN|CreditCardNumber)\b'            => 0.1 ],
  [ qr'^(Heuristics\.)?Phishing\.'                       => 0.1 ],
  [ qr'^(Email|HTML)\.Phishing\.(?!.*Sanesecurity)'      => 0.1 ],
  [ qr'^Email\.Spam\.Bounce(\.[^., ]*)*\.Sanesecurity\.' => 0   ],
  [ qr'^Email\.Spammail\b'                               => 0.1 ],
  [ qr'^MSRBL-(Images|SPAM)\b'                           => 0.1 ],
  [ qr'^VX\.Honeypot-SecuriteInfo\.com\.Joke'            => 0.1 ],
  [ qr'^VX\.not-virus_(Hoax|Joke)\..*-SecuriteInfo\.com(\.|\z)' => 0.1 ],
  [ qr'^Email\.Spam.*-SecuriteInfo\.com(\.|\z)'          => 0.1 ],
  [ qr'^Safebrowsing\.'                                  => 0.1 ],
  [ qr'^INetMsg\.SpamDomain'                             => 0.1 ],
  [ qr'^Doppelstern\.(Spam|Scam|Phishing|Junk|Lott|Loan)'=> 0.1 ],
  [ qr'^ScamNailer\.'                                    => 0.1 ],
  [ qr'^HTML/Bankish'                                    => 0.1 ],
  [ qr'^MBL_NA\.UNOFFICIAL'                              => 0.1 ],
  [ qr'^MBL_'                                            => undef ],
));

Note: I'm setting the spam score to 30 to make sure they are set to spam even if they have other valid scores. Anything declared as undefined will be marked as a virus

To apply everything we need to restart amavis

service amavis restart

That's it !

To test everything, follow the instructions here. All 3 tests should be marked as Viruses if everything was done correctly.

2

Re: [Tutorial] Increasing ClamAV effectiveness

Thanks for sharing, i will check the tutorials later to see whether we can integrate it into iRedMail directly. smile

----

Does my reply help a little? How about buying me a cup of coffee ($5) as an encouragement?

buy me a cup of coffee

3

Re: [Tutorial] Increasing ClamAV effectiveness

Dear ZhangHuangbin, is IRedMail use the clamscan command to check mail or clamdscan?
In my system (IRedMail-0.9.6, CentOS 6.7, MySQL) it shows different results:

# clamdscan test.msg
/tmp/test.msg: OK

----------- SCAN SUMMARY -----------
Infected files: 0
Time: 0.001 sec (0 m 0 s)

or

# clamscan test.msg
test.msg: Sanesecurity.TestSig_Type3_Bdy.4.UNOFFICIAL FOUND

----------- SCAN SUMMARY -----------
Known viruses: 6449849
Engine version: 0.99.2
Scanned directories: 0
Scanned files: 1
Infected files: 1
Data scanned: 0.00 MB
Data read: 0.00 MB (ratio 0.00:1)
Time: 21.308 sec (0 m 21 s)

4

Re: [Tutorial] Increasing ClamAV effectiveness

Amavisd is configured to pipe email to ClamAV socket, not running clam* commands directly.

----

Does my reply help a little? How about buying me a cup of coffee ($5) as an encouragement?

buy me a cup of coffee

5

Re: [Tutorial] Increasing ClamAV effectiveness

clamdscan command use ClamAV daemon. In my case to enable SaneSecurity signatures to check I was need to set

OfficialDatabaseOnly  no

in clamd.conf
Thanks a lot to topicstarter for a useful idea!

6

Re: [Tutorial] Increasing ClamAV effectiveness

According to the Donate page of Sanesecurity website, I'm hesitate to integrate them:
http://sanesecurity.com/donate/

iRedMail has few hundred deployments every day, i believe 99.9% users will NOT notice that Sanesecurity asks for a donation for personal and commercial use (it's free to use, but a donation is appreciated). So our integration brings them nothing, except network traffic and possible workload.

Any suggestions?

----

Does my reply help a little? How about buying me a cup of coffee ($5) as an encouragement?

buy me a cup of coffee

7

Re: [Tutorial] Increasing ClamAV effectiveness

IMHO people who really want to maintain his servers are able to integrate such services by himself. And IMHO your decision is right.

8

Re: [Tutorial] Increasing ClamAV effectiveness

Hey Zhang,

I think those points are all valid. In my opinion the best way to approach this is maybe not to add it to the base but instead add a doc on how to do so with a disclaimer at the top.

Like Serge said, anyone maintaining their own server can integrate this easily.

Regards

9

Re: [Tutorial] Increasing ClamAV effectiveness

Would also be nice to an alternative for the Sanesecurity hosted signatures. Just so you have a choice once Sanesecurity decides to charge for it, etc.

10

Re: [Tutorial] Increasing ClamAV effectiveness

Quack66 wrote:

======================== Required information ====
- iRedMail version (check /etc/iredmail-release): 0.9.6
- Linux/BSD distribution name and version: Ubuntu 16.04 LTS
- Store mail accounts in which backend (LDAP/MySQL/PGSQL):  PGSQL
- Web server (Apache or Nginx): Apache
- Manage mail accounts with iRedAdmin-Pro? Yes
- [IMPORTANT] Related original log or error message is required if you're experiencing an issue.
====

As you probably know, Clamav is fine as an AV but is not on-par with paid commercial product and sometime viruses can slip through.

Thanks to Quack66, the OP, for this tutorial - it worked well for me !

11

Re: [Tutorial] Increasing ClamAV effectiveness

awkpavo wrote:

Would also be nice to an alternative for the Sanesecurity hosted signatures. Just so you have a choice once Sanesecurity decides to charge for it, etc.

Copying the Sanesecurity files and hosting on your own server doesn't change the fact: you use their work, but return nothing.

----

Does my reply help a little? How about buying me a cup of coffee ($5) as an encouragement?

buy me a cup of coffee

12 (edited by Quack66 2017-06-01 03:50:20)

Re: [Tutorial] Increasing ClamAV effectiveness

Updates:
- Removed SecurityInfo from the list of DBs since they are no longer hosted on SaneSecurity. Check their website here to add their DB. You will need to register

- Removed Malware patrol since the you need to register for free version and they are no longer hosted on SaneSecurity