1 Topic by broth (edited by broth 2020-08-02 01:33:03)

  • broth
  • Member
  • Offline
  • Registered: 2012-04-22
  • Posts: 145

Topic: iRedMail new install 1.3.1: Autolearn spam/ham uses bad database path

==== REQUIRED BASIC INFO OF YOUR IREDMAIL SERVER ====
- iRedMail version (check /etc/iredmail-release): 1.3.1
- Deployed with iRedMail Easy or the downloadable installer? installer
- Linux/BSD distribution name and version: Debian 10
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): MySQL
- Web server (Apache or Nginx): nginx
- Manage mail accounts with iRedAdmin-Pro? yes
- [IMPORTANT] Related original log or error message is required if you're experiencing an issue.
====

Dear Zhang,

on my new Debian 10 iRedMail 1.3.1 server I implemented the autolean spam/ham feature from https://docs.iredmail.org/dovecot.imapsieve.html

It looks like the cron script does not use the correct spamassassin database path.

Background:

Spam/ham is learned by following script 

# iRedMail: Scan reported mails.
*/10   *   *   *   *   /bin/bash /etc/dovecot/sieve/scan_reported_mails.sh

That script uses

# `sa-learn` command, with optional arguments.
export SA_LEARN="sa-learn -u ${AMAVISD_USER}"

To learn spam but the database is saved to /root/.spamassassin

It should be placed in /var/lib/amavis/.spamassassin

Proposed fix:

Script /etc/dovecot/sieve/scan_reported_mails.sh should use

# `sa-learn` command, with optional arguments.
export SA_LEARN="sa-learn --dbpath /var/lib/amavis/.spamassassin -u ${AMAVISD_USER}"

Tested and working on my installation.

Thanks!
Best regards,
Bernhard

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: iRedMail new install 1.3.1: Autolearn spam/ham uses bad database path

Did you set correct user name in variable "AMAVISD_USER="?
With "-u ${AMAVISD_USER}", sa-learn should use files under Amavisd daemon user's home directory.

3 Reply by broth

  • broth
  • Member
  • Offline
  • Registered: 2012-04-22
  • Posts: 145

Re: iRedMail new install 1.3.1: Autolearn spam/ham uses bad database path

@Zhang:

Yes, the user is correct:

passwd

amavis:x:112:120:AMaViS system user,,,:/var/lib/amavis:/bin/sh

scan_reported_mails.sh

# The Amavisd daemon user.
# Note: on OpenBSD, it's "_vscan". On FreeBSD, it's "vscan".
export AMAVISD_USER='amavis'

According to the man page sa-learn will only act on behalf of the user. I can't find any notes about setting the users home directory as well.

Have you tried to reproduce the issue? Is it working well on your installation?

4 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: iRedMail new install 1.3.1: Autolearn spam/ham uses bad database path

I don’t have such testing environment right now, cannot reproduce the issue.

Could you help run the “sa-learn —debug” and “-u amavis” and check which db path it uses? Also enable debug argument in the scan_reported_mails.sh script and compare the db path.

5 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: iRedMail new install 1.3.1: Autolearn spam/ham uses bad database path

Btw, if you deploy iRedMail server with our iRedMail Easy platform, there’s an option to store bayes in SQL database and no such issue.

6 Reply by broth (edited by broth 2020-08-02 21:07:18)

  • broth
  • Member
  • Offline
  • Registered: 2012-04-22
  • Posts: 145

Re: iRedMail new install 1.3.1: Autolearn spam/ham uses bad database path

ZhangHuangbin wrote:

Could you help run the “sa-learn —debug” and “-u amavis” and check which db path it uses? Also enable debug argument in the scan_reported_mails.sh script and compare the db path.

Done, path is as predicted

root@debmail1:~# sa-learn --debug -u amavis --dump magic 2>/tmp/log 
0.000          0          3          0  non-token data: bayes db version
0.000          0          3          0  non-token data: nspam
0.000          0          0          0  non-token data: nham
0.000          0        906          0  non-token data: ntokens
0.000          0 1596024494          0  non-token data: oldest atime
0.000          0 1596274918          0  non-token data: newest atime
0.000          0          0          0  non-token data: last journal sync atime
0.000          0          0          0  non-token data: last expiry atime
0.000          0          0          0  non-token data: last expire atime delta
0.000          0          0          0  non-token data: last expire reduction count
root@debmail1:~# cat /tmp/log | grep root
Aug  2 15:01:22.257 [16791] dbg: bayes: tie-ing to DB file R/O /root/.spamassassin/bayes_toks
Aug  2 15:01:22.257 [16791] dbg: bayes: tie-ing to DB file R/O /root/.spamassassin/bayes_seen
Aug  2 15:01:22.259 [16791] dbg: bayes: tie-ing to DB file R/O /root/.spamassassin/bayes_toks
Aug  2 15:01:22.259 [16791] dbg: bayes: tie-ing to DB file R/O /root/.spamassassin/bayes_seen
root@debmail1:~#

Same result with debug flag in cron script.

I think there is a need to update your howto page and implement a fix.

7 Reply by broth

  • broth
  • Member
  • Offline
  • Registered: 2012-04-22
  • Posts: 145

Re: iRedMail new install 1.3.1: Autolearn spam/ham uses bad database path

ZhangHuangbin wrote:

Btw, if you deploy iRedMail server with our iRedMail Easy platform, there’s an option to store bayes in SQL database and no such issue.

Thanks but we have no plans to use iRedMail Easy platform. Bayes in filesystem ist just perfect for our usecase.

8 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: iRedMail new install 1.3.1: Autolearn spam/ham uses bad database path

Could you replace the "export SA_LEARN=" line by below one and try again?

export SA_LEARN="sa-learn -u ${AMAVISD_USER} --dbpath ~${AMAVISD_USER}/.spamassassin"

9 Reply by broth

  • broth
  • Member
  • Offline
  • Registered: 2012-04-22
  • Posts: 145

Re: iRedMail new install 1.3.1: Autolearn spam/ham uses bad database path

Looks like its working.
I need to wait for some spam to get classified in order to confirm.

The new iRedMail 1.3.1 is very good in filtering spam actually smile
Thanks Zhang for your great work!

10 Reply by broth (edited by broth 2020-08-03 03:31:23)

  • broth
  • Member
  • Offline
  • Registered: 2012-04-22
  • Posts: 145

Re: iRedMail new install 1.3.1: Autolearn spam/ham uses bad database path

I re-ran the script with a spam message and debug enabled.

Unfortunately it saves data now to /root, files .spamassassin.mutex .spamassassin_seen .spamassassin_toks

So for the meantime my proposal is the only one working on Debian 10.5 by setting --dbpath /var/lib/amavis/.spamassassin

11 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: iRedMail new install 1.3.1: Autolearn spam/ham uses bad database path

broth wrote:

So for the meantime my proposal is the only one working on Debian 10.5 by setting --dbpath /var/lib/amavis/.spamassassin

this is weird. I don’t see difference between “—dbpath ~${AMAVISD_USER}/.spamassassin” and “—dbpath /var/lib/amavis/.spamassassin”.

12 Reply by broth

  • broth
  • Member
  • Offline
  • Registered: 2012-04-22
  • Posts: 145

Re: iRedMail new install 1.3.1: Autolearn spam/ham uses bad database path

This is tricky. I haven't found yet the reason why but that works for me:

# `sa-learn` command, with optional arguments.
export SA_HOME=$(eval echo ~$AMAVISD_USER)
export SA_LEARN="sa-learn -u ${AMAVISD_USER} --dbpath ${SA_HOME}/.spamassassin"

eval is important, without it doesn't work

13 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: iRedMail new install 1.3.1: Autolearn spam/ham uses bad database path

If you copied the command in my previous reply directly, there's incorrect dashes caused by phone auto-correction. sad

14 Reply by broth

  • broth
  • Member
  • Offline
  • Registered: 2012-04-22
  • Posts: 145

Re: iRedMail new install 1.3.1: Autolearn spam/ham uses bad database path

Don't worry, I did enter the command by hand.

My modification from post 12 is still working well on my server.

15 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: iRedMail new install 1.3.1: Autolearn spam/ham uses bad database path

Fixed the script. Thanks for the patience. smile