iRedMail — Spam tagging and spam threshold

Re: Spam tagging and spam threshold

null@example.com (cvcvelo) — Sat, 19 Feb 2022 15:21:58 +0000

meszaros.karoly wrote:

Does anyone have any suggestions?

I migrated my iRedMail box to Debian and have not had this problem there. Sorry, can't comment on the correctness of the data typing change.

Re: Spam tagging and spam threshold

null@example.com (meszaros.karoly) — Sat, 19 Feb 2022 15:10:31 +0000

ZhangHuangbin wrote:

nuwinfo wrote:
Are there any news here?
No plan to switch to rspamd shortly. sorry.
Amavisd-new is still active: https://gitlab.com/amavis/amavis

I have the same problem.
amavisd-new-2.12.2, perl5 (revision 5 version 34 subversion 0), 10.5.15-MariaDB, for FreeBSD13.0

I wrote a "check" perl script that reads the policy table values. Here the float values are displayed correctly (3.000 , 6.900, etc). However, the amavisd-new log shows spam_tag_level=>"0", so I suspect a faulty value conversion for amavisd (and not due to p5-DBD-mysql).

As a workaround, I changed the float type in the database to varchar and -for now- it seems fine.

Am I wrong? Does anyone have any suggestions?

Regard
k

Re: Spam tagging and spam threshold

null@example.com (ZhangHuangbin) — Mon, 04 Nov 2019 04:37:53 +0000

nuwinfo wrote:

Are there any news here?

No plan to switch to rspamd shortly. sorry.
Amavisd-new is still active: https://gitlab.com/amavis/amavis

Re: Spam tagging and spam threshold

null@example.com (nuwinfo) — Thu, 31 Oct 2019 13:57:18 +0000

ZhangHuangbin wrote:

rspamd is on my radar. If no more development/maintenance happened to Amavisd-new, we may switch to rspamd in 2019.

Are there any news here?

Re: Spam tagging and spam threshold

null@example.com (ZhangHuangbin) — Tue, 09 Oct 2018 08:39:02 +0000

rspamd is on my radar. If no more development/maintenance happened to Amavisd-new, we may switch to rspamd in 2019.

Re: Spam tagging and spam threshold

null@example.com (cvcvelo) — Mon, 08 Oct 2018 17:35:19 +0000

An update: 30 days ago I filed bug 231250 against amavisd-new in the FreeBSD bugzilla database, and contacted the port maintainer about a week before that. There's been no response to either post.

Since then there has been a long thread on the amavisd-new mailing list concluding that further development of amavisd-new is dead.

Postfix guru Ralf Hildebrandt noted that his organization, a German telecommunications company, recently migrated away from amavisd-new and now uses rspamd instead:

https://github.com/rspamd/rspamd

I have no experience with rspamd. What I can say at this point is that:

(1) the root cause of the bad tags is in the way amavisd-new handles perl data types such as perl doubles;

(2) the likelihood of a fix in amavisd-new is approximately 0.00 percent; and

(3) it's probably time to start looking at alternatives to amavisd-new.

Thanks!

Re: Spam tagging and spam threshold

null@example.com (ZhangHuangbin) — Sun, 02 Sep 2018 15:38:47 +0000

You'd better file bugs for all.

Re: Spam tagging and spam threshold

null@example.com (cvcvelo) — Sun, 02 Sep 2018 15:28:44 +0000

Thanks, ZHB. I’m doing option 1 now, and just yesterday upgraded ports using “portmaster -aD -x p5-DBD-mysql” — not a great option considering 4.037 has other security vulnerabilities.

I’m already doing option 4, but so far have not heard anything back from the port maintainer or the amavisd mailing list. I may also file a bug report against one or both packages in the FreeBSD bugzilla bug tracker.

BTW there’s also this thread claiming it’s an amavisd problem:

https://de.postfix.org/pipermail/amavis … 04711.html

Quoting: "The change is that [as of 4.038] p5-DBD-mysql now returns mysql doubles as perl doubles and mysql floats as perl floats (and not as a string anymore). This should be adressed [sic] by amavisd."

I don’t know which port is at fault but will files bugs against both since the bottom line is that one of them returns bad data.

Thanks again.

Re: Spam tagging and spam threshold

null@example.com (ZhangHuangbin) — Sun, 02 Sep 2018 15:09:44 +0000

- Both 1+3 will cause some issue during package upgrade. you have to skip this package.
- #2 is ok, but if this is a production server, the cost may be higher than 1 or 3.

Here's option 4:

Contact software developer and package maintainer, push them to fix it and release a new version.

Re: Spam tagging and spam threshold

null@example.com (cvcvelo) — Mon, 27 Aug 2018 15:36:02 +0000

After much research, I now believe the root cause of the tagged_above=0 headers is a type-conversion bug in the p5-DBD-mysql module:

https://github.com/perl5-dbi/DBD-mysql/issues/78

https://lists.amavis.org/pipermail/amav … 04674.html

This explains why user amavisd gets correct results in MySQL queries but iRedMail lookups get incorrect (0) results.

My options are these:

1. Revert to the old p5-DBD-mysql-4.037 module, the last version not to have the type-conversion bug. I have verified that this version does not have the type-conversion bug, and returns expected values from MySQL lookups. However, this is a quite old version (current on FreeBSD is 4.046) and has several other security vulnerabilities.

2. Switch away from MySQL or MariaDB to PostgreSQL. OpenLDAP isn't an option for me as I don't know it, and even switching between SQL databases is, for me, something like changing engines on a jet airline while it's in the air.

3. Stay on the current broken 4.046 module and live with a some false positives.

Of these three options, which would you recommend?

Thanks!

Re: Spam tagging and spam threshold

null@example.com (cvcvelo) — Fri, 01 Jun 2018 23:04:30 +0000

OK, thanks. Please let me know if you need additional details about the amavisd config or database settings.

The final post in this thread suggests there's a change in perl or Amavisd which affected SQL queries, but provides no useful details:

https://www.howtoforge.com/community/th … pam.75725/

Certainly there is a problem with the tagged_above=0 label, but I don't know if that's the only reason for the false positives, or even if it's related.

Re: Spam tagging and spam threshold

null@example.com (ZhangHuangbin) — Fri, 01 Jun 2018 08:09:06 +0000

cvcvelo wrote:

What else might explain this? Thanks.

I tried to figure it out on another production server, but no result yet. It looks like a Amavisd bug, but not sure.

Re: Spam tagging and spam threshold

null@example.com (cvcvelo) — Thu, 31 May 2018 04:02:59 +0000

ZhangHuangbin wrote:

If you're running Amavisd-new-2.11.x, could you try this patch for Amavisd and test again?
https://marc.info/?l=amavis-user&m= … 91&w=2

Thanks for the speedy reply. This server runs amavisd-new-2.11.0_2,1 and the executable at /usr/local/sbin/amavisd already has the one line added by that patch (but I desintalled, recompiled, and reinstalled the port, then restarted the service, just to be sure).

Regarding the 0 scores: The system correctly classifies most spam and ham messages. It's not like it tags every message as spam.

The problem is the occasional false positive such as the one in the previous example with a score of just 0.105 and the system threshold set in iRedAdmin Pro to 6.0. I have not noticed a pattern, other that false positives often occur with domains new to the system. This is a real problem; the "tagged_above=0" message is annoying but most users don't read message headers. However, they do notice when "good" stuff ends up in their spam folders or is tagged as spam in the subject line.

What else might explain this? Thanks.

Re: Spam tagging and spam threshold

null@example.com (ZhangHuangbin) — Thu, 31 May 2018 03:42:52 +0000

If you're running Amavisd-new-2.11.x, could you try this patch for Amavisd and test again?
https://marc.info/?l=amavis-user&m= … 91&w=2

Re: Spam tagging and spam threshold

null@example.com (ZhangHuangbin) — Thu, 31 May 2018 03:41:31 +0000

Which Amavisd release are you running? I think this version has some kind of bug. Check this log line:

May 30 17:29:35.825 mail8.example.com /usr/local/sbin/amavisd[93839]: (93839-01) lookup_sql(someuser@domain2.org) matches, result=(id=>"2", priority=>"0", policy_id=>"2", email=>"@.", fullname=>-, id=>"2", policy_name=>"@.", virus_lover=>"N", spam_lover=>"N", unchecked_lover=>-, banned_files_lover=>"N", bad_header_lover=>"N", bypass_virus_checks=>"N", bypass_spam_checks=>"N", bypass_banned_checks=>"N", bypass_header_checks=>"N", virus_quarantine_to=>"", spam_quarantine_to=>"", banned_quarantine_to=>"", unchecked_quarantine_to=>-, bad_header_quarantine_to=>"", clean_quarantine_to=>-, archive_quarantine_to=>-, spam_tag_level=>"0", spam_tag2_level=>"0", spam_tag3_level=>"0", spam_kill_level=>"0", spam_dsn_cutoff_level=>-, spam_quarantine_cutoff_level=>-, addr_extension_virus=>-, addr_extension_spam=>-, addr_extension_banned=>-, addr_extension_bad_header=>-, warnvirusrecip=>-, warnbannedrecip=>-, warnbadhrecip=>-, newvirus_admin=>-, virus_admin=>-, banned_admin=>-, bad_header_admin=>-, spam_admin=>-, spam_subject_tag=>-, spam_subject_tag2=>-, spam_subject_tag3=>-, message_size_limit=>-, banned_rulenames=>-, disclaimer_options=>-, forward_method=>-, sa_userconf=>-, sa_username=>-, id=>"2", local=>-)

It founds the spam policy by sql query, but the spam tag levels are always 0. That's why Amavisd inserted mail header:

 ... tagged_above=0 required=0 ...