You are not logged in. Please login or register.
iRedMail → iRedAdmin-Pro Support → Spam tagging and spam threshold
OK, here is an example of a message that scored 0.105 and still was classified as spam.
With amavisd in debug mode, the output was too large for pastebin, so I used another service called TinyPaste. Here are the message headers and debug output:
Thanks in advance for further troubleshooting clues.
----
Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.
Which Amavisd release are you running? I think this version has some kind of bug. Check this log line:
May 30 17:29:35.825 mail8.example.com /usr/local/sbin/amavisd[93839]: (93839-01) lookup_sql(someuser@domain2.org) matches, result=(id=>"2", priority=>"0", policy_id=>"2", email=>"@.", fullname=>-, id=>"2", policy_name=>"@.", virus_lover=>"N", spam_lover=>"N", unchecked_lover=>-, banned_files_lover=>"N", bad_header_lover=>"N", bypass_virus_checks=>"N", bypass_spam_checks=>"N", bypass_banned_checks=>"N", bypass_header_checks=>"N", virus_quarantine_to=>"", spam_quarantine_to=>"", banned_quarantine_to=>"", unchecked_quarantine_to=>-, bad_header_quarantine_to=>"", clean_quarantine_to=>-, archive_quarantine_to=>-, spam_tag_level=>"0", spam_tag2_level=>"0", spam_tag3_level=>"0", spam_kill_level=>"0", spam_dsn_cutoff_level=>-, spam_quarantine_cutoff_level=>-, addr_extension_virus=>-, addr_extension_spam=>-, addr_extension_banned=>-, addr_extension_bad_header=>-, warnvirusrecip=>-, warnbannedrecip=>-, warnbadhrecip=>-, newvirus_admin=>-, virus_admin=>-, banned_admin=>-, bad_header_admin=>-, spam_admin=>-, spam_subject_tag=>-, spam_subject_tag2=>-, spam_subject_tag3=>-, message_size_limit=>-, banned_rulenames=>-, disclaimer_options=>-, forward_method=>-, sa_userconf=>-, sa_username=>-, id=>"2", local=>-)
It founds the spam policy by sql query, but the spam tag levels are always 0. That's why Amavisd inserted mail header:
... tagged_above=0 required=0 ...If you're running Amavisd-new-2.11.x, could you try this patch for Amavisd and test again?
https://marc.info/?l=amavis-user&m= … 91&w=2
If you're running Amavisd-new-2.11.x, could you try this patch for Amavisd and test again?
https://marc.info/?l=amavis-user&m= … 91&w=2
Thanks for the speedy reply. This server runs amavisd-new-2.11.0_2,1 and the executable at /usr/local/sbin/amavisd already has the one line added by that patch (but I desintalled, recompiled, and reinstalled the port, then restarted the service, just to be sure).
Regarding the 0 scores: The system correctly classifies most spam and ham messages. It's not like it tags every message as spam.
The problem is the occasional false positive such as the one in the previous example with a score of just 0.105 and the system threshold set in iRedAdmin Pro to 6.0. I have not noticed a pattern, other that false positives often occur with domains new to the system. This is a real problem; the "tagged_above=0" message is annoying but most users don't read message headers. However, they do notice when "good" stuff ends up in their spam folders or is tagged as spam in the subject line.
What else might explain this? Thanks.
What else might explain this? Thanks.
I tried to figure it out on another production server, but no result yet. It looks like a Amavisd bug, but not sure.
OK, thanks. Please let me know if you need additional details about the amavisd config or database settings.
The final post in this thread suggests there's a change in perl or Amavisd which affected SQL queries, but provides no useful details:
https://www.howtoforge.com/community/th … pam.75725/
Certainly there is a problem with the tagged_above=0 label, but I don't know if that's the only reason for the false positives, or even if it's related.
After much research, I now believe the root cause of the tagged_above=0 headers is a type-conversion bug in the p5-DBD-mysql module:
https://github.com/perl5-dbi/DBD-mysql/issues/78
https://lists.amavis.org/pipermail/amav … 04674.html
This explains why user amavisd gets correct results in MySQL queries but iRedMail lookups get incorrect (0) results.
My options are these:
1. Revert to the old p5-DBD-mysql-4.037 module, the last version not to have the type-conversion bug. I have verified that this version does not have the type-conversion bug, and returns expected values from MySQL lookups. However, this is a quite old version (current on FreeBSD is 4.046) and has several other security vulnerabilities.
2. Switch away from MySQL or MariaDB to PostgreSQL. OpenLDAP isn't an option for me as I don't know it, and even switching between SQL databases is, for me, something like changing engines on a jet airline while it's in the air.
3. Stay on the current broken 4.046 module and live with a some false positives.
Of these three options, which would you recommend?
Thanks!
- Both 1+3 will cause some issue during package upgrade. you have to skip this package.
- #2 is ok, but if this is a production server, the cost may be higher than 1 or 3.
Here's option 4:
Contact software developer and package maintainer, push them to fix it and release a new version.
Thanks, ZHB. I’m doing option 1 now, and just yesterday upgraded ports using “portmaster -aD -x p5-DBD-mysql” — not a great option considering 4.037 has other security vulnerabilities.
I’m already doing option 4, but so far have not heard anything back from the port maintainer or the amavisd mailing list. I may also file a bug report against one or both packages in the FreeBSD bugzilla bug tracker.
BTW there’s also this thread claiming it’s an amavisd problem:
https://de.postfix.org/pipermail/amavis … 04711.html
Quoting: "The change is that [as of 4.038] p5-DBD-mysql now returns mysql doubles as perl doubles and mysql floats as perl floats (and not as a string anymore). This should be adressed [sic] by amavisd."
I don’t know which port is at fault but will files bugs against both since the bottom line is that one of them returns bad data.
Thanks again.
You'd better file bugs for all. 
An update: 30 days ago I filed bug 231250 against amavisd-new in the FreeBSD bugzilla database, and contacted the port maintainer about a week before that. There's been no response to either post.
Since then there has been a long thread on the amavisd-new mailing list concluding that further development of amavisd-new is dead.
Postfix guru Ralf Hildebrandt noted that his organization, a German telecommunications company, recently migrated away from amavisd-new and now uses rspamd instead:
https://github.com/rspamd/rspamd
I have no experience with rspamd. What I can say at this point is that:
(1) the root cause of the bad tags is in the way amavisd-new handles perl data types such as perl doubles;
(2) the likelihood of a fix in amavisd-new is approximately 0.00 percent; and
(3) it's probably time to start looking at alternatives to amavisd-new.
Thanks!
rspamd is on my radar. If no more development/maintenance happened to Amavisd-new, we may switch to rspamd in 2019.
rspamd is on my radar. If no more development/maintenance happened to Amavisd-new, we may switch to rspamd in 2019.
Are there any news here?
Are there any news here?
No plan to switch to rspamd shortly. sorry.
Amavisd-new is still active: https://gitlab.com/amavis/amavis
nuwinfo wrote:Are there any news here?
No plan to switch to rspamd shortly. sorry.
Amavisd-new is still active: https://gitlab.com/amavis/amavis
I have the same problem.
amavisd-new-2.12.2, perl5 (revision 5 version 34 subversion 0), 10.5.15-MariaDB, for FreeBSD13.0
I wrote a "check" perl script that reads the policy table values. Here the float values are displayed correctly (3.000 , 6.900, etc). However, the amavisd-new log shows spam_tag_level=>"0", so I suspect a faulty value conversion for amavisd (and not due to p5-DBD-mysql).
As a workaround, I changed the float type in the database to varchar and -for now- it seems fine.
Am I wrong? Does anyone have any suggestions?
Regard
k
Does anyone have any suggestions?
I migrated my iRedMail box to Debian and have not had this problem there. Sorry, can't comment on the correctness of the data typing change.
iRedMail → iRedAdmin-Pro Support → Spam tagging and spam threshold
Powered by PunBB, supported by Informer Technologies, Inc.
Currently installed 2 official extensions. Copyright © 2003–2010 PunBB.
Generated in 0.010 seconds (63% PHP - 37% DB) with 10 queries