1 Topic by dcherring

  • dcherring
  • Member
  • Offline
  • Registered: 2019-08-29
  • Posts: 4

Topic: All user email addresses fail email address validation

I've been struggling to figure out a problem where whenever me or any of my email users subscribe to most any mailing list, we never receive the confirmation email that we need in order to confirm our subscription.  When we subscribe to these same mailing lists using, say for example, a gmail account, it works perfectly every time. It works never on iredmail.

After much researching and head scratching, I have discovered that what is happening is that my iredmail servers (I have two of them) fail any and all email validation services in use out there. These mailing list owners try to validate email addresses before adding them to their lists, in order to prevent crap addresses getting added to their lists.

Seems reasonable enough.

But, why are they all seemingly unable to validate an address on iredmail servers? Is there a way to correct this?


My config: 0.9.7 PGSQL, deployed via downloadable, ubuntu 16.04.6, nginx, not Pro.

Thanks!


==== REQUIRED BASIC INFO OF YOUR IREDMAIL SERVER ====
- iRedMail version (check /etc/iredmail-release):
- Deployed with iRedMail Easy or the downloadable installer?
- Linux/BSD distribution name and version:
- Store mail accounts in which backend (LDAP/MySQL/PGSQL):
- Web server (Apache or Nginx):
- Manage mail accounts with iRedAdmin-Pro?
- [IMPORTANT] Related original log or error message is required if you're experiencing an issue.
====

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,094

Re: All user email addresses fail email address validation

dcherring wrote:

I have discovered that what is happening is that my iredmail servers (I have two of them) fail any and all email validation services in use out there.

Do you host same domain on both iRedMail servers? Are all mail accounts synced between them?

3 Reply by dcherring

  • dcherring
  • Member
  • Offline
  • Registered: 2019-08-29
  • Posts: 4

Re: All user email addresses fail email address validation

ZhangHuangbin wrote:
dcherring wrote:

I have discovered that what is happening is that my iredmail servers (I have two of them) fail any and all email validation services in use out there.

Do you host same domain on both iRedMail servers? Are all mail accounts synced between them?

No, the two servers are completely separate, hosting completely different domains and are not related or connected in any way. Thus nothing is synced between them. I only mention the fact that I have two to show that both have the problem and it's maybe not an isolated issue.

4 Reply by dcherring

  • dcherring
  • Member
  • Offline
  • Registered: 2019-08-29
  • Posts: 4

Re: All user email addresses fail email address validation

Well, further research shows that the email address validators in use out there behave like spambots. The actions they take to validate an address are largely the same actions the bots take when they are trying to guess email addresses. So it would seem that iRedMail blocking address validators is perhaps a normal consequence of blocking spambots. 

This is both good and bad. At least in my way of thinking.

Good: iRedMail is doing exactly what it's supposed to, and doing it well.  ;-)

Bad: As people begin to make greater and greater use of email address validators, iRedMail and other servers like it using similar strategies to block the bots will become less and less useful for valid email.

Does anyone have any thoughts about how to get around this? Are there other ways to protect ourselves from spam that would play nicer with address validators?

5 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,094

Re: All user email addresses fail email address validation

If you're sending normal email, you must know someone's (valid) email address in some way, for example, you get the email address from your client's website, or your client gave you the card with his/her contact information like email or phone number, etc.

Not sure i understand the "validators" correctly, but it sounds like spammer's weapon to validate whether the target recipient addresses are valid, then they can save some system resource by avoiding sending email to invalid addresses.

Postfix has option to allow such validating, but it's disabled by iRedMail by default  ("disable_vrfy_command = yes" in /etc/postfix/main.cf), and we will not enable it with our default setting.

6 Reply by dcherring

  • dcherring
  • Member
  • Offline
  • Registered: 2019-08-29
  • Posts: 4

Re: All user email addresses fail email address validation

ZhangHuangbin wrote:

If you're sending normal email, you must know someone's (valid) email address in some way, for example, you get the email address from your client's website, or your client gave you the card with his/her contact information like email or phone number, etc.

Not sure i understand the "validators" correctly, but it sounds like spammer's weapon to validate whether the target recipient addresses are valid, then they can save some system resource by avoiding sending email to invalid addresses.

Postfix has option to allow such validating, but it's disabled by iRedMail by default  ("disable_vrfy_command = yes" in /etc/postfix/main.cf), and we will not enable it with our default setting.

To explain the email address validator, take for example a person or entity that publishes an email based newsletter. They put up an opt-in page where you give your name and email address.  In exchange for that, they usually give you a free gift or something in addition to sending  you their newsletter.  What was happening was that people would go to these opt-in pages and give a totally fake or bogus email address to get the free gift and thus not actually sign up for anything.  The people running these newsletter lists are now preventing that from happening by running an email address validator at the point where one enters their email address on the opt-in page. If it validates, great.  If not, the person who entered a bad address is ignored.

During the course of my figuring this out, I discovered that, as you pointed out, spammers do the exact same thing as these email address validators. The spammers aren't associated with an opt-in page, but they do hit our servers with numerous random verify commands to try and guess/harvest email addresses. So turns out it is the same basic underlying technology in both cases, just used in different ways for different reasons.

So when one enters their email address on one of these opt-in pages, one would reasonably expect the address to be validated and thus we join the list and get the goodies, whatever they are. But we don't want spammers to be able to come along and pound our server to guess our addresses.

It seems we have a catch-22 situation here. Block spammers, lose valid functionality...enable valid functionality, open to spammers.

I honestly don't know the answer to this, if there is one. I am hoping someone else more versed in this would have an idea or  suggestion as to how to have it both ways as it were -- to block spammers and allow valid use of email address validation.

I hope this explanation helps.
Thank you!

7 Reply by jasongo

  • jasongo
  • Member
  • Offline
  • Registered: 2019-09-01
  • Posts: 8

Re: All user email addresses fail email address validation

I think the following information are needed to fully inspect your case:

1. What's the IP addresses of your mail servers? Include both IPv4 and IPv6 addresses attached to your server.

I believe that some "email address verification" service like Zerobounce, etc., will try to scan the IP addresses of your email domain and will try to check if it's listed in any of the blacklists (not only the spammers blacklist, but also to other malicious listing available to them).

2. What's the SPF, DKIM, and DMARC policies you have?

Although these are needed by the sender, of which in your case your email addresses are receivers, a properly configured sending policies will signal the address validation that your email domain is not some kind of a fly-by-night email. It's one of the simplest preliminary check whether the email domain were configured for legitimate use.

3. What's the hostname of your email server? Is it the same as the output of your smtp port when you telnet it (port 25 or 587)?

4. By any chance, does your postscreen configuration have some weird custom configuration that may block some IP addresses? Like you have some DNBSL and you configured to block almost anyone qualifying to it? Have you tried re-adjusting these?

5. What's the postscreen pre-greet delay that you have? 5 sec? 15 sec? The longer it is, the more chances that some mail servers will simply stop sending emails because they thought (and they are lousily configured) that your email server had time-outs when in fact your server is just checking their IP address reputation.

6. Can you check in phpmyadmin under the iredapd database -> greylisting_tracking tables if the domain of the website you signed up are listed in the greylisted domains? And how many blocks and passes did that email had? iRedmail tries to do greylisting for new email domains that sends email to your server and asks them to send again after a few minutes. Legitimate senders will re-attempt delivery while other will just stop resending. In this case, the website you signed up for might be the culprit (typical of mass senders who just wants to send as fast as they can...).

7. Have you inspected your /var/log/mail.log? Is there any indication that emails coming from the website you signed up are being blocked by postfix/amavis? What other messages it contained? That might give us some clues.

8. What email validation services are you using? I have used Zerobounce in the past and it's good enough. Have you tried Mailgun's email validation? ElasticEmail? These email address validators are more reputable and can be trusted. The other email validation service might be doing it "strictly" because they are paid based on how many email address you validate with them. The more you validate, the more they earn and so they might be trying to manipulate some results just for you to try validating again...that's just my conspiracy thinking.

8 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,094

Re: All user email addresses fail email address validation

dcherring wrote:

To explain the email address validator, take for example a person or entity that publishes an email based newsletter. They put up an opt-in page where you give your name and email address.  In exchange for that, they usually give you a free gift or something in addition to sending  you their newsletter.  What was happening was that people would go to these opt-in pages and give a totally fake or bogus email address to get the free gift and thus not actually sign up for anything.  The people running these newsletter lists are now preventing that from happening by running an email address validator at the point where one enters their email address on the opt-in page. If it validates, great.  If not, the person who entered a bad address is ignored.

The procedure to valid email address is wrong, and this is a normal way but works fine:

When user submits an email address from the opt-in page, usually we send an email (with a http link in email) to the address and ask user to click the http link, the link is served by your server, when the link is accessed, your system marks this email address as valid. then you can send the free gift.

btw, on the opt-in page, usually we don't say something like "The email address you input exists in our license database", this will give spammer clear signal that the email address is valid. Instead using something like "We have sent you email to verify the email address ownership moment ago if the address is valid, please login to your mailbox and click the link in email to verify you're the owner."