1 Topic by bmoore (edited by bmoore 2013-12-17 04:39:25)

  • bmoore
  • Member
  • Offline
  • Registered: 2013-08-27
  • Posts: 14

Topic: Line breaks being stripped out of email

==== Required information ====
- iRedMail version: 0.8.5
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): mysql
- Linux/BSD distribution name and version: Debian 7 Wheezy
- Related log if you're reporting an issue:
====

We have some proprietary business software that generates email messages for certain things. That server uses sendmail for this task (no changes made to this system). We migrated to iRedMail for our company email. Now when we get the messages the line breaks are removed so that:

Item1: some words
Item2: more words
Item3: even more words

becomes:

Item1: some wordsItem2: some wordsItem3: even more words

I added the line in postfix: sendmail_fix_line_endings = never (since always is default, but I tried that too).

There was no noticeable change. We didn't have this problem before but we do now. The problem appears in Roundcube, Thunderbird and Outlook. When I send the messages to a gmail account the line breaks are displayed properly leading me to believe it's an issue with the new email server on the receiving end. Any ideas or suggestions?

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,091

Re: Line breaks being stripped out of email

I didn't receive similar report before, no idea yet.

*) Did you change any config files after initial iRedMail installation?
*) Could you please show us output of below two commands?

# postconf -n
# dovecot -n

*) is your email sent as plain text or HTML format?
*) What's the line break do you use in email? '\r\n', '\n'? Or you didn't specify this at all. You can try to view it in terminal with command 'cat -v -e email.eml'.

P.S. As mentioned in Postfix document, 'sendmail_fix_line_endings' exists in Postfix-2.9 and later versions.

3 Reply by bmoore (edited by bmoore 2013-12-18 22:20:41)

  • bmoore
  • Member
  • Offline
  • Registered: 2013-08-27
  • Posts: 14

Re: Line breaks being stripped out of email

I made a couple of minor changes to lessen some of the restrictions on email because a few messages weren't coming through. Email is sent in plain text. The email is generated through software we purchased so I'm not sure if they used new lines or carriage returns (or both) for line breaks. We did have an issue in the past with MS Outlook stripping out the line breaks, but this happens outside of Outlook and I've made sure to set Outlook not to remove the line breaks. Here is the output minus identifying info. If you want the full output I could email it to you. Thanks for your help.

/home/username# postconf -n
alias_database = hash:/etc/postfix/aliases
alias_maps = hash:/etc/postfix/aliases
allow_min_user = no
allow_percent_hack = no
append_dot_mydomain = no
biff = no
bounce_queue_lifetime = 4h
broken_sasl_auth_clients = yes
config_directory = /etc/postfix
content_filter = smtp-amavis:[127.0.0.1]:10024
delay_warning_time = 0h
disable_vrfy_command = yes
dovecot_destination_recipient_limit = 1
enable_original_recipient = no
home_mailbox = Maildir/
inet_interfaces = all
inet_protocols = ipv4
mailbox_command = /usr/lib/dovecot/deliver
mailbox_size_limit = 0
maximal_backoff_time = 4000s
maximal_queue_lifetime = 4h
message_size_limit = 15728640
minimal_backoff_time = 300s
mydestination = $myhostname, localhost, localhost.localdomain, localhost.$myhostname
mydomain = domainname.com
myhostname = mail.domainname.com
mynetworks = 127.0.0.0/8, 172.xxx.xxx.0/24
mynetworks_style = host
myorigin = mail.domainname.com
proxy_read_maps = $canonical_maps $lmtp_generic_maps $local_recipient_maps $mydestination $mynetworks $recipient_bcc_maps $recipient_canonical_maps $relay_domains $relay_recipient_maps $relocated_maps $sender_bcc_maps $sender_canonical_maps $smtp_generic_maps $smtpd_sender_login_maps $transport_maps $virtual_alias_domains $virtual_alias_maps $virtual_mailbox_domains $virtual_mailbox_maps $smtpd_sender_restrictions
queue_run_delay = 300s
readme_directory = no
recipient_bcc_maps = proxy:mysql:/etc/postfix/mysql/recipient_bcc_maps_user.cf, proxy:mysql:/etc/postfix/mysql/recipient_bcc_maps_domain.cf
recipient_delimiter = +
relay_domains = $mydestination, proxy:mysql:/etc/postfix/mysql/relay_domains.cf
relayhost =
sender_bcc_maps = proxy:mysql:/etc/postfix/mysql/sender_bcc_maps_user.cf, proxy:mysql:/etc/postfix/mysql/sender_bcc_maps_domain.cf
sendmail_fix_line_endings = always
smtp-amavis_destination_recipient_limit = 1
smtp_data_init_timeout = 240s
smtp_data_xfer_timeout = 600s
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)
smtpd_data_restrictions = reject_unauth_pipelining
smtpd_end_of_data_restrictions =
smtpd_helo_required = no
smtpd_helo_restrictions = permit_mynetworks, permit_sasl_authenticated
smtpd_recipient_restrictions = reject_unknown_sender_domain, permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination
smtpd_reject_unlisted_recipient = yes
smtpd_reject_unlisted_sender = no
smtpd_sasl_auth_enable = yes
smtpd_sasl_authenticated_header = no
smtpd_sasl_local_domain = domainname.local
smtpd_sasl_path = ./dovecot-auth
smtpd_sasl_security_options = noanonymous
smtpd_sasl_type = dovecot
smtpd_sender_login_maps = proxy:mysql:/etc/postfix/mysql/sender_login_maps.cf
smtpd_sender_restrictions = permit_mynetworks, reject_sender_login_mismatch, permit_sasl_authenticated
smtpd_tls_CAfile = /etc/ssl/certs/iRedMail_CA.pem
smtpd_tls_cert_file = /etc/ssl/certs/iRedMail_CA.pem
smtpd_tls_key_file = /etc/ssl/private/iRedMail.key
smtpd_tls_loglevel = 0
smtpd_tls_security_level = may
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtpd_use_tls = yes
swap_bangpath = no
tls_random_source = dev:/dev/urandom
transport_maps = proxy:mysql:/etc/postfix/mysql/transport_maps_user.cf, proxy:mysql:/etc/postfix/mysql/transport_maps_domain.cf
virtual_alias_domains =
virtual_alias_maps = proxy:mysql:/etc/postfix/mysql/virtual_alias_maps.cf, proxy:mysql:/etc/postfix/mysql/domain_alias_maps.cf, proxy:mysql:/etc/postfix/mysql/catchall_maps.cf, proxy:mysql:/etc/postfix/mysql/domain_alias_catchall_maps.cf
virtual_gid_maps = static:2000
virtual_mailbox_base = /var/vmail
virtual_mailbox_domains = proxy:mysql:/etc/postfix/mysql/virtual_mailbox_domains.cf
virtual_mailbox_maps = proxy:mysql:/etc/postfix/mysql/virtual_mailbox_maps.cf
virtual_minimum_uid = 2000
virtual_transport = dovecot
virtual_uid_maps = static:2000

/home/username# dovecot -n
# 2.1.7: /etc/dovecot/dovecot.conf
# OS: Linux 3.2.0-4-amd64 x86_64 Debian 7.2
auth_master_user_separator = *
auth_mechanisms = PLAIN LOGIN
auth_verbose_passwords = sha1
dict {
  acl = mysql:/etc/dovecot/dovecot-share-folder.conf
  quotadict = mysql:/etc/dovecot/dovecot-used-quota.conf
}
disable_plaintext_auth = no
first_valid_uid = 2000
last_valid_uid = 2000
listen = *
log_path = /var/log/dovecot.log
mail_gid = 2000
mail_location = maildir:/%Lh/Maildir/:INDEX=/%Lh/Maildir/
mail_plugins = quota
mail_uid = 2000
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave
namespace {
  inbox = yes
  location =
  prefix =
  separator = /
  type = private
}
namespace {
  list = children
  location = maildir:/%%Lh/Maildir/:INDEX=/%%Lh/Maildir/Shared/%%u
  prefix = Shared/%%u/
  separator = /
  subscriptions = yes
  type = shared
}
passdb {
  args = /etc/dovecot/dovecot-mysql.conf
  driver = sql
}
passdb {
  args = /etc/dovecot/dovecot-master-users-password
  driver = passwd-file
  master = yes
}
plugin {
  acl = vfile
  acl_shared_dict = proxy::acl
  auth_socket_path = /var/run/dovecot/auth-master
  autocreate = INBOX
  autocreate2 = Sent
  autocreate3 = Trash
  autocreate4 = Drafts
  autocreate5 = Junk
  autosubscribe = INBOX
  autosubscribe2 = Sent
  autosubscribe3 = Trash
  autosubscribe4 = Drafts
  autosubscribe5 = Junk
  quota = dict:user::proxy::quotadict
  quota_rule = *:storage=1G
  quota_warning = storage=85%% quota-warning 85 %u
  quota_warning2 = storage=90%% quota-warning 90 %u
  quota_warning3 = storage=95%% quota-warning 95 %u
  sieve = /%Lh/sieve/dovecot.sieve
  sieve_dir = /%Lh/sieve
  sieve_global_dir = /var/vmail/sieve
  sieve_global_path = /var/vmail/sieve/dovecot.sieve
}
protocols = pop3 imap sieve
service auth {
  unix_listener /var/spool/postfix/dovecot-auth {
    group = postfix
    mode = 0666
    user = postfix
  }
  unix_listener auth-master {
    group = vmail
    mode = 0666
    user = vmail
  }
  unix_listener auth-userdb {
    group = vmail
    mode = 0660
    user = vmail
  }
}
service dict {
  unix_listener dict {
    group = vmail
    mode = 0660
    user = vmail
  }
}
service imap-login {
  process_limit = 500
  service_count = 1
}
service pop3-login {
  service_count = 1
}
service quota-warning {
  executable = script /usr/local/bin/dovecot-quota-warning.sh
  unix_listener quota-warning {
    group = vmail
    mode = 0660
    user = vmail
  }
}
ssl_cert = </etc/ssl/certs/iRedMail_CA.pem
ssl_key = </etc/ssl/private/iRedMail.key
userdb {
  args = /etc/dovecot/dovecot-mysql.conf
  driver = sql
}
protocol lda {
  auth_socket_path = /var/run/dovecot/auth-master
  lda_mailbox_autocreate = yes
  log_path = /var/log/sieve.log
  mail_plugins = quota sieve autocreate
  postmaster_address = root
}
protocol imap {
  imap_client_workarounds = tb-extra-mailbox-sep
  mail_plugins = quota imap_quota autocreate
}
protocol pop3 {
  mail_plugins = quota
  pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
  pop3_uidl_format = %08Xu%08Xv
}

/home/username/Downloads# cat -v -e testing.eml
Return-Path: <root@domainname.com>^M$
Delivered-To: bmoore@domainname.com^M$
Received: from localhost (localhost.localdomain [127.0.0.1])^M$
    by mail.domainname.com (Postfix) with ESMTP id 4C70E3C1F6D^M$
    for <bmoore@domainname.com>; Wed, 18 Dec 2013 07:51:53 -0600 (CST)^M$
X-Virus-Scanned: Debian amavisd-new at mail.domainname.com^M$
Received: from mail.domainname.com ([127.0.0.1])^M$
    by localhost (mail.domainname.com [127.0.0.1]) (amavisd-new, port 10024)^M$
    with ESMTP id ODbVuNp29Ddi for <bmoore@domainname.com>;^M$
    Wed, 18 Dec 2013 07:51:52 -0600 (CST)^M$
Received: from server.domainname.com (server.domainname.local [172.xxx.xxx.xxx])^M$
    by mail.domainname.com (Postfix) with ESMTPS id A08773C1F6C^M$
    for <bmoore@domainname.com>; Wed, 18 Dec 2013 07:51:52 -0600 (CST)^M$
Received: from domainname.com (localhost.localdomain [127.0.0.1])^M$
    by server.domainname.com (x.xx.x/x.xx.x) with ESMTP id rBIDpx7j027025^M$
    for <bmoore@domainname.com>; Wed, 18 Dec 2013 07:51:59 -0600^M$
Received: (from root@localhost)^M$
    by domainname.com (x.xx.x/x.xx.x/Submit) id rBIDpxkb027021^M$
    for bmoore@domainname.com; Wed, 18 Dec 2013 07:51:59 -0600^M$
Date: Wed, 18 Dec 2013 07:51:59 -0600^M$
From: root <root@domainname.com>^M$
Message-Id: <201312181351.rBIDpxkb027021@domainname.com>^M$
To: bmoore@domainname.com^M$
Subject: testing^M$
^M$
Event Name: oe-eventnameEvent Desc: OE Approval of an OrderCustomer #: 123456Customer Name: CompanyName: 1234567-00Sales Rep In: 1Sales Rep Out: 053Stage: 1Taken By: user1Transaction Type: soWarehouse: Operator: user1Customer PO #: 12345Promise Date: 12/18/13Req Ship Date: 12/18/13^M$

4 Reply by riverco (edited by riverco 2013-12-19 00:09:15)

  • riverco
  • Member
  • Offline
  • Registered: 2013-11-18
  • Posts: 28

Re: Line breaks being stripped out of email

Hi,

i do not know if this helps but i thought this info is useful it has happened to me also with proprietary feeds. (be patient with me i tried explaining with 2 quotes the solution is down the end although not necessarily a easy one)

--- Disclaimer : Try this on a test machine first ---

Your send app isn't 100% good (this can happen very rarely with some of those apps that are written to spit out simple text messages and the makers were not experts in email, this doesn't always cause problems and sometimes it does, the issue did not happen because probably your other mail system used another MTA (sendmail, Exim, etc..) or a more recent postfix ) but that usually doesn't matter however postfix does a few things by design with messages :

Quote :
The SMTP record delimiter is CRLF. Postfix removes it (as well as
invalid CR characters at the end of a record) while receiving mail
via SMTP, and adds it when sending mail via SMTP.

The UNIX text record delimiter is LF. Postfix removes it (as well
as invalid CR characters at the end of a record) while receiving
mail via the Postfix sendmail command, and adds it when delivering
mail to local mailbox, maildir, or command.

The DOS/WINDOWS text record delimiter is CRLF. When someone sends
mail as text/plain or text/quoted-printable from DOS/WINDOWS to
UNIX, those CRLF will become LF. Just like UNIX LF will become CRLF
when sending text/plain or text/quoted-printable from UNIX to
DOS/WINDOWS.

-->>>Solution : (and this is the only one i so far found which i had to do on  one installation to make it work, is upgrade postfix :
postfix config file :

   sendmail_fix_line_endings (default: always)
       Controls  how  the Postfix sendmail command converts email message line
       endings from <CR><LF> into UNIX format (<LF>).

       always Always convert message lines ending in <CR><LF>. This setting is
              the default with Postfix 2.9 and later.

       strict Convert message lines ending in <CR><LF> only if the first input
              line ends in <CR><LF>. This setting is backwards-compatible with
              Postfix 2.8 and earlier.

       never  Never  convert  message  lines  ending in <CR><LF>. This setting
              exists for completeness only.

       This feature is available in Postfix 2.9 and later.

Maybe Zhang knows another way ?

kind regards,

5 Reply by bmoore

  • bmoore
  • Member
  • Offline
  • Registered: 2013-08-27
  • Posts: 14

Re: Line breaks being stripped out of email

I tried using sendmail_fix_line_endings as my research was pointing me there as well. It had no effect. I read that if a plain text message gets encoded differently that might alter the line breaks as well but I wasn't able to determine a way to alter that in postfix. I'm using postfix 2.9.6.

6 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,091

Re: Line breaks being stripped out of email

Hi @bmoore,

As you can see in the 'cat -v -e' command output, line break will be shown as '^M$', but your mail body doesn't have line breaks at all. I suggest you contact your application provider for a potential solution (i don't mean it's their issue, just a potential opportunity).

P.S. You modified many Postfix settings which i think it's bad idea. But i don't know your requirements, so it's fine if you understand what you're doing.

smtpd_helo_required
smtpd_recipient_restrictions
smtpd_reject_unlisted_sender
smtpd_sasl_local_domain

7 Reply by bmoore (edited by bmoore 2013-12-19 21:38:29)

  • bmoore
  • Member
  • Offline
  • Registered: 2013-08-27
  • Posts: 14

Re: Line breaks being stripped out of email

I had to change those settings because they were preventing some email from being delivered/sent in certain situations. We didn't have issues with line breaks with our previous email provider and if they are sent to a gmail or yahoo account then there aren't any issues either. The problem is only present when they are delivered to the new email server setup with iRedMail. I've been pretty stumped because they have line breaks at the source when being sent but not when they reach the destination. I haven't been able to determine where in the chain this is happening.

Well, thanks for looking into it.