1 Topic by mkr

  • mkr
  • Member
  • Offline
  • Registered: 2023-05-08
  • Posts: 15

Topic: deduplication

==== REQUIRED BASIC INFO OF YOUR IREDMAIL SERVER ====
- iRedMail version (check /etc/iredmail-release): 1.6.4
- Deployed with iRedMail Easy or the downloadable installer?
- Linux/BSD distribution name and version: Alma 9.2
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): MySQL
- Web server (Apache or Nginx):
- Manage mail accounts with iRedAdmin-Pro?
- [IMPORTANT] Related original log or error message is required if you're experiencing an issue.
====

How to achieve deduplication in IRedmail. Suppose if a mail was sent to 10 recipients, 10 copies are being stored under individual Maildir folders. This is increasing storage

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2 Reply by Cthulhu

  • Cthulhu
  • Member
  • Offline
  • Registered: 2019-06-04
  • Posts: 859

Re: deduplication

How did you send them, one by one or with CC/BCC ?

3 Reply by mkr

  • mkr
  • Member
  • Offline
  • Registered: 2023-05-08
  • Posts: 15

Re: deduplication

Cthulhu wrote:

How did you send them, one by one or with CC/BCC ?

The recipients in To are 10.

4 Reply by Cthulhu

  • Cthulhu
  • Member
  • Offline
  • Registered: 2019-06-04
  • Posts: 859

Re: deduplication

if you send mails, they are only stored if the mail client does so, so this is not related to iredmail, but to your MUA

5 Reply by mkr

  • mkr
  • Member
  • Offline
  • Registered: 2023-05-08
  • Posts: 15

Re: deduplication

I feel my question has been misunderstood. When a single email was sent with 10 recipients in TO, that email instance is being stored in dovecot maildir folders of that 10 users which implies 10 copies. This is increasing storage. Lot of New generation email softwares keep a single copy and have pointers in DB or somewhere which they call deduplication(Storing single copy instead of 10 copies). How can we achive that here?

6 Reply by Cthulhu

  • Cthulhu
  • Member
  • Offline
  • Registered: 2019-06-04
  • Posts: 859

Re: deduplication

messages are stored as files and not in a database, only possibility would be shared mailboxes but then when someone deletes the mail, it is deleted for everyone

and those are not duplicates, they are individual mails for every single user

7 Reply by mkr

  • mkr
  • Member
  • Offline
  • Registered: 2023-05-08
  • Posts: 15

Re: deduplication

So,How can we achive deduplication in IRedmail for regular mailboxes.

8 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 30,916

Re: deduplication

FYI https://doc.dovecot.org/3.0/man/doveadm-deduplicate.1/

If you sent email with attachment to many recipients, the mailbox size may increase quickly since every recipient gets the message at almost same size. But if you sent plain messages, the size is usually small and acceptable.

MTA, LDA and milter may add additional headers to messages for different recipients (e.g. X-Envelope-To:, Delivered-To:), or modify mail body (e.g. append disclaimer text at the end), so the messages delivered to mailboxes may have different email headers or body, as @Cthulhu mentioned, "they are individual mails for every single user", hence it's impossible to deduplicate since they're not same anymore.

There's one way to save disk space and deduplicate: run an email archiving software to archive all in/out emails. Although email with attachment was sent to many recipients, the attachment (e.g. a PDF file, a PNG image) is always same and only one copy is saved / archived. You're free to remove old emails (e.g. 1 year old, 2 years old, etc) from mailboxes, and view or restore old emails from the web UI offered by archiving software. The side affect is, smaller mailbox gets better IMAP performance since IMAP server handles less messages.

AD here: don't forget to check out the lightweight on-premises email archiving software developed by iRedMail team: Spider Email Archiver. https://spiderd.io/