1 Topic by jader

  • jader
  • Member
  • Offline
  • Registered: 2023-07-20
  • Posts: 2

Topic: spider-import = too many e-mails on INBOX

I'm trying the Spider, so far so good BUT I have a VIP user (a director!) with +77000 e-mail in INBOX.
Her SENT folder  has +20000 messages too... and spider-import is getting timeout and other errors:

2023-07-19 19:57:47 INFO user@domain.com.br|[INBOX.enviados] Archiving 7720 messages.
2023-07-19 19:57:47 WARN user@domain.com.br|failed in fetching full mail message: imap: connection closed
2023-07-19 19:57:47 INFO user@domain.com.br|Logged out.
imap/client: 2023/07/19 19:57:57 error reading response: unexpected EOF
2023-07-19 19:57:57 WARN user@domain.com.br|Failed in fetching mail headers: imap: connection closed
2023-07-19 19:57:57 INFO user@domain.com.br|[INBOX] Archiving 8004 messages.
2023-07-19 19:57:57 WARN user@domain.com.br|failed in fetching full mail message: imap: connection closed
2023-07-19 19:57:57 INFO user@domain.com.br|Logged out.

The first run I did on own Spider server and CPU spiked so I even try to use "--parallel-msg=1" to slow down it. Now I'm running it on other server... but still getting errors.
Any tips or bypass solution to eliminate importation problems ?

Regards,

Jáder

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,081

Re: spider-import = too many e-mails on INBOX

Hi Jader,

- Which IMAP server are you running on email server? Dovecot, Cyrus-IMAP, or other? It says "imap: connection closed", seems IMAP server has some issue with the fetch, probably too many emails and cannot respond in time. Try to increase the IMAP read timeout interval in IMAP server if possible.

- What's the mailbox format? If it's Maildir, you can copy "spider-import" command to email server (i hope it's a Linux x86_64 box?), then run spider-import to import from Maildir mailbox directly. FYI https://spiderd.io/docs/import-maildir.html

3 Reply by jader

  • jader
  • Member
  • Offline
  • Registered: 2023-07-20
  • Posts: 2

Re: spider-import = too many e-mails on INBOX

Hi, thanks by your quick answer.
I read the options for spider-import and change timeout to 20 or 30s and because I choose to start using an external server  I change the paralells to 50.
So my load average was tolerable (very powerfull CPU on importing host) and no timeout because of so many paralls!
I'll reimport all again!
Thank you!

4 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,081

Re: spider-import = too many e-mails on INBOX

Great. Contact us if you have any question or issue. smile
Forum topic closed for now, free free to create a new one.