1 Topic by schnappi (edited by schnappi 2025-01-07 09:14:19)

  • schnappi
  • Member
  • Offline
  • Registered: 2015-02-13
  • Posts: 166

Topic: connect to 127.0.0.1[127.0.0.1]:10026: Connection refused

Immediately after upgrading to Debian 12 started to see the following in postfix/amavis/mail log files:

connect to 127.0.0.1[127.0.0.1]:10026: Connection refused

The postfix queue started to fillup with "stuck" messages. Ran 

postqueue -f

and some messages would process, but then everything got stuck again. In order to process about ~150 messages, had to restart amavis and then run "postqueue -f" about 5-7 times.

After cleaning the queue, running the below indicates an empty queue for a short period of time, then the queue starts to fill up again.

mailq

Every time that get "connect to 127.0.0.1[127.0.0.1]:10026: Connection refused" the amavis task is running and using a good amount of CPU, but not maxing out RAM. The amavis task never fails per se (service is still running), but the queue stops processing, and the amavis task stops using a good amount of cpu corresponding with ceasing to process the queue.

Every time the queue starts to fill up again,  "netstat -tulnp" shows that ports 10024 and 10026 disappear as open. I believe that the cause of this issue may be that these ports are closing unexpectedly, but not sure how to stop them from closing unexpectedly.

Any suggestions to resolve the underlying issue would be appreciated.

==== REQUIRED BASIC INFO OF YOUR IREDMAIL SERVER ====
- iRedMail version (check /etc/iredmail-release): 1.7.1
- Deployed with iRedMail Easy or the downloadable installer? Downloadable installer
- Linux/BSD distribution name and version: Debian 12
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): MariaDB
- Web server (Apache or Nginx): Nginx
- Manage mail accounts with iRedAdmin-Pro?
- [IMPORTANT] Related original log or error message is required if you're experiencing an issue.
====

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2 Reply by Cthulhu (edited by Cthulhu 2025-01-07 06:33:04)

  • Cthulhu
  • Member
  • Offline
  • Registered: 2019-06-04
  • Posts: 925

Re: connect to 127.0.0.1[127.0.0.1]:10026: Connection refused

10024 and 10026 both are used my amavis

what are the specs?

and what causes amavis to crash?

3 Reply by schnappi (edited by schnappi 2025-01-07 10:27:30)

  • schnappi
  • Member
  • Offline
  • Registered: 2015-02-13
  • Posts: 166

Re: connect to 127.0.0.1[127.0.0.1]:10026: Connection refused

I was incorrect. The amavis service stopping is causing the issue to occur.

Machine is a 1 core CPU with 2 GB of RAM and 2 GB swap space. Have been running iRedMail for years on this machine with Debian 11 with no or minimal issues. Was avoiding upgrading to Debian 12 to avoid an issue like this.

Here are the results of "netstat -tulnp" when the issue is occurring:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:24242         0.0.0.0:*               LISTEN      722/dovecot
tcp        0      0 127.0.0.1:7790          0.0.0.0:*               LISTEN      800/uwsgi
tcp        0      0 127.0.0.1:7791          0.0.0.0:*               LISTEN      795/uwsgi
tcp        0      0 127.0.0.1:7778          0.0.0.0:*               LISTEN      1129/python3
tcp        0      0 127.0.0.1:7779          0.0.0.0:*               LISTEN      1129/python3
tcp        0      0 127.0.0.1:7777          0.0.0.0:*               LISTEN      1129/python3
tcp        0      0 0.0.0.0:995             0.0.0.0:*               LISTEN      722/dovecot
tcp        0      0 0.0.0.0:993             0.0.0.0:*               LISTEN      722/dovecot
tcp        0      0 0.0.0.0:587             0.0.0.0:*               LISTEN      5254/master
tcp        0      0 127.0.0.1:783           0.0.0.0:*               LISTEN      737/perl
tcp        0      0 127.0.0.1:9999          0.0.0.0:*               LISTEN      733/php-fpm: master
tcp        0      0 127.0.0.1:10025         0.0.0.0:*               LISTEN      5254/master
tcp        0      0 127.0.0.1:10028         0.0.0.0:*               LISTEN      5254/master
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      887/mariadbd
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      1003/nginx: master
tcp        0      0 127.0.0.1:4190          0.0.0.0:*               LISTEN      722/dovecot
tcp        0      0 127.0.0.1:24            0.0.0.0:*               LISTEN      722/dovecot
tcp        0      0 127.0.0.1:12340         0.0.0.0:*               LISTEN      722/dovecot
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      794/unbound
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN      5254/master
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      962/sshd: /usr/sbin
tcp        0      0 0.0.0.0:110             0.0.0.0:*               LISTEN      722/dovecot
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      1003/nginx: master
tcp        0      0 0.0.0.0:143             0.0.0.0:*               LISTEN      722/dovecot
tcp6       0      0 :::995                  :::*                    LISTEN      722/dovecot
tcp6       0      0 :::993                  :::*                    LISTEN      722/dovecot
tcp6       0      0 ::1:53                  :::*                    LISTEN      794/unbound
tcp6       0      0 :::587                  :::*                    LISTEN      5254/master
tcp6       0      0 ::1:783                 :::*                    LISTEN      737/perl
tcp6       0      0 :::443                  :::*                    LISTEN      1003/nginx: master
tcp6       0      0 :::25                   :::*                    LISTEN      5254/master
tcp6       0      0 :::22                   :::*                    LISTEN      962/sshd: /usr/sbin
tcp6       0      0 :::110                  :::*                    LISTEN      722/dovecot
tcp6       0      0 :::80                   :::*                    LISTEN      1003/nginx: master
tcp6       0      0 :::143                  :::*                    LISTEN      722/dovecot
udp        0      0 127.0.0.1:53            0.0.0.0:*                           794/unbound
udp        0      0 0.0.0.0:68              0.0.0.0:*                           520/dhclient
udp        0      0 0.0.0.0:68              0.0.0.0:*                           416/dhclient
udp        0      0 51.222.25.58:123        0.0.0.0:*                           983/ntpd
udp        0      0 127.0.0.1:123           0.0.0.0:*                           983/ntpd
udp        0      0 0.0.0.0:123             0.0.0.0:*                           983/ntpd
udp6       0      0 ::1:53                  :::*                                794/unbound
udp6       0      0 fe80::f816:3eff:fe1:123 :::*                                983/ntpd
udp6       0      0 2607:5300:205:200:::123 :::*                                983/ntpd
udp6       0      0 ::1:123                 :::*                                983/ntpd
udp6       0      0 :::123                  :::*                                983/ntpd

Here are the results of "netstat -tulnp" after restarting the amavis service (notice that ports 10024, 10026, 10027 are now closed):

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:24242         0.0.0.0:*               LISTEN      722/dovecot
tcp        0      0 127.0.0.1:7790          0.0.0.0:*               LISTEN      800/uwsgi
tcp        0      0 127.0.0.1:7791          0.0.0.0:*               LISTEN      795/uwsgi
tcp        0      0 127.0.0.1:7778          0.0.0.0:*               LISTEN      1129/python3
tcp        0      0 127.0.0.1:7779          0.0.0.0:*               LISTEN      1129/python3
tcp        0      0 127.0.0.1:7777          0.0.0.0:*               LISTEN      1129/python3
tcp        0      0 0.0.0.0:995             0.0.0.0:*               LISTEN      722/dovecot
tcp        0      0 0.0.0.0:993             0.0.0.0:*               LISTEN      722/dovecot
tcp        0      0 0.0.0.0:587             0.0.0.0:*               LISTEN      5254/master
tcp        0      0 127.0.0.1:9998          0.0.0.0:*               LISTEN      26964/amavisd (mast
tcp        0      0 127.0.0.1:783           0.0.0.0:*               LISTEN      737/perl
tcp        0      0 127.0.0.1:9999          0.0.0.0:*               LISTEN      733/php-fpm: master
tcp        0      0 127.0.0.1:10026         0.0.0.0:*               LISTEN      26964/amavisd (mast
tcp        0      0 127.0.0.1:10027         0.0.0.0:*               LISTEN      26964/amavisd (mast
tcp        0      0 127.0.0.1:10024         0.0.0.0:*               LISTEN      26964/amavisd (mast
tcp        0      0 127.0.0.1:10025         0.0.0.0:*               LISTEN      5254/master
tcp        0      0 127.0.0.1:10028         0.0.0.0:*               LISTEN      5254/master
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      887/mariadbd
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      1003/nginx: master
tcp        0      0 127.0.0.1:4190          0.0.0.0:*               LISTEN      722/dovecot
tcp        0      0 127.0.0.1:24            0.0.0.0:*               LISTEN      722/dovecot
tcp        0      0 127.0.0.1:12340         0.0.0.0:*               LISTEN      722/dovecot
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      794/unbound
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN      5254/master
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      962/sshd: /usr/sbin
tcp        0      0 0.0.0.0:110             0.0.0.0:*               LISTEN      722/dovecot
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      1003/nginx: master
tcp        0      0 0.0.0.0:143             0.0.0.0:*               LISTEN      722/dovecot
tcp6       0      0 :::995                  :::*                    LISTEN      722/dovecot
tcp6       0      0 :::993                  :::*                    LISTEN      722/dovecot
tcp6       0      0 ::1:53                  :::*                    LISTEN      794/unbound
tcp6       0      0 :::587                  :::*                    LISTEN      5254/master
tcp6       0      0 ::1:783                 :::*                    LISTEN      737/perl
tcp6       0      0 :::443                  :::*                    LISTEN      1003/nginx: master
tcp6       0      0 :::25                   :::*                    LISTEN      5254/master
tcp6       0      0 :::22                   :::*                    LISTEN      962/sshd: /usr/sbin
tcp6       0      0 :::110                  :::*                    LISTEN      722/dovecot
tcp6       0      0 :::80                   :::*                    LISTEN      1003/nginx: master
tcp6       0      0 :::143                  :::*                    LISTEN      722/dovecot
udp        0      0 127.0.0.1:53            0.0.0.0:*                           794/unbound
udp        0      0 0.0.0.0:68              0.0.0.0:*                           520/dhclient
udp        0      0 0.0.0.0:68              0.0.0.0:*                           416/dhclient
udp        0      0 51.222.25.58:123        0.0.0.0:*                           983/ntpd
udp        0      0 127.0.0.1:123           0.0.0.0:*                           983/ntpd
udp        0      0 0.0.0.0:123             0.0.0.0:*                           983/ntpd
udp6       0      0 ::1:53                  :::*                                794/unbound
udp6       0      0 fe80::f816:3eff:fe1:123 :::*                                983/ntpd
udp6       0      0 2607:5300:205:200:::123 :::*                                983/ntpd
udp6       0      0 ::1:123                 :::*                                983/ntpd
udp6       0      0 :::123                  :::*                                983/ntpd

4 Reply by Cthulhu

  • Cthulhu
  • Member
  • Offline
  • Registered: 2019-06-04
  • Posts: 925

Re: connect to 127.0.0.1[127.0.0.1]:10026: Connection refused

min specs are 4gb of ram, so your problem is caused by running on half of min specs, amavis gets terminated because of OOM (out of memory kill)

only thing you can do is upgrade your specs and respect system requirements, they are documented here:

https://docs.iredmail.org/install.iredm … buntu.html

That is not a upgrade problem, thats a VPS problem

5 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,165

Re: connect to 127.0.0.1[127.0.0.1]:10026: Connection refused

ClamAV requires more memory nowadays, so you'd better increase the memory to at least 4GB and larger swap disk. Otherwise you have to disable clamav which is not recommended.

6 Reply by schnappi

  • schnappi
  • Member
  • Offline
  • Registered: 2015-02-13
  • Posts: 166

Re: connect to 127.0.0.1[127.0.0.1]:10026: Connection refused

The newer version of ClamAV that installed with Debian 12 must not be able to function for extended periods of time with 2 GB of RAM and 2 GB of swap space.

This setup worked fined with Debian 11 for years. While the issue is not upgrade related per se, the issue is still update related-albeit updating with specs below the recommended level.

For right now am running a cron job to restart amavis every 10 minutes and running "postqueue -f." May look into installing the backported old  version of ClamAV available in Debian 11 to see if this resolves the issue.

7 Reply by Cthulhu

  • Cthulhu
  • Member
  • Offline
  • Registered: 2019-06-04
  • Posts: 925

Re: connect to 127.0.0.1[127.0.0.1]:10026: Connection refused

Just upgrade the server to at least 4gb of ram

8 Reply by schnappi

  • schnappi
  • Member
  • Offline
  • Registered: 2015-02-13
  • Posts: 166

Re: connect to 127.0.0.1[127.0.0.1]:10026: Connection refused

Curiously, more often than not, the process being killed more than clamd is swapoff. I don't understand why swapoff is being killed, when that is the process needed to use the virtual memory.

Tried changing the swappiness value, but it was already at 1. Changing it to 60 did nothing.

9 Reply by Cthulhu

  • Cthulhu
  • Member
  • Offline
  • Registered: 2019-06-04
  • Posts: 925

Re: connect to 127.0.0.1[127.0.0.1]:10026: Connection refused

If processes reserve more memory than the sum of the swap area and part of the RAM, the allocation fails.

As already said: increase the specs or live with the problems

10 Reply by schnappi (edited by schnappi 2025-01-09 10:06:34)

  • schnappi
  • Member
  • Offline
  • Registered: 2015-02-13
  • Posts: 166

Re: connect to 127.0.0.1[127.0.0.1]:10026: Connection refused

I agree, which is why I am surprised that processes are being killed for lack of memory at all when the sum of physical ram and swap space equals 4 GB.

I think there is more to this than simply lack of memory. I will up the swap space to 4 GB and see if processes are still being killed when 6 GB of memory are available.

11 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,165

Re: connect to 127.0.0.1[127.0.0.1]:10026: Connection refused

I suggest don't count swap as memory at all.
Also, increase the swap too.

By the way, do you use swap file or a disk partition as swap?

12 Reply by schnappi

  • schnappi
  • Member
  • Offline
  • Registered: 2015-02-13
  • Posts: 166

Re: connect to 127.0.0.1[127.0.0.1]:10026: Connection refused

ZhangHuangbin wrote:

I suggest don't count swap as memory at all.
Also, increase the swap too.

By the way, do you use swap file or a disk partition as swap?

Swap file.