Topic: Large I/O on server (DDoS or whatever...)
- iRedMail version: from 0.7.0 updated to 0.8.3
- Store mail accounts in which backend (MySQL):
- Linux/BSD distribution name and version: Debian Squeeze
====
Today I was faced with strange problem. Catastrophic response from my server.
iostat showed me a large I/O. Tps on disk (600-800) and avg-cpu %idle parameter high CPU usage (0.03). That situation was constantly for hours.
$ iostat
avg-cpu: %user %nice %system %iowait %steal %idle
42.93 0.00 42.35 14.69 0.00 0.03Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 727.68 3763.78 8132.43 17387617 37569536
I checked incoming sources to my server and was terrified. 50+ IPs from around the world sending requests. (it is impossible to be a visitors, because the web presentation is from small country with local domain and have 5-15 visits per day)
$ netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
1 108.23.241.78
1 113.208.32.112
1 115.114.58.8
1 12.154.55.40
1 12.34.246.167
1 130.206.18.132
1 138.141.100.73
1 158.109.36.179
1 174.132.149.218
1 193.222.78.6
1 193.222.84.70
1 193.38.113.62
1 195.186.227.50
1 195.186.99.50
1 199.250.129.2
1 200.230.71.10
1 200.230.71.55
1 205.166.218.186
1 205.178.149.7
1 205.188.100.58
1 205.188.159.42
1 205.188.190.2
1 205.188.59.193
1 207.115.17.26
1 207.172.157.20
1 207.46.163.30
1 208.36.123.165
1 208.70.88.10
1 208.91.197.128
1 209.145.111.61
1 209.33.205.5
1 212.166.70.250
1 212.170.233.86
1 212.59.199.125
1 213.221.143.235
1 213.46.255.200
1 213.55.128.8
1 213.55.128.9
1 216.99.131.15
1 216.99.131.16
1 217.114.0.53
1 217.76.128.34
1 220.181.15.194
1 50.19.104.123
1 62.14.4.83
1 62.201.1.2
1 64.12.90.34
1 64.12.90.65
1 64.12.90.97
1 64.12.90.98
1 65.205.78.58
1 65.55.39.12
1 65.55.92.136
1 66.232.205.163
1 66.246.235.44
1 66.40.20.29
1 67.220.48.93
1 68.232.135.213
1 69.64.147.249
1 72.167.238.201
1 72.9.240.201
1 74.54.41.162
1 80.64.32.18
1 80.67.172.24
1 82.98.86.161
1 82.98.86.167
1 89.108.104.4
1 94.189.240.125
1 98.139.214.154
1 Address
1 servers)
2
2 12.102.252.75
2 205.188.103.1
2 205.188.103.2
2 205.188.146.193
2 205.188.156.193
2 209.202.254.14
2 212.40.2.32
2 212.82.111.207
2 213.133.98.98
2 64.12.137.162
2 64.12.90.33
2 64.38.116.12
2 71.74.56.244
2 77.238.177.9
3 205.188.155.110
3 213.133.100.100
3 64.12.137.161
3 65.54.188.110
3 65.55.37.104
3 65.55.37.120
3 65.55.37.88
3 65.55.92.152
3 65.55.92.184
3 71.74.56.243
3 77.238.184.241
3 80.57.35.169
4 204.127.208.75
4 205.188.146.194
4 64.12.139.193
4 65.54.188.126
4 65.55.37.72
5 65.54.188.72
5 65.54.188.94
6 65.55.92.168
8 68.1.17.3
11 61.19.251.167
13 115.78.225.198
13 68.6.19.3
13 98.138.206.39
14 173.194.70.27
14 66.216.129.171
16 115.115.125.50
18 98.136.217.192
23 190.69.78.140
57 91.119.178.10
So after I tried to stop apache, turn off server for a while, drop packets from couple IPs:
$ /sbin/iptables -I INPUT -s x.x.x.x -p tcp -j DROP
$ /sbin/iptables -I INPUT -s x.x.x.x -p udp -j DROP
result was the same.
I installed iRedMail 15 days ago, so I decided to try uninstall and to my surprise that solved a problem tps on disk (2.07) and avg-cpu %idle parameter high CPU usage (97.81).
Really liked iRedMail solution so I would like to consult with someone on this forum about which configuration of iRedMail could cause this situation at all.
----
Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.