ISPconfig 3 not responding to requests on 8080 suddenly

Discussion in 'Installation/Configuration' started by Airbag888, Oct 14, 2014.

  1. Airbag888

    Airbag888 New Member

    All of a sudden we noticed emails were no longer coming in and trying to access ISPconfig 3 as usual via https and port 8080 isn't working. Sites hosted are being served though and I can still access the Digital ocean droplet.

    Anything that I can do to revive via CLI without rebooting?

    Thanks

    EDIT:
    Managed to get access so the problem is the emails no longer being received.
     
    Last edited: Oct 14, 2014
  2. srijan

    srijan New Member HowtoForge Supporter

    Do you use dovecot or courier as imap and pop3 server?

    Are you able to send and receive emails with webmail? An easy test if the local mailsystem works is to login to webmail with a account hosted on the server and send a email to the same address that you used to login to webmail. The email shall arrive within a minute back to your account when the local mailsystem works. If the email does not arrive, then check the mail.log
     
  3. Airbag888

    Airbag888 New Member

    Hi, I'm using Dovecot. I've scrupulously followed the HowTo guide for Debian.

    Right now amavisd stopped running (according to ps wwwaux | grep amavisd )

    In the mail-log I'm seeing this:

    Code:
    Oct 14 15:16:03 andromeda postfix/smtp[9007]: warning: numeric domain name in resource data of MX record for pulik.in: 5.133.183.108
    Oct 14 15:18:04 andromeda amavis[6354]: (06354-11) (!!)file(1) utility (/usr/bin/file) FAILED: run_command: can't fork: Cannot allocate memory at /usr/sbin/amavisd-new line 3586.
    Oct 14 15:21:24 andromeda amavis[6406]: (06406-11) (!!)file(1) utility (/usr/bin/file) FAILED: run_command: can't fork: Cannot allocate memory at /usr/sbin/amavisd-new line 3586, <GEN36> line 209397.
    Oct 14 15:36:03 andromeda postfix/smtp[9607]: warning: numeric domain name in resource data of MX record for pulik.in: 5.133.183.108
    Oct 14 15:36:03 andromeda postfix/smtp[9608]: warning: numeric domain name in resource data of MX record for pulik.in: 5.133.183.108
    Oct 14 15:39:02 andromeda amavis[6354]: (06354-12) (!!)file(1) utility (/usr/bin/file) FAILED: run_command: can't fork: Cannot allocate memory at /usr/sbin/amavisd-new line 3586.
    Oct 14 15:39:32 andromeda amavis[6406]: (06406-12) (!!)file(1) utility (/usr/bin/file) FAILED: run_command: can't fork: Cannot allocate memory at /usr/sbin/amavisd-new line 3586, <GEN36> line 209910.
    Oct 14 15:40:39 andromeda amavis[6354]: (06354-13) (!!)file(1) utility (/usr/bin/file) FAILED: run_command: can't fork: Cannot allocate memory at /usr/sbin/amavisd-new line 3586, <GEN36> line 7327.
    Oct 14 15:54:42 andromeda amavis[6406]: (06406-13) (!!)file(1) utility (/usr/bin/file) FAILED: run_command: can't fork: Cannot allocate memory at /usr/sbin/amavisd-new line 3586, <GEN36> line 349261.
    Oct 14 15:58:37 andromeda amavis[6354]: (06354-14) (!!)file(1) utility (/usr/bin/file) FAILED: run_command: can't fork: Cannot allocate memory at /usr/sbin/amavisd-new line 3586, <GEN36> line 76803.
    Oct 14 16:01:03 andromeda postfix/smtp[10388]: warning: numeric domain name in resource data of MX record for pulik.in: 5.133.183.108
    Oct 14 16:09:03 andromeda amavis[6406]: (06406-14) (!!)file(1) utility (/usr/bin/file) FAILED: run_command: can't fork: Cannot allocate memory at /usr/sbin/amavisd-new line 3586.
    Oct 14 16:10:03 andromeda amavis[6354]: (06354-15) (!!)file(1) utility (/usr/bin/file) FAILED: run_command: can't fork: Cannot allocate memory at /usr/sbin/amavisd-new line 3586.
    Oct 14 16:10:57 andromeda amavis[6406]: (06406-15) (!!)file(1) utility (/usr/bin/file) FAILED: run_command: can't fork: Cannot allocate memory at /usr/sbin/amavisd-new line 3586.
    Oct 14 16:12:02 andromeda dovecot: master: Error: service(pop3): fork() failed: Cannot allocate memory
    Oct 14 16:12:02 andromeda dovecot: master: Error: service(pop3): command startup failed, throttling for 2 secs
    Oct 14 16:12:02 andromeda dovecot: pop3-login: Error: read(pop3) failed: Connection reset by peer
    Oct 14 16:14:07 andromeda amavis[6354]: (06354-16) (!!)file(1) utility (/usr/bin/file) FAILED: run_command: can't fork: Cannot allocate memory at /usr/sbin/amavisd-new line 3586, <GEN36> line 77168.
    Oct 14 16:14:12 andromeda amavis[2230]: (!)Net::Server: 2014/10/14-16:14:12 Bad fork [Cannot allocate memory]\n at line 139 in file /usr/share/perl5/Net/Server/PreForkSimple.pm
    Oct 14 16:16:03 andromeda postfix/smtp[10841]: warning: numeric domain name in resource data of MX record for pulik.in: 5.133.183.108
    I tried restarting with service amavis restart but I get
    Code:
    root@andromeda:~# service amavis restart
    Stopping amavisd: (not running).
    Starting amavisd: (failed).
    
    Memory currently.

    Code:
    root@andromeda:~# cat /proc/meminfo
    MemTotal:        1027076 kB
    MemFree:           80640 kB
    Buffers:            3768 kB
    Cached:            74508 kB
    SwapCached:            0 kB
    Active:           825724 kB
    Inactive:          53992 kB
    Active(anon):     801528 kB
    Inactive(anon):    37524 kB
    Active(file):      24196 kB
    Inactive(file):    16468 kB
    Unevictable:           0 kB
    Mlocked:               0 kB
    SwapTotal:             0 kB
    SwapFree:              0 kB
    Dirty:                12 kB
    Writeback:             0 kB
    AnonPages:        801476 kB
    Mapped:            50316 kB
    Shmem:             37612 kB
    Slab:              24248 kB
    SReclaimable:       8232 kB
    SUnreclaim:        16016 kB
    KernelStack:        1856 kB
    PageTables:        28712 kB
    NFS_Unstable:          0 kB
    Bounce:                0 kB
    WritebackTmp:          0 kB
    CommitLimit:      513536 kB
    Committed_AS:    2091212 kB
    VmallocTotal:   34359738367 kB
    VmallocUsed:        4976 kB
    VmallocChunk:   34359733383 kB
    HardwareCorrupted:     0 kB
    AnonHugePages:         0 kB
    HugePages_Total:       0
    HugePages_Free:        0
    HugePages_Rsvd:        0
    HugePages_Surp:        0
    Hugepagesize:       2048 kB
    DirectMap4k:       40952 kB
    DirectMap2M:     1007616 kB
    
    Code:
    root@andromeda:~# free
                 total       used       free     shared    buffers     cached
    Mem:       1027076     947752      79324          0       4436      77220
    -/+ buffers/cache:     866096     160980
    Swap:            0          0          0
    
     
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    There is not enough free memory on your server. you should try to restart it. If this wont help then you have to assign more memory to your droplet.
     
  5. Airbag888

    Airbag888 New Member

    The service works once the droplet is restarted. However eventually it fails as described. I'll look into allocating more memory. Silly me thought with nginx I'd be ok for longer.

    This is memory usage after a restart:

    Code:
                 total       used       free     shared    buffers     cached
    Mem:       1027076     957528      69548          0       6168     112800
    -/+ buffers/cache:     838560     188516
    Swap:            0          0          0
    
    It seems more memory is 'used' after the restart..

    Btw clamav isn't 'working' could it be causing a memory leak over time? I've done the installation at least 5 different times from scratch following the instructions to the letter and clamav never worked.
     
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    There are no known problems with clamav. Which errors do you get in the clamav log?
     
  7. Airbag888

    Airbag888 New Member

    That's weird I know I had errors at install all the times I did the install and I had a full page of red clamav errors in the log but after restart it's all gone.
    I'll update here or create a new thread when I get them.

    So the server has 1GB RAM. It's hosting a few (8) small low traffic sites with some 30 mailboxes spread throughout running on nginx (which you helped me with before)

    Would you say 1GB isn't enough? Particularly as per memory showed by the 'free' command?
     
  8. srijan

    srijan New Member HowtoForge Supporter

    You can start with a new thread mentioning the reference of this tread with link.

    Yes, Ideally 1Gb is sufficient, please check which processes are eating the RAM Run
    It will enlist top 10 maximum cpu eating programs.
     
  9. Airbag888

    Airbag888 New Member

    Hi,

    This is what I'm getting

    Code:
    root@andromeda:~# ps -eo pcpu,pid,user,args | sort -k1 -r | head -10
    %CPU   PID USER     COMMAND
     1.3 12818 root     -bash
     0.3 11598 amavis   /usr/sbin/amavisd-new (ch6-avail)
     0.1  8872 amavis   /usr/sbin/amavisd-new (ch13-avail)
     0.1  2799 clamav   /usr/bin/freshclam -d --quiet --config-file=/etc/clamav/freshclam.conf
     0.1 12813 root     sshd: root@pts/1
     0.0     9 root     [khelper]
     0.0    99 root     [scsi_eh_1]
     0.0     8 root     [cpuset]
     0.0    85 root     [scsi_eh_0]
    
     
  10. srijan

    srijan New Member HowtoForge Supporter

    Please check once with
    and further press key e
     
  11. Airbag888

    Airbag888 New Member

    Hi

    Code:
    top - 18:32:38 up 10:38,  2 users,  load average: 0.01, 0.02, 0.05
    Tasks: 126 total,   1 running, 125 sleeping,   0 stopped,   0 zombie
    %Cpu(s):  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    KiB Mem:   1027076 total,   843324 used,   183752 free,     8720 buffers
    KiB Swap:        0 total,        0 used,        0 free,   122336 cached
    
      PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
     2972 list      20   0 52856 7928  236 S   0.6  0.8   0:05.16 python
    23019 root      20   0 23360 1496 1064 R   0.6  0.1   0:00.08 top
        1 root      20   0 10648  132    0 S   0.0  0.0   0:01.59 init
        2 root      20   0     0    0    0 S   0.0  0.0   0:00.00 kthreadd
        3 root      20   0     0    0    0 S   0.0  0.0   0:01.79 ksoftirqd/0
        5 root      20   0     0    0    0 S   0.0  0.0   0:00.00 kworker/u:0
        6 root      rt   0     0    0    0 S   0.0  0.0   0:00.00 migration/0
        7 root      rt   0     0    0    0 S   0.0  0.0   0:00.26 watchdog/0
        8 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 cpuset
        9 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 khelper
       10 root      20   0     0    0    0 S   0.0  0.0   0:00.00 kdevtmpfs
       11 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 netns
       12 root      20   0     0    0    0 S   0.0  0.0   0:00.13 sync_supers
       13 root      20   0     0    0    0 S   0.0  0.0   0:00.00 bdi-default
       14 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 kintegrityd
       15 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 kblockd
       17 root      20   0     0    0    0 S   0.0  0.0   0:00.09 khungtaskd
       18 root      20   0     0    0    0 S   0.0  0.0   0:23.36 kswapd0
       19 root      25   5     0    0    0 S   0.0  0.0   0:00.00 ksmd
       20 root      39  19     0    0    0 S   0.0  0.0   0:00.00 khugepaged
       21 root      20   0     0    0    0 S   0.0  0.0   0:00.00 fsnotify_mark
       22 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 crypto
       74 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 ata_sff
       85 root      20   0     0    0    0 S   0.0  0.0   0:00.00 scsi_eh_0
       99 root      20   0     0    0    0 S   0.0  0.0   0:00.01 scsi_eh_1
      100 root      20   0     0    0    0 S   0.0  0.0   0:00.00 kworker/u:1
      137 root      20   0     0    0    0 S   0.0  0.0   0:01.19 jbd2/vda-8
      138 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 ext4-dio-unwrit
      288 root      20   0 21352  496    4 S   0.0  0.0   0:00.07 udevd
      389 root      20   0 21348  496    0 S   0.0  0.0   0:00.00 udevd
      390 root      20   0 21348  496    0 S   0.0  0.0   0:00.00 udevd
     2014 root      20   0     0    0    0 S   0.0  0.0   0:01.27 flush-254:0
     2080 www-data  20   0 10476   92    0 S   0.0  0.0   0:00.00 fcgiwrap
     2122 root      20   0  324m  10m 4708 S   0.0  1.1   0:03.11 php5-fpm
     2124 web2      20   0  320m 6484    0 S   0.0  0.6   0:00.00 php5-fpm
     2125 web2      20   0  320m 6488    0 S   0.0  0.6   0:00.00 php5-fpm
     2128 web14     20   0  338m  53m  31m S   0.0  5.3   0:04.65 php5-fpm
     2129 web14     20   0  333m  47m  31m S   0.0  4.7   0:03.66 php5-fpm
     2130 web8      20   0  320m 6488    0 S   0.0  0.6   0:00.00 php5-fpm
     2132 web8      20   0  320m 6488    0 S   0.0  0.6   0:00.00 php5-fpm
     2134 web6      20   0  342m  53m  29m S   0.0  5.4   0:01.48 php5-fpm
     2139 web6      20   0  328m  32m  22m S   0.0  3.3   0:01.43 php5-fpm
     2140 web9      20   0  320m 6488    0 S   0.0  0.6   0:00.00 php5-fpm
     2142 web9      20   0  320m 6488    0 S   0.0  0.6   0:00.00 php5-fpm
     2144 web5      20   0  344m  59m  31m S   0.0  5.9   0:03.92 php5-fpm
     2145 web5      20   0  414m  79m  31m S   0.0  7.9   0:03.70 php5-fpm
     2146 ispconfi  20   0  327m  29m  22m S   0.0  3.0   0:00.73 php5-fpm
     2147 ispconfi  20   0  328m  30m  21m S   0.0  3.0   0:00.78 php5-fpm
     2148 root      20   0 52968  904   80 S   0.0  0.1   0:03.68 rsyslogd
     2149 web10     20   0  320m 6488    0 S   0.0  0.6   0:00.00 php5-fpm
     2150 web10     20   0  320m 6488    0 S   0.0  0.6   0:00.00 php5-fpm
    
    
    Pressing 'e' gives
    Unknown command - try 'h' for help
     
  12. Airbag888

    Airbag888 New Member

    Just to update. After increasing the RAM I stopped getting the error. Thanks all for the help! So now we're at 2GB droplet.
     

Share This Page