My server has crashed unexpectedly for 2 days

Discussion in 'Server Operation' started by arastirici, Oct 7, 2009.

  1. arastirici

    arastirici New Member

    I use Opensuse11

    My server has crashed for 2 days. I had to restart by the button.
    After the restart, the server works for a time about half a day. and it has crash unexptectedly.

    When I look at the /var/log/messages, I see something messages as below.

    Oct 7 10:15:09 php logrotate: ALERT exited abnormally with [1]
    Oct 7 10:15:09 php logrotate: Reload httpd2 (graceful restart)..done
    Oct 7 10:15:09 php logrotate: Reload httpd2 (graceful restart)..done
    Oct 7 10:15:09 php logrotate: ^G/usr/bin/mysqladmin: connect to server at 'localhost' failed
    Oct 7 10:15:09 php logrotate: error: 'Access denied for user 'root'@'localhost' (using password: NO)'
    Oct 7 10:15:09 php logrotate: /etc/logrotate.d/mysql failed, probably because
    Oct 7 10:15:09 php logrotate: the root acount is protected by password.
    Oct 7 10:15:09 php logrotate: See comments in /etc/logrotate.d/mysql on how to fix this
    Oct 7 10:15:09 php logrotate: error: error running non-shared postrotate script for /var/lib/mysql/mysqld.log of '/var/lib/mysql/my
    Oct 7 10:15:12 php su: (to nobody) root on none
    Oct 7 10:15:32 php syslog-ng[2885]: last message repeated 2 times
    Oct 7 09:47:10 php smartd[3017]: Device: /dev/sda, SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 57 to 58
    Oct 7 09:47:10 php smartd[3017]: Device: /dev/sda, SMART Usage Attribute: 194 Temperature_Celsius changed from 43 to 42
    Oct 7 10:23:21 php kernel: __journal_remove_journal_head: freeing b_frozen_data
    Oct 7 10:25:52 php kernel: Bad page state in process 'kswapd0'
    Oct 7 10:25:52 php kernel: page:c1800000 flags:0x80000008 mapping:d4e86564 mapcount:0 count:0
    Oct 7 10:25:52 php kernel: Trying to fix it up, but a reboot is needed
    Oct 7 10:25:52 php kernel: Backtrace:
    Oct 7 10:25:52 php kernel: Pid: 20, comm: kswapd0 Tainted: G N 2.6.25.16-0.1-pae #1
    Oct 7 10:25:52 php kernel: [<c01071d9>] dump_trace+0x63/0x227
    Oct 7 10:25:52 php kernel: [<c0107c8a>] show_trace+0x15/0x29
    Oct 7 10:25:52 php kernel: [<c02e2c89>] dump_stack+0x5b/0x65
    Oct 7 10:25:52 php kernel: [<c015d9c9>] bad_page+0x55/0x82
    Oct 7 10:25:52 php kernel: [<c015e495>] free_hot_cold_page+0x63/0x1b2
    Oct 7 10:25:52 php kernel: [<c015e601>] __pagevec_free+0x1d/0x25
    Oct 7 10:25:52 php kernel: [<c0160b62>] release_pages+0x14b/0x153
    Oct 7 10:25:52 php kernel: [<c0160f18>] __pagevec_release+0x18/0x21
    Oct 7 10:25:52 php kernel: [<c0162903>] shrink_inactive_list+0x334/0x33f
    Oct 7 10:25:52 php kernel: [<c01629d9>] shrink_zone+0xcb/0xea
    Oct 7 10:25:52 php kernel: [<c0162eaa>] kswapd+0x380/0x4ba
    Oct 7 10:25:52 php kernel: [<c0135f07>] kthread+0x3b/0x61
    Oct 7 10:25:52 php kernel: [<c0106d37>] kernel_thread_helper+0x7/0x10
    Oct 7 10:25:52 php kernel: =======================

    What may the problem be ?
     
  2. damir

    damir New Member

    Seems to be hardware related, have you installed anything hardware related?

    Can you run free command and df -h and paste the output.
     
  3. arastirici

    arastirici New Member

    I haven't installed recently any hardware

    php:/ # free
    total used free shared buffers cached
    Mem: 4139476 388724 3750752 0 22240 198332
    -/+ buffers/cache: 168152 3971324
    Swap: 5124724 0 5124724


    php:/ # df -h
    Filesystem Size Used Avail Use% Mounted on
    /dev/sda1 97G 5.3G 86G 6% /
    udev 2.0G 92K 2.0G 1% /dev
    /dev/sda2 39G 750M 36G 3% /var
    /dev/sda4 114G 66G 43G 61% /www
     
  4. damir

    damir New Member

    There is no swap partition, now it is possible that you are running out of RAM and because swap is missing system crashes.
     
  5. arastirici

    arastirici New Member

    When I run top, swap exist;

    top - 12:15:05 up 1:26, 3 users, load average: 0.00, 0.00, 0.00
    Tasks: 112 total, 1 running, 110 sleeping, 0 stopped, 1 zombie
    Cpu(s): 0.6%us, 0.1%sy, 0.0%ni, 98.1%id, 1.2%wa, 0.0%hi, 0.0%si, 0.0%st
    Mem: 4139476k total, 370808k used, 3768668k free, 32000k buffers
    Swap: 5124724k total, 0k used, 5124724k free, 223684k cached

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    1 root 20 0 772 304 256 S 0 0.0 0:00.76 init
    2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd
    3 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/0
    4 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0
    5 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/1
    6 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1
    7 root 15 -5 0 0 0 S 0 0.0 0:00.04 events/0
    8 root 15 -5 0 0 0 S 0 0.0 0:00.06 events/1
    9 root 15 -5 0 0 0 S 0 0.0 0:00.00 khelper
    10 root 15 -5 0 0 0 S 0 0.0 0:00.02 kblockd/0
    11 root 15 -5 0 0 0 S 0 0.0 0:00.02 kblockd/1
    12 root 15 -5 0 0 0 S 0 0.0 0:00.00 kacpid
    13 root 15 -5 0 0 0 S 0 0.0 0:00.00 kacpi_notify
    14 root 15 -5 0 0 0 S 0 0.0 0:00.00 cqueue
    15 root 15 -5 0 0 0 S 0 0.0 0:00.00 kseriod
    16 root 15 -5 0 0 0 S 0 0.0 0:00.36 kondemand/0
    17 root 15 -5 0 0 0 S 0 0.0 0:00.44 kondemand/1
    18 root 20 0 0 0 0 S 0 0.0 0:00.00 pdflush
    19 root 20 0 0 0 0 S 0 0.0 0:00.06 pdflush
    20 root 15 -5 0 0 0 S 0 0.0 0:00.00 kswapd0
     
  6. damir

    damir New Member

    What is the output od fdisk -l
     
  7. arastirici

    arastirici New Member

    php:/ # fdisk -l

    Disk /dev/sda: 320.0 GB, 320072933376 bytes
    255 heads, 63 sectors/track, 38913 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes
    Disk identifier: 0x90909090

    Device Boot Start End Blocks Id System
    /dev/sda1 * 263 13011 102406342+ 83 Linux
    /dev/sda2 18112 23211 40965750 83 Linux
    /dev/sda3 23212 23849 5124735 82 Linux swap / Solaris
    /dev/sda4 23850 38913 121001580 83 Linux
     
  8. damir

    damir New Member

    I think the problem is that you have to little of RAM memory, wich after a while runs out and system tries to swap but can't find the swap partition and crashes.

    Try from comandline:

    swapon /dev/sda3

    Than edit /etc/fstab and add following:

    /dev/sda3 none swap sw 0 0

    Now if memory runs out, OS is gonna swap to hardrive and it shouldn't crash.

    The best solution is to add more RAM, because RAM is cheap and it's gonna help your system stay online :)
     
  9. arastirici

    arastirici New Member

    Meanwhile, My server has crashed again.
    it has been waiting during the boot. there is "grub loading stage2" on the screen.

    What do I have to do ?
     
  10. arastirici

    arastirici New Member

    I booted the server by opensuse installer. it repaired problems on the disk because of I had to reset on the button.
    I also changed its RAMs.
    Now, the system has been up about 24 hours.
     

Share This Page