Hello, my www and mail server hanged suddenly several times for last three days. The only one solution was to reset computer after every situation after which soft raid starts replicating. Configuration: Debian 6.0.5 Kernel 2.6.32-5-amd64 Pentium 4 3.0GHz RAM 4GB 4 x 1TB HDD in soft raid 10 md0 ext2 /boot ~100MB md2 ext4 / ~2TB What additional information should I attach to help in problem analyze? Can you help me? Here are log samples, these errors can appear in logs every 2 minutes: syslog: Code: Sep 3 01:02:45 web kernel: [9020573.486487] BUG: Bad page map in process rkhunter pte:8000000057494045 pmd:dd23f067 Sep 3 01:02:45 web kernel: [9020573.486498] page:ffffea0001318060 flags:010000000002003c count:1 mapcount:-1 mapping:ffff88011db39518 index:296 Sep 3 01:02:45 web kernel: [9020573.486506] addr:0000000000e55000 vm_flags:00100073 anon_vma:ffff8800ce79a560 mapping:(null) index:e55 Sep 3 01:02:45 web kernel: [9020573.486515] Pid: 31311, comm: rkhunter Not tainted 2.6.32-5-amd64 #1 Sep 3 01:02:45 web kernel: [9020573.486519] Call Trace: Sep 3 01:02:45 web kernel: [9020573.486534] [<ffffffff810cb0eb>] ? print_bad_pte+0x232/0x24a Sep 3 01:02:45 web kernel: [9020573.486543] [<ffffffff810cc19b>] ? unmap_vmas+0x62d/0x931 Sep 3 01:02:45 web kernel: [9020573.486552] [<ffffffff810b44ad>] ? lock_page+0x9/0x1f Sep 3 01:02:45 web kernel: [9020573.486562] [<ffffffff810d08a0>] ? exit_mmap+0xc4/0x148 Sep 3 01:02:45 web kernel: [9020573.486570] [<ffffffff8104bca9>] ? mmput+0x3c/0xdf Sep 3 01:02:45 web kernel: [9020573.486578] [<ffffffff8104f942>] ? exit_mm+0x102/0x10d Sep 3 01:02:45 web kernel: [9020573.486586] [<ffffffff81051367>] ? do_exit+0x1f8/0x6c9 Sep 3 01:02:45 web kernel: [9020573.486594] [<ffffffff810518ae>] ? do_group_exit+0x76/0x9d Sep 3 01:02:45 web kernel: [9020573.486601] [<ffffffff810518e7>] ? sys_exit_group+0x12/0x16 Sep 3 01:02:45 web kernel: [9020573.486609] [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b Sep 3 01:02:45 web kernel: [9020573.486614] Disabling lock debugging due to kernel taint Sep 3 01:02:45 web kernel: [9020573.486690] BUG: Bad page state in process rkhunter pfn:57494 Sep 3 01:02:45 web kernel: [9020573.486697] page:ffffea0001318060 flags:010000000002001c count:0 mapcount:-1 mapping:ffff88011db39518 index:296 Sep 3 01:02:45 web kernel: [9020573.486705] Pid: 31311, comm: rkhunter Tainted: G B 2.6.32-5-amd64 #1 Sep 3 01:02:45 web kernel: [9020573.486709] Call Trace: Sep 3 01:02:45 web kernel: [9020573.486716] [<ffffffff8104e553>] ? release_console_sem+0x17e/0x1af Sep 3 01:02:45 web kernel: [9020573.486724] [<ffffffff810b7d89>] ? bad_page+0x116/0x129 Sep 3 01:02:45 web kernel: [9020573.486731] [<ffffffff810b8272>] ? free_pages_check+0x38/0x57 Sep 3 01:02:45 web kernel: [9020573.486738] [<ffffffff810b95d4>] ? free_hot_cold_page+0x46/0x191 Sep 3 01:02:45 web kernel: [9020573.486745] [<ffffffff810b9788>] ? __pagevec_free+0x69/0x80 Sep 3 01:02:45 web kernel: [9020573.486753] [<ffffffff810bc68b>] ? release_pages+0x17b/0x18d Sep 3 01:02:45 web kernel: [9020573.486762] [<ffffffff810168c1>] ? sched_clock+0x5/0x8 Sep 3 01:02:45 web kernel: [9020573.486770] [<ffffffff812faedd>] ? dump_stack+0x69/0x6f Sep 3 01:02:45 web kernel: [9020573.486778] [<ffffffff810d91ad>] ? free_pages_and_swap_cache+0x57/0x73 Sep 3 01:02:45 web kernel: [9020573.486786] [<ffffffff810cc219>] ? unmap_vmas+0x6ab/0x931 Sep 3 01:02:45 web kernel: [9020573.486794] [<ffffffff810b44ad>] ? lock_page+0x9/0x1f Sep 3 01:02:45 web kernel: [9020573.486801] [<ffffffff810d08a0>] ? exit_mmap+0xc4/0x148 Sep 3 01:02:45 web kernel: [9020573.486807] [<ffffffff8104bca9>] ? mmput+0x3c/0xdf Sep 3 01:02:45 web kernel: [9020573.486814] [<ffffffff8104f942>] ? exit_mm+0x102/0x10d Sep 3 01:02:45 web kernel: [9020573.486821] [<ffffffff81051367>] ? do_exit+0x1f8/0x6c9 Sep 3 01:02:45 web kernel: [9020573.486828] [<ffffffff810518ae>] ? do_group_exit+0x76/0x9d Sep 3 01:02:45 web kernel: [9020573.486835] [<ffffffff810518e7>] ? sys_exit_group+0x12/0x16 Sep 3 01:02:45 web kernel: [9020573.486842] [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b kern.log: Code: Sep 4 15:28:02 web kernel: [97653.620196] php[2237]: segfault at 692e3e6d ip 00007f72d9e55739 sp 00007fff355a0368 error 4 in mysql.so[7f72d9e51000+b000] Sep 4 15:28:02 web kernel: [97653.620839] BUG: Bad page map in process php pte:c6403025 pmd:5ca3f067 Sep 4 15:28:02 web kernel: [97653.620848] page:ffffea0002b5e0a8 flags:0100000000020068 count:1 mapcount:-1 mapping:ffff88003e7a58f0 index:0 Sep 4 15:28:02 web kernel: [97653.620855] addr:00007f72d9e55000 vm_flags:08000075 anon_vma:(null) mapping:ffff8801117f33d8 index:4 Sep 4 15:28:02 web kernel: [97653.620865] vma->vm_ops->fault: filemap_fault+0x0/0x2f6 Sep 4 15:28:02 web kernel: [97653.620893] vma->vm_file->f_op->mmap: ext4_file_mmap+0x0/0x47 [ext4] Sep 4 15:28:02 web kernel: [97653.620899] Pid: 2237, comm: php Not tainted 2.6.32-5-amd64 #1 Sep 4 15:28:02 web kernel: [97653.620903] Call Trace: Sep 4 15:28:02 web kernel: [97653.620914] [<ffffffff810cb0eb>] ? print_bad_pte+0x232/0x24a Sep 4 15:28:02 web kernel: [97653.620922] [<ffffffff810cc19b>] ? unmap_vmas+0x62d/0x931 Sep 4 15:28:02 web kernel: [97653.620930] [<ffffffff810d08a0>] ? exit_mmap+0xc4/0x148 Sep 4 15:28:02 web kernel: [97653.620938] [<ffffffff8104bca9>] ? mmput+0x3c/0xdf Sep 4 15:28:02 web kernel: [97653.620944] [<ffffffff8104f942>] ? exit_mm+0x102/0x10d Sep 4 15:28:02 web kernel: [97653.620950] [<ffffffff81051367>] ? do_exit+0x1f8/0x6c9 Sep 4 15:28:02 web kernel: [97653.620958] [<ffffffff810e60a1>] ? virt_to_head_page+0x9/0x2a Sep 4 15:28:02 web kernel: [97653.620965] [<ffffffff810518ae>] ? do_group_exit+0x76/0x9d Sep 4 15:28:02 web kernel: [97653.620973] [<ffffffff8105e19b>] ? get_signal_to_deliver+0x310/0x339 Sep 4 15:28:02 web kernel: [97653.620981] [<ffffffff810fe2ae>] ? d_path+0xc2/0xd2 Sep 4 15:28:02 web kernel: [97653.620989] [<ffffffff81010037>] ? do_notify_resume+0x87/0x73f Sep 4 15:28:02 web kernel: [97653.620999] [<ffffffff812ff1c5>] ? do_page_fault+0x1bf/0x2fc Sep 4 15:28:02 web kernel: [97653.621006] [<ffffffff810115dc>] ? retint_signal+0x48/0x8c Sep 4 15:28:02 web kernel: [97653.621010] Disabling lock debugging due to kernel taint Sep 4 15:28:02 web kernel: [97653.621198] BUG: Bad page state in process php pfn:c6403 Sep 4 15:28:02 web kernel: [97653.621205] page:ffffea0002b5e0a8 flags:0100000000020008 count:0 mapcount:-1 mapping:ffff88003e7a58f0 index:0 Sep 4 15:28:02 web kernel: [97653.621211] Pid: 2237, comm: php Tainted: G B 2.6.32-5-amd64 #1 Sep 4 15:28:02 web kernel: [97653.621215] Call Trace: Sep 4 15:28:02 web kernel: [97653.621222] [<ffffffff810b7d89>] ? bad_page+0x116/0x129 Sep 4 15:28:02 web kernel: [97653.621229] [<ffffffff810b8272>] ? free_pages_check+0x38/0x57 Sep 4 15:28:02 web kernel: [97653.621235] [<ffffffff810b95d4>] ? free_hot_cold_page+0x46/0x191 Sep 4 15:28:02 web kernel: [97653.621241] [<ffffffff810b9788>] ? __pagevec_free+0x69/0x80 Sep 4 15:28:02 web kernel: [97653.621248] [<ffffffff810bc68b>] ? release_pages+0x17b/0x18d Sep 4 15:28:02 web kernel: [97653.621257] [<ffffffff810d91ad>] ? free_pages_and_swap_cache+0x57/0x73 Sep 4 15:28:02 web kernel: [97653.621264] [<ffffffff810cc219>] ? unmap_vmas+0x6ab/0x931 Sep 4 15:28:02 web kernel: [97653.621271] [<ffffffff810d08a0>] ? exit_mmap+0xc4/0x148 Sep 4 15:28:02 web kernel: [97653.621277] [<ffffffff8104bca9>] ? mmput+0x3c/0xdf Sep 4 15:28:02 web kernel: [97653.621283] [<ffffffff8104f942>] ? exit_mm+0x102/0x10d Sep 4 15:28:02 web kernel: [97653.621289] [<ffffffff81051367>] ? do_exit+0x1f8/0x6c9 Sep 4 15:28:02 web kernel: [97653.621296] [<ffffffff810e60a1>] ? virt_to_head_page+0x9/0x2a Sep 4 15:28:02 web kernel: [97653.621302] [<ffffffff810518ae>] ? do_group_exit+0x76/0x9d Sep 4 15:28:02 web kernel: [97653.621309] [<ffffffff8105e19b>] ? get_signal_to_deliver+0x310/0x339 Sep 4 15:28:02 web kernel: [97653.621316] [<ffffffff810fe2ae>] ? d_path+0xc2/0xd2 Sep 4 15:28:02 web kernel: [97653.621323] [<ffffffff81010037>] ? do_notify_resume+0x87/0x73f Sep 4 15:28:02 web kernel: [97653.621330] [<ffffffff812ff1c5>] ? do_page_fault+0x1bf/0x2fc Sep 4 15:28:02 web kernel: [97653.621337] [<ffffffff810115dc>] ? retint_signal+0x48/0x8c Sep 5 00:44:15 web kernel: [131026.568002] BUG: soft lockup - CPU#0 stuck for 61s! [kswapd0:31] Sep 5 00:44:15 web kernel: [131026.568002] Modules linked in: ip6table_filter ip6_tables nls_utf8 isofs udf xt_multiport cpufreq_stats cpufreq_powersave cpufreq_userspace cpufreq_conservative parport_pc ppdev lp parport xt_tcpudp xt_state ipt_LOG nf_conntrack_ftp iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter sco bridge rfcomm ip_tables stp x_tables bnep l2cap bluetooth rfkill binfmt_misc quota_v2 quota_tree fuse ext2 loop firewire_sbp2 snd_hda_codec_analog snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_seq snd_timer snd_seq_device i2c_i801 snd i2c_core pcspkr evdev asus_atk0110 soundcore snd_page_alloc processor button ext4 mbcache jbd2 crc16 raid10 raid1 md_mod sg usbhid hid sr_mod sd_mod crc_t10dif cdrom pata_marvell firewire_ohci ata_generic ahci ata_piix thermal sky2 firewire_core crc_itu_t libata uhci_hcd ehci_hcd scsi_mod usbcore nls_base thermal_sys [last unloaded: scsi_wait_scan] Sep 5 00:44:15 web kernel: [131026.568002] CPU 0: Sep 5 00:44:15 web kernel: [131026.568002] Modules linked in: ip6table_filter ip6_tables nls_utf8 isofs udf xt_multiport cpufreq_stats cpufreq_powersave cpufreq_userspace cpufreq_conservative parport_pc ppdev lp parport xt_tcpudp xt_state ipt_LOG nf_conntrack_ftp iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter sco bridge rfcomm ip_tables stp x_tables bnep l2cap bluetooth rfkill binfmt_misc quota_v2 quota_tree fuse ext2 loop firewire_sbp2 snd_hda_codec_analog snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_seq snd_timer snd_seq_device i2c_i801 snd i2c_core pcspkr evdev asus_atk0110 soundcore snd_page_alloc processor button ext4 mbcache jbd2 crc16 raid10 raid1 md_mod sg usbhid hid sr_mod sd_mod crc_t10dif cdrom pata_marvell firewire_ohci ata_generic ahci ata_piix thermal sky2 firewire_core crc_itu_t libata uhci_hcd ehci_hcd scsi_mod usbcore nls_base thermal_sys [last unloaded: scsi_wait_scan] Sep 5 00:44:15 web kernel: [131026.568002] Pid: 31, comm: kswapd0 Tainted: G B 2.6.32-5-amd64 #1 System Product Name Sep 5 00:44:15 web kernel: [131026.568002] RIP: 0010:[<ffffffff810b4289>] [<ffffffff810b4289>] find_get_pages+0x5f/0xbb Sep 5 00:44:15 web kernel: [131026.568002] RSP: 0018:ffff88011ceabbc0 EFLAGS: 00000297 Sep 5 00:44:15 web kernel: [131026.568002] RAX: ffffffffffffffff RBX: ffff88011ceabc50 RCX: 0000000000000000 Sep 5 00:44:15 web kernel: [131026.568002] RDX: 0000000000000000 RSI: ffffea0002b5e0b0 RDI: ffffea0002b5e0a8 Sep 5 00:44:15 web kernel: [131026.568002] RBP: ffffffff8101166e R08: ffff88011fc03080 R09: ffff880000045480 Sep 5 00:44:15 web kernel: [131026.568002] R10: 0000000000000002 R11: ffff88011364caf0 R12: 0000000000000020 Sep 5 00:44:15 web kernel: [131026.568002] R13: ffff88011ceabc90 R14: fffffffffffffffe R15: 0000000000000000 Sep 5 00:44:15 web kernel: [131026.568002] FS: 0000000000000000(0000) GS:ffff880005400000(0000) knlGS:0000000000000000 Sep 5 00:44:15 web kernel: [131026.568002] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Sep 5 00:44:15 web kernel: [131026.568002] CR2: 00000000014cd098 CR3: 0000000001001000 CR4: 00000000000006f0 Sep 5 00:44:15 web kernel: [131026.568002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Sep 5 00:44:15 web kernel: [131026.568002] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Sep 5 00:44:15 web kernel: [131026.568002] Call Trace: Sep 5 00:44:15 web kernel: [131026.568002] [<ffffffff810bc3b8>] ? pagevec_lookup+0x17/0x1e Sep 5 00:44:15 web kernel: [131026.568002] [<ffffffff810bd175>] ? invalidate_mapping_pages+0xb9/0xdb Sep 5 00:44:15 web kernel: [131026.568002] [<ffffffff810fd771>] ? d_kill+0x58/0x61 Sep 5 00:44:15 web kernel: [131026.568002] [<ffffffff810bb55d>] ? throttle_vm_writeout+0x30/0x8d Sep 5 00:44:15 web kernel: [131026.568002] [<ffffffff81100b17>] ? shrink_icache_memory+0xfc/0x228 Sep 5 00:44:15 web kernel: [131026.568002] [<ffffffff810bf779>] ? shrink_slab+0xe0/0x153 Sep 5 00:44:15 web kernel: [131026.568002] [<ffffffff810c001c>] ? kswapd+0x4d9/0x686 Sep 5 00:44:15 web kernel: [131026.568002] [<ffffffff810bd693>] ? isolate_pages_global+0x0/0x20f Sep 5 00:44:15 web kernel: [131026.568002] [<ffffffff81065042>] ? autoremove_wake_function+0x0/0x2e Sep 5 00:44:15 web kernel: [131026.568002] [<ffffffff810bfb43>] ? kswapd+0x0/0x686 Sep 5 00:44:15 web kernel: [131026.568002] [<ffffffff81064d75>] ? kthread+0x79/0x81 Sep 5 00:44:15 web kernel: [131026.568002] [<ffffffff81011baa>] ? child_rip+0xa/0x20 Sep 5 00:44:15 web kernel: [131026.568002] [<ffffffff81064cfc>] ? kthread+0x0/0x81 Sep 5 00:44:15 web kernel: [131026.568002] [<ffffffff81011ba0>] ? child_rip+0x0/0x20 Sep 5 00:45:20 web kernel: [131092.068003] BUG: soft lockup - CPU#0 stuck for 61s! [kswapd0:31] Sep 5 00:45:20 web kernel: [131092.068003] Modules linked in: ip6table_filter ip6_tables nls_utf8 isofs udf xt_multiport cpufreq_stats cpufreq_powersave cpufreq_userspace cpufreq_conservative parport_pc ppdev lp parport xt_tcpudp xt_state ipt_LOG nf_conntrack_ftp iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter sco bridge rfcomm ip_tables stp x_tables bnep l2cap bluetooth rfkill binfmt_misc quota_v2 quota_tree fuse ext2 loop firewire_sbp2 snd_hda_codec_analog snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_seq snd_timer snd_seq_device i2c_i801 snd i2c_core pcspkr evdev asus_atk0110 soundcore snd_page_alloc processor button ext4 mbcache jbd2 crc16 raid10 raid1 md_mod sg usbhid hid sr_mod sd_mod crc_t10dif cdrom pata_marvell firewire_ohci ata_generic ahci ata_piix thermal sky2 firewire_core crc_itu_t libata uhci_hcd ehci_hcd scsi_mod usbcore nls_base thermal_sys [last unloaded: scsi_wait_scan] Sep 5 00:45:20 web kernel: [131092.068003] CPU 0: Sep 5 00:45:20 web kernel: [131092.068003] Modules linked in: ip6table_filter ip6_tables nls_utf8 isofs udf xt_multiport cpufreq_stats cpufreq_powersave cpufreq_userspace cpufreq_conservative parport_pc ppdev lp parport xt_tcpudp xt_state ipt_LOG nf_conntrack_ftp iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter sco bridge rfcomm ip_tables stp x_tables bnep l2cap bluetooth rfkill binfmt_misc quota_v2 quota_tree fuse ext2 loop firewire_sbp2 snd_hda_codec_analog snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_seq snd_timer snd_seq_device i2c_i801 snd i2c_core pcspkr evdev asus_atk0110 soundcore snd_page_alloc processor button ext4 mbcache jbd2 crc16 raid10 raid1 md_mod sg usbhid hid sr_mod sd_mod crc_t10dif cdrom pata_marvell firewire_ohci ata_generic ahci ata_piix thermal sky2 firewire_core crc_itu_t libata uhci_hcd ehci_hcd scsi_mod usbcore nls_base thermal_sys [last unloaded: scsi_wait_scan] Sep 5 09:15:08 web kernel: [ 0.000000] Initializing cgroup subsys cpuset Sep 5 09:15:08 web kernel: [ 0.000000] Initializing cgroup subsys cpu Sep 5 09:15:08 web kernel: [ 0.000000] Linux version 2.6.32-5-amd64 (Debian 2.6.32-45) ([email protected]) (gcc version 4.3.5 (Debian 4.3.5-4) ) #1 SMP Sun May 6 04:00:17 UTC 2012 Sep 5 09:15:08 web kernel: [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-2.6.32-5-amd64 root=UUID=4f6538ff-3a89-4bbf-8c5f-1b990bc4782f ro quiet ...
