Howto resolve sudden server crashes? Possibly spamassassin or clamscan?

Discussion in 'General' started by linus, Jan 31, 2007.

  1. linus

    linus Member

    Hello and thank you for your wonderful Ispconfig! It is a brilliant set of tools!

    Now I wonder how I should get rid of problems that causes my server to crash several times a year. It did it again this morning at 5:57:18. When I plugged the monitor there was a dump ending with text "closing the console". I was able to ping the machine. The webpages weren't served and the browser went on waiting. At the crash-time in the maillog there was a usual activity probably from spam-mail. I suspect clamscan and spamassassin, as previously spamassassin has generated segfaults.

    How should one resolve errors like this? This is the relevant part of the /var/log/messages-file:

    Jan 31 05:57:18 oslo kernel: general protection fault: 0000 [1] SMP
    Jan 31 05:57:18 oslo kernel: last sysfs file: /devices/system/cpu/cpu0/cpufreq/scaling_setspeed
    Jan 31 05:57:18 oslo kernel: CPU 0
    Jan 31 05:57:18 oslo kernel:
    Modules linked in: xt_tcpudp xt_state ipt_LOG ip_conntrack_ftp
    iptable_mangle iptable_nat ip_nat ip_conntrack nfnetlink iptable_filter
    ip_tables x_tables autofs4 hidp rfcomm l2cap bluetooth sunrpc ipv6 dm_mirror
    dm_mod video button battery acpi_memhotplug ac lp parport_pc parport snd_via82xx gameport
    snd_ac97_codec snd_ac97_bus snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss
    floppy ehci_hcd snd_pcm via_velocity ohci1394 serio_raw i2c_viapro i2c_core uhci_hcd ieee1394 snd_timer snd_page_alloc snd_mpu401_uart

    Jan 31 05:57:18 oslo kernel: Pid: 23720, comm: clamscan Not tainted 2.6.17-1.2187_FC5 #1
    Jan 31 05:57:18 oslo kernel: RIP: 0010:[<ffffffff802075d3>] <ffffffff802075d3>{find_get_page+45}
    Jan 31 05:57:18 oslo kernel: RSP: 0018:ffff81007c46dc68 EFLAGS: 00010006
    Jan 31 05:57:18 oslo kernel: RAX: 2f2f3a7074746822 RBX: 2f2f3a7074746822 RCX: 0000000000000000
    Jan 31 05:57:18 oslo kernel: RDX: 0000000000000000 RSI: 0000000000000100 RDI: ffff81004ec5c4c0
    Jan 31 05:57:18 oslo kernel: RBP: ffff81002ceacc30 R08: 0000000000000004 R09: ffffffff8020caf1
    Jan 31 05:57:18 oslo kernel: R10: ffff81007c46dde8 R11: 0000000000000246 R12: 0000000000000100
    Jan 31 05:57:18 oslo kernel: R13: 00000000000000ff R14: 0000000000000000 R15: ffff81007c46dd88
    Jan 31 05:57:18 oslo kernel: FS: 00002aaaaac13bc0(0000) GS:ffffffff8069e000(0000) knlGS:00000000f7fe56c0
    Jan 31 05:57:18 oslo kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
    Jan 31 05:57:18 oslo kernel: CR2: 000000000051c000 CR3: 0000000070b71000 CR4: 00000000000006e0
    Jan 31 05:57:18 oslo kernel: Process clamscan (pid: 23720, threadinfo ffff81007c46c000, task ffff81001cdf2860)
    Jan 31 05:57:18 oslo kernel: Stack: ffff81007c46dde8 0000000000000000 0000000000000100 ffffffff8020bab4
    Jan 31 05:57:18 oslo kernel: ffffffff8020caf1 ffff81007c46de88 ffff81006bfe0200 ffff81006bfe0280
    Jan 31 05:57:18 oslo kernel: ffff81002ceacc18 ffff81002ceaca88
    Jan 31 05:57:18 oslo kernel: Call Trace: <ffffffff8020bab4>{do_generic_mapping_read+313}
    Jan 31 05:57:18 oslo kernel: <ffffffff8020caf1>{file_read_actor+0} <ffffffff8020bee8>{__generic_file_aio_read+351}
    Jan 31 05:57:18 oslo kernel: <ffffffff802175ff>{generic_file_aio_read+52} <ffffffff8020c891>{do_sync_read+199}
    Jan 31 05:57:18 oslo kernel: <ffffffff80269d8e>{_spin_lock_irqsave+9} <ffffffff8029f419>{autoremove_wake_function+0}
    Jan 31 05:57:18 oslo kernel: <ffffffff80269d4b>{_spin_unlock_irq+9} <ffffffff802677b6>{thread_return+0}
    Jan 31 05:57:18 oslo kernel: <ffffffff80267814>{thread_return+94} <ffffffff8020ae31>{vfs_read+203}
    Jan 31 05:57:18 oslo kernel: <ffffffff802111e3>{sys_read+69} <ffffffff802629ce>{system_call+126}
    Jan 31 05:57:18 oslo kernel:
    Jan 31 05:57:18 oslo kernel: Code: 8b 00 48 89 da f6 c4 40 74 04 48 8b 53 10 f0 ff 42 08 48 89
    Jan 31 05:57:18 oslo kernel: RIP <ffffffff802075d3>{find_get_page+45} RSP <ffff81007c46dc68>
    Jan 31 05:57:18 oslo kernel: <3>BUG: sleeping function called from invalid context at include/linux/rwsem.h:43
    Jan 31 05:57:18 oslo kernel: in_atomic():0, irqs_disabled():1
    Jan 31 05:57:18 oslo kernel:
    Jan 31 05:57:18 oslo kernel: Call Trace: <ffffffff80299774>{blocking_notifier_call_chain+31}
    Jan 31 05:57:18 oslo kernel: <ffffffff80215bf6>{do_exit+32} <ffffffff8027071f>{kernel_math_error+0}
    Jan 31 05:57:18 oslo kernel: <ffffffff8026a9ee>{do_general_protection+254} <ffffffff802638a5>{error_exit+0}
    Jan 31 05:57:19 oslo kernel: <ffffffff8020caf1>{file_read_actor+0} <ffffffff802075d3>{find_get_page+45}
    Jan 31 05:57:19 oslo kernel: <ffffffff802075cb>{find_get_page+37} <ffffffff8020bab4>{do_generic_mapping_read+313}
    Jan 31 05:57:19 oslo kernel: <ffffffff8020caf1>{file_read_actor+0} <ffffffff8020bee8>{__generic_file_aio_read+351}
    Jan 31 05:57:19 oslo kernel: <ffffffff802175ff>{generic_file_aio_read+52} <ffffffff8020c891>{do_sync_read+199}
    Jan 31 05:57:19 oslo kernel: <ffffffff80269d8e>{_spin_lock_irqsave+9} <ffffffff8029f419>{autoremove_wake_function+0}
    Jan 31 05:57:19 oslo kernel: <ffffffff80269d4b>{_spin_unlock_irq+9} <ffffffff802677b6>{thread_return+0}
    Jan 31 05:57:19 oslo kernel: <ffffffff80267814>{thread_return+94} <ffffffff8020ae31>{vfs_read+203}
    Jan 31 05:57:19 oslo kernel: <ffffffff802111e3>{sys_read+69} <ffffffff802629ce>{system_call+126}


    I'd be glad if someone has a clue on how to proceed.

    Vielen danke!
     
  2. martinfst

    martinfst Member Moderator

    A segfault should not (does not, I have never seen it in >20 year of U/Linux experience) lead to a system crash, unless you have been toying around with device drivers and made a booboo.

    For a default installed system I'd run several (extensive) memory checks. Most likely it will appear to be a faulty module. Next to try is a flacky disk. Any read/write errors in your kern.log file?
     
  3. linus

    linus Member

    Further reading

    Hello and thank you very much for helping!

    Do you have any command line program that you could recommend, sufficient for testing the memory? Memtest86 from a floppy?

    Is finding faulty modules possible for regular mortal people?

    The only kernel messages I found in /var/log/messages (except for the bootup ones) were (no read or write errors):

    Jan 16 00:48:14 oslo kernel: clamscan[339]: segfault at 00000051000000cc rip 00002aaaaaab6df6 rsp 00007fff274384c0 error 4
    Jan 16 00:49:22 oslo kernel: spamassassin[520]: segfault at 0000000000000000 rip 0000003ae4d68d94 rsp 00007fff9d937940 error 4

    Jan 18 14:51:51 oslo kernel: ispconfig_httpd[23396]: segfault at 00007fff3ebd4ff0 rip 0000003ae4040bc4 rsp 00007fff3ebd4fd0 error 6

    Jan 23 15:47:11 oslo kernel: ispconfig_httpd[744]: segfault at 00007fff3ebd4f40 rip 0000003ae4040bc4 rsp 00007fff3ebd4f20 error 6
     
  4. martinfst

    martinfst Member Moderator

    Could it be that you mixed 32 and 64 bit libraries? Or you installed / updated to a 64 bit kernel afterwards?

    Yes, Memtest86 or the hardware tests from the manufacturer. Usually systems come with a diagnostics CD/Floppy, which does contain a memory test set. Some BIOS versions include a memtest in the bios-setup.
     

Share This Page