hi I have a lvm setup with 4 hard disks in total 2TB size. Reiserfs is the file system used. Well coming to the problem. One of the hard disk has around 130 badblocks. I ran badblocks against the disk which is /dev/sda and it has given the block entries which are corrupted in a file. What is the best solution to fix this issue. I have a huge database running on that server which doesnt want to start due to this isssue. After doing little search i found that if the problem is on a single hard drive with ext2 file system The blocks exist on the sda4 partition on my case fsck -t ext -l badblocks-logfile /dev/sda4 should fix the issue if its reiserfs then reiserfsck -B badblocks-logfile /dev/sda4 should i do the same with lvm and will it work fine ? what am worried is if am going to attempt the above step will it disturb the current lvm setup or is there any other bestway to do it. I did come across another document which i couldnt follow the first step itself. http://smartmontools.sourceforge.net/BadBlockHowTo.txt Any help would be much appreciated.
I didnt try it because when i started with smartctl at the begining it failed so just left it. I will try from this part and will get back to you. Thank you for your reply.
I Followed each step carefully mentioned in the url http://smartmontools.sourceforge.net/BadBlockHowTo.txt by Federic BOITEUX and here are the Results of the test: Step 1 smartctl -l selftest /dev/sda === START OF READ SMART DATA SECTION === SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed: read failure 90% 13365 227328439 Step 2 Disk /dev/sda: 60801 cylinders, 255 heads, 63 sectors/track Units = sectors of 512 bytes, counting from 0 Device Boot Start End #sectors Id System /dev/sda1 * 63 401624 401562 83 Linux /dev/sda2 401625 12691349 12289725 82 Linux swap / Solaris /dev/sda3 12691350 24981074 12289725 83 Linux /dev/sda4 24981075 976768064 951786990 8e Linux LVM Step 3 (227328439 - 24981075) = 202347364 Step 4 pvdisplay -c /dev/sda4 | awk -F: '{print $8}' 4096 To get its size in LBA block size (512 bytes or 0.5 KB), we multiply this number by 2 : 4096 * 2 = 8192 blocks for each PE. Either, you can look in /etc/lvm/backup : # grep pe_start $(grep -l /dev/sda4 /etc/lvm/backup/*) pe_start = 384 Step 5 Then, we search in which PE is the badblock, calculating the PE rank in which the faulty block of the partition is : physical partition's bad block number / sizeof(PE) = 202347364 / 8192 = 24700.6059 Step 6 server#lvdisplay --maps --- Logical volume --- LV Name /dev/vg/lv VG Name vg LV UUID zkcUSw-Dpum-aIXr-jE2Y-ob8Z-eqlF-AKQWnP LV Write Access read/write LV Status available # open 2 LV Size 1.81 TB Current LE 473886 Segments 4 Allocation inherit Read ahead sectors 0 Block device 253:0 --- Segments --- Logical extent 0 to 119233: Type linear Physical volume /dev/sdb1 Physical extents 0 to 119233 Logical extent 119234 to 238467: Type linear Physical volume /dev/sdd1 Physical extents 0 to 119233 Logical extent 238468 to 354651: Type linear Physical volume /dev/sda4 Physical extents 0 to 116183 Logical extent 354652 to 473885: Type linear Physical volume /dev/sde1 Physical extents 0 to 119233 Step 7 * bad block number for the filesystem : --------------------------------------- Since my physical extent for the partition /dev/sda4 starts from 0 Physical extent (0 * 8192) + 384 = 384 (202347364 - 384) = 202346980 /(sizeof(fs block) / 512) 202346980 / (4096/512) = 202346980 / 8 = 25293372.5 As we can see from my lvdisplay all harddisks physical extents start from 0, so if i use the formula using physical extent its not working out. I need a formula which would get me the right block using logical extent range Logical extent 238468 to 354651: the calculated value in Step 5 24700.6059 also comes within the range. I tried using the same formula in Step 7 but with the logical extent values but the outputs are negative. Logical extent 238468 to 354651: (238468 * 8192) + 384 = 1953529856 + 384 = 1953530240 (202347364 - 1953530240) = -1751182876 / 8 = -218897859.5 Step 8 * Test of the fs bad block : dd if=/dev/vg/lv of=block25293372 bs=4096 count=1 skip=25293372 This test returns successful which means the calculated block is wrong for the lvm. #smartctl -A /dev/sda smartctl version 5.37 [i386-redhat-linux-gnu] Copyright (C) 2002-6 Bruce Allen Home page is http://smartmontools.sourceforge.net/ === START OF READ SMART DATA SECTION === SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0003 218 217 021 Pre-fail Always - 6100 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 98 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 200 200 051 Pre-fail Always - 0 9 Power_On_Hours 0x0032 081 081 000 Old_age Always - 14034 10 Spin_Retry_Count 0x0013 100 253 051 Pre-fail Always - 0 11 Calibration_Retry_Count 0x0012 100 253 051 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 97 194 Temperature_Celsius 0x0022 253 253 000 Old_age Always - 44 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0012 200 200 000 Old_age Always - 3 198 Offline_Uncorrectable 0x0010 200 200 000 Old_age Offline - 1 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0009 200 200 051 Pre-fail Offline - 0 As we can see from the above output there are 3 currently pending sectors and 1 Offline Uncorrectable sector. Badblocks sees around 100 badblocks in the /dev/sda hard disk. Also Mutt has two mails saying the following which confirms there is badblocks in the /dev/sda hard disk. The following warning/error was logged by the smartd daemon: Device: /dev/sda, 3 Currently unreadable (pending) sectors For details see host's SYSLOG (default: /var/log/messages). The following warning/error was logged by the smartd daemon: Device: /dev/sda, 1 Offline uncorrectable sectors For details see host's SYSLOG (default: /var/log/messages). Apart from the above steps i have tried reiserfsck --check /dev/vg/lv which said no corruptions found. debugreiserfs -B badblock.log /dev/vg/lv gave me no currently marked badblocks in the file system. This step is mentioned here http://chichkin_i.zelnet.ru/bad-block-handling.html I Really dont know how to fix this. Any help would be much appreciated. Thanks. Mohan.
