December 2010 - WLUG - wlug.mailman3.com

Re: [Wlug] Dec Meeting announcement / linux-raid problem

by Eric Martin

On Mon, Dec 20, 2010 at 4:52 PM, John Stoffel <john(a)stoffel.org> wrote: > >>>>> "Eric" == Eric Martin <eric.joshua.martin(a)gmail.com> writes: > > Eric> Now, the main reason for my email. My mythbox / home file > Eric> server runs linux soft-raid in RAID5 with 3 500GB disks. One > Eric> disk went bad a while ago and the array never rebuilt itself. > > So you only had two 500Gb disks left in the array? > Correct. I just received my new disk yesterday and dd_rescued the data from the bad disk onto a fresh disk. the array won't start, so here's the info you asked for > > Eric> The other day, the second disk went bad. Am I hosed? > > Possibly, it depends on how bad the second disk is. What I would do > is try to use dd_rescue to copy the 2nd bad disk onto a new disk > (possibly your original bad disk if you feel brave!) and then try to > re-assemble your raid 5 using that. You might or might not have > corruption in the filesystem, so make sure you run an fsck on it. > > Now, in the future, you should run a weekly check of the array for bad > blocks or other problems, so that you get notified if a disk dies > silently. I use the following crontab entry: > BTW, this is what I was missing. there was no warning that my disk was bad! Like I said, I have backups but the last one failed so I want something tighter > > > > Eric> I've been googling for 'rebuild bad linux software raid' but all > Eric> I get is the rebuild command. Also, I don't see any tools that > Eric> will move bad data to another spot on the disk. This is my > Eric> first time using software raid so I'm in a bit over my head. > > The first thing is to ask for help on the linux-raid mailing list, > which is hosted on vger.kernel.org. > > But somethings you can do to help is to give us more information. > Like: > > cat /proc/mdstat > Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md2 : inactive sda3[0](S) 4000064 blocks md3 : inactive sda4[0] 483315456 blocks unused devices: <none> > > mdadm -E /dev/sd... > > or /dev/hd... depending on whether your SATA or IDE drives. > Basically, use the devices you got from the /proc/mdstat output as > your basis. > > Give us this output, and we should be able to help you more. > livecd / # mdadm -E /dev/sda3 /dev/sda3: Magic : a92b4efc Version : 00.90.00 UUID : 8374ea27:6e191996:e56f6693:e45468a9 Creation Time : Sat Jul 11 17:14:31 2009 Raid Level : raid5 Used Dev Size : 4000064 (3.81 GiB 4.10 GB) Array Size : 8000128 (7.63 GiB 8.19 GB) Raid Devices : 3 Total Devices : 2 Preferred Minor : 0 Update Time : Mon Dec 13 03:59:27 2010 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 1 Spare Devices : 0 Checksum : fc28b820 - correct Events : 0.496842 Layout : left-symmetric Chunk Size : 64K Number Major Minor RaidDevice State this 0 8 3 0 active sync /dev/sda3 0 0 8 3 0 active sync /dev/sda3 1 1 8 19 1 active sync 2 2 0 0 2 faulty removed livecd / # mdadm -E /dev/sda4 /dev/sda4: Magic : a92b4efc Version : 00.90.00 UUID : c7b07c90:cbd50faf:bc824667:2504996b Creation Time : Sat Jul 11 16:52:52 2009 Raid Level : raid5 Used Dev Size : 483315456 (460.93 GiB 494.92 GB) Array Size : 966630912 (921.85 GiB 989.83 GB) Raid Devices : 3 Total Devices : 2 Preferred Minor : 3 Update Time : Thu Dec 9 11:13:25 2010 State : clean Active Devices : 1 Working Devices : 1 Failed Devices : 2 Spare Devices : 0 Checksum : d43b9ad8 - correct Events : 0.15550817 Layout : left-symmetric Chunk Size : 64K Number Major Minor RaidDevice State this 0 8 4 0 active sync /dev/sda4 0 0 8 4 0 active sync /dev/sda4 1 1 0 0 1 faulty removed 2 2 0 0 2 faulty removed livecd / # mdadm -E /dev/sdb3 mdadm: cannot open /dev/sdb3: No such file or directory livecd / # mdadm -E /dev/sdb4 mdadm: cannot open /dev/sdb4: No such file or directory livecd / # mdadm -E /dev/sdc4 /dev/sdc4: Magic : a92b4efc Version : 00.90.00 UUID : c7b07c90:cbd50faf:bc824667:2504996b Creation Time : Sat Jul 11 16:52:52 2009 Raid Level : raid5 Used Dev Size : 483315456 (460.93 GiB 494.92 GB) Array Size : 966630912 (921.85 GiB 989.83 GB) Raid Devices : 3 Total Devices : 2 Preferred Minor : 3 Update Time : Thu Dec 9 11:13:06 2010 State : active Active Devices : 2 Working Devices : 2 Failed Devices : 1 Spare Devices : 0 Checksum : d34e516a - correct Events : 0.15550815 Layout : left-symmetric Chunk Size : 64K Number Major Minor RaidDevice State this 1 8 20 1 active sync 0 0 8 4 0 active sync /dev/sda4 1 1 8 20 1 active sync 2 2 0 0 2 faulty removed livecd / # mdadm -E /dev/sdc3 /dev/sdc3: Magic : a92b4efc Version : 00.90.00 UUID : 8374ea27:6e191996:e56f6693:e45468a9 Creation Time : Sat Jul 11 17:14:31 2009 Raid Level : raid5 Used Dev Size : 4000064 (3.81 GiB 4.10 GB) Array Size : 8000128 (7.63 GiB 8.19 GB) Raid Devices : 3 Total Devices : 2 Preferred Minor : 0 Update Time : Mon Dec 13 03:59:27 2010 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 1 Spare Devices : 0 Checksum : fc28b832 - correct Events : 0.496842 Layout : left-symmetric Chunk Size : 64K Number Major Minor RaidDevice State this 1 8 19 1 active sync 0 0 8 3 0 active sync /dev/sda3 1 1 8 19 1 active sync 2 2 0 0 2 faulty removed /dev/sda is a good disk, /dev/sdc is the bad disk and /dev/sdb is the good disk that has the clone of /dev/sdc. Curiously, mdadm -E doesn't work on /dev/sdb even though the partitions are setup correctly thanks! > > John > > > > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org > http://mail.wlug.org/mailman/listinfo/wlug > -- Eric Martin

Dec. 21, 2010

Announce: Next WLUG meeting & meetings for 2011

by Chuck Anderson

Hi folks, Our next WLUG meeting is scheduled to be held on Wednesday, January 12 at 7:00 PM at the WPI campus in Worcester, MA USA. The meeting room is in WPI's Morgan Hall (building 'H' on the campus map [1]), down the hallway to the right of the dining hall entrance. [1] http://www.wpi.edu/Images/CMS/WPI/walkingmap.pdf Also, I have normalized the meeting schedule to be the 2nd Wednesday of each month. We'll try this for awhile and see how it goes. As always, please email me if you are interested in presenting a topic for any of our future meetings. We're a very laid back group, so don't feel you need to have a polished presentation in order to speak. Demos are always fun too. Some topic ideas we've had are: - Linux Android Development - Linux on Wireless Routers: OpenWRT, DD-WRT, etc. - IRC basics: how to chat on #wlug-ma - PGP/GPG keysigning party - Compare ZFS vs. BTRFS vs. ... - Virtualization Technologies KVM, Xen, VMWare - Demo the latest version of your favorite Linux distribution Happy Holidays and I look forward to seeing you next year, Chuck President, Worcester Linux Users' Group

Dec. 20, 2010

Re: [Wlug] Dec Meeting announcement / linux-raid problem

by John Stoffel

>>>>> "Eric" == Eric Martin <eric.joshua.martin(a)gmail.com> writes: Eric> Now, the main reason for my email. My mythbox / home file Eric> server runs linux soft-raid in RAID5 with 3 500GB disks. One Eric> disk went bad a while ago and the array never rebuilt itself. So you only had two 500Gb disks left in the array? Eric> The other day, the second disk went bad. Am I hosed? Possibly, it depends on how bad the second disk is. What I would do is try to use dd_rescue to copy the 2nd bad disk onto a new disk (possibly your original bad disk if you feel brave!) and then try to re-assemble your raid 5 using that. You might or might not have corruption in the filesystem, so make sure you run an fsck on it. Now, in the future, you should run a weekly check of the array for bad blocks or other problems, so that you get notified if a disk dies silently. I use the following crontab entry: # # cron.d/mdadm -- schedules periodic redundancy checks of MD devices # # Copyright © martin f. krafft <madduck(a)madduck.net> # distributed under the terms of the Artistic Licence 2.0 # # By default, run at 00:57 on every Sunday, but do nothing unless the day of # the month is less than or equal to 7. Thus, only run on the first Sunday of # each month. crontab(5) sucks, unfortunately, in this regard; therefore this # hack (see #380425). 57 0 * * 0 root if [ -x /usr/share/mdadm/checkarray ] && [ $(date +\%d) -le 7 ]; then /usr/share/mdadm/checkarray --cron --all --idle --quiet; fi and I get a nice weekly report on both arrays on my main filerserver at thome. Eric> I've been googling for 'rebuild bad linux software raid' but all Eric> I get is the rebuild command. Also, I don't see any tools that Eric> will move bad data to another spot on the disk. This is my Eric> first time using software raid so I'm in a bit over my head. The first thing is to ask for help on the linux-raid mailing list, which is hosted on vger.kernel.org. But somethings you can do to help is to give us more information. Like: cat /proc/mdstat mdadm -E /dev/sd... or /dev/hd... depending on whether your SATA or IDE drives. Basically, use the devices you got from the /proc/mdstat output as your basis. Give us this output, and we should be able to help you more. John

Dec. 20, 2010

Re: [Wlug] disk/Raid ideas

by Theo Van Dinter

Did the second drive completely fail or is it just a bad block? Does the array work at all? If the disk is marked bad (RAID metadata), then maybe you can mark it as ok, get the array online, and copy off as much data as possible (starting with the most important files first, of course). You may be able to dd the data per disk, skipping any bad blocks, to another disk and then possibly be able to access some of the array that way. And of course ... you do have a backup of that data somewhere, right? :) In general, some rules of thumb are: - RAID is not a replacement for backups - RAID 5 becomes more problematic as disk sizes + number of disks increase, look at something like RAID 6 (raid-dp, raidz2, whatever your implementation calls it) instead. - Be aggressive about fixing bad disks in order to shorten the catastrophe window. On Sat, Dec 18, 2010 at 1:03 PM, Eric Martin <eric.joshua.martin(a)gmail.com> wrote: > It's raid 5, and I can't recover the group because of a hd error on two > disks. I wanted to see if I could move that data somewhere else to force a > rebuild but it looks like a no. Oh well, to the backup drive. > > On Sat, Dec 18, 2010 at 1:00 PM, Jason Couture <plaguethenet(a)gmail.com> > wrote: >> >> What kind of raid? If it was raid 5 I don't believe you can rebuild the >> array unless you have 2 surviving disks, so unless you can somehow recover >> one of the other two it would seem you are sol. Been a long time since I've >> done anything but 0, 1 and 0+1 so I could be wrong. >> >> Sent from my android wireless device. >> >> On Dec 18, 2010 12:48 PM, "Doug Mildram" <dmildram(a)gmail.com> wrote: >> >> You're hosed (i think) unless >> >> -- you're rich and you send the disks to a shop for megabucks (they may >> start with a trivial purchase of 2-3 identical model disks) >> >> -- OR IF your problem was NOT the actual platters/heads, YOU obtain 1-2 >> same-model disks, swap the board from a replacement disk onto your hosed >> disk ( w/ sm torx tool and brave fingers ) >> >> I've not done this myself yet. >> >> I HAVE had "stiction" problems where (after a power out or a "rest" so a >> disk bearing/lubricant gets a chance to harden) the disk just isnt spinning >> up. Of course, tapping/whacking it at the right/initial moment will usually >> cure that and you can feel if it's spinning or not....if you can hold it in >> your hand ...OR somehow get hands-on during powerup. >> >> >> >> _______________________________________________ >> Wlug mailing list >> Wlug(a)mail.wlug.org >> http://mail.wlug.org/mailman/listinfo/wlug >> >> >> _______________________________________________ >> Wlug mailing list >> Wlug(a)mail.wlug.org >> http://mail.wlug.org/mailman/listinfo/wlug >> > > > > -- > Eric Martin > > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org > http://mail.wlug.org/mailman/listinfo/wlug > >

Dec. 19, 2010

Re: [Wlug] Dec Meeting announcement / linux-raid problem

by soup

I don't run software raid so I'm not sure how it handles filesystem errors but I've used ddrescue to recover drives/arrays that were written off. It's fairly easy to use but can take quite a while depending on what the drive failure was. Simple bad sectors in a bad place it will usually chew through pretty quick, though. Take care, James On 12/18/2010 1:01 PM, Eric Martin wrote: > I have backups of some stuff but I have a program that isn't great > about writing stuff properly so I wanted to see if I could get more. > If I can't it's ok, the backups will suffice. I was just hoping for > more :) The replacement is on it's way so this is a moot point pretty > fast. > > On Sat, Dec 18, 2010 at 11:28 AM, James Gray <jamespgray(a)gmail.com > <mailto:jamespgray@gmail.com>> wrote: > > It appears there has been very little traffic as of late. Did you > compare the messages you've received to the wlug archives? > http://mail.wlug.org/pipermail/wlug/ > > As for your RAID issue you I don't have any advice as I have virtually > no experience with it. I assume if you had backups you won't have > bothered writing the group. > > On Fri, Dec 17, 2010 at 6:30 AM, Eric Martin > <eric.joshua.martin(a)gmail.com > <mailto:eric.joshua.martin@gmail.com>> wrote: > > > > I just realized that I haven't received anything from WLUG in > over a month, including the announcement for this past meeting. > Did anybody else have this problem or is it just me? > > > > Now, the main reason for my email. My mythbox / home file > server runs linux soft-raid in RAID5 with 3 500GB disks. One disk > went bad a while ago and the array never rebuilt itself. The > other day, the second disk went bad. Am I hosed? I've been > googling for 'rebuild bad linux software raid' but all I get is > the rebuild command. Also, I don't see any tools that will move > bad data to another spot on the disk. This is my first time using > software raid so I'm in a bit over my head. > > > > thanks! > > > > -- > > Eric Martin > > > > _______________________________________________ > > Wlug mailing list > > Wlug(a)mail.wlug.org <mailto:Wlug@mail.wlug.org> > > http://mail.wlug.org/mailman/listinfo/wlug > > > > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org <mailto:Wlug@mail.wlug.org> > http://mail.wlug.org/mailman/listinfo/wlug > > > > > -- > Eric Martin > > > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org > http://mail.wlug.org/mailman/listinfo/wlug

Dec. 19, 2010

Re: [Wlug] Dec Meeting announcement / linux-raid problem

by Jason Couture

I hear you man, good luck. At least you had the backups. Sent from my android wireless device. On Dec 18, 2010 1:02 PM, "Eric Martin" <eric.joshua.martin(a)gmail.com> wrote: I have backups of some stuff but I have a program that isn't great about writing stuff properly so I wanted to see if I could get more. If I can't it's ok, the backups will suffice. I was just hoping for more :) The replacement is on it's way so this is a moot point pretty fast. On Sat, Dec 18, 2010 at 11:28 AM, James Gray <jamespgray(a)gmail.com> wrote: > > It appears there ha... -- Eric Martin _______________________________________________ Wlug mailing list Wlug(a)mail.wlug.org http://mail.wlug.org/mailman/listinfo/wlug

Dec. 18, 2010

Re: [Wlug] disk/Raid ideas

by Eric Martin

It's raid 5, and I can't recover the group because of a hd error on two disks. I wanted to see if I could move that data somewhere else to force a rebuild but it looks like a no. Oh well, to the backup drive. On Sat, Dec 18, 2010 at 1:00 PM, Jason Couture <plaguethenet(a)gmail.com>wrote: > What kind of raid? If it was raid 5 I don't believe you can rebuild the > array unless you have 2 surviving disks, so unless you can somehow recover > one of the other two it would seem you are sol. Been a long time since I've > done anything but 0, 1 and 0+1 so I could be wrong. > > Sent from my android wireless device. > > On Dec 18, 2010 12:48 PM, "Doug Mildram" <dmildram(a)gmail.com> wrote: > > You're hosed (i think) unless > > -- you're rich and you send the disks to a shop for megabucks (they may > start with a trivial purchase of 2-3 identical model disks) > > -- OR IF your problem was NOT the actual platters/heads, YOU obtain 1-2 > same-model disks, swap the board from a replacement disk onto your hosed > disk ( w/ sm torx tool and brave fingers ) > > I've not done this myself yet. > > I HAVE had "stiction" problems where (after a power out or a "rest" so a > disk bearing/lubricant gets a chance to harden) the disk just isnt spinning > up. Of course, tapping/whacking it at the right/initial moment will usually > cure that and you can feel if it's spinning or not....if you can hold it in > your hand ...OR somehow get hands-on during powerup. > > > > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org > http://mail.wlug.org/mailman/listinfo/wlug > > > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org > http://mail.wlug.org/mailman/listinfo/wlug > > -- Eric Martin

Dec. 18, 2010

Re: [Wlug] Dec Meeting announcement / linux-raid problem

by Eric Martin

I have backups of some stuff but I have a program that isn't great about writing stuff properly so I wanted to see if I could get more. If I can't it's ok, the backups will suffice. I was just hoping for more :) The replacement is on it's way so this is a moot point pretty fast. On Sat, Dec 18, 2010 at 11:28 AM, James Gray <jamespgray(a)gmail.com> wrote: > It appears there has been very little traffic as of late. Did you > compare the messages you've received to the wlug archives? > http://mail.wlug.org/pipermail/wlug/ > > As for your RAID issue you I don't have any advice as I have virtually > no experience with it. I assume if you had backups you won't have > bothered writing the group. > > On Fri, Dec 17, 2010 at 6:30 AM, Eric Martin > <eric.joshua.martin(a)gmail.com> wrote: > > > > I just realized that I haven't received anything from WLUG in over a > month, including the announcement for this past meeting. Did anybody else > have this problem or is it just me? > > > > Now, the main reason for my email. My mythbox / home file server runs > linux soft-raid in RAID5 with 3 500GB disks. One disk went bad a while ago > and the array never rebuilt itself. The other day, the second disk went > bad. Am I hosed? I've been googling for 'rebuild bad linux software raid' > but all I get is the rebuild command. Also, I don't see any tools that will > move bad data to another spot on the disk. This is my first time using > software raid so I'm in a bit over my head. > > > > thanks! > > > > -- > > Eric Martin > > > > _______________________________________________ > > Wlug mailing list > > Wlug(a)mail.wlug.org > > http://mail.wlug.org/mailman/listinfo/wlug > > > > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org > http://mail.wlug.org/mailman/listinfo/wlug > -- Eric Martin

Dec. 18, 2010

Re: [Wlug] disk/Raid ideas

by Jason Couture

What kind of raid? If it was raid 5 I don't believe you can rebuild the array unless you have 2 surviving disks, so unless you can somehow recover one of the other two it would seem you are sol. Been a long time since I've done anything but 0, 1 and 0+1 so I could be wrong. Sent from my android wireless device. On Dec 18, 2010 12:48 PM, "Doug Mildram" <dmildram(a)gmail.com> wrote: You're hosed (i think) unless -- you're rich and you send the disks to a shop for megabucks (they may start with a trivial purchase of 2-3 identical model disks) -- OR IF your problem was NOT the actual platters/heads, YOU obtain 1-2 same-model disks, swap the board from a replacement disk onto your hosed disk ( w/ sm torx tool and brave fingers ) I've not done this myself yet. I HAVE had "stiction" problems where (after a power out or a "rest" so a disk bearing/lubricant gets a chance to harden) the disk just isnt spinning up. Of course, tapping/whacking it at the right/initial moment will usually cure that and you can feel if it's spinning or not....if you can hold it in your hand ...OR somehow get hands-on during powerup. _______________________________________________ Wlug mailing list Wlug(a)mail.wlug.org http://mail.wlug.org/mailman/listinfo/wlug

Dec. 18, 2010

disk/Raid ideas

by Doug Mildram

You're hosed (i think) unless -- you're rich and you send the disks to a shop for megabucks (they may start with a trivial purchase of 2-3 identical model disks) -- OR IF your problem was NOT the actual platters/heads, YOU obtain 1-2 same-model disks, swap the board from a replacement disk onto your hosed disk ( w/ sm torx tool and brave fingers ) I've not done this myself yet. I HAVE had "stiction" problems where (after a power out or a "rest" so a disk bearing/lubricant gets a chance to harden) the disk just isnt spinning up. Of course, tapping/whacking it at the right/initial moment will usually cure that and you can feel if it's spinning or not....if you can hold it in your hand ...OR somehow get hands-on during powerup.

Dec. 18, 2010