March 2010 - WLUG - wlug.mailman3.com

Re: [Wlug] interesting article on RAID

by John Stoffel

Theo> Yeah, it's complicated by the fact that a lot of RAID systems Theo> will consider any I/O error to mean the entire device is bad. This is why Netapp has moved to their double-parity raid sets. It's a help, but not perfect. I'm going to read the article in a few and then comment in particular. Theo> ZFS does a particularly good job here, I've seen single raidz2 Theo> sets able to recover from having 4 simultaneously failing disks. Now that's interesting. How was performance during this issue? Theo> I've had other vendor's arrays start failing disks while Theo> reconstructing from a single failure. Thankfully it was only 1 Theo> additional failure on a 2 parity set. Still PITA. I once had a double disk failure in a Netapp. Ouch! This was before their double parity stuff was widely deployed. One disk died due to head problems, the other died due to the controller board on the disk dying. I managed to swap the good board from the bad hardware disk onto the good hardware, bad board disk, re-insert it and have the system come back up and rebuild. Big sigh of relief there for sure! Another thing I like about alot of newer raid implementations (including Linux RAID, assuming you use it) is the consistency checks or scrubbing that you can run, to hopefully catch errors before they become problems. Reading the disk(s) and checking blocks for consistency, and even re-writing them to make sure the data is good. John

March 12, 2010

Re: [Wlug] interesting article on RAID

by Theo Van Dinter

Yeah, it's complicated by the fact that a lot of RAID systems will consider any I/O error to mean the entire device is bad. ZFS does a particularly good job here, I've seen single raidz2 sets able to recover from having 4 simultaneously failing disks. I've had other vendor's arrays start failing disks while reconstructing from a single failure. Thankfully it was only 1 additional failure on a 2 parity set. Still PITA. On Thu, Mar 11, 2010 at 8:09 PM, Chuck Anderson <cra(a)wpi.edu> wrote: > Triple-Parity RAID and Beyond > by Adam Leventhal | December 17, 2009 > > http://queue.acm.org/detail.cfm?id=1670144 > > As hard-drive capacities continue to outpace their throughput, the > time has come for a new level of RAID.

March 12, 2010

interesting article on RAID

by Chuck Anderson

Triple-Parity RAID and Beyond by Adam Leventhal | December 17, 2009 http://queue.acm.org/detail.cfm?id=1670144 As hard-drive capacities continue to outpace their throughput, the time has come for a new level of RAID. -- - Chuck (372 Days until IPv4 depletion: http://ipv4depletion.com/)

March 12, 2010

Meeting: BtrFS: Now (Thurs March 11 7pm)

by Chuck Anderson

Frank is giving a talk on BtrFS right now. It seems the website wasn't updated (but is now). -- - Chuck (372 Days until IPv4 depletion: http://ipv4depletion.com/) On Mon, Feb 22, 2010 at 06:18:44AM -0500, Frank Sweetser wrote: > I've noticed that not too many people have putting in meeting topic requests, > so I thought that if anyone's interested I'd volunteer to throw together some > slides on BtrFS. > > For those of you who haven't heard of it, BtrFS is the next generation linux > filesystem. Though still in heavy development, it was included in the > mainstream kernel last year. It promises to scale to filesystems well beyond > what any of the current offerings can handle (including ext4), as well as > offering far greater flexibility and better data security for data admins. > > If you have anything else that you'd like to hear about, either instead of or > in addition to BtrFS, chime in, and hopefully someone else on the list will be > able to make it happen! > > -- > Frank Sweetser fs at wpi.edu | For every problem, there is a solution that > WPI Senior Network Engineer | is simple, elegant, and wrong. - HL Mencken > GPG fingerprint = 6174 1257 129E 0D21 D8D4 E8A3 8E39 29E3 E2E8 8CEC

March 12, 2010

Re: [Wlug] Meeting topics

by Jeff Moyer

"John Stoffel" <john(a)stoffel.org> writes: >>>>>> "Jeff" == Jeff Moyer <jmoyer(a)redhat.com> writes: > Jeff> I've seen many claims that SSDs are more reliable than > Jeff> traditional HDDs, but I haven't seen data to back up those > Jeff> claims. > > Some of the arguements I've read on SSD reliability are quite > interesting. If you do the math on how much data you need to write to > destroy an SSD with proper (note!) wear levelling, it's really quite > high. But I think the real key is to write data to them in blocks of > 512 bytes as much as possible, and to not write smaller amounts if at > all possible. Did you mean to say 512KB? You can't write smaller than the exported block size (which will be either 512 bytes or 4k). Perhaps what you meant to say was that you should try to issue writes that are aligned on and multiples of the erase block size. > Jeff> My main concern is that SSDs are still young, so they haven't > Jeff> had much soak time. Whether or not the flash is reliable is > Jeff> only a small part of the battle. Buggy firmware is a much more > Jeff> likely problem at this stage in the game. No single point of > Jeff> failure is a good design point when dealing with your data. ;-) > > I think this is a key issue, the firmware and controllers, not the > flash chips themselves. > > Jeff> I remain cautiously optimistic (though I've already bricked one > Jeff> drive). > > Details? During TRIM testing, we managed to get an OCZ Vertex to spew I/O errors, and then finally fail to be recognized by the BIOS upon reboot. Note that this isn't a normal workload (the drive is being pounded on) and TRIM is still very new. So, I am not saying that the drives are not reliable in general. I simply don't know. Cheers, Jeff

March 2, 2010

Re: [Wlug] Meeting topics

by John Stoffel

>>>>> "Jeff" == Jeff Moyer <jmoyer(a)redhat.com> writes: Jeff> Chuck Anderson <cra(a)WPI.EDU> writes: >> On Mon, Mar 01, 2010 at 04:46:44PM -0500, Jim Dibb wrote: >>> On Mon, Mar 1, 2010 at 4:21 PM, Chuck Anderson <cra(a)wpi.edu> wrote: >>>> I'm interested. I was considering using SSDs for my boot drive. Is >>>> RAID protection necessary/advisable using SSDs? >>> >>> SSDs have much higher reliability than HDD. >>> Mirroring is the only RAID level that does not impose a write penalty. >>> Mirroring SSDs is expensive >>> >>> Based on these 3 points, I'd say a backup is a much better solution than >>> RAID, unless you want to (R)AID-0 them into a stripe for even more insane >>> performance. >>> >>> But then again, you didn't state how important your uptime of your system >>> is. Sorry for leaving that bit out.... >> >> Uptime/availability isn't that important, but backups are since RAID >> *is* my backup (plus rsync to other RAIDed systems for the most >> important data). With traditional disk drives, it hasn't been too >> expensive to slap a 2nd system drive in for RAID 1/mirroring. SSDs >> are much more expensive, so if as you say they are more reliable, >> perhaps I can forgo the expense of doing RAID on them. Jeff> I've seen many claims that SSDs are more reliable than Jeff> traditional HDDs, but I haven't seen data to back up those Jeff> claims. Some of the arguements I've read on SSD reliability are quite interesting. If you do the math on how much data you need to write to destroy an SSD with proper (note!) wear levelling, it's really quite high. But I think the real key is to write data to them in blocks of 512 bytes as much as possible, and to not write smaller amounts if at all possible. Jeff> My main concern is that SSDs are still young, so they haven't Jeff> had much soak time. Whether or not the flash is reliable is Jeff> only a small part of the battle. Buggy firmware is a much more Jeff> likely problem at this stage in the game. No single point of Jeff> failure is a good design point when dealing with your data. ;-) I think this is a key issue, the firmware and controllers, not the flash chips themselves. Jeff> I remain cautiously optimistic (though I've already bricked one Jeff> drive). Details? John

March 2, 2010

Re: [Wlug] Meeting topics

by Jeff Moyer

Chuck Anderson <cra(a)WPI.EDU> writes: > On Mon, Mar 01, 2010 at 04:46:44PM -0500, Jim Dibb wrote: >> On Mon, Mar 1, 2010 at 4:21 PM, Chuck Anderson <cra(a)wpi.edu> wrote: >>> I'm interested. I was considering using SSDs for my boot drive. Is >>> RAID protection necessary/advisable using SSDs? >> >> SSDs have much higher reliability than HDD. >> Mirroring is the only RAID level that does not impose a write penalty. >> Mirroring SSDs is expensive >> >> Based on these 3 points, I'd say a backup is a much better solution than >> RAID, unless you want to (R)AID-0 them into a stripe for even more insane >> performance. >> >> But then again, you didn't state how important your uptime of your system >> is. Sorry for leaving that bit out.... > > Uptime/availability isn't that important, but backups are since RAID > *is* my backup (plus rsync to other RAIDed systems for the most > important data). With traditional disk drives, it hasn't been too > expensive to slap a 2nd system drive in for RAID 1/mirroring. SSDs > are much more expensive, so if as you say they are more reliable, > perhaps I can forgo the expense of doing RAID on them. I've seen many claims that SSDs are more reliable than traditional HDDs, but I haven't seen data to back up those claims. My main concern is that SSDs are still young, so they haven't had much soak time. Whether or not the flash is reliable is only a small part of the battle. Buggy firmware is a much more likely problem at this stage in the game. No single point of failure is a good design point when dealing with your data. ;-) I remain cautiously optimistic (though I've already bricked one drive). Cheers, Jeff

March 1, 2010

vmbuilder - Ubuntu Lucid?

by John Stoffel

Hi, Anyone else using vmbuilder with Ubuntu's 10.x Lucid Lynx (Alpha3, I know) and cursing the change in options that just happened recently? I've been playing around building VMs (based on Karmic) and tossing them away since it was so quick. Now that I've upgrade to a newer version of Lucid, it's all busted. Sigh... This would be a good talk too, about building and using VMs. Here's the script I whipped up to help automate things, which is now busted. Sigh... I realize I need to just look in the docs, but they a) suck and b) aren't easily found. Nor is any list of changes in the options listed nicely either. Grrr!!!! THanks, John #!/bin/sh # My own script for building and starting VMs more easily, based on # the instructions given at: # # http://www.howtoforge.com/virtualization-with-kvm-on-ubuntu-9.10 # # Version 0.1 - 12/29/2009 # if [ $# -ne 3 ]; then echo "Usage: $0 hostname ip username" echo " IP is last octet of the 192.168.1/24 subnet" echo exit fi euid=`id -u` if [ $euid -ne 0 ]; then echo "Error! Script must run with root privs. Try sudo $0 ...." echo exit fi hostname=$1 ip=$2 username=$3 vmbase="/var/vm" subnet="192.168.1" suite="karmic" bridge="br0" mem=256 mirror="http://192.168.1.3:9999/ubuntu" # LVM stuff volgroup="virtmachs" #--------------------------------------------------------------------- # Make the directory where we store the VM config #--------------------------------------------------------------------- if [ -d ${vmbase}/${hostname} ]; then /bin/rm -rf ${vmbase}/${hostname} fi mkdir -p ${vmbase}/${hostname} cd ${vmbase}/${hostname} #--------------------------------------------------------------------- # Copy over the template, but use the fixed one instead in vm0 #--------------------------------------------------------------------- mkdir -p ${vmbase}/${hostname}/mytemplates/libvirt cp ${vmbase}/vm0/mytemplates/libvirt/* ${vmbase}/${hostname}/mytemplates/libvirt/ #--------------------------------------------------------------------- # Setup partitions file #--------------------------------------------------------------------- cat >vmbuilder.partition <<EOF root 8000 swap 2000 /var 10000 EOF #--------------------------------------------------------------------- # Setup the boot.sh file #--------------------------------------------------------------------- cat >boot.sh <<EOF # This script will run the first time the virtual machine boots # It is run as root. # Expire the user account /usr/bin/passwd -e ${username} # Set the root password /usr/sbin/usermod -p '$1$MbDcN3NU$ndC6TvQ1.a1VG0jbeW3hD.' root # Install openssh-server apt-get update apt-get install -qqy --force-yes openssh-server nfs-client nfs-common mkdir /local /staging /home echo "jfsnew:/home /home nfs soft,sec=sys,vers=3,tcp,rsize=32768,wsize=32768,nolock 0 0" >> /etc/fstab echo "jfsnew:/local /local nfs soft,sec=sys,vers=3,tcp,rsize=32768,wsize=32768,nolock 0 0" >> /etc/fstab echo "jfsnew:/staging /staging nfs soft,sec=sys,vers=3,tcp,rsize=32768,wsize=32768,nolock 0 0" >> /etc/fstab mount /home mount /local mount /staging echo "America/New_York" > /etc/timezone cp /usr/share/zoneinfo/America/New_York /etc/localtime EOF #--------------------------------------------------------------------- # Create LVM volume for VM #--------------------------------------------------------------------- if [ ! -e /dev/${volgroup}/${hostname} ] then lvcreate -L20G -n ${hostname} ${volgroup} else echo echo "Skipping lvcreate, Logical Volume ${hostname} already exists." echo fi #--------------------------------------------------------------------- # Run vmbuilder #--------------------------------------------------------------------- echo vmbuilder kvm ubuntu --suite=${suite} --flavour=virtual --arch=amd64 --mirror=${mirror} -o --libvirt=qemu:///system --tmpfs=- --ip="${subnet}.${ip}" --part=vmbuilder.partition --templates=mytemplates --user=${username} --name=${username} --pass=${username} --addpkg=vim-nox --addpkg=unattended-upgrades --addpkg=acpid --raw=/dev/virtmachs/${hostname} --firstboot=boot.sh --mem=${mem} --hostname=${hostname} --bridge=${bridge} vmbuilder kvm ubuntu --suite=${suite} --flavour=virtual --arch=amd64 --mirror=${mirror} -o --libvirt=qemu:///system --tmpfs=- --ip="${subnet}.${ip}" --gateway="${subnet}.254" --part=vmbuilder.partition --templates=mytemplates --user=${username} --name=${username} --pass=${username} --addpkg=vim-nox --addpkg=unattended-upgrades --addpkg=acpid --raw=/dev/virtmachs/${hostname} --firstboot=boot.sh --mem=${mem} --hostname=${hostname} --bridge=${bridge} #--------------------------------------------------------------------- # Startup the VM #--------------------------------------------------------------------- if [ -e /etc/libvirt/qemu/${hostname}.xml ] then virsh define /etc/libvirt/qemu/${hostname}.xml virsh start ${hostname} else echo "Error! VM ${hostname} not created properly." echo exit fi

March 1, 2010

Re: [Wlug] Meeting topics

by Chuck Anderson

On Mon, Mar 01, 2010 at 04:46:44PM -0500, Jim Dibb wrote: > On Mon, Mar 1, 2010 at 4:21 PM, Chuck Anderson <cra(a)wpi.edu> wrote: >> I'm interested. I was considering using SSDs for my boot drive. Is >> RAID protection necessary/advisable using SSDs? > > SSDs have much higher reliability than HDD. > Mirroring is the only RAID level that does not impose a write penalty. > Mirroring SSDs is expensive > > Based on these 3 points, I'd say a backup is a much better solution than > RAID, unless you want to (R)AID-0 them into a stripe for even more insane > performance. > > But then again, you didn't state how important your uptime of your system > is. Sorry for leaving that bit out.... Uptime/availability isn't that important, but backups are since RAID *is* my backup (plus rsync to other RAIDed systems for the most important data). With traditional disk drives, it hasn't been too expensive to slap a 2nd system drive in for RAID 1/mirroring. SSDs are much more expensive, so if as you say they are more reliable, perhaps I can forgo the expense of doing RAID on them.

March 1, 2010

Re: [Wlug] Meeting topics

by Jeff Moyer

Chuck Anderson <cra(a)WPI.EDU> writes: > On Sun, Feb 28, 2010 at 06:26:31PM -0500, Jeff Moyer wrote: >> If anyone is interested in a talk on SSDs, chime in. I can present my >> experiences with these devices. I've evaluated the performance of >> several vendors' products, ranging from the poor to the exemplary. I >> can talk about the modifications made in the block layer to accommodate >> for these devices, and the current implementation of the block discard >> support (which maps to ATA TRIM, which is important for device lognevity >> and performance) and how that is wired up to file systems such as ext4. > > I'm interested. I was considering using SSDs for my boot drive. Is > RAID protection necessary/advisable using SSDs? The answer is, as always, it depends. I'll cover this in more depth when I give the talk, but any RAID algorithm that needs to initialize every block of the device is going to hurt performance on SSDs. There is some question about the durability/reliability of the drives, and some of those concerns are valid, others aren't. If you are diligent about backups, or if the disk just holds the OS, I wouldn't worry too much about RAID. HTH, Jeff

March 1, 2010