Tim> My first experiment was simply setting up a distributed volume
Tim> that needs to have each of the "bricks" as they refer to them as
Tim> being reliable members.
Nice.
Tim> However you can configure it to mirror and strip across the
Tim> "bricks" so they don't have to be. In fact you can setup
Tim> stripped mirrors, etc.
That would be safer for sure. I can just see a network problem taking
down a bunch of bricks and leading to a split-brain situation if
you're not careful.
Tim> I've now got a number of these storage nodes as I call them. It
Tim> generally consists of a 1U box, stuffed full of ram, connected to
Tim> a JBOD chassis full of disks. I then use ZFS to create storage
Tim> pools.
So I used to like ZFS and think it was "da bomb" but after using it
for several years, and esp after using the ZFS appliances from
Sun/Oracle, I'm not really enamored of the entire design any more.
I think they have some great ideas in terms of checksumming all the
files and metadata. But the layering (or lack of) disks/devices
inside zpools just drives me nuts. It's just really inflexible and
can get you in trouble if you're not careful.
Tim> The problem with this is that your scale is limited to the size
Tim> of a single "node" and while you can play games with autofs it's
Tim> not a cohesive filesystem.
Can you give some more details of your setup?
Tim> This solves that problem. My only complaint is that it's fuse based.
Yeah, not really high performance at the end of the day.
Tim> Lustre is a kernel loaded filesystem that'll do the same thing as
Tim> gluster. However, it doesn't have any of the redunancy stuff,
Tim> it simply assumes your underlying storage is reliable.
This is the key/kicker for all these systems.
I'm waiting for someone to come up with an opensource sharding system
where you use erasure codes for low-level block storage so that you
have alot of the RAID6 advantages, but even more reliability AND
speed. But it's also a hard hard problem space to get right.
Thanks for sharing!
John
