- WLUG - wlug.mailman3.com

Re: [Wlug] (no subject)
by John Stoffel 29 Mar '17

29 Mar '17

Tim> My first experiment was simply setting up a distributed volume Tim> that needs to have each of the "bricks" as they refer to them as Tim> being reliable members. Nice. Tim> However you can configure it to mirror and strip across the Tim> "bricks" so they don't have to be. In fact you can setup Tim> stripped mirrors, etc. That would be safer for sure. I can just see a network problem taking down a bunch of bricks and leading to a split-brain situation if you're not careful. Tim> I've now got a number of these storage nodes as I call them. It Tim> generally consists of a 1U box, stuffed full of ram, connected to Tim> a JBOD chassis full of disks. I then use ZFS to create storage Tim> pools. So I used to like ZFS and think it was "da bomb" but after using it for several years, and esp after using the ZFS appliances from Sun/Oracle, I'm not really enamored of the entire design any more. I think they have some great ideas in terms of checksumming all the files and metadata. But the layering (or lack of) disks/devices inside zpools just drives me nuts. It's just really inflexible and can get you in trouble if you're not careful. Tim> The problem with this is that your scale is limited to the size Tim> of a single "node" and while you can play games with autofs it's Tim> not a cohesive filesystem. Can you give some more details of your setup? Tim> This solves that problem. My only complaint is that it's fuse based. Yeah, not really high performance at the end of the day. Tim> Lustre is a kernel loaded filesystem that'll do the same thing as Tim> gluster. However, it doesn't have any of the redunancy stuff, Tim> it simply assumes your underlying storage is reliable. This is the key/kicker for all these systems. I'm waiting for someone to come up with an opensource sharding system where you use erasure codes for low-level block storage so that you have alot of the RAID6 advantages, but even more reliability AND speed. But it's also a hard hard problem space to get right. Thanks for sharing! John

1 0

Re: [Wlug] (no subject)
by Tim Keller 28 Mar '17

28 Mar '17

My first experiment was simply setting up a distributed volume that needs to have each of the "bricks" as they refer to them as being reliable members. However you can configure it to mirror and strip across the "bricks" so they don't have to be. In fact you can setup stripped mirrors, etc. I've now got a number of these storage nodes as I call them. It generally consists of a 1U box, stuffed full of ram, connected to a JBOD chassis full of disks. I then use ZFS to create storage pools. The problem with this is that your scale is limited to the size of a single "node" and while you can play games with autofs it's not a cohesive filesystem. This solves that problem. My only complaint is that it's fuse based. Lustre is a kernel loaded filesystem that'll do the same thing as gluster. However, it doesn't have any of the redunancy stuff, it simply assumes your underlying storage is reliable. Tim. On Tue, Mar 28, 2017 at 11:08 AM, John Stoffel <john(a)stoffel.org> wrote: > > Tim> The subject for that was supposed to be "Fun with glusterfs" > Tim> On Tue, Mar 28, 2017 at 10:51 AM, Tim Keller <turbofx(a)gmail.com> > wrote: > > Tim> I've been messing around with "glusterfs" and it's pretty cool. > > Tim> I build 3 machine running RHEL7.2 each with an extra 250GB disk. > > Tim> I then setup a filesystem that stripes across all three machines. > > Tim> So it looks like a single 750GB filesystem. Over gigE I'm seeing > okay > Tim> performance ~50MB/s. > > Tim> Cool stuff. Next on my list of filesystems to play with it > Lustre. > > Sounds neat. Does glusterfs support RAID5/6 or does it depend on the > underlying nodes to be reliable? > > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org > http://mail.wlug.org/mailman/listinfo/wlug > -- I am leery of the allegiances of any politician who refers to their constituents as "consumers".

1 0

Re: [Wlug] (no subject)
by John Stoffel 28 Mar '17

28 Mar '17

Tim> The subject for that was supposed to be "Fun with glusterfs" Tim> On Tue, Mar 28, 2017 at 10:51 AM, Tim Keller <turbofx(a)gmail.com> wrote: Tim> I've been messing around with "glusterfs" and it's pretty cool. Tim> I build 3 machine running RHEL7.2 each with an extra 250GB disk. Tim> I then setup a filesystem that stripes across all three machines. Tim> So it looks like a single 750GB filesystem. Over gigE I'm seeing okay Tim> performance ~50MB/s. Tim> Cool stuff. Next on my list of filesystems to play with it Lustre. Sounds neat. Does glusterfs support RAID5/6 or does it depend on the underlying nodes to be reliable?

1 0

(no subject)
by Tim Keller 28 Mar '17

28 Mar '17

I've been messing around with "glusterfs" and it's pretty cool. I build 3 machine running RHEL7.2 each with an extra 250GB disk. I then setup a filesystem that stripes across all three machines. So it looks like a single 750GB filesystem. Over gigE I'm seeing okay performance ~50MB/s. Cool stuff. Next on my list of filesystems to play with it Lustre. Tim. -- I am leery of the allegiances of any politician who refers to their constituents as "consumers".

1 1

Re: [Wlug] awk help
by Mike Peckar 15 Mar '17

15 Mar '17

Nathan’s solution worked, thanks to all for the replies. I do feel better this was not a solution that could fit into a tweet. Theo’s is elegant, but there are spaces in the data, and John’s would have work, but I can’t use perl in the environment I’m working in. Mike awk -F, '{linecount[$3]++;tmp=dup[$3];if(length(tmp)==0){dup[$3]=$0}else{dup[$3]=dup[$3]"\n"$0}} END {for (count in linecount){if(linecount[count]>1){print dup[count]}}}' Hopefully this is not a homework problem. On Mon, Mar 13, 2017 at 7:20 PM, Mike Peckar <fog(a)fognet.com> wrote: Hey now, folks, This seemed like it should be simple, but I’m at wits end. I simply want to find duplicates in the third column of a csv file, and output the duplicate line _and_ the original line that matched it. There’s a million examples out there that will output just the duplicate but not both. In the data below, I’m looking for lines that match in the 3rd column… Normal,Server,xldspntc02,,10.33.52.185, Normal,Server,xldspntc02,,10.33.52.186, Normal,Server,xldspntc04,,10.33.52.187, Normal,Server,xldspntcs01,10.33.16.198, Normal,Server,xldspntcs01,,10.33.16.199, Normal,Server,xldsps01,10.33.16.162, Normal,Server,xldsps02,10.33.16.163, My desired output would be: Normal,Server,xldspntc02,,10.33.52.185, Normal,Server,xldspntc02,,10.33.52.186, Normal,Server,xldspntcs01,10.33.16.198, Normal,Server,xldspntcs01,,10.33.16.199, $ awk -F, 'dup[$3]++' file.csv I played around with the prev variable, but could not pumb it out fully, e.g { print prev } Mike _______________________________________________ Wlug mailing list Wlug(a)mail.wlug.org http://mail.wlug.org/mailman/listinfo/wlug -- Nathan Panike

1 0

Re: [Wlug] awk help
by John Stoffel 15 Mar '17

15 Mar '17

>>>>> "Theo" == Theo Van Dinter <felicity(a)kluge.net> writes: Theo> Assuming there's no spaces in the csv, you could do: Theo> awk -F, '{print $0,$3}' | sort -k2 | uniq -D -f 1 | awk '{print $1}' Nice use of 'uniq -f 1' to re-read the file and do the de-duping. Theo> $ cat 1 Theo> Normal,Server,xldspntcs01,10.33.16.198, Theo> Normal,Server,xldspntc02,,10.33.52.185, Theo> Normal,Server,xldsps01,10.33.16.162, Theo> Normal,Server,xldspntc04,,10.33.52.187, Theo> Normal,Server,xldspntcs01,,10.33.16.199, Theo> Normal,Server,xldsps02,10.33.16.163, Theo> Normal,Server,xldspntc02,,10.33.52.186, Theo> $ cat 1 | awk -F, '{print $0,$3}' | sort -k2 | uniq -D -f 1 | awk '{print $1}' Theo> Normal,Server,xldspntc02,,10.33.52.185, Theo> Normal,Server,xldspntc02,,10.33.52.186, Theo> Normal,Server,xldspntcs01,,10.33.16.199, Theo> Normal,Server,xldspntcs01,10.33.16.198, Theo> On Mon, Mar 13, 2017 at 7:20 PM, Mike Peckar <fog(a)fognet.com> wrote: Theo> Hey now, folks, Theo> Theo> This seemed like it should be simple, but I’m at wits end. I simply Theo> want to find duplicates in the third column of a csv file, and output Theo> the duplicate line _and_ the original line that matched it. There’s a Theo> million examples out there that will output just the duplicate but not Theo> both. Theo> Theo> In the data below, I’m looking for lines that match in the 3^rd column… Theo> Theo> Normal,Server,xldspntc02,,10.33.52.185, Theo> Normal,Server,xldspntc02,,10.33.52.186, Theo> Normal,Server,xldspntc04,,10.33.52.187, Theo> Normal,Server,xldspntcs01,10.33.16.198, Theo> Normal,Server,xldspntcs01,,10.33.16.199, Theo> Normal,Server,xldsps01,10.33.16.162, Theo> Normal,Server,xldsps02,10.33.16.163, Theo> Theo> My desired output would be: Theo> Theo> Normal,Server,xldspntc02,,10.33.52.185, Theo> Normal,Server,xldspntc02,,10.33.52.186, Theo> Normal,Server,xldspntcs01,10.33.16.198, Theo> Normal,Server,xldspntcs01,,10.33.16.199, Theo> Theo> $ awk -F, 'dup[$3]++' file.csv Theo> Theo> I played around with the prev variable, but could not pumb it out Theo> fully, e.g { print prev } Theo> Theo> Mike Theo> _______________________________________________ Theo> Wlug mailing list Theo> Wlug(a)mail.wlug.org Theo> http://mail.wlug.org/mailman/listinfo/wlug Theo> _______________________________________________ Theo> Wlug mailing list Theo> Wlug(a)mail.wlug.org Theo> http://mail.wlug.org/mailman/listinfo/wlug

1 0

Re: [Wlug] awk help
by Theo Van Dinter 15 Mar '17

15 Mar '17

Assuming there's no spaces in the csv, you could do: awk -F, '{print $0,$3}' | sort -k2 | uniq -D -f 1 | awk '{print $1}' eg: $ cat 1 Normal,Server,xldspntcs01,10.33.16.198, Normal,Server,xldspntc02,,10.33.52.185, Normal,Server,xldsps01,10.33.16.162, Normal,Server,xldspntc04,,10.33.52.187, Normal,Server,xldspntcs01,,10.33.16.199, Normal,Server,xldsps02,10.33.16.163, Normal,Server,xldspntc02,,10.33.52.186, $ cat 1 | awk -F, '{print $0,$3}' | sort -k2 | uniq -D -f 1 | awk '{print $1}' Normal,Server,xldspntc02,,10.33.52.185, Normal,Server,xldspntc02,,10.33.52.186, Normal,Server,xldspntcs01,,10.33.16.199, Normal,Server,xldspntcs01,10.33.16.198, On Mon, Mar 13, 2017 at 7:20 PM, Mike Peckar <fog(a)fognet.com> wrote: > Hey now, folks, > > > > This seemed like it should be simple, but I’m at wits end. I simply want > to find duplicates in the third column of a csv file, and output the > duplicate line _*and*_ the original line that matched it. There’s a > million examples out there that will output just the duplicate but not both. > > > > In the data below, I’m looking for lines that match in the 3rd column… > > > > Normal,Server,xldspntc02,,10.33.52.185, > > Normal,Server,xldspntc02,,10.33.52.186, > > Normal,Server,xldspntc04,,10.33.52.187, > > Normal,Server,xldspntcs01,10.33.16.198, > > Normal,Server,xldspntcs01,,10.33.16.199, > > Normal,Server,xldsps01,10.33.16.162, > > Normal,Server,xldsps02,10.33.16.163, > > > > My desired output would be: > > > > Normal,Server,xldspntc02,,10.33.52.185, > > Normal,Server,xldspntc02,,10.33.52.186, > > Normal,Server,xldspntcs01,10.33.16.198, > > Normal,Server,xldspntcs01,,10.33.16.199, > > > > $ awk -F, 'dup[$3]++' file.csv > > > > I played around with the prev variable, but could not pumb it out fully, > e.g { print prev } > > > > Mike > > > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org > http://mail.wlug.org/mailman/listinfo/wlug > >

1 0

Re: [Wlug] awk help
by John Stoffel 14 Mar '17

14 Mar '17

>>>>> "Mike" == Mike Peckar <fog(a)fognet.com> writes: Mike> This seemed like it should be simple, but I’m at wits end. I Mike> simply want to find duplicates in the third column of a csv Mike> file, and output the duplicate line _and_ the original line Mike> that matched it. There’s a million examples out there that Mike> will output just the duplicate but not both. Mike> In the data below, I’m looking for lines that match in the 3^rd column… The sorting part is easy... sort -k 3 -t "," <file> Now to find the duplicates... I'd probably jump to a perl script: perl -e '@a=(<>);foreach @a {@t=split(",",$_) {push @{$t{$t[2]}},$_; } foreach sort (keys %t) { if ($#{$t{$_}} > 0) { print @{$t{$_}}, "\n";}}' Should also do the right thing. First it splits into keys, stuffs the line into an assoc array of arrays. Then it sorts the assoc array and prints out those with more than one entry in it. Admittedly done off the top of my head, without any actual testing. :-) Mike> Normal,Server,xldspntc02,,10.33.52.185, Mike> Normal,Server,xldspntc02,,10.33.52.186, Mike> Normal,Server,xldspntc04,,10.33.52.187, Mike> Normal,Server,xldspntcs01,10.33.16.198, Mike> Normal,Server,xldspntcs01,,10.33.16.199, Mike> Normal,Server,xldsps01,10.33.16.162, Mike> Normal,Server,xldsps02,10.33.16.163, Mike> My desired output would be: Mike> Normal,Server,xldspntc02,,10.33.52.185, Mike> Normal,Server,xldspntc02,,10.33.52.186, Mike> Normal,Server,xldspntcs01,10.33.16.198, Mike> Normal,Server,xldspntcs01,,10.33.16.199, Mike> $ awk -F, 'dup[$3]++' file.csv Mike> I played around with the prev variable, but could not pumb it out fully, Mike> e.g { print prev } Mike> Mike Mike> _______________________________________________ Mike> Wlug mailing list Mike> Wlug(a)mail.wlug.org Mike> http://mail.wlug.org/mailman/listinfo/wlug

1 0

Re: [Wlug] awk help
by Nathan W. Panike 14 Mar '17

14 Mar '17

This seems to work: awk -F, '{linecount[$3]++;tmp=dup[$3];if(length(tmp)==0){dup[$3]=$0}else{dup[$3]=dup[$3]"\n"$0}} END {for (count in linecount){if(linecount[count]>1){print dup[count]}}}' Hopefully this is not a homework problem. On Mon, Mar 13, 2017 at 7:20 PM, Mike Peckar <fog(a)fognet.com> wrote: > Hey now, folks, > > > > This seemed like it should be simple, but I’m at wits end. I simply want > to find duplicates in the third column of a csv file, and output the > duplicate line _*and*_ the original line that matched it. There’s a > million examples out there that will output just the duplicate but not both. > > > > In the data below, I’m looking for lines that match in the 3rd column… > > > > Normal,Server,xldspntc02,,10.33.52.185, > > Normal,Server,xldspntc02,,10.33.52.186, > > Normal,Server,xldspntc04,,10.33.52.187, > > Normal,Server,xldspntcs01,10.33.16.198, > > Normal,Server,xldspntcs01,,10.33.16.199, > > Normal,Server,xldsps01,10.33.16.162, > > Normal,Server,xldsps02,10.33.16.163, > > > > My desired output would be: > > > > Normal,Server,xldspntc02,,10.33.52.185, > > Normal,Server,xldspntc02,,10.33.52.186, > > Normal,Server,xldspntcs01,10.33.16.198, > > Normal,Server,xldspntcs01,,10.33.16.199, > > > > $ awk -F, 'dup[$3]++' file.csv > > > > I played around with the prev variable, but could not pumb it out fully, > e.g { print prev } > > > > Mike > > > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org > http://mail.wlug.org/mailman/listinfo/wlug > > -- Nathan Panike

1 0

awk help
by Mike Peckar 13 Mar '17

13 Mar '17

Hey now, folks, This seemed like it should be simple, but I’m at wits end. I simply want to find duplicates in the third column of a csv file, and output the duplicate line _and_ the original line that matched it. There’s a million examples out there that will output just the duplicate but not both. In the data below, I’m looking for lines that match in the 3rd column… Normal,Server,xldspntc02,,10.33.52.185, Normal,Server,xldspntc02,,10.33.52.186, Normal,Server,xldspntc04,,10.33.52.187, Normal,Server,xldspntcs01,10.33.16.198, Normal,Server,xldspntcs01,,10.33.16.199, Normal,Server,xldsps01,10.33.16.162, Normal,Server,xldsps02,10.33.16.163, My desired output would be: Normal,Server,xldspntc02,,10.33.52.185, Normal,Server,xldspntc02,,10.33.52.186, Normal,Server,xldspntcs01,10.33.16.198, Normal,Server,xldspntcs01,,10.33.16.199, $ awk -F, 'dup[$3]++' file.csv I played around with the prev variable, but could not pumb it out fully, e.g { print prev } Mike

1 0