- WLUG - wlug.mailman3.com

Topics for next meeting (Tim Keller)
by joel d April 26, 2017

April 26, 2017

All three topics are interesting. Plex holds more interest for me currently as I plan to soon build a new home NAS for professional, personal and entertainment data. Would want to include Plex in those plans. RetroPie is interesting too. Built one with my 10 year old son. He learned a lot. Dad can still win at Atari Combat. I could bring it and talk about it, but would likely be a relatively short discussion. BitCoin for me has less interest. Technically it's harder to mine for BitCoin prime numbers every month. As a financial currency it is not very strong, not quite taking the world by storm. I like it's government independence, but prepper friends prefer precious metals. Joel ------------------- Date: Tue, 25 Apr 2017 14:32:50 -0400 From: Tim Keller <turbofx(a)gmail.com> Subject: [Wlug] Topics for next meeting To: Worcester Linux Users Group <wlug(a)mail.wlug.org> Before the next meeting suddenly appears and we're scrambling for a topic, is there something that people are burning to know about? We talked about doing a talk about plex, I'm not sure how interested people are. Another topic that came up was a talk about setting up a RetroPi system. Another topic that came up later was BitCoin. Personally I've always found bitcoin an enigma. If anybody in the group understands it I'd love to learn about it and possibly what tools there are on linux for mining and storing bitcoins. Thanks, Tim. -- I am leery of the allegiances of any politician who refers to their constituents as "consumers".

1 0

Topics for next meeting
by Tim Keller April 25, 2017

April 25, 2017

Before the next meeting suddenly appears and we're scrambling for a topic, is there something that people are burning to know about? We talked about doing a talk about plex, I'm not sure how interested people are. Another topic that came up was a talk about setting up a RetroPi system. Another topic that came up later was BitCoin. Personally I've always found bitcoin an enigma. If anybody in the group understands it I'd love to learn about it and possibly what tools there are on linux for mining and storing bitcoins. Thanks, Tim. -- I am leery of the allegiances of any politician who refers to their constituents as "consumers".

1 0

Humble Book Bundle: Python presented by No Starch Press (pay what you want and help charity)
by Eric Martin April 16, 2017

April 16, 2017

I figured some of you might find some use for this. It has some great books in it, including black hat python and teaching your kids to program. $15 for a bunch of DRM free books is hard to beat. https://www.humblebundle.com/books/python-book-bundle -- Eric

1 0

Reminder! WLUG April Meeting tonight! Topic: GlusterFS
by Tim Keller April 13, 2017

April 13, 2017

Hey Gang, Don't forget we've got a meeting tonight! Time: 7pm Location: WPI Student Center Rom 331 (mid Century Room) Speaker: Tim Keller Topic: GlusterFS GlusterFS is a distributed clustered file system that lets you take a whole bunch of servers and combine their storage into a single large file system. Snacks and Refreshments will be provided. After the meeting we'll head down to Tech for some pizza! Thanks, Tim. -- I am leery of the allegiances of any politician who refers to their constituents as "consumers".

1 0

Reminder! Meeting tomorrow! Apr. 13th, 2017! Topic: glusterfs
by Tim Keller April 12, 2017

April 12, 2017

Hey Gang, Tomorrow we've got a meeting! Time: 7pm Location: WPI Student Center Room 331 Topic: glusterfs What is glusterfs? It's a distributed clustered file system. I recently did a talk about GFS2 and while interesting, it requires centralized storage, a knowledge about stuff like iscsi, etc. It's a beast to setup and isn't that flexible. glusterfs is a different animal. It's a flexible overlay filesystem that let's you take a bunch of basic servers and cluster them into a big homogenous filesystem. If you have four systems, each with 4TB worth of disk, you can make yourself a nice 16TB filesystem, or you can setup an 8TB filesystem with mirroring that spans all the machines or a couple of filesystems that do both depending on your needs. What's great about glusterfs is that it sits atop an existing filesystems on the machines and uses TCP to communicate with both fs clients and the node members. It also has a built in NFS server that allows it to be used by other non linux machines. As usual there will be snacks and refreshments and afterwards we'll head to tech for pizza. Thanks, Tim. -- I am leery of the allegiances of any politician who refers to their constituents as "consumers".

1 0

Reminder! Meeting April 13th @ 7PM
by Tim Keller April 9, 2017

April 9, 2017

Hey Gang, This is reminder that we've got a meeting this week! Location: Campus Center Room 331 Time: 7PM Topic: To be decided! Let me know if there's something that you have a burning urge to talk about! Thanks, Tim. Per our last meeting, here were some of the topics... Media server bake off Kodi (Xbmc) PLEX media center /server Open media vault (runs PLEX) Open ELEC (runs Kodi) Linux Arcade machines Research Distros Home Backup Server (zfs for linux) Bake off of backup solutions Sysadmin automation tools Salt / Puppet / Chef / Ansible Nicholas Howgraham-Grave Talk about finding flaws in RSA SHA1 collison stuff Git development Methods Virtualization on linux KVM / Vmware / OVB / Xen / Qubes Privacy on Linux: What to do when you're convinced the goverment is watching. Encrypted FS 2FA Isolated applications What distro is snowden running? 3d Printing and linux Tools for linux -- I am leery of the allegiances of any politician who refers to their constituents as "consumers".

1 0

Re: [Wlug] video from Identity Management, FreeIPA, and Kerberos talk
by Chuck Anderson April 5, 2017

April 5, 2017

Someone tried to download a "high" quality file. Sorry there is nothing but this "low" quality one available right now. I might be able to make a higher quality one, but it takes like 4-8 hours for each try, and I figured the video quality isn't that important--just use the PDF slides with this 278MB video file. The audio should be good quality. On Wed, Apr 05, 2017 at 04:38:24PM -0400, Tim Keller wrote: > Nice!!! > > On Wed, Apr 5, 2017 at 4:07 PM, Chuck Anderson <cra(a)wpi.edu> wrote: > > > It only took me a year and a half, but I finally was able to get the > > 27gig AVCHD files from the video camera converted to a usable video > > file format. Please find the video (and slides) here: > > > > http://www.wlug.org/ > > > > December 10, 2015 - Dmitri Pal, Red Hat, Inc. - Identity Management, > > FreeIPA, and Kerberos (slides) (video) > > > > Direct links: > > > > http://www.wlug.org/files/Identity%20Management%20for% > > 20Modern%20Infrastructure%20(WPI)%20low.mp4 > > > > http://www.wlug.org/files/Identity%20Management%20for% > > 20Modern%20Infrastructure%20(WPI).pdf

1 0

Re: [Wlug] video from Identity Management, FreeIPA, and Kerberos talk
by Tim Keller April 5, 2017

April 5, 2017

Nice!!! On Wed, Apr 5, 2017 at 4:07 PM, Chuck Anderson <cra(a)wpi.edu> wrote: > It only took me a year and a half, but I finally was able to get the > 27gig AVCHD files from the video camera converted to a usable video > file format. Please find the video (and slides) here: > > http://www.wlug.org/ > > December 10, 2015 - Dmitri Pal, Red Hat, Inc. - Identity Management, > FreeIPA, and Kerberos (slides) (video) > > Direct links: > > http://www.wlug.org/files/Identity%20Management%20for% > 20Modern%20Infrastructure%20(WPI)%20low.mp4 > > http://www.wlug.org/files/Identity%20Management%20for% > 20Modern%20Infrastructure%20(WPI).pdf > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org > http://mail.wlug.org/mailman/listinfo/wlug > -- I am leery of the allegiances of any politician who refers to their constituents as "consumers".

1 0

video from Identity Management, FreeIPA, and Kerberos talk
by Chuck Anderson April 5, 2017

April 5, 2017

It only took me a year and a half, but I finally was able to get the 27gig AVCHD files from the video camera converted to a usable video file format. Please find the video (and slides) here: http://www.wlug.org/ December 10, 2015 - Dmitri Pal, Red Hat, Inc. - Identity Management, FreeIPA, and Kerberos (slides) (video) Direct links: http://www.wlug.org/files/Identity%20Management%20for%20Modern%20Infrastruc… http://www.wlug.org/files/Identity%20Management%20for%20Modern%20Infrastruc…

1 0

Ideas for next meeting (April 13th)
by Tim Keller April 4, 2017

April 4, 2017

Hey Gang, A couple of ideas have been bandied about for the next meeting. 1. A round table discussion regarding internet security. 2. glusterfs. This is a distributed clustering filesystem that lets you tie a bunch of systems into a single cohesive filesystem. 3. ? If people have ideas, I'd love to hear them. I made a list at the last meeting but I don't have it on me... Thanks, Tim. -- I am leery of the allegiances of any politician who refers to their constituents as "consumers".

1 0

Re: [Wlug] (no subject)
by Tim Keller March 29, 2017

March 29, 2017

Lustre solves the problem by getting out of the way. The metadata server simply tells the client on which of the nodes the pieces of the file resides. Reading the documentation, when it comes to Lustre, split brain can't happen because the moment any of the meta controllers go offline, the whole thing stops. Definitely with ZFS if you don't start with a plan you're going to have a bad day. The current design for a node is a 1U HP DL360G7 server with ~144 or 288GB of ram. Internal storage is handled by the smart array controller. The next one I build I think I'm going to buy a 9200-8i and route the cables so that all the internal and external storage is JBOD. My current HBA card dejour for external storage is the LSI 9200-8e. As for external JBOD enclosures, there are LOTS of choices. Generally I've stuck with the promise J610s. It's essentially the "expansion" cabinet for a "smart" promise array. It's beauty is in it's abject stupidity. As for drives, currently all my stuff is running 4TB Seagate SAS drives. For "mainline" nodes, I configure them as raid10 pools. This gets me ~29TB of storage per pool For "backup" nodes, I configure them as raidz2 pools. All my backup nodes are 32 disk boxes so the pools are 116TB. Obviously this design is a balance between cost and performance. If I need more slots I'll use a 380 instead of 360 and I'll stick in an Intel X540-T2 10GigE card. It's fun to imagine if you scaled up the disks and the interconnects what a filesystem would look like. Tim. On Tue, Mar 28, 2017 at 8:45 PM, John Stoffel <john(a)stoffel.org> wrote: > > Tim> My first experiment was simply setting up a distributed volume > Tim> that needs to have each of the "bricks" as they refer to them as > Tim> being reliable members. > > Nice. > > Tim> However you can configure it to mirror and strip across the > Tim> "bricks" so they don't have to be. In fact you can setup > Tim> stripped mirrors, etc. > > That would be safer for sure. I can just see a network problem taking > down a bunch of bricks and leading to a split-brain situation if > you're not careful. > > Tim> I've now got a number of these storage nodes as I call them. It > Tim> generally consists of a 1U box, stuffed full of ram, connected to > Tim> a JBOD chassis full of disks. I then use ZFS to create storage > Tim> pools. > > So I used to like ZFS and think it was "da bomb" but after using it > for several years, and esp after using the ZFS appliances from > Sun/Oracle, I'm not really enamored of the entire design any more. > > I think they have some great ideas in terms of checksumming all the > files and metadata. But the layering (or lack of) disks/devices > inside zpools just drives me nuts. It's just really inflexible and > can get you in trouble if you're not careful. > > Tim> The problem with this is that your scale is limited to the size > Tim> of a single "node" and while you can play games with autofs it's > Tim> not a cohesive filesystem. > > Can you give some more details of your setup? > > Tim> This solves that problem. My only complaint is that it's fuse based. > > Yeah, not really high performance at the end of the day. > > Tim> Lustre is a kernel loaded filesystem that'll do the same thing as > Tim> gluster. However, it doesn't have any of the redunancy stuff, > Tim> it simply assumes your underlying storage is reliable. > > This is the key/kicker for all these systems. > > I'm waiting for someone to come up with an opensource sharding system > where you use erasure codes for low-level block storage so that you > have alot of the RAID6 advantages, but even more reliability AND > speed. But it's also a hard hard problem space to get right. > > Thanks for sharing! > John > > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org > http://mail.wlug.org/mailman/listinfo/wlug > -- I am leery of the allegiances of any politician who refers to their constituents as "consumers".

1 0

Re: [Wlug] (no subject)
by John Stoffel March 29, 2017

March 29, 2017

Tim> My first experiment was simply setting up a distributed volume Tim> that needs to have each of the "bricks" as they refer to them as Tim> being reliable members. Nice. Tim> However you can configure it to mirror and strip across the Tim> "bricks" so they don't have to be. In fact you can setup Tim> stripped mirrors, etc. That would be safer for sure. I can just see a network problem taking down a bunch of bricks and leading to a split-brain situation if you're not careful. Tim> I've now got a number of these storage nodes as I call them. It Tim> generally consists of a 1U box, stuffed full of ram, connected to Tim> a JBOD chassis full of disks. I then use ZFS to create storage Tim> pools. So I used to like ZFS and think it was "da bomb" but after using it for several years, and esp after using the ZFS appliances from Sun/Oracle, I'm not really enamored of the entire design any more. I think they have some great ideas in terms of checksumming all the files and metadata. But the layering (or lack of) disks/devices inside zpools just drives me nuts. It's just really inflexible and can get you in trouble if you're not careful. Tim> The problem with this is that your scale is limited to the size Tim> of a single "node" and while you can play games with autofs it's Tim> not a cohesive filesystem. Can you give some more details of your setup? Tim> This solves that problem. My only complaint is that it's fuse based. Yeah, not really high performance at the end of the day. Tim> Lustre is a kernel loaded filesystem that'll do the same thing as Tim> gluster. However, it doesn't have any of the redunancy stuff, Tim> it simply assumes your underlying storage is reliable. This is the key/kicker for all these systems. I'm waiting for someone to come up with an opensource sharding system where you use erasure codes for low-level block storage so that you have alot of the RAID6 advantages, but even more reliability AND speed. But it's also a hard hard problem space to get right. Thanks for sharing! John

1 0

Re: [Wlug] (no subject)
by Tim Keller March 28, 2017

March 28, 2017

My first experiment was simply setting up a distributed volume that needs to have each of the "bricks" as they refer to them as being reliable members. However you can configure it to mirror and strip across the "bricks" so they don't have to be. In fact you can setup stripped mirrors, etc. I've now got a number of these storage nodes as I call them. It generally consists of a 1U box, stuffed full of ram, connected to a JBOD chassis full of disks. I then use ZFS to create storage pools. The problem with this is that your scale is limited to the size of a single "node" and while you can play games with autofs it's not a cohesive filesystem. This solves that problem. My only complaint is that it's fuse based. Lustre is a kernel loaded filesystem that'll do the same thing as gluster. However, it doesn't have any of the redunancy stuff, it simply assumes your underlying storage is reliable. Tim. On Tue, Mar 28, 2017 at 11:08 AM, John Stoffel <john(a)stoffel.org> wrote: > > Tim> The subject for that was supposed to be "Fun with glusterfs" > Tim> On Tue, Mar 28, 2017 at 10:51 AM, Tim Keller <turbofx(a)gmail.com> > wrote: > > Tim> I've been messing around with "glusterfs" and it's pretty cool. > > Tim> I build 3 machine running RHEL7.2 each with an extra 250GB disk. > > Tim> I then setup a filesystem that stripes across all three machines. > > Tim> So it looks like a single 750GB filesystem. Over gigE I'm seeing > okay > Tim> performance ~50MB/s. > > Tim> Cool stuff. Next on my list of filesystems to play with it > Lustre. > > Sounds neat. Does glusterfs support RAID5/6 or does it depend on the > underlying nodes to be reliable? > > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org > http://mail.wlug.org/mailman/listinfo/wlug > -- I am leery of the allegiances of any politician who refers to their constituents as "consumers".

1 0

Re: [Wlug] (no subject)
by John Stoffel March 28, 2017

March 28, 2017

Tim> The subject for that was supposed to be "Fun with glusterfs" Tim> On Tue, Mar 28, 2017 at 10:51 AM, Tim Keller <turbofx(a)gmail.com> wrote: Tim> I've been messing around with "glusterfs" and it's pretty cool. Tim> I build 3 machine running RHEL7.2 each with an extra 250GB disk. Tim> I then setup a filesystem that stripes across all three machines. Tim> So it looks like a single 750GB filesystem. Over gigE I'm seeing okay Tim> performance ~50MB/s. Tim> Cool stuff. Next on my list of filesystems to play with it Lustre. Sounds neat. Does glusterfs support RAID5/6 or does it depend on the underlying nodes to be reliable?

1 0

(no subject)
by Tim Keller March 28, 2017

March 28, 2017

I've been messing around with "glusterfs" and it's pretty cool. I build 3 machine running RHEL7.2 each with an extra 250GB disk. I then setup a filesystem that stripes across all three machines. So it looks like a single 750GB filesystem. Over gigE I'm seeing okay performance ~50MB/s. Cool stuff. Next on my list of filesystems to play with it Lustre. Tim. -- I am leery of the allegiances of any politician who refers to their constituents as "consumers".

1 1

Re: [Wlug] awk help
by Mike Peckar March 15, 2017

March 15, 2017

Nathan’s solution worked, thanks to all for the replies. I do feel better this was not a solution that could fit into a tweet. Theo’s is elegant, but there are spaces in the data, and John’s would have work, but I can’t use perl in the environment I’m working in. Mike awk -F, '{linecount[$3]++;tmp=dup[$3];if(length(tmp)==0){dup[$3]=$0}else{dup[$3]=dup[$3]"\n"$0}} END {for (count in linecount){if(linecount[count]>1){print dup[count]}}}' Hopefully this is not a homework problem. On Mon, Mar 13, 2017 at 7:20 PM, Mike Peckar <fog(a)fognet.com> wrote: Hey now, folks, This seemed like it should be simple, but I’m at wits end. I simply want to find duplicates in the third column of a csv file, and output the duplicate line _and_ the original line that matched it. There’s a million examples out there that will output just the duplicate but not both. In the data below, I’m looking for lines that match in the 3rd column… Normal,Server,xldspntc02,,10.33.52.185, Normal,Server,xldspntc02,,10.33.52.186, Normal,Server,xldspntc04,,10.33.52.187, Normal,Server,xldspntcs01,10.33.16.198, Normal,Server,xldspntcs01,,10.33.16.199, Normal,Server,xldsps01,10.33.16.162, Normal,Server,xldsps02,10.33.16.163, My desired output would be: Normal,Server,xldspntc02,,10.33.52.185, Normal,Server,xldspntc02,,10.33.52.186, Normal,Server,xldspntcs01,10.33.16.198, Normal,Server,xldspntcs01,,10.33.16.199, $ awk -F, 'dup[$3]++' file.csv I played around with the prev variable, but could not pumb it out fully, e.g { print prev } Mike _______________________________________________ Wlug mailing list Wlug(a)mail.wlug.org http://mail.wlug.org/mailman/listinfo/wlug -- Nathan Panike

1 0

Re: [Wlug] awk help
by John Stoffel March 15, 2017

March 15, 2017

>>>>> "Theo" == Theo Van Dinter <felicity(a)kluge.net> writes: Theo> Assuming there's no spaces in the csv, you could do: Theo> awk -F, '{print $0,$3}' | sort -k2 | uniq -D -f 1 | awk '{print $1}' Nice use of 'uniq -f 1' to re-read the file and do the de-duping. Theo> $ cat 1 Theo> Normal,Server,xldspntcs01,10.33.16.198, Theo> Normal,Server,xldspntc02,,10.33.52.185, Theo> Normal,Server,xldsps01,10.33.16.162, Theo> Normal,Server,xldspntc04,,10.33.52.187, Theo> Normal,Server,xldspntcs01,,10.33.16.199, Theo> Normal,Server,xldsps02,10.33.16.163, Theo> Normal,Server,xldspntc02,,10.33.52.186, Theo> $ cat 1 | awk -F, '{print $0,$3}' | sort -k2 | uniq -D -f 1 | awk '{print $1}' Theo> Normal,Server,xldspntc02,,10.33.52.185, Theo> Normal,Server,xldspntc02,,10.33.52.186, Theo> Normal,Server,xldspntcs01,,10.33.16.199, Theo> Normal,Server,xldspntcs01,10.33.16.198, Theo> On Mon, Mar 13, 2017 at 7:20 PM, Mike Peckar <fog(a)fognet.com> wrote: Theo> Hey now, folks, Theo> Theo> This seemed like it should be simple, but I’m at wits end. I simply Theo> want to find duplicates in the third column of a csv file, and output Theo> the duplicate line _and_ the original line that matched it. There’s a Theo> million examples out there that will output just the duplicate but not Theo> both. Theo> Theo> In the data below, I’m looking for lines that match in the 3^rd column… Theo> Theo> Normal,Server,xldspntc02,,10.33.52.185, Theo> Normal,Server,xldspntc02,,10.33.52.186, Theo> Normal,Server,xldspntc04,,10.33.52.187, Theo> Normal,Server,xldspntcs01,10.33.16.198, Theo> Normal,Server,xldspntcs01,,10.33.16.199, Theo> Normal,Server,xldsps01,10.33.16.162, Theo> Normal,Server,xldsps02,10.33.16.163, Theo> Theo> My desired output would be: Theo> Theo> Normal,Server,xldspntc02,,10.33.52.185, Theo> Normal,Server,xldspntc02,,10.33.52.186, Theo> Normal,Server,xldspntcs01,10.33.16.198, Theo> Normal,Server,xldspntcs01,,10.33.16.199, Theo> Theo> $ awk -F, 'dup[$3]++' file.csv Theo> Theo> I played around with the prev variable, but could not pumb it out Theo> fully, e.g { print prev } Theo> Theo> Mike Theo> _______________________________________________ Theo> Wlug mailing list Theo> Wlug(a)mail.wlug.org Theo> http://mail.wlug.org/mailman/listinfo/wlug Theo> _______________________________________________ Theo> Wlug mailing list Theo> Wlug(a)mail.wlug.org Theo> http://mail.wlug.org/mailman/listinfo/wlug

1 0

Re: [Wlug] awk help
by Theo Van Dinter March 15, 2017

March 15, 2017

Assuming there's no spaces in the csv, you could do: awk -F, '{print $0,$3}' | sort -k2 | uniq -D -f 1 | awk '{print $1}' eg: $ cat 1 Normal,Server,xldspntcs01,10.33.16.198, Normal,Server,xldspntc02,,10.33.52.185, Normal,Server,xldsps01,10.33.16.162, Normal,Server,xldspntc04,,10.33.52.187, Normal,Server,xldspntcs01,,10.33.16.199, Normal,Server,xldsps02,10.33.16.163, Normal,Server,xldspntc02,,10.33.52.186, $ cat 1 | awk -F, '{print $0,$3}' | sort -k2 | uniq -D -f 1 | awk '{print $1}' Normal,Server,xldspntc02,,10.33.52.185, Normal,Server,xldspntc02,,10.33.52.186, Normal,Server,xldspntcs01,,10.33.16.199, Normal,Server,xldspntcs01,10.33.16.198, On Mon, Mar 13, 2017 at 7:20 PM, Mike Peckar <fog(a)fognet.com> wrote: > Hey now, folks, > > > > This seemed like it should be simple, but I’m at wits end. I simply want > to find duplicates in the third column of a csv file, and output the > duplicate line _*and*_ the original line that matched it. There’s a > million examples out there that will output just the duplicate but not both. > > > > In the data below, I’m looking for lines that match in the 3rd column… > > > > Normal,Server,xldspntc02,,10.33.52.185, > > Normal,Server,xldspntc02,,10.33.52.186, > > Normal,Server,xldspntc04,,10.33.52.187, > > Normal,Server,xldspntcs01,10.33.16.198, > > Normal,Server,xldspntcs01,,10.33.16.199, > > Normal,Server,xldsps01,10.33.16.162, > > Normal,Server,xldsps02,10.33.16.163, > > > > My desired output would be: > > > > Normal,Server,xldspntc02,,10.33.52.185, > > Normal,Server,xldspntc02,,10.33.52.186, > > Normal,Server,xldspntcs01,10.33.16.198, > > Normal,Server,xldspntcs01,,10.33.16.199, > > > > $ awk -F, 'dup[$3]++' file.csv > > > > I played around with the prev variable, but could not pumb it out fully, > e.g { print prev } > > > > Mike > > > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org > http://mail.wlug.org/mailman/listinfo/wlug > >

1 0

Re: [Wlug] awk help
by John Stoffel March 14, 2017

March 14, 2017

>>>>> "Mike" == Mike Peckar <fog(a)fognet.com> writes: Mike> This seemed like it should be simple, but I’m at wits end. I Mike> simply want to find duplicates in the third column of a csv Mike> file, and output the duplicate line _and_ the original line Mike> that matched it. There’s a million examples out there that Mike> will output just the duplicate but not both. Mike> In the data below, I’m looking for lines that match in the 3^rd column… The sorting part is easy... sort -k 3 -t "," <file> Now to find the duplicates... I'd probably jump to a perl script: perl -e '@a=(<>);foreach @a {@t=split(",",$_) {push @{$t{$t[2]}},$_; } foreach sort (keys %t) { if ($#{$t{$_}} > 0) { print @{$t{$_}}, "\n";}}' Should also do the right thing. First it splits into keys, stuffs the line into an assoc array of arrays. Then it sorts the assoc array and prints out those with more than one entry in it. Admittedly done off the top of my head, without any actual testing. :-) Mike> Normal,Server,xldspntc02,,10.33.52.185, Mike> Normal,Server,xldspntc02,,10.33.52.186, Mike> Normal,Server,xldspntc04,,10.33.52.187, Mike> Normal,Server,xldspntcs01,10.33.16.198, Mike> Normal,Server,xldspntcs01,,10.33.16.199, Mike> Normal,Server,xldsps01,10.33.16.162, Mike> Normal,Server,xldsps02,10.33.16.163, Mike> My desired output would be: Mike> Normal,Server,xldspntc02,,10.33.52.185, Mike> Normal,Server,xldspntc02,,10.33.52.186, Mike> Normal,Server,xldspntcs01,10.33.16.198, Mike> Normal,Server,xldspntcs01,,10.33.16.199, Mike> $ awk -F, 'dup[$3]++' file.csv Mike> I played around with the prev variable, but could not pumb it out fully, Mike> e.g { print prev } Mike> Mike Mike> _______________________________________________ Mike> Wlug mailing list Mike> Wlug(a)mail.wlug.org Mike> http://mail.wlug.org/mailman/listinfo/wlug

1 0

Re: [Wlug] awk help
by Nathan W. Panike March 14, 2017

March 14, 2017

This seems to work: awk -F, '{linecount[$3]++;tmp=dup[$3];if(length(tmp)==0){dup[$3]=$0}else{dup[$3]=dup[$3]"\n"$0}} END {for (count in linecount){if(linecount[count]>1){print dup[count]}}}' Hopefully this is not a homework problem. On Mon, Mar 13, 2017 at 7:20 PM, Mike Peckar <fog(a)fognet.com> wrote: > Hey now, folks, > > > > This seemed like it should be simple, but I’m at wits end. I simply want > to find duplicates in the third column of a csv file, and output the > duplicate line _*and*_ the original line that matched it. There’s a > million examples out there that will output just the duplicate but not both. > > > > In the data below, I’m looking for lines that match in the 3rd column… > > > > Normal,Server,xldspntc02,,10.33.52.185, > > Normal,Server,xldspntc02,,10.33.52.186, > > Normal,Server,xldspntc04,,10.33.52.187, > > Normal,Server,xldspntcs01,10.33.16.198, > > Normal,Server,xldspntcs01,,10.33.16.199, > > Normal,Server,xldsps01,10.33.16.162, > > Normal,Server,xldsps02,10.33.16.163, > > > > My desired output would be: > > > > Normal,Server,xldspntc02,,10.33.52.185, > > Normal,Server,xldspntc02,,10.33.52.186, > > Normal,Server,xldspntcs01,10.33.16.198, > > Normal,Server,xldspntcs01,,10.33.16.199, > > > > $ awk -F, 'dup[$3]++' file.csv > > > > I played around with the prev variable, but could not pumb it out fully, > e.g { print prev } > > > > Mike > > > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org > http://mail.wlug.org/mailman/listinfo/wlug > > -- Nathan Panike

1 0

awk help
by Mike Peckar March 13, 2017

March 13, 2017

Hey now, folks, This seemed like it should be simple, but I’m at wits end. I simply want to find duplicates in the third column of a csv file, and output the duplicate line _and_ the original line that matched it. There’s a million examples out there that will output just the duplicate but not both. In the data below, I’m looking for lines that match in the 3rd column… Normal,Server,xldspntc02,,10.33.52.185, Normal,Server,xldspntc02,,10.33.52.186, Normal,Server,xldspntc04,,10.33.52.187, Normal,Server,xldspntcs01,10.33.16.198, Normal,Server,xldspntcs01,,10.33.16.199, Normal,Server,xldsps01,10.33.16.162, Normal,Server,xldsps02,10.33.16.163, My desired output would be: Normal,Server,xldspntc02,,10.33.52.185, Normal,Server,xldspntc02,,10.33.52.186, Normal,Server,xldspntcs01,10.33.16.198, Normal,Server,xldspntcs01,,10.33.16.199, $ awk -F, 'dup[$3]++' file.csv I played around with the prev variable, but could not pumb it out fully, e.g { print prev } Mike

1 0

Re: [Wlug] (for meeting projectors) laptops,VGA-out,linux....
by Chuck Anderson March 10, 2017

March 10, 2017

On Fri, Mar 10, 2017 at 10:37:53AM -0500, Tim Keller wrote: > Thanks for doing some great digging on the projector stuff.. It's only > 2017... Projectors should just *work*. They do "just work" in some OSes/desktop environments.

1 0

Re: [Wlug] (for meeting projectors) laptops,VGA-out,linux....
by Tim Keller March 10, 2017

March 10, 2017

Playing with Cinnamon last night I'm going to take the dive and throw it on a RHEL7 VM and see if I like it better than Mate. If so, I'll swap over my desktop and see... Thanks for doing some great digging on the projector stuff.. It's only 2017... Projectors should just *work*. On Fri, Mar 10, 2017 at 9:12 AM, Doug Mildram <dmildram(a)gmail.com> wrote: > doug> Thanks Tim for bringing a functional laptop w/linux for last night's > meeting. Our VGA-out projector worked fine as he installed a linux Mint VM > (Mate Cinnamon) onto it. > > When that finished, I wanted to show/compare Mint Mate, which I had > already on my laptop. > > When I unplugged Tim's laptop from VGA-out and tried my own thinkpad > running Mint Mate (dualbooted, my alternate OS).....well,.....chirp. > > The projector stopped projecting. My Fn-F7 key seemed to be trying to > cycle thru the blind toggles/choices (Duplicate display? screen-only? > VGA-only might expect), but no video came out, I had to give up, not having > tried VGA-out ever while running linux on my laptop. We put Tim's back > onto VGA-out, and it also seemed dead, but we chose not to fight it, > meeting being close enough to stop caring about projector, and it "wasnt a > proud moment" for me. > > SO I WENT HOME and plugged my (same) laptop into my TV's VGA-in port. > Started by booting win7. I'd never noticed a detail : at WHICH POINT does > this start working....should I expect to see the dualboot/GRUB menu on TV? > Apparently not. Soon enough though, Win7 (the OS preinstalled by > Lenovo/thinkpad) came up and VGA-out/TV started working as usual. > (somewhere AFTER "Resuming Windows", it starts working). > To control VGA-out, Fn-F7 works ...I've got (4) choices: Duplicate, > Extend(laptop-screen=#1, TV-VGAout=screen2), Projector Only (turns off > laptop display), and Disconnect Projector (only laptop display ON). No > doubt (i surmise), thanks to windows drivers. > > Feels like I learned: OS controls VGA-out (somehow naively I was inclined > to believe this was more of an OS-independent hardware function). > > OK, now rebooted laptop/win7 to try Linux Mint from a win7 state with > "Duplicate" working. "Shutting down Windows" displayed....laptop goes thru > boot/BIOS?...TV is blank now, I've learned to expect this....Linux kernel > boots, and I see some of the minimal console text messages as usual ALSO ON > TV, laptop-screen AND TV quickly go to graphical mode, also displaying the > Mate background... > > and this is where I was the most surprised. > > TV/VGA-out appeared "frozen' with the Mate background, appearing much like > my laptop screen BUT MINUS anything else (no bottom launch/status/"panel" > bar. Whatever I launch on my laptop screen, does not appear on VGA-out. > Fascinating! > > Wed eve late, at my house, I could swear Fn-F7 still wasnt doing anything. > But this Thurs AM, it DOES start working a bit (same 4 choices, but blind, > and it's strange. TV/VGA-out doesnt show EVERYTHING). > I wonder if I'd rebooted at the meeting, would that have helped, probably. > Today I play with it some more (ALT-F2 helps when I'm in that semi-blind > VGA-out situation, getting SOME things to show on TV) even though I'm not > likely to use this feature much?ever? at home, unless I break a leg. > > Caveat, practice what you bring for meetings to project-OUT! > HDMI is also an option in our WPI meeting/conf rms if you have it. > > > ---trivia for Mate interest... > (Mate = lightweight alternate to Cinnamon. > Their start/launch menus are different, and I can find good stuff in > Mate's Control Center. Cinnamon's Control center looks like this: > https://docs.google.com/file/d/0B2V313X0LzmQclJCVkI5aFhIeDA/edit > > Couldnt find a good Mate comparison Control Center screenshot > (not sharing some high-ad google'd choices) > but I did like this decent page mostly cuz i love keyb shortcuts: > > http://securitronlinux.com/bejiitaswrath/some-useful- > tips-for-the-linux-mate-desktop/ > where I learned > CTL-ALT-d (lowercase "d" not DEL, which offers to reboot) > .....lowers-or-raises what you're doing, a great "quick hide from > over-shoulder" ninja move. (sorry to degrade tricks/skill(s) of a Real > Ninja) > > Stay warm! --doug > > > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org > http://mail.wlug.org/mailman/listinfo/wlug > > -- I am leery of the allegiances of any politician who refers to their constituents as "consumers".

1 0

(for meeting projectors) laptops,VGA-out,linux....
by Doug Mildram March 10, 2017

March 10, 2017

doug> Thanks Tim for bringing a functional laptop w/linux for last night's meeting. Our VGA-out projector worked fine as he installed a linux Mint VM (Mate Cinnamon) onto it. When that finished, I wanted to show/compare Mint Mate, which I had already on my laptop. When I unplugged Tim's laptop from VGA-out and tried my own thinkpad running Mint Mate (dualbooted, my alternate OS).....well,.....chirp. The projector stopped projecting. My Fn-F7 key seemed to be trying to cycle thru the blind toggles/choices (Duplicate display? screen-only? VGA-only might expect), but no video came out, I had to give up, not having tried VGA-out ever while running linux on my laptop. We put Tim's back onto VGA-out, and it also seemed dead, but we chose not to fight it, meeting being close enough to stop caring about projector, and it "wasnt a proud moment" for me. SO I WENT HOME and plugged my (same) laptop into my TV's VGA-in port. Started by booting win7. I'd never noticed a detail : at WHICH POINT does this start working....should I expect to see the dualboot/GRUB menu on TV? Apparently not. Soon enough though, Win7 (the OS preinstalled by Lenovo/thinkpad) came up and VGA-out/TV started working as usual. (somewhere AFTER "Resuming Windows", it starts working). To control VGA-out, Fn-F7 works ...I've got (4) choices: Duplicate, Extend(laptop-screen=#1, TV-VGAout=screen2), Projector Only (turns off laptop display), and Disconnect Projector (only laptop display ON). No doubt (i surmise), thanks to windows drivers. Feels like I learned: OS controls VGA-out (somehow naively I was inclined to believe this was more of an OS-independent hardware function). OK, now rebooted laptop/win7 to try Linux Mint from a win7 state with "Duplicate" working. "Shutting down Windows" displayed....laptop goes thru boot/BIOS?...TV is blank now, I've learned to expect this....Linux kernel boots, and I see some of the minimal console text messages as usual ALSO ON TV, laptop-screen AND TV quickly go to graphical mode, also displaying the Mate background... and this is where I was the most surprised. TV/VGA-out appeared "frozen' with the Mate background, appearing much like my laptop screen BUT MINUS anything else (no bottom launch/status/"panel" bar. Whatever I launch on my laptop screen, does not appear on VGA-out. Fascinating! Wed eve late, at my house, I could swear Fn-F7 still wasnt doing anything. But this Thurs AM, it DOES start working a bit (same 4 choices, but blind, and it's strange. TV/VGA-out doesnt show EVERYTHING). I wonder if I'd rebooted at the meeting, would that have helped, probably. Today I play with it some more (ALT-F2 helps when I'm in that semi-blind VGA-out situation, getting SOME things to show on TV) even though I'm not likely to use this feature much?ever? at home, unless I break a leg. Caveat, practice what you bring for meetings to project-OUT! HDMI is also an option in our WPI meeting/conf rms if you have it. ---trivia for Mate interest... (Mate = lightweight alternate to Cinnamon. Their start/launch menus are different, and I can find good stuff in Mate's Control Center. Cinnamon's Control center looks like this: https://docs.google.com/file/d/0B2V313X0LzmQclJCVkI5aFhIeDA/edit Couldnt find a good Mate comparison Control Center screenshot (not sharing some high-ad google'd choices) but I did like this decent page mostly cuz i love keyb shortcuts: http://securitronlinux.com/bejiitaswrath/some-useful-tips-for-the-linux-mat… where I learned CTL-ALT-d (lowercase "d" not DEL, which offers to reboot) .....lowers-or-raises what you're doing, a great "quick hide from over-shoulder" ninja move. (sorry to degrade tricks/skill(s) of a Real Ninja) Stay warm! --doug

1 0

Reminder! Meeting TONIGHT March 9th, 2017! Topics: KVM, Virtualized windows apps, Linux Mint!
by Tim Keller March 9, 2017

March 9, 2017

Hey Gang, First the important details! Location: Rm 331 (Hagglund Room) @ 7PM Topic: read below... Tonight we've got a meeting! After the last meeting a couple of people were asking me questions about KVM (Kernel based Virtual Machine). New member Al Reinhart asked me a question about the best ways to run windows apps on linux, either via wine or through KVM / Oracle Virtual Box. He also mentioned Linux.Encoder (the first linux ransomware) which I didn't even know exists. Long time member Doug Mildram also wrote me about Linux Mint. A distribution I know quite well! They'll be some conversation and we'll kick off an install and talk about it's merits as a greate distribution. As usual, there'll be refreshments and snacks and afterwards we'll head to Tech for pizza! Later, Tim. -- I am leery of the allegiances of any politician who refers to their constituents as "consumers".

1 0