July 2007 - WLUG - wlug.mailman3.com

Re: [Wlug] Any SQL-types out there? (more lab automation stuff)

by Chuck Anderson

On Wed, Jul 25, 2007 at 09:29:43AM -0400, Stephen Daukas wrote: > There are literally hundreds of files of data in CSV format, and I intend to > process each by itself, preserving the original, and creating a new > "adjusted" file for the next step. (The next step would be to concatenate > all files together for numeric processing.) Original files must be > preserved (chain of custody requirements), so I am opening the file, > selecting records using SQL, and am now ready to do the rearranging/writing > to a new file... So you are selecting records out of a CSV file using SQL? How about this: select columnB,columnG,columnA,columnC,columnF from foo; That will take the columns and return them in the given order.

July 25, 2007

Re: [Wlug] Any SQL-types out there? (more lab automation stuff)

by Michael Long

Stephen Daukas wrote: > There are a lot of things I'd rather use, but I have to stick with the > IT standards... > Most of my work has been in SQL Server shops where MS is the preferred technology. Having said that I would push for a Perl solution in this case because what you want to do does not lend itself to SQL easily. Perl will get the job done faster and cheaper. Not only that it is portable across platforms. You will have significantly less maintenance over the long term. -mike

July 25, 2007

Re: [Wlug] Any SQL-types out there? (more lab automation stuff)

by Stephen Daukas

There are a lot of things I'd rather use, but I have to stick with the IT standards... On 7/25/07, Michael Zarozinski <MichaelZ(a)aiguru.com> wrote: > > I think you'd be far better off using a scripting language such as perl, > it's far easier to manipulate text files with perl. > > Michael Z > > Stephen Daukas wrote: > > Good morning! > > > > In the continuing saga of automating various activities where I > > work, I have taken on processing data files. (Automating data upload > > is still somewhere in the air.)... > > > > This is rather simple in concept, but a bit less obvious in practice > > (aren't they all!): > > > > I would like to open up a CSV file and rearrange the columns according > > to a standardized order. Not every data file will have all of the > > standard columns (different probes collect different data), but column > > headers are standardized. So, I need to inspect each column header so > > that I can place it and its entire column of data into the > > proper position in the table, with empty columns inserted where needed > > to preserve the standard sequence of columns. In other words: > > > > A, D, C, G, H becomes A, [B], C, D, [E], [F], G, H > > where [ ] denote inserted empty columns > > > > There are literally hundreds of files of data in CSV format, and I > > intend to process each by itself, preserving the original, and > > creating a new "adjusted" file for the next step. (The next step > > would be to concatenate all files together for numeric processing.) > > Original files must be preserved (chain of custody requirements), so I > > am opening the file, selecting records using SQL, and am now ready to > > do the rearranging/writing to a new file... > > > > I didn't know if there is a "well known" SQL way of doing this (e.g., > > data manipulation statements of some sort), so I thought I'd see if > > any WLUGgers are also SQL-types. I have a few ideas for how to get > > this done, but they are "programmer-centric" algorithms, which tend to > > be different from "SQL-centric" algorithms... > > > > The reason for SQL is that I have to do this on a MS platform because > > that is the standard. But, if most of the real work is done in SQL, > > then I can easily move to another platform in the future and only have > > to change the "shell code". I am using VBscript as the "shell > > language" because it works with all MS apps and file system objects > > inclusive of CSV files. > > > > Thanks for any help!!! > > > > Best regards, > > Steve > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > Wlug mailing list > > Wlug(a)mail.wlug.org > > http://mail.wlug.org/mailman/listinfo/wlug > > > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org > http://mail.wlug.org/mailman/listinfo/wlug >

July 25, 2007

Re: [Wlug] Any SQL-types out there? (more lab automation stuff)

by Michael Zarozinski

I think you'd be far better off using a scripting language such as perl, it's far easier to manipulate text files with perl. Michael Z Stephen Daukas wrote: > Good morning! > > In the continuing saga of automating various activities where I > work, I have taken on processing data files. (Automating data upload > is still somewhere in the air.)... > > This is rather simple in concept, but a bit less obvious in practice > (aren't they all!): > > I would like to open up a CSV file and rearrange the columns according > to a standardized order. Not every data file will have all of the > standard columns (different probes collect different data), but column > headers are standardized. So, I need to inspect each column header so > that I can place it and its entire column of data into the > proper position in the table, with empty columns inserted where needed > to preserve the standard sequence of columns. In other words: > > A, D, C, G, H becomes A, [B], C, D, [E], [F], G, H > where [ ] denote inserted empty columns > > There are literally hundreds of files of data in CSV format, and I > intend to process each by itself, preserving the original, and > creating a new "adjusted" file for the next step. (The next step > would be to concatenate all files together for numeric processing.) > Original files must be preserved (chain of custody requirements), so I > am opening the file, selecting records using SQL, and am now ready to > do the rearranging/writing to a new file... > > I didn't know if there is a "well known" SQL way of doing this (e.g., > data manipulation statements of some sort), so I thought I'd see if > any WLUGgers are also SQL-types. I have a few ideas for how to get > this done, but they are "programmer-centric" algorithms, which tend to > be different from "SQL-centric" algorithms... > > The reason for SQL is that I have to do this on a MS platform because > that is the standard. But, if most of the real work is done in SQL, > then I can easily move to another platform in the future and only have > to change the "shell code". I am using VBscript as the "shell > language" because it works with all MS apps and file system objects > inclusive of CSV files. > > Thanks for any help!!! > > Best regards, > Steve > ------------------------------------------------------------------------ > > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org > http://mail.wlug.org/mailman/listinfo/wlug >

July 25, 2007

Any SQL-types out there? (more lab automation stuff)

by Stephen Daukas

Good morning! In the continuing saga of automating various activities where I work, I have taken on processing data files. (Automating data upload is still somewhere in the air.)... This is rather simple in concept, but a bit less obvious in practice (aren't they all!): I would like to open up a CSV file and rearrange the columns according to a standardized order. Not every data file will have all of the standard columns (different probes collect different data), but column headers are standardized. So, I need to inspect each column header so that I can place it and its entire column of data into the proper position in the table, with empty columns inserted where needed to preserve the standard sequence of columns. In other words: A, D, C, G, H becomes A, [B], C, D, [E], [F], G, H where [ ] denote inserted empty columns There are literally hundreds of files of data in CSV format, and I intend to process each by itself, preserving the original, and creating a new "adjusted" file for the next step. (The next step would be to concatenate all files together for numeric processing.) Original files must be preserved (chain of custody requirements), so I am opening the file, selecting records using SQL, and am now ready to do the rearranging/writing to a new file... I didn't know if there is a "well known" SQL way of doing this (e.g., data manipulation statements of some sort), so I thought I'd see if any WLUGgers are also SQL-types. I have a few ideas for how to get this done, but they are "programmer-centric" algorithms, which tend to be different from "SQL-centric" algorithms... The reason for SQL is that I have to do this on a MS platform because that is the standard. But, if most of the real work is done in SQL, then I can easily move to another platform in the future and only have to change the "shell code". I am using VBscript as the "shell language" because it works with all MS apps and file system objects inclusive of CSV files. Thanks for any help!!! Best regards, Steve

July 25, 2007

Re: [Wlug] WLUG BBQ !!!

by Gregory Avedissian

I'd like to go to the BBQ, but I need help with transportation. Could someone give me a ride? I live in Worcester. Thanks, Greg Andy Stewart wrote: > > HI everybody, > > The WLUG BBQ is coming up fast: > > Date: Thursday, July 26th, 2007 > > Time: 6:00 PM > > Place: Eric Martin's house > > Address: 33 Lansing Avenue, Worcester, MA 01605 > This is 2.5 miles from WPI. > Look for the penguin! > > WLUG will supply burgers, dogs, munchies, soda, and the like. Thanks, > Eric, for hosting the BBQ this year! > > See ya there! > > Andy > > _______________________________________________ Wlug mailing list Wlug(a)mail.wlug.org http://mail.wlug.org/mailman/listinfo/wlug

July 25, 2007

Thanks!

by Stephen Daukas

Hi all! Thanks for all of the posts on "project management software"! They were helpful! Regards, Steve

July 25, 2007

Re: [Wlug] Hard drive configuration

by Chuck Anderson

On Tue, Jul 24, 2007 at 05:36:16PM -0400, Michael Long wrote: > /dev/sdb3 / reiserfs acl,user_xattr 1 1 > /dev/sdb1 /boot reiserfs acl,user_xattr 1 2 > /dev/sda1 /raid reiserfs defaults 1 2 > /dev/sdb2 swap swap pri=42 0 0 > > I downloaded OpenSuSE 10.2 and attempted to upgrade the system. When > running either an upgrade or new install it tells me that /dev/sda3 is > defined as /dev/sdb3 in fstab. > > Bottom line when I install opensuse 10.2 it sees my raid as sdb and the > system is currently setup to have it as sda. I do not understand why my > current configuration is the opposite of what a new install is finding. > Where do I go from here? The order sda sdb sdc etc. depends on the order the SCSI buses were probed in. Different kernels may find hardware in a different order, or modules may load in a different order causing this. In fact, in Fedora 7+, they stopped using the old IDE driver and switched to the new ATA driver using libata. Now all /dev/hdX devices appear as /dev/sdX. The solution is to avoid using device names directly, and instead use filesystem labels, uuids, and/or LVM. e.g. this will label your devices (caveat: I haven't used reiserfs or SuSE before so you'll probably want to google this and read more about it): reiserfstune -l raid /dev/sda1 reiserfstune -l boot /dev/sdb1 reiserfstune -l root /dev/sdb3 (For Ext2/Ext3, you'd use e.g. "e2label /dev/sda1 raid") swapoff /dev/sdb2 mkswap -L swap /dev/sdb2 swapon -L swap Then you change your /etc/fstab like this: LABEL=root / reiserfs acl,user_xattr 1 1 LABEL=boot /boot reiserfs acl,user_xattr 1 2 LABEL=raid /raid reiserfs defaults 1 2 LABEL=swap swap swap pri=42 0 0 GRUB/Lilo will have to be dealt with separately. For example, my grub.conf (menu.lst) has this in it: title Fedora (2.6.21-1.3194.fc7) root (hd0,1) kernel /vmlinuz-2.6.21-1.3194.fc7 ro root=/dev/VGSystem/LVRoot rhgb quiet initrd /initrd-2.6.21-1.3194.fc7.img but I believe you can say "root=LABEL=root" instead. UUID's work similarly to labels.

July 25, 2007

Re: [Wlug] Hard drive configuration

by Robert L Krawitz

Date: Tue, 24 Jul 2007 17:36:16 -0400 From: Michael Long <mlong(a)datalong.com> I hope someone can point me in the right direction. I have a SuSE 9.3 system that was fully configured when I bought it. When looking at fstab the drives are configured as follows: /dev/sdb3 / reiserfs acl,user_xattr 1 1 /dev/sdb1 /boot reiserfs acl,user_xattr 1 2 /dev/sda1 /raid reiserfs defaults 1 2 /dev/sdb2 swap swap pri=42 0 0 I downloaded OpenSuSE 10.2 and attempted to upgrade the system. When running either an upgrade or new install it tells me that /dev/sda3 is defined as /dev/sdb3 in fstab. Bottom line when I install opensuse 10.2 it sees my raid as sdb and the system is currently setup to have it as sda. I do not understand why my current configuration is the opposite of what a new install is finding. Where do I go from here? Somewhere between 10.0 and 10.2 they changed the drive layout, so this may be correct.

July 25, 2007

Evolution plugin for MS Exchange

by chuck

Hi All, I'm running kubuntu feisty with evolution 2.10.1. I'm trying to use the exchange plugin to connect to the MS 2003 Exchange Server at work. I have all the info about the implementation of Exchange from the IT folks, but I just can't make the plugin work. When I enter in my password, I get "Can not authenticate. (password incorrect?)" error. For the "Identity" - Username is my name (not surprising...) - Email address is my fully qualified email address. For the "Receiving email" - Server type is Microsoft exchange - Username is my domainname\username (Note: I've tried it without the domainname, still no joy). - OWA URL is the webmail URL (this has been confirmed by the IT folks) - Authenticate - I enter my password - seems to be ok. - Authentication Type - I've tried both secure and plaintext For "Receiving options" - Global Address List/Active directory settings - There's no entry I've looked at the Ubuntu launchpad website and there seems to be a few issues with the plugin. When evolution starts, it tries to search for mail folders, but finds nothing. I can't even terminate the process. I have to find out what pid number it is and terminate it with the 'kill <pid>' command. Has anyone managed to get this working? Is there some "magic" tweaking that I need to do? TIA, -Chuck

July 24, 2007