- WLUG - wlug.mailman3.com

Last-Minute Emergency, Unable to Attend

by Terry S

Hello, I had a phone call at 6:07 that required me to go out for a while, and did not have time to email. (I'm the one with the camera.) Hopefully, someone else just "happened" to have one, but I can take pictures at our next meeting if needed. How did the key-signing go? Terry

May 17, 2007

Re: [Wlug] large file performance on linux

by Jeff Moyer

==> On Wed, 16 May 2007 22:19:14 +0000, brad noyes <maitre(a)ccs.neu.edu> said: brad> All, brad> here's a summary of what i have tried so far. I have tried using the O_DIRECT brad> and O_SYNC flags with open(2) and i have been using a non-journalling brad> filesystem, ext2. None of these options have yielded any better performance at brad> the rates i'm looking for. brad> I have not tried AIO. AIO will allow you to drive your disk subsystem better than synchronous writes; I really think this would solve your problem. For an example of an AIO app, you can google for aio-stress.c. Or, send me some code and I'll help get you started. -Jeff

May 16, 2007

Re: [Wlug] large file performance on linux

by brad noyes

All, here's a summary of what i have tried so far. I have tried using the O_DIRECT and O_SYNC flags with open(2) and i have been using a non-journalling filesystem, ext2. None of these options have yielded any better performance at the rates i'm looking for. I have not tried AIO. I don't think i can use a non 32-bit kernel on the xeons. Correct me if i'm wrong (it's happened before). As a temporary solution i created a memory filesystem, by $> mount -t tmpfs tmpfs -o size=6G /data/memory The writes are fast however i need another process to progressively move data out of the memory filesystem since i don't have enough memory to hold all 22GB of data i need. For those interested i have attached a small program which will use different options to open and profile the fwrite calls. To compile simply do $> sh ./testwrite.c $> ./testwrite -h $> ./testwrite -l 16 file.out I just wanted to keep you all updated. Thanks for all your help. I'll keep trying various ideas. I wish i could make it to the meeting to buy you all pizza for your help. Perhaps next month. Thanks, -- Brad

May 16, 2007

Re: [Wlug] Meeting tomight

by Brains

freak4uxxx(a)gmail.com wrote: > I might be slightly delayed tonight, small family thing that needs to be taken care of. I shouldn't be late but I just wanted to let people know. > > -Eric I suspect a lot of people are going to be running a little late this evening, including me... Frank "Rain, sleet, or snow, the LUGman marches on..."

May 16, 2007

Meeting tomight

by freak4uxxx＠gmail.com

I might be slightly delayed tonight, small family thing that needs to be taken care of. I shouldn't be late but I just wanted to let people know. -Eric

May 16, 2007

Re: [Wlug] PGP Key Signing Party

by Mike

Sorry guys. I don't think I am going to be showing up to do the inclement weather. If you see anyone posing as me, well, then it is not me. I can always trade keys w/ peeps next time. Cheers, Mike <quote who="Andy Stewart"> > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Eric Martin wrote: >> Just a reminder that we are having a PGP Key Signing Party tomorrow for >> WLUG. If you're name isn't listed below, I do not have your info (and >> you should get it to me if you want to be on the list). I know Andy >> said noon tomorrow but I'll take stuff until 4pm. Don't forget your two >> forms of ID and I'll see you tomorrow. >> >> Eric Martin >> Gregory M. Avedissian >> Walt Sawyer >> Alexander Pennace >> Doug Mildram >> Keith Wright >> Alex Weiss >> Andrew M. Stewart >> James Philip Gray >> >> -Eric > > HI Eric, > > Thanks for sending out the meeting reminder! > > I'll see you all on Wednesday evening with my shiny new passport. :-) > > Later, > > Andy > > > - -- > Andy Stewart, Founder > Worcester Linux Users' Group (http://www.wlug.org) > Chelmsford Linux Meetup Group (http://linux.meetup.com/393) > Amateur Radio: KB1OIQ > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.2 (GNU/Linux) > Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org > > iD8DBQFGSkvGHl0iXDssISsRApWmAJ9HpIERvKHryJXnIeBo8ZfoJnqBygCeIF2I > tQxc9xAb6JDHR8dbSTc3yNA= > =v7Xx > -----END PGP SIGNATURE----- > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org > http://mail.wlug.org/mailman/listinfo/wlug > -- They who would give up an essential liberty for temporary security, deserve neither liberty or security --Ben Franklin

May 16, 2007

Re: [Wlug] large file performance on linux

by Andre Lehovich

--- brad noyes <maitre(a)ccs.neu.edu> wrote: > Do you have any suggestions on how to speed up performance > on these writes, filesystem options, kernel options, other strategies, etc? How are you doing the writes, using fwrite(3), write(2), or mmap(2)? I've seen dramatic speedups reading large files using memory-mapped I/O. > - I have also tried spooling over several files (a la multiple volumes) > but i see no difference in performance. In fact, i think this actually > hinders performance a bit. Do you mean multiple physical spindles or drives here? (I'd expect a slow down using multiple partitions on the same physical drive.) > - I keep my own giant memory buffer where all the data is stored and > then it is written to disk in a background thread. This helps, but > i run out of space in the buffer before i finish taking data. With memory-mapped I/O you could just maintain the buffer and have the kernel page it out to disk for you as needed; no need to shuffle stuff around with a background thread. --Andre ____________________________________________________________________________________Building a website is a piece of cake. Yahoo! Small Business gives you all the tools to get online. http://smallbusiness.yahoo.com/webhosting

May 16, 2007

Re: [Wlug] large file performance on linux

by Jeff Moyer

==> On Wed, 16 May 2007 19:19:45 +0000, brad noyes <maitre(a)ccs.neu.edu> said: brad> On Wed, May 16, 2007 at 02:40:38PM -0400, Jeff Moyer wrote: brad> > ==> On Wed, 16 May 2007 12:59:54 -0400, Jeff Moyer <jmoyer(a)redhat.com> said: brad> > brad> > Jeff> Look under /proc/sys/vm. Documentation for these variables might be brad> > Jeff> in Documentation/filesystems/proc.txt (it's not always up-to-date). brad> > Jeff> But, as I said, I don't think this is the right avenue to explore. brad> > Jeff> You can get more predictable results by using AIO+O_DIRECT (or maybe brad> > Jeff> even O_SYNC as another mentioned). brad> > brad> > One other thing worth mentioning is that you should be doing I/O in brad> > large block sizes. What size are you currently using for your write brad> > buffers? brad> > brad> i'm writing in 16777216 byte chunks. That happens to be evenly divisible by 512 brad> for the O_DIRECT flag. However every time i try to use that flag the file gets brad> created, but nothing gets written. I've been looking online for an example. Are you aligning your buffers on a 512 byte boundary? Man posix_memalign. brad> I don't know if this means anything, but i ran brad> hdparm -T /dev/sdb1 brad> Timing cached reads: 1369MB in 2.00 seconds == 698.14MB/sec brad> hdparm -T --direct /dev/sdb1 brad> Timing O_DIRECT cached reads: 136 in 2.00 seconds == 66.54MB/sec brad> It really seems that 53MB/s shouldn't be hard. I have fairly heavy hardware, brad> scsi320 in a raid1 configuration. So long as nothing else is going on at the time. =) -Jeff

May 16, 2007

Re: [Wlug] large file performance on linux

by Karl Hiramoto

brad noyes wrote: > On Wed, May 16, 2007 at 02:40:38PM -0400, Jeff Moyer wrote: > >> ==> On Wed, 16 May 2007 12:59:54 -0400, Jeff Moyer <jmoyer(a)redhat.com> said: >> >> Jeff> Look under /proc/sys/vm. Documentation for these variables might be >> Jeff> in Documentation/filesystems/proc.txt (it's not always up-to-date). >> Jeff> But, as I said, I don't think this is the right avenue to explore. >> Jeff> You can get more predictable results by using AIO+O_DIRECT (or maybe >> Jeff> even O_SYNC as another mentioned). >> >> One other thing worth mentioning is that you should be doing I/O in >> large block sizes. What size are you currently using for your write >> buffers? >> >> > i'm writing in 16777216 byte chunks. That happens to be evenly divisible by 512 > for the O_DIRECT flag. However every time i try to use that flag the file gets > created, but nothing gets written. I've been looking online for an example. > > I don't know if this means anything, but i ran > hdparm -T /dev/sdb1 > Timing cached reads: 1369MB in 2.00 seconds == 698.14MB/sec > hdparm -T --direct /dev/sdb1 > Timing O_DIRECT cached reads: 136 in 2.00 seconds == 66.54MB/sec > > > It really seems that 53MB/s shouldn't be hard. I have fairly heavy hardware, > scsi320 in a raid1 configuration. > > -- Brad > _______________________________________________ > That is CACHED read.. your uncached write will be slower.. run a bonnie++ benchmark on your disks. You might want to switch to raid 0 or raid 1+0. And use a 64 bit kernel to be able to allocate more than 3G of ram. -- -- Karl Hiramoto http://karl.hiramoto.org/ US VOIP: (1) 603.966.4448 Spain Casa (34) 951.273.347 Spain Mobil (34) 617.463.826 Yahoo_IM = karl_hiramoto GTalk= karl.hiramoto [at] gmail [d0t] com -- Everyone complains of his memory, no one of his judgement.

May 16, 2007

Re: [Wlug] large file performance on linux

by brad noyes

On Wed, May 16, 2007 at 03:19:17PM -0400, John Stoffel wrote: > >>>>> "brad" == brad noyes <maitre(a)ccs.neu.edu> writes: > > brad> I am seeing some really slow performance regarding large files > brad> on linux. I write a lot of data points from a light sensor. The > brad> stream is about 53 Mb/s and i need to keep this rate for 7 > brad> minutes, that's a total of about 22Gb. I can sustain 53Mb/s > brad> pretty well until the file grows to over 1Gb or so, then things > brad> hit the wall and the writes to the filesystem can't keep up. The > brad> writes go from 20ms in duration to 500ms. I assume the > brad> filesystem/operating system is caching writes. Do you have any > brad> suggestions on how to speed up performance on these writes, > brad> filesystem options, kernel options, other strategies, etc? > > You've already had a good bunch of suggestions, but I've got some > questions on your hardware. > i don't mind answering more questions :). > - cpu? dual xeons (not sure if they are hyper threaded or dual core) cpu MHz: 3067.044 cache size: 512 KB > - memory - 12gb I know > - disk(s) SCSI hard drives, i believe they are SCSI 320. I have tried in a raid1 as well as stand alone. > - RAID setup at all? tried using RAID1. I'm afraid to try RAID0 b/c the data is pretty vital, but i may try it. > > One way to get more performance would be to add another disk or two > and to stripe your data between them. This assumes you have enough > PCI bus bandwidth available as well. You don't say how you're > capturing the light sensor data, but it's obviosly not over a serial > port or some other slow device. Network? So if you've got 53 > Mbyte/second comming into the system, and another 53Mbytes/second > writing out to disk, then you're starting to get close to the > 132Mbytes/sec bandwidth of the PCI bus. > Good point. Its not network, but it is still on the PCI bus. It doesn't seem that bandwidth is really a problem b/c it works great for the first minute. Once the file grows past 1GB the writes are extremely slow. Like i said above i might try the RAID0 configuration. > Finding a motherboard with two or more PCI busses would help. Or > something with PCI-E busses. It all depends on your budget and the > data acquisition tool you're using. > If i were to redesign this system i would really like to use PXI, which was really meant for this sort of thing. I'm really retrofitting my design to to something different than intended. Thanks -- Brad

May 16, 2007