May 2007 - WLUG - wlug.mailman3.com

Re: [Wlug] large file performance on linux

by brad noyes

All, here's a summary of what i have tried so far. I have tried using the O_DIRECT and O_SYNC flags with open(2) and i have been using a non-journalling filesystem, ext2. None of these options have yielded any better performance at the rates i'm looking for. I have not tried AIO. I don't think i can use a non 32-bit kernel on the xeons. Correct me if i'm wrong (it's happened before). As a temporary solution i created a memory filesystem, by $> mount -t tmpfs tmpfs -o size=6G /data/memory The writes are fast however i need another process to progressively move data out of the memory filesystem since i don't have enough memory to hold all 22GB of data i need. For those interested i have attached a small program which will use different options to open and profile the fwrite calls. To compile simply do $> sh ./testwrite.c $> ./testwrite -h $> ./testwrite -l 16 file.out I just wanted to keep you all updated. Thanks for all your help. I'll keep trying various ideas. I wish i could make it to the meeting to buy you all pizza for your help. Perhaps next month. Thanks, -- Brad

May 16, 2007

Re: [Wlug] Meeting tomight

by Brains

freak4uxxx(a)gmail.com wrote: > I might be slightly delayed tonight, small family thing that needs to be taken care of. I shouldn't be late but I just wanted to let people know. > > -Eric I suspect a lot of people are going to be running a little late this evening, including me... Frank "Rain, sleet, or snow, the LUGman marches on..."

May 16, 2007

Meeting tomight

by freak4uxxx＠gmail.com

I might be slightly delayed tonight, small family thing that needs to be taken care of. I shouldn't be late but I just wanted to let people know. -Eric

May 16, 2007

Re: [Wlug] PGP Key Signing Party

by Mike

Sorry guys. I don't think I am going to be showing up to do the inclement weather. If you see anyone posing as me, well, then it is not me. I can always trade keys w/ peeps next time. Cheers, Mike <quote who="Andy Stewart"> > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Eric Martin wrote: >> Just a reminder that we are having a PGP Key Signing Party tomorrow for >> WLUG. If you're name isn't listed below, I do not have your info (and >> you should get it to me if you want to be on the list). I know Andy >> said noon tomorrow but I'll take stuff until 4pm. Don't forget your two >> forms of ID and I'll see you tomorrow. >> >> Eric Martin >> Gregory M. Avedissian >> Walt Sawyer >> Alexander Pennace >> Doug Mildram >> Keith Wright >> Alex Weiss >> Andrew M. Stewart >> James Philip Gray >> >> -Eric > > HI Eric, > > Thanks for sending out the meeting reminder! > > I'll see you all on Wednesday evening with my shiny new passport. :-) > > Later, > > Andy > > > - -- > Andy Stewart, Founder > Worcester Linux Users' Group (http://www.wlug.org) > Chelmsford Linux Meetup Group (http://linux.meetup.com/393) > Amateur Radio: KB1OIQ > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.2 (GNU/Linux) > Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org > > iD8DBQFGSkvGHl0iXDssISsRApWmAJ9HpIERvKHryJXnIeBo8ZfoJnqBygCeIF2I > tQxc9xAb6JDHR8dbSTc3yNA= > =v7Xx > -----END PGP SIGNATURE----- > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org > http://mail.wlug.org/mailman/listinfo/wlug > -- They who would give up an essential liberty for temporary security, deserve neither liberty or security --Ben Franklin

May 16, 2007

Re: [Wlug] large file performance on linux

by Andre Lehovich

--- brad noyes <maitre(a)ccs.neu.edu> wrote: > Do you have any suggestions on how to speed up performance > on these writes, filesystem options, kernel options, other strategies, etc? How are you doing the writes, using fwrite(3), write(2), or mmap(2)? I've seen dramatic speedups reading large files using memory-mapped I/O. > - I have also tried spooling over several files (a la multiple volumes) > but i see no difference in performance. In fact, i think this actually > hinders performance a bit. Do you mean multiple physical spindles or drives here? (I'd expect a slow down using multiple partitions on the same physical drive.) > - I keep my own giant memory buffer where all the data is stored and > then it is written to disk in a background thread. This helps, but > i run out of space in the buffer before i finish taking data. With memory-mapped I/O you could just maintain the buffer and have the kernel page it out to disk for you as needed; no need to shuffle stuff around with a background thread. --Andre ____________________________________________________________________________________Building a website is a piece of cake. Yahoo! Small Business gives you all the tools to get online. http://smallbusiness.yahoo.com/webhosting

May 16, 2007

Re: [Wlug] large file performance on linux

by Jeff Moyer

==> On Wed, 16 May 2007 19:19:45 +0000, brad noyes <maitre(a)ccs.neu.edu> said: brad> On Wed, May 16, 2007 at 02:40:38PM -0400, Jeff Moyer wrote: brad> > ==> On Wed, 16 May 2007 12:59:54 -0400, Jeff Moyer <jmoyer(a)redhat.com> said: brad> > brad> > Jeff> Look under /proc/sys/vm. Documentation for these variables might be brad> > Jeff> in Documentation/filesystems/proc.txt (it's not always up-to-date). brad> > Jeff> But, as I said, I don't think this is the right avenue to explore. brad> > Jeff> You can get more predictable results by using AIO+O_DIRECT (or maybe brad> > Jeff> even O_SYNC as another mentioned). brad> > brad> > One other thing worth mentioning is that you should be doing I/O in brad> > large block sizes. What size are you currently using for your write brad> > buffers? brad> > brad> i'm writing in 16777216 byte chunks. That happens to be evenly divisible by 512 brad> for the O_DIRECT flag. However every time i try to use that flag the file gets brad> created, but nothing gets written. I've been looking online for an example. Are you aligning your buffers on a 512 byte boundary? Man posix_memalign. brad> I don't know if this means anything, but i ran brad> hdparm -T /dev/sdb1 brad> Timing cached reads: 1369MB in 2.00 seconds == 698.14MB/sec brad> hdparm -T --direct /dev/sdb1 brad> Timing O_DIRECT cached reads: 136 in 2.00 seconds == 66.54MB/sec brad> It really seems that 53MB/s shouldn't be hard. I have fairly heavy hardware, brad> scsi320 in a raid1 configuration. So long as nothing else is going on at the time. =) -Jeff

May 16, 2007

Re: [Wlug] large file performance on linux

by Karl Hiramoto

brad noyes wrote: > On Wed, May 16, 2007 at 02:40:38PM -0400, Jeff Moyer wrote: > >> ==> On Wed, 16 May 2007 12:59:54 -0400, Jeff Moyer <jmoyer(a)redhat.com> said: >> >> Jeff> Look under /proc/sys/vm. Documentation for these variables might be >> Jeff> in Documentation/filesystems/proc.txt (it's not always up-to-date). >> Jeff> But, as I said, I don't think this is the right avenue to explore. >> Jeff> You can get more predictable results by using AIO+O_DIRECT (or maybe >> Jeff> even O_SYNC as another mentioned). >> >> One other thing worth mentioning is that you should be doing I/O in >> large block sizes. What size are you currently using for your write >> buffers? >> >> > i'm writing in 16777216 byte chunks. That happens to be evenly divisible by 512 > for the O_DIRECT flag. However every time i try to use that flag the file gets > created, but nothing gets written. I've been looking online for an example. > > I don't know if this means anything, but i ran > hdparm -T /dev/sdb1 > Timing cached reads: 1369MB in 2.00 seconds == 698.14MB/sec > hdparm -T --direct /dev/sdb1 > Timing O_DIRECT cached reads: 136 in 2.00 seconds == 66.54MB/sec > > > It really seems that 53MB/s shouldn't be hard. I have fairly heavy hardware, > scsi320 in a raid1 configuration. > > -- Brad > _______________________________________________ > That is CACHED read.. your uncached write will be slower.. run a bonnie++ benchmark on your disks. You might want to switch to raid 0 or raid 1+0. And use a 64 bit kernel to be able to allocate more than 3G of ram. -- -- Karl Hiramoto http://karl.hiramoto.org/ US VOIP: (1) 603.966.4448 Spain Casa (34) 951.273.347 Spain Mobil (34) 617.463.826 Yahoo_IM = karl_hiramoto GTalk= karl.hiramoto [at] gmail [d0t] com -- Everyone complains of his memory, no one of his judgement.

May 16, 2007

Re: [Wlug] large file performance on linux

by brad noyes

On Wed, May 16, 2007 at 03:19:17PM -0400, John Stoffel wrote: > >>>>> "brad" == brad noyes <maitre(a)ccs.neu.edu> writes: > > brad> I am seeing some really slow performance regarding large files > brad> on linux. I write a lot of data points from a light sensor. The > brad> stream is about 53 Mb/s and i need to keep this rate for 7 > brad> minutes, that's a total of about 22Gb. I can sustain 53Mb/s > brad> pretty well until the file grows to over 1Gb or so, then things > brad> hit the wall and the writes to the filesystem can't keep up. The > brad> writes go from 20ms in duration to 500ms. I assume the > brad> filesystem/operating system is caching writes. Do you have any > brad> suggestions on how to speed up performance on these writes, > brad> filesystem options, kernel options, other strategies, etc? > > You've already had a good bunch of suggestions, but I've got some > questions on your hardware. > i don't mind answering more questions :). > - cpu? dual xeons (not sure if they are hyper threaded or dual core) cpu MHz: 3067.044 cache size: 512 KB > - memory - 12gb I know > - disk(s) SCSI hard drives, i believe they are SCSI 320. I have tried in a raid1 as well as stand alone. > - RAID setup at all? tried using RAID1. I'm afraid to try RAID0 b/c the data is pretty vital, but i may try it. > > One way to get more performance would be to add another disk or two > and to stripe your data between them. This assumes you have enough > PCI bus bandwidth available as well. You don't say how you're > capturing the light sensor data, but it's obviosly not over a serial > port or some other slow device. Network? So if you've got 53 > Mbyte/second comming into the system, and another 53Mbytes/second > writing out to disk, then you're starting to get close to the > 132Mbytes/sec bandwidth of the PCI bus. > Good point. Its not network, but it is still on the PCI bus. It doesn't seem that bandwidth is really a problem b/c it works great for the first minute. Once the file grows past 1GB the writes are extremely slow. Like i said above i might try the RAID0 configuration. > Finding a motherboard with two or more PCI busses would help. Or > something with PCI-E busses. It all depends on your budget and the > data acquisition tool you're using. > If i were to redesign this system i would really like to use PXI, which was really meant for this sort of thing. I'm really retrofitting my design to to something different than intended. Thanks -- Brad

May 16, 2007

Re: [Wlug] large file performance on linux

by brad noyes

On Wed, May 16, 2007 at 02:40:38PM -0400, Jeff Moyer wrote: > ==> On Wed, 16 May 2007 12:59:54 -0400, Jeff Moyer <jmoyer(a)redhat.com> said: > > Jeff> Look under /proc/sys/vm. Documentation for these variables might be > Jeff> in Documentation/filesystems/proc.txt (it's not always up-to-date). > Jeff> But, as I said, I don't think this is the right avenue to explore. > Jeff> You can get more predictable results by using AIO+O_DIRECT (or maybe > Jeff> even O_SYNC as another mentioned). > > One other thing worth mentioning is that you should be doing I/O in > large block sizes. What size are you currently using for your write > buffers? > i'm writing in 16777216 byte chunks. That happens to be evenly divisible by 512 for the O_DIRECT flag. However every time i try to use that flag the file gets created, but nothing gets written. I've been looking online for an example. I don't know if this means anything, but i ran hdparm -T /dev/sdb1 Timing cached reads: 1369MB in 2.00 seconds == 698.14MB/sec hdparm -T --direct /dev/sdb1 Timing O_DIRECT cached reads: 136 in 2.00 seconds == 66.54MB/sec It really seems that 53MB/s shouldn't be hard. I have fairly heavy hardware, scsi320 in a raid1 configuration. -- Brad

May 16, 2007

Re: [Wlug] large file performance on linux

by John Stoffel

>>>>> "brad" == brad noyes <maitre(a)ccs.neu.edu> writes: brad> I am seeing some really slow performance regarding large files brad> on linux. I write a lot of data points from a light sensor. The brad> stream is about 53 Mb/s and i need to keep this rate for 7 brad> minutes, that's a total of about 22Gb. I can sustain 53Mb/s brad> pretty well until the file grows to over 1Gb or so, then things brad> hit the wall and the writes to the filesystem can't keep up. The brad> writes go from 20ms in duration to 500ms. I assume the brad> filesystem/operating system is caching writes. Do you have any brad> suggestions on how to speed up performance on these writes, brad> filesystem options, kernel options, other strategies, etc? You've already had a good bunch of suggestions, but I've got some questions on your hardware. - cpu? - memory - 12gb I know - disk(s) - RAID setup at all? One way to get more performance would be to add another disk or two and to stripe your data between them. This assumes you have enough PCI bus bandwidth available as well. You don't say how you're capturing the light sensor data, but it's obviosly not over a serial port or some other slow device. Network? So if you've got 53 Mbyte/second comming into the system, and another 53Mbytes/second writing out to disk, then you're starting to get close to the 132Mbytes/sec bandwidth of the PCI bus. Finding a motherboard with two or more PCI busses would help. Or something with PCI-E busses. It all depends on your budget and the data acquisition tool you're using. John

May 16, 2007